[pve-devel] [PATCH storage 1/2] rbd: improve handling of missing images

[pve-devel] [PATCH storage 1/2] rbd: improve handling of missing images

[Aaron Lauterer]

It can happen, that an RBD image isn't cleaned up 100%. Calling 'rbd ls
-l' will then show errors that it is not possible to open the image in
question:
```
rbd: error opening vm-103-disk-1: (2) No such file or directory
rbd: listing images failed: (2) No such file or directory
```

Originally we only showed the last error line which is too generic and
doesn't give a good hint what is actually wrong.

We can improve that by catching these specific errors and add the
problematic disk imageto the returned list with a size of '-1'.
This way, normal operation won't be affected by errors and users can
still see that there is a broken image.

When the 'rbd rm' command is used on such an image, it will clean up
whatever is still left.
But for that to work, we also need to handle these errors in the
'rbd_ls_snap' sub as it is called from 'free_image'.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
---

Instead of parsing the error, we could have also ignored these specific
errors and run 'rbd ls' (without -l) a second time and compare the
outputs as the non -l variant will list the broken image as well.
But then we need to run a potentially expensive command twice.

Not sure if the way I parse the errors and handle the decision if we
have an error on which we should die ($show_err) is the most elegant
way.

 PVE/Storage/RBDPlugin.pm | 52 +++++++++++++++++++++++++++++++++++-----
 1 file changed, 46 insertions(+), 6 deletions(-)

diff --git a/PVE/Storage/RBDPlugin.pm b/PVE/Storage/RBDPlugin.pm
index 9047504..60fd353 100644
--- a/PVE/Storage/RBDPlugin.pm
+++ b/PVE/Storage/RBDPlugin.pm
@@ -168,6 +168,8 @@ my $krbd_feature_update = sub {
     }
 };
 
+my $missing_image_err_regex = '((?:vm|base)-\d+-.*): \(2\) No such file or directory$';
+
 sub run_rbd_command {
     my ($cmd, %args) = @_;
 
@@ -206,13 +208,28 @@ sub rbd_ls {
     my $raw = '';
     my $parser = sub { $raw .= shift };
 
+    my $show_err = 1;
+    my $missing_images = {};
+    my $err_parser = sub {
+	my $line = shift;
+	if ($line =~ m/$missing_image_err_regex/) {
+	    $show_err = 0;
+	    $missing_images->{$1} = 1;
+	} elsif ($line ne "rbd: listing images failed: (2) No such file or directory") {
+	    # this generic error is shown after the image specific "No such file..." one,
+	    # ignore it but not other errors
+	    $show_err = 1;
+	    die $line;
+	}
+    };
+
     my $cmd = $rbd_cmd->($scfg, $storeid, 'ls', '-l', '--format', 'json');
     eval {
-	run_rbd_command($cmd, errmsg => "rbd error", errfunc => sub {}, outfunc => $parser);
+	run_rbd_command($cmd, errmsg => "rbd error", errfunc => $err_parser, outfunc => $parser);
     };
     my $err = $@;
 
-    die $err if $err && $err !~ m/doesn't contain rbd images/ ;
+    die $err if $err && $show_err && $err !~ m/doesn't contain rbd images/ ;
 
     my $result;
     if ($raw eq '') {
@@ -223,6 +240,13 @@ sub rbd_ls {
 	die "got unexpected data from rbd ls: '$raw'\n";
     }
 
+    for my $image (keys %$missing_images) {
+	push @$result, {
+	    image => $image,
+	    size => -1,
+	};
+    }
+
     my $list = {};
 
     foreach my $el (@$result) {
@@ -250,7 +274,20 @@ sub rbd_ls_snap {
     my $cmd = $rbd_cmd->($scfg, $storeid, 'snap', 'ls', $name, '--format', 'json');
 
     my $raw = '';
-    run_rbd_command($cmd, errmsg => "rbd error", errfunc => sub {}, outfunc => sub { $raw .= shift; });
+    my $show_err = 0;
+    my $err_parser = sub {
+	my $line = shift;
+	if ($line !~ m/$missing_image_err_regex/) {
+	    $show_err = 1;
+	    die $line;
+	}
+    };
+    eval {
+	run_rbd_command($cmd, errmsg => "rbd error", errfunc => $err_parser, outfunc => sub { $raw .= shift; });
+    };
+    my $err = $@;
+    die $err if $err && $show_err;
+    return {} if $err && !$show_err; # could not open image, probably missing
 
     my $list;
     if ($raw =~ m/^(\[.*\])$/s) { # untaint
@@ -594,10 +631,13 @@ sub free_image {
 
     $class->deactivate_volume($storeid, $scfg, $volname);
 
-    my $cmd = $rbd_cmd->($scfg, $storeid, 'snap', 'purge',  $name);
-    run_rbd_command($cmd, errmsg => "rbd snap purge '$name' error");
 
-    $cmd = $rbd_cmd->($scfg, $storeid, 'rm', $name);
+    if (keys %{$snaps}) {
+	my $cmd = $rbd_cmd->($scfg, $storeid, 'snap', 'purge',  $name);
+	run_rbd_command($cmd, errmsg => "rbd snap purge '$name' error");
+    }
+
+    my $cmd = $rbd_cmd->($scfg, $storeid, 'rm', $name);
     run_rbd_command($cmd, errmsg => "rbd rm '$name' error");
 
     return undef;
-- 
2.30.2

[pve-devel] [PATCH widget-toolkit 2/2] utils: format_size: show negative size as NA

[Aaron Lauterer]

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
---

AFAIK we do not have negative sizes anywhere, and if, it is an
indication that something is wrong.

 src/Utils.js | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/src/Utils.js b/src/Utils.js
index ef72630..8cdbe86 100644
--- a/src/Utils.js
+++ b/src/Utils.js
@@ -688,6 +688,9 @@ utilities: {
     },
 
     format_size: function(size, useSI) {
+	if (size < 0) {
+	    return gettext("N/A");
+	}
 	let units = ['', 'K', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y'];
 	let order = 0;
 	const baseValue = useSI ? 1000 : 1024;
-- 
2.30.2

Re: [pve-devel] [PATCH storage 1/2] rbd: improve handling of missing images

[Aaron Lauterer]

ping?

Re: [pve-devel] [PATCH widget-toolkit 2/2] utils: format_size: show negative size as NA

[Thomas Lamprecht]

Am 19/04/2023 um 12:34 schrieb Aaron Lauterer:
> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
> ---
> 
> AFAIK we do not have negative sizes anywhere, and if, it is an
> indication that something is wrong.

above belongs in the commit message, additionaly some background for why doing
this now (i.e., did you run into this or what made you make this change?)

> 
>  src/Utils.js | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/src/Utils.js b/src/Utils.js
> index ef72630..8cdbe86 100644
> --- a/src/Utils.js
> +++ b/src/Utils.js
> @@ -688,6 +688,9 @@ utilities: {
>      },
>  
>      format_size: function(size, useSI) {
> +	if (size < 0) {
> +	    return gettext("N/A");

catching this seems OK, but I'd rather just return the value then, as "N/A" (Not
Applicable) doesn't really makes sense here and just hides a potential underlying
problem.

> +	}
>  	let units = ['', 'K', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y'];
>  	let order = 0;
>  	const baseValue = useSI ? 1000 : 1024;

Re: [pve-devel] [PATCH widget-toolkit 2/2] utils: format_size: show negative size as NA

[Aaron Lauterer]

On 6/1/23 16:22, Thomas Lamprecht wrote:
> Am 19/04/2023 um 12:34 schrieb Aaron Lauterer:
>> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
>> ---
>>
>> AFAIK we do not have negative sizes anywhere, and if, it is an
>> indication that something is wrong.
> 
> above belongs in the commit message, additionaly some background for why doing
> this now (i.e., did you run into this or what made you make this change?)
> 

good point. It happens with the first patch of the series, when we return '-1' 
to indicate a broken RBD image.


>>
>>   src/Utils.js | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/src/Utils.js b/src/Utils.js
>> index ef72630..8cdbe86 100644
>> --- a/src/Utils.js
>> +++ b/src/Utils.js
>> @@ -688,6 +688,9 @@ utilities: {
>>       },
>>   
>>       format_size: function(size, useSI) {
>> +	if (size < 0) {
>> +	    return gettext("N/A");
> 
> catching this seems OK, but I'd rather just return the value then, as "N/A" (Not
> Applicable) doesn't really makes sense here and just hides a potential underlying
> problem.

Since 'format_size' is used in many places all over the place, what about only 
checking for it in the content view, where we really shouldn't expect a negative 
size?
I think showing N/A instead of '-1 B' is more obvious. Something like this:

diff --git a/www/manager6/storage/ContentView.js 
b/www/manager6/storage/ContentView.js
index 2761b48e..c7b3d5ef 100644
--- a/www/manager6/storage/ContentView.js
+++ b/www/manager6/storage/ContentView.js
@@ -182,7 +182,12 @@ Ext.define('PVE.storage.ContentView', {
             'size': {
                 header: gettext('Size'),
                 width: 100,
-               renderer: Proxmox.Utils.format_size,
+               renderer: function(size) {
+                   if (Number(size) === -1) {
+                       return gettext("N/A");
+                   }
+                   return Proxmox.Utils.format_size(size);
+               },
                 dataIndex: 'size',
             },
         };