From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 6F9C46CCBB for ; Fri, 24 Sep 2021 13:31:56 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5D6DDA342 for ; Fri, 24 Sep 2021 13:31:26 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 9A196A336 for ; Fri, 24 Sep 2021 13:31:24 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 6ACEF4323D for ; Fri, 24 Sep 2021 13:31:24 +0200 (CEST) To: Dominik Csapak , Proxmox VE development discussion References: <20210917130227.248852-1-f.ebner@proxmox.com> <20210917130227.248852-3-f.ebner@proxmox.com> <1e14cc90-2a2d-b59e-0574-718e3128873f@proxmox.com> <889814f6-92b8-304f-8012-d3a86be38ac2@proxmox.com> From: Fabian Ebner Message-ID: Date: Fri, 24 Sep 2021 13:31:21 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: <889814f6-92b8-304f-8012-d3a86be38ac2@proxmox.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.067 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.001 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [btrfsplugin.pm, dirplugin.pm, storage.pm, cephfsplugin.pm, pbsplugin.pm, nfsplugin.pm, cifsplugin.pm, plugin.pm, content.pm] URI_NOVOWEL 0.5 URI hostname has long non-vowel sequence Subject: Re: [pve-devel] [RFC storage 2/6] add generalized functions to manage volume attributes X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Sep 2021 11:31:56 -0000 Am 24.09.21 um 13:16 schrieb Dominik Csapak: > On 9/24/21 13:05, Fabian Ebner wrote: >> Am 24.09.21 um 10:54 schrieb Dominik Csapak: >>> On 9/17/21 15:02, Fabian Ebner wrote: >>>> replacing the ones for handling notes. The generic implementation in >>>> Plugin.pm will fall back to the methods for notes to ensure backwards >>>> compatibility with external plugins. >>>> >>>> This is mainly done to avoid the need to add new methods every time a >>>> new attribute is added. >>>> >>>> Not adding a timeout parameter like the notes functions have, because >>>> it was not used and can still be added if it ever is needed in the >>>> future. >>>> >>>> For get_volume_attribute, undef will indicate that the attribute is >>>> not supported. This makes it possible to distinguish "not supported" >>>> from "error getting the attribute", which is useful when the attribute >>>> is important for an operation. For example, free_image checking for >>>> protection (introduced in a later patch) can abort if getting the >>>> 'protected' attribute fails. >>>> >>>> Suggested-by: Thomas Lamprecht >>>> Signed-off-by: Fabian Ebner >>>> --- >>>> >>>> Hope I didn't miss a way this would break external plugins. >>> >>> i think it would break a plugin that is not derived from >>> the bas Plugin.pm no? >>> >>> an example: >>> >>> i have a CustomCIFSPlugin that uses CIFSPlugin as base >>> >>> i have implemented a custom 'get_volume_notes' function >>> >>> after this patch, 'get_volume_attribute' will be called, which >>> falls back to the CIFSPlugin versionwhich calls the DirPlugin >>> 'get_volume_attribute' instead of my custom 'get_volume_notes' >>> >> >> Right. Having the fall-back only in Plugin.pm would not be enough then. >> >>> since we did not document the custom plugins very well, i am >>> not sure if that's even a supported scenario, or if a custom >>> plugin always have to use the 'Plugin' as a base >>> >>> if it is supported, we could only solve it by either make >>> it a breaking api change, or by keeping the '*_volume_notes' >>> around and fall back to *_volume_attribute implmentation >>> >> >> That wouldn't solve the problem described above, would it? You still >> follow the same call-path and land in DirPlugin's get_volume_attribute >> in the end. >> >> Yes, it would be necessary to keep the *_volume_notes functions >> around, but I think you need to have the opposite fall-back, i.e. >> *_volume_attribute has to call *_volume_notes to catch derived >> implementations like Plugin.pm. > > > i meant in e.g. Content.pm that would call 'get_volume_notes'. > this would call the correct one (if it exists) > and a default which calls 'get_volume_attribute' > would catch all plugins that have no 'get_volume_notes' ? Yes, it works if the outside still calls the old ones, i.e. having a fallback in Storage.pm's wrapper would make the above work. Then it'd be Storage.pm get_volume_attribute calls $plugin->get_volume_notes calls $plugin->get_volume_attribute, with the last call happening if the Plugin is recent enough to have implemented the fallback. > > but i think it could work both ways, though i like your suggested > approach more since we use the new sub everywhere outside the plugins > >> >>> or did i make a mistake anywhere in my assumptions? >>> >>>> >>>> Requires an APIAGE+APIVER bump. >>>> >>>> The alternative to undef indicating "not supported" would be to add a >>>> separate method like supports_volume_attribute. Or adapt >>>> volume_has_feature, but it doesn't really feel like the right fit as >>>> it's currently intended for features for guest images only. >>>> >>>>   PVE/API2/Storage/Content.pm |  7 +++--- >>>>   PVE/Storage.pm              | 12 +++++----- >>>>   PVE/Storage/BTRFSPlugin.pm  |  4 ++-- >>>>   PVE/Storage/CIFSPlugin.pm   | 10 ++++---- >>>>   PVE/Storage/CephFSPlugin.pm | 10 ++++---- >>>>   PVE/Storage/DirPlugin.pm    | 47 >>>> +++++++++++++++++++++++-------------- >>>>   PVE/Storage/NFSPlugin.pm    | 10 ++++---- >>>>   PVE/Storage/PBSPlugin.pm    | 28 ++++++++++++++-------- >>>>   PVE/Storage/Plugin.pm       | 36 ++++++++++++++++++++++++++++ >>>>   9 files changed, 113 insertions(+), 51 deletions(-) >>>> >>>> diff --git a/PVE/API2/Storage/Content.pm b/PVE/API2/Storage/Content.pm >>>> index 4d0ceb6..b3dc593 100644 >>>> --- a/PVE/API2/Storage/Content.pm >>>> +++ b/PVE/API2/Storage/Content.pm >>>> @@ -321,11 +321,12 @@ __PACKAGE__->register_method ({ >>>>           format => $format, >>>>       }; >>>> -    # not all storages/types support notes, so ignore errors here >>>> +    # keep going if fetching an optional attribute fails >>>>       eval { >>>> -        my $notes = PVE::Storage::get_volume_notes($cfg, $volid); >>>> +        my $notes = PVE::Storage::get_volume_attribute($cfg, >>>> $volid, 'notes'); >>>>           $entry->{notes} = $notes if defined($notes); >>>>       }; >>>> +    warn $@ if $@; >>>>       return $entry; >>>>       }}); >>>> @@ -371,7 +372,7 @@ __PACKAGE__->register_method ({ >>>>       PVE::Storage::check_volume_access($rpcenv, $authuser, $cfg, >>>> undef, $volid); >>>>       if (exists $param->{notes}) { >>>> -        PVE::Storage::update_volume_notes($cfg, $volid, >>>> $param->{notes}); >>>> +        PVE::Storage::update_volume_attribute($cfg, $volid, >>>> 'notes', $param->{notes}); >>>>       } >>>>       return undef; >>>> diff --git a/PVE/Storage.pm b/PVE/Storage.pm >>>> index 7b319fe..729c90e 100755 >>>> --- a/PVE/Storage.pm >>>> +++ b/PVE/Storage.pm >>>> @@ -215,24 +215,24 @@ sub file_size_info { >>>>       return PVE::Storage::Plugin::file_size_info($filename, $timeout); >>>>   } >>>> -sub get_volume_notes { >>>> -    my ($cfg, $volid, $timeout) = @_; >>>> +sub get_volume_attribute { >>>> +    my ($cfg, $volid, $attribute) = @_; >>>>       my ($storeid, $volname) = parse_volume_id($volid); >>>>       my $scfg = storage_config($cfg, $storeid); >>>>       my $plugin = PVE::Storage::Plugin->lookup($scfg->{type}); >>>> -    return $plugin->get_volume_notes($scfg, $storeid, $volname, >>>> $timeout); >>>> +    return $plugin->get_volume_attribute($scfg, $storeid, $volname, >>>> $attribute); >>>>   } >>>> -sub update_volume_notes { >>>> -    my ($cfg, $volid, $notes, $timeout) = @_; >>>> +sub update_volume_attribute { >>>> +    my ($cfg, $volid, $attribute, $value) = @_; >>>>       my ($storeid, $volname) = parse_volume_id($volid); >>>>       my $scfg = storage_config($cfg, $storeid); >>>>       my $plugin = PVE::Storage::Plugin->lookup($scfg->{type}); >>>> -    $plugin->update_volume_notes($scfg, $storeid, $volname, $notes, >>>> $timeout); >>>> +    return $plugin->update_volume_attribute($scfg, $storeid, >>>> $volname, $attribute, $value); >>>>   } >>>>   sub volume_size_info { >>>> diff --git a/PVE/Storage/BTRFSPlugin.pm b/PVE/Storage/BTRFSPlugin.pm >>>> index dbc1244..61bede2 100644 >>>> --- a/PVE/Storage/BTRFSPlugin.pm >>>> +++ b/PVE/Storage/BTRFSPlugin.pm >>>> @@ -136,9 +136,9 @@ sub status { >>>>       return PVE::Storage::DirPlugin::status($class, $storeid, >>>> $scfg, $cache); >>>>   } >>>> -# TODO: sub get_volume_notes {} >>>> +# TODO: sub get_volume_attribute {} >>>> -# TODO: sub update_volume_notes {} >>>> +# TODO: sub update_volume_attribute {} >>>>   # croak would not include the caller from within this module >>>>   sub __error { >>>> diff --git a/PVE/Storage/CIFSPlugin.pm b/PVE/Storage/CIFSPlugin.pm >>>> index c5f3894..bee6885 100644 >>>> --- a/PVE/Storage/CIFSPlugin.pm >>>> +++ b/PVE/Storage/CIFSPlugin.pm >>>> @@ -286,13 +286,15 @@ sub check_connection { >>>>       return 1; >>>>   } >>>> -sub get_volume_notes { >>>> +sub get_volume_attribute { >>>>       my $class = shift; >>>> -    PVE::Storage::DirPlugin::get_volume_notes($class, @_); >>>> + >>>> +    return PVE::Storage::DirPlugin::get_volume_attribute($class, @_); >>>>   } >>>> -sub update_volume_notes { >>>> +sub update_volume_attribute { >>>>       my $class = shift; >>>> -    PVE::Storage::DirPlugin::update_volume_notes($class, @_); >>>> + >>>> +    return PVE::Storage::DirPlugin::update_volume_attribute($class, >>>> @_); >>>>   } >>>>   1; >>>> diff --git a/PVE/Storage/CephFSPlugin.pm b/PVE/Storage/CephFSPlugin.pm >>>> index 3b9a791..2c06804 100644 >>>> --- a/PVE/Storage/CephFSPlugin.pm >>>> +++ b/PVE/Storage/CephFSPlugin.pm >>>> @@ -232,14 +232,16 @@ sub deactivate_storage { >>>>       } >>>>   } >>>> -sub get_volume_notes { >>>> +sub get_volume_attribute { >>>>       my $class = shift; >>>> -    PVE::Storage::DirPlugin::get_volume_notes($class, @_); >>>> + >>>> +    return PVE::Storage::DirPlugin::get_volume_attribute($class, @_); >>>>   } >>>> -sub update_volume_notes { >>>> +sub update_volume_attribute { >>>>       my $class = shift; >>>> -    PVE::Storage::DirPlugin::update_volume_notes($class, @_); >>>> + >>>> +    return PVE::Storage::DirPlugin::update_volume_attribute($class, >>>> @_); >>>>   } >>>>   1; >>>> diff --git a/PVE/Storage/DirPlugin.pm b/PVE/Storage/DirPlugin.pm >>>> index 0423e5f..6ab9ef2 100644 >>>> --- a/PVE/Storage/DirPlugin.pm >>>> +++ b/PVE/Storage/DirPlugin.pm >>>> @@ -87,32 +87,43 @@ sub parse_is_mountpoint { >>>>       return $is_mp; # contains a path >>>>   } >>>> -sub get_volume_notes { >>>> -    my ($class, $scfg, $storeid, $volname, $timeout) = @_; >>>> +sub get_volume_attribute { >>>> +    my ($class, $scfg, $storeid, $volname, $attribute) = @_; >>>> -    my $path = $class->filesystem_path($scfg, $volname); >>>> -    $path .= $class->SUPER::NOTES_EXT; >>>> +    my ($vtype) = $class->parse_volname($volname); >>>> +    return if $vtype ne 'backup'; >>>> -    return PVE::Tools::file_get_contents($path) if -f $path; >>>> +    if ($attribute eq 'notes') { >>>> +    my $path = $class->filesystem_path($scfg, $volname); >>>> +    $path .= $class->SUPER::NOTES_EXT; >>>> -    return ''; >>>> -} >>>> +    return PVE::Tools::file_get_contents($path) if -f $path; >>>> -sub update_volume_notes { >>>> -    my ($class, $scfg, $storeid, $volname, $notes, $timeout) = @_; >>>> +    return ''; >>>> +    } >>>> -    my ($vtype) = $class->parse_volname($volname); >>>> -    die "only backups can have notes\n" if $vtype ne 'backup'; >>>> +    return; >>>> +} >>>> -    my $path = $class->filesystem_path($scfg, $volname); >>>> -    $path .= $class->SUPER::NOTES_EXT; >>>> +sub update_volume_attribute { >>>> +    my ($class, $scfg, $storeid, $volname, $attribute, $value) = @_; >>>> -    if (defined($notes) && $notes ne '') { >>>> -    PVE::Tools::file_set_contents($path, $notes); >>>> -    } elsif (-e $path) { >>>> -    unlink $path or die "could not delete notes - $!\n"; >>>> +    my ($vtype) = $class->parse_volname($volname); >>>> +    die "only backups support attribute '$attribute'\n" if $vtype >>>> ne 'backup'; >>>> + >>>> +    if ($attribute eq 'notes') { >>>> +    my $path = $class->filesystem_path($scfg, $volname); >>>> +    $path .= $class->SUPER::NOTES_EXT; >>>> + >>>> +    if (defined($value) && $value ne '') { >>>> +        PVE::Tools::file_set_contents($path, $value); >>>> +    } elsif (-e $path) { >>>> +        unlink $path or die "could not delete notes - $!\n"; >>>> +    } >>>> +    return; >>>>       } >>>> -    return; >>>> + >>>> +    die "attribute '$attribute' is not supported for storage type >>>> '$scfg->{type}'\n"; >>>>   } >>>>   sub status { >>>> diff --git a/PVE/Storage/NFSPlugin.pm b/PVE/Storage/NFSPlugin.pm >>>> index 39bf15a..3a6efe5 100644 >>>> --- a/PVE/Storage/NFSPlugin.pm >>>> +++ b/PVE/Storage/NFSPlugin.pm >>>> @@ -188,13 +188,15 @@ sub check_connection { >>>>       return 1; >>>>   } >>>> -sub get_volume_notes { >>>> +sub get_volume_attribute { >>>>       my $class = shift; >>>> -    PVE::Storage::DirPlugin::get_volume_notes($class, @_); >>>> + >>>> +    return PVE::Storage::DirPlugin::get_volume_attribute($class, @_); >>>>   } >>>> -sub update_volume_notes { >>>> +sub update_volume_attribute { >>>>       my $class = shift; >>>> -    PVE::Storage::DirPlugin::update_volume_notes($class, @_); >>>> + >>>> +    return PVE::Storage::DirPlugin::update_volume_attribute($class, >>>> @_); >>>>   } >>>>   1; >>>> diff --git a/PVE/Storage/PBSPlugin.pm b/PVE/Storage/PBSPlugin.pm >>>> index bb1c382..d8e1ac8 100644 >>>> --- a/PVE/Storage/PBSPlugin.pm >>>> +++ b/PVE/Storage/PBSPlugin.pm >>>> @@ -782,24 +782,32 @@ sub deactivate_volume { >>>>       return 1; >>>>   } >>>> -sub get_volume_notes { >>>> -    my ($class, $scfg, $storeid, $volname, $timeout) = @_; >>>> +sub get_volume_attribute { >>>> +    my ($class, $scfg, $storeid, $volname, $attribute) = @_; >>>> + >>>> +    if ($attribute eq 'notes') { >>>> +    my (undef, $name,  undef, undef, undef, undef, $format) = >>>> $class->parse_volname($volname); >>>> -    my (undef, $name,  undef, undef, undef, undef, $format) = >>>> $class->parse_volname($volname); >>>> +    my $data = run_client_cmd($scfg, $storeid, "snapshot", [ >>>> "notes", "show", $name ]); >>>> -    my $data = run_client_cmd($scfg, $storeid, "snapshot", [ >>>> "notes", "show", $name ]); >>>> +    return $data->{notes} // ''; >>>> +    } >>>> -    return $data->{notes}; >>>> +    return; >>>>   } >>>> -sub update_volume_notes { >>>> -    my ($class, $scfg, $storeid, $volname, $notes, $timeout) = @_; >>>> +sub update_volume_attribute { >>>> +    my ($class, $scfg, $storeid, $volname, $attribute, $value) = @_; >>>> -    my (undef, $name,  undef, undef, undef, undef, $format) = >>>> $class->parse_volname($volname); >>>> +    if ($attribute eq 'notes') { >>>> +    my (undef, $name,  undef, undef, undef, undef, $format) = >>>> $class->parse_volname($volname); >>>> -    run_client_cmd($scfg, $storeid, "snapshot", [ "notes", >>>> "update", $name, $notes ], 1); >>>> +    run_client_cmd($scfg, $storeid, "snapshot", [ "notes", >>>> "update", $name, $value ], 1); >>>> -    return undef; >>>> +    return; >>>> +    } >>>> + >>>> +    die "attribute '$attribute' is not supported for storage type >>>> '$scfg->{type}'\n"; >>>>   } >>>>   sub volume_size_info { >>>> diff --git a/PVE/Storage/Plugin.pm b/PVE/Storage/Plugin.pm >>>> index 417d1fd..1182008 100644 >>>> --- a/PVE/Storage/Plugin.pm >>>> +++ b/PVE/Storage/Plugin.pm >>>> @@ -851,18 +851,52 @@ sub file_size_info { >>>>       return wantarray ? ($size, $format, $used, $parent, >>>> $st->ctime) : $size; >>>>   } >>>> +# FIXME remove on the next APIAGE reset. >>>> +# Deprecated, use get_volume_attribute instead. >>>>   sub get_volume_notes { >>>>       my ($class, $scfg, $storeid, $volname, $timeout) = @_; >>>>       die "volume notes are not supported for $class"; >>>>   } >>>> +# FIXME remove on the next APIAGE reset. >>>> +# Deprecated, use update_volume_attribute instead. >>>>   sub update_volume_notes { >>>>       my ($class, $scfg, $storeid, $volname, $notes, $timeout) = @_; >>>>       die "volume notes are not supported for $class"; >>>>   } >>>> +# Returns undef if the attribute is not supported for the volume. >>>> +# Should die if there is an error fetching the attribute. >>>> +# Possible attributes: >>>> +# notes     - user-provided comments/notes. >>>> +sub get_volume_attribute { >>>> +    my ($class, $scfg, $storeid, $volname, $attribute) = @_; >>>> + >>>> +    if ($attribute eq 'notes') { >>>> +     my $notes = eval { $class->get_volume_notes($scfg, $storeid, >>>> $volname); }; >>>> +     if (my $err = $@) { >>>> +         return if $err =~ m/^volume notes are not supported/; >>>> +         die $err; >>>> +     } >>>> +     return $notes; >>>> +    } >>>> + >>>> +    return; >>>> +} >>>> + >>>> +# Dies if the attribute is not supported for the volume. >>>> +sub update_volume_attribute { >>>> +    my ($class, $scfg, $storeid, $volname, $attribute, $value) = @_; >>>> + >>>> +    if ($attribute eq 'notes') { >>>> +    $class->update_volume_notes($scfg, $storeid, $volname, $value); >>>> +    } >>>> + >>>> +    die "attribute '$attribute' is not supported for storage type >>>> '$scfg->{type}'\n"; >>>> +} >>>> + >>>>   sub volume_size_info { >>>>       my ($class, $scfg, $storeid, $volname, $timeout) = @_; >>>>       my $path = $class->filesystem_path($scfg, $volname); >>>> @@ -1098,6 +1132,8 @@ my $get_subdir_files = sub { >>>>       return $res; >>>>   }; >>>> +# If attributes are set on a volume, they should be included in the >>>> result. >>>> +# See get_volume_attribute for a list of possible attributes. >>>>   sub list_volumes { >>>>       my ($class, $storeid, $scfg, $vmid, $content_types) = @_; >>>> >>> >>> > >