From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 7605091672 for ; Wed, 21 Dec 2022 11:09:41 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 4FB265FA4 for ; Wed, 21 Dec 2022 11:09:11 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 21 Dec 2022 11:09:09 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 24F7742523 for ; Wed, 21 Dec 2022 11:09:09 +0100 (CET) Date: Wed, 21 Dec 2022 11:09:01 +0100 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= To: Proxmox VE development discussion References: <20221220131638.223940-1-f.ebner@proxmox.com> In-Reply-To: <20221220131638.223940-1-f.ebner@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.16.0 (https://github.com/astroidmail/astroid) Message-Id: <1671616821.xwylqnkt0c.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL 0.134 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [zfspoolplugin.pm, proxmox.com] Subject: [pve-devel] applied-series: [PATCH v2 storage 1/3] zfs: list: only X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Dec 2022 10:09:41 -0000 thanks! some possible room for further improvements: - zfs_list_zvol could add `-d1` to only list $scfg->{pool} and direct child= ren (then we don't need to filter out any indirect descendants, just the "poo= l" itself..) - $list in zfs_list_zvol could be initialized as `{}`, then we don't need a= n explicit fallback in list_images On December 20, 2022 2:16 pm, Fiona Ebner wrote: > The plugin for remote ZFS storages currently also uses the same > list_images() as the plugin for local ZFS storages. There is only > one cache which does not remember the target host where the > information originated. >=20 > This is problematic for rescan in qemu-server: > 1. Via list_images() and zfs_list_zvol(), ZFSPlugin.pm's zfs_request() > is executed for a remote ZFS. > 2. $cache->{zfs} is set to the result of scanning the images there. > 3. Another list_images() for a local ZFS storage happens and uses the > cache with the wrong information. >=20 > The only two operations using the cache currently are > 1. Disk rescan in qemu-server which is affected by the issue. It is > done as part of (or as a) long-running operations. > 2. prepare_local_job for pvesr, but the non-local ZFS plugin doesn't > support replication, so it cannot trigger there. The cache only > helps if there is a replicated guest with volumes on different > ZFS storages, but otherwise it will be less efficient than no > cache or querying every storage by itself. >=20 > Fix the issue by making the cache $storeid-specific, which effectively > makes the cache superfluous, because there is no caller using the same > $storeid multiple times. As argued above, this should not really hurt > the callers described above much and actually improves performance for > all other callers. >=20 > Signed-off-by: Fiona Ebner > --- >=20 > Changes from v1: > * Always only list images for $scfg->{pool} (and drop patch that > would only do so for the remote ZFS plugin). > * This makes the cache useless, so add a patch removing it. > * Also add a patch for cleanup. >=20 > See here for previous discussion: > https://lists.proxmox.com/pipermail/pve-devel/2022-November/054485.html >=20 > PVE/Storage/ZFSPoolPlugin.pm | 25 +++++++++++++++++++------ > 1 file changed, 19 insertions(+), 6 deletions(-) >=20 > diff --git a/PVE/Storage/ZFSPoolPlugin.pm b/PVE/Storage/ZFSPoolPlugin.pm > index e264fde..0f16e7d 100644 > --- a/PVE/Storage/ZFSPoolPlugin.pm > +++ b/PVE/Storage/ZFSPoolPlugin.pm > @@ -254,11 +254,12 @@ sub free_image { > sub list_images { > my ($class, $storeid, $scfg, $vmid, $vollist, $cache) =3D @_; > =20 > - $cache->{zfs} =3D $class->zfs_list_zvol($scfg) if !$cache->{zfs}; > - my $zfspool =3D $scfg->{pool}; > + $cache->{zfs}->{$storeid} =3D $class->zfs_list_zvol($scfg) > + if !$cache->{zfs}->{$storeid}; > + > my $res =3D []; > =20 > - if (my $dat =3D $cache->{zfs}->{$zfspool}) { > + if (my $dat =3D $cache->{zfs}->{$storeid}) { > =20 > foreach my $image (keys %$dat) { > =20 > @@ -375,20 +376,32 @@ sub zfs_delete_zvol { > sub zfs_list_zvol { > my ($class, $scfg) =3D @_; > =20 > - my $text =3D $class->zfs_request($scfg, 10, 'list', '-o', 'name,vols= ize,origin,type,refquota', '-t', 'volume,filesystem', '-Hrp'); > + my $text =3D $class->zfs_request( > + $scfg, > + 10, > + 'list', > + '-o', > + 'name,volsize,origin,type,refquota', > + '-t', > + 'volume,filesystem', > + '-Hrp', > + $scfg->{pool}, > + ); > my $zvols =3D zfs_parse_zvol_list($text); > return undef if !$zvols; > =20 > my $list =3D (); > foreach my $zvol (@$zvols) { > - my $pool =3D $zvol->{pool}; > + # The "pool" in $scfg is not the same as ZFS pool, so it's necessary to= filter here. > + next if $scfg->{pool} ne $zvol->{pool}; > + > my $name =3D $zvol->{name}; > my $parent =3D $zvol->{origin}; > if($zvol->{origin} && $zvol->{origin} =3D~ m/^$scfg->{pool}\/(\S+)$/){ > $parent =3D $1; > } > =20 > - $list->{$pool}->{$name} =3D { > + $list->{$name} =3D { > name =3D> $name, > size =3D> $zvol->{size}, > parent =3D> $parent, > --=20 > 2.30.2 >=20 >=20 >=20 > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel >=20 >=20 >=20