From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 1684C75118 for ; Wed, 23 Jun 2021 14:15:43 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 0C314AAE5 for ; Wed, 23 Jun 2021 14:15:13 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id C051FAADA for ; Wed, 23 Jun 2021 14:15:10 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 97E2A4389E for ; Wed, 23 Jun 2021 14:15:10 +0200 (CEST) Date: Wed, 23 Jun 2021 14:15:00 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= To: Proxmox VE development discussion References: <20210622121828.84178-1-w.bumiller@proxmox.com> <20210622121828.84178-2-w.bumiller@proxmox.com> In-Reply-To: <20210622121828.84178-2-w.bumiller@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.15.0 (https://github.com/astroidmail/astroid) Message-Id: <1624446697.vi3jolanip.astroid@nora.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL 0.398 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URI_NOVOWEL 0.5 URI hostname has long non-vowel sequence Subject: Re: [pve-devel] [PATCH v2 storage 1/5] add BTRFS storage plugin X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jun 2021 12:15:43 -0000 On June 22, 2021 2:18 pm, Wolfgang Bumiller wrote: > This is mostly the same as a directory storage, with 2 major > differences: >=20 > * 'subvol' volumes are actual btrfs subvolumes and therefore > allow snapshots > * 'raw' files are placed *into* a subvolume and therefore > also allow snapshots, the raw file for volume > `btrstore:100/vm-100-disk-1.raw` can be found under > `$path/images/100/vm-100-disk-1/disk.raw` > * in both cases, snapshots add an '@name' suffix to the > subvolume's directory name, so snapshot 'foo' of the above > would be found under > `$path/images/100/vm-100-disk-1@foo/disk.raw` > or for format "subvol": > `$path/images/100/subvol-100-disk-1.subvol@foo` >=20 > Note that qgroups aren't included in btrfs-send streams, > therefore for now we will only be using *unsized* subvolumes > for containers and place a regular raw+ext4 file for sized > containers. > We could extend the import/export stream format to include > the information at the front (similar to how we do the > "tar+size" format, but we need to include the size of all > the contained snapshots as well, since they can technically > change). (But before enabling quotas we should do some > performance testing on bigger file systems with multiple > snapshots as there are quite a few reports of the fs slowing > down considerably in such scenarios). >=20 > Signed-off-by: Wolfgang Bumiller > --- > Changes to v1: > raw files are created manually with sysopen/truncate and > ioctls for setting the NOCOW bit have been added (this > are made optional later in the series) >=20 >=20 > PVE/Storage.pm | 2 + > PVE/Storage/BTRFSPlugin.pm | 634 +++++++++++++++++++++++++++++++++++++ > PVE/Storage/Makefile | 1 + > 3 files changed, 637 insertions(+) > create mode 100644 PVE/Storage/BTRFSPlugin.pm >=20 > diff --git a/PVE/Storage.pm b/PVE/Storage.pm > index ea887a4..e0d6f44 100755 > --- a/PVE/Storage.pm > +++ b/PVE/Storage.pm > @@ -38,6 +38,7 @@ use PVE::Storage::GlusterfsPlugin; > use PVE::Storage::ZFSPoolPlugin; > use PVE::Storage::ZFSPlugin; > use PVE::Storage::PBSPlugin; > +use PVE::Storage::BTRFSPlugin; > =20 > # Storage API version. Increment it on changes in storage API interface. > use constant APIVER =3D> 8; > @@ -60,6 +61,7 @@ PVE::Storage::GlusterfsPlugin->register(); > PVE::Storage::ZFSPoolPlugin->register(); > PVE::Storage::ZFSPlugin->register(); > PVE::Storage::PBSPlugin->register(); > +PVE::Storage::BTRFSPlugin->register(); > =20 > # load third-party plugins > if ( -d '/usr/share/perl5/PVE/Storage/Custom' ) { > diff --git a/PVE/Storage/BTRFSPlugin.pm b/PVE/Storage/BTRFSPlugin.pm > new file mode 100644 > index 0000000..370d848 > --- /dev/null > +++ b/PVE/Storage/BTRFSPlugin.pm > @@ -0,0 +1,634 @@ > +package PVE::Storage::BTRFSPlugin; > + > +use strict; > +use warnings; > + > +use base qw(PVE::Storage::Plugin); > + > +use Fcntl qw(S_ISDIR O_WRONLY O_CREAT O_EXCL); > +use File::Basename qw(dirname); > +use File::Path qw(mkpath); > +use IO::Dir; > + > +use PVE::Tools qw(run_command); > + > +use PVE::Storage::DirPlugin; > + > +use constant { > + BTRFS_FIRST_FREE_OBJECTID =3D> 256, > + FS_NOCOW_FL =3D> 0x00800000, > + FS_IOC_GETFLAGS =3D> 0x40086602, > + FS_IOC_SETFLAGS =3D> 0x80086601, > +}; > + > +# Configuration (similar to DirPlugin) > + > +sub type { > + return 'btrfs'; > +} > + > +sub plugindata { > + return { > + content =3D> [ > + { > + images =3D> 1, > + rootdir =3D> 1, > + vztmpl =3D> 1, > + iso =3D> 1, > + backup =3D> 1, > + snippets =3D> 1, > + none =3D> 1, > + }, > + { images =3D> 1, rootdir =3D> 1 }, > + ], > + format =3D> [ { raw =3D> 1, qcow2 =3D> 1, vmdk =3D> 1, subvol =3D> 1 },= 'raw', ], I am not really convinced we need this (or rather, that it doesn't=20 cause more confusion than convenience. if I want to use btrfs without=20 any btrfs features, I can already use any btrfs-backed dir as dir=20 storage (just like with ZFS, or $storage-of-my-choice-providing-a-directory= ). e.g., having a qcow2 file on btrfs storage support snapshots but not use=20 btrfs functionality for that seems... weird? > + }; > +} > + > +sub options { > + return { > + path =3D> { fixed =3D> 1 }, > + nodes =3D> { optional =3D> 1 }, > + shared =3D> { optional =3D> 1 }, > + disable =3D> { optional =3D> 1 }, > + maxfiles =3D> { optional =3D> 1 }, > + content =3D> { optional =3D> 1 }, > + format =3D> { optional =3D> 1 }, > + is_mountpoint =3D> { optional =3D> 1 }, > + # TODO: The new variant of mkdir with `populate` vs `create`... > + }; > +} > + > +# Storage implementation > +# > +# We use the same volume names are directory plugins, but map *raw* disk= image file names into a > +# subdirectory. > +# > +# `vm-VMID-disk-ID.raw` > +# -> `images/VMID/vm-VMID-disk-ID/disk.raw` > +# where the `vm-VMID-disk-ID/` subdirectory is a btrfs subvolume > + > +# Reuse `DirPlugin`'s `check_config`. This simply checks for invalid pat= hs. > +sub check_config { > + my ($self, $sectionId, $config, $create, $skipSchemaCheck) =3D @_; > + return PVE::Storage::DirPlugin::check_config($self, $sectionId, $con= fig, $create, $skipSchemaCheck); > +} > + > +sub activate_storage { > + my ($class, $storeid, $scfg, $cache) =3D @_; > + return PVE::Storage::DirPlugin::activate_storage($class, $storeid, $= scfg, $cache); shouldn't this check that we can actually do btrfs stuff on $path? > +} > + > +sub status { > + my ($class, $storeid, $scfg, $cache) =3D @_; > + return PVE::Storage::DirPlugin::status($class, $storeid, $scfg, $cac= he); > +} > + > +# TODO: sub get_volume_notes {} > + > +# TODO: sub update_volume_notes {} > + > +# croak would not include the caller from within this module > +sub __error { > + my ($msg) =3D @_; > + my (undef, $f, $n) =3D caller(1); > + die "$msg at $f: $n\n"; > +} this is kind of a new style of error handling - might be worthy of a=20 mention somewhere ;) > + > +# Given a name (eg. `vm-VMID-disk-ID.raw`), take the part up to the form= at suffix as the name of > +# the subdirectory (subvolume). > +sub raw_name_to_dir($) { > + my ($raw) =3D @_; > + > + # For the subvolume directory Strip the `.` suffix: > + if ($raw =3D~ /^(.*)\.raw$/) { > + return $1; > + } > + > + __error "internal error: bad disk name: $raw"; > +} > + > +sub raw_file_to_subvol($) { > + my ($file) =3D @_; > + > + if ($file =3D~ m|^(.*)/disk\.raw$|) { > + return "$1"; > + } > + > + __error "internal error: bad raw path: $file"; > +} > + > +sub filesystem_path { > + my ($class, $scfg, $volname, $snapname) =3D @_; > + > + my ($vtype, $name, $vmid, undef, undef, $isBase, $format) =3D > + $class->parse_volname($volname); > + > + my $path =3D $class->get_subdir($scfg, $vtype); > + > + $path .=3D "/$vmid" if $vtype eq 'images'; > + > + if ($format eq 'raw') { > + my $dir =3D raw_name_to_dir($name); > + if ($snapname) { > + $dir .=3D "\@$snapname"; > + } > + $path .=3D "/$dir/disk.raw"; > + } elsif ($format eq 'subvol') { > + $path .=3D "/$name"; > + if ($snapname) { > + $path .=3D "\@$snapname"; > + } > + } else { > + $path .=3D "/$name"; > + } > + > + return wantarray ? ($path, $vmid, $vtype) : $path; > +} > + > +sub btrfs_cmd { > + my ($class, $cmd, $outfunc) =3D @_; > + > + my $msg =3D ''; > + my $func; > + if (defined($outfunc)) { > + $func =3D sub { > + my $part =3D &$outfunc(@_); > + $msg .=3D $part if defined($part); > + }; > + } else { > + $func =3D sub { $msg .=3D "$_[0]\n" }; > + } > + run_command(['btrfs', '-q', @$cmd], errmsg =3D> 'btrfs error', outfu= nc =3D> $func); > + > + return $msg; > +} > + > +sub btrfs_get_subvol_id { > + my ($class, $path) =3D @_; > + my $info =3D $class->btrfs_cmd(['subvolume', 'show', '--', $path]); > + if ($info !~ /^\s*(?:Object|Subvolume) ID:\s*(\d+)$/m) { > + die "failed to get btrfs subvolume ID from: $info\n"; ugh at yet another parser for CLI tool output.. (not your fault I=20 know..) > + } > + return $1; > +} > + > +my sub chattr : prototype($$$) { > + my ($fh, $mask, $xor) =3D @_; > + > + my $flags =3D pack('L!', 0); > + ioctl($fh, FS_IOC_GETFLAGS, $flags) or die "FS_IOC_GETFLAGS failed -= $!\n"; > + $flags =3D pack('L!', (unpack('L!', $flags) & $mask) ^ $xor); > + ioctl($fh, FS_IOC_SETFLAGS, $flags) or die "FS_IOC_SETFLAGS failed -= $!\n"; > + return 1; > +} > + > +sub create_base { > + my ($class, $storeid, $scfg, $volname) =3D @_; > + > + my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $format) = =3D > + $class->parse_volname($volname); > + > + my $newname =3D $name; > + $newname =3D~ s/^vm-/base-/; > + > + # If we're not working with a 'raw' file, which is the only thing th= at's "different" for btrfs, > + # or a subvolume, we forward to the DirPlugin > + if ($format ne 'raw' && $format ne 'subvol') { > + return PVE::Storage::DirPlugin::create_base(@_); > + } see above.. > + > + my $path =3D $class->filesystem_path($scfg, $volname); > + my $newvolname =3D $basename ? "$basevmid/$basename/$vmid/$newname" = : "$vmid/$newname"; > + my $newpath =3D $class->filesystem_path($scfg, $newvolname); > + > + my $subvol =3D $path; > + my $newsubvol =3D $newpath; > + if ($format eq 'raw') { > + $subvol =3D raw_file_to_subvol($subvol); > + $newsubvol =3D raw_file_to_subvol($newsubvol); > + } > + > + rename($subvol, $newsubvol) > + || die "rename '$subvol' to '$newsubvol' failed - $!\n"; > + eval { $class->btrfs_cmd(['property', 'set', $newsubvol, 'ro', 'true= ']) }; > + warn $@ if $@; > + > + return $newvolname; > +} > + > +sub clone_image { > + my ($class, $scfg, $storeid, $volname, $vmid, $snap) =3D @_; > + > + my ($vtype, $basename, $basevmid, undef, undef, $isBase, $format) = =3D > + $class->parse_volname($volname); > + > + # If we're not working with a 'raw' file, which is the only thing th= at's "different" for btrfs, > + # or a subvolume, we forward to the DirPlugin > + if ($format ne 'raw' && $format ne 'subvol') { > + return PVE::Storage::DirPlugin::clone_image(@_); > + } see above.. > + my $imagedir =3D $class->get_subdir($scfg, 'images'); > + $imagedir .=3D "/$vmid"; > + mkpath $imagedir; > + > + my $path =3D $class->filesystem_path($scfg, $volname); > + my $newname =3D $class->find_free_diskname($storeid, $scfg, $vmid, $= format, 1); > + > + # For btrfs subvolumes we don't actually need the "link": > + #my $newvolname =3D "$basevmid/$basename/$vmid/$newname"; > + my $newvolname =3D "$vmid/$newname"; > + my $newpath =3D $class->filesystem_path($scfg, $newvolname); > + > + my $subvol =3D $path; > + my $newsubvol =3D $newpath; > + if ($format eq 'raw') { > + $subvol =3D raw_file_to_subvol($subvol); > + $newsubvol =3D raw_file_to_subvol($newsubvol); > + } so actually these are two blocks for getting at $subvol and $newsubvol,=20 one for raw one for subvol. structuring it as such would make it more=20 readable IMHO.. > + > + $class->btrfs_cmd(['subvolume', 'snapshot', '--', $subvol, $newsubvo= l]); > + > + return $newvolname; > +} > + > +sub alloc_image { > + my ($class, $storeid, $scfg, $vmid, $fmt, $name, $size) =3D @_; > + > + if ($fmt ne 'raw' && $fmt ne 'subvol') { > + return PVE::Storage::DirPlugin::alloc_image(@_); > + } > + > + # From Plugin.pm: > + > + my $imagedir =3D $class->get_subdir($scfg, 'images') . "/$vmid"; > + > + mkpath $imagedir; > + > + $name =3D $class->find_free_diskname($storeid, $scfg, $vmid, $fmt, 1= ) if !$name; > + > + my (undef, $tmpfmt) =3D PVE::Storage::Plugin::parse_name_dir($name); > + > + die "illegal name '$name' - wrong extension for format ('$tmpfmt != =3D '$fmt')\n" > + if $tmpfmt ne $fmt; > + > + # End copy from Plugin.pm > + > + my $subvol =3D "$imagedir/$name"; > + # .raw is not part of the directory name > + $subvol =3D~ s/\.raw$//; why not use your helpers for this (raw_.._to_..) in a similar vain, there are three "add '/disk.raw' to path" sites=20 that might warrant a helper.. > + > + die "disk image '$subvol' already exists\n" if -e $subvol; > + > + my $path; > + if ($fmt eq 'raw') { > + $path =3D "$subvol/disk.raw"; > + } > + > + if ($fmt eq 'subvol' && !!$size) { > + # NOTE: `btrfs send/recv` actually drops quota information so supportin= g subvolumes with > + # quotas doesn't play nice with send/recv. > + die "btrfs quotas are currently not supported, use an unsized subvolume= or a raw file\n"; > + } > + > + $class->btrfs_cmd(['subvolume', 'create', '--', $subvol]); > + > + eval { > + if ($fmt eq 'subvol') { > + # Nothing to do for now... > + > + # This is how we *would* do it: > + # # Use the subvol's default 0/$id qgroup > + # eval { > + # # This call should happen at storage creation instead and the= refore governed by a > + # # configuration option! > + # # $class->btrfs_cmd(['quota', 'enable', $subvol]); > + # my $id =3D $class->btrfs_get_subvol_id($subvol); > + # $class->btrfs_cmd(['qgroup', 'limit', "${size}k", "0/$id", $s= ubvol]); > + # }; > + } elsif ($fmt eq 'raw') { > + sysopen my $fh, $path, O_WRONLY | O_CREAT | O_EXCL > + or die "failed to create raw file '$path' - $!\n"; > + chattr($fh, ~FS_NOCOW_FL, FS_NOCOW_FL); > + truncate($fh, $size * 1024) > + or die "failed to set file size for '$path' - $!\n"; > + close($fh); > + } else { > + die "internal format error (format =3D $fmt)\n"; > + } > + }; > + > + if (my $err =3D $@) { > + eval { $class->btrfs_cmd(['subvolume', 'delete', '--', $subvol]); }; > + warn $@ if $@; > + die $err; > + } > + > + return "$vmid/$name"; > +} > + > +# Same as btrfsprogs does: > +my sub path_is_subvolume : prototype($) { > + my ($path) =3D @_; > + my @stat =3D stat($path) > + or die "stat failed on '$path' - $!\n"; > + my ($ino, $mode) =3D @stat[1, 2]; > + return S_ISDIR($mode) && $ino =3D=3D BTRFS_FIRST_FREE_OBJECTID; > +} > + > +my $BTRFS_VOL_REGEX =3D qr/((?:vm|base|subvol)-\d+-disk-\d+(?:\.subvol)?= )(?:\@(\S+))$/; > + > +# Calls `$code->($volume, $name, $snapshot)` for each subvol in a direct= ory matching our volume > +# regex. > +my sub foreach_subvol : prototype($$) { > + my ($dir, $code) =3D @_; > + > + dir_glob_foreach($dir, $BTRFS_VOL_REGEX, sub { > + my ($volume, $name, $snapshot) =3D ($1, $2, $3); > + return if !path_is_subvolume("$dir/$volume"); > + $code->($volume, $name, $snapshot); > + }) > +} > + > +sub free_image { > + my ($class, $storeid, $scfg, $volname, $isBase, $_format) =3D @_; > + > + my (undef, undef, $vmid, undef, undef, undef, $format) =3D > + $class->parse_volname($volname); > + > + if ($format ne 'subvol' && $format ne 'raw') { > + return PVE::Storage::DirPlugin::free_image(@_); > + } > + > + my $path =3D $class->filesystem_path($scfg, $volname); > + > + my $subvol =3D $path; > + if ($format eq 'raw') { > + $subvol =3D raw_file_to_subvol($path); > + } > + > + my $dir =3D dirname($subvol); > + my @snapshot_vols; > + foreach_subvol($dir, sub { > + my ($volume, $name, $snapshot) =3D @_; > + return if !defined $snapshot; > + push @snapshot_vols, "$dir/$volume"; > + }); > + > + $class->btrfs_cmd(['subvolume', 'delete', '--', @snapshot_vols, $sub= vol]); > + # try to cleanup directory to not clutter storage with empty $vmid d= irs if > + # all images from a guest got deleted > + rmdir($dir); > + > + return undef; > +} > + > +# Currently not used because quotas clash with send/recv. > +# my sub btrfs_subvol_quota { > +# my ($class, $path) =3D @_; > +# my $id =3D '0/' . $class->btrfs_get_subvol_id($path); > +# my $search =3D qr/^\Q$id\E\s+(\d)+\s+\d+\s+(\d+)\s*$/; > +# my ($used, $size); > +# $class->btrfs_cmd(['qgroup', 'show', '--raw', '-rf', '--', $path],= sub { > +# return if defined($size); > +# if ($_[0] =3D~ $search) { > +# ($used, $size) =3D ($1, $2); > +# } > +# }); > +# if (!defined($size)) { > +# # syslog should include more information: > +# syslog('err', "failed to get subvolume size for: $path (id $id)"); > +# # UI should only see the last path component: > +# $path =3D~ s|^.*/||; > +# die "failed to get subvolume size for $path\n"; > +# } > +# return wantarray ? ($used, $size) : $size; > +# } > + > +sub volume_size_info { > + my ($class, $scfg, $storeid, $volname, $timeout) =3D @_; > + > + my $path =3D $class->filesystem_path($scfg, $volname); > + > + my $format =3D ($class->parse_volname($volname))[6]; > + > + if ($format eq 'subvol') { > + my $ctime =3D (stat($path))[10]; > + my ($used, $size) =3D (0, 0); > + #my ($used, $size) =3D btrfs_subvol_quota($class, $path); # uses wantar= ray > + return wantarray ? ($size, 'subvol', $used, undef, $ctime) : 1; > + } > + > + return PVE::Storage::Plugin::file_size_info($path, $timeout); > +} > + > +sub volume_resize { > + my ($class, $scfg, $storeid, $volname, $size, $running) =3D @_; > + > + my $format =3D ($class->parse_volname($volname))[6]; > + if ($format eq 'subvol') { > + my $path =3D $class->filesystem_path($scfg, $volname); > + my $id =3D '0/' . $class->btrfs_get_subvol_id($path); > + $class->btrfs_cmd(['qgroup', 'limit', '--', "${size}k", "0/$id", $path]= ); > + return undef; > + } > + > + return PVE::Storage::Plugin::volume_resize(@_); > +} > + > +sub volume_snapshot { > + my ($class, $scfg, $storeid, $volname, $snap) =3D @_; > + > + my ($name, $vmid, $format) =3D ($class->parse_volname($volname))[1,2= ,6]; > + if ($format ne 'subvol' && $format ne 'raw') { > + return PVE::Storage::Plugin::volume_snapshot(@_); > + } > + > + my $path =3D $class->filesystem_path($scfg, $volname); > + my $snap_path =3D $class->filesystem_path($scfg, $volname, $snap); > + > + if ($format eq 'raw') { > + $path =3D raw_file_to_subvol($path); > + $snap_path =3D raw_file_to_subvol($snap_path); > + } this is a repeating pattern (volname + optional snapshot name =3D> path(s)=20 based on format) > + > + my $snapshot_dir =3D $class->get_subdir($scfg, 'images') . "/$vmid"; > + mkpath $snapshot_dir; > + > + $class->btrfs_cmd(['subvolume', 'snapshot', '-r', '--', $path, $snap= _path]); > + return undef; > +} > + > +sub volume_rollback_is_possible { > + my ($class, $scfg, $storeid, $volname, $snap) =3D @_;=20 > + > + return 1;=20 snapshot creation was delegated to Plugin.pm for non-raw and non-subvol,=20 so if we don't drop that part the same logic would need to apply here.. > +} > + > +sub volume_snapshot_rollback { > + my ($class, $scfg, $storeid, $volname, $snap) =3D @_; > + > + my ($name, $format) =3D ($class->parse_volname($volname))[1,6]; > + > + if ($format ne 'subvol' && $format ne 'raw') { > + return PVE::Storage::Plugin::volume_snapshot_rollback(@_); > + } > + > + my $path =3D $class->filesystem_path($scfg, $volname); > + my $snap_path =3D $class->filesystem_path($scfg, $volname, $snap); > + > + if ($format eq 'raw') { > + $path =3D raw_file_to_subvol($path); > + $snap_path =3D raw_file_to_subvol($snap_path); > + } > + > + # Simple version would be: > + # rename old to temp > + # create new > + # on error rename temp back > + # But for atomicity in case the rename after create-failure *also* f= ails, we create the new > + # subvol first, then use RENAME_EXCHANGE,=20 > + my $tmp_path =3D "$path.tmp.$$"; > + $class->btrfs_cmd(['subvolume', 'snapshot', '--', $snap_path, $tmp_p= ath]); > + # The paths are absolute, so pass -1 as file descriptors. > + my $ok =3D PVE::Tools::renameat2(-1, $tmp_path, -1, $path, &PVE::Too= ls::RENAME_EXCHANGE); > + > + eval { $class->btrfs_cmd(['subvolume', 'delete', '--', $tmp_path]) }= ; > + warn "failed to remove '$tmp_path' subvolume: $@" if $@; > + > + if (!$ok) { > + die "failed to rotate '$tmp_path' into place at '$path' - $!\n"; > + } > + > + return undef; > +} > + > +sub volume_snapshot_delete { > + my ($class, $scfg, $storeid, $volname, $snap, $running) =3D @_; > + > + my ($name, $vmid, $format) =3D ($class->parse_volname($volname))[1,2= ,6]; > + > + if ($format ne 'subvol' && $format ne 'raw') { > + return PVE::Storage::Plugin::volume_snapshot_delete(@_); > + } > + > + my $path =3D $class->filesystem_path($scfg, $volname, $snap); > + > + if ($format eq 'raw') { > + $path =3D raw_file_to_subvol($path); > + } > + > + $class->btrfs_cmd(['subvolume', 'delete', '--', $path]); > + > + return undef; > +} > + > +sub volume_has_feature { > + my ($class, $scfg, $feature, $storeid, $volname, $snapname, $running= ) =3D @_; > + > + my $features =3D { > + snapshot =3D> { > + current =3D> { qcow2 =3D> 1, raw =3D> 1, subvol =3D> 1 }, > + snap =3D> { qcow2 =3D> 1, raw =3D> 1, subvol =3D> 1 } > + }, > + clone =3D> { > + base =3D> { qcow2 =3D> 1, raw =3D> 1, subvol =3D> 1, vmdk =3D> 1 }, > + current =3D> { raw =3D> 1 }, > + snap =3D> { raw =3D> 1 }, > + }, > + template =3D> { current =3D> { qcow2 =3D> 1, raw =3D> 1, vmdk =3D> 1, s= ubvol =3D> 1 } }, > + copy =3D> { > + base =3D> { qcow2 =3D> 1, raw =3D> 1, subvol =3D> 1, vmdk =3D> 1 }, > + current =3D> { qcow2 =3D> 1, raw =3D> 1, subvol =3D> 1, vmdk =3D> 1= }, > + snap =3D> { qcow2 =3D> 1, raw =3D> 1, subvol =3D> 1 }, > + }, > + sparseinit =3D> { base =3D> {qcow2 =3D> 1, raw =3D> 1, vmdk =3D> 1 }, > + current =3D> {qcow2 =3D> 1, raw =3D> 1, vmdk =3D> 1 } }, > + }; some of that is duplicated from the regular plugin, so if we don't drop=20 the qcow2/.. support here we need to forward those based on $format.. > + > + my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $format) = =3D > + $class->parse_volname($volname); > + > + my $key =3D undef; > + if ($snapname) { > + $key =3D 'snap'; > + } else { > + $key =3D $isBase ? 'base' : 'current'; > + } > + > + return 1 if defined($features->{$feature}->{$key}->{$format}); > + > + return undef; > +} > + > +sub list_images { > + my ($class, $storeid, $scfg, $vmid, $vollist, $cache) =3D @_; > + my $imagedir =3D $class->get_subdir($scfg, 'images'); > + > + my $res =3D []; > + > + # Copied from Plugin.pm, with file_size_info calls adapted: > + foreach my $fn (<$imagedir/[0-9][0-9]*/*>) { > + # different to in Plugin.pm the regex below also excludes '@' as valid = file name > + next if $fn !~ m@^(/.+/(\d+)/([^/\@.]+(?:\.(qcow2|vmdk|subvol))?))$@; > + $fn =3D $1; # untaint > + > + my $owner =3D $2; > + my $name =3D $3; > + my $ext =3D $4; > + > + next if !$vollist && defined($vmid) && ($owner ne $vmid); > + > + my $volid =3D "$storeid:$owner/$name"; > + my ($size, $format, $used, $parent, $ctime); > + > + if (!$ext) { # raw > + $volid .=3D '.raw'; > + ($size, $format, $used, $parent, $ctime) =3D PVE::Storage::Plugin::= file_size_info("$fn/disk.raw"); > + } elsif ($ext eq 'subvol') { > + ($used, $size) =3D (0, 0); > + #($used, $size) =3D btrfs_subvol_quota($class, $fn); > + $format =3D 'subvol'; > + } else { > + ($size, $format, $used, $parent, $ctime) =3D PVE::Storage::Plugin::= file_size_info($fn); > + } > + next if !($format && defined($size)); > + > + if ($vollist) { > + next if ! grep { $_ eq $volid } @$vollist; > + } > + > + my $info =3D { > + volid =3D> $volid, format =3D> $format, > + size =3D> $size, vmid =3D> $owner, used =3D> $used, parent =3D> $pa= rent, > + }; > + > + $info->{ctime} =3D $ctime if $ctime; > + > + push @$res, $info; > + } > + > + return $res; > +} > + > +# For now we don't implement `btrfs send/recv` as it needs some updates = to our import/export API > +# first! > + > +sub volume_export_formats { > + return PVE::Storage::DirPlugin::volume_export_formats(@_); > +} > + > +sub volume_export { > + return PVE::Storage::DirPlugin::volume_export(@_); > +} > + > +sub volume_import_formats { > + return PVE::Storage::DirPlugin::volume_import_formats(@_); > +} > + > +sub volume_import { > + return PVE::Storage::DirPlugin::volume_import(@_); > +} > + > +1 > diff --git a/PVE/Storage/Makefile b/PVE/Storage/Makefile > index 91b9238..857c485 100644 > --- a/PVE/Storage/Makefile > +++ b/PVE/Storage/Makefile > @@ -12,6 +12,7 @@ SOURCES=3D \ > ZFSPoolPlugin.pm \ > ZFSPlugin.pm \ > PBSPlugin.pm \ > + BTRFSPlugin.pm \ > LvmThinPlugin.pm > =20 > .PHONY: install > --=20 > 2.30.2 >=20 >=20 >=20 > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel >=20 >=20 >=20