From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <f.gruenbichler@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id CEC8D7148E
for <pve-devel@lists.proxmox.com>; Thu, 10 Jun 2021 14:41:03 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id B65162F6F9
for <pve-devel@lists.proxmox.com>; Thu, 10 Jun 2021 14:40:33 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
[94.136.29.106])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS id AB84C2F6EE
for <pve-devel@lists.proxmox.com>; Thu, 10 Jun 2021 14:40:31 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 7F78842B97
for <pve-devel@lists.proxmox.com>; Thu, 10 Jun 2021 14:40:31 +0200 (CEST)
Date: Thu, 10 Jun 2021 14:40:22 +0200
From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= <f.gruenbichler@proxmox.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
References: <20210609131852.167416-1-w.bumiller@proxmox.com>
<20210609131852.167416-4-w.bumiller@proxmox.com>
In-Reply-To: <20210609131852.167416-4-w.bumiller@proxmox.com>
MIME-Version: 1.0
User-Agent: astroid/0.15.0 (https://github.com/astroidmail/astroid)
Message-Id: <1623328690.owhbf7mh8s.astroid@nora.none>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.613 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [zfspoolplugin.pm, lvmthinplugin.pm, storage.pm, btrfsplugin.pm,
pbsplugin.pm, plugin.pm, zfsplugin.pm, proxmox.com]
URI_NOVOWEL 0.5 URI hostname has long non-vowel sequence
Subject: Re: [pve-devel] [PATCH storage 2/4] add BTRFS storage plugin
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Thu, 10 Jun 2021 12:41:03 -0000
On June 9, 2021 3:18 pm, Wolfgang Bumiller wrote:
> This is mostly the same as a directory storage, with 2 major
> differences:
>=20
> * 'subvol' volumes are actual btrfs subvolumes and therefore
> allow snapshots
> * 'raw' files are placed *into* a subvolume and therefore
> also allow snapshots, the raw file for volume
> `btrstore:100/vm-100-disk-1.raw` can be found under
> `$path/images/100/vm-100-disk-1/disk.raw`
> * snapshots add an '@name' suffix to the subvolume's name,
> so snapshot 'foo' of the above would be found under
> `$path/images/100/vm-100-disk-1@foo/disk.raw`
>=20
> Note that qgroups aren't included in btrfs-send streams,
> therefore for now we will only be using *unsized* subvolumes
> for containers and place a regular raw+ext4 file for sized
> containers.
> We could extend the import/export stream format to include
> the information at the front (similar to how we do the
> "tar+size" format, but we need to include the size of all
> the contained snapshots as well, since they can technically
> change). (But before enabling quotas we should do some
> performance testing on bigger file systems with multiple
> snapshots as there are quite a few reports of the fs slowing
> down considerably in such scenarios).
this should probably be mentioned prominently in the docs somewhere,=20
else it could cause quite some confusion. also is ext4 in raw image on=20
btrfs really faster than btrfs with quotas? Oo we really should do some=20
benchmarks first before going down either route..
>=20
> Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
> ---
> PVE/Storage.pm | 2 +
> PVE/Storage/BTRFSPlugin.pm | 616 +++++++++++++++++++++++++++++++++++++
> PVE/Storage/Makefile | 1 +
> 3 files changed, 619 insertions(+)
> create mode 100644 PVE/Storage/BTRFSPlugin.pm
>=20
> diff --git a/PVE/Storage.pm b/PVE/Storage.pm
> index aa36bad..f9f8d16 100755
> --- a/PVE/Storage.pm
> +++ b/PVE/Storage.pm
> @@ -38,6 +38,7 @@ use PVE::Storage::GlusterfsPlugin;
> use PVE::Storage::ZFSPoolPlugin;
> use PVE::Storage::ZFSPlugin;
> use PVE::Storage::PBSPlugin;
> +use PVE::Storage::BTRFSPlugin;
> =20
> # Storage API version. Increment it on changes in storage API interface.
> use constant APIVER =3D> 8;
> @@ -60,6 +61,7 @@ PVE::Storage::GlusterfsPlugin->register();
> PVE::Storage::ZFSPoolPlugin->register();
> PVE::Storage::ZFSPlugin->register();
> PVE::Storage::PBSPlugin->register();
> +PVE::Storage::BTRFSPlugin->register();
> =20
> # load third-party plugins
> if ( -d '/usr/share/perl5/PVE/Storage/Custom' ) {
> diff --git a/PVE/Storage/BTRFSPlugin.pm b/PVE/Storage/BTRFSPlugin.pm
> new file mode 100644
> index 0000000..733be6e
> --- /dev/null
> +++ b/PVE/Storage/BTRFSPlugin.pm
> @@ -0,0 +1,616 @@
> +package PVE::Storage::BTRFSPlugin;
> +
> +use strict;
> +use warnings;
> +
> +use base qw(PVE::Storage::Plugin);
> +
> +use Fcntl qw(S_ISDIR);
> +use File::Basename qw(dirname);
> +use File::Path qw(mkpath);
> +use IO::Dir;
> +
> +use PVE::Tools qw(run_command);
> +
> +use PVE::Storage::DirPlugin;
> +
> +use constant {
> + BTRFS_FIRST_FREE_OBJECTID =3D> 256,
> +};
> +
> +# Configuration (similar to DirPlugin)
> +
> +sub type {
> + return 'btrfs';
> +}
> +
> +sub plugindata {
> + return {
> + content =3D> [
> + {
> + images =3D> 1,
> + rootdir =3D> 1,
> + vztmpl =3D> 1,
> + iso =3D> 1,
> + backup =3D> 1,
> + snippets =3D> 1,
> + none =3D> 1,
> + },
> + { images =3D> 1, rootdir =3D> 1 },
> + ],
> + format =3D> [ { raw =3D> 1, qcow2 =3D> 1, vmdk =3D> 1, subvol =3D> 1 },=
'raw', ],
> + };
> +}
> +
> +sub options {
> + return {
> + path =3D> { fixed =3D> 1 },
> + nodes =3D> { optional =3D> 1 },
> + shared =3D> { optional =3D> 1 },
> + disable =3D> { optional =3D> 1 },
> + maxfiles =3D> { optional =3D> 1 },
> + content =3D> { optional =3D> 1 },
> + format =3D> { optional =3D> 1 },
> + is_mountpoint =3D> { optional =3D> 1 },
> + # TODO: The new variant of mkdir with `populate` vs `create`...
> + };
> +}
> +
> +# Storage implementation
> +#
> +# We use the same volume names are directory plugins, but map *raw* disk=
image file names into a
> +# subdirectory.
> +#
> +# `vm-VMID-disk-ID.raw`
> +# -> `images/VMID/vm-VMID-disk-ID/disk.raw`
> +# where the `vm-VMID-disk-ID/` subdirectory is a btrfs subvolume
> +
> +# Reuse `DirPlugin`'s `check_config`. This simply checks for invalid pat=
hs.
> +sub check_config {
> + my ($self, $sectionId, $config, $create, $skipSchemaCheck) =3D @_;
> + return PVE::Storage::DirPlugin::check_config($self, $sectionId, $con=
fig, $create, $skipSchemaCheck);
> +}
> +
> +sub activate_storage {
> + my ($class, $storeid, $scfg, $cache) =3D @_;
> + return PVE::Storage::DirPlugin::activate_storage($class, $storeid, $=
scfg, $cache);
> +}
> +
> +sub status {
> + my ($class, $storeid, $scfg, $cache) =3D @_;
> + return PVE::Storage::DirPlugin::status($class, $storeid, $scfg, $cac=
he);
> +}
> +
> +# TODO: sub get_volume_notes {}
> +
> +# TODO: sub update_volume_notes {}
> +
> +# croak would not include the caller from within this module
> +sub __error {
> + my ($msg) =3D @_;
> + my (undef, $f, $n) =3D caller(1);
> + die "$msg at $f: $n\n";
> +}
> +
> +# Given a name (eg. `vm-VMID-disk-ID.raw`), take the part up to the form=
at suffix as the name of
> +# the subdirectory (subvolume).
> +sub raw_name_to_dir($) {
> + my ($raw) =3D @_;
> +
> + # For the subvolume directory Strip the `.<format>` suffix:
> + if ($raw =3D~ /^(.*)\.raw$/) {
> + return $1;
> + }
> +
> + __error "internal error: bad disk name: $raw";
> +}
> +
> +sub raw_file_to_subvol($) {
> + my ($file) =3D @_;
> +
> + if ($file =3D~ m|^(.*)/disk\.raw$|) {
> + return "$1";
> + }
> +
> + __error "internal error: bad raw path: $file";
> +}
> +
> +sub filesystem_path {
> + my ($class, $scfg, $volname, $snapname) =3D @_;
> +
> + my ($vtype, $name, $vmid, undef, undef, $isBase, $format) =3D
> + $class->parse_volname($volname);
> +
> + my $path =3D $class->get_subdir($scfg, $vtype);
> +
> + $path .=3D "/$vmid" if $vtype eq 'images';
> +
> + if ($format eq 'raw') {
> + my $dir =3D raw_name_to_dir($name);
> + if ($snapname) {
> + $dir .=3D "\@$snapname";
> + }
> + $path .=3D "/$dir/disk.raw";
> + } elsif ($format eq 'subvol') {
> + $path .=3D "/$name";
> + if ($snapname) {
> + $path .=3D "\@$snapname";
> + }
> + } else {
> + $path .=3D "/$name";
> + }
> +
> + return wantarray ? ($path, $vmid, $vtype) : $path;
> +}
> +
> +sub btrfs_cmd {
> + my ($class, $cmd, $outfunc) =3D @_;
> +
> + my $msg =3D '';
> + my $func;
> + if (defined($outfunc)) {
> + $func =3D sub {
> + my $part =3D &$outfunc(@_);
> + $msg .=3D $part if defined($part);
> + };
> + } else {
> + $func =3D sub { $msg .=3D "$_[0]\n" };
> + }
> + run_command(['btrfs', '-q', @$cmd], errmsg =3D> 'btrfs error', outfu=
nc =3D> $func);
> +
> + return $msg;
> +}
> +
> +sub btrfs_get_subvol_id {
> + my ($class, $path) =3D @_;
> + my $info =3D $class->btrfs_cmd(['subvolume', 'show', '--', $path]);
> + if ($info !~ /^\s*(?:Object|Subvolume) ID:\s*(\d+)$/m) {
> + die "failed to get btrfs subvolume ID from: $info\n";
> + }
> + return $1;
> +}
> +
> +sub create_base {
> + my ($class, $storeid, $scfg, $volname) =3D @_;
> +
> + my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $format) =
=3D
> + $class->parse_volname($volname);
> +
> + my $newname =3D $name;
> + $newname =3D~ s/^vm-/base-/;
> +
> + # If we're not working with a 'raw' file, which is the only thing th=
at's "different" for btrfs,
> + # or a subvolume, we forward to the DirPlugin
> + if ($format ne 'raw' && $format ne 'subvol') {
> + return PVE::Storage::DirPlugin::create_base(@_);
> + }
> +
> + my $path =3D $class->filesystem_path($scfg, $volname);
> + my $newvolname =3D $basename ? "$basevmid/$basename/$vmid/$newname" =
: "$vmid/$newname";
> + my $newpath =3D $class->filesystem_path($scfg, $newvolname);
> +
> + my $subvol =3D $path;
> + my $newsubvol =3D $newpath;
> + if ($format eq 'raw') {
> + $subvol =3D raw_file_to_subvol($subvol);
> + $newsubvol =3D raw_file_to_subvol($newsubvol);
> + }
> +
> + rename($subvol, $newsubvol)
> + || die "rename '$subvol' to '$newsubvol' failed - $!\n";
> + eval { $class->btrfs_cmd(['property', 'set', $newsubvol, 'ro', 'true=
']) };
> + warn $@ if $@;
> +
> + return $newvolname;
> +}
> +
> +sub clone_image {
> + my ($class, $scfg, $storeid, $volname, $vmid, $snap) =3D @_;
> +
> + my ($vtype, $basename, $basevmid, undef, undef, $isBase, $format) =
=3D
> + $class->parse_volname($volname);
> +
> + # If we're not working with a 'raw' file, which is the only thing th=
at's "different" for btrfs,
> + # or a subvolume, we forward to the DirPlugin
> + if ($format ne 'raw' && $format ne 'subvol') {
> + return PVE::Storage::DirPlugin::clone_image(@_);
> + }
> +
> + my $imagedir =3D $class->get_subdir($scfg, 'images');
> + $imagedir .=3D "/$vmid";
> + mkpath $imagedir;
> +
> + my $path =3D $class->filesystem_path($scfg, $volname);
> + my $newname =3D $class->find_free_diskname($storeid, $scfg, $vmid, $=
format, 1);
> +
> + # For btrfs subvolumes we don't actually need the "link":
> + #my $newvolname =3D "$basevmid/$basename/$vmid/$newname";
> + my $newvolname =3D "$vmid/$newname";
> + my $newpath =3D $class->filesystem_path($scfg, $newvolname);
> +
> + my $subvol =3D $path;
> + my $newsubvol =3D $newpath;
> + if ($format eq 'raw') {
> + $subvol =3D raw_file_to_subvol($subvol);
> + $newsubvol =3D raw_file_to_subvol($newsubvol);
> + }
> +
> + $class->btrfs_cmd(['subvolume', 'snapshot', '--', $subvol, $newsubvo=
l]);
> +
> + return $newvolname;
> +}
> +
> +sub alloc_image {
> + my ($class, $storeid, $scfg, $vmid, $fmt, $name, $size) =3D @_;
> +
> + if ($fmt ne 'raw' && $fmt ne 'subvol') {
> + return PVE::Storage::DirPlugin::alloc_image(@_);
> + }
> +
> + # From Plugin.pm:
> +
> + my $imagedir =3D $class->get_subdir($scfg, 'images') . "/$vmid";
> +
> + mkpath $imagedir;
> +
> + $name =3D $class->find_free_diskname($storeid, $scfg, $vmid, $fmt, 1=
) if !$name;
> +
> + my (undef, $tmpfmt) =3D PVE::Storage::Plugin::parse_name_dir($name);
> +
> + die "illegal name '$name' - wrong extension for format ('$tmpfmt !=
=3D '$fmt')\n"
> + if $tmpfmt ne $fmt;
> +
> + # End copy from Plugin.pm
> +
> + my $subvol =3D "$imagedir/$name";
> + # .raw is not part of the directory name
> + $subvol =3D~ s/\.raw$//;
> +
> + die "disk image '$subvol' already exists\n" if -e $subvol;
> +
> + my $path;
> + if ($fmt eq 'raw') {
> + $path =3D "$subvol/disk.raw";
> + }
> +
> + $class->btrfs_cmd(['subvolume', 'create', '--', $subvol]);
> +
> + if ($fmt eq 'subvol' && !!$size) {
> + # NOTE: `btrfs send/recv` actually drops quota information so supportin=
g subvolumes with
> + # quotas doesn't play nice with send/recv.
> + die "btrfs quotas are currently not supported, use an unsized subvolume=
or a raw file\n";
> +
> + # This is how we *would* do it:
> + # # Use the subvol's default 0/$id qgroup
> + # eval {
> + # # This call should happen at storage creation instead and therefo=
re governed by a
> + # # configuration option!
> + # # $class->btrfs_cmd(['quota', 'enable', $subvol]);
> + # my $id =3D $class->btrfs_get_subvol_id($subvol);
> + # $class->btrfs_cmd(['qgroup', 'limit', "${size}k", "0/$id", $subvo=
l]);
> + # };
> + } elsif ($fmt eq 'raw') {
> + # From Plugin.pm (minus the qcow2 part):
> + my $cmd =3D ['/usr/bin/qemu-img', 'create'];
> +
> + push @$cmd, '-f', $fmt, $path, "${size}K";
> +
> + eval { run_command($cmd, errmsg =3D> "unable to create image"); };
> + } else {
> + # be clear that after this if/else we're handling the result of an eval=
block:
> + $@ =3D undef;
> + }
> +
> + if (my $err =3D $@) {
> + eval { $class->btrfs_cmd(['subvolume', 'delete', '--', $subvol]); };
> + warn $@ if $@;
> + die $err;
> + }
> +
> + return "$vmid/$name";
> +}
> +
> +# Same as btrfsprogs does:
> +my sub path_is_subvolume : prototype($) {
> + my ($path) =3D @_;
> + my @stat =3D stat($path)
> + or die "stat failed on '$path' - $!\n";
> + my ($ino, $mode) =3D @stat[1, 2];
> + return S_ISDIR($mode) && $ino =3D=3D BTRFS_FIRST_FREE_OBJECTID;
> +}
> +
> +my $BTRFS_VOL_REGEX =3D qr/((?:vm|base|subvol)-\d+-disk-\d+(?:\.subvol)?=
)(?:\@(\S+))$/;
> +
> +# Calls `$code->($volume, $name, $snapshot)` for each subvol in a direct=
ory matching our volume
> +# regex.
> +my sub foreach_subvol : prototype($$) {
> + my ($dir, $code) =3D @_;
> +
> + dir_glob_foreach($dir, $BTRFS_VOL_REGEX, sub {
> + my ($volume, $name, $snapshot) =3D ($1, $2, $3);
> + return if !path_is_subvolume("$dir/$volume");
> + $code->($volume, $name, $snapshot);
> + })
> +}
> +
> +sub free_image {
> + my ($class, $storeid, $scfg, $volname, $isBase, $_format) =3D @_;
> +
> + my (undef, undef, $vmid, undef, undef, undef, $format) =3D
> + $class->parse_volname($volname);
> +
> + if ($format ne 'subvol' && $format ne 'raw') {
> + return PVE::Storage::DirPlugin::free_image(@_);
> + }
> +
> + my $path =3D $class->filesystem_path($scfg, $volname);
> +
> + my $subvol =3D $path;
> + if ($format eq 'raw') {
> + $subvol =3D raw_file_to_subvol($path);
> + }
> +
> + my $dir =3D dirname($subvol);
> + my @snapshot_vols;
> + foreach_subvol($dir, sub {
> + my ($volume, $name, $snapshot) =3D @_;
> + return if !defined $snapshot;
> + push @snapshot_vols, "$dir/$volume";
> + });
> +
> + $class->btrfs_cmd(['subvolume', 'delete', '--', @snapshot_vols, $sub=
vol]);
> + # try to cleanup directory to not clutter storage with empty $vmid d=
irs if
> + # all images from a guest got deleted
> + rmdir($dir);
> +
> + return undef;
> +}
> +
> +# Currently not used because quotas clash with send/recv.
> +# my sub btrfs_subvol_quota {
> +# my ($class, $path) =3D @_;
> +# my $id =3D '0/' . $class->btrfs_get_subvol_id($path);
> +# my $search =3D qr/^\Q$id\E\s+(\d)+\s+\d+\s+(\d+)\s*$/;
> +# my ($used, $size);
> +# $class->btrfs_cmd(['qgroup', 'show', '--raw', '-rf', '--', $path],=
sub {
> +# return if defined($size);
> +# if ($_[0] =3D~ $search) {
> +# ($used, $size) =3D ($1, $2);
> +# }
> +# });
> +# if (!defined($size)) {
> +# # syslog should include more information:
> +# syslog('err', "failed to get subvolume size for: $path (id $id)");
> +# # UI should only see the last path component:
> +# $path =3D~ s|^.*/||;
> +# die "failed to get subvolume size for $path\n";
> +# }
> +# return wantarray ? ($used, $size) : $size;
> +# }
> +
> +sub volume_size_info {
> + my ($class, $scfg, $storeid, $volname, $timeout) =3D @_;
> +
> + my $path =3D $class->filesystem_path($scfg, $volname);
> +
> + my $format =3D ($class->parse_volname($volname))[6];
> +
> + if ($format eq 'subvol') {
> + my $ctime =3D (stat($path))[10];
> + my ($used, $size) =3D (0, 0);
> + #my ($used, $size) =3D btrfs_subvol_quota($class, $path); # uses wantar=
ray
> + return wantarray ? ($size, 'subvol', $used, undef, $ctime) : 1;
> + }
> +
> + return PVE::Storage::Plugin::file_size_info($path, $timeout);
> +}
> +
> +sub volume_resize {
> + my ($class, $scfg, $storeid, $volname, $size, $running) =3D @_;
> +
> + my $format =3D ($class->parse_volname($volname))[6];
> + if ($format eq 'subvol') {
> + my $path =3D $class->filesystem_path($scfg, $volname);
> + my $id =3D '0/' . $class->btrfs_get_subvol_id($path);
> + $class->btrfs_cmd(['qgroup', 'limit', '--', "${size}k", "0/$id", $path]=
);
> + return undef;
> + }
> +
> + return PVE::Storage::Plugin::volume_resize(@_);
> +}
> +
> +sub volume_snapshot {
> + my ($class, $scfg, $storeid, $volname, $snap) =3D @_;
> +
> + my ($name, $vmid, $format) =3D ($class->parse_volname($volname))[1,2=
,6];
> + if ($format ne 'subvol' && $format ne 'raw') {
> + return PVE::Storage::Plugin::volume_snapshot(@_);
> + }
> +
> + my $path =3D $class->filesystem_path($scfg, $volname);
> + my $snap_path =3D $class->filesystem_path($scfg, $volname, $snap);
> +
> + if ($format eq 'raw') {
> + $path =3D raw_file_to_subvol($path);
> + $snap_path =3D raw_file_to_subvol($snap_path);
> + }
> +
> + my $snapshot_dir =3D $class->get_subdir($scfg, 'images') . "/$vmid";
> + mkpath $snapshot_dir;
> +
> + $class->btrfs_cmd(['subvolume', 'snapshot', '-r', '--', $path, $snap=
_path]);
> + return undef;
> +}
> +
> +sub volume_rollback_is_possible {
> + my ($class, $scfg, $storeid, $volname, $snap) =3D @_;=20
> +
> + return 1;=20
> +}
> +
> +sub volume_snapshot_rollback {
> + my ($class, $scfg, $storeid, $volname, $snap) =3D @_;
> +
> + my ($name, $format) =3D ($class->parse_volname($volname))[1,6];
> +
> + if ($format ne 'subvol' && $format ne 'raw') {
> + return PVE::Storage::Plugin::volume_snapshot_rollback(@_);
> + }
> +
> + my $path =3D $class->filesystem_path($scfg, $volname);
> + my $snap_path =3D $class->filesystem_path($scfg, $volname, $snap);
> +
> + if ($format eq 'raw') {
> + $path =3D raw_file_to_subvol($path);
> + $snap_path =3D raw_file_to_subvol($snap_path);
> + }
> +
> + # Simple version would be:
> + # rename old to temp
> + # create new
> + # on error rename temp back
> + # But for atomicity in case the rename after create-failure *also* f=
ails, we create the new
> + # subvol first, then use RENAME_EXCHANGE,=20
> + my $tmp_path =3D "$path.tmp.$$";
> + $class->btrfs_cmd(['subvolume', 'snapshot', '--', $snap_path, $tmp_p=
ath]);
> + # The paths are absolute, so pass -1 as file descriptors.
> + my $ok =3D PVE::Tools::renameat2(-1, $tmp_path, -1, $path, &PVE::Too=
ls::RENAME_EXCHANGE);
> +
> + eval { $class->btrfs_cmd(['subvolume', 'delete', '--', $tmp_path]) }=
;
> + warn "failed to remove '$tmp_path' subvolume: $@" if $@;
> +
> + if (!$ok) {
> + die "failed to rotate '$tmp_path' into place at '$path' - $!\n";
> + }
> +
> + return undef;
> +}
> +
> +sub volume_snapshot_delete {
> + my ($class, $scfg, $storeid, $volname, $snap, $running) =3D @_;
> +
> + my ($name, $vmid, $format) =3D ($class->parse_volname($volname))[1,2=
,6];
> +
> + if ($format ne 'subvol' && $format ne 'raw') {
> + return PVE::Storage::Plugin::volume_snapshot_delete(@_);
> + }
> +
> + my $path =3D $class->filesystem_path($scfg, $volname, $snap);
> +
> + if ($format eq 'raw') {
> + $path =3D raw_file_to_subvol($path);
> + }
> +
> + $class->btrfs_cmd(['subvolume', 'delete', '--', $path]);
> +
> + return undef;
> +}
> +
> +sub volume_has_feature {
> + my ($class, $scfg, $feature, $storeid, $volname, $snapname, $running=
) =3D @_;
> +
> + my $features =3D {
> + snapshot =3D> {
> + current =3D> { qcow2 =3D> 1, raw =3D> 1, subvol =3D> 1 },
> + snap =3D> { qcow2 =3D> 1, raw =3D> 1, subvol =3D> 1 }
> + },
> + clone =3D> {
> + base =3D> { qcow2 =3D> 1, raw =3D> 1, subvol =3D> 1, vmdk =3D> 1 },
> + current =3D> { raw =3D> 1 },
> + snap =3D> { raw =3D> 1 },
> + },
> + template =3D> { current =3D> { qcow2 =3D> 1, raw =3D> 1, vmdk =3D> 1, s=
ubvol =3D> 1 } },
> + clone =3D> {
> + base =3D> { qcow2 =3D> 1, raw =3D> 1, subvol =3D> 1, vmdk =3D> 1 },
> + current =3D> { qcow2 =3D> 1, raw =3D> 1, subvol =3D> 1, vmdk =3D> 1=
},
> + snap =3D> { qcow2 =3D> 1, raw =3D> 1, subvol =3D> 1 },
> + },
> + sparseinit =3D> { base =3D> {qcow2 =3D> 1, raw =3D> 1, vmdk =3D> 1 },
> + current =3D> {qcow2 =3D> 1, raw =3D> 1, vmdk =3D> 1 } },
> + };
> +
> + my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $format) =
=3D
> + $class->parse_volname($volname);
> +
> + my $key =3D undef;
> + if ($snapname) {
> + $key =3D 'snap';
> + } else {
> + $key =3D $isBase ? 'base' : 'current';
> + }
> +
> + return 1 if defined($features->{$feature}->{$key}->{$format});
> +
> + return undef;
> +}
> +
> +sub list_images {
> + my ($class, $storeid, $scfg, $vmid, $vollist, $cache) =3D @_;
> + my $imagedir =3D $class->get_subdir($scfg, 'images');
> +
> + my $res =3D [];
> +
> + # Copied from Plugin.pm, with file_size_info calls adapted:
> + foreach my $fn (<$imagedir/[0-9][0-9]*/*>) {
> + # different to in Plugin.pm the regex below also excludes '@' as valid =
file name
> + next if $fn !~ m@^(/.+/(\d+)/([^/\@.]+(?:\.(qcow2|vmdk|subvol))?))$@;
> + $fn =3D $1; # untaint
> +
> + my $owner =3D $2;
> + my $name =3D $3;
> + my $ext =3D $4;
> +
> + next if !$vollist && defined($vmid) && ($owner ne $vmid);
> +
> + my $volid =3D "$storeid:$owner/$name";
> + my ($size, $format, $used, $parent, $ctime);
> +
> + if (!$ext) { # raw
> + $volid .=3D '.raw';
> + ($size, $format, $used, $parent, $ctime) =3D PVE::Storage::Plugin::=
file_size_info("$fn/disk.raw");
> + } elsif ($ext eq 'subvol') {
> + ($used, $size) =3D (0, 0);
> + #($used, $size) =3D btrfs_subvol_quota($class, $fn);
> + $format =3D 'subvol';
> + } else {
> + ($size, $format, $used, $parent, $ctime) =3D PVE::Storage::Plugin::=
file_size_info($fn);
> + }
> + next if !($format && defined($size));
> +
> + if ($vollist) {
> + next if ! grep { $_ eq $volid } @$vollist;
> + }
> +
> + my $info =3D {
> + volid =3D> $volid, format =3D> $format,
> + size =3D> $size, vmid =3D> $owner, used =3D> $used, parent =3D> $pa=
rent,
> + };
> +
> + $info->{ctime} =3D $ctime if $ctime;
> +
> + push @$res, $info;
> + }
> +
> + return $res;
> +}
> +
> +# For now we don't implement `btrfs send/recv` as it needs some updates =
to our import/export API
> +# first!
> +
> +sub volume_export_formats {
> + return PVE::Storage::DirPlugin::volume_export_formats(@_);
> +}
> +
> +sub volume_export {
> + return PVE::Storage::DirPlugin::volume_export(@_);
> +}
> +
> +sub volume_import_formats {
> + return PVE::Storage::DirPlugin::volume_import_formats(@_);
> +}
> +
> +sub volume_import {
> + return PVE::Storage::DirPlugin::volume_import(@_);
> +}
> +
> +1
> diff --git a/PVE/Storage/Makefile b/PVE/Storage/Makefile
> index 91b9238..857c485 100644
> --- a/PVE/Storage/Makefile
> +++ b/PVE/Storage/Makefile
> @@ -12,6 +12,7 @@ SOURCES=3D \
> ZFSPoolPlugin.pm \
> ZFSPlugin.pm \
> PBSPlugin.pm \
> + BTRFSPlugin.pm \
> LvmThinPlugin.pm
> =20
> .PHONY: install
> --=20
> 2.30.2
>=20
>=20
>=20
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>=20
>=20
>=20