From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id EE28B1FF130 for ; Mon, 20 Jul 2026 16:31:00 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 8A23421565; Mon, 20 Jul 2026 16:30:59 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784557809; x=1785162609; darn=lists.proxmox.com; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to:content-type; bh=bwj2+TIlkZDJNHAnWONxxIF9mzHH7g1OFq+rLWm7c+4=; b=lab9K0cpryp90bTWF2sryaukdvEd0F8W3rkIylVtd3qf+J6xrMFMb8+izkTUYJRyqv w3b3xCP4BkGgyZsP1jp42L8k/+GTEWjAJCQv4uKqmU8qWHiF0rRqM8dIYxy+ac2eI3uV 8bq43U/A1U/bpFtHu1Fbj1nHlRdXEHoei5aDMe4Y8cECsoYy9bxR9FgU+IMoF1IFWHCG knDyoj9tcL7Sc16n20EDe7ZNopdcaz4tI11MFs1flJjLlszP9iqXHfeP2XJ4AJK13Xav 5A+10MRFAjkgnwmIL57y1WF1MySCcZGRsotPBh4f2/NiO97F2HBf8T/6TM7lKWYL0rZL 1MEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784557809; x=1785162609; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=bwj2+TIlkZDJNHAnWONxxIF9mzHH7g1OFq+rLWm7c+4=; b=X5sAxVwntyFyVBCebRs6sc/t9eIladZgKAN/BQBt7xRnOiCYTfYss16EVBms2d8m6E w4jDrz3yW1K9K5pl445B0jKJQlx05MgLDUJ/1d+9fKSqmQ8RiuHupoCJzcKHa8FlRGQq 42xldlxJN/vEdiI2S9XW2MtZx94VgWbBAGXsQcp8urnz5lhLFdlX1/dRtOivHOLgjnd/ YC0esHmWFd0EPS3PZ9hkw4BDr3WMwOCdnm+BtZcJNiQZ6nYqPtTUyvsz1Gp5pqzZcrw9 R1guKML64CS88CfZqppLXRBPX3syECgQwoJo0zUep1lydXko7US+OVGh3Y0WcH/92NZu X/Kg== X-Gm-Message-State: AOJu0Yz8rj9jGral6zxcQRdgBsBpCST0usn7/N6KrbMP3U9cGwm3W+2l /92xp0DHgjMTcsLICWcIXzEjd+cL3mO/Ljr3wnzCHEWI5tUP7DnA17bL X-Gm-Gg: AfdE7cnvf2UvCOYCEKDZ8JG+dz3kNV4pLIR8U2VgZbO9cjxMoFh2yJK0b60l3mBur0g 3CnkYODJj8+hvPhpfGHRTYXadRaW0YT79IBfIjkkjgfrqPj5KFgwMvoAw+2jn45ZX7Fl5TeaunH 6PtlifkBd09oT9XC3Psl5ap8+/S0lXyi+iriGclYR5MJtCmZQMdtR52fUAY24RZ2UeKEp1R8efD E/ur4ZXb5MyEuKt1qtvtQsoVjNDXrF2PH0xiPWCQOl4rMUzx9j0V5Ss+AVaLP3o0sNbOcU1PG70 JA9hJlC/pNElmmJqSiB6nTj2rfI0V+1DY4rh5no/Zs7DIY+/Wv4zaGfQjYqHvHTe08qOxLbwph2 EvUXM7Y3usI+4525pw6oAuWUEVYbrdt9eaG94evqGEy7nqPjDrrQLqUB9DqdIrOuVGC8s/e9F X-Received: by 2002:a05:6300:95:b0:3a2:f7bd:a9a5 with SMTP id adf61e73a8af0-3c3ad95b577mr14996202637.38.1784557808739; Mon, 20 Jul 2026 07:30:08 -0700 (PDT) Date: Mon, 20 Jul 2026 07:30:08 -0700 (PDT) From: Ciro Iriarte To: pve-devel@lists.proxmox.com Subject: [RFC PATCH storage 3/5] dir: implement copy-offload via reflink (FICLONE) Message-ID: <20260720.3.copyoffload@cyruspy.gmail.com> In-Reply-To: <20260720.0.copyoffload@cyruspy.gmail.com> References: <20260720.0.copyoffload@cyruspy.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.067 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: LIYX5ATJABUOIZTXODXWWGLE5ITQQA5X X-Message-ID-Hash: LIYX5ATJABUOIZTXODXWWGLE5ITQQA5X X-MailFrom: cyruspy@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: A filesystem that supports FICLONE -- XFS with reflink=1, btrfs, or ZFS with block cloning -- can produce a full copy by sharing extents copy-on-write. The result is INDEPENDENT immediately: the extents are reference counted, so deleting the source does not affect the copy. That makes this the cheapest member of the 'copy-offload-atomic' class. There is no data movement and nothing to wait for, so copy_image_status() reports 'complete' on the first poll and the asynchronous machinery simply does not engage. Today a full clone on a directory storage byte-copies the whole image through qemu-img convert even when the filesystem underneath could share the extents for free. Three things are refused rather than half-done: - A qcow2 that HAS A BACKING FILE. A byte-identical copy of an overlay inherits the dependency: it passes qemu-img check and then breaks the moment the base is removed, which is exactly the "readable but not independent" failure the hook's contract exists to prevent. Only a real convert can flatten it, so the feature is not advertised for such volumes -- deliberately not merely rejected in prepare, because failing there would abort the clone instead of letting it take the normal host-side path. - Format conversion. FICLONE copies bytes; qcow2 -> raw needs qemu-img. - A source snapshot, and a target on a different filesystem. prepare() creates the target file rather than only choosing a name. The caller releases the storage lock between prepare and start, so a name that was merely chosen could be taken by a concurrent allocation, and the caller's rollback would then delete that other operation's volume. An empty file costs nothing. start() uses 'cp --reflink=always' so an unsupported filesystem fails loudly instead of silently degrading into a full byte copy that would block a caller which may be holding a guest frozen. Verified on a loop-backed XFS (reflink=1) by driving the hooks directly: the clone is byte-identical, status is 'complete' on the first poll, and the clone still matches after the source is deleted. The guards were checked too -- a snapshot source and a format conversion are both refused, a qcow2 with a backing file is not advertised, and a plain qcow2 is. Independence and zero space use were separately confirmed on XFS, btrfs and ZFS, where a 256 MiB clone added 0 MiB of allocation. Generated-By: Claude (https://claude.ai) Signed-off-by: Ciro Iriarte Co-Authored-By: Claude --- src/PVE/Storage/DirPlugin.pm | 140 +++++++++++++++++++++++++++++++++++ 1 file changed, 140 insertions(+) diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm index 80c4a03..cbc7357 100644 --- a/src/PVE/Storage/DirPlugin.pm +++ b/src/PVE/Storage/DirPlugin.pm @@ -8,6 +8,7 @@ use Encode qw(decode encode); use File::Path; use File::Spec; use IO::File; +use JSON; use POSIX; use PVE::Storage::Plugin; @@ -95,6 +96,8 @@ sub options { bwlimit => { optional => 1 }, preallocation => { optional => 1 }, 'snapshot-as-volume-chain' => { optional => 1, fixed => 1 }, + 'copy-offload' => { optional => 1 }, + 'copy-offload-timeout' => { optional => 1 }, }; } @@ -323,4 +326,141 @@ sub volume_qemu_snapshot_method { return $scfg->{'snapshot-as-volume-chain'} ? 'mixed' : 'qemu'; } + +# qemu_img_info() returns raw JSON text, so decode before use. Returns the backing +# filename, or undef when there is none / the image cannot be inspected. +my sub qcow2_backing_file { + my ($path) = @_; + my $json = eval { PVE::Storage::Common::qemu_img_info($path, undef, 10) }; + return undef if $@ || !$json; + my $info = eval { decode_json($json) }; + return undef if $@ || ref($info) ne 'HASH'; + return $info->{'backing-filename'}; +} + +sub volume_has_feature { + my ($class, $scfg, $feature, $storeid, $volname, $snapname, $running, $opts) = @_; + + if ($feature eq 'copy-offload-atomic') { + # reflink clones a whole file; there is no way to pick a snapshot out of one + return 0 if $snapname; + + my ($vtype, undef, undef, undef, undef, undef, $format) = + eval { $class->parse_volname($volname) }; + return 0 if $@ || !defined($vtype) || $vtype ne 'images'; + return 0 if $format ne 'raw' && $format ne 'qcow2'; + + # A qcow2 with a backing file cannot be flattened by a byte-identical copy, so + # do not advertise it -- otherwise the clone would fail in prepare instead of + # quietly taking the normal host-side path. + if ($format eq 'qcow2') { + my $path = eval { $class->filesystem_path($scfg, $volname) }; + return 0 if $@ || !defined($path); + return 0 if qcow2_backing_file($path); + } + + return 1; + } + + return $class->SUPER::volume_has_feature( + $scfg, $feature, $storeid, $volname, $snapname, $running, $opts, + ); +} + +# ---- storage-offloaded full copy via reflink ------------------------------------- +# +# A filesystem that supports FICLONE (XFS with reflink=1, btrfs, ZFS with block +# cloning) can produce a full copy by sharing extents copy-on-write. Unlike an RBD +# clone or a ZFS clone-from-snapshot, the result is INDEPENDENT straight away: the +# extents are reference counted, so deleting the source does not affect the copy. +# +# That makes this the cheapest possible member of the 'copy-offload-atomic' class -- +# instant, no extra space, and nothing to wait for. copy_image_status() reports +# 'complete' on the first poll because there is no background work. + +# Same filesystem? FICLONE cannot cross one, and the caller may be copying between +# two different storages that happen to be directories. +my sub same_filesystem { + my ($a, $b) = @_; + my $da = (stat($a))[0]; + my $db = (stat($b))[0]; + return defined($da) && defined($db) && $da == $db; +} + +sub copy_image_prepare { + my ( + $class, $scfg, $storeid, $volname, + $target_scfg, $target_storeid, $target_vmid, $snap, $opts, + ) = @_; + + die "copy offload cannot copy from a snapshot\n" if defined($snap); + + my ($vtype, undef, undef, undef, undef, undef, $format) = $class->parse_volname($volname); + die "copy offload only handles VM images, not '$vtype'\n" if $vtype ne 'images'; + + # FICLONE copies bytes; it cannot convert between formats. + my $target_format = $opts->{format} // $format; + die "copy offload cannot convert '$format' to '$target_format'\n" + if $target_format ne $format; + + my $path = $class->filesystem_path($scfg, $volname); + + # A qcow2 with a backing file is NOT independent, and a byte-identical copy of it + # inherits that dependency -- it would pass qemu-img check and then break when the + # base is removed. Only a real convert can flatten it. + die "copy offload cannot flatten '$volname': it has a backing file\n" + if $format eq 'qcow2' && qcow2_backing_file($path); + + my $target_dir = $class->get_subdir($target_scfg, 'images') . "/$target_vmid"; + mkpath $target_dir; + + die "copy offload requires source and target on the same filesystem\n" + if !same_filesystem($path, $target_dir); + + # the trailing 1 adds the format suffix; without it the volname does not parse + my $name = + $class->find_free_diskname($target_storeid, $target_scfg, $target_vmid, $format, 1); + + # Reserve the name by creating the file. The caller drops the storage lock between + # prepare and start, so a name that was merely chosen could be taken by a + # concurrent allocation -- and the caller's rollback would then delete somebody + # else's volume. An empty file costs nothing and makes the target freeable. + my $target_path = "$target_dir/$name"; + my $fh = IO::File->new($target_path, O_WRONLY | O_CREAT | O_EXCL, 0640) + or die "unable to reserve '$target_path' - $!\n"; + close($fh); + + return "$target_vmid/$name"; +} + +sub copy_image_start { + my ( + $class, $scfg, $storeid, $volname, + $target_scfg, $target_storeid, $target_volname, $snap, + ) = @_; + + my $src = $class->filesystem_path($scfg, $volname); + my $dst = $class->filesystem_path($target_scfg, $target_volname); + + # --reflink=always so an unsupported filesystem fails loudly rather than silently + # turning this into a full byte copy that blocks the caller -- which may be holding + # a guest frozen. + eval { PVE::Tools::run_command(['/bin/cp', '--reflink=always', '--', $src, $dst]) }; + if (my $err = $@) { + unlink($dst); + die "reflink copy of '$volname' failed - $err"; + } + + return; +} + +sub copy_image_status { + my ($class, $scfg, $storeid, $volname, $source) = @_; + + # FICLONE is synchronous and the extents are reference counted, so the copy is + # already independent of its source. Nothing to poll and nothing to clean up. + return { state => 'complete' }; +} + + 1; -- 2.54.0