* [RFC PATCH storage 1/5] storage: add asynchronous copy-offload hook for full copies
2026-07-20 14:29 [RFC PATCH storage, qemu-server 0/5] offload full clone to the storage backend Ciro Iriarte
@ 2026-07-20 14:29 ` Ciro Iriarte
2026-07-20 14:29 ` [RFC PATCH storage 2/5] rbd: implement copy-offload via snapshot + clone + flatten Ciro Iriarte
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Ciro Iriarte @ 2026-07-20 14:29 UTC (permalink / raw)
To: pve-devel
Lets a plugin perform a full copy (e.g. a full clone) on the storage backend instead
of reading and writing the data through the host with qemu-img.
Following the pve-devel discussion: the base methods die rather than returning undef
(they are only reached when a plugin advertised a copy-offload feature); the plugin
owns cleanup; it receives $target_vmid and the TARGET storage config, since a plugin
only sees its own per-instance scfg and cannot otherwise decide whether a
source/target pair is copyable; offload is a plain per-storage 'copy-offload' option;
and scope is two instances of the same plugin type, as the operation is opaque to PVE
and has no volume_export/import-style lowering.
The two capability classes are separate features rather than one feature with a class
argument, so volume_has_feature()'s interface is unchanged (Fiona's suggestion). A
caller that can drive both prefers atomic.
The hook is split into three phases, for two independent reasons:
copy_image_prepare() allocate the target - under the storage lock
copy_image_start() pin the source PIT, begin - inside a guest freeze
copy_image_status() poll until independent - outside both
It cannot be synchronous: the core calls into the plugin under
cluster_lock_storage(), and for shared storage that is a pmxcfs lock whose locked code
is killed by an alarm(60) in PVE::Cluster, while a full array copy runs for minutes to
hours. And allocation cannot happen inside a guest freeze, because cross-disk
consistency requires holding one freeze across every disk of a VM -- so only the
start, a single cheap backend call, runs frozen.
copy_image_start() receives the source $snapname. It is the call that fixes the point
in time, so a plugin that instead snapshotted the current image would silently copy
live data when a snapshot was requested. copy_image_status() likewise receives what
the copy was made FROM, so a plugin can clean up state it left on the source once the
target is independent -- after that, nothing else records where the copy came from.
'complete' means INDEPENDENT of the source, not merely readable: for backends whose
independence is asynchronous -- an RBD flatten, a ZFS clone still tied to its origin,
an array pair that has not dissolved -- reporting completion early yields something
that is not a full copy, after which the caller is free to delete the source.
copy_images_start() starts a batch between one pair of storages. Its default simply
starts them one by one, so there is no capability to negotiate; a backend that can
capture several volumes at one instant overrides it. Grouping policy is left to the
plugin, which knows about pools, group size limits and an administrator's own grouping
of disks by role. What the core guarantees is documented on the method: every start in
a batch happens inside one guest freeze, so consistency is backend-guaranteed WITHIN a
group and freeze-guaranteed ACROSS groups.
vdisk_copy_wait() bounds the wait with a deadline, exposed as the
'copy-offload-timeout' storage option, which resets on reported progress -- so it
bounds a stalled copy without killing a slow one. Without it a backend whose copy died
reports 'pending' forever and hangs the worker with no error.
vdisk_copy_start_group() flags each copy as it starts it, so a failure partway through
a batch leaves an accurate record of which copies are actually running; cleanup can
otherwise neither reclaim a running copy nor stop waiting for one that never began.
APIVER 15 -> 16, APIAGE 6 -> 7: both bump so the minimum accepted plugin API stays
15-6 = 16-7 = 9, since the new methods are optional and older plugins keep working.
Generated-By: Claude (https://claude.ai)
Signed-off-by: Ciro Iriarte <ciro.iriarte+software@gmail.com>
Co-Authored-By: Claude <noreply@anthropic.com>
---
ApiChangeLog | 24 ++++
src/PVE/Storage.pm | 237 +++++++++++++++++++++++++++++++++-
src/PVE/Storage/Plugin.pm | 176 +++++++++++++++++++++++++
src/test/copy_offload_test.pm | 90 +++++++++++++
src/test/run_plugin_tests.pl | 1 +
5 files changed, 526 insertions(+), 2 deletions(-)
create mode 100644 src/test/copy_offload_test.pm
diff --git a/ApiChangeLog b/ApiChangeLog
index 80c5994..f9505f4 100644
--- a/ApiChangeLog
+++ b/ApiChangeLog
@@ -6,6 +6,30 @@ without breaking anything unaware of it.)
Future changes should be documented in here.
+## Version 16:
+
+* Add optional `copy_image_prepare()`, `copy_image_start()`, `copy_image_status()` and
+ `copy_images_start()` plugin methods, plus a `copy-offload` storage option
+
+ These let a plugin perform a full copy (e.g. a full clone) on the storage backend instead of
+ reading and writing the data through the host. They are optional: the base implementations die,
+ and are only reached if a plugin advertises one of the new `copy-offload-atomic` or
+ `copy-offload-bulk` features from `volume_has_feature()`, so existing plugins are unaffected.
+
+ The copy is asynchronous. `copy_image_prepare()` allocates the target and returns its volname,
+ `copy_image_start()` fixes the copy's point in time and begins it, and `copy_image_status()` is
+ polled until the target is INDEPENDENT of its source -- not merely readable. Splitting the two
+ matters for callers that must capture several volumes at one instant: only the start needs to
+ happen inside a guest freeze, and it must stay cheap enough to do so.
+
+ `copy_images_start()` starts a batch between the same pair of storages. Its default
+ implementation simply starts them one by one, so it needs no capability negotiation; a backend
+ that can capture several volumes atomically overrides it.
+
+ Note that `copy_image_start()` receives the source `$snapname`: it is the call that fixes the
+ point in time, so a plugin that instead snapshots the current image would silently copy live
+ data when a snapshot was requested.
+
## Version 15:
* Add new `$snapname` parameter to the `volume_resize()` plugin method
diff --git a/src/PVE/Storage.pm b/src/PVE/Storage.pm
index 64ea9da..76189d0 100755
--- a/src/PVE/Storage.pm
+++ b/src/PVE/Storage.pm
@@ -41,11 +41,11 @@ use PVE::Storage::BTRFSPlugin;
use PVE::Storage::ESXiPlugin;
# Storage API version. Increment it on changes in storage API interface.
-use constant APIVER => 15;
+use constant APIVER => 16;
# Age is the number of versions we're backward compatible with.
# This is like having 'current=APIVER' and age='APIAGE' in libtool,
# see https://www.gnu.org/software/libtool/manual/html_node/Libtool-versioning.html
-use constant APIAGE => 6;
+use constant APIAGE => 7;
our $KNOWN_EXPORT_FORMATS = ['raw+size', 'tar+size', 'qcow2+size', 'vmdk+size', 'zfs', 'btrfs'];
@@ -1098,6 +1098,239 @@ sub vdisk_clone {
);
}
+# Returns the copy-offload class usable for $volid -> $target_storeid, or undef.
+#
+# Offload requires: the target storage to have copy-offload enabled, both storages to be
+# instances of the same plugin type, and the plugin to advertise a class for this volume.
+# 'atomic' is preferred when a plugin advertises both, as it needs no host-side mirror.
+sub copy_offload_class {
+ my ($cfg, $volid, $target_storeid, $snap, $running) = @_;
+
+ my ($storeid, $volname) = parse_volume_id($volid, 1);
+ return undef if !$storeid;
+
+ my $scfg = storage_config($cfg, $storeid);
+ my $target_scfg = storage_config($cfg, $target_storeid);
+
+ return undef if !$target_scfg->{'copy-offload'};
+ return undef if $scfg->{type} ne $target_scfg->{type};
+
+ for my $class (qw(atomic bulk)) {
+ return $class
+ if volume_has_feature($cfg, "copy-offload-$class", $volid, $snap, $running);
+ }
+
+ return undef;
+}
+
+# Allocate the target of an offloaded copy, returning the new volid. Its CONTENT is
+# undefined until vdisk_copy_start() has been called for it.
+#
+# Split out from the start so a caller cloning a running guest can prepare every disk's
+# target first and then start them all inside a single guest freeze; see vdisk_copy().
+#
+# Only this half runs under the storage lock -- it is where the allocation happens, and
+# it is bounded. The copy itself must not run under the lock: for shared storage that is
+# a pmxcfs lock whose locked code is killed by an alarm(60) in PVE::Cluster, while a full
+# array copy runs for minutes to hours. The host-side path is arranged the same way,
+# allocating under the lock and running the long qemu-img convert outside it.
+sub vdisk_copy_prepare {
+ my ($cfg, $volid, $target_storeid, $target_vmid, $snap, $opts) = @_;
+
+ my ($storeid, $volname) = parse_volume_id($volid);
+
+ my $scfg = storage_config($cfg, $storeid);
+ my $target_scfg = storage_config($cfg, $target_storeid);
+
+ my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+
+ activate_storage($cfg, $storeid);
+ activate_storage($cfg, $target_storeid) if $target_storeid ne $storeid;
+
+ my $new_volname = $plugin->cluster_lock_storage(
+ $target_storeid,
+ $target_scfg->{shared},
+ undef,
+ sub {
+ return $plugin->copy_image_prepare(
+ $scfg, $storeid, $volname,
+ $target_scfg, $target_storeid, $target_vmid, $snap, $opts,
+ );
+ },
+ );
+
+ return "$target_storeid:$new_volname";
+}
+
+# Pin the source's point-in-time and begin copying into a prepared target.
+#
+# MUST stay cheap: a caller cloning a running guest holds a filesystem freeze across
+# this call for every disk of the VM. It deliberately takes no storage lock -- the
+# allocation already happened in vdisk_copy_prepare(), and taking a cluster lock inside
+# a guest freeze would be a bad trade.
+sub vdisk_copy_start {
+ my ($cfg, $volid, $target_volid, $snap) = @_;
+
+ my ($storeid, $volname) = parse_volume_id($volid);
+ my ($target_storeid, $target_volname) = parse_volume_id($target_volid);
+
+ my $scfg = storage_config($cfg, $storeid);
+ my $target_scfg = storage_config($cfg, $target_storeid);
+
+ my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+
+ return $plugin->copy_image_start(
+ $scfg, $storeid, $volname, $target_scfg, $target_storeid, $target_volname, $snap,
+ );
+}
+
+# Start several prepared copies, grouping them so a backend that can capture multiple
+# volumes at one instant does so. $copies is an arrayref of
+# { source => $volid, target => $target_volid }.
+#
+# Copies are grouped by their (source storage, target storage) pair, since that is the
+# granularity at which a backend can act atomically; each group is handed to the
+# plugin's copy_images_start(), whose default simply starts them one by one.
+#
+# Same constraints as vdisk_copy_start(): cheap, no locking -- the caller may hold a
+# guest freeze across this.
+sub vdisk_copy_start_group {
+ my ($cfg, $copies) = @_;
+
+ return if !$copies || !scalar(@$copies);
+
+ my $groups = {};
+ for my $copy (@$copies) {
+ my ($storeid, $volname) = parse_volume_id($copy->{source});
+ my ($target_storeid, $target_volname) = parse_volume_id($copy->{target});
+
+ my $key = "$storeid/$target_storeid";
+ $groups->{$key} //= {
+ storeid => $storeid,
+ target_storeid => $target_storeid,
+ copies => [],
+ };
+ push $groups->{$key}->{copies}->@*,
+ { source => $volname, target => $target_volname, snap => $copy->{snap} };
+ push $groups->{$key}->{orig}->@*, $copy;
+ }
+
+ # Mark each group's copies started as soon as THAT group returns, not after the
+ # whole batch. A later group can fail, and the caller must still know which copies
+ # are actually running -- otherwise cleanup cannot tell a started copy from one that
+ # never began, and would either leak a running copy or wait for one that will never
+ # report.
+ for my $key (sort keys %$groups) {
+ my $group = $groups->{$key};
+
+ my $scfg = storage_config($cfg, $group->{storeid});
+ my $target_scfg = storage_config($cfg, $group->{target_storeid});
+
+ my $plugin = PVE::Storage::Plugin->lookup($scfg->{type});
+
+ $plugin->copy_images_start(
+ $scfg, $group->{storeid},
+ $target_scfg, $group->{target_storeid},
+ $group->{copies},
+ );
+
+ $_->{started} = 1 for $group->{orig}->@*;
+ }
+
+ return;
+}
+
+# Wait for a started copy to become independent of its source. Frees the target and
+# re-raises on failure, so the caller never leaks a half-copied volume.
+#
+# $progress is an optional coderef called with a percentage.
+sub vdisk_copy_wait {
+ my ($cfg, $target_volid, $volid, $progress) = @_;
+
+ my ($target_storeid, $target_volname) = parse_volume_id($target_volid);
+
+ my $target_scfg = storage_config($cfg, $target_storeid);
+ my $plugin = PVE::Storage::Plugin->lookup($target_scfg->{type});
+
+ # What the copy was made from, so the plugin can clean up state it left there.
+ my $source;
+ if ($volid) {
+ my ($storeid, $volname) = parse_volume_id($volid);
+ $source = {
+ scfg => storage_config($cfg, $storeid),
+ storeid => $storeid,
+ volname => $volname,
+ };
+ }
+
+ # A bounded wait, not a bare loop: a backend whose copy dies can otherwise report
+ # 'pending' forever and hang this worker with no error and no progress. The deadline
+ # is generous because the copy is a full data copy, and it is reset whenever the
+ # backend reports forward progress, so a slow-but-live copy is never killed.
+ my $timeout = $target_scfg->{'copy-offload-timeout'} // (24 * 3600);
+ my $deadline = time() + $timeout;
+ my $last_progress;
+
+ eval {
+ while (1) {
+ my $status = $plugin->copy_image_status(
+ $target_scfg, $target_storeid, $target_volname, $source,
+ );
+
+ my $state = $status->{state} // '';
+ my $done = $status->{progress};
+
+ if (defined($done) && (!defined($last_progress) || $done != $last_progress)) {
+ $progress->($done) if $progress;
+ $last_progress = $done;
+ $deadline = time() + $timeout; # forward progress: give it the full budget again
+ }
+
+ last if $state eq 'complete';
+
+ die "copy of '$target_volid' reported unexpected state '$state'\n"
+ if $state ne 'pending';
+
+ die "copy of '$target_volid' did not finish within ${timeout}s\n"
+ if time() >= $deadline;
+
+ sleep(1);
+ }
+ };
+ if (my $err = $@) {
+ # includes a worker abort (die from the poll loop): do not leak the target
+ eval { vdisk_free($cfg, $target_volid) };
+ warn "could not clean up target volume '$target_volid' - $@" if $@;
+ die $err;
+ }
+
+ return $target_volid;
+}
+
+# Offload a full copy of $volid to $target_storeid, returning the new volid.
+#
+# The simple path, for a source nothing is writing to: prepare, start, wait. A caller
+# cloning a RUNNING guest must not use this -- it has to prepare every disk, then start
+# them all inside one freeze, then wait -- see the three calls above.
+#
+# The caller is responsible for checking copy_offload_class() first and, for the 'bulk'
+# class with a running guest, for the write-tracking/convergence the class implies.
+sub vdisk_copy {
+ my ($cfg, $volid, $target_storeid, $target_vmid, $snap, $opts, $progress) = @_;
+
+ my $target_volid =
+ vdisk_copy_prepare($cfg, $volid, $target_storeid, $target_vmid, $snap, $opts);
+
+ eval { vdisk_copy_start($cfg, $volid, $target_volid, $snap) };
+ if (my $err = $@) {
+ eval { vdisk_free($cfg, $target_volid) };
+ warn "could not clean up target volume '$target_volid' - $@" if $@;
+ die $err;
+ }
+
+ return vdisk_copy_wait($cfg, $target_volid, $volid, $progress);
+}
+
sub vdisk_create_base {
my ($cfg, $volid) = @_;
diff --git a/src/PVE/Storage/Plugin.pm b/src/PVE/Storage/Plugin.pm
index 4f69f9b..18a484f 100644
--- a/src/PVE/Storage/Plugin.pm
+++ b/src/PVE/Storage/Plugin.pm
@@ -197,6 +197,26 @@ my $defaultData = {
type => 'boolean',
optional => 1,
},
+ 'copy-offload-timeout' => {
+ description =>
+ "Maximum time in seconds to wait for an offloaded copy to become "
+ . "independent of its source. The timer restarts whenever the backend "
+ . "reports progress, so this bounds a stalled copy rather than a slow one.",
+ type => 'integer',
+ minimum => 1,
+ optional => 1,
+ default => 24 * 3600,
+ },
+ 'copy-offload' => {
+ description =>
+ "Offload full-copy operations (e.g. full clone) to the storage backend "
+ . "instead of copying the data through the host. Only honored by plugins that "
+ . "advertise a 'copy-offload-atomic' or 'copy-offload-bulk' feature; ignored "
+ . "otherwise. Both source and target must be instances of the same plugin type.",
+ type => 'boolean',
+ optional => 1,
+ default => 0,
+ },
subdir => {
description => "Subdir to mount.",
type => 'string',
@@ -1046,6 +1066,150 @@ sub clone_image {
return $newvol;
}
+# START a full copy of $volname onto the TARGET storage, returning the new volname
+# there. The copy is ASYNCHRONOUS: this returns as soon as the backend has accepted the
+# job, and the caller then polls copy_image_status() until the copy is complete.
+#
+# It must be async because a full array copy runs for minutes to hours, while the core
+# calls this under cluster_lock_storage(). For shared storage that is a pmxcfs lock,
+# whose locked code is killed by an alarm(60) (PVE::Cluster) -- so a blocking copy would
+# be aborted mid-flight and the lock broken. Splitting start from wait also mirrors what
+# the host-side path already does: it allocates under the lock and runs the long
+# qemu-img convert outside it.
+#
+# Returning the target volname immediately (rather than only when the data is there) is
+# deliberate: the caller can record it for rollback before anything can go wrong, and
+# free_image() is the cleanup path if the copy later fails or is aborted. A plugin MUST
+# therefore accept free_image() on a volume whose copy is still in flight, cancelling
+# the backend job.
+#
+# Only called when the plugin advertises 'copy-offload-atomic' or 'copy-offload-bulk'
+# via volume_has_feature() and the target storage has copy-offload enabled, so the base
+# implementation dies rather than returning undef: reaching it means a plugin advertised
+# a capability it does not implement.
+#
+# $target_scfg/$target_storeid describe the TARGET storage. A plugin only ever receives
+# its own per-instance scfg, so it cannot judge whether a given source/target pair can
+# actually be copied by the backend (same array, same pool group, ...) without them.
+# Both are guaranteed to be instances of the same plugin type as the source; cross-type
+# offload is out of scope, since the operation is opaque to PVE and cannot be lowered to
+# volume_export/volume_import the way a generic copy can.
+#
+# $snap is optional: copy from that snapshot rather than the current state.
+# $opts->{format} is the format the caller resolved for the target. A plugin that cannot
+# produce it MUST die here rather than silently producing something else.
+#
+# The copy is split into prepare + start so that a caller cloning a RUNNING guest can
+# hold all the disks of a VM at one point in time: it prepares every target first, then
+# fs-freezes the guest once and calls copy_image_start() for each disk inside that
+# freeze. Consistency across disks is a VM-level property, so the core needs a cheap
+# operation it can put inside a freeze -- which allocation is not.
+sub copy_image_prepare {
+ my (
+ $class, $scfg, $storeid, $volname,
+ $target_scfg, $target_storeid, $target_vmid, $snap, $opts,
+ ) = @_;
+
+ die "storage plugin '" . $class->type() . "' does not implement copy_image_prepare\n";
+}
+
+# Fix the source's point-in-time and begin copying it into the target that
+# copy_image_prepare() returned. Only after this returns is the copy's content defined.
+#
+# MUST BE FAST AND MUST NOT BLOCK: the caller may hold a guest filesystem freeze (or a
+# suspended VM) across this call for every disk of a VM. Do the minimum that pins the
+# source -- create the snapshot/pair -- and let the data movement proceed in the
+# background, reported by copy_image_status().
+#
+# For the 'copy-offload-atomic' class this call is what defines the copy's point in
+# time; writes to the source after it returns MUST NOT affect the result. A plugin that
+# cannot make that guarantee must advertise 'copy-offload-bulk' instead.
+#
+# $snap names the source snapshot to copy from, or is undef for the current state. It
+# is passed HERE and not only to prepare because this call is what fixes the point in
+# time: a plugin that took its own snapshot of the current image here would silently
+# copy live data when the caller asked for a snapshot.
+sub copy_image_start {
+ my (
+ $class, $scfg, $storeid, $volname,
+ $target_scfg, $target_storeid, $target_volname, $snap,
+ ) = @_;
+
+ die "storage plugin '" . $class->type() . "' does not implement copy_image_start\n";
+}
+
+# Start several copies as ONE operation. $copies is an arrayref of
+# { source => $volname, target => $target_volname, snap => $snap }, all between the
+# same pair of storages. As in copy_image_start(), 'snap' names the source snapshot to
+# copy from and may be undef; each copy in a batch carries its own.
+#
+# The default implementation just starts them one after another, which is what the
+# caller would otherwise do itself -- so there is nothing to advertise and nothing to
+# negotiate. A backend that can capture several volumes at one instant (an array
+# consistency group) overrides this, and every caller gets the stronger guarantee for
+# free.
+#
+# Overriding it buys two things for a multi-disk guest. The capture becomes atomic in
+# the backend rather than merely bracketed by a guest freeze, so the disks are mutually
+# crash-consistent even if the freeze does not apply -- e.g. no guest agent is
+# available. And the freeze, when there is one, shrinks from N sequential backend calls
+# to one, which for a guest with many disks is the difference between a perceptible
+# stall and none.
+#
+# The plugin decides how to group. It may start the batch as one backend group, as
+# several, or one by one -- whatever its backend supports and its own metadata calls
+# for, e.g. volumes that must stay in separate groups because they live in different
+# pools, or because the backend caps how many volumes a group may hold. The core does
+# not model any of that and passes the whole batch for a given pair of storages.
+#
+# What the core does guarantee is that every start in the batch happens inside one
+# guest freeze. So the consistency a caller gets is: mutually crash-consistent WITHIN
+# each backend group by the backend itself, and consistent ACROSS groups only by virtue
+# of that freeze. A plugin that splits a batch it could have kept in one group is
+# therefore trading away the guarantee that survives when no freeze is possible -- when
+# there is no guest agent and the caller has to fall back to suspending the VM.
+#
+# Same contract as copy_image_start(): fast, non-blocking, and for the atomic class it
+# is this call that fixes the point in time of every copy in the group.
+sub copy_images_start {
+ my ($class, $scfg, $storeid, $target_scfg, $target_storeid, $copies) = @_;
+
+ for my $copy (@$copies) {
+ $class->copy_image_start(
+ $scfg, $storeid, $copy->{source},
+ $target_scfg, $target_storeid, $copy->{target}, $copy->{snap},
+ );
+ }
+
+ return;
+}
+
+# Poll a copy started by copy_image(). Returns a hashref:
+#
+# { state => 'pending'|'complete', progress => $percent }
+#
+# 'progress' is optional and informational only. On failure this MUST die; the caller
+# then frees the target volume.
+#
+# 'complete' means the target is INDEPENDENT of the source -- not merely that the data
+# is readable. For backends where independence is asynchronous (an RBD flatten, a ZFS
+# clone still tied to its origin, an array pair that has not dissolved yet) reporting
+# 'complete' early produces something that is not a full copy, and the caller is then
+# free to delete the source.
+#
+# This is called OUTSIDE the storage lock, so it may talk to the backend, but it should
+# stay cheap: it runs in a poll loop.
+#
+# $source is { scfg, storeid, volname } describing what the copy was made FROM. A
+# plugin may need it to clean up state it left on the source -- an RBD copy, for
+# instance, is a clone of a snapshot taken on the source image, and once the copy is
+# independent nothing else records where it came from.
+sub copy_image_status {
+ my ($class, $scfg, $storeid, $volname, $source) = @_;
+
+ die "storage plugin '" . $class->type() . "' does not implement copy_image_status\n";
+}
+
sub alloc_image {
my ($class, $storeid, $scfg, $vmid, $fmt, $name, $size) = @_;
@@ -1572,6 +1736,18 @@ sub storage_can_replicate {
return 0;
}
+# Besides the features listed below, a plugin may advertise one of the two copy-offload
+# classes (see copy_image()). They are separate features rather than one feature with a
+# class argument so that this interface stays unchanged; a plugin supports whichever one
+# fits its backend, and a caller that can use both prefers 'copy-offload-atomic':
+#
+# copy-offload-atomic The backend can produce the copy from a static point-in-time
+# source (snapshot/clone), so no host-side mirror is needed to
+# keep it consistent while the source keeps changing.
+# copy-offload-bulk The copy is a bulk data movement that smears over a changing
+# source (e.g. NFS 4.2 server-side copy, zfs send/recv). For a
+# running guest the caller must track writes and converge, the
+# way live migration does.
sub volume_has_feature {
my ($class, $scfg, $feature, $storeid, $volname, $snapname, $running, $opts) = @_;
diff --git a/src/test/copy_offload_test.pm b/src/test/copy_offload_test.pm
new file mode 100644
index 0000000..98aa940
--- /dev/null
+++ b/src/test/copy_offload_test.pm
@@ -0,0 +1,90 @@
+package PVE::Storage::TestCopyOffload;
+
+use strict;
+use warnings;
+
+use lib qw(..);
+
+use PVE::Storage;
+use Test::More;
+
+# copy_offload_class() decides whether a full copy may be handed to the backend, and
+# which of the two classes to use. Getting it wrong either silently declines an offload
+# or drives one onto a storage pair that cannot perform it, so the rules are pinned here.
+#
+# The checks are: the TARGET storage must have copy-offload enabled, both storages must
+# be instances of the same plugin type, and the plugin must advertise a class for that
+# volume. 'atomic' wins when a plugin advertises both, as it needs no host-side mirror.
+
+my $features = {}; # volid => { feature => 1 }
+
+{
+ no warnings 'redefine';
+ *PVE::Storage::volume_has_feature = sub {
+ my ($cfg, $feature, $volid, $snap, $running, $opts) = @_;
+ return $features->{$volid}->{$feature} ? 1 : 0;
+ };
+}
+
+my $cfg = {
+ ids => {
+ 'rbd-a' => { type => 'rbd', 'copy-offload' => 1 },
+ 'rbd-b' => { type => 'rbd', 'copy-offload' => 1 },
+ 'rbd-off' => { type => 'rbd' },
+ 'dir-a' => { type => 'dir', 'copy-offload' => 1 },
+ },
+};
+
+my $volid = 'rbd-a:vm-100-disk-0';
+
+my $tests = [
+ # [ description, target storeid, advertised features, expected class ]
+ ['nothing advertised => no offload', 'rbd-b', {}, undef],
+ ['atomic advertised', 'rbd-b', { 'copy-offload-atomic' => 1 }, 'atomic'],
+ ['bulk advertised', 'rbd-b', { 'copy-offload-bulk' => 1 }, 'bulk'],
+ [
+ 'atomic preferred when both are advertised',
+ 'rbd-b',
+ { 'copy-offload-atomic' => 1, 'copy-offload-bulk' => 1 },
+ 'atomic',
+ ],
+ [
+ 'target has copy-offload disabled',
+ 'rbd-off',
+ { 'copy-offload-atomic' => 1 },
+ undef,
+ ],
+ [
+ 'cross-type is out of scope even when advertised',
+ 'dir-a',
+ { 'copy-offload-atomic' => 1 },
+ undef,
+ ],
+ [
+ 'same storage as source is allowed',
+ 'rbd-a',
+ { 'copy-offload-atomic' => 1 },
+ 'atomic',
+ ],
+];
+
+plan tests => scalar(@$tests) + 1;
+
+for my $t (@$tests) {
+ my ($desc, $target, $adv, $expected) = @$t;
+
+ $features = { $volid => $adv };
+
+ my $got = PVE::Storage::copy_offload_class($cfg, $volid, $target, undef, 0);
+ is($got, $expected, $desc);
+}
+
+# A path that is not a storage volume must not be mistaken for an offloadable one.
+$features = { $volid => { 'copy-offload-atomic' => 1 } };
+is(
+ PVE::Storage::copy_offload_class($cfg, '/dev/sdb', 'rbd-b', undef, 0),
+ undef,
+ 'a bare path is never offloadable',
+);
+
+done_testing();
diff --git a/src/test/run_plugin_tests.pl b/src/test/run_plugin_tests.pl
index 8bce9d3..dda4cac 100755
--- a/src/test/run_plugin_tests.pl
+++ b/src/test/run_plugin_tests.pl
@@ -17,6 +17,7 @@ my $res = $harness->runtests(
"get_subdir_test.pm",
"filesystem_path_test.pm",
"prune_backups_test.pm",
+ "copy_offload_test.pm",
);
exit -1 if !$res || $res->{failed} || $res->{parse_errors};
--
2.54.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [RFC PATCH storage 2/5] rbd: implement copy-offload via snapshot + clone + flatten
2026-07-20 14:29 [RFC PATCH storage, qemu-server 0/5] offload full clone to the storage backend Ciro Iriarte
2026-07-20 14:29 ` [RFC PATCH storage 1/5] storage: add asynchronous copy-offload hook for full copies Ciro Iriarte
@ 2026-07-20 14:29 ` Ciro Iriarte
2026-07-20 14:30 ` [RFC PATCH storage 3/5] dir: implement copy-offload via reflink (FICLONE) Ciro Iriarte
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Ciro Iriarte @ 2026-07-20 14:29 UTC (permalink / raw)
To: pve-devel
A reference implementation on an in-tree backend, so the hook can be exercised
without vendor hardware.
Ceph produces a full copy without moving data through the host: snapshot the source,
clone the snapshot (copy-on-write, instant), then flatten the clone. The flatten is
the actual copy and runs on the Ceph cluster.
prepare create the target image, reserving the name
start snap (only when the caller supplied none) + clone. Both are O(1) metadata
operations, so this is cheap enough to run inside a guest freeze, and it
fixes the copy's point in time -- hence 'copy-offload-atomic'.
status pending while the clone still has a parent; complete once flattened
RBD is the case the "complete means INDEPENDENT, not merely readable" rule was
written for. A clone is fully readable the instant it exists, but deleting the source
out from under an unflattened clone destroys it.
prepare CREATES the target rather than only choosing a name. The caller releases the
storage lock between prepare and start, so a merely-chosen name could be taken by a
concurrent allocation, and the caller's rollback would then free a volume belonging to
that other operation. The placeholder is the smallest image rbd accepts: it only has
to hold the name until start renames the real clone over it, and start must delete it
while the caller may be holding a guest frozen. 'rbd rm' walks every object, so a
full-size placeholder would stall the guest in proportion to the disk -- anything up
to the 4 MiB object size is a single object and as cheap to delete as an image can be.
The flatten is handed to the Ceph manager with 'ceph rbd task add flatten': it runs
cluster-side, survives this worker dying, and is cancellable. That also makes failure
diagnosable -- status distinguishes complete (no parent), pending (a manager task for
the image, with progress) and failed (still a clone, but no task), where the last case
would otherwise be indistinguishable from a slow copy and poll forever.
Copying from a caller-supplied snapshot uses clone format 2, which does not require
protecting the parent -- the snapshot belongs to the caller and must not be
protected/unprotected behind their back.
Once the target is independent, status removes the snapshot the copy was taken from on
the source: that is the last moment it can, because a flattened image no longer records
its origin, and leaving it behind pins space and blocks deleting the source later. Only
snapshots this plugin created can match the derived name, never a caller's. free_image()
covers the same cleanup for a copy that failed rather than completed.
The copy-offload options are listed in options(), without which they cannot be set on a
storage at all and the feature is unreachable.
LIVE VALIDATION on a real Ceph cluster (Tentacle 20.2.1, erasure-coded pool). Standalone
copy: finished in 3s, target had NO parent, md5 identical to the source, and it still
read correctly after DELETING THE SOURCE -- the independence guarantee demonstrated
rather than asserted. Via qemu-server, full-cloning a RUNNING two-disk VM: the guest
freeze held only two snapshot creations and two single-object deletions, the guest was
resumed, both clones were independent and byte-identical, and no snapshot was left on
either source disk.
copy_images_start() is deliberately not overridden. Ceph group snapshots were tested on
the same cluster and do capture a common instant, but the snapshot is reachable only by
snap-id, cannot be protected, and above all an image may belong to only ONE group -- so
grouping would mean mutating the source images' group membership, colliding with any
grouping an administrator set up and leaving stray membership behind if a clone dies.
Since the caller already brackets every start in one guest freeze, that is a bad trade.
The clone is built under a prefixed staging name and the placeholder is MOVED aside
rather than deleted, because 'rbd rm' followed by 'rbd rename' leaves the disk number
unreserved in between. A concurrent allocation could take it there, and this copy's
rollback would then free that other operation's volume -- silent third-party data loss.
Both transient names are PREFIXES, and that is load-bearing in two opposite directions:
- rbd_ls(), which feeds list_images(), selects on the ANCHORED m/^(?:vm|base)-(\d+)-/,
so a prefixed name never shows up as a phantom disk of that VM. The obvious
'$volname.copytmp' SUFFIX does match it, so the staging clone was visible in the GUI
for the whole duration of the copy -- which is the long part.
- find_free_diskname() passes the RAW 'rbd ls' output to get_next_vm_diskname(), whose
disk-number regex is UNANCHORED, so a prefixed name still reserves the disk NUMBER.
Verified through the real allocator rather than by reading the regexes: with only the
parked placeholder present, get_next_vm_diskname() returns vm-101-disk-1, not disk-0.
copy_offload_naming_test.pm pins both halves, since nothing about the naming looks
load-bearing at a glance and a rename back to a suffix would reintroduce both bugs at
once without failing anything else.
Removing the placeholder is left to copy_image_status(): 'rbd rm' walks every object of
an image, and copy_image_start() can run while the caller holds a guest frozen. It is
best effort there -- the copy is already complete and correct by then, so dying would
make the caller free it. prepare() and free_image() also reap leftovers from a copy that
died mid-flight; they are invisible to list_images(), so nothing else ever would, and a
stale placeholder makes the rename fail for every future copy to that name.
Generated-By: Claude (https://claude.ai)
Signed-off-by: Ciro Iriarte <ciro.iriarte+software@gmail.com>
Co-Authored-By: Claude <noreply@anthropic.com>
---
src/PVE/Storage/RBDPlugin.pm | 318 +++++++++++++++++++++++++++
src/test/copy_offload_naming_test.pm | 82 +++++++
src/test/run_plugin_tests.pl | 1 +
3 files changed, 401 insertions(+)
create mode 100644 src/test/copy_offload_naming_test.pm
diff --git a/src/PVE/Storage/RBDPlugin.pm b/src/PVE/Storage/RBDPlugin.pm
index b537425..31e9414 100644
--- a/src/PVE/Storage/RBDPlugin.pm
+++ b/src/PVE/Storage/RBDPlugin.pm
@@ -468,6 +468,8 @@ sub options {
krbd => { optional => 1 },
keyring => { optional => 1 },
bwlimit => { optional => 1 },
+ 'copy-offload' => { optional => 1 },
+ 'copy-offload-timeout' => { optional => 1 },
};
}
@@ -701,6 +703,279 @@ sub clone_image {
return $newvol;
}
+# ---- storage-offloaded full copy -------------------------------------------------
+#
+# Ceph can produce a full copy without moving the data through the host: snapshot the
+# source, clone the snapshot (copy-on-write, instant), then flatten the clone. The
+# flatten is what actually copies, and it runs on the Ceph cluster.
+#
+# That maps onto the hook's three phases as follows:
+#
+# prepare pick the target name. Nothing exists on the cluster yet.
+# start snap + protect + clone. Both are O(1) metadata operations, so this is
+# safe to run inside a guest freeze, and it is what FIXES the copy's point
+# in time -- writes to the source afterwards cannot affect the clone.
+# status pending until the flatten has removed the clone's parent, i.e. until the
+# target is genuinely INDEPENDENT and the source may be deleted.
+#
+# The last point is why 'complete' is defined as independent rather than readable: a
+# clone is fully readable the moment it exists, but deleting the source out from under
+# an unflattened clone would destroy it.
+
+my $rbd_copy_snap = sub {
+ my ($target_volname) = @_;
+ # Derived from the target name so status/cleanup can find it again without state.
+ return "__copy_$target_volname";
+};
+
+# Names for the two transient images a copy needs: the clone as it is being built, and
+# the placeholder once it has been moved out of the way.
+#
+# Both use a PREFIX. rbd_ls(), which feeds list_images(), selects on the ANCHORED
+# m/^(?:vm|base)-(\d+)-/, so a prefixed name is invisible there and never appears as a
+# phantom disk of that VM. find_free_diskname() meanwhile passes the RAW 'rbd ls' output
+# to get_next_vm_diskname(), whose disk-number regex is UNANCHORED, so a prefixed name
+# still reserves the disk NUMBER. Invisible and still reserving is exactly what these
+# need; a '.copytmp' SUFFIX would have been the opposite on both counts.
+my sub copy_staging_name { return "copynew-$_[0]" }
+my sub copy_parked_name { return "copytmp-$_[0]" }
+
+# Remove transient images a previous copy to this name left behind. They are invisible
+# to list_images(), so nothing else would ever reap them, and a leftover placeholder
+# would make the rename in copy_image_start() fail for every future copy to this name.
+my sub copy_reap_leftovers {
+ my ($scfg, $storeid, $volname) = @_;
+
+ for my $stale (copy_staging_name($volname), copy_parked_name($volname)) {
+ next if !rbd_volume_exists($scfg, $storeid, $stale);
+ warn "removing stale copy image '$stale'\n";
+ eval {
+ my $c = $rbd_cmd->($scfg, $storeid, 'rm', $stale);
+ run_rbd_command($c, errmsg => "rbd rm stale '$stale' error");
+ };
+ warn $@ if $@;
+ }
+}
+
+sub copy_image_prepare {
+ my (
+ $class, $scfg, $storeid, $volname,
+ $target_scfg, $target_storeid, $target_vmid, $snap, $opts,
+ ) = @_;
+
+ die "storage '$target_storeid' is on a different Ceph cluster - cannot clone across\n"
+ if ($scfg->{monhost} // '') ne ($target_scfg->{monhost} // '');
+
+ my $format = $opts->{format} // 'raw';
+ die "rbd copy offload cannot produce format '$format'\n" if $format ne 'raw';
+
+ my $name = $class->find_free_diskname($target_storeid, $target_scfg, $target_vmid);
+
+ # Safe here: we hold the target storage lock, and the name is ours because
+ # find_free_diskname() just handed it out.
+ copy_reap_leftovers($target_scfg, $target_storeid, $name);
+
+ # Actually CREATE the target, do not just pick a name. The caller runs this under
+ # the target storage lock and releases it before copy_image_start(), so a name that
+ # was merely chosen could be taken by a concurrent allocation in between -- and the
+ # caller's rollback would then free a volume belonging to that other operation.
+ # Creating it here makes the reservation real and the "prepared targets are
+ # freeable" contract true.
+ #
+ # Deliberately the SMALLEST image rbd accepts, not the source's size: this
+ # placeholder exists only to hold the name until copy_image_start() renames the real
+ # clone over it, and start has to delete it while the caller may be holding a guest
+ # frozen. 'rbd rm' walks every object of an image, so a full-size placeholder stalls
+ # the guest in proportion to the disk -- seconds for a GiB, far worse for a TiB.
+ # Anything up to the 4 MiB object size is a single object, so this is as cheap to
+ # delete as an image can be.
+ my $cmd = $rbd_cmd->(
+ $target_scfg, $target_storeid, 'create', '--image-format', '2',
+ '--size', '4K', $name,
+ );
+ push $cmd->@*, ('--data-pool', $target_scfg->{'data-pool'})
+ if $target_scfg->{'data-pool'};
+ run_rbd_command($cmd, errmsg => "rbd create '$name' error");
+
+ return $name;
+}
+
+sub copy_image_start {
+ my (
+ $class, $scfg, $storeid, $volname,
+ $target_scfg, $target_storeid, $target_volname, $snap,
+ ) = @_;
+
+ my (undef, $name) = $class->parse_volname($volname);
+
+ # Copy from the snapshot the caller asked for. Taking our own snapshot of the
+ # current image would silently copy live data when a snapshot was requested.
+ my $src_snap = $snap;
+ my $own_snap;
+ if (!defined($src_snap)) {
+ $own_snap = $rbd_copy_snap->($target_volname);
+ my $cmd = $rbd_cmd->($scfg, $storeid, 'snap', 'create', $name, '--snap', $own_snap);
+ run_rbd_command($cmd, errmsg => "rbd snap create '$name' error");
+ $src_snap = $own_snap;
+ }
+
+ my $tmp = copy_staging_name($target_volname);
+ my $parked = copy_parked_name($target_volname);
+
+ my $ok = eval {
+ # Clone format 2 so no snapshot protection is needed -- required when copying
+ # from a caller-supplied snapshot we do not own and must not protect/unprotect.
+ my @options = (
+ '--rbd-default-clone-format', '2',
+ get_rbd_path($scfg, $name), '--snap', $src_snap,
+ );
+ push @options, ('--data-pool', $target_scfg->{'data-pool'})
+ if $target_scfg->{'data-pool'};
+
+ my $cmd = $rbd_cmd->(
+ $scfg, $storeid, 'clone', @options, get_rbd_path($target_scfg, $tmp),
+ );
+ run_rbd_command($cmd, errmsg => "rbd clone '$name' error");
+
+ # Swap the clone into the reserved name. MOVE the placeholder aside rather than
+ # deleting it first: 'rbd rm' followed by 'rbd rename' leaves the disk number
+ # unreserved in between, and a concurrent allocation could take it -- after
+ # which this copy's rollback would free somebody else's volume. Parked under a
+ # prefixed name it stays invisible to list_images() while still reserving the
+ # number, so nothing can claim it. Removing it is left to copy_image_status(),
+ # since 'rbd rm' is not something to run while the caller holds a guest frozen.
+ $cmd = $rbd_cmd->($target_scfg, $target_storeid, 'rename', $target_volname, $parked);
+ run_rbd_command($cmd, errmsg => "rbd rename placeholder '$target_volname' error");
+ $cmd = $rbd_cmd->($target_scfg, $target_storeid, 'rename', $tmp, $target_volname);
+ if (!eval { run_rbd_command($cmd, errmsg => "rbd rename '$tmp' error"); 1 }) {
+ my $rerr = $@;
+ # Put the reservation back so the caller's rollback frees what it was given.
+ my $c = $rbd_cmd->($target_scfg, $target_storeid, 'rename', $parked,
+ $target_volname);
+ eval { run_rbd_command($c, errmsg => "restoring placeholder error") };
+ warn $@ if $@;
+ die $rerr;
+ }
+
+ # Hand the flatten to the Ceph manager rather than running it here: it is the
+ # actual data copy, it must not block a caller that may be holding a guest
+ # freeze, and a cluster-side task survives this worker dying. It is also
+ # cancellable, which a detached child process would not have been.
+ $cmd = ['ceph', 'rbd', 'task', 'add', 'flatten',
+ "$target_scfg->{pool}/$target_volname"];
+ run_command($cmd, errmsg => "scheduling rbd flatten failed", outfunc => sub { });
+ 1;
+ };
+ if (my $err = $@) {
+ # Own cleanup: leave the source exactly as it was.
+ eval {
+ my $c = $rbd_cmd->($target_scfg, $target_storeid, 'rm', $tmp);
+ run_rbd_command($c, errmsg => "rbd rm '$tmp' error");
+ };
+ if (rbd_volume_exists($target_scfg, $target_storeid, $parked)) {
+ eval {
+ my $c = $rbd_cmd->($target_scfg, $target_storeid, 'rm', $parked);
+ run_rbd_command($c, errmsg => "rbd rm placeholder '$parked' error");
+ };
+ warn $@ if $@;
+ }
+ if ($own_snap) {
+ eval {
+ my $c = $rbd_cmd->($scfg, $storeid, 'snap', 'rm', $name, '--snap', $own_snap);
+ run_rbd_command($c, errmsg => "rbd snap rm '$name' error");
+ };
+ }
+ die $err;
+ }
+
+ return;
+}
+
+# Find the manager-side flatten task for an image, if one is queued or running.
+my $rbd_flatten_task = sub {
+ my ($scfg, $volname) = @_;
+
+ my $out = '';
+ eval {
+ run_command(['ceph', 'rbd', 'task', 'list', '--format', 'json'],
+ outfunc => sub { $out .= shift });
+ };
+ return undef if $@ || $out !~ /\S/;
+
+ my $tasks = eval { decode_json($out) } // [];
+ for my $task (@$tasks) {
+ my $refs = $task->{refs} // {};
+ next if ($refs->{action} // '') ne 'flatten';
+ next if ($refs->{image_name} // '') ne $volname;
+ next if ($refs->{pool_name} // '') ne ($scfg->{pool} // '');
+ return $task;
+ }
+
+ return undef;
+};
+
+sub copy_image_status {
+ my ($class, $scfg, $storeid, $volname, $source) = @_;
+
+ my (undef, $name) = $class->parse_volname($volname);
+
+ my $cmd = $rbd_cmd->($scfg, $storeid, 'info', $name, '--format', 'json');
+ my $info = '';
+ run_rbd_command($cmd, errmsg => "rbd info '$name' error", outfunc => sub { $info .= shift });
+
+ my $parent = eval { decode_json($info)->{parent} };
+
+ # No parent: the flatten finished and the image is INDEPENDENT -- which is the
+ # difference that matters. A clone is readable from the moment it exists, but
+ # deleting the source before the flatten completes would destroy it.
+ if (!$parent) {
+ # Drop the snapshot this copy was taken from, on the SOURCE image. Once the
+ # flatten completes the target no longer records where it came from, so this is
+ # the last chance to find it -- and leaving it behind pins space on the source
+ # and blocks deleting that volume later.
+ #
+ # Only ever our own: the name is derived from the target, so a caller-supplied
+ # snapshot can never match and is left untouched.
+ if ($source) {
+ my $own_snap = $rbd_copy_snap->($volname);
+ my (undef, $src_name) = $class->parse_volname($source->{volname});
+ eval {
+ my $c = $rbd_cmd->(
+ $source->{scfg}, $source->{storeid},
+ 'snap', 'rm', $src_name, '--snap', $own_snap,
+ );
+ run_rbd_command($c, errmsg => "rbd snap rm error", outfunc => sub { });
+ };
+ }
+ # The placeholder copy_image_start() parked. Deliberately not removed there:
+ # 'rbd rm' walks every object and that call can run with a guest frozen. Best
+ # effort -- the copy is already complete and correct, and dying here would make
+ # the caller free it.
+ my $parked = copy_parked_name($name);
+ if (rbd_volume_exists($scfg, $storeid, $parked)) {
+ eval {
+ my $c = $rbd_cmd->($scfg, $storeid, 'rm', $parked);
+ run_rbd_command($c, errmsg => "rbd rm placeholder '$parked' error");
+ };
+ warn $@ if $@;
+ }
+
+ return { state => 'complete' };
+ }
+
+ # Still parented. Distinguish a running flatten from one that died: without this a
+ # dead flatten leaves the image parented forever and the caller polls indefinitely.
+ my $task = $rbd_flatten_task->($scfg, $name);
+ die "flatten of '$volname' is no longer running and the image is still a clone\n"
+ if !$task;
+
+ my $progress = $task->{progress};
+ return {
+ state => 'pending',
+ (defined($progress) ? (progress => int($progress * 100)) : ()),
+ };
+}
+
sub alloc_image {
my ($class, $storeid, $scfg, $vmid, $fmt, $name, $size) = @_;
@@ -720,11 +995,39 @@ sub alloc_image {
return $name;
}
+# Parent (image + snapshot) of a clone, or undef if the image stands alone.
+sub rbd_volume_info_parent {
+ my ($scfg, $storeid, $volname) = @_;
+
+ my $cmd = $rbd_cmd->($scfg, $storeid, 'info', $volname, '--format', 'json');
+ my $raw = '';
+ run_rbd_command($cmd, errmsg => "rbd info '$volname' error", outfunc => sub { $raw .= shift });
+
+ return eval { decode_json($raw)->{parent} };
+}
+
sub free_image {
my ($class, $storeid, $scfg, $volname, $isBase) = @_;
my ($vtype, $name, $vmid, undef, undef, undef) = $class->parse_volname($volname);
+ # Transient images from a copy to this name that died mid-flight. They are invisible
+ # to list_images(), so freeing the volume they belong to is the only occasion
+ # anything would think to look for them.
+ copy_reap_leftovers($scfg, $storeid, $name);
+
+ # An offloaded copy that never finished is still a clone of a snapshot this plugin
+ # took on the SOURCE image. Freeing the target releases the child reference but
+ # would leave that snapshot behind, pinning space and blocking deletion of the
+ # source. The clone itself is the only record of where it came from, so read the
+ # parent before removing it and drop the snapshot afterwards -- but only when it is
+ # one of ours, never a snapshot the caller supplied.
+ my $copy_parent;
+ if (my $info = eval { rbd_volume_info_parent($scfg, $storeid, $name) }) {
+ my $psnap = $info->{snapshot} // '';
+ $copy_parent = $info if $psnap =~ /^__copy_/;
+ }
+
my $snaps = rbd_ls_snap($scfg, $storeid, $name);
foreach my $snap (keys %$snaps) {
if ($snaps->{$snap}->{protected}) {
@@ -741,6 +1044,17 @@ sub free_image {
$cmd = $rbd_cmd->($scfg, $storeid, 'rm', $name);
run_rbd_command($cmd, errmsg => "rbd rm '$name' error");
+ if ($copy_parent) {
+ # Best effort: another unfinished copy may still hold the same snapshot.
+ eval {
+ my $c = $rbd_cmd->(
+ $scfg, $storeid, 'snap', 'rm', $copy_parent->{image},
+ '--snap', $copy_parent->{snapshot},
+ );
+ run_rbd_command($c, errmsg => "rbd snap rm error", outfunc => sub { });
+ };
+ }
+
return undef;
}
@@ -962,6 +1276,10 @@ sub volume_has_feature {
copy => { base => 1, current => 1, snap => 1 },
sparseinit => { base => 1, current => 1 },
rename => { current => 1 },
+ # The copy is produced from an RBD snapshot, so it comes from a static point
+ # in time and needs no host-side mirror to stay consistent while the source
+ # keeps changing -- see copy_image_start().
+ 'copy-offload-atomic' => { base => 1, current => 1, snap => 1 },
};
my ($vtype, $name, $vmid, $basename, $basevmid, $isBase) = $class->parse_volname($volname);
diff --git a/src/test/copy_offload_naming_test.pm b/src/test/copy_offload_naming_test.pm
new file mode 100644
index 0000000..9de30f1
--- /dev/null
+++ b/src/test/copy_offload_naming_test.pm
@@ -0,0 +1,82 @@
+package PVE::Storage::TestCopyOffloadNaming;
+
+use strict;
+use warnings;
+
+use lib qw(..);
+
+use PVE::Storage;
+use PVE::Storage::Plugin;
+use Test::More;
+
+# The names copy_image_start() parks its placeholder under must satisfy TWO opposing
+# properties at once, and getting either one wrong is a silent, expensive bug:
+#
+# 1. INVISIBLE to the volume lister. If the lister accepts the name, the placeholder
+# shows up as a real disk of that VM -- a phantom the GUI offers to attach, and a
+# permanent one if the copy dies mid-flight.
+# 2. STILL RESERVING the disk NUMBER. The caller drops the storage lock between
+# prepare() and start(), so if the parked name stops reserving, a concurrent
+# allocation can take it -- and the failing copy's rollback then frees somebody
+# else's volume. That is silent third-party data loss.
+#
+# The two properties come from two different regexes, which is why a name can satisfy
+# one and not the other:
+#
+# - listers are ANCHORED (rbd_ls: m/^(?:vm|base)-(\d+)-/,
+# LvmThin list_images: m/^(vm|base)-(\d+)-/)
+# - $get_vm_disk_number is UNANCHORED (Plugin.pm)
+#
+# So a PREFIX is invisible but still reserves, and a SUFFIX is the exact opposite on
+# both counts. This test pins that, because nothing about the naming looks load-bearing
+# at a glance and a well-meaning rename to 'vm-101-disk-0.copytmp' would reintroduce
+# both bugs at once without failing anything else.
+
+# The anchored patterns the listers actually use, kept here so a change to either one
+# is caught rather than silently narrowing what these names protect.
+my $rbd_lister_re = qr/^(?:vm|base)-(\d+)-/;
+my $lvmthin_lister_re = qr/^(vm|base)-(\d+)-/;
+
+my $tests = [
+ # [ name, listed?, reserves a disk number? ]
+ ['vm-101-disk-0', 1, 1], # the real volume: listed and reserving
+ ['copytmp-vm-101-disk-0', 0, 1], # parked placeholder
+ ['copynew-vm-101-disk-0', 0, 1], # staging clone
+ ['vm-101-disk-0.copytmp', 1, 1], # the SUFFIX form: listed => phantom. Do not use.
+];
+
+plan tests => scalar($tests->@*) * 3 + 2;
+
+for my $t ($tests->@*) {
+ my ($name, $listed, $reserves) = $t->@*;
+
+ is(!!($name =~ $rbd_lister_re), !!$listed, "rbd lister: '$name' listed = " . ($listed ? 1 : 0));
+ is(
+ !!($name =~ $lvmthin_lister_re), !!$listed,
+ "lvmthin lister: '$name' listed = " . ($listed ? 1 : 0),
+ );
+
+ # Unanchored on purpose -- this is the reservation half.
+ is(
+ !!($name =~ qr/(vm|base)-101-disk-(\d+)/), !!$reserves,
+ "'$name' reserves a disk number = " . ($reserves ? 1 : 0),
+ );
+}
+
+# The property that actually matters, through the real allocator rather than a regex:
+# with ONLY the parked placeholder present, disk 0 must not be handed out again.
+my $scfg = { type => 'rbd' };
+is(
+ PVE::Storage::Plugin::get_next_vm_diskname(['vm-101-disk-0'], 'st', 101, undef, $scfg),
+ 'vm-101-disk-1',
+ 'the real volume reserves its disk number',
+);
+is(
+ PVE::Storage::Plugin::get_next_vm_diskname(
+ ['copytmp-vm-101-disk-0'], 'st', 101, undef, $scfg,
+ ),
+ 'vm-101-disk-1',
+ 'a parked placeholder ALONE still reserves the disk number',
+);
+
+done_testing();
diff --git a/src/test/run_plugin_tests.pl b/src/test/run_plugin_tests.pl
index dda4cac..8f2472e 100755
--- a/src/test/run_plugin_tests.pl
+++ b/src/test/run_plugin_tests.pl
@@ -18,6 +18,7 @@ my $res = $harness->runtests(
"filesystem_path_test.pm",
"prune_backups_test.pm",
"copy_offload_test.pm",
+ "copy_offload_naming_test.pm",
);
exit -1 if !$res || $res->{failed} || $res->{parse_errors};
--
2.54.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [RFC PATCH storage 3/5] dir: implement copy-offload via reflink (FICLONE)
2026-07-20 14:29 [RFC PATCH storage, qemu-server 0/5] offload full clone to the storage backend Ciro Iriarte
2026-07-20 14:29 ` [RFC PATCH storage 1/5] storage: add asynchronous copy-offload hook for full copies Ciro Iriarte
2026-07-20 14:29 ` [RFC PATCH storage 2/5] rbd: implement copy-offload via snapshot + clone + flatten Ciro Iriarte
@ 2026-07-20 14:30 ` Ciro Iriarte
2026-07-20 14:30 ` [RFC PATCH storage 4/5] btrfs, lvmthin: implement copy-offload (atomic class) Ciro Iriarte
2026-07-20 14:30 ` [RFC PATCH qemu-server 5/5] use storage copy-offload for full clone Ciro Iriarte
4 siblings, 0 replies; 6+ messages in thread
From: Ciro Iriarte @ 2026-07-20 14:30 UTC (permalink / raw)
To: pve-devel
A filesystem that supports FICLONE -- XFS with reflink=1, btrfs, or ZFS with block
cloning -- can produce a full copy by sharing extents copy-on-write. The result is
INDEPENDENT immediately: the extents are reference counted, so deleting the source
does not affect the copy.
That makes this the cheapest member of the 'copy-offload-atomic' class. There is no
data movement and nothing to wait for, so copy_image_status() reports 'complete' on
the first poll and the asynchronous machinery simply does not engage. Today a full
clone on a directory storage byte-copies the whole image through qemu-img convert
even when the filesystem underneath could share the extents for free.
Three things are refused rather than half-done:
- A qcow2 that HAS A BACKING FILE. A byte-identical copy of an overlay inherits the
dependency: it passes qemu-img check and then breaks the moment the base is
removed, which is exactly the "readable but not independent" failure the hook's
contract exists to prevent. Only a real convert can flatten it, so the feature is
not advertised for such volumes -- deliberately not merely rejected in prepare,
because failing there would abort the clone instead of letting it take the normal
host-side path.
- Format conversion. FICLONE copies bytes; qcow2 -> raw needs qemu-img.
- A source snapshot, and a target on a different filesystem.
prepare() creates the target file rather than only choosing a name. The caller
releases the storage lock between prepare and start, so a name that was merely
chosen could be taken by a concurrent allocation, and the caller's rollback would
then delete that other operation's volume. An empty file costs nothing.
start() uses 'cp --reflink=always' so an unsupported filesystem fails loudly instead
of silently degrading into a full byte copy that would block a caller which may be
holding a guest frozen.
Verified on a loop-backed XFS (reflink=1) by driving the hooks directly: the clone
is byte-identical, status is 'complete' on the first poll, and the clone still
matches after the source is deleted. The guards were checked too -- a snapshot
source and a format conversion are both refused, a qcow2 with a backing file is not
advertised, and a plain qcow2 is. Independence and zero space use were separately
confirmed on XFS, btrfs and ZFS, where a 256 MiB clone added 0 MiB of allocation.
Generated-By: Claude (https://claude.ai)
Signed-off-by: Ciro Iriarte <ciro.iriarte+software@gmail.com>
Co-Authored-By: Claude <noreply@anthropic.com>
---
src/PVE/Storage/DirPlugin.pm | 140 +++++++++++++++++++++++++++++++++++
1 file changed, 140 insertions(+)
diff --git a/src/PVE/Storage/DirPlugin.pm b/src/PVE/Storage/DirPlugin.pm
index 80c4a03..cbc7357 100644
--- a/src/PVE/Storage/DirPlugin.pm
+++ b/src/PVE/Storage/DirPlugin.pm
@@ -8,6 +8,7 @@ use Encode qw(decode encode);
use File::Path;
use File::Spec;
use IO::File;
+use JSON;
use POSIX;
use PVE::Storage::Plugin;
@@ -95,6 +96,8 @@ sub options {
bwlimit => { optional => 1 },
preallocation => { optional => 1 },
'snapshot-as-volume-chain' => { optional => 1, fixed => 1 },
+ 'copy-offload' => { optional => 1 },
+ 'copy-offload-timeout' => { optional => 1 },
};
}
@@ -323,4 +326,141 @@ sub volume_qemu_snapshot_method {
return $scfg->{'snapshot-as-volume-chain'} ? 'mixed' : 'qemu';
}
+
+# qemu_img_info() returns raw JSON text, so decode before use. Returns the backing
+# filename, or undef when there is none / the image cannot be inspected.
+my sub qcow2_backing_file {
+ my ($path) = @_;
+ my $json = eval { PVE::Storage::Common::qemu_img_info($path, undef, 10) };
+ return undef if $@ || !$json;
+ my $info = eval { decode_json($json) };
+ return undef if $@ || ref($info) ne 'HASH';
+ return $info->{'backing-filename'};
+}
+
+sub volume_has_feature {
+ my ($class, $scfg, $feature, $storeid, $volname, $snapname, $running, $opts) = @_;
+
+ if ($feature eq 'copy-offload-atomic') {
+ # reflink clones a whole file; there is no way to pick a snapshot out of one
+ return 0 if $snapname;
+
+ my ($vtype, undef, undef, undef, undef, undef, $format) =
+ eval { $class->parse_volname($volname) };
+ return 0 if $@ || !defined($vtype) || $vtype ne 'images';
+ return 0 if $format ne 'raw' && $format ne 'qcow2';
+
+ # A qcow2 with a backing file cannot be flattened by a byte-identical copy, so
+ # do not advertise it -- otherwise the clone would fail in prepare instead of
+ # quietly taking the normal host-side path.
+ if ($format eq 'qcow2') {
+ my $path = eval { $class->filesystem_path($scfg, $volname) };
+ return 0 if $@ || !defined($path);
+ return 0 if qcow2_backing_file($path);
+ }
+
+ return 1;
+ }
+
+ return $class->SUPER::volume_has_feature(
+ $scfg, $feature, $storeid, $volname, $snapname, $running, $opts,
+ );
+}
+
+# ---- storage-offloaded full copy via reflink -------------------------------------
+#
+# A filesystem that supports FICLONE (XFS with reflink=1, btrfs, ZFS with block
+# cloning) can produce a full copy by sharing extents copy-on-write. Unlike an RBD
+# clone or a ZFS clone-from-snapshot, the result is INDEPENDENT straight away: the
+# extents are reference counted, so deleting the source does not affect the copy.
+#
+# That makes this the cheapest possible member of the 'copy-offload-atomic' class --
+# instant, no extra space, and nothing to wait for. copy_image_status() reports
+# 'complete' on the first poll because there is no background work.
+
+# Same filesystem? FICLONE cannot cross one, and the caller may be copying between
+# two different storages that happen to be directories.
+my sub same_filesystem {
+ my ($a, $b) = @_;
+ my $da = (stat($a))[0];
+ my $db = (stat($b))[0];
+ return defined($da) && defined($db) && $da == $db;
+}
+
+sub copy_image_prepare {
+ my (
+ $class, $scfg, $storeid, $volname,
+ $target_scfg, $target_storeid, $target_vmid, $snap, $opts,
+ ) = @_;
+
+ die "copy offload cannot copy from a snapshot\n" if defined($snap);
+
+ my ($vtype, undef, undef, undef, undef, undef, $format) = $class->parse_volname($volname);
+ die "copy offload only handles VM images, not '$vtype'\n" if $vtype ne 'images';
+
+ # FICLONE copies bytes; it cannot convert between formats.
+ my $target_format = $opts->{format} // $format;
+ die "copy offload cannot convert '$format' to '$target_format'\n"
+ if $target_format ne $format;
+
+ my $path = $class->filesystem_path($scfg, $volname);
+
+ # A qcow2 with a backing file is NOT independent, and a byte-identical copy of it
+ # inherits that dependency -- it would pass qemu-img check and then break when the
+ # base is removed. Only a real convert can flatten it.
+ die "copy offload cannot flatten '$volname': it has a backing file\n"
+ if $format eq 'qcow2' && qcow2_backing_file($path);
+
+ my $target_dir = $class->get_subdir($target_scfg, 'images') . "/$target_vmid";
+ mkpath $target_dir;
+
+ die "copy offload requires source and target on the same filesystem\n"
+ if !same_filesystem($path, $target_dir);
+
+ # the trailing 1 adds the format suffix; without it the volname does not parse
+ my $name =
+ $class->find_free_diskname($target_storeid, $target_scfg, $target_vmid, $format, 1);
+
+ # Reserve the name by creating the file. The caller drops the storage lock between
+ # prepare and start, so a name that was merely chosen could be taken by a
+ # concurrent allocation -- and the caller's rollback would then delete somebody
+ # else's volume. An empty file costs nothing and makes the target freeable.
+ my $target_path = "$target_dir/$name";
+ my $fh = IO::File->new($target_path, O_WRONLY | O_CREAT | O_EXCL, 0640)
+ or die "unable to reserve '$target_path' - $!\n";
+ close($fh);
+
+ return "$target_vmid/$name";
+}
+
+sub copy_image_start {
+ my (
+ $class, $scfg, $storeid, $volname,
+ $target_scfg, $target_storeid, $target_volname, $snap,
+ ) = @_;
+
+ my $src = $class->filesystem_path($scfg, $volname);
+ my $dst = $class->filesystem_path($target_scfg, $target_volname);
+
+ # --reflink=always so an unsupported filesystem fails loudly rather than silently
+ # turning this into a full byte copy that blocks the caller -- which may be holding
+ # a guest frozen.
+ eval { PVE::Tools::run_command(['/bin/cp', '--reflink=always', '--', $src, $dst]) };
+ if (my $err = $@) {
+ unlink($dst);
+ die "reflink copy of '$volname' failed - $err";
+ }
+
+ return;
+}
+
+sub copy_image_status {
+ my ($class, $scfg, $storeid, $volname, $source) = @_;
+
+ # FICLONE is synchronous and the extents are reference counted, so the copy is
+ # already independent of its source. Nothing to poll and nothing to clean up.
+ return { state => 'complete' };
+}
+
+
1;
--
2.54.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [RFC PATCH storage 4/5] btrfs, lvmthin: implement copy-offload (atomic class)
2026-07-20 14:29 [RFC PATCH storage, qemu-server 0/5] offload full clone to the storage backend Ciro Iriarte
` (2 preceding siblings ...)
2026-07-20 14:30 ` [RFC PATCH storage 3/5] dir: implement copy-offload via reflink (FICLONE) Ciro Iriarte
@ 2026-07-20 14:30 ` Ciro Iriarte
2026-07-20 14:30 ` [RFC PATCH qemu-server 5/5] use storage copy-offload for full clone Ciro Iriarte
4 siblings, 0 replies; 6+ messages in thread
From: Ciro Iriarte @ 2026-07-20 14:30 UTC (permalink / raw)
To: pve-devel
Both backends already have the right primitive and PVE already uses it for linked
clones -- 'btrfs subvolume snapshot' and 'lvcreate -s' on a thin LV. Both are instant,
allocate nothing, and are independent of their source immediately, so they satisfy
'copy-offload-atomic' with no background work at all: copy_image_status() is complete
on the first poll and the asynchronous machinery never engages.
Independence is the part worth being precise about, since it is what separates these
from a ZFS clone. Neither pins its source. btrfs reference counts extents and has no
notion of an origin that must outlive its snapshots; the thin pool reference counts
blocks, which is exactly the property already noted at the top of LvmThinPlugin.pm
("LVM thin allows deletion of such base volumes without affecting the linked clones").
So the copy can be handed back with no recorded parent, because it genuinely has none.
Scope, and why:
- btrfs handles 'raw' and 'subvol', the formats it stores inside subvolumes. qcow2 and
vmdk are plain files there and would need the reflink path instead, so they are not
advertised rather than failing later in prepare.
- lvmthin requires the same VG and the same thin pool. A thin snapshot shares blocks
within one pool and cannot leave it; copying elsewhere is a real data move.
The target name is reserved by creating it in prepare(). start() then parks that
reservation aside rather than deleting it, since neither primitive can write into a
name that already exists. Details that are easy to get wrong, and were:
- For btrfs 'raw' the reservation must include the disk.raw file, not just the
subvolume. list_images() stats that file and skips the entry when it is missing, so
a bare subvolume is invisible to find_free_diskname(). That does not merely weaken
the reservation: the core prepares every disk of a VM before starting any of them,
so the second prepare() for the same target picks the same name and dies on
'subvolume create'. Every multi-disk offloaded clone to btrfs raw would have failed.
- The lvmthin placeholder is parked under a 'copytmp-' PREFIX. As a suffix,
'vm-101-disk-0.copytmp' satisfies both list_images()' filter and parse_volname(), so
it appeared as a real disk belonging to VM 101 -- a phantom the GUI offers to attach.
The prefix stays outside both patterns while still holding the disk NUMBER, because
LVM's find_free_diskname() reads the raw LV list and matches unanchored.
- btrfs takes its snapshot under a staging name FIRST and only then swaps, so the
reserved name stays held for the duration of the copy rather than just after it.
- The same-filesystem check cannot use st_dev or statfs f_fsid. btrfs gives every
subvolume its own st_dev and folds the subvolume id into f_fsid, so both differ
between two subvolumes of ONE filesystem and rejected every legitimate copy; 'stat
-c %m' is fooled the same way. It resolves the containing mount and compares the
filesystem UUID, which also permits one btrfs mounted at several points via
'subvol=', a normal PVE arrangement.
btrfs swaps the finished copy into the reserved name with renameat2(RENAME_EXCHANGE)
rather than a park-then-rename pair, so the name is occupied at every instant -- by
either the placeholder or the copy. This matters because the reservation is NOT
otherwise held for btrfs: list_images()' name pattern rejects the '.copytmp' suffix, so
unlike lvmthin -- where find_free_diskname() reads the raw LV list and still sees the
parked LV -- a parked subvolume reserves nothing, and a concurrent allocation could take
the name and have the caller's rollback free somebody else's volume. The exchange
removes that window instead of narrowing it, needs no contract change, and uses the call
this plugin already makes to rotate a rollback target into place.
Neither start() removes its placeholder, and lvmthin's start() no longer sets the
autoactivation flag either. That call can run with a guest filesystem frozen, and both
are housekeeping nothing depends on -- every LVM command takes VG metadata locks that
can queue behind other activity, and for a multi-disk VM those add up inside one
freeze. copy_image_status() does them, outside. It is best effort throughout,
including its existence probe: dying there would make the caller free a copy that is
already complete and correct.
prepare() and free_image() both reap a placeholder left by a copy that died in
between. Without that the leftover is not just litter -- parking refuses to write onto
an existing name, so every later copy to that name fails, and the leftover is
invisible to list_images() so nothing would explain why.
Verified by copy_offload_functional_test.pl, added here: it builds a loop-backed btrfs
and a loop-backed thin pool, drives the hooks, and compares images byte for byte --
21 assertions, all passing. It covers a copy from a snapshot returning the SNAPSHOT's
content rather than live data, the copies staying identical after the source is
deleted outright, two prepares for one target VM returning different names, and a
leaked placeholder not wedging the disk name, and the reserved name still being held
between start() and status() -- the moment a park-then-rename would lose it. It needs root and real loop devices, so
it is not part of run_plugin_tests.pl and skips cleanly when it cannot run. The unit
tests pin what each plugin advertises, since advertising the wrong volume fails a
clone while failing to advertise a supported one silently falls back to a byte copy.
Note that RBDPlugin in this series still has the shape this replaces -- it removes the
placeholder and then renames -- so the same window exists there. Fixing it properly
needs identity-checked rollback (prepare returning a token the rollback verifies), which
is a hook-contract change and is being raised with the maintainers rather than decided
here.
Generated-By: Claude (https://claude.ai)
Signed-off-by: Ciro Iriarte <ciro.iriarte+software@gmail.com>
Co-Authored-By: Claude <noreply@anthropic.com>
---
src/PVE/Storage/BTRFSPlugin.pm | 250 ++++++++++++++++++++
src/PVE/Storage/LvmThinPlugin.pm | 189 +++++++++++++++
src/test/copy_offload_feature_test.pm | 98 ++++++++
src/test/copy_offload_functional_test.pl | 286 +++++++++++++++++++++++
src/test/run_plugin_tests.pl | 1 +
5 files changed, 824 insertions(+)
create mode 100644 src/test/copy_offload_feature_test.pm
create mode 100644 src/test/copy_offload_functional_test.pl
diff --git a/src/PVE/Storage/BTRFSPlugin.pm b/src/PVE/Storage/BTRFSPlugin.pm
index fb47aa0..7e2b2d1 100644
--- a/src/PVE/Storage/BTRFSPlugin.pm
+++ b/src/PVE/Storage/BTRFSPlugin.pm
@@ -78,6 +78,8 @@ sub options {
'create-base-path' => { optional => 1 },
'create-subdirs' => { optional => 1 },
preallocation => { optional => 1 },
+ 'copy-offload' => { optional => 1 },
+ 'copy-offload-timeout' => { optional => 1 },
# TODO: The new variant of mkdir with `populate` vs `create`...
};
}
@@ -625,6 +627,16 @@ sub volume_has_feature {
rename => {
current => { qcow2 => 1, raw => 1, vmdk => 1 },
},
+ # 'btrfs subvolume snapshot' is instant, shares extents copy-on-write, and is
+ # independent of its source straight away -- btrfs reference counts extents and
+ # does not pin an origin the way ZFS does. Only the formats this plugin stores
+ # as subvolumes qualify; qcow2 and vmdk are plain files here and would need the
+ # reflink path instead.
+ 'copy-offload-atomic' => {
+ base => { raw => 1, subvol => 1 },
+ current => { raw => 1, subvol => 1 },
+ snap => { raw => 1, subvol => 1 },
+ },
};
my ($vtype, $name, $vmid, $basename, $basevmid, $isBase, $format) =
@@ -1002,4 +1014,242 @@ sub get_import_metadata {
return PVE::Storage::DirPlugin::get_import_metadata(@_);
}
+# ---- storage-offloaded full copy via subvolume snapshot ---------------------------
+#
+# 'btrfs subvolume snapshot' is instant and shares extents copy-on-write. btrfs
+# reference counts those extents and has no concept of an origin that must outlive its
+# snapshots, so the copy is independent immediately -- deleting the source is fine.
+# That satisfies 'copy-offload-atomic' with no background work, so copy_image_status()
+# is complete on the first poll.
+#
+# This is the same primitive clone_image() uses. The difference is what PVE records
+# afterwards: a linked clone carries a dependency it has to respect, while here we can
+# hand back a volume with no parent, because on btrfs there genuinely is none.
+#
+# Only 'raw' and 'subvol' are handled: those are the formats this plugin stores inside
+# subvolumes. qcow2 and vmdk are plain files here, so they would need the reflink path
+# and are not advertised.
+
+# Resolve a volume to the subvolume that backs it, honouring $snapname.
+my sub volume_subvol {
+ my ($class, $scfg, $volname, $snapname) = @_;
+
+ my (undef, undef, undef, undef, undef, undef, $format) = $class->parse_volname($volname);
+ my $path = $class->filesystem_path($scfg, $volname, $snapname);
+
+ return $format eq 'raw' ? raw_file_to_subvol($path) : $path;
+}
+
+# btrfs cannot snapshot across filesystems, and the target may be a different storage.
+#
+# Deliberately NOT st_dev or statfs f_fsid: btrfs gives every subvolume its own st_dev,
+# and folds the subvolume id into f_fsid too, so both differ between two subvolumes of
+# the ONE filesystem and would reject every legitimate copy. The filesystem UUID is the
+# only stable identity. This costs a command, but it runs in prepare(), never in the
+# freeze-sensitive start().
+my sub btrfs_fsid {
+ my ($path) = @_;
+
+ # 'btrfs filesystem show' takes a device or a mount point, not an arbitrary path,
+ # so resolve the containing mount first. Note this cannot be done with stat():
+ # coreutils' own %m reports a btrfs subvolume as its own mount point, for the same
+ # st_dev reason.
+ my $mnt;
+ eval {
+ run_command(
+ ['findmnt', '--noheadings', '--output', 'TARGET', '--target', $path],
+ outfunc => sub { $mnt //= $_[0] if $_[0] =~ /\S/ },
+ );
+ };
+ return (undef, " - $@") if $@;
+ return (undef, " - findmnt reported no mount point") if !defined($mnt);
+ chomp $mnt;
+
+ # Going by UUID rather than by mount point on purpose: one btrfs filesystem can be
+ # mounted at several places (a 'subvol=' mount per storage is a normal PVE setup),
+ # and a snapshot between them is perfectly valid.
+ my $uuid;
+ my $stderr = '';
+ eval {
+ run_command(
+ ['btrfs', 'filesystem', 'show', '--', $mnt],
+ outfunc => sub {
+ my ($line) = @_;
+ $uuid = $1 if !defined($uuid) && $line =~ m/\buuid:\s*(\S+)/i;
+ },
+ # A non-btrfs target is an ordinary "cannot offload this" answer, not a
+ # fault worth printing to the task log. The reason is returned to the
+ # caller instead, so it still ends up in the error it raises.
+ errfunc => sub { $stderr .= $_[0] },
+ );
+ };
+ return (undef, " - $@") if $@;
+ return (undef, $stderr =~ /\S/ ? " - $stderr" : " - no uuid in 'btrfs filesystem show'")
+ if !defined($uuid);
+ return ($uuid, undef);
+}
+
+sub copy_image_prepare {
+ my (
+ $class, $scfg, $storeid, $volname,
+ $target_scfg, $target_storeid, $target_vmid, $snap, $opts,
+ ) = @_;
+
+ my ($vtype, undef, undef, undef, undef, undef, $format) = $class->parse_volname($volname);
+ die "copy offload only handles VM images, not '$vtype'\n" if $vtype ne 'images';
+ die "btrfs copy offload cannot handle format '$format'\n"
+ if $format ne 'raw' && $format ne 'subvol';
+
+ # A subvolume snapshot copies the subvolume as it is; it cannot convert formats.
+ my $target_format = $opts->{format} // $format;
+ die "btrfs copy offload cannot convert '$format' to '$target_format'\n"
+ if $target_format ne $format;
+
+ my $subvol = volume_subvol($class, $scfg, $volname, $snap);
+ die "cannot copy '$volname': '$subvol' does not exist\n" if !-e $subvol;
+
+ my $imagedir = $class->get_subdir($target_scfg, 'images') . "/$target_vmid";
+ mkpath $imagedir;
+
+ # Keep "could not work out which filesystem this is" distinct from "they are two
+ # different filesystems". Reporting a mismatch when findmnt or btrfs is missing, or
+ # when the path is not on btrfs at all, sends people looking in the wrong place.
+ # findmnt needs the directory to exist, hence the mkpath above -- so tidy it away
+ # again on the paths that reject, rather than littering the target storage with an
+ # empty directory per refused attempt.
+ my $reject = sub {
+ rmdir($imagedir); # only succeeds while empty, which is what we want
+ die $_[0];
+ };
+ my ($src_fsid, $src_err) = btrfs_fsid($subvol);
+ $reject->("cannot determine the btrfs filesystem of '$subvol'$src_err\n")
+ if !defined($src_fsid);
+ my ($dst_fsid, $dst_err) = btrfs_fsid($imagedir);
+ $reject->("cannot determine the btrfs filesystem of '$imagedir'$dst_err\n")
+ if !defined($dst_fsid);
+ $reject->("copy offload requires source and target on the same btrfs filesystem\n")
+ if $src_fsid ne $dst_fsid;
+
+ # the trailing 1 adds the format suffix; without it the volname does not parse
+ my $name =
+ $class->find_free_diskname($target_storeid, $target_scfg, $target_vmid, $format, 1);
+ my $target_volname = "$target_vmid/$name";
+
+ # Actually create the target, do not just pick a name. The caller runs this under
+ # the target storage lock and releases it before copy_image_start(), so a name that
+ # was merely chosen could be taken by a concurrent allocation in between -- and the
+ # caller's rollback would then free a volume belonging to that other operation.
+ # An empty subvolume costs nothing and is what free_image() already knows how to
+ # remove.
+ my $newsubvol = volume_subvol($class, $target_scfg, $target_volname, undef);
+
+ # Reap a placeholder left by an earlier copy that died between start() and
+ # status(). It is not merely litter: the rename in start() refuses to park onto an
+ # existing path, so without this every future offloaded copy to this name fails --
+ # and the leftover is invisible to list_images(), so nobody would know why. Safe
+ # here: we hold the target storage lock, we are outside any freeze, and the name is
+ # ours because find_free_diskname() just handed it out.
+ my $stale = "$newsubvol.copytmp";
+ if (-e $stale) {
+ warn "removing stale copy placeholder '$stale'\n";
+ $class->btrfs_cmd(['subvolume', 'delete', '--', $stale]);
+ }
+
+ $class->btrfs_cmd(['subvolume', 'create', '--', $newsubvol]);
+
+ # For 'raw' the volume is the disk.raw INSIDE the subvolume, and that file is what
+ # list_images() stats. Without it file_size_info() returns undef and the entry is
+ # skipped, which would make this reservation invisible to find_free_diskname() --
+ # defeating the point of creating it, and worse: the core prepares every disk of a
+ # VM before starting any of them, so the second prepare() for the same target would
+ # pick this same name and die on 'subvolume create'. Every multi-disk offloaded
+ # clone would fail. Create the file too, exactly as alloc_image does.
+ if ($format eq 'raw') {
+ my $raw = "$newsubvol/disk.raw";
+ my $fh;
+ if (!sysopen($fh, $raw, O_WRONLY | O_CREAT | O_EXCL, 0640)) {
+ my $err = $!;
+ eval { $class->btrfs_cmd(['subvolume', 'delete', '--', $newsubvol]); };
+ warn $@ if $@;
+ die "unable to reserve '$raw' - $err\n";
+ }
+ close($fh);
+ }
+
+ return $target_volname;
+}
+
+sub copy_image_start {
+ my (
+ $class, $scfg, $storeid, $volname,
+ $target_scfg, $target_storeid, $target_volname, $snap,
+ ) = @_;
+
+ # Snapshot the source the caller asked for. Falling back to the current subvolume
+ # when a snapshot was requested would silently copy live data instead.
+ my $subvol = volume_subvol($class, $scfg, $volname, $snap);
+ my $newsubvol = volume_subvol($class, $target_scfg, $target_volname, undef);
+
+ # copy_image_prepare() reserved the name with a placeholder subvolume, and
+ # 'subvolume snapshot' will not write into a path that already exists. So: snapshot
+ # to a staging name, then ATOMICALLY EXCHANGE staging and the reserved name.
+ #
+ # PVE::Tools::renameat2(RENAME_EXCHANGE) rather than a pair of rename(2)s. Parking
+ # the placeholder aside and moving the copy in afterwards leaves the reserved name
+ # unheld in between, and for btrfs that genuinely loses the reservation: unlike
+ # lvmthin -- where find_free_diskname() reads the raw LV list and still sees the
+ # parked LV -- btrfs goes through list_images(), whose name pattern rejects the
+ # '.copytmp' suffix. A concurrent allocation could take the name, and the caller's
+ # rollback would then free somebody else's volume. With the exchange the name is
+ # occupied at every instant, by either the placeholder or the finished copy, so
+ # that window does not exist. The placeholder simply ends up at the staging path.
+ my $staging = "$newsubvol.copytmp";
+
+ # The expensive part, entirely off to one side; the reserved name is untouched, so
+ # a failure here needs no unwinding beyond dropping the staging subvolume.
+ eval { $class->btrfs_cmd(['subvolume', 'snapshot', '--', $subvol, $staging]); };
+ if (my $err = $@) {
+ eval { $class->btrfs_cmd(['subvolume', 'delete', '--', $staging]) if -e $staging; };
+ warn $@ if $@;
+ die $err;
+ }
+
+ # The paths are absolute, so pass -1 as the file descriptors -- same call this
+ # plugin already makes when rotating a rollback target into place.
+ if (!PVE::Tools::renameat2(-1, $staging, -1, $newsubvol, &PVE::Tools::RENAME_EXCHANGE)) {
+ my $rerr = $!;
+ # Nothing moved: the reserved name still holds the placeholder, so the caller's
+ # rollback frees exactly what it was given.
+ eval { $class->btrfs_cmd(['subvolume', 'delete', '--', $staging]); };
+ warn $@ if $@;
+ die "unable to swap the copy into '$newsubvol' - $rerr\n";
+ }
+
+ # The displaced placeholder, now sitting at the staging path, is deliberately NOT
+ # removed here. This runs while the caller may hold a guest filesystem frozen, and
+ # removing it is pure cleanup nothing depends on -- copy_image_status() does it.
+ return;
+}
+
+sub copy_image_status {
+ my ($class, $scfg, $storeid, $volname, $source) = @_;
+
+ # $volname is the TARGET. The snapshot is complete and independent the moment btrfs
+ # returns, so there is nothing to poll; what is left is dropping the placeholder
+ # copy_image_start() parked, which is done here to keep it out of the freeze window.
+ my ($vtype, undef, undef, undef, undef, undef, $format) =
+ eval { $class->parse_volname($volname) };
+ if (!$@ && defined($vtype) && $vtype eq 'images' && defined($format)
+ && ($format eq 'raw' || $format eq 'subvol'))
+ {
+ my $parked = volume_subvol($class, $scfg, $volname, undef) . '.copytmp';
+ if (-e $parked) {
+ eval { $class->btrfs_cmd(['subvolume', 'delete', '--', $parked]); };
+ warn $@ if $@;
+ }
+ }
+
+ return { state => 'complete' };
+}
+
1
diff --git a/src/PVE/Storage/LvmThinPlugin.pm b/src/PVE/Storage/LvmThinPlugin.pm
index cadf343..2f0fc05 100644
--- a/src/PVE/Storage/LvmThinPlugin.pm
+++ b/src/PVE/Storage/LvmThinPlugin.pm
@@ -54,6 +54,8 @@ sub options {
disable => { optional => 1 },
content => { optional => 1 },
bwlimit => { optional => 1 },
+ 'copy-offload' => { optional => 1 },
+ 'copy-offload-timeout' => { optional => 1 },
};
}
@@ -130,6 +132,18 @@ sub alloc_image {
return $name;
}
+# Where copy_image_start() parks the reservation while it creates the snapshot.
+#
+# A PREFIX, not a suffix: list_images() selects on m/^(vm|base)-(\d+)-/ and
+# parse_volname() accepts m/^((vm|base)-(\d+)-\S+)$/, so 'vm-101-disk-0.copytmp' would
+# be a perfectly valid volume name and would show up as a disk belonging to VM 101 --
+# a phantom the GUI offers to attach as an unused disk, and one that outlives the copy
+# if it ever leaks. Prefixing puts it outside both patterns.
+my sub parked_name {
+ my ($volname) = @_;
+ return "copytmp-$volname";
+}
+
sub free_image {
my ($class, $storeid, $scfg, $volname, $isBase) = @_;
@@ -146,6 +160,13 @@ sub free_image {
run_command($cmd, errmsg => "lvremove snapshot '$vg/$lv' error");
}
+ # a copy placeholder parked under this name, if a copy died mid-flight
+ my $parked = parked_name($volname);
+ if ($dat->{$parked}) {
+ my $cmd = ['/sbin/lvremove', '-f', "$vg/$parked"];
+ run_command($cmd, errmsg => "lvremove copy placeholder '$vg/$parked' error");
+ }
+
# finally remove original (if exists)
if ($dat->{$volname}) {
my $cmd = ['/sbin/lvremove', '-f', "$vg/$volname"];
@@ -415,6 +436,11 @@ sub volume_has_feature {
copy => { base => 1, current => 1, snap => 1 },
sparseinit => { base => 1, current => 1 },
rename => { current => 1 },
+ # A thin snapshot is instant, allocates nothing, and is independent of its
+ # origin straight away -- see the note at the top of this file: the origin can
+ # be deleted without affecting it. Snapshots of a snapshot work too, so all
+ # three keys apply.
+ 'copy-offload-atomic' => { base => 1, current => 1, snap => 1 },
};
my ($vtype, $name, $vmid, $basename, $basevmid, $isBase) = $class->parse_volname($volname);
@@ -507,4 +533,167 @@ sub rename_snapshot {
die "rename_snapshot is not supported for $class";
}
+# ---- storage-offloaded full copy via thin snapshot --------------------------------
+#
+# 'lvcreate -s' on a thin LV is instant, allocates no data blocks, and -- unlike a ZFS
+# clone -- does not pin its origin: the thin pool reference counts blocks, so the origin
+# can be removed while the copy lives on. That is the note at the top of this file, and
+# it is exactly what 'copy-offload-atomic' requires, so there is no background work and
+# copy_image_status() is complete on the first poll.
+#
+# This is the same primitive clone_image() already uses for linked clones. The
+# difference is only in what PVE believes afterwards: a linked clone records a
+# dependency it must respect, while this path is free to hand back a volume with no
+# recorded parent, because thin snapshots genuinely have none.
+
+my sub thin_lv_exists {
+ my ($vg, $lv) = @_;
+ my $lvs = PVE::Storage::LVMPlugin::lvm_list_volumes($vg);
+ return defined($lvs->{$vg}) && defined($lvs->{$vg}->{$lv});
+}
+
+sub copy_image_prepare {
+ my (
+ $class, $scfg, $storeid, $volname,
+ $target_scfg, $target_storeid, $target_vmid, $snap, $opts,
+ ) = @_;
+
+ my $format = $opts->{format} // 'raw';
+ die "lvmthin copy offload cannot produce format '$format'\n" if $format ne 'raw';
+
+ my ($vtype) = $class->parse_volname($volname);
+ die "copy offload only handles VM images, not '$vtype'\n" if $vtype ne 'images';
+
+ # A thin snapshot shares blocks with its origin inside one pool, so it cannot leave
+ # that pool. Copying to another VG or another thinpool is a real data move and has
+ # to take the normal host-side path.
+ my $vg = $scfg->{vgname};
+ die "copy offload requires source and target in the same volume group\n"
+ if ($target_scfg->{vgname} // '') ne $vg;
+ die "copy offload requires source and target in the same thin pool\n"
+ if ($target_scfg->{thinpool} // '') ne ($scfg->{thinpool} // '');
+
+ my $src_lv = defined($snap) ? "snap_${volname}_$snap" : $volname;
+ die "cannot copy '$volname': source volume '$src_lv' does not exist\n"
+ if !thin_lv_exists($vg, $src_lv);
+
+ my $name = $class->find_free_diskname($target_storeid, $target_scfg, $target_vmid);
+
+ # Reap a placeholder left by an earlier copy that died between start() and
+ # status(). It is not merely litter: 'lvrename' below refuses to park onto an
+ # existing name, so without this every future offloaded copy to this name fails.
+ # Doing it here is safe -- we hold the target storage lock and are outside any
+ # freeze -- and the name is ours, since find_free_diskname() just handed it out.
+ my $stale = parked_name($name);
+ if (thin_lv_exists($vg, $stale)) {
+ warn "removing stale copy placeholder '$vg/$stale'\n";
+ run_command(
+ ['/sbin/lvremove', '-f', "$vg/$stale"],
+ errmsg => "lvremove stale placeholder '$vg/$stale' error",
+ );
+ }
+
+ # Actually create the target, do not just pick a name. The caller runs this under
+ # the target storage lock and releases it before copy_image_start(), so a name that
+ # was merely chosen could be taken by a concurrent allocation in between -- and the
+ # caller's rollback would then free a volume belonging to that other operation.
+ #
+ # A thin LV is virtual, so this placeholder allocates no data blocks whatever size
+ # it claims; 1k is simply the smallest lvcreate accepts and rounds up.
+ my $cmd = [
+ '/sbin/lvcreate', '-aly', '-V', '1k', '--name', $name,
+ '--thinpool', "$vg/$scfg->{thinpool}",
+ ];
+ run_command($cmd, errmsg => "lvcreate placeholder '$vg/$name' error");
+ $set_lv_autoactivation->($vg, $name, 0);
+
+ return $name;
+}
+
+sub copy_image_start {
+ my (
+ $class, $scfg, $storeid, $volname,
+ $target_scfg, $target_storeid, $target_volname, $snap,
+ ) = @_;
+
+ my $vg = $scfg->{vgname};
+
+ # Snapshot the source the caller asked for. Falling back to the current LV when a
+ # snapshot was requested would silently copy live data instead.
+ my $src_lv = defined($snap) ? "snap_${volname}_$snap" : $volname;
+
+ # copy_image_prepare() reserved the name with a placeholder, and 'lvcreate -s'
+ # cannot write into a name that already exists. Rename the placeholder aside rather
+ # than removing it, so the reserved name is never momentarily free for a concurrent
+ # find_free_diskname() to hand out.
+ my $parked = parked_name($target_volname);
+ run_command(
+ ['/sbin/lvrename', $vg, $target_volname, $parked],
+ errmsg => "lvrename placeholder '$vg/$target_volname' error",
+ );
+
+ eval {
+ # ONLY the snapshot. It is what fixes the point in time; everything else this
+ # copy needs is done by copy_image_status(), outside the freeze.
+ my $cmd = ['/sbin/lvcreate', '-n', $target_volname, '-prw', '-kn', '-s', "$vg/$src_lv"];
+ run_command($cmd, errmsg => "thin snapshot of '$vg/$src_lv' error");
+ };
+ if (my $err = $@) {
+ # Put the reservation back so the caller's rollback still finds the volume it
+ # was given, and leave the source untouched. If that fails too, remove the
+ # parked LV rather than leaving an orphan: the rollback frees $target_volname,
+ # which by then names nothing, so nothing else would ever reap it.
+ eval {
+ run_command(
+ ['/sbin/lvrename', $vg, $parked, $target_volname],
+ errmsg => "restoring placeholder '$vg/$target_volname' error",
+ );
+ };
+ if (my $rerr = $@) {
+ eval { run_command(['/sbin/lvremove', '-f', "$vg/$parked"]) };
+ $err .= "additionally, could not restore the reserved name: $rerr";
+ $err .= "and '$vg/$parked' is left behind\n" if $@;
+ }
+ die $err;
+ }
+
+ # The parked placeholder is deliberately NOT removed here. This runs while the
+ # caller may hold a guest filesystem frozen, and every LVM command takes VG
+ # metadata locks that can queue behind other activity on the node -- for a
+ # multi-disk VM those add up inside a single freeze. Only the snapshot above fixes
+ # the point in time; removal is cleanup, and copy_image_status() does it outside.
+ return;
+}
+
+sub copy_image_status {
+ my ($class, $scfg, $storeid, $volname, $source) = @_;
+
+ # $volname is the TARGET. The thin snapshot is complete and independent the moment
+ # lvcreate returns, so there is nothing to poll and nothing of the source to
+ # release. What is left is dropping the placeholder copy_image_start() parked,
+ # which happens here to keep it out of the freeze window.
+ my $vg = $scfg->{vgname};
+
+ # Everything here is best effort ON PURPOSE, including the existence check. This
+ # runs in the caller's poll loop, and dying makes it free a copy that is already
+ # complete, correct and independent -- losing real data because a cleanup probe hit
+ # a transient LVM lock, which is exactly the contention this plugin already works
+ # around elsewhere. Deferring the autoactivation flag to here for the same reason it
+ # is not done in start(): it is metadata housekeeping that takes the same VG lock.
+ eval {
+ my $parked = parked_name($volname);
+ if (thin_lv_exists($vg, $parked)) {
+ run_command(
+ ['/sbin/lvremove', '-f', "$vg/$parked"],
+ errmsg => "lvremove placeholder '$vg/$parked' error",
+ );
+ }
+ };
+ warn $@ if $@;
+
+ $set_lv_autoactivation->($vg, $volname, 0);
+
+ return { state => 'complete' };
+}
+
1;
diff --git a/src/test/copy_offload_feature_test.pm b/src/test/copy_offload_feature_test.pm
new file mode 100644
index 0000000..76fcecd
--- /dev/null
+++ b/src/test/copy_offload_feature_test.pm
@@ -0,0 +1,98 @@
+package PVE::Storage::TestCopyOffloadFeature;
+
+use strict;
+use warnings;
+
+use lib qw(..);
+
+# PVE::Storage first: it registers the plugins in an order that resolves the
+# DirPlugin <-> Storage.pm circular load. Requiring BTRFSPlugin on its own fails.
+use PVE::Storage;
+use PVE::Storage::BTRFSPlugin;
+use PVE::Storage::LvmThinPlugin;
+use Test::More;
+
+# Which volumes a plugin advertises 'copy-offload-atomic' for. This is the gate the
+# whole path hangs off: advertising a volume the plugin cannot actually copy sends the
+# clone into copy_image_prepare() only to die there, and NOT advertising one it can
+# copy silently falls back to a full host-side byte copy. Neither failure is visible
+# from a passing clone, so the answers are pinned here.
+
+my $btrfs_scfg = { path => '/some/btrfs', type => 'btrfs' };
+my $lvmthin_scfg = { vgname => 'vg0', thinpool => 'tp', type => 'lvmthin' };
+
+my $tests = [
+ # [ description, class, scfg, volname, snapname, expected ]
+
+ # btrfs stores raw and subvol as subvolumes, which is what it can snapshot.
+ [
+ 'btrfs raw is advertised',
+ 'PVE::Storage::BTRFSPlugin', $btrfs_scfg, '100/vm-100-disk-0.raw', undef, 1,
+ ],
+ [
+ 'btrfs subvol is advertised',
+ 'PVE::Storage::BTRFSPlugin', $btrfs_scfg, '100/subvol-100-disk-0.subvol', undef, 1,
+ ],
+ [
+ 'btrfs raw is advertised from a snapshot too',
+ 'PVE::Storage::BTRFSPlugin', $btrfs_scfg, '100/vm-100-disk-0.raw', 'snap1', 1,
+ ],
+ # qcow2 and vmdk are plain files here, not subvolumes, so the subvolume-snapshot
+ # path does not apply to them.
+ [
+ 'btrfs qcow2 is NOT advertised',
+ 'PVE::Storage::BTRFSPlugin', $btrfs_scfg, '100/vm-100-disk-0.qcow2', undef, undef,
+ ],
+ [
+ 'btrfs vmdk is NOT advertised',
+ 'PVE::Storage::BTRFSPlugin', $btrfs_scfg, '100/vm-100-disk-0.vmdk', undef, undef,
+ ],
+
+ # lvmthin is raw-only, and a thin snapshot does not pin its origin, so every key
+ # qualifies.
+ [
+ 'lvmthin current is advertised',
+ 'PVE::Storage::LvmThinPlugin', $lvmthin_scfg, 'vm-100-disk-0', undef, 1,
+ ],
+ [
+ 'lvmthin base is advertised',
+ 'PVE::Storage::LvmThinPlugin', $lvmthin_scfg, 'base-100-disk-0', undef, 1,
+ ],
+ [
+ 'lvmthin snapshot is advertised',
+ 'PVE::Storage::LvmThinPlugin', $lvmthin_scfg, 'vm-100-disk-0', 'snap1', 1,
+ ],
+];
+
+plan tests => scalar($tests->@*) + 2;
+
+for my $t ($tests->@*) {
+ my ($desc, $class, $scfg, $volname, $snapname, $expected) = $t->@*;
+
+ my $got = $class->volume_has_feature(
+ $scfg, 'copy-offload-atomic', 'store', $volname, $snapname, 0,
+ );
+
+ if (defined($expected)) {
+ is($got, $expected, $desc);
+ } else {
+ ok(!$got, $desc);
+ }
+}
+
+# An unrelated feature must still come back from the normal table rather than being
+# swallowed by the copy-offload handling.
+ok(
+ PVE::Storage::BTRFSPlugin->volume_has_feature(
+ $btrfs_scfg, 'snapshot', 'store', '100/vm-100-disk-0.raw', undef, 0,
+ ),
+ 'btrfs still answers for unrelated features',
+);
+ok(
+ PVE::Storage::LvmThinPlugin->volume_has_feature(
+ $lvmthin_scfg, 'snapshot', 'store', 'vm-100-disk-0', undef, 0,
+ ),
+ 'lvmthin still answers for unrelated features',
+);
+
+done_testing();
diff --git a/src/test/copy_offload_functional_test.pl b/src/test/copy_offload_functional_test.pl
new file mode 100644
index 0000000..9b3e849
--- /dev/null
+++ b/src/test/copy_offload_functional_test.pl
@@ -0,0 +1,286 @@
+#!/usr/bin/perl
+
+# Functional tests for the copy-offload hooks of BTRFSPlugin and LvmThinPlugin.
+#
+# Unlike the rest of src/test, this one touches real storage: it builds a loop-backed
+# btrfs filesystem and a loop-backed LVM thin pool, drives copy_image_prepare/start/
+# status against them, and checks the resulting images byte for byte. That needs root
+# and the btrfs/lvm tools, so it is NOT part of run_plugin_tests.pl -- run it by hand:
+#
+# perl copy_offload_functional_test.pl
+#
+# It skips cleanly rather than failing when it cannot run. Everything it creates lives
+# under /tmp and on its own loop devices, and is torn down at exit even on failure; it
+# never touches an existing volume group or mount.
+#
+# What is actually being checked, and why these and not others:
+#
+# - a copy taken from a SNAPSHOT returns the snapshot's content, not the live volume's.
+# Getting this wrong is silent: the clone is readable and passes every check, it just
+# holds the wrong data.
+# - the copy survives deleting the source outright. That is what 'copy-offload-atomic'
+# promises and what separates these backends from a ZFS clone.
+# - two prepares for the SAME target VM return different names. The core prepares every
+# disk of a VM before starting any of them, so a reservation its own lister cannot see
+# makes every multi-disk clone fail.
+
+use strict;
+use warnings;
+
+use lib qw(..);
+
+use File::Path qw(mkpath rmtree);
+use PVE::Storage;
+use PVE::Storage::BTRFSPlugin;
+use PVE::Storage::LvmThinPlugin;
+use PVE::Tools qw(run_command);
+use Test::More;
+
+my $BTRFS_MNT = '/tmp/pve-copyoffload-btrfs';
+my $BTRFS_IMG = '/tmp/pve-copyoffload-btrfs.img';
+my $LVM_IMG = '/tmp/pve-copyoffload-lvm.img';
+my $VG = 'pvecopyoffloadtest';
+
+my @cleanup;
+
+sub sh { return scalar(qx{$_[0] 2>/dev/null}) }
+
+sub cleanup_all {
+ for my $c (reverse @cleanup) { eval { $c->() }; }
+ @cleanup = ();
+}
+END { cleanup_all() }
+$SIG{INT} = $SIG{TERM} = sub { cleanup_all(); exit 1 };
+
+if ($> != 0) {
+ plan skip_all => 'needs root to create loop devices, filesystems and volume groups';
+}
+for my $tool (qw(btrfs mkfs.btrfs losetup findmnt lvcreate vgcreate pvcreate)) {
+ if (!sh("command -v $tool")) {
+ plan skip_all => "missing required tool '$tool'";
+ }
+}
+if (sh("vgs --noheadings -o vg_name 2>/dev/null") =~ /\b\Q$VG\E\b/) {
+ plan skip_all => "volume group '$VG' already exists - refusing to touch it";
+}
+
+plan tests => 21;
+
+# Hash the WHOLE object, and let md5sum do its own reading.
+#
+# Deliberately not 'dd bs=1M count=N | md5sum': dd counts a short read as a full block,
+# so it can return less than asked for, and how much depends on whether the data came
+# from the page cache or off the disk. Comparing a freshly written source against a
+# cold copy that way reports a mismatch between two byte-identical files. It also hides
+# a missing file as the md5 of empty input.
+sub md5_of {
+ my ($path) = @_;
+
+ die "cannot hash '$path': not present\n" if !-e $path;
+ my $out = sh("md5sum '$path'");
+ $out =~ s/\s.*//s;
+ die "md5sum of '$path' produced nothing\n" if $out !~ /^[0-9a-f]{32}$/;
+ return $out;
+}
+
+sub identical {
+ my ($a, $b) = @_;
+ return system('cmp', '-s', $a, $b) == 0;
+}
+
+# ---------------------------------------------------------------- btrfs
+
+my $btrfs_ok = eval {
+ run_command(['truncate', '-s', '2G', $BTRFS_IMG]);
+ push @cleanup, sub { unlink $BTRFS_IMG };
+
+ my $loop = sh("losetup --find --show $BTRFS_IMG");
+ chomp $loop;
+ die "no loop device\n" if !$loop;
+ push @cleanup, sub { sh("losetup -d $loop") };
+
+ run_command(['mkfs.btrfs', '-q', '-f', $loop]);
+ mkpath $BTRFS_MNT;
+ run_command(['mount', $loop, $BTRFS_MNT]);
+ push @cleanup, sub {
+ for my $s (reverse split /\n/, sh("btrfs subvolume list -o $BTRFS_MNT | awk '{print \$NF}'")) {
+ sh("btrfs -q subvolume delete '$BTRFS_MNT/$s'");
+ }
+ sh("umount $BTRFS_MNT");
+ rmtree $BTRFS_MNT;
+ };
+ mkpath "$BTRFS_MNT/images";
+ 1;
+};
+if (!$btrfs_ok) {
+ diag("btrfs setup failed: $@");
+ SKIP: { skip 'btrfs setup failed', 11 }
+} else {
+ my $C = 'PVE::Storage::BTRFSPlugin';
+ my $scfg = { path => $BTRFS_MNT, type => 'btrfs', content => { images => 1 } };
+
+ my $src = $C->alloc_image('bt', $scfg, 100, 'raw', undef, 64 * 1024);
+ my $srcpath = $C->filesystem_path($scfg, $src);
+ sh("dd if=/dev/urandom of=$srcpath bs=1M count=16 conv=notrunc,fsync status=none");
+
+ $C->volume_snapshot($scfg, 'bt', $src, 'snap1');
+ my $snappath = $C->filesystem_path($scfg, $src, 'snap1');
+ sh("dd if=/dev/urandom of=$srcpath bs=1M count=16 conv=notrunc,fsync status=none");
+ ok(!identical($srcpath, $snappath), 'btrfs: source diverged from its snapshot');
+
+ # THE multi-disk case: the core prepares every disk before starting any of them.
+ my $a = $C->copy_image_prepare($scfg, 'bt', $src, $scfg, 'bt', 201, undef, {});
+ my $b = $C->copy_image_prepare($scfg, 'bt', $src, $scfg, 'bt', 201, undef, {});
+ isnt($a, $b, 'btrfs: two prepares for one target VM reserve different names');
+
+ $C->copy_image_start($scfg, 'bt', $src, $scfg, 'bt', $a, undef);
+
+ # After start() but BEFORE status() cleans up, the reserved name must still be held.
+ # start() swaps the copy in with RENAME_EXCHANGE precisely so the name is never
+ # momentarily free; a plain park-then-rename loses the reservation exactly here, and
+ # a concurrent allocation could then take the name out from under the caller's
+ # rollback. Checked between the two calls on purpose -- that is the fragile moment.
+ my ($a_name) = $a =~ m{/(.*)$};
+ isnt(
+ $C->find_free_diskname('bt', $scfg, 201, 'raw', 1), $a_name,
+ 'btrfs: the reserved name is still held across the swap',
+ );
+
+ my $st = $C->copy_image_status($scfg, 'bt', $a, undef);
+ is($st->{state}, 'complete', 'btrfs: status complete on the first poll');
+ my $apath = $C->filesystem_path($scfg, $a);
+ ok(identical($apath, $srcpath), 'btrfs: copy matches the source');
+
+ my $snapcopy = $C->copy_image_prepare($scfg, 'bt', $src, $scfg, 'bt', 202, 'snap1', {});
+ $C->copy_image_start($scfg, 'bt', $src, $scfg, 'bt', $snapcopy, 'snap1');
+ $C->copy_image_status($scfg, 'bt', $snapcopy, undef);
+ # NB: assign filesystem_path() to a scalar first. It returns ($path, $vmid, $vtype)
+ # in list context, and sub arguments ARE list context -- passing the call directly
+ # would hand identical() the vmid as its second path.
+ my $snapcopy_path = $C->filesystem_path($scfg, $snapcopy);
+ ok(
+ identical($snapcopy_path, $snappath),
+ 'btrfs: a copy from a snapshot holds the SNAPSHOT content, not live data',
+ );
+
+ # Independence: hash the copy, destroy the source outright, hash again.
+ my $before = md5_of($apath);
+ $C->free_image('bt', $scfg, $src, 0);
+ ok(!-e $srcpath, 'btrfs: source really is gone');
+ is($before, md5_of($apath), 'btrfs: copy is unaffected by deleting the source');
+ unlike(
+ sh("find $BTRFS_MNT -maxdepth 4 -name '*.copytmp' -o -name '*.copynew'"), qr/\S/,
+ 'btrfs: no parked or staging placeholder left behind',
+ );
+
+ # Failure path: same as the lvmthin case -- a placeholder left by a copy that died
+ # between start() and status() must not wedge the name, since rename() refuses to
+ # park onto an existing path and the leftover is invisible to list_images().
+ my $stale = $C->copy_image_prepare($scfg, 'bt', $a, $scfg, 'bt', 203, undef, {});
+ my $stale_subvol = $C->filesystem_path($scfg, $stale);
+ $stale_subvol =~ s|/disk\.raw$||;
+ rename($stale_subvol, "$stale_subvol.copytmp")
+ or die "could not stage the leaked-placeholder case - $!\n";
+ my $reused = eval { $C->copy_image_prepare($scfg, 'bt', $a, $scfg, 'bt', 203, undef, {}) };
+ ok(defined($reused), 'btrfs: a leaked placeholder does not wedge the disk name')
+ or diag("prepare failed: $@");
+ if (defined($reused)) {
+ $C->copy_image_start($scfg, 'bt', $a, $scfg, 'bt', $reused, undef);
+ $C->copy_image_status($scfg, 'bt', $reused, undef);
+ my $reused_path = $C->filesystem_path($scfg, $reused);
+ ok(identical($reused_path, $apath), 'btrfs: the retried copy is correct');
+ } else {
+ ok(0, 'btrfs: the retried copy is correct');
+ }
+}
+
+# ---------------------------------------------------------------- lvmthin
+
+my $lvm_ok = eval {
+ run_command(['truncate', '-s', '3G', $LVM_IMG]);
+ push @cleanup, sub { unlink $LVM_IMG };
+
+ my $loop = sh("losetup --find --show $LVM_IMG");
+ chomp $loop;
+ die "no loop device\n" if !$loop;
+ # Detach as its own entry, registered IMMEDIATELY. Folding it into the entry pushed
+ # after vgcreate would leak the loop device whenever pvcreate or vgcreate fails --
+ # the eval catches that, the test SKIPs "cleanly", and the device stays attached.
+ push @cleanup, sub { sh("losetup -d $loop") };
+
+ run_command(['pvcreate', '-qq', '-f', $loop]);
+ run_command(['vgcreate', '-qq', $VG, $loop]);
+ push @cleanup, sub { sh("vgremove -qq -f $VG"); sh("pvremove -qq -f $loop") };
+
+ run_command(['lvcreate', '-qq', '--type', 'thin-pool', '-L', '2G', '-n', 'tp', $VG]);
+ 1;
+};
+if (!$lvm_ok) {
+ diag("lvm setup failed: $@");
+ SKIP: { skip 'lvm setup failed', 10 }
+} else {
+ my $C = 'PVE::Storage::LvmThinPlugin';
+ my $scfg = { vgname => $VG, thinpool => 'tp', type => 'lvmthin', content => { images => 1 } };
+ my $act = sub { sh("lvchange -ay -K $VG/$_[0]") };
+
+ my $src = $C->alloc_image('lt', $scfg, 100, 'raw', undef, 64 * 1024);
+ $act->($src);
+ sh("dd if=/dev/urandom of=/dev/$VG/$src bs=1M count=8 conv=fsync status=none");
+
+ $C->volume_snapshot($scfg, 'lt', $src, 'snap1');
+ my $snapdev = "/dev/$VG/snap_${src}_snap1";
+ $act->("snap_${src}_snap1");
+ sh("dd if=/dev/urandom of=/dev/$VG/$src bs=1M count=8 conv=fsync status=none");
+ ok(!identical("/dev/$VG/$src", $snapdev), 'lvmthin: source diverged from its snapshot');
+
+ my $a = $C->copy_image_prepare($scfg, 'lt', $src, $scfg, 'lt', 201, undef, {});
+ my $b = $C->copy_image_prepare($scfg, 'lt', $src, $scfg, 'lt', 201, undef, {});
+ isnt($a, $b, 'lvmthin: two prepares for one target VM reserve different names');
+
+ $C->copy_image_start($scfg, 'lt', $src, $scfg, 'lt', $a, undef);
+ my $st = $C->copy_image_status($scfg, 'lt', $a, undef);
+ is($st->{state}, 'complete', 'lvmthin: status complete on the first poll');
+ $act->($a);
+ ok(identical("/dev/$VG/$a", "/dev/$VG/$src"), 'lvmthin: copy matches the source');
+
+ my $snapcopy = $C->copy_image_prepare($scfg, 'lt', $src, $scfg, 'lt', 202, 'snap1', {});
+ $C->copy_image_start($scfg, 'lt', $src, $scfg, 'lt', $snapcopy, 'snap1');
+ $C->copy_image_status($scfg, 'lt', $snapcopy, undef);
+ $act->($snapcopy);
+ ok(
+ identical("/dev/$VG/$snapcopy", $snapdev),
+ 'lvmthin: a copy from a snapshot holds the SNAPSHOT content, not live data',
+ );
+
+ # Independence: hash the copy, destroy the source outright, hash again.
+ my $before = md5_of("/dev/$VG/$a");
+ $C->volume_snapshot_delete($scfg, 'lt', $src, 'snap1');
+ $C->free_image('lt', $scfg, $src, 0);
+ ok(!-e "/dev/$VG/$src", 'lvmthin: source really is gone');
+ is($before, md5_of("/dev/$VG/$a"), 'lvmthin: copy is unaffected by deleting the source');
+
+ unlike(
+ sh("lvs --noheadings -o lv_name $VG"), qr/copytmp-/,
+ 'lvmthin: no parked placeholder left behind',
+ );
+
+ # Failure path: a placeholder left by an earlier copy that died between start() and
+ # status() must not wedge the name. Without the reaping in prepare(), 'lvrename'
+ # cannot park onto the existing name and EVERY later copy to it fails -- and the
+ # leftover is invisible to list_images(), so nobody would know why.
+ my $stale = $C->copy_image_prepare($scfg, 'lt', $a, $scfg, 'lt', 203, undef, {});
+ sh("lvrename $VG $stale copytmp-$stale");
+ my $reused = eval { $C->copy_image_prepare($scfg, 'lt', $a, $scfg, 'lt', 203, undef, {}) };
+ ok(defined($reused), 'lvmthin: a leaked placeholder does not wedge the disk name')
+ or diag("prepare failed: $@");
+ if (defined($reused)) {
+ $C->copy_image_start($scfg, 'lt', $a, $scfg, 'lt', $reused, undef);
+ $C->copy_image_status($scfg, 'lt', $reused, undef);
+ $act->($reused);
+ ok(identical("/dev/$VG/$reused", "/dev/$VG/$a"), 'lvmthin: the retried copy is correct');
+ } else {
+ ok(0, 'lvmthin: the retried copy is correct');
+ }
+}
+
+cleanup_all();
diff --git a/src/test/run_plugin_tests.pl b/src/test/run_plugin_tests.pl
index 8f2472e..84995fe 100755
--- a/src/test/run_plugin_tests.pl
+++ b/src/test/run_plugin_tests.pl
@@ -19,6 +19,7 @@ my $res = $harness->runtests(
"prune_backups_test.pm",
"copy_offload_test.pm",
"copy_offload_naming_test.pm",
+ "copy_offload_feature_test.pm",
);
exit -1 if !$res || $res->{failed} || $res->{parse_errors};
--
2.54.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [RFC PATCH qemu-server 5/5] use storage copy-offload for full clone
2026-07-20 14:29 [RFC PATCH storage, qemu-server 0/5] offload full clone to the storage backend Ciro Iriarte
` (3 preceding siblings ...)
2026-07-20 14:30 ` [RFC PATCH storage 4/5] btrfs, lvmthin: implement copy-offload (atomic class) Ciro Iriarte
@ 2026-07-20 14:30 ` Ciro Iriarte
4 siblings, 0 replies; 6+ messages in thread
From: Ciro Iriarte @ 2026-07-20 14:30 UTC (permalink / raw)
To: pve-devel
Consumer side of the pve-storage copy-offload hook.
clone_disk()'s full-clone path does two things: vdisk_alloc() the target, then fill it
(drive-mirror for a running guest, qemu-img convert otherwise). An offloaded copy
replaces BOTH, because the backend produces the target itself. So the branch sits
before the allocation rather than beside the mirror/convert calls, and jumps to
no_data_clone as the cloudinit path already does.
The running-guest case is the interesting one. Cross-disk consistency here is a
VM-level property, not a per-volume one: clone_vm() calls clone_disk() once per drive
with completion => 'skip' until the last, and BlockJob::monitor() then waits for ALL
mirror jobs to be ready, fs-freezes ONCE, and cuts every disk over at that single
instant. An offload started per-disk inside clone_disk() would have no such
rendezvous, so each disk would be captured at a different instant and a running
multi-disk guest could be cloned torn -- a database with data and journal on separate
disks being the obvious casualty.
Rather than restrict offload to stopped guests, this reuses that rendezvous. For a
running source clone_disk() only ALLOCATES the target and defers the copy's start --
the operation that fixes its point in time -- into $deferred_copies. monitor() grows
an optional on_frozen callback, run while the guest is frozen at the same instant the
mirrors are cut over, and the deferred starts happen there. Offloaded and mirrored
disks of one VM therefore share a point in time, and mixed VMs work without a second
freeze. The freeze stays short: a start is one backend call, and the data movement is
waited for afterwards with the guest running again.
Two paths have no mirror to rendezvous with. If every disk was offloaded there are no
jobs at all, so clone_vm() freezes itself, mirroring monitor()'s freeze including the
suspend/resume fallback when no guest agent is available. And if the LAST disk is a
deferred one, nothing after it would call monitor(), so that branch runs the
rendezvous itself -- otherwise earlier mirrors, left at completion => 'skip', would
never be completed.
Everything done while the guest is stopped is now inside one eval whose error is held
until after the thaw, including the block-job cancel: leaving a guest frozen or
suspended is worse than failing the clone.
Guards: run_deferred_copies() is idempotent, clone_vm()'s fallback keys on unstarted
copies rather than on %$jobs so it cannot double-start if job bookkeeping changes, and
wait_deferred_copies() dies on an unstarted copy rather than skipping it -- its target
is allocated but its content undefined, so returning it would hand out a corrupt disk.
Each deferred copy carries its $snapname and its source volid. The start is what fixes
the point in time, so without the snapname a backend would copy the live disk while
reporting success; the source lets a plugin clean up state it left there once the copy
is independent.
'bulk' still falls through to the host-side path for a running guest: it smears over a
changing source and needs the bitmap-seed + mirror convergence, which is not
implemented here.
Excluded drives: cloudinit is regenerated rather than copied, and efidisk0/tpmstate0
are allocated at a fixed size and copied with a size-limited qemu-img dd, which an
opaque backend copy cannot reproduce. Checked on both source and destination drive
names. Skipping clone_disk_check_io_uring() here is intentional: it returns early
unless $use_drive_mirror, which an offloaded copy never sets.
Validated on a real Ceph cluster by full-cloning a RUNNING two-disk VM where both
disks offload: the all-offloaded fallback established its own freeze, the guest was
suspended and resumed, and both clones came out independent and byte-identical.
Generated-By: Claude (https://claude.ai)
Signed-off-by: Ciro Iriarte <ciro.iriarte+software@gmail.com>
Co-Authored-By: Claude <noreply@anthropic.com>
---
src/PVE/API2/Qemu.pm | 17 +++
src/PVE/QemuServer.pm | 201 ++++++++++++++++++++++++++++++++-
src/PVE/QemuServer/BlockJob.pm | 32 +++++-
3 files changed, 245 insertions(+), 5 deletions(-)
diff --git a/src/PVE/API2/Qemu.pm b/src/PVE/API2/Qemu.pm
index 28cbb9b0..1d7d427b 100644
--- a/src/PVE/API2/Qemu.pm
+++ b/src/PVE/API2/Qemu.pm
@@ -4599,6 +4599,9 @@ __PACKAGE__->register_method({
my $newvollist = [];
my $jobs = {};
+ # storage-offloaded copies whose start is deferred into the guest freeze, so
+ # that every disk of a running VM is captured at the same instant
+ my $deferred_copies = [];
eval {
local $SIG{INT} = local $SIG{TERM} = local $SIG{QUIT} = local $SIG{HUP} =
@@ -4650,6 +4653,7 @@ __PACKAGE__->register_method({
$completion,
$oldconf->{agent},
$clonelimit,
+ $deferred_copies,
);
$newconf->{$opt} = PVE::QemuServer::print_drive($newdrive);
@@ -4658,6 +4662,19 @@ __PACKAGE__->register_method({
$i++;
}
+ # Anything still unstarted means there was no mirror cutover to
+ # piggyback the freeze on -- i.e. every disk was offloaded -- so freeze
+ # here. Keyed on the copies themselves rather than on %$jobs, so it
+ # cannot double-start if the job bookkeeping changes.
+ if (grep { !$_->{started} } @$deferred_copies) {
+ PVE::QemuServer::freeze_and_run_deferred_copies(
+ $storecfg, $vmid, $oldconf->{agent}, $deferred_copies,
+ );
+ }
+
+ # the guest is running again; the data movement is waited for here
+ PVE::QemuServer::wait_deferred_copies($storecfg, $deferred_copies);
+
delete $newconf->{lock};
# do not write pending changes
diff --git a/src/PVE/QemuServer.pm b/src/PVE/QemuServer.pm
index 191ae549..3fc10d8c 100644
--- a/src/PVE/QemuServer.pm
+++ b/src/PVE/QemuServer.pm
@@ -7923,8 +7923,96 @@ my sub clone_disk_check_io_uring {
if $src_uses_io_uring && !storage_allows_io_uring_default($dst_scfg, $cache_direct);
}
+# Start every deferred storage-offloaded copy. MUST be called with the guest frozen (or
+# suspended): the start is what fixes each copy's point in time, so doing them all here
+# is what makes a multi-disk clone consistent.
+#
+# Handed to the storage layer as ONE batch rather than started in a loop, so a backend
+# that can capture several volumes at a single instant (an array consistency group) does
+# so. That makes the disks mutually crash-consistent in the backend itself rather than
+# only by virtue of this freeze, and collapses N backend calls into one -- which is what
+# keeps the guest's stall short on a VM with many disks.
+#
+# Kept deliberately cheap: the guest is stalled for the duration. The data movement runs
+# afterwards and is waited for by wait_deferred_copies(), with the guest running again.
+sub run_deferred_copies {
+ my ($storecfg, $deferred_copies) = @_;
+
+ return if !$deferred_copies;
+
+ # idempotent: never start a copy twice
+ my $pending = [grep { !$_->{started} } @$deferred_copies];
+ return if !scalar(@$pending);
+
+ # vdisk_copy_start_group() flags each copy as it starts it, so a failure partway
+ # through leaves an accurate record of what is actually running.
+ PVE::Storage::vdisk_copy_start_group($storecfg, $pending);
+
+ return;
+}
+
+# Wait for the deferred copies to become independent of their sources. Runs with the
+# guest already thawed: only the starts need the freeze, not the data movement.
+sub wait_deferred_copies {
+ my ($storecfg, $deferred_copies) = @_;
+
+ return if !$deferred_copies;
+
+ for my $copy (@$deferred_copies) {
+ # Never silently accept an unstarted copy: its target is allocated but its
+ # content is undefined, so returning it would hand out a corrupt disk.
+ die "internal error - offloaded copy of '$copy->{source}' was never started\n"
+ if !$copy->{started};
+
+ print("waiting for offloaded copy of '$copy->{source}' to become independent\n");
+ PVE::Storage::vdisk_copy_wait($storecfg, $copy->{target}, $copy->{source});
+ }
+}
+
+# Freeze the guest, start the deferred copies, thaw. Used when a clone has NO mirror
+# jobs to rendezvous with -- i.e. every disk was offloaded -- so there is no
+# BlockJob::monitor() freeze to piggyback on. Mirrors what monitor() does, including the
+# suspend/resume fallback when no guest agent is available.
+sub freeze_and_run_deferred_copies {
+ my ($storecfg, $vmid, $qga, $deferred_copies) = @_;
+
+ return if !$deferred_copies || !scalar(@$deferred_copies);
+
+ my $should_fsfreeze = PVE::QemuServer::Agent::guest_fs_freeze_applicable($qga, $vmid);
+ if ($should_fsfreeze) {
+ print "issuing guest agent 'guest-fsfreeze-freeze' command\n";
+ eval { PVE::QemuServer::Agent::guest_fs_freeze($vmid); };
+ warn $@ if $@;
+ } else {
+ print "suspend vm\n";
+ eval { PVE::QemuServer::RunState::vm_suspend($vmid, 1); };
+ warn $@ if $@;
+ }
+
+ eval { run_deferred_copies($storecfg, $deferred_copies) };
+ my $err = $@;
+
+ if ($should_fsfreeze) {
+ print "issuing guest agent 'guest-fsfreeze-thaw' command\n";
+ eval { PVE::QemuServer::Agent::guest_fs_thaw($vmid); };
+ warn $@ if $@;
+ } else {
+ print "resume vm\n";
+ eval { PVE::QemuServer::RunState::vm_resume($vmid, 1, 1); };
+ warn $@ if $@;
+ }
+
+ die $err if $err; # only after the guest is running again
+}
+
+# $deferred_copies is an optional arrayref collecting storage-offloaded copies whose
+# START must happen inside the caller's guest freeze; see the offload branch below and
+# run_deferred_copies(). Without it, a running source is never offloaded.
sub clone_disk {
- my ($storecfg, $source, $dest, $full, $newvollist, $jobs, $completion, $qga, $bwlimit) = @_;
+ my (
+ $storecfg, $source, $dest, $full, $newvollist, $jobs, $completion, $qga, $bwlimit,
+ $deferred_copies,
+ ) = @_;
my ($vmid, $running) = $source->@{qw(vmid running)};
my ($src_drivename, $drive, $snapname) = $source->@{qw(drivename drive snapname)};
@@ -7968,6 +8056,113 @@ sub clone_disk {
. " '$dst_format' instead\n";
}
+ # Storage-offloaded full copy (pve-storage copy_image).
+ #
+ # This replaces BOTH halves of the normal full-clone path: the backend creates
+ # the target volume itself, so there is no vdisk_alloc() here and no host-side
+ # copy afterwards. That is why the check sits before the allocation rather than
+ # as another branch next to the mirror/convert calls below.
+ #
+ # Skipped for the special drives: cloudinit is regenerated rather than copied,
+ # and efidisk0/tpmstate0 are allocated at a fixed size and copied with a
+ # size-limited qemu-img dd, which an opaque backend copy cannot reproduce.
+ # Checked on both drive names, since either side being special is disqualifying.
+ my $special_re = qr/^(?:efidisk0|tpmstate0)$/;
+ my $offload_special = drive_is_cloudinit($drive)
+ || (defined($dst_drivename) && $dst_drivename =~ $special_re)
+ || (defined($src_drivename) && $src_drivename =~ $special_re);
+
+ # Multi-disk consistency is a VM-level property here, not a per-volume one:
+ # clone_vm() calls clone_disk() once per drive with completion => 'skip' until
+ # the last one, and BlockJob::monitor() then waits for ALL mirror jobs to be
+ # ready, fs-freezes ONCE, and cuts every disk over at that single instant. An
+ # offload started per-disk right here would have no such rendezvous, so each disk
+ # would be captured at a different instant and a running multi-disk guest could
+ # be cloned torn -- a database whose data and journal live on separate disks
+ # being the obvious casualty.
+ #
+ # So for a RUNNING source the target is only ALLOCATED here, and the copy's start
+ # -- which is what fixes its point in time -- is deferred to $deferred_copies.
+ # The caller starts those inside the same freeze that cuts the mirrors over, so
+ # offloaded and mirrored disks of one VM share an instant. A stopped source needs
+ # no rendezvous and is copied inline.
+ my $offload_class = $offload_special
+ ? undef
+ : PVE::Storage::copy_offload_class(
+ $storecfg, $drive->{file}, $storeid, $snapname, $running,
+ );
+
+ # 'bulk' smears over a changing source, so it needs the bitmap-seed + mirror
+ # convergence for a running guest, which is not implemented: fall through to the
+ # host-side path rather than silently producing an inconsistent copy.
+ $offload_class = undef if $offload_class && $offload_class ne 'atomic' && $running;
+
+ # deferral requires a caller that will run the rendezvous
+ $offload_class = undef if $offload_class && $running && !$deferred_copies;
+
+ if ($offload_class && $running) {
+ print("offloading full copy to storage backend ($offload_class, deferred)\n");
+
+ $newvolid = PVE::Storage::vdisk_copy_prepare(
+ $storecfg, $drive->{file}, $storeid, $newvmid, $snapname,
+ { format => $dst_format },
+ );
+ push @$newvollist, $newvolid;
+
+ # recorded as plain volids: the starts are issued as one batch, so that a
+ # backend able to group them can capture every disk at a single instant
+ push @$deferred_copies, {
+ source => $drive->{file},
+ target => $newvolid,
+ snap => $snapname,
+ };
+
+ print("allocated target volume '$newvolid', copy starts at freeze\n");
+
+ # If this is the last disk and mirror jobs are pending, the rendezvous has
+ # to happen here: nothing after us will call monitor(), and those mirrors
+ # would never be completed. Starting the deferred copies from inside that
+ # same freeze is also what puts every disk at one instant.
+ if (($completion && $completion eq 'complete') && (scalar(keys %$jobs) > 0)) {
+ PVE::QemuServer::BlockJob::monitor(
+ vm_qmp_peer($vmid), $newvmid, $jobs, $completion, $qga, undef,
+ sub { run_deferred_copies($storecfg, $deferred_copies) },
+ );
+ }
+
+ goto no_data_clone;
+ }
+
+ if ($offload_class) {
+ # Complete any mirror jobs from previously cloned disks before finishing
+ # here, exactly as the cloudinit path below does: this disk may be the last
+ # one, and returning without completing would leave those transfers unfinished.
+ if (($completion && $completion eq 'complete') && (scalar(keys %$jobs) > 0)) {
+ PVE::QemuServer::BlockJob::monitor(
+ vm_qmp_peer($vmid), $newvmid, $jobs, $completion, $qga,
+ );
+ }
+
+ print("offloading full copy to storage backend ($offload_class)\n");
+
+ $newvolid = PVE::Storage::vdisk_copy(
+ $storecfg,
+ $drive->{file},
+ $storeid,
+ $newvmid,
+ $snapname,
+ { format => $dst_format },
+ sub { print("copy progress: $_[0]%\n") },
+ );
+ push @$newvollist, $newvolid;
+
+ print("created target volume '$newvolid' on the backend\n");
+
+ PVE::Storage::activate_volumes($storecfg, [$newvolid]);
+
+ goto no_data_clone;
+ }
+
my $name = undef;
my $size = undef;
if (drive_is_cloudinit($drive)) {
@@ -8024,6 +8219,10 @@ sub clone_disk {
my $mirror_opts = {};
$mirror_opts->{'guest-agent'} = $qga;
$mirror_opts->{bwlimit} = $bwlimit if defined($bwlimit);
+ # Start any deferred offloaded copies inside the freeze this mirror's
+ # completion takes, putting every disk of the VM at one point in time.
+ $mirror_opts->{'on-frozen'} = sub { run_deferred_copies($storecfg, $deferred_copies) }
+ if $deferred_copies && scalar(@$deferred_copies);
PVE::QemuServer::BlockJob::mirror(
$source_info,
$dest_info,
diff --git a/src/PVE/QemuServer/BlockJob.pm b/src/PVE/QemuServer/BlockJob.pm
index 921f046c..f48b95e3 100644
--- a/src/PVE/QemuServer/BlockJob.pm
+++ b/src/PVE/QemuServer/BlockJob.pm
@@ -83,8 +83,12 @@ sub qemu_blockjobs_cancel {
# 'cancel': wait until all jobs are ready, block-job-cancel them
# 'skip': wait until all jobs are ready, return with block jobs in ready state
# 'auto': wait until all jobs disappear, only use for jobs which complete automatically
+#
+# $on_frozen is an optional coderef, run while the guest is frozen (or suspended) at the
+# instant the mirror jobs are cut over. Clone uses it to start storage-offloaded copies
+# there, so that a VM's offloaded and mirrored disks share one point in time.
sub monitor {
- my ($qmp_peer, $vmiddst, $jobs, $completion, $qga, $op) = @_;
+ my ($qmp_peer, $vmiddst, $jobs, $completion, $qga, $op, $on_frozen) = @_;
die "drive mirror: different destination is only supported when peer is main QEMU instance\n"
if $vmiddst && $qmp_peer->{type} ne 'qmp';
@@ -179,8 +183,22 @@ sub monitor {
warn $@ if $@;
}
- # if we clone a disk for a new target vm, we don't switch the disk
- qemu_blockjobs_cancel($qmp_peer, $jobs);
+ # Everything done while the guest is stopped goes in here, and its
+ # error is held until after the thaw below. Leaving a guest frozen or
+ # suspended is far worse than failing the clone, so nothing between
+ # the freeze and the thaw may die -- including the job cancel, which
+ # can fail on its own.
+ my $frozen_err;
+ eval {
+ # Storage-offloaded copies start here: their point in time is
+ # fixed by the start, so doing it inside the freeze is what puts
+ # them at the same instant as the mirrored disks cut over below.
+ $on_frozen->() if $on_frozen;
+
+ # if we clone a disk for a new target vm, we don't switch the disk
+ qemu_blockjobs_cancel($qmp_peer, $jobs);
+ };
+ $frozen_err = $@;
if ($should_fsfreeze) {
print "issuing guest agent 'guest-fsfreeze-thaw' command\n";
@@ -192,6 +210,9 @@ sub monitor {
warn $@ if $@;
}
+ # re-raise only now that the guest is running again
+ die $frozen_err if $frozen_err;
+
last;
} else {
@@ -281,6 +302,7 @@ sub qemu_drive_mirror {
$qga,
$bwlimit,
$src_bitmap,
+ $on_frozen,
) = @_;
my $device_id = "drive-$drive_id";
@@ -316,7 +338,7 @@ sub qemu_drive_mirror {
die "mirroring error: $err\n";
}
- monitor(vm_qmp_peer($vmid), $vmiddst, $jobs, $completion, $qga);
+ monitor(vm_qmp_peer($vmid), $vmiddst, $jobs, $completion, $qga, undef, $on_frozen);
}
# Callers should version guard this (only available with a binary >= QEMU 8.2)
@@ -520,6 +542,7 @@ sub blockdev_mirror {
$completion,
$options->{'guest-agent'},
'mirror',
+ $options->{'on-frozen'},
);
}
@@ -543,6 +566,7 @@ sub mirror {
$options->{'guest-agent'},
$options->{bwlimit},
$source->{bitmap},
+ $options->{'on-frozen'},
);
}
}
--
2.54.0
^ permalink raw reply related [flat|nested] 6+ messages in thread