[RFC storage, qemu-server] offload full/live clone to storage backend

[RFC storage, qemu-server] offload full/live clone to storage backend

[Ciro Iriarte]

Hello,

I would like feedback on a storage-plugin contract that lets a backend
perform full-clone copies itself, instead of PVE moving every block
through the host.

Decision I am asking about
--------------------------
For a full clone where source and destination live on the same
offload-capable storage, should PVE delegate the copy to the storage
plugin (array-internal copy) rather than run qemu-img convert /
drive-mirror on the host? If yes, I propose the minimal contract below.

Background
----------
clone_image already offloads LINKED clones (CoW). Full clones do not
come through it: clone_disk() does vdisk_alloc + qemu-img convert
(offline) or drive-mirror (running), reading and writing the whole
volume over the host even when the array could copy it internally in
near-constant time with zero host I/O. volume_has_feature returns
copy => 1 today, but that only means "may be copied", not "the storage
can copy it itself" -- there is no hook for the storage to do the copy.

Proposed contract (pve-storage)
-------------------------------
- Add an optional plugin method, copy_image($scfg, $storeid,
  $src_volname, $vmid, $dst_format, $opts), that allocates and copies
  on the backend atomically and returns the new volname. Base class
  returns undef (unsupported), so existing plugins are unaffected --
  the driver is the sole source of the capability.
- Add a copy-offload feature to volume_has_feature, negotiated with the
  target storeid/format, so the driver also decides per copy whether
  this specific src->dst pair can be offloaded.
- Add a per-storage copy-offload option as a policy override (default
  on), exposed only in the options of drivers that advertise the
  capability, so it can disable offload but never enable it where
  unsupported. Setting it to 0 routes that storage back to the
  host-assisted path with no other change.
- Bump the storage APIVER/APIAGE.

Consumer (qemu-server)
----------------------
- In clone_disk(), in the offline full path, when src and dst are the
  same storage, formats match, the disk is not a special disk
  (cloudinit/efidisk0/tpmstate0), and the plugin advertises
  copy-offload, call the new copy_image hook instead of qemu-img
  convert. Otherwise run the existing path unchanged.
- A failed offload aborts the clone (existing rollback frees the
  target); it does not silently fall back to a host copy, so
  misbehaviour surfaces as a clean error rather than a bad volume.

Live clone (running VM)
-----------------------
A clone is not a migration: the source keeps running on its own volume,
so the target only needs a consistent point-in-time copy, not
drive-mirror convergence. For a running VM, quiesce the guest once
around the whole disk set (guest-fsfreeze-freeze) at the clone loop
level, flush QEMU's caches, run the array copy per disk, then thaw.
Online offload runs only when the freeze succeeds, giving an
FS-consistent copy; if there is no agent or the freeze fails, fall back
to drive-mirror. This needs no separate switch -- the same copy-offload
option governs it, and the freeze requirement keeps it from ever
producing a crash-consistent copy silently.

Out of scope
------------
- Cross-storage and format-converting clones (inherently
  host-mediated).
- VM move-disk needs no separate work: it already calls
  clone_disk(full) and its "same storage, same format" case is
  rejected, so it inherits this hook without ever mis-firing.
- Container volumes (storage_migrate path) -- a possible follow-up.

If the contract looks acceptable, I will follow up with [RFC PATCH]
series against pve-storage and then qemu-server. My CLA is on file.

Tracking/details:
https://github.com/ciroiriarte/pve-FCLUPlugin/issues/11

Thanks,
Ciro Iriarte

Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Fabian Grünbichler]

CCing Fiona and Max

On June 30, 2026 9:20 pm, Ciro Iriarte wrote:
> Hello,
> 
> I would like feedback on a storage-plugin contract that lets a backend
> perform full-clone copies itself, instead of PVE moving every block
> through the host.
> 
> Decision I am asking about
> --------------------------
> For a full clone where source and destination live on the same
> offload-capable storage, should PVE delegate the copy to the storage
> plugin (array-internal copy) rather than run qemu-img convert /
> drive-mirror on the host? If yes, I propose the minimal contract below.
> 
> Background
> ----------
> clone_image already offloads LINKED clones (CoW). Full clones do not
> come through it: clone_disk() does vdisk_alloc + qemu-img convert
> (offline) or drive-mirror (running), reading and writing the whole
> volume over the host even when the array could copy it internally in
> near-constant time with zero host I/O. volume_has_feature returns
> copy => 1 today, but that only means "may be copied", not "the storage
> can copy it itself" -- there is no hook for the storage to do the copy.
> 
> Proposed contract (pve-storage)
> -------------------------------
> - Add an optional plugin method, copy_image($scfg, $storeid,
>   $src_volname, $vmid, $dst_format, $opts), that allocates and copies
>   on the backend atomically and returns the new volname. Base class
>   returns undef (unsupported), so existing plugins are unaffected --
>   the driver is the sole source of the capability.

I don't think "atomically" is the right word here - if the copying
fails, the plugin would need to undo the allocation (if needed) as well.

$vmid should maybe be $target_vmid to make it explicit (it can be the
same as the $vmid encoded in the $src_volname). see further below as
well.

> - Add a copy-offload feature to volume_has_feature, negotiated with the
>   target storeid/format, so the driver also decides per copy whether
>   this specific src->dst pair can be offloaded.

the target storage and format can be passed in via $opts, but we'd also
need the target storage config (all the plugin methods get the
per-instance config, and not the global one), else the plugin cannot
really decide whether the source and target storage are a valid offload
combination?

> - Add a per-storage copy-offload option as a policy override (default
>   on), exposed only in the options of drivers that advertise the
>   capability, so it can disable offload but never enable it where
>   unsupported. Setting it to 0 routes that storage back to the
>   host-assisted path with no other change.

this can just be a regular option, and plugins enabling copy-offload
would then need to also add support/honor it

> - Bump the storage APIVER/APIAGE.
> 
> Consumer (qemu-server)
> ----------------------
> - In clone_disk(), in the offline full path, when src and dst are the
>   same storage, formats match, the disk is not a special disk
>   (cloudinit/efidisk0/tpmstate0), and the plugin advertises
>   copy-offload, call the new copy_image hook instead of qemu-img
>   convert. Otherwise run the existing path unchanged.
> - A failed offload aborts the clone (existing rollback frees the
>   target); it does not silently fall back to a host copy, so
>   misbehaviour surfaces as a clean error rather than a bad volume.
>
> Live clone (running VM)
> -----------------------
> A clone is not a migration: the source keeps running on its own volume,
> so the target only needs a consistent point-in-time copy, not
> drive-mirror convergence. For a running VM, quiesce the guest once
> around the whole disk set (guest-fsfreeze-freeze) at the clone loop
> level, flush QEMU's caches, run the array copy per disk, then thaw.

an offloaded copy might still take a while, so I don't think this is the
approach we want. instead, we probably want to invert it, and use the
same mechanism for regular "live" move disk and "live" cloning of whole
VMs:

1. create bitmap
2. copy disk using offload mechanism (inconsistent state)
3. mirror using bitmap (all writes between 1 and 3 will be done once
more on the qemu level, but that is just the delta)
4. auto-converge for move disk, repeat 1-3 for other disks and complete
when all are done when cloning

we are using the same mechanism atm when live-migrating VMs between
nodes when replication is enabled - it allows us to piggy-back on top of
the replicated volumes and only transfer the delta.

> Online offload runs only when the freeze succeeds, giving an
> FS-consistent copy; if there is no agent or the freeze fails, fall back
> to drive-mirror. This needs no separate switch -- the same copy-offload
> option governs it, and the freeze requirement keeps it from ever
> producing a crash-consistent copy silently.

I think the question of how to incorporate freeze for cloning running
VMs is a bit orthogonal? atm we don't freeze, so there is no cross-disk
consistency when cloning a running VM, other than that provided by the
mirror blockjob.

> 
> Out of scope
> ------------
> - Cross-storage and format-converting clones (inherently
>   host-mediated).

agreed. cross-storage could potentially be a follow-up, and we might
want to account for it in the copy_image signature/parameters?

> - VM move-disk needs no separate work: it already calls
>   clone_disk(full) and its "same storage, same format" case is
>   rejected, so it inherits this hook without ever mis-firing.

see above

> - Container volumes (storage_migrate path) -- a possible follow-up.

this would probably be nice to have. we've had requests to allow using
storage_migrate (or volume_export/volume_import) for copying container
volumes - it can be faster than the current rsync-based approach, and a
storage-offloaded variant would likely be even faster in many cases..

but it is a bit of its own series, as long as it's kept in mind when
developing this one it should be fine.

> 
> If the contract looks acceptable, I will follow up with [RFC PATCH]
> series against pve-storage and then qemu-server. My CLA is on file.

I'd suggest waiting a few more days in case additional feedback comes
in,  but the plan doesn't sound too bad to me :)

Thanks for wanting to tackle this!

> Tracking/details:
> https://github.com/ciroiriarte/pve-FCLUPlugin/issues/11
> 
> Thanks,
> Ciro Iriarte
> 
> 
> 
> 

Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Fiona Ebner]

Adding on top of Fabian's feedback:

Am 07.07.26 um 3:48 PM schrieb Fabian Grünbichler:
> On June 30, 2026 9:20 pm, Ciro Iriarte wrote:
>> -------------------------------
>> - Add an optional plugin method, copy_image($scfg, $storeid,
>>   $src_volname, $vmid, $dst_format, $opts), that allocates and copies
>>   on the backend atomically and returns the new volname. Base class
>>   returns undef (unsupported), so existing plugins are unaffected --
>>   the driver is the sole source of the capability.

I think the base class should die, because if it doesn't advertise the
feature, the method should never be reachable anyways.

>> Online offload runs only when the freeze succeeds, giving an
>> FS-consistent copy; if there is no agent or the freeze fails, fall back
>> to drive-mirror. This needs no separate switch -- the same copy-offload
>> option governs it, and the freeze requirement keeps it from ever
>> producing a crash-consistent copy silently.
> 
> I think the question of how to incorporate freeze for cloning running
> VMs is a bit orthogonal? atm we don't freeze, so there is no cross-disk
> consistency when cloning a running VM, other than that provided by the
> mirror blockjob.

No, we do freeze when cloning a running VM:
https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuServer/BlockJob.pm;h=921f046cd9ad917a1e365bd152cb250d656beec4;hb=HEAD#l168

I'm in favor of the bitmap-mirroring approach Fabian suggested, so in
the end, it would just use the existing BlockJob.pm mirror code for
completion.

>>
>> Out of scope
>> ------------
>> - Cross-storage and format-converting clones (inherently
>>   host-mediated).
> 
> agreed. cross-storage could potentially be a follow-up, and we might
> want to account for it in the copy_image signature/parameters?

I think we'd need an abstraction similar to
volume_{export,import}{,formats} so that plugins don't deal with
internal information from other plugins?

Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Fabian Grünbichler]

On July 8, 2026 10:17 am, Fiona Ebner wrote:
> Adding on top of Fabian's feedback:
> 
> Am 07.07.26 um 3:48 PM schrieb Fabian Grünbichler:
>> On June 30, 2026 9:20 pm, Ciro Iriarte wrote:
>>> -------------------------------
>>> - Add an optional plugin method, copy_image($scfg, $storeid,
>>>   $src_volname, $vmid, $dst_format, $opts), that allocates and copies
>>>   on the backend atomically and returns the new volname. Base class
>>>   returns undef (unsupported), so existing plugins are unaffected --
>>>   the driver is the sole source of the capability.
> 
> I think the base class should die, because if it doesn't advertise the
> feature, the method should never be reachable anyways.

I agree, it's also more consistent with existing code

>>> Online offload runs only when the freeze succeeds, giving an
>>> FS-consistent copy; if there is no agent or the freeze fails, fall back
>>> to drive-mirror. This needs no separate switch -- the same copy-offload
>>> option governs it, and the freeze requirement keeps it from ever
>>> producing a crash-consistent copy silently.
>> 
>> I think the question of how to incorporate freeze for cloning running
>> VMs is a bit orthogonal? atm we don't freeze, so there is no cross-disk
>> consistency when cloning a running VM, other than that provided by the
>> mirror blockjob.
> 
> No, we do freeze when cloning a running VM:
> https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuServer/BlockJob.pm;h=921f046cd9ad917a1e365bd152cb250d656beec4;hb=HEAD#l168

ah, thanks, missed that.

> I'm in favor of the bitmap-mirroring approach Fabian suggested, so in
> the end, it would just use the existing BlockJob.pm mirror code for
> completion.

but that sitll applies :)

>>>
>>> Out of scope
>>> ------------
>>> - Cross-storage and format-converting clones (inherently
>>>   host-mediated).
>> 
>> agreed. cross-storage could potentially be a follow-up, and we might
>> want to account for it in the copy_image signature/parameters?
> 
> I think we'd need an abstraction similar to
> volume_{export,import}{,formats} so that plugins don't deal with
> internal information from other plugins?

I guess a valid baseline assumption is that offloaded copies can only
ever occur between two storage instances of the same plugin type? how
the plugin then decides that the two instances can be used for copying
or not is up to the plugin (e.g., zfs can send/recv locally, a storage
appliance might have a copy feature between namespaces, but not
between storage pools or appliances, ..).

volume_export/import can be split like that because we lower to some
transport format, but that is not possible for offloading, which is
opaque from PVE's point of view..

Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Fiona Ebner]

Am 08.07.26 um 10:43 AM schrieb Fabian Grünbichler:
> On July 8, 2026 10:17 am, Fiona Ebner wrote:
>>>>
>>>> Out of scope
>>>> ------------
>>>> - Cross-storage and format-converting clones (inherently
>>>>   host-mediated).
>>>
>>> agreed. cross-storage could potentially be a follow-up, and we might
>>> want to account for it in the copy_image signature/parameters?
>>
>> I think we'd need an abstraction similar to
>> volume_{export,import}{,formats} so that plugins don't deal with
>> internal information from other plugins?
> 
> I guess a valid baseline assumption is that offloaded copies can only
> ever occur between two storage instances of the same plugin type? how
> the plugin then decides that the two instances can be used for copying
> or not is up to the plugin (e.g., zfs can send/recv locally, a storage
> appliance might have a copy feature between namespaces, but not
> between storage pools or appliances, ..).
> 
> volume_export/import can be split like that because we lower to some
> transport format, but that is not possible for offloading, which is
> opaque from PVE's point of view..

Different storage types based on the same underlying technology (e.g.
ZFS) could still do it. That said, I don't mind not designing for this
but just for offloading between storages of the same type.

Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Ciro Iriarte]

On July 9, 2026 9:56 am, Fiona Ebner wrote:
> Different storage types based on the same underlying technology (e.g.
> ZFS) could still do it. That said, I don't mind not designing for this
> but just for offloading between storages of the same type.

Thanks both for the thorough feedback. Consolidated reply and a revised
proposal below.

Accepted as-is:
- Base-class copy_image dies instead of returning undef (only reachable
  when the plugin advertises the capability).
- Drop "atomically" -- the plugin owns cleanup and undoes its own
  allocation on failure; I'll document that as the contract.
- Rename $vmid -> $target_vmid, and pass the target storage *config* so
  the plugin can judge whether a source/target pair is a valid offload.
- copy-offload as a plain per-storage option that capable plugins honor.
- Baseline: offload only between two instances of the same plugin type,
  the plugin deciding whether those instances can actually copy.
  Cross-type stays out of scope; since offload is opaque to PVE there's
  no volume_export/import-style lowering to generalize it, so I'll leave
  room in the signature but not design for it.

The substantive point is separating two things "offload" blurs:
point-in-time consistency of the copy, and independence of the result.

Consistency: the bitmap + mirror model is the right completion path for
any *smeared* copy -- a non-atomic, out-of-band bulk copy written while
the source keeps changing. NFS 4.2 server-side copy (copy_file_range) is
exactly that. Migration (move-disk, live migrate) needs it too, since
the target must converge to current state and cut over -- reusing the
replicated-migration mechanism fits well, agreed.

But snapshot-based backends never smear: rbd snap+clone, lvm-thin/btrfs
snapshots, zfs clone all copy from a static point-in-time source. There
a host-side drive-mirror only re-writes blocks that are already correct
-- not incorrect, just host I/O for work the array already did with
none. Those should skip the host mirror: freeze once (PVE already does,
per BlockJob.pm:168), snapshot/clone, thaw. That yields a valid
point-in-time clone -- same as the mirror-at-cutover PVE produces today,
minus the host data movement.

Independence differs per backend, and this is what determines whether a
snapshot is a valid *full* clone:
- lvm-thin, btrfs: reference-counted, independent immediately; source
  can be removed at once.
- rbd: depends on a protected parent snapshot until `rbd flatten`
  finishes; flatten runs on the Ceph cluster rather than as a host-side
  qemu-img copy, but the dependency is real until it completes.
- zfs: a clone depends on its origin snapshot; `zfs promote` only
  re-parents that dependency (no help when the template must stay the
  clone source), and the only true full-copy route is a bulk send/recv,
  which is not instant. So "instant + independent + host-free" isn't
  achievable for zfs full clones -- that one wants the bulk path.

So copy_image should advertise which case it is, and the core needs an
independence barrier: the hook returns only once the target is
independent, or the core records the pending dependency and refuses to
delete the source/template/parent snapshot until the backend signals
completion. Otherwise "snapshot and return" is a broken full clone for
rbd/zfs.

Concretely, two capability classes on top of the option:
- copy-offload => atomic: static point-in-time source, no host mirror
  (freeze -> snapshot/clone -> thaw [-> background flatten]).
- copy-offload => bulk: no instant independent snapshot -- needs a full
  data copy (NFS 4.2 SSC, non-atomic SAN copy, zfs send/recv); the
  running-VM case then needs the bitmap + mirror completion.

qemu-server then branches: atomic clones skip the mirror; bulk clones
and all move-disk/migration take the bitmap-seed + mirror path, reusing
the existing BlockJob.pm completion code.

Two QEMU details I'd like to confirm for the bulk path: the dirty bitmap
must be created before the array copy starts, so out-of-band writes are
tracked; and the target can't be attached writable during the copy --
blockdev-add + mirror only after the array reports the copy done, the
mirror then healing the tracked blocks.

Does that split sound acceptable? If so I'll send the pve-storage RFC
PATCH first (hook + option + capability + APIVER/APIAGE bump), then the
qemu-server consumer.

Thanks,
Ciro