[RFC storage, qemu-server] offload full/live clone to storage backend

[RFC storage, qemu-server] offload full/live clone to storage backend

[Ciro Iriarte]

Hello,

I would like feedback on a storage-plugin contract that lets a backend
perform full-clone copies itself, instead of PVE moving every block
through the host.

Decision I am asking about
--------------------------
For a full clone where source and destination live on the same
offload-capable storage, should PVE delegate the copy to the storage
plugin (array-internal copy) rather than run qemu-img convert /
drive-mirror on the host? If yes, I propose the minimal contract below.

Background
----------
clone_image already offloads LINKED clones (CoW). Full clones do not
come through it: clone_disk() does vdisk_alloc + qemu-img convert
(offline) or drive-mirror (running), reading and writing the whole
volume over the host even when the array could copy it internally in
near-constant time with zero host I/O. volume_has_feature returns
copy => 1 today, but that only means "may be copied", not "the storage
can copy it itself" -- there is no hook for the storage to do the copy.

Proposed contract (pve-storage)
-------------------------------
- Add an optional plugin method, copy_image($scfg, $storeid,
  $src_volname, $vmid, $dst_format, $opts), that allocates and copies
  on the backend atomically and returns the new volname. Base class
  returns undef (unsupported), so existing plugins are unaffected --
  the driver is the sole source of the capability.
- Add a copy-offload feature to volume_has_feature, negotiated with the
  target storeid/format, so the driver also decides per copy whether
  this specific src->dst pair can be offloaded.
- Add a per-storage copy-offload option as a policy override (default
  on), exposed only in the options of drivers that advertise the
  capability, so it can disable offload but never enable it where
  unsupported. Setting it to 0 routes that storage back to the
  host-assisted path with no other change.
- Bump the storage APIVER/APIAGE.

Consumer (qemu-server)
----------------------
- In clone_disk(), in the offline full path, when src and dst are the
  same storage, formats match, the disk is not a special disk
  (cloudinit/efidisk0/tpmstate0), and the plugin advertises
  copy-offload, call the new copy_image hook instead of qemu-img
  convert. Otherwise run the existing path unchanged.
- A failed offload aborts the clone (existing rollback frees the
  target); it does not silently fall back to a host copy, so
  misbehaviour surfaces as a clean error rather than a bad volume.

Live clone (running VM)
-----------------------
A clone is not a migration: the source keeps running on its own volume,
so the target only needs a consistent point-in-time copy, not
drive-mirror convergence. For a running VM, quiesce the guest once
around the whole disk set (guest-fsfreeze-freeze) at the clone loop
level, flush QEMU's caches, run the array copy per disk, then thaw.
Online offload runs only when the freeze succeeds, giving an
FS-consistent copy; if there is no agent or the freeze fails, fall back
to drive-mirror. This needs no separate switch -- the same copy-offload
option governs it, and the freeze requirement keeps it from ever
producing a crash-consistent copy silently.

Out of scope
------------
- Cross-storage and format-converting clones (inherently
  host-mediated).
- VM move-disk needs no separate work: it already calls
  clone_disk(full) and its "same storage, same format" case is
  rejected, so it inherits this hook without ever mis-firing.
- Container volumes (storage_migrate path) -- a possible follow-up.

If the contract looks acceptable, I will follow up with [RFC PATCH]
series against pve-storage and then qemu-server. My CLA is on file.

Tracking/details:
https://github.com/ciroiriarte/pve-FCLUPlugin/issues/11

Thanks,
Ciro Iriarte

Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Fabian Grünbichler]

CCing Fiona and Max

On June 30, 2026 9:20 pm, Ciro Iriarte wrote:
> Hello,
> 
> I would like feedback on a storage-plugin contract that lets a backend
> perform full-clone copies itself, instead of PVE moving every block
> through the host.
> 
> Decision I am asking about
> --------------------------
> For a full clone where source and destination live on the same
> offload-capable storage, should PVE delegate the copy to the storage
> plugin (array-internal copy) rather than run qemu-img convert /
> drive-mirror on the host? If yes, I propose the minimal contract below.
> 
> Background
> ----------
> clone_image already offloads LINKED clones (CoW). Full clones do not
> come through it: clone_disk() does vdisk_alloc + qemu-img convert
> (offline) or drive-mirror (running), reading and writing the whole
> volume over the host even when the array could copy it internally in
> near-constant time with zero host I/O. volume_has_feature returns
> copy => 1 today, but that only means "may be copied", not "the storage
> can copy it itself" -- there is no hook for the storage to do the copy.
> 
> Proposed contract (pve-storage)
> -------------------------------
> - Add an optional plugin method, copy_image($scfg, $storeid,
>   $src_volname, $vmid, $dst_format, $opts), that allocates and copies
>   on the backend atomically and returns the new volname. Base class
>   returns undef (unsupported), so existing plugins are unaffected --
>   the driver is the sole source of the capability.

I don't think "atomically" is the right word here - if the copying
fails, the plugin would need to undo the allocation (if needed) as well.

$vmid should maybe be $target_vmid to make it explicit (it can be the
same as the $vmid encoded in the $src_volname). see further below as
well.

> - Add a copy-offload feature to volume_has_feature, negotiated with the
>   target storeid/format, so the driver also decides per copy whether
>   this specific src->dst pair can be offloaded.

the target storage and format can be passed in via $opts, but we'd also
need the target storage config (all the plugin methods get the
per-instance config, and not the global one), else the plugin cannot
really decide whether the source and target storage are a valid offload
combination?

> - Add a per-storage copy-offload option as a policy override (default
>   on), exposed only in the options of drivers that advertise the
>   capability, so it can disable offload but never enable it where
>   unsupported. Setting it to 0 routes that storage back to the
>   host-assisted path with no other change.

this can just be a regular option, and plugins enabling copy-offload
would then need to also add support/honor it

> - Bump the storage APIVER/APIAGE.
> 
> Consumer (qemu-server)
> ----------------------
> - In clone_disk(), in the offline full path, when src and dst are the
>   same storage, formats match, the disk is not a special disk
>   (cloudinit/efidisk0/tpmstate0), and the plugin advertises
>   copy-offload, call the new copy_image hook instead of qemu-img
>   convert. Otherwise run the existing path unchanged.
> - A failed offload aborts the clone (existing rollback frees the
>   target); it does not silently fall back to a host copy, so
>   misbehaviour surfaces as a clean error rather than a bad volume.
>
> Live clone (running VM)
> -----------------------
> A clone is not a migration: the source keeps running on its own volume,
> so the target only needs a consistent point-in-time copy, not
> drive-mirror convergence. For a running VM, quiesce the guest once
> around the whole disk set (guest-fsfreeze-freeze) at the clone loop
> level, flush QEMU's caches, run the array copy per disk, then thaw.

an offloaded copy might still take a while, so I don't think this is the
approach we want. instead, we probably want to invert it, and use the
same mechanism for regular "live" move disk and "live" cloning of whole
VMs:

1. create bitmap
2. copy disk using offload mechanism (inconsistent state)
3. mirror using bitmap (all writes between 1 and 3 will be done once
more on the qemu level, but that is just the delta)
4. auto-converge for move disk, repeat 1-3 for other disks and complete
when all are done when cloning

we are using the same mechanism atm when live-migrating VMs between
nodes when replication is enabled - it allows us to piggy-back on top of
the replicated volumes and only transfer the delta.

> Online offload runs only when the freeze succeeds, giving an
> FS-consistent copy; if there is no agent or the freeze fails, fall back
> to drive-mirror. This needs no separate switch -- the same copy-offload
> option governs it, and the freeze requirement keeps it from ever
> producing a crash-consistent copy silently.

I think the question of how to incorporate freeze for cloning running
VMs is a bit orthogonal? atm we don't freeze, so there is no cross-disk
consistency when cloning a running VM, other than that provided by the
mirror blockjob.

> 
> Out of scope
> ------------
> - Cross-storage and format-converting clones (inherently
>   host-mediated).

agreed. cross-storage could potentially be a follow-up, and we might
want to account for it in the copy_image signature/parameters?

> - VM move-disk needs no separate work: it already calls
>   clone_disk(full) and its "same storage, same format" case is
>   rejected, so it inherits this hook without ever mis-firing.

see above

> - Container volumes (storage_migrate path) -- a possible follow-up.

this would probably be nice to have. we've had requests to allow using
storage_migrate (or volume_export/volume_import) for copying container
volumes - it can be faster than the current rsync-based approach, and a
storage-offloaded variant would likely be even faster in many cases..

but it is a bit of its own series, as long as it's kept in mind when
developing this one it should be fine.

> 
> If the contract looks acceptable, I will follow up with [RFC PATCH]
> series against pve-storage and then qemu-server. My CLA is on file.

I'd suggest waiting a few more days in case additional feedback comes
in,  but the plan doesn't sound too bad to me :)

Thanks for wanting to tackle this!

> Tracking/details:
> https://github.com/ciroiriarte/pve-FCLUPlugin/issues/11
> 
> Thanks,
> Ciro Iriarte
> 
> 
> 
> 

Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Fiona Ebner]

Adding on top of Fabian's feedback:

Am 07.07.26 um 3:48 PM schrieb Fabian Grünbichler:
> On June 30, 2026 9:20 pm, Ciro Iriarte wrote:
>> -------------------------------
>> - Add an optional plugin method, copy_image($scfg, $storeid,
>>   $src_volname, $vmid, $dst_format, $opts), that allocates and copies
>>   on the backend atomically and returns the new volname. Base class
>>   returns undef (unsupported), so existing plugins are unaffected --
>>   the driver is the sole source of the capability.

I think the base class should die, because if it doesn't advertise the
feature, the method should never be reachable anyways.

>> Online offload runs only when the freeze succeeds, giving an
>> FS-consistent copy; if there is no agent or the freeze fails, fall back
>> to drive-mirror. This needs no separate switch -- the same copy-offload
>> option governs it, and the freeze requirement keeps it from ever
>> producing a crash-consistent copy silently.
> 
> I think the question of how to incorporate freeze for cloning running
> VMs is a bit orthogonal? atm we don't freeze, so there is no cross-disk
> consistency when cloning a running VM, other than that provided by the
> mirror blockjob.

No, we do freeze when cloning a running VM:
https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuServer/BlockJob.pm;h=921f046cd9ad917a1e365bd152cb250d656beec4;hb=HEAD#l168

I'm in favor of the bitmap-mirroring approach Fabian suggested, so in
the end, it would just use the existing BlockJob.pm mirror code for
completion.

>>
>> Out of scope
>> ------------
>> - Cross-storage and format-converting clones (inherently
>>   host-mediated).
> 
> agreed. cross-storage could potentially be a follow-up, and we might
> want to account for it in the copy_image signature/parameters?

I think we'd need an abstraction similar to
volume_{export,import}{,formats} so that plugins don't deal with
internal information from other plugins?