[RFC storage, qemu-server] offload full/live clone to storage backend

[RFC storage, qemu-server] offload full/live clone to storage backend

[Ciro Iriarte]

Hello,

I would like feedback on a storage-plugin contract that lets a backend
perform full-clone copies itself, instead of PVE moving every block
through the host.

Decision I am asking about
--------------------------
For a full clone where source and destination live on the same
offload-capable storage, should PVE delegate the copy to the storage
plugin (array-internal copy) rather than run qemu-img convert /
drive-mirror on the host? If yes, I propose the minimal contract below.

Background
----------
clone_image already offloads LINKED clones (CoW). Full clones do not
come through it: clone_disk() does vdisk_alloc + qemu-img convert
(offline) or drive-mirror (running), reading and writing the whole
volume over the host even when the array could copy it internally in
near-constant time with zero host I/O. volume_has_feature returns
copy => 1 today, but that only means "may be copied", not "the storage
can copy it itself" -- there is no hook for the storage to do the copy.

Proposed contract (pve-storage)
-------------------------------
- Add an optional plugin method, copy_image($scfg, $storeid,
  $src_volname, $vmid, $dst_format, $opts), that allocates and copies
  on the backend atomically and returns the new volname. Base class
  returns undef (unsupported), so existing plugins are unaffected --
  the driver is the sole source of the capability.
- Add a copy-offload feature to volume_has_feature, negotiated with the
  target storeid/format, so the driver also decides per copy whether
  this specific src->dst pair can be offloaded.
- Add a per-storage copy-offload option as a policy override (default
  on), exposed only in the options of drivers that advertise the
  capability, so it can disable offload but never enable it where
  unsupported. Setting it to 0 routes that storage back to the
  host-assisted path with no other change.
- Bump the storage APIVER/APIAGE.

Consumer (qemu-server)
----------------------
- In clone_disk(), in the offline full path, when src and dst are the
  same storage, formats match, the disk is not a special disk
  (cloudinit/efidisk0/tpmstate0), and the plugin advertises
  copy-offload, call the new copy_image hook instead of qemu-img
  convert. Otherwise run the existing path unchanged.
- A failed offload aborts the clone (existing rollback frees the
  target); it does not silently fall back to a host copy, so
  misbehaviour surfaces as a clean error rather than a bad volume.

Live clone (running VM)
-----------------------
A clone is not a migration: the source keeps running on its own volume,
so the target only needs a consistent point-in-time copy, not
drive-mirror convergence. For a running VM, quiesce the guest once
around the whole disk set (guest-fsfreeze-freeze) at the clone loop
level, flush QEMU's caches, run the array copy per disk, then thaw.
Online offload runs only when the freeze succeeds, giving an
FS-consistent copy; if there is no agent or the freeze fails, fall back
to drive-mirror. This needs no separate switch -- the same copy-offload
option governs it, and the freeze requirement keeps it from ever
producing a crash-consistent copy silently.

Out of scope
------------
- Cross-storage and format-converting clones (inherently
  host-mediated).
- VM move-disk needs no separate work: it already calls
  clone_disk(full) and its "same storage, same format" case is
  rejected, so it inherits this hook without ever mis-firing.
- Container volumes (storage_migrate path) -- a possible follow-up.

If the contract looks acceptable, I will follow up with [RFC PATCH]
series against pve-storage and then qemu-server. My CLA is on file.

Tracking/details:
https://github.com/ciroiriarte/pve-FCLUPlugin/issues/11

Thanks,
Ciro Iriarte

Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Fabian Grünbichler]

CCing Fiona and Max

On June 30, 2026 9:20 pm, Ciro Iriarte wrote:
> Hello,
> 
> I would like feedback on a storage-plugin contract that lets a backend
> perform full-clone copies itself, instead of PVE moving every block
> through the host.
> 
> Decision I am asking about
> --------------------------
> For a full clone where source and destination live on the same
> offload-capable storage, should PVE delegate the copy to the storage
> plugin (array-internal copy) rather than run qemu-img convert /
> drive-mirror on the host? If yes, I propose the minimal contract below.
> 
> Background
> ----------
> clone_image already offloads LINKED clones (CoW). Full clones do not
> come through it: clone_disk() does vdisk_alloc + qemu-img convert
> (offline) or drive-mirror (running), reading and writing the whole
> volume over the host even when the array could copy it internally in
> near-constant time with zero host I/O. volume_has_feature returns
> copy => 1 today, but that only means "may be copied", not "the storage
> can copy it itself" -- there is no hook for the storage to do the copy.
> 
> Proposed contract (pve-storage)
> -------------------------------
> - Add an optional plugin method, copy_image($scfg, $storeid,
>   $src_volname, $vmid, $dst_format, $opts), that allocates and copies
>   on the backend atomically and returns the new volname. Base class
>   returns undef (unsupported), so existing plugins are unaffected --
>   the driver is the sole source of the capability.

I don't think "atomically" is the right word here - if the copying
fails, the plugin would need to undo the allocation (if needed) as well.

$vmid should maybe be $target_vmid to make it explicit (it can be the
same as the $vmid encoded in the $src_volname). see further below as
well.

> - Add a copy-offload feature to volume_has_feature, negotiated with the
>   target storeid/format, so the driver also decides per copy whether
>   this specific src->dst pair can be offloaded.

the target storage and format can be passed in via $opts, but we'd also
need the target storage config (all the plugin methods get the
per-instance config, and not the global one), else the plugin cannot
really decide whether the source and target storage are a valid offload
combination?

> - Add a per-storage copy-offload option as a policy override (default
>   on), exposed only in the options of drivers that advertise the
>   capability, so it can disable offload but never enable it where
>   unsupported. Setting it to 0 routes that storage back to the
>   host-assisted path with no other change.

this can just be a regular option, and plugins enabling copy-offload
would then need to also add support/honor it

> - Bump the storage APIVER/APIAGE.
> 
> Consumer (qemu-server)
> ----------------------
> - In clone_disk(), in the offline full path, when src and dst are the
>   same storage, formats match, the disk is not a special disk
>   (cloudinit/efidisk0/tpmstate0), and the plugin advertises
>   copy-offload, call the new copy_image hook instead of qemu-img
>   convert. Otherwise run the existing path unchanged.
> - A failed offload aborts the clone (existing rollback frees the
>   target); it does not silently fall back to a host copy, so
>   misbehaviour surfaces as a clean error rather than a bad volume.
>
> Live clone (running VM)
> -----------------------
> A clone is not a migration: the source keeps running on its own volume,
> so the target only needs a consistent point-in-time copy, not
> drive-mirror convergence. For a running VM, quiesce the guest once
> around the whole disk set (guest-fsfreeze-freeze) at the clone loop
> level, flush QEMU's caches, run the array copy per disk, then thaw.

an offloaded copy might still take a while, so I don't think this is the
approach we want. instead, we probably want to invert it, and use the
same mechanism for regular "live" move disk and "live" cloning of whole
VMs:

1. create bitmap
2. copy disk using offload mechanism (inconsistent state)
3. mirror using bitmap (all writes between 1 and 3 will be done once
more on the qemu level, but that is just the delta)
4. auto-converge for move disk, repeat 1-3 for other disks and complete
when all are done when cloning

we are using the same mechanism atm when live-migrating VMs between
nodes when replication is enabled - it allows us to piggy-back on top of
the replicated volumes and only transfer the delta.

> Online offload runs only when the freeze succeeds, giving an
> FS-consistent copy; if there is no agent or the freeze fails, fall back
> to drive-mirror. This needs no separate switch -- the same copy-offload
> option governs it, and the freeze requirement keeps it from ever
> producing a crash-consistent copy silently.

I think the question of how to incorporate freeze for cloning running
VMs is a bit orthogonal? atm we don't freeze, so there is no cross-disk
consistency when cloning a running VM, other than that provided by the
mirror blockjob.

> 
> Out of scope
> ------------
> - Cross-storage and format-converting clones (inherently
>   host-mediated).

agreed. cross-storage could potentially be a follow-up, and we might
want to account for it in the copy_image signature/parameters?

> - VM move-disk needs no separate work: it already calls
>   clone_disk(full) and its "same storage, same format" case is
>   rejected, so it inherits this hook without ever mis-firing.

see above

> - Container volumes (storage_migrate path) -- a possible follow-up.

this would probably be nice to have. we've had requests to allow using
storage_migrate (or volume_export/volume_import) for copying container
volumes - it can be faster than the current rsync-based approach, and a
storage-offloaded variant would likely be even faster in many cases..

but it is a bit of its own series, as long as it's kept in mind when
developing this one it should be fine.

> 
> If the contract looks acceptable, I will follow up with [RFC PATCH]
> series against pve-storage and then qemu-server. My CLA is on file.

I'd suggest waiting a few more days in case additional feedback comes
in,  but the plan doesn't sound too bad to me :)

Thanks for wanting to tackle this!

> Tracking/details:
> https://github.com/ciroiriarte/pve-FCLUPlugin/issues/11
> 
> Thanks,
> Ciro Iriarte
> 
> 
> 
> 

Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Fiona Ebner]

Adding on top of Fabian's feedback:

Am 07.07.26 um 3:48 PM schrieb Fabian Grünbichler:
> On June 30, 2026 9:20 pm, Ciro Iriarte wrote:
>> -------------------------------
>> - Add an optional plugin method, copy_image($scfg, $storeid,
>>   $src_volname, $vmid, $dst_format, $opts), that allocates and copies
>>   on the backend atomically and returns the new volname. Base class
>>   returns undef (unsupported), so existing plugins are unaffected --
>>   the driver is the sole source of the capability.

I think the base class should die, because if it doesn't advertise the
feature, the method should never be reachable anyways.

>> Online offload runs only when the freeze succeeds, giving an
>> FS-consistent copy; if there is no agent or the freeze fails, fall back
>> to drive-mirror. This needs no separate switch -- the same copy-offload
>> option governs it, and the freeze requirement keeps it from ever
>> producing a crash-consistent copy silently.
> 
> I think the question of how to incorporate freeze for cloning running
> VMs is a bit orthogonal? atm we don't freeze, so there is no cross-disk
> consistency when cloning a running VM, other than that provided by the
> mirror blockjob.

No, we do freeze when cloning a running VM:
https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuServer/BlockJob.pm;h=921f046cd9ad917a1e365bd152cb250d656beec4;hb=HEAD#l168

I'm in favor of the bitmap-mirroring approach Fabian suggested, so in
the end, it would just use the existing BlockJob.pm mirror code for
completion.

>>
>> Out of scope
>> ------------
>> - Cross-storage and format-converting clones (inherently
>>   host-mediated).
> 
> agreed. cross-storage could potentially be a follow-up, and we might
> want to account for it in the copy_image signature/parameters?

I think we'd need an abstraction similar to
volume_{export,import}{,formats} so that plugins don't deal with
internal information from other plugins?

Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Fabian Grünbichler]

On July 8, 2026 10:17 am, Fiona Ebner wrote:
> Adding on top of Fabian's feedback:
> 
> Am 07.07.26 um 3:48 PM schrieb Fabian Grünbichler:
>> On June 30, 2026 9:20 pm, Ciro Iriarte wrote:
>>> -------------------------------
>>> - Add an optional plugin method, copy_image($scfg, $storeid,
>>>   $src_volname, $vmid, $dst_format, $opts), that allocates and copies
>>>   on the backend atomically and returns the new volname. Base class
>>>   returns undef (unsupported), so existing plugins are unaffected --
>>>   the driver is the sole source of the capability.
> 
> I think the base class should die, because if it doesn't advertise the
> feature, the method should never be reachable anyways.

I agree, it's also more consistent with existing code

>>> Online offload runs only when the freeze succeeds, giving an
>>> FS-consistent copy; if there is no agent or the freeze fails, fall back
>>> to drive-mirror. This needs no separate switch -- the same copy-offload
>>> option governs it, and the freeze requirement keeps it from ever
>>> producing a crash-consistent copy silently.
>> 
>> I think the question of how to incorporate freeze for cloning running
>> VMs is a bit orthogonal? atm we don't freeze, so there is no cross-disk
>> consistency when cloning a running VM, other than that provided by the
>> mirror blockjob.
> 
> No, we do freeze when cloning a running VM:
> https://git.proxmox.com/?p=qemu-server.git;a=blob;f=src/PVE/QemuServer/BlockJob.pm;h=921f046cd9ad917a1e365bd152cb250d656beec4;hb=HEAD#l168

ah, thanks, missed that.

> I'm in favor of the bitmap-mirroring approach Fabian suggested, so in
> the end, it would just use the existing BlockJob.pm mirror code for
> completion.

but that sitll applies :)

>>>
>>> Out of scope
>>> ------------
>>> - Cross-storage and format-converting clones (inherently
>>>   host-mediated).
>> 
>> agreed. cross-storage could potentially be a follow-up, and we might
>> want to account for it in the copy_image signature/parameters?
> 
> I think we'd need an abstraction similar to
> volume_{export,import}{,formats} so that plugins don't deal with
> internal information from other plugins?

I guess a valid baseline assumption is that offloaded copies can only
ever occur between two storage instances of the same plugin type? how
the plugin then decides that the two instances can be used for copying
or not is up to the plugin (e.g., zfs can send/recv locally, a storage
appliance might have a copy feature between namespaces, but not
between storage pools or appliances, ..).

volume_export/import can be split like that because we lower to some
transport format, but that is not possible for offloading, which is
opaque from PVE's point of view..