Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Ciro Iriarte <cyruspy@gmail.com>]

On July 13, 2026 4:03 pm, Fiona Ebner wrote:
> I guess it's easiest to just have two different features to avoid the
> need to change the interface for volume_has_feature().

Done -- `copy-offload-atomic` and `copy-offload-bulk`, volume_has_feature()
untouched.

Sorry for the delay; the design changed twice while I was building it.

Two things I got wrong, both of which changed the shape.

First, my argument that a clone needs no host-side mirror only holds for a
single disk. For several disks the freeze in monitor() is what puts them at
one instant, and an offload started per-disk in clone_disk() misses that
rendezvous entirely -- data and journal on separate disks would clone torn.
Fabian's point about unifying with the mirror machinery stands for the
running case.

So I use that rendezvous instead of avoiding it: for a running source
clone_disk() only allocates the target and defers the copy's start, and
monitor() takes an optional on_frozen callback that runs the deferred starts
inside the same freeze that cuts the mirrors over. Offloaded and mirrored
disks then share an instant, and mixed VMs need no second freeze. Only the
data movement happens after the thaw.

Second, a synchronous hook cannot work on shared storage: the plugin is
called under cluster_lock_storage(), and PVE::Cluster wraps that in
alarm(60), while a full copy runs for minutes or hours. I had missed this.
So the hook is now three phases:

  copy_image_prepare()  allocate the target       - under the storage lock
  copy_image_start()    pin the source PIT, begin - inside the guest freeze
  copy_image_status()   poll until independent    - outside both

The freeze is the second reason for the split: allocation is far too
expensive to hold a guest frozen across, so only the start runs there. On a
VSP the point in time is fixed by one group action, so the freeze holds a
single request regardless of disk count.

'complete' means independent of the source, not merely readable -- the
distinction that matters for rbd flatten, a zfs clone still tied to its
origin, or an array pair that has not dissolved.

The series includes an RBD implementation, so the hook can be exercised
without vendor hardware: snapshot, clone, flatten. On a Ceph Tentacle
cluster the copy finished in 3s, the target had no parent, md5 matched the
source, and it still read correctly after I deleted the source.

The flatten goes to `ceph rbd task add flatten` rather than a forked child --
cluster-side, survives the worker dying, cancellable. It also makes failure
visible: status can tell complete (no parent) from pending (a manager task
exists) from failed (still a clone, no task). Without that last state a dead
flatten looks exactly like a slow one and the caller polls forever.

One bug worth flagging since it was mine and it was silent: prepare received
$snap but start did not, and start is what fixes the point in time. RBD
therefore snapshotted the current image while advertising the feature for the
'snap' key, so a clone from a snapshot would have copied live data. The
parameter had gone to the phase that allocates rather than the one that
captures. Fixed, and each entry of a batch carries its own $snap now. A
related one: prepare reserved only a name, so a concurrent allocation could
take it and rollback would free the other operation's volume -- prepare
creates the target now.

Open questions:

- move-disk. Same mechanics, but it needs converge-to-current, which is the
  bulk + bitmap/mirror path I have not implemented. Tell me if you would
  rather they land together.

- copy_images_start() starts a batch between one pair of storages, with a
  default that just loops, so there is nothing to negotiate. It exists
  because an array can capture several volumes at one instant. Nothing
  in-tree overrides it: Ceph group snapshots do work, but an image can only
  be in one group, so grouping would mutate the source's group membership and
  collide with an admin's own. If that is too speculative without an in-tree
  user, it is the last commit and drops cleanly.

- prepare returns a volname. A token would let a plugin carry reservation
  state without inventing a name, at the cost of messier rollback.

Three commits: the hook (with ApiChangeLog and tests for the eligibility
rules), the RBD implementation, the qemu-server consumer. It compiles against
a real tree and the storage half is validated end-to-end on Ceph; the freeze
bracket has only been exercised offline so far. bulk with a running guest
still falls through to the host path.

Happy to post the patches now, or after you have looked at the shape.

Thanks,
Ciro