Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Ciro Iriarte <cyruspy@gmail.com>]

On July 9, 2026 9:56 am, Fiona Ebner wrote:
> Different storage types based on the same underlying technology (e.g.
> ZFS) could still do it. That said, I don't mind not designing for this
> but just for offloading between storages of the same type.

Thanks both for the thorough feedback. Consolidated reply and a revised
proposal below.

Accepted as-is:
- Base-class copy_image dies instead of returning undef (only reachable
  when the plugin advertises the capability).
- Drop "atomically" -- the plugin owns cleanup and undoes its own
  allocation on failure; I'll document that as the contract.
- Rename $vmid -> $target_vmid, and pass the target storage *config* so
  the plugin can judge whether a source/target pair is a valid offload.
- copy-offload as a plain per-storage option that capable plugins honor.
- Baseline: offload only between two instances of the same plugin type,
  the plugin deciding whether those instances can actually copy.
  Cross-type stays out of scope; since offload is opaque to PVE there's
  no volume_export/import-style lowering to generalize it, so I'll leave
  room in the signature but not design for it.

The substantive point is separating two things "offload" blurs:
point-in-time consistency of the copy, and independence of the result.

Consistency: the bitmap + mirror model is the right completion path for
any *smeared* copy -- a non-atomic, out-of-band bulk copy written while
the source keeps changing. NFS 4.2 server-side copy (copy_file_range) is
exactly that. Migration (move-disk, live migrate) needs it too, since
the target must converge to current state and cut over -- reusing the
replicated-migration mechanism fits well, agreed.

But snapshot-based backends never smear: rbd snap+clone, lvm-thin/btrfs
snapshots, zfs clone all copy from a static point-in-time source. There
a host-side drive-mirror only re-writes blocks that are already correct
-- not incorrect, just host I/O for work the array already did with
none. Those should skip the host mirror: freeze once (PVE already does,
per BlockJob.pm:168), snapshot/clone, thaw. That yields a valid
point-in-time clone -- same as the mirror-at-cutover PVE produces today,
minus the host data movement.

Independence differs per backend, and this is what determines whether a
snapshot is a valid *full* clone:
- lvm-thin, btrfs: reference-counted, independent immediately; source
  can be removed at once.
- rbd: depends on a protected parent snapshot until `rbd flatten`
  finishes; flatten runs on the Ceph cluster rather than as a host-side
  qemu-img copy, but the dependency is real until it completes.
- zfs: a clone depends on its origin snapshot; `zfs promote` only
  re-parents that dependency (no help when the template must stay the
  clone source), and the only true full-copy route is a bulk send/recv,
  which is not instant. So "instant + independent + host-free" isn't
  achievable for zfs full clones -- that one wants the bulk path.

So copy_image should advertise which case it is, and the core needs an
independence barrier: the hook returns only once the target is
independent, or the core records the pending dependency and refuses to
delete the source/template/parent snapshot until the backend signals
completion. Otherwise "snapshot and return" is a broken full clone for
rbd/zfs.

Concretely, two capability classes on top of the option:
- copy-offload => atomic: static point-in-time source, no host mirror
  (freeze -> snapshot/clone -> thaw [-> background flatten]).
- copy-offload => bulk: no instant independent snapshot -- needs a full
  data copy (NFS 4.2 SSC, non-atomic SAN copy, zfs send/recv); the
  running-VM case then needs the bitmap + mirror completion.

qemu-server then branches: atomic clones skip the mirror; bulk clones
and all move-disk/migration take the bitmap-seed + mirror path, reusing
the existing BlockJob.pm completion code.

Two QEMU details I'd like to confirm for the bulk path: the dirty bitmap
must be created before the array copy starts, so out-of-band writes are
tracked; and the target can't be attached writable during the copy --
blockdev-add + mirror only after the array reports the copy done, the
mirror then healing the tracked blocks.

Does that split sound acceptable? If so I'll send the pve-storage RFC
PATCH first (hook + option + capability + APIVER/APIAGE bump), then the
qemu-server consumer.

Thanks,
Ciro