Re: [RFC storage, qemu-server] offload full/live clone to storage backend

[Fiona Ebner <f.ebner@proxmox.com>]

Am 10.07.26 um 4:18 PM schrieb Ciro Iriarte:
> On July 9, 2026 9:56 am, Fiona Ebner wrote:
>> Different storage types based on the same underlying technology (e.g.
>> ZFS) could still do it. That said, I don't mind not designing for this
>> but just for offloading between storages of the same type.
> 
> Thanks both for the thorough feedback. Consolidated reply and a revised
> proposal below.
> 
> Accepted as-is:
> - Base-class copy_image dies instead of returning undef (only reachable
>   when the plugin advertises the capability).
> - Drop "atomically" -- the plugin owns cleanup and undoes its own
>   allocation on failure; I'll document that as the contract.
> - Rename $vmid -> $target_vmid, and pass the target storage *config* so
>   the plugin can judge whether a source/target pair is a valid offload.
> - copy-offload as a plain per-storage option that capable plugins honor.
> - Baseline: offload only between two instances of the same plugin type,
>   the plugin deciding whether those instances can actually copy.
>   Cross-type stays out of scope; since offload is opaque to PVE there's
>   no volume_export/import-style lowering to generalize it, so I'll leave
>   room in the signature but not design for it.
> 
> The substantive point is separating two things "offload" blurs:
> point-in-time consistency of the copy, and independence of the result.
> 
> Consistency: the bitmap + mirror model is the right completion path for
> any *smeared* copy -- a non-atomic, out-of-band bulk copy written while
> the source keeps changing. NFS 4.2 server-side copy (copy_file_range) is
> exactly that. Migration (move-disk, live migrate) needs it too, since
> the target must converge to current state and cut over -- reusing the
> replicated-migration mechanism fits well, agreed.
> 
> But snapshot-based backends never smear: rbd snap+clone, lvm-thin/btrfs
> snapshots, zfs clone all copy from a static point-in-time source. There
> a host-side drive-mirror only re-writes blocks that are already correct
> -- not incorrect, just host I/O for work the array already did with
> none. Those should skip the host mirror: freeze once (PVE already does,
> per BlockJob.pm:168), snapshot/clone, thaw. That yields a valid
> point-in-time clone -- same as the mirror-at-cutover PVE produces today,
> minus the host data movement.
> 
> Independence differs per backend, and this is what determines whether a
> snapshot is a valid *full* clone:
> - lvm-thin, btrfs: reference-counted, independent immediately; source
>   can be removed at once.
> - rbd: depends on a protected parent snapshot until `rbd flatten`
>   finishes; flatten runs on the Ceph cluster rather than as a host-side
>   qemu-img copy, but the dependency is real until it completes.
> - zfs: a clone depends on its origin snapshot; `zfs promote` only
>   re-parents that dependency (no help when the template must stay the
>   clone source), and the only true full-copy route is a bulk send/recv,
>   which is not instant. So "instant + independent + host-free" isn't
>   achievable for zfs full clones -- that one wants the bulk path.
> 
> So copy_image should advertise which case it is, and the core needs an
> independence barrier: the hook returns only once the target is
> independent, or the core records the pending dependency and refuses to
> delete the source/template/parent snapshot until the backend signals
> completion. Otherwise "snapshot and return" is a broken full clone for
> rbd/zfs.
> 
> Concretely, two capability classes on top of the option:
> - copy-offload => atomic: static point-in-time source, no host mirror
>   (freeze -> snapshot/clone -> thaw [-> background flatten]).
> - copy-offload => bulk: no instant independent snapshot -- needs a full
>   data copy (NFS 4.2 SSC, non-atomic SAN copy, zfs send/recv); the
>   running-VM case then needs the bitmap + mirror completion.

I guess it's easiest to just have two different features to avoid the
need to change the interface for volume_has_feature(). Each plugin with
copy-offload support just needs to support the one that's better for it
and qemu-server prefers the atomic one if available.

> qemu-server then branches: atomic clones skip the mirror; bulk clones
> and all move-disk/migration take the bitmap-seed + mirror path, reusing
> the existing BlockJob.pm completion code.
> 
> Two QEMU details I'd like to confirm for the bulk path: the dirty bitmap
> must be created before the array copy starts, so out-of-band writes are
> tracked; and the target can't be attached writable during the copy --
> blockdev-add + mirror only after the array reports the copy done, the
> mirror then healing the tracked blocks.

Yes.

FWIW, I don't see an obvious blocker with the rest of the proposal either.

> Does that split sound acceptable? If so I'll send the pve-storage RFC
> PATCH first (hook + option + capability + APIVER/APIAGE bump), then the
> qemu-server consumer.

Could you include a rough draft for the qemu-server part as well? It's
totally fine if not complete, but it would be useful to see the changes
in context already.

Best Regards,
Fiona