From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 3FF521FF0E1 for ; Mon, 13 Jul 2026 16:03:24 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 360392133D; Mon, 13 Jul 2026 16:03:23 +0200 (CEST) Message-ID: Date: Mon, 13 Jul 2026 16:03:17 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC storage, qemu-server] offload full/live clone to storage backend To: Ciro Iriarte , pve-devel@lists.proxmox.com References: <6a4416f5.01f0a1da.1533d8.2fdf@mx.google.com> <1783430000.wo44ge3ggk.astroid@yuna.none> <90aae63f-8d67-4a26-ab6b-ed32fd1537ce@proxmox.com> <1783500024.h4hi20lbpy.astroid@yuna.none> <6a50ff20.d6ce4a42.63dff.76fa@mx.google.com> Content-Language: en-US From: Fiona Ebner In-Reply-To: <6a50ff20.d6ce4a42.63dff.76fa@mx.google.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1783951382958 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.325 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: GFYPOPMJ5TGXEBUEFCSLDJKDVECQMZZX X-Message-ID-Hash: GFYPOPMJ5TGXEBUEFCSLDJKDVECQMZZX X-MailFrom: f.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Am 10.07.26 um 4:18 PM schrieb Ciro Iriarte: > On July 9, 2026 9:56 am, Fiona Ebner wrote: >> Different storage types based on the same underlying technology (e.g. >> ZFS) could still do it. That said, I don't mind not designing for this >> but just for offloading between storages of the same type. > > Thanks both for the thorough feedback. Consolidated reply and a revised > proposal below. > > Accepted as-is: > - Base-class copy_image dies instead of returning undef (only reachable > when the plugin advertises the capability). > - Drop "atomically" -- the plugin owns cleanup and undoes its own > allocation on failure; I'll document that as the contract. > - Rename $vmid -> $target_vmid, and pass the target storage *config* so > the plugin can judge whether a source/target pair is a valid offload. > - copy-offload as a plain per-storage option that capable plugins honor. > - Baseline: offload only between two instances of the same plugin type, > the plugin deciding whether those instances can actually copy. > Cross-type stays out of scope; since offload is opaque to PVE there's > no volume_export/import-style lowering to generalize it, so I'll leave > room in the signature but not design for it. > > The substantive point is separating two things "offload" blurs: > point-in-time consistency of the copy, and independence of the result. > > Consistency: the bitmap + mirror model is the right completion path for > any *smeared* copy -- a non-atomic, out-of-band bulk copy written while > the source keeps changing. NFS 4.2 server-side copy (copy_file_range) is > exactly that. Migration (move-disk, live migrate) needs it too, since > the target must converge to current state and cut over -- reusing the > replicated-migration mechanism fits well, agreed. > > But snapshot-based backends never smear: rbd snap+clone, lvm-thin/btrfs > snapshots, zfs clone all copy from a static point-in-time source. There > a host-side drive-mirror only re-writes blocks that are already correct > -- not incorrect, just host I/O for work the array already did with > none. Those should skip the host mirror: freeze once (PVE already does, > per BlockJob.pm:168), snapshot/clone, thaw. That yields a valid > point-in-time clone -- same as the mirror-at-cutover PVE produces today, > minus the host data movement. > > Independence differs per backend, and this is what determines whether a > snapshot is a valid *full* clone: > - lvm-thin, btrfs: reference-counted, independent immediately; source > can be removed at once. > - rbd: depends on a protected parent snapshot until `rbd flatten` > finishes; flatten runs on the Ceph cluster rather than as a host-side > qemu-img copy, but the dependency is real until it completes. > - zfs: a clone depends on its origin snapshot; `zfs promote` only > re-parents that dependency (no help when the template must stay the > clone source), and the only true full-copy route is a bulk send/recv, > which is not instant. So "instant + independent + host-free" isn't > achievable for zfs full clones -- that one wants the bulk path. > > So copy_image should advertise which case it is, and the core needs an > independence barrier: the hook returns only once the target is > independent, or the core records the pending dependency and refuses to > delete the source/template/parent snapshot until the backend signals > completion. Otherwise "snapshot and return" is a broken full clone for > rbd/zfs. > > Concretely, two capability classes on top of the option: > - copy-offload => atomic: static point-in-time source, no host mirror > (freeze -> snapshot/clone -> thaw [-> background flatten]). > - copy-offload => bulk: no instant independent snapshot -- needs a full > data copy (NFS 4.2 SSC, non-atomic SAN copy, zfs send/recv); the > running-VM case then needs the bitmap + mirror completion. I guess it's easiest to just have two different features to avoid the need to change the interface for volume_has_feature(). Each plugin with copy-offload support just needs to support the one that's better for it and qemu-server prefers the atomic one if available. > qemu-server then branches: atomic clones skip the mirror; bulk clones > and all move-disk/migration take the bitmap-seed + mirror path, reusing > the existing BlockJob.pm completion code. > > Two QEMU details I'd like to confirm for the bulk path: the dirty bitmap > must be created before the array copy starts, so out-of-band writes are > tracked; and the target can't be attached writable during the copy -- > blockdev-add + mirror only after the array reports the copy done, the > mirror then healing the tracked blocks. Yes. FWIW, I don't see an obvious blocker with the rest of the proposal either. > Does that split sound acceptable? If so I'll send the pve-storage RFC > PATCH first (hook + option + capability + APIVER/APIAGE bump), then the > qemu-server consumer. Could you include a rough draft for the qemu-server part as well? It's totally fine if not complete, but it would be useful to see the changes in context already. Best Regards, Fiona