From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 9423F1FF146 for ; Tue, 07 Jul 2026 15:49:30 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id BB7C82143B; Tue, 07 Jul 2026 15:49:27 +0200 (CEST) Date: Tue, 07 Jul 2026 15:48:49 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= Subject: Re: [RFC storage, qemu-server] offload full/live clone to storage backend To: Ciro Iriarte , pve-devel@lists.proxmox.com References: <6a4416f5.01f0a1da.1533d8.2fdf@mx.google.com> In-Reply-To: <6a4416f5.01f0a1da.1533d8.2fdf@mx.google.com> MIME-Version: 1.0 User-Agent: astroid/0.17.0 (https://github.com/astroidmail/astroid) Message-Id: <1783430000.wo44ge3ggk.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1783432124742 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.400 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 27AFKEABDUQD3UW4TWD5OQTGHR5RYL7J X-Message-ID-Hash: 27AFKEABDUQD3UW4TWD5OQTGHR5RYL7J X-MailFrom: f.gruenbichler@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: CCing Fiona and Max On June 30, 2026 9:20 pm, Ciro Iriarte wrote: > Hello, >=20 > I would like feedback on a storage-plugin contract that lets a backend > perform full-clone copies itself, instead of PVE moving every block > through the host. >=20 > Decision I am asking about > -------------------------- > For a full clone where source and destination live on the same > offload-capable storage, should PVE delegate the copy to the storage > plugin (array-internal copy) rather than run qemu-img convert / > drive-mirror on the host? If yes, I propose the minimal contract below. >=20 > Background > ---------- > clone_image already offloads LINKED clones (CoW). Full clones do not > come through it: clone_disk() does vdisk_alloc + qemu-img convert > (offline) or drive-mirror (running), reading and writing the whole > volume over the host even when the array could copy it internally in > near-constant time with zero host I/O. volume_has_feature returns > copy =3D> 1 today, but that only means "may be copied", not "the storage > can copy it itself" -- there is no hook for the storage to do the copy. >=20 > Proposed contract (pve-storage) > ------------------------------- > - Add an optional plugin method, copy_image($scfg, $storeid, > $src_volname, $vmid, $dst_format, $opts), that allocates and copies > on the backend atomically and returns the new volname. Base class > returns undef (unsupported), so existing plugins are unaffected -- > the driver is the sole source of the capability. I don't think "atomically" is the right word here - if the copying fails, the plugin would need to undo the allocation (if needed) as well. $vmid should maybe be $target_vmid to make it explicit (it can be the same as the $vmid encoded in the $src_volname). see further below as well. > - Add a copy-offload feature to volume_has_feature, negotiated with the > target storeid/format, so the driver also decides per copy whether > this specific src->dst pair can be offloaded. the target storage and format can be passed in via $opts, but we'd also need the target storage config (all the plugin methods get the per-instance config, and not the global one), else the plugin cannot really decide whether the source and target storage are a valid offload combination? > - Add a per-storage copy-offload option as a policy override (default > on), exposed only in the options of drivers that advertise the > capability, so it can disable offload but never enable it where > unsupported. Setting it to 0 routes that storage back to the > host-assisted path with no other change. this can just be a regular option, and plugins enabling copy-offload would then need to also add support/honor it > - Bump the storage APIVER/APIAGE. >=20 > Consumer (qemu-server) > ---------------------- > - In clone_disk(), in the offline full path, when src and dst are the > same storage, formats match, the disk is not a special disk > (cloudinit/efidisk0/tpmstate0), and the plugin advertises > copy-offload, call the new copy_image hook instead of qemu-img > convert. Otherwise run the existing path unchanged. > - A failed offload aborts the clone (existing rollback frees the > target); it does not silently fall back to a host copy, so > misbehaviour surfaces as a clean error rather than a bad volume. > > Live clone (running VM) > ----------------------- > A clone is not a migration: the source keeps running on its own volume, > so the target only needs a consistent point-in-time copy, not > drive-mirror convergence. For a running VM, quiesce the guest once > around the whole disk set (guest-fsfreeze-freeze) at the clone loop > level, flush QEMU's caches, run the array copy per disk, then thaw. an offloaded copy might still take a while, so I don't think this is the approach we want. instead, we probably want to invert it, and use the same mechanism for regular "live" move disk and "live" cloning of whole VMs: 1. create bitmap 2. copy disk using offload mechanism (inconsistent state) 3. mirror using bitmap (all writes between 1 and 3 will be done once more on the qemu level, but that is just the delta) 4. auto-converge for move disk, repeat 1-3 for other disks and complete when all are done when cloning we are using the same mechanism atm when live-migrating VMs between nodes when replication is enabled - it allows us to piggy-back on top of the replicated volumes and only transfer the delta. > Online offload runs only when the freeze succeeds, giving an > FS-consistent copy; if there is no agent or the freeze fails, fall back > to drive-mirror. This needs no separate switch -- the same copy-offload > option governs it, and the freeze requirement keeps it from ever > producing a crash-consistent copy silently. I think the question of how to incorporate freeze for cloning running VMs is a bit orthogonal? atm we don't freeze, so there is no cross-disk consistency when cloning a running VM, other than that provided by the mirror blockjob. >=20 > Out of scope > ------------ > - Cross-storage and format-converting clones (inherently > host-mediated). agreed. cross-storage could potentially be a follow-up, and we might want to account for it in the copy_image signature/parameters? > - VM move-disk needs no separate work: it already calls > clone_disk(full) and its "same storage, same format" case is > rejected, so it inherits this hook without ever mis-firing. see above > - Container volumes (storage_migrate path) -- a possible follow-up. this would probably be nice to have. we've had requests to allow using storage_migrate (or volume_export/volume_import) for copying container volumes - it can be faster than the current rsync-based approach, and a storage-offloaded variant would likely be even faster in many cases.. but it is a bit of its own series, as long as it's kept in mind when developing this one it should be fine. >=20 > If the contract looks acceptable, I will follow up with [RFC PATCH] > series against pve-storage and then qemu-server. My CLA is on file. I'd suggest waiting a few more days in case additional feedback comes in, but the plan doesn't sound too bad to me :) Thanks for wanting to tackle this! > Tracking/details: > https://github.com/ciroiriarte/pve-FCLUPlugin/issues/11 >=20 > Thanks, > Ciro Iriarte >=20 >=20 >=20 >=20