From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 750E31FF0E7 for ; Fri, 10 Jul 2026 16:18:50 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 97A102142F; Fri, 10 Jul 2026 16:18:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1783693089; x=1784297889; darn=lists.proxmox.com; h=content-type:references:in-reply-to:subject:cc:to:from:date :message-id:from:to:cc:subject:date:message-id:reply-to:content-type; bh=xKdpif3tIYJk1+P7EPsdHVJubABSAfAc+MLeyR+RR3s=; b=V+lV2JN+D3luXlhJrfpvpEWSLAFr58HbljPOrgTeZeYy6eYwmYoDjW8b8LaFQhtFAm bpCTy7/KFYl6YvcKmDKvliujtXwsH7YTv9pbPx7poQMLYx3Xld+DEgoBtMpYK1PTWJqV YKDT6mzs0DUVuMvtv1lXM2676irKIy1JG+TfSNJLH3k4Tupw1DtFru1ptYXDPj/aZEU0 uVhARj3JIsAePseSLyiAQxVLzJqRKvsk8mC8kN2GnSnilJlv4We6Lb47KE8avJmgKy8i UPYX6edkX+t+OgEty/TOOn51QR1K2cUquOQ3pIOkIL0LoJZ9tqkfqKfb3OeVTQuQXccH riFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783693089; x=1784297889; h=content-type:references:in-reply-to:subject:cc:to:from:date :message-id:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=xKdpif3tIYJk1+P7EPsdHVJubABSAfAc+MLeyR+RR3s=; b=nPKSpDouYkRqksQorBPCkyCnsc8MKcnXL+fGxEzVUY+3ouGtQ4yp+YssewkvSPv81d GVFsUSJ/LyeoHsim7wQXR9qPrbp+6ftBpi+Sbjyy3MjtStJ8KCCXnd4zyYuje9L8IuKD 371f7egMIN2dJOZIAQMHTEdUNHBYHZyQ7R8SUMa5nbQLs4bPeySYQN8BQsAdFrOgLD6j kCke5BR6mqfFJ57vNRe2EzFlmkyORQFiI3716VxWkk2vlyyZKBuwZruTZJwG/VvJijs5 qOYgvSei8MrnQZeEzO1tD4NJU5VmITcYJdFmITuSpIB1h9jDkmCSbxS3HkYIe5eY07dl pV8A== X-Gm-Message-State: AOJu0YzLMj11g0YIDKrht6OhXrqz/gPu4jgHyMDQINvWLa+/6xqYtOGH OwLmckK3OHH092prTmmjGAFsLSIFN0hzCeAy7cPsibSg/sPNbBtmBiLIRimnUjmg0VE= X-Gm-Gg: AfdE7ckOjBM+glpzoEWYOlugGJkO6RsXDmISQxBCy46m6c3fRFT961AmOwmnSSeffv9 yCiXxr+zdL4MVusfSjAkWxnSasbJemF1iqKwbtBADo3esz3EmBXGbOU85jTb3JHICFANKhB/RAW xQfm/yNtWUUptekGgJkiIYGL+Nv8qVVFD9RzVkW9OB+kzwYVFWz3ATnI155D5Tx5vnmhLDM8H40 3TWM06g1yhztjAEDs57XVgLUzW3IgqQW62hYFTLco+CtZO1P9yiJahtjBVpyHHxn0DizRQRotpu 30HGuSxA7xBtIlHAxO+gQu9vapTpXumyYLyCN0kA9csFUJgnqab4kY08vQmPaj/Teny3pKlR5U0 yECr1z/3ibr0DiZUYEmTyr+o5eW8WovZDuTnmCp1AId8yPaz6CGWSkjQkOYR8RnAlPEuhrvx7fW d2cQEAKhJktj90qaqptQ== X-Received: by 2002:a05:690c:7406:b0:80c:85b6:75bf with SMTP id 00721157ae682-81dbf588f4fmr92697777b3.72.1783693088671; Fri, 10 Jul 2026 07:18:08 -0700 (PDT) Message-ID: <6a50ff20.d6ce4a42.63dff.76fa@mx.google.com> Date: Fri, 10 Jul 2026 07:18:08 -0700 (PDT) From: Ciro Iriarte To: pve-devel@lists.proxmox.com Subject: Re: [RFC storage, qemu-server] offload full/live clone to storage backend In-Reply-To: References: <6a4416f5.01f0a1da.1533d8.2fdf@mx.google.com> <1783430000.wo44ge3ggk.astroid@yuna.none> <90aae63f-8d67-4a26-ab6b-ed32fd1537ce@proxmox.com> <1783500024.h4hi20lbpy.astroid@yuna.none> Content-Type: text/plain; charset=UTF-8 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.400 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: ZMQVA76NTJ3XRZVLHIECUQKP5YHNXZEY X-Message-ID-Hash: ZMQVA76NTJ3XRZVLHIECUQKP5YHNXZEY X-MailFrom: cyruspy@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On July 9, 2026 9:56 am, Fiona Ebner wrote: > Different storage types based on the same underlying technology (e.g. > ZFS) could still do it. That said, I don't mind not designing for this > but just for offloading between storages of the same type. Thanks both for the thorough feedback. Consolidated reply and a revised proposal below. Accepted as-is: - Base-class copy_image dies instead of returning undef (only reachable when the plugin advertises the capability). - Drop "atomically" -- the plugin owns cleanup and undoes its own allocation on failure; I'll document that as the contract. - Rename $vmid -> $target_vmid, and pass the target storage *config* so the plugin can judge whether a source/target pair is a valid offload. - copy-offload as a plain per-storage option that capable plugins honor. - Baseline: offload only between two instances of the same plugin type, the plugin deciding whether those instances can actually copy. Cross-type stays out of scope; since offload is opaque to PVE there's no volume_export/import-style lowering to generalize it, so I'll leave room in the signature but not design for it. The substantive point is separating two things "offload" blurs: point-in-time consistency of the copy, and independence of the result. Consistency: the bitmap + mirror model is the right completion path for any *smeared* copy -- a non-atomic, out-of-band bulk copy written while the source keeps changing. NFS 4.2 server-side copy (copy_file_range) is exactly that. Migration (move-disk, live migrate) needs it too, since the target must converge to current state and cut over -- reusing the replicated-migration mechanism fits well, agreed. But snapshot-based backends never smear: rbd snap+clone, lvm-thin/btrfs snapshots, zfs clone all copy from a static point-in-time source. There a host-side drive-mirror only re-writes blocks that are already correct -- not incorrect, just host I/O for work the array already did with none. Those should skip the host mirror: freeze once (PVE already does, per BlockJob.pm:168), snapshot/clone, thaw. That yields a valid point-in-time clone -- same as the mirror-at-cutover PVE produces today, minus the host data movement. Independence differs per backend, and this is what determines whether a snapshot is a valid *full* clone: - lvm-thin, btrfs: reference-counted, independent immediately; source can be removed at once. - rbd: depends on a protected parent snapshot until `rbd flatten` finishes; flatten runs on the Ceph cluster rather than as a host-side qemu-img copy, but the dependency is real until it completes. - zfs: a clone depends on its origin snapshot; `zfs promote` only re-parents that dependency (no help when the template must stay the clone source), and the only true full-copy route is a bulk send/recv, which is not instant. So "instant + independent + host-free" isn't achievable for zfs full clones -- that one wants the bulk path. So copy_image should advertise which case it is, and the core needs an independence barrier: the hook returns only once the target is independent, or the core records the pending dependency and refuses to delete the source/template/parent snapshot until the backend signals completion. Otherwise "snapshot and return" is a broken full clone for rbd/zfs. Concretely, two capability classes on top of the option: - copy-offload => atomic: static point-in-time source, no host mirror (freeze -> snapshot/clone -> thaw [-> background flatten]). - copy-offload => bulk: no instant independent snapshot -- needs a full data copy (NFS 4.2 SSC, non-atomic SAN copy, zfs send/recv); the running-VM case then needs the bitmap + mirror completion. qemu-server then branches: atomic clones skip the mirror; bulk clones and all move-disk/migration take the bitmap-seed + mirror path, reusing the existing BlockJob.pm completion code. Two QEMU details I'd like to confirm for the bulk path: the dirty bitmap must be created before the array copy starts, so out-of-band writes are tracked; and the target can't be attached writable during the copy -- blockdev-add + mirror only after the array reports the copy done, the mirror then healing the tracked blocks. Does that split sound acceptable? If so I'll send the pve-storage RFC PATCH first (hook + option + capability + APIVER/APIAGE bump), then the qemu-server consumer. Thanks, Ciro