From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id C63B81FF0B2 for ; Mon, 07 Sep 2026 07:09:01 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 7627421500; Mon, 07 Sep 2026 07:08:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1788757724; x=1789362524; darn=lists.proxmox.com; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=k1CMDbSEIQ9rmQdoJQyl/bOBjmkrwCUOb4JJC5eYQHo=; b=VMg8pVKehqlpjnV6FP006u3nozlv967CBOGn0B1MiMnJdyqabdfkc7GX3SAoZPb2tf XSazHybwuoLaOeV8KHmIzrR3FfN4acoUU7piV4D+OPF4k5ar04AA6VjLJR24Gnq5n5Tl reXTv1lJPwvQdJlQ18a0RbA9wCi1PRhtMkr04aKGi87scJv/lBOwPSAwBADMRl1TLitO Ql8E588HM+wXGjdHEFC60b7UXgU5usG5NruMNZDxJtENC+Z3LeF9vjpTaxMmz5nEuzPT /OApDoM38JHKgyXjpbowVQOpct9yaCbKLGnNf/456QjJ3PFGdDy0oxXm2vW03BN0mhFT 7KKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1788757724; x=1789362524; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=k1CMDbSEIQ9rmQdoJQyl/bOBjmkrwCUOb4JJC5eYQHo=; b=Ej+xwtLHYLwo7zrSR1JjHJlCJLFSdVwaR8GR3EYGIznCjTLfysGnhrJK7S2pAmj0BA grVIvMXdq8RXePPDGLhJf7Yckv6JLcc9I2+IVkWTncnVoPnprS9CO7fConQct0gy6ZSP 3V2Ltn2STANzDv8pqWGat5JEUl3UACE9WgW7bm7D5mN0UcpkEvYJ2ovbmX796BSNpEIm RsJ5YX5V7b5iLq/WYaeOP4c6GCS+PcKCyNsVLvreDtB8tSM5D24KPDEzbSm4BNLx49W4 VHHrZrRS/ih1AtXYOlQmrhxy9Vc9EJjPGsvEkPSCvc8V0Or+NRoh7RCp87Nxw2uZEaxW qyyg== X-Gm-Message-State: AFuF++nCqhfFgzFSsdPQQqYiY2/y4jdYuK9koUi7Q/shpVsOJSdp/Lj8 7DBNLZ5AZroIVB43t/l3WB37ERDJaYURQ7FtT3YwMmz5IdwRnGRWsWEd X-Gm-Gg: AYBFou1Vy4GqgNVbYFpVxg35hxq+EZ5eC0klxkH3+C92kqjUrz8maT9g0fUO5ZvMHOB 5W3Sqcikvse5B/1psuPD11a81L39M5TcY/ssEdJgMHD/i5UcbtK7dLjTnk4MCGqV0f5GDC8+ecn Q6AkrrwWpPxvoh5iv6QLkeOc9woSxpi77d07h/RKWIiyLEvXODEsxLhOwE3R8Qbz84k4yYlbkAd 26ZinjO/iNEtArXNY3/YCj/aoN1+/JtiTUBU9iSM3J2uTPLf+rAq2WRh5lsBqS+Fky2F4UujjWc HkuyqbW6B+Zm944GizrWFRglU90BGJBhhI2J/aNugpEtgvTViCJG5eB4e0oJFk82LR0ZGOYTKJZ MnzP1TeRCU+szEBYEFmIl1OlEbaIIAy/GIHIgiqlZsM1UChWxo6S7XLpWKlEKvddgXy+bo3Kotx bQV2P66j4aeZBeRVT5WsjXgCDl7IyZb422RuENiZLxRAeCHa/s9i+3qiU= X-Received: by 2002:a17:90b:4c44:b0:38e:97f0:aa4b with SMTP id 98e67ed59e1d1-39b261cdb38mr30116785a91.13.1788757724344; Sun, 06 Sep 2026 22:08:44 -0700 (PDT) From: Ciro Iriarte To: pve-devel@lists.proxmox.com Subject: [RFC PATCH docs] pvesm: document copy-offload and add it to the storage capability table Date: Mon, 07 Sep 2026 02:06:38 -0300 Message-ID: <20260907.docs.copyoffload@cyruspy.gmail.com> In-Reply-To: <20260720.0.copyoffload@cyruspy.gmail.com> References: <20260720.0.copyoffload@cyruspy.gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.300 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 7JQ3XTVC6EYY6VH5IF4QLI3RSNAOROHD X-Message-ID-Hash: 7JQ3XTVC6EYY6VH5IF4QLI3RSNAOROHD X-MailFrom: cyruspy@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The capability table is where people look to decide which storage to use, so a full-clone offload belongs in it next to Snapshots rather than only in the option list. Adds a "Clone offload" column and documents both `copy-offload` and `copy-offload-timeout` among the common storage properties. The column says what is implemented today, not what is theoretically possible, and the footnote says so explicitly -- NFS and CIFS can both do server-side copies and are marked `no` only because nothing calls them yet. Overstating it would send people looking for a switch that does nothing. Two caveats are called out because both are silent failures otherwise: - dir only qualifies when the underlying filesystem supports reflinks, which is a property of how the filesystem was created (XFS needs reflink=1) rather than of the storage configuration, and is not visible from the PVE side at all. - a qcow2 with a backing file is excluded, because a copy of an overlay keeps the overlay's dependency on its base. It would look like a working clone right up until the base is removed. The timeout description follows the schema rather than paraphrasing it: the default is one day, and the timer restarts on reported progress, so it bounds a stalled copy rather than a slow one. A large disk on a busy backend will not trip it. Documents the storage-side series on the matching `copy-image-offload` branch. Generated-By: Claude (https://claude.ai) Signed-off-by: Ciro Iriarte Co-Authored-By: Claude --- pvesm.adoc | 68 +++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 52 insertions(+), 16 deletions(-) diff --git a/pvesm.adoc b/pvesm.adoc index 5bd24b2..daa6984 100644 --- a/pvesm.adoc +++ b/pvesm.adoc @@ -65,23 +65,23 @@ nodes that can be accessed as RBD (RADOS Block Device). .Available storage types -[width="100%",cols="<2d,1*m,4*d",options="header"] +[width="100%",cols="<2d,1*m,5*d",options="header"] |======================================================================== -|Description |Plugin type |Level |Shared|Snapshots|Stable -|ZFS (local) |zfspool |both^1^|no |yes |yes -|Directory |dir |file |no |yes^2^ |yes -|BTRFS |btrfs |file |no |yes |TP^5^ -|NFS |nfs |file |yes |yes^2^ |yes -|CIFS |cifs |file |yes |yes^2^ |yes -|Proxmox Backup |pbs |both |yes |n/a |yes -|CephFS |cephfs |file |yes |yes |yes -|LVM |lvm |block |no^3^ |yes^4^ |yes -|LVM-thin |lvmthin |block |no |yes |yes -|iSCSI/kernel |iscsi |block |yes^3^|yes^4^ |yes -|iSCSI/libiscsi |iscsidirect |block |yes^3^|yes^4^ |yes -|FC/SAS |native^6^ |block |yes^3^|yes^4^ |yes -|Ceph/RBD |rbd |block |yes |yes |yes -|ZFS over iSCSI |zfs |block |yes |yes |yes +|Description |Plugin type |Level |Shared|Snapshots|Clone offload^7^|Stable +|ZFS (local) |zfspool |both^1^|no |yes |no |yes +|Directory |dir |file |no |yes^2^ |yes^8^ |yes +|BTRFS |btrfs |file |no |yes |yes |TP^5^ +|NFS |nfs |file |yes |yes^2^ |no |yes +|CIFS |cifs |file |yes |yes^2^ |no |yes +|Proxmox Backup |pbs |both |yes |n/a |n/a |yes +|CephFS |cephfs |file |yes |yes |no |yes +|LVM |lvm |block |no^3^ |yes^4^ |no |yes +|LVM-thin |lvmthin |block |no |yes |yes |yes +|iSCSI/kernel |iscsi |block |yes^3^|yes^4^ |no |yes +|iSCSI/libiscsi |iscsidirect |block |yes^3^|yes^4^ |no |yes +|FC/SAS |native^6^ |block |yes^3^|yes^4^ |n/a |yes +|Ceph/RBD |rbd |block |yes |yes |yes |yes +|ZFS over iSCSI |zfs |block |yes |yes |no |yes |======================================================================== ^1^: Disk images for VMs are stored in ZFS volume (zvol) datasets, which provide @@ -112,6 +112,17 @@ xref:pvesm_lvm_config[LVM configuration] section. ^6^ Fibre Channel (FC) and SAS block storage is handled directly by the host without a dedicated storage plugin. +^7^ Whether a *full* clone can be handed to the storage instead of being copied +byte by byte by the host. Off by default; enable it per storage with the +`copy-offload` option. This only applies within one storage backend -- copying +between two different backends always goes through the host. Storages marked +`no` are not necessarily incapable, they simply have no implementation yet. + +^8^ Only when the underlying filesystem supports reflinks (XFS created with +`reflink=1`, btrfs, or ZFS with block cloning enabled), and not for a 'qcow2' +image that has a backing file, since copying such an image would reproduce its +dependency on the base rather than a standalone disk. + Thin Provisioning ~~~~~~~~~~~~~~~~~ @@ -277,6 +288,31 @@ file-based storages. The default is `metadata`, which is treated like `off` for `raw` images. When using network storages in combination with large `qcow2` images, using `off` can help to avoid timeouts. +copy-offload:: + +Let the storage perform full clones itself instead of the host copying the image +byte by byte with `qemu-img convert`. Disabled by default. Depending on the +backend this can make a full clone close to instant and cost little or no extra +space, because the copy shares blocks with its source copy-on-write. The result +is still a normal, independent disk: the source can be deleted afterwards. + +It only takes effect when both storages are instances of the same plugin and +that plugin supports it -- see the ``Clone offload'' column above. Anything else +falls back to the usual host-side copy, so turning this on is safe even where it +cannot be used. + +copy-offload-timeout:: + +How long to wait, in seconds, for an offloaded copy to become independent of its +source before giving up and cleaning up the unfinished target (default: 86400, +one day). The timer restarts every time the backend reports progress, so this +bounds a copy that has stalled rather than one that is merely slow -- a large +disk on a busy backend will not trip it. + +Only relevant for backends that copy in the background, such as Ceph/RBD, where +the clone is readable immediately but has to be flattened before it stops +depending on its source. Backends that finish instantly never reach it. + WARNING: It is not advisable to use the same storage pool on different {pve} clusters. Some storage operation need exclusive access to the storage, so proper locking is required. While this is implemented -- 2.54.0