From: Ciro Iriarte <cyruspy@gmail.com>
To: pve-devel@lists.proxmox.com
Subject: [RFC PATCH docs] pvesm: document copy-offload and add it to the storage capability table
Date: Mon, 07 Sep 2026 02:06:38 -0300 [thread overview]
Message-ID: <20260907.docs.copyoffload@cyruspy.gmail.com> (raw)
In-Reply-To: <20260720.0.copyoffload@cyruspy.gmail.com>
The capability table is where people look to decide which storage to use, so a
full-clone offload belongs in it next to Snapshots rather than only in the option
list. Adds a "Clone offload" column and documents both `copy-offload` and
`copy-offload-timeout` among the common storage properties.
The column says what is implemented today, not what is theoretically possible, and
the footnote says so explicitly -- NFS and CIFS can both do server-side copies and
are marked `no` only because nothing calls them yet. Overstating it would send people
looking for a switch that does nothing.
Two caveats are called out because both are silent failures otherwise:
- dir only qualifies when the underlying filesystem supports reflinks, which is a
property of how the filesystem was created (XFS needs reflink=1) rather than of the
storage configuration, and is not visible from the PVE side at all.
- a qcow2 with a backing file is excluded, because a copy of an overlay keeps the
overlay's dependency on its base. It would look like a working clone right up until
the base is removed.
The timeout description follows the schema rather than paraphrasing it: the default is
one day, and the timer restarts on reported progress, so it bounds a stalled copy
rather than a slow one. A large disk on a busy backend will not trip it.
Documents the storage-side series on the matching `copy-image-offload` branch.
Generated-By: Claude (https://claude.ai)
Signed-off-by: Ciro Iriarte <ciro.iriarte+software@gmail.com>
Co-Authored-By: Claude <noreply@anthropic.com>
---
pvesm.adoc | 68 +++++++++++++++++++++++++++++++++++++++++-------------
1 file changed, 52 insertions(+), 16 deletions(-)
diff --git a/pvesm.adoc b/pvesm.adoc
index 5bd24b2..daa6984 100644
--- a/pvesm.adoc
+++ b/pvesm.adoc
@@ -65,23 +65,23 @@ nodes that can be accessed as RBD (RADOS Block Device).
.Available storage types
-[width="100%",cols="<2d,1*m,4*d",options="header"]
+[width="100%",cols="<2d,1*m,5*d",options="header"]
|========================================================================
-|Description |Plugin type |Level |Shared|Snapshots|Stable
-|ZFS (local) |zfspool |both^1^|no |yes |yes
-|Directory |dir |file |no |yes^2^ |yes
-|BTRFS |btrfs |file |no |yes |TP^5^
-|NFS |nfs |file |yes |yes^2^ |yes
-|CIFS |cifs |file |yes |yes^2^ |yes
-|Proxmox Backup |pbs |both |yes |n/a |yes
-|CephFS |cephfs |file |yes |yes |yes
-|LVM |lvm |block |no^3^ |yes^4^ |yes
-|LVM-thin |lvmthin |block |no |yes |yes
-|iSCSI/kernel |iscsi |block |yes^3^|yes^4^ |yes
-|iSCSI/libiscsi |iscsidirect |block |yes^3^|yes^4^ |yes
-|FC/SAS |native^6^ |block |yes^3^|yes^4^ |yes
-|Ceph/RBD |rbd |block |yes |yes |yes
-|ZFS over iSCSI |zfs |block |yes |yes |yes
+|Description |Plugin type |Level |Shared|Snapshots|Clone offload^7^|Stable
+|ZFS (local) |zfspool |both^1^|no |yes |no |yes
+|Directory |dir |file |no |yes^2^ |yes^8^ |yes
+|BTRFS |btrfs |file |no |yes |yes |TP^5^
+|NFS |nfs |file |yes |yes^2^ |no |yes
+|CIFS |cifs |file |yes |yes^2^ |no |yes
+|Proxmox Backup |pbs |both |yes |n/a |n/a |yes
+|CephFS |cephfs |file |yes |yes |no |yes
+|LVM |lvm |block |no^3^ |yes^4^ |no |yes
+|LVM-thin |lvmthin |block |no |yes |yes |yes
+|iSCSI/kernel |iscsi |block |yes^3^|yes^4^ |no |yes
+|iSCSI/libiscsi |iscsidirect |block |yes^3^|yes^4^ |no |yes
+|FC/SAS |native^6^ |block |yes^3^|yes^4^ |n/a |yes
+|Ceph/RBD |rbd |block |yes |yes |yes |yes
+|ZFS over iSCSI |zfs |block |yes |yes |no |yes
|========================================================================
^1^: Disk images for VMs are stored in ZFS volume (zvol) datasets, which provide
@@ -112,6 +112,17 @@ xref:pvesm_lvm_config[LVM configuration] section.
^6^ Fibre Channel (FC) and SAS block storage is handled directly by the host
without a dedicated storage plugin.
+^7^ Whether a *full* clone can be handed to the storage instead of being copied
+byte by byte by the host. Off by default; enable it per storage with the
+`copy-offload` option. This only applies within one storage backend -- copying
+between two different backends always goes through the host. Storages marked
+`no` are not necessarily incapable, they simply have no implementation yet.
+
+^8^ Only when the underlying filesystem supports reflinks (XFS created with
+`reflink=1`, btrfs, or ZFS with block cloning enabled), and not for a 'qcow2'
+image that has a backing file, since copying such an image would reproduce its
+dependency on the base rather than a standalone disk.
+
Thin Provisioning
~~~~~~~~~~~~~~~~~
@@ -277,6 +288,31 @@ file-based storages. The default is `metadata`, which is treated like `off` for
`raw` images. When using network storages in combination with large `qcow2`
images, using `off` can help to avoid timeouts.
+copy-offload::
+
+Let the storage perform full clones itself instead of the host copying the image
+byte by byte with `qemu-img convert`. Disabled by default. Depending on the
+backend this can make a full clone close to instant and cost little or no extra
+space, because the copy shares blocks with its source copy-on-write. The result
+is still a normal, independent disk: the source can be deleted afterwards.
+
+It only takes effect when both storages are instances of the same plugin and
+that plugin supports it -- see the ``Clone offload'' column above. Anything else
+falls back to the usual host-side copy, so turning this on is safe even where it
+cannot be used.
+
+copy-offload-timeout::
+
+How long to wait, in seconds, for an offloaded copy to become independent of its
+source before giving up and cleaning up the unfinished target (default: 86400,
+one day). The timer restarts every time the backend reports progress, so this
+bounds a copy that has stalled rather than one that is merely slow -- a large
+disk on a busy backend will not trip it.
+
+Only relevant for backends that copy in the background, such as Ceph/RBD, where
+the clone is readable immediately but has to be flattened before it stops
+depending on its source. Backends that finish instantly never reach it.
+
WARNING: It is not advisable to use the same storage pool on different
{pve} clusters. Some storage operation need exclusive access to the
storage, so proper locking is required. While this is implemented
--
2.54.0
prev parent reply other threads:[~2026-09-07 5:09 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-07-20 14:29 [RFC PATCH storage, qemu-server 0/5] offload full clone to the storage backend Ciro Iriarte
2026-07-20 14:29 ` [RFC PATCH storage 1/5] storage: add asynchronous copy-offload hook for full copies Ciro Iriarte
2026-07-20 14:29 ` [RFC PATCH storage 2/5] rbd: implement copy-offload via snapshot + clone + flatten Ciro Iriarte
2026-07-20 14:30 ` [RFC PATCH storage 3/5] dir: implement copy-offload via reflink (FICLONE) Ciro Iriarte
2026-07-20 14:30 ` [RFC PATCH storage 4/5] btrfs, lvmthin: implement copy-offload (atomic class) Ciro Iriarte
2026-07-20 14:30 ` [RFC PATCH qemu-server 5/5] use storage copy-offload for full clone Ciro Iriarte
2026-07-22 15:34 ` [RFC PATCH storage] rbd: fix copy-offload flatten status race on fast small copies Ciro Iriarte
2026-09-07 5:06 ` Ciro Iriarte [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260907.docs.copyoffload@cyruspy.gmail.com \
--to=cyruspy@gmail.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.