From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 36AD81FF0B7 for ; Fri, 13 Nov 2026 12:46:22 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 21A802175A; Fri, 13 Nov 2026 12:46:15 +0100 (CET) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Fri, 13 Nov 2026 12:46:08 +0100 Message-Id: Subject: Re: [PATCH] PVE backup: vma reader: merge contiguous writes on restore From: "Elias Huhsovitz" To: "Thomas Lamprecht" , X-Mailer: aerc 0.20.0 References: <20261109160210.3380582-1-t.lamprecht@proxmox.com> In-Reply-To: <20261109160210.3380582-1-t.lamprecht@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794570368815 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.418 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 25YCYOFHZY3XHRCY5P5JVR3XU3WG4R2R X-Message-ID-Hash: 25YCYOFHZY3XHRCY5P5JVR3XU3WG4R2R X-MailFrom: e.huhsovitz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: I was a bit too slow with my review, since the code was applied yesterday, but I will document my findings anyway: I tested this by replacing /usr/bin/vma and /usr/bin/qemu-system-x86_64 with the patches build and dropping caches before each restore. I restored a 6.4 GiB .vma.zst (ext4 on virtual disk) to lvm thin and local-zfs. lvmthin: Old binaries: 2min 35s New binaries: 2min 14s local-zfs: virtually identical I only 3 did runs for each restore, so the numbers might be influenced by external factors. Nevertheless: While the time for lvmthin has decreased, I did not observe such a massive decrease in restore time, as the commit message suggested. (I didn't go out of my way to create a massive sparse vm. I simply tested this with a regular ubuntu VM.) 1 nit inline, otherwise this seems like a solid improvment to me: Reviewed-by: Elias Huhsovitz Tested-by: Elias Huhsovitz On Mon Nov 9, 2026 at 5:02 PM CET, Thomas Lamprecht wrote: > Restoring wrote every 64 KiB cluster, or every 4 KiB block of a > cluster with zero blocks, as a separate synchronous request. With the > page cache this mostly costs extra system calls, as writeback merges the > data again there. Without page cache =E2=80=93 like cache=3Dnone that qem= u-server > uses for restores to ZFS pools =E2=80=93 each request goes to the storage= on its > own and has to complete before vma continues reading the archive. > > To improve this, queue writes that continue the previous one on the same > target and issue them as a single request once the target changes, a gap > occurs, or the extent is done, as the queued data points into the extent > buffer. The vma writer puts up to 58 of the 64 KiB clusters into an > extent, so a request can now cover about 3.6 MiB. Writes to each target > stay synchronous and in order, and a successful restore produces the > same data. > > One small drawback: QEMU admits a throttled request as a whole and > delays the next one, so a bandwidth limit now takes effect in steps of > one such request instead of one cluster. Over a longer restore the > average rate stays the same, but it can be "burstier" if the limit is > less than ~30 Mbps (one 3.6 MiB batched write submitted). Might be > improvable, but not sure if its really worth the complexity. > > A simple benchmark of restoring a 3 GiB image from a zstd-compressed > archive like qemu-server does (inside a VM with ext4 on a virtual disk) > took on average 45s instead of 66s for a sparse target and 38s instead > of 83s when also writing zeroes, both with cache=3Dnone; so a rather > noticable improvement. With the page cache, the before/after times > stayed within the measurement noise. > > Signed-off-by: Thomas Lamprecht > --- > > Patch on top of the current pve downstream series. This fell out of > reviewing the impact that Elias noticed in Markus' "{pbs-,vma-}restore: > disable caching for LVM-thin" series [0], where vma restore performed > worse after over PBS restore. > > [0]: https://lore.proxmox.com/pve-devel/20260908123727.285848-1-m.frank@p= roxmox.com/ > > vma-reader.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++----- > 1 file changed, 70 insertions(+), 7 deletions(-) > > diff --git a/vma-reader.c b/vma-reader.c > index 2bce1b9d70..4209d6dc45 100644 > --- a/vma-reader.c > +++ b/vma-reader.c > @@ -49,6 +49,10 @@ struct VmaReader { > int64_t zero_cluster_data; > int64_t partial_zero_cluster_data; > int clusters_read_per; > + /* queued writes, covering one contiguous range of a single target *= / > + BlockBackend *batch_target; > + int64_t batch_offset; > + QEMUIOVector batch_qiov; > }; > =20 > static guint > @@ -210,6 +214,8 @@ void vma_reader_destroy(VmaReader *vmar) > g_free(vmar->head_data); > } > =20 > + qemu_iovec_destroy(&vmar->batch_qiov); > + > g_free(vmar); > =20 > }; > @@ -372,6 +378,7 @@ VmaReader *vma_reader_create(const char *filename, Er= ror **errp) > assert(errp); > =20 > VmaReader *vmar =3D g_new0(VmaReader, 1); > + qemu_iovec_init(&vmar->batch_qiov, 1); > =20 > if (strcmp(filename, "-") =3D=3D 0) { > vmar->fd =3D dup(0); > @@ -523,6 +530,64 @@ static size_t full_write(int fd, void *buf, size_t l= en) > return total; > } > =20 > +static int restore_flush_writes(VmaReader *vmar, Error **errp) > +{ > + QEMUIOVector *qiov =3D &vmar->batch_qiov; > + BlockBackend *target =3D vmar->batch_target; > + > + if (!qiov->size) { > + return 0; > + } > + > + int res =3D blk_pwritev(target, vmar->batch_offset, qiov->size, qiov= , 0); > + qemu_iovec_reset(qiov); > + if (res < 0) { > + bdrv_graph_rdlock_main_loop(); > + error_setg(errp, "blk_pwrite to %s failed (%d)", > + bdrv_get_device_name(blk_bs(target)), res); nit: The error should be "blk_pwritev to ...", since we are calling=20 blk_pwritev instead of blk_pwrite. > + bdrv_graph_rdunlock_main_loop(); > + return -1; > + } > + > + return 0; > +} > + > +/* > + * Writing every cluster, or every 4 KiB block of a partial cluster, on = its > + * own is slow without the host page cache, which otherwise merges them.= So > + * extend the queued write as long as the data continues it on the same = target. > + */ > +static int restore_queue_write(VmaReader *vmar, BlockBackend *target, > + unsigned char *buf, int64_t offset, > + size_t bytes, Error **errp) > +{ > + QEMUIOVector *qiov =3D &vmar->batch_qiov; > + > + if (qiov->size && (target !=3D vmar->batch_target || > + offset !=3D vmar->batch_offset + (int64_t)qiov->s= ize || > + qiov->niov >=3D blk_get_max_iov(target))) { > + if (restore_flush_writes(vmar, errp) < 0) { > + return -1; > + } > + } > + > + if (!qiov->size) { > + vmar->batch_target =3D target; > + vmar->batch_offset =3D offset; > + } > + > + struct iovec *last =3D qiov->niov ? &qiov->iov[qiov->niov - 1] : NUL= L; > + if (last && (unsigned char *)last->iov_base + last->iov_len =3D=3D b= uf) { > + /* adjacent in memory, keep the vector short */ > + last->iov_len +=3D bytes; > + qiov->size +=3D bytes; > + } else { > + qemu_iovec_add(qiov, buf, bytes); > + } > + > + return 0; > +} > + > static int restore_write_data(VmaReader *vmar, guint8 dev_id, > BlockBackend *target, int vmstate_fd, > unsigned char *buf, int64_t sector_num, > @@ -540,12 +605,9 @@ static int restore_write_data(VmaReader *vmar, guint= 8 dev_id, > } > } > } else { > - int res =3D blk_pwrite(target, sector_num * BDRV_SECTOR_SIZE, nb= _sectors * BDRV_SECTOR_SIZE, buf, 0); > - if (res < 0) { > - bdrv_graph_rdlock_main_loop(); > - error_setg(errp, "blk_pwrite to %s failed (%d)", > - bdrv_get_device_name(blk_bs(target)), res); > - bdrv_graph_rdunlock_main_loop(); > + if (restore_queue_write(vmar, target, buf, > + sector_num * BDRV_SECTOR_SIZE, > + nb_sectors * BDRV_SECTOR_SIZE, errp) < 0= ) { > return -1; > } > } > @@ -720,7 +782,8 @@ static int restore_extent(VmaReader *vmar, unsigned c= har *buf, > return -1; > } > =20 > - return 0; > + /* queued writes point into the extent buffer, which gets reused */ > + return restore_flush_writes(vmar, errp); > } > =20 > static int vma_reader_restore_full(VmaReader *vmar, int vmstate_fd,