public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: "Elias Huhsovitz" <e.huhsovitz@proxmox.com>
To: "Thomas Lamprecht" <t.lamprecht@proxmox.com>,
	<pve-devel@lists.proxmox.com>
Subject: Re: [PATCH] PVE backup: vma reader: merge contiguous writes on restore
Date: Fri, 13 Nov 2026 12:46:08 +0100	[thread overview]
Message-ID: <DMU23Q7GMC5Q.15U75O6NTCZ7R@proxmox.com> (raw)
In-Reply-To: <20261109160210.3380582-1-t.lamprecht@proxmox.com>

I was a bit too slow with my review, since the code was applied
yesterday, but I will document my findings anyway:

I tested this by replacing /usr/bin/vma and /usr/bin/qemu-system-x86_64
with the patches build and dropping caches before each restore.

I restored a 6.4 GiB .vma.zst (ext4 on virtual disk) to lvm thin and
local-zfs.

lvmthin:
Old binaries: 2min 35s
New binaries: 2min 14s

local-zfs:
virtually identical

I only 3 did runs for each restore, so the numbers might be influenced
by external factors.

Nevertheless: While the time for lvmthin has decreased, I did not
observe such a massive decrease in restore time, as the commit message
suggested. (I didn't go out of my way to create a massive sparse vm. I
simply tested this with a regular ubuntu VM.)

1 nit inline, otherwise this seems like a solid improvment to me:

Reviewed-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>
Tested-by: Elias Huhsovitz <e.huhsovitz@proxmox.com>

On Mon Nov 9, 2026 at 5:02 PM CET, Thomas Lamprecht wrote:
> Restoring wrote every 64 KiB cluster, or every 4 KiB block of a
> cluster with zero blocks, as a separate synchronous request. With the
> page cache this mostly costs extra system calls, as writeback merges the
> data again there. Without page cache – like cache=none that qemu-server
> uses for restores to ZFS pools – each request goes to the storage on its
> own and has to complete before vma continues reading the archive.
>
> To improve this, queue writes that continue the previous one on the same
> target and issue them as a single request once the target changes, a gap
> occurs, or the extent is done, as the queued data points into the extent
> buffer. The vma writer puts up to 58 of the 64 KiB clusters into an
> extent, so a request can now cover about 3.6 MiB. Writes to each target
> stay synchronous and in order, and a successful restore produces the
> same data.
>
> One small drawback: QEMU admits a throttled request as a whole and
> delays the next one, so a bandwidth limit now takes effect in steps of
> one such request instead of one cluster. Over a longer restore the
> average rate stays the same, but it can be "burstier" if the limit is
> less than ~30 Mbps (one 3.6 MiB batched write submitted). Might be
> improvable, but not sure if its really worth the complexity.
>
> A simple benchmark of restoring a 3 GiB image from a zstd-compressed
> archive like qemu-server does (inside a VM with ext4 on a virtual disk)
> took on average 45s instead of 66s for a sparse target and 38s instead
> of 83s when also writing zeroes, both with cache=none; so a rather
> noticable improvement. With the page cache, the before/after times
> stayed within the measurement noise.
>
> Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
> ---
>
> Patch on top of the current pve downstream series. This fell out of
> reviewing the impact that Elias noticed in Markus' "{pbs-,vma-}restore:
> disable caching for LVM-thin" series [0], where vma restore performed
> worse after over PBS restore.
>
> [0]: https://lore.proxmox.com/pve-devel/20260908123727.285848-1-m.frank@proxmox.com/
>
>  vma-reader.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++-----
>  1 file changed, 70 insertions(+), 7 deletions(-)
>
> diff --git a/vma-reader.c b/vma-reader.c
> index 2bce1b9d70..4209d6dc45 100644
> --- a/vma-reader.c
> +++ b/vma-reader.c
> @@ -49,6 +49,10 @@ struct VmaReader {
>      int64_t zero_cluster_data;
>      int64_t partial_zero_cluster_data;
>      int clusters_read_per;
> +    /* queued writes, covering one contiguous range of a single target */
> +    BlockBackend *batch_target;
> +    int64_t batch_offset;
> +    QEMUIOVector batch_qiov;
>  };
>  
>  static guint
> @@ -210,6 +214,8 @@ void vma_reader_destroy(VmaReader *vmar)
>          g_free(vmar->head_data);
>      }
>  
> +    qemu_iovec_destroy(&vmar->batch_qiov);
> +
>      g_free(vmar);
>  
>  };
> @@ -372,6 +378,7 @@ VmaReader *vma_reader_create(const char *filename, Error **errp)
>      assert(errp);
>  
>      VmaReader *vmar = g_new0(VmaReader, 1);
> +    qemu_iovec_init(&vmar->batch_qiov, 1);
>  
>      if (strcmp(filename, "-") == 0) {
>          vmar->fd = dup(0);
> @@ -523,6 +530,64 @@ static size_t full_write(int fd, void *buf, size_t len)
>      return total;
>  }
>  
> +static int restore_flush_writes(VmaReader *vmar, Error **errp)
> +{
> +    QEMUIOVector *qiov = &vmar->batch_qiov;
> +    BlockBackend *target = vmar->batch_target;
> +
> +    if (!qiov->size) {
> +        return 0;
> +    }
> +
> +    int res = blk_pwritev(target, vmar->batch_offset, qiov->size, qiov, 0);
> +    qemu_iovec_reset(qiov);
> +    if (res < 0) {
> +        bdrv_graph_rdlock_main_loop();
> +        error_setg(errp, "blk_pwrite to %s failed (%d)",
> +                   bdrv_get_device_name(blk_bs(target)), res);

nit: The error should be "blk_pwritev to ...", since we are calling 
blk_pwritev instead of blk_pwrite.

> +        bdrv_graph_rdunlock_main_loop();
> +        return -1;
> +    }
> +
> +    return 0;
> +}
> +
> +/*
> + * Writing every cluster, or every 4 KiB block of a partial cluster, on its
> + * own is slow without the host page cache, which otherwise merges them. So
> + * extend the queued write as long as the data continues it on the same target.
> + */
> +static int restore_queue_write(VmaReader *vmar, BlockBackend *target,
> +                               unsigned char *buf, int64_t offset,
> +                               size_t bytes, Error **errp)
> +{
> +    QEMUIOVector *qiov = &vmar->batch_qiov;
> +
> +    if (qiov->size && (target != vmar->batch_target ||
> +                       offset != vmar->batch_offset + (int64_t)qiov->size ||
> +                       qiov->niov >= blk_get_max_iov(target))) {
> +        if (restore_flush_writes(vmar, errp) < 0) {
> +            return -1;
> +        }
> +    }
> +
> +    if (!qiov->size) {
> +        vmar->batch_target = target;
> +        vmar->batch_offset = offset;
> +    }
> +
> +    struct iovec *last = qiov->niov ? &qiov->iov[qiov->niov - 1] : NULL;
> +    if (last && (unsigned char *)last->iov_base + last->iov_len == buf) {
> +        /* adjacent in memory, keep the vector short */
> +        last->iov_len += bytes;
> +        qiov->size += bytes;
> +    } else {
> +        qemu_iovec_add(qiov, buf, bytes);
> +    }
> +
> +    return 0;
> +}
> +
>  static int restore_write_data(VmaReader *vmar, guint8 dev_id,
>                                BlockBackend *target, int vmstate_fd,
>                                unsigned char *buf, int64_t sector_num,
> @@ -540,12 +605,9 @@ static int restore_write_data(VmaReader *vmar, guint8 dev_id,
>              }
>          }
>      } else {
> -        int res = blk_pwrite(target, sector_num * BDRV_SECTOR_SIZE, nb_sectors * BDRV_SECTOR_SIZE, buf, 0);
> -        if (res < 0) {
> -            bdrv_graph_rdlock_main_loop();
> -            error_setg(errp, "blk_pwrite to %s failed (%d)",
> -                       bdrv_get_device_name(blk_bs(target)), res);
> -            bdrv_graph_rdunlock_main_loop();
> +        if (restore_queue_write(vmar, target, buf,
> +                                sector_num * BDRV_SECTOR_SIZE,
> +                                nb_sectors * BDRV_SECTOR_SIZE, errp) < 0) {
>              return -1;
>          }
>      }
> @@ -720,7 +782,8 @@ static int restore_extent(VmaReader *vmar, unsigned char *buf,
>          return -1;
>      }
>  
> -    return 0;
> +    /* queued writes point into the extent buffer, which gets reused */
> +    return restore_flush_writes(vmar, errp);
>  }
>  
>  static int vma_reader_restore_full(VmaReader *vmar, int vmstate_fd,





      parent reply	other threads:[~2026-11-13 11:46 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-09 16:02 [PATCH] PVE backup: vma reader: merge contiguous writes on restore Thomas Lamprecht
2026-11-12 22:35 ` applied: " Thomas Lamprecht
2026-11-13 11:46 ` Elias Huhsovitz [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMU23Q7GMC5Q.15U75O6NTCZ7R@proxmox.com \
    --to=e.huhsovitz@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    --cc=t.lamprecht@proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal