all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH qemu] fix #8343, #8361: avoid QMP timeouts and VM stalls in backups of RBD disks
Date: Fri, 20 Nov 2026 12:38:01 +0100	[thread overview]
Message-ID: <20261120113834.4069058-1-t.lamprecht@proxmox.com> (raw)

For an RBD image with fast-diff, a backup queries the block status of
each chunk, and each query waits until the requests in flight on the
image have completed. For a disk without an iothread, QEMU waited for
that in its main loop. Since pve-qemu-kvm 11.1.3-2, backups to a
Proxmox Backup Server process chunks in parallel, which keeps more
reads in flight and so makes these waits longer. The guest could stall
meanwhile, and a backup failed when one of its QMP commands timed out.

Add a patch that runs these queries in a worker thread instead.
Backups of a running VM can take longer with it, as the guest no longer
stalls and its I/O competes with the backup. It does not fix bug #7909,
the slow reads of RBD disks in backups.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 ...uery-block-status-in-a-worker-thread.patch | 135 ++++++++++++++++++
 debian/patches/series                         |   1 +
 2 files changed, 136 insertions(+)
 create mode 100644 debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch

diff --git a/debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch b/debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch
new file mode 100644
index 0000000000..ec8600779d
--- /dev/null
+++ b/debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch
@@ -0,0 +1,135 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: Thomas Lamprecht <t.lamprecht@proxmox.com>
+Date: Fri, 20 Nov 2026 11:46:52 +0100
+Subject: [PATCH] block/rbd: query block status in a worker thread
+
+To answer a block status query for an image with fast-diff, the driver
+calls rbd_diff_iterate2(). That call blocks, and before it reports
+anything, librbd waits until the requests in flight on the image have
+completed. So the query blocks the thread of the image's AioContext.
+For a disk without an iothread, that is the main loop, which holds the
+BQL meanwhile.
+
+A backup queries the block status of each chunk before copying it,
+while up to max-workers reads of other chunks are in flight. Each of
+these queries then blocks the main loop until those reads are done.
+With slow OSDs, QMP commands can time out, and the vCPUs stall while
+they wait for the BQL. In Proxmox VE, this got much more likely once
+backups to a Proxmox Backup Server started to process chunks in
+parallel, which keeps more reads in flight.
+
+Call rbd_diff_iterate2() in a worker thread of the AioContext's thread
+pool, so that the coroutine yields while librbd waits. A query still
+waits as long, but the AioContext can handle other requests and events
+in the meantime. Reading a backup's data does not get faster, as each
+query still waits for the reads in flight before the next read can
+start. If the guest writes during a backup, the backup can even take
+longer, as the guest no longer stalls and its requests compete with the
+backup's.
+
+Test setup: a three-node Ceph cluster with slow OSDs, and a VM whose
+8 GiB disk without iothread holds 3 GiB of data. It is backed up with
+16 workers to a Proxmox Backup Server in three ways: stopped with data
+that the server already has, stopped with new data, and running while
+the guest writes. The last row is a stopped VM with a 256 GiB disk that
+has 8192 alternating allocated and unallocated runs. Before is
+pve-qemu-kvm 11.1.3-3, after is the same with this patch:
+
+                     runs  longest QMP reply    backup time
+                           before     after     before    after
+  stopped, known      3    436-472ms  1-3ms     4.9-6.0s  5.9-6.9s
+  stopped, new data   2    6.2-8.7s   2ms       26-32s    23-31s
+  running, writing    3    6.0-10.1s  13-89ms   16s       32-57s
+  256 GiB, fragmented 2    246-379ms  3-4ms     33-35s    24-25s
+
+The fragmented image was backed up faster, probably because the main
+loop no longer waits for its 65536 queries.
+
+Without this patch, two of the three backups of the running VM failed
+with a timeout of 'query-pbs-bitmap-info', so its backup time before is
+from one run. Its guest was set to write 4 KiB to a random offset every
+20 ms on average, one write at a time, next to other I/O. The 99th
+percentile latency of these writes went from 1.2-2.3 s to 0.14-0.44 s,
+and the guest completed 15-24 of them per second instead of 3-6.
+
+With an iothread, QMP was not affected before, but the guest's writes
+to that disk were, as the iothread blocked instead. With one run each,
+the 99th percentile latency of the 4 KiB writes went from 443 to 100 ms,
+while the backup took 20 s instead of 12 s.
+
+Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
+---
+ block/rbd.c | 32 ++++++++++++++++++++++++++++++--
+ 1 file changed, 30 insertions(+), 2 deletions(-)
+
+diff --git a/block/rbd.c b/block/rbd.c
+index 2a70b5a983..d3b6107b18 100644
+--- a/block/rbd.c
++++ b/block/rbd.c
+@@ -21,6 +21,7 @@
+ #include "block/block-io.h"
+ #include "block/block_int.h"
+ #include "block/qdict.h"
++#include "block/thread-pool.h"
+ #include "crypto/secret.h"
+ #include "qemu/cutils.h"
+ #include "system/replay.h"
+@@ -120,6 +121,13 @@ typedef struct RBDDiffIterateReq {
+     bool exists;
+ } RBDDiffIterateReq;
+ 
++typedef struct RBDDiffIterateCall {
++    rbd_image_t image;
++    uint64_t offs;
++    uint64_t len;
++    RBDDiffIterateReq *req;
++} RBDDiffIterateCall;
++
+ static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx,
+                             BlockdevOptionsRbd *opts, bool cache,
+                             const char *keypairs, const char *secretid,
+@@ -1530,6 +1538,14 @@ static int qemu_rbd_diff_iterate_cb(uint64_t offs, size_t len,
+     return 0;
+ }
+ 
++static int qemu_rbd_diff_iterate_worker(void *opaque)
++{
++    RBDDiffIterateCall *call = opaque;
++
++    return rbd_diff_iterate2(call->image, NULL, call->offs, call->len, true,
++                             true, qemu_rbd_diff_iterate_cb, call->req);
++}
++
+ static int coroutine_fn qemu_rbd_co_block_status(BlockDriverState *bs,
+                                                  unsigned int mode,
+                                                  int64_t offset, int64_t bytes,
+@@ -1539,6 +1555,7 @@ static int coroutine_fn qemu_rbd_co_block_status(BlockDriverState *bs,
+     BDRVRBDState *s = bs->opaque;
+     int status, r;
+     RBDDiffIterateReq req = { .offs = offset };
++    RBDDiffIterateCall call;
+     uint64_t features, flags;
+     uint64_t head = 0;
+ 
+@@ -1604,8 +1621,19 @@ static int coroutine_fn qemu_rbd_co_block_status(BlockDriverState *bs,
+     bytes += head;
+ #endif
+ 
+-    r = rbd_diff_iterate2(s->image, NULL, req.offs, bytes, true, true,
+-                          qemu_rbd_diff_iterate_cb, &req);
++    /*
++     * rbd_diff_iterate2() blocks until the requests in flight on the image
++     * have completed, which can take a long time while a backup reads from it.
++     * Call it in a worker thread, so that it does not block the AioContext,
++     * which is the main loop for a disk without an iothread.
++     */
++    call = (RBDDiffIterateCall) {
++        .image = s->image,
++        .offs = req.offs,
++        .len = bytes,
++        .req = &req,
++    };
++    r = thread_pool_submit_co(qemu_rbd_diff_iterate_worker, &call);
+     if (r < 0 && r != QEMU_RBD_EXIT_DIFF_ITERATE2) {
+         return status;
+     }
diff --git a/debian/patches/series b/debian/patches/series
index 7b0cba5ff6..bc240a0141 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -4,6 +4,7 @@ extra/0003-block-vmdk-tolerate-known-seSparse-reserved1-bit.patch
 extra/0004-hw-scsi-lsi53c895a-gracefully-handle-re-entrant-DMA.patch
 extra/0005-qapi-virtio-Restore-backend-features-in-x-query-virt.patch
 extra/0006-qapi-monitor-Handle-missing-vhost-user-protocol-feat.patch
+extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch
 bitmap-mirror/0001-drive-mirror-add-support-for-sync-bitmap-mode-never.patch
 bitmap-mirror/0002-drive-mirror-add-support-for-conditional-and-always-.patch
 bitmap-mirror/0003-mirror-add-check-for-bitmap-mode-without-bitmap.patch
-- 
2.47.3





             reply	other threads:[~2026-11-20 11:38 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 11:38 Thomas Lamprecht [this message]
2026-11-20 14:26 ` applied: [PATCH qemu] fix #8343, #8361: avoid QMP timeouts and VM stalls in backups of RBD disks Fiona Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120113834.4069058-1-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal