From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH qemu] fix #8343, #8361: avoid QMP timeouts and VM stalls in backups of RBD disks
Date: Fri, 20 Nov 2026 12:38:01 +0100 [thread overview]
Message-ID: <20261120113834.4069058-1-t.lamprecht@proxmox.com> (raw)
For an RBD image with fast-diff, a backup queries the block status of
each chunk, and each query waits until the requests in flight on the
image have completed. For a disk without an iothread, QEMU waited for
that in its main loop. Since pve-qemu-kvm 11.1.3-2, backups to a
Proxmox Backup Server process chunks in parallel, which keeps more
reads in flight and so makes these waits longer. The guest could stall
meanwhile, and a backup failed when one of its QMP commands timed out.
Add a patch that runs these queries in a worker thread instead.
Backups of a running VM can take longer with it, as the guest no longer
stalls and its I/O competes with the backup. It does not fix bug #7909,
the slow reads of RBD disks in backups.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
...uery-block-status-in-a-worker-thread.patch | 135 ++++++++++++++++++
debian/patches/series | 1 +
2 files changed, 136 insertions(+)
create mode 100644 debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch
diff --git a/debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch b/debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch
new file mode 100644
index 0000000000..ec8600779d
--- /dev/null
+++ b/debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch
@@ -0,0 +1,135 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: Thomas Lamprecht <t.lamprecht@proxmox.com>
+Date: Fri, 20 Nov 2026 11:46:52 +0100
+Subject: [PATCH] block/rbd: query block status in a worker thread
+
+To answer a block status query for an image with fast-diff, the driver
+calls rbd_diff_iterate2(). That call blocks, and before it reports
+anything, librbd waits until the requests in flight on the image have
+completed. So the query blocks the thread of the image's AioContext.
+For a disk without an iothread, that is the main loop, which holds the
+BQL meanwhile.
+
+A backup queries the block status of each chunk before copying it,
+while up to max-workers reads of other chunks are in flight. Each of
+these queries then blocks the main loop until those reads are done.
+With slow OSDs, QMP commands can time out, and the vCPUs stall while
+they wait for the BQL. In Proxmox VE, this got much more likely once
+backups to a Proxmox Backup Server started to process chunks in
+parallel, which keeps more reads in flight.
+
+Call rbd_diff_iterate2() in a worker thread of the AioContext's thread
+pool, so that the coroutine yields while librbd waits. A query still
+waits as long, but the AioContext can handle other requests and events
+in the meantime. Reading a backup's data does not get faster, as each
+query still waits for the reads in flight before the next read can
+start. If the guest writes during a backup, the backup can even take
+longer, as the guest no longer stalls and its requests compete with the
+backup's.
+
+Test setup: a three-node Ceph cluster with slow OSDs, and a VM whose
+8 GiB disk without iothread holds 3 GiB of data. It is backed up with
+16 workers to a Proxmox Backup Server in three ways: stopped with data
+that the server already has, stopped with new data, and running while
+the guest writes. The last row is a stopped VM with a 256 GiB disk that
+has 8192 alternating allocated and unallocated runs. Before is
+pve-qemu-kvm 11.1.3-3, after is the same with this patch:
+
+ runs longest QMP reply backup time
+ before after before after
+ stopped, known 3 436-472ms 1-3ms 4.9-6.0s 5.9-6.9s
+ stopped, new data 2 6.2-8.7s 2ms 26-32s 23-31s
+ running, writing 3 6.0-10.1s 13-89ms 16s 32-57s
+ 256 GiB, fragmented 2 246-379ms 3-4ms 33-35s 24-25s
+
+The fragmented image was backed up faster, probably because the main
+loop no longer waits for its 65536 queries.
+
+Without this patch, two of the three backups of the running VM failed
+with a timeout of 'query-pbs-bitmap-info', so its backup time before is
+from one run. Its guest was set to write 4 KiB to a random offset every
+20 ms on average, one write at a time, next to other I/O. The 99th
+percentile latency of these writes went from 1.2-2.3 s to 0.14-0.44 s,
+and the guest completed 15-24 of them per second instead of 3-6.
+
+With an iothread, QMP was not affected before, but the guest's writes
+to that disk were, as the iothread blocked instead. With one run each,
+the 99th percentile latency of the 4 KiB writes went from 443 to 100 ms,
+while the backup took 20 s instead of 12 s.
+
+Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
+---
+ block/rbd.c | 32 ++++++++++++++++++++++++++++++--
+ 1 file changed, 30 insertions(+), 2 deletions(-)
+
+diff --git a/block/rbd.c b/block/rbd.c
+index 2a70b5a983..d3b6107b18 100644
+--- a/block/rbd.c
++++ b/block/rbd.c
+@@ -21,6 +21,7 @@
+ #include "block/block-io.h"
+ #include "block/block_int.h"
+ #include "block/qdict.h"
++#include "block/thread-pool.h"
+ #include "crypto/secret.h"
+ #include "qemu/cutils.h"
+ #include "system/replay.h"
+@@ -120,6 +121,13 @@ typedef struct RBDDiffIterateReq {
+ bool exists;
+ } RBDDiffIterateReq;
+
++typedef struct RBDDiffIterateCall {
++ rbd_image_t image;
++ uint64_t offs;
++ uint64_t len;
++ RBDDiffIterateReq *req;
++} RBDDiffIterateCall;
++
+ static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx,
+ BlockdevOptionsRbd *opts, bool cache,
+ const char *keypairs, const char *secretid,
+@@ -1530,6 +1538,14 @@ static int qemu_rbd_diff_iterate_cb(uint64_t offs, size_t len,
+ return 0;
+ }
+
++static int qemu_rbd_diff_iterate_worker(void *opaque)
++{
++ RBDDiffIterateCall *call = opaque;
++
++ return rbd_diff_iterate2(call->image, NULL, call->offs, call->len, true,
++ true, qemu_rbd_diff_iterate_cb, call->req);
++}
++
+ static int coroutine_fn qemu_rbd_co_block_status(BlockDriverState *bs,
+ unsigned int mode,
+ int64_t offset, int64_t bytes,
+@@ -1539,6 +1555,7 @@ static int coroutine_fn qemu_rbd_co_block_status(BlockDriverState *bs,
+ BDRVRBDState *s = bs->opaque;
+ int status, r;
+ RBDDiffIterateReq req = { .offs = offset };
++ RBDDiffIterateCall call;
+ uint64_t features, flags;
+ uint64_t head = 0;
+
+@@ -1604,8 +1621,19 @@ static int coroutine_fn qemu_rbd_co_block_status(BlockDriverState *bs,
+ bytes += head;
+ #endif
+
+- r = rbd_diff_iterate2(s->image, NULL, req.offs, bytes, true, true,
+- qemu_rbd_diff_iterate_cb, &req);
++ /*
++ * rbd_diff_iterate2() blocks until the requests in flight on the image
++ * have completed, which can take a long time while a backup reads from it.
++ * Call it in a worker thread, so that it does not block the AioContext,
++ * which is the main loop for a disk without an iothread.
++ */
++ call = (RBDDiffIterateCall) {
++ .image = s->image,
++ .offs = req.offs,
++ .len = bytes,
++ .req = &req,
++ };
++ r = thread_pool_submit_co(qemu_rbd_diff_iterate_worker, &call);
+ if (r < 0 && r != QEMU_RBD_EXIT_DIFF_ITERATE2) {
+ return status;
+ }
diff --git a/debian/patches/series b/debian/patches/series
index 7b0cba5ff6..bc240a0141 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -4,6 +4,7 @@ extra/0003-block-vmdk-tolerate-known-seSparse-reserved1-bit.patch
extra/0004-hw-scsi-lsi53c895a-gracefully-handle-re-entrant-DMA.patch
extra/0005-qapi-virtio-Restore-backend-features-in-x-query-virt.patch
extra/0006-qapi-monitor-Handle-missing-vhost-user-protocol-feat.patch
+extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch
bitmap-mirror/0001-drive-mirror-add-support-for-sync-bitmap-mode-never.patch
bitmap-mirror/0002-drive-mirror-add-support-for-conditional-and-always-.patch
bitmap-mirror/0003-mirror-add-check-for-bitmap-mode-without-bitmap.patch
--
2.47.3
next reply other threads:[~2026-11-20 11:38 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 11:38 Thomas Lamprecht [this message]
2026-11-20 14:26 ` applied: [PATCH qemu] fix #8343, #8361: avoid QMP timeouts and VM stalls in backups of RBD disks Fiona Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120113834.4069058-1-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox