public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH qemu] fix #8343, #8361: avoid QMP timeouts and VM stalls in backups of RBD disks
Date: Fri, 20 Nov 2026 12:38:01 +0100	[thread overview]
Message-ID: <20261120113834.4069058-1-t.lamprecht@proxmox.com> (raw)

For an RBD image with fast-diff, a backup queries the block status of
each chunk, and each query waits until the requests in flight on the
image have completed. For a disk without an iothread, QEMU waited for
that in its main loop. Since pve-qemu-kvm 11.1.3-2, backups to a
Proxmox Backup Server process chunks in parallel, which keeps more
reads in flight and so makes these waits longer. The guest could stall
meanwhile, and a backup failed when one of its QMP commands timed out.

Add a patch that runs these queries in a worker thread instead.
Backups of a running VM can take longer with it, as the guest no longer
stalls and its I/O competes with the backup. It does not fix bug #7909,
the slow reads of RBD disks in backups.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 ...uery-block-status-in-a-worker-thread.patch | 135 ++++++++++++++++++
 debian/patches/series                         |   1 +
 2 files changed, 136 insertions(+)
 create mode 100644 debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch

diff --git a/debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch b/debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch
new file mode 100644
index 0000000000..ec8600779d
--- /dev/null
+++ b/debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch
@@ -0,0 +1,135 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: Thomas Lamprecht <t.lamprecht@proxmox.com>
+Date: Fri, 20 Nov 2026 11:46:52 +0100
+Subject: [PATCH] block/rbd: query block status in a worker thread
+
+To answer a block status query for an image with fast-diff, the driver
+calls rbd_diff_iterate2(). That call blocks, and before it reports
+anything, librbd waits until the requests in flight on the image have
+completed. So the query blocks the thread of the image's AioContext.
+For a disk without an iothread, that is the main loop, which holds the
+BQL meanwhile.
+
+A backup queries the block status of each chunk before copying it,
+while up to max-workers reads of other chunks are in flight. Each of
+these queries then blocks the main loop until those reads are done.
+With slow OSDs, QMP commands can time out, and the vCPUs stall while
+they wait for the BQL. In Proxmox VE, this got much more likely once
+backups to a Proxmox Backup Server started to process chunks in
+parallel, which keeps more reads in flight.
+
+Call rbd_diff_iterate2() in a worker thread of the AioContext's thread
+pool, so that the coroutine yields while librbd waits. A query still
+waits as long, but the AioContext can handle other requests and events
+in the meantime. Reading a backup's data does not get faster, as each
+query still waits for the reads in flight before the next read can
+start. If the guest writes during a backup, the backup can even take
+longer, as the guest no longer stalls and its requests compete with the
+backup's.
+
+Test setup: a three-node Ceph cluster with slow OSDs, and a VM whose
+8 GiB disk without iothread holds 3 GiB of data. It is backed up with
+16 workers to a Proxmox Backup Server in three ways: stopped with data
+that the server already has, stopped with new data, and running while
+the guest writes. The last row is a stopped VM with a 256 GiB disk that
+has 8192 alternating allocated and unallocated runs. Before is
+pve-qemu-kvm 11.1.3-3, after is the same with this patch:
+
+                     runs  longest QMP reply    backup time
+                           before     after     before    after
+  stopped, known      3    436-472ms  1-3ms     4.9-6.0s  5.9-6.9s
+  stopped, new data   2    6.2-8.7s   2ms       26-32s    23-31s
+  running, writing    3    6.0-10.1s  13-89ms   16s       32-57s
+  256 GiB, fragmented 2    246-379ms  3-4ms     33-35s    24-25s
+
+The fragmented image was backed up faster, probably because the main
+loop no longer waits for its 65536 queries.
+
+Without this patch, two of the three backups of the running VM failed
+with a timeout of 'query-pbs-bitmap-info', so its backup time before is
+from one run. Its guest was set to write 4 KiB to a random offset every
+20 ms on average, one write at a time, next to other I/O. The 99th
+percentile latency of these writes went from 1.2-2.3 s to 0.14-0.44 s,
+and the guest completed 15-24 of them per second instead of 3-6.
+
+With an iothread, QMP was not affected before, but the guest's writes
+to that disk were, as the iothread blocked instead. With one run each,
+the 99th percentile latency of the 4 KiB writes went from 443 to 100 ms,
+while the backup took 20 s instead of 12 s.
+
+Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
+---
+ block/rbd.c | 32 ++++++++++++++++++++++++++++++--
+ 1 file changed, 30 insertions(+), 2 deletions(-)
+
+diff --git a/block/rbd.c b/block/rbd.c
+index 2a70b5a983..d3b6107b18 100644
+--- a/block/rbd.c
++++ b/block/rbd.c
+@@ -21,6 +21,7 @@
+ #include "block/block-io.h"
+ #include "block/block_int.h"
+ #include "block/qdict.h"
++#include "block/thread-pool.h"
+ #include "crypto/secret.h"
+ #include "qemu/cutils.h"
+ #include "system/replay.h"
+@@ -120,6 +121,13 @@ typedef struct RBDDiffIterateReq {
+     bool exists;
+ } RBDDiffIterateReq;
+ 
++typedef struct RBDDiffIterateCall {
++    rbd_image_t image;
++    uint64_t offs;
++    uint64_t len;
++    RBDDiffIterateReq *req;
++} RBDDiffIterateCall;
++
+ static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx,
+                             BlockdevOptionsRbd *opts, bool cache,
+                             const char *keypairs, const char *secretid,
+@@ -1530,6 +1538,14 @@ static int qemu_rbd_diff_iterate_cb(uint64_t offs, size_t len,
+     return 0;
+ }
+ 
++static int qemu_rbd_diff_iterate_worker(void *opaque)
++{
++    RBDDiffIterateCall *call = opaque;
++
++    return rbd_diff_iterate2(call->image, NULL, call->offs, call->len, true,
++                             true, qemu_rbd_diff_iterate_cb, call->req);
++}
++
+ static int coroutine_fn qemu_rbd_co_block_status(BlockDriverState *bs,
+                                                  unsigned int mode,
+                                                  int64_t offset, int64_t bytes,
+@@ -1539,6 +1555,7 @@ static int coroutine_fn qemu_rbd_co_block_status(BlockDriverState *bs,
+     BDRVRBDState *s = bs->opaque;
+     int status, r;
+     RBDDiffIterateReq req = { .offs = offset };
++    RBDDiffIterateCall call;
+     uint64_t features, flags;
+     uint64_t head = 0;
+ 
+@@ -1604,8 +1621,19 @@ static int coroutine_fn qemu_rbd_co_block_status(BlockDriverState *bs,
+     bytes += head;
+ #endif
+ 
+-    r = rbd_diff_iterate2(s->image, NULL, req.offs, bytes, true, true,
+-                          qemu_rbd_diff_iterate_cb, &req);
++    /*
++     * rbd_diff_iterate2() blocks until the requests in flight on the image
++     * have completed, which can take a long time while a backup reads from it.
++     * Call it in a worker thread, so that it does not block the AioContext,
++     * which is the main loop for a disk without an iothread.
++     */
++    call = (RBDDiffIterateCall) {
++        .image = s->image,
++        .offs = req.offs,
++        .len = bytes,
++        .req = &req,
++    };
++    r = thread_pool_submit_co(qemu_rbd_diff_iterate_worker, &call);
+     if (r < 0 && r != QEMU_RBD_EXIT_DIFF_ITERATE2) {
+         return status;
+     }
diff --git a/debian/patches/series b/debian/patches/series
index 7b0cba5ff6..bc240a0141 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -4,6 +4,7 @@ extra/0003-block-vmdk-tolerate-known-seSparse-reserved1-bit.patch
 extra/0004-hw-scsi-lsi53c895a-gracefully-handle-re-entrant-DMA.patch
 extra/0005-qapi-virtio-Restore-backend-features-in-x-query-virt.patch
 extra/0006-qapi-monitor-Handle-missing-vhost-user-protocol-feat.patch
+extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch
 bitmap-mirror/0001-drive-mirror-add-support-for-sync-bitmap-mode-never.patch
 bitmap-mirror/0002-drive-mirror-add-support-for-conditional-and-always-.patch
 bitmap-mirror/0003-mirror-add-check-for-bitmap-mode-without-bitmap.patch
-- 
2.47.3





             reply	other threads:[~2026-11-20 11:38 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-20 11:38 Thomas Lamprecht [this message]
2026-11-20 14:26 ` applied: [PATCH qemu] fix #8343, #8361: avoid QMP timeouts and VM stalls in backups of RBD disks Fiona Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120113834.4069058-1-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal