From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 01D701FF0AF for ; Fri, 20 Nov 2026 12:38:48 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id EA5A0216FD; Fri, 20 Nov 2026 12:38:45 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH qemu] fix #8343, #8361: avoid QMP timeouts and VM stalls in backups of RBD disks Date: Fri, 20 Nov 2026 12:38:01 +0100 Message-ID: <20261120113834.4069058-1-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795174721256 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.148 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: KK5FIB6V7F2JBLIVKLRF6UX2AC3OL53H X-Message-ID-Hash: KK5FIB6V7F2JBLIVKLRF6UX2AC3OL53H X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: For an RBD image with fast-diff, a backup queries the block status of each chunk, and each query waits until the requests in flight on the image have completed. For a disk without an iothread, QEMU waited for that in its main loop. Since pve-qemu-kvm 11.1.3-2, backups to a Proxmox Backup Server process chunks in parallel, which keeps more reads in flight and so makes these waits longer. The guest could stall meanwhile, and a backup failed when one of its QMP commands timed out. Add a patch that runs these queries in a worker thread instead. Backups of a running VM can take longer with it, as the guest no longer stalls and its I/O competes with the backup. It does not fix bug #7909, the slow reads of RBD disks in backups. Signed-off-by: Thomas Lamprecht --- ...uery-block-status-in-a-worker-thread.patch | 135 ++++++++++++++++++ debian/patches/series | 1 + 2 files changed, 136 insertions(+) create mode 100644 debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch diff --git a/debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch b/debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch new file mode 100644 index 0000000000..ec8600779d --- /dev/null +++ b/debian/patches/extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch @@ -0,0 +1,135 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: Thomas Lamprecht +Date: Fri, 20 Nov 2026 11:46:52 +0100 +Subject: [PATCH] block/rbd: query block status in a worker thread + +To answer a block status query for an image with fast-diff, the driver +calls rbd_diff_iterate2(). That call blocks, and before it reports +anything, librbd waits until the requests in flight on the image have +completed. So the query blocks the thread of the image's AioContext. +For a disk without an iothread, that is the main loop, which holds the +BQL meanwhile. + +A backup queries the block status of each chunk before copying it, +while up to max-workers reads of other chunks are in flight. Each of +these queries then blocks the main loop until those reads are done. +With slow OSDs, QMP commands can time out, and the vCPUs stall while +they wait for the BQL. In Proxmox VE, this got much more likely once +backups to a Proxmox Backup Server started to process chunks in +parallel, which keeps more reads in flight. + +Call rbd_diff_iterate2() in a worker thread of the AioContext's thread +pool, so that the coroutine yields while librbd waits. A query still +waits as long, but the AioContext can handle other requests and events +in the meantime. Reading a backup's data does not get faster, as each +query still waits for the reads in flight before the next read can +start. If the guest writes during a backup, the backup can even take +longer, as the guest no longer stalls and its requests compete with the +backup's. + +Test setup: a three-node Ceph cluster with slow OSDs, and a VM whose +8 GiB disk without iothread holds 3 GiB of data. It is backed up with +16 workers to a Proxmox Backup Server in three ways: stopped with data +that the server already has, stopped with new data, and running while +the guest writes. The last row is a stopped VM with a 256 GiB disk that +has 8192 alternating allocated and unallocated runs. Before is +pve-qemu-kvm 11.1.3-3, after is the same with this patch: + + runs longest QMP reply backup time + before after before after + stopped, known 3 436-472ms 1-3ms 4.9-6.0s 5.9-6.9s + stopped, new data 2 6.2-8.7s 2ms 26-32s 23-31s + running, writing 3 6.0-10.1s 13-89ms 16s 32-57s + 256 GiB, fragmented 2 246-379ms 3-4ms 33-35s 24-25s + +The fragmented image was backed up faster, probably because the main +loop no longer waits for its 65536 queries. + +Without this patch, two of the three backups of the running VM failed +with a timeout of 'query-pbs-bitmap-info', so its backup time before is +from one run. Its guest was set to write 4 KiB to a random offset every +20 ms on average, one write at a time, next to other I/O. The 99th +percentile latency of these writes went from 1.2-2.3 s to 0.14-0.44 s, +and the guest completed 15-24 of them per second instead of 3-6. + +With an iothread, QMP was not affected before, but the guest's writes +to that disk were, as the iothread blocked instead. With one run each, +the 99th percentile latency of the 4 KiB writes went from 443 to 100 ms, +while the backup took 20 s instead of 12 s. + +Signed-off-by: Thomas Lamprecht +--- + block/rbd.c | 32 ++++++++++++++++++++++++++++++-- + 1 file changed, 30 insertions(+), 2 deletions(-) + +diff --git a/block/rbd.c b/block/rbd.c +index 2a70b5a983..d3b6107b18 100644 +--- a/block/rbd.c ++++ b/block/rbd.c +@@ -21,6 +21,7 @@ + #include "block/block-io.h" + #include "block/block_int.h" + #include "block/qdict.h" ++#include "block/thread-pool.h" + #include "crypto/secret.h" + #include "qemu/cutils.h" + #include "system/replay.h" +@@ -120,6 +121,13 @@ typedef struct RBDDiffIterateReq { + bool exists; + } RBDDiffIterateReq; + ++typedef struct RBDDiffIterateCall { ++ rbd_image_t image; ++ uint64_t offs; ++ uint64_t len; ++ RBDDiffIterateReq *req; ++} RBDDiffIterateCall; ++ + static int qemu_rbd_connect(rados_t *cluster, rados_ioctx_t *io_ctx, + BlockdevOptionsRbd *opts, bool cache, + const char *keypairs, const char *secretid, +@@ -1530,6 +1538,14 @@ static int qemu_rbd_diff_iterate_cb(uint64_t offs, size_t len, + return 0; + } + ++static int qemu_rbd_diff_iterate_worker(void *opaque) ++{ ++ RBDDiffIterateCall *call = opaque; ++ ++ return rbd_diff_iterate2(call->image, NULL, call->offs, call->len, true, ++ true, qemu_rbd_diff_iterate_cb, call->req); ++} ++ + static int coroutine_fn qemu_rbd_co_block_status(BlockDriverState *bs, + unsigned int mode, + int64_t offset, int64_t bytes, +@@ -1539,6 +1555,7 @@ static int coroutine_fn qemu_rbd_co_block_status(BlockDriverState *bs, + BDRVRBDState *s = bs->opaque; + int status, r; + RBDDiffIterateReq req = { .offs = offset }; ++ RBDDiffIterateCall call; + uint64_t features, flags; + uint64_t head = 0; + +@@ -1604,8 +1621,19 @@ static int coroutine_fn qemu_rbd_co_block_status(BlockDriverState *bs, + bytes += head; + #endif + +- r = rbd_diff_iterate2(s->image, NULL, req.offs, bytes, true, true, +- qemu_rbd_diff_iterate_cb, &req); ++ /* ++ * rbd_diff_iterate2() blocks until the requests in flight on the image ++ * have completed, which can take a long time while a backup reads from it. ++ * Call it in a worker thread, so that it does not block the AioContext, ++ * which is the main loop for a disk without an iothread. ++ */ ++ call = (RBDDiffIterateCall) { ++ .image = s->image, ++ .offs = req.offs, ++ .len = bytes, ++ .req = &req, ++ }; ++ r = thread_pool_submit_co(qemu_rbd_diff_iterate_worker, &call); + if (r < 0 && r != QEMU_RBD_EXIT_DIFF_ITERATE2) { + return status; + } diff --git a/debian/patches/series b/debian/patches/series index 7b0cba5ff6..bc240a0141 100644 --- a/debian/patches/series +++ b/debian/patches/series @@ -4,6 +4,7 @@ extra/0003-block-vmdk-tolerate-known-seSparse-reserved1-bit.patch extra/0004-hw-scsi-lsi53c895a-gracefully-handle-re-entrant-DMA.patch extra/0005-qapi-virtio-Restore-backend-features-in-x-query-virt.patch extra/0006-qapi-monitor-Handle-missing-vhost-user-protocol-feat.patch +extra/0007-block-rbd-query-block-status-in-a-worker-thread.patch bitmap-mirror/0001-drive-mirror-add-support-for-sync-bitmap-mode-never.patch bitmap-mirror/0002-drive-mirror-add-support-for-conditional-and-always-.patch bitmap-mirror/0003-mirror-add-check-for-bitmap-mode-without-bitmap.patch -- 2.47.3