From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 0EF121FF13A for ; Wed, 27 May 2026 18:12:44 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id E1D531599; Wed, 27 May 2026 18:12:42 +0200 (CEST) Message-ID: <66def9af-7504-441a-b605-6705f76fb047@proxmox.com> Date: Wed, 27 May 2026 18:12:08 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v1 proxmox-backup] api: do not block tokio worker threads during chunk inserts To: Robert Obkircher , pbs-devel@lists.proxmox.com References: <20260527123809.238964-1-r.obkircher@proxmox.com> Content-Language: en-US, de-DE From: Christian Ebner In-Reply-To: <20260527123809.238964-1-r.obkircher@proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1779898302461 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.331 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Message-ID-Hash: NCVZZQVBFV26KTBHERXUFHNUCOVWMJTU X-Message-ID-Hash: NCVZZQVBFV26KTBHERXUFHNUCOVWMJTU X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 5/27/26 2:37 PM, Robert Obkircher wrote: > Move synchronous operations off the worker threads to prevent blocking > the I/O and timer drivers of the entire runtime. This is especially > important for S3 uploads, which wait up to 3 hours for the chunk lock. > > Also prevent worker starvation, which could happen because S3 uploads > are wrapped in proxmox_async::runtime::block_on, which prevents other > futures from running in the current thread. > > In the backtrace from the linked forum post, two workers were waiting > for chunk locks (presumably due to duplicates) while the remaining 19 > were stuck because block_on called std::thread::park. > > Fixes: https://forum.proxmox.com/threads/183705 > Signed-off-by: Robert Obkircher > --- Gave this patch a spin, changes look good to me although a second pair of eyes is highly appreciated given the scope. In order to look for performance regressions, I did also some basic benchmarks: Performed backups of a linux kernel git repo with size 11.488 GiB (compressed 9.96 GiB). For each patch state and target datastore, performed 5 runs removing the backup snapshot again after each run to force re-upload of chunks, but left chunk in store to avoid overhead of insert/upload. In-memory state cleared after each run by restarting proxmox-backup-proxy.service. delta RSS was obtained by taking difference of max and min value of output as reported by: `watch -n 1 "ps -p $(pidof proxmox-backup-proxy) -o rss | tail -n 1 | tee -a ps-rss.out"` Regular datastore and S3 store backed by Ceph RGW using a local datastore cache on virtualized regular datastore - unpatched ----------------------------- runtime (s) | delta RSS (KiB) 25.24 ± 0.09 | 31772 ----------------------------- S3 datastore - unpatched ----------------------------- runtime (s) | delta RSS (KiB) 25.19 ± 0.14 | 28072 ----------------------------- regular datastore - patched ----------------------------- runtime (s) | delta RSS (KiB) 25.28 ± 0.08 | 26184 ----------------------------- S3 datastore - patched ----------------------------- runtime (s) | delta rss (KiB) 25.42 ± 0.15 | 35024 ----------------------------- Consider: Reviewed-by: Christian Ebner Tested-by: Christian Ebner