From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id D524B1FF0DF for ; Fri, 28 Aug 2026 12:30:09 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 787DC214DB; Fri, 28 Aug 2026 12:30:09 +0200 (CEST) Message-ID: <3d2c267b-e018-4b8e-83c4-665e57ece282@proxmox.com> Date: Fri, 28 Aug 2026 12:30:03 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH proxmox-backup v2 1/3] server: pull: run blocking file operations on the blocking pool To: Jakob Klocker , pbs-devel@lists.proxmox.com References: <20260821111826.299588-1-j.klocker@proxmox.com> <20260821111826.299588-2-j.klocker@proxmox.com> Content-Language: en-US, de-DE From: Christian Ebner In-Reply-To: <20260821111826.299588-2-j.klocker@proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1787912994055 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.699 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_SHORT 0.001 Use of a URL Shortener for very short URL RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: UCQ3HR5FFL4EAHFIOBJNH4OQK3XYUXNI X-Message-ID-Hash: UCQ3HR5FFL4EAHFIOBJNH4OQK3XYUXNI X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: One comment inline, with that addressed consider: Reviewed-by: Christian Ebner On 8/21/26 1:18 PM, Jakob Klocker wrote: > The atomic renames and the unreferenced-file cleanup are blocking > operations that ran directly on the runtime's worker threads, stalling > other tasks scheduled there for their duration. > > Use the tokio counterpart for the renames and move the cleanup onto the > blocking thread pool. > > Signed-off-by: Jakob Klocker > --- > src/server/pull.rs | 9 +++++---- > 1 file changed, 5 insertions(+), 4 deletions(-) > > diff --git a/src/server/pull.rs b/src/server/pull.rs > index 4eb5bcf11..fd401ac85 100644 > --- a/src/server/pull.rs > +++ b/src/server/pull.rs > @@ -600,7 +600,7 @@ async fn pull_single_archive<'a>( > } else { > tmp_path > }; > - if let Err(err) = std::fs::rename(&source_path, &path) { > + if let Err(err) = tokio::fs::rename(&source_path, &path).await { comment: there is an additional std::fs::remove_file() in the pull code, e.g. removing the temp file just above this (not seen in diff). All of these should be moved to tokio::fs::remove_file(). We need to be careful with performance when using tokio::fs over std::fs as stated in [0], but these should be fine to call on the blocking task pool. That this might have performance implications might be also noted in the commit message. [0] https://docs.rs/tokio/latest/tokio/fs/index.html > bail!("{archive_prefix}: Atomic rename file {path:?} failed - {err}"); > } > > @@ -890,7 +890,7 @@ async fn pull_snapshot<'a>( > nix::unistd::fsync(tmp_manifest_file.as_raw_fd())?; > } > > - if let Err(err) = std::fs::rename(&tmp_manifest_name, &manifest_name) { > + if let Err(err) = tokio::fs::rename(&tmp_manifest_name, &manifest_name).await { > bail!("{prefix}: Atomic rename file {manifest_name:?} failed - {err}"); > } > if let DatastoreBackend::S3(s3_client) = backend { > @@ -910,8 +910,9 @@ async fn pull_snapshot<'a>( > > fetch_log(crypt_config).await?; > > - snapshot > - .cleanup_unreferenced_files(&manifest) > + let snapshot = snapshot.clone(); > + tokio::task::spawn_blocking(move || snapshot.cleanup_unreferenced_files(&manifest)) > + .await? > .map_err(|err| format_err!("{prefix}: failed to cleanup unreferenced files - {err}"))?; > > Ok(Some(sync_stats))