From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id CCFE71FF0B6 for ; Thu, 12 Nov 2026 19:02:12 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 3CA62216A3; Thu, 12 Nov 2026 19:02:06 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox v3 04/11] s3-client: avoid retry bursts by adding jitter to retry backoff delay Date: Thu, 12 Nov 2026 19:01:12 +0100 Message-ID: <20261112180119.660923-5-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261112180119.660923-1-c.ebner@proxmox.com> References: <20261112180119.660923-1-c.ebner@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794506501696 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.411 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: PWKHWAXZNDBHDLY57WF6JGF6K5OZ2VMA X-Message-ID-Hash: PWKHWAXZNDBHDLY57WF6JGF6K5OZ2VMA X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Requests failing at the same time, are all retried after exactly the same delay due to the deterministic backoff calculation. This causes bursts of retries hitting the server at once, which makes it more likely for them to fail again. Therefore, randomize the delay uniformly between zero and the computed backoff, which spreads the retries over the whole backoff window. If no random value can be generated, wait for the full backoff rather than not at all. This follows the AWS SDKs implementation guide. Link: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html Signed-off-by: Christian Ebner --- proxmox-s3-client/src/client.rs | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs index c43d0973..bb9132b8 100644 --- a/proxmox-s3-client/src/client.rs +++ b/proxmox-s3-client/src/client.rs @@ -580,7 +580,7 @@ impl S3Client { _ => retry_reason, }; - tokio::time::sleep(retry_backoff(retry_reason, retry)).await; + tokio::time::sleep(retry_backoff_with_jitter(retry_reason, retry)).await; } bail!("failed to send request exceeding retries"); @@ -1095,6 +1095,17 @@ impl RetryReason { } } + +/// Helper to get exponential backoff duration based on retry reason and count with +/// uniform jitter. +/// +/// Calculates how long to wait before the next retry attempt for a request. +/// The duration is limited by an upper bond of [`S3_RETRY_MAX_BACKOFF`], jitter applied +/// afterwards. +fn retry_backoff_with_jitter(reason: RetryReason, retry: usize) -> Duration { + with_jitter(retry_backoff(reason, retry)) +} + /// Helper to get exponential backoff duration based on retry reason and count. /// /// Calculates how long to wait before the next retry attempt for a request. @@ -1109,6 +1120,17 @@ fn retry_backoff(reason: RetryReason, retry: usize) -> Duration { }) } +/// Randomize the given backoff uniformly between zero and its full duration, so clients which +/// failed at the same time do not retry in lockstep. +fn with_jitter(backoff: Duration) -> Duration { + let mut random = [0u8; 8]; + match openssl::rand::rand_bytes(&mut random) { + Ok(()) => backoff.mul_f64(u64::from_ne_bytes(random) as f64 / u64::MAX as f64), + // rather wait too long than retry without any delay + Err(_) => backoff, + } +} + #[cfg(test)] mod tests { use super::*; @@ -1154,6 +1176,15 @@ mod tests { ); } + #[test] + fn with_jitter_within_backoff() { + let backoff = Duration::from_secs(1); + let delays: Vec = (0..1000).map(|_| with_jitter(backoff)).collect(); + assert!(delays.iter().all(|delay| *delay <= backoff)); + assert!(delays.iter().any(|delay| *delay != delays[0])); + assert_eq!(with_jitter(Duration::ZERO), Duration::ZERO); + } + #[test] fn retry_backoff_is_capped() { assert_eq!( -- 2.47.3