From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id B8BFF1FF0B7 for ; Fri, 13 Nov 2026 15:21:56 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 828F02163D; Fri, 13 Nov 2026 15:21:55 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox v4 04/11] s3-client: avoid retry bursts by adding jitter to retry backoff delay Date: Fri, 13 Nov 2026 15:21:41 +0100 Message-ID: <20261113142148.569471-5-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261113142148.569471-1-c.ebner@proxmox.com> References: <20261113142148.569471-1-c.ebner@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794579712549 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.399 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: B6GZV5DTH4POX3INGCVDCJ3GBWIHNM5H X-Message-ID-Hash: B6GZV5DTH4POX3INGCVDCJ3GBWIHNM5H X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Requests failing at the same time, are all retried after exactly the same delay due to the deterministic backoff calculation. This causes bursts of retries hitting the server at once, which makes it more likely for them to fail again. Therefore, randomize the delay uniformly between zero and the computed backoff, which spreads the retries over the whole backoff window. If no random value can be generated, wait for the full backoff rather than not at all. This follows the AWS SDKs implementation guide. Link: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html Signed-off-by: Christian Ebner --- proxmox-s3-client/src/client.rs | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs index 2d04ccb7..149b421c 100644 --- a/proxmox-s3-client/src/client.rs +++ b/proxmox-s3-client/src/client.rs @@ -580,7 +580,7 @@ impl S3Client { _ => retry_reason, }; - tokio::time::sleep(retry_backoff(retry_reason, retry)).await; + tokio::time::sleep(retry_backoff_with_jitter(retry_reason, retry)).await; } bail!("failed to send request exceeding retries"); @@ -1095,6 +1095,16 @@ impl RetryReason { } } +/// Helper to get exponential backoff duration based on retry reason and count with +/// uniform jitter. +/// +/// Calculates how long to wait before the next retry attempt for a request. +/// The duration is limited by an upper bond of [`S3_RETRY_MAX_BACKOFF`], jitter applied +/// afterwards. +fn retry_backoff_with_jitter(reason: RetryReason, retry: usize) -> Duration { + with_jitter(retry_backoff(reason, retry)) +} + /// Helper to get exponential backoff duration based on retry reason and count. /// /// Calculates how long to wait before the next retry attempt for a request. @@ -1109,6 +1119,17 @@ fn retry_backoff(reason: RetryReason, retry: usize) -> Duration { }) } +/// Randomize the given backoff uniformly between zero and its full duration, so clients which +/// failed at the same time do not retry in lockstep. +fn with_jitter(backoff: Duration) -> Duration { + let mut random = [0u8; 8]; + match openssl::rand::rand_bytes(&mut random) { + Ok(()) => backoff.mul_f64(u64::from_ne_bytes(random) as f64 / u64::MAX as f64), + // rather wait too long than retry without any delay + Err(_) => backoff, + } +} + #[cfg(test)] mod tests { use super::*; @@ -1154,6 +1175,15 @@ mod tests { ); } + #[test] + fn with_jitter_within_backoff() { + let backoff = Duration::from_secs(1); + let delays: Vec = (0..1000).map(|_| with_jitter(backoff)).collect(); + assert!(delays.iter().all(|delay| *delay <= backoff)); + assert!(delays.iter().any(|delay| *delay != delays[0])); + assert_eq!(with_jitter(Duration::ZERO), Duration::ZERO); + } + #[test] fn retry_backoff_is_capped() { assert_eq!( -- 2.47.3