public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [PATCH proxmox v4 04/11] s3-client: avoid retry bursts by adding jitter to retry backoff delay
Date: Fri, 13 Nov 2026 15:21:41 +0100	[thread overview]
Message-ID: <20261113142148.569471-5-c.ebner@proxmox.com> (raw)
In-Reply-To: <20261113142148.569471-1-c.ebner@proxmox.com>

Requests failing at the same time, are all retried after exactly the
same delay due to the deterministic backoff calculation. This causes
bursts of retries hitting the server at once, which makes it more
likely for them to fail again.

Therefore, randomize the delay uniformly between zero and the
computed backoff, which spreads the retries over the whole backoff
window. If no random value can be generated, wait for the full
backoff rather than not at all.

This follows the AWS SDKs implementation guide.

Link: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 proxmox-s3-client/src/client.rs | 32 +++++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)

diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 2d04ccb7..149b421c 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -580,7 +580,7 @@ impl S3Client {
                 _ => retry_reason,
             };
 
-            tokio::time::sleep(retry_backoff(retry_reason, retry)).await;
+            tokio::time::sleep(retry_backoff_with_jitter(retry_reason, retry)).await;
         }
 
         bail!("failed to send request exceeding retries");
@@ -1095,6 +1095,16 @@ impl RetryReason {
     }
 }
 
+/// Helper to get exponential backoff duration based on retry reason and count with
+/// uniform jitter.
+///
+/// Calculates how long to wait before the next retry attempt for a request.
+/// The duration is limited by an upper bond of [`S3_RETRY_MAX_BACKOFF`], jitter applied
+/// afterwards.
+fn retry_backoff_with_jitter(reason: RetryReason, retry: usize) -> Duration {
+    with_jitter(retry_backoff(reason, retry))
+}
+
 /// Helper to get exponential backoff duration based on retry reason and count.
 ///
 /// Calculates how long to wait before the next retry attempt for a request.
@@ -1109,6 +1119,17 @@ fn retry_backoff(reason: RetryReason, retry: usize) -> Duration {
         })
 }
 
+/// Randomize the given backoff uniformly between zero and its full duration, so clients which
+/// failed at the same time do not retry in lockstep.
+fn with_jitter(backoff: Duration) -> Duration {
+    let mut random = [0u8; 8];
+    match openssl::rand::rand_bytes(&mut random) {
+        Ok(()) => backoff.mul_f64(u64::from_ne_bytes(random) as f64 / u64::MAX as f64),
+        // rather wait too long than retry without any delay
+        Err(_) => backoff,
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -1154,6 +1175,15 @@ mod tests {
         );
     }
 
+    #[test]
+    fn with_jitter_within_backoff() {
+        let backoff = Duration::from_secs(1);
+        let delays: Vec<Duration> = (0..1000).map(|_| with_jitter(backoff)).collect();
+        assert!(delays.iter().all(|delay| *delay <= backoff));
+        assert!(delays.iter().any(|delay| *delay != delays[0]));
+        assert_eq!(with_jitter(Duration::ZERO), Duration::ZERO);
+    }
+
     #[test]
     fn retry_backoff_is_capped() {
         assert_eq!(
-- 
2.47.3





  parent reply	other threads:[~2026-11-13 14:21 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13 14:21 [PATCH proxmox{,-backup} v4 00/11] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 01/11] s3-client: also retry requests on bad gateway responses Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 02/11] s3-client: distinguish transient errors and throttled requests Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 03/11] s3-client: detect throttling request from the response body on error Christian Ebner
2026-11-13 14:21 ` Christian Ebner [this message]
2026-11-13 14:21 ` [PATCH proxmox v4 05/11] s3-client: honor response header directed retry delays Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 06/11] s3-client: add additional backoff delay between upload retries Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox v4 07/11] s3-client: add helper to fetch object contents with retry Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox-backup v4 08/11] s3: retry fetching object contents on body collection errors Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox-backup v4 09/11] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox-backup v4 10/11] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-13 14:21 ` [PATCH proxmox-backup v4 11/11] verify: s3: stop early if the endpoint appears unreachable Christian Ebner
2026-11-16  9:55 ` superseded: [PATCH proxmox{,-backup} v4 00/11] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261113142148.569471-5-c.ebner@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal