From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [PATCH proxmox v5 04/12] s3-client: avoid retry bursts by adding jitter to retry backoff delay
Date: Mon, 16 Nov 2026 10:54:26 +0100 [thread overview]
Message-ID: <20261116095434.110541-5-c.ebner@proxmox.com> (raw)
In-Reply-To: <20261116095434.110541-1-c.ebner@proxmox.com>
Requests failing at the same time, are all retried after exactly the
same delay due to the deterministic backoff calculation. This causes
bursts of retries hitting the server at once, which makes it more
likely for them to fail again.
Therefore, randomize the delay uniformly between zero and the
computed backoff, which spreads the retries over the whole backoff
window. If no random value can be generated, wait for the full
backoff rather than not at all.
This follows the AWS SDKs implementation guide.
Link: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
proxmox-s3-client/src/client.rs | 32 +++++++++++++++++++++++++++++++-
1 file changed, 31 insertions(+), 1 deletion(-)
diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 2d04ccb7..149b421c 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -580,7 +580,7 @@ impl S3Client {
_ => retry_reason,
};
- tokio::time::sleep(retry_backoff(retry_reason, retry)).await;
+ tokio::time::sleep(retry_backoff_with_jitter(retry_reason, retry)).await;
}
bail!("failed to send request exceeding retries");
@@ -1095,6 +1095,16 @@ impl RetryReason {
}
}
+/// Helper to get exponential backoff duration based on retry reason and count with
+/// uniform jitter.
+///
+/// Calculates how long to wait before the next retry attempt for a request.
+/// The duration is limited by an upper bond of [`S3_RETRY_MAX_BACKOFF`], jitter applied
+/// afterwards.
+fn retry_backoff_with_jitter(reason: RetryReason, retry: usize) -> Duration {
+ with_jitter(retry_backoff(reason, retry))
+}
+
/// Helper to get exponential backoff duration based on retry reason and count.
///
/// Calculates how long to wait before the next retry attempt for a request.
@@ -1109,6 +1119,17 @@ fn retry_backoff(reason: RetryReason, retry: usize) -> Duration {
})
}
+/// Randomize the given backoff uniformly between zero and its full duration, so clients which
+/// failed at the same time do not retry in lockstep.
+fn with_jitter(backoff: Duration) -> Duration {
+ let mut random = [0u8; 8];
+ match openssl::rand::rand_bytes(&mut random) {
+ Ok(()) => backoff.mul_f64(u64::from_ne_bytes(random) as f64 / u64::MAX as f64),
+ // rather wait too long than retry without any delay
+ Err(_) => backoff,
+ }
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -1154,6 +1175,15 @@ mod tests {
);
}
+ #[test]
+ fn with_jitter_within_backoff() {
+ let backoff = Duration::from_secs(1);
+ let delays: Vec<Duration> = (0..1000).map(|_| with_jitter(backoff)).collect();
+ assert!(delays.iter().all(|delay| *delay <= backoff));
+ assert!(delays.iter().any(|delay| *delay != delays[0]));
+ assert_eq!(with_jitter(Duration::ZERO), Duration::ZERO);
+ }
+
#[test]
fn retry_backoff_is_capped() {
assert_eq!(
--
2.47.3
next prev parent reply other threads:[~2026-11-16 9:54 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 9:54 [PATCH proxmox{,-backup} v5 00/12] fix #7606: verify: s3: handle transient fetch errors and add retry Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 01/12] s3-client: also retry requests on bad gateway responses Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 02/12] s3-client: distinguish transient errors and throttled requests Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 03/12] s3-client: detect throttling request from the response body on error Christian Ebner
2026-11-16 9:54 ` Christian Ebner [this message]
2026-11-16 9:54 ` [PATCH proxmox v5 05/12] s3-client: honor response header directed retry delays Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 06/12] s3-client: add additional backoff delay between upload retries Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 07/12] s3-client: add helper to fetch object contents with retry Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox v5 08/12] s3-client: add shared rate limiter based retry quota Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox-backup v5 09/12] s3: retry fetching object contents on body collection errors Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox-backup v5 10/12] datastore: snapshot: propagate errors in manifest update callback Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox-backup v5 11/12] fix #7606: verify: s3: handle transient chunk load errors Christian Ebner
2026-11-16 9:54 ` [PATCH proxmox-backup v5 12/12] verify: s3: stop early if the endpoint appears unreachable Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116095434.110541-5-c.ebner@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.