From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 47B1D1FF139 for ; Tue, 24 Feb 2026 14:49:38 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5F15CBD24; Tue, 24 Feb 2026 14:50:34 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox v2 3/3] fix #6858: s3-client: retry request on 500, 503 and 504 status codes Date: Tue, 24 Feb 2026 14:49:44 +0100 Message-ID: <20260224134944.593408-4-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260224134944.593408-1-c.ebner@proxmox.com> References: <20260224134944.593408-1-c.ebner@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1771940984350 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.051 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: ZR5FUB7YV33L7NA2XONF4B5N2UJP2MRK X-Message-ID-Hash: ZR5FUB7YV33L7NA2XONF4B5N2UJP2MRK X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Follow the best practices for AWS S3 error handling [0] and perform retries on requests with http status code 500 or 503 in the response. Further, do the same for 504 gateway timeout errors encountered by some users in the community forum [1] in combination with Hetzner's S3 storage offerings. This is done for all requests unconditionally, maximum number of retires and optional request timeout being honored. [0] https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorBestPractices.html [1] https://forum.proxmox.com/threads/180956/ Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=6858 Signed-off-by: Christian Ebner --- changes since version 1: - return the last error if retries are exhausted - consider also 504 gateway timeout as retryable proxmox-s3-client/src/client.rs | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs index f3e5eb45..35c80948 100644 --- a/proxmox-s3-client/src/client.rs +++ b/proxmox-s3-client/src/client.rs @@ -388,13 +388,18 @@ impl S3Client { self.client.request(request).await }; - match response { - Ok(response) => return Ok(response), - Err(err) => { - if retry >= MAX_S3_HTTP_REQUEST_RETRY - 1 { - return Err(err.into()); - } - } + let do_retry = match &response { + Ok(response) => matches!( + response.status(), + StatusCode::INTERNAL_SERVER_ERROR + | StatusCode::SERVICE_UNAVAILABLE + | StatusCode::GATEWAY_TIMEOUT + ), + Err(_) => true, + }; + + if !do_retry || retry >= MAX_S3_HTTP_REQUEST_RETRY - 1 { + return Ok(response?); } let backoff_secs = S3_HTTP_REQUEST_RETRY_BACKOFF_DEFAULT * 3_u32.pow(retry as u32); -- 2.47.3