From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 519321FF139 for ; Tue, 24 Feb 2026 14:49:07 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 65160BB3A; Tue, 24 Feb 2026 14:50:03 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Subject: [PATCH proxmox v2 0/3] fix #6858: implement retry logic for transient API errors Date: Tue, 24 Feb 2026 14:49:41 +0100 Message-ID: <20260224134944.593408-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1771940983626 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.051 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: OZ4X2KKERCNWKLZJEMX2HADT5GWRD75P X-Message-ID-Hash: OZ4X2KKERCNWKLZJEMX2HADT5GWRD75P X-MailFrom: c.ebner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: These patches implement the best practice [0] on handling S3 API response status codes 500, 503 by retrying the requests after an exponential backoff time. Do the same for status code 504, as this is returned by some storage providers if overwhelmed [1]. The first 2 patches contain a small fix to avoid additional response latency in case of request timeout being reached and reorganize the code for better logical flow. The final patch then adds the additional response status code checks for retires. Link to the issue in bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6858 [0] https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorBestPractices.html [1] https://forum.proxmox.com/threads/180956/ Changes since version 1 (thanks @Fabian for review): - return the last error if retries are exhausted - consider also 504 gateway timeout as retryable Christian Ebner (3): s3-client: early return when request timeout deadline reached s3-client: move exponential backoff to after the response state check fix #6858: s3-client: retry request on 500, 503 and 504 status codes proxmox-s3-client/src/client.rs | 38 ++++++++++++++++----------------- 1 file changed, 19 insertions(+), 19 deletions(-) -- 2.47.3