From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 5AE471FF0AE for ; Tue, 29 Sep 2026 11:21:43 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 85D49214E0; Tue, 29 Sep 2026 11:21:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=krasovs.ky; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm3; t=1790673282; x=1790759682; bh=MB5evYIdLPfd13LL88qkLTYoS1dk4YJtkLYbPX0fEnI=; b= oTJ3biN4xXXFRNIOXcvFxDhAyx6CvqlT8hjpW67ld9sHZrqMJ6ZB5hDKTE83MkYq +naXkWSlmK2tw6T2u1xaCv35qqiLJUpH4bRpIjQiLJJY7h3Bu7z6MLky/Wezh0dM Pqbu6Gzuaozhw3dU/Oz1h2/8BjpYTyawHSvjyuzPROr4T2bxLG+nw4agXHAWEb6q ukzJ6K4Y+5QRrGEktl6z/AWIvD3zO6TWdYN73w3bUYO5HWqHYEOtNW80V6D+HzH4 uKyC44q8Y5KBeT8hWBe7BhHb7GXc4m0WuuacoLvUsyIpN1fkgGKZ1bGza+C7M6pI Hqzlu409OE+9vpMuJ0H87A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1790673282; x= 1790759682; bh=MB5evYIdLPfd13LL88qkLTYoS1dk4YJtkLYbPX0fEnI=; b=K b4GEgkqNrW617Yv0X8DuN7CLdY7mxDW+Fk6oP/inqUoCi4FwY1N8PxEfkrlMm4sK YiaL0V+IYoRB45mGkJe83V4CTP1+CR0aYzVlfMJ/P5Rxgvik4pl/iNQzjOGvOTjW 4NdCCcyPioSrSlvEF6GCU95JqaSe122Ez4cWBfGqDo2k0Rn+jJ2W+UwrzhVGVXJC HEvgTUoeva8GLr5R4GxKdRxICk2hPZSwp0wf+pgCy8Ro5GL7VdZgEsNsN38R86YK 7++vjog2uGtHOP+e3v8PHFq5Ese+9SbmV0RIe1ARhsqAouHRt6pk5aicIHZ2rZAq Q981tR1i+0SiMf95NJTtw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTEOTJnsZUZMyXUiRjdXrp9S5ojezLANJOaijxIPe79DajKgSBc8e5ZudToPBfFSzV FeZXZu4W2L1L/vFMRzvRC2yYeBGCZzocXGYjckF6dToI1XhfxYm5vouILd0s1g2WC5S6h+ /kbUnkrNFv/85hs5vDvJXPr+NDccEsWBpWCjrjkruxhhiNHgYqwwqWzvMBImm7cGXIPBbf Ry2Jhkh6HakyI1HGDLrtQTqxOtRiNerKNYeOb5Wyv3kUNoDrPZ/HZOjS2SX2IzLPE87O91 YYCGqRNcu3Kn+sT2LC4zi4U22d5O61t44FR6jx218Wh/eKcLU68/EL80Mn21M6WaxWqMPV xa7We5JEVefH3rLLPqn39YKCFBM1+9oFpLmY+VKx524dg4lH61mGtWtnPW4HjURzg8S+yp SVjsv7rX9ncnDkWd5vWMqnkSU/H2U5bJETN6uKjq9OHD2iijEituKQL/YUAwCskVzENnl7 e85s5mPDun4uC2cJ8tB0ySjK7tjmqoJd90OcyT6dltvYh8L1o5su/FqV/15Mzq3SXTrFE6 n3Qq06wIivHc56sWP1DfcYxJ8SZ8+iCJ9cHyyvGIiB36tsHrDDjdGUs/LKbJzt0/7nYw7W IsIVJ8+toW0ZgGlpLPojATYOl9SzRyEH9dbAiPIf6xkKfwiemxGyWcysG9Ng X-ME-Proxy: Feedback-ID: i867e4b57:Fastmail From: Savely Krasovsky To: Christian Ebner Subject: Re: [PATCH proxmox-backup 0/2] fix #7606: verify: s3: handle transient fetch errors and add retry Date: Tue, 29 Sep 2026 11:14:39 +0200 Message-ID: <20260929091439.579314-1-git@krasovs.ky> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260903121557.420018-1-c.ebner@proxmox.com> References: <20260903121557.420018-1-c.ebner@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_CSS_A 0.1 Contains URL's A record listed in the Spamhaus CSS blocklist [109.245.66.74] Message-ID-Hash: OVU4CZ6DMTOH6GGM6Y27FTZHUCLMTULJ X-Message-ID-Hash: OVU4CZ6DMTOH6GGM6Y27FTZHUCLMTULJ X-MailFrom: git@krasovs.ky X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pbs-devel@lists.proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Backup Server development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi Christian, Do you have an update on the status of this series? Is there anything still needed before it can be merged? I'm running PBS 4.2.6-1 with an S3-backed datastore on Backblaze B2. A single backup is over 1 TB, and I'm seeing errors like this on every verification run: can't verify chunk, load failed - error reading a body from connection The latest verification job ran for 48 hours and 26 minutes before finishing with "verification failed". It reported four such failures: three while checking a file backup and one while checking a VM disk. There were no checksum mismatch errors in the logs. With backups this large, rerunning verification takes a long time and requires a lot of S3 reads, only to hit similar errors again. Retrying these transient failures would be very helpful for this workload. Please let me know if any additional logs or details would help. Thanks, Savely