From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 6A1ED1FF15C for ; Fri, 14 Nov 2025 23:16:48 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id DDCFA1BA44; Fri, 14 Nov 2025 23:16:47 +0100 (CET) From: Thomas Lamprecht To: pbs-devel@lists.proxmox.com, Christian Ebner Date: Fri, 14 Nov 2025 23:14:59 +0100 Message-ID: <176315850792.218994.15570328140358401661.b4-ty@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20251114131901.441650-1-c.ebner@proxmox.com> References: <20251114131901.441650-1-c.ebner@proxmox.com> MIME-Version: 1.0 X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1763158577833 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.150 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_LOTSOFHASH 0.25 Emails with lots of hash-like gibberish RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] [PATCH proxmox-backup v6 00/21] fix chunk upload/insert, rename corrupt chunks and GC race conditions for s3 backend X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" On Fri, 14 Nov 2025 14:18:40 +0100, Christian Ebner wrote: > These patches fix possible race conditions on datastores with s3 backend for > chunk insert, renaming of corrupt chunks during verification and cleanup during > garbage collection. Further, the patches assure consistency between the chunk > marker file of the local datastore cache, the s3 object store and the in-memory > LRU cache during state changes occurring by one of the above mentioned operations. > > Consistency is achieved by using a per-chunk file locking mechanism. File locks > are stored on the predefined location for datastore file locks, using the same > `.chunks/prefix/digest` folder layout for consistency and to keep readdir and > other fs operations performant. > > [...] Applied, thanks! [01/21] datastore: GC: drop overly verbose info message during s3 chunk sweep commit: d326fc5c055fd2783cff5817aa9c3d1df3bc4092 [02/21] chunk store: implement per-chunk file locking helper for s3 backend commit: adcc39540a3ba070d4627f0785dd113427d8c92c [03/21] datastore: acquire chunk store mutex lock when renaming corrupt chunk commit: e35597690a064be6efe78e5df5c196bc3c9ecd7b [04/21] datastore: get per-chunk file lock for chunk rename on s3 backend commit: 5eabe8b9222a108daf335a6eba7fbd64d9329d6b [05/21] fix #6961: datastore: verify: evict corrupt chunks from in-memory LRU cache commit: 34a395e4f95976f8d96253c6cdb0ccc11ba21604 [06/21] datastore: add locking to protect against races on chunk insert for s3 commit: 59414b6c9c6e5958d0dc5c0356e754095637f8a3 [07/21] GC: fix race with chunk upload/insert on s3 backends commit: 86d5d073037132e6e9aeb01f8e6aaf311dadcd10 [08/21] chunk store: reduce exposure of clear_chunk() to crate only commit: 9bd3f5a65110e6bafe5921cb4a7fe24da726cc98 [09/21] chunk store: make chunk removal a helper method of the chunk store commit: 9312c9de136f3df8001b5b397a51ed69fa3ee818 [10/21] store: split insert_chunk into wrapper + unsafe locked implementation commit: 25db75fb4b23356b28717d2bc88b6826486c851a [11/21] store: cache: move Mutex acquire to cache insertion commit: e4905d1cadfb17d85cd350d9815e4787c888eea8 [12/21] chunk store: rename cache-specific helpers commit: 76ab83194af389d954241eb4d10126e32236b714 [13/21] GC: cleanup chunk markers from cache in phase 3 on s3 backends commit: 77438efecb8ff3d5e51b04650597ba738d3419b6 [14/21] GC: touch bad chunk files independent of backend type commit: f46fd187003eb2da355ff8da19aa71cb8d22878f [15/21] GC: guard missing marker file insertion for s3 backed stores commit: 014f22d51df63d3721fb0a96861bd08978bc8064 [16/21] GC: s3: track if a chunk marker file is missing since a bad chunk commit: 7548f8f84ff60e8dcb2787f296db6bc34554f8d7 [17/21] chunk store: add helpers marking missing local chunk markers as expected commit: abfad2f38cd3bf94c2df5f6b97f8b192417641c0 [18/21] GC: assure chunk exists on s3 store when creating missing chunk marker commit: 9510ef1af395b0e744a938155ff796be9ce67ff6 [19/21] datastore: document s3 backend specific locking restrictions commit: c1b003afd932cca0d35e319d413382268e112470 [20/21] GC: fix: don't drop bad extension for S3 object to chunk path helper commit: 27e3f5b7bb50cc8b419c9b51b560344469629da5 [21/21] GC: clean up bad chunks from the filesystem only commit: 36f62da10d1cb40a19e32b8a0d3d4bcb8ac322bf _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel