From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id D7F4EAA19 for ; Fri, 8 Sep 2023 09:41:41 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id ABFA3E26B for ; Fri, 8 Sep 2023 09:41:11 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Fri, 8 Sep 2023 09:41:10 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id C01314348E for ; Fri, 8 Sep 2023 09:41:09 +0200 (CEST) Message-ID: Date: Fri, 8 Sep 2023 09:41:08 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 To: Thomas Lamprecht , Proxmox Backup Server development discussion References: <20230905093747.54879-1-g.goller@proxmox.com> <482af171-fcb0-4dc6-b662-66d1511fc4a3@proxmox.com> Content-Language: en-US From: Gabriel Goller In-Reply-To: <482af171-fcb0-4dc6-b662-66d1511fc4a3@proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.312 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -1.473 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [datastore.rs] Subject: Re: [pbs-devel] [PATCH proxmox-backup] fix #4823: datastore: ignore vanished files when walking directory X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Sep 2023 07:41:41 -0000 On 9/7/23 17:05, Thomas Lamprecht wrote: > On 05/09/2023 11:37, Gabriel Goller wrote: >> When walking through a datastore on a GC run, it can >> happen that the snapshot is deleted, and then walked over. >> For example: >> - read dir entry for group >> - walk entries (snapshots) >> - snapshot X is removed/pruned >> - walking reaches snapshot X, but ENOENT >> Previously we bailed here, now we just ignore it. >> > looks mostly fine, some style nits and a actual comment inline. > >> Signed-off-by: Gabriel Goller >> --- >> pbs-datastore/src/datastore.rs | 28 +++++++++++++++++----------- >> 1 file changed, 17 insertions(+), 11 deletions(-) >> >> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs >> index fe75d9b5..d135ad90 100644 >> --- a/pbs-datastore/src/datastore.rs >> +++ b/pbs-datastore/src/datastore.rs >> @@ -869,18 +869,24 @@ impl DataStore { >> let handle_entry_err = |err: walkdir::Error| { >> if let Some(inner) = err.io_error() { >> if let Some(path) = err.path() { >> - if inner.kind() == io::ErrorKind::PermissionDenied { >> - // only allow to skip ext4 fsck directory, avoid GC if, for example, >> - // a user got file permissions wrong on datastore rsync to new server >> - if err.depth() > 1 || !path.ends_with("lost+found") { >> - bail!("cannot continue garbage-collection safely, permission denied on: {:?}", path) >> + match inner.kind() { >> + io::ErrorKind::PermissionDenied => { > that's some deep indentation level.. not a must, but maybe you find > some good/simple way to refactor some of this to make it a bit less > crowded here (if, then in a separate patch please) I could check if `err` is an `io:Error`, thus returning early. Then calling `.unwrap()` to get the actual `io::Error` later on. >> + // only allow to skip ext4 fsck directory, avoid GC if, for example, >> + // a user got file permissions wrong on datastore rsync to new server >> + if err.depth() > 1 || !path.ends_with("lost+found") { >> + bail!("cannot continue garbage-collection safely, permission denied on: {:?}", path) >> + } >> + } >> + io::ErrorKind::NotFound => { >> + // ignore vanished file > would be still good to log that here, at least at debug level > if it can be noisy; but as there wasn't many that run into this > in the four years of PBS existing I'd guess a always visible > level is fine as long as the log message doesn't sounds scary. How about a "ignoring/skipping vanished file: {path}" on the info log level? >> + } >> + _ => { >> + bail!( >> + "unexpected error on datastore traversal: {} - {:?}", >> + inner, >> + path > as already mentioned once, please use captured variables directly > in the format strings for new additions or lines that you touch anyway. > E.g.: > > bail!("unexpected error on datastore traversal: {inner} - {path:?}") > >> + ) >> } >> - } else { >> - bail!( >> - "unexpected error on datastore traversal: {} - {:?}", >> - inner, >> - path >> - ) > same here > >> } >> } else { >> bail!("unexpected error on datastore traversal: {}", inner)