From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 76BC0920C6 for ; Fri, 5 Apr 2024 10:09:00 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5091DF307 for ; Fri, 5 Apr 2024 10:08:30 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Fri, 5 Apr 2024 10:08:29 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 76BE245880 for ; Fri, 5 Apr 2024 10:08:29 +0200 (CEST) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable In-Reply-To: <20240328123707.336951-46-c.ebner@proxmox.com> References: <20240328123707.336951-1-c.ebner@proxmox.com> <20240328123707.336951-46-c.ebner@proxmox.com> From: Fabian =?utf-8?q?Gr=C3=BCnbichler?= To: Christian Ebner , pbs-devel@lists.proxmox.com Date: Fri, 05 Apr 2024 10:08:22 +0200 Message-ID: <171230450235.1926770.8602698179855647404@yuna.proxmox.com> User-Agent: alot/0.10 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.058 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] [PATCH v3 proxmox-backup 45/58] client: pxar: add method for metadata comparison X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2024 08:09:00 -0000 Quoting Christian Ebner (2024-03-28 13:36:54) > Adds a method to compare the metadata of the current file entry > against the metadata of the entry looked up in the previous backup > snapshot. >=20 > If the metadata matched, the start offset for the payload stream is > returned. >=20 > This is in preparation for reusing payload chunks for unchanged files. >=20 > Signed-off-by: Christian Ebner > --- > changes since version 2: > - refactored to new padding based threshold >=20 > pbs-client/src/pxar/create.rs | 31 ++++++++++++++++++++++++++++++- > 1 file changed, 30 insertions(+), 1 deletion(-) >=20 > diff --git a/pbs-client/src/pxar/create.rs b/pbs-client/src/pxar/create.rs > index 79925bba2..c64084a74 100644 > --- a/pbs-client/src/pxar/create.rs > +++ b/pbs-client/src/pxar/create.rs > @@ -21,7 +21,7 @@ use pbs_datastore::index::IndexFile; > use proxmox_sys::error::SysError; > use pxar::accessor::aio::{Accessor, Directory}; > use pxar::encoder::{LinkOffset, PayloadOffset, SeqWrite}; > -use pxar::Metadata; > +use pxar::{EntryKind, Metadata}; > =20 > use proxmox_io::vec; > use proxmox_lang::c_str; > @@ -466,6 +466,35 @@ impl Archiver { > .boxed() > } > =20 > + async fn is_reusable_entry( > + &mut self, > + previous_metadata_accessor: &mut Directory>, > + file_name: &Path, > + stat: &FileStat, > + metadata: &Metadata, > + ) -> Result, Error> { > + if stat.st_nlink > 1 { > + log::debug!("re-encode: {file_name:?} has hardlinks."); > + return Ok(None); > + } it would be nice if we had a way to handle those as well.. what's the curre= nt blocker? shouldn't we be able to use the same scheme as for regular archive= s? first encounter adds (possibly re-uses) the payload and remembers the offse= t, subsequent ones just add another reference/meta entry? > + > + if let Some(file_entry) =3D previous_metadata_accessor.lookup(fi= le_name).await? { > + if metadata =3D=3D file_entry.metadata() { > + if let EntryKind::File { payload_offset, .. } =3D file_e= ntry.entry().kind() { > + log::debug!("possible re-use: {file_name:?} at offse= t {payload_offset:?} has unchanged metadata."); > + return Ok(*payload_offset); > + } > + log::debug!("re-encode: {file_name:?} not a regular file= ."); > + return Ok(None); > + } > + log::debug!("re-encode: {file_name:?} metadata did not match= ."); > + return Ok(None); > + } > + > + log::debug!("re-encode: {file_name:?} not found in previous arch= ive."); > + Ok(None) > + } > + > /// openat() wrapper which allows but logs `EACCES` and turns `ENOEN= T` into `None`. > /// > /// The `existed` flag is set when iterating through a directory to = note that we know the file > --=20 > 2.39.2 >=20 >=20 >=20 > _______________________________________________ > pbs-devel mailing list > pbs-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel >=20 >