From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id F29739231D for ; Fri, 5 Apr 2024 13:28:59 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id E454D12776 for ; Fri, 5 Apr 2024 13:28:59 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Fri, 5 Apr 2024 13:28:59 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 0921146466 for ; Fri, 5 Apr 2024 13:28:59 +0200 (CEST) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable In-Reply-To: <20240328123707.336951-38-c.ebner@proxmox.com> References: <20240328123707.336951-1-c.ebner@proxmox.com> <20240328123707.336951-38-c.ebner@proxmox.com> From: Fabian =?utf-8?q?Gr=C3=BCnbichler?= To: Christian Ebner , pbs-devel@lists.proxmox.com Date: Fri, 05 Apr 2024 13:28:50 +0200 Message-ID: <171231653078.2418221.16619231788922937847@yuna.proxmox.com> User-Agent: alot/0.10 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.058 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [create.rs] Subject: Re: [pbs-devel] [PATCH v3 proxmox-backup 37/58] client: pxar: helper for lookup of reusable dynamic entries X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2024 11:29:00 -0000 Quoting Christian Ebner (2024-03-28 13:36:46) > The helper method allows to lookup the entries of a dynamic index > which fully cover a given offset range. Further, the helper returns > the start padding from the start offset of the dynamic index entry > to the start offset of the given range and the end padding. >=20 > This will be used to lookup size and digest for chunks covering the > payload range of a regular file in order to re-use found chunks by > indexing them in the archives index file instead of re-encoding the > payload. >=20 > Signed-off-by: Christian Ebner > --- > changes since version 2: > - moved this from the dynamic index to the pxar create as suggested > - refactored and optimized search, going for linear search to find the > end entry > - reworded commit message >=20 > pbs-client/src/pxar/create.rs | 63 +++++++++++++++++++++++++++++++++++ > 1 file changed, 63 insertions(+) >=20 > diff --git a/pbs-client/src/pxar/create.rs b/pbs-client/src/pxar/create.rs > index 2bb5a6253..e2d3954ca 100644 > --- a/pbs-client/src/pxar/create.rs > +++ b/pbs-client/src/pxar/create.rs > @@ -2,6 +2,7 @@ use std::collections::{HashMap, HashSet}; > use std::ffi::{CStr, CString, OsStr}; > use std::fmt; > use std::io::{self, Read}; > +use std::ops::Range; > use std::os::unix::ffi::OsStrExt; > use std::os::unix::io::{AsRawFd, FromRawFd, IntoRawFd, OwnedFd, RawFd}; > use std::path::{Path, PathBuf}; > @@ -16,6 +17,7 @@ use nix::fcntl::OFlag; > use nix::sys::stat::{FileStat, Mode}; > =20 > use pathpatterns::{MatchEntry, MatchFlag, MatchList, MatchType, PatternF= lag}; > +use pbs_datastore::index::IndexFile; > use proxmox_sys::error::SysError; > use pxar::encoder::{LinkOffset, SeqWrite}; > use pxar::Metadata; > @@ -25,6 +27,7 @@ use proxmox_lang::c_str; > use proxmox_sys::fs::{self, acl, xattr}; > =20 > use pbs_datastore::catalog::BackupCatalogWriter; > +use pbs_datastore::dynamic_index::DynamicIndexReader; > =20 > use crate::pxar::metadata::errno_is_unsupported; > use crate::pxar::tools::assert_single_path_component; > @@ -791,6 +794,66 @@ impl Archiver { > } > } > =20 > +/// Dynamic Entry reusable by payload references > +#[derive(Clone, Debug)] > +#[repr(C)] > +pub struct ReusableDynamicEntry { > + size_le: u64, I don't think the `le` here makes sense, this is never stored on disk.. > + digest: [u8; 32], > +}