From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 5514EB8E6D for ; Tue, 12 Mar 2024 09:50:48 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 37A4B14164 for ; Tue, 12 Mar 2024 09:50:48 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 12 Mar 2024 09:50:47 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 5851A489B0 for ; Tue, 12 Mar 2024 09:50:47 +0100 (CET) Date: Tue, 12 Mar 2024 09:50:40 +0100 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= To: Proxmox Backup Server development discussion References: <20240305092703.126906-1-c.ebner@proxmox.com> <20240305092703.126906-25-c.ebner@proxmox.com> In-Reply-To: <20240305092703.126906-25-c.ebner@proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.16.0 (https://github.com/astroidmail/astroid) Message-Id: <1710232817.oiegytfp1w.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL 0.065 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pbs-devel] [RFC v2 proxmox-backup 24/36] index: fetch chunk form index by start/end-offset X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Mar 2024 08:50:48 -0000 s/form/from/ but the commit title seems a bit off anyway - this doesn't fetch any chunk On March 5, 2024 10:26 am, Christian Ebner wrote: > In preparation for injecting reused payload chunks in payload streams > for regular files with unchanged metaddata. >=20 > Adds a function to get a list of DynamicEntry's from a chunk index by > given start and end offset, the range of which will be contained > within these returned chunks. >=20 > In addition to the list of index entries, the padding to the start of > the requested start offset from the first chunk is returned, as well > as the end padding following the requested payload to the actual > chunk end. >=20 > The padding is used for calculation of the payload reference offset > written to the metadata archive, required for payload access during > decoding. >=20 > Signed-off-by: Christian Ebner > --- > changes since version 1: > - s/Appendable/Reusable/ incorrect naming leftover from previous approach >=20 > pbs-datastore/src/dynamic_index.rs | 55 ++++++++++++++++++++++++++++++ > 1 file changed, 55 insertions(+) >=20 > diff --git a/pbs-datastore/src/dynamic_index.rs b/pbs-datastore/src/dynam= ic_index.rs > index 71a5082e..85c86b32 100644 > --- a/pbs-datastore/src/dynamic_index.rs > +++ b/pbs-datastore/src/dynamic_index.rs > @@ -74,6 +74,26 @@ impl DynamicEntry { > } > } > =20 > +/// Dynamic Entry reusable by payload references > +#[derive(Clone, Debug)] > +#[repr(C)] > +pub struct ReusableDynamicEntry { > + size_le: u64, > + digest: [u8; 32], > +} > + > +impl ReusableDynamicEntry { > + #[inline] > + pub fn size(&self) -> u64 { > + u64::from_le(self.size_le) > + } > + > + #[inline] > + pub fn digest(&self) -> [u8; 32] { > + self.digest.clone() > + } > +} > + > pub struct DynamicIndexReader { > _file: File, > pub size: usize, > @@ -188,6 +208,41 @@ impl DynamicIndexReader { > self.binary_search(middle_idx + 1, middle_end, end_idx, end,= offset) > } > } > + > + /// List of chunk indices containing the data from start_offset to e= nd_offset > + pub fn indices( and tbh, this also seems like a very generic name ;) there's also only a single call site, so maybe this could live as a helper there? or do we expect to need this somewhere else in the near future? > + &self, > + start_offset: u64, > + end_offset: u64, > + ) -> Result<(Vec, u64, u64), Error> { > + let end_idx =3D self.index.len() - 1; this would then be self.index_count() - 1; > + let chunk_end =3D self.chunk_end(end_idx); > + let start =3D self.binary_search(0, 0, end_idx, chunk_end, start= _offset)?; > + let end =3D self.binary_search(0, 0, end_idx, chunk_end, end_off= set - 1)?; these three are pub fn > + > + let offset_first =3D if start =3D=3D 0 { > + 0 > + } else { > + self.index[start - 1].end() this would be self.chunk_end(start - 1) > + }; > + > + let padding_start =3D start_offset - offset_first; > + let padding_end =3D self.index[end].end() - end_offset; > + also chunk_end > + let mut indices =3D Vec::new(); > + let mut prev_end =3D offset_first; > + for dynamic_entry in &self.index[start..end + 1] { also pub via index() > + let size =3D dynamic_entry.end() - prev_end; > + let reusable_dynamic_entry =3D ReusableDynamicEntry { > + size_le: size.to_le(), > + digest: dynamic_entry.digest.clone(), > + }; > + prev_end +=3D size; > + indices.push(reusable_dynamic_entry); > + } > + > + Ok((indices, padding_start, padding_end)) the end padding is currently not even used at the call site? > + } > } > =20 > impl IndexFile for DynamicIndexReader { > --=20 > 2.39.2 >=20 >=20 >=20 > _______________________________________________ > pbs-devel mailing list > pbs-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel >=20 >=20 >=20