Re: [pbs-devel] [RFC v2 proxmox-backup 24/36] index: fetch chunk form index by start/end-offset

["Fabian Grünbichler" <f.gruenbichler@proxmox.com>]

s/form/from/

but the commit title seems a bit off anyway - this doesn't fetch any
chunk

On March 5, 2024 10:26 am, Christian Ebner wrote:
> In preparation for injecting reused payload chunks in payload streams
> for regular files with unchanged metaddata.
> 
> Adds a function to get a list of DynamicEntry's from a chunk index by
> given start and end offset, the range of which will be contained
> within these returned chunks.
> 
> In addition to the list of index entries, the padding to the start of
> the requested start offset from the first chunk is returned, as well
> as the end padding following the requested payload to the actual
> chunk end.
> 
> The padding is used for calculation of the payload reference  offset
> written to the metadata archive, required for payload access during
> decoding.
> 
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
> changes since version 1:
> - s/Appendable/Reusable/ incorrect naming leftover from previous approach
> 
>  pbs-datastore/src/dynamic_index.rs | 55 ++++++++++++++++++++++++++++++
>  1 file changed, 55 insertions(+)
> 
> diff --git a/pbs-datastore/src/dynamic_index.rs b/pbs-datastore/src/dynamic_index.rs
> index 71a5082e..85c86b32 100644
> --- a/pbs-datastore/src/dynamic_index.rs
> +++ b/pbs-datastore/src/dynamic_index.rs
> @@ -74,6 +74,26 @@ impl DynamicEntry {
>      }
>  }
>  
> +/// Dynamic Entry reusable by payload references
> +#[derive(Clone, Debug)]
> +#[repr(C)]
> +pub struct ReusableDynamicEntry {
> +    size_le: u64,
> +    digest: [u8; 32],
> +}
> +
> +impl ReusableDynamicEntry {
> +    #[inline]
> +    pub fn size(&self) -> u64 {
> +        u64::from_le(self.size_le)
> +    }
> +
> +    #[inline]
> +    pub fn digest(&self) -> [u8; 32] {
> +        self.digest.clone()
> +    }
> +}
> +
>  pub struct DynamicIndexReader {
>      _file: File,
>      pub size: usize,
> @@ -188,6 +208,41 @@ impl DynamicIndexReader {
>              self.binary_search(middle_idx + 1, middle_end, end_idx, end, offset)
>          }
>      }
> +
> +    /// List of chunk indices containing the data from start_offset to end_offset
> +    pub fn indices(

and tbh, this also seems like a very generic name ;) there's also only a
single call site, so maybe this could live as a helper there? or do we
expect to need this somewhere else in the near future?

> +        &self,
> +        start_offset: u64,
> +        end_offset: u64,
> +    ) -> Result<(Vec<ReusableDynamicEntry>, u64, u64), Error> {
> +        let end_idx = self.index.len() - 1;

this would then be self.index_count() - 1;

> +        let chunk_end = self.chunk_end(end_idx);
> +        let start = self.binary_search(0, 0, end_idx, chunk_end, start_offset)?;
> +        let end = self.binary_search(0, 0, end_idx, chunk_end, end_offset - 1)?;

these three are pub fn

> +
> +        let offset_first = if start == 0 {
> +            0
> +        } else {
> +            self.index[start - 1].end()

this would be self.chunk_end(start - 1)

> +        };
> +
> +        let padding_start = start_offset - offset_first;
> +        let padding_end = self.index[end].end() - end_offset;
> +

also chunk_end

> +        let mut indices = Vec::new();
> +        let mut prev_end = offset_first;
> +        for dynamic_entry in &self.index[start..end + 1] {

also pub via index()

> +            let size = dynamic_entry.end() - prev_end;
> +            let reusable_dynamic_entry = ReusableDynamicEntry {
> +                size_le: size.to_le(),
> +                digest: dynamic_entry.digest.clone(),
> +            };
> +            prev_end += size;
> +            indices.push(reusable_dynamic_entry);
> +        }
> +
> +        Ok((indices, padding_start, padding_end))

the end padding is currently not even used at the call site?

> +    }
>  }
>  
>  impl IndexFile for DynamicIndexReader {
> -- 
> 2.39.2
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
>