Re: [pbs-devel] [RFC pxar 7/20] fix #3174: encoder: add helper to incr encoder pos

[Christian Ebner <c.ebner@proxmox.com>]

> On 27.09.2023 14:07 CEST Wolfgang Bumiller <w.bumiller@proxmox.com> wrote:
> 
>  
> 'incr' :S
> 
> On Fri, Sep 22, 2023 at 09:16:08AM +0200, Christian Ebner wrote:
> > Adds a helper to allow to increase the encoder position by a given
> > size. This is used to increase the position when adding an appendix
> > section to the pxar stream, as these bytes are never encoded directly
> > but rather referenced by already existing chunks.
> 
> Exposing this seems like a weird choice to me. Why exactly is this
> needed? Why don't we instead expose methods to actually write the
> appendix section instead?

This is needed in order to increase the byte offset of the encoder itself.
The appendix section is a list of chunks which are injected in the chunk
stream on upload, but never really consumed by the encoder and subsequently
the chunker itself. So there is no direct writing of the appendix section to
the stream.

By adding the bytes, consistency with the rest of the pxar archive is assured,
as these chunks/bytes are present during decoding.

> 
> > 
> > Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> > ---
> >  src/encoder/aio.rs  | 5 +++++
> >  src/encoder/mod.rs  | 6 ++++++
> >  src/encoder/sync.rs | 5 +++++
> >  3 files changed, 16 insertions(+)
> > 
> > diff --git a/src/encoder/aio.rs b/src/encoder/aio.rs
> > index 3b6c900..3587f65 100644
> > --- a/src/encoder/aio.rs
> > +++ b/src/encoder/aio.rs
> > @@ -112,6 +112,11 @@ impl<'a, T: SeqWrite + 'a> Encoder<'a, T> {
> >          self.inner.finish().await
> >      }
> >  
> > +    /// Add size to encoders position and return new position.
> > +    pub fn position_add(&mut self, size: u64) -> u64 {
> > +        self.inner.position_add(size)
> > +    }
> > +
> >      /// Add reference to archive appendix
> >      pub async fn add_appendix_ref<PF: AsRef<Path>>(
> >          &mut self,
> > diff --git a/src/encoder/mod.rs b/src/encoder/mod.rs
> > index a513ce6..abe21f2 100644
> > --- a/src/encoder/mod.rs
> > +++ b/src/encoder/mod.rs
> > @@ -626,6 +626,12 @@ impl<'a, T: SeqWrite + 'a> EncoderImpl<'a, T> {
> >          self.state.write_position
> >      }
> >  
> > +    #[inline]
> > +    pub fn position_add(&mut self, size: u64) -> u64 {
> > +        self.state.write_position += size;
> > +        self.state.write_position
> > +    }
> > +
> >      pub async fn create_directory(
> >          &mut self,
> >          file_name: &Path,
> > diff --git a/src/encoder/sync.rs b/src/encoder/sync.rs
> > index 372ca12..b3d7c44 100644
> > --- a/src/encoder/sync.rs
> > +++ b/src/encoder/sync.rs
> > @@ -110,6 +110,11 @@ impl<'a, T: SeqWrite + 'a> Encoder<'a, T> {
> >          poll_result_once(self.inner.finish())
> >      }
> >  
> > +    /// Add size to encoders position and return new position.
> > +    pub fn position_add(&mut self, size: u64) -> u64 {
> > +        self.inner.position_add(size)
> > +    }
> > +
> >      /// Add reference to archive appendix
> >      pub async fn add_appendix_ref<PF: AsRef<Path>>(
> >          &mut self,
> > -- 
> > 2.39.2