From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <c.ebner@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 1B905EB5D
for <pbs-devel@lists.proxmox.com>; Wed, 27 Sep 2023 14:20:51 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id D7382BE57
for <pbs-devel@lists.proxmox.com>; Wed, 27 Sep 2023 14:20:20 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
[94.136.29.106])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pbs-devel@lists.proxmox.com>; Wed, 27 Sep 2023 14:20:19 +0200 (CEST)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 8BE4748DD0
for <pbs-devel@lists.proxmox.com>; Wed, 27 Sep 2023 14:20:19 +0200 (CEST)
Date: Wed, 27 Sep 2023 14:20:18 +0200 (CEST)
From: Christian Ebner <c.ebner@proxmox.com>
To: Wolfgang Bumiller <w.bumiller@proxmox.com>
Cc: pbs-devel@lists.proxmox.com
Message-ID: <526287774.5120.1695817218589@webmail.proxmox.com>
In-Reply-To: <dhmslgqzkgvkwd7eozdfkdk56unn37nmkiizkj3ck45aspnf5z@aawsoehpx6gs>
References: <20230922071621.12670-1-c.ebner@proxmox.com>
<20230922071621.12670-8-c.ebner@proxmox.com>
<dhmslgqzkgvkwd7eozdfkdk56unn37nmkiizkj3ck45aspnf5z@aawsoehpx6gs>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Priority: 3
Importance: Normal
X-Mailer: Open-Xchange Mailer v7.10.6-Rev50
X-Originating-Client: open-xchange-appsuite
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.098 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [aio.rs, sync.rs, mod.rs]
Subject: Re: [pbs-devel] [RFC pxar 7/20] fix #3174: encoder: add helper to
incr encoder pos
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
<pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Wed, 27 Sep 2023 12:20:51 -0000
> On 27.09.2023 14:07 CEST Wolfgang Bumiller <w.bumiller@proxmox.com> wrote:
>
>
> 'incr' :S
>
> On Fri, Sep 22, 2023 at 09:16:08AM +0200, Christian Ebner wrote:
> > Adds a helper to allow to increase the encoder position by a given
> > size. This is used to increase the position when adding an appendix
> > section to the pxar stream, as these bytes are never encoded directly
> > but rather referenced by already existing chunks.
>
> Exposing this seems like a weird choice to me. Why exactly is this
> needed? Why don't we instead expose methods to actually write the
> appendix section instead?
This is needed in order to increase the byte offset of the encoder itself.
The appendix section is a list of chunks which are injected in the chunk
stream on upload, but never really consumed by the encoder and subsequently
the chunker itself. So there is no direct writing of the appendix section to
the stream.
By adding the bytes, consistency with the rest of the pxar archive is assured,
as these chunks/bytes are present during decoding.
>
> >
> > Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> > ---
> > src/encoder/aio.rs | 5 +++++
> > src/encoder/mod.rs | 6 ++++++
> > src/encoder/sync.rs | 5 +++++
> > 3 files changed, 16 insertions(+)
> >
> > diff --git a/src/encoder/aio.rs b/src/encoder/aio.rs
> > index 3b6c900..3587f65 100644
> > --- a/src/encoder/aio.rs
> > +++ b/src/encoder/aio.rs
> > @@ -112,6 +112,11 @@ impl<'a, T: SeqWrite + 'a> Encoder<'a, T> {
> > self.inner.finish().await
> > }
> >
> > + /// Add size to encoders position and return new position.
> > + pub fn position_add(&mut self, size: u64) -> u64 {
> > + self.inner.position_add(size)
> > + }
> > +
> > /// Add reference to archive appendix
> > pub async fn add_appendix_ref<PF: AsRef<Path>>(
> > &mut self,
> > diff --git a/src/encoder/mod.rs b/src/encoder/mod.rs
> > index a513ce6..abe21f2 100644
> > --- a/src/encoder/mod.rs
> > +++ b/src/encoder/mod.rs
> > @@ -626,6 +626,12 @@ impl<'a, T: SeqWrite + 'a> EncoderImpl<'a, T> {
> > self.state.write_position
> > }
> >
> > + #[inline]
> > + pub fn position_add(&mut self, size: u64) -> u64 {
> > + self.state.write_position += size;
> > + self.state.write_position
> > + }
> > +
> > pub async fn create_directory(
> > &mut self,
> > file_name: &Path,
> > diff --git a/src/encoder/sync.rs b/src/encoder/sync.rs
> > index 372ca12..b3d7c44 100644
> > --- a/src/encoder/sync.rs
> > +++ b/src/encoder/sync.rs
> > @@ -110,6 +110,11 @@ impl<'a, T: SeqWrite + 'a> Encoder<'a, T> {
> > poll_result_once(self.inner.finish())
> > }
> >
> > + /// Add size to encoders position and return new position.
> > + pub fn position_add(&mut self, size: u64) -> u64 {
> > + self.inner.position_add(size)
> > + }
> > +
> > /// Add reference to archive appendix
> > pub async fn add_appendix_ref<PF: AsRef<Path>>(
> > &mut self,
> > --
> > 2.39.2