From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 1B905EB5D for ; Wed, 27 Sep 2023 14:20:51 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D7382BE57 for ; Wed, 27 Sep 2023 14:20:20 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 27 Sep 2023 14:20:19 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 8BE4748DD0 for ; Wed, 27 Sep 2023 14:20:19 +0200 (CEST) Date: Wed, 27 Sep 2023 14:20:18 +0200 (CEST) From: Christian Ebner To: Wolfgang Bumiller Cc: pbs-devel@lists.proxmox.com Message-ID: <526287774.5120.1695817218589@webmail.proxmox.com> In-Reply-To: References: <20230922071621.12670-1-c.ebner@proxmox.com> <20230922071621.12670-8-c.ebner@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Priority: 3 Importance: Normal X-Mailer: Open-Xchange Mailer v7.10.6-Rev50 X-Originating-Client: open-xchange-appsuite X-SPAM-LEVEL: Spam detection results: 0 AWL 0.098 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [aio.rs, sync.rs, mod.rs] Subject: Re: [pbs-devel] [RFC pxar 7/20] fix #3174: encoder: add helper to incr encoder pos X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Sep 2023 12:20:51 -0000 > On 27.09.2023 14:07 CEST Wolfgang Bumiller wrote: > > > 'incr' :S > > On Fri, Sep 22, 2023 at 09:16:08AM +0200, Christian Ebner wrote: > > Adds a helper to allow to increase the encoder position by a given > > size. This is used to increase the position when adding an appendix > > section to the pxar stream, as these bytes are never encoded directly > > but rather referenced by already existing chunks. > > Exposing this seems like a weird choice to me. Why exactly is this > needed? Why don't we instead expose methods to actually write the > appendix section instead? This is needed in order to increase the byte offset of the encoder itself. The appendix section is a list of chunks which are injected in the chunk stream on upload, but never really consumed by the encoder and subsequently the chunker itself. So there is no direct writing of the appendix section to the stream. By adding the bytes, consistency with the rest of the pxar archive is assured, as these chunks/bytes are present during decoding. > > > > > Signed-off-by: Christian Ebner > > --- > > src/encoder/aio.rs | 5 +++++ > > src/encoder/mod.rs | 6 ++++++ > > src/encoder/sync.rs | 5 +++++ > > 3 files changed, 16 insertions(+) > > > > diff --git a/src/encoder/aio.rs b/src/encoder/aio.rs > > index 3b6c900..3587f65 100644 > > --- a/src/encoder/aio.rs > > +++ b/src/encoder/aio.rs > > @@ -112,6 +112,11 @@ impl<'a, T: SeqWrite + 'a> Encoder<'a, T> { > > self.inner.finish().await > > } > > > > + /// Add size to encoders position and return new position. > > + pub fn position_add(&mut self, size: u64) -> u64 { > > + self.inner.position_add(size) > > + } > > + > > /// Add reference to archive appendix > > pub async fn add_appendix_ref>( > > &mut self, > > diff --git a/src/encoder/mod.rs b/src/encoder/mod.rs > > index a513ce6..abe21f2 100644 > > --- a/src/encoder/mod.rs > > +++ b/src/encoder/mod.rs > > @@ -626,6 +626,12 @@ impl<'a, T: SeqWrite + 'a> EncoderImpl<'a, T> { > > self.state.write_position > > } > > > > + #[inline] > > + pub fn position_add(&mut self, size: u64) -> u64 { > > + self.state.write_position += size; > > + self.state.write_position > > + } > > + > > pub async fn create_directory( > > &mut self, > > file_name: &Path, > > diff --git a/src/encoder/sync.rs b/src/encoder/sync.rs > > index 372ca12..b3d7c44 100644 > > --- a/src/encoder/sync.rs > > +++ b/src/encoder/sync.rs > > @@ -110,6 +110,11 @@ impl<'a, T: SeqWrite + 'a> Encoder<'a, T> { > > poll_result_once(self.inner.finish()) > > } > > > > + /// Add size to encoders position and return new position. > > + pub fn position_add(&mut self, size: u64) -> u64 { > > + self.inner.position_add(size) > > + } > > + > > /// Add reference to archive appendix > > pub async fn add_appendix_ref>( > > &mut self, > > -- > > 2.39.2