all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [PATCH v5 pxar 3/28] fix #3174: encoder: calc filename + metadata byte size
Date: Wed, 15 Nov 2023 16:47:48 +0100	[thread overview]
Message-ID: <20231115154813.281564-4-c.ebner@proxmox.com> (raw)
In-Reply-To: <20231115154813.281564-1-c.ebner@proxmox.com>

Introduce SeqSink and impl SeqWrite in order to create an encoder
implementation which instead of writing data to a stream, consumes
the encoded stream and returns the consumed bytes for that stream.

Based on this, implement a helper function `byte_len` which returns the
byte size of the filename entry and metadata entry as encoded by the
archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
Changes since v4:
- Refactor and rename `bytes_len` to `encoded_size`

Changes since v3:
- no changes

Changes since v2:
- no changes

Changes since v1:
- Instead of calculating the metadata size based on the known encoding
  sizes, implement an Encoder instance which counts the encoded bytes.

 src/encoder/mod.rs  | 35 +++++++++++++++++++++++++++++++++++
 src/encoder/sync.rs |  5 +++++
 2 files changed, 40 insertions(+)

diff --git a/src/encoder/mod.rs b/src/encoder/mod.rs
index 0d342ec..860c21f 100644
--- a/src/encoder/mod.rs
+++ b/src/encoder/mod.rs
@@ -85,6 +85,24 @@ where
     }
 }
 
+#[derive(Default)]
+/// Sink to consume sequential byte stream
+pub struct SeqSink;
+
+impl SeqWrite for SeqSink {
+    fn poll_seq_write(
+        self: Pin<&mut Self>,
+        _cx: &mut Context,
+        buf: &[u8],
+    ) -> Poll<io::Result<usize>> {
+        Poll::Ready(Ok(buf.len()))
+    }
+
+    fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context) -> Poll<io::Result<()>> {
+        Poll::Ready(Ok(()))
+    }
+}
+
 /// awaitable verison of `poll_seq_write`.
 async fn seq_write<T: SeqWrite + ?Sized>(
     output: &mut T,
@@ -183,6 +201,23 @@ where
     seq_write_pxar_entry(output, htype, buf, position).await
 }
 
+/// Calculate the encoded byte len of filename and metadata struct
+pub async fn encoded_size(filename: &std::ffi::CStr, metadata: &Metadata) -> io::Result<u64> {
+    let mut this = EncoderImpl {
+        output: EncoderOutput::Owned(SeqSink::default()),
+        state: EncoderState::default(),
+        parent: None,
+        finished: false,
+        file_copy_buffer: Arc::new(Mutex::new(unsafe {
+            crate::util::vec_new_uninitialized(1024 * 1024)
+        })),
+    };
+
+    this.start_file_do(Some(metadata), filename.to_bytes())
+        .await?;
+    Ok(this.position())
+}
+
 /// Error conditions caused by wrong usage of this crate.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
 pub enum EncodeError {
diff --git a/src/encoder/sync.rs b/src/encoder/sync.rs
index 1ec91b8..93c3b2c 100644
--- a/src/encoder/sync.rs
+++ b/src/encoder/sync.rs
@@ -228,3 +228,8 @@ impl<T: io::Write> SeqWrite for StandardWriter<T> {
         Poll::Ready(self.pin_to_inner().and_then(|inner| inner.flush()))
     }
 }
+
+/// Calculate the encoded byte len of filename and metadata struct
+pub fn encoded_size(filename: &std::ffi::CStr, metadata: &Metadata) -> io::Result<u64> {
+    poll_result_once(crate::encoder::encoded_size(filename, metadata))
+}
-- 
2.39.2





  parent reply	other threads:[~2023-11-15 15:48 UTC|newest]

Thread overview: 29+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-11-15 15:47 [pbs-devel] [PATCH-SERIES v5 pxar proxmox-backup proxmox-widget-toolkit 00/28] fix #3174: improve file-level backup Christian Ebner
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 pxar 1/28] fix #3174: decoder: factor out skip_bytes from skip_entry Christian Ebner
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 pxar 2/28] fix #3174: decoder: impl skip_bytes for sync dec Christian Ebner
2023-11-15 15:47 ` Christian Ebner [this message]
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 pxar 4/28] fix #3174: enc/dec: impl PXAR_APPENDIX_REF entrytype Christian Ebner
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 pxar 5/28] fix #3174: enc/dec: impl PXAR_APPENDIX entrytype Christian Ebner
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 pxar 6/28] fix #3174: encoder: helper to add to encoder position Christian Ebner
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 pxar 7/28] fix #3174: enc/dec: impl PXAR_APPENDIX_TAIL entrytype Christian Ebner
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 pxar 8/28] fix #3174: enc/dec: introduce pxar format version 2 Christian Ebner
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 proxmox-backup 09/28] fix #3174: index: add fn index list from start/end-offsets Christian Ebner
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 proxmox-backup 10/28] fix #3174: index: add fn digest for DynamicEntry Christian Ebner
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 proxmox-backup 11/28] fix #3174: api: double catalog upload size Christian Ebner
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 proxmox-backup 12/28] fix #3174: catalog: introduce extended format v2 Christian Ebner
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 proxmox-backup 13/28] fix #3174: archiver/extractor: impl appendix ref Christian Ebner
2023-11-15 15:47 ` [pbs-devel] [PATCH v5 proxmox-backup 14/28] fix #3174: catalog: add specialized Archive entry Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 15/28] fix #3174: extractor: impl seq restore from appendix Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 16/28] fix #3174: archiver: store ref to previous backup Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 17/28] fix #3174: upload stream: impl reused chunk injector Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 18/28] fix #3174: chunker: add forced boundaries Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 19/28] fix #3174: backup writer: inject queued chunk in upload steam Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 20/28] fix #3174: archiver: reuse files with unchanged metadata Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 21/28] fix #3174: specs: add backup detection mode specification Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 22/28] fix #3174: client: Add detection mode to backup creation Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 23/28] test-suite: add detection mode change benchmark Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 24/28] test-suite: Add bin to deb, add shell completions Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 25/28] catalog: fetch offset and size for files and refs Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 26/28] pxar: add heuristic to reduce reused chunk fragmentation Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-backup 27/28] catalog: use format version 2 conditionally Christian Ebner
2023-11-15 15:48 ` [pbs-devel] [PATCH v5 proxmox-widget-toolkit 28/28] file-browser: support pxar archive and fileref types Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20231115154813.281564-4-c.ebner@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal