From: Christian Ebner <c.ebner@proxmox.com>
To: "Proxmox Backup Server development discussion"
<pbs-devel@lists.proxmox.com>,
"Fabian Grünbichler" <f.gruenbichler@proxmox.com>
Subject: Re: [pbs-devel] [PATCH v3 pxar 13/58] format: add pxar format version entry
Date: Wed, 3 Apr 2024 15:31:59 +0200 [thread overview]
Message-ID: <23fbf7c2-bcd8-44a3-8dda-902404607171@proxmox.com> (raw)
In-Reply-To: <1712143636.hs6xlwzexi.astroid@yuna.none>
On 4/3/24 13:41, Fabian Grünbichler wrote:
>> @@ -258,7 +261,16 @@ impl<I: SeqRead> DecoderImpl<I> {
>> loop {
>> match self.state {
>> State::Eof => return Ok(None),
>> - State::Begin => return self.read_next_entry().await.map(Some),
>> + State::Begin => {
>> + let entry = self.read_next_entry().await.map(Some);
>> + if let Ok(Some(ref entry)) = entry {
>> + if let EntryKind::Version(version) = entry.kind() {
>> + self.version = version.clone();
>> + return self.read_next_entry().await.map(Some);
>> + }
>> + }
>> + return entry;
>
> a bit unsure here, if we want to enforce the order, wouldn't it be more
> clean to transition to a new state here rather than adding more nested
> ifs over time? ;)
Okay, agreed, will try to de-clutter this a bit by adding a Prelude
state as suggested in your other reply.
>> + 1u64 => Ok(format::FormatVersion::Version1),
>
> this should never happen though, right?
No, right... Will remove it.
>
>> + 2u64 => Ok(format::FormatVersion::Version2),
>
> also this (cted below)
>
>> + _ => io_bail!("unexpected pxar format version"),
>
> this should maybe include the value? ;)
Okay, will add that.
>
>> + }
>> + }
>> }
>>
>> /// Reader for file contents inside a pxar archive.
>> diff --git a/src/encoder/mod.rs b/src/encoder/mod.rs
>> index 88c0ed5..9270153 100644
>> --- a/src/encoder/mod.rs
>> +++ b/src/encoder/mod.rs
>> @@ -17,7 +17,7 @@ use endian_trait::Endian;
>>
>> use crate::binary_tree_array;
>> use crate::decoder::{self, SeqRead};
>> -use crate::format::{self, GoodbyeItem, PayloadRef};
>> +use crate::format::{self, FormatVersion, GoodbyeItem, PayloadRef};
>> use crate::Metadata;
>>
>> pub mod aio;
>> @@ -307,6 +307,8 @@ pub(crate) struct EncoderImpl<'a, T: SeqWrite + 'a> {
>> /// Since only the "current" entry can be actively writing files, we share the file copy
>> /// buffer.
>> file_copy_buffer: Arc<Mutex<Vec<u8>>>,
>> + /// Pxar format version to encode
>> + version: format::FormatVersion,
>> }
>>
>> impl<'a, T: SeqWrite + 'a> EncoderImpl<'a, T> {
>> @@ -320,11 +322,14 @@ impl<'a, T: SeqWrite + 'a> EncoderImpl<'a, T> {
>> }
>>
>> let mut state = EncoderState::default();
>> - if let Some(payload_output) = payload_output.as_mut() {
>> + let version = if let Some(payload_output) = payload_output.as_mut() {
>> let header = format::Header::with_content_size(format::PXAR_PAYLOAD_START_MARKER, 0);
>> header.check_header_size()?;
>> seq_write_struct(payload_output, header, &mut state.payload_write_position).await?;
>> - }
>> + format::FormatVersion::Version2
>> + } else {
>> + format::FormatVersion::default()
>
> shouldn't this be Version1 instead of default()? they are the same
> *now*, but that might not be the case forever?
Okay, will set this to the new version.
>
>> + };
>>
>> let mut this = Self {
>> output,
>> @@ -334,8 +339,10 @@ impl<'a, T: SeqWrite + 'a> EncoderImpl<'a, T> {
>> file_copy_buffer: Arc::new(Mutex::new(unsafe {
>> crate::util::vec_new_uninitialized(1024 * 1024)
>> })),
>> + version,
>> };
>>
>> + this.encode_format_version().await?;
>> this.encode_metadata(metadata).await?;
>> let state = this.state_mut()?;
>> state.files_offset = state.position();
>> @@ -522,6 +529,10 @@ impl<'a, T: SeqWrite + 'a> EncoderImpl<'a, T> {
>> file_size: u64,
>> payload_offset: PayloadOffset,
>> ) -> io::Result<()> {
>> + if self.version == FormatVersion::Version1 {
>> + io_bail!("payload references not supported pxar format version 1");
>> + }
>> +
>> if self.payload_output.as_mut().is_none() {
>> io_bail!("unable to add payload reference");
>> }
>> @@ -729,6 +740,26 @@ impl<'a, T: SeqWrite + 'a> EncoderImpl<'a, T> {
>> Ok(())
>> }
>>
>> + async fn encode_format_version(&mut self) -> io::Result<()> {
>> + let version_bytes = match self.version {
>> + format::FormatVersion::Version1 => return Ok(()),
>> + format::FormatVersion::Version2 => 2u64.to_le_bytes(),
>
> (cted from above) and this here should maybe go together?
Not sure I understand, above is in the decoder, this in the encoder...
What was your intention? Move the encoding/decoding to the FormatVersion
type?
>
>> + };
>> +
>> + let (output, state) = self.output_state()?;
>> + if state.write_position != 0 {
>> + io_bail!("pxar format version must be encoded at the beginning of an archive");
>
> should this also be enforced while decoding?
Okay, will see to add a check for that as well.
>
> should we also encode a/the version of the payload archive?
The payload archive has the PAYLOAD_START_MARKER, which can be changed
just like the pxar entries v2?
>
>> + }
>> +
>> + seq_write_pxar_entry(
>> + output,
>> + format::PXAR_FORMAT_VERSION,
>> + &version_bytes,
>> + &mut state.write_position,
>> + )
>> + .await
>> + }
>> +
>> async fn encode_metadata(&mut self, metadata: &Metadata) -> io::Result<()> {
>> let (output, state) = self.output_state()?;
>> seq_write_pxar_struct_entry(
>> diff --git a/src/format/mod.rs b/src/format/mod.rs
>> index a672d19..2bf33c9 100644
>> --- a/src/format/mod.rs
>> +++ b/src/format/mod.rs
>> @@ -6,6 +6,7 @@
>> //! item data.
>> //!
>> //! An archive contains items in the following order:
>> +//! * `FORMAT_VERSION` -- (optional for v1), version of encoding format
>> //! * `ENTRY` -- containing general stat() data and related bits
>> //! * `XATTR` -- one extended attribute
>> //! * ... -- more of these when there are multiple defined
>> @@ -80,6 +81,8 @@ pub mod mode {
>> }
>>
>> // Generated by `cargo run --example mk-format-hashes`
>> +/// Pxar format version entry, fallback to version 1 if not present
>> +pub const PXAR_FORMAT_VERSION: u64 = 0x730f6c75df16a40d;
>> /// Beginning of an entry (current version).
>> pub const PXAR_ENTRY: u64 = 0xd5956474e588acef;
>> /// Previous version of the entry struct
>> @@ -186,6 +189,7 @@ impl Header {
>> impl Display for Header {
>> fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
>> let readable = match self.htype {
>> + PXAR_FORMAT_VERSION => "FORMAT_VERSION",
>> PXAR_FILENAME => "FILENAME",
>> PXAR_SYMLINK => "SYMLINK",
>> PXAR_HARDLINK => "HARDLINK",
>> @@ -551,6 +555,13 @@ impl From<&std::fs::Metadata> for Stat {
>> }
>> }
>>
>> +#[derive(Clone, Debug, Default, PartialEq)]
>> +pub enum FormatVersion {
>> + #[default]
>> + Version1,
>> + Version2,
>> +}
>> +
>> #[derive(Clone, Debug)]
>> pub struct Filename {
>> pub name: Vec<u8>,
>> diff --git a/src/lib.rs b/src/lib.rs
>> index ef81a85..a87b5ac 100644
>> --- a/src/lib.rs
>> +++ b/src/lib.rs
>> @@ -342,6 +342,9 @@ impl Acl {
>> /// Identifies whether the entry is a file, symlink, directory, etc.
>> #[derive(Clone, Debug)]
>> pub enum EntryKind {
>> + /// Pxar file format version
>> + Version(format::FormatVersion),
>> +
>> /// Symbolic links.
>> Symlink(format::Symlink),
>>
>> --
>> 2.39.2
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
next prev parent reply other threads:[~2024-04-03 13:32 UTC|newest]
Thread overview: 122+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-28 12:36 [pbs-devel] [PATCH v3 pxar proxmox-backup 00/58] fix #3174: improve file-level backup Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 01/58] encoder: fix two typos in comments Christian Ebner
2024-04-03 9:12 ` [pbs-devel] applied: " Fabian Grünbichler
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 02/58] format/examples: add PXAR_PAYLOAD_REF entry header Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 03/58] decoder: add method to read payload references Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 04/58] decoder: factor out skip part from skip_entry Christian Ebner
2024-04-03 9:18 ` Fabian Grünbichler
2024-04-03 11:02 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 05/58] encoder: add optional output writer for file payloads Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 06/58] encoder: move to stack based state tracking Christian Ebner
2024-04-03 9:54 ` Fabian Grünbichler
2024-04-03 11:01 ` Christian Ebner
2024-04-04 8:48 ` Fabian Grünbichler
2024-04-04 9:04 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 07/58] decoder/accessor: add optional payload input stream Christian Ebner
2024-04-03 10:38 ` Fabian Grünbichler
2024-04-03 11:47 ` Christian Ebner
2024-04-03 12:18 ` Christian Ebner
2024-04-04 8:46 ` Fabian Grünbichler
2024-04-04 9:49 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 08/58] encoder: add payload reference capability Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 09/58] encoder: add payload position capability Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 10/58] encoder: add payload advance capability Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 11/58] encoder/format: finish payload stream with marker Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 12/58] format: add payload stream start marker Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 13/58] format: add pxar format version entry Christian Ebner
2024-04-03 11:41 ` Fabian Grünbichler
2024-04-03 13:31 ` Christian Ebner [this message]
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 pxar 14/58] format/encoder/decoder: add entry type cli params Christian Ebner
2024-04-03 12:01 ` Fabian Grünbichler
2024-04-03 14:41 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 15/58] client: pxar: switch to stack based encoder state Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 16/58] client: backup writer: only borrow http client Christian Ebner
2024-04-08 9:04 ` [pbs-devel] applied: " Fabian Grünbichler
2024-04-08 9:17 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 17/58] client: backup: factor out extension from backup target Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 18/58] client: backup: early check for fixed index type Christian Ebner
2024-04-08 9:05 ` [pbs-devel] applied: " Fabian Grünbichler
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 19/58] client: pxar: combine writer params into struct Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 20/58] client: backup: split payload to dedicated stream Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 21/58] client: helper: add helpers for creating reader instances Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 22/58] client: helper: add method for split archive name mapping Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 23/58] client: restore: read payload from dedicated index Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 24/58] tools: cover meta extension for pxar archives Christian Ebner
2024-04-04 9:01 ` Fabian Grünbichler
2024-04-04 9:06 ` Christian Ebner
2024-04-04 9:10 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 25/58] restore: " Christian Ebner
2024-04-04 9:02 ` Fabian Grünbichler
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 26/58] client: mount: make split pxar archives mountable Christian Ebner
2024-04-04 9:43 ` Fabian Grünbichler
2024-04-04 13:29 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 27/58] api: datastore: refactor getting local chunk reader Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 28/58] api: datastore: attach optional payload " Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 29/58] catalog: shell: factor out pxar fuse reader instantiation Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 30/58] catalog: shell: redirect payload reader for split streams Christian Ebner
2024-04-04 9:49 ` Fabian Grünbichler
2024-04-04 15:52 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 31/58] www: cover meta extension for pxar archives Christian Ebner
2024-04-04 10:01 ` Fabian Grünbichler
2024-04-04 14:51 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 32/58] pxar: add optional payload input for achive restore Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 33/58] pxar: add more context to extraction error Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 34/58] client: pxar: include payload offset in output Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 35/58] pxar: show padding in debug output on archive list Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 36/58] datastore: dynamic index: add method to get digest Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 37/58] client: pxar: helper for lookup of reusable dynamic entries Christian Ebner
2024-04-04 12:54 ` Fabian Grünbichler
2024-04-04 17:13 ` Christian Ebner
2024-04-05 7:22 ` Christian Ebner
2024-04-05 11:28 ` Fabian Grünbichler
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 38/58] upload stream: impl reused chunk injector Christian Ebner
2024-04-04 14:24 ` Fabian Grünbichler
2024-04-05 10:26 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 39/58] client: chunk stream: add struct to hold injection state Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 40/58] client: chunk stream: add dynamic entries injection queues Christian Ebner
2024-04-04 14:52 ` Fabian Grünbichler
2024-04-08 13:54 ` Christian Ebner
2024-04-09 7:19 ` Fabian Grünbichler
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 41/58] specs: add backup detection mode specification Christian Ebner
2024-04-04 14:54 ` Fabian Grünbichler
2024-04-08 13:36 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 42/58] client: implement prepare reference method Christian Ebner
2024-04-05 8:01 ` Fabian Grünbichler
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 43/58] client: pxar: implement store to insert chunks on caching Christian Ebner
2024-04-05 7:52 ` Fabian Grünbichler
2024-04-09 9:12 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 44/58] client: pxar: add previous reference to archiver Christian Ebner
2024-04-04 15:04 ` Fabian Grünbichler
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 45/58] client: pxar: add method for metadata comparison Christian Ebner
2024-04-05 8:08 ` Fabian Grünbichler
2024-04-05 8:14 ` Christian Ebner
2024-04-09 12:52 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 46/58] pxar: caching: add look-ahead cache types Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 47/58] client: pxar: add look-ahead caching Christian Ebner
2024-04-05 8:33 ` Fabian Grünbichler
2024-04-09 14:53 ` Christian Ebner
[not found] ` <<dce38c53-f3e7-47ac-b1fd-a63daaabbcec@proxmox.com>
2024-04-10 7:03 ` Fabian Grünbichler
2024-04-10 7:11 ` Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 48/58] fix #3174: client: pxar: enable caching and meta comparison Christian Ebner
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 49/58] client: backup: increase average chunk size for metadata Christian Ebner
2024-04-05 9:42 ` Fabian Grünbichler
2024-04-05 10:49 ` Dietmar Maurer
2024-04-08 8:28 ` Fabian Grünbichler
2024-03-28 12:36 ` [pbs-devel] [PATCH v3 proxmox-backup 50/58] client: backup writer: add injected chunk count to stats Christian Ebner
2024-03-28 12:37 ` [pbs-devel] [PATCH v3 proxmox-backup 51/58] pxar: create: show chunk injection stats debug output Christian Ebner
2024-04-05 9:47 ` Fabian Grünbichler
2024-04-10 10:00 ` Christian Ebner
2024-03-28 12:37 ` [pbs-devel] [PATCH v3 proxmox-backup 52/58] client: pxar: add entry kind format version Christian Ebner
2024-03-28 12:37 ` [pbs-devel] [PATCH v3 proxmox-backup 53/58] client: pxar: opt encode cli exclude patterns as CliParams Christian Ebner
2024-04-05 9:49 ` Fabian Grünbichler
2024-03-28 12:37 ` [pbs-devel] [PATCH v3 proxmox-backup 54/58] client: pxar: add flow chart for metadata change detection Christian Ebner
2024-04-05 10:16 ` Fabian Grünbichler
2024-04-10 10:04 ` Christian Ebner
2024-03-28 12:37 ` [pbs-devel] [PATCH v3 proxmox-backup 55/58] docs: describe file format for split payload files Christian Ebner
2024-04-05 10:26 ` Fabian Grünbichler
2024-03-28 12:37 ` [pbs-devel] [PATCH v3 proxmox-backup 56/58] docs: add section describing change detection mode Christian Ebner
2024-04-05 11:22 ` Fabian Grünbichler
2024-03-28 12:37 ` [pbs-devel] [PATCH v3 proxmox-backup 57/58] test-suite: add detection mode change benchmark Christian Ebner
2024-03-28 12:37 ` [pbs-devel] [PATCH v3 proxmox-backup 58/58] test-suite: add bin to deb, add shell completions Christian Ebner
2024-04-05 11:39 ` [pbs-devel] [PATCH v3 pxar proxmox-backup 00/58] fix #3174: improve file-level backup Fabian Grünbichler
2024-04-29 12:13 ` Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=23fbf7c2-bcd8-44a3-8dda-902404607171@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=f.gruenbichler@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox