[pbs-devel] [PATCH pxar/backup 0/6] bump timestamps to 96 bit

[pbs-devel] [PATCH pxar/backup 0/6] bump timestamps to 96 bit

[Wolfgang Bumiller]

So apparently modification time values of *before* Jan 1 1970 are a
thing, so here's support for that...

This bumps the `Entry` struct in pxar (meaning an API bump), and still
supports reading old archives.

Note that I've introduced a new `StatxTimestamp` struct. I chose this
name as the `statx(2)` manpage's `struct statx_timestamp` is the only
struct which actually documents the fact that only the seconds are
signed, and the nanoseconds are *positive* and *relative* to the
seconds, iow. a timestamp of "-3.5 seconds" is represented as "-4
seconds, plus 500_000_000 nanoseconds". (The only other time I found
this to be explicitly mentioned is in the `chrono` crate's
`TimeZone::timestamp` method which explicitly creates a "DateTime from
the number of non-leap seconds since (...) 1970 (...) and the number of
nanoseconds since the last whole non-leap second.".

Wolfgang Bumiller (6):
 pxar:
  add format description to format module
  introduce StatxTimestamp helper type
  update mk-format-hashes for a new ENTRY
  implement Entry v2
  add entry v1 compatiblity test
  bump version to 0.3.0-1

 Cargo.toml                   |   2 +-
 debian/changelog             |   8 ++
 examples/mk-format-hashes.rs |  11 ++-
 src/decoder/mod.rs           |  21 +++-
 src/errors.rs                |  25 -----
 src/format/mod.rs            | 179 ++++++++++++++++++++++++++++++++---
 src/lib.rs                   |  28 +++---
 tests/compat.rs              | 136 ++++++++++++++++++++++++++
 8 files changed, 353 insertions(+), 57 deletions(-)
 delete mode 100644 src/errors.rs
 create mode 100644 tests/compat.rs

 backup:
  update to pxar 0.3 to support negative timestamps

-- 
2.20.1

[pbs-devel] [PATCH pxar 1/6] add format description to format module

[Wolfgang Bumiller]

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
 src/format/mod.rs | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/src/format/mod.rs b/src/format/mod.rs
index 2a7d377..0f9db79 100644
--- a/src/format/mod.rs
+++ b/src/format/mod.rs
@@ -4,6 +4,36 @@
 //!
 //! The Archive contains a list of items. Each item starts with a `Header`, followed by the
 //! item data.
+//!
+//! An archive contains items in the following order:
+//!  * `ENTRY`              -- containing general stat() data and related bits
+//!   * `XATTR`             -- one extended attribute
+//!   * ...                 -- more of these when there are multiple defined
+//!   * `ACL_USER`          -- one `USER ACL` entry
+//!   * ...                 -- more of these when there are multiple defined
+//!   * `ACL_GROUP`         -- one `GROUP ACL` entry
+//!   * ...                 -- more of these when there are multiple defined
+//!   * `ACL_GROUP_OBJ`     -- The `ACL_GROUP_OBJ`
+//!   * `ACL_DEFAULT`       -- The various default ACL fields if there's one defined
+//!   * `ACL_DEFAULT_USER`  -- one USER ACL entry
+//!   * ...                 -- more of these when multiple are defined
+//!   * `ACL_DEFAULT_GROUP` -- one GROUP ACL entry
+//!   * ...                 -- more of these when multiple are defined
+//!   * `FCAPS`             -- file capability in Linux disk format
+//!   * `QUOTA_PROJECT_ID`  -- the ext4/xfs quota project ID
+//!   * `PAYLOAD`           -- file contents, if it is one
+//!   * `SYMLINK`           -- symlink target, if it is one
+//!   * `DEVICE`            -- device major/minor, if it is a block/char device
+//!
+//!   If we are serializing a directory, then this is followed by:
+//!
+//!   * `FILENAME`          -- name of the first directory entry (strictly ordered!)
+//!   * `<archive>`         -- serialization of the first directory entry's metadata and contents,
+//!                            following the exact same archive format
+//!   * `FILENAME`          -- name of the second directory entry (strictly ordered!)
+//!   * `<archive>`         -- serialization of the second directory entry
+//!   * ...
+//!   * `GOODBYE`           -- lookup table at the end of a list of directory entries
 
 use std::cmp::Ordering;
 use std::ffi::{CStr, OsStr};
-- 
2.20.1

[pbs-devel] [PATCH backup] update to pxar 0.3 to support negative timestamps

[Wolfgang Bumiller]

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
 Cargo.toml           |  2 +-
 src/pxar/create.rs   |  9 ++++-----
 src/pxar/fuse.rs     | 17 ++++++-----------
 src/pxar/metadata.rs | 18 ++++++------------
 src/pxar/tools.rs    |  6 ++----
 5 files changed, 19 insertions(+), 33 deletions(-)

diff --git a/Cargo.toml b/Cargo.toml
index 258a9bb2..23063956 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -43,7 +43,7 @@ proxmox = { version = "0.2.1", features = [ "sortable-macro", "api-macro", "webs
 #proxmox = { git = "ssh://gitolite3@proxdev.maurer-it.com/rust/proxmox", version = "0.1.2", features = [ "sortable-macro", "api-macro" ] }
 #proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro", "websocket" ] }
 proxmox-fuse = "0.1.0"
-pxar = { version = "0.2.1", features = [ "tokio-io", "futures-io" ] }
+pxar = { version = "0.3.0", features = [ "tokio-io", "futures-io" ] }
 #pxar = { path = "../pxar", features = [ "tokio-io", "futures-io" ] }
 regex = "1.2"
 rustyline = "6"
diff --git a/src/pxar/create.rs b/src/pxar/create.rs
index f0831526..d0c5ae81 100644
--- a/src/pxar/create.rs
+++ b/src/pxar/create.rs
@@ -1,5 +1,4 @@
 use std::collections::{HashSet, HashMap};
-use std::convert::TryFrom;
 use std::ffi::{CStr, CString, OsStr};
 use std::fmt;
 use std::io::{self, Read, Write};
@@ -696,16 +695,16 @@ fn get_metadata(fd: RawFd, stat: &FileStat, flags: Flags, fs_magic: i64) -> Resu
     // required for some of these
     let proc_path = Path::new("/proc/self/fd/").join(fd.to_string());
 
-    let mtime = u64::try_from(stat.st_mtime * 1_000_000_000 + stat.st_mtime_nsec)
-        .map_err(|_| format_err!("file with negative mtime"))?;
-
     let mut meta = Metadata {
         stat: pxar::Stat {
             mode: u64::from(stat.st_mode),
             flags: 0,
             uid: stat.st_uid,
             gid: stat.st_gid,
-            mtime,
+            mtime: pxar::format::StatxTimestamp {
+                secs: stat.st_mtime,
+                nanos: stat.st_mtime_nsec as u32,
+            },
         },
         ..Default::default()
     };
diff --git a/src/pxar/fuse.rs b/src/pxar/fuse.rs
index 1534ff29..652b9219 100644
--- a/src/pxar/fuse.rs
+++ b/src/pxar/fuse.rs
@@ -673,11 +673,6 @@ fn to_stat(inode: u64, entry: &pxar::Entry) -> Result<libc::stat, Error> {
 
     let metadata = entry.metadata();
 
-    let time = i64::try_from(metadata.stat.mtime)
-        .map_err(|_| format_err!("mtime does not fit into a signed 64 bit integer"))?;
-    let sec = time / 1_000_000_000;
-    let nsec = time % 1_000_000_000;
-
     let mut stat: libc::stat = unsafe { mem::zeroed() };
     stat.st_ino = inode;
     stat.st_nlink = nlink;
@@ -687,11 +682,11 @@ fn to_stat(inode: u64, entry: &pxar::Entry) -> Result<libc::stat, Error> {
         .map_err(|err| format_err!("size does not fit into st_size field: {}", err))?;
     stat.st_uid = metadata.stat.uid;
     stat.st_gid = metadata.stat.gid;
-    stat.st_atime = sec;
-    stat.st_atime_nsec = nsec;
-    stat.st_mtime = sec;
-    stat.st_mtime_nsec = nsec;
-    stat.st_ctime = sec;
-    stat.st_ctime_nsec = nsec;
+    stat.st_atime = metadata.stat.mtime.secs;
+    stat.st_atime_nsec = metadata.stat.mtime.nanos as _;
+    stat.st_mtime = metadata.stat.mtime.secs;
+    stat.st_mtime_nsec = metadata.stat.mtime.nanos as _;
+    stat.st_ctime = metadata.stat.mtime.secs;
+    stat.st_ctime_nsec = metadata.stat.mtime.nanos as _;
     Ok(stat)
 }
diff --git a/src/pxar/metadata.rs b/src/pxar/metadata.rs
index 2cbed756..6df00230 100644
--- a/src/pxar/metadata.rs
+++ b/src/pxar/metadata.rs
@@ -37,26 +37,20 @@ fn allow_notsupp_remember<E: SysError>(err: E, not_supp: &mut bool) -> Result<()
     }
 }
 
-fn nsec_to_update_timespec(mtime_nsec: u64) -> [libc::timespec; 2] {
+fn timestamp_to_update_timespec(mtime: &pxar::format::StatxTimestamp) -> [libc::timespec; 2] {
     // restore mtime
     const UTIME_OMIT: i64 = (1 << 30) - 2;
-    const NANOS_PER_SEC: i64 = 1_000_000_000;
 
-    let sec = (mtime_nsec as i64) / NANOS_PER_SEC;
-    let nsec = (mtime_nsec as i64) % NANOS_PER_SEC;
-
-    let times: [libc::timespec; 2] = [
+    [
         libc::timespec {
             tv_sec: 0,
             tv_nsec: UTIME_OMIT,
         },
         libc::timespec {
-            tv_sec: sec,
-            tv_nsec: nsec,
+            tv_sec: mtime.secs,
+            tv_nsec: mtime.nanos as _,
         },
-    ];
-
-    times
+    ]
 }
 
 //
@@ -130,7 +124,7 @@ pub fn apply(flags: Flags, metadata: &Metadata, fd: RawFd, file_name: &CStr) ->
         libc::utimensat(
             libc::AT_FDCWD,
             c_proc_path.as_ptr(),
-            nsec_to_update_timespec(metadata.stat.mtime).as_ptr(),
+            timestamp_to_update_timespec(&metadata.stat.mtime).as_ptr(),
             0,
         )
     });
diff --git a/src/pxar/tools.rs b/src/pxar/tools.rs
index ec5c13b2..0fdb033d 100644
--- a/src/pxar/tools.rs
+++ b/src/pxar/tools.rs
@@ -120,8 +120,7 @@ pub fn format_single_line_entry(entry: &Entry) -> String {
     let mode_string = mode_string(entry);
 
     let meta = entry.metadata();
-    let mtime = meta.mtime_as_duration();
-    let mtime = chrono::Local.timestamp(mtime.as_secs() as i64, mtime.subsec_nanos());
+    let mtime = chrono::Local.timestamp(meta.stat.mtime.secs, meta.stat.mtime.nanos);
 
     let (size, link) = match entry.kind() {
         EntryKind::File { size, .. } => (format!("{}", *size), String::new()),
@@ -148,8 +147,7 @@ pub fn format_multi_line_entry(entry: &Entry) -> String {
     let mode_string = mode_string(entry);
 
     let meta = entry.metadata();
-    let mtime = meta.mtime_as_duration();
-    let mtime = chrono::Local.timestamp(mtime.as_secs() as i64, mtime.subsec_nanos());
+    let mtime = chrono::Local.timestamp(meta.stat.mtime.secs, meta.stat.mtime.nanos);
 
     let (size, link, type_name) = match entry.kind() {
         EntryKind::File { size, .. } => (format!("{}", *size), String::new(), "file"),
-- 
2.20.1

[pbs-devel] [PATCH pxar 2/6] introduce StatxTimestamp helper type

[Wolfgang Bumiller]

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
 src/format/mod.rs | 103 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 103 insertions(+)

diff --git a/src/format/mod.rs b/src/format/mod.rs
index 0f9db79..085b8b9 100644
--- a/src/format/mod.rs
+++ b/src/format/mod.rs
@@ -43,6 +43,7 @@ use std::io;
 use std::mem::size_of;
 use std::os::unix::ffi::OsStrExt;
 use std::path::Path;
+use std::time::{Duration, SystemTime};
 
 use endian_trait::Endian;
 use siphasher::sip::SipHasher24;
@@ -194,6 +195,108 @@ impl Display for Header {
     }
 }
 
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub enum SignedDuration {
+    Positive(Duration),
+    Negative(Duration),
+}
+
+#[derive(Clone, Debug, Default, Endian, Eq, PartialEq)]
+#[repr(C)]
+pub struct StatxTimestamp {
+    /// Seconds since the epoch (unix time).
+    pub secs: i64,
+
+    /// Nanoseconds since this struct's `secs`.
+    pub nanos: u32,
+}
+
+impl From<SystemTime> for StatxTimestamp {
+    fn from(time: SystemTime) -> Self {
+        match time.duration_since(SystemTime::UNIX_EPOCH) {
+            Ok(positive) => Self::from_duration_since_epoch(positive),
+            Err(negative) => Self::from_duration_before_epoch(negative.duration()),
+        }
+    }
+}
+
+impl StatxTimestamp {
+    /// `const` version of `Default`
+    pub const fn zero() -> Self {
+        Self { secs: 0, nanos: 0 }
+    }
+
+    /// Turn a positive duration relative to the unix epoch into a time stamp.
+    pub fn from_duration_since_epoch(duration: Duration) -> Self {
+        Self {
+            secs: duration.as_secs() as i64,
+            nanos: duration.subsec_nanos(),
+        }
+    }
+
+    /// Turn a *negative* duration from relative to the unix epoch into a time stamp.
+    pub fn from_duration_before_epoch(duration: Duration) -> Self {
+        match duration.subsec_nanos() {
+            0 => Self {
+                secs: duration.as_secs() as i64,
+                nanos: 0,
+            },
+            nanos => Self {
+                secs: -(duration.as_secs() as i64) - 1,
+                nanos: 1_000_000_000 - nanos,
+            },
+        }
+    }
+
+    /// Get the duration since the epoch. an `Ok` value is a positive duration, an `Err` value is a
+    /// negative duration.
+    pub fn to_duration(&self) -> SignedDuration {
+        if self.secs >= 0 {
+            SignedDuration::Positive(Duration::new(self.secs as u64, self.nanos))
+        } else {
+            SignedDuration::Negative(Duration::new(
+                -(self.secs + 1) as u64,
+                1_000_000_000 - self.nanos,
+            ))
+        }
+    }
+
+    /// Get a `std::time::SystemTime` from this time stamp.
+    pub fn system_time(&self) -> SystemTime {
+        match self.to_duration() {
+            SignedDuration::Positive(positive) => SystemTime::UNIX_EPOCH + positive,
+            SignedDuration::Negative(negative) => SystemTime::UNIX_EPOCH - negative,
+        }
+    }
+}
+
+#[test]
+fn test_statx_timestamp() {
+    const MAY_1_2015_1530: i64 = 1430487000;
+    let system_time = SystemTime::UNIX_EPOCH + Duration::new(MAY_1_2015_1530 as u64, 1_000_000);
+    let tx = StatxTimestamp::from(system_time);
+    assert_eq!(
+        tx,
+        StatxTimestamp {
+            secs: MAY_1_2015_1530,
+            nanos: 1_000_000,
+        }
+    );
+    assert_eq!(tx.system_time(), system_time);
+
+    const MAY_1_1960_1530: i64 = -305112600;
+    let system_time = SystemTime::UNIX_EPOCH - Duration::new(-MAY_1_1960_1530 as u64, 1_000_000);
+    let tx = StatxTimestamp::from(system_time);
+    assert_eq!(
+        tx,
+        StatxTimestamp {
+            secs: MAY_1_1960_1530 - 1,
+            nanos: 999_000_000,
+        }
+    );
+    assert_eq!(tx.system_time(), system_time);
+}
+
 #[derive(Clone, Debug, Default, Endian)]
 #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
 #[repr(C)]
-- 
2.20.1

[pbs-devel] [PATCH pxar 3/6] update mk-format-hashes for a new ENTRY

[Wolfgang Bumiller]

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
 examples/mk-format-hashes.rs | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/examples/mk-format-hashes.rs b/examples/mk-format-hashes.rs
index d2c9925..4ab6fff 100644
--- a/examples/mk-format-hashes.rs
+++ b/examples/mk-format-hashes.rs
@@ -1,7 +1,16 @@
 use pxar::format::hash_filename;
 
 const CONSTANTS: &[(&str, &str, &str)] = &[
-    ("", "PXAR_ENTRY", "__PROXMOX_FORMAT_ENTRY__"),
+    (
+        "Beginning of an entry (current version).",
+        "PXAR_ENTRY",
+        "__PROXMOX_FORMAT_ENTRY_V2__",
+    ),
+    (
+        "Previous version of the entry struct",
+        "PXAR_ENTRY_V1",
+        "__PROXMOX_FORMAT_ENTRY__",
+    ),
     ("", "PXAR_FILENAME", "__PROXMOX_FORMAT_FILENAME__"),
     ("", "PXAR_SYMLINK", "__PROXMOX_FORMAT_SYMLINK__"),
     ("", "PXAR_DEVICE", "__PROXMOX_FORMAT_DEVICE__"),
-- 
2.20.1

[pbs-devel] [PATCH pxar 4/6] implement Entry v2

[Wolfgang Bumiller]

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
 src/decoder/mod.rs | 21 ++++++++++++++++-----
 src/errors.rs      | 25 -------------------------
 src/format/mod.rs  | 46 ++++++++++++++++++++++++++++++++++------------
 src/lib.rs         | 28 +++++++++++++++-------------
 4 files changed, 65 insertions(+), 55 deletions(-)
 delete mode 100644 src/errors.rs

diff --git a/src/decoder/mod.rs b/src/decoder/mod.rs
index 9410942..e5b6d2c 100644
--- a/src/decoder/mod.rs
+++ b/src/decoder/mod.rs
@@ -360,11 +360,22 @@ impl<I: SeqRead> DecoderImpl<I> {
             self.entry.kind = EntryKind::Hardlink(self.read_hardlink().await?);
 
             Ok(Some(self.entry.take()))
-        } else if header.htype == format::PXAR_ENTRY {
-            self.entry.metadata = Metadata {
-                stat: seq_read_entry(&mut self.input).await?,
-                ..Default::default()
-            };
+        } else if header.htype == format::PXAR_ENTRY || header.htype == format::PXAR_ENTRY_V1 {
+            if header.htype == format::PXAR_ENTRY {
+                self.entry.metadata = Metadata {
+                    stat: seq_read_entry(&mut self.input).await?,
+                    ..Default::default()
+                };
+            } else if header.htype == format::PXAR_ENTRY_V1 {
+                let stat: format::Entry_V1 = seq_read_entry(&mut self.input).await?;
+
+                self.entry.metadata = Metadata {
+                    stat: stat.into(),
+                    ..Default::default()
+                };
+            } else {
+                unreachable!();
+            }
 
             self.current_header = unsafe { mem::zeroed() };
 
diff --git a/src/errors.rs b/src/errors.rs
deleted file mode 100644
index 8983e98..0000000
--- a/src/errors.rs
+++ /dev/null
@@ -1,25 +0,0 @@
-use std::error::Error;
-use std::fmt;
-
-#[derive(Clone, Debug)]
-pub enum TimeError {
-    Underflow(std::time::SystemTimeError),
-    Overflow(std::time::Duration),
-}
-
-impl fmt::Display for TimeError {
-    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        match self {
-            TimeError::Underflow(st) => st.fmt(f),
-            TimeError::Overflow(dur) => write!(f, "timestamp out of range: {:?}", dur),
-        }
-    }
-}
-
-impl Error for TimeError {}
-
-impl From<std::time::SystemTimeError> for TimeError {
-    fn from(t: std::time::SystemTimeError) -> Self {
-        TimeError::Underflow(t)
-    }
-}
diff --git a/src/format/mod.rs b/src/format/mod.rs
index 085b8b9..ec61e15 100644
--- a/src/format/mod.rs
+++ b/src/format/mod.rs
@@ -78,7 +78,10 @@ pub mod mode {
     pub const ISVTX  : u64 = 0o0001000;
 }
 
-pub const PXAR_ENTRY: u64 = 0x11da850a1c1cceff;
+/// Beginning of an entry (current version).
+pub const PXAR_ENTRY: u64 = 0xd5956474e588acef;
+/// Previous version of the entry struct
+pub const PXAR_ENTRY_V1: u64 = 0x11da850a1c1cceff;
 pub const PXAR_FILENAME: u64 = 0x16701121063917b3;
 pub const PXAR_SYMLINK: u64 = 0x27f971e7dbf5dc5f;
 pub const PXAR_DEVICE: u64 = 0x9fc9e906586d5ce9;
@@ -300,7 +303,7 @@ fn test_statx_timestamp() {
 #[derive(Clone, Debug, Default, Endian)]
 #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
 #[repr(C)]
-pub struct Entry {
+pub struct Entry_V1 {
     pub mode: u64,
     pub flags: u64,
     pub uid: u32,
@@ -308,6 +311,29 @@ pub struct Entry {
     pub mtime: u64,
 }
 
+impl Into<Entry> for Entry_V1 {
+    fn into(self) -> Entry {
+        Entry {
+            mode: self.mode,
+            flags: self.flags,
+            uid: self.uid,
+            gid: self.gid,
+            mtime: StatxTimestamp::from_duration_since_epoch(Duration::from_nanos(self.mtime)),
+        }
+    }
+}
+
+#[derive(Clone, Debug, Default, Endian)]
+#[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
+#[repr(C)]
+pub struct Entry {
+    pub mode: u64,
+    pub flags: u64,
+    pub uid: u32,
+    pub gid: u32,
+    pub mtime: StatxTimestamp,
+}
+
 /// Builder pattern methods.
 impl Entry {
     pub const fn mode(self, mode: u64) -> Self {
@@ -326,7 +352,7 @@ impl Entry {
         Self { gid, ..self }
     }
 
-    pub const fn mtime(self, mtime: u64) -> Self {
+    pub const fn mtime(self, mtime: StatxTimestamp) -> Self {
         Self { mtime, ..self }
     }
 
@@ -363,9 +389,10 @@ impl Entry {
 
 /// Convenience accessor methods.
 impl Entry {
-    /// Get the mtime as duration since the epoch.
-    pub fn mtime_as_duration(&self) -> std::time::Duration {
-        std::time::Duration::from_nanos(self.mtime)
+    /// Get the mtime as duration since the epoch. an `Ok` value is a positive duration, an `Err`
+    /// value is a negative duration.
+    pub fn mtime_as_duration(&self) -> SignedDuration {
+        self.mtime.to_duration()
     }
 
     /// Get the file type portion of the mode bitfield.
@@ -449,12 +476,7 @@ impl From<&std::fs::Metadata> for Entry {
             .mode(meta.mode() as u64);
 
         let this = match meta.modified() {
-            Ok(mtime) => this.mtime(
-                mtime
-                    .duration_since(std::time::SystemTime::UNIX_EPOCH)
-                    .map(|dur| dur.as_nanos() as u64)
-                    .unwrap_or(0u64),
-            ),
+            Ok(mtime) => this.mtime(mtime.into()),
             Err(_) => this,
         };
 
diff --git a/src/lib.rs b/src/lib.rs
index 5d1b781..a58ec30 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -2,7 +2,6 @@
 //!
 //! This implements a reader and writer for the proxmox archive format (.pxar).
 
-use std::convert::TryFrom;
 use std::ffi::OsStr;
 use std::mem;
 use std::path::{Path, PathBuf};
@@ -20,7 +19,6 @@ pub mod accessor;
 pub mod binary_tree_array;
 pub mod decoder;
 pub mod encoder;
-pub mod errors;
 
 /// Reexport of `format::Entry`. Since this conveys mostly information found via the `stat` syscall
 /// we mostly use this name for public interfaces.
@@ -125,8 +123,9 @@ impl Metadata {
         self.stat.is_socket()
     }
 
-    /// Get the mtime as duration since the epoch.
-    pub fn mtime_as_duration(&self) -> std::time::Duration {
+    /// Get the mtime as duration since the epoch. an `Ok` value is a positive duration, an `Err`
+    /// value is a negative duration.
+    pub fn mtime_as_duration(&self) -> format::SignedDuration {
         self.stat.mtime_as_duration()
     }
 
@@ -165,7 +164,7 @@ impl MetadataBuilder {
                     flags: 0,
                     uid: 0,
                     gid: 0,
-                    mtime: 0,
+                    mtime: format::StatxTimestamp::zero(),
                 },
                 xattrs: Vec::new(),
                 acl: Acl {
@@ -198,17 +197,20 @@ impl MetadataBuilder {
         self
     }
 
-    /// Set the modification time from a system time.
-    pub fn mtime(self, mtime: std::time::SystemTime) -> Result<Self, errors::TimeError> {
-        self.mtime_unix(mtime.duration_since(std::time::SystemTime::UNIX_EPOCH)?)
+    /// Set the modification time from a statx timespec value.
+    pub fn mtime_full(mut self, mtime: format::StatxTimestamp) -> Self {
+        self.inner.stat.mtime = mtime;
+        self
     }
 
     /// Set the modification time from a duration since the epoch (`SystemTime::UNIX_EPOCH`).
-    pub fn mtime_unix(mut self, mtime: std::time::Duration) -> Result<Self, errors::TimeError> {
-        let nanos =
-            u64::try_from(mtime.as_nanos()).map_err(|_| errors::TimeError::Overflow(mtime))?;
-        self.inner.stat.mtime = nanos;
-        Ok(self)
+    pub fn mtime_unix(self, mtime: std::time::Duration) -> Self {
+        self.mtime_full(format::StatxTimestamp::from_duration_since_epoch(mtime))
+    }
+
+    /// Set the modification time from a system time.
+    pub fn mtime(self, mtime: std::time::SystemTime) -> Self {
+        self.mtime_full(mtime.into())
     }
 
     /// Set the ownership information.
-- 
2.20.1

[pbs-devel] [PATCH pxar 5/6] add entry v1 compatiblity test

[Wolfgang Bumiller]

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
 tests/compat.rs | 136 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 136 insertions(+)
 create mode 100644 tests/compat.rs

diff --git a/tests/compat.rs b/tests/compat.rs
new file mode 100644
index 0000000..53e2d09
--- /dev/null
+++ b/tests/compat.rs
@@ -0,0 +1,136 @@
+//! Test for old timestamp format compatibility.
+
+use std::io::{self, Read};
+
+use endian_trait::Endian;
+
+use pxar::{decoder, format, EntryKind};
+
+fn write_raw_struct<T: Endian, W: io::Write + ?Sized>(output: &mut W, data: T) -> io::Result<()> {
+    let data = data.to_le();
+    output.write_all(unsafe {
+        std::slice::from_raw_parts(&data as *const T as *const u8, std::mem::size_of::<T>())
+    })
+}
+
+fn write_data<W>(output: &mut W, htype: u64, data: &[u8]) -> io::Result<()>
+where
+    W: io::Write + ?Sized,
+{
+    let header = format::Header::with_content_size(htype, data.len() as u64);
+    header.check_header_size()?;
+    write_raw_struct(output, header)?;
+    output.write_all(data)
+}
+
+fn write_entry<T, W>(output: &mut W, htype: u64, data: T) -> io::Result<()>
+where
+    T: Endian,
+    W: io::Write + ?Sized,
+{
+    let data = data.to_le();
+    let data = unsafe {
+        std::slice::from_raw_parts(&data as *const T as *const u8, std::mem::size_of::<T>())
+    };
+    write_data(output, htype, data)
+}
+
+const MAY_1_2015_1530: u64 = 1430487000u64;
+
+const FILE_NAME: &str = "file.txt";
+const FILE_NAME_BYTES: &[u8] = b"file.txt\0";
+const FILE_CONTENT: &[u8] = b"This is a small text file.\n";
+const ROOT_STAT: format::Entry_V1 = format::Entry_V1 {
+    mode: format::mode::IFDIR | 0o755,
+    flags: 0,
+    uid: 1000,
+    gid: 1000,
+    mtime: MAY_1_2015_1530 * 1_000_000_000u64,
+};
+const FILE_STAT: format::Entry_V1 = format::Entry_V1 {
+    mode: format::mode::IFREG | 0o644,
+    flags: 0,
+    uid: 1000,
+    gid: 1000,
+    mtime: MAY_1_2015_1530 * 1_000_000_000u64,
+};
+
+fn create_archive() -> io::Result<Vec<u8>> {
+    let mut out = Vec::new();
+
+    write_entry(&mut out, format::PXAR_ENTRY_V1, ROOT_STAT.clone())?;
+
+    let file_offset = out.len();
+    write_data(&mut out, format::PXAR_FILENAME, FILE_NAME_BYTES)?;
+    write_entry(&mut out, format::PXAR_ENTRY_V1, FILE_STAT.clone())?;
+    write_data(&mut out, format::PXAR_PAYLOAD, FILE_CONTENT)?;
+
+    let mut gbt = Vec::new();
+    write_raw_struct(
+        &mut gbt,
+        format::GoodbyeItem::new(
+            FILE_NAME.as_bytes(),
+            file_offset as u64,
+            FILE_CONTENT.len() as u64,
+        ),
+    )?;
+
+    let gbt_size = gbt.len();
+    write_raw_struct(
+        &mut gbt,
+        format::GoodbyeItem {
+            hash: format::PXAR_GOODBYE_TAIL_MARKER,
+            offset: out.len() as u64,
+            size: gbt_size as u64,
+        },
+    )?;
+
+    write_data(&mut out, format::PXAR_GOODBYE, &{ gbt })?;
+
+    Ok(out)
+}
+
+#[test]
+fn test_archive() {
+    let archive = create_archive().expect("failed to create test archive");
+    let mut input = &archive[..];
+    let mut decoder = decoder::Decoder::from_std(&mut input).expect("failed to create decoder");
+
+    let item = decoder
+        .next()
+        .expect("missing root directory in test archive")
+        .expect("failed to extract root directory from test archive");
+    match item.kind() {
+        EntryKind::Directory => (),
+        other => panic!("unexpected root entry in archive: {:?}", other),
+    }
+    assert_eq!(item.file_name(), "");
+    assert_eq!(item.metadata().stat, ROOT_STAT.into());
+    assert_eq!(
+        item.metadata().stat.mtime,
+        format::StatxTimestamp {
+            secs: MAY_1_2015_1530 as i64,
+            nanos: 0,
+        },
+    );
+
+    let item = decoder
+        .next()
+        .expect("missing file entry in test archive")
+        .expect("failed to extract file entry from test archive");
+    match item.kind() {
+        EntryKind::File { size, .. } => assert_eq!(*size, FILE_CONTENT.len() as u64),
+        other => panic!("unexpected file entry in archive: {:?}", other),
+    }
+    assert_eq!(item.file_name(), FILE_NAME);
+    assert_eq!(item.metadata().stat, FILE_STAT.into());
+    let mut content = Vec::new();
+    decoder
+        .contents()
+        .expect("failed to get contents for file entry")
+        .read_to_end(&mut content)
+        .expect("failed to read test file contents");
+    assert_eq!(&content[..], FILE_CONTENT);
+
+    assert!(decoder.next().is_none(), "expected end of test archive");
+}
-- 
2.20.1

[pbs-devel] [PATCH pxar 6/6] bump version to 0.3.0-1

[Wolfgang Bumiller]

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
 Cargo.toml       | 2 +-
 debian/changelog | 8 ++++++++
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/Cargo.toml b/Cargo.toml
index c2f0f96..c1b53fb 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "pxar"
-version = "0.2.1"
+version = "0.3.0"
 authors = ["Wolfgang Bumiller <w.bumiller@proxmox.com>"]
 edition = "2018"
 license = "AGPL-3"
diff --git a/debian/changelog b/debian/changelog
index 5901b61..b1a1852 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+rust-pxar (0.3.0-1) pve; urgency=medium
+
+  * introduce 96 bit time stamp type with support for negative timestamps
+
+  * introduce PXAR_ENTRY version 2 header type
+
+ -- Proxmox Support Team <support@proxmox.com>  Tue, 28 Jul 2020 11:56:47 +0200
+
 rust-pxar (0.2.1-1) pve; urgency=medium
 
   * sync encoder: fix metadata lifetime leaking into encoder
-- 
2.20.1

[pbs-devel] [PATCH pxar v2 2/6] introduce StatxTimestamp helper type

[Wolfgang Bumiller]

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
changes to v1:
  Add testcase with nanos=0
  Fix nanos=0 case in `from_duration_before_epoch`.
  (added comment about why `to_duration` doesn't need special handling,
  that is because std::time::Duration already does that)

 src/format/mod.rs | 115 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 115 insertions(+)

diff --git a/src/format/mod.rs b/src/format/mod.rs
index 0f9db79..1a38157 100644
--- a/src/format/mod.rs
+++ b/src/format/mod.rs
@@ -43,6 +43,7 @@ use std::io;
 use std::mem::size_of;
 use std::os::unix::ffi::OsStrExt;
 use std::path::Path;
+use std::time::{Duration, SystemTime};
 
 use endian_trait::Endian;
 use siphasher::sip::SipHasher24;
@@ -194,6 +195,120 @@ impl Display for Header {
     }
 }
 
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub enum SignedDuration {
+    Positive(Duration),
+    Negative(Duration),
+}
+
+#[derive(Clone, Debug, Default, Endian, Eq, PartialEq)]
+#[repr(C)]
+pub struct StatxTimestamp {
+    /// Seconds since the epoch (unix time).
+    pub secs: i64,
+
+    /// Nanoseconds since this struct's `secs`.
+    pub nanos: u32,
+}
+
+impl From<SystemTime> for StatxTimestamp {
+    fn from(time: SystemTime) -> Self {
+        match time.duration_since(SystemTime::UNIX_EPOCH) {
+            Ok(positive) => Self::from_duration_since_epoch(positive),
+            Err(negative) => Self::from_duration_before_epoch(negative.duration()),
+        }
+    }
+}
+
+impl StatxTimestamp {
+    /// `const` version of `Default`
+    pub const fn zero() -> Self {
+        Self { secs: 0, nanos: 0 }
+    }
+
+    /// Turn a positive duration relative to the unix epoch into a time stamp.
+    pub fn from_duration_since_epoch(duration: Duration) -> Self {
+        Self {
+            secs: duration.as_secs() as i64,
+            nanos: duration.subsec_nanos(),
+        }
+    }
+
+    /// Turn a *negative* duration from relative to the unix epoch into a time stamp.
+    pub fn from_duration_before_epoch(duration: Duration) -> Self {
+        match duration.subsec_nanos() {
+            0 => Self {
+                secs: -(duration.as_secs() as i64),
+                nanos: 0,
+            },
+            nanos => Self {
+                secs: -(duration.as_secs() as i64) - 1,
+                nanos: 1_000_000_000 - nanos,
+            },
+        }
+    }
+
+    /// Get the duration since the epoch. an `Ok` value is a positive duration, an `Err` value is a
+    /// negative duration.
+    pub fn to_duration(&self) -> SignedDuration {
+        if self.secs >= 0 {
+            SignedDuration::Positive(Duration::new(self.secs as u64, self.nanos))
+        } else {
+            // this handles the nanos=0 case as `Duration::new()` performs the carry-over.
+            SignedDuration::Negative(Duration::new(
+                -(self.secs + 1) as u64,
+                1_000_000_000 - self.nanos,
+            ))
+        }
+    }
+
+    /// Get a `std::time::SystemTime` from this time stamp.
+    pub fn system_time(&self) -> SystemTime {
+        match self.to_duration() {
+            SignedDuration::Positive(positive) => SystemTime::UNIX_EPOCH + positive,
+            SignedDuration::Negative(negative) => SystemTime::UNIX_EPOCH - negative,
+        }
+    }
+}
+
+#[test]
+fn test_statx_timestamp() {
+    const MAY_1_2015_1530: i64 = 1430487000;
+    let system_time = SystemTime::UNIX_EPOCH + Duration::new(MAY_1_2015_1530 as u64, 1_000_000);
+    let tx = StatxTimestamp::from(system_time);
+    assert_eq!(
+        tx,
+        StatxTimestamp {
+            secs: MAY_1_2015_1530,
+            nanos: 1_000_000,
+        }
+    );
+    assert_eq!(tx.system_time(), system_time);
+
+    const MAY_1_1960_1530: i64 = -305112600;
+    let system_time = SystemTime::UNIX_EPOCH - Duration::new(-MAY_1_1960_1530 as u64, 1_000_000);
+    let tx = StatxTimestamp::from(system_time);
+    assert_eq!(
+        tx,
+        StatxTimestamp {
+            secs: MAY_1_1960_1530 - 1,
+            nanos: 999_000_000,
+        }
+    );
+    assert_eq!(tx.system_time(), system_time);
+
+    let system_time = SystemTime::UNIX_EPOCH - Duration::new(-MAY_1_1960_1530 as u64, 0);
+    let tx = StatxTimestamp::from(system_time);
+    assert_eq!(
+        tx,
+        StatxTimestamp {
+            secs: MAY_1_1960_1530,
+            nanos: 0,
+        }
+    );
+    assert_eq!(tx.system_time(), system_time);
+}
+
 #[derive(Clone, Debug, Default, Endian)]
 #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
 #[repr(C)]
-- 
2.20.1

[pbs-devel] applied: [PATCH pxar/backup 0/6] bump timestamps to 96 bit

[Dietmar Maurer]

applied

> On 07/28/2020 12:33 PM Wolfgang Bumiller <w.bumiller@proxmox.com> wrote:
> 
>  
> So apparently modification time values of *before* Jan 1 1970 are a
> thing, so here's support for that...
> 
> This bumps the `Entry` struct in pxar (meaning an API bump), and still
> supports reading old archives.
> 
> Note that I've introduced a new `StatxTimestamp` struct. I chose this
> name as the `statx(2)` manpage's `struct statx_timestamp` is the only
> struct which actually documents the fact that only the seconds are
> signed, and the nanoseconds are *positive* and *relative* to the
> seconds, iow. a timestamp of "-3.5 seconds" is represented as "-4
> seconds, plus 500_000_000 nanoseconds". (The only other time I found
> this to be explicitly mentioned is in the `chrono` crate's
> `TimeZone::timestamp` method which explicitly creates a "DateTime from
> the number of non-leap seconds since (...) 1970 (...) and the number of
> nanoseconds since the last whole non-leap second.".
> 
> Wolfgang Bumiller (6):
>  pxar:
>   add format description to format module
>   introduce StatxTimestamp helper type
>   update mk-format-hashes for a new ENTRY
>   implement Entry v2
>   add entry v1 compatiblity test
>   bump version to 0.3.0-1
> 
>  Cargo.toml                   |   2 +-
>  debian/changelog             |   8 ++
>  examples/mk-format-hashes.rs |  11 ++-
>  src/decoder/mod.rs           |  21 +++-
>  src/errors.rs                |  25 -----
>  src/format/mod.rs            | 179 ++++++++++++++++++++++++++++++++---
>  src/lib.rs                   |  28 +++---
>  tests/compat.rs              | 136 ++++++++++++++++++++++++++
>  8 files changed, 353 insertions(+), 57 deletions(-)
>  delete mode 100644 src/errors.rs
>  create mode 100644 tests/compat.rs
> 
>  backup:
>   update to pxar 0.3 to support negative timestamps
> 
> -- 
> 2.20.1
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] applied: [PATCH backup] update to pxar 0.3 to support negative timestamps

[Dietmar Maurer]

applied