* [pbs-devel] [PATCH pxar 1/6] add format description to format module
2020-07-28 10:33 [pbs-devel] [PATCH pxar/backup 0/6] bump timestamps to 96 bit Wolfgang Bumiller
@ 2020-07-28 10:33 ` Wolfgang Bumiller
2020-07-28 10:33 ` [pbs-devel] [PATCH backup] update to pxar 0.3 to support negative timestamps Wolfgang Bumiller
` (6 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Wolfgang Bumiller @ 2020-07-28 10:33 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
src/format/mod.rs | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/src/format/mod.rs b/src/format/mod.rs
index 2a7d377..0f9db79 100644
--- a/src/format/mod.rs
+++ b/src/format/mod.rs
@@ -4,6 +4,36 @@
//!
//! The Archive contains a list of items. Each item starts with a `Header`, followed by the
//! item data.
+//!
+//! An archive contains items in the following order:
+//! * `ENTRY` -- containing general stat() data and related bits
+//! * `XATTR` -- one extended attribute
+//! * ... -- more of these when there are multiple defined
+//! * `ACL_USER` -- one `USER ACL` entry
+//! * ... -- more of these when there are multiple defined
+//! * `ACL_GROUP` -- one `GROUP ACL` entry
+//! * ... -- more of these when there are multiple defined
+//! * `ACL_GROUP_OBJ` -- The `ACL_GROUP_OBJ`
+//! * `ACL_DEFAULT` -- The various default ACL fields if there's one defined
+//! * `ACL_DEFAULT_USER` -- one USER ACL entry
+//! * ... -- more of these when multiple are defined
+//! * `ACL_DEFAULT_GROUP` -- one GROUP ACL entry
+//! * ... -- more of these when multiple are defined
+//! * `FCAPS` -- file capability in Linux disk format
+//! * `QUOTA_PROJECT_ID` -- the ext4/xfs quota project ID
+//! * `PAYLOAD` -- file contents, if it is one
+//! * `SYMLINK` -- symlink target, if it is one
+//! * `DEVICE` -- device major/minor, if it is a block/char device
+//!
+//! If we are serializing a directory, then this is followed by:
+//!
+//! * `FILENAME` -- name of the first directory entry (strictly ordered!)
+//! * `<archive>` -- serialization of the first directory entry's metadata and contents,
+//! following the exact same archive format
+//! * `FILENAME` -- name of the second directory entry (strictly ordered!)
+//! * `<archive>` -- serialization of the second directory entry
+//! * ...
+//! * `GOODBYE` -- lookup table at the end of a list of directory entries
use std::cmp::Ordering;
use std::ffi::{CStr, OsStr};
--
2.20.1
^ permalink raw reply [flat|nested] 11+ messages in thread* [pbs-devel] [PATCH backup] update to pxar 0.3 to support negative timestamps
2020-07-28 10:33 [pbs-devel] [PATCH pxar/backup 0/6] bump timestamps to 96 bit Wolfgang Bumiller
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 1/6] add format description to format module Wolfgang Bumiller
@ 2020-07-28 10:33 ` Wolfgang Bumiller
2020-07-29 6:32 ` [pbs-devel] applied: " Dietmar Maurer
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 2/6] introduce StatxTimestamp helper type Wolfgang Bumiller
` (5 subsequent siblings)
7 siblings, 1 reply; 11+ messages in thread
From: Wolfgang Bumiller @ 2020-07-28 10:33 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
Cargo.toml | 2 +-
src/pxar/create.rs | 9 ++++-----
src/pxar/fuse.rs | 17 ++++++-----------
src/pxar/metadata.rs | 18 ++++++------------
src/pxar/tools.rs | 6 ++----
5 files changed, 19 insertions(+), 33 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index 258a9bb2..23063956 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -43,7 +43,7 @@ proxmox = { version = "0.2.1", features = [ "sortable-macro", "api-macro", "webs
#proxmox = { git = "ssh://gitolite3@proxdev.maurer-it.com/rust/proxmox", version = "0.1.2", features = [ "sortable-macro", "api-macro" ] }
#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro", "websocket" ] }
proxmox-fuse = "0.1.0"
-pxar = { version = "0.2.1", features = [ "tokio-io", "futures-io" ] }
+pxar = { version = "0.3.0", features = [ "tokio-io", "futures-io" ] }
#pxar = { path = "../pxar", features = [ "tokio-io", "futures-io" ] }
regex = "1.2"
rustyline = "6"
diff --git a/src/pxar/create.rs b/src/pxar/create.rs
index f0831526..d0c5ae81 100644
--- a/src/pxar/create.rs
+++ b/src/pxar/create.rs
@@ -1,5 +1,4 @@
use std::collections::{HashSet, HashMap};
-use std::convert::TryFrom;
use std::ffi::{CStr, CString, OsStr};
use std::fmt;
use std::io::{self, Read, Write};
@@ -696,16 +695,16 @@ fn get_metadata(fd: RawFd, stat: &FileStat, flags: Flags, fs_magic: i64) -> Resu
// required for some of these
let proc_path = Path::new("/proc/self/fd/").join(fd.to_string());
- let mtime = u64::try_from(stat.st_mtime * 1_000_000_000 + stat.st_mtime_nsec)
- .map_err(|_| format_err!("file with negative mtime"))?;
-
let mut meta = Metadata {
stat: pxar::Stat {
mode: u64::from(stat.st_mode),
flags: 0,
uid: stat.st_uid,
gid: stat.st_gid,
- mtime,
+ mtime: pxar::format::StatxTimestamp {
+ secs: stat.st_mtime,
+ nanos: stat.st_mtime_nsec as u32,
+ },
},
..Default::default()
};
diff --git a/src/pxar/fuse.rs b/src/pxar/fuse.rs
index 1534ff29..652b9219 100644
--- a/src/pxar/fuse.rs
+++ b/src/pxar/fuse.rs
@@ -673,11 +673,6 @@ fn to_stat(inode: u64, entry: &pxar::Entry) -> Result<libc::stat, Error> {
let metadata = entry.metadata();
- let time = i64::try_from(metadata.stat.mtime)
- .map_err(|_| format_err!("mtime does not fit into a signed 64 bit integer"))?;
- let sec = time / 1_000_000_000;
- let nsec = time % 1_000_000_000;
-
let mut stat: libc::stat = unsafe { mem::zeroed() };
stat.st_ino = inode;
stat.st_nlink = nlink;
@@ -687,11 +682,11 @@ fn to_stat(inode: u64, entry: &pxar::Entry) -> Result<libc::stat, Error> {
.map_err(|err| format_err!("size does not fit into st_size field: {}", err))?;
stat.st_uid = metadata.stat.uid;
stat.st_gid = metadata.stat.gid;
- stat.st_atime = sec;
- stat.st_atime_nsec = nsec;
- stat.st_mtime = sec;
- stat.st_mtime_nsec = nsec;
- stat.st_ctime = sec;
- stat.st_ctime_nsec = nsec;
+ stat.st_atime = metadata.stat.mtime.secs;
+ stat.st_atime_nsec = metadata.stat.mtime.nanos as _;
+ stat.st_mtime = metadata.stat.mtime.secs;
+ stat.st_mtime_nsec = metadata.stat.mtime.nanos as _;
+ stat.st_ctime = metadata.stat.mtime.secs;
+ stat.st_ctime_nsec = metadata.stat.mtime.nanos as _;
Ok(stat)
}
diff --git a/src/pxar/metadata.rs b/src/pxar/metadata.rs
index 2cbed756..6df00230 100644
--- a/src/pxar/metadata.rs
+++ b/src/pxar/metadata.rs
@@ -37,26 +37,20 @@ fn allow_notsupp_remember<E: SysError>(err: E, not_supp: &mut bool) -> Result<()
}
}
-fn nsec_to_update_timespec(mtime_nsec: u64) -> [libc::timespec; 2] {
+fn timestamp_to_update_timespec(mtime: &pxar::format::StatxTimestamp) -> [libc::timespec; 2] {
// restore mtime
const UTIME_OMIT: i64 = (1 << 30) - 2;
- const NANOS_PER_SEC: i64 = 1_000_000_000;
- let sec = (mtime_nsec as i64) / NANOS_PER_SEC;
- let nsec = (mtime_nsec as i64) % NANOS_PER_SEC;
-
- let times: [libc::timespec; 2] = [
+ [
libc::timespec {
tv_sec: 0,
tv_nsec: UTIME_OMIT,
},
libc::timespec {
- tv_sec: sec,
- tv_nsec: nsec,
+ tv_sec: mtime.secs,
+ tv_nsec: mtime.nanos as _,
},
- ];
-
- times
+ ]
}
//
@@ -130,7 +124,7 @@ pub fn apply(flags: Flags, metadata: &Metadata, fd: RawFd, file_name: &CStr) ->
libc::utimensat(
libc::AT_FDCWD,
c_proc_path.as_ptr(),
- nsec_to_update_timespec(metadata.stat.mtime).as_ptr(),
+ timestamp_to_update_timespec(&metadata.stat.mtime).as_ptr(),
0,
)
});
diff --git a/src/pxar/tools.rs b/src/pxar/tools.rs
index ec5c13b2..0fdb033d 100644
--- a/src/pxar/tools.rs
+++ b/src/pxar/tools.rs
@@ -120,8 +120,7 @@ pub fn format_single_line_entry(entry: &Entry) -> String {
let mode_string = mode_string(entry);
let meta = entry.metadata();
- let mtime = meta.mtime_as_duration();
- let mtime = chrono::Local.timestamp(mtime.as_secs() as i64, mtime.subsec_nanos());
+ let mtime = chrono::Local.timestamp(meta.stat.mtime.secs, meta.stat.mtime.nanos);
let (size, link) = match entry.kind() {
EntryKind::File { size, .. } => (format!("{}", *size), String::new()),
@@ -148,8 +147,7 @@ pub fn format_multi_line_entry(entry: &Entry) -> String {
let mode_string = mode_string(entry);
let meta = entry.metadata();
- let mtime = meta.mtime_as_duration();
- let mtime = chrono::Local.timestamp(mtime.as_secs() as i64, mtime.subsec_nanos());
+ let mtime = chrono::Local.timestamp(meta.stat.mtime.secs, meta.stat.mtime.nanos);
let (size, link, type_name) = match entry.kind() {
EntryKind::File { size, .. } => (format!("{}", *size), String::new(), "file"),
--
2.20.1
^ permalink raw reply [flat|nested] 11+ messages in thread* [pbs-devel] [PATCH pxar 2/6] introduce StatxTimestamp helper type
2020-07-28 10:33 [pbs-devel] [PATCH pxar/backup 0/6] bump timestamps to 96 bit Wolfgang Bumiller
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 1/6] add format description to format module Wolfgang Bumiller
2020-07-28 10:33 ` [pbs-devel] [PATCH backup] update to pxar 0.3 to support negative timestamps Wolfgang Bumiller
@ 2020-07-28 10:33 ` Wolfgang Bumiller
2020-07-28 14:05 ` [pbs-devel] [PATCH pxar v2 " Wolfgang Bumiller
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 3/6] update mk-format-hashes for a new ENTRY Wolfgang Bumiller
` (4 subsequent siblings)
7 siblings, 1 reply; 11+ messages in thread
From: Wolfgang Bumiller @ 2020-07-28 10:33 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
src/format/mod.rs | 103 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 103 insertions(+)
diff --git a/src/format/mod.rs b/src/format/mod.rs
index 0f9db79..085b8b9 100644
--- a/src/format/mod.rs
+++ b/src/format/mod.rs
@@ -43,6 +43,7 @@ use std::io;
use std::mem::size_of;
use std::os::unix::ffi::OsStrExt;
use std::path::Path;
+use std::time::{Duration, SystemTime};
use endian_trait::Endian;
use siphasher::sip::SipHasher24;
@@ -194,6 +195,108 @@ impl Display for Header {
}
}
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub enum SignedDuration {
+ Positive(Duration),
+ Negative(Duration),
+}
+
+#[derive(Clone, Debug, Default, Endian, Eq, PartialEq)]
+#[repr(C)]
+pub struct StatxTimestamp {
+ /// Seconds since the epoch (unix time).
+ pub secs: i64,
+
+ /// Nanoseconds since this struct's `secs`.
+ pub nanos: u32,
+}
+
+impl From<SystemTime> for StatxTimestamp {
+ fn from(time: SystemTime) -> Self {
+ match time.duration_since(SystemTime::UNIX_EPOCH) {
+ Ok(positive) => Self::from_duration_since_epoch(positive),
+ Err(negative) => Self::from_duration_before_epoch(negative.duration()),
+ }
+ }
+}
+
+impl StatxTimestamp {
+ /// `const` version of `Default`
+ pub const fn zero() -> Self {
+ Self { secs: 0, nanos: 0 }
+ }
+
+ /// Turn a positive duration relative to the unix epoch into a time stamp.
+ pub fn from_duration_since_epoch(duration: Duration) -> Self {
+ Self {
+ secs: duration.as_secs() as i64,
+ nanos: duration.subsec_nanos(),
+ }
+ }
+
+ /// Turn a *negative* duration from relative to the unix epoch into a time stamp.
+ pub fn from_duration_before_epoch(duration: Duration) -> Self {
+ match duration.subsec_nanos() {
+ 0 => Self {
+ secs: duration.as_secs() as i64,
+ nanos: 0,
+ },
+ nanos => Self {
+ secs: -(duration.as_secs() as i64) - 1,
+ nanos: 1_000_000_000 - nanos,
+ },
+ }
+ }
+
+ /// Get the duration since the epoch. an `Ok` value is a positive duration, an `Err` value is a
+ /// negative duration.
+ pub fn to_duration(&self) -> SignedDuration {
+ if self.secs >= 0 {
+ SignedDuration::Positive(Duration::new(self.secs as u64, self.nanos))
+ } else {
+ SignedDuration::Negative(Duration::new(
+ -(self.secs + 1) as u64,
+ 1_000_000_000 - self.nanos,
+ ))
+ }
+ }
+
+ /// Get a `std::time::SystemTime` from this time stamp.
+ pub fn system_time(&self) -> SystemTime {
+ match self.to_duration() {
+ SignedDuration::Positive(positive) => SystemTime::UNIX_EPOCH + positive,
+ SignedDuration::Negative(negative) => SystemTime::UNIX_EPOCH - negative,
+ }
+ }
+}
+
+#[test]
+fn test_statx_timestamp() {
+ const MAY_1_2015_1530: i64 = 1430487000;
+ let system_time = SystemTime::UNIX_EPOCH + Duration::new(MAY_1_2015_1530 as u64, 1_000_000);
+ let tx = StatxTimestamp::from(system_time);
+ assert_eq!(
+ tx,
+ StatxTimestamp {
+ secs: MAY_1_2015_1530,
+ nanos: 1_000_000,
+ }
+ );
+ assert_eq!(tx.system_time(), system_time);
+
+ const MAY_1_1960_1530: i64 = -305112600;
+ let system_time = SystemTime::UNIX_EPOCH - Duration::new(-MAY_1_1960_1530 as u64, 1_000_000);
+ let tx = StatxTimestamp::from(system_time);
+ assert_eq!(
+ tx,
+ StatxTimestamp {
+ secs: MAY_1_1960_1530 - 1,
+ nanos: 999_000_000,
+ }
+ );
+ assert_eq!(tx.system_time(), system_time);
+}
+
#[derive(Clone, Debug, Default, Endian)]
#[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
#[repr(C)]
--
2.20.1
^ permalink raw reply [flat|nested] 11+ messages in thread* [pbs-devel] [PATCH pxar v2 2/6] introduce StatxTimestamp helper type
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 2/6] introduce StatxTimestamp helper type Wolfgang Bumiller
@ 2020-07-28 14:05 ` Wolfgang Bumiller
0 siblings, 0 replies; 11+ messages in thread
From: Wolfgang Bumiller @ 2020-07-28 14:05 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
changes to v1:
Add testcase with nanos=0
Fix nanos=0 case in `from_duration_before_epoch`.
(added comment about why `to_duration` doesn't need special handling,
that is because std::time::Duration already does that)
src/format/mod.rs | 115 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 115 insertions(+)
diff --git a/src/format/mod.rs b/src/format/mod.rs
index 0f9db79..1a38157 100644
--- a/src/format/mod.rs
+++ b/src/format/mod.rs
@@ -43,6 +43,7 @@ use std::io;
use std::mem::size_of;
use std::os::unix::ffi::OsStrExt;
use std::path::Path;
+use std::time::{Duration, SystemTime};
use endian_trait::Endian;
use siphasher::sip::SipHasher24;
@@ -194,6 +195,120 @@ impl Display for Header {
}
}
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub enum SignedDuration {
+ Positive(Duration),
+ Negative(Duration),
+}
+
+#[derive(Clone, Debug, Default, Endian, Eq, PartialEq)]
+#[repr(C)]
+pub struct StatxTimestamp {
+ /// Seconds since the epoch (unix time).
+ pub secs: i64,
+
+ /// Nanoseconds since this struct's `secs`.
+ pub nanos: u32,
+}
+
+impl From<SystemTime> for StatxTimestamp {
+ fn from(time: SystemTime) -> Self {
+ match time.duration_since(SystemTime::UNIX_EPOCH) {
+ Ok(positive) => Self::from_duration_since_epoch(positive),
+ Err(negative) => Self::from_duration_before_epoch(negative.duration()),
+ }
+ }
+}
+
+impl StatxTimestamp {
+ /// `const` version of `Default`
+ pub const fn zero() -> Self {
+ Self { secs: 0, nanos: 0 }
+ }
+
+ /// Turn a positive duration relative to the unix epoch into a time stamp.
+ pub fn from_duration_since_epoch(duration: Duration) -> Self {
+ Self {
+ secs: duration.as_secs() as i64,
+ nanos: duration.subsec_nanos(),
+ }
+ }
+
+ /// Turn a *negative* duration from relative to the unix epoch into a time stamp.
+ pub fn from_duration_before_epoch(duration: Duration) -> Self {
+ match duration.subsec_nanos() {
+ 0 => Self {
+ secs: -(duration.as_secs() as i64),
+ nanos: 0,
+ },
+ nanos => Self {
+ secs: -(duration.as_secs() as i64) - 1,
+ nanos: 1_000_000_000 - nanos,
+ },
+ }
+ }
+
+ /// Get the duration since the epoch. an `Ok` value is a positive duration, an `Err` value is a
+ /// negative duration.
+ pub fn to_duration(&self) -> SignedDuration {
+ if self.secs >= 0 {
+ SignedDuration::Positive(Duration::new(self.secs as u64, self.nanos))
+ } else {
+ // this handles the nanos=0 case as `Duration::new()` performs the carry-over.
+ SignedDuration::Negative(Duration::new(
+ -(self.secs + 1) as u64,
+ 1_000_000_000 - self.nanos,
+ ))
+ }
+ }
+
+ /// Get a `std::time::SystemTime` from this time stamp.
+ pub fn system_time(&self) -> SystemTime {
+ match self.to_duration() {
+ SignedDuration::Positive(positive) => SystemTime::UNIX_EPOCH + positive,
+ SignedDuration::Negative(negative) => SystemTime::UNIX_EPOCH - negative,
+ }
+ }
+}
+
+#[test]
+fn test_statx_timestamp() {
+ const MAY_1_2015_1530: i64 = 1430487000;
+ let system_time = SystemTime::UNIX_EPOCH + Duration::new(MAY_1_2015_1530 as u64, 1_000_000);
+ let tx = StatxTimestamp::from(system_time);
+ assert_eq!(
+ tx,
+ StatxTimestamp {
+ secs: MAY_1_2015_1530,
+ nanos: 1_000_000,
+ }
+ );
+ assert_eq!(tx.system_time(), system_time);
+
+ const MAY_1_1960_1530: i64 = -305112600;
+ let system_time = SystemTime::UNIX_EPOCH - Duration::new(-MAY_1_1960_1530 as u64, 1_000_000);
+ let tx = StatxTimestamp::from(system_time);
+ assert_eq!(
+ tx,
+ StatxTimestamp {
+ secs: MAY_1_1960_1530 - 1,
+ nanos: 999_000_000,
+ }
+ );
+ assert_eq!(tx.system_time(), system_time);
+
+ let system_time = SystemTime::UNIX_EPOCH - Duration::new(-MAY_1_1960_1530 as u64, 0);
+ let tx = StatxTimestamp::from(system_time);
+ assert_eq!(
+ tx,
+ StatxTimestamp {
+ secs: MAY_1_1960_1530,
+ nanos: 0,
+ }
+ );
+ assert_eq!(tx.system_time(), system_time);
+}
+
#[derive(Clone, Debug, Default, Endian)]
#[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
#[repr(C)]
--
2.20.1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [pbs-devel] [PATCH pxar 3/6] update mk-format-hashes for a new ENTRY
2020-07-28 10:33 [pbs-devel] [PATCH pxar/backup 0/6] bump timestamps to 96 bit Wolfgang Bumiller
` (2 preceding siblings ...)
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 2/6] introduce StatxTimestamp helper type Wolfgang Bumiller
@ 2020-07-28 10:33 ` Wolfgang Bumiller
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 4/6] implement Entry v2 Wolfgang Bumiller
` (3 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Wolfgang Bumiller @ 2020-07-28 10:33 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
examples/mk-format-hashes.rs | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/examples/mk-format-hashes.rs b/examples/mk-format-hashes.rs
index d2c9925..4ab6fff 100644
--- a/examples/mk-format-hashes.rs
+++ b/examples/mk-format-hashes.rs
@@ -1,7 +1,16 @@
use pxar::format::hash_filename;
const CONSTANTS: &[(&str, &str, &str)] = &[
- ("", "PXAR_ENTRY", "__PROXMOX_FORMAT_ENTRY__"),
+ (
+ "Beginning of an entry (current version).",
+ "PXAR_ENTRY",
+ "__PROXMOX_FORMAT_ENTRY_V2__",
+ ),
+ (
+ "Previous version of the entry struct",
+ "PXAR_ENTRY_V1",
+ "__PROXMOX_FORMAT_ENTRY__",
+ ),
("", "PXAR_FILENAME", "__PROXMOX_FORMAT_FILENAME__"),
("", "PXAR_SYMLINK", "__PROXMOX_FORMAT_SYMLINK__"),
("", "PXAR_DEVICE", "__PROXMOX_FORMAT_DEVICE__"),
--
2.20.1
^ permalink raw reply [flat|nested] 11+ messages in thread* [pbs-devel] [PATCH pxar 4/6] implement Entry v2
2020-07-28 10:33 [pbs-devel] [PATCH pxar/backup 0/6] bump timestamps to 96 bit Wolfgang Bumiller
` (3 preceding siblings ...)
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 3/6] update mk-format-hashes for a new ENTRY Wolfgang Bumiller
@ 2020-07-28 10:33 ` Wolfgang Bumiller
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 5/6] add entry v1 compatiblity test Wolfgang Bumiller
` (2 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Wolfgang Bumiller @ 2020-07-28 10:33 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
src/decoder/mod.rs | 21 ++++++++++++++++-----
src/errors.rs | 25 -------------------------
src/format/mod.rs | 46 ++++++++++++++++++++++++++++++++++------------
src/lib.rs | 28 +++++++++++++++-------------
4 files changed, 65 insertions(+), 55 deletions(-)
delete mode 100644 src/errors.rs
diff --git a/src/decoder/mod.rs b/src/decoder/mod.rs
index 9410942..e5b6d2c 100644
--- a/src/decoder/mod.rs
+++ b/src/decoder/mod.rs
@@ -360,11 +360,22 @@ impl<I: SeqRead> DecoderImpl<I> {
self.entry.kind = EntryKind::Hardlink(self.read_hardlink().await?);
Ok(Some(self.entry.take()))
- } else if header.htype == format::PXAR_ENTRY {
- self.entry.metadata = Metadata {
- stat: seq_read_entry(&mut self.input).await?,
- ..Default::default()
- };
+ } else if header.htype == format::PXAR_ENTRY || header.htype == format::PXAR_ENTRY_V1 {
+ if header.htype == format::PXAR_ENTRY {
+ self.entry.metadata = Metadata {
+ stat: seq_read_entry(&mut self.input).await?,
+ ..Default::default()
+ };
+ } else if header.htype == format::PXAR_ENTRY_V1 {
+ let stat: format::Entry_V1 = seq_read_entry(&mut self.input).await?;
+
+ self.entry.metadata = Metadata {
+ stat: stat.into(),
+ ..Default::default()
+ };
+ } else {
+ unreachable!();
+ }
self.current_header = unsafe { mem::zeroed() };
diff --git a/src/errors.rs b/src/errors.rs
deleted file mode 100644
index 8983e98..0000000
--- a/src/errors.rs
+++ /dev/null
@@ -1,25 +0,0 @@
-use std::error::Error;
-use std::fmt;
-
-#[derive(Clone, Debug)]
-pub enum TimeError {
- Underflow(std::time::SystemTimeError),
- Overflow(std::time::Duration),
-}
-
-impl fmt::Display for TimeError {
- fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
- match self {
- TimeError::Underflow(st) => st.fmt(f),
- TimeError::Overflow(dur) => write!(f, "timestamp out of range: {:?}", dur),
- }
- }
-}
-
-impl Error for TimeError {}
-
-impl From<std::time::SystemTimeError> for TimeError {
- fn from(t: std::time::SystemTimeError) -> Self {
- TimeError::Underflow(t)
- }
-}
diff --git a/src/format/mod.rs b/src/format/mod.rs
index 085b8b9..ec61e15 100644
--- a/src/format/mod.rs
+++ b/src/format/mod.rs
@@ -78,7 +78,10 @@ pub mod mode {
pub const ISVTX : u64 = 0o0001000;
}
-pub const PXAR_ENTRY: u64 = 0x11da850a1c1cceff;
+/// Beginning of an entry (current version).
+pub const PXAR_ENTRY: u64 = 0xd5956474e588acef;
+/// Previous version of the entry struct
+pub const PXAR_ENTRY_V1: u64 = 0x11da850a1c1cceff;
pub const PXAR_FILENAME: u64 = 0x16701121063917b3;
pub const PXAR_SYMLINK: u64 = 0x27f971e7dbf5dc5f;
pub const PXAR_DEVICE: u64 = 0x9fc9e906586d5ce9;
@@ -300,7 +303,7 @@ fn test_statx_timestamp() {
#[derive(Clone, Debug, Default, Endian)]
#[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
#[repr(C)]
-pub struct Entry {
+pub struct Entry_V1 {
pub mode: u64,
pub flags: u64,
pub uid: u32,
@@ -308,6 +311,29 @@ pub struct Entry {
pub mtime: u64,
}
+impl Into<Entry> for Entry_V1 {
+ fn into(self) -> Entry {
+ Entry {
+ mode: self.mode,
+ flags: self.flags,
+ uid: self.uid,
+ gid: self.gid,
+ mtime: StatxTimestamp::from_duration_since_epoch(Duration::from_nanos(self.mtime)),
+ }
+ }
+}
+
+#[derive(Clone, Debug, Default, Endian)]
+#[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
+#[repr(C)]
+pub struct Entry {
+ pub mode: u64,
+ pub flags: u64,
+ pub uid: u32,
+ pub gid: u32,
+ pub mtime: StatxTimestamp,
+}
+
/// Builder pattern methods.
impl Entry {
pub const fn mode(self, mode: u64) -> Self {
@@ -326,7 +352,7 @@ impl Entry {
Self { gid, ..self }
}
- pub const fn mtime(self, mtime: u64) -> Self {
+ pub const fn mtime(self, mtime: StatxTimestamp) -> Self {
Self { mtime, ..self }
}
@@ -363,9 +389,10 @@ impl Entry {
/// Convenience accessor methods.
impl Entry {
- /// Get the mtime as duration since the epoch.
- pub fn mtime_as_duration(&self) -> std::time::Duration {
- std::time::Duration::from_nanos(self.mtime)
+ /// Get the mtime as duration since the epoch. an `Ok` value is a positive duration, an `Err`
+ /// value is a negative duration.
+ pub fn mtime_as_duration(&self) -> SignedDuration {
+ self.mtime.to_duration()
}
/// Get the file type portion of the mode bitfield.
@@ -449,12 +476,7 @@ impl From<&std::fs::Metadata> for Entry {
.mode(meta.mode() as u64);
let this = match meta.modified() {
- Ok(mtime) => this.mtime(
- mtime
- .duration_since(std::time::SystemTime::UNIX_EPOCH)
- .map(|dur| dur.as_nanos() as u64)
- .unwrap_or(0u64),
- ),
+ Ok(mtime) => this.mtime(mtime.into()),
Err(_) => this,
};
diff --git a/src/lib.rs b/src/lib.rs
index 5d1b781..a58ec30 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -2,7 +2,6 @@
//!
//! This implements a reader and writer for the proxmox archive format (.pxar).
-use std::convert::TryFrom;
use std::ffi::OsStr;
use std::mem;
use std::path::{Path, PathBuf};
@@ -20,7 +19,6 @@ pub mod accessor;
pub mod binary_tree_array;
pub mod decoder;
pub mod encoder;
-pub mod errors;
/// Reexport of `format::Entry`. Since this conveys mostly information found via the `stat` syscall
/// we mostly use this name for public interfaces.
@@ -125,8 +123,9 @@ impl Metadata {
self.stat.is_socket()
}
- /// Get the mtime as duration since the epoch.
- pub fn mtime_as_duration(&self) -> std::time::Duration {
+ /// Get the mtime as duration since the epoch. an `Ok` value is a positive duration, an `Err`
+ /// value is a negative duration.
+ pub fn mtime_as_duration(&self) -> format::SignedDuration {
self.stat.mtime_as_duration()
}
@@ -165,7 +164,7 @@ impl MetadataBuilder {
flags: 0,
uid: 0,
gid: 0,
- mtime: 0,
+ mtime: format::StatxTimestamp::zero(),
},
xattrs: Vec::new(),
acl: Acl {
@@ -198,17 +197,20 @@ impl MetadataBuilder {
self
}
- /// Set the modification time from a system time.
- pub fn mtime(self, mtime: std::time::SystemTime) -> Result<Self, errors::TimeError> {
- self.mtime_unix(mtime.duration_since(std::time::SystemTime::UNIX_EPOCH)?)
+ /// Set the modification time from a statx timespec value.
+ pub fn mtime_full(mut self, mtime: format::StatxTimestamp) -> Self {
+ self.inner.stat.mtime = mtime;
+ self
}
/// Set the modification time from a duration since the epoch (`SystemTime::UNIX_EPOCH`).
- pub fn mtime_unix(mut self, mtime: std::time::Duration) -> Result<Self, errors::TimeError> {
- let nanos =
- u64::try_from(mtime.as_nanos()).map_err(|_| errors::TimeError::Overflow(mtime))?;
- self.inner.stat.mtime = nanos;
- Ok(self)
+ pub fn mtime_unix(self, mtime: std::time::Duration) -> Self {
+ self.mtime_full(format::StatxTimestamp::from_duration_since_epoch(mtime))
+ }
+
+ /// Set the modification time from a system time.
+ pub fn mtime(self, mtime: std::time::SystemTime) -> Self {
+ self.mtime_full(mtime.into())
}
/// Set the ownership information.
--
2.20.1
^ permalink raw reply [flat|nested] 11+ messages in thread* [pbs-devel] [PATCH pxar 5/6] add entry v1 compatiblity test
2020-07-28 10:33 [pbs-devel] [PATCH pxar/backup 0/6] bump timestamps to 96 bit Wolfgang Bumiller
` (4 preceding siblings ...)
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 4/6] implement Entry v2 Wolfgang Bumiller
@ 2020-07-28 10:33 ` Wolfgang Bumiller
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 6/6] bump version to 0.3.0-1 Wolfgang Bumiller
2020-07-29 6:14 ` [pbs-devel] applied: [PATCH pxar/backup 0/6] bump timestamps to 96 bit Dietmar Maurer
7 siblings, 0 replies; 11+ messages in thread
From: Wolfgang Bumiller @ 2020-07-28 10:33 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
tests/compat.rs | 136 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 136 insertions(+)
create mode 100644 tests/compat.rs
diff --git a/tests/compat.rs b/tests/compat.rs
new file mode 100644
index 0000000..53e2d09
--- /dev/null
+++ b/tests/compat.rs
@@ -0,0 +1,136 @@
+//! Test for old timestamp format compatibility.
+
+use std::io::{self, Read};
+
+use endian_trait::Endian;
+
+use pxar::{decoder, format, EntryKind};
+
+fn write_raw_struct<T: Endian, W: io::Write + ?Sized>(output: &mut W, data: T) -> io::Result<()> {
+ let data = data.to_le();
+ output.write_all(unsafe {
+ std::slice::from_raw_parts(&data as *const T as *const u8, std::mem::size_of::<T>())
+ })
+}
+
+fn write_data<W>(output: &mut W, htype: u64, data: &[u8]) -> io::Result<()>
+where
+ W: io::Write + ?Sized,
+{
+ let header = format::Header::with_content_size(htype, data.len() as u64);
+ header.check_header_size()?;
+ write_raw_struct(output, header)?;
+ output.write_all(data)
+}
+
+fn write_entry<T, W>(output: &mut W, htype: u64, data: T) -> io::Result<()>
+where
+ T: Endian,
+ W: io::Write + ?Sized,
+{
+ let data = data.to_le();
+ let data = unsafe {
+ std::slice::from_raw_parts(&data as *const T as *const u8, std::mem::size_of::<T>())
+ };
+ write_data(output, htype, data)
+}
+
+const MAY_1_2015_1530: u64 = 1430487000u64;
+
+const FILE_NAME: &str = "file.txt";
+const FILE_NAME_BYTES: &[u8] = b"file.txt\0";
+const FILE_CONTENT: &[u8] = b"This is a small text file.\n";
+const ROOT_STAT: format::Entry_V1 = format::Entry_V1 {
+ mode: format::mode::IFDIR | 0o755,
+ flags: 0,
+ uid: 1000,
+ gid: 1000,
+ mtime: MAY_1_2015_1530 * 1_000_000_000u64,
+};
+const FILE_STAT: format::Entry_V1 = format::Entry_V1 {
+ mode: format::mode::IFREG | 0o644,
+ flags: 0,
+ uid: 1000,
+ gid: 1000,
+ mtime: MAY_1_2015_1530 * 1_000_000_000u64,
+};
+
+fn create_archive() -> io::Result<Vec<u8>> {
+ let mut out = Vec::new();
+
+ write_entry(&mut out, format::PXAR_ENTRY_V1, ROOT_STAT.clone())?;
+
+ let file_offset = out.len();
+ write_data(&mut out, format::PXAR_FILENAME, FILE_NAME_BYTES)?;
+ write_entry(&mut out, format::PXAR_ENTRY_V1, FILE_STAT.clone())?;
+ write_data(&mut out, format::PXAR_PAYLOAD, FILE_CONTENT)?;
+
+ let mut gbt = Vec::new();
+ write_raw_struct(
+ &mut gbt,
+ format::GoodbyeItem::new(
+ FILE_NAME.as_bytes(),
+ file_offset as u64,
+ FILE_CONTENT.len() as u64,
+ ),
+ )?;
+
+ let gbt_size = gbt.len();
+ write_raw_struct(
+ &mut gbt,
+ format::GoodbyeItem {
+ hash: format::PXAR_GOODBYE_TAIL_MARKER,
+ offset: out.len() as u64,
+ size: gbt_size as u64,
+ },
+ )?;
+
+ write_data(&mut out, format::PXAR_GOODBYE, &{ gbt })?;
+
+ Ok(out)
+}
+
+#[test]
+fn test_archive() {
+ let archive = create_archive().expect("failed to create test archive");
+ let mut input = &archive[..];
+ let mut decoder = decoder::Decoder::from_std(&mut input).expect("failed to create decoder");
+
+ let item = decoder
+ .next()
+ .expect("missing root directory in test archive")
+ .expect("failed to extract root directory from test archive");
+ match item.kind() {
+ EntryKind::Directory => (),
+ other => panic!("unexpected root entry in archive: {:?}", other),
+ }
+ assert_eq!(item.file_name(), "");
+ assert_eq!(item.metadata().stat, ROOT_STAT.into());
+ assert_eq!(
+ item.metadata().stat.mtime,
+ format::StatxTimestamp {
+ secs: MAY_1_2015_1530 as i64,
+ nanos: 0,
+ },
+ );
+
+ let item = decoder
+ .next()
+ .expect("missing file entry in test archive")
+ .expect("failed to extract file entry from test archive");
+ match item.kind() {
+ EntryKind::File { size, .. } => assert_eq!(*size, FILE_CONTENT.len() as u64),
+ other => panic!("unexpected file entry in archive: {:?}", other),
+ }
+ assert_eq!(item.file_name(), FILE_NAME);
+ assert_eq!(item.metadata().stat, FILE_STAT.into());
+ let mut content = Vec::new();
+ decoder
+ .contents()
+ .expect("failed to get contents for file entry")
+ .read_to_end(&mut content)
+ .expect("failed to read test file contents");
+ assert_eq!(&content[..], FILE_CONTENT);
+
+ assert!(decoder.next().is_none(), "expected end of test archive");
+}
--
2.20.1
^ permalink raw reply [flat|nested] 11+ messages in thread* [pbs-devel] [PATCH pxar 6/6] bump version to 0.3.0-1
2020-07-28 10:33 [pbs-devel] [PATCH pxar/backup 0/6] bump timestamps to 96 bit Wolfgang Bumiller
` (5 preceding siblings ...)
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 5/6] add entry v1 compatiblity test Wolfgang Bumiller
@ 2020-07-28 10:33 ` Wolfgang Bumiller
2020-07-29 6:14 ` [pbs-devel] applied: [PATCH pxar/backup 0/6] bump timestamps to 96 bit Dietmar Maurer
7 siblings, 0 replies; 11+ messages in thread
From: Wolfgang Bumiller @ 2020-07-28 10:33 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
Cargo.toml | 2 +-
debian/changelog | 8 ++++++++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/Cargo.toml b/Cargo.toml
index c2f0f96..c1b53fb 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,6 +1,6 @@
[package]
name = "pxar"
-version = "0.2.1"
+version = "0.3.0"
authors = ["Wolfgang Bumiller <w.bumiller@proxmox.com>"]
edition = "2018"
license = "AGPL-3"
diff --git a/debian/changelog b/debian/changelog
index 5901b61..b1a1852 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+rust-pxar (0.3.0-1) pve; urgency=medium
+
+ * introduce 96 bit time stamp type with support for negative timestamps
+
+ * introduce PXAR_ENTRY version 2 header type
+
+ -- Proxmox Support Team <support@proxmox.com> Tue, 28 Jul 2020 11:56:47 +0200
+
rust-pxar (0.2.1-1) pve; urgency=medium
* sync encoder: fix metadata lifetime leaking into encoder
--
2.20.1
^ permalink raw reply [flat|nested] 11+ messages in thread* [pbs-devel] applied: [PATCH pxar/backup 0/6] bump timestamps to 96 bit
2020-07-28 10:33 [pbs-devel] [PATCH pxar/backup 0/6] bump timestamps to 96 bit Wolfgang Bumiller
` (6 preceding siblings ...)
2020-07-28 10:33 ` [pbs-devel] [PATCH pxar 6/6] bump version to 0.3.0-1 Wolfgang Bumiller
@ 2020-07-29 6:14 ` Dietmar Maurer
7 siblings, 0 replies; 11+ messages in thread
From: Dietmar Maurer @ 2020-07-29 6:14 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Wolfgang Bumiller
applied
> On 07/28/2020 12:33 PM Wolfgang Bumiller <w.bumiller@proxmox.com> wrote:
>
>
> So apparently modification time values of *before* Jan 1 1970 are a
> thing, so here's support for that...
>
> This bumps the `Entry` struct in pxar (meaning an API bump), and still
> supports reading old archives.
>
> Note that I've introduced a new `StatxTimestamp` struct. I chose this
> name as the `statx(2)` manpage's `struct statx_timestamp` is the only
> struct which actually documents the fact that only the seconds are
> signed, and the nanoseconds are *positive* and *relative* to the
> seconds, iow. a timestamp of "-3.5 seconds" is represented as "-4
> seconds, plus 500_000_000 nanoseconds". (The only other time I found
> this to be explicitly mentioned is in the `chrono` crate's
> `TimeZone::timestamp` method which explicitly creates a "DateTime from
> the number of non-leap seconds since (...) 1970 (...) and the number of
> nanoseconds since the last whole non-leap second.".
>
> Wolfgang Bumiller (6):
> pxar:
> add format description to format module
> introduce StatxTimestamp helper type
> update mk-format-hashes for a new ENTRY
> implement Entry v2
> add entry v1 compatiblity test
> bump version to 0.3.0-1
>
> Cargo.toml | 2 +-
> debian/changelog | 8 ++
> examples/mk-format-hashes.rs | 11 ++-
> src/decoder/mod.rs | 21 +++-
> src/errors.rs | 25 -----
> src/format/mod.rs | 179 ++++++++++++++++++++++++++++++++---
> src/lib.rs | 28 +++---
> tests/compat.rs | 136 ++++++++++++++++++++++++++
> 8 files changed, 353 insertions(+), 57 deletions(-)
> delete mode 100644 src/errors.rs
> create mode 100644 tests/compat.rs
>
> backup:
> update to pxar 0.3 to support negative timestamps
>
> --
> 2.20.1
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 11+ messages in thread