[pbs-devel] [PATCH proxmox/widget-toolkit/proxmox-backup] add tar.zst support for file download

[pbs-devel] [PATCH proxmox/widget-toolkit/proxmox-backup] add tar.zst support for file download

[Dominik Csapak]

this series adds support for downloading .tar.zst files instead of .zip
when downloading directories of unencrypted pxar archives

PVE support is TBD, but should not be that hard to do. (Only missing
part is to add a directory directly for the restore-daemon)

we must bump 'proxmox-compression' and depend accordingly
proxmox-backup ui patch depends on new widget-toolkit
adds a new dependency on 'librust-tar-dev'

proxmox:

Dominik Csapak (2):
  proxmox-compression: add async tar builder
  proxmox-compression: add streaming zstd encoder

 proxmox-compression/Cargo.toml  |   2 +
 proxmox-compression/src/lib.rs  |   2 +
 proxmox-compression/src/tar.rs  | 172 ++++++++++++++++++++++++++++++++
 proxmox-compression/src/zstd.rs | 126 +++++++++++++++++++++++
 4 files changed, 302 insertions(+)
 create mode 100644 proxmox-compression/src/tar.rs
 create mode 100644 proxmox-compression/src/zstd.rs

proxmox-widget-toolkit:

Dominik Csapak (1):
  window/FileBrowser: add optional 'tar.zst' button

 src/window/FileBrowser.js | 39 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 34 insertions(+), 5 deletions(-)

proxmox-backup:

Dominik Csapak (3):
  pbs-client: add 'create_tar' helper function
  api: admin/datastore: add tar support for pxar_file_download
  ui: datastore/Content: enable tar download in ui

 Cargo.toml                     |   1 +
 pbs-client/Cargo.toml          |   1 +
 pbs-client/src/pxar/extract.rs | 211 ++++++++++++++++++++++++++++++++-
 pbs-client/src/pxar/mod.rs     |   2 +-
 src/api2/admin/datastore.rs    |  33 ++++--
 www/datastore/Content.js       |   1 +
 6 files changed, 237 insertions(+), 12 deletions(-)

-- 
2.30.2

[pbs-devel] [PATCH proxmox 1/2] proxmox-compression: add async tar builder

[Dominik Csapak]

inspired by tar::Builder, but limited to the things we need and using
AsyncRead+AsyncWrite instead of the sync variants.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 proxmox-compression/Cargo.toml |   1 +
 proxmox-compression/src/lib.rs |   1 +
 proxmox-compression/src/tar.rs | 172 +++++++++++++++++++++++++++++++++
 3 files changed, 174 insertions(+)
 create mode 100644 proxmox-compression/src/tar.rs

diff --git a/proxmox-compression/Cargo.toml b/proxmox-compression/Cargo.toml
index 0b9edf5..c3f7f49 100644
--- a/proxmox-compression/Cargo.toml
+++ b/proxmox-compression/Cargo.toml
@@ -17,6 +17,7 @@ flate2 = "1.0"
 futures = "0.3"
 tokio = { version = "1.6", features = [ "fs", "io-util"] }
 walkdir = "2"
+tar = "0.4"
 
 proxmox-time = { path = "../proxmox-time", version = "1" }
 proxmox-io = { path = "../proxmox-io", version = "1", features = [ "tokio" ] }
diff --git a/proxmox-compression/src/lib.rs b/proxmox-compression/src/lib.rs
index 05cf06b..e9dd113 100644
--- a/proxmox-compression/src/lib.rs
+++ b/proxmox-compression/src/lib.rs
@@ -1,4 +1,5 @@
 mod compression;
 pub use compression::*;
 
+pub mod tar;
 pub mod zip;
diff --git a/proxmox-compression/src/tar.rs b/proxmox-compression/src/tar.rs
new file mode 100644
index 0000000..59a8cc1
--- /dev/null
+++ b/proxmox-compression/src/tar.rs
@@ -0,0 +1,172 @@
+//! tar helper
+use std::io;
+use std::os::unix::ffi::OsStrExt;
+use std::path::{Component, Path, PathBuf};
+use std::str;
+
+use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt};
+
+use tar::{EntryType, Header};
+
+/// An async Builder for tar archives based on [tar::Builder]
+///
+/// Wraps an inner [AsyncWrite] struct to write into.
+/// Must call [finish()](Builder::finish) to write trailer + close
+/// # Example
+///
+/// ```
+/// use tar::{EntryType, Header};
+/// use proxmox_compression::tar::Builder;
+///
+/// # async fn foo() {
+/// let mut tar = Builder::new(Vec::new());
+///
+/// // Add file
+/// let mut header = Header::new_gnu();
+/// let mut data: &[u8] = &[1, 2, 3];
+/// header.set_size(data.len() as u64);
+/// tar.add_entry(&mut header, "foo", data).await.unwrap();
+///
+/// // Add symlink
+/// let mut header = Header::new_gnu();
+/// header.set_entry_type(EntryType::Symlink);
+/// tar.add_link(&mut header, "bar", "foo").await.unwrap();
+///
+/// // must call finish at the end
+/// let data = tar.finish().await.unwrap();
+/// # }
+/// ```
+pub struct Builder<W: AsyncWrite + Unpin> {
+    inner: W,
+}
+
+impl<W: AsyncWrite + Unpin> Builder<W> {
+    /// Takes an AsyncWriter as target
+    pub fn new(inner: W) -> Builder<W> {
+        Builder {
+            inner,
+        }
+    }
+
+    async fn add<R: AsyncRead + Unpin>(
+        &mut self,
+        header: &Header,
+        mut data: R,
+    ) -> io::Result<()> {
+        append_data(&mut self.inner, header, &mut data).await
+    }
+
+    /// Adds a new entry to this archive with the specified path.
+    pub async fn add_entry<P: AsRef<Path>, R: AsyncRead + Unpin>(
+        &mut self,
+        header: &mut Header,
+        path: P,
+        data: R,
+    ) -> io::Result<()> {
+        append_path_header(&mut self.inner, header, path.as_ref()).await?;
+        header.set_cksum();
+        self.add(&header, data).await
+    }
+
+    /// Adds a new link (symbolic or hard) entry to this archive with the specified path and target.
+    pub async fn add_link<P: AsRef<Path>, T: AsRef<Path>>(
+        &mut self,
+        header: &mut Header,
+        path: P,
+        target: T,
+    ) -> io::Result<()> {
+        append_path_header(&mut self.inner, header, path.as_ref()).await?;
+
+        // try to set the linkame, fallback to gnu extension header otherwise
+        if let Err(err) = header.set_link_name(target.as_ref()) {
+            let link_name = target.as_ref().as_os_str().as_bytes();
+            if link_name.len() < header.as_old().linkname.len() {
+                return Err(err);
+            }
+            // add trailing '\0'
+            let mut ext_data = link_name.chain(tokio::io::repeat(0).take(1));
+            let extension = get_gnu_header(link_name.len() as u64 + 1, EntryType::GNULongLink);
+            append_data(&mut self.inner, &extension, &mut ext_data).await?;
+        }
+        header.set_cksum();
+        self.add(&header, tokio::io::empty()).await
+    }
+
+    /// Finish the archive and flush the underlying writer
+    ///
+    /// Consumes the Builder. This must be called when finishing the archive.
+    /// Flushes the inner writer and returns it.
+    pub async fn finish(mut self) -> io::Result<W> {
+        self.inner.write_all(&[0; 1024]).await?;
+        self.inner.flush().await?;
+        Ok(self.inner)
+    }
+}
+
+async fn append_data<W: AsyncWrite + Unpin, R: AsyncRead + Unpin>(
+    mut dst: &mut W,
+    header: &Header,
+    mut data: &mut R,
+) -> io::Result<()> {
+    dst.write_all(header.as_bytes()).await?;
+    let len = tokio::io::copy(&mut data, &mut dst).await?;
+
+    // Pad with zeros if necessary.
+    let buf = [0; 512];
+    let remaining = 512 - (len % 512);
+    if remaining < 512 {
+        dst.write_all(&buf[..remaining as usize]).await?;
+    }
+
+    Ok(())
+}
+
+fn get_gnu_header(size: u64, entry_type: EntryType) -> Header {
+    let mut header = Header::new_gnu();
+    let name = b"././@LongLink";
+    header.as_gnu_mut().unwrap().name[..name.len()].clone_from_slice(&name[..]);
+    header.set_mode(0o644);
+    header.set_uid(0);
+    header.set_gid(0);
+    header.set_mtime(0);
+    header.set_size(size);
+    header.set_entry_type(entry_type);
+    header.set_cksum();
+    header
+}
+
+// tries to set the path in header, or add a gnu header with 'LongName'
+async fn append_path_header<W: AsyncWrite + Unpin>(
+    dst: &mut W,
+    header: &mut Header,
+    path: &Path,
+) -> io::Result<()> {
+    let mut relpath = PathBuf::new();
+    let components = path.components();
+    for comp in components {
+        if Component::RootDir == comp {
+            continue;
+        }
+        relpath.push(comp);
+    }
+    // try to set the path directly, fallback to gnu extension header otherwise
+    if let Err(err) = header.set_path(&relpath) {
+        let data = relpath.as_os_str().as_bytes();
+        let max = header.as_old().name.len();
+        if data.len() < max {
+            return Err(err);
+        }
+        // add trailing '\0'
+        let mut ext_data = data.chain(tokio::io::repeat(0).take(1));
+        let extension = get_gnu_header(data.len() as u64 + 1, EntryType::GNULongName);
+        append_data(dst, &extension, &mut ext_data).await?;
+
+        // add the path as far as we can
+        let truncated = match str::from_utf8(&data[..max]) {
+            Ok(truncated) => truncated,
+            Err(err) => str::from_utf8(&data[..err.valid_up_to()]).unwrap(),
+        };
+        header.set_path(truncated)?;
+    }
+    Ok(())
+}
-- 
2.30.2

[pbs-devel] [PATCH proxmox 2/2] proxmox-compression: add streaming zstd encoder

[Dominik Csapak]

similar to our DeflateEncoder, takes a Stream and implements it itself,
so that we can use it as an adapter for async api calls

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 proxmox-compression/Cargo.toml  |   1 +
 proxmox-compression/src/lib.rs  |   1 +
 proxmox-compression/src/zstd.rs | 126 ++++++++++++++++++++++++++++++++
 3 files changed, 128 insertions(+)
 create mode 100644 proxmox-compression/src/zstd.rs

diff --git a/proxmox-compression/Cargo.toml b/proxmox-compression/Cargo.toml
index c3f7f49..5ca67b2 100644
--- a/proxmox-compression/Cargo.toml
+++ b/proxmox-compression/Cargo.toml
@@ -18,6 +18,7 @@ futures = "0.3"
 tokio = { version = "1.6", features = [ "fs", "io-util"] }
 walkdir = "2"
 tar = "0.4"
+zstd = { version = "0.6", features = []}
 
 proxmox-time = { path = "../proxmox-time", version = "1" }
 proxmox-io = { path = "../proxmox-io", version = "1", features = [ "tokio" ] }
diff --git a/proxmox-compression/src/lib.rs b/proxmox-compression/src/lib.rs
index e9dd113..1fcfb97 100644
--- a/proxmox-compression/src/lib.rs
+++ b/proxmox-compression/src/lib.rs
@@ -3,3 +3,4 @@ pub use compression::*;
 
 pub mod tar;
 pub mod zip;
+pub mod zstd;
diff --git a/proxmox-compression/src/zstd.rs b/proxmox-compression/src/zstd.rs
new file mode 100644
index 0000000..0b480f6
--- /dev/null
+++ b/proxmox-compression/src/zstd.rs
@@ -0,0 +1,126 @@
+//! zstd helper
+use std::io;
+use std::pin::Pin;
+use std::task::{Context, Poll};
+
+use anyhow::{format_err, Error};
+use bytes::Bytes;
+use futures::ready;
+use futures::stream::Stream;
+use zstd::stream::raw::{Encoder, Operation, OutBuffer};
+
+use proxmox_io::ByteBuffer;
+
+const BUFFER_SIZE: usize = 8192;
+
+#[derive(Eq, PartialEq)]
+enum EncoderState {
+    Reading,
+    Writing,
+    Finishing,
+    Finished,
+}
+
+/// An async ZstdEncoder that implements [Stream] for another [Stream]
+///
+/// Useful for on-the-fly zstd compression in streaming api calls
+pub struct ZstdEncoder<'a, T> {
+    inner: T,
+    compressor: Encoder<'a>,
+    buffer: ByteBuffer,
+    input_buffer: Bytes,
+    state: EncoderState,
+}
+
+impl<'a, T> ZstdEncoder<'a, T> {
+    /// Returns a new [ZstdEncoder] with default level 3
+    pub fn new(inner: T) -> Result<Self, io::Error> {
+        Self::with_quality(inner, 3)
+    }
+
+    /// Returns a new [ZstdEncoder] with the given level
+    pub fn with_quality(inner: T, level: i32) -> Result<Self, io::Error> {
+        Ok(Self {
+            inner,
+            compressor: Encoder::new(level)?,
+            buffer: ByteBuffer::with_capacity(BUFFER_SIZE),
+            input_buffer: Bytes::new(),
+            state: EncoderState::Reading,
+        })
+    }
+
+    /// Returns the wrapped [Stream]
+    pub fn into_inner(self) -> T {
+        self.inner
+    }
+
+    fn encode(&mut self, inbuf: &[u8]) -> Result<zstd::stream::raw::Status, io::Error> {
+        let res = self
+            .compressor
+            .run_on_buffers(inbuf, self.buffer.get_free_mut_slice())?;
+        self.buffer.add_size(res.bytes_written);
+
+        Ok(res)
+    }
+
+    fn finish(&mut self) -> Result<usize, io::Error> {
+        let mut outbuf = OutBuffer::around(self.buffer.get_free_mut_slice());
+        let res = self.compressor.finish(&mut outbuf, true);
+        let size = outbuf.pos;
+        drop(outbuf);
+        self.buffer.add_size(size);
+        res
+    }
+}
+
+impl<'a, T, O> Stream for ZstdEncoder<'a, T>
+where
+    T: Stream<Item = Result<O, Error>> + Unpin,
+    O: Into<Bytes>,
+{
+    type Item = Result<Bytes, Error>;
+
+    fn poll_next(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
+        let this = self.get_mut();
+
+        loop {
+            match this.state {
+                EncoderState::Reading => {
+                    if let Some(res) = ready!(Pin::new(&mut this.inner).poll_next(cx)) {
+                        let buf = res?;
+                        this.input_buffer = buf.into();
+                        this.state = EncoderState::Writing;
+                    } else {
+                        this.state = EncoderState::Finishing;
+                    }
+                }
+                EncoderState::Writing => {
+                    if this.input_buffer.is_empty() {
+                        return Poll::Ready(Some(Err(format_err!("empty input during write"))));
+                    }
+                    let mut buf = this.input_buffer.split_off(0);
+                    let status = this.encode(&buf[..])?;
+                    this.input_buffer = buf.split_off(status.bytes_read);
+                    if this.input_buffer.is_empty() {
+                        this.state = EncoderState::Reading;
+                    }
+                    if this.buffer.is_full() {
+                        let bytes = this.buffer.remove_data(this.buffer.len()).to_vec();
+                        return Poll::Ready(Some(Ok(bytes.into())));
+                    }
+                }
+                EncoderState::Finishing => {
+                    let remaining = this.finish()?;
+                    if remaining == 0 {
+                        this.state = EncoderState::Finished;
+                    }
+                    if !this.buffer.is_empty() {
+                        let bytes = this.buffer.remove_data(this.buffer.len()).to_vec();
+                        return Poll::Ready(Some(Ok(bytes.into())));
+                    }
+                }
+                EncoderState::Finished => return Poll::Ready(None),
+            }
+        }
+    }
+}
-- 
2.30.2

[pbs-devel] [PATCH widget-toolkit 1/1] window/FileBrowser: add optional 'tar.zst' button

[Dominik Csapak]

only show it when enabled in config (so that we can hide it where
that is not supported, which is in PVE right now)

also changes the text between 'Download' and 'Download .zip' depending
if the selected entry is a directory or not

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/window/FileBrowser.js | 39 ++++++++++++++++++++++++++++++++++-----
 1 file changed, 34 insertions(+), 5 deletions(-)

diff --git a/src/window/FileBrowser.js b/src/window/FileBrowser.js
index 99a7a85..2efa988 100644
--- a/src/window/FileBrowser.js
+++ b/src/window/FileBrowser.js
@@ -75,6 +75,9 @@ Ext.define("Proxmox.window.FileBrowser", {
 	    'f': true, // "normal" files
 	    'd': true, // directories
 	},
+
+	// set to true to show the tar download button
+	enableTar: false,
     },
 
     controller: {
@@ -89,7 +92,15 @@ Ext.define("Proxmox.window.FileBrowser", {
 	    return url.href;
 	},
 
-	downloadFile: function() {
+	downloadTar: function() {
+	    this.downloadFile(true);
+	},
+
+	downloadZip: function() {
+	    this.downloadFile(false);
+	},
+
+	downloadFile: function(tar) {
 	    let me = this;
 	    let view = me.getView();
 	    let tree = me.lookup('tree');
@@ -105,7 +116,12 @@ Ext.define("Proxmox.window.FileBrowser", {
 	    params.filepath = data.filepath;
 	    atag.download = data.text;
 	    if (data.type === 'd') {
-		atag.download += ".zip";
+		if (tar) {
+		    params.tar = 1;
+		    atag.download += ".tar.zst";
+		} else {
+		    atag.download += ".zip";
+		}
 	    }
 	    atag.href = me.buildUrl(view.downloadURL, params);
 	    atag.click();
@@ -120,12 +136,18 @@ Ext.define("Proxmox.window.FileBrowser", {
 
 	    let data = selection[0].data;
 	    let canDownload = view.downloadURL && view.downloadableFileTypes[data.type];
-	    me.lookup('downloadBtn').setDisabled(!canDownload);
+	    let zipBtn = me.lookup('downloadBtn');
+	    let tarBtn = me.lookup('downloadTar');
+	    zipBtn.setDisabled(!canDownload);
+	    tarBtn.setDisabled(!canDownload);
+	    zipBtn.setText(data.type === 'd' ? gettext('Download .zip') : gettext('Download'));
+	    tarBtn.setVisible(data.type === 'd' && view.enableTar);
 	},
 
 	errorHandler: function(error, msg) {
 	    let me = this;
 	    me.lookup('downloadBtn').setDisabled(true);
+	    me.lookup('downloadTar').setDisabled(true);
 	    if (me.initialLoadDone) {
 		Ext.Msg.alert(gettext('Error'), msg);
 		return true;
@@ -245,8 +267,15 @@ Ext.define("Proxmox.window.FileBrowser", {
 
     buttons: [
 	{
-	    text: gettext('Download'),
-	    handler: 'downloadFile',
+	    text: gettext('Download .tar.zst'),
+	    handler: 'downloadTar',
+	    reference: 'downloadTar',
+	    hidden: true,
+	    disabled: true,
+	},
+	{
+	    text: gettext('Download .zip'),
+	    handler: 'downloadZip',
 	    reference: 'downloadBtn',
 	    disabled: true,
 	},
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 1/3] pbs-client: add 'create_tar' helper function

[Dominik Csapak]

similar to create_zip, uses an accessor to write a tar into an output
that implements AsyncWrite, but we use a Decoder to iterate instead
of having a recursive function. This is done so that we get the
entries in the correct order, and it should be faster as well.

Includes files, directories, symlinks, hardlink, block/char devs, fifos
into the tar. If the hardlink points to outside the current dir to
archive, promote the first instance to a 'real' file, and use a
hardlink for the rest.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 pbs-client/Cargo.toml          |   1 +
 pbs-client/src/pxar/extract.rs | 211 ++++++++++++++++++++++++++++++++-
 pbs-client/src/pxar/mod.rs     |   2 +-
 3 files changed, 211 insertions(+), 3 deletions(-)

diff --git a/pbs-client/Cargo.toml b/pbs-client/Cargo.toml
index d713a3ca..68a777b0 100644
--- a/pbs-client/Cargo.toml
+++ b/pbs-client/Cargo.toml
@@ -27,6 +27,7 @@ tokio = { version = "1.6", features = [ "fs", "signal" ] }
 tokio-stream = "0.1.0"
 tower-service = "0.3.0"
 xdg = "2.2"
+tar = "0.4"
 
 pathpatterns = "0.1.2"
 
diff --git a/pbs-client/src/pxar/extract.rs b/pbs-client/src/pxar/extract.rs
index b1f8718e..a0efcbe4 100644
--- a/pbs-client/src/pxar/extract.rs
+++ b/pbs-client/src/pxar/extract.rs
@@ -1,9 +1,10 @@
 //! Code for extraction of pxar contents onto the file system.
 
+use std::collections::HashMap;
 use std::convert::TryFrom;
 use std::ffi::{CStr, CString, OsStr, OsString};
 use std::io;
-use std::os::unix::ffi::OsStrExt;
+use std::os::unix::ffi::{OsStrExt, OsStringExt};
 use std::os::unix::io::{AsRawFd, FromRawFd, RawFd};
 use std::path::{Path, PathBuf};
 use std::sync::{Arc, Mutex};
@@ -17,7 +18,7 @@ use nix::sys::stat::Mode;
 
 use pathpatterns::{MatchEntry, MatchList, MatchType};
 use pxar::accessor::aio::{Accessor, FileContents, FileEntry};
-use pxar::decoder::aio::Decoder;
+use pxar::decoder::{aio::Decoder, Contents};
 use pxar::format::Device;
 use pxar::{Entry, EntryKind, Metadata};
 
@@ -501,6 +502,212 @@ impl Extractor {
     }
 }
 
+fn add_metadata_to_header(header: &mut tar::Header, metadata: &Metadata) {
+    header.set_mode(metadata.stat.mode as u32);
+    header.set_mtime(metadata.stat.mtime.secs as u64);
+    header.set_uid(metadata.stat.uid as u64);
+    header.set_gid(metadata.stat.gid as u64);
+}
+
+async fn tar_add_file<'a, W, T>(
+    tar: &mut proxmox_compression::tar::Builder<W>,
+    contents: Option<Contents<'a, T>>,
+    size: u64,
+    metadata: &Metadata,
+    path: &Path,
+) -> Result<(), Error>
+where
+    T: pxar::decoder::SeqRead + Unpin + Send + Sync + 'static,
+    W: tokio::io::AsyncWrite + Unpin + Send + 'static,
+{
+    let mut header = tar::Header::new_gnu();
+    header.set_entry_type(tar::EntryType::Regular);
+    header.set_size(size);
+    add_metadata_to_header(&mut header, metadata);
+    header.set_cksum();
+    match contents {
+        Some(content) => tar.add_entry(&mut header, path, content).await,
+        None => tar.add_entry(&mut header, path, tokio::io::empty()).await,
+    }
+    .map_err(|err| format_err!("could not send file entry: {}", err))?;
+    Ok(())
+}
+
+// converts to a pathbuf and removes the trailing '\0'
+fn link_to_pathbuf(link: &[u8]) -> PathBuf {
+    let len = link.len();
+    let mut buf = Vec::with_capacity(len);
+    buf.extend_from_slice(&link[..len - 1]);
+    OsString::from_vec(buf).into()
+}
+
+/// Creates a tar file from `path` and writes it into `output`
+pub async fn create_tar<T, W, P>(
+    output: W,
+    accessor: Accessor<T>,
+    path: P,
+    verbose: bool,
+) -> Result<(), Error>
+where
+    T: Clone + pxar::accessor::ReadAt + Unpin + Send + Sync + 'static,
+    W: tokio::io::AsyncWrite + Unpin + Send + 'static,
+    P: AsRef<Path>,
+{
+    let root = accessor.open_root().await?;
+    let file = root
+        .lookup(&path)
+        .await?
+        .ok_or(format_err!("error opening '{:?}'", path.as_ref()))?;
+
+    let mut prefix = PathBuf::new();
+    let mut components = file.entry().path().components();
+    components.next_back(); // discard last
+    for comp in components {
+        prefix.push(comp);
+    }
+
+    let mut tarencoder = proxmox_compression::tar::Builder::new(output);
+    let mut hardlinks: HashMap<PathBuf, PathBuf> = HashMap::new();
+
+    if let Ok(dir) = file.enter_directory().await {
+        let mut decoder = dir.decode_full().await?;
+        decoder.enable_goodbye_entries(false);
+        while let Some(entry) = decoder.next().await {
+            let entry = entry.map_err(|err| format_err!("cannot decode entry: {}", err))?;
+
+            let metadata = entry.metadata();
+            let path = entry.path().strip_prefix(&prefix)?.to_path_buf();
+
+            match entry.kind() {
+                EntryKind::File { .. } => {
+                    let size = decoder.content_size().unwrap_or(0);
+                    tar_add_file(&mut tarencoder, decoder.contents(), size, &metadata, &path)
+                        .await?
+                }
+                EntryKind::Hardlink(link) => {
+                    if !link.data.is_empty() {
+                        let entry = root
+                            .lookup(&path)
+                            .await?
+                            .ok_or(format_err!("error looking up '{:?}'", path))?;
+                        let realfile = accessor.follow_hardlink(&entry).await?;
+                        let metadata = realfile.entry().metadata();
+                        let realpath = link_to_pathbuf(&link.data);
+
+                        if verbose {
+                            eprintln!("adding '{}' to tar", path.display());
+                        }
+
+                        let stripped_path = match realpath.strip_prefix(&prefix) {
+                            Ok(path) => path,
+                            Err(_) => {
+                                // outside of our tar archive, add the first occurrance to the tar
+                                if let Some(path) = hardlinks.get(&realpath) {
+                                    path
+                                } else {
+                                    let size = decoder.content_size().unwrap_or(0);
+                                    tar_add_file(
+                                        &mut tarencoder,
+                                        decoder.contents(),
+                                        size,
+                                        metadata,
+                                        &path,
+                                    )
+                                    .await?;
+                                    hardlinks.insert(realpath, path);
+                                    continue;
+                                }
+                            }
+                        };
+                        let mut header = tar::Header::new_gnu();
+                        header.set_entry_type(tar::EntryType::Link);
+                        add_metadata_to_header(&mut header, metadata);
+                        header.set_size(0);
+                        tarencoder
+                            .add_link(&mut header, path, stripped_path)
+                            .await
+                            .map_err(|err| format_err!("could not send hardlink entry: {}", err))?;
+                    }
+                }
+                EntryKind::Symlink(link) if !link.data.is_empty() => {
+                    if verbose {
+                        eprintln!("adding '{}' to tar", path.display());
+                    }
+                    let realpath = link_to_pathbuf(&link.data);
+                    let mut header = tar::Header::new_gnu();
+                    header.set_entry_type(tar::EntryType::Symlink);
+                    add_metadata_to_header(&mut header, metadata);
+                    header.set_size(0);
+                    tarencoder
+                        .add_link(&mut header, path, realpath)
+                        .await
+                        .map_err(|err| format_err!("could not send symlink entry: {}", err))?;
+                }
+                EntryKind::Fifo => {
+                    if verbose {
+                        eprintln!("adding '{}' to tar", path.display());
+                    }
+                    let mut header = tar::Header::new_gnu();
+                    header.set_entry_type(tar::EntryType::Fifo);
+                    add_metadata_to_header(&mut header, metadata);
+                    header.set_size(0);
+                    header.set_device_major(0)?;
+                    header.set_device_minor(0)?;
+                    header.set_cksum();
+                    tarencoder
+                        .add_entry(&mut header, path, tokio::io::empty())
+                        .await
+                        .map_err(|err| format_err!("could not send fifo entry: {}", err))?;
+                }
+                EntryKind::Directory => {
+                    if verbose {
+                        eprintln!("adding '{}' to tar", path.display());
+                    }
+                    // we cannot add the root path itself
+                    if path != Path::new("/") {
+                        let mut header = tar::Header::new_gnu();
+                        header.set_entry_type(tar::EntryType::Directory);
+                        add_metadata_to_header(&mut header, metadata);
+                        header.set_size(0);
+                        header.set_cksum();
+                        tarencoder
+                            .add_entry(&mut header, path, tokio::io::empty())
+                            .await
+                            .map_err(|err| format_err!("could not send dir entry: {}", err))?;
+                    }
+                }
+                EntryKind::Device(device) => {
+                    if verbose {
+                        eprintln!("adding '{}' to tar", path.display());
+                    }
+                    let entry_type = if metadata.stat.is_chardev() {
+                        tar::EntryType::Char
+                    } else {
+                        tar::EntryType::Block
+                    };
+                    let mut header = tar::Header::new_gnu();
+                    header.set_entry_type(entry_type);
+                    header.set_device_major(device.major as u32)?;
+                    header.set_device_minor(device.minor as u32)?;
+                    add_metadata_to_header(&mut header, metadata);
+                    header.set_size(0);
+                    tarencoder
+                        .add_entry(&mut header, path, tokio::io::empty())
+                        .await
+                        .map_err(|err| format_err!("could not send device entry: {}", err))?;
+                }
+                _ => {} // ignore all else
+            }
+        }
+    }
+
+    tarencoder.finish().await.map_err(|err| {
+        eprintln!("error during finishing of zip: {}", err);
+        err
+    })?;
+    Ok(())
+}
+
 pub async fn create_zip<T, W, P>(
     output: W,
     decoder: Accessor<T>,
diff --git a/pbs-client/src/pxar/mod.rs b/pbs-client/src/pxar/mod.rs
index f20a1f9e..725fc2d9 100644
--- a/pbs-client/src/pxar/mod.rs
+++ b/pbs-client/src/pxar/mod.rs
@@ -59,7 +59,7 @@ pub use flags::Flags;
 
 pub use create::{create_archive, PxarCreateOptions};
 pub use extract::{
-    create_zip, extract_archive, extract_sub_dir, extract_sub_dir_seq, ErrorHandler,
+    create_tar, create_zip, extract_archive, extract_sub_dir, extract_sub_dir_seq, ErrorHandler,
     PxarExtractOptions,
 };
 
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 2/3] api: admin/datastore: add tar support for pxar_file_download

[Dominik Csapak]

by using the newly added 'create_tar' and the 'ZstdEncoder'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 Cargo.toml                  |  1 +
 src/api2/admin/datastore.rs | 33 ++++++++++++++++++++++++---------
 2 files changed, 25 insertions(+), 9 deletions(-)

diff --git a/Cargo.toml b/Cargo.toml
index bd21117a..c7677c33 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -105,6 +105,7 @@ proxmox-uuid = "1"
 proxmox-serde = "0.1"
 proxmox-shared-memory = "0.2"
 proxmox-sys = { version = "0.2", features = [ "sortable-macro" ] }
+proxmox-compression = "0.1"
 
 
 proxmox-acme-rs = "0.4"
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index ef82b426..02e92640 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -12,6 +12,7 @@ use hyper::{header, Body, Response, StatusCode};
 use serde_json::{json, Value};
 use tokio_stream::wrappers::ReceiverStream;
 
+use proxmox_compression::zstd::ZstdEncoder;
 use proxmox_sys::sortable;
 use proxmox_sys::fs::{
     file_read_firstline, file_read_optional_string, replace_file, CreateOptions,
@@ -40,7 +41,7 @@ use pbs_api_types::{ Authid, BackupContent, Counts, CryptMode,
     PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_VERIFY,
 
 };
-use pbs_client::pxar::create_zip;
+use pbs_client::pxar::{create_tar, create_zip};
 use pbs_datastore::{
     check_backup_owner, DataStore, BackupDir, BackupGroup, StoreProgress, LocalChunkReader,
     CATALOG_NAME,
@@ -1432,6 +1433,7 @@ pub const API_METHOD_PXAR_FILE_DOWNLOAD: ApiMethod = ApiMethod::new(
             ("backup-id", false,  &BACKUP_ID_SCHEMA),
             ("backup-time", false, &BACKUP_TIME_SCHEMA),
             ("filepath", false, &StringSchema::new("Base64 encoded path").schema()),
+            ("tar", true, &BooleanSchema::new("Download as .tar.zst").schema()),
         ]),
     )
 ).access(None, &Permission::Privilege(
@@ -1460,6 +1462,8 @@ pub fn pxar_file_download(
         let backup_id = required_string_param(&param, "backup-id")?;
         let backup_time = required_integer_param(&param, "backup-time")?;
 
+        let tar = param["tar"].as_bool().unwrap_or(false);
+
         let backup_dir = BackupDir::new(backup_type, backup_id, backup_time)?;
 
         check_priv_or_backup_owner(&datastore, backup_dir.group(), &auth_id, PRIV_DATASTORE_READ)?;
@@ -1519,15 +1523,26 @@ pub fn pxar_file_download(
                     }),
             ),
             EntryKind::Directory => {
-                let (sender, receiver) = tokio::sync::mpsc::channel(100);
+                let (sender, receiver) = tokio::sync::mpsc::channel::<Result<_, Error>>(100);
                 let channelwriter = AsyncChannelWriter::new(sender, 1024 * 1024);
-                proxmox_rest_server::spawn_internal_task(
-                    create_zip(channelwriter, decoder, path.clone(), false)
-                );
-                Body::wrap_stream(ReceiverStream::new(receiver).map_err(move |err| {
-                    eprintln!("error during streaming of zip '{:?}' - {}", path, err);
-                    err
-                }))
+                if tar {
+                    proxmox_rest_server::spawn_internal_task(
+                        create_tar(channelwriter, decoder, path.clone(), false)
+                    );
+                    let zstdstream = ZstdEncoder::new(ReceiverStream::new(receiver))?;
+                    Body::wrap_stream(zstdstream.map_err(move |err| {
+                        eprintln!("error during streaming of tar.zst '{:?}' - {}", path, err);
+                        err
+                    }))
+                } else {
+                    proxmox_rest_server::spawn_internal_task(
+                        create_zip(channelwriter, decoder, path.clone(), false)
+                    );
+                    Body::wrap_stream(ReceiverStream::new(receiver).map_err(move |err| {
+                        eprintln!("error during streaming of zip '{:?}' - {}", path, err);
+                        err
+                    }))
+                }
             }
             other => bail!("cannot download file of type {:?}", other),
         };
-- 
2.30.2

[pbs-devel] [PATCH proxmox-backup 3/3] ui: datastore/Content: enable tar download in ui

[Dominik Csapak]

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 www/datastore/Content.js | 1 +
 1 file changed, 1 insertion(+)

diff --git a/www/datastore/Content.js b/www/datastore/Content.js
index d0498bc7..1be63e0c 100644
--- a/www/datastore/Content.js
+++ b/www/datastore/Content.js
@@ -618,6 +618,7 @@ Ext.define('PBS.DataStoreContent', {
 		title: `${type}/${id}/${timetext}`,
 		listURL: `/api2/json/admin/datastore/${view.datastore}/catalog`,
 		downloadURL: `/api2/json/admin/datastore/${view.datastore}/pxar-file-download`,
+		enableTar: true,
 		extraParams: {
 		    'backup-id': id,
 		    'backup-time': (time.getTime()/1000).toFixed(0),
-- 
2.30.2

[pbs-devel] applied-series: [PATCH proxmox 1/2] proxmox-compression: add async tar builder

[Wolfgang Bumiller]

applied series for proxmox-compression crate

[pbs-devel] applied-series: [PATCH proxmox-backup 1/3] pbs-client: add 'create_tar' helper function

[Wolfgang Bumiller]

applied pbs series with some followup fixups

[pbs-devel] applied: [PATCH widget-toolkit 1/1] window/FileBrowser: add optional 'tar.zst' button

[Wolfgang Bumiller]

applied