[pve-devel] [PATCH proxmox-backup/common/storage/wt v2] add tar.zst download in pve

[pve-devel] [PATCH proxmox-backup/common/storage/wt v2] add tar.zst download in pve

[Dominik Csapak]

like we have the tar.zst download button for pbs itself, add it for
pve for both vms and container file-restore

pve-storage depends on pve-common, which depends on proxmox-backup,
(we must add the recently bumped proxmox-compression as a
dependency; also for the restore-daemon)

changes from v1:
* split format into 'format' and 'zstd'
* also use those parameters for file-restore
  (keep 'tar' for the pve api for gui compatibility)
* use an '#[api]'  enum for the format type

i kept the 'tar' for the pve api, since that way it makes it easily
compatible with the current gui. I'd understand that we maybe also
here want to have the 'format'/'zst' options and simply pass them
through

in that case, we "have" to adapt the pbs-api too since there we already
have the 'tar' parameter, or we have  to have two codepaths for the gui
depending which api we want to use.

No problem for a v3 either way, there just was not really an feedback
on that in my previous versions.

proxmox-backup:

Dominik Csapak (4):
  file-restore: update dependencies
  pbs-api-types: add FileRestoreFormat type
  restore-daemon: add 'format' and 'zstd' parameters to the 'extract'
    handler
  file-restore: add 'format' and 'zstd' parameters to 'extract' command

 pbs-api-types/src/file_restore.rs             |  15 +++
 proxmox-file-restore/Cargo.toml               |   4 +-
 proxmox-file-restore/src/block_driver.rs      |  12 ++-
 proxmox-file-restore/src/block_driver_qemu.rs |  15 +--
 proxmox-file-restore/src/main.rs              | 100 +++++++++++++++---
 .../src/proxmox_restore_daemon/api.rs         |  46 ++++++--
 6 files changed, 157 insertions(+), 35 deletions(-)

pve-common:

Dominik Csapak (1):
  PBSClient: add 'tar' parameter to file_restore_extract

 src/PVE/PBSClient.pm | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

pve-storage:

Dominik Csapak (1):
  api/filerestore: add 'tar' parameter to 'download' api

 PVE/API2/Storage/FileRestore.pm | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

proxmox-widget-toolkit:

Dominik Csapak (1):
  window/FileBrowser: enable tar button by default

 src/window/FileBrowser.js | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

-- 
2.30.2

[pve-devel] [PATCH proxmox-backup v2 1/4] file-restore: update dependencies

[Dominik Csapak]

logrotate is now a feature of proxmox-sys
proxmox-compression is it's own crate now

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 proxmox-file-restore/Cargo.toml | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/proxmox-file-restore/Cargo.toml b/proxmox-file-restore/Cargo.toml
index 6342c4e3..9ffac708 100644
--- a/proxmox-file-restore/Cargo.toml
+++ b/proxmox-file-restore/Cargo.toml
@@ -24,7 +24,8 @@ proxmox-router = { version = "1.2.4", features = [ "cli" ] }
 proxmox-schema = { version = "1.3.1", features = [ "api-macro" ] }
 proxmox-time = "1"
 proxmox-uuid = "1"
-proxmox-sys = "0.3.1"
+proxmox-sys = { version = "0.3.1", features = [ "logrotate" ] }
+proxmox-compression = "0.1"
 
 pbs-api-types = { path = "../pbs-api-types" }
 pbs-buildcfg = { path = "../pbs-buildcfg" }
-- 
2.30.2

Re: [pve-devel] [pbs-devel] [PATCH proxmox-backup v2 1/4] file-restore: update dependencies

[Thomas Lamprecht]

Am 13/07/2022 um 11:43 schrieb Dominik Csapak:
> logrotate is now a feature of proxmox-sys

Ok, but depreacated as Fabian patched this directly already

> proxmox-compression is it's own crate now

this is not yet used so makes no sense to have it in the same
patch as an unrelated fix, it belongs in patch 3/4 (fixed up).

> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
>  proxmox-file-restore/Cargo.toml | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/proxmox-file-restore/Cargo.toml b/proxmox-file-restore/Cargo.toml
> index 6342c4e3..9ffac708 100644
> --- a/proxmox-file-restore/Cargo.toml
> +++ b/proxmox-file-restore/Cargo.toml
> @@ -24,7 +24,8 @@ proxmox-router = { version = "1.2.4", features = [ "cli" ] }
>  proxmox-schema = { version = "1.3.1", features = [ "api-macro" ] }
>  proxmox-time = "1"
>  proxmox-uuid = "1"
> -proxmox-sys = "0.3.1"
> +proxmox-sys = { version = "0.3.1", features = [ "logrotate" ] }
> +proxmox-compression = "0.1"
>  
>  pbs-api-types = { path = "../pbs-api-types" }
>  pbs-buildcfg = { path = "../pbs-buildcfg" }

Re: [pve-devel] [pbs-devel] [PATCH proxmox-backup v2 1/4] file-restore: update dependencies

[Thomas Lamprecht]

Am 05/10/2022 um 18:51 schrieb Thomas Lamprecht:
>  it belongs in patch 3/4 (fixed up).

4/4 sorry

[pve-devel] [PATCH proxmox-backup v2 2/4] pbs-api-types: add FileRestoreFormat type

[Dominik Csapak]

intended for passing the format to the file-restore client/daemon

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 pbs-api-types/src/file_restore.rs | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/pbs-api-types/src/file_restore.rs b/pbs-api-types/src/file_restore.rs
index 5748f3a7..90657d65 100644
--- a/pbs-api-types/src/file_restore.rs
+++ b/pbs-api-types/src/file_restore.rs
@@ -13,3 +13,18 @@ pub struct RestoreDaemonStatus {
     /// not set, as then the status call will have reset the timer before returning the value
     pub timeout: i64,
 }
+
+#[api]
+#[derive(Serialize, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "kebab-case")]
+/// The desired format of the result.
+pub enum FileRestoreFormat {
+    /// Plain file (only works for single files)
+    Plain,
+    /// PXAR archive
+    Pxar,
+    /// ZIP archive
+    Zip,
+    /// TAR archive
+    Tar,
+}
-- 
2.30.2

[pve-devel] applied: [pbs-devel] [PATCH proxmox-backup v2 2/4] pbs-api-types: add FileRestoreFormat type

[Thomas Lamprecht]

Am 13/07/2022 um 11:43 schrieb Dominik Csapak:
> intended for passing the format to the file-restore client/daemon
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
>  pbs-api-types/src/file_restore.rs | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
>

applied, thanks!

[pve-devel] applied: [pbs-devel] [PATCH proxmox-backup v2 2/4] pbs-api-types: add FileRestoreFormat type

[Thomas Lamprecht]

Am 13/07/2022 um 11:43 schrieb Dominik Csapak:
> intended for passing the format to the file-restore client/daemon
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
>  pbs-api-types/src/file_restore.rs | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
>

applied, thanks!

[pve-devel] [PATCH proxmox-backup v2 3/4] restore-daemon: add 'format' and 'zstd' parameters to the 'extract' handler

[Dominik Csapak]

'format' can be 'plain', 'pxar', 'zip' or 'tar',  and it returns the
content in the given format (with fallback to the old behaviour if not
given)

the 'zstd' denotes if the output should be zstd compressed

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 .../src/proxmox_restore_daemon/api.rs         | 46 +++++++++++++++----
 1 file changed, 38 insertions(+), 8 deletions(-)

diff --git a/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs b/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
index 3cc9c370..dd2a13cf 100644
--- a/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
+++ b/proxmox-restore-daemon/src/proxmox_restore_daemon/api.rs
@@ -13,7 +13,7 @@ use serde_json::Value;
 use tokio::sync::Semaphore;
 
 use pathpatterns::{MatchEntry, MatchPattern, MatchType, Pattern};
-use proxmox_compression::zip::zip_directory;
+use proxmox_compression::{tar::tar_directory, zip::zip_directory, zstd::ZstdEncoder};
 use proxmox_router::{
     list_subdirs_api_method, ApiHandler, ApiMethod, ApiResponseFuture, Permission, Router,
     RpcEnvironment, SubdirMap,
@@ -22,7 +22,7 @@ use proxmox_schema::*;
 use proxmox_sys::fs::read_subdir;
 use proxmox_sys::sortable;
 
-use pbs_api_types::file_restore::RestoreDaemonStatus;
+use pbs_api_types::file_restore::{FileRestoreFormat, RestoreDaemonStatus};
 use pbs_client::pxar::{create_archive, Flags, PxarCreateOptions, ENCODER_MAX_ENTRIES};
 use pbs_datastore::catalog::{ArchiveEntry, DirEntryAttribute};
 use pbs_tools::json::required_string_param;
@@ -237,11 +237,19 @@ pub const API_METHOD_EXTRACT: ApiMethod = ApiMethod::new(
                 true,
                 &BooleanSchema::new(concat!(
                     "if true, return a pxar archive, otherwise either the ",
-                    "file content or the directory as a zip file"
+                    "file content or the directory as a zip file. DEPRECATED: use 'format' instead."
                 ))
                 .default(true)
                 .schema()
-            )
+            ),
+            ("format", true, &FileRestoreFormat::API_SCHEMA,),
+            (
+                "zstd",
+                true,
+                &BooleanSchema::new(concat!("if true, zstd compresses the result.",))
+                    .default(false)
+                    .schema()
+            ),
         ]),
     ),
 )
@@ -271,7 +279,13 @@ fn extract(
         }
         let path = Path::new(OsStr::from_bytes(&path[..]));
 
-        let pxar = param["pxar"].as_bool().unwrap_or(true);
+        let format = match (param["format"].as_str(), param["pxar"].as_bool()) {
+            (Some(format), None) => format.to_string(),
+            (Some(_), Some(_)) => bail!("cannot set 'pxar' and 'format' simultaneously"),
+            // FIXME, pxar 'false' defaulted to either zip or plain, remove with 3.0
+            (None, Some(false) | None) => String::new(),
+            (None, Some(true)) => "pxar".to_string(),
+        };
 
         let query_result = proxmox_async::runtime::block_in_place(move || {
             let mut disk_state = crate::DISK_STATE.lock().unwrap();
@@ -291,7 +305,7 @@ fn extract(
 
         let (mut writer, reader) = tokio::io::duplex(1024 * 64);
 
-        if pxar {
+        if format == "pxar" {
             tokio::spawn(async move {
                 let _inhibitor = _inhibitor;
                 let _permit = _permit;
@@ -349,12 +363,23 @@ fn extract(
                     error!("pxar streaming task failed - {}", err);
                 }
             });
+        } else if format == "tar" {
+            tokio::spawn(async move {
+                let _inhibitor = _inhibitor;
+                let _permit = _permit;
+                if let Err(err) = tar_directory(&mut writer, &vm_path).await {
+                    error!("file or dir streaming task failed - {}", err);
+                }
+            });
         } else {
+            if format == "plain" && vm_path.is_dir() {
+                bail!("cannot stream dir with format 'plain'");
+            }
             tokio::spawn(async move {
                 let _inhibitor = _inhibitor;
                 let _permit = _permit;
                 let result = async move {
-                    if vm_path.is_dir() {
+                    if vm_path.is_dir() || format == "zip" {
                         zip_directory(&mut writer, &vm_path).await?;
                         Ok(())
                     } else if vm_path.is_file() {
@@ -377,7 +402,12 @@ fn extract(
 
         let stream = tokio_util::io::ReaderStream::new(reader);
 
-        let body = Body::wrap_stream(stream);
+        let body = if param["zstd"].as_bool().unwrap_or(false) {
+            let stream = ZstdEncoder::new(stream)?;
+            Body::wrap_stream(stream)
+        } else {
+            Body::wrap_stream(stream)
+        };
         Ok(Response::builder()
             .status(StatusCode::OK)
             .header(header::CONTENT_TYPE, "application/octet-stream")
-- 
2.30.2

[pve-devel] applied: [pbs-devel] [PATCH proxmox-backup v2 3/4] restore-daemon: add 'format' and 'zstd' parameters to the 'extract' handler

[Thomas Lamprecht]

Am 13/07/2022 um 11:43 schrieb Dominik Csapak:
> 'format' can be 'plain', 'pxar', 'zip' or 'tar',  and it returns the
> content in the given format (with fallback to the old behaviour if not
> given)
> 
> the 'zstd' denotes if the output should be zstd compressed
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
>  .../src/proxmox_restore_daemon/api.rs         | 46 +++++++++++++++----
>  1 file changed, 38 insertions(+), 8 deletions(-)
> 
>

applied, thanks!

[pve-devel] [PATCH proxmox-backup v2 4/4] file-restore: add 'format' and 'zstd' parameters to 'extract' command

[Dominik Csapak]

if the target ist stdout, we can now specify the exact format by making use of
the new 'format' parameter of the restore daemons 'extract' api

note that extracting a pxar from a source pxar (container/host backups)
won't work currently since we would have to reencode as pxar first

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 proxmox-file-restore/Cargo.toml               |   1 +
 proxmox-file-restore/src/block_driver.rs      |  12 ++-
 proxmox-file-restore/src/block_driver_qemu.rs |  15 +--
 proxmox-file-restore/src/main.rs              | 100 +++++++++++++++---
 4 files changed, 102 insertions(+), 26 deletions(-)

diff --git a/proxmox-file-restore/Cargo.toml b/proxmox-file-restore/Cargo.toml
index 9ffac708..c0daf530 100644
--- a/proxmox-file-restore/Cargo.toml
+++ b/proxmox-file-restore/Cargo.toml
@@ -14,6 +14,7 @@ log = "0.4"
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
 tokio = { version = "1.6", features = [ "io-std", "rt", "rt-multi-thread", "time" ] }
+tokio-util = { version = "0.7", features = ["io"] }
 
 pxar = { version = "0.10.1", features = [ "tokio-io" ] }
 
diff --git a/proxmox-file-restore/src/block_driver.rs b/proxmox-file-restore/src/block_driver.rs
index eb6de82c..3c774e5d 100644
--- a/proxmox-file-restore/src/block_driver.rs
+++ b/proxmox-file-restore/src/block_driver.rs
@@ -11,7 +11,7 @@ use serde_json::{json, Value};
 use proxmox_router::cli::*;
 use proxmox_schema::api;
 
-use pbs_api_types::{BackupDir, BackupNamespace};
+use pbs_api_types::{file_restore::FileRestoreFormat, BackupDir, BackupNamespace};
 use pbs_client::BackupRepository;
 use pbs_datastore::catalog::ArchiveEntry;
 use pbs_datastore::manifest::BackupManifest;
@@ -55,7 +55,8 @@ pub trait BlockRestoreDriver {
         details: SnapRestoreDetails,
         img_file: String,
         path: Vec<u8>,
-        pxar: bool,
+        format: Option<FileRestoreFormat>,
+        zstd: bool,
     ) -> Async<Result<Box<dyn tokio::io::AsyncRead + Unpin + Send>, Error>>;
 
     /// Return status of all running/mapped images, result value is (id, extra data), where id must
@@ -101,10 +102,13 @@ pub async fn data_extract(
     details: SnapRestoreDetails,
     img_file: String,
     path: Vec<u8>,
-    pxar: bool,
+    format: Option<FileRestoreFormat>,
+    zstd: bool,
 ) -> Result<Box<dyn tokio::io::AsyncRead + Send + Unpin>, Error> {
     let driver = driver.unwrap_or(DEFAULT_DRIVER).resolve();
-    driver.data_extract(details, img_file, path, pxar).await
+    driver
+        .data_extract(details, img_file, path, format, zstd)
+        .await
 }
 
 #[api(
diff --git a/proxmox-file-restore/src/block_driver_qemu.rs b/proxmox-file-restore/src/block_driver_qemu.rs
index 55d15e8d..736ae2fd 100644
--- a/proxmox-file-restore/src/block_driver_qemu.rs
+++ b/proxmox-file-restore/src/block_driver_qemu.rs
@@ -10,7 +10,7 @@ use serde_json::json;
 
 use proxmox_sys::fs::lock_file;
 
-use pbs_api_types::{BackupDir, BackupNamespace};
+use pbs_api_types::{file_restore::FileRestoreFormat, BackupDir, BackupNamespace};
 use pbs_client::{BackupRepository, VsockClient, DEFAULT_VSOCK_PORT};
 use pbs_datastore::catalog::ArchiveEntry;
 
@@ -217,7 +217,8 @@ impl BlockRestoreDriver for QemuBlockDriver {
         details: SnapRestoreDetails,
         img_file: String,
         mut path: Vec<u8>,
-        pxar: bool,
+        format: Option<FileRestoreFormat>,
+        zstd: bool,
     ) -> Async<Result<Box<dyn tokio::io::AsyncRead + Unpin + Send>, Error>> {
         async move {
             let client = ensure_running(&details).await?;
@@ -226,13 +227,13 @@ impl BlockRestoreDriver for QemuBlockDriver {
             }
             let path = base64::encode(img_file.bytes().chain(path).collect::<Vec<u8>>());
             let (mut tx, rx) = tokio::io::duplex(1024 * 4096);
+            let mut data = json!({ "path": path, "zstd": zstd });
+            if let Some(format) = format {
+                data["format"] = serde_json::to_value(format)?;
+            }
             tokio::spawn(async move {
                 if let Err(err) = client
-                    .download(
-                        "api2/json/extract",
-                        Some(json!({ "path": path, "pxar": pxar })),
-                        &mut tx,
-                    )
+                    .download("api2/json/extract", Some(data), &mut tx)
                     .await
                 {
                     log::error!("reading file extraction stream failed - {}", err);
diff --git a/proxmox-file-restore/src/main.rs b/proxmox-file-restore/src/main.rs
index 562c8ca7..d5deb44a 100644
--- a/proxmox-file-restore/src/main.rs
+++ b/proxmox-file-restore/src/main.rs
@@ -4,8 +4,11 @@ use std::path::PathBuf;
 use std::sync::Arc;
 
 use anyhow::{bail, format_err, Error};
+use futures::StreamExt;
 use serde_json::{json, Value};
+use tokio::io::AsyncWriteExt;
 
+use proxmox_compression::zstd::ZstdEncoder;
 use proxmox_router::cli::{
     complete_file_name, default_table_format_options, format_and_print_result_full,
     get_output_format, init_cli_logger, run_cli_command, CliCommand, CliCommandMap, CliEnvironment,
@@ -17,8 +20,8 @@ use proxmox_sys::fs::{create_path, CreateOptions};
 use pxar::accessor::aio::Accessor;
 use pxar::decoder::aio::Decoder;
 
-use pbs_api_types::{BackupDir, BackupNamespace, CryptMode};
-use pbs_client::pxar::{create_zip, extract_sub_dir, extract_sub_dir_seq};
+use pbs_api_types::{file_restore::FileRestoreFormat, BackupDir, BackupNamespace, CryptMode};
+use pbs_client::pxar::{create_tar, create_zip, extract_sub_dir, extract_sub_dir_seq};
 use pbs_client::tools::{
     complete_group_or_snapshot, complete_repository, connect, extract_repository_from_value,
     key_source::{
@@ -346,9 +349,19 @@ async fn list(
                 description: "Group/Snapshot path.",
             },
             "path": {
-                description: "Path to restore. Directories will be restored as .zip files if extracted to stdout.",
+                description: "Path to restore. Directories will be restored as archive files if extracted to stdout.",
                 type: String,
             },
+            "format": {
+                type: FileRestoreFormat,
+                optional: true,
+            },
+            "zstd": {
+                type: bool,
+                description: "If true, output will be zstd compressed.",
+                optional: true,
+                default: false,
+            },
             "base64": {
                 type: Boolean,
                 description: "If set, 'path' will be interpreted as base64 encoded.",
@@ -392,6 +405,8 @@ async fn extract(
     path: String,
     base64: bool,
     target: Option<String>,
+    format: Option<FileRestoreFormat>,
+    zstd: bool,
     param: Value,
 ) -> Result<(), Error> {
     let repo = extract_repository_from_value(&param)?;
@@ -450,7 +465,7 @@ async fn extract(
             let archive_size = reader.archive_size();
             let reader = LocalDynamicReadAt::new(reader);
             let decoder = Accessor::new(reader, archive_size).await?;
-            extract_to_target(decoder, &path, target).await?;
+            extract_to_target(decoder, &path, target, format, zstd).await?;
         }
         ExtractPath::VM(file, path) => {
             let details = SnapRestoreDetails {
@@ -466,7 +481,15 @@ async fn extract(
             };
 
             if let Some(mut target) = target {
-                let reader = data_extract(driver, details, file, path.clone(), true).await?;
+                let reader = data_extract(
+                    driver,
+                    details,
+                    file,
+                    path.clone(),
+                    Some(FileRestoreFormat::Pxar),
+                    false,
+                )
+                .await?;
                 let decoder = Decoder::from_tokio(reader).await?;
                 extract_sub_dir_seq(&target, decoder).await?;
 
@@ -477,7 +500,8 @@ async fn extract(
                     format_err!("unable to remove temporary .pxarexclude-cli file - {}", e)
                 })?;
             } else {
-                let mut reader = data_extract(driver, details, file, path.clone(), false).await?;
+                let mut reader =
+                    data_extract(driver, details, file, path.clone(), format, zstd).await?;
                 tokio::io::copy(&mut reader, &mut tokio::io::stdout()).await?;
             }
         }
@@ -493,29 +517,75 @@ async fn extract_to_target<T>(
     decoder: Accessor<T>,
     path: &[u8],
     target: Option<PathBuf>,
+    format: Option<FileRestoreFormat>,
+    zstd: bool,
 ) -> Result<(), Error>
 where
     T: pxar::accessor::ReadAt + Clone + Send + Sync + Unpin + 'static,
 {
     let path = if path.is_empty() { b"/" } else { path };
+    let path = OsStr::from_bytes(path);
+
+    if let Some(target) = target {
+        extract_sub_dir(target, decoder, path).await?;
+    } else {
+        extract_archive(decoder, path, format, zstd).await?;
+    }
+
+    Ok(())
+}
 
+async fn extract_archive<T>(
+    decoder: Accessor<T>,
+    path: &OsStr,
+    format: Option<FileRestoreFormat>,
+    zstd: bool,
+) -> Result<(), Error>
+where
+    T: pxar::accessor::ReadAt + Clone + Send + Sync + Unpin + 'static,
+{
+    let path = path.to_owned();
     let root = decoder.open_root().await?;
     let file = root
-        .lookup(OsStr::from_bytes(path))
+        .lookup(&path)
         .await?
-        .ok_or_else(|| format_err!("error opening '{:?}'", path))?;
+        .ok_or_else(|| format_err!("error opening '{:?}'", &path))?;
 
-    if let Some(target) = target {
-        extract_sub_dir(target, decoder, OsStr::from_bytes(path)).await?;
+    let (mut writer, mut reader) = tokio::io::duplex(1024 * 1024);
+    if file.is_regular_file() {
+        match format {
+            Some(FileRestoreFormat::Plain) | None => {}
+            _ => bail!("cannot extract single files as archive"),
+        }
+        tokio::spawn(
+            async move { tokio::io::copy(&mut file.contents().await?, &mut writer).await },
+        );
     } else {
-        match file.kind() {
-            pxar::EntryKind::File { .. } => {
-                tokio::io::copy(&mut file.contents().await?, &mut tokio::io::stdout()).await?;
+        match format {
+            Some(FileRestoreFormat::Pxar) => {
+                bail!("pxar target not supported for pxar source");
             }
-            _ => {
-                create_zip(tokio::io::stdout(), decoder, OsStr::from_bytes(path)).await?;
+            Some(FileRestoreFormat::Plain) => {
+                bail!("plain file not supported for non-regular files");
             }
+            Some(FileRestoreFormat::Zip) | None => {
+                tokio::spawn(create_zip(writer, decoder, path));
+            }
+            Some(FileRestoreFormat::Tar) => {
+                tokio::spawn(create_tar(writer, decoder, path));
+            }
+        }
+    }
+
+    if zstd {
+        let mut zstdstream = ZstdEncoder::new(tokio_util::io::ReaderStream::new(reader))?;
+        let mut stdout = tokio::io::stdout();
+        while let Some(buf) = zstdstream.next().await {
+            let buf = buf?;
+            stdout.write_all(&buf).await?;
         }
+    } else {
+        tokio::io::copy(&mut reader, &mut tokio::io::stdout()).await?;
     }
 
     Ok(())
-- 
2.30.2

[pve-devel] applied: [pbs-devel] [PATCH proxmox-backup v2 4/4] file-restore: add 'format' and 'zstd' parameters to 'extract' command

[Thomas Lamprecht]

Am 13/07/2022 um 11:43 schrieb Dominik Csapak:
> if the target ist stdout, we can now specify the exact format by making use of
> the new 'format' parameter of the restore daemons 'extract' api
> 
> note that extracting a pxar from a source pxar (container/host backups)
> won't work currently since we would have to reencode as pxar first
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
>  proxmox-file-restore/Cargo.toml               |   1 +
>  proxmox-file-restore/src/block_driver.rs      |  12 ++-
>  proxmox-file-restore/src/block_driver_qemu.rs |  15 +--
>  proxmox-file-restore/src/main.rs              | 100 +++++++++++++++---
>  4 files changed, 102 insertions(+), 26 deletions(-)
> 
>

applied, with the addition of the proxmox-compression dependency to Cargo.toml
squashed in, thanks!

[pve-devel] [PATCH common v2 1/1] PBSClient: add 'tar' parameter to file_restore_extract

[Dominik Csapak]

so that we can get a 'tar.zst' from proxmox-file-restore by giving
'--format tar --zstd' to the file-restore binary

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/PVE/PBSClient.pm | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/src/PVE/PBSClient.pm b/src/PVE/PBSClient.pm
index 37385d7..08e69eb 100644
--- a/src/PVE/PBSClient.pm
+++ b/src/PVE/PBSClient.pm
@@ -407,7 +407,7 @@ sub file_restore_extract_prepare {
 
 # this blocks while data is transfered, call this from a background worker
 sub file_restore_extract {
-    my ($self, $output_file, $snapshot, $filepath, $base64) = @_;
+    my ($self, $output_file, $snapshot, $filepath, $base64, $tar) = @_;
 
     (my $namespace, $snapshot) = split_namespaced_parameter($snapshot);
 
@@ -421,10 +421,15 @@ sub file_restore_extract {
 	my $fn = fileno($fh);
 	my $errfunc = sub { print $_[0], "\n"; };
 
+	my $cmd = [ $snapshot, $filepath, "-", "--base64", $base64 ? 1 : 0];
+	if ($tar) {
+	    push @$cmd, '--format', 'tar', '--zstd', 1;
+	}
+
 	return run_raw_client_cmd(
 	    $self,
             "extract",
-	    [ $snapshot, $filepath, "-", "--base64", $base64 ? 1 : 0 ],
+	    $cmd,
 	    binary => "proxmox-file-restore",
 	    namespace => $namespace,
 	    errfunc => $errfunc,
-- 
2.30.2

[pve-devel] [PATCH storage v2 1/1] api/filerestore: add 'tar' parameter to 'download' api

[Dominik Csapak]

to be able to download 'tar.zst' archives

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 PVE/API2/Storage/FileRestore.pm | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/PVE/API2/Storage/FileRestore.pm b/PVE/API2/Storage/FileRestore.pm
index 5630f52..1c44b3c 100644
--- a/PVE/API2/Storage/FileRestore.pm
+++ b/PVE/API2/Storage/FileRestore.pm
@@ -156,6 +156,12 @@ __PACKAGE__->register_method ({
 		description => 'base64-path to the directory or file to download.',
 		type => 'string',
 	    },
+	    tar => {
+		description => "Download dirs as 'tar.zst' instead of 'zip'.",
+		type => 'boolean',
+		optional => 1,
+		default => 0,
+	    },
 	},
     },
     returns => {
@@ -171,6 +177,7 @@ __PACKAGE__->register_method ({
 	my $path = extract_param($param, 'filepath');
 	my $storeid = extract_param($param, 'storage');
 	my $volid = $parse_volname_or_id->($storeid, $param->{volume});
+	my $tar = extract_param($param, 'tar') // 0;
 
 	my $cfg = PVE::Storage::config();
 	my $scfg = PVE::Storage::storage_config($cfg, $storeid);
@@ -188,7 +195,7 @@ __PACKAGE__->register_method ({
 	$rpcenv->fork_worker('pbs-download', undef, $user, sub {
 	    my $name = decode_base64($path);
 	    print "Starting download of file: $name\n";
-	    $client->file_restore_extract($fifo, [$scfg->{namespace}, $snap], $path, 1);
+	    $client->file_restore_extract($fifo, [$scfg->{namespace}, $snap], $path, 1, $tar);
 	});
 
 	my $ret = {
-- 
2.30.2

[pve-devel] [PATCH widget-toolkit v2 1/1] window/FileBrowser: enable tar button by default

[Dominik Csapak]

all endpoints now can handle the 'tar' parameter, so add it for all

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/window/FileBrowser.js | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/src/window/FileBrowser.js b/src/window/FileBrowser.js
index a519d6b..c2f485a 100644
--- a/src/window/FileBrowser.js
+++ b/src/window/FileBrowser.js
@@ -61,10 +61,6 @@ Ext.define("Proxmox.window.FileBrowser", {
 	    'd': true, // directories
 	},
 
-	// enable tar download, this will add a menu to the "Download" button when the selection
-	// can be downloaded as `.tar` files
-	enableTar: false,
-
 	// prefix to prepend to downloaded file names
 	downloadPrefix: '',
     },
@@ -127,7 +123,7 @@ Ext.define("Proxmox.window.FileBrowser", {
 	    view.lookup('selectText').setText(st);
 
 	    let canDownload = view.downloadURL && view.downloadableFileTypes[data.type];
-	    let enableMenu = view.enableTar && data.type === 'd';
+	    let enableMenu = data.type === 'd';
 
 	    let downloadBtn = view.lookup('downloadBtn');
 	    downloadBtn.setDisabled(!canDownload || enableMenu);
-- 
2.30.2

Re: [pve-devel] [PATCH proxmox-backup/common/storage/wt v2] add tar.zst download in pve

[Dominik Csapak]

ping (sorry i waited for such a long time ;) )

it probably does not apply anymore, but i'd like to get feedback
before starting the rebase