[pbs-devel] [PATCH v6 proxmox-backup 0/4] fix #2996: client: allow optional match patterns for restore

[pbs-devel] [PATCH v6 proxmox-backup 0/4] fix #2996: client: allow optional match patterns for restore

[Christian Ebner]

This patches implement the api types to allow input validation for
pathpatterns and reuse them in the pxar-bin, the catalog shell as
well as the newly exposed optional restore patterns to the backup
clients restore command.

Patterns are parsed and passed along to the preexisting restore
logic via the `PxarExtractOptions`.

To correctly work also with split pxar archives, this patches depend
on the following patch being applied to the pxar repo first:
https://lore.proxmox.com/pbs-devel/20240918150047.485551-1-c.ebner@proxmox.com/

changes since version 5:
- rebased onto current master

changes since version 4:
- rebased onto current master
- fixed passing patterns via cli for pxar extract

changes since version 3:
- s/matches/patterns for bail message, thanks for testing and
  catching this Gabriel!

changes since version 2:
- added API types as suggested
- reuse same API types for proxmox-backup-client catalog shell and
  restore as well as the pxar extract
- use simple reference instead of `as_slice()` when passing vector of
  patterns

Link to bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=2996

Christian Ebner (4):
  api-types: implement dedicated api type for match patterns
  pxar: bin: use dedicated api type for restore pattern
  client: catalog shell: use dedicated api type for patterns
  fix #2996: client: allow optional match patterns for restore

 pbs-api-types/src/lib.rs          |  3 ++
 pbs-api-types/src/pathpatterns.rs | 55 +++++++++++++++++++++++++++++++
 pbs-client/src/catalog_shell.rs   |  7 ++--
 proxmox-backup-client/src/main.rs | 29 +++++++++++++---
 pxar-bin/Cargo.toml               |  1 +
 pxar-bin/src/main.rs              | 26 +++++++--------
 6 files changed, 99 insertions(+), 22 deletions(-)
 create mode 100644 pbs-api-types/src/pathpatterns.rs

-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH v6 proxmox-backup 1/4] api-types: implement dedicated api type for match patterns

[Christian Ebner]

Introduces a dedicated api type `PathPattern` and the corresponding
format and input validation schema. Further, add a `PathPatterns`
type for collections of path patterns and implement required traits
to be able to replace currently defined api parameters.

In preparation for using this common api type for all api endpoints
exposing a match pattern parameter.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 5:
- no changes

 pbs-api-types/src/lib.rs          |  3 ++
 pbs-api-types/src/pathpatterns.rs | 55 +++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)
 create mode 100644 pbs-api-types/src/pathpatterns.rs

diff --git a/pbs-api-types/src/lib.rs b/pbs-api-types/src/lib.rs
index 460c7da7c..75dc42407 100644
--- a/pbs-api-types/src/lib.rs
+++ b/pbs-api-types/src/lib.rs
@@ -143,6 +143,9 @@ pub use ad::*;
 mod remote;
 pub use remote::*;
 
+mod pathpatterns;
+pub use pathpatterns::*;
+
 mod tape;
 pub use tape::*;
 
diff --git a/pbs-api-types/src/pathpatterns.rs b/pbs-api-types/src/pathpatterns.rs
new file mode 100644
index 000000000..c40926a44
--- /dev/null
+++ b/pbs-api-types/src/pathpatterns.rs
@@ -0,0 +1,55 @@
+use proxmox_schema::{const_regex, ApiStringFormat, ApiType, ArraySchema, Schema, StringSchema};
+
+use serde::{Deserialize, Serialize};
+
+const_regex! {
+     pub PATH_PATTERN_REGEX = concat!(r"^.+[^\\]$");
+}
+
+pub const PATH_PATTERN_FORMAT: ApiStringFormat = ApiStringFormat::Pattern(&PATH_PATTERN_REGEX);
+
+pub const PATH_PATTERN_SCHEMA: Schema =
+    StringSchema::new("Path or match pattern for matching filenames.")
+        .format(&PATH_PATTERN_FORMAT)
+        .schema();
+
+pub const PATH_PATTERN_LIST_SCHEMA: Schema = ArraySchema::new(
+    "List of paths or match patterns for matching filenames.",
+    &PATH_PATTERN_SCHEMA,
+)
+.schema();
+
+#[derive(Default, Deserialize, Serialize)]
+/// Path or path pattern for filename matching
+pub struct PathPattern {
+    pattern: String,
+}
+
+impl ApiType for PathPattern {
+    const API_SCHEMA: Schema = PATH_PATTERN_SCHEMA;
+}
+
+impl AsRef<[u8]> for PathPattern {
+    fn as_ref(&self) -> &[u8] {
+        self.pattern.as_bytes()
+    }
+}
+
+#[derive(Default, Deserialize, Serialize)]
+/// Array of paths and/or path patterns for filename matching
+pub struct PathPatterns {
+    patterns: Vec<PathPattern>,
+}
+
+impl ApiType for PathPatterns {
+    const API_SCHEMA: Schema = PATH_PATTERN_LIST_SCHEMA;
+}
+
+impl IntoIterator for PathPatterns {
+    type Item = PathPattern;
+    type IntoIter = std::vec::IntoIter<PathPattern>;
+
+    fn into_iter(self) -> Self::IntoIter {
+        self.patterns.into_iter()
+    }
+}
-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH v6 proxmox-backup 2/4] pxar: bin: use dedicated api type for restore pattern

[Christian Ebner]

Instead of taking a plain string as input parameter, use the
corresponding api type performing additional input validation.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 5:
- rebased to current master

 pxar-bin/Cargo.toml  |  1 +
 pxar-bin/src/main.rs | 26 +++++++++++++-------------
 2 files changed, 14 insertions(+), 13 deletions(-)

diff --git a/pxar-bin/Cargo.toml b/pxar-bin/Cargo.toml
index d0d7ab24d..37c980e28 100644
--- a/pxar-bin/Cargo.toml
+++ b/pxar-bin/Cargo.toml
@@ -25,5 +25,6 @@ proxmox-router = { workspace = true, features = ["cli", "server"] }
 proxmox-schema = { workspace = true, features = [ "api-macro" ] }
 proxmox-sys.workspace = true
 
+pbs-api-types.workspace = true
 pbs-client.workspace = true
 pbs-pxar-fuse.workspace = true
diff --git a/pxar-bin/src/main.rs b/pxar-bin/src/main.rs
index 2fc0d1bb5..eb3580d92 100644
--- a/pxar-bin/src/main.rs
+++ b/pxar-bin/src/main.rs
@@ -9,9 +9,11 @@ use std::sync::Arc;
 use anyhow::{bail, format_err, Error};
 use futures::future::FutureExt;
 use futures::select;
+use serde_json::Value;
 use tokio::signal::unix::{signal, SignalKind};
 
 use pathpatterns::{MatchEntry, MatchType, PatternFlag};
+use pbs_api_types::PathPatterns;
 use pbs_client::pxar::tools::format_single_line_entry;
 use pbs_client::pxar::{
     Flags, OverwriteFlags, PxarExtractOptions, PxarWriters, ENCODER_MAX_ENTRIES,
@@ -53,12 +55,7 @@ fn extract_archive_from_reader<R: std::io::Read>(
                 description: "Archive name.",
             },
             pattern: {
-                description: "List of paths or pattern matching files to restore",
-                type: Array,
-                items: {
-                    type: String,
-                    description: "Path or pattern matching files to restore.",
-                },
+                type: PathPatterns,
                 optional: true,
             },
             target: {
@@ -144,7 +141,6 @@ fn extract_archive_from_reader<R: std::io::Read>(
 #[allow(clippy::too_many_arguments)]
 fn extract_archive(
     archive: String,
-    pattern: Option<Vec<String>>,
     target: Option<String>,
     no_xattrs: bool,
     no_fcaps: bool,
@@ -161,6 +157,7 @@ fn extract_archive(
     strict: bool,
     payload_input: Option<String>,
     prelude_target: Option<String>,
+    param: Value,
 ) -> Result<(), Error> {
     let mut feature_flags = Flags::DEFAULT;
     if no_xattrs {
@@ -190,7 +187,6 @@ fn extract_archive(
         overwrite_flags.insert(OverwriteFlags::all());
     }
 
-    let pattern = pattern.unwrap_or_default();
     let target = target.as_ref().map_or_else(|| ".", String::as_str);
 
     let mut match_list = Vec::new();
@@ -204,11 +200,15 @@ fn extract_archive(
         }
     }
 
-    for entry in pattern {
-        match_list.push(
-            MatchEntry::parse_pattern(entry, PatternFlag::PATH_NAME, MatchType::Include)
-                .map_err(|err| format_err!("error in pattern: {}", err))?,
-        );
+    if let Some(pattern) = param["pattern"].as_array() {
+        for p in pattern {
+            if let Some(entry) = p.as_str() {
+                match_list.push(
+                    MatchEntry::parse_pattern(entry, PatternFlag::PATH_NAME, MatchType::Include)
+                        .map_err(|err| format_err!("error in pattern: {err}"))?,
+                );
+            }
+        }
     }
 
     let extract_match_default = match_list.is_empty();
-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH v6 proxmox-backup 3/4] client: catalog shell: use dedicated api type for patterns

[Christian Ebner]

Use the common api type with schema based input validation for all
match pattern parameters exposed via the api macro.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 5:
- no changes

 pbs-client/src/catalog_shell.rs | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/pbs-client/src/catalog_shell.rs b/pbs-client/src/catalog_shell.rs
index 8c8e9a654..7e69970fe 100644
--- a/pbs-client/src/catalog_shell.rs
+++ b/pbs-client/src/catalog_shell.rs
@@ -14,6 +14,7 @@ use nix::fcntl::OFlag;
 use nix::sys::stat::Mode;
 
 use pathpatterns::{MatchEntry, MatchList, MatchPattern, MatchType, PatternFlag};
+use pbs_api_types::PathPattern;
 use proxmox_router::cli::{self, CliCommand, CliCommandMap, CliHelper, CommandLineInterface};
 use proxmox_schema::api;
 use proxmox_sys::fs::{create_path, CreateOptions};
@@ -240,8 +241,7 @@ async fn list_selected_command(patterns: bool) -> Result<(), Error> {
     input: {
         properties: {
             pattern: {
-                type: String,
-                description: "Match pattern for matching files in the catalog."
+                type: PathPattern,
             },
             select: {
                 type: bool,
@@ -282,9 +282,8 @@ async fn restore_selected_command(target: String) -> Result<(), Error> {
                 description: "target path for restore on local filesystem."
             },
             pattern: {
-                type: String,
+                type: PathPattern,
                 optional: true,
-                description: "match pattern to limit files for restore."
             }
         }
     }
-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH v6 proxmox-backup 4/4] fix #2996: client: allow optional match patterns for restore

[Christian Ebner]

When the user is only interested in a subset of the entries stored in
a file-level backup, it is convenient to be able to provide a list of
match patterns for the entries intended to be restored.

The required restore logic is already in place. Therefore, expose it
for the `proxmox-backup-client restore` command by adding the optional
array of patterns as command line argument and parse these before
passing them via the pxar restore options to the archive extractor.

Link to bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=2996

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 5:
- no changes

 proxmox-backup-client/src/main.rs | 29 ++++++++++++++++++++++++-----
 1 file changed, 24 insertions(+), 5 deletions(-)

diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
index e4034aa99..817235dbe 100644
--- a/proxmox-backup-client/src/main.rs
+++ b/proxmox-backup-client/src/main.rs
@@ -26,9 +26,9 @@ use pxar::accessor::{MaybeReady, ReadAt, ReadAtOperation};
 
 use pbs_api_types::{
     Authid, BackupDir, BackupGroup, BackupNamespace, BackupPart, BackupType, ClientRateLimitConfig,
-    CryptMode, Fingerprint, GroupListItem, PruneJobOptions, PruneListItem, RateLimitConfig,
-    SnapshotListItem, StorageStatus, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
-    BACKUP_TYPE_SCHEMA,
+    CryptMode, Fingerprint, GroupListItem, PathPatterns, PruneJobOptions, PruneListItem,
+    RateLimitConfig, SnapshotListItem, StorageStatus, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA,
+    BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA,
 };
 use pbs_client::catalog_shell::Shell;
 use pbs_client::pxar::{ErrorHandler as PxarErrorHandler, MetadataArchiveReader, PxarPrevRef};
@@ -1394,6 +1394,10 @@ We do not extract '.pxar' archives when writing to standard output.
                 type: ClientRateLimitConfig,
                 flatten: true,
             },
+            pattern: {
+                type: PathPatterns,
+                optional: true,
+            },
             "allow-existing-dirs": {
                 type: Boolean,
                 description: "Do not fail if directories already exists.",
@@ -1503,6 +1507,21 @@ async fn restore(
     let target = json::required_string_param(&param, "target")?;
     let target = if target == "-" { None } else { Some(target) };
 
+    let mut match_list = Vec::new();
+    if let Some(pattern) = param["pattern"].as_array() {
+        if target.is_none() {
+            bail!("patterns not allowed when restoring to stdout");
+        }
+
+        for p in pattern {
+            if let Some(pattern) = p.as_str() {
+                let match_entry =
+                    MatchEntry::parse_pattern(pattern, PatternFlag::PATH_NAME, MatchType::Include)?;
+                match_list.push(match_entry);
+            }
+        }
+    };
+
     let crypto = crypto_parameters(&param)?;
 
     let crypt_config = match crypto.enc_key {
@@ -1622,8 +1641,8 @@ async fn restore(
         let prelude_path = param["prelude-target"].as_str().map(PathBuf::from);
 
         let options = pbs_client::pxar::PxarExtractOptions {
-            match_list: &[],
-            extract_match_default: true,
+            match_list: &match_list,
+            extract_match_default: match_list.is_empty(),
             allow_existing_dirs,
             overwrite_flags,
             on_error,
-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel