[pbs-devel] [PATCH v3 proxmox-backup 0/4] fix #2996: client: allow optional match patterns for restore

[pbs-devel] [PATCH v3 proxmox-backup 0/4] fix #2996: client: allow optional match patterns for restore

[Christian Ebner]

This patches implement the api types to allow input validation for
pathpatterns and reuse them in the pxar-bin, the catalog shell as
well as the newly exposed optional restore patterns to the backup
clients restore command.

Patterns are parsed and passed along to the preexisting restore
logic via the `PxarExtractOptions`.

changes since version 2:
- added API types as suggested
- reuse same API types for proxmox-backup-client catalog shell and
  restore as well as the pxar extract
- use simple reference instead of `as_slice()` when passing vector of
  patterns

Link to bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=2996

proxmox-backup:

Christian Ebner (4):
  api-types: implement dedicated api type for match patterns
  pxar: bin: use dedicated api type for restore pattern
  client: catalog shell: use dedicated api type for patterns
  fix #2996: client: allow optional match patterns for restore

 pbs-api-types/src/lib.rs          |  3 ++
 pbs-api-types/src/pathpatterns.rs | 55 +++++++++++++++++++++++++++++++
 pbs-client/src/catalog_shell.rs   |  7 ++--
 proxmox-backup-client/src/main.rs | 27 ++++++++++++---
 pxar-bin/Cargo.toml               |  1 +
 pxar-bin/src/main.rs              | 10 ++----
 6 files changed, 88 insertions(+), 15 deletions(-)
 create mode 100644 pbs-api-types/src/pathpatterns.rs

-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH v3 proxmox-backup 1/4] api-types: implement dedicated api type for match patterns

[Christian Ebner]

Introduces a dedicated api type `PathPattern` and the corresponding
format and input validation schema. Further, add a `PathPatterns`
type for collections of path patterns and implement required traits
to be able to replace currently defined api parameters.

In preparation for using this common api type for all api endpoints
exposing a match pattern parameter.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 2:
- not present

 pbs-api-types/src/lib.rs          |  3 ++
 pbs-api-types/src/pathpatterns.rs | 55 +++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)
 create mode 100644 pbs-api-types/src/pathpatterns.rs

diff --git a/pbs-api-types/src/lib.rs b/pbs-api-types/src/lib.rs
index a3ad185b6..c144ad0b2 100644
--- a/pbs-api-types/src/lib.rs
+++ b/pbs-api-types/src/lib.rs
@@ -136,6 +136,9 @@ pub use ad::*;
 mod remote;
 pub use remote::*;
 
+mod pathpatterns;
+pub use pathpatterns::*;
+
 mod tape;
 pub use tape::*;
 
diff --git a/pbs-api-types/src/pathpatterns.rs b/pbs-api-types/src/pathpatterns.rs
new file mode 100644
index 000000000..c40926a44
--- /dev/null
+++ b/pbs-api-types/src/pathpatterns.rs
@@ -0,0 +1,55 @@
+use proxmox_schema::{const_regex, ApiStringFormat, ApiType, ArraySchema, Schema, StringSchema};
+
+use serde::{Deserialize, Serialize};
+
+const_regex! {
+     pub PATH_PATTERN_REGEX = concat!(r"^.+[^\\]$");
+}
+
+pub const PATH_PATTERN_FORMAT: ApiStringFormat = ApiStringFormat::Pattern(&PATH_PATTERN_REGEX);
+
+pub const PATH_PATTERN_SCHEMA: Schema =
+    StringSchema::new("Path or match pattern for matching filenames.")
+        .format(&PATH_PATTERN_FORMAT)
+        .schema();
+
+pub const PATH_PATTERN_LIST_SCHEMA: Schema = ArraySchema::new(
+    "List of paths or match patterns for matching filenames.",
+    &PATH_PATTERN_SCHEMA,
+)
+.schema();
+
+#[derive(Default, Deserialize, Serialize)]
+/// Path or path pattern for filename matching
+pub struct PathPattern {
+    pattern: String,
+}
+
+impl ApiType for PathPattern {
+    const API_SCHEMA: Schema = PATH_PATTERN_SCHEMA;
+}
+
+impl AsRef<[u8]> for PathPattern {
+    fn as_ref(&self) -> &[u8] {
+        self.pattern.as_bytes()
+    }
+}
+
+#[derive(Default, Deserialize, Serialize)]
+/// Array of paths and/or path patterns for filename matching
+pub struct PathPatterns {
+    patterns: Vec<PathPattern>,
+}
+
+impl ApiType for PathPatterns {
+    const API_SCHEMA: Schema = PATH_PATTERN_LIST_SCHEMA;
+}
+
+impl IntoIterator for PathPatterns {
+    type Item = PathPattern;
+    type IntoIter = std::vec::IntoIter<PathPattern>;
+
+    fn into_iter(self) -> Self::IntoIter {
+        self.patterns.into_iter()
+    }
+}
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH v3 proxmox-backup 2/4] pxar: bin: use dedicated api type for restore pattern

[Christian Ebner]

Instead of taking a plain string as input parameter, use the
corresponding api type performing additional input validation.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 2:
- not present

 pxar-bin/Cargo.toml  |  1 +
 pxar-bin/src/main.rs | 10 +++-------
 2 files changed, 4 insertions(+), 7 deletions(-)

diff --git a/pxar-bin/Cargo.toml b/pxar-bin/Cargo.toml
index bb010ff78..aa53a1cf0 100644
--- a/pxar-bin/Cargo.toml
+++ b/pxar-bin/Cargo.toml
@@ -25,6 +25,7 @@ proxmox-router = { workspace = true, features = ["cli", "server"] }
 proxmox-schema = { workspace = true, features = [ "api-macro" ] }
 proxmox-sys.workspace = true
 
+pbs-api-types.workspace = true
 pbs-client.workspace = true
 pbs-pxar-fuse.workspace = true
 pbs-tools.workspace = true
diff --git a/pxar-bin/src/main.rs b/pxar-bin/src/main.rs
index e62348e25..1ec1ff13d 100644
--- a/pxar-bin/src/main.rs
+++ b/pxar-bin/src/main.rs
@@ -12,6 +12,7 @@ use futures::select;
 use tokio::signal::unix::{signal, SignalKind};
 
 use pathpatterns::{MatchEntry, MatchType, PatternFlag};
+use pbs_api_types::PathPatterns;
 use pbs_client::pxar::{
     format_single_line_entry, Flags, OverwriteFlags, PxarExtractOptions, PxarWriters,
     ENCODER_MAX_ENTRIES,
@@ -52,12 +53,7 @@ fn extract_archive_from_reader<R: std::io::Read>(
                 description: "Archive name.",
             },
             pattern: {
-                description: "List of paths or pattern matching files to restore",
-                type: Array,
-                items: {
-                    type: String,
-                    description: "Path or pattern matching files to restore.",
-                },
+                type: PathPatterns,
                 optional: true,
             },
             target: {
@@ -143,7 +139,7 @@ fn extract_archive_from_reader<R: std::io::Read>(
 #[allow(clippy::too_many_arguments)]
 fn extract_archive(
     archive: String,
-    pattern: Option<Vec<String>>,
+    pattern: Option<PathPatterns>,
     target: Option<String>,
     no_xattrs: bool,
     no_fcaps: bool,
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH v3 proxmox-backup 3/4] client: catalog shell: use dedicated api type for patterns

[Christian Ebner]

Use the common api type with schema based input validation for all
match pattern parameters exposed via the api macro.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 2:
- not present

 pbs-client/src/catalog_shell.rs | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/pbs-client/src/catalog_shell.rs b/pbs-client/src/catalog_shell.rs
index 349bb7cbc..6afde0e0a 100644
--- a/pbs-client/src/catalog_shell.rs
+++ b/pbs-client/src/catalog_shell.rs
@@ -14,6 +14,7 @@ use nix::fcntl::OFlag;
 use nix::sys::stat::Mode;
 
 use pathpatterns::{MatchEntry, MatchList, MatchPattern, MatchType, PatternFlag};
+use pbs_api_types::PathPattern;
 use proxmox_router::cli::{self, CliCommand, CliCommandMap, CliHelper, CommandLineInterface};
 use proxmox_schema::api;
 use proxmox_sys::fs::{create_path, CreateOptions};
@@ -240,8 +241,7 @@ async fn list_selected_command(patterns: bool) -> Result<(), Error> {
     input: {
         properties: {
             pattern: {
-                type: String,
-                description: "Match pattern for matching files in the catalog."
+                type: PathPattern,
             },
             select: {
                 type: bool,
@@ -282,9 +282,8 @@ async fn restore_selected_command(target: String) -> Result<(), Error> {
                 description: "target path for restore on local filesystem."
             },
             pattern: {
-                type: String,
+                type: PathPattern,
                 optional: true,
-                description: "match pattern to limit files for restore."
             }
         }
     }
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH v3 proxmox-backup 4/4] fix #2996: client: allow optional match patterns for restore

[Christian Ebner]

When the user is only interested in a subset of the entries stored in
a file-level backup, it is convenient to be able to provide a list of
match patterns for the entries intended to be restored.

The required restore logic is already in place. Therefore, expose it
for the `proxmox-backup-client restore` command by adding the optional
array of patterns as command line argument and parse these before
passing them via the pxar restore options to the archive extractor.

Link to bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=2996

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 2:
- use of `PathPatterns` instead of Array of Strings for pattern
  parameters
- use simple reference instead of `as_slice()` when passing vector of
  patterns

 proxmox-backup-client/src/main.rs | 27 +++++++++++++++++++++++----
 1 file changed, 23 insertions(+), 4 deletions(-)

diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
index f1c7fbf93..9cc6f889a 100644
--- a/proxmox-backup-client/src/main.rs
+++ b/proxmox-backup-client/src/main.rs
@@ -26,8 +26,8 @@ use pxar::accessor::{MaybeReady, ReadAt, ReadAtOperation};
 
 use pbs_api_types::{
     Authid, BackupDir, BackupGroup, BackupNamespace, BackupPart, BackupType, CryptMode,
-    Fingerprint, GroupListItem, PruneJobOptions, PruneListItem, RateLimitConfig, SnapshotListItem,
-    StorageStatus, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
+    Fingerprint, GroupListItem, PathPatterns, PruneJobOptions, PruneListItem, RateLimitConfig,
+    SnapshotListItem, StorageStatus, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
     BACKUP_TYPE_SCHEMA, TRAFFIC_CONTROL_BURST_SCHEMA, TRAFFIC_CONTROL_RATE_SCHEMA,
 };
 use pbs_client::catalog_shell::Shell;
@@ -1389,6 +1389,10 @@ We do not extract '.pxar' archives when writing to standard output.
 
 "###
             },
+            pattern: {
+                type: PathPatterns,
+                optional: true,
+            },
             rate: {
                 schema: TRAFFIC_CONTROL_RATE_SCHEMA,
                 optional: true,
@@ -1514,6 +1518,21 @@ async fn restore(
     let target = json::required_string_param(&param, "target")?;
     let target = if target == "-" { None } else { Some(target) };
 
+    let mut match_list = Vec::new();
+    if let Some(pattern) = param["pattern"].as_array() {
+        if target.is_none() {
+            bail!("matches not allowed when restoring to stdout");
+        }
+
+        for p in pattern {
+            if let Some(pattern) = p.as_str() {
+                let match_entry =
+                    MatchEntry::parse_pattern(pattern, PatternFlag::PATH_NAME, MatchType::Include)?;
+                match_list.push(match_entry);
+            }
+        }
+    };
+
     let crypto = crypto_parameters(&param)?;
 
     let crypt_config = match crypto.enc_key {
@@ -1635,8 +1654,8 @@ async fn restore(
             .map(|path| PathBuf::from(path));
 
         let options = pbs_client::pxar::PxarExtractOptions {
-            match_list: &[],
-            extract_match_default: true,
+            match_list: &match_list,
+            extract_match_default: match_list.is_empty(),
             allow_existing_dirs,
             overwrite_flags,
             on_error,
-- 
2.39.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH v3 proxmox-backup 4/4] fix #2996: client: allow optional match patterns for restore

[Gabriel Goller]

On 13.06.2024 12:56, Christian Ebner wrote:
>diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
>index f1c7fbf93..9cc6f889a 100644
>--- a/proxmox-backup-client/src/main.rs
>+++ b/proxmox-backup-client/src/main.rs
>@@ -1389,6 +1389,10 @@ We do not extract '.pxar' archives when writing to standard output.
>
> "###
>             },
>+            pattern: {
>+                type: PathPatterns,
>+                optional: true,
>+            },
>             rate: {
>                 schema: TRAFFIC_CONTROL_RATE_SCHEMA,
>                 optional: true,
>@@ -1514,6 +1518,21 @@ async fn restore(
>     let target = json::required_string_param(&param, "target")?;
>     let target = if target == "-" { None } else { Some(target) };
>
>+    let mut match_list = Vec::new();
>+    if let Some(pattern) = param["pattern"].as_array() {
>+        if target.is_none() {
>+            bail!("matches not allowed when restoring to stdout");

s/matches/patterns

>+        }
>+
>+        for p in pattern {
>+            if let Some(pattern) = p.as_str() {
>+                let match_entry =
>+                    MatchEntry::parse_pattern(pattern, PatternFlag::PATH_NAME, MatchType::Include)?;
>+                match_list.push(match_entry);
>+            }
>+        }
>+    };
>+
>     let crypto = crypto_parameters(&param)?;
>
>     let crypt_config = match crypto.enc_key {

Otherwise the pattern-restore part seems to work nicely! Consider:

Tested-by: Gabriel Goller <g.goller@proxmox.com>



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH v3 proxmox-backup 0/4] fix #2996: client: allow optional match patterns for restore

[Christian Ebner]

Superseded by version 4:
https://lists.proxmox.com/pipermail/pbs-devel/2024-June/009866.html


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel