[pbs-devel] [PATCH v7 proxmox-backup 0/3] add proxmox-backup-debug binary

[pbs-devel] [PATCH v7 proxmox-backup 0/3] add proxmox-backup-debug binary

[Hannes Laimer]

Adds a new proxmox-backup-debug binary for recovery or inspection of
index, blob and chunk files

v7:
  - handle 'text' output-format properly
  - describe inspect file a little more

v6(based on Wolfgang Bumiller <w.bumiller@proxmox.com>'s feedback):
  - removed unnecessary SCHEMA references
  - added few args with arg_param
  - removed unnecessary Result wrapping
  - moved logic for checking whether decode is to stdout into
    decode_blob function
  - improved argument description

v5:
  - improved some parts based on Dominik Csapak <d.csapak@proxmox.com>
    feedback on v4
  - changes are described in the patches themself

v4:
  - rewrote parts based on Wolfgang Bumiller <w.bumiller@proxmox.com>
    feedback on v3
  - check filetype mostly by magic number, except when
    looking for index files

v3:
  - move description into commit message

v2:
  - appiled suggestions + rewrote a few parts
  - renamed 'restore' to 'debug'<Paste>

Hannes Laimer (3):
  add chunk inspection to pb-debug
  add file inspection to pb-debug
  add index recovery to pb-debug

 Makefile                                |   3 +-
 src/bin/proxmox-backup-debug.rs         |  15 ++
 src/bin/proxmox_backup_debug/inspect.rs | 335 ++++++++++++++++++++++++
 src/bin/proxmox_backup_debug/mod.rs     |   4 +
 src/bin/proxmox_backup_debug/recover.rs | 119 +++++++++
 src/tools/mod.rs                        |  13 +
 6 files changed, 488 insertions(+), 1 deletion(-)
 create mode 100644 src/bin/proxmox-backup-debug.rs
 create mode 100644 src/bin/proxmox_backup_debug/inspect.rs
 create mode 100644 src/bin/proxmox_backup_debug/mod.rs
 create mode 100644 src/bin/proxmox_backup_debug/recover.rs

-- 
2.30.2

[pbs-devel] [PATCH v7 proxmox-backup 1/3] add chunk inspection to pb-debug

[Hannes Laimer]

Adds possibility to inspect chunks and find indexes that reference the
chunk. Options:
 - chunk: path to the chunk file
 - [opt] decode: path to a file or to stdout(-), if specified, the
   chunk will be decoded into the specified location
 - [opt] digest: needed when searching for references, if set, it will
   be used for verification when decoding
 - [opt] keyfile: path to a keyfile, needed if decode is specified and
   the data was encrypted
 - [opt] reference-filter: path in which indexes that reference the
   chunk should be searched, can be a group, snapshot or the whole
   datastore, if not specified no references will be searched
 - [default=true] use-filename-as-digest: use chunk-filename as digest,
   if no digest is specified

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 Makefile                                |   3 +-
 src/bin/proxmox-backup-debug.rs         |  13 ++
 src/bin/proxmox_backup_debug/inspect.rs | 213 ++++++++++++++++++++++++
 src/bin/proxmox_backup_debug/mod.rs     |   2 +
 src/tools/mod.rs                        |  13 ++
 5 files changed, 243 insertions(+), 1 deletion(-)
 create mode 100644 src/bin/proxmox-backup-debug.rs
 create mode 100644 src/bin/proxmox_backup_debug/inspect.rs
 create mode 100644 src/bin/proxmox_backup_debug/mod.rs

diff --git a/Makefile b/Makefile
index d6c1acf5..14c07ca0 100644
--- a/Makefile
+++ b/Makefile
@@ -17,7 +17,8 @@ USR_BIN := \
 
 # Binaries usable by admins
 USR_SBIN := \
-	proxmox-backup-manager
+	proxmox-backup-manager \
+	proxmox-backup-debug \
 
 # Binaries for services:
 SERVICE_BIN := \
diff --git a/src/bin/proxmox-backup-debug.rs b/src/bin/proxmox-backup-debug.rs
new file mode 100644
index 00000000..897fa221
--- /dev/null
+++ b/src/bin/proxmox-backup-debug.rs
@@ -0,0 +1,13 @@
+use proxmox::api::cli::*;
+
+mod proxmox_backup_debug;
+use proxmox_backup_debug::inspect_commands;
+
+fn main() {
+    proxmox_backup::tools::setup_safe_path_env();
+
+    let cmd_def = CliCommandMap::new().insert("inspect", inspect_commands());
+
+    let rpcenv = CliEnvironment::new();
+    run_cli_command(cmd_def, rpcenv, Some(|future| pbs_runtime::main(future)));
+}
diff --git a/src/bin/proxmox_backup_debug/inspect.rs b/src/bin/proxmox_backup_debug/inspect.rs
new file mode 100644
index 00000000..5a0979d2
--- /dev/null
+++ b/src/bin/proxmox_backup_debug/inspect.rs
@@ -0,0 +1,213 @@
+use std::path::Path;
+
+use anyhow::{format_err, Error};
+use proxmox::api::cli::{
+    format_and_print_result, get_output_format, CliCommand, CliCommandMap, CommandLineInterface,
+};
+use proxmox::api::{api, cli::*};
+use serde_json::{json, Value};
+use walkdir::WalkDir;
+
+use proxmox_backup::backup::{
+    load_and_decrypt_key, CryptConfig, DataBlob, DynamicIndexReader, FixedIndexReader, IndexFile,
+};
+
+use pbs_client::tools::key_source::get_encryption_key_password;
+
+use proxmox_backup::tools::outfile_or_stdout;
+
+/// Decodes a blob and writes its content either to stdout or into a file
+fn decode_blob(
+    mut output_path: Option<&Path>,
+    key_file: Option<&Path>,
+    digest: Option<&[u8; 32]>,
+    blob: &DataBlob,
+) -> Result<(), Error> {
+    let mut crypt_conf_opt = None;
+    let crypt_conf;
+
+    if blob.is_encrypted() && key_file.is_some() {
+        let (key, _created, _fingerprint) =
+            load_and_decrypt_key(&key_file.unwrap(), &get_encryption_key_password)?;
+        crypt_conf = CryptConfig::new(key)?;
+        crypt_conf_opt = Some(&crypt_conf);
+    }
+
+    output_path = match output_path {
+        Some(path) if path.eq(Path::new("-")) => None,
+        _ => output_path,
+    };
+
+    outfile_or_stdout(output_path)?.write_all(blob.decode(crypt_conf_opt, digest)?.as_slice())?;
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            chunk: {
+                description: "The chunk file.",
+                type: String,
+            },
+            "reference-filter": {
+                description: "Path to the directory that should be searched for references.",
+                type: String,
+                optional: true,
+            },
+            "digest": {
+                description: "Needed when searching for references, if set, it will be used for verification when decoding.",
+                type: String,
+                optional: true,
+            },
+            "decode": {
+                description: "Path to the file to which the chunk should be decoded, '-' -> decode to stdout.",
+                type: String,
+                optional: true,
+            },
+            "keyfile": {
+                description: "Path to the keyfile with which the chunk was encrypted.",
+                type: String,
+                optional: true,
+            },
+            "use-filename-as-digest": {
+                description: "The filename should be used as digest for reference search and decode verification, if no digest is specified.",
+                type: bool,
+                optional: true,
+                default: true,
+            },
+            "output-format": {
+                schema: OUTPUT_FORMAT,
+                optional: true,
+            },
+        }
+    }
+)]
+/// Inspect a chunk
+fn inspect_chunk(
+    chunk: String,
+    reference_filter: Option<String>,
+    mut digest: Option<String>,
+    decode: Option<String>,
+    keyfile: Option<String>,
+    use_filename_as_digest: bool,
+    param: Value,
+) -> Result<(), Error> {
+    let output_format = get_output_format(&param);
+    let chunk_path = Path::new(&chunk);
+
+    if digest.is_none() && use_filename_as_digest {
+        digest = Some(if let Some((_, filename)) = chunk.rsplit_once("/") {
+            String::from(filename)
+        } else {
+            chunk.clone()
+        });
+    };
+
+    let digest_raw: Option<[u8; 32]> = digest
+        .map(|ref d| {
+            proxmox::tools::hex_to_digest(d)
+                .map_err(|e| format_err!("could not parse chunk - {}", e))
+        })
+        .map_or(Ok(None), |r| r.map(Some))?;
+
+    let search_path = reference_filter.as_ref().map(Path::new);
+    let key_file_path = keyfile.as_ref().map(Path::new);
+    let decode_output_path = decode.as_ref().map(Path::new);
+
+    let blob = DataBlob::load_from_reader(
+        &mut std::fs::File::open(&chunk_path)
+            .map_err(|e| format_err!("could not open chunk file - {}", e))?,
+    )?;
+
+    let referenced_by = if let (Some(search_path), Some(digest_raw)) = (search_path, digest_raw) {
+        let mut references = Vec::new();
+        for entry in WalkDir::new(search_path)
+            .follow_links(false)
+            .into_iter()
+            .filter_map(|e| e.ok())
+        {
+            use std::os::unix::ffi::OsStrExt;
+            let file_name = entry.file_name().as_bytes();
+
+            let index: Box<dyn IndexFile> = if file_name.ends_with(b".fidx") {
+                match FixedIndexReader::open(entry.path()) {
+                    Ok(index) => Box::new(index),
+                    Err(_) => continue,
+                }
+            } else if file_name.ends_with(b".didx") {
+                match DynamicIndexReader::open(entry.path()) {
+                    Ok(index) => Box::new(index),
+                    Err(_) => continue,
+                }
+            } else {
+                continue;
+            };
+
+            for pos in 0..index.index_count() {
+                if let Some(index_chunk_digest) = index.index_digest(pos) {
+                    if digest_raw.eq(index_chunk_digest) {
+                        references.push(entry.path().to_string_lossy().into_owned());
+                        break;
+                    }
+                }
+            }
+        }
+        if !references.is_empty() {
+            Some(references)
+        } else {
+            None
+        }
+    } else {
+        None
+    };
+
+    if decode_output_path.is_some() {
+        decode_blob(
+            decode_output_path,
+            key_file_path,
+            digest_raw.as_ref(),
+            &blob,
+        )?;
+    }
+
+    let crc_status = format!(
+        "{}({})",
+        blob.compute_crc(),
+        blob.verify_crc().map_or("BAD", |_| "OK")
+    );
+
+    let val = match referenced_by {
+        Some(references) => json!({
+            "crc": crc_status,
+            "encryption": blob.crypt_mode()?,
+            "referenced-by": references
+        }),
+        None => json!({
+             "crc": crc_status,
+             "encryption": blob.crypt_mode()?,
+        }),
+    };
+
+    if output_format == "text" {
+        println!("CRC: {}", val["crc"]);
+        println!("encryption: {}", val["encryption"]);
+        if let Some(refs) = val["referenced-by"].as_array() {
+            println!("referenced by:");
+            for reference in refs {
+                println!("  {}", reference);
+            }
+        }
+    } else {
+        format_and_print_result(&val, &output_format);
+    }
+    Ok(())
+}
+
+pub fn inspect_commands() -> CommandLineInterface {
+    let cmd_def = CliCommandMap::new().insert(
+        "chunk",
+        CliCommand::new(&API_METHOD_INSPECT_CHUNK).arg_param(&["chunk"]),
+    );
+
+    cmd_def.into()
+}
diff --git a/src/bin/proxmox_backup_debug/mod.rs b/src/bin/proxmox_backup_debug/mod.rs
new file mode 100644
index 00000000..644583db
--- /dev/null
+++ b/src/bin/proxmox_backup_debug/mod.rs
@@ -0,0 +1,2 @@
+mod inspect;
+pub use inspect::*;
diff --git a/src/tools/mod.rs b/src/tools/mod.rs
index b6c55ac2..dfff8c10 100644
--- a/src/tools/mod.rs
+++ b/src/tools/mod.rs
@@ -2,7 +2,10 @@
 //!
 //! This is a collection of small and useful tools.
 use std::any::Any;
+use std::fs::File;
+use std::io::{stdout, Write};
 use std::os::unix::io::RawFd;
+use std::path::Path;
 
 use anyhow::{bail, format_err, Error};
 use openssl::hash::{hash, DigestBytes, MessageDigest};
@@ -224,3 +227,13 @@ pub fn create_run_dir() -> Result<(), Error> {
     let _: bool = create_path(pbs_buildcfg::PROXMOX_BACKUP_RUN_DIR_M!(), None, Some(opts))?;
     Ok(())
 }
+
+/// Returns either a new file, if a path is given, or stdout, if no path is given.
+pub fn outfile_or_stdout<P: AsRef<Path>>(path: Option<P>) -> Result<Box<dyn Write>, Error> {
+    if let Some(path) = path {
+        let f = File::create(path)?;
+        Ok(Box::new(f) as Box<dyn Write>)
+    } else {
+        Ok(Box::new(stdout()) as Box<dyn Write>)
+    }
+}
-- 
2.30.2

[pbs-devel] [PATCH v7 proxmox-backup 2/3] add file inspection to pb-debug

[Hannes Laimer]

Adds possibility to inspect .blob, .fidx and .didx files. For index
files a list of the chunks referenced will be printed in addition to
some other information. .blob files can be decoded into file or directly
into stdout. Without decode the tool just prints the size and encryption
mode of the blob file. Options:
 - file: path to the file
 - [opt] decode: path to a file or stdout(-), if specidied, the file will be
   decoded into the specified location [only for blob files, no effect
   with index files]
 - [opt] keyfile: path to a keyfile, needed if decode is specified and the
   data was encrypted

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 src/bin/proxmox_backup_debug/inspect.rs | 132 +++++++++++++++++++++++-
 1 file changed, 127 insertions(+), 5 deletions(-)

diff --git a/src/bin/proxmox_backup_debug/inspect.rs b/src/bin/proxmox_backup_debug/inspect.rs
index 5a0979d2..c06875d0 100644
--- a/src/bin/proxmox_backup_debug/inspect.rs
+++ b/src/bin/proxmox_backup_debug/inspect.rs
@@ -1,6 +1,9 @@
+use std::collections::HashSet;
+use std::fs::File;
+use std::io::{Read, Seek, SeekFrom};
 use std::path::Path;
 
-use anyhow::{format_err, Error};
+use anyhow::{bail, format_err, Error};
 use proxmox::api::cli::{
     format_and_print_result, get_output_format, CliCommand, CliCommandMap, CommandLineInterface,
 };
@@ -10,6 +13,8 @@ use walkdir::WalkDir;
 
 use proxmox_backup::backup::{
     load_and_decrypt_key, CryptConfig, DataBlob, DynamicIndexReader, FixedIndexReader, IndexFile,
+    COMPRESSED_BLOB_MAGIC_1_0, DYNAMIC_SIZED_CHUNK_INDEX_1_0, ENCRYPTED_BLOB_MAGIC_1_0,
+    ENCR_COMPR_BLOB_MAGIC_1_0, FIXED_SIZED_CHUNK_INDEX_1_0, UNCOMPRESSED_BLOB_MAGIC_1_0,
 };
 
 use pbs_client::tools::key_source::get_encryption_key_password;
@@ -203,11 +208,128 @@ fn inspect_chunk(
     Ok(())
 }
 
+#[api(
+    input: {
+        properties: {
+            file: {
+                description: "Path to the file.",
+                type: String,
+            },
+            "decode": {
+                description: "Path to the file to which the file should be decoded, '-' -> decode to stdout.",
+                type: String,
+                optional: true,
+            },
+            "keyfile": {
+                description: "Path to the keyfile with which the file was encrypted.",
+                type: String,
+                optional: true,
+            },
+            "output-format": {
+                schema: OUTPUT_FORMAT,
+                optional: true,
+            },
+        }
+    }
+)]
+/// Inspect a file, for blob file without decode only the size and encryption mode is printed
+fn inspect_file(
+    file: String,
+    decode: Option<String>,
+    keyfile: Option<String>,
+    param: Value,
+) -> Result<(), Error> {
+    let output_format = get_output_format(&param);
+
+    let mut file = File::open(Path::new(&file))?;
+    let mut magic = [0; 8];
+    file.read_exact(&mut magic)?;
+    file.seek(SeekFrom::Start(0))?;
+    let val = match magic {
+        UNCOMPRESSED_BLOB_MAGIC_1_0
+        | COMPRESSED_BLOB_MAGIC_1_0
+        | ENCRYPTED_BLOB_MAGIC_1_0
+        | ENCR_COMPR_BLOB_MAGIC_1_0 => {
+            let data_blob = DataBlob::load_from_reader(&mut file)?;
+            let key_file_path = keyfile.as_ref().map(Path::new);
+
+            let decode_output_path = decode.as_ref().map(Path::new);
+
+            if decode_output_path.is_some() {
+                decode_blob(decode_output_path, key_file_path, None, &data_blob)?;
+            }
+
+            let crypt_mode = data_blob.crypt_mode()?;
+            json!({
+                "encryption": crypt_mode,
+                "size": data_blob.raw_size(),
+            })
+        }
+        FIXED_SIZED_CHUNK_INDEX_1_0 | DYNAMIC_SIZED_CHUNK_INDEX_1_0 => {
+            let index: Box<dyn IndexFile> = match magic {
+                FIXED_SIZED_CHUNK_INDEX_1_0 => {
+                    Box::new(FixedIndexReader::new(file)?) as Box<dyn IndexFile>
+                }
+                DYNAMIC_SIZED_CHUNK_INDEX_1_0 => {
+                    Box::new(DynamicIndexReader::new(file)?) as Box<dyn IndexFile>
+                }
+                _ => bail!(format_err!("This is technically not possible")),
+            };
+
+            let mut ctime_str = index.index_ctime().to_string();
+            if let Ok(s) = proxmox::tools::time::strftime_local("%c", index.index_ctime()) {
+                ctime_str = s;
+            }
+
+            let mut chunk_digests = HashSet::new();
+
+            for pos in 0..index.index_count() {
+                let digest = index.index_digest(pos).unwrap();
+                chunk_digests.insert(proxmox::tools::digest_to_hex(digest));
+            }
+
+            json!({
+                "size": index.index_size(),
+                "ctime": ctime_str,
+                "chunk-digests": chunk_digests
+            })
+        }
+        _ => bail!(format_err!(
+            "Only .blob, .fidx and .didx files may be inspected"
+        )),
+    };
+
+    if output_format == "text" {
+        println!("size: {}", val["size"]);
+        if let Some(encryption) = val["encryption"].as_str() {
+            println!("encryption: {}", encryption);
+        }
+        if let Some(ctime) = val["ctime"].as_str() {
+            println!("creation time: {}", ctime);
+        }
+        if let Some(chunks) = val["chunk-digests"].as_array() {
+            println!("chunks:");
+            for chunk in chunks {
+                println!("  {}", chunk);
+            }
+        }
+    } else {
+        format_and_print_result(&val, &output_format);
+    }
+
+    Ok(())
+}
+
 pub fn inspect_commands() -> CommandLineInterface {
-    let cmd_def = CliCommandMap::new().insert(
-        "chunk",
-        CliCommand::new(&API_METHOD_INSPECT_CHUNK).arg_param(&["chunk"]),
-    );
+    let cmd_def = CliCommandMap::new()
+        .insert(
+            "chunk",
+            CliCommand::new(&API_METHOD_INSPECT_CHUNK).arg_param(&["chunk"]),
+        )
+        .insert(
+            "file",
+            CliCommand::new(&API_METHOD_INSPECT_FILE).arg_param(&["file"]),
+        );
 
     cmd_def.into()
 }
-- 
2.30.2

[pbs-devel] [PATCH v7 proxmox-backup 3/3] add index recovery to pb-debug

[Hannes Laimer]

Adds possibility to recover data from an index file. Options:
 - chunks: path to the directory where the chunks are saved
 - file: the index file that should be recovered(must be either .fidx or
   didx)
 - [opt] keyfile: path to a keyfile, if the data was encrypted, a keyfile is
   needed
 - [opt] skip-crc: boolean, if true, read chunks wont be verified with their
   crc-sum, increases the restore speed by a lot

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 src/bin/proxmox-backup-debug.rs         |   6 +-
 src/bin/proxmox_backup_debug/mod.rs     |   2 +
 src/bin/proxmox_backup_debug/recover.rs | 119 ++++++++++++++++++++++++
 3 files changed, 125 insertions(+), 2 deletions(-)
 create mode 100644 src/bin/proxmox_backup_debug/recover.rs

diff --git a/src/bin/proxmox-backup-debug.rs b/src/bin/proxmox-backup-debug.rs
index 897fa221..b3360c93 100644
--- a/src/bin/proxmox-backup-debug.rs
+++ b/src/bin/proxmox-backup-debug.rs
@@ -1,12 +1,14 @@
 use proxmox::api::cli::*;
 
 mod proxmox_backup_debug;
-use proxmox_backup_debug::inspect_commands;
+use proxmox_backup_debug::{inspect_commands, recover_commands};
 
 fn main() {
     proxmox_backup::tools::setup_safe_path_env();
 
-    let cmd_def = CliCommandMap::new().insert("inspect", inspect_commands());
+    let cmd_def = CliCommandMap::new()
+        .insert("inspect", inspect_commands())
+        .insert("recover", recover_commands());
 
     let rpcenv = CliEnvironment::new();
     run_cli_command(cmd_def, rpcenv, Some(|future| pbs_runtime::main(future)));
diff --git a/src/bin/proxmox_backup_debug/mod.rs b/src/bin/proxmox_backup_debug/mod.rs
index 644583db..62df7754 100644
--- a/src/bin/proxmox_backup_debug/mod.rs
+++ b/src/bin/proxmox_backup_debug/mod.rs
@@ -1,2 +1,4 @@
 mod inspect;
 pub use inspect::*;
+mod recover;
+pub use recover::*;
diff --git a/src/bin/proxmox_backup_debug/recover.rs b/src/bin/proxmox_backup_debug/recover.rs
new file mode 100644
index 00000000..706223b1
--- /dev/null
+++ b/src/bin/proxmox_backup_debug/recover.rs
@@ -0,0 +1,119 @@
+use std::fs::File;
+use std::io::{Read, Seek, SeekFrom, Write};
+use std::path::Path;
+
+use anyhow::{bail, format_err, Error};
+
+use proxmox::api::api;
+use proxmox::api::cli::{CliCommand, CliCommandMap, CommandLineInterface};
+use proxmox_backup::backup::{DYNAMIC_SIZED_CHUNK_INDEX_1_0, FIXED_SIZED_CHUNK_INDEX_1_0};
+use serde_json::Value;
+
+use proxmox_backup::backup::{
+    load_and_decrypt_key, CryptConfig, DataBlob, DynamicIndexReader, FixedIndexReader, IndexFile,
+};
+
+use pbs_client::tools::key_source::get_encryption_key_password;
+
+use proxmox::tools::digest_to_hex;
+
+#[api(
+    input: {
+        properties: {
+            file: {
+                description: "Path to the index file, either .fidx or .didx.",
+                type: String,
+            },
+            chunks: {
+                description: "Path to the directorty that contains the chunks, usually <datastore>/.chunks.",
+                type: String,
+            },
+            "keyfile": {
+                description: "Path to a keyfile, if the data was encrypted, a keyfile is needed for decryption.",
+                type: String,
+                optional: true,
+            },
+            "skip-crc": {
+                description: "Skip the crc verification, increases the restore speed by lot.",
+                type: Boolean,
+                optional: true,
+                default: false,
+            }
+        }
+    }
+)]
+/// Restore the data from an index file, given the directory of where chunks
+/// are saved, the index file and a keyfile, if needed for decryption.
+fn recover_index(
+    file: String,
+    chunks: String,
+    keyfile: Option<String>,
+    skip_crc: bool,
+    _param: Value,
+) -> Result<(), Error> {
+    let file_path = Path::new(&file);
+    let chunks_path = Path::new(&chunks);
+
+    let key_file_path = keyfile.as_ref().map(Path::new);
+
+    let mut file = File::open(Path::new(&file))?;
+    let mut magic = [0; 8];
+    file.read_exact(&mut magic)?;
+    file.seek(SeekFrom::Start(0))?;
+    let index: Box<dyn IndexFile> = match magic {
+        FIXED_SIZED_CHUNK_INDEX_1_0 => Box::new(FixedIndexReader::new(file)?) as Box<dyn IndexFile>,
+        DYNAMIC_SIZED_CHUNK_INDEX_1_0 => {
+            Box::new(DynamicIndexReader::new(file)?) as Box<dyn IndexFile>
+        }
+        _ => bail!(format_err!(
+            "index file must either be a .fidx or a .didx file"
+        )),
+    };
+
+    let crypt_conf_opt = if let Some(key_file_path) = key_file_path {
+        let (key, _created, _fingerprint) =
+            load_and_decrypt_key(&key_file_path, &get_encryption_key_password)?;
+        Some(CryptConfig::new(key)?)
+    } else {
+        None
+    };
+
+    let output_filename = file_path.file_stem().unwrap().to_str().unwrap();
+    let output_path = Path::new(output_filename);
+    let mut output_file = File::create(output_path)
+        .map_err(|e| format_err!("could not create output file - {}", e))?;
+
+    let mut data = Vec::with_capacity(4 * 1024 * 1024);
+    for pos in 0..index.index_count() {
+        let chunk_digest = index.index_digest(pos).unwrap();
+        let digest_str = digest_to_hex(chunk_digest);
+        let digest_prefix = &digest_str[0..4];
+        let chunk_path = chunks_path.join(digest_prefix).join(digest_str);
+        let mut chunk_file = std::fs::File::open(&chunk_path)
+            .map_err(|e| format_err!("could not open chunk file - {}", e))?;
+
+        data.clear();
+        chunk_file.read_to_end(&mut data)?;
+        let chunk_blob = DataBlob::from_raw(data.clone())?;
+
+        if !skip_crc {
+            chunk_blob.verify_crc()?;
+        }
+
+        output_file.write_all(
+            chunk_blob
+                .decode(crypt_conf_opt.as_ref(), Some(chunk_digest))?
+                .as_slice(),
+        )?;
+    }
+
+    Ok(())
+}
+
+pub fn recover_commands() -> CommandLineInterface {
+    let cmd_def = CliCommandMap::new().insert(
+        "index",
+        CliCommand::new(&API_METHOD_RECOVER_INDEX).arg_param(&["file", "chunks"]),
+    );
+    cmd_def.into()
+}
-- 
2.30.2

[pbs-devel] applied: [PATCH v7 proxmox-backup 0/3] add proxmox-backup-debug binary

[Wolfgang Bumiller]

applied series

On Mon, Aug 30, 2021 at 10:53:36AM +0200, Hannes Laimer wrote:
> Adds a new proxmox-backup-debug binary for recovery or inspection of
> index, blob and chunk files
> 
> v7:
>   - handle 'text' output-format properly
>   - describe inspect file a little more
> 
> v6(based on Wolfgang Bumiller <w.bumiller@proxmox.com>'s feedback):
>   - removed unnecessary SCHEMA references
>   - added few args with arg_param
>   - removed unnecessary Result wrapping
>   - moved logic for checking whether decode is to stdout into
>     decode_blob function
>   - improved argument description
> 
> v5:
>   - improved some parts based on Dominik Csapak <d.csapak@proxmox.com>
>     feedback on v4
>   - changes are described in the patches themself
> 
> v4:
>   - rewrote parts based on Wolfgang Bumiller <w.bumiller@proxmox.com>
>     feedback on v3
>   - check filetype mostly by magic number, except when
>     looking for index files
> 
> v3:
>   - move description into commit message
> 
> v2:
>   - appiled suggestions + rewrote a few parts
>   - renamed 'restore' to 'debug'<Paste>
> 
> Hannes Laimer (3):
>   add chunk inspection to pb-debug
>   add file inspection to pb-debug
>   add index recovery to pb-debug
> 
>  Makefile                                |   3 +-
>  src/bin/proxmox-backup-debug.rs         |  15 ++
>  src/bin/proxmox_backup_debug/inspect.rs | 335 ++++++++++++++++++++++++
>  src/bin/proxmox_backup_debug/mod.rs     |   4 +
>  src/bin/proxmox_backup_debug/recover.rs | 119 +++++++++
>  src/tools/mod.rs                        |  13 +
>  6 files changed, 488 insertions(+), 1 deletion(-)
>  create mode 100644 src/bin/proxmox-backup-debug.rs
>  create mode 100644 src/bin/proxmox_backup_debug/inspect.rs
>  create mode 100644 src/bin/proxmox_backup_debug/mod.rs
>  create mode 100644 src/bin/proxmox_backup_debug/recover.rs
> 
> -- 
> 2.30.2