[pbs-devel] [PATCH proxmox-backup 1/3] backup: ensure no fixed index writers are left over either

[pbs-devel] [PATCH proxmox-backup 1/3] backup: ensure no fixed index writers are left over either

[Stefan Reiter]

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
---
 src/api2/backup/environment.rs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 973563d3..f635c6f7 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -457,11 +457,11 @@ impl BackupEnvironment {
     /// Mark backup as finished
     pub fn finish_backup(&self) -> Result<(), Error> {
         let mut state = self.state.lock().unwrap();
-        // test if all writer are correctly closed
 
         state.ensure_unfinished()?;
 
-        if state.dynamic_writers.len() != 0 {
+        // test if all writer are correctly closed
+        if state.dynamic_writers.len() != 0 || state.fixed_writers.len() != 0 {
             bail!("found open index writer - unable to finish backup");
         }
 
-- 
2.20.1

[pbs-devel] [PATCH RESEND proxmox-backup 2/3] backup: check verify state of previous backup before allowing reuse

[Stefan Reiter]

Do not allow clients to reuse chunks from the previous backup if it has
a failed validation result. This would result in a new "successful"
backup that potentially references broken chunks.

If the previous backup has not been verified, assume it is fine and
continue on.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
---

Patch 3 is much more useful with this one applied first, so here it is again
instead of a ping for easy review/application :)

 src/api2/backup.rs | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/src/api2/backup.rs b/src/api2/backup.rs
index ad608d85..c0b1d985 100644
--- a/src/api2/backup.rs
+++ b/src/api2/backup.rs
@@ -652,6 +652,19 @@ fn download_previous(
             None => bail!("no previous backup"),
         };
 
+        let (manifest, _) = env.datastore.load_manifest(&last_backup.backup_dir)?;
+        let verify = manifest.unprotected["verify_state"].clone();
+        match serde_json::from_value::<SnapshotVerifyState>(verify) {
+            Ok(verify) => {
+                if verify.state != "ok" {
+                    bail!("previous backup has failed verification");
+                }
+            },
+            Err(_) => {
+                // no verify state found, ignore and treat as valid
+            }
+        };
+
         let mut path = env.datastore.snapshot_path(&last_backup.backup_dir);
         path.push(&archive_name);
 
-- 
2.20.1

[pbs-devel] [PATCH v3 proxmox-backup 3/3] backup: check all referenced chunks actually exist

[Stefan Reiter]

A client can omit uploading chunks in the "known_chunks" list, those
then also won't be written on the server side. Check all those chunks
mentioned in the index but not uploaded for existance and report an
error if they don't exist instead of marking a potentially broken backup
as "successful".

This is only important if the base snapshot references corrupted chunks,
but has not been negatively verified. Also, it is important to only
verify this at the end, *after* all index writers are closed, since only
then can it be guaranteed that no GC will sweep referenced chunks away.

If a chunk is found missing, also mark the previous backup with a
verification failure, since we know the missing chunk has to referenced
in it (only way it could have been inserted to known_chunks with
checked=false). This has the benefit of automatically doing a
full-upload backup if the user attempts to retry after seeing the new
error, instead of requiring a manual verify or forget.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
---

v3:
* reuse known_chunks map instead of new HashSet
* create named typed for known_chunks
* refactor check into helper function
* mark previous backup with 'bad' verify to make next backup succeed

@Dietmar: this patch is useful since the last snapshot (base) might be corrupted
but we don't know since it might not have been verified. Of course, actually
corrupted chunks are still not detected (expensive), but at least missing ones.

v2 here: https://lists.proxmox.com/pipermail/pbs-devel/2020-September/000572.html


 src/api2/backup/environment.rs | 58 ++++++++++++++++++++++++++++++----
 1 file changed, 52 insertions(+), 6 deletions(-)

diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index f635c6f7..22b96c22 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -9,7 +9,7 @@ use proxmox::tools::digest_to_hex;
 use proxmox::tools::fs::{replace_file, CreateOptions};
 use proxmox::api::{RpcEnvironment, RpcEnvironmentType};
 
-use crate::api2::types::Userid;
+use crate::api2::types::{Userid, SnapshotVerifyState};
 use crate::backup::*;
 use crate::server::WorkerTask;
 use crate::server::formatter::*;
@@ -66,13 +66,16 @@ struct FixedWriterState {
     incremental: bool,
 }
 
+// key=digest, value=(length, existance checked)
+type KnownChunksMap = HashMap<[u8;32], (u32, bool)>;
+
 struct SharedBackupState {
     finished: bool,
     uid_counter: usize,
     file_counter: usize, // successfully uploaded files
     dynamic_writers: HashMap<usize, DynamicWriterState>,
     fixed_writers: HashMap<usize, FixedWriterState>,
-    known_chunks: HashMap<[u8;32], u32>,
+    known_chunks: KnownChunksMap,
     backup_size: u64, // sums up size of all files
     backup_stat: UploadStatistic,
 }
@@ -153,7 +156,7 @@ impl BackupEnvironment {
 
         state.ensure_unfinished()?;
 
-        state.known_chunks.insert(digest, length);
+        state.known_chunks.insert(digest, (length, false));
 
         Ok(())
     }
@@ -195,7 +198,7 @@ impl BackupEnvironment {
         if is_duplicate { data.upload_stat.duplicates += 1; }
 
         // register chunk
-        state.known_chunks.insert(digest, size);
+        state.known_chunks.insert(digest, (size, true));
 
         Ok(())
     }
@@ -228,7 +231,7 @@ impl BackupEnvironment {
         if is_duplicate { data.upload_stat.duplicates += 1; }
 
         // register chunk
-        state.known_chunks.insert(digest, size);
+        state.known_chunks.insert(digest, (size, true));
 
         Ok(())
     }
@@ -237,7 +240,7 @@ impl BackupEnvironment {
         let state = self.state.lock().unwrap();
 
         match state.known_chunks.get(digest) {
-            Some(len) => Some(*len),
+            Some((len, _)) => Some(*len),
             None => None,
         }
     }
@@ -454,6 +457,47 @@ impl BackupEnvironment {
         Ok(())
     }
 
+    /// Ensure all chunks referenced in this backup actually exist.
+    /// Only call *after* all writers have been closed, to avoid race with GC.
+    /// In case of error, mark the previous backup as 'verify failed'.
+    fn verify_chunk_existance(&self, known_chunks: &KnownChunksMap) -> Result<(), Error> {
+        for (digest, (_, checked)) in known_chunks.iter() {
+            if !checked && !self.datastore.chunk_path(digest).0.exists() {
+                let mark_msg = if let Some(ref last_backup) = self.last_backup {
+                    let last_dir = &last_backup.backup_dir;
+                    let verify_state = SnapshotVerifyState {
+                        state: "failed".to_owned(),
+                        upid: self.worker.upid().clone(),
+                    };
+
+                    let res = proxmox::try_block!{
+                        let (mut manifest, _) = self.datastore.load_manifest(last_dir)?;
+                        manifest.unprotected["verify_state"] = serde_json::to_value(verify_state)?;
+                        self.datastore.store_manifest(last_dir, serde_json::to_value(manifest)?)
+                    };
+
+                    if let Err(err) = res {
+                        format!("tried marking previous snapshot as bad, \
+                                but got error accessing manifest: {}", err)
+                    } else {
+                        "marked previous snapshot as bad, please use \
+                        'verify' for a detailed check".to_owned()
+                    }
+                } else {
+                    "internal error: no base backup registered to mark invalid".to_owned()
+                };
+
+                bail!(
+                    "chunk '{}' was attempted to be reused but doesn't exist - {}",
+                    digest_to_hex(digest),
+                    mark_msg
+                );
+            }
+        }
+
+        Ok(())
+    }
+
     /// Mark backup as finished
     pub fn finish_backup(&self) -> Result<(), Error> {
         let mut state = self.state.lock().unwrap();
@@ -490,6 +534,8 @@ impl BackupEnvironment {
             }
         }
 
+        self.verify_chunk_existance(&state.known_chunks)?;
+
         // marks the backup as successful
         state.finished = true;
 
-- 
2.20.1

[pbs-devel] applied: [PATCH proxmox-backup 1/3] backup: ensure no fixed index writers are left over either

[Dietmar Maurer]

applied

> On 09/08/2020 3:29 PM Stefan Reiter <s.reiter@proxmox.com> wrote:
> 
>  
> Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
> ---
>  src/api2/backup/environment.rs | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
> index 973563d3..f635c6f7 100644
> --- a/src/api2/backup/environment.rs
> +++ b/src/api2/backup/environment.rs
> @@ -457,11 +457,11 @@ impl BackupEnvironment {
>      /// Mark backup as finished
>      pub fn finish_backup(&self) -> Result<(), Error> {
>          let mut state = self.state.lock().unwrap();
> -        // test if all writer are correctly closed
>  
>          state.ensure_unfinished()?;
>  
> -        if state.dynamic_writers.len() != 0 {
> +        // test if all writer are correctly closed
> +        if state.dynamic_writers.len() != 0 || state.fixed_writers.len() != 0 {
>              bail!("found open index writer - unable to finish backup");
>          }
>  
> -- 
> 2.20.1
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH RESEND proxmox-backup 2/3] backup: check verify state of previous backup before allowing reuse

[Dietmar Maurer]

> On 09/08/2020 3:29 PM Stefan Reiter <s.reiter@proxmox.com> wrote:
> 
>  
> Do not allow clients to reuse chunks from the previous backup if it has
> a failed validation result. This would result in a new "successful"
> backup that potentially references broken chunks.

This patch does not prevent that, because in create_fixed_index():

 
        let last_backup = match &env.last_backup {
            Some(info) => info,
            None => {
                bail!("cannot reuse index - no previous backup exists");
            }
        };

So we allow reuse there!

I suggest to do those checks earlier in upgrade_to_backup_protocol()

> 
> If the previous backup has not been verified, assume it is fine and
> continue on.
> 
> Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
> ---
> 
> Patch 3 is much more useful with this one applied first, so here it is again
> instead of a ping for easy review/application :)
> 
>  src/api2/backup.rs | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/src/api2/backup.rs b/src/api2/backup.rs
> index ad608d85..c0b1d985 100644
> --- a/src/api2/backup.rs
> +++ b/src/api2/backup.rs
> @@ -652,6 +652,19 @@ fn download_previous(
>              None => bail!("no previous backup"),
>          };
>  
> +        let (manifest, _) = env.datastore.load_manifest(&last_backup.backup_dir)?;
> +        let verify = manifest.unprotected["verify_state"].clone();
> +        match serde_json::from_value::<SnapshotVerifyState>(verify) {
> +            Ok(verify) => {
> +                if verify.state != "ok" {
> +                    bail!("previous backup has failed verification");
> +                }
> +            },
> +            Err(_) => {
> +                // no verify state found, ignore and treat as valid
> +            }
> +        };
> +
>          let mut path = env.datastore.snapshot_path(&last_backup.backup_dir);
>          path.push(&archive_name);
>  
> -- 
> 2.20.1
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel