From: Dominik Csapak <d.csapak@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [PATCH proxmox-backup v2] chunk_store: insert_chunk: write chunk again if it is empty on disk
Date: Mon, 9 May 2022 16:09:06 +0200 [thread overview]
Message-ID: <20220509140906.3488498-1-d.csapak@proxmox.com> (raw)
and issue a warning. We can do this, because we know an empty chunk cannot be
valid, and we (assumedly) have a valid chunk in memory.
Having empty chunks on disk is currently possible when PBS crashes, but the
rename of the chunk was flushed to disk, when the actual data was not.
If it's not empty but there is a size mismatch, return an error.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
replaces: https://lists.proxmox.com/pipermail/pbs-devel/2022-May/005042.html
changes from v1:
* warn only on old_size == 0
* fix old_size/new_size
* expand commit message
we could do further checks here, but those should be caught by verify.
the 'real' fix for most of the 0-size chunks will be to either fsync
the chunks after write/rename, or do a syncfs before the backup ends
pbs-datastore/src/chunk_store.rs | 20 +++++++++++++-------
1 file changed, 13 insertions(+), 7 deletions(-)
diff --git a/pbs-datastore/src/chunk_store.rs b/pbs-datastore/src/chunk_store.rs
index 8d7df513..85e7cc71 100644
--- a/pbs-datastore/src/chunk_store.rs
+++ b/pbs-datastore/src/chunk_store.rs
@@ -458,17 +458,26 @@ impl ChunkStore {
let lock = self.mutex.lock();
+ let raw_data = chunk.raw_data();
+ let encoded_size = raw_data.len() as u64;
+
if let Ok(metadata) = std::fs::metadata(&chunk_path) {
- if metadata.is_file() {
- self.touch_chunk(digest)?;
- return Ok((true, metadata.len()));
- } else {
+ if !metadata.is_file() {
bail!(
"Got unexpected file type on store '{}' for chunk {}",
self.name,
digest_str
);
}
+ let old_size = metadata.len();
+ if encoded_size == old_size {
+ self.touch_chunk(digest)?;
+ return Ok((true, old_size));
+ } else if old_size == 0 {
+ log::warn!("found empty chunk '{digest_str}' on disk, overwriting");
+ } else {
+ bail!("found chunk size mismatch for '{digest_str}': old {old_size} - new {encoded_size}");
+ }
}
let mut tmp_path = chunk_path.clone();
@@ -483,9 +492,6 @@ impl ChunkStore {
)
})?;
- let raw_data = chunk.raw_data();
- let encoded_size = raw_data.len() as u64;
-
file.write_all(raw_data).map_err(|err| {
format_err!(
"writing temporary chunk on store '{}' failed for {} - {}",
--
2.30.2
next reply other threads:[~2022-05-09 14:09 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-05-09 14:09 Dominik Csapak [this message]
2022-05-10 7:47 ` [pbs-devel] applied: " Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220509140906.3488498-1-d.csapak@proxmox.com \
--to=d.csapak@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox