* [pbs-devel] [PATCH proxmox-backup v2] chunk_store: insert_chunk: write chunk again if it is empty on disk
@ 2022-05-09 14:09 Dominik Csapak
2022-05-10 7:47 ` [pbs-devel] applied: " Thomas Lamprecht
0 siblings, 1 reply; 2+ messages in thread
From: Dominik Csapak @ 2022-05-09 14:09 UTC (permalink / raw)
To: pbs-devel
and issue a warning. We can do this, because we know an empty chunk cannot be
valid, and we (assumedly) have a valid chunk in memory.
Having empty chunks on disk is currently possible when PBS crashes, but the
rename of the chunk was flushed to disk, when the actual data was not.
If it's not empty but there is a size mismatch, return an error.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
replaces: https://lists.proxmox.com/pipermail/pbs-devel/2022-May/005042.html
changes from v1:
* warn only on old_size == 0
* fix old_size/new_size
* expand commit message
we could do further checks here, but those should be caught by verify.
the 'real' fix for most of the 0-size chunks will be to either fsync
the chunks after write/rename, or do a syncfs before the backup ends
pbs-datastore/src/chunk_store.rs | 20 +++++++++++++-------
1 file changed, 13 insertions(+), 7 deletions(-)
diff --git a/pbs-datastore/src/chunk_store.rs b/pbs-datastore/src/chunk_store.rs
index 8d7df513..85e7cc71 100644
--- a/pbs-datastore/src/chunk_store.rs
+++ b/pbs-datastore/src/chunk_store.rs
@@ -458,17 +458,26 @@ impl ChunkStore {
let lock = self.mutex.lock();
+ let raw_data = chunk.raw_data();
+ let encoded_size = raw_data.len() as u64;
+
if let Ok(metadata) = std::fs::metadata(&chunk_path) {
- if metadata.is_file() {
- self.touch_chunk(digest)?;
- return Ok((true, metadata.len()));
- } else {
+ if !metadata.is_file() {
bail!(
"Got unexpected file type on store '{}' for chunk {}",
self.name,
digest_str
);
}
+ let old_size = metadata.len();
+ if encoded_size == old_size {
+ self.touch_chunk(digest)?;
+ return Ok((true, old_size));
+ } else if old_size == 0 {
+ log::warn!("found empty chunk '{digest_str}' on disk, overwriting");
+ } else {
+ bail!("found chunk size mismatch for '{digest_str}': old {old_size} - new {encoded_size}");
+ }
}
let mut tmp_path = chunk_path.clone();
@@ -483,9 +492,6 @@ impl ChunkStore {
)
})?;
- let raw_data = chunk.raw_data();
- let encoded_size = raw_data.len() as u64;
-
file.write_all(raw_data).map_err(|err| {
format_err!(
"writing temporary chunk on store '{}' failed for {} - {}",
--
2.30.2
^ permalink raw reply [flat|nested] 2+ messages in thread
* [pbs-devel] applied: [PATCH proxmox-backup v2] chunk_store: insert_chunk: write chunk again if it is empty on disk
2022-05-09 14:09 [pbs-devel] [PATCH proxmox-backup v2] chunk_store: insert_chunk: write chunk again if it is empty on disk Dominik Csapak
@ 2022-05-10 7:47 ` Thomas Lamprecht
0 siblings, 0 replies; 2+ messages in thread
From: Thomas Lamprecht @ 2022-05-10 7:47 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Dominik Csapak
Am 5/9/22 um 16:09 schrieb Dominik Csapak:
> and issue a warning. We can do this, because we know an empty chunk cannot be
> valid, and we (assumedly) have a valid chunk in memory.
>
> Having empty chunks on disk is currently possible when PBS crashes, but the
> rename of the chunk was flushed to disk, when the actual data was not.
>
> If it's not empty but there is a size mismatch, return an error.
>
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
> replaces: https://lists.proxmox.com/pipermail/pbs-devel/2022-May/005042.html
>
> changes from v1:
> * warn only on old_size == 0
> * fix old_size/new_size
> * expand commit message
>
> we could do further checks here, but those should be caught by verify.
> the 'real' fix for most of the 0-size chunks will be to either fsync
> the chunks after write/rename, or do a syncfs before the backup ends
>
> pbs-datastore/src/chunk_store.rs | 20 +++++++++++++-------
> 1 file changed, 13 insertions(+), 7 deletions(-)
>
>
applied, thanks!
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2022-05-10 7:48 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-05-09 14:09 [pbs-devel] [PATCH proxmox-backup v2] chunk_store: insert_chunk: write chunk again if it is empty on disk Dominik Csapak
2022-05-10 7:47 ` [pbs-devel] applied: " Thomas Lamprecht
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox