[pbs-devel] [PATCH proxmox/proxmox-backup] restore files from pxar sparsely

[pbs-devel] [PATCH proxmox/proxmox-backup] restore files from pxar sparsely

[Dominik Csapak]

to be able to restore containers with big sparse files

ideally(?) we would save hole information directly in the pxar archive
and not even use zero chunks, so that we can have smaller
pxar archives, and accurately restore sparse files like they were before

for now, restore all files sparsely

changes from RFC:
* drop the zero module of proxmox, rust can generate fast code by itself

proxmox:

Dominik Csapak (1):
  proxmox: add sparse_copy(_async) to tools::io

 proxmox/src/tools/io/mod.rs | 70 +++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)

proxmox-backup:

Dominik Csapak (1):
  pxar/extract: if possible create files sparesly

 src/pxar/extract.rs | 32 ++++++++++++++++++++++++++------
 1 file changed, 26 insertions(+), 6 deletions(-)

-- 
2.20.1

[pbs-devel] [PATCH proxmox 1/1] proxmox: add sparse_copy(_async) to tools::io

[Dominik Csapak]

this is able to seek the target instead of writing zeroes, which
generates sparse files where supported

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
changes from RFC:
* use minimal buffer_is_zero version that generates fast code on release
  compilation
 proxmox/src/tools/io/mod.rs | 70 +++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)

diff --git a/proxmox/src/tools/io/mod.rs b/proxmox/src/tools/io/mod.rs
index 2e92ebb..299d19c 100644
--- a/proxmox/src/tools/io/mod.rs
+++ b/proxmox/src/tools/io/mod.rs
@@ -3,8 +3,78 @@
 //! The [`ReadExt`] trait provides additional operations for handling byte buffers for types
 //! implementing [`Read`](std::io::Read).
 
+use std::io::{self, Read, Write, Seek, SeekFrom, ErrorKind};
+
 mod read;
 pub use read::*;
 
 mod write;
 pub use write::*;
+
+fn buffer_is_zero(buf: &[u8]) -> bool {
+    !buf
+        .chunks(128)
+        .map(|aa|
+            aa.iter().fold(0, |a, b| a|b) != 0
+        ).any(|a| a)
+}
+
+/// copy similar to io::copy, but seeks the target when encountering
+/// zero bytes instead of writing them
+pub fn sparse_copy<R: Read + ?Sized, W: Write + Seek + ?Sized>(
+    reader: &mut R,
+    writer: &mut W,
+) -> Result<u64, io::Error> {
+    let mut buf = crate::tools::byte_buffer::ByteBuffer::new();
+    let mut written = 0;
+    loop {
+        let len = match buf.read_from(reader) {
+            Ok(0) => return Ok(written),
+            Ok(len) => len,
+            Err(ref e) if e.kind() == ErrorKind::Interrupted => continue,
+            Err(e) => return Err(e),
+        };
+
+        if buffer_is_zero(&buf[..]) {
+            writer.seek(SeekFrom::Current(len as i64))?;
+        } else {
+            writer.write_all(&buf[..])?;
+        }
+        buf.clear();
+        written += len as u64;
+    }
+}
+
+#[cfg(feature = "tokio")]
+use tokio::io::{AsyncReadExt, AsyncWriteExt, AsyncSeekExt};
+
+#[cfg(feature = "tokio")]
+/// copy similar to tokio::io::copy, but seeks the target when encountering
+/// zero bytes instead of writing them
+pub async fn sparse_copy_async<R, W>(
+    reader: &mut R,
+    writer: &mut W,
+) -> Result<u64, io::Error>
+where
+    R: AsyncReadExt + Unpin,
+    W: AsyncWriteExt + AsyncSeekExt + Unpin,
+{
+    let mut buf = crate::tools::byte_buffer::ByteBuffer::new();
+    let mut written = 0;
+    loop {
+        let len = match buf.read_from_async(reader).await {
+            Ok(0) => return Ok(written),
+            Ok(len) => len,
+            Err(ref e) if e.kind() == ErrorKind::Interrupted => continue,
+            Err(e) => return Err(e),
+        };
+
+        if buffer_is_zero(&buf[..]) {
+            writer.seek(SeekFrom::Current(len as i64)).await?;
+        } else {
+            writer.write_all(&buf[..]).await?;
+        }
+        buf.clear();
+        written += len as u64;
+    }
+}
-- 
2.20.1

[pbs-devel] [PATCH proxmox-backup 1/1] pxar/extract: if possible create files sparesly

[Dominik Csapak]

instead of filling them with zeroes

this fixes an issue where we could not restore a container with large
sparse files in the backup (e.g. a 10GiB sparse file in a container
with a 8GiB disk)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/pxar/extract.rs | 32 ++++++++++++++++++++++++++------
 1 file changed, 26 insertions(+), 6 deletions(-)

diff --git a/src/pxar/extract.rs b/src/pxar/extract.rs
index ed238a2c..dd084ead 100644
--- a/src/pxar/extract.rs
+++ b/src/pxar/extract.rs
@@ -18,7 +18,10 @@ use pxar::format::Device;
 use pxar::Metadata;
 
 use proxmox::c_result;
-use proxmox::tools::fs::{create_path, CreateOptions};
+use proxmox::tools::{
+    fs::{create_path, CreateOptions},
+    io::{sparse_copy, sparse_copy_async}
+};
 
 use crate::pxar::dir_stack::PxarDirStack;
 use crate::pxar::metadata;
@@ -392,6 +395,11 @@ impl Extractor {
             )
         };
 
+        let copy_sparse = match nix::unistd::ftruncate(file.as_raw_fd(), size as i64) {
+            Ok(_) => true,
+            Err(_) => false,
+        };
+
         metadata::apply_initial_flags(
             self.feature_flags,
             metadata,
@@ -399,8 +407,12 @@ impl Extractor {
             &mut self.on_error,
         )?;
 
-        let extracted = io::copy(&mut *contents, &mut file)
-            .map_err(|err| format_err!("failed to copy file contents: {}", err))?;
+        let extracted = if copy_sparse {
+            sparse_copy(&mut *contents, &mut file)
+        } else {
+            io::copy(&mut *contents, &mut file)
+        }.map_err(|err| format_err!("failed to copy file contents: {}", err))?;
+
         if size != extracted {
             bail!("extracted {} bytes of a file of {} bytes", extracted, size);
         }
@@ -434,6 +446,11 @@ impl Extractor {
             )
         });
 
+        let copy_sparse = match nix::unistd::ftruncate(file.as_raw_fd(), size as i64) {
+            Ok(_) => true,
+            Err(_) => false,
+        };
+
         metadata::apply_initial_flags(
             self.feature_flags,
             metadata,
@@ -441,9 +458,12 @@ impl Extractor {
             &mut self.on_error,
         )?;
 
-        let extracted = tokio::io::copy(&mut *contents, &mut file)
-            .await
-            .map_err(|err| format_err!("failed to copy file contents: {}", err))?;
+        let extracted = if copy_sparse {
+            sparse_copy_async(&mut *contents, &mut file).await
+        } else {
+            tokio::io::copy(&mut *contents, &mut file).await
+        }.map_err(|err| format_err!("failed to copy file contents: {}", err))?;
+
         if size != extracted {
             bail!("extracted {} bytes of a file of {} bytes", extracted, size);
         }
-- 
2.20.1