[pbs-devel] [PATCH proxmox/proxmox-backup v3] restore files from pxar sparsely

[pbs-devel] [PATCH proxmox/proxmox-backup v3] restore files from pxar sparsely

[Dominik Csapak]

to be able to restore containers with big sparse files

ideally(?) we would save hole information directly in the pxar archive
and not even use zero chunks, so that we can have smaller
pxar archives, and accurately restore sparse files like they were before

for now, restore all files sparsely

i am not sure if v2 or this is better, since before we could
restore if truncate did not work, now we fail
but we have probably less io.

changes from v2:
* always sparse copy and truncate after

changes from RFC:
* drop the zero module of proxmox, rust can generate fast code by itself

proxmox:

Dominik Csapak (1):
  proxmox: add sparse_copy(_async) to tools::io

 proxmox/src/tools/io/mod.rs | 70 +++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)

proxmox-backup:

Dominik Csapak (1):
  pxar/extract: if possible create files sparesly

 src/pxar/extract.rs | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

-- 
2.20.1

[pbs-devel] [PATCH proxmox v3 1/1] proxmox: add sparse_copy(_async) to tools::io

[Dominik Csapak]

this is able to seek the target instead of writing zeroes, which
generates sparse files where supported

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 proxmox/src/tools/io/mod.rs | 70 +++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)

diff --git a/proxmox/src/tools/io/mod.rs b/proxmox/src/tools/io/mod.rs
index 2e92ebb..299d19c 100644
--- a/proxmox/src/tools/io/mod.rs
+++ b/proxmox/src/tools/io/mod.rs
@@ -3,8 +3,78 @@
 //! The [`ReadExt`] trait provides additional operations for handling byte buffers for types
 //! implementing [`Read`](std::io::Read).
 
+use std::io::{self, Read, Write, Seek, SeekFrom, ErrorKind};
+
 mod read;
 pub use read::*;
 
 mod write;
 pub use write::*;
+
+fn buffer_is_zero(buf: &[u8]) -> bool {
+    !buf
+        .chunks(128)
+        .map(|aa|
+            aa.iter().fold(0, |a, b| a|b) != 0
+        ).any(|a| a)
+}
+
+/// copy similar to io::copy, but seeks the target when encountering
+/// zero bytes instead of writing them
+pub fn sparse_copy<R: Read + ?Sized, W: Write + Seek + ?Sized>(
+    reader: &mut R,
+    writer: &mut W,
+) -> Result<u64, io::Error> {
+    let mut buf = crate::tools::byte_buffer::ByteBuffer::new();
+    let mut written = 0;
+    loop {
+        let len = match buf.read_from(reader) {
+            Ok(0) => return Ok(written),
+            Ok(len) => len,
+            Err(ref e) if e.kind() == ErrorKind::Interrupted => continue,
+            Err(e) => return Err(e),
+        };
+
+        if buffer_is_zero(&buf[..]) {
+            writer.seek(SeekFrom::Current(len as i64))?;
+        } else {
+            writer.write_all(&buf[..])?;
+        }
+        buf.clear();
+        written += len as u64;
+    }
+}
+
+#[cfg(feature = "tokio")]
+use tokio::io::{AsyncReadExt, AsyncWriteExt, AsyncSeekExt};
+
+#[cfg(feature = "tokio")]
+/// copy similar to tokio::io::copy, but seeks the target when encountering
+/// zero bytes instead of writing them
+pub async fn sparse_copy_async<R, W>(
+    reader: &mut R,
+    writer: &mut W,
+) -> Result<u64, io::Error>
+where
+    R: AsyncReadExt + Unpin,
+    W: AsyncWriteExt + AsyncSeekExt + Unpin,
+{
+    let mut buf = crate::tools::byte_buffer::ByteBuffer::new();
+    let mut written = 0;
+    loop {
+        let len = match buf.read_from_async(reader).await {
+            Ok(0) => return Ok(written),
+            Ok(len) => len,
+            Err(ref e) if e.kind() == ErrorKind::Interrupted => continue,
+            Err(e) => return Err(e),
+        };
+
+        if buffer_is_zero(&buf[..]) {
+            writer.seek(SeekFrom::Current(len as i64)).await?;
+        } else {
+            writer.write_all(&buf[..]).await?;
+        }
+        buf.clear();
+        written += len as u64;
+    }
+}
-- 
2.20.1

[pbs-devel] [PATCH proxmox-backup v3 1/1] pxar/extract: if possible create files sparesly

[Dominik Csapak]

instead of filling them with zeroes

this fixes an issue where we could not restore a container with large
sparse files in the backup (e.g. a 10GiB sparse file in a container
with a 8GiB disk)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
 src/pxar/extract.rs | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)

diff --git a/src/pxar/extract.rs b/src/pxar/extract.rs
index 9cf3f928..7ae7c862 100644
--- a/src/pxar/extract.rs
+++ b/src/pxar/extract.rs
@@ -18,7 +18,10 @@ use pxar::format::Device;
 use pxar::Metadata;
 
 use proxmox::c_result;
-use proxmox::tools::fs::{create_path, CreateOptions};
+use proxmox::tools::{
+    fs::{create_path, CreateOptions},
+    io::{sparse_copy, sparse_copy_async}
+};
 
 use crate::pxar::dir_stack::PxarDirStack;
 use crate::pxar::metadata;
@@ -398,6 +401,7 @@ impl Extractor {
             )
         };
 
+
         metadata::apply_initial_flags(
             self.feature_flags,
             metadata,
@@ -405,12 +409,19 @@ impl Extractor {
             &mut self.on_error,
         )?;
 
-        let extracted = io::copy(&mut *contents, &mut file)
+        let extracted = sparse_copy(&mut *contents, &mut file)
             .map_err(|err| format_err!("failed to copy file contents: {}", err))?;
+
         if size != extracted {
             bail!("extracted {} bytes of a file of {} bytes", extracted, size);
         }
 
+        while match nix::unistd::ftruncate(file.as_raw_fd(), size as i64) {
+            Ok(_) => false,
+            Err(nix::Error::Sys(errno)) if errno == nix::errno::Errno::EINTR => true,
+            Err(err) => bail!("error setting file size: {}", err),
+        } {};
+
         metadata::apply(
             self.feature_flags,
             metadata,
@@ -447,13 +458,20 @@ impl Extractor {
             &mut self.on_error,
         )?;
 
-        let extracted = tokio::io::copy(&mut *contents, &mut file)
+        let extracted = sparse_copy_async(&mut *contents, &mut file)
             .await
             .map_err(|err| format_err!("failed to copy file contents: {}", err))?;
+
         if size != extracted {
             bail!("extracted {} bytes of a file of {} bytes", extracted, size);
         }
 
+        while match nix::unistd::ftruncate(file.as_raw_fd(), size as i64) {
+            Ok(_) => false,
+            Err(nix::Error::Sys(errno)) if errno == nix::errno::Errno::EINTR => true,
+            Err(err) => bail!("error setting file size: {}", err),
+        } {};
+
         metadata::apply(
             self.feature_flags,
             metadata,
-- 
2.20.1