* [pbs-devel] [PATCH proxmox/proxmox-backup v3] restore files from pxar sparsely
@ 2021-02-09 9:36 Dominik Csapak
2021-02-09 9:36 ` [pbs-devel] [PATCH proxmox v3 1/1] proxmox: add sparse_copy(_async) to tools::io Dominik Csapak
2021-02-09 9:37 ` [pbs-devel] [PATCH proxmox-backup v3 1/1] pxar/extract: if possible create files sparesly Dominik Csapak
0 siblings, 2 replies; 3+ messages in thread
From: Dominik Csapak @ 2021-02-09 9:36 UTC (permalink / raw)
To: pbs-devel
to be able to restore containers with big sparse files
ideally(?) we would save hole information directly in the pxar archive
and not even use zero chunks, so that we can have smaller
pxar archives, and accurately restore sparse files like they were before
for now, restore all files sparsely
i am not sure if v2 or this is better, since before we could
restore if truncate did not work, now we fail
but we have probably less io.
changes from v2:
* always sparse copy and truncate after
changes from RFC:
* drop the zero module of proxmox, rust can generate fast code by itself
proxmox:
Dominik Csapak (1):
proxmox: add sparse_copy(_async) to tools::io
proxmox/src/tools/io/mod.rs | 70 +++++++++++++++++++++++++++++++++++++
1 file changed, 70 insertions(+)
proxmox-backup:
Dominik Csapak (1):
pxar/extract: if possible create files sparesly
src/pxar/extract.rs | 24 +++++++++++++++++++++---
1 file changed, 21 insertions(+), 3 deletions(-)
--
2.20.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* [pbs-devel] [PATCH proxmox v3 1/1] proxmox: add sparse_copy(_async) to tools::io
2021-02-09 9:36 [pbs-devel] [PATCH proxmox/proxmox-backup v3] restore files from pxar sparsely Dominik Csapak
@ 2021-02-09 9:36 ` Dominik Csapak
2021-02-09 9:37 ` [pbs-devel] [PATCH proxmox-backup v3 1/1] pxar/extract: if possible create files sparesly Dominik Csapak
1 sibling, 0 replies; 3+ messages in thread
From: Dominik Csapak @ 2021-02-09 9:36 UTC (permalink / raw)
To: pbs-devel
this is able to seek the target instead of writing zeroes, which
generates sparse files where supported
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
proxmox/src/tools/io/mod.rs | 70 +++++++++++++++++++++++++++++++++++++
1 file changed, 70 insertions(+)
diff --git a/proxmox/src/tools/io/mod.rs b/proxmox/src/tools/io/mod.rs
index 2e92ebb..299d19c 100644
--- a/proxmox/src/tools/io/mod.rs
+++ b/proxmox/src/tools/io/mod.rs
@@ -3,8 +3,78 @@
//! The [`ReadExt`] trait provides additional operations for handling byte buffers for types
//! implementing [`Read`](std::io::Read).
+use std::io::{self, Read, Write, Seek, SeekFrom, ErrorKind};
+
mod read;
pub use read::*;
mod write;
pub use write::*;
+
+fn buffer_is_zero(buf: &[u8]) -> bool {
+ !buf
+ .chunks(128)
+ .map(|aa|
+ aa.iter().fold(0, |a, b| a|b) != 0
+ ).any(|a| a)
+}
+
+/// copy similar to io::copy, but seeks the target when encountering
+/// zero bytes instead of writing them
+pub fn sparse_copy<R: Read + ?Sized, W: Write + Seek + ?Sized>(
+ reader: &mut R,
+ writer: &mut W,
+) -> Result<u64, io::Error> {
+ let mut buf = crate::tools::byte_buffer::ByteBuffer::new();
+ let mut written = 0;
+ loop {
+ let len = match buf.read_from(reader) {
+ Ok(0) => return Ok(written),
+ Ok(len) => len,
+ Err(ref e) if e.kind() == ErrorKind::Interrupted => continue,
+ Err(e) => return Err(e),
+ };
+
+ if buffer_is_zero(&buf[..]) {
+ writer.seek(SeekFrom::Current(len as i64))?;
+ } else {
+ writer.write_all(&buf[..])?;
+ }
+ buf.clear();
+ written += len as u64;
+ }
+}
+
+#[cfg(feature = "tokio")]
+use tokio::io::{AsyncReadExt, AsyncWriteExt, AsyncSeekExt};
+
+#[cfg(feature = "tokio")]
+/// copy similar to tokio::io::copy, but seeks the target when encountering
+/// zero bytes instead of writing them
+pub async fn sparse_copy_async<R, W>(
+ reader: &mut R,
+ writer: &mut W,
+) -> Result<u64, io::Error>
+where
+ R: AsyncReadExt + Unpin,
+ W: AsyncWriteExt + AsyncSeekExt + Unpin,
+{
+ let mut buf = crate::tools::byte_buffer::ByteBuffer::new();
+ let mut written = 0;
+ loop {
+ let len = match buf.read_from_async(reader).await {
+ Ok(0) => return Ok(written),
+ Ok(len) => len,
+ Err(ref e) if e.kind() == ErrorKind::Interrupted => continue,
+ Err(e) => return Err(e),
+ };
+
+ if buffer_is_zero(&buf[..]) {
+ writer.seek(SeekFrom::Current(len as i64)).await?;
+ } else {
+ writer.write_all(&buf[..]).await?;
+ }
+ buf.clear();
+ written += len as u64;
+ }
+}
--
2.20.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v3 1/1] pxar/extract: if possible create files sparesly
2021-02-09 9:36 [pbs-devel] [PATCH proxmox/proxmox-backup v3] restore files from pxar sparsely Dominik Csapak
2021-02-09 9:36 ` [pbs-devel] [PATCH proxmox v3 1/1] proxmox: add sparse_copy(_async) to tools::io Dominik Csapak
@ 2021-02-09 9:37 ` Dominik Csapak
1 sibling, 0 replies; 3+ messages in thread
From: Dominik Csapak @ 2021-02-09 9:37 UTC (permalink / raw)
To: pbs-devel
instead of filling them with zeroes
this fixes an issue where we could not restore a container with large
sparse files in the backup (e.g. a 10GiB sparse file in a container
with a 8GiB disk)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
| 24 +++++++++++++++++++++---
1 file changed, 21 insertions(+), 3 deletions(-)
--git a/src/pxar/extract.rs b/src/pxar/extract.rs
index 9cf3f928..7ae7c862 100644
--- a/src/pxar/extract.rs
+++ b/src/pxar/extract.rs
@@ -18,7 +18,10 @@ use pxar::format::Device;
use pxar::Metadata;
use proxmox::c_result;
-use proxmox::tools::fs::{create_path, CreateOptions};
+use proxmox::tools::{
+ fs::{create_path, CreateOptions},
+ io::{sparse_copy, sparse_copy_async}
+};
use crate::pxar::dir_stack::PxarDirStack;
use crate::pxar::metadata;
@@ -398,6 +401,7 @@ impl Extractor {
)
};
+
metadata::apply_initial_flags(
self.feature_flags,
metadata,
@@ -405,12 +409,19 @@ impl Extractor {
&mut self.on_error,
)?;
- let extracted = io::copy(&mut *contents, &mut file)
+ let extracted = sparse_copy(&mut *contents, &mut file)
.map_err(|err| format_err!("failed to copy file contents: {}", err))?;
+
if size != extracted {
bail!("extracted {} bytes of a file of {} bytes", extracted, size);
}
+ while match nix::unistd::ftruncate(file.as_raw_fd(), size as i64) {
+ Ok(_) => false,
+ Err(nix::Error::Sys(errno)) if errno == nix::errno::Errno::EINTR => true,
+ Err(err) => bail!("error setting file size: {}", err),
+ } {};
+
metadata::apply(
self.feature_flags,
metadata,
@@ -447,13 +458,20 @@ impl Extractor {
&mut self.on_error,
)?;
- let extracted = tokio::io::copy(&mut *contents, &mut file)
+ let extracted = sparse_copy_async(&mut *contents, &mut file)
.await
.map_err(|err| format_err!("failed to copy file contents: {}", err))?;
+
if size != extracted {
bail!("extracted {} bytes of a file of {} bytes", extracted, size);
}
+ while match nix::unistd::ftruncate(file.as_raw_fd(), size as i64) {
+ Ok(_) => false,
+ Err(nix::Error::Sys(errno)) if errno == nix::errno::Errno::EINTR => true,
+ Err(err) => bail!("error setting file size: {}", err),
+ } {};
+
metadata::apply(
self.feature_flags,
metadata,
--
2.20.1
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-02-09 9:37 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-09 9:36 [pbs-devel] [PATCH proxmox/proxmox-backup v3] restore files from pxar sparsely Dominik Csapak
2021-02-09 9:36 ` [pbs-devel] [PATCH proxmox v3 1/1] proxmox: add sparse_copy(_async) to tools::io Dominik Csapak
2021-02-09 9:37 ` [pbs-devel] [PATCH proxmox-backup v3 1/1] pxar/extract: if possible create files sparesly Dominik Csapak
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox