From: Dominik Csapak <d.csapak@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [PATCH proxmox-backup 2/2] tape/helpers/snapshot_reader: sort chunks by inode (per index)
Date: Fri, 18 Jun 2021 11:29:10 +0200 [thread overview]
Message-ID: <20210618092910.4518-2-d.csapak@proxmox.com> (raw)
In-Reply-To: <20210618092910.4518-1-d.csapak@proxmox.com>
sort the chunks we want to backup to tape by inode, to gain some
speed on spinning disks. this is done per index, not globally.
costs a bit memory, but not too much, about 16 bytes per chunk which
would mean ~4MiB for a 1TiB index with 4MiB chunks.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
---
this resulted in a speedup in my setup of between 20 and 30%
(single spinner with random snapshots, from 17-26MiB/s to 30-40MiB/s)
we already do this for verification, but got no real feedback on it,
so either it does not make that much of a difference in the real world,
or it is not that visible on verification (since that varies very much
anyway)
src/tape/helpers/snapshot_reader.rs | 20 +++++++++++++-------
1 file changed, 13 insertions(+), 7 deletions(-)
diff --git a/src/tape/helpers/snapshot_reader.rs b/src/tape/helpers/snapshot_reader.rs
index 7b272e37..416c88c1 100644
--- a/src/tape/helpers/snapshot_reader.rs
+++ b/src/tape/helpers/snapshot_reader.rs
@@ -107,7 +107,7 @@ impl SnapshotReader {
pub struct SnapshotChunkIterator<'a> {
snapshot_reader: &'a SnapshotReader,
todo_list: Vec<String>,
- current_index: Option<(Arc<Box<dyn IndexFile>>, usize)>,
+ current_index: Option<(Arc<Box<dyn IndexFile + Send>>, usize, Vec<(usize, u64)>)>,
}
impl <'a> Iterator for SnapshotChunkIterator<'a> {
@@ -119,20 +119,26 @@ impl <'a> Iterator for SnapshotChunkIterator<'a> {
if self.current_index.is_none() {
if let Some(filename) = self.todo_list.pop() {
let file = self.snapshot_reader.open_file(&filename)?;
- let index: Box<dyn IndexFile> = match archive_type(&filename)? {
+ let index: Box<dyn IndexFile + Send> = match archive_type(&filename)? {
ArchiveType::FixedIndex => Box::new(FixedIndexReader::new(file)?),
ArchiveType::DynamicIndex => Box::new(DynamicIndexReader::new(file)?),
_ => bail!("SnapshotChunkIterator: got unknown file type - internal error"),
};
- self.current_index = Some((Arc::new(index), 0));
+
+ let datastore =
+ DataStore::lookup_datastore(self.snapshot_reader.datastore_name())?;
+ let order = datastore.get_chunks_in_order(&index, |_| false, |_| Ok(()))?;
+
+ self.current_index = Some((Arc::new(index), 0, order));
} else {
return Ok(None);
}
}
- let (index, pos) = self.current_index.take().unwrap();
- if pos < index.index_count() {
- let digest = *index.index_digest(pos).unwrap();
- self.current_index = Some((index, pos + 1));
+ let (index, pos, list) = self.current_index.take().unwrap();
+ if pos < list.len() {
+ let (real_pos, _) = list[pos];
+ let digest = *index.index_digest(real_pos).unwrap();
+ self.current_index = Some((index, pos + 1, list));
return Ok(Some(digest));
} else {
// pop next index
--
2.20.1
next prev parent reply other threads:[~2021-06-18 9:29 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-18 9:29 [pbs-devel] [PATCH proxmox-backup 1/2] backup/datastore: refactor chunk inode sorting to the datastore Dominik Csapak
2021-06-18 9:29 ` Dominik Csapak [this message]
2021-06-18 15:02 ` [pbs-devel] [PATCH proxmox-backup 2/2] tape/helpers/snapshot_reader: sort chunks by inode (per index) Thomas Lamprecht
2021-07-12 4:23 ` [pbs-devel] applied-series: [PATCH proxmox-backup 1/2] backup/datastore: refactor chunk inode sorting to the datastore Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210618092910.4518-2-d.csapak@proxmox.com \
--to=d.csapak@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox