From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [RFC v2 proxmox-backup 08/23] fix #3174: index: add fn index list from start/end-offsets
Date: Mon, 9 Oct 2023 13:51:24 +0200 [thread overview]
Message-ID: <20231009115139.1417886-9-c.ebner@proxmox.com> (raw)
In-Reply-To: <20231009115139.1417886-1-c.ebner@proxmox.com>
Adds a function to get a list of DynamicEntry's from a chunk index by
given start and end offset, which should be contained within these
chunks.
This is needed in order to reference file payloads and reuse the
chunks containing them from a previous backup run.
The index entries are normalized, meaning each entries end() is equal to
the chunk size.
In addition to the list of index entries, the padding to the start of
the requested start offset from the first chunk is returned. This is
needed for calculation of the appendix section offsets during
decoding.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
Changes since version 1:
- remove unneeded total size calculation, not needed
pbs-datastore/src/dynamic_index.rs | 32 ++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/pbs-datastore/src/dynamic_index.rs b/pbs-datastore/src/dynamic_index.rs
index 71a5082e..11b8381d 100644
--- a/pbs-datastore/src/dynamic_index.rs
+++ b/pbs-datastore/src/dynamic_index.rs
@@ -188,6 +188,38 @@ impl DynamicIndexReader {
self.binary_search(middle_idx + 1, middle_end, end_idx, end, offset)
}
}
+
+ /// List of chunk indices containing the data from start_offset to end_offset
+ pub fn indices(
+ &self,
+ start_offset: u64,
+ end_offset: u64,
+ ) -> Result<(Vec<DynamicEntry>, u64), Error> {
+ let end_idx = self.index.len() - 1;
+ let chunk_end = self.chunk_end(end_idx);
+ let start = self.binary_search(0, 0, end_idx, chunk_end, start_offset)?;
+ let end = self.binary_search(0, 0, end_idx, chunk_end, end_offset)?;
+
+ let offset_first = if start == 0 {
+ 0
+ } else {
+ self.index[start - 1].end()
+ };
+
+ let padding_start = start_offset - offset_first;
+
+ let mut indices = Vec::new();
+ let mut prev_end = offset_first;
+ for dynamic_entry in &self.index[start..end + 1] {
+ let mut dynamic_entry = dynamic_entry.clone();
+ // Normalize end to be equal to size
+ dynamic_entry.end_le -= prev_end;
+ prev_end += dynamic_entry.end_le;
+ indices.push(dynamic_entry);
+ }
+
+ Ok((indices, padding_start))
+ }
}
impl IndexFile for DynamicIndexReader {
--
2.39.2
next prev parent reply other threads:[~2023-10-09 11:52 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-09 11:51 [pbs-devel] [RFC v2 pxar proxmox-backup 00/23] fix #3174: improve file-level backup Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 pxar 1/23] fix #3174: decoder: factor out skip_bytes from skip_entry Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 pxar 2/23] fix #3174: decoder: impl skip_bytes for sync dec Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 pxar 3/23] fix #3174: encoder: calc filename + metadata byte size Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 pxar 4/23] fix #3174: enc/dec: impl PXAR_APPENDIX_REF entrytype Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 pxar 5/23] fix #3174: enc/dec: impl PXAR_APPENDIX entrytype Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 pxar 6/23] fix #3174: encoder: helper to add to encoder position Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 pxar 7/23] fix #3174: enc/dec: impl PXAR_APPENDIX_TAIL entrytype Christian Ebner
2023-10-09 11:51 ` Christian Ebner [this message]
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 09/23] fix #3174: index: add fn digest for DynamicEntry Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 10/23] fix #3174: api: double catalog upload size Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 11/23] fix #3174: catalog: introduce extended format v2 Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 12/23] fix #3174: archiver/extractor: impl appendix ref Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 13/23] fix #3174: catalog: add specialized Archive entry Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 14/23] fix #3174: extractor: impl seq restore from appendix Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 15/23] fix #3174: archiver: store ref to previous backup Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 16/23] fix #3174: upload stream: impl reused chunk injector Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 17/23] fix #3174: chunker: add forced boundaries Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 18/23] fix #3174: backup writer: inject queued chunk in upload steam Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 19/23] fix #3174: archiver: reuse files with unchanged metadata Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 20/23] fix #3174: schema: add backup detection mode schema Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 21/23] fix #3174: client: Add detection mode to backup creation Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 22/23] test-suite: add detection mode change benchmark Christian Ebner
2023-10-09 11:51 ` [pbs-devel] [RFC v2 proxmox-backup 23/23] test-suite: Add bin to deb, add shell completions Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231009115139.1417886-9-c.ebner@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox