From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [PATCH v4 proxmox-backup 24/26] catalog: fetch offset and size for files and refs
Date: Thu, 9 Nov 2023 19:46:12 +0100 [thread overview]
Message-ID: <20231109184614.1611127-25-c.ebner@proxmox.com> (raw)
In-Reply-To: <20231109184614.1611127-1-c.ebner@proxmox.com>
Allows to fetch the pxar archive offsets and file size for regular files
and appendix referenced files.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
Changes since version 2:
- no present in version 2
pbs-datastore/src/catalog.rs | 70 ++++++++++++++++++++++++++++++++++++
1 file changed, 70 insertions(+)
diff --git a/pbs-datastore/src/catalog.rs b/pbs-datastore/src/catalog.rs
index 220313c6..fe076a94 100644
--- a/pbs-datastore/src/catalog.rs
+++ b/pbs-datastore/src/catalog.rs
@@ -1,3 +1,4 @@
+use std::collections::BTreeMap;
use std::ffi::{CStr, CString, OsStr};
use std::fmt;
use std::io::{Read, Seek, SeekFrom, Write};
@@ -1118,6 +1119,75 @@ impl<R: Read + Seek> CatalogReader<R> {
Ok(res)
}
+
+ /// Get all File and AppendixRef entries with their pxar archive offset and size
+ pub fn fetch_offsets(&mut self) -> Result<BTreeMap<u64, u64>, Error> {
+ let root = self.root()?;
+ let mut list = BTreeMap::new();
+ match root {
+ DirEntry {
+ attr: DirEntryAttribute::Directory { start },
+ ..
+ } => self.fetch_offsets_from_dir(std::path::Path::new("./"), start, &mut list, None)?,
+ _ => bail!("unexpected root entry type, not a directory!"),
+ }
+ Ok(list)
+ }
+
+ fn fetch_offsets_from_dir(
+ &mut self,
+ prefix: &std::path::Path,
+ start: u64,
+ list: &mut BTreeMap<u64, u64>,
+ appendix_start: Option<AppendixStartOffset>,
+ ) -> Result<(), Error> {
+ let data = self.read_raw_dirinfo_block(start)?;
+
+ DirInfo::parse(
+ &data,
+ self.magic,
+ |etype, name_bytes, offset, size, _mtime, _ctime, link_offset| {
+ let mut path = std::path::PathBuf::from(prefix);
+ let name: &OsStr = OsStrExt::from_bytes(name_bytes);
+ path.push(name);
+
+ match etype {
+ CatalogEntryType::Archive => {
+ if offset > start {
+ bail!("got wrong archive offset ({} > {})", offset, start);
+ }
+ let pos = start - offset;
+ let appendix_start = self.appendix_offset(name_bytes)?;
+ self.fetch_offsets_from_dir(&path, pos, list, appendix_start)?;
+ }
+ CatalogEntryType::Directory => {
+ if offset > start {
+ bail!("got wrong directory offset ({} > {})", offset, start);
+ }
+ let pos = start - offset;
+ self.fetch_offsets_from_dir(&path, pos, list, appendix_start)?;
+ }
+ CatalogEntryType::AppendixRef => {
+ if let Some(Offset::AppendixRefOffset { offset }) = link_offset {
+ if let Some(appendix_start) = appendix_start {
+ list.insert(appendix_start.raw() + offset, size);
+ } else {
+ bail!("missing required appendix start offset");
+ }
+ }
+ }
+ CatalogEntryType::File => {
+ if let Some(Offset::FileOffset { offset }) = link_offset {
+ list.insert(offset, size);
+ }
+ }
+ _ => {}
+ }
+ Ok(true)
+ },
+ )?;
+ Ok(())
+ }
}
/// Serialize i64 as short, variable length byte sequence
--
2.39.2
next prev parent reply other threads:[~2023-11-09 18:47 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-09 18:45 [pbs-devel] [PATCH-SERIES v4 pxar proxmox-backup proxmox-widget-toolkit 00/26] fix #3174: improve file-level backup Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 1/26] fix #3174: decoder: factor out skip_bytes from skip_entry Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 2/26] fix #3174: decoder: impl skip_bytes for sync dec Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 3/26] fix #3174: encoder: calc filename + metadata byte size Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 4/26] fix #3174: enc/dec: impl PXAR_APPENDIX_REF entrytype Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 5/26] fix #3174: enc/dec: impl PXAR_APPENDIX entrytype Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 6/26] fix #3174: encoder: helper to add to encoder position Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 pxar 7/26] fix #3174: enc/dec: impl PXAR_APPENDIX_TAIL entrytype Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 proxmox-backup 08/26] fix #3174: index: add fn index list from start/end-offsets Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 proxmox-backup 09/26] fix #3174: index: add fn digest for DynamicEntry Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 proxmox-backup 10/26] fix #3174: api: double catalog upload size Christian Ebner
2023-11-09 18:45 ` [pbs-devel] [PATCH v4 proxmox-backup 11/26] fix #3174: catalog: introduce extended format v2 Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 12/26] fix #3174: archiver/extractor: impl appendix ref Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 13/26] fix #3174: catalog: add specialized Archive entry Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 14/26] fix #3174: extractor: impl seq restore from appendix Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 15/26] fix #3174: archiver: store ref to previous backup Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 16/26] fix #3174: upload stream: impl reused chunk injector Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 17/26] fix #3174: chunker: add forced boundaries Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 18/26] fix #3174: backup writer: inject queued chunk in upload steam Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 19/26] fix #3174: archiver: reuse files with unchanged metadata Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 20/26] fix #3174: specs: add backup detection mode specification Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 21/26] fix #3174: client: Add detection mode to backup creation Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 22/26] test-suite: add detection mode change benchmark Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 23/26] test-suite: Add bin to deb, add shell completions Christian Ebner
2023-11-09 18:46 ` Christian Ebner [this message]
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-backup 25/26] pxar: add heuristic to reduce reused chunk fragmentation Christian Ebner
2023-11-09 18:46 ` [pbs-devel] [PATCH v4 proxmox-widget-toolkit 26/26] file-browser: support pxar archive and fileref types Christian Ebner
2023-11-13 14:23 ` [pbs-devel] [PATCH-SERIES v4 pxar proxmox-backup proxmox-widget-toolkit 00/26] fix #3174: improve file-level backup Fabian Grünbichler
2023-11-13 15:14 ` Christian Ebner
2023-11-13 15:21 ` Christian Ebner
2023-11-13 15:35 ` Fabian Grünbichler
2023-11-13 15:45 ` Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231109184614.1611127-25-c.ebner@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox