From: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [pve-devel] [PATCH follow-up proxmox-offline-mirror 8/7] mirror: handle indices which are only available compressed
Date: Thu, 15 Sep 2022 16:36:15 +0200 [thread overview]
Message-ID: <20220915143615.1673687-1-f.gruenbichler@proxmox.com> (raw)
In-Reply-To: <20220915130918.727902-1-f.gruenbichler@proxmox.com>
there are repositories out there that not only skip serving the
uncompressed version, but not even reference it in their Release
file(s).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
best viewed with -w, the signal-desktop repository is one such example.
src/mirror.rs | 56 ++++++++++++++++++++++++++++-----------------------
1 file changed, 31 insertions(+), 25 deletions(-)
diff --git a/src/mirror.rs b/src/mirror.rs
index 3370ca4..f5ee48e 100644
--- a/src/mirror.rs
+++ b/src/mirror.rs
@@ -232,25 +232,30 @@ fn fetch_index_file(
config: &ParsedMirrorConfig,
prefix: &Path,
reference: &FileReference,
- uncompressed: &FileReference,
+ uncompressed: Option<&FileReference>,
by_hash: bool,
) -> Result<FetchResult, Error> {
let url = get_dist_url(&config.repository, &reference.path);
let path = get_dist_path(&config.repository, prefix, &reference.path);
- let uncompressed_path = get_dist_path(&config.repository, prefix, &uncompressed.path);
-
- if config.pool.contains(&reference.checksums) && config.pool.contains(&uncompressed.checksums) {
- let data = config
- .pool
- .get_contents(&uncompressed.checksums, config.verify)?;
-
- // Ensure they're linked at current path
- config.pool.lock()?.link_file(&reference.checksums, &path)?;
- config
- .pool
- .lock()?
- .link_file(&uncompressed.checksums, &uncompressed_path)?;
- return Ok(FetchResult { data, fetched: 0 });
+
+ if let Some(uncompressed) = uncompressed {
+ let uncompressed_path = get_dist_path(&config.repository, prefix, &uncompressed.path);
+
+ if config.pool.contains(&reference.checksums)
+ && config.pool.contains(&uncompressed.checksums)
+ {
+ let data = config
+ .pool
+ .get_contents(&uncompressed.checksums, config.verify)?;
+
+ // Ensure they're linked at current path
+ config.pool.lock()?.link_file(&reference.checksums, &path)?;
+ config
+ .pool
+ .lock()?
+ .link_file(&uncompressed.checksums, &uncompressed_path)?;
+ return Ok(FetchResult { data, fetched: 0 });
+ }
}
let urls = if by_hash {
@@ -307,12 +312,15 @@ fn fetch_index_file(
};
let locked = &config.pool.lock()?;
- if !locked.contains(&uncompressed.checksums) {
- locked.add_file(decompressed, &uncompressed.checksums, config.sync)?;
- }
+ if let Some(uncompressed) = uncompressed {
+ if !locked.contains(&uncompressed.checksums) {
+ locked.add_file(decompressed, &uncompressed.checksums, config.sync)?;
+ }
- // Ensure it's linked at current path
- locked.link_file(&uncompressed.checksums, &uncompressed_path)?;
+ // Ensure it's linked at current path
+ let uncompressed_path = get_dist_path(&config.repository, prefix, &uncompressed.path);
+ locked.link_file(&uncompressed.checksums, &uncompressed_path)?;
+ }
Ok(FetchResult {
data: decompressed.to_owned(),
@@ -566,15 +574,13 @@ pub fn create_snapshot(
for basename in references {
println!("\tFetching '{basename}'..");
let files = release.files.get(basename).unwrap();
- let uncompressed_ref = files
- .iter()
- .find(|reference| reference.path == *basename)
- .ok_or_else(|| format_err!("Found derived reference without base reference."))?;
+ let uncompressed_ref = files.iter().find(|reference| reference.path == *basename);
+
let mut package_index_data = None;
for reference in files {
// if both compressed and uncompressed are referenced, the uncompressed file may not exist on the server
- if reference == uncompressed_ref && files.len() > 1 {
+ if Some(reference) == uncompressed_ref && files.len() > 1 {
continue;
}
--
2.30.2
next prev parent reply other threads:[~2022-09-15 14:36 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-15 13:09 [pve-devel] [PATCH proxmox-apt/proxmox-offline-mirror 0/7] misc improvements Fabian Grünbichler
2022-09-15 13:09 ` [pve-devel] [PATCH proxmox-offline-mirror 1/4] mirror: use xz multi decoder Fabian Grünbichler
2022-09-15 13:09 ` [pve-devel] [PATCH proxmox-apt 1/3] release: add Commands file reference type Fabian Grünbichler
2022-09-15 13:09 ` [pve-devel] [PATCH proxmox-offline-mirror 2/4] mirror: skip failed, non Packages references Fabian Grünbichler
2022-09-15 13:09 ` [pve-devel] [PATCH proxmox-apt 2/3] release: add 'architecture' helper Fabian Grünbichler
2022-09-15 13:09 ` [pve-devel] [PATCH proxmox-offline-mirror 3/4] mirror: support acquiring indices by hash Fabian Grünbichler
2022-09-15 14:34 ` [pve-devel] [PATCH FIXUP proxmox-offline-mirror] clippy fix Fabian Grünbichler
2022-09-15 13:09 ` [pve-devel] [PATCH proxmox-apt 3/3] release: fix typo in 'Acquire-By-Hash' Fabian Grünbichler
2022-09-15 13:09 ` [pve-devel] [PATCH proxmox-offline-mirror 4/4] mirror: use new architecture helper Fabian Grünbichler
2022-09-15 14:36 ` Fabian Grünbichler [this message]
2022-09-16 12:31 ` [pve-devel] applied-series: [PATCH proxmox-apt/proxmox-offline-mirror 0/7] misc improvements Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220915143615.1673687-1-f.gruenbichler@proxmox.com \
--to=f.gruenbichler@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox