[pve-devel] [PATCH proxmox-apt/proxmox-offline-mirror 0/7] misc improvements

[pve-devel] [PATCH proxmox-apt/proxmox-offline-mirror 0/7] misc improvements

[Fabian Grünbichler]

this series adds some features and fixes some issues that pop up when
attempting to mirror Ubuntu repositories, and should also improve
resilience with other third-party repositories.

tested with Ubuntu Jammy (main, security and updates repositories).

[pve-devel] [PATCH proxmox-offline-mirror 1/4] mirror: use xz multi decoder

[Fabian Grünbichler]

Ubuntu's Packages.xz files require it, because they contain multiple
streams.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
 src/mirror.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/mirror.rs b/src/mirror.rs
index 78a493b..14b0c6a 100644
--- a/src/mirror.rs
+++ b/src/mirror.rs
@@ -277,7 +277,7 @@ fn fetch_index_file(
             &buf[..]
         }
         Some(CompressionType::Lzma) | Some(CompressionType::Xz) => {
-            let mut xz = xz2::read::XzDecoder::new(raw);
+            let mut xz = xz2::read::XzDecoder::new_multi_decoder(raw);
             xz.read_to_end(&mut buf)?;
             &buf[..]
         }
-- 
2.30.2

[pve-devel] [PATCH proxmox-apt 1/3] release: add Commands file reference type

[Fabian Grünbichler]

used by command-not-found to lookup which package ships which command.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---

Notes:
    this is technically a breaking change, but the only user of this already
    has a fallback match arm. I wonder whether we should mark this as
    non-exhaustive?

 src/deb822/release_file.rs | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)

diff --git a/src/deb822/release_file.rs b/src/deb822/release_file.rs
index 6668450..2b7245b 100644
--- a/src/deb822/release_file.rs
+++ b/src/deb822/release_file.rs
@@ -51,6 +51,8 @@ pub type Component = String;
 /// `Packages` and `Sources` will contain further reference to binary or source package files.
 ///  These are handled in `PackagesFile` and `SourcesFile` respectively.
 pub enum FileReferenceType {
+    /// A `Commands` index listing command to package mappings
+    Commands(Architecture, Option<CompressionType>),
     /// A `Contents` index listing contents of binary packages
     Contents(Architecture, Option<CompressionType>),
     /// A `Contents` index listing contents of binary udeb packages
@@ -123,6 +125,20 @@ impl FileReferenceType {
                         Ok(FileReferenceType::Unknown)
                     }
                 }
+                "cnf" => {
+                    if let Some(rest) = rest.strip_prefix("Commands-") {
+                        if let Some((arch, ext)) = rest.rsplit_once('.') {
+                            Ok(FileReferenceType::Commands(
+                                arch.to_owned(),
+                                FileReferenceType::match_compression(ext).ok().flatten(),
+                            ))
+                        } else {
+                            Ok(FileReferenceType::Commands(rest.to_owned(), None))
+                        }
+                    } else {
+                        Ok(FileReferenceType::Unknown)
+                    }
+                },
                 "dep11" => {
                     if let Some((_path, ext)) = rest.rsplit_once('.') {
                         Ok(FileReferenceType::Dep11(
@@ -198,7 +214,8 @@ impl FileReferenceType {
 
     pub fn compression(&self) -> Option<CompressionType> {
         match *self {
-            FileReferenceType::Contents(_, comp)
+            FileReferenceType::Commands(_, comp)
+            | FileReferenceType::Contents(_, comp)
             | FileReferenceType::ContentsUdeb(_, comp)
             | FileReferenceType::Packages(_, comp)
             | FileReferenceType::Sources(comp)
-- 
2.30.2

[pve-devel] [PATCH proxmox-offline-mirror 2/4] mirror: skip failed, non Packages references

[Fabian Grünbichler]

these contain extra data that is not that important for the main
repository use case - providing deb packages.

if they are not retrievable (e.g., Ubuntu *only* provides some of they
via by-hash, which proxmox-offline-mirror doesn't yet support) a warning
should be enough, instead of failing the whole snapshot creation.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
 src/mirror.rs | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/src/mirror.rs b/src/mirror.rs
index 14b0c6a..f910e6a 100644
--- a/src/mirror.rs
+++ b/src/mirror.rs
@@ -534,6 +534,7 @@ pub fn create_snapshot(
 
     let mut packages_size = 0_usize;
     let mut packages_indices = HashMap::new();
+    let mut failed_references = Vec::new();
     for (component, references) in per_component {
         println!("\nFetching indices for component '{component}'");
         let mut component_deb_size = 0;
@@ -555,7 +556,18 @@ pub fn create_snapshot(
                 }
 
                 // this will ensure the uncompressed file will be written locally
-                let res = fetch_index_file(&config, prefix, reference, uncompressed_ref)?;
+                let res = match fetch_index_file(&config, prefix, reference, uncompressed_ref) {
+                    Ok(res) => res,
+                    Err(err) if !reference.file_type.is_package_index() => {
+                        eprintln!(
+                            "Failed to fetch '{:?}' type reference '{}', skipping - {err}",
+                            reference.file_type, reference.path
+                        );
+                        failed_references.push(reference);
+                        continue;
+                    }
+                    Err(err) => bail!(err),
+                };
                 fetch_progress.update(&res);
 
                 if package_index_data.is_none() && reference.file_type.is_package_index() {
@@ -577,6 +589,12 @@ pub fn create_snapshot(
         total_progress += fetch_progress;
     }
     println!("Total deb size: {packages_size}");
+    if !failed_references.is_empty() {
+        eprintln!("Failed to download non-package-index references:");
+        for reference in failed_references {
+            eprintln!("\t{}", reference.path);
+        }
+    }
 
     println!("\nFetching packages..");
     for (basename, references) in packages_indices {
-- 
2.30.2

[pve-devel] [PATCH proxmox-apt 2/3] release: add 'architecture' helper

[Fabian Grünbichler]

which returns if a file reference is architecture specific, and for
which architecture it is relevant.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
 src/deb822/release_file.rs | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/src/deb822/release_file.rs b/src/deb822/release_file.rs
index 2b7245b..5888728 100644
--- a/src/deb822/release_file.rs
+++ b/src/deb822/release_file.rs
@@ -228,6 +228,22 @@ impl FileReferenceType {
         }
     }
 
+    pub fn architecture(&self) -> Option<&Architecture> {
+        match self {
+            FileReferenceType::Commands(arch, _)
+            | FileReferenceType::Contents(arch, _)
+            | FileReferenceType::ContentsUdeb(arch, _)
+            | FileReferenceType::Packages(arch, _) => Some(arch),
+            FileReferenceType::PseudoRelease(arch) => arch.as_ref(),
+            FileReferenceType::Unknown
+            | FileReferenceType::PDiff
+            | FileReferenceType::Sources(_)
+            | FileReferenceType::Dep11(_)
+            | FileReferenceType::Translation(_)
+            | FileReferenceType::Ignored => None,
+        }
+    }
+
     pub fn is_package_index(&self) -> bool {
         matches!(self, FileReferenceType::Packages(_, _))
     }
-- 
2.30.2

[pve-devel] [PATCH proxmox-offline-mirror 3/4] mirror: support acquiring indices by hash

[Fabian Grünbichler]

requires proxmox-apt > 0.9.1, since earlier versions misdetect by-hash
support in the release file.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
 src/mirror.rs | 47 ++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 38 insertions(+), 9 deletions(-)

diff --git a/src/mirror.rs b/src/mirror.rs
index f910e6a..6cbd680 100644
--- a/src/mirror.rs
+++ b/src/mirror.rs
@@ -233,6 +233,7 @@ fn fetch_index_file(
     prefix: &Path,
     reference: &FileReference,
     uncompressed: &FileReference,
+    by_hash: bool,
 ) -> Result<FetchResult, Error> {
     let url = get_dist_url(&config.repository, &reference.path);
     let path = get_dist_path(&config.repository, prefix, &reference.path);
@@ -252,14 +253,36 @@ fn fetch_index_file(
         return Ok(FetchResult { data, fetched: 0 });
     }
 
-    let res = fetch_plain_file(
-        config,
-        &url,
-        &path,
-        reference.size,
-        &reference.checksums,
-        true,
-    )?;
+    let urls = if by_hash {
+        let mut urls = Vec::new();
+        if let Some((base_url, _file_name)) = url.rsplit_once('/') {
+            if let Some(sha512) = reference.checksums.sha512 {
+                urls.push(format!("{base_url}/by-hash/SHA512/{}", hex::encode(sha512)));
+            }
+            if let Some(sha256) = reference.checksums.sha256 {
+                urls.push(format!("{base_url}/by-hash/SHA256/{}", hex::encode(sha256)));
+            }
+        }
+        urls.push(url);
+        urls
+    } else {
+        vec![url]
+    };
+
+    let res = urls
+        .iter()
+        .fold(None, |res, url| match res {
+            Some(Ok(res)) => Some(Ok(res)),
+            _ => Some(fetch_plain_file(
+                config,
+                &url,
+                &path,
+                reference.size,
+                &reference.checksums,
+                true,
+            )),
+        })
+        .ok_or_else(|| format_err!("Failed to retrieve {}", reference.path))??;
 
     let mut buf = Vec::new();
     let raw = res.data_ref();
@@ -556,7 +579,13 @@ pub fn create_snapshot(
                 }
 
                 // this will ensure the uncompressed file will be written locally
-                let res = match fetch_index_file(&config, prefix, reference, uncompressed_ref) {
+                let res = match fetch_index_file(
+                    &config,
+                    prefix,
+                    reference,
+                    uncompressed_ref,
+                    release.aquire_by_hash,
+                ) {
                     Ok(res) => res,
                     Err(err) if !reference.file_type.is_package_index() => {
                         eprintln!(
-- 
2.30.2

[pve-devel] [PATCH proxmox-apt 3/3] release: fix typo in 'Acquire-By-Hash'

[Fabian Grünbichler]

to allow detection of repositories that support downloading indices via
their hash instead of their filename.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
 src/deb822/release_file.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/deb822/release_file.rs b/src/deb822/release_file.rs
index 5888728..355b246 100644
--- a/src/deb822/release_file.rs
+++ b/src/deb822/release_file.rs
@@ -342,7 +342,7 @@ impl TryFrom<ReleaseFileRaw> for ReleaseFile {
         parsed.suite = value.suite;
         parsed.version = value.version;
 
-        parsed.aquire_by_hash = match value.extra_fields.get("Aquire-By-Hash") {
+        parsed.aquire_by_hash = match value.extra_fields.get("Acquire-By-Hash") {
             Some(val) => *val == "yes",
             None => false,
         };
-- 
2.30.2

[pve-devel] [PATCH proxmox-offline-mirror 4/4] mirror: use new architecture helper

[Fabian Grünbichler]

in order to avoid having a list of arch-specific references on two
places.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
requires proxmox-apt > 0.9.1 with the new helper

 src/mirror.rs | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/src/mirror.rs b/src/mirror.rs
index 6cbd680..3370ca4 100644
--- a/src/mirror.rs
+++ b/src/mirror.rs
@@ -501,14 +501,14 @@ pub fn create_snapshot(
             || match &reference.file_type {
                 FileReferenceType::Ignored => true,
                 FileReferenceType::PDiff => true, // would require fetching the patches as well
-                FileReferenceType::Contents(arch, _)
-                | FileReferenceType::ContentsUdeb(arch, _)
-                | FileReferenceType::Packages(arch, _)
-                | FileReferenceType::PseudoRelease(Some(arch)) => {
-                    !binary || !config.architectures.contains(arch)
-                }
                 FileReferenceType::Sources(_) => !source,
-                _ => false,
+                _ => {
+                    if let Some(arch) = reference.file_type.architecture() {
+                        !binary || !config.architectures.contains(arch)
+                    } else {
+                        false
+                    }
+                }
             };
         if skip {
             println!("Skipping {}", reference.path);
-- 
2.30.2

[pve-devel] [PATCH FIXUP proxmox-offline-mirror] clippy fix

[Fabian Grünbichler]

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
not really important, but can be folded into the patch since it's not
yet applied ;)

 src/mirror.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/mirror.rs b/src/mirror.rs
index f5ee48e..a1fc1a0 100644
--- a/src/mirror.rs
+++ b/src/mirror.rs
@@ -280,7 +280,7 @@ fn fetch_index_file(
             Some(Ok(res)) => Some(Ok(res)),
             _ => Some(fetch_plain_file(
                 config,
-                &url,
+                url,
                 &path,
                 reference.size,
                 &reference.checksums,
-- 
2.30.2

[pve-devel] [PATCH follow-up proxmox-offline-mirror 8/7] mirror: handle indices which are only available compressed

[Fabian Grünbichler]

there are repositories out there that not only skip serving the
uncompressed version, but not even reference it in their Release
file(s).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
best viewed with -w, the signal-desktop repository is one such example.

 src/mirror.rs | 56 ++++++++++++++++++++++++++++-----------------------
 1 file changed, 31 insertions(+), 25 deletions(-)

diff --git a/src/mirror.rs b/src/mirror.rs
index 3370ca4..f5ee48e 100644
--- a/src/mirror.rs
+++ b/src/mirror.rs
@@ -232,25 +232,30 @@ fn fetch_index_file(
     config: &ParsedMirrorConfig,
     prefix: &Path,
     reference: &FileReference,
-    uncompressed: &FileReference,
+    uncompressed: Option<&FileReference>,
     by_hash: bool,
 ) -> Result<FetchResult, Error> {
     let url = get_dist_url(&config.repository, &reference.path);
     let path = get_dist_path(&config.repository, prefix, &reference.path);
-    let uncompressed_path = get_dist_path(&config.repository, prefix, &uncompressed.path);
-
-    if config.pool.contains(&reference.checksums) && config.pool.contains(&uncompressed.checksums) {
-        let data = config
-            .pool
-            .get_contents(&uncompressed.checksums, config.verify)?;
-
-        // Ensure they're linked at current path
-        config.pool.lock()?.link_file(&reference.checksums, &path)?;
-        config
-            .pool
-            .lock()?
-            .link_file(&uncompressed.checksums, &uncompressed_path)?;
-        return Ok(FetchResult { data, fetched: 0 });
+
+    if let Some(uncompressed) = uncompressed {
+        let uncompressed_path = get_dist_path(&config.repository, prefix, &uncompressed.path);
+
+        if config.pool.contains(&reference.checksums)
+            && config.pool.contains(&uncompressed.checksums)
+        {
+            let data = config
+                .pool
+                .get_contents(&uncompressed.checksums, config.verify)?;
+
+            // Ensure they're linked at current path
+            config.pool.lock()?.link_file(&reference.checksums, &path)?;
+            config
+                .pool
+                .lock()?
+                .link_file(&uncompressed.checksums, &uncompressed_path)?;
+            return Ok(FetchResult { data, fetched: 0 });
+        }
     }
 
     let urls = if by_hash {
@@ -307,12 +312,15 @@ fn fetch_index_file(
     };
 
     let locked = &config.pool.lock()?;
-    if !locked.contains(&uncompressed.checksums) {
-        locked.add_file(decompressed, &uncompressed.checksums, config.sync)?;
-    }
+    if let Some(uncompressed) = uncompressed {
+        if !locked.contains(&uncompressed.checksums) {
+            locked.add_file(decompressed, &uncompressed.checksums, config.sync)?;
+        }
 
-    // Ensure it's linked at current path
-    locked.link_file(&uncompressed.checksums, &uncompressed_path)?;
+        // Ensure it's linked at current path
+        let uncompressed_path = get_dist_path(&config.repository, prefix, &uncompressed.path);
+        locked.link_file(&uncompressed.checksums, &uncompressed_path)?;
+    }
 
     Ok(FetchResult {
         data: decompressed.to_owned(),
@@ -566,15 +574,13 @@ pub fn create_snapshot(
         for basename in references {
             println!("\tFetching '{basename}'..");
             let files = release.files.get(basename).unwrap();
-            let uncompressed_ref = files
-                .iter()
-                .find(|reference| reference.path == *basename)
-                .ok_or_else(|| format_err!("Found derived reference without base reference."))?;
+            let uncompressed_ref = files.iter().find(|reference| reference.path == *basename);
+
             let mut package_index_data = None;
 
             for reference in files {
                 // if both compressed and uncompressed are referenced, the uncompressed file may not exist on the server
-                if reference == uncompressed_ref && files.len() > 1 {
+                if Some(reference) == uncompressed_ref && files.len() > 1 {
                     continue;
                 }
 
-- 
2.30.2

[pve-devel] applied-series: [PATCH proxmox-apt/proxmox-offline-mirror 0/7] misc improvements

[Thomas Lamprecht]

Am 15/09/2022 um 15:09 schrieb Fabian Grünbichler:
> this series adds some features and fixes some issues that pop up when
> attempting to mirror Ubuntu repositories, and should also improve
> resilience with other third-party repositories.
> 
> tested with Ubuntu Jammy (main, security and updates repositories).
> 

applied series, with the fixup squashed into the respective commit and a dependency
bump for proxmox-apt, thanks!