[pve-devel] [PATCH proxmox-offline-mirror 1/3] mirror: also filter {component}/{section}

[pve-devel] [PATCH proxmox-offline-mirror 1/3] mirror: also filter {component}/{section}

[Fabian Grünbichler]

so that a single filter 'games' covers all related sections for a stock
Debian repository.

this also has the side-effect that package downloads are now batched by
component.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
 src/mirror.rs | 58 +++++++++++++++++++++++++++++++++++----------------
 1 file changed, 40 insertions(+), 18 deletions(-)

diff --git a/src/mirror.rs b/src/mirror.rs
index e6aca8a..ea593a9 100644
--- a/src/mirror.rs
+++ b/src/mirror.rs
@@ -504,6 +504,7 @@ fn convert_to_globset(config: &ParsedMirrorConfig) -> Result<Option<GlobSet>, Er
 
 fn fetch_binary_packages(
     config: &ParsedMirrorConfig,
+    component: &str,
     packages_indices: HashMap<&String, PackagesFile>,
     dry_run: bool,
     prefix: &Path,
@@ -523,9 +524,13 @@ fn fetch_binary_packages(
         let mut fetch_progress = Progress::new();
         let mut skip_count = 0usize;
         let mut skip_bytes = 0usize;
+
         for package in references.files {
             if let Some(ref sections) = &config.skip.skip_sections {
-                if sections.iter().any(|section| package.section == *section) {
+                if sections.iter().any(|section| {
+                    package.section == *section
+                        || package.section == format!("{component}/{section}")
+                }) {
                     println!(
                         "\tskipping {} - {}b (section '{}')",
                         package.package, package.size, package.section
@@ -615,6 +620,7 @@ fn fetch_binary_packages(
 
 fn fetch_source_packages(
     config: &ParsedMirrorConfig,
+    component: &str,
     source_packages_indices: HashMap<&String, SourcesFile>,
     dry_run: bool,
     prefix: &Path,
@@ -636,10 +642,10 @@ fn fetch_source_packages(
         let mut skip_bytes = 0usize;
         for package in references.source_packages {
             if let Some(ref sections) = &config.skip.skip_sections {
-                if sections
-                    .iter()
-                    .any(|section| package.section.as_ref() == Some(section))
-                {
+                if sections.iter().any(|section| {
+                    package.section.as_ref() == Some(section)
+                        || package.section == Some(format!("{component}/{section}"))
+                }) {
                     println!(
                         "\tskipping {} - {}b (section '{}')",
                         package.package,
@@ -897,9 +903,13 @@ pub fn create_snapshot(
     println!();
 
     let mut packages_size = 0_usize;
-    let mut packages_indices = HashMap::new();
-
-    let mut source_packages_indices = HashMap::new();
+    let mut per_component_indices: HashMap<
+        String,
+        (
+            HashMap<&String, PackagesFile>,
+            HashMap<&String, SourcesFile>,
+        ),
+    > = HashMap::new();
 
     let mut failed_references = Vec::new();
     for (component, references) in per_component {
@@ -909,6 +919,9 @@ pub fn create_snapshot(
 
         let mut fetch_progress = Progress::new();
 
+        let (packages_indices, source_packages_indices) =
+            per_component_indices.entry(component.clone()).or_default();
+
         for basename in references {
             println!("\tFetching '{basename}'..");
             let files = release.files.get(basename).unwrap();
@@ -1001,17 +1014,26 @@ pub fn create_snapshot(
         }
     }
 
-    println!("\nFetching packages..");
-
-    fetch_binary_packages(&config, packages_indices, dry_run, prefix, &mut progress)?;
+    for (component, (packages_indices, source_packages_indices)) in per_component_indices {
+        println!("\nFetching {component} packages..");
+        fetch_binary_packages(
+            &config,
+            &component,
+            packages_indices,
+            dry_run,
+            prefix,
+            &mut progress,
+        )?;
 
-    fetch_source_packages(
-        &config,
-        source_packages_indices,
-        dry_run,
-        prefix,
-        &mut progress,
-    )?;
+        fetch_source_packages(
+            &config,
+            &component,
+            source_packages_indices,
+            dry_run,
+            prefix,
+            &mut progress,
+        )?;
+    }
 
     if dry_run {
         println!(
-- 
2.30.2

[pve-devel] [PATCH proxmox-offline-mirror 2/3] mirror setup: query filters in guided mode

[Fabian Grünbichler]

with a somewhat sensible default of filtering the games and debug
sections - which already reduces a mirror of PVE + Debian bullseye by
about 27% (105GB->77GB).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
 src/bin/proxmox-offline-mirror.rs | 42 ++++++++++++++++++++-----------
 1 file changed, 28 insertions(+), 14 deletions(-)

diff --git a/src/bin/proxmox-offline-mirror.rs b/src/bin/proxmox-offline-mirror.rs
index 07b6ce6..67036f1 100644
--- a/src/bin/proxmox-offline-mirror.rs
+++ b/src/bin/proxmox-offline-mirror.rs
@@ -96,7 +96,20 @@ fn derive_debian_repo(
     release: &Release,
     variant: &DebianVariant,
     components: &str,
-) -> (String, String, String) {
+) -> Result<(String, String, String, SkipConfig), Error> {
+    println!("Configure filters for Debian mirror {release} / {variant}:");
+    let skip_sections = match read_string_from_tty("\tEnter list of package sections to be skipped ('-' for None):", Some("debug,games"))?.as_str() {
+        "-" => None,
+        list => Some(list.split(',').map(|v| v.trim().to_owned()).collect::<Vec<String>>()),
+    };
+    let skip_packages = match read_string_from_tty("\tEnter list of package names/name globs to be skipped ('-' for None):", None)?.as_str() {
+        "-" => None,
+        list => Some(list.split(',').map(|v| v.trim().to_owned()).collect::<Vec<String>>()),
+    };
+    let filters = SkipConfig {
+        skip_packages,
+        skip_sections,
+    };
     let url = match (release, variant) {
         (Release::Bullseye, DebianVariant::Main) => "http://deb.debian.org/debian bullseye",
         (Release::Bullseye, DebianVariant::Security) => {
@@ -138,14 +151,14 @@ fn derive_debian_repo(
 
     let suggested_id = format!("debian_{release}_{variant}");
 
-    (url, key.to_string(), suggested_id)
+    Ok((url, key.to_string(), suggested_id, filters))
 }
 
 fn action_add_mirror(config: &SectionConfigData) -> Result<Vec<MirrorConfig>, Error> {
     let mut use_subscription = None;
     let mut extra_repos = Vec::new();
 
-    let (repository, key_path, architectures, suggested_id) = if read_bool_from_tty(
+    let (repository, key_path, architectures, suggested_id, skip) = if read_bool_from_tty(
         "Guided Setup",
         Some(true),
     )? {
@@ -163,7 +176,7 @@ fn action_add_mirror(config: &SectionConfigData) -> Result<Vec<MirrorConfig>, Er
 
         let mut add_debian_repo = false;
 
-        let (url, key_path, suggested_id) = match dist {
+        let (url, key_path, suggested_id, skip) = match dist {
             Distro::Debian => {
                 let variants = &[
                     (DebianVariant::Main, "Main repository"),
@@ -179,7 +192,7 @@ fn action_add_mirror(config: &SectionConfigData) -> Result<Vec<MirrorConfig>, Er
                     Some("main contrib non-free"),
                 )?;
 
-                derive_debian_repo(release, variant, &components)
+                derive_debian_repo(release, variant, &components)?
             }
             Distro::PveCeph => {
                 enum CephRelease {
@@ -231,7 +244,7 @@ fn action_add_mirror(config: &SectionConfigData) -> Result<Vec<MirrorConfig>, Er
                 );
                 let suggested_id = format!("ceph_{ceph_release}_{release}");
 
-                (url, key.to_string(), suggested_id)
+                (url, key.to_string(), suggested_id, SkipConfig::default())
             }
             product => {
                 let variants = &[
@@ -272,7 +285,7 @@ fn action_add_mirror(config: &SectionConfigData) -> Result<Vec<MirrorConfig>, Er
                     Some(true),
                 )?;
 
-                (url, key.to_string(), suggested_id)
+                (url, key.to_string(), suggested_id, SkipConfig::default())
             }
         };
 
@@ -283,23 +296,24 @@ fn action_add_mirror(config: &SectionConfigData) -> Result<Vec<MirrorConfig>, Er
                 release,
                 &DebianVariant::Main,
                 "main contrib",
-            ));
+            )?);
             extra_repos.push(derive_debian_repo(
                 release,
                 &DebianVariant::Updates,
                 "main contrib",
-            ));
+            )?);
             extra_repos.push(derive_debian_repo(
                 release,
                 &DebianVariant::Security,
                 "main contrib",
-            ));
+            )?);
         }
         (
             format!("deb {url}"),
             key_path,
             architectures,
             Some(suggested_id),
+            skip,
         )
     } else {
         let repo = read_string_from_tty("Enter repository line in sources.list format", None)?;
@@ -329,7 +343,7 @@ fn action_add_mirror(config: &SectionConfigData) -> Result<Vec<MirrorConfig>, Er
         )?
         .clone();
 
-        (repo, key_path, architectures, None)
+        (repo, key_path, architectures, None, SkipConfig::default())
     };
 
     if !Path::new(&key_path).exists() {
@@ -371,7 +385,7 @@ fn action_add_mirror(config: &SectionConfigData) -> Result<Vec<MirrorConfig>, Er
 
     let mut configs = Vec::with_capacity(extra_repos.len() + 1);
 
-    for (url, key_path, suggested_id) in extra_repos {
+    for (url, key_path, suggested_id, skip) in extra_repos {
         if config.sections.contains_key(&suggested_id) {
             eprintln!("config section '{suggested_id}' already exists, skipping..");
         } else {
@@ -387,7 +401,7 @@ fn action_add_mirror(config: &SectionConfigData) -> Result<Vec<MirrorConfig>, Er
                 base_dir: base_dir.clone(),
                 use_subscription: None,
                 ignore_errors: false,
-                skip: SkipConfig::default(), // TODO sensible default?
+                skip,
             });
         }
     }
@@ -402,7 +416,7 @@ fn action_add_mirror(config: &SectionConfigData) -> Result<Vec<MirrorConfig>, Er
         base_dir,
         use_subscription,
         ignore_errors: false,
-        skip: SkipConfig::default(),
+        skip,
     };
 
     configs.push(main_config);
-- 
2.30.2

[pve-devel] [PATCH proxmox-offline-mirror 3/3] docs: add section/package filters

[Fabian Grünbichler]

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
---
 docs/offline-mirror.rst | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/docs/offline-mirror.rst b/docs/offline-mirror.rst
index ac78e9a..fde27ed 100644
--- a/docs/offline-mirror.rst
+++ b/docs/offline-mirror.rst
@@ -52,6 +52,34 @@ To create the first (and subsequent) snapshots, the following command can be use
   snapshot can take both time and require significant disk space. This is especially true for the
   initial snapshot, as subsequent ones will re-use unchanged package files and indices.
 
+Reducing Mirror Scope
+---------------------
+
+There are different mechanisms for reducing a mirror's scope (and correspondingly, the amount of
+traffic and disk space required to keep it synced):
+
+- architecture filters
+- components (as part of the `repository` specification)
+- package name and section filters
+
+By default, only packages for the architectures `all` (see note above) and `amd64` are mirrored.
+
+Optionally, it's possible to setup filters for downloaded binary or source packages via the
+`--skip-packages` and `--skip-sections` options. The package filters support globbing, for example
+`linux-image-*` will skip all packages with a name starting with `linux-image-`. The section
+filters match the full value, or the value prefixed with the package's component (for example,
+`games` will match both the section `games`, as well as `non-free/games` in a packages index of the
+`non-free` component).
+
+Some examples for packages and section filters:
+
+- `--skip-packages 'linux-image-*'` - filter Debian linux kernel image packages
+- `--skip-sections 'games'` - filter sections containing game packages
+- `--skip-sections 'debug'` - filter sections containing debug information
+
+Please refer to https://packages.debian.org/bullseye/ for a list of Debian archive sections and
+their contents.
+
 Space Management
 ----------------
 
-- 
2.30.2

[pve-devel] applied-series: [PATCH proxmox-offline-mirror 1/3] mirror: also filter {component}/{section}

[Wolfgang Bumiller]

applied series, thanks