[pmg-devel] [PATCH-SERIES pmg-rs/pmg-api/pmg-gui] add APT repositories API/UI

[pmg-devel] [PATCH-SERIES pmg-rs/pmg-api/pmg-gui] add APT repositories API/UI

[Fabian Ebner]

Nearly identical to what's in PVE. I ran into very occasional errors
like
    error: invalid type: string "3", expected usize (500)
    error: expected bool value (500)
when using the enabled/disabled button, so I added int() coercions to
make perlmod always happy.

Dependency bump from pmg-api to pmg-rs needed.


pmg-rs:

Fabian Ebner (2):
  bump perlmod dependency
  add bindings for proxmox-apt

 Cargo.toml              |   6 +-
 Makefile                |   7 ++
 debian/control          |   2 +
 src/apt/mod.rs          |   1 +
 src/apt/repositories.rs | 159 ++++++++++++++++++++++++++++++++++++++++
 src/lib.rs              |   1 +
 6 files changed, 175 insertions(+), 1 deletion(-)
 create mode 100644 src/apt/mod.rs
 create mode 100644 src/apt/repositories.rs


pmg-api:

Fabian Ebner (1):
  api: apt: add calls for repositories

 src/PMG/API2/APT.pm | 292 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 292 insertions(+)


pmg-gui:

Fabian Ebner (1):
  add panel for APT repositories

 js/ServerAdministration.js | 7 +++++++
 1 file changed, 7 insertions(+)

-- 
2.30.2

[pmg-devel] [PATCH pmg-rs 1/2] bump perlmod dependency

[Fabian Ebner]

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
 Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Cargo.toml b/Cargo.toml
index 27d219b..d942daa 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -21,7 +21,7 @@ serde = "1.0"
 serde_bytes = "0.11.3"
 serde_json = "1.0"
 
-perlmod = { version = "0.4.2", features = [ "exporter" ] }
+perlmod = { version = "0.5.2", features = [ "exporter" ] }
 #perlmod = { path = "../perlmod/perlmod", features = [ "exporter" ] }
 
 proxmox-acme-rs = { version = "0.2.2", features = ["client"] }
-- 
2.30.2

[pmg-devel] [PATCH pmg-rs 2/2] add bindings for proxmox-apt

[Fabian Ebner]

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
 Cargo.toml              |   4 +
 Makefile                |   7 ++
 debian/control          |   2 +
 src/apt/mod.rs          |   1 +
 src/apt/repositories.rs | 159 ++++++++++++++++++++++++++++++++++++++++
 src/lib.rs              |   1 +
 6 files changed, 174 insertions(+)
 create mode 100644 src/apt/mod.rs
 create mode 100644 src/apt/repositories.rs

diff --git a/Cargo.toml b/Cargo.toml
index d942daa..553d5eb 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -24,5 +24,9 @@ serde_json = "1.0"
 perlmod = { version = "0.5.2", features = [ "exporter" ] }
 #perlmod = { path = "../perlmod/perlmod", features = [ "exporter" ] }
 
+proxmox = { version = "0.11.5", default-features = false }
+
 proxmox-acme-rs = { version = "0.2.2", features = ["client"] }
 #proxmox-acme-rs = { path = "../proxmox-acme-rs", features = ["client"] }
+
+proxmox-apt = "0.4.0"
diff --git a/Makefile b/Makefile
index 9d475c7..67eeb15 100644
--- a/Makefile
+++ b/Makefile
@@ -14,8 +14,12 @@ DEBS=$(MAIN_DEB) $(DBGSYM_DEB)
 
 DESTDIR=
 
+PM_DIRS := \
+	PMG/RS/APT
+
 PM_FILES := \
 	PMG/RS/Acme.pm \
+	PMG/RS/APT/Repositories.pm \
 	PMG/RS/CSR.pm
 
 ifeq ($(BUILD_MODE), release)
@@ -40,6 +44,9 @@ install: target/release/libpmg_rs.so
 	install -d -m755 $(DESTDIR)$(PERL_INSTALLVENDORARCH)/auto
 	install -m644 target/release/libpmg_rs.so $(DESTDIR)$(PERL_INSTALLVENDORARCH)/auto/libpmg_rs.so
 	install -d -m755 $(DESTDIR)$(PERL_INSTALLVENDORLIB)/PMG/RS
+	for i in $(PM_DIRS); do \
+	  install -d -m755 $(DESTDIR)$(PERL_INSTALLVENDORLIB)/$$i; \
+	done
 	for i in $(PM_FILES); do \
 	  install -m644 $$i $(DESTDIR)$(PERL_INSTALLVENDORLIB)/$$i; \
 	done
diff --git a/debian/control b/debian/control
index 677a28d..fe5dad6 100644
--- a/debian/control
+++ b/debian/control
@@ -5,6 +5,8 @@ Maintainer: Proxmox Support Team <support@proxmox.com>
 Build-Depends: debhelper (>= 12),
                librust-curl-dev,
                librust-proxmox-acme-rs+client-dev,
+               librust-proxmox-apt-dev,
+               librust-proxmox-dev,
                librust-perlmod-macro-dev,
                librust-anyhow-dev,
                librust-openssl-dev,
diff --git a/src/apt/mod.rs b/src/apt/mod.rs
new file mode 100644
index 0000000..574c1a7
--- /dev/null
+++ b/src/apt/mod.rs
@@ -0,0 +1 @@
+mod repositories;
diff --git a/src/apt/repositories.rs b/src/apt/repositories.rs
new file mode 100644
index 0000000..81552e3
--- /dev/null
+++ b/src/apt/repositories.rs
@@ -0,0 +1,159 @@
+#[perlmod::package(name = "PMG::RS::APT::Repositories", lib = "pmg_rs")]
+mod export {
+    use std::convert::TryInto;
+
+    use anyhow::{bail, Error};
+    use serde::{Deserialize, Serialize};
+
+    use proxmox_apt::repositories::{
+        APTRepositoryFile, APTRepositoryFileError, APTRepositoryHandle, APTRepositoryInfo,
+        APTStandardRepository,
+    };
+
+    #[derive(Deserialize, Serialize)]
+    #[serde(rename_all = "kebab-case")]
+    /// Result for the repositories() function
+    pub struct RepositoriesResult {
+        /// Successfully parsed files.
+        pub files: Vec<APTRepositoryFile>,
+
+        /// Errors for files that could not be parsed or read.
+        pub errors: Vec<APTRepositoryFileError>,
+
+        /// Common digest for successfully parsed files.
+        pub digest: String,
+
+        /// Additional information/warnings about repositories.
+        pub infos: Vec<APTRepositoryInfo>,
+
+        /// Standard repositories and their configuration status.
+        pub standard_repos: Vec<APTStandardRepository>,
+    }
+
+    #[derive(Deserialize, Serialize)]
+    #[serde(rename_all = "kebab-case")]
+    /// For changing an existing repository.
+    pub struct ChangeProperties {
+        /// Whether the repository should be enabled or not.
+        pub enabled: Option<bool>,
+    }
+
+    /// Get information about configured and standard repositories.
+    #[export]
+    pub fn repositories() -> Result<RepositoriesResult, Error> {
+        let (files, errors, digest) = proxmox_apt::repositories::repositories()?;
+        let digest = proxmox::tools::digest_to_hex(&digest);
+
+        let infos = proxmox_apt::repositories::check_repositories(&files)?;
+        let standard_repos = proxmox_apt::repositories::standard_repositories("pmg", &files);
+
+        Ok(RepositoriesResult {
+            files,
+            errors,
+            digest,
+            infos,
+            standard_repos,
+        })
+    }
+
+    /// Add the repository identified by the `handle`.
+    /// If the repository is already configured, it will be set to enabled.
+    ///
+    /// The `digest` parameter asserts that the configuration has not been modified.
+    #[export]
+    pub fn add_repository(handle: &str, digest: Option<&str>) -> Result<(), Error> {
+        let (mut files, errors, current_digest) = proxmox_apt::repositories::repositories()?;
+
+        let handle: APTRepositoryHandle = handle.try_into()?;
+
+        if let Some(digest) = digest {
+            let expected_digest = proxmox::tools::hex_to_digest(digest)?;
+            if expected_digest != current_digest {
+                bail!("detected modified configuration - file changed by other user? Try again.");
+            }
+        }
+
+        // check if it's already configured first
+        for file in files.iter_mut() {
+            for repo in file.repositories.iter_mut() {
+                if repo.is_referenced_repository(handle, "pmg") {
+                    if repo.enabled {
+                        return Ok(());
+                    }
+
+                    repo.set_enabled(true);
+                    file.write()?;
+
+                    return Ok(());
+                }
+            }
+        }
+
+        let (repo, path) = proxmox_apt::repositories::get_standard_repository(handle, "pmg")?;
+
+        if let Some(error) = errors.iter().find(|error| error.path == path) {
+            bail!(
+                "unable to parse existing file {} - {}",
+                error.path,
+                error.error,
+            );
+        }
+
+        if let Some(file) = files.iter_mut().find(|file| file.path == path) {
+            file.repositories.push(repo);
+
+            file.write()?;
+        } else {
+            let mut file = match APTRepositoryFile::new(&path)? {
+                Some(file) => file,
+                None => bail!("invalid path - {}", path),
+            };
+
+            file.repositories.push(repo);
+
+            file.write()?;
+        }
+
+        Ok(())
+    }
+
+    /// Change the properties of the specified repository.
+    ///
+    /// The `digest` parameter asserts that the configuration has not been modified.
+    #[export]
+    pub fn change_repository(
+        path: &str,
+        index: usize,
+        options: ChangeProperties,
+        digest: Option<&str>,
+    ) -> Result<(), Error> {
+        let (mut files, errors, current_digest) = proxmox_apt::repositories::repositories()?;
+
+        if let Some(digest) = digest {
+            let expected_digest = proxmox::tools::hex_to_digest(digest)?;
+            if expected_digest != current_digest {
+                bail!("detected modified configuration - file changed by other user? Try again.");
+            }
+        }
+
+        if let Some(error) = errors.iter().find(|error| error.path == path) {
+            bail!("unable to parse file {} - {}", error.path, error.error);
+        }
+
+        if let Some(file) = files.iter_mut().find(|file| file.path == path) {
+            if let Some(repo) = file.repositories.get_mut(index) {
+                if let Some(enabled) = options.enabled {
+                    repo.set_enabled(enabled);
+                }
+
+                file.write()?;
+            } else {
+                bail!("invalid index - {}", index);
+            }
+        } else {
+            bail!("invalid path - {}", path);
+        }
+
+        Ok(())
+    }
+}
diff --git a/src/lib.rs b/src/lib.rs
index 9822106..47e61b5 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -1,2 +1,3 @@
 pub mod acme;
+pub mod apt;
 pub mod csr;
-- 
2.30.2

[pmg-devel] [PATCH pmg-api 1/1] api: apt: add calls for repositories

[Fabian Ebner]

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
 src/PMG/API2/APT.pm | 292 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 292 insertions(+)

diff --git a/src/PMG/API2/APT.pm b/src/PMG/API2/APT.pm
index dafe4f4..ce54b0f 100644
--- a/src/PMG/API2/APT.pm
+++ b/src/PMG/API2/APT.pm
@@ -20,6 +20,7 @@ use PVE::JSONSchema qw(get_standard_option);
 use PMG::RESTEnvironment;
 use PMG::pmgcfg;
 use PMG::Config;
+use PMG::RS::APT::Repositories;
 
 use AptPkg::Cache;
 use AptPkg::Version;
@@ -63,6 +64,7 @@ __PACKAGE__->register_method({
 
 	my $res = [
 	    { id => 'changelog' },
+	    { id => 'repositories' },
 	    { id => 'update' },
 	    { id => 'versions' },
 	];
@@ -469,6 +471,296 @@ __PACKAGE__->register_method({
 	return $data;
     }});
 
+__PACKAGE__->register_method({
+    name => 'repositories',
+    path => 'repositories',
+    method => 'GET',
+    proxyto => 'node',
+    description => "Get APT repository information.",
+    permissions => {
+	check => ['perm', '/nodes/{node}', [ 'Sys.Audit' ]],
+    },
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	},
+    },
+    returns => {
+	type => "object",
+	description => "Result from parsing the APT repository files in /etc/apt/.",
+	properties => {
+	    files => {
+		type => "array",
+		description => "List of parsed repository files.",
+		items => {
+		    type => "object",
+		    properties => {
+			path => {
+			    type => "string",
+			    description => "Path to the problematic file.",
+			},
+			'file-type' => {
+			    type => "string",
+			    enum => [ 'list', 'sources' ],
+			    description => "Format of the file.",
+			},
+			repositories => {
+			    type => "array",
+			    description => "The parsed repositories.",
+			    items => {
+				type => "object",
+				properties => {
+				    Types => {
+					type => "array",
+					description => "List of package types.",
+					items => {
+					    type => "string",
+					    enum => [ 'deb', 'deb-src' ],
+					},
+				    },
+				    URIs => {
+					description => "List of repository URIs.",
+					type => "array",
+					items => {
+					    type => "string",
+					},
+				    },
+				    Suites => {
+					type => "array",
+					description => "List of package distribuitions",
+					items => {
+					    type => "string",
+					},
+				    },
+				    Components => {
+					type => "array",
+					description => "List of repository components",
+					optional => 1, # not present if suite is absolute
+					items => {
+					    type => "string",
+					},
+				    },
+				    Options => {
+					type => "array",
+					description => "Additional options",
+					optional => 1,
+					items => {
+					    type => "object",
+					    properties => {
+						Key => {
+						    type => "string",
+						},
+						Values => {
+						    type => "array",
+						    items => {
+							type => "string",
+						    },
+						},
+					    },
+					},
+				    },
+				    Comment => {
+					type => "string",
+					description => "Associated comment",
+					optional => 1,
+				    },
+				    FileType => {
+					type => "string",
+					enum => [ 'list', 'sources' ],
+					description => "Format of the defining file.",
+				    },
+				    Enabled => {
+					type => "boolean",
+					description => "Whether the repository is enabled or not",
+				    },
+				},
+			    },
+			},
+			digest => {
+			    type => "array",
+			    description => "Digest of the file as bytes.",
+			    items => {
+				type => "integer",
+			    },
+			},
+		    },
+		},
+	    },
+	    errors => {
+		type => "array",
+		description => "List of problematic repository files.",
+		items => {
+		    type => "object",
+		    properties => {
+			path => {
+			    type => "string",
+			    description => "Path to the problematic file.",
+			},
+			error => {
+			    type => "string",
+			    description => "The error message",
+			},
+		    },
+		},
+	    },
+	    digest => {
+		type => "string",
+		description => "Common digest of all files.",
+	    },
+	    infos => {
+		type => "array",
+		description => "Additional information/warnings for APT repositories.",
+		items => {
+		    type => "object",
+		    properties => {
+			path => {
+			    type => "string",
+			    description => "Path to the associated file.",
+			},
+			index => {
+			    type => "string",
+			    description => "Index of the associated repository within the file.",
+			},
+			property => {
+			    type => "string",
+			    description => "Property from which the info originates.",
+			    optional => 1,
+			},
+			kind => {
+			    type => "string",
+			    description => "Kind of the information (e.g. warning).",
+			},
+			message => {
+			    type => "string",
+			    description => "Information message.",
+			}
+		    },
+		},
+	    },
+	    'standard-repos' => {
+		type => "array",
+		description => "List of standard repositories and their configuration status",
+		items => {
+		    type => "object",
+		    properties => {
+			handle => {
+			    type => "string",
+			    description => "Handle to identify the repository.",
+			},
+			name => {
+			    type => "string",
+			    description => "Display name of the repository.",
+			},
+			description => {
+			    type => "string",
+			    description => "Description of the repository.",
+			},
+			status => {
+			    type => "boolean",
+			    optional => 1,
+			    description => "Indicating enabled/disabled status, if the " .
+				"repository is configured.",
+			},
+		    },
+		},
+	    },
+	},
+    },
+    code => sub {
+	my ($param) = @_;
+
+	return PMG::RS::APT::Repositories::repositories();
+    }});
+
+__PACKAGE__->register_method({
+    name => 'add_repository',
+    path => 'repositories',
+    method => 'PUT',
+    description => "Add a standard repository to the configuration",
+    permissions => {
+	check => ['perm', '/nodes/{node}', [ 'Sys.Modify' ]],
+    },
+    protected => 1,
+    proxyto => 'node',
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    handle => {
+		type => 'string',
+		description => "Handle that identifies a repository.",
+	    },
+	    digest => {
+		type => "string",
+		description => "Digest to detect modifications.",
+		maxLength => 80,
+		optional => 1,
+	    },
+	},
+    },
+    returns => {
+	type => 'null',
+    },
+    code => sub {
+	my ($param) = @_;
+
+	PMG::RS::APT::Repositories::add_repository($param->{handle}, $param->{digest});
+    }});
+
+__PACKAGE__->register_method({
+    name => 'change_repository',
+    path => 'repositories',
+    method => 'POST',
+    description => "Change the properties of a repository. Currently only allows enabling/disabling.",
+    permissions => {
+	check => ['perm', '/nodes/{node}', [ 'Sys.Modify' ]],
+    },
+    protected => 1,
+    proxyto => 'node',
+    parameters => {
+	additionalProperties => 0,
+	properties => {
+	    node => get_standard_option('pve-node'),
+	    path => {
+		type => 'string',
+		description => "Path to the containing file.",
+	    },
+	    index => {
+		type => 'integer',
+		description => "Index within the file (starting from 0).",
+	    },
+	    enabled => {
+		type => 'boolean',
+		description => "Whether the repository should be enabled or not.",
+		optional => 1,
+	    },
+	    digest => {
+		type => "string",
+		description => "Digest to detect modifications.",
+		maxLength => 80,
+		optional => 1,
+	    },
+	},
+    },
+    returns => {
+	type => 'null',
+    },
+    code => sub {
+	my ($param) = @_;
+
+	my $options = {
+	    enabled => int($param->{enabled}),
+	};
+
+	PMG::RS::APT::Repositories::change_repository(
+	    $param->{path},
+	    int($param->{index}),
+	    $options,
+	    $param->{digest}
+	);
+    }});
+
 __PACKAGE__->register_method({
     name => 'versions',
     path => 'versions',
-- 
2.30.2

[pmg-devel] [PATCH pmg-gui 1/1] add panel for APT repositories

[Fabian Ebner]

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
 js/ServerAdministration.js | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/js/ServerAdministration.js b/js/ServerAdministration.js
index 76f569d..76bdac7 100644
--- a/js/ServerAdministration.js
+++ b/js/ServerAdministration.js
@@ -47,6 +47,13 @@ Ext.define('PMG.ServerAdministration', {
 	    itemId: 'updates',
 	    nodename: Proxmox.NodeName,
 	},
+	{
+	    xtype: 'proxmoxNodeAPTRepositories',
+	    title: gettext('Repositories'),
+	    itemId: 'aptrepositories',
+	    nodename: 'localhost',
+	    product: 'Proxmox Mail Gateway',
+	},
 	{
 	    xtype: 'proxmoxJournalView',
 	    itemId: 'logs',
-- 
2.30.2

[pmg-devel] applied-series: [PATCH-SERIES pmg-rs/pmg-api/pmg-gui] add APT repositories API/UI

[Thomas Lamprecht]

On 13.07.21 10:04, Fabian Ebner wrote:
> Nearly identical to what's in PVE. I ran into very occasional errors
> like
>     error: invalid type: string "3", expected usize (500)
>     error: expected bool value (500)
> when using the enabled/disabled button, so I added int() coercions to
> make perlmod always happy.
> 
> Dependency bump from pmg-api to pmg-rs needed.
> 
> 
> pmg-rs:
> 
> Fabian Ebner (2):
>   bump perlmod dependency
>   add bindings for proxmox-apt
> 
>  Cargo.toml              |   6 +-
>  Makefile                |   7 ++
>  debian/control          |   2 +
>  src/apt/mod.rs          |   1 +
>  src/apt/repositories.rs | 159 ++++++++++++++++++++++++++++++++++++++++
>  src/lib.rs              |   1 +
>  6 files changed, 175 insertions(+), 1 deletion(-)
>  create mode 100644 src/apt/mod.rs
>  create mode 100644 src/apt/repositories.rs
> 
> 
> pmg-api:
> 
> Fabian Ebner (1):
>   api: apt: add calls for repositories
> 
>  src/PMG/API2/APT.pm | 292 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 292 insertions(+)
> 
> 
> pmg-gui:
> 
> Fabian Ebner (1):
>   add panel for APT repositories
> 
>  js/ServerAdministration.js | 7 +++++++
>  1 file changed, 7 insertions(+)
> 

Another reuse for that whole stack, great! now we need more projects to increase reuse factors ;-P

applied series, thanks!