Re: [pve-devel] [pbs-devel] [RFC v5 pve-rs 16/23] add perlmod crate with initial APT module

Re: [pve-devel] [pbs-devel] [RFC v5 pve-rs 16/23] add perlmod crate with initial APT module

[Wolfgang Bumiller]

> On 05/31/2021 3:17 PM Fabian Ebner <f.ebner@proxmox.com> wrote:
> 
>  
> Am 31.05.21 um 14:55 schrieb Wolfgang Bumiller:
> > Please don't use 'perlmod' as the name of a crate which also depends on
> > a different 'perlmod' crate, that won't be fun for long.
> > 
> > Call it pve-rs ;-)
> >
> 
> I called it perlmod, because that's what was suggested in the (outdated) 

That was probably just confusingly written but actually meant to suggest 'pve-pl'
(also bad enough ^^) *utilizing* perlmod ;-)

> README. Should it even be a sub-crate then, or organized like pmg-rs?

Either is fine with me. I expect it to end up being multiple crates at some point,
(IMO more likely for pve-rs than pmg-rs), but the end goal of that would probably
be to use them in pbs/pmg, too, so dependin gon their size they might live in another
repo then anyway, who knows.

I mean we can always change it, so I do *slightly* lean more towards pmg-rs like,
with a possible inline workspace in the near future, but only slightly.

[pve-devel] [PATCH-SERIES v5] APT repositories API/UI

[Fabian Ebner]

List the configured repositories, have some basic checks for them, and
allow upgrading the package distribution before a major release.

The plan is to use perlmod to make the Rust implementation available for PVE+PMG
as well.

For PBS, everything beside the upgrade call should be ready (the upgrade
call should work too, but what's the best way to enable/disable the call
pre/post-upgrade?)

For PVE, it's all RFC.

Note that pve-rs patches are based on top of Wolfang's staff repository. I hope
I didn't completly misinterpret the inteded packaging/structure there, tried to
base it off pmg-rs and the README.

For perlmod (still feels like magic to me), I couldn't figure out how to have an
Option<String> as an argument for an exported function. Tried using undef, but
got
     error: cannot deserialize weird magic perl values (10)
Worked around by using empty string as a None replacement instead.


Changes from v4:
    * some minor style improvements
    * call cargo clean with make clean in proxmox-apt
    * require 'deb' package type to detect enterprise/no-subscription repos
    * moved common_digest helper to the library
    * add replace_suite function in proxmox-apt and refactored suite_is_variant
      for re-use
    * ui: add reload button
    * ui: don't pass undefined digest parameter
    * add upgrade button
    * add RFC for PBS upgrade call
    * add RFCs for PVE integration


Still missing (intended as followups):
    * integration in PMG.


Changes v3 -> v4:
    * incorporate Fabian G.'s feedback:
        * switch to a per-file approach
        * check for official host names
        * fix case-sensitivity issue for .sources keys
        * include digests
    * fix write issue when there are no components (in case of an absolute suite)
    * add more tests


Changes v2 -> v3:
    * incorporate Wolfgang's feedback
    * improve main warning's UI

Changes v1 -> v2:
    * Perl -> Rust
    * PVE -> PBS
    * Don't rely on regexes for parsing.
    * Add writer and tests.
    * UI: pin warnings to the repository they're for.
    * Keep order of options consistent with configuration.
    * Smaller things noted on the individual patches.


proxmox-apt:

Fabian Ebner (6):
  initial commit
  add files for Debian packaging
  add functions to check for Proxmox repositories
  add check_repositories function
  add common_digest helper
  add replace_suite function and constants for the current/next stable
    suites


proxmox-backup:

Fabian Ebner (5):
  depend on new proxmox-apt crate
  api: apt: add repositories call
  ui: add APT repositories
  add check_repositories_call
  RFC: add upgrade_repositories call

 Cargo.toml                  |   1 +
 debian/control              |   1 +
 src/api2/node/apt.rs        | 176 +++++++++++++++++++++++++++++++++++-
 www/ServerAdministration.js |   8 ++
 4 files changed, 185 insertions(+), 1 deletion(-)


proxmox-widget-toolkit:

Fabian Ebner (3):
  add UI for APT repositories
  APT repositories: add warnings
  add upgrade button

 src/Makefile                |   1 +
 src/node/APTRepositories.js | 465 ++++++++++++++++++++++++++++++++++++
 2 files changed, 466 insertions(+)
 create mode 100644 src/node/APTRepositories.js


pve-rs:

Fabian Ebner (5):
  add cargo config
  add perlmod crate with initial APT module
  add files for Debian packaging
  update .gitignore
  RFC: add upgrade_repositories wrapper

 .cargo/config                   |  5 ++
 .gitignore                      |  6 +++
 Cargo.toml                      |  3 +-
 Makefile                        | 73 ++++++++++++++++++++++++++
 debian/changelog                |  5 ++
 debian/compat                   |  1 +
 debian/control                  | 20 +++++++
 debian/copyright                | 16 ++++++
 debian/rules                    |  7 +++
 debian/triggers                 |  1 +
 perlmod/Cargo.toml              | 17 ++++++
 perlmod/src/apt/mod.rs          |  1 +
 perlmod/src/apt/repositories.rs | 92 +++++++++++++++++++++++++++++++++
 perlmod/src/lib.rs              |  1 +
 14 files changed, 247 insertions(+), 1 deletion(-)
 create mode 100644 .cargo/config
 create mode 100644 Makefile
 create mode 100644 debian/changelog
 create mode 100644 debian/compat
 create mode 100644 debian/control
 create mode 100644 debian/copyright
 create mode 100755 debian/rules
 create mode 100644 debian/triggers
 create mode 100644 perlmod/Cargo.toml
 create mode 100644 perlmod/src/apt/mod.rs
 create mode 100644 perlmod/src/apt/repositories.rs
 create mode 100644 perlmod/src/lib.rs


pve-manager:

Fabian Ebner (4):
  api: apt: add call to list repositories
  ui: add panel for listing APT repositories
  api: apt: add call for repository check
  api: apt: add upgrade repos call

 PVE/API2/APT.pm             | 265 ++++++++++++++++++++++++++++++++++++
 www/manager6/node/Config.js |   8 ++
 2 files changed, 273 insertions(+)

-- 
2.20.1

[pve-devel] [RFC v5 pve-rs 16/23] add perlmod crate with initial APT module

[Fabian Ebner]

and disable diskmanage for now, whose Cargo.toml references local paths.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---

New in v5.

 Cargo.toml                      |  3 +-
 perlmod/Cargo.toml              | 17 ++++++++++
 perlmod/src/apt/mod.rs          |  1 +
 perlmod/src/apt/repositories.rs | 55 +++++++++++++++++++++++++++++++++
 perlmod/src/lib.rs              |  1 +
 5 files changed, 76 insertions(+), 1 deletion(-)
 create mode 100644 perlmod/Cargo.toml
 create mode 100644 perlmod/src/apt/mod.rs
 create mode 100644 perlmod/src/apt/repositories.rs
 create mode 100644 perlmod/src/lib.rs

diff --git a/Cargo.toml b/Cargo.toml
index d16a498..0ffd4a2 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,4 +1,5 @@
 [workspace]
 members = [
-    "diskmanage",
+#    "diskmanage",
+    "perlmod",
 ]
diff --git a/perlmod/Cargo.toml b/perlmod/Cargo.toml
new file mode 100644
index 0000000..397f444
--- /dev/null
+++ b/perlmod/Cargo.toml
@@ -0,0 +1,17 @@
+[package]
+name = "pve-rs"
+version = "0.1.0"
+authors = ["Proxmox Support Team <support@proxmox.com>"]
+edition = "2018"
+license = "AGPL-3"
+description = "Perl bindings for proxmox-apt"
+homepage = "https://www.proxmox.com"
+
+[lib]
+crate-type = [ "cdylib" ]
+
+[dependencies]
+anyhow = "1.0"
+proxmox = { version = "0.11.5" }
+proxmox-apt = "0.1.0"
+perlmod = { version = "0.4.3", features = [ "exporter" ] }
diff --git a/perlmod/src/apt/mod.rs b/perlmod/src/apt/mod.rs
new file mode 100644
index 0000000..574c1a7
--- /dev/null
+++ b/perlmod/src/apt/mod.rs
@@ -0,0 +1 @@
+mod repositories;
diff --git a/perlmod/src/apt/repositories.rs b/perlmod/src/apt/repositories.rs
new file mode 100644
index 0000000..37a62f2
--- /dev/null
+++ b/perlmod/src/apt/repositories.rs
@@ -0,0 +1,55 @@
+#[perlmod::package(name = "PVE::RS::APT::Repositories", lib = "pve_rs")]
+mod export {
+    use anyhow::{bail, Error};
+
+    use perlmod::{to_value, Value};
+
+    #[export(raw_return)]
+    fn repositories() -> Result<(Value, Value, Value), Error> {
+        let (files, errors) = proxmox_apt::repositories::repositories()?;
+
+        if files.is_empty() {
+            bail!("no APT repository files could be parsed!");
+        }
+
+        let common_digest = proxmox_apt::repositories::common_digest(&files);
+
+        let hex_digest = proxmox::tools::digest_to_hex(&common_digest);
+
+        Ok((
+            to_value(&files)?,
+            to_value(&errors)?,
+            to_value(&hex_digest)?,
+        ))
+    }
+
+    #[export(raw_return)]
+    fn check_repositories(digest: String) -> Result<(Value, Value, Value), Error> {
+        let (files, _) = proxmox_apt::repositories::repositories()?;
+
+        if files.is_empty() {
+            bail!("no APT repository files could be parsed!");
+        }
+
+        if !digest.is_empty() {
+            let expected_digest = proxmox::tools::hex_to_digest(&digest)?;
+            let current_digest = proxmox_apt::repositories::common_digest(&files);
+            if current_digest != expected_digest {
+                bail!("detected modified configuration - file changed by other user? Try again.");
+            }
+        }
+
+        let infos = proxmox_apt::repositories::check_repositories(&files);
+
+        let enterprise_enabled =
+            proxmox_apt::repositories::enterprise_repository_enabled(&files, "pve");
+        let no_subscription_enabled =
+            proxmox_apt::repositories::no_subscription_repository_enabled(&files, "pve");
+
+        Ok((
+            to_value(&infos)?,
+            to_value(&enterprise_enabled)?,
+            to_value(&no_subscription_enabled)?,
+        ))
+    }
+}
diff --git a/perlmod/src/lib.rs b/perlmod/src/lib.rs
new file mode 100644
index 0000000..10b3376
--- /dev/null
+++ b/perlmod/src/lib.rs
@@ -0,0 +1 @@
+pub mod apt;
-- 
2.20.1

Re: [pve-devel] [pbs-devel] [RFC v5 pve-rs 16/23] add perlmod crate with initial APT module

[Wolfgang Bumiller]

Please don't use 'perlmod' as the name of a crate which also depends on
a different 'perlmod' crate, that won't be fun for long.

Call it pve-rs ;-)

On Fri, May 28, 2021 at 04:29:55PM +0200, Fabian Ebner wrote:
> and disable diskmanage for now, whose Cargo.toml references local paths.
> 
> Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
> ---
> 
> New in v5.
> 
>  Cargo.toml                      |  3 +-
>  perlmod/Cargo.toml              | 17 ++++++++++
>  perlmod/src/apt/mod.rs          |  1 +
>  perlmod/src/apt/repositories.rs | 55 +++++++++++++++++++++++++++++++++
>  perlmod/src/lib.rs              |  1 +
>  5 files changed, 76 insertions(+), 1 deletion(-)
>  create mode 100644 perlmod/Cargo.toml
>  create mode 100644 perlmod/src/apt/mod.rs
>  create mode 100644 perlmod/src/apt/repositories.rs
>  create mode 100644 perlmod/src/lib.rs

Re: [pve-devel] [pbs-devel] [RFC v5 pve-rs 16/23] add perlmod crate with initial APT module

[Fabian Ebner]

Am 31.05.21 um 14:55 schrieb Wolfgang Bumiller:
> Please don't use 'perlmod' as the name of a crate which also depends on
> a different 'perlmod' crate, that won't be fun for long.
> 
> Call it pve-rs ;-)
>

I called it perlmod, because that's what was suggested in the (outdated) 
README. Should it even be a sub-crate then, or organized like pmg-rs?

> On Fri, May 28, 2021 at 04:29:55PM +0200, Fabian Ebner wrote:
>> and disable diskmanage for now, whose Cargo.toml references local paths.
>>
>> Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
>> ---
>>
>> New in v5.
>>
>>   Cargo.toml                      |  3 +-
>>   perlmod/Cargo.toml              | 17 ++++++++++
>>   perlmod/src/apt/mod.rs          |  1 +
>>   perlmod/src/apt/repositories.rs | 55 +++++++++++++++++++++++++++++++++
>>   perlmod/src/lib.rs              |  1 +
>>   5 files changed, 76 insertions(+), 1 deletion(-)
>>   create mode 100644 perlmod/Cargo.toml
>>   create mode 100644 perlmod/src/apt/mod.rs
>>   create mode 100644 perlmod/src/apt/repositories.rs
>>   create mode 100644 perlmod/src/lib.rs

Re: [pve-devel] [pbs-devel] [RFC v5 pve-rs 16/23] add perlmod crate with initial APT module

[Wolfgang Bumiller]

On Fri, May 28, 2021 at 04:29:55PM +0200, Fabian Ebner wrote:
> and disable diskmanage for now, whose Cargo.toml references local paths.
> 
> Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
> ---
> 
> New in v5.
> 
>  Cargo.toml                      |  3 +-
>  perlmod/Cargo.toml              | 17 ++++++++++
>  perlmod/src/apt/mod.rs          |  1 +
>  perlmod/src/apt/repositories.rs | 55 +++++++++++++++++++++++++++++++++
>  perlmod/src/lib.rs              |  1 +
>  5 files changed, 76 insertions(+), 1 deletion(-)
>  create mode 100644 perlmod/Cargo.toml
>  create mode 100644 perlmod/src/apt/mod.rs
>  create mode 100644 perlmod/src/apt/repositories.rs
>  create mode 100644 perlmod/src/lib.rs
> 
> diff --git a/Cargo.toml b/Cargo.toml
> index d16a498..0ffd4a2 100644
> --- a/Cargo.toml
> +++ b/Cargo.toml
> @@ -1,4 +1,5 @@
>  [workspace]
>  members = [
> -    "diskmanage",
> +#    "diskmanage",
> +    "perlmod",
>  ]
> diff --git a/perlmod/Cargo.toml b/perlmod/Cargo.toml
> new file mode 100644
> index 0000000..397f444
> --- /dev/null
> +++ b/perlmod/Cargo.toml
> @@ -0,0 +1,17 @@
> +[package]
> +name = "pve-rs"
> +version = "0.1.0"
> +authors = ["Proxmox Support Team <support@proxmox.com>"]
> +edition = "2018"
> +license = "AGPL-3"
> +description = "Perl bindings for proxmox-apt"
> +homepage = "https://www.proxmox.com"
> +
> +[lib]
> +crate-type = [ "cdylib" ]
> +
> +[dependencies]
> +anyhow = "1.0"
> +proxmox = { version = "0.11.5" }
> +proxmox-apt = "0.1.0"
> +perlmod = { version = "0.4.3", features = [ "exporter" ] }
> diff --git a/perlmod/src/apt/mod.rs b/perlmod/src/apt/mod.rs
> new file mode 100644
> index 0000000..574c1a7
> --- /dev/null
> +++ b/perlmod/src/apt/mod.rs
> @@ -0,0 +1 @@
> +mod repositories;
> diff --git a/perlmod/src/apt/repositories.rs b/perlmod/src/apt/repositories.rs
> new file mode 100644
> index 0000000..37a62f2
> --- /dev/null
> +++ b/perlmod/src/apt/repositories.rs
> @@ -0,0 +1,55 @@
> +#[perlmod::package(name = "PVE::RS::APT::Repositories", lib = "pve_rs")]
> +mod export {
> +    use anyhow::{bail, Error};
> +
> +    use perlmod::{to_value, Value};
> +
> +    #[export(raw_return)]
> +    fn repositories() -> Result<(Value, Value, Value), Error> {

Please also add some documentation to exported methods, especially if
you skip writing out the return type.
Ideally we'd also add some kind of doc generation to perlmod in the
future which could in theory also link to the types from the foreign
crates you're returning here, so please assume that that's already the
case ;-)

Come to think of it, since we need a `mod` statement anyway, it probably
makes sense to use `pub fn` for exported functions (since the `mod`
itself is not-`pub`).
This also means we could utilize `#![deny(missing_docs)]` inside those
`mod`s ;-)

> +        let (files, errors) = proxmox_apt::repositories::repositories()?;
> +
> +        if files.is_empty() {
> +            bail!("no APT repository files could be parsed!");
> +        }
> +
> +        let common_digest = proxmox_apt::repositories::common_digest(&files);
> +
> +        let hex_digest = proxmox::tools::digest_to_hex(&common_digest);
> +
> +        Ok((
> +            to_value(&files)?,
> +            to_value(&errors)?,
> +            to_value(&hex_digest)?,
> +        ))
> +    }
> +
> +    #[export(raw_return)]
> +    fn check_repositories(digest: String) -> Result<(Value, Value, Value), Error> {

^ same