[pbs-devel] [PATCH backup 1/2] move subscription API path to /nodes

[pbs-devel] [PATCH backup 1/2] move subscription API path to /nodes

[Stefan Reiter]

This aligns it with PVE and allows the widget toolkit's update window
"refresh" to work without modifications once POST /apt/update is
implemented.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
---

I believe it makes more sense here in general, it is certainly node specific.

 src/api2.rs                         | 2 --
 src/api2/node.rs                    | 2 ++
 src/api2/{ => node}/subscription.rs | 8 ++++++++
 www/Dashboard.js                    | 2 +-
 www/Subscription.js                 | 2 +-
 5 files changed, 12 insertions(+), 4 deletions(-)
 rename src/api2/{ => node}/subscription.rs (88%)

diff --git a/src/api2.rs b/src/api2.rs
index 178f3f45..85d29ed2 100644
--- a/src/api2.rs
+++ b/src/api2.rs
@@ -4,7 +4,6 @@ pub mod backup;
 pub mod config;
 pub mod node;
 pub mod reader;
-mod subscription;
 pub mod status;
 pub mod types;
 pub mod version;
@@ -26,7 +25,6 @@ pub const SUBDIRS: SubdirMap = &[
     ("pull", &pull::ROUTER),
     ("reader", &reader::ROUTER),
     ("status", &status::ROUTER),
-    ("subscription", &subscription::ROUTER),
     ("version", &version::ROUTER),
 ];
 
diff --git a/src/api2/node.rs b/src/api2/node.rs
index 13ff282c..e67cab4e 100644
--- a/src/api2/node.rs
+++ b/src/api2/node.rs
@@ -9,6 +9,7 @@ mod syslog;
 mod journal;
 mod services;
 mod status;
+mod subscription;
 pub(crate) mod rrd;
 pub mod disks;
 
@@ -20,6 +21,7 @@ pub const SUBDIRS: SubdirMap = &[
     ("rrd", &rrd::ROUTER),
     ("services", &services::ROUTER),
     ("status", &status::ROUTER),
+    ("subscription", &subscription::ROUTER),
     ("syslog", &syslog::ROUTER),
     ("tasks", &tasks::ROUTER),
     ("time", &time::ROUTER),
diff --git a/src/api2/subscription.rs b/src/api2/node/subscription.rs
similarity index 88%
rename from src/api2/subscription.rs
rename to src/api2/node/subscription.rs
index 7d1b0237..186019cb 100644
--- a/src/api2/subscription.rs
+++ b/src/api2/node/subscription.rs
@@ -5,8 +5,16 @@ use proxmox::api::{api, Router, Permission};
 
 use crate::tools;
 use crate::config::acl::PRIV_SYS_AUDIT;
+use crate::api2::types::NODE_SCHEMA;
 
 #[api(
+    input: {
+        properties: {
+            node: {
+                schema: NODE_SCHEMA,
+            },
+        },
+    },
     returns: {
         description: "Subscription status.",
         properties: {
diff --git a/www/Dashboard.js b/www/Dashboard.js
index 7f699d93..d6dc40d2 100644
--- a/www/Dashboard.js
+++ b/www/Dashboard.js
@@ -209,7 +209,7 @@ Ext.define('PBS.Dashboard', {
 		autoDestroy: true,
 		proxy: {
 		    type: 'proxmox',
-		    url: '/api2/json/subscription'
+		    url: '/api2/json/nodes/localhost/subscription'
 		},
 		listeners: {
 		    load: 'updateSubscription'
diff --git a/www/Subscription.js b/www/Subscription.js
index f5a64459..8746091f 100644
--- a/www/Subscription.js
+++ b/www/Subscription.js
@@ -37,7 +37,7 @@ Ext.define('PBS.Subscription', {
 	    me.rstore.load();
 	};
 
-	var baseurl = '/subscription';
+	var baseurl = '/nodes/localhost/subscription';
 
 	var render_status = function(value) {
 
-- 
2.20.1

[pbs-devel] [PATCH v2 backup 2/2] add .../apt/update API call

[Stefan Reiter]

Depends on patched apt-pkg-native-rs. Changelog-URL detection is
inspired by PVE perl code for now, though marked with fixme to use 'apt
changelog' later on, if/when our repos have APT-compatible changelogs
set up.

list_installed_apt_packages iterates all packages and creates an
APTUpdateInfo with detailed information for every package matched by the
given filter Fn.

Sadly, libapt-pkg has some questionable design choices regarding their
use of 'iterators', which means quite a bit of nesting...

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
---

v2:
* Include feedback from Fabian:
** Update Cargo.toml with now packaged apt-pkg-native (still no update on my PR)
** Use proxmox.com changelog logic for all origin=Proxmox packages, not just pbs
** Change serde naming to PascalCase for APTUpdateInfo
** Add FIXME to changelog detection
** Update comments


 Cargo.toml           |   1 +
 src/api2/node.rs     |   2 +
 src/api2/node/apt.rs | 211 +++++++++++++++++++++++++++++++++++++++++++
 src/api2/types.rs    |  27 ++++++
 4 files changed, 241 insertions(+)
 create mode 100644 src/api2/node/apt.rs

diff --git a/Cargo.toml b/Cargo.toml
index 355217eb..b0881319 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -14,6 +14,7 @@ name = "proxmox_backup"
 path = "src/lib.rs"
 
 [dependencies]
+apt-pkg-native = "0.3.1" # custom patched version
 base64 = "0.12"
 bitflags = "1.2.1"
 bytes = "0.5"
diff --git a/src/api2/node.rs b/src/api2/node.rs
index e67cab4e..e8800e4c 100644
--- a/src/api2/node.rs
+++ b/src/api2/node.rs
@@ -12,8 +12,10 @@ mod status;
 mod subscription;
 pub(crate) mod rrd;
 pub mod disks;
+mod apt;
 
 pub const SUBDIRS: SubdirMap = &[
+    ("apt", &apt::ROUTER),
     ("disks", &disks::ROUTER),
     ("dns", &dns::ROUTER),
     ("journal", &journal::ROUTER),
diff --git a/src/api2/node/apt.rs b/src/api2/node/apt.rs
new file mode 100644
index 00000000..55d360fe
--- /dev/null
+++ b/src/api2/node/apt.rs
@@ -0,0 +1,211 @@
+use apt_pkg_native::Cache;
+use anyhow::{Error, bail};
+use serde_json::{json, Value};
+
+use proxmox::{list_subdirs_api_method, const_regex};
+use proxmox::api::{api, Router, Permission, SubdirMap};
+
+use crate::config::acl::PRIV_SYS_AUDIT;
+use crate::api2::types::{APTUpdateInfo, NODE_SCHEMA};
+
+const_regex! {
+    VERSION_EPOCH_REGEX = r"^\d+:";
+    FILENAME_EXTRACT_REGEX = r"^.*/.*?_(.*)_Packages$";
+}
+
+// FIXME: Replace with call to 'apt changelog <pkg> --print-uris'. Currently
+// not possible as our packages do not have a URI set in their Release file
+fn get_changelog_url(
+    package: &str,
+    filename: &str,
+    source_pkg: &str,
+    version: &str,
+    source_version: &str,
+    origin: &str,
+    component: &str,
+) -> Result<String, Error> {
+    if origin == "" {
+        bail!("no origin available for package {}", package);
+    }
+
+    if origin == "Debian" {
+        let source_version = (VERSION_EPOCH_REGEX.regex_obj)().replace_all(source_version, "");
+
+        let prefix = if source_pkg.starts_with("lib") {
+            source_pkg.get(0..4)
+        } else {
+            source_pkg.get(0..1)
+        };
+
+        let prefix = match prefix {
+            Some(p) => p,
+            None => bail!("cannot get starting characters of package name '{}'", package)
+        };
+
+        // note: security updates seem to not always upload a changelog for
+        // their package version, so this only works *most* of the time
+        return Ok(format!("https://metadata.ftp-master.debian.org/changelogs/main/{}/{}/{}_{}_changelog",
+                          prefix, source_pkg, source_pkg, source_version));
+
+    } else if origin == "Proxmox" {
+        let version = (VERSION_EPOCH_REGEX.regex_obj)().replace_all(version, "");
+
+        let base = match (FILENAME_EXTRACT_REGEX.regex_obj)().captures(filename) {
+            Some(captures) => {
+                let base_capture = captures.get(1);
+                match base_capture {
+                    Some(base_underscore) => base_underscore.as_str().replace("_", "/"),
+                    None => bail!("incompatible filename, cannot find regex group")
+                }
+            },
+            None => bail!("incompatible filename, doesn't match regex")
+        };
+
+        return Ok(format!("http://download.proxmox.com/{}/{}_{}.changelog",
+                          base, package, version));
+    }
+
+    bail!("unknown origin ({}) or component ({})", origin, component)
+}
+
+fn list_installed_apt_packages<F: Fn(&str, &str, &str) -> bool>(filter: F)
+    -> Vec<APTUpdateInfo> {
+
+    let mut ret = Vec::new();
+
+    // note: this is not an 'apt update', it just re-reads the cache from disk
+    let mut cache = Cache::get_singleton();
+    cache.reload();
+
+    let mut cache_iter = cache.iter();
+
+    loop {
+        let view = match cache_iter.next() {
+            Some(view) => view,
+            None => break
+        };
+
+        let current_version = match view.current_version() {
+            Some(vers) => vers,
+            None => continue
+        };
+        let candidate_version = match view.candidate_version() {
+            Some(vers) => vers,
+            // if there's no candidate (i.e. no update) get info of currently
+            // installed version instead
+            None => current_version.clone()
+        };
+
+        let package = view.name();
+        if filter(&package, &current_version, &candidate_version) {
+            let mut origin_res = "unknown".to_owned();
+            let mut section_res = "unknown".to_owned();
+            let mut priority_res = "unknown".to_owned();
+            let mut change_log_url = "".to_owned();
+            let mut short_desc = package.clone();
+            let mut long_desc = "".to_owned();
+
+            // get additional information via nested APT 'iterators'
+            let mut view_iter = view.versions();
+            while let Some(ver) = view_iter.next() {
+                if ver.version() == candidate_version {
+                    if let Some(section) = ver.section() {
+                        section_res = section;
+                    }
+
+                    if let Some(prio) = ver.priority_type() {
+                        priority_res = prio;
+                    }
+
+                    // assume every package has only one origin file (not
+                    // origin, but origin *file*, for some reason those seem to
+                    // be different concepts in APT)
+                    let mut origin_iter = ver.origin_iter();
+                    let origin = origin_iter.next();
+                    if let Some(origin) = origin {
+
+                        if let Some(sd) = origin.short_desc() {
+                            short_desc = sd;
+                        }
+
+                        if let Some(ld) = origin.long_desc() {
+                            long_desc = ld;
+                        }
+
+                        // the package files appear in priority order, meaning
+                        // the one for the candidate version is first
+                        let mut pkg_iter = origin.file();
+                        let pkg_file = pkg_iter.next();
+                        if let Some(pkg_file) = pkg_file {
+                            if let Some(origin_name) = pkg_file.origin() {
+                                origin_res = origin_name;
+                            }
+
+                            let filename = pkg_file.file_name();
+                            let source_pkg = ver.source_package();
+                            let source_ver = ver.source_version();
+                            let component = pkg_file.component();
+
+                            // build changelog URL from gathered information
+                            // ignore errors, use empty changelog instead
+                            let url = get_changelog_url(&package, &filename, &source_pkg,
+                                &candidate_version, &source_ver, &origin_res, &component);
+                            if let Ok(url) = url {
+                                change_log_url = url;
+                            }
+                        }
+                    }
+
+                    break;
+                }
+            }
+
+            let info = APTUpdateInfo {
+                package,
+                title: short_desc,
+                arch: view.arch(),
+                description: long_desc,
+                change_log_url,
+                origin: origin_res,
+                version: candidate_version,
+                old_version: current_version,
+                priority: priority_res,
+                section: section_res,
+            };
+            ret.push(info);
+        }
+    }
+
+    return ret;
+}
+
+#[api(
+    input: {
+        properties: {
+            node: {
+                schema: NODE_SCHEMA,
+            },
+        },
+    },
+    returns: {
+        description: "A list of packages with available updates.",
+        type: Array,
+        items: { type: APTUpdateInfo },
+    },
+    access: {
+        permission: &Permission::Privilege(&[], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// List available APT updates
+fn apt_update_available(_param: Value) -> Result<Value, Error> {
+    let ret = list_installed_apt_packages(|_pkg, cur_ver, can_ver| cur_ver != can_ver);
+    Ok(json!(ret))
+}
+
+const SUBDIRS: SubdirMap = &[
+    ("update", &Router::new().get(&API_METHOD_APT_UPDATE_AVAILABLE)),
+];
+
+pub const ROUTER: Router = Router::new()
+    .get(&list_subdirs_api_method!(SUBDIRS))
+    .subdirs(SUBDIRS);
diff --git a/src/api2/types.rs b/src/api2/types.rs
index 0d0fab3b..f6972c8b 100644
--- a/src/api2/types.rs
+++ b/src/api2/types.rs
@@ -962,3 +962,30 @@ pub enum RRDTimeFrameResolution {
     /// 1 week => last 490 days
     Year = 60*10080,
 }
+
+#[api()]
+#[derive(Serialize, Deserialize)]
+#[serde(rename_all = "PascalCase")]
+/// Describes a package for which an update is available.
+pub struct APTUpdateInfo {
+    /// Package name
+    pub package: String,
+    /// Package title
+    pub title: String,
+    /// Package architecture
+    pub arch: String,
+    /// Human readable package description
+    pub description: String,
+    /// New version to be updated to
+    pub version: String,
+    /// Old version currently installed
+    pub old_version: String,
+    /// Package origin
+    pub origin: String,
+    /// Package priority in human-readable form
+    pub priority: String,
+    /// Package section
+    pub section: String,
+    /// URL under which the package's changelog can be retrieved
+    pub change_log_url: String,
+}
-- 
2.20.1

[pbs-devel] applied: [PATCH backup 1/2] move subscription API path to /nodes

[Thomas Lamprecht]

On 21.07.20 13:41, Stefan Reiter wrote:
> This aligns it with PVE and allows the widget toolkit's update window
> "refresh" to work without modifications once POST /apt/update is
> implemented.
> 
> Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
> ---
> 
> I believe it makes more sense here in general, it is certainly node specific.
> 
>  src/api2.rs                         | 2 --
>  src/api2/node.rs                    | 2 ++
>  src/api2/{ => node}/subscription.rs | 8 ++++++++
>  www/Dashboard.js                    | 2 +-
>  www/Subscription.js                 | 2 +-
>  5 files changed, 12 insertions(+), 4 deletions(-)
>  rename src/api2/{ => node}/subscription.rs (88%)
> 
>

applied, thanks!

[pbs-devel] applied: [PATCH v2 backup 2/2] add .../apt/update API call

[Thomas Lamprecht]

On 21.07.20 13:41, Stefan Reiter wrote:
> Depends on patched apt-pkg-native-rs. Changelog-URL detection is
> inspired by PVE perl code for now, though marked with fixme to use 'apt
> changelog' later on, if/when our repos have APT-compatible changelogs
> set up.
> 
> list_installed_apt_packages iterates all packages and creates an
> APTUpdateInfo with detailed information for every package matched by the
> given filter Fn.
> 
> Sadly, libapt-pkg has some questionable design choices regarding their
> use of 'iterators', which means quite a bit of nesting...
> 
> Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
> ---
> 
> v2:
> * Include feedback from Fabian:
> ** Update Cargo.toml with now packaged apt-pkg-native (still no update on my PR)
> ** Use proxmox.com changelog logic for all origin=Proxmox packages, not just pbs
> ** Change serde naming to PascalCase for APTUpdateInfo
> ** Add FIXME to changelog detection
> ** Update comments
> 
> 
>  Cargo.toml           |   1 +
>  src/api2/node.rs     |   2 +
>  src/api2/node/apt.rs | 211 +++++++++++++++++++++++++++++++++++++++++++
>  src/api2/types.rs    |  27 ++++++
>  4 files changed, 241 insertions(+)
>  create mode 100644 src/api2/node/apt.rs
> 
>

applied, thanks! Adding additionally the POST call on it for `apt-get update` wouldn't
have been really more work ;)