* [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI
@ 2021-04-02 11:20 Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 01/10] initial commit Fabian Ebner
` (10 more replies)
0 siblings, 11 replies; 13+ messages in thread
From: Fabian Ebner @ 2021-04-02 11:20 UTC (permalink / raw)
To: pbs-devel
List the configured repositories and have some basic checks for them.
The plan is to use perlmod to make the Rust implementation available for PVE+PMG
as well.
Changes from v3:
* incorporate Fabian G.'s feedback:
* switch to a per-file approach
* check for official host names
* fix case-sensitivity issue for .sources keys
* include digests
* fix write issue when there are no components (in case of an absolute suite)
* add more tests
Still missing (intended as followups):
* Upgrade suite/distribuiton button to be used before major release
upgrades (but it's really simply to add that now).
* perlmod magic and integration in PVE and PMG.
Changes v2 -> v3:
* incorporate Wolfgang's feedback
* improve main warning's UI
Changes v1 -> v2:
* Perl -> Rust
* PVE -> PBS
* Don't rely on regexes for parsing.
* Add writer and tests.
* UI: pin warnings to the repository they're for.
* Keep order of options consistent with configuration.
* Smaller things noted on the individual patches.
proxmox-apt:
Fabian Ebner (4):
initial commit
add files for Debian packaging
add functions to check for Proxmox repositories
add check_repositories function
proxmox-backup:
Fabian Ebner (4):
depend on new proxmox-apt crate
api: apt: add repositories call
ui: add panel for APT repositories
api: apt: add check_repositories_call
Cargo.toml | 1 +
debian/control | 1 +
src/api2/node/apt.rs | 149 +++++++++++++++++++++++++++++++++++-
www/ServerAdministration.js | 8 ++
4 files changed, 158 insertions(+), 1 deletion(-)
proxmox-widget-toolkit:
Fabian Ebner (2):
add UI for APT repositories
APT repositories: add warnings
src/Makefile | 1 +
src/node/APTRepositories.js | 415 ++++++++++++++++++++++++++++++++++++
2 files changed, 416 insertions(+)
create mode 100644 src/node/APTRepositories.js
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* [pbs-devel] [PATCH v4 proxmox-apt 01/10] initial commit
2021-04-02 11:20 [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
@ 2021-04-02 11:20 ` Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 02/10] add files for Debian packaging Fabian Ebner
` (9 subsequent siblings)
10 siblings, 0 replies; 13+ messages in thread
From: Fabian Ebner @ 2021-04-02 11:20 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v3:
Switching to a per-file approach as Fabian G. suggested led to what's
essentially a re-write of all the handling in repositories/mod.rs. There's now
an APTRepositoryFile struct that is used for name-check/read/write/parse, and
other functions take slices of the new struct. The new struct is also where the
digest from the raw contents is stored. I removed the path and number from the
APTRepository struct, because this information can/should be extracted from the
new file-based struct. The parsers also don't need the path anymore, for errors
it's added in when bubbling up.
Other changes:
* also recognize lower-case keys for types, uris, components, suites
and add test (again thanks to Fabian G.'s feedback)
* add tests for += -= options and absolute suites
* don't write components for .sources files when there are none (in case of
an absolute suite).
* show diff command for test failure for quick comparsion
* remove superfluous comment.clear() after mem::take calls (forgot this last
time)
* have parse_repositories return its result again as opposed to taking a
vector, because it's more natural with the per-file approach
.cargo/config | 5 +
.gitignore | 4 +
Cargo.toml | 23 ++
rustfmt.toml | 1 +
src/lib.rs | 3 +
src/repositories/check.rs | 47 ++++
src/repositories/file.rs | 96 +++++++
src/repositories/list_parser.rs | 171 ++++++++++++
src/repositories/mod.rs | 224 ++++++++++++++++
src/repositories/sources_parser.rs | 204 +++++++++++++++
src/repositories/writer.rs | 92 +++++++
src/types.rs | 246 ++++++++++++++++++
tests/repositories.rs | 131 ++++++++++
.../absolute_suite.list | 5 +
.../absolute_suite.sources | 5 +
tests/sources.list.d.expected/case.sources | 16 ++
.../sources.list.d.expected/multiline.sources | 10 +
| 6 +
.../pbs-enterprise.list | 2 +
tests/sources.list.d.expected/pve.list | 13 +
tests/sources.list.d.expected/standard.list | 7 +
.../sources.list.d.expected/standard.sources | 11 +
tests/sources.list.d/absolute_suite.list | 3 +
tests/sources.list.d/absolute_suite.sources | 5 +
tests/sources.list.d/case.sources | 17 ++
tests/sources.list.d/multiline.sources | 11 +
| 3 +
tests/sources.list.d/pbs-enterprise.list | 1 +
tests/sources.list.d/pve.list | 10 +
tests/sources.list.d/standard.list | 6 +
tests/sources.list.d/standard.sources | 10 +
31 files changed, 1388 insertions(+)
create mode 100644 .cargo/config
create mode 100644 .gitignore
create mode 100644 Cargo.toml
create mode 100644 rustfmt.toml
create mode 100644 src/lib.rs
create mode 100644 src/repositories/check.rs
create mode 100644 src/repositories/file.rs
create mode 100644 src/repositories/list_parser.rs
create mode 100644 src/repositories/mod.rs
create mode 100644 src/repositories/sources_parser.rs
create mode 100644 src/repositories/writer.rs
create mode 100644 src/types.rs
create mode 100644 tests/repositories.rs
create mode 100644 tests/sources.list.d.expected/absolute_suite.list
create mode 100644 tests/sources.list.d.expected/absolute_suite.sources
create mode 100644 tests/sources.list.d.expected/case.sources
create mode 100644 tests/sources.list.d.expected/multiline.sources
create mode 100644 tests/sources.list.d.expected/options_comment.list
create mode 100644 tests/sources.list.d.expected/pbs-enterprise.list
create mode 100644 tests/sources.list.d.expected/pve.list
create mode 100644 tests/sources.list.d.expected/standard.list
create mode 100644 tests/sources.list.d.expected/standard.sources
create mode 100644 tests/sources.list.d/absolute_suite.list
create mode 100644 tests/sources.list.d/absolute_suite.sources
create mode 100644 tests/sources.list.d/case.sources
create mode 100644 tests/sources.list.d/multiline.sources
create mode 100644 tests/sources.list.d/options_comment.list
create mode 100644 tests/sources.list.d/pbs-enterprise.list
create mode 100644 tests/sources.list.d/pve.list
create mode 100644 tests/sources.list.d/standard.list
create mode 100644 tests/sources.list.d/standard.sources
diff --git a/.cargo/config b/.cargo/config
new file mode 100644
index 0000000..3b5b6e4
--- /dev/null
+++ b/.cargo/config
@@ -0,0 +1,5 @@
+[source]
+[source.debian-packages]
+directory = "/usr/share/cargo/registry"
+[source.crates-io]
+replace-with = "debian-packages"
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..24917d4
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,4 @@
+Cargo.lock
+target/
+tests/sources.list.d.actual
+tests/sources.list.d.digest
diff --git a/Cargo.toml b/Cargo.toml
new file mode 100644
index 0000000..9350e3b
--- /dev/null
+++ b/Cargo.toml
@@ -0,0 +1,23 @@
+[package]
+name = "proxmox-apt"
+version = "0.1.0"
+authors = [
+ "Fabian Ebner <f.ebner@proxmox.com>",
+ "Proxmox Support Team <support@proxmox.com>",
+]
+edition = "2018"
+license = "AGPL-3"
+description = "Proxmox library for APT"
+homepage = "https://www.proxmox.com"
+
+exclude = [ "debian" ]
+
+[lib]
+name = "proxmox_apt"
+path = "src/lib.rs"
+
+[dependencies]
+anyhow = "1.0"
+openssl = "0.10"
+proxmox = { version = "0.11.0", features = [ "api-macro" ] }
+serde = { version = "1.0", features = ["derive"] }
diff --git a/rustfmt.toml b/rustfmt.toml
new file mode 100644
index 0000000..32a9786
--- /dev/null
+++ b/rustfmt.toml
@@ -0,0 +1 @@
+edition = "2018"
diff --git a/src/lib.rs b/src/lib.rs
new file mode 100644
index 0000000..b065c0f
--- /dev/null
+++ b/src/lib.rs
@@ -0,0 +1,3 @@
+pub mod types;
+
+pub mod repositories;
diff --git a/src/repositories/check.rs b/src/repositories/check.rs
new file mode 100644
index 0000000..87fbbac
--- /dev/null
+++ b/src/repositories/check.rs
@@ -0,0 +1,47 @@
+use anyhow::{bail, Error};
+
+use crate::types::{APTRepository, APTRepositoryFileType};
+
+impl APTRepository {
+ /// Makes sure that all basic properties of a repository are present and
+ /// not obviously invalid.
+ pub fn basic_check(&self) -> Result<(), Error> {
+ if self.types.is_empty() {
+ bail!("missing package type(s)");
+ }
+ if self.uris.is_empty() {
+ bail!("missing URI(s)");
+ }
+ if self.suites.is_empty() {
+ bail!("missing suite(s)");
+ }
+
+ for uri in self.uris.iter() {
+ if !uri.contains(':') || uri.len() < 3 {
+ bail!("invalid URI: '{}'", uri);
+ }
+ }
+
+ for suite in self.suites.iter() {
+ if !suite.ends_with('/') && self.components.is_empty() {
+ bail!("missing component(s)");
+ } else if suite.ends_with('/') && !self.components.is_empty() {
+ bail!("absolute suite '{}' does not allow component(s)", suite);
+ }
+ }
+
+ if self.file_type == APTRepositoryFileType::List {
+ if self.types.len() > 1 {
+ bail!("more than one package type");
+ }
+ if self.uris.len() > 1 {
+ bail!("more than one URI");
+ }
+ if self.suites.len() > 1 {
+ bail!("more than one suite");
+ }
+ }
+
+ Ok(())
+ }
+}
diff --git a/src/repositories/file.rs b/src/repositories/file.rs
new file mode 100644
index 0000000..e264ec6
--- /dev/null
+++ b/src/repositories/file.rs
@@ -0,0 +1,96 @@
+use std::convert::TryFrom;
+use std::path::{Path, PathBuf};
+
+use anyhow::{format_err, Error};
+
+use crate::types::{APTRepositoryFile, APTRepositoryFileError, APTRepositoryFileType};
+
+impl APTRepositoryFile {
+ /// Creates a new `APTRepositoryFile` without parsing.
+ ///
+ /// If the file is hidden or the path points to a directory, `Ok(None)` is
+ /// returned, while invalid file names yield an error.
+ pub fn new<P: AsRef<Path>>(path: P) -> Result<Option<Self>, APTRepositoryFileError> {
+ let path: PathBuf = path.as_ref().to_path_buf();
+
+ let new_err = |path_string: String, err: &str| APTRepositoryFileError {
+ path: path_string,
+ error: err.to_string(),
+ };
+
+ let path_string = path
+ .clone()
+ .into_os_string()
+ .into_string()
+ .map_err(|os_string| {
+ new_err(
+ os_string.to_string_lossy().to_string(),
+ "path is not valid unicode",
+ )
+ })?;
+
+ let new_err = |err| new_err(path_string.clone(), err);
+
+ if path.is_dir() {
+ return Ok(None);
+ }
+
+ let file_name = match path.file_name() {
+ Some(file_name) => file_name
+ .to_os_string()
+ .into_string()
+ .map_err(|_| new_err("invalid path"))?,
+ None => return Err(new_err("invalid path")),
+ };
+
+ if file_name.starts_with('.') {
+ return Ok(None);
+ }
+
+ let extension = match path.extension() {
+ Some(extension) => extension
+ .to_os_string()
+ .into_string()
+ .map_err(|_| new_err("invalid path"))?,
+ None => return Err(new_err("invalid extension")),
+ };
+
+ let file_type = APTRepositoryFileType::try_from(&extension[..])
+ .map_err(|_| new_err("invalid extension"))?;
+
+ if !file_name
+ .chars()
+ .all(|x| x.is_ascii_alphanumeric() || x == '_' || x == '-' || x == '.')
+ {
+ return Err(new_err("invalid characters in file name"));
+ }
+
+ Ok(Some(Self {
+ path: path_string,
+ file_type,
+ repositories: vec![],
+ digest: None,
+ }))
+ }
+
+ /// Check if the file exists.
+ pub fn exists(&self) -> bool {
+ PathBuf::from(&self.path).exists()
+ }
+
+ pub fn read_with_digest(&self) -> Result<(Vec<u8>, [u8; 32]), APTRepositoryFileError> {
+ let content = std::fs::read(&self.path).map_err(|err| self.err(format_err!("{}", err)))?;
+
+ let digest = openssl::sha::sha256(&content);
+
+ Ok((content, digest))
+ }
+
+ /// Create an `APTRepositoryFileError`.
+ pub fn err(&self, error: Error) -> APTRepositoryFileError {
+ APTRepositoryFileError {
+ path: self.path.clone(),
+ error: error.to_string(),
+ }
+ }
+}
diff --git a/src/repositories/list_parser.rs b/src/repositories/list_parser.rs
new file mode 100644
index 0000000..6c9f898
--- /dev/null
+++ b/src/repositories/list_parser.rs
@@ -0,0 +1,171 @@
+use std::convert::TryInto;
+use std::io::BufRead;
+use std::iter::{Iterator, Peekable};
+use std::str::SplitAsciiWhitespace;
+
+use anyhow::{bail, format_err, Error};
+
+use super::APTRepositoryParser;
+use crate::types::{APTRepository, APTRepositoryFileType, APTRepositoryOption};
+
+pub struct APTListFileParser<R: BufRead> {
+ input: R,
+ line_nr: usize,
+ comment: String,
+}
+
+impl<R: BufRead> APTListFileParser<R> {
+ pub fn new(reader: R) -> Self {
+ Self {
+ input: reader,
+ line_nr: 0,
+ comment: String::new(),
+ }
+ }
+
+ /// Helper to parse options from the existing token stream.
+ ///
+ /// Also returns `Ok(())` if there are no options.
+ ///
+ /// Errors when options are invalid or not closed by `']'`.
+ fn parse_options(
+ options: &mut Vec<APTRepositoryOption>,
+ tokens: &mut Peekable<SplitAsciiWhitespace>,
+ ) -> Result<(), Error> {
+ let mut option = match tokens.peek() {
+ Some(token) => {
+ match token.strip_prefix('[') {
+ Some(option) => option,
+ None => return Ok(()), // doesn't look like options
+ }
+ }
+ None => return Ok(()),
+ };
+
+ tokens.next(); // avoid reading the beginning twice
+
+ let mut finished = false;
+ loop {
+ if let Some(stripped) = option.strip_suffix(']') {
+ option = stripped;
+ if option.is_empty() {
+ break;
+ }
+ finished = true; // but still need to handle the last one
+ };
+
+ if let Some(mid) = option.find('=') {
+ let (key, mut value_str) = option.split_at(mid);
+ value_str = &value_str[1..];
+
+ if key.is_empty() {
+ bail!("option has no key: '{}'", option);
+ }
+
+ if value_str.is_empty() {
+ bail!("option has no value: '{}'", option);
+ }
+
+ let values: Vec<String> = value_str
+ .split(',')
+ .map(|value| value.to_string())
+ .collect();
+
+ options.push(APTRepositoryOption {
+ key: key.to_string(),
+ values,
+ });
+ } else if !option.is_empty() {
+ bail!("got invalid option - '{}'", option);
+ }
+
+ if finished {
+ break;
+ }
+
+ option = match tokens.next() {
+ Some(option) => option,
+ None => bail!("options not closed by ']'"),
+ }
+ }
+
+ Ok(())
+ }
+
+ /// Parse a repository or comment in one-line format.
+ ///
+ /// Commented out repositories are also detected and returned with the
+ /// `enabled` property set to `false`.
+ ///
+ /// If the line contains a repository, `self.comment` is added to the
+ /// `comment` property.
+ ///
+ /// If the line contains a comment, it is added to `self.comment`.
+ fn parse_one_line(&mut self, mut line: &str) -> Result<Option<APTRepository>, Error> {
+ line = line.trim_matches(|c| char::is_ascii_whitespace(&c));
+
+ // check for commented out repository first
+ if let Some(commented_out) = line.strip_prefix('#') {
+ if let Ok(Some(mut repo)) = self.parse_one_line(commented_out) {
+ repo.set_enabled(false);
+ return Ok(Some(repo));
+ }
+ }
+
+ let mut repo = APTRepository::new(APTRepositoryFileType::List);
+
+ // now handle "real" comment
+ if let Some(comment_start) = line.find('#') {
+ let (line_start, comment) = line.split_at(comment_start);
+ self.comment = format!("{}{}\n", self.comment, &comment[1..]);
+ line = line_start;
+ }
+
+ let mut tokens = line.split_ascii_whitespace().peekable();
+
+ match tokens.next() {
+ Some(package_type) => {
+ repo.types.push(package_type.try_into()?);
+ }
+ None => return Ok(None), // empty line
+ }
+
+ Self::parse_options(&mut repo.options, &mut tokens)?;
+
+ // the rest of the line is just '<uri> <suite> [<components>...]'
+ let mut tokens = tokens.map(str::to_string);
+ repo.uris
+ .push(tokens.next().ok_or_else(|| format_err!("missing URI"))?);
+ repo.suites
+ .push(tokens.next().ok_or_else(|| format_err!("missing suite"))?);
+ repo.components.extend(tokens);
+
+ repo.comment = std::mem::take(&mut self.comment);
+
+ Ok(Some(repo))
+ }
+}
+
+impl<R: BufRead> APTRepositoryParser for APTListFileParser<R> {
+ fn parse_repositories(&mut self) -> Result<Vec<APTRepository>, Error> {
+ let mut repos = vec![];
+ let mut line = String::new();
+
+ loop {
+ self.line_nr += 1;
+ line.clear();
+
+ match self.input.read_line(&mut line) {
+ Err(err) => bail!("input error - {}", err),
+ Ok(0) => break,
+ Ok(_) => match self.parse_one_line(&line) {
+ Ok(Some(repo)) => repos.push(repo),
+ Ok(None) => continue,
+ Err(err) => bail!("malformed entry on line {} - {}", self.line_nr, err),
+ },
+ }
+ }
+
+ Ok(repos)
+ }
+}
diff --git a/src/repositories/mod.rs b/src/repositories/mod.rs
new file mode 100644
index 0000000..187ead3
--- /dev/null
+++ b/src/repositories/mod.rs
@@ -0,0 +1,224 @@
+use std::path::PathBuf;
+
+use anyhow::{bail, format_err, Error};
+
+use crate::types::{
+ APTRepository, APTRepositoryFile, APTRepositoryFileError, APTRepositoryFileType,
+ APTRepositoryOption,
+};
+
+mod list_parser;
+use list_parser::APTListFileParser;
+
+mod sources_parser;
+use sources_parser::APTSourcesFileParser;
+
+mod check;
+mod file;
+mod writer;
+
+const APT_SOURCES_LIST_FILENAME: &str = "/etc/apt/sources.list";
+const APT_SOURCES_LIST_DIRECTORY: &str = "/etc/apt/sources.list.d/";
+
+impl APTRepository {
+ /// Crates an empty repository.
+ fn new(file_type: APTRepositoryFileType) -> Self {
+ Self {
+ types: vec![],
+ uris: vec![],
+ suites: vec![],
+ components: vec![],
+ options: vec![],
+ comment: String::new(),
+ file_type,
+ enabled: true,
+ }
+ }
+
+ /// Changes the `enabled` flag and makes sure the `Enabled` option for
+ /// `APTRepositoryPackageType::Sources` repositories is updated too.
+ fn set_enabled(&mut self, enabled: bool) {
+ self.enabled = enabled;
+
+ if self.file_type == APTRepositoryFileType::Sources {
+ let enabled_string = match enabled {
+ true => "true".to_string(),
+ false => "false".to_string(),
+ };
+ for option in self.options.iter_mut() {
+ if option.key == "Enabled" {
+ option.values = vec![enabled_string];
+ return;
+ }
+ }
+ self.options.push(APTRepositoryOption {
+ key: "Enabled".to_string(),
+ values: vec![enabled_string],
+ });
+ }
+ }
+}
+
+trait APTRepositoryParser {
+ /// Parse all repositories including the disabled ones and push them onto
+ /// the provided vector.
+ fn parse_repositories(&mut self) -> Result<Vec<APTRepository>, Error>;
+}
+
+impl APTRepositoryFile {
+ /// Parses the APT repositories configured in the file on disk, including
+ /// disabled ones.
+ ///
+ /// Resets the current repositories and digest, even on failure.
+ pub fn parse(&mut self) -> Result<(), APTRepositoryFileError> {
+ self.repositories.clear();
+ self.digest = None;
+
+ let (content, digest) = self.read_with_digest()?;
+
+ let mut parser: Box<dyn APTRepositoryParser> = match self.file_type {
+ APTRepositoryFileType::List => Box::new(APTListFileParser::new(&content[..])),
+ APTRepositoryFileType::Sources => Box::new(APTSourcesFileParser::new(&content[..])),
+ };
+
+ let repos = parser.parse_repositories().map_err(|err| self.err(err))?;
+
+ for (n, repo) in repos.iter().enumerate() {
+ repo.basic_check()
+ .map_err(|err| self.err(format_err!("check for repository {} - {}", n + 1, err)))?;
+ }
+
+ self.repositories = repos;
+ self.digest = Some(digest);
+
+ Ok(())
+ }
+
+ /// Writes the repositories to the file on disk.
+ ///
+ /// If a digest is provided, checks that the current content of the file still
+ /// produces the same one.
+ pub fn write(&self) -> Result<(), APTRepositoryFileError> {
+ if let Some(digest) = self.digest {
+ if !self.exists() {
+ return Err(self.err(format_err!("digest specified, but file does not exist")));
+ }
+
+ let (_, current_digest) = self.read_with_digest()?;
+ if digest != current_digest {
+ return Err(self.err(format_err!("digest mismatch")));
+ }
+ }
+
+ let mut content = vec![];
+
+ for (n, repo) in self.repositories.iter().enumerate() {
+ repo.basic_check()
+ .map_err(|err| self.err(format_err!("check for repository {} - {}", n + 1, err)))?;
+
+ repo.write(&mut content)
+ .map_err(|err| self.err(format_err!("writing repository {} - {}", n + 1, err)))?;
+ }
+
+ let path = PathBuf::from(&self.path);
+ let dir = match path.parent() {
+ Some(dir) => dir,
+ None => return Err(self.err(format_err!("invalid path"))),
+ };
+
+ std::fs::create_dir_all(dir)
+ .map_err(|err| self.err(format_err!("unable to create parent dir - {}", err)))?;
+
+ let pid = std::process::id();
+ let mut tmp_path = path.clone();
+ tmp_path.set_extension("tmp");
+ tmp_path.set_extension(format!("{}", pid));
+
+ if let Err(err) = std::fs::write(&tmp_path, content) {
+ let _ = std::fs::remove_file(&tmp_path);
+ return Err(self.err(format_err!("writing {:?} failed - {}", path, err)));
+ }
+
+ if let Err(err) = std::fs::rename(&tmp_path, &path) {
+ let _ = std::fs::remove_file(&tmp_path);
+ return Err(self.err(format_err!("rename failed for {:?} - {}", path, err)));
+ }
+
+ Ok(())
+ }
+}
+
+/// Returns all APT repositories configured in `/etc/apt/sources.list` and
+/// in `/etc/apt/sources.list.d` including disabled repositories.
+///
+/// Returns the parsable files with their repositories and a list of errors for
+/// files that could not be read or parsed.
+///
+/// The digest is guaranteed to be set for each successfully parsed file.
+pub fn repositories() -> Result<(Vec<APTRepositoryFile>, Vec<APTRepositoryFileError>), Error> {
+ let mut files = vec![];
+ let mut errors = vec![];
+
+ let sources_list_path = PathBuf::from(APT_SOURCES_LIST_FILENAME);
+
+ let sources_list_d_path = PathBuf::from(APT_SOURCES_LIST_DIRECTORY);
+
+ match APTRepositoryFile::new(sources_list_path) {
+ Ok(Some(mut file)) => match file.parse() {
+ Ok(()) => files.push(file),
+ Err(err) => errors.push(err),
+ },
+ _ => bail!("internal error with '{}'", APT_SOURCES_LIST_FILENAME),
+ }
+
+ if !sources_list_d_path.exists() {
+ return Ok((files, errors));
+ }
+
+ if !sources_list_d_path.is_dir() {
+ errors.push(APTRepositoryFileError {
+ path: APT_SOURCES_LIST_DIRECTORY.to_string(),
+ error: "not a directory!".to_string(),
+ });
+ return Ok((files, errors));
+ }
+
+ for entry in std::fs::read_dir(sources_list_d_path)? {
+ let path = entry?.path();
+
+ match APTRepositoryFile::new(path) {
+ Ok(Some(mut file)) => match file.parse() {
+ Ok(()) => {
+ if file.digest.is_none() {
+ bail!("internal error - digest not set");
+ }
+ files.push(file);
+ }
+ Err(err) => errors.push(err),
+ },
+ Ok(None) => (),
+ Err(err) => errors.push(err),
+ }
+ }
+
+ Ok((files, errors))
+}
+
+/// Write the repositories for each file.
+///
+/// Returns an error for each file that could not be written successfully.
+pub fn write_repositories(files: &[APTRepositoryFile]) -> Result<(), Vec<APTRepositoryFileError>> {
+ let mut errors = vec![];
+
+ for file in files {
+ if let Err(err) = file.write() {
+ errors.push(err);
+ }
+ }
+
+ if !errors.is_empty() {
+ return Err(errors);
+ }
+
+ Ok(())
+}
diff --git a/src/repositories/sources_parser.rs b/src/repositories/sources_parser.rs
new file mode 100644
index 0000000..6065a32
--- /dev/null
+++ b/src/repositories/sources_parser.rs
@@ -0,0 +1,204 @@
+use std::convert::TryInto;
+use std::io::BufRead;
+use std::iter::Iterator;
+
+use anyhow::{bail, Error};
+
+use crate::types::{
+ APTRepository, APTRepositoryFileType, APTRepositoryOption, APTRepositoryPackageType,
+};
+
+use super::APTRepositoryParser;
+
+pub struct APTSourcesFileParser<R: BufRead> {
+ input: R,
+ stanza_nr: usize,
+ comment: String,
+}
+
+/// See `man sources.list` and `man deb822` for the format specification.
+impl<R: BufRead> APTSourcesFileParser<R> {
+ pub fn new(reader: R) -> Self {
+ Self {
+ input: reader,
+ stanza_nr: 1,
+ comment: String::new(),
+ }
+ }
+
+ /// Based on APT's `StringToBool` in `strutl.cc`
+ fn string_to_bool(string: &str, default: bool) -> bool {
+ let string = string.trim_matches(|c| char::is_ascii_whitespace(&c));
+ let string = string.to_lowercase();
+
+ match &string[..] {
+ "1" | "yes" | "true" | "with" | "on" | "enable" => true,
+ "0" | "no" | "false" | "without" | "off" | "disable" => false,
+ _ => default,
+ }
+ }
+
+ /// Checks if `key` is valid according to deb822
+ fn valid_key(key: &str) -> bool {
+ if key.starts_with('-') {
+ return false;
+ };
+ return key.chars().all(|c| matches!(c, '!'..='9' | ';'..='~'));
+ }
+
+ /// Try parsing a repository in stanza format from `lines`.
+ ///
+ /// Returns `Ok(None)` when no stanza can be found.
+ ///
+ /// Comments are added to `self.comments`. If a stanza can be found,
+ /// `self.comment` is added to the repository's `comment` property.
+ ///
+ /// Fully commented out stanzas are treated as comments.
+ fn parse_stanza(&mut self, lines: &str) -> Result<Option<APTRepository>, Error> {
+ let mut repo = APTRepository::new(APTRepositoryFileType::Sources);
+
+ // Values may be folded into multiple lines.
+ // Those lines have to start with a space or a tab.
+ let lines = lines.replace("\n ", " ");
+ let lines = lines.replace("\n\t", " ");
+
+ let mut got_something = false;
+
+ for line in lines.lines() {
+ let line = line.trim_matches(|c| char::is_ascii_whitespace(&c));
+ if line.is_empty() {
+ continue;
+ }
+
+ if let Some(commented_out) = line.strip_prefix('#') {
+ self.comment = format!("{}{}\n", self.comment, commented_out);
+ continue;
+ }
+
+ if let Some(mid) = line.find(':') {
+ let (key, value_str) = line.split_at(mid);
+ let value_str = &value_str[1..];
+ let key = key.trim_matches(|c| char::is_ascii_whitespace(&c));
+
+ if key.is_empty() {
+ bail!("option has no key: '{}'", line);
+ }
+
+ if value_str.is_empty() {
+ // ignored by APT
+ eprintln!("option has no value: '{}'", line);
+ continue;
+ }
+
+ if !Self::valid_key(key) {
+ // ignored by APT
+ eprintln!("option with invalid key '{}'", key);
+ continue;
+ }
+
+ let values: Vec<String> = value_str
+ .split_ascii_whitespace()
+ .map(|value| value.to_string())
+ .collect();
+
+ match &key.to_lowercase()[..] {
+ "types" => {
+ if !repo.types.is_empty() {
+ eprintln!("key 'Types' was defined twice");
+ }
+ let mut types = Vec::<APTRepositoryPackageType>::new();
+ for package_type in values {
+ types.push((&package_type[..]).try_into()?);
+ }
+ repo.types = types;
+ }
+ "uris" => {
+ if !repo.uris.is_empty() {
+ eprintln!("key 'URIs' was defined twice");
+ }
+ repo.uris = values;
+ }
+ "suites" => {
+ if !repo.suites.is_empty() {
+ eprintln!("key 'Suites' was defined twice");
+ }
+ repo.suites = values;
+ }
+ "components" => {
+ if !repo.components.is_empty() {
+ eprintln!("key 'Components' was defined twice");
+ }
+ repo.components = values;
+ }
+ "enabled" => {
+ repo.set_enabled(Self::string_to_bool(value_str, true));
+ }
+ _ => repo.options.push(APTRepositoryOption {
+ key: key.to_string(),
+ values,
+ }),
+ }
+ } else {
+ bail!("got invalid line - '{:?}'", line);
+ }
+
+ got_something = true;
+ }
+
+ if !got_something {
+ return Ok(None);
+ }
+
+ repo.comment = std::mem::take(&mut self.comment);
+
+ Ok(Some(repo))
+ }
+
+ /// Helper function for `parse_repositories`.
+ fn try_parse_stanza(
+ &mut self,
+ lines: &str,
+ repos: &mut Vec<APTRepository>,
+ ) -> Result<(), Error> {
+ match self.parse_stanza(&lines[..]) {
+ Ok(Some(repo)) => {
+ repos.push(repo);
+ self.stanza_nr += 1;
+ }
+ Ok(None) => (),
+ Err(err) => bail!("malformed entry in stanza {} - {}", self.stanza_nr, err),
+ }
+
+ Ok(())
+ }
+}
+
+impl<R: BufRead> APTRepositoryParser for APTSourcesFileParser<R> {
+ fn parse_repositories(&mut self) -> Result<Vec<APTRepository>, Error> {
+ let mut repos = vec![];
+ let mut lines = String::new();
+
+ loop {
+ let old_length = lines.len();
+ match self.input.read_line(&mut lines) {
+ Err(err) => bail!("input error - {}", err),
+ Ok(0) => {
+ self.try_parse_stanza(&lines[..], &mut repos)?;
+ break;
+ }
+ Ok(_) => {
+ if (&lines[old_length..])
+ .trim_matches(|c| char::is_ascii_whitespace(&c))
+ .is_empty()
+ {
+ // detected end of stanza
+ self.try_parse_stanza(&lines[..], &mut repos)?;
+ lines.clear();
+ }
+ }
+ }
+ }
+
+ Ok(repos)
+ }
+}
diff --git a/src/repositories/writer.rs b/src/repositories/writer.rs
new file mode 100644
index 0000000..d9e937c
--- /dev/null
+++ b/src/repositories/writer.rs
@@ -0,0 +1,92 @@
+use std::io::Write;
+
+use anyhow::{bail, Error};
+
+use crate::types::{APTRepository, APTRepositoryFileType};
+
+impl APTRepository {
+ /// Writes a repository in the corresponding format followed by a blank.
+ ///
+ /// Expects that `basic_check()` for the repository was successful.
+ pub fn write(&self, w: &mut dyn Write) -> Result<(), Error> {
+ match self.file_type {
+ APTRepositoryFileType::List => write_one_line(self, w),
+ APTRepositoryFileType::Sources => write_stanza(self, w),
+ }
+ }
+}
+
+/// Writes a repository in one-line format followed by a blank line.
+///
+/// Expects that `repo.file_type == APTRepositoryFileType::List`.
+///
+/// Expects that `basic_check()` for the repository was successful.
+fn write_one_line(repo: &APTRepository, w: &mut dyn Write) -> Result<(), Error> {
+ if repo.file_type != APTRepositoryFileType::List {
+ bail!("not a .list repository");
+ }
+
+ if !repo.comment.is_empty() {
+ for line in repo.comment.lines() {
+ writeln!(w, "#{}", line)?;
+ }
+ }
+
+ if !repo.enabled {
+ write!(w, "# ")?;
+ }
+
+ write!(w, "{} ", repo.types[0])?;
+
+ if !repo.options.is_empty() {
+ write!(w, "[ ")?;
+ repo.options
+ .iter()
+ .try_for_each(|option| write!(w, "{}={} ", option.key, option.values.join(",")))?;
+ write!(w, "] ")?;
+ };
+
+ write!(w, "{} ", repo.uris[0])?;
+ write!(w, "{} ", repo.suites[0])?;
+ writeln!(w, "{}", repo.components.join(" "))?;
+
+ writeln!(w)?;
+
+ Ok(())
+}
+
+/// Writes a single stanza followed by a blank line.
+///
+/// Expects that `repo.file_type == APTRepositoryFileType::Sources`.
+fn write_stanza(repo: &APTRepository, w: &mut dyn Write) -> Result<(), Error> {
+ if repo.file_type != APTRepositoryFileType::Sources {
+ bail!("not a .sources repository");
+ }
+
+ if !repo.comment.is_empty() {
+ for line in repo.comment.lines() {
+ writeln!(w, "#{}", line)?;
+ }
+ }
+
+ write!(w, "Types:")?;
+ repo.types
+ .iter()
+ .try_for_each(|package_type| write!(w, " {}", package_type))?;
+ writeln!(w)?;
+
+ writeln!(w, "URIs: {}", repo.uris.join(" "))?;
+ writeln!(w, "Suites: {}", repo.suites.join(" "))?;
+
+ if !repo.components.is_empty() {
+ writeln!(w, "Components: {}", repo.components.join(" "))?;
+ }
+
+ for option in repo.options.iter() {
+ writeln!(w, "{}: {}", option.key, option.values.join(" "))?;
+ }
+
+ writeln!(w)?;
+
+ Ok(())
+}
diff --git a/src/types.rs b/src/types.rs
new file mode 100644
index 0000000..45b8455
--- /dev/null
+++ b/src/types.rs
@@ -0,0 +1,246 @@
+use std::convert::TryFrom;
+use std::fmt::Display;
+
+use anyhow::{bail, Error};
+use serde::{Deserialize, Serialize};
+
+use proxmox::api::api;
+
+#[api]
+#[derive(Debug, Copy, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "lowercase")]
+pub enum APTRepositoryFileType {
+ /// One-line-style format
+ List,
+ /// DEB822-style format
+ Sources,
+}
+
+impl TryFrom<&str> for APTRepositoryFileType {
+ type Error = Error;
+
+ fn try_from(string: &str) -> Result<Self, Error> {
+ match &string[..] {
+ "list" => Ok(APTRepositoryFileType::List),
+ "sources" => Ok(APTRepositoryFileType::Sources),
+ _ => bail!("invalid file type '{}'", string),
+ }
+ }
+}
+
+impl Display for APTRepositoryFileType {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match self {
+ APTRepositoryFileType::List => write!(f, "list"),
+ APTRepositoryFileType::Sources => write!(f, "sources"),
+ }
+ }
+}
+
+#[api]
+#[derive(Debug, Copy, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+pub enum APTRepositoryPackageType {
+ /// Debian package
+ Deb,
+ /// Debian source package
+ DebSrc,
+}
+
+impl TryFrom<&str> for APTRepositoryPackageType {
+ type Error = Error;
+
+ fn try_from(string: &str) -> Result<Self, Error> {
+ match &string[..] {
+ "deb" => Ok(APTRepositoryPackageType::Deb),
+ "deb-src" => Ok(APTRepositoryPackageType::DebSrc),
+ _ => bail!("invalid package type '{}'", string),
+ }
+ }
+}
+
+impl Display for APTRepositoryPackageType {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match self {
+ APTRepositoryPackageType::Deb => write!(f, "deb"),
+ APTRepositoryPackageType::DebSrc => write!(f, "deb-src"),
+ }
+ }
+}
+
+#[api(
+ properties: {
+ Key: {
+ description: "Option key.",
+ type: String,
+ },
+ Values: {
+ description: "Option values.",
+ type: Array,
+ items: {
+ description: "Value.",
+ type: String,
+ },
+ },
+ },
+)]
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "PascalCase")] // for consistency
+/// Additional options for an APT repository.
+/// Used for both single- and mutli-value options.
+pub struct APTRepositoryOption {
+ /// Option key.
+ pub key: String,
+ /// Option value(s).
+ pub values: Vec<String>,
+}
+
+#[api(
+ properties: {
+ Types: {
+ description: "List of package types.",
+ type: Array,
+ items: {
+ type: APTRepositoryPackageType,
+ },
+ },
+ URIs: {
+ description: "List of repository URIs.",
+ type: Array,
+ items: {
+ description: "Repository URI.",
+ type: String,
+ },
+ },
+ Suites: {
+ description: "List of distributions.",
+ type: Array,
+ items: {
+ description: "Package distribution.",
+ type: String,
+ },
+ },
+ Components: {
+ description: "List of repository components.",
+ type: Array,
+ items: {
+ description: "Repository component.",
+ type: String,
+ },
+ },
+ Options: {
+ type: Array,
+ optional: true,
+ items: {
+ type: APTRepositoryOption,
+ },
+ },
+ Comment: {
+ description: "Associated comment.",
+ type: String,
+ optional: true,
+ },
+ FileType: {
+ type: APTRepositoryFileType,
+ },
+ Enabled: {
+ description: "Whether the repository is enabled or not.",
+ type: Boolean,
+ },
+ },
+)]
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "PascalCase")]
+/// Describes an APT repository.
+pub struct APTRepository {
+ /// List of package types.
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ pub types: Vec<APTRepositoryPackageType>,
+
+ /// List of repository URIs.
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ #[serde(rename = "URIs")]
+ pub uris: Vec<String>,
+
+ /// List of package distributions.
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ pub suites: Vec<String>,
+
+ /// List of repository components.
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ pub components: Vec<String>,
+
+ /// Additional options.
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ pub options: Vec<APTRepositoryOption>,
+
+ /// Associated comment.
+ #[serde(skip_serializing_if = "String::is_empty")]
+ pub comment: String,
+
+ /// Format of the defining file.
+ pub file_type: APTRepositoryFileType,
+
+ /// Whether the repository is enabled or not.
+ pub enabled: bool,
+}
+
+#[api(
+ properties: {
+ file_type: {
+ type: APTRepositoryFileType,
+ },
+ repositories: {
+ description: "List of APT repositories.",
+ type: Array,
+ items: {
+ type: APTRepository,
+ },
+ },
+ digest: {
+ description: "Digest for the content of the file.",
+ optional: true,
+ type: Array,
+ items: {
+ description: "Digest byte.",
+ type: Integer,
+ },
+ },
+ },
+)]
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "lowercase")]
+/// Represents an abstract APT repository file.
+pub struct APTRepositoryFile {
+ /// The path to the file.
+ pub path: String,
+ /// The type of the file.
+ pub file_type: APTRepositoryFileType,
+ /// List of repositories in the file.
+ pub repositories: Vec<APTRepository>,
+ /// Digest of the original contents.
+ pub digest: Option<[u8; 32]>,
+}
+
+#[api]
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "lowercase")]
+/// Error type for problems with APT repository files.
+pub struct APTRepositoryFileError {
+ /// The path to the problematic file.
+ pub path: String,
+ /// The error message.
+ pub error: String,
+}
+
+impl Display for APTRepositoryFileError {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ write!(f, "proxmox-apt error for '{}' - {}", self.path, self.error)
+ }
+}
+
+impl std::error::Error for APTRepositoryFileError {
+ fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
+ None
+ }
+}
diff --git a/tests/repositories.rs b/tests/repositories.rs
new file mode 100644
index 0000000..348c496
--- /dev/null
+++ b/tests/repositories.rs
@@ -0,0 +1,131 @@
+use std::path::PathBuf;
+
+use anyhow::{bail, format_err, Error};
+
+use proxmox_apt::repositories::write_repositories;
+use proxmox_apt::types::APTRepositoryFile;
+
+#[test]
+fn test_parse_write() -> Result<(), Error> {
+ let test_dir = std::env::current_dir()?.join("tests");
+ let read_dir = test_dir.join("sources.list.d");
+ let write_dir = test_dir.join("sources.list.d.actual");
+ let expected_dir = test_dir.join("sources.list.d.expected");
+
+ if write_dir.is_dir() {
+ std::fs::remove_dir_all(&write_dir)
+ .map_err(|err| format_err!("unable to remove dir {:?} - {}", write_dir, err))?;
+ }
+
+ std::fs::create_dir_all(&write_dir)
+ .map_err(|err| format_err!("unable to create dir {:?} - {}", write_dir, err))?;
+
+ let mut files = vec![];
+ let mut errors = vec![];
+
+ for entry in std::fs::read_dir(read_dir)? {
+ let path = entry?.path();
+
+ match APTRepositoryFile::new(&path)? {
+ Some(mut file) => match file.parse() {
+ Ok(()) => files.push(file),
+ Err(err) => errors.push(err),
+ },
+ None => bail!("unexpected None for '{:?}'", path),
+ }
+ }
+
+ assert!(errors.is_empty());
+
+ for file in files.iter_mut() {
+ let path = PathBuf::from(&file.path);
+ let new_path = write_dir.join(path.file_name().unwrap());
+ file.path = new_path.into_os_string().into_string().unwrap();
+ file.digest = None;
+ }
+
+ write_repositories(&files).map_err(|err| format_err!("{:?}", err))?;
+
+ let mut expected_count = 0;
+
+ for entry in std::fs::read_dir(expected_dir)? {
+ expected_count += 1;
+
+ let expected_path = entry?.path();
+ let actual_path = write_dir.join(expected_path.file_name().unwrap());
+
+ let expected_contents = std::fs::read(&expected_path)
+ .map_err(|err| format_err!("unable to read {:?} - {}", expected_path, err))?;
+
+ let actual_contents = std::fs::read(&actual_path)
+ .map_err(|err| format_err!("unable to read {:?} - {}", actual_path, err))?;
+
+ assert_eq!(
+ expected_contents, actual_contents,
+ "Use\n\ndiff {:?} {:?}\n\nif you're not fluent in byte decimals",
+ expected_path, actual_path
+ );
+ }
+
+ let actual_count = std::fs::read_dir(write_dir)?.count();
+
+ assert_eq!(expected_count, actual_count);
+
+ Ok(())
+}
+
+#[test]
+fn test_digest() -> Result<(), Error> {
+ let test_dir = std::env::current_dir()?.join("tests");
+ let read_dir = test_dir.join("sources.list.d");
+ let write_dir = test_dir.join("sources.list.d.digest");
+
+ if write_dir.is_dir() {
+ std::fs::remove_dir_all(&write_dir)
+ .map_err(|err| format_err!("unable to remove dir {:?} - {}", write_dir, err))?;
+ }
+
+ std::fs::create_dir_all(&write_dir)
+ .map_err(|err| format_err!("unable to create dir {:?} - {}", write_dir, err))?;
+
+ let file_name = "standard.list";
+
+ let path = read_dir.join(file_name);
+
+ let mut file = APTRepositoryFile::new(&path)?.unwrap();
+ file.parse()?;
+
+ let new_path = write_dir.join(path.file_name().unwrap());
+ file.path = new_path.clone().into_os_string().into_string().unwrap();
+
+ let old_digest = file.digest.unwrap();
+ let mut files = vec![file];
+
+ // file does not exist yet...
+ assert!(files.first().unwrap().read_with_digest().is_err());
+ assert!(write_repositories(&files).is_err());
+
+ // ...but it should work if there's no digest
+ files[0].digest = None;
+ write_repositories(&files).map_err(|err| format_err!("{:?}", err))?;
+
+ // overwrite with old contents...
+ std::fs::copy(path, new_path)?;
+
+ // modify the repo
+ let mut file = files.first_mut().unwrap();
+ let mut repo = file.repositories.first_mut().unwrap();
+ repo.enabled = !repo.enabled;
+
+ // ...then it should work
+ file.digest = Some(old_digest);
+ write_repositories(&files).map_err(|err| format_err!("{:?}", err))?;
+
+ // expect a different digest, because the repo was modified
+ let (_, new_digest) = files.first().unwrap().read_with_digest()?;
+ assert_ne!(old_digest, new_digest);
+
+ assert!(write_repositories(&files).is_err());
+
+ Ok(())
+}
diff --git a/tests/sources.list.d.expected/absolute_suite.list b/tests/sources.list.d.expected/absolute_suite.list
new file mode 100644
index 0000000..525389c
--- /dev/null
+++ b/tests/sources.list.d.expected/absolute_suite.list
@@ -0,0 +1,5 @@
+# From Debian Administrator's Handbook
+deb http://packages.falcot.com/ updates/
+
+deb http://packages.falcot.com/ internal/
+
diff --git a/tests/sources.list.d.expected/absolute_suite.sources b/tests/sources.list.d.expected/absolute_suite.sources
new file mode 100644
index 0000000..51e4d56
--- /dev/null
+++ b/tests/sources.list.d.expected/absolute_suite.sources
@@ -0,0 +1,5 @@
+# From Debian Administrator's Handbook
+Types: deb
+URIs: http://packages.falcot.com/
+Suites: updates/ internal/
+
diff --git a/tests/sources.list.d.expected/case.sources b/tests/sources.list.d.expected/case.sources
new file mode 100644
index 0000000..307aab6
--- /dev/null
+++ b/tests/sources.list.d.expected/case.sources
@@ -0,0 +1,16 @@
+# comment in here
+Types: deb deb-src
+URIs: http://ftp.at.debian.org/debian
+Suites: buster-updates
+Components: main contrib
+languages: it de fr
+Enabled: false
+languages-Add: ja
+languages-Remove: de
+
+# comment in here
+Types: deb deb-src
+URIs: http://ftp.at.debian.org/debian
+Suites: buster
+Components: main contrib
+
diff --git a/tests/sources.list.d.expected/multiline.sources b/tests/sources.list.d.expected/multiline.sources
new file mode 100644
index 0000000..d96acea
--- /dev/null
+++ b/tests/sources.list.d.expected/multiline.sources
@@ -0,0 +1,10 @@
+# comment in here
+Types: deb deb-src
+URIs: http://ftp.at.debian.org/debian
+Suites: buster buster-updates
+Components: main contrib
+Languages: it de fr
+Enabled: false
+Languages-Add: ja
+Languages-Remove: de
+
--git a/tests/sources.list.d.expected/options_comment.list b/tests/sources.list.d.expected/options_comment.list
new file mode 100644
index 0000000..8c905c0
--- /dev/null
+++ b/tests/sources.list.d.expected/options_comment.list
@@ -0,0 +1,6 @@
+# comment
+deb [ lang=it,de arch=amd64 ] http://ftp.at.debian.org/debian buster main contrib
+
+# non-free :(
+deb [ lang=it,de arch=amd64 lang+=fr lang-=de ] http://ftp.at.debian.org/debian buster non-free
+
diff --git a/tests/sources.list.d.expected/pbs-enterprise.list b/tests/sources.list.d.expected/pbs-enterprise.list
new file mode 100644
index 0000000..acb2990
--- /dev/null
+++ b/tests/sources.list.d.expected/pbs-enterprise.list
@@ -0,0 +1,2 @@
+deb https://enterprise.proxmox.com/debian/pbs buster pbs-enterprise
+
diff --git a/tests/sources.list.d.expected/pve.list b/tests/sources.list.d.expected/pve.list
new file mode 100644
index 0000000..127a49a
--- /dev/null
+++ b/tests/sources.list.d.expected/pve.list
@@ -0,0 +1,13 @@
+deb http://ftp.debian.org/debian buster main contrib
+
+deb http://ftp.debian.org/debian buster-updates main contrib
+
+# PVE pve-no-subscription repository provided by proxmox.com,
+# NOT recommended for production use
+deb http://download.proxmox.com/debian/pve buster pve-no-subscription
+
+# deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise
+
+# security updates
+deb http://security.debian.org/debian-security buster/updates main contrib
+
diff --git a/tests/sources.list.d.expected/standard.list b/tests/sources.list.d.expected/standard.list
new file mode 100644
index 0000000..63c1b60
--- /dev/null
+++ b/tests/sources.list.d.expected/standard.list
@@ -0,0 +1,7 @@
+deb http://ftp.at.debian.org/debian buster main contrib
+
+deb http://ftp.at.debian.org/debian buster-updates main contrib
+
+# security updates
+deb http://security.debian.org buster/updates main contrib
+
diff --git a/tests/sources.list.d.expected/standard.sources b/tests/sources.list.d.expected/standard.sources
new file mode 100644
index 0000000..56ce280
--- /dev/null
+++ b/tests/sources.list.d.expected/standard.sources
@@ -0,0 +1,11 @@
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: buster buster-updates
+Components: main contrib
+
+# security updates
+Types: deb
+URIs: http://security.debian.org
+Suites: buster/updates
+Components: main contrib
+
diff --git a/tests/sources.list.d/absolute_suite.list b/tests/sources.list.d/absolute_suite.list
new file mode 100644
index 0000000..ed576bf
--- /dev/null
+++ b/tests/sources.list.d/absolute_suite.list
@@ -0,0 +1,3 @@
+# From Debian Administrator's Handbook
+deb http://packages.falcot.com/ updates/
+deb http://packages.falcot.com/ internal/
diff --git a/tests/sources.list.d/absolute_suite.sources b/tests/sources.list.d/absolute_suite.sources
new file mode 100644
index 0000000..51e4d56
--- /dev/null
+++ b/tests/sources.list.d/absolute_suite.sources
@@ -0,0 +1,5 @@
+# From Debian Administrator's Handbook
+Types: deb
+URIs: http://packages.falcot.com/
+Suites: updates/ internal/
+
diff --git a/tests/sources.list.d/case.sources b/tests/sources.list.d/case.sources
new file mode 100644
index 0000000..8979d0c
--- /dev/null
+++ b/tests/sources.list.d/case.sources
@@ -0,0 +1,17 @@
+tYpeS: deb deb-src
+uRis: http://ftp.at.debian.org/debian
+suiTes: buster-updates
+# comment in here
+CompOnentS: main contrib
+languages: it
+ de
+ fr
+Enabled: off
+languages-Add: ja
+languages-Remove: de
+
+types: deb deb-src
+Uris: http://ftp.at.debian.org/debian
+suites: buster
+# comment in here
+components: main contrib
diff --git a/tests/sources.list.d/multiline.sources b/tests/sources.list.d/multiline.sources
new file mode 100644
index 0000000..bdbce29
--- /dev/null
+++ b/tests/sources.list.d/multiline.sources
@@ -0,0 +1,11 @@
+Types: deb deb-src
+URIs: http://ftp.at.debian.org/debian
+Suites: buster buster-updates
+# comment in here
+Components: main contrib
+Languages: it
+ de
+ fr
+Enabled: off
+Languages-Add: ja
+Languages-Remove: de
--git a/tests/sources.list.d/options_comment.list b/tests/sources.list.d/options_comment.list
new file mode 100644
index 0000000..6b73053
--- /dev/null
+++ b/tests/sources.list.d/options_comment.list
@@ -0,0 +1,3 @@
+deb [ lang=it,de arch=amd64 ] http://ftp.at.debian.org/debian buster main contrib # comment
+deb [ lang=it,de arch=amd64 lang+=fr lang-=de ] http://ftp.at.debian.org/debian buster non-free # non-free :(
+
diff --git a/tests/sources.list.d/pbs-enterprise.list b/tests/sources.list.d/pbs-enterprise.list
new file mode 100644
index 0000000..5f8763c
--- /dev/null
+++ b/tests/sources.list.d/pbs-enterprise.list
@@ -0,0 +1 @@
+deb https://enterprise.proxmox.com/debian/pbs buster pbs-enterprise
diff --git a/tests/sources.list.d/pve.list b/tests/sources.list.d/pve.list
new file mode 100644
index 0000000..6213f72
--- /dev/null
+++ b/tests/sources.list.d/pve.list
@@ -0,0 +1,10 @@
+deb http://ftp.debian.org/debian buster main contrib
+deb http://ftp.debian.org/debian buster-updates main contrib
+
+# PVE pve-no-subscription repository provided by proxmox.com,
+# NOT recommended for production use
+deb http://download.proxmox.com/debian/pve buster pve-no-subscription
+# deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise
+
+# security updates
+deb http://security.debian.org/debian-security buster/updates main contrib
diff --git a/tests/sources.list.d/standard.list b/tests/sources.list.d/standard.list
new file mode 100644
index 0000000..26db887
--- /dev/null
+++ b/tests/sources.list.d/standard.list
@@ -0,0 +1,6 @@
+deb http://ftp.at.debian.org/debian buster main contrib
+
+deb http://ftp.at.debian.org/debian buster-updates main contrib
+
+# security updates
+deb http://security.debian.org buster/updates main contrib
diff --git a/tests/sources.list.d/standard.sources b/tests/sources.list.d/standard.sources
new file mode 100644
index 0000000..605202e
--- /dev/null
+++ b/tests/sources.list.d/standard.sources
@@ -0,0 +1,10 @@
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: buster buster-updates
+Components: main contrib
+
+# security updates
+Types: deb
+URIs: http://security.debian.org
+Suites: buster/updates
+Components: main contrib
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* [pbs-devel] [PATCH v4 proxmox-apt 02/10] add files for Debian packaging
2021-04-02 11:20 [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 01/10] initial commit Fabian Ebner
@ 2021-04-02 11:20 ` Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 03/10] add functions to check for Proxmox repositories Fabian Ebner
` (8 subsequent siblings)
10 siblings, 0 replies; 13+ messages in thread
From: Fabian Ebner @ 2021-04-02 11:20 UTC (permalink / raw)
To: pbs-devel
The Makefile is based on the one from Mira's conntrack series, as it already got
some review.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
No changes from v3.
.gitignore | 5 ++++
Makefile | 64 ++++++++++++++++++++++++++++++++++++++++++++
debian/changelog | 5 ++++
debian/copyright | 16 +++++++++++
debian/debcargo.toml | 7 +++++
5 files changed, 97 insertions(+)
create mode 100644 Makefile
create mode 100644 debian/changelog
create mode 100644 debian/copyright
create mode 100644 debian/debcargo.toml
diff --git a/.gitignore b/.gitignore
index 24917d4..db6f13e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,3 +2,8 @@ Cargo.lock
target/
tests/sources.list.d.actual
tests/sources.list.d.digest
+proxmox-apt-*/
+*proxmox-apt*.buildinfo
+*proxmox-apt*.tar.?z
+*proxmox-apt*.changes
+*proxmox-apt*.deb
diff --git a/Makefile b/Makefile
new file mode 100644
index 0000000..de3d620
--- /dev/null
+++ b/Makefile
@@ -0,0 +1,64 @@
+include /usr/share/dpkg/pkg-info.mk
+include /usr/share/dpkg/architecture.mk
+
+PACKAGE=proxmox-apt
+BUILDDIR ?= $(PACKAGE)-$(DEB_VERSION_UPSTREAM)
+BUILDDIR_TMP ?= $(BUILDDIR).tmp
+
+DEB=librust-$(PACKAGE)-dev_$(DEB_VERSION_UPSTREAM_REVISION)_$(DEB_BUILD_ARCH).deb
+DSC=rust-$(PACKAGE)_$(DEB_VERSION_UPSTREAM_REVISION).dsc
+
+ifeq ($(BUILD_MODE), release)
+CARGO_BUILD_ARGS += --release
+COMPILEDIR := target/release
+else
+COMPILEDIR := target/debug
+endif
+
+all: cargo-build $(SUBDIRS)
+
+.PHONY: cargo-build
+cargo-build:
+ cargo build $(CARGO_BUILD_ARGS)
+
+.PHONY: build
+build:
+ rm -rf $(BUILDDIR) $(BUILDDIR_TMP); mkdir $(BUILDDIR_TMP)
+ debcargo package \
+ --config debian/debcargo.toml \
+ --changelog-ready \
+ --no-overlay-write-back \
+ --directory $(BUILDDIR_TMP) \
+ $(PACKAGE) \
+ $(shell dpkg-parsechangelog -l debian/changelog -SVersion | sed -e 's/-.*//')
+ rm -f $(BUILDDIR_TMP)/Cargo.lock
+ find $(BUILDDIR_TMP)/debian -name "*.hint" -delete
+ mv $(BUILDDIR_TMP) $(BUILDDIR)
+
+.PHONY: deb
+deb: $(DEB)
+$(DEB): build
+ cd $(BUILDDIR); dpkg-buildpackage -b -us -uc --no-pre-clean
+ lintian $(DEB)
+
+.PHONY: dsc
+dsc: $(DSC)
+$(DSC): build
+ cd $(BUILDDIR); dpkg-buildpackage -S -us -uc -d -nc
+ lintian $(DSC)
+
+.PHONY: dinstall
+dinstall: $(DEB)
+ dpkg -i $(DEB)
+
+.PHONY: upload
+upload: $(DEB) $(DBG_DEB)
+ tar cf - $(DEB) $(DBG_DEB) | ssh -X repoman@repo.proxmox.com -- upload --product pbs,pmg,pve --dist buster --arch $(DEB_BUILD_ARCH)
+
+.PHONY: distclean
+distclean: clean
+
+.PHONY: clean
+clean:
+ rm -rf *.deb *.buildinfo *.changes *.dsc rust-$(PACKAGE)_*.tar.?z $(BUILDDIR) $(BUILDDIR_TMP)
+ find . -name '*~' -exec rm {} ';'
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..11e26ed
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+rust-proxmox-apt (0.1.0-1) unstable; urgency=medium
+
+ * Initial release.
+
+ -- Proxmox Support Team <support@proxmox.com> Thu, 18 Feb 2021 10:20:44 +0100
diff --git a/debian/copyright b/debian/copyright
new file mode 100644
index 0000000..5661ef6
--- /dev/null
+++ b/debian/copyright
@@ -0,0 +1,16 @@
+Copyright (C) 2021 Proxmox Server Solutions GmbH
+
+This software is written by Proxmox Server Solutions GmbH <support@proxmox.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
diff --git a/debian/debcargo.toml b/debian/debcargo.toml
new file mode 100644
index 0000000..74e3854
--- /dev/null
+++ b/debian/debcargo.toml
@@ -0,0 +1,7 @@
+overlay = "."
+crate_src_path = ".."
+maintainer = "Proxmox Support Team <support@proxmox.com>"
+
+[source]
+vcs_git = "git://git.proxmox.com/git/proxmox-apt.git"
+vcs_browser = "https://git.proxmox.com/?p=proxmox-apt.git"
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* [pbs-devel] [PATCH v4 proxmox-apt 03/10] add functions to check for Proxmox repositories
2021-04-02 11:20 [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 01/10] initial commit Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 02/10] add files for Debian packaging Fabian Ebner
@ 2021-04-02 11:20 ` Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 04/10] add check_repositories function Fabian Ebner
` (7 subsequent siblings)
10 siblings, 0 replies; 13+ messages in thread
From: Fabian Ebner @ 2021-04-02 11:20 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v3:
* switched to taking APTRepositoryFile slice as input.
src/repositories/check.rs | 42 +++++++++++++++++++++++++++++++++++++++
src/repositories/mod.rs | 20 +++++++++++++++++++
tests/repositories.rs | 34 ++++++++++++++++++++++++++++++-
3 files changed, 95 insertions(+), 1 deletion(-)
diff --git a/src/repositories/check.rs b/src/repositories/check.rs
index 87fbbac..d0656cd 100644
--- a/src/repositories/check.rs
+++ b/src/repositories/check.rs
@@ -44,4 +44,46 @@ impl APTRepository {
Ok(())
}
+
+ /// Checks if the repository is the no-subscription repository of the specified
+ /// Proxmox product.
+ pub fn is_no_subscription(&self, product: &str) -> bool {
+ let base_uri = "http://download.proxmox.com/debian";
+ let no_subscription_uri = format!("{}/{}", base_uri, product);
+ let no_subscription_component = format!("{}-no-subscription", product);
+
+ if self
+ .uris
+ .iter()
+ .any(|uri| uri.trim_end_matches('/') == no_subscription_uri)
+ {
+ return self
+ .components
+ .iter()
+ .any(|comp| *comp == no_subscription_component);
+ } else {
+ false
+ }
+ }
+
+ /// Checks if the repository is the enterprise repository of the specified
+ /// Proxmox product.
+ pub fn is_enterprise(&self, product: &str) -> bool {
+ let base_uri = "https://enterprise.proxmox.com/debian";
+ let enterprise_uri = format!("{}/{}", base_uri, product);
+ let enterprise_component = format!("{}-enterprise", product);
+
+ if self
+ .uris
+ .iter()
+ .any(|uri| uri.trim_end_matches('/') == enterprise_uri)
+ {
+ return self
+ .components
+ .iter()
+ .any(|comp| *comp == enterprise_component);
+ } else {
+ false
+ }
+ }
}
diff --git a/src/repositories/mod.rs b/src/repositories/mod.rs
index 187ead3..b7919a9 100644
--- a/src/repositories/mod.rs
+++ b/src/repositories/mod.rs
@@ -148,6 +148,26 @@ impl APTRepositoryFile {
}
}
+/// Checks if the enterprise repository for the specified Proxmox product is
+/// configured and enabled.
+pub fn enterprise_repository_enabled(files: &[APTRepositoryFile], product: &str) -> bool {
+ files.iter().any(|file| {
+ file.repositories
+ .iter()
+ .any(|repo| repo.enabled && repo.is_enterprise(product))
+ })
+}
+
+/// Checks if the no-subscription repository for the specified Proxmox product
+/// is configured and enabled.
+pub fn no_subscription_repository_enabled(files: &[APTRepositoryFile], product: &str) -> bool {
+ files.iter().any(|file| {
+ file.repositories
+ .iter()
+ .any(|repo| repo.enabled && repo.is_no_subscription(product))
+ })
+}
+
/// Returns all APT repositories configured in `/etc/apt/sources.list` and
/// in `/etc/apt/sources.list.d` including disabled repositories.
///
diff --git a/tests/repositories.rs b/tests/repositories.rs
index 348c496..d23beac 100644
--- a/tests/repositories.rs
+++ b/tests/repositories.rs
@@ -2,7 +2,9 @@ use std::path::PathBuf;
use anyhow::{bail, format_err, Error};
-use proxmox_apt::repositories::write_repositories;
+use proxmox_apt::repositories::{
+ enterprise_repository_enabled, no_subscription_repository_enabled, write_repositories,
+};
use proxmox_apt::types::APTRepositoryFile;
#[test]
@@ -129,3 +131,33 @@ fn test_digest() -> Result<(), Error> {
Ok(())
}
+
+#[test]
+fn test_proxmox_repositories() -> Result<(), Error> {
+ let test_dir = std::env::current_dir()?.join("tests");
+ let read_dir = test_dir.join("sources.list.d");
+
+ let pve_list = read_dir.join("pve.list");
+ let mut file = APTRepositoryFile::new(&pve_list)?.unwrap();
+ file.parse()?;
+
+ let files = vec![file];
+
+ assert_eq!(false, enterprise_repository_enabled(&files, "pbs"));
+ assert_eq!(false, enterprise_repository_enabled(&files, "pve"));
+ assert_eq!(false, no_subscription_repository_enabled(&files, "pmg"));
+ assert_eq!(true, no_subscription_repository_enabled(&files, "pve"));
+
+ let pbs_list = read_dir.join("pbs-enterprise.list");
+ let mut file = APTRepositoryFile::new(&pbs_list)?.unwrap();
+ file.parse()?;
+
+ let files = vec![file];
+
+ assert_eq!(true, enterprise_repository_enabled(&files, "pbs"));
+ assert_eq!(false, enterprise_repository_enabled(&files, "pve"));
+ assert_eq!(false, no_subscription_repository_enabled(&files, "pmg"));
+ assert_eq!(false, no_subscription_repository_enabled(&files, "pve"));
+
+ Ok(())
+}
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* [pbs-devel] [PATCH v4 proxmox-apt 04/10] add check_repositories function
2021-04-02 11:20 [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
` (2 preceding siblings ...)
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 03/10] add functions to check for Proxmox repositories Fabian Ebner
@ 2021-04-02 11:20 ` Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-backup 05/10] depend on new proxmox-apt crate Fabian Ebner
` (6 subsequent siblings)
10 siblings, 0 replies; 13+ messages in thread
From: Fabian Ebner @ 2021-04-02 11:20 UTC (permalink / raw)
To: pbs-devel
which checks for bad suites and official URIs.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v3:
* switched to taking APTRepositoryFile slice as input
* don't use vec! for a constant array
* replace APTRepositoryWarning with a more general APTRepositoryInfo
* warn about 'testing' when checking suites
* warn about 'bullseye', but have it be a special warning so the UI
can avoid showing it while a major upgrade happens
* check for official host names and return a 'badge' info
* add/adapt tests
src/repositories/check.rs | 151 +++++++++++++++++++++-
src/repositories/mod.rs | 19 ++-
src/types.rs | 19 +++
tests/repositories.rs | 97 +++++++++++++-
tests/sources.list.d.expected/bad.sources | 30 +++++
tests/sources.list.d/bad.sources | 29 +++++
6 files changed, 341 insertions(+), 4 deletions(-)
create mode 100644 tests/sources.list.d.expected/bad.sources
create mode 100644 tests/sources.list.d/bad.sources
diff --git a/src/repositories/check.rs b/src/repositories/check.rs
index d0656cd..809b7bc 100644
--- a/src/repositories/check.rs
+++ b/src/repositories/check.rs
@@ -1,6 +1,22 @@
use anyhow::{bail, Error};
-use crate::types::{APTRepository, APTRepositoryFileType};
+use crate::types::{
+ APTRepository, APTRepositoryFile, APTRepositoryFileType, APTRepositoryInfo,
+ APTRepositoryPackageType,
+};
+
+/// Checks if `suite` is some variant of `base_suite`, e.g. `buster-backports`
+/// is a variant of `buster`.
+fn suite_is_variant(suite: &str, base_suite: &str) -> bool {
+ matches!(
+ suite.strip_prefix(base_suite),
+ Some("")
+ | Some("-backports")
+ | Some("-backports-sloppy")
+ | Some("-updates")
+ | Some("/updates")
+ )
+}
impl APTRepository {
/// Makes sure that all basic properties of a repository are present and
@@ -86,4 +102,137 @@ impl APTRepository {
false
}
}
+
+ /// Checks if old or unstable suites are configured and also that the
+ /// `stable` keyword is not used.
+ fn check_suites(&self, add_info: &mut dyn FnMut(String, String)) {
+ let old_suites = [
+ "lenny",
+ "squeeze",
+ "wheezy",
+ "jessie",
+ "stretch",
+ "oldoldstable",
+ "oldstable",
+ ];
+
+ let next_suite = "bullseye";
+
+ let new_suites = ["testing", "unstable", "sid", "experimental"];
+
+ if self
+ .types
+ .iter()
+ .any(|package_type| *package_type == APTRepositoryPackageType::Deb)
+ {
+ for suite in self.suites.iter() {
+ if old_suites
+ .iter()
+ .any(|base_suite| suite_is_variant(suite, base_suite))
+ {
+ add_info(
+ "warning".to_string(),
+ format!("old suite '{}' configured!", suite),
+ );
+ }
+
+ if suite_is_variant(suite, next_suite) {
+ add_info(
+ "ignore-pre-upgrade-warning".to_string(),
+ format!("suite '{}' should not be used in production!", suite),
+ );
+ }
+
+ if new_suites
+ .iter()
+ .any(|base_suite| suite_is_variant(suite, base_suite))
+ {
+ add_info(
+ "warning".to_string(),
+ format!("suite '{}' should not be used in production!", suite),
+ );
+ }
+
+ if suite_is_variant(suite, "stable") {
+ add_info(
+ "warning".to_string(),
+ "use the name of the stable distribution instead of 'stable'!".to_string(),
+ );
+ }
+ }
+ }
+ }
+
+ /// Checks if an official host is configured in the repository.
+ fn check_uris(&self) -> Option<(String, String)> {
+ let official_host = |domains: &Vec<&str>| {
+ match domains.len() {
+ 3 => match domains[0] {
+ "ftp" | "deb" | "security" => domains[1] == "debian" && domains[2] == "org",
+ "download" | "enterprise" => domains[1] == "proxmox" && domains[2] == "com",
+ _ => false,
+ },
+ // ftp.*.debian.org
+ 4 => domains[0] == "ftp" && domains[2] == "debian" && domains[3] == "org",
+ _ => false,
+ }
+ };
+
+ for uri in self.uris.iter() {
+ if let Some(begin) = uri.find("://") {
+ let mut host = uri.split_at(begin + 3).1;
+ if let Some(end) = host.find('/') {
+ host = host.split_at(end).0;
+ } // otherwise assume everything belongs is the host
+
+ let domains = host.split('.').collect();
+
+ if official_host(&domains) {
+ return Some(("badge".to_string(), "official host name".to_string()));
+ }
+ }
+ }
+
+ None
+ }
+}
+
+impl APTRepositoryFile {
+ /// Checks if old or unstable suites are configured and also that the
+ /// `stable` keyword is not used.
+ pub fn check_suites(&self) -> Vec<APTRepositoryInfo> {
+ let mut infos = vec![];
+
+ for (n, repo) in self.repositories.iter().enumerate() {
+ let mut add_info = |kind, message| {
+ infos.push(APTRepositoryInfo {
+ path: self.path.clone(),
+ number: n + 1,
+ kind,
+ message,
+ })
+ };
+ repo.check_suites(&mut add_info);
+ }
+
+ infos
+ }
+
+ /// Checks for official URIs.
+ pub fn check_uris(&self) -> Vec<APTRepositoryInfo> {
+ let mut infos = vec![];
+
+ for (n, repo) in self.repositories.iter().enumerate() {
+ if let Some((kind, message)) = repo.check_uris() {
+ infos.push(APTRepositoryInfo {
+ path: self.path.clone(),
+ number: n + 1,
+ kind,
+ message,
+ });
+ }
+ }
+
+ infos
+ }
}
diff --git a/src/repositories/mod.rs b/src/repositories/mod.rs
index b7919a9..c2bbc06 100644
--- a/src/repositories/mod.rs
+++ b/src/repositories/mod.rs
@@ -4,7 +4,7 @@ use anyhow::{bail, format_err, Error};
use crate::types::{
APTRepository, APTRepositoryFile, APTRepositoryFileError, APTRepositoryFileType,
- APTRepositoryOption,
+ APTRepositoryInfo, APTRepositoryOption,
};
mod list_parser;
@@ -148,6 +148,23 @@ impl APTRepositoryFile {
}
}
+/// Provides additional information about the repositories.
+///
+/// The kind of information can be:
+/// `warnings` for bad suites.
+/// `ignore-pre-upgrade-warning` when the next stable suite is configured.
+/// `badge` for official URIs.
+pub fn check_repositories(files: &[APTRepositoryFile]) -> Vec<APTRepositoryInfo> {
+ let mut infos = vec![];
+
+ for file in files.iter() {
+ infos.append(&mut file.check_suites());
+ infos.append(&mut file.check_uris());
+ }
+
+ infos
+}
+
/// Checks if the enterprise repository for the specified Proxmox product is
/// configured and enabled.
pub fn enterprise_repository_enabled(files: &[APTRepositoryFile], product: &str) -> bool {
diff --git a/src/types.rs b/src/types.rs
index 45b8455..94b6411 100644
--- a/src/types.rs
+++ b/src/types.rs
@@ -244,3 +244,22 @@ impl std::error::Error for APTRepositoryFileError {
None
}
}
+
+#[api]
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
+#[serde(rename_all = "lowercase")]
+/// Additional information for a repository.
+pub struct APTRepositoryInfo {
+ /// Path to the defining file.
+ #[serde(skip_serializing_if = "String::is_empty")]
+ pub path: String,
+
+ /// Number of the associated respository within the file.
+ pub number: usize,
+
+ /// Info kind (e.g. "warning")
+ pub kind: String,
+
+ /// Info message
+ pub message: String,
+}
diff --git a/tests/repositories.rs b/tests/repositories.rs
index d23beac..2b9b208 100644
--- a/tests/repositories.rs
+++ b/tests/repositories.rs
@@ -3,9 +3,10 @@ use std::path::PathBuf;
use anyhow::{bail, format_err, Error};
use proxmox_apt::repositories::{
- enterprise_repository_enabled, no_subscription_repository_enabled, write_repositories,
+ check_repositories, enterprise_repository_enabled, no_subscription_repository_enabled,
+ write_repositories,
};
-use proxmox_apt::types::APTRepositoryFile;
+use proxmox_apt::types::{APTRepositoryFile, APTRepositoryInfo};
#[test]
fn test_parse_write() -> Result<(), Error> {
@@ -161,3 +162,95 @@ fn test_proxmox_repositories() -> Result<(), Error> {
Ok(())
}
+
+#[test]
+fn test_check_repositories() -> Result<(), Error> {
+ let test_dir = std::env::current_dir()?.join("tests");
+ let read_dir = test_dir.join("sources.list.d");
+
+ let absolute_suite_list = read_dir.join("absolute_suite.list");
+ let mut file = APTRepositoryFile::new(&absolute_suite_list)?.unwrap();
+ file.parse()?;
+
+ let infos = check_repositories(&vec![file]);
+
+ assert_eq!(infos.is_empty(), true);
+ let pve_list = read_dir.join("pve.list");
+ let mut file = APTRepositoryFile::new(&pve_list)?.unwrap();
+ file.parse()?;
+
+ let path_string = pve_list.into_os_string().into_string().unwrap();
+
+ let mut expected_infos = vec![];
+ for n in 1..=5 {
+ expected_infos.push(APTRepositoryInfo {
+ path: path_string.clone(),
+ number: n,
+ kind: "badge".to_string(),
+ message: "official host name".to_string(),
+ });
+ }
+
+ let mut infos = check_repositories(&vec![file]);
+
+ assert_eq!(infos.sort(), expected_infos.sort());
+
+ let bad_sources = read_dir.join("bad.sources");
+ let mut file = APTRepositoryFile::new(&bad_sources)?.unwrap();
+ file.parse()?;
+
+ let path_string = bad_sources.into_os_string().into_string().unwrap();
+
+ let mut expected_infos = vec![
+ APTRepositoryInfo {
+ path: path_string.clone(),
+ number: 1,
+ kind: "warning".to_string(),
+ message: "suite 'sid' should not be used in production!".to_string(),
+ },
+ APTRepositoryInfo {
+ path: path_string.clone(),
+ number: 2,
+ kind: "warning".to_string(),
+ message: "old suite 'lenny-backports' configured!".to_string(),
+ },
+ APTRepositoryInfo {
+ path: path_string.clone(),
+ number: 3,
+ kind: "warning".to_string(),
+ message: "old suite 'stretch/updates' configured!".to_string(),
+ },
+ APTRepositoryInfo {
+ path: path_string.clone(),
+ number: 4,
+ kind: "warning".to_string(),
+ message: "use the name of the stable distribution instead of 'stable'!".to_string(),
+ },
+ APTRepositoryInfo {
+ path: path_string.clone(),
+ number: 5,
+ kind: "ignore-pre-upgrade-warning".to_string(),
+ message: "suite 'bullseye' should not be used in production!".to_string(),
+ },
+ APTRepositoryInfo {
+ path: path_string.clone(),
+ number: 6,
+ kind: "warning".to_string(),
+ message: "suite 'testing' should not be used in production!".to_string(),
+ },
+ ];
+ for n in 1..=6 {
+ expected_infos.push(APTRepositoryInfo {
+ path: path_string.clone(),
+ number: n,
+ kind: "badge".to_string(),
+ message: "official URI".to_string(),
+ });
+ }
+
+ let mut infos = check_repositories(&vec![file]);
+
+ assert_eq!(infos.sort(), expected_infos.sort());
+
+ Ok(())
+}
diff --git a/tests/sources.list.d.expected/bad.sources b/tests/sources.list.d.expected/bad.sources
new file mode 100644
index 0000000..36ff7a0
--- /dev/null
+++ b/tests/sources.list.d.expected/bad.sources
@@ -0,0 +1,30 @@
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: sid
+Components: main contrib
+
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: lenny-backports
+Components: contrib
+
+Types: deb
+URIs: http://security.debian.org
+Suites: stretch/updates
+Components: main contrib
+
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: stable
+Components: main
+
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: bullseye
+Components: main
+
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: testing
+Components: main
+
diff --git a/tests/sources.list.d/bad.sources b/tests/sources.list.d/bad.sources
new file mode 100644
index 0000000..6f2524a
--- /dev/null
+++ b/tests/sources.list.d/bad.sources
@@ -0,0 +1,29 @@
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: sid
+Components: main contrib
+
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: lenny-backports
+Components: contrib
+
+Types: deb
+URIs: http://security.debian.org
+Suites: stretch/updates
+Components: main contrib
+
+Suites: stable
+URIs: http://ftp.at.debian.org/debian
+Components: main
+Types: deb
+
+Suites: bullseye
+URIs: http://ftp.at.debian.org/debian
+Components: main
+Types: deb
+
+Suites: testing
+URIs: http://ftp.at.debian.org/debian
+Components: main
+Types: deb
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* [pbs-devel] [PATCH v4 proxmox-backup 05/10] depend on new proxmox-apt crate
2021-04-02 11:20 [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
` (3 preceding siblings ...)
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 04/10] add check_repositories function Fabian Ebner
@ 2021-04-02 11:20 ` Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-backup 06/10] api: apt: add repositories call Fabian Ebner
` (5 subsequent siblings)
10 siblings, 0 replies; 13+ messages in thread
From: Fabian Ebner @ 2021-04-02 11:20 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v3:
* also update debian/control
Cargo.toml | 1 +
debian/control | 1 +
2 files changed, 2 insertions(+)
diff --git a/Cargo.toml b/Cargo.toml
index 69b07d41..fdacf46f 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -51,6 +51,7 @@ pathpatterns = "0.1.2"
proxmox = { version = "0.11.0", features = [ "sortable-macro", "api-macro", "websocket" ] }
#proxmox = { git = "git://git.proxmox.com/git/proxmox", version = "0.1.2", features = [ "sortable-macro", "api-macro" ] }
#proxmox = { path = "../proxmox/proxmox", features = [ "sortable-macro", "api-macro", "websocket" ] }
+proxmox-apt = "0.1.0"
proxmox-fuse = "0.1.1"
pxar = { version = "0.10.1", features = [ "tokio-io" ] }
#pxar = { path = "../pxar", features = [ "tokio-io" ] }
diff --git a/debian/control b/debian/control
index 5cea2ed2..271d1d27 100644
--- a/debian/control
+++ b/debian/control
@@ -40,6 +40,7 @@ Build-Depends: debhelper (>= 11),
librust-proxmox-0.11+default-dev,
librust-proxmox-0.11+sortable-macro-dev,
librust-proxmox-0.11+websocket-dev,
+ librust-proxmox-apt-0.1+default-dev,
librust-proxmox-fuse-0.1+default-dev (>= 0.1.1-~~),
librust-pxar-0.10+default-dev (>= 0.10.1-~~),
librust-pxar-0.10+tokio-io-dev (>= 0.10.1-~~),
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* [pbs-devel] [PATCH v4 proxmox-backup 06/10] api: apt: add repositories call
2021-04-02 11:20 [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
` (4 preceding siblings ...)
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-backup 05/10] depend on new proxmox-apt crate Fabian Ebner
@ 2021-04-02 11:20 ` Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-backup 07/10] ui: add panel for APT repositories Fabian Ebner
` (4 subsequent siblings)
10 siblings, 0 replies; 13+ messages in thread
From: Fabian Ebner @ 2021-04-02 11:20 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v3:
* adapt to new library behavior
* add helper to calculate common digest
* return successfully parsed files and errors separately
src/api2/node/apt.rs | 83 +++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 82 insertions(+), 1 deletion(-)
diff --git a/src/api2/node/apt.rs b/src/api2/node/apt.rs
index e77b89fa..c91d03e1 100644
--- a/src/api2/node/apt.rs
+++ b/src/api2/node/apt.rs
@@ -1,16 +1,19 @@
use anyhow::{Error, bail, format_err};
use serde_json::{json, Value};
+use std::collections::BTreeMap;
use std::collections::HashMap;
use proxmox::list_subdirs_api_method;
use proxmox::api::{api, RpcEnvironment, RpcEnvironmentType, Permission};
use proxmox::api::router::{Router, SubdirMap};
+use proxmox_apt::types::{APTRepositoryFile, APTRepositoryFileError};
+
use crate::server::WorkerTask;
use crate::tools::{apt, http, subscription};
use crate::config::acl::{PRIV_SYS_AUDIT, PRIV_SYS_MODIFY};
-use crate::api2::types::{Authid, APTUpdateInfo, NODE_SCHEMA, UPID_SCHEMA};
+use crate::api2::types::{Authid, APTUpdateInfo, NODE_SCHEMA, PROXMOX_CONFIG_DIGEST_SCHEMA, UPID_SCHEMA};
#[api(
input: {
@@ -350,8 +353,86 @@ pub fn get_versions() -> Result<Vec<APTUpdateInfo>, Error> {
Ok(packages)
}
+/// Calculates a common digest for successfully parsed repository files.
+///
+/// The digest is invariant with respect to file order.
+///
+/// Files without a digest are ignored.
+fn repositories_common_digest(files: &Vec<APTRepositoryFile>) -> [u8; 32] {
+ let mut digests = BTreeMap::new();
+
+ for file in files.iter() {
+ digests.insert(file.path.clone(), &file.digest);
+ }
+
+ let mut common_raw = Vec::<u8>::with_capacity(digests.len() * 32);
+ for digest in digests.values() {
+ match digest {
+ Some(digest) => common_raw.extend_from_slice(&digest[..]),
+ None => (),
+ }
+ }
+
+ openssl::sha::sha256(&common_raw[..])
+}
+
+#[api(
+ input: {
+ properties: {
+ node: {
+ schema: NODE_SCHEMA,
+ },
+ },
+ },
+ returns: {
+ type: Object,
+ description: "Result from parsing the APT repository files in /etc/apt/.",
+ properties: {
+ files: {
+ description: "List of parsed repository files.",
+ type: Array,
+ items: {
+ type: APTRepositoryFile,
+ }
+ },
+ errors: {
+ description: "List of problematic files.",
+ type: Array,
+ items: {
+ type: APTRepositoryFileError,
+ }
+ },
+ digest: {
+ schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+ },
+ },
+ },
+ access: {
+ permission: &Permission::Privilege(&[], PRIV_SYS_AUDIT, false),
+ },
+)]
+/// Get APT repository information
+pub fn get_repositories() -> Result<Value, Error> {
+ let (files, errors) = proxmox_apt::repositories::repositories()?;
+
+ if files.len() == 0 {
+ bail!("no APT repository files could be parsed!");
+ }
+
+ let common_digest = repositories_common_digest(&files);
+
+ let hex_digest = proxmox::tools::digest_to_hex(&common_digest);
+
+ Ok(json!({
+ "files": files,
+ "errors": errors,
+ "digest": hex_digest,
+ }))
+}
+
const SUBDIRS: SubdirMap = &[
("changelog", &Router::new().get(&API_METHOD_APT_GET_CHANGELOG)),
+ ("repositories", &Router::new().get(&API_METHOD_GET_REPOSITORIES)),
("update", &Router::new()
.get(&API_METHOD_APT_UPDATE_AVAILABLE)
.post(&API_METHOD_APT_UPDATE_DATABASE)
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* [pbs-devel] [PATCH v4 proxmox-backup 07/10] ui: add panel for APT repositories
2021-04-02 11:20 [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
` (5 preceding siblings ...)
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-backup 06/10] api: apt: add repositories call Fabian Ebner
@ 2021-04-02 11:20 ` Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-backup 08/10] api: apt: add check_repositories_call Fabian Ebner
` (3 subsequent siblings)
10 siblings, 0 replies; 13+ messages in thread
From: Fabian Ebner @ 2021-04-02 11:20 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v3:
* add majorUpgradesAllowed flag
www/ServerAdministration.js | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/www/ServerAdministration.js b/www/ServerAdministration.js
index 0d803ac4..16de687e 100644
--- a/www/ServerAdministration.js
+++ b/www/ServerAdministration.js
@@ -53,6 +53,14 @@ Ext.define('PBS.ServerAdministration', {
itemId: 'updates',
nodename: 'localhost',
},
+ {
+ xtype: 'proxmoxNodeAPTRepositories',
+ title: gettext('Repositories'),
+ iconCls: 'fa fa-files-o',
+ itemId: 'aptrepositories',
+ nodename: 'localhost',
+ majorUpgradeAllowed: false,
+ },
{
xtype: 'proxmoxJournalView',
itemId: 'logs',
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* [pbs-devel] [PATCH v4 proxmox-backup 08/10] api: apt: add check_repositories_call
2021-04-02 11:20 [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
` (6 preceding siblings ...)
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-backup 07/10] ui: add panel for APT repositories Fabian Ebner
@ 2021-04-02 11:20 ` Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 widget-toolkit 09/10] add UI for APT repositories Fabian Ebner
` (2 subsequent siblings)
10 siblings, 0 replies; 13+ messages in thread
From: Fabian Ebner @ 2021-04-02 11:20 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v3:
* adapt to new library interface
* check against previous digest if provided
src/api2/node/apt.rs | 68 +++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 67 insertions(+), 1 deletion(-)
diff --git a/src/api2/node/apt.rs b/src/api2/node/apt.rs
index c91d03e1..91b159cd 100644
--- a/src/api2/node/apt.rs
+++ b/src/api2/node/apt.rs
@@ -7,7 +7,7 @@ use proxmox::list_subdirs_api_method;
use proxmox::api::{api, RpcEnvironment, RpcEnvironmentType, Permission};
use proxmox::api::router::{Router, SubdirMap};
-use proxmox_apt::types::{APTRepositoryFile, APTRepositoryFileError};
+use proxmox_apt::types::{APTRepositoryFile, APTRepositoryFileError, APTRepositoryInfo};
use crate::server::WorkerTask;
use crate::tools::{apt, http, subscription};
@@ -376,6 +376,71 @@ fn repositories_common_digest(files: &Vec<APTRepositoryFile>) -> [u8; 32] {
openssl::sha::sha256(&common_raw[..])
}
+#[api(
+ input: {
+ properties: {
+ node: {
+ schema: NODE_SCHEMA,
+ },
+ digest: {
+ schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+ optional: true,
+ },
+ },
+ },
+ returns: {
+ type: Object,
+ description: "Additional sanity checks for the configured APT repositories.",
+ properties: {
+ warnings: {
+ description: "Additional information/warnings for APT repositories.",
+ type: Array,
+ items: {
+ type: APTRepositoryInfo,
+ },
+ },
+ enterprise: {
+ description: "Whether the enterprise repository is enabled or not.",
+ type: Boolean,
+ },
+ nosubscription: {
+ description: "Whether the no-subscription repository is enabled or not.",
+ type: Boolean,
+ },
+ },
+ },
+ access: {
+ permission: &Permission::Privilege(&[], PRIV_SYS_AUDIT, false),
+ },
+)]
+/// Additional sanity checks for the configured APT repositories.
+pub fn check_repositories(digest: Option<String>) -> Result<Value, Error> {
+ let (files, _) = proxmox_apt::repositories::repositories()?;
+
+ if files.len() == 0 {
+ bail!("no APT repository files could be parsed!");
+ }
+
+ if let Some(digest) = digest {
+ let expected_digest = proxmox::tools::hex_to_digest(&digest)?;
+ let current_digest = repositories_common_digest(&files);
+ crate::tools::detect_modified_configuration_file(¤t_digest, &expected_digest)?;
+ }
+
+ let infos = proxmox_apt::repositories::check_repositories(&files);
+
+ let enterprise_enabled =
+ proxmox_apt::repositories::enterprise_repository_enabled(&files, "pbs");
+ let no_subscription_enabled =
+ proxmox_apt::repositories::no_subscription_repository_enabled(&files, "pbs");
+
+ Ok(json!({
+ "infos": infos,
+ "enterprise": enterprise_enabled,
+ "nosubscription": no_subscription_enabled
+ }))
+}
+
#[api(
input: {
properties: {
@@ -432,6 +497,7 @@ pub fn get_repositories() -> Result<Value, Error> {
const SUBDIRS: SubdirMap = &[
("changelog", &Router::new().get(&API_METHOD_APT_GET_CHANGELOG)),
+ ("checkrepositories", &Router::new().get(&API_METHOD_CHECK_REPOSITORIES)),
("repositories", &Router::new().get(&API_METHOD_GET_REPOSITORIES)),
("update", &Router::new()
.get(&API_METHOD_APT_UPDATE_AVAILABLE)
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* [pbs-devel] [PATCH v4 widget-toolkit 09/10] add UI for APT repositories
2021-04-02 11:20 [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
` (7 preceding siblings ...)
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-backup 08/10] api: apt: add check_repositories_call Fabian Ebner
@ 2021-04-02 11:20 ` Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 widget-toolkit 10/10] APT repositories: add warnings Fabian Ebner
2021-05-10 5:54 ` [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
10 siblings, 0 replies; 13+ messages in thread
From: Fabian Ebner @ 2021-04-02 11:20 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v3:
* adapt to new API behavior, add panel for the per-file errors
* use <br> instead of '\n', to have more readable .sources options
* don't reload() on init, but only on activation
* better variable names in renderers
src/Makefile | 1 +
src/node/APTRepositories.js | 252 ++++++++++++++++++++++++++++++++++++
2 files changed, 253 insertions(+)
create mode 100644 src/node/APTRepositories.js
diff --git a/src/Makefile b/src/Makefile
index 44c11ea..dd3f1f9 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -65,6 +65,7 @@ JSSRC= \
window/ACMEPluginEdit.js \
window/ACMEDomains.js \
node/APT.js \
+ node/APTRepositories.js \
node/NetworkEdit.js \
node/NetworkView.js \
node/DNSEdit.js \
diff --git a/src/node/APTRepositories.js b/src/node/APTRepositories.js
new file mode 100644
index 0000000..0ad034c
--- /dev/null
+++ b/src/node/APTRepositories.js
@@ -0,0 +1,252 @@
+Ext.define('apt-repolist', {
+ extend: 'Ext.data.Model',
+ fields: [
+ 'Path',
+ 'Number',
+ 'FileType',
+ 'Enabled',
+ 'Comment',
+ 'Types',
+ 'URIs',
+ 'Suites',
+ 'Components',
+ 'Options',
+ ],
+});
+
+Ext.define('Proxmox.node.APTRepositoriesErrors', {
+ extend: 'Ext.grid.GridPanel',
+
+ xtype: 'proxmoxNodeAPTRepositoriesErrors',
+
+ title: gettext('Errors'),
+
+ store: {},
+
+ viewConfig: {
+ stripeRows: false,
+ getRowClass: () => 'proxmox-invalid-row',
+ },
+
+ columns: [
+ {
+ header: gettext('File'),
+ dataIndex: 'path',
+ renderer: function(value, cell, record) {
+ return "<i class='pve-grid-fa fa fa-fw " +
+ "fa-exclamation-triangle'></i>" + value;
+ },
+ width: 350,
+ },
+ {
+ header: gettext('Error'),
+ dataIndex: 'error',
+ flex: 1,
+ },
+ ],
+});
+
+Ext.define('Proxmox.node.APTRepositoriesGrid', {
+ extend: 'Ext.grid.GridPanel',
+
+ xtype: 'proxmoxNodeAPTRepositoriesGrid',
+
+ title: gettext('APT Repositories'),
+
+ sortableColumns: false,
+
+ columns: [
+ {
+ header: gettext('Enabled'),
+ dataIndex: 'Enabled',
+ renderer: Proxmox.Utils.format_enabled_toggle,
+ width: 90,
+ },
+ {
+ header: gettext('Types'),
+ dataIndex: 'Types',
+ renderer: function(types, cell, record) {
+ return types.join(' ');
+ },
+ width: 100,
+ },
+ {
+ header: gettext('URIs'),
+ dataIndex: 'URIs',
+ renderer: function(uris, cell, record) {
+ return uris.join(' ');
+ },
+ width: 350,
+ },
+ {
+ header: gettext('Suites'),
+ dataIndex: 'Suites',
+ renderer: function(suites, cell, record) {
+ return suites.join(' ');
+ },
+ width: 130,
+ },
+ {
+ header: gettext('Components'),
+ dataIndex: 'Components',
+ renderer: function(components, cell, record) {
+ return components.join(' ');
+ },
+ width: 170,
+ },
+ {
+ header: gettext('Options'),
+ dataIndex: 'Options',
+ renderer: function(options, cell, record) {
+ if (!options) {
+ return '';
+ }
+
+ let filetype = record.data.FileType;
+ let text = '';
+
+ options.forEach(function(option) {
+ let key = option.Key;
+ if (filetype === 'list') {
+ let values = option.Values.join(',');
+ text += `${key}=${values} `;
+ } else if (filetype === 'sources') {
+ let values = option.Values.join(' ');
+ text += `${key}: ${values}<br>`;
+ } else {
+ throw "unkown file type";
+ }
+ });
+ return text;
+ },
+ flex: 1,
+ },
+ {
+ header: gettext('Comment'),
+ dataIndex: 'Comment',
+ flex: 2,
+ },
+ ],
+
+ initComponent: function() {
+ let me = this;
+
+ let store = Ext.create('Ext.data.Store', {
+ model: 'apt-repolist',
+ groupField: 'Path',
+ sorters: [
+ {
+ property: 'Number',
+ direction: 'ASC',
+ },
+ ],
+ });
+
+ let groupingFeature = Ext.create('Ext.grid.feature.Grouping', {
+ groupHeaderTpl: '{[ "File: " + values.name ]} ({rows.length} ' +
+ 'repositor{[values.rows.length > 1 ? "ies" : "y"]})',
+ enableGroupingMenu: false,
+ });
+
+ let sm = Ext.create('Ext.selection.RowModel', {});
+
+ Ext.apply(me, {
+ store: store,
+ selModel: sm,
+ features: [groupingFeature],
+ });
+
+ me.callParent();
+ },
+});
+
+Ext.define('Proxmox.node.APTRepositories', {
+ extend: 'Ext.panel.Panel',
+
+ xtype: 'proxmoxNodeAPTRepositories',
+
+ digest: undefined,
+
+ viewModel: {
+ data: {
+ errorCount: 0,
+ },
+ formulas: {
+ noErrors: (get) => get('errorCount') === 0,
+ },
+ },
+
+ items: [
+ {
+ xtype: 'proxmoxNodeAPTRepositoriesErrors',
+ name: 'repositoriesErrors',
+ hidden: true,
+ bind: {
+ hidden: '{noErrors}',
+ },
+ },
+ {
+ xtype: 'proxmoxNodeAPTRepositoriesGrid',
+ name: 'repositoriesGrid',
+ },
+ ],
+
+ listeners: {
+ activate: function() {
+ let me = this;
+ let vm = me.getViewModel();
+ let repoGrid = me.down('proxmoxNodeAPTRepositoriesGrid');
+ let errorGrid = me.down('proxmoxNodeAPTRepositoriesErrors');
+
+ me.store.load(function(records, operation, success) {
+ let gridData = [];
+ let errors = [];
+ let digest;
+
+ if (success && records.length > 0) {
+ let data = records[0].data;
+ let files = data.files;
+ errors = data.errors;
+ digest = data.digest;
+
+ files.forEach(function(file) {
+ for (let n = 0; n < file.repositories.length; n++) {
+ let repo = file.repositories[n];
+ repo.Path = file.path;
+ repo.Number = n + 1;
+ gridData.push(repo);
+ }
+ });
+ }
+
+ me.digest = digest;
+
+ repoGrid.store.loadData(gridData);
+
+ vm.set('errorCount', errors.length);
+ errorGrid.store.loadData(errors);
+ });
+ },
+ },
+
+ initComponent: function() {
+ let me = this;
+
+ if (!me.nodename) {
+ throw "no node name specified";
+ }
+
+ let store = Ext.create('Ext.data.Store', {
+ proxy: {
+ type: 'proxmox',
+ url: `/api2/json/nodes/${me.nodename}/apt/repositories`,
+ },
+ });
+
+ Ext.apply(me, { store: store });
+
+ Proxmox.Utils.monStoreErrors(me, me.store, true);
+
+ me.callParent();
+ },
+});
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* [pbs-devel] [PATCH v4 widget-toolkit 10/10] APT repositories: add warnings
2021-04-02 11:20 [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
` (8 preceding siblings ...)
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 widget-toolkit 09/10] add UI for APT repositories Fabian Ebner
@ 2021-04-02 11:20 ` Fabian Ebner
2021-05-10 5:54 ` [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
10 siblings, 0 replies; 13+ messages in thread
From: Fabian Ebner @ 2021-04-02 11:20 UTC (permalink / raw)
To: pbs-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v3:
* adapt to API changes, handle the special "ignore-pre-upgrade" warning and
show when a repository has an offical URI
* use digest for checkrepositories call
* let the main panel be the target for monStoreErrrors
* The main warning didn't look good as a label, especially when
there are also unparsable files. Having it as a panel title felt
more uniform and prominent, but I'm open for alternatives.
src/node/APTRepositories.js | 167 +++++++++++++++++++++++++++++++++++-
1 file changed, 165 insertions(+), 2 deletions(-)
diff --git a/src/node/APTRepositories.js b/src/node/APTRepositories.js
index 0ad034c..a2ff0fe 100644
--- a/src/node/APTRepositories.js
+++ b/src/node/APTRepositories.js
@@ -3,6 +3,7 @@ Ext.define('apt-repolist', {
fields: [
'Path',
'Number',
+ 'OfficialHost',
'FileType',
'Enabled',
'Comment',
@@ -56,6 +57,22 @@ Ext.define('Proxmox.node.APTRepositoriesGrid', {
sortableColumns: false,
columns: [
+ {
+ header: gettext('Official'),
+ dataIndex: 'OfficialHost',
+ renderer: function(value, cell, record) {
+ let icon = (cls) => `<i class="fa fa-fw ${cls}"></i>`;
+
+ if (value === undefined || value === null) {
+ return icon('fa-question-circle-o');
+ }
+ if (!value) {
+ return icon('fa-times critical');
+ }
+ return icon('fa-check good');
+ },
+ width: 70,
+ },
{
header: gettext('Enabled'),
dataIndex: 'Enabled',
@@ -128,9 +145,87 @@ Ext.define('Proxmox.node.APTRepositoriesGrid', {
},
],
+ check_repositories: function(gridData) {
+ let me = this;
+ let panel = me.up('proxmoxNodeAPTRepositories');
+ let vm = panel.getViewModel();
+
+ Proxmox.Utils.API2Request({
+ url: `/nodes/${me.nodename}/apt/checkrepositories`,
+ method: 'GET',
+ params: {
+ digest: panel.digest,
+ },
+ failure: function(response, opts) {
+ me.rowBodyFeature.getAdditionalData = function() {
+ return {
+ rowBody: undefined,
+ rowBodyCls: Ext.baseCSSPrefix + 'grid-row-body-hidden',
+ };
+ };
+ me.store.loadData(gridData);
+ Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+ },
+ success: function(response, opts) {
+ const data = response.result.data;
+
+ vm.set('enterpriseRepo', data.enterprise);
+ vm.set('noSubscriptionRepo', data.nosubscription);
+
+ let warnings = {};
+ let officialHosts = {};
+
+ let addLine = function(obj, key, line) {
+ if (obj[key]) {
+ obj[key] += "\n";
+ obj[key] += line;
+ } else {
+ obj[key] = line;
+ }
+ };
+
+ for (const info of data.infos) {
+ const key = `${info.path}:${info.number}`;
+ if (info.kind === 'warning' ||
+ (info.kind === 'ignore-pre-upgrade-warning' && !me.majorUpgradeAllowed)) {
+ addLine(warnings, key, gettext('Warning') + ": " + info.message);
+ } else if (info.kind === 'badge' && info.message === 'official host name') {
+ officialHosts[key] = true;
+ }
+ }
+
+ gridData.forEach(function(record) {
+ const key = `${record.Path}:${record.Number}`;
+ record.OfficialHost = !!officialHosts[key];
+ });
+
+ me.rowBodyFeature.getAdditionalData = function(innerData, rowIndex, record, orig) {
+ let headerCt = this.view.headerCt;
+ let colspan = headerCt.getColumnCount();
+
+ const key = `${innerData.Path}:${innerData.Number}`;
+ const warning_text = warnings[key];
+
+ return {
+ rowBody: '<div style="color: red; white-space: pre-line">' +
+ Ext.String.htmlEncode(warning_text) + '</div>',
+ rowBodyCls: warning_text ? '' : Ext.baseCSSPrefix + 'grid-row-body-hidden',
+ rowBodyColspan: colspan,
+ };
+ };
+
+ me.store.loadData(gridData);
+ },
+ });
+ },
+
initComponent: function() {
let me = this;
+ if (!me.nodename) {
+ throw "no node name specified";
+ }
+
let store = Ext.create('Ext.data.Store', {
model: 'apt-repolist',
groupField: 'Path',
@@ -142,6 +237,8 @@ Ext.define('Proxmox.node.APTRepositoriesGrid', {
],
});
+ let rowBodyFeature = Ext.create('Ext.grid.feature.RowBody', {});
+
let groupingFeature = Ext.create('Ext.grid.feature.Grouping', {
groupHeaderTpl: '{[ "File: " + values.name ]} ({rows.length} ' +
'repositor{[values.rows.length > 1 ? "ies" : "y"]})',
@@ -153,7 +250,8 @@ Ext.define('Proxmox.node.APTRepositoriesGrid', {
Ext.apply(me, {
store: store,
selModel: sm,
- features: [groupingFeature],
+ rowBodyFeature: rowBodyFeature,
+ features: [groupingFeature, rowBodyFeature],
});
me.callParent();
@@ -164,19 +262,59 @@ Ext.define('Proxmox.node.APTRepositories', {
extend: 'Ext.panel.Panel',
xtype: 'proxmoxNodeAPTRepositories',
+ mixins: ['Proxmox.Mixin.CBind'],
digest: undefined,
viewModel: {
data: {
errorCount: 0,
+ subscriptionActive: '',
+ noSubscriptionRepo: '',
+ enterpriseRepo: '',
},
formulas: {
noErrors: (get) => get('errorCount') === 0,
+ mainWarning: function(get) {
+ if (get('subscriptionActive') === '' ||
+ get('noSubscriptionRepo') === '' ||
+ get('enterpriseRepo') === '') {
+ return '';
+ }
+
+ let withStyle = (msg) => "<div style='color:red;'><i class='fa fa-fw " +
+ "fa-exclamation-triangle'></i>" + gettext('Warning') + ': ' + msg + "</div>";
+
+ if (!get('subscriptionActive') && get('enterpriseRepo')) {
+ return withStyle(gettext('The enterprise repository is ' +
+ 'configured, but there is no active subscription!'));
+ }
+
+ if (get('noSubscriptionRepo')) {
+ return withStyle(gettext('The no-subscription repository is ' +
+ 'not recommended for production use!'));
+ }
+
+ if (!get('enterpriseRepo') && !get('noSubscriptionRepo')) {
+ return withStyle(gettext('Neither the enterprise repository ' +
+ 'nor the no-subscription repository is configured!'));
+ }
+
+ return '';
+ },
},
},
items: [
+ {
+ title: gettext('Warning'),
+ name: 'repositoriesMainWarning',
+ xtype: 'panel',
+ bind: {
+ title: '{mainWarning}',
+ hidden: '{!mainWarning}',
+ },
+ },
{
xtype: 'proxmoxNodeAPTRepositoriesErrors',
name: 'repositoriesErrors',
@@ -188,9 +326,32 @@ Ext.define('Proxmox.node.APTRepositories', {
{
xtype: 'proxmoxNodeAPTRepositoriesGrid',
name: 'repositoriesGrid',
+ cbind: {
+ nodename: '{nodename}',
+ majorUpgradeAllowed: '{majorUpgradeAllowed}',
+ },
},
],
+ check_subscription: function() {
+ let me = this;
+ let vm = me.getViewModel();
+
+ Proxmox.Utils.API2Request({
+ url: `/nodes/${me.nodename}/subscription`,
+ method: 'GET',
+ failure: function(response, opts) {
+ Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+ },
+ success: function(response, opts) {
+ const res = response.result;
+ const subscription = !(res === null || res === undefined ||
+ !res || res.data.status.toLowerCase() !== 'active');
+ vm.set('subscriptionActive', subscription);
+ },
+ });
+ },
+
listeners: {
activate: function() {
let me = this;
@@ -221,11 +382,13 @@ Ext.define('Proxmox.node.APTRepositories', {
me.digest = digest;
- repoGrid.store.loadData(gridData);
+ repoGrid.check_repositories(gridData); // loads gridData after updating it
vm.set('errorCount', errors.length);
errorGrid.store.loadData(errors);
});
+
+ me.check_subscription();
},
},
--
2.20.1
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI
2021-04-02 11:20 [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
` (9 preceding siblings ...)
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 widget-toolkit 10/10] APT repositories: add warnings Fabian Ebner
@ 2021-05-10 5:54 ` Fabian Ebner
2021-05-10 13:22 ` Thomas Lamprecht
10 siblings, 1 reply; 13+ messages in thread
From: Fabian Ebner @ 2021-05-10 5:54 UTC (permalink / raw)
To: pbs-devel
Ping
Am 02.04.21 um 13:20 schrieb Fabian Ebner:
> List the configured repositories and have some basic checks for them.
>
> The plan is to use perlmod to make the Rust implementation available for PVE+PMG
> as well.
>
>
> Changes from v3:
> * incorporate Fabian G.'s feedback:
> * switch to a per-file approach
> * check for official host names
> * fix case-sensitivity issue for .sources keys
> * include digests
> * fix write issue when there are no components (in case of an absolute suite)
> * add more tests
>
>
> Still missing (intended as followups):
> * Upgrade suite/distribuiton button to be used before major release
> upgrades (but it's really simply to add that now).
> * perlmod magic and integration in PVE and PMG.
>
>
> Changes v2 -> v3:
> * incorporate Wolfgang's feedback
> * improve main warning's UI
>
> Changes v1 -> v2:
> * Perl -> Rust
> * PVE -> PBS
> * Don't rely on regexes for parsing.
> * Add writer and tests.
> * UI: pin warnings to the repository they're for.
> * Keep order of options consistent with configuration.
> * Smaller things noted on the individual patches.
>
> proxmox-apt:
>
> Fabian Ebner (4):
> initial commit
> add files for Debian packaging
> add functions to check for Proxmox repositories
> add check_repositories function
>
>
> proxmox-backup:
>
> Fabian Ebner (4):
> depend on new proxmox-apt crate
> api: apt: add repositories call
> ui: add panel for APT repositories
> api: apt: add check_repositories_call
>
> Cargo.toml | 1 +
> debian/control | 1 +
> src/api2/node/apt.rs | 149 +++++++++++++++++++++++++++++++++++-
> www/ServerAdministration.js | 8 ++
> 4 files changed, 158 insertions(+), 1 deletion(-)
>
>
> proxmox-widget-toolkit:
>
> Fabian Ebner (2):
> add UI for APT repositories
> APT repositories: add warnings
>
> src/Makefile | 1 +
> src/node/APTRepositories.js | 415 ++++++++++++++++++++++++++++++++++++
> 2 files changed, 416 insertions(+)
> create mode 100644 src/node/APTRepositories.js
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI
2021-05-10 5:54 ` [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
@ 2021-05-10 13:22 ` Thomas Lamprecht
0 siblings, 0 replies; 13+ messages in thread
From: Thomas Lamprecht @ 2021-05-10 13:22 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Ebner
On 10.05.21 07:54, Fabian Ebner wrote:
> Ping
>
I have this series on my todo list, no worries. Independent of review outcome:
I do not plan to apply this now, but at earliest for when master is set at a
2.0/7.0 release.
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2021-05-10 13:22 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-02 11:20 [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 01/10] initial commit Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 02/10] add files for Debian packaging Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 03/10] add functions to check for Proxmox repositories Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-apt 04/10] add check_repositories function Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-backup 05/10] depend on new proxmox-apt crate Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-backup 06/10] api: apt: add repositories call Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-backup 07/10] ui: add panel for APT repositories Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 proxmox-backup 08/10] api: apt: add check_repositories_call Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 widget-toolkit 09/10] add UI for APT repositories Fabian Ebner
2021-04-02 11:20 ` [pbs-devel] [PATCH v4 widget-toolkit 10/10] APT repositories: add warnings Fabian Ebner
2021-05-10 5:54 ` [pbs-devel] [PATCH-SERIES v4] APT repositories API/UI Fabian Ebner
2021-05-10 13:22 ` Thomas Lamprecht
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal