* [pve-devel] [PATCH v7 proxmox-apt 1/5] initial commit
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
@ 2021-06-23 13:38 ` Fabian Ebner
2021-06-23 13:38 ` [pve-devel] [PATCH v7 proxmox-apt 2/5] add files for Debian packaging Fabian Ebner
` (11 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Fabian Ebner @ 2021-06-23 13:38 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v6:
* make common_digest helper private and return the common digest together
with the parsed repository files.
* keep impl blocks and type declarations together
* base tests on bullseye
* ignore file extensions ignored by APT (e.g. ".orig")
* file.write() now removes a file if no repositories are left
* get rid of write_repositories() function: It's just a loop over
file.write() and returning a Vec<> of errors is akward. Just let the
caller iterate themselves and decide how to react when an error occurs.
* prefer kebab-case over lowercase (only affects one parameter, 'file-type')
.cargo/config | 5 +
.gitignore | 4 +
Cargo.toml | 23 ++
rustfmt.toml | 1 +
src/lib.rs | 1 +
src/repositories/file.rs | 274 ++++++++++++++
src/repositories/file/list_parser.rs | 172 +++++++++
src/repositories/file/sources_parser.rs | 204 ++++++++++
src/repositories/mod.rs | 107 ++++++
src/repositories/repository.rs | 353 ++++++++++++++++++
tests/repositories.rs | 162 ++++++++
.../absolute_suite.list | 5 +
.../absolute_suite.sources | 5 +
tests/sources.list.d.expected/case.sources | 16 +
.../sources.list.d.expected/multiline.sources | 10 +
| 6 +
.../pbs-enterprise.list | 2 +
tests/sources.list.d.expected/pve.list | 13 +
tests/sources.list.d.expected/standard.list | 7 +
.../sources.list.d.expected/standard.sources | 11 +
tests/sources.list.d/absolute_suite.list | 4 +
tests/sources.list.d/absolute_suite.sources | 5 +
tests/sources.list.d/case.sources | 17 +
tests/sources.list.d/multiline.sources | 11 +
| 3 +
tests/sources.list.d/pbs-enterprise.list | 1 +
tests/sources.list.d/pve.list | 10 +
tests/sources.list.d/standard.list | 6 +
tests/sources.list.d/standard.sources | 10 +
29 files changed, 1448 insertions(+)
create mode 100644 .cargo/config
create mode 100644 .gitignore
create mode 100644 Cargo.toml
create mode 100644 rustfmt.toml
create mode 100644 src/lib.rs
create mode 100644 src/repositories/file.rs
create mode 100644 src/repositories/file/list_parser.rs
create mode 100644 src/repositories/file/sources_parser.rs
create mode 100644 src/repositories/mod.rs
create mode 100644 src/repositories/repository.rs
create mode 100644 tests/repositories.rs
create mode 100644 tests/sources.list.d.expected/absolute_suite.list
create mode 100644 tests/sources.list.d.expected/absolute_suite.sources
create mode 100644 tests/sources.list.d.expected/case.sources
create mode 100644 tests/sources.list.d.expected/multiline.sources
create mode 100644 tests/sources.list.d.expected/options_comment.list
create mode 100644 tests/sources.list.d.expected/pbs-enterprise.list
create mode 100644 tests/sources.list.d.expected/pve.list
create mode 100644 tests/sources.list.d.expected/standard.list
create mode 100644 tests/sources.list.d.expected/standard.sources
create mode 100644 tests/sources.list.d/absolute_suite.list
create mode 100644 tests/sources.list.d/absolute_suite.sources
create mode 100644 tests/sources.list.d/case.sources
create mode 100644 tests/sources.list.d/multiline.sources
create mode 100644 tests/sources.list.d/options_comment.list
create mode 100644 tests/sources.list.d/pbs-enterprise.list
create mode 100644 tests/sources.list.d/pve.list
create mode 100644 tests/sources.list.d/standard.list
create mode 100644 tests/sources.list.d/standard.sources
diff --git a/.cargo/config b/.cargo/config
new file mode 100644
index 0000000..3b5b6e4
--- /dev/null
+++ b/.cargo/config
@@ -0,0 +1,5 @@
+[source]
+[source.debian-packages]
+directory = "/usr/share/cargo/registry"
+[source.crates-io]
+replace-with = "debian-packages"
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..24917d4
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,4 @@
+Cargo.lock
+target/
+tests/sources.list.d.actual
+tests/sources.list.d.digest
diff --git a/Cargo.toml b/Cargo.toml
new file mode 100644
index 0000000..24f734b
--- /dev/null
+++ b/Cargo.toml
@@ -0,0 +1,23 @@
+[package]
+name = "proxmox-apt"
+version = "0.1.0"
+authors = [
+ "Fabian Ebner <f.ebner@proxmox.com>",
+ "Proxmox Support Team <support@proxmox.com>",
+]
+edition = "2018"
+license = "AGPL-3"
+description = "Proxmox library for APT"
+homepage = "https://www.proxmox.com"
+
+exclude = [ "debian" ]
+
+[lib]
+name = "proxmox_apt"
+path = "src/lib.rs"
+
+[dependencies]
+anyhow = "1.0"
+openssl = "0.10"
+proxmox = { version = "0.11.5", features = [ "api-macro" ] }
+serde = { version = "1.0", features = ["derive"] }
diff --git a/rustfmt.toml b/rustfmt.toml
new file mode 100644
index 0000000..32a9786
--- /dev/null
+++ b/rustfmt.toml
@@ -0,0 +1 @@
+edition = "2018"
diff --git a/src/lib.rs b/src/lib.rs
new file mode 100644
index 0000000..21b552a
--- /dev/null
+++ b/src/lib.rs
@@ -0,0 +1 @@
+pub mod repositories;
diff --git a/src/repositories/file.rs b/src/repositories/file.rs
new file mode 100644
index 0000000..bc48bf2
--- /dev/null
+++ b/src/repositories/file.rs
@@ -0,0 +1,274 @@
+use std::convert::TryFrom;
+use std::fmt::Display;
+use std::path::{Path, PathBuf};
+
+use anyhow::{format_err, Error};
+use serde::{Deserialize, Serialize};
+
+use crate::repositories::repository::{APTRepository, APTRepositoryFileType};
+
+use proxmox::api::api;
+
+mod list_parser;
+use list_parser::APTListFileParser;
+
+mod sources_parser;
+use sources_parser::APTSourcesFileParser;
+
+trait APTRepositoryParser {
+ /// Parse all repositories including the disabled ones and push them onto
+ /// the provided vector.
+ fn parse_repositories(&mut self) -> Result<Vec<APTRepository>, Error>;
+}
+
+#[api(
+ properties: {
+ "file-type": {
+ type: APTRepositoryFileType,
+ },
+ repositories: {
+ description: "List of APT repositories.",
+ type: Array,
+ items: {
+ type: APTRepository,
+ },
+ },
+ digest: {
+ description: "Digest for the content of the file.",
+ optional: true,
+ type: Array,
+ items: {
+ description: "Digest byte.",
+ type: Integer,
+ },
+ },
+ },
+)]
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "kebab-case")]
+/// Represents an abstract APT repository file.
+pub struct APTRepositoryFile {
+ /// The path to the file.
+ pub path: String,
+
+ /// The type of the file.
+ pub file_type: APTRepositoryFileType,
+
+ /// List of repositories in the file.
+ pub repositories: Vec<APTRepository>,
+
+ /// Digest of the original contents.
+ pub digest: Option<[u8; 32]>,
+}
+
+#[api]
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "kebab-case")]
+/// Error type for problems with APT repository files.
+pub struct APTRepositoryFileError {
+ /// The path to the problematic file.
+ pub path: String,
+
+ /// The error message.
+ pub error: String,
+}
+
+impl Display for APTRepositoryFileError {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ write!(f, "proxmox-apt error for '{}' - {}", self.path, self.error)
+ }
+}
+
+impl std::error::Error for APTRepositoryFileError {
+ fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
+ None
+ }
+}
+
+impl APTRepositoryFile {
+ /// Creates a new `APTRepositoryFile` without parsing.
+ ///
+ /// If the file is hidden, the path points to a directory, or the extension
+ /// is usually ignored by APT (e.g. `.orig`), `Ok(None)` is returned, while
+ /// invalid file names yield an error.
+ pub fn new<P: AsRef<Path>>(path: P) -> Result<Option<Self>, APTRepositoryFileError> {
+ let path: PathBuf = path.as_ref().to_path_buf();
+
+ let new_err = |path_string: String, err: &str| APTRepositoryFileError {
+ path: path_string,
+ error: err.to_string(),
+ };
+
+ let path_string = path
+ .clone()
+ .into_os_string()
+ .into_string()
+ .map_err(|os_string| {
+ new_err(
+ os_string.to_string_lossy().to_string(),
+ "path is not valid unicode",
+ )
+ })?;
+
+ let new_err = |err| new_err(path_string.clone(), err);
+
+ if path.is_dir() {
+ return Ok(None);
+ }
+
+ let file_name = match path.file_name() {
+ Some(file_name) => file_name
+ .to_os_string()
+ .into_string()
+ .map_err(|_| new_err("invalid path"))?,
+ None => return Err(new_err("invalid path")),
+ };
+
+ if file_name.starts_with('.') || file_name.ends_with('~') {
+ return Ok(None);
+ }
+
+ let extension = match path.extension() {
+ Some(extension) => extension
+ .to_os_string()
+ .into_string()
+ .map_err(|_| new_err("invalid path"))?,
+ None => return Err(new_err("invalid extension")),
+ };
+
+ // See APT's apt-pkg/init.cc
+ if extension.starts_with("dpkg-")
+ || extension.starts_with("ucf-")
+ || matches!(
+ extension.as_str(),
+ "disabled" | "bak" | "save" | "orig" | "distUpgrade"
+ )
+ {
+ return Ok(None);
+ }
+
+ let file_type = APTRepositoryFileType::try_from(&extension[..])
+ .map_err(|_| new_err("invalid extension"))?;
+
+ if !file_name
+ .chars()
+ .all(|x| x.is_ascii_alphanumeric() || x == '_' || x == '-' || x == '.')
+ {
+ return Err(new_err("invalid characters in file name"));
+ }
+
+ Ok(Some(Self {
+ path: path_string,
+ file_type,
+ repositories: vec![],
+ digest: None,
+ }))
+ }
+
+ /// Check if the file exists.
+ pub fn exists(&self) -> bool {
+ PathBuf::from(&self.path).exists()
+ }
+
+ pub fn read_with_digest(&self) -> Result<(Vec<u8>, [u8; 32]), APTRepositoryFileError> {
+ let content = std::fs::read(&self.path).map_err(|err| self.err(format_err!("{}", err)))?;
+
+ let digest = openssl::sha::sha256(&content);
+
+ Ok((content, digest))
+ }
+
+ /// Create an `APTRepositoryFileError`.
+ pub fn err(&self, error: Error) -> APTRepositoryFileError {
+ APTRepositoryFileError {
+ path: self.path.clone(),
+ error: error.to_string(),
+ }
+ }
+
+ /// Parses the APT repositories configured in the file on disk, including
+ /// disabled ones.
+ ///
+ /// Resets the current repositories and digest, even on failure.
+ pub fn parse(&mut self) -> Result<(), APTRepositoryFileError> {
+ self.repositories.clear();
+ self.digest = None;
+
+ let (content, digest) = self.read_with_digest()?;
+
+ let mut parser: Box<dyn APTRepositoryParser> = match self.file_type {
+ APTRepositoryFileType::List => Box::new(APTListFileParser::new(&content[..])),
+ APTRepositoryFileType::Sources => Box::new(APTSourcesFileParser::new(&content[..])),
+ };
+
+ let repos = parser.parse_repositories().map_err(|err| self.err(err))?;
+
+ for (n, repo) in repos.iter().enumerate() {
+ repo.basic_check()
+ .map_err(|err| self.err(format_err!("check for repository {} - {}", n + 1, err)))?;
+ }
+
+ self.repositories = repos;
+ self.digest = Some(digest);
+
+ Ok(())
+ }
+
+ /// Writes the repositories to the file on disk.
+ ///
+ /// If a digest is provided, checks that the current content of the file still
+ /// produces the same one.
+ pub fn write(&self) -> Result<(), APTRepositoryFileError> {
+ if let Some(digest) = self.digest {
+ if !self.exists() {
+ return Err(self.err(format_err!("digest specified, but file does not exist")));
+ }
+
+ let (_, current_digest) = self.read_with_digest()?;
+ if digest != current_digest {
+ return Err(self.err(format_err!("digest mismatch")));
+ }
+ }
+
+ if self.repositories.is_empty() {
+ return std::fs::remove_file(&self.path)
+ .map_err(|err| self.err(format_err!("unable to remove file - {}", err)));
+ }
+
+ let mut content = vec![];
+
+ for (n, repo) in self.repositories.iter().enumerate() {
+ repo.basic_check()
+ .map_err(|err| self.err(format_err!("check for repository {} - {}", n + 1, err)))?;
+
+ repo.write(&mut content)
+ .map_err(|err| self.err(format_err!("writing repository {} - {}", n + 1, err)))?;
+ }
+
+ let path = PathBuf::from(&self.path);
+ let dir = match path.parent() {
+ Some(dir) => dir,
+ None => return Err(self.err(format_err!("invalid path"))),
+ };
+
+ std::fs::create_dir_all(dir)
+ .map_err(|err| self.err(format_err!("unable to create parent dir - {}", err)))?;
+
+ let pid = std::process::id();
+ let mut tmp_path = path.clone();
+ tmp_path.set_extension("tmp");
+ tmp_path.set_extension(format!("{}", pid));
+
+ if let Err(err) = std::fs::write(&tmp_path, content) {
+ let _ = std::fs::remove_file(&tmp_path);
+ return Err(self.err(format_err!("writing {:?} failed - {}", path, err)));
+ }
+
+ if let Err(err) = std::fs::rename(&tmp_path, &path) {
+ let _ = std::fs::remove_file(&tmp_path);
+ return Err(self.err(format_err!("rename failed for {:?} - {}", path, err)));
+ }
+
+ Ok(())
+ }
+}
diff --git a/src/repositories/file/list_parser.rs b/src/repositories/file/list_parser.rs
new file mode 100644
index 0000000..86c9955
--- /dev/null
+++ b/src/repositories/file/list_parser.rs
@@ -0,0 +1,172 @@
+use std::convert::TryInto;
+use std::io::BufRead;
+use std::iter::{Iterator, Peekable};
+use std::str::SplitAsciiWhitespace;
+
+use anyhow::{bail, format_err, Error};
+
+use crate::repositories::{APTRepository, APTRepositoryFileType, APTRepositoryOption};
+
+use super::APTRepositoryParser;
+
+pub struct APTListFileParser<R: BufRead> {
+ input: R,
+ line_nr: usize,
+ comment: String,
+}
+
+impl<R: BufRead> APTListFileParser<R> {
+ pub fn new(reader: R) -> Self {
+ Self {
+ input: reader,
+ line_nr: 0,
+ comment: String::new(),
+ }
+ }
+
+ /// Helper to parse options from the existing token stream.
+ ///
+ /// Also returns `Ok(())` if there are no options.
+ ///
+ /// Errors when options are invalid or not closed by `']'`.
+ fn parse_options(
+ options: &mut Vec<APTRepositoryOption>,
+ tokens: &mut Peekable<SplitAsciiWhitespace>,
+ ) -> Result<(), Error> {
+ let mut option = match tokens.peek() {
+ Some(token) => {
+ match token.strip_prefix('[') {
+ Some(option) => option,
+ None => return Ok(()), // doesn't look like options
+ }
+ }
+ None => return Ok(()),
+ };
+
+ tokens.next(); // avoid reading the beginning twice
+
+ let mut finished = false;
+ loop {
+ if let Some(stripped) = option.strip_suffix(']') {
+ option = stripped;
+ if option.is_empty() {
+ break;
+ }
+ finished = true; // but still need to handle the last one
+ };
+
+ if let Some(mid) = option.find('=') {
+ let (key, mut value_str) = option.split_at(mid);
+ value_str = &value_str[1..];
+
+ if key.is_empty() {
+ bail!("option has no key: '{}'", option);
+ }
+
+ if value_str.is_empty() {
+ bail!("option has no value: '{}'", option);
+ }
+
+ let values: Vec<String> = value_str
+ .split(',')
+ .map(|value| value.to_string())
+ .collect();
+
+ options.push(APTRepositoryOption {
+ key: key.to_string(),
+ values,
+ });
+ } else if !option.is_empty() {
+ bail!("got invalid option - '{}'", option);
+ }
+
+ if finished {
+ break;
+ }
+
+ option = match tokens.next() {
+ Some(option) => option,
+ None => bail!("options not closed by ']'"),
+ }
+ }
+
+ Ok(())
+ }
+
+ /// Parse a repository or comment in one-line format.
+ ///
+ /// Commented out repositories are also detected and returned with the
+ /// `enabled` property set to `false`.
+ ///
+ /// If the line contains a repository, `self.comment` is added to the
+ /// `comment` property.
+ ///
+ /// If the line contains a comment, it is added to `self.comment`.
+ fn parse_one_line(&mut self, mut line: &str) -> Result<Option<APTRepository>, Error> {
+ line = line.trim_matches(|c| char::is_ascii_whitespace(&c));
+
+ // check for commented out repository first
+ if let Some(commented_out) = line.strip_prefix('#') {
+ if let Ok(Some(mut repo)) = self.parse_one_line(commented_out) {
+ repo.set_enabled(false);
+ return Ok(Some(repo));
+ }
+ }
+
+ let mut repo = APTRepository::new(APTRepositoryFileType::List);
+
+ // now handle "real" comment
+ if let Some(comment_start) = line.find('#') {
+ let (line_start, comment) = line.split_at(comment_start);
+ self.comment = format!("{}{}\n", self.comment, &comment[1..]);
+ line = line_start;
+ }
+
+ let mut tokens = line.split_ascii_whitespace().peekable();
+
+ match tokens.next() {
+ Some(package_type) => {
+ repo.types.push(package_type.try_into()?);
+ }
+ None => return Ok(None), // empty line
+ }
+
+ Self::parse_options(&mut repo.options, &mut tokens)?;
+
+ // the rest of the line is just '<uri> <suite> [<components>...]'
+ let mut tokens = tokens.map(str::to_string);
+ repo.uris
+ .push(tokens.next().ok_or_else(|| format_err!("missing URI"))?);
+ repo.suites
+ .push(tokens.next().ok_or_else(|| format_err!("missing suite"))?);
+ repo.components.extend(tokens);
+
+ repo.comment = std::mem::take(&mut self.comment);
+
+ Ok(Some(repo))
+ }
+}
+
+impl<R: BufRead> APTRepositoryParser for APTListFileParser<R> {
+ fn parse_repositories(&mut self) -> Result<Vec<APTRepository>, Error> {
+ let mut repos = vec![];
+ let mut line = String::new();
+
+ loop {
+ self.line_nr += 1;
+ line.clear();
+
+ match self.input.read_line(&mut line) {
+ Err(err) => bail!("input error - {}", err),
+ Ok(0) => break,
+ Ok(_) => match self.parse_one_line(&line) {
+ Ok(Some(repo)) => repos.push(repo),
+ Ok(None) => continue,
+ Err(err) => bail!("malformed entry on line {} - {}", self.line_nr, err),
+ },
+ }
+ }
+
+ Ok(repos)
+ }
+}
diff --git a/src/repositories/file/sources_parser.rs b/src/repositories/file/sources_parser.rs
new file mode 100644
index 0000000..e824f3d
--- /dev/null
+++ b/src/repositories/file/sources_parser.rs
@@ -0,0 +1,204 @@
+use std::convert::TryInto;
+use std::io::BufRead;
+use std::iter::Iterator;
+
+use anyhow::{bail, Error};
+
+use crate::repositories::{
+ APTRepository, APTRepositoryFileType, APTRepositoryOption, APTRepositoryPackageType,
+};
+
+use super::APTRepositoryParser;
+
+pub struct APTSourcesFileParser<R: BufRead> {
+ input: R,
+ stanza_nr: usize,
+ comment: String,
+}
+
+/// See `man sources.list` and `man deb822` for the format specification.
+impl<R: BufRead> APTSourcesFileParser<R> {
+ pub fn new(reader: R) -> Self {
+ Self {
+ input: reader,
+ stanza_nr: 1,
+ comment: String::new(),
+ }
+ }
+
+ /// Based on APT's `StringToBool` in `strutl.cc`
+ fn string_to_bool(string: &str, default: bool) -> bool {
+ let string = string.trim_matches(|c| char::is_ascii_whitespace(&c));
+ let string = string.to_lowercase();
+
+ match &string[..] {
+ "1" | "yes" | "true" | "with" | "on" | "enable" => true,
+ "0" | "no" | "false" | "without" | "off" | "disable" => false,
+ _ => default,
+ }
+ }
+
+ /// Checks if `key` is valid according to deb822
+ fn valid_key(key: &str) -> bool {
+ if key.starts_with('-') {
+ return false;
+ };
+ return key.chars().all(|c| matches!(c, '!'..='9' | ';'..='~'));
+ }
+
+ /// Try parsing a repository in stanza format from `lines`.
+ ///
+ /// Returns `Ok(None)` when no stanza can be found.
+ ///
+ /// Comments are added to `self.comments`. If a stanza can be found,
+ /// `self.comment` is added to the repository's `comment` property.
+ ///
+ /// Fully commented out stanzas are treated as comments.
+ fn parse_stanza(&mut self, lines: &str) -> Result<Option<APTRepository>, Error> {
+ let mut repo = APTRepository::new(APTRepositoryFileType::Sources);
+
+ // Values may be folded into multiple lines.
+ // Those lines have to start with a space or a tab.
+ let lines = lines.replace("\n ", " ");
+ let lines = lines.replace("\n\t", " ");
+
+ let mut got_something = false;
+
+ for line in lines.lines() {
+ let line = line.trim_matches(|c| char::is_ascii_whitespace(&c));
+ if line.is_empty() {
+ continue;
+ }
+
+ if let Some(commented_out) = line.strip_prefix('#') {
+ self.comment = format!("{}{}\n", self.comment, commented_out);
+ continue;
+ }
+
+ if let Some(mid) = line.find(':') {
+ let (key, value_str) = line.split_at(mid);
+ let value_str = &value_str[1..];
+ let key = key.trim_matches(|c| char::is_ascii_whitespace(&c));
+
+ if key.is_empty() {
+ bail!("option has no key: '{}'", line);
+ }
+
+ if value_str.is_empty() {
+ // ignored by APT
+ eprintln!("option has no value: '{}'", line);
+ continue;
+ }
+
+ if !Self::valid_key(key) {
+ // ignored by APT
+ eprintln!("option with invalid key '{}'", key);
+ continue;
+ }
+
+ let values: Vec<String> = value_str
+ .split_ascii_whitespace()
+ .map(|value| value.to_string())
+ .collect();
+
+ match &key.to_lowercase()[..] {
+ "types" => {
+ if !repo.types.is_empty() {
+ eprintln!("key 'Types' was defined twice");
+ }
+ let mut types = Vec::<APTRepositoryPackageType>::new();
+ for package_type in values {
+ types.push((&package_type[..]).try_into()?);
+ }
+ repo.types = types;
+ }
+ "uris" => {
+ if !repo.uris.is_empty() {
+ eprintln!("key 'URIs' was defined twice");
+ }
+ repo.uris = values;
+ }
+ "suites" => {
+ if !repo.suites.is_empty() {
+ eprintln!("key 'Suites' was defined twice");
+ }
+ repo.suites = values;
+ }
+ "components" => {
+ if !repo.components.is_empty() {
+ eprintln!("key 'Components' was defined twice");
+ }
+ repo.components = values;
+ }
+ "enabled" => {
+ repo.set_enabled(Self::string_to_bool(value_str, true));
+ }
+ _ => repo.options.push(APTRepositoryOption {
+ key: key.to_string(),
+ values,
+ }),
+ }
+ } else {
+ bail!("got invalid line - '{:?}'", line);
+ }
+
+ got_something = true;
+ }
+
+ if !got_something {
+ return Ok(None);
+ }
+
+ repo.comment = std::mem::take(&mut self.comment);
+
+ Ok(Some(repo))
+ }
+
+ /// Helper function for `parse_repositories`.
+ fn try_parse_stanza(
+ &mut self,
+ lines: &str,
+ repos: &mut Vec<APTRepository>,
+ ) -> Result<(), Error> {
+ match self.parse_stanza(lines) {
+ Ok(Some(repo)) => {
+ repos.push(repo);
+ self.stanza_nr += 1;
+ }
+ Ok(None) => (),
+ Err(err) => bail!("malformed entry in stanza {} - {}", self.stanza_nr, err),
+ }
+
+ Ok(())
+ }
+}
+
+impl<R: BufRead> APTRepositoryParser for APTSourcesFileParser<R> {
+ fn parse_repositories(&mut self) -> Result<Vec<APTRepository>, Error> {
+ let mut repos = vec![];
+ let mut lines = String::new();
+
+ loop {
+ let old_length = lines.len();
+ match self.input.read_line(&mut lines) {
+ Err(err) => bail!("input error - {}", err),
+ Ok(0) => {
+ self.try_parse_stanza(&lines[..], &mut repos)?;
+ break;
+ }
+ Ok(_) => {
+ if (&lines[old_length..])
+ .trim_matches(|c| char::is_ascii_whitespace(&c))
+ .is_empty()
+ {
+ // detected end of stanza
+ self.try_parse_stanza(&lines[..], &mut repos)?;
+ lines.clear();
+ }
+ }
+ }
+ }
+
+ Ok(repos)
+ }
+}
diff --git a/src/repositories/mod.rs b/src/repositories/mod.rs
new file mode 100644
index 0000000..d540bcb
--- /dev/null
+++ b/src/repositories/mod.rs
@@ -0,0 +1,107 @@
+use std::collections::BTreeMap;
+use std::path::PathBuf;
+
+use anyhow::{bail, Error};
+
+mod repository;
+pub use repository::{
+ APTRepository, APTRepositoryFileType, APTRepositoryOption, APTRepositoryPackageType,
+};
+
+mod file;
+pub use file::{APTRepositoryFile, APTRepositoryFileError};
+
+const APT_SOURCES_LIST_FILENAME: &str = "/etc/apt/sources.list";
+const APT_SOURCES_LIST_DIRECTORY: &str = "/etc/apt/sources.list.d/";
+
+/// Calculates a common digest for successfully parsed repository files.
+///
+/// The digest is invariant with respect to file order.
+///
+/// Files without a digest are ignored.
+fn common_digest(files: &[APTRepositoryFile]) -> [u8; 32] {
+ let mut digests = BTreeMap::new();
+
+ for file in files.iter() {
+ digests.insert(file.path.clone(), &file.digest);
+ }
+
+ let mut common_raw = Vec::<u8>::with_capacity(digests.len() * 32);
+ for digest in digests.values() {
+ match digest {
+ Some(digest) => common_raw.extend_from_slice(&digest[..]),
+ None => (),
+ }
+ }
+
+ openssl::sha::sha256(&common_raw[..])
+}
+
+/// Returns all APT repositories configured in `/etc/apt/sources.list` and
+/// in `/etc/apt/sources.list.d` including disabled repositories.
+///
+/// Returns the succesfully parsed files, a list of errors for files that could
+/// not be read or parsed and a common digest for the succesfully parsed files.
+///
+/// The digest is guaranteed to be set for each successfully parsed file.
+pub fn repositories() -> Result<
+ (
+ Vec<APTRepositoryFile>,
+ Vec<APTRepositoryFileError>,
+ [u8; 32],
+ ),
+ Error,
+> {
+ let to_result = |files: Vec<APTRepositoryFile>, errors: Vec<APTRepositoryFileError>| {
+ let common_digest = common_digest(&files);
+
+ (files, errors, common_digest)
+ };
+
+ let mut files = vec![];
+ let mut errors = vec![];
+
+ let sources_list_path = PathBuf::from(APT_SOURCES_LIST_FILENAME);
+
+ let sources_list_d_path = PathBuf::from(APT_SOURCES_LIST_DIRECTORY);
+
+ match APTRepositoryFile::new(sources_list_path) {
+ Ok(Some(mut file)) => match file.parse() {
+ Ok(()) => files.push(file),
+ Err(err) => errors.push(err),
+ },
+ _ => bail!("internal error with '{}'", APT_SOURCES_LIST_FILENAME),
+ }
+
+ if !sources_list_d_path.exists() {
+ return Ok(to_result(files, errors));
+ }
+
+ if !sources_list_d_path.is_dir() {
+ errors.push(APTRepositoryFileError {
+ path: APT_SOURCES_LIST_DIRECTORY.to_string(),
+ error: "not a directory!".to_string(),
+ });
+ return Ok(to_result(files, errors));
+ }
+
+ for entry in std::fs::read_dir(sources_list_d_path)? {
+ let path = entry?.path();
+
+ match APTRepositoryFile::new(path) {
+ Ok(Some(mut file)) => match file.parse() {
+ Ok(()) => {
+ if file.digest.is_none() {
+ bail!("internal error - digest not set");
+ }
+ files.push(file);
+ }
+ Err(err) => errors.push(err),
+ },
+ Ok(None) => (),
+ Err(err) => errors.push(err),
+ }
+ }
+
+ Ok(to_result(files, errors))
+}
diff --git a/src/repositories/repository.rs b/src/repositories/repository.rs
new file mode 100644
index 0000000..9ee7df9
--- /dev/null
+++ b/src/repositories/repository.rs
@@ -0,0 +1,353 @@
+use std::convert::TryFrom;
+use std::fmt::Display;
+use std::io::Write;
+
+use anyhow::{bail, Error};
+use serde::{Deserialize, Serialize};
+
+use proxmox::api::api;
+
+#[api]
+#[derive(Debug, Copy, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "lowercase")]
+pub enum APTRepositoryFileType {
+ /// One-line-style format
+ List,
+ /// DEB822-style format
+ Sources,
+}
+
+impl TryFrom<&str> for APTRepositoryFileType {
+ type Error = Error;
+
+ fn try_from(string: &str) -> Result<Self, Error> {
+ match string {
+ "list" => Ok(APTRepositoryFileType::List),
+ "sources" => Ok(APTRepositoryFileType::Sources),
+ _ => bail!("invalid file type '{}'", string),
+ }
+ }
+}
+
+impl Display for APTRepositoryFileType {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match self {
+ APTRepositoryFileType::List => write!(f, "list"),
+ APTRepositoryFileType::Sources => write!(f, "sources"),
+ }
+ }
+}
+
+#[api]
+#[derive(Debug, Copy, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+pub enum APTRepositoryPackageType {
+ /// Debian package
+ Deb,
+ /// Debian source package
+ DebSrc,
+}
+
+impl TryFrom<&str> for APTRepositoryPackageType {
+ type Error = Error;
+
+ fn try_from(string: &str) -> Result<Self, Error> {
+ match string {
+ "deb" => Ok(APTRepositoryPackageType::Deb),
+ "deb-src" => Ok(APTRepositoryPackageType::DebSrc),
+ _ => bail!("invalid package type '{}'", string),
+ }
+ }
+}
+
+impl Display for APTRepositoryPackageType {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match self {
+ APTRepositoryPackageType::Deb => write!(f, "deb"),
+ APTRepositoryPackageType::DebSrc => write!(f, "deb-src"),
+ }
+ }
+}
+
+#[api(
+ properties: {
+ Key: {
+ description: "Option key.",
+ type: String,
+ },
+ Values: {
+ description: "Option values.",
+ type: Array,
+ items: {
+ description: "Value.",
+ type: String,
+ },
+ },
+ },
+)]
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "PascalCase")] // for consistency
+/// Additional options for an APT repository.
+/// Used for both single- and mutli-value options.
+pub struct APTRepositoryOption {
+ /// Option key.
+ pub key: String,
+ /// Option value(s).
+ pub values: Vec<String>,
+}
+
+#[api(
+ properties: {
+ Types: {
+ description: "List of package types.",
+ type: Array,
+ items: {
+ type: APTRepositoryPackageType,
+ },
+ },
+ URIs: {
+ description: "List of repository URIs.",
+ type: Array,
+ items: {
+ description: "Repository URI.",
+ type: String,
+ },
+ },
+ Suites: {
+ description: "List of distributions.",
+ type: Array,
+ items: {
+ description: "Package distribution.",
+ type: String,
+ },
+ },
+ Components: {
+ description: "List of repository components.",
+ type: Array,
+ items: {
+ description: "Repository component.",
+ type: String,
+ },
+ },
+ Options: {
+ type: Array,
+ optional: true,
+ items: {
+ type: APTRepositoryOption,
+ },
+ },
+ Comment: {
+ description: "Associated comment.",
+ type: String,
+ optional: true,
+ },
+ FileType: {
+ type: APTRepositoryFileType,
+ },
+ Enabled: {
+ description: "Whether the repository is enabled or not.",
+ type: Boolean,
+ },
+ },
+)]
+#[derive(Debug, Clone, Serialize, Deserialize)]
+#[serde(rename_all = "PascalCase")]
+/// Describes an APT repository.
+pub struct APTRepository {
+ /// List of package types.
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ pub types: Vec<APTRepositoryPackageType>,
+
+ /// List of repository URIs.
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ #[serde(rename = "URIs")]
+ pub uris: Vec<String>,
+
+ /// List of package distributions.
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ pub suites: Vec<String>,
+
+ /// List of repository components.
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ pub components: Vec<String>,
+
+ /// Additional options.
+ #[serde(skip_serializing_if = "Vec::is_empty")]
+ pub options: Vec<APTRepositoryOption>,
+
+ /// Associated comment.
+ #[serde(skip_serializing_if = "String::is_empty")]
+ pub comment: String,
+
+ /// Format of the defining file.
+ pub file_type: APTRepositoryFileType,
+
+ /// Whether the repository is enabled or not.
+ pub enabled: bool,
+}
+
+impl APTRepository {
+ /// Crates an empty repository.
+ pub fn new(file_type: APTRepositoryFileType) -> Self {
+ Self {
+ types: vec![],
+ uris: vec![],
+ suites: vec![],
+ components: vec![],
+ options: vec![],
+ comment: String::new(),
+ file_type,
+ enabled: true,
+ }
+ }
+
+ /// Changes the `enabled` flag and makes sure the `Enabled` option for
+ /// `APTRepositoryPackageType::Sources` repositories is updated too.
+ pub fn set_enabled(&mut self, enabled: bool) {
+ self.enabled = enabled;
+
+ if self.file_type == APTRepositoryFileType::Sources {
+ let enabled_string = match enabled {
+ true => "true".to_string(),
+ false => "false".to_string(),
+ };
+ for option in self.options.iter_mut() {
+ if option.key == "Enabled" {
+ option.values = vec![enabled_string];
+ return;
+ }
+ }
+ self.options.push(APTRepositoryOption {
+ key: "Enabled".to_string(),
+ values: vec![enabled_string],
+ });
+ }
+ }
+
+ /// Makes sure that all basic properties of a repository are present and
+ /// not obviously invalid.
+ pub fn basic_check(&self) -> Result<(), Error> {
+ if self.types.is_empty() {
+ bail!("missing package type(s)");
+ }
+ if self.uris.is_empty() {
+ bail!("missing URI(s)");
+ }
+ if self.suites.is_empty() {
+ bail!("missing suite(s)");
+ }
+
+ for uri in self.uris.iter() {
+ if !uri.contains(':') || uri.len() < 3 {
+ bail!("invalid URI: '{}'", uri);
+ }
+ }
+
+ for suite in self.suites.iter() {
+ if !suite.ends_with('/') && self.components.is_empty() {
+ bail!("missing component(s)");
+ } else if suite.ends_with('/') && !self.components.is_empty() {
+ bail!("absolute suite '{}' does not allow component(s)", suite);
+ }
+ }
+
+ if self.file_type == APTRepositoryFileType::List {
+ if self.types.len() > 1 {
+ bail!("more than one package type");
+ }
+ if self.uris.len() > 1 {
+ bail!("more than one URI");
+ }
+ if self.suites.len() > 1 {
+ bail!("more than one suite");
+ }
+ }
+
+ Ok(())
+ }
+
+ /// Writes a repository in the corresponding format followed by a blank.
+ ///
+ /// Expects that `basic_check()` for the repository was successful.
+ pub fn write(&self, w: &mut dyn Write) -> Result<(), Error> {
+ match self.file_type {
+ APTRepositoryFileType::List => write_one_line(self, w),
+ APTRepositoryFileType::Sources => write_stanza(self, w),
+ }
+ }
+}
+
+/// Writes a repository in one-line format followed by a blank line.
+///
+/// Expects that `repo.file_type == APTRepositoryFileType::List`.
+///
+/// Expects that `basic_check()` for the repository was successful.
+fn write_one_line(repo: &APTRepository, w: &mut dyn Write) -> Result<(), Error> {
+ if repo.file_type != APTRepositoryFileType::List {
+ bail!("not a .list repository");
+ }
+
+ if !repo.comment.is_empty() {
+ for line in repo.comment.lines() {
+ writeln!(w, "#{}", line)?;
+ }
+ }
+
+ if !repo.enabled {
+ write!(w, "# ")?;
+ }
+
+ write!(w, "{} ", repo.types[0])?;
+
+ if !repo.options.is_empty() {
+ write!(w, "[ ")?;
+ repo.options
+ .iter()
+ .try_for_each(|option| write!(w, "{}={} ", option.key, option.values.join(",")))?;
+ write!(w, "] ")?;
+ };
+
+ write!(w, "{} ", repo.uris[0])?;
+ write!(w, "{} ", repo.suites[0])?;
+ writeln!(w, "{}", repo.components.join(" "))?;
+
+ writeln!(w)?;
+
+ Ok(())
+}
+
+/// Writes a single stanza followed by a blank line.
+///
+/// Expects that `repo.file_type == APTRepositoryFileType::Sources`.
+fn write_stanza(repo: &APTRepository, w: &mut dyn Write) -> Result<(), Error> {
+ if repo.file_type != APTRepositoryFileType::Sources {
+ bail!("not a .sources repository");
+ }
+
+ if !repo.comment.is_empty() {
+ for line in repo.comment.lines() {
+ writeln!(w, "#{}", line)?;
+ }
+ }
+
+ write!(w, "Types:")?;
+ repo.types
+ .iter()
+ .try_for_each(|package_type| write!(w, " {}", package_type))?;
+ writeln!(w)?;
+
+ writeln!(w, "URIs: {}", repo.uris.join(" "))?;
+ writeln!(w, "Suites: {}", repo.suites.join(" "))?;
+
+ if !repo.components.is_empty() {
+ writeln!(w, "Components: {}", repo.components.join(" "))?;
+ }
+
+ for option in repo.options.iter() {
+ writeln!(w, "{}: {}", option.key, option.values.join(" "))?;
+ }
+
+ writeln!(w)?;
+
+ Ok(())
+}
diff --git a/tests/repositories.rs b/tests/repositories.rs
new file mode 100644
index 0000000..477d718
--- /dev/null
+++ b/tests/repositories.rs
@@ -0,0 +1,162 @@
+use std::path::PathBuf;
+
+use anyhow::{bail, format_err, Error};
+
+use proxmox_apt::repositories::APTRepositoryFile;
+
+#[test]
+fn test_parse_write() -> Result<(), Error> {
+ let test_dir = std::env::current_dir()?.join("tests");
+ let read_dir = test_dir.join("sources.list.d");
+ let write_dir = test_dir.join("sources.list.d.actual");
+ let expected_dir = test_dir.join("sources.list.d.expected");
+
+ if write_dir.is_dir() {
+ std::fs::remove_dir_all(&write_dir)
+ .map_err(|err| format_err!("unable to remove dir {:?} - {}", write_dir, err))?;
+ }
+
+ std::fs::create_dir_all(&write_dir)
+ .map_err(|err| format_err!("unable to create dir {:?} - {}", write_dir, err))?;
+
+ let mut files = vec![];
+ let mut errors = vec![];
+
+ for entry in std::fs::read_dir(read_dir)? {
+ let path = entry?.path();
+
+ match APTRepositoryFile::new(&path)? {
+ Some(mut file) => match file.parse() {
+ Ok(()) => files.push(file),
+ Err(err) => errors.push(err),
+ },
+ None => bail!("unexpected None for '{:?}'", path),
+ }
+ }
+
+ assert!(errors.is_empty());
+
+ for file in files.iter_mut() {
+ let path = PathBuf::from(&file.path);
+ let new_path = write_dir.join(path.file_name().unwrap());
+ file.path = new_path.into_os_string().into_string().unwrap();
+ file.digest = None;
+ file.write()?;
+ }
+
+ let mut expected_count = 0;
+
+ for entry in std::fs::read_dir(expected_dir)? {
+ expected_count += 1;
+
+ let expected_path = entry?.path();
+ let actual_path = write_dir.join(expected_path.file_name().unwrap());
+
+ let expected_contents = std::fs::read(&expected_path)
+ .map_err(|err| format_err!("unable to read {:?} - {}", expected_path, err))?;
+
+ let actual_contents = std::fs::read(&actual_path)
+ .map_err(|err| format_err!("unable to read {:?} - {}", actual_path, err))?;
+
+ assert_eq!(
+ expected_contents, actual_contents,
+ "Use\n\ndiff {:?} {:?}\n\nif you're not fluent in byte decimals",
+ expected_path, actual_path
+ );
+ }
+
+ let actual_count = std::fs::read_dir(write_dir)?.count();
+
+ assert_eq!(expected_count, actual_count);
+
+ Ok(())
+}
+
+#[test]
+fn test_digest() -> Result<(), Error> {
+ let test_dir = std::env::current_dir()?.join("tests");
+ let read_dir = test_dir.join("sources.list.d");
+ let write_dir = test_dir.join("sources.list.d.digest");
+
+ if write_dir.is_dir() {
+ std::fs::remove_dir_all(&write_dir)
+ .map_err(|err| format_err!("unable to remove dir {:?} - {}", write_dir, err))?;
+ }
+
+ std::fs::create_dir_all(&write_dir)
+ .map_err(|err| format_err!("unable to create dir {:?} - {}", write_dir, err))?;
+
+ let path = read_dir.join("standard.list");
+
+ let mut file = APTRepositoryFile::new(&path)?.unwrap();
+ file.parse()?;
+
+ let new_path = write_dir.join(path.file_name().unwrap());
+ file.path = new_path.clone().into_os_string().into_string().unwrap();
+
+ let old_digest = file.digest.unwrap();
+
+ // file does not exist yet...
+ assert!(file.read_with_digest().is_err());
+ assert!(file.write().is_err());
+
+ // ...but it should work if there's no digest
+ file.digest = None;
+ file.write()?;
+
+ // overwrite with old contents...
+ std::fs::copy(path, new_path)?;
+
+ // modify the repo
+ let mut repo = file.repositories.first_mut().unwrap();
+ repo.enabled = !repo.enabled;
+
+ // ...then it should work
+ file.digest = Some(old_digest);
+ file.write()?;
+
+ // expect a different digest, because the repo was modified
+ let (_, new_digest) = file.read_with_digest()?;
+ assert_ne!(old_digest, new_digest);
+
+ assert!(file.write().is_err());
+
+ Ok(())
+}
+
+#[test]
+fn test_empty_write() -> Result<(), Error> {
+ let test_dir = std::env::current_dir()?.join("tests");
+ let read_dir = test_dir.join("sources.list.d");
+ let write_dir = test_dir.join("sources.list.d.remove");
+
+ if write_dir.is_dir() {
+ std::fs::remove_dir_all(&write_dir)
+ .map_err(|err| format_err!("unable to remove dir {:?} - {}", write_dir, err))?;
+ }
+
+ std::fs::create_dir_all(&write_dir)
+ .map_err(|err| format_err!("unable to create dir {:?} - {}", write_dir, err))?;
+
+ let path = read_dir.join("standard.list");
+
+ let mut file = APTRepositoryFile::new(&path)?.unwrap();
+ file.parse()?;
+
+ let new_path = write_dir.join(path.file_name().unwrap());
+ file.path = new_path.clone().into_os_string().into_string().unwrap();
+
+ file.digest = None;
+
+ file.write()?;
+
+ assert!(file.exists());
+
+ file.repositories.clear();
+
+ file.write()?;
+
+ assert!(!file.exists());
+
+ Ok(())
+}
diff --git a/tests/sources.list.d.expected/absolute_suite.list b/tests/sources.list.d.expected/absolute_suite.list
new file mode 100644
index 0000000..af6b966
--- /dev/null
+++ b/tests/sources.list.d.expected/absolute_suite.list
@@ -0,0 +1,5 @@
+# From Debian Administrator's Handbook
+deb http://packages.falcot.com/ updates/
+
+deb http://user.name@packages.falcot.com:80/ internal/
+
diff --git a/tests/sources.list.d.expected/absolute_suite.sources b/tests/sources.list.d.expected/absolute_suite.sources
new file mode 100644
index 0000000..51e4d56
--- /dev/null
+++ b/tests/sources.list.d.expected/absolute_suite.sources
@@ -0,0 +1,5 @@
+# From Debian Administrator's Handbook
+Types: deb
+URIs: http://packages.falcot.com/
+Suites: updates/ internal/
+
diff --git a/tests/sources.list.d.expected/case.sources b/tests/sources.list.d.expected/case.sources
new file mode 100644
index 0000000..0c71323
--- /dev/null
+++ b/tests/sources.list.d.expected/case.sources
@@ -0,0 +1,16 @@
+# comment in here
+Types: deb deb-src
+URIs: http://ftp.at.debian.org/debian
+Suites: bullseye-updates
+Components: main contrib
+languages: it de fr
+Enabled: false
+languages-Add: ja
+languages-Remove: de
+
+# comment in here
+Types: deb deb-src
+URIs: http://ftp.at.debian.org/debian
+Suites: bullseye
+Components: main contrib
+
diff --git a/tests/sources.list.d.expected/multiline.sources b/tests/sources.list.d.expected/multiline.sources
new file mode 100644
index 0000000..fa7a9e9
--- /dev/null
+++ b/tests/sources.list.d.expected/multiline.sources
@@ -0,0 +1,10 @@
+# comment in here
+Types: deb deb-src
+URIs: http://ftp.at.debian.org/debian
+Suites: bullseye bullseye-updates
+Components: main contrib
+Languages: it de fr
+Enabled: false
+Languages-Add: ja
+Languages-Remove: de
+
--git a/tests/sources.list.d.expected/options_comment.list b/tests/sources.list.d.expected/options_comment.list
new file mode 100644
index 0000000..caef5e0
--- /dev/null
+++ b/tests/sources.list.d.expected/options_comment.list
@@ -0,0 +1,6 @@
+# comment
+deb [ lang=it,de arch=amd64 ] http://ftp.at.debian.org/debian bullseye main contrib
+
+# non-free :(
+deb [ lang=it,de arch=amd64 lang+=fr lang-=de ] http://ftp.at.debian.org/debian bullseye non-free
+
diff --git a/tests/sources.list.d.expected/pbs-enterprise.list b/tests/sources.list.d.expected/pbs-enterprise.list
new file mode 100644
index 0000000..cb6e779
--- /dev/null
+++ b/tests/sources.list.d.expected/pbs-enterprise.list
@@ -0,0 +1,2 @@
+deb https://enterprise.proxmox.com/debian/pbs bullseye pbs-enterprise
+
diff --git a/tests/sources.list.d.expected/pve.list b/tests/sources.list.d.expected/pve.list
new file mode 100644
index 0000000..95725ab
--- /dev/null
+++ b/tests/sources.list.d.expected/pve.list
@@ -0,0 +1,13 @@
+deb http://ftp.debian.org/debian bullseye main contrib
+
+deb http://ftp.debian.org/debian bullseye-updates main contrib
+
+# PVE pve-no-subscription repository provided by proxmox.com,
+# NOT recommended for production use
+deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
+
+# deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise
+
+# security updates
+deb http://security.debian.org/debian-security bullseye-security main contrib
+
diff --git a/tests/sources.list.d.expected/standard.list b/tests/sources.list.d.expected/standard.list
new file mode 100644
index 0000000..51f7ed0
--- /dev/null
+++ b/tests/sources.list.d.expected/standard.list
@@ -0,0 +1,7 @@
+deb http://ftp.at.debian.org/debian bullseye main contrib
+
+deb http://ftp.at.debian.org/debian bullseye-updates main contrib
+
+# security updates
+deb http://security.debian.org bullseye-security main contrib
+
diff --git a/tests/sources.list.d.expected/standard.sources b/tests/sources.list.d.expected/standard.sources
new file mode 100644
index 0000000..85539b3
--- /dev/null
+++ b/tests/sources.list.d.expected/standard.sources
@@ -0,0 +1,11 @@
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: bullseye bullseye-updates
+Components: main contrib
+
+# security updates
+Types: deb
+URIs: http://security.debian.org
+Suites: bullseye-security
+Components: main contrib
+
diff --git a/tests/sources.list.d/absolute_suite.list b/tests/sources.list.d/absolute_suite.list
new file mode 100644
index 0000000..b690d30
--- /dev/null
+++ b/tests/sources.list.d/absolute_suite.list
@@ -0,0 +1,4 @@
+# From Debian Administrator's Handbook
+deb http://packages.falcot.com/ updates/
+
+deb http://user.name@packages.falcot.com:80/ internal/
diff --git a/tests/sources.list.d/absolute_suite.sources b/tests/sources.list.d/absolute_suite.sources
new file mode 100644
index 0000000..51e4d56
--- /dev/null
+++ b/tests/sources.list.d/absolute_suite.sources
@@ -0,0 +1,5 @@
+# From Debian Administrator's Handbook
+Types: deb
+URIs: http://packages.falcot.com/
+Suites: updates/ internal/
+
diff --git a/tests/sources.list.d/case.sources b/tests/sources.list.d/case.sources
new file mode 100644
index 0000000..1a69d14
--- /dev/null
+++ b/tests/sources.list.d/case.sources
@@ -0,0 +1,17 @@
+tYpeS: deb deb-src
+uRis: http://ftp.at.debian.org/debian
+suiTes: bullseye-updates
+# comment in here
+CompOnentS: main contrib
+languages: it
+ de
+ fr
+Enabled: off
+languages-Add: ja
+languages-Remove: de
+
+types: deb deb-src
+Uris: http://ftp.at.debian.org/debian
+suites: bullseye
+# comment in here
+components: main contrib
diff --git a/tests/sources.list.d/multiline.sources b/tests/sources.list.d/multiline.sources
new file mode 100644
index 0000000..683e7e5
--- /dev/null
+++ b/tests/sources.list.d/multiline.sources
@@ -0,0 +1,11 @@
+Types: deb deb-src
+URIs: http://ftp.at.debian.org/debian
+Suites: bullseye bullseye-updates
+# comment in here
+Components: main contrib
+Languages: it
+ de
+ fr
+Enabled: off
+Languages-Add: ja
+Languages-Remove: de
--git a/tests/sources.list.d/options_comment.list b/tests/sources.list.d/options_comment.list
new file mode 100644
index 0000000..7cc810e
--- /dev/null
+++ b/tests/sources.list.d/options_comment.list
@@ -0,0 +1,3 @@
+deb [ lang=it,de arch=amd64 ] http://ftp.at.debian.org/debian bullseye main contrib # comment
+deb [ lang=it,de arch=amd64 lang+=fr lang-=de ] http://ftp.at.debian.org/debian bullseye non-free # non-free :(
+
diff --git a/tests/sources.list.d/pbs-enterprise.list b/tests/sources.list.d/pbs-enterprise.list
new file mode 100644
index 0000000..4592181
--- /dev/null
+++ b/tests/sources.list.d/pbs-enterprise.list
@@ -0,0 +1 @@
+deb https://enterprise.proxmox.com/debian/pbs bullseye pbs-enterprise
diff --git a/tests/sources.list.d/pve.list b/tests/sources.list.d/pve.list
new file mode 100644
index 0000000..c6d451a
--- /dev/null
+++ b/tests/sources.list.d/pve.list
@@ -0,0 +1,10 @@
+deb http://ftp.debian.org/debian bullseye main contrib
+deb http://ftp.debian.org/debian bullseye-updates main contrib
+
+# PVE pve-no-subscription repository provided by proxmox.com,
+# NOT recommended for production use
+deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
+# deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise
+
+# security updates
+deb http://security.debian.org/debian-security bullseye-security main contrib
diff --git a/tests/sources.list.d/standard.list b/tests/sources.list.d/standard.list
new file mode 100644
index 0000000..5fc952e
--- /dev/null
+++ b/tests/sources.list.d/standard.list
@@ -0,0 +1,6 @@
+deb http://ftp.at.debian.org/debian bullseye main contrib
+
+deb http://ftp.at.debian.org/debian bullseye-updates main contrib
+
+# security updates
+deb http://security.debian.org bullseye-security main contrib
diff --git a/tests/sources.list.d/standard.sources b/tests/sources.list.d/standard.sources
new file mode 100644
index 0000000..0b0e83e
--- /dev/null
+++ b/tests/sources.list.d/standard.sources
@@ -0,0 +1,10 @@
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: bullseye bullseye-updates
+Components: main contrib
+
+# security updates
+Types: deb
+URIs: http://security.debian.org
+Suites: bullseye-security
+Components: main contrib
--
2.30.2
^ permalink raw reply [flat|nested] 15+ messages in thread* [pve-devel] [PATCH v7 proxmox-apt 2/5] add files for Debian packaging
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
2021-06-23 13:38 ` [pve-devel] [PATCH v7 proxmox-apt 1/5] initial commit Fabian Ebner
@ 2021-06-23 13:38 ` Fabian Ebner
2021-06-23 13:38 ` [pve-devel] [PATCH v7 proxmox-apt 3/5] add more functions to check repositories Fabian Ebner
` (10 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Fabian Ebner @ 2021-06-23 13:38 UTC (permalink / raw)
To: pve-devel
The Makefile is based on the one from Mira's conntrack series, as it already got
some review.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v6:
* (automatically) update debian/control for bullseye
.gitignore | 5 ++++
Makefile | 67 ++++++++++++++++++++++++++++++++++++++++++++
debian/changelog | 5 ++++
debian/control | 43 ++++++++++++++++++++++++++++
debian/copyright | 16 +++++++++++
debian/debcargo.toml | 7 +++++
6 files changed, 143 insertions(+)
create mode 100644 Makefile
create mode 100644 debian/changelog
create mode 100644 debian/control
create mode 100644 debian/copyright
create mode 100644 debian/debcargo.toml
diff --git a/.gitignore b/.gitignore
index 24917d4..db6f13e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,3 +2,8 @@ Cargo.lock
target/
tests/sources.list.d.actual
tests/sources.list.d.digest
+proxmox-apt-*/
+*proxmox-apt*.buildinfo
+*proxmox-apt*.tar.?z
+*proxmox-apt*.changes
+*proxmox-apt*.deb
diff --git a/Makefile b/Makefile
new file mode 100644
index 0000000..e7f0725
--- /dev/null
+++ b/Makefile
@@ -0,0 +1,67 @@
+include /usr/share/dpkg/pkg-info.mk
+include /usr/share/dpkg/architecture.mk
+
+PACKAGE=proxmox-apt
+BUILDDIR ?= $(PACKAGE)-$(DEB_VERSION_UPSTREAM)
+BUILDDIR_TMP ?= $(BUILDDIR).tmp
+
+DEB=librust-$(PACKAGE)-dev_$(DEB_VERSION_UPSTREAM_REVISION)_$(DEB_BUILD_ARCH).deb
+DSC=rust-$(PACKAGE)_$(DEB_VERSION_UPSTREAM_REVISION).dsc
+
+ifeq ($(BUILD_MODE), release)
+CARGO_BUILD_ARGS += --release
+COMPILEDIR := target/release
+else
+COMPILEDIR := target/debug
+endif
+
+all: cargo-build $(SUBDIRS)
+
+.PHONY: cargo-build
+cargo-build:
+ cargo build $(CARGO_BUILD_ARGS)
+
+.PHONY: build
+build:
+ rm -rf $(BUILDDIR) $(BUILDDIR_TMP); mkdir $(BUILDDIR_TMP)
+ rm -f debian/control
+ debcargo package \
+ --config debian/debcargo.toml \
+ --changelog-ready \
+ --no-overlay-write-back \
+ --directory $(BUILDDIR_TMP) \
+ $(PACKAGE) \
+ $(shell dpkg-parsechangelog -l debian/changelog -SVersion | sed -e 's/-.*//')
+ cp $(BUILDDIR_TMP)/debian/control debian/control
+ rm -f $(BUILDDIR_TMP)/Cargo.lock
+ find $(BUILDDIR_TMP)/debian -name "*.hint" -delete
+ mv $(BUILDDIR_TMP) $(BUILDDIR)
+
+.PHONY: deb
+deb: $(DEB)
+$(DEB): build
+ cd $(BUILDDIR); dpkg-buildpackage -b -us -uc --no-pre-clean
+ lintian $(DEB)
+
+.PHONY: dsc
+dsc: $(DSC)
+$(DSC): build
+ cd $(BUILDDIR); dpkg-buildpackage -S -us -uc -d -nc
+ lintian $(DSC)
+
+.PHONY: dinstall
+dinstall: $(DEB)
+ dpkg -i $(DEB)
+
+.PHONY: upload
+upload: $(DEB) $(DBG_DEB)
+ tar cf - $(DEB) $(DBG_DEB) | ssh -X repoman@repo.proxmox.com -- upload --product pbs,pmg,pve --dist buster --arch $(DEB_BUILD_ARCH)
+
+.PHONY: distclean
+distclean: clean
+
+.PHONY: clean
+clean:
+ cargo clean
+ rm -rf *.deb *.buildinfo *.changes *.dsc rust-$(PACKAGE)_*.tar.?z $(BUILDDIR) $(BUILDDIR_TMP)
+ find . -name '*~' -exec rm {} ';'
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 0000000..11e26ed
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+rust-proxmox-apt (0.1.0-1) unstable; urgency=medium
+
+ * Initial release.
+
+ -- Proxmox Support Team <support@proxmox.com> Thu, 18 Feb 2021 10:20:44 +0100
diff --git a/debian/control b/debian/control
new file mode 100644
index 0000000..e0938bf
--- /dev/null
+++ b/debian/control
@@ -0,0 +1,43 @@
+Source: rust-proxmox-apt
+Section: rust
+Priority: optional
+Build-Depends: debhelper (>= 12),
+ dh-cargo (>= 24),
+ cargo:native <!nocheck>,
+ rustc:native <!nocheck>,
+ libstd-rust-dev <!nocheck>,
+ librust-anyhow-1+default-dev <!nocheck>,
+ librust-openssl-0.10+default-dev <!nocheck>,
+ librust-proxmox-0.11+api-macro-dev (>= 0.11.5-~~) <!nocheck>,
+ librust-proxmox-0.11+default-dev (>= 0.11.5-~~) <!nocheck>,
+ librust-serde-1+default-dev <!nocheck>,
+ librust-serde-1+derive-dev <!nocheck>
+Maintainer: Proxmox Support Team <support@proxmox.com>
+Standards-Version: 4.5.1
+Vcs-Git: git://git.proxmox.com/git/proxmox-apt.git
+Vcs-Browser: https://git.proxmox.com/?p=proxmox-apt.git
+Homepage: https://www.proxmox.com
+Rules-Requires-Root: no
+
+Package: librust-proxmox-apt-dev
+Architecture: any
+Multi-Arch: same
+Depends:
+ ${misc:Depends},
+ librust-anyhow-1+default-dev,
+ librust-openssl-0.10+default-dev,
+ librust-proxmox-0.11+api-macro-dev (>= 0.11.5-~~),
+ librust-proxmox-0.11+default-dev (>= 0.11.5-~~),
+ librust-serde-1+default-dev,
+ librust-serde-1+derive-dev
+Provides:
+ librust-proxmox-apt+default-dev (= ${binary:Version}),
+ librust-proxmox-apt-0-dev (= ${binary:Version}),
+ librust-proxmox-apt-0+default-dev (= ${binary:Version}),
+ librust-proxmox-apt-0.1-dev (= ${binary:Version}),
+ librust-proxmox-apt-0.1+default-dev (= ${binary:Version}),
+ librust-proxmox-apt-0.1.0-dev (= ${binary:Version}),
+ librust-proxmox-apt-0.1.0+default-dev (= ${binary:Version})
+Description: Proxmox library for APT - Rust source code
+ This package contains the source for the Rust proxmox-apt crate, packaged by
+ debcargo for use with cargo and dh-cargo.
diff --git a/debian/copyright b/debian/copyright
new file mode 100644
index 0000000..5661ef6
--- /dev/null
+++ b/debian/copyright
@@ -0,0 +1,16 @@
+Copyright (C) 2021 Proxmox Server Solutions GmbH
+
+This software is written by Proxmox Server Solutions GmbH <support@proxmox.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
diff --git a/debian/debcargo.toml b/debian/debcargo.toml
new file mode 100644
index 0000000..74e3854
--- /dev/null
+++ b/debian/debcargo.toml
@@ -0,0 +1,7 @@
+overlay = "."
+crate_src_path = ".."
+maintainer = "Proxmox Support Team <support@proxmox.com>"
+
+[source]
+vcs_git = "git://git.proxmox.com/git/proxmox-apt.git"
+vcs_browser = "https://git.proxmox.com/?p=proxmox-apt.git"
--
2.30.2
^ permalink raw reply [flat|nested] 15+ messages in thread* [pve-devel] [PATCH v7 proxmox-apt 3/5] add more functions to check repositories
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
2021-06-23 13:38 ` [pve-devel] [PATCH v7 proxmox-apt 1/5] initial commit Fabian Ebner
2021-06-23 13:38 ` [pve-devel] [PATCH v7 proxmox-apt 2/5] add files for Debian packaging Fabian Ebner
@ 2021-06-23 13:38 ` Fabian Ebner
2021-06-23 13:38 ` [pve-devel] [PATCH v7 proxmox-apt 4/5] add handling of Proxmox standard repositories Fabian Ebner
` (9 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Fabian Ebner @ 2021-06-23 13:38 UTC (permalink / raw)
To: pve-devel
Currently includes check for suites and check for official URIs
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v6:
* move impl blocks and APTRepositoryInfo type and helpers into
repository.rs and mod.rs
* limit host_from_uri to http(s)
* fix test by not using assert_eq!(a.sort(), b.sort()) (sort() returns (),
so this was always true...)
* simplify host matching in check_uris()
* make check_suites dependent on the current release from /etc/os-release
* add 'property' to APTRepositoryInfo to know where it originates from
* use index (starting from 0) instead of number (starting from 1) in
APTRepositoryInfo. Originally it came from line number, but each
APTRepositoryFile contains an array of repos and thus index is the better
fit.
src/repositories/file.rs | 119 +++++++++++++++++++++-
src/repositories/mod.rs | 21 +++-
src/repositories/release.rs | 42 ++++++++
src/repositories/repository.rs | 58 +++++++++++
tests/repositories.rs | 106 ++++++++++++++++++-
tests/sources.list.d.expected/bad.sources | 30 ++++++
tests/sources.list.d.expected/pve.list | 2 +
tests/sources.list.d/bad.sources | 29 ++++++
tests/sources.list.d/pve.list | 2 +
9 files changed, 405 insertions(+), 4 deletions(-)
create mode 100644 src/repositories/release.rs
create mode 100644 tests/sources.list.d.expected/bad.sources
create mode 100644 tests/sources.list.d/bad.sources
diff --git a/src/repositories/file.rs b/src/repositories/file.rs
index bc48bf2..6225f1c 100644
--- a/src/repositories/file.rs
+++ b/src/repositories/file.rs
@@ -2,10 +2,13 @@ use std::convert::TryFrom;
use std::fmt::Display;
use std::path::{Path, PathBuf};
-use anyhow::{format_err, Error};
+use anyhow::{bail, format_err, Error};
use serde::{Deserialize, Serialize};
-use crate::repositories::repository::{APTRepository, APTRepositoryFileType};
+use crate::repositories::release::{get_current_release_codename, DEBIAN_SUITES};
+use crate::repositories::repository::{
+ APTRepository, APTRepositoryFileType, APTRepositoryPackageType,
+};
use proxmox::api::api;
@@ -85,6 +88,29 @@ impl std::error::Error for APTRepositoryFileError {
}
}
+#[api]
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)]
+#[serde(rename_all = "kebab-case")]
+/// Additional information for a repository.
+pub struct APTRepositoryInfo {
+ /// Path to the defining file.
+ #[serde(skip_serializing_if = "String::is_empty")]
+ pub path: String,
+
+ /// Index of the associated respository within the file (starting from 0).
+ pub index: usize,
+
+ /// The property from which the info originates (e.g. "Suites")
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub property: Option<String>,
+
+ /// Info kind (e.g. "warning")
+ pub kind: String,
+
+ /// Info message
+ pub message: String,
+}
+
impl APTRepositoryFile {
/// Creates a new `APTRepositoryFile` without parsing.
///
@@ -271,4 +297,93 @@ impl APTRepositoryFile {
Ok(())
}
+
+ /// Checks if old or unstable suites are configured and also that the
+ /// `stable` keyword is not used.
+ pub fn check_suites(&self) -> Result<Vec<APTRepositoryInfo>, Error> {
+ let mut infos = vec![];
+
+ for (n, repo) in self.repositories.iter().enumerate() {
+ if !repo
+ .types
+ .iter()
+ .any(|package_type| *package_type == APTRepositoryPackageType::Deb)
+ {
+ continue;
+ }
+
+ let mut add_info = |kind, message| {
+ infos.push(APTRepositoryInfo {
+ path: self.path.clone(),
+ index: n,
+ property: Some("Suites".to_string()),
+ kind,
+ message,
+ })
+ };
+
+ let current_suite = get_current_release_codename()?;
+
+ let current_index = match DEBIAN_SUITES
+ .iter()
+ .position(|&suite| suite == current_suite)
+ {
+ Some(index) => index,
+ None => bail!("unknown release {}", current_suite),
+ };
+
+ for (n, suite) in DEBIAN_SUITES.iter().enumerate() {
+ if repo.has_suite_variant(suite) {
+ if n < current_index {
+ add_info(
+ "warning".to_string(),
+ format!("old suite '{}' configured!", suite),
+ );
+ }
+
+ if n == current_index + 1 {
+ add_info(
+ "ignore-pre-upgrade-warning".to_string(),
+ format!("suite '{}' should not be used in production!", suite),
+ );
+ }
+
+ if n > current_index + 1 {
+ add_info(
+ "warning".to_string(),
+ format!("suite '{}' should not be used in production!", suite),
+ );
+ }
+ }
+ }
+
+ if repo.has_suite_variant("stable") {
+ add_info(
+ "warning".to_string(),
+ "use the name of the stable distribution instead of 'stable'!".to_string(),
+ );
+ }
+ }
+
+ Ok(infos)
+ }
+
+ /// Checks for official URIs.
+ pub fn check_uris(&self) -> Vec<APTRepositoryInfo> {
+ let mut infos = vec![];
+
+ for (n, repo) in self.repositories.iter().enumerate() {
+ if repo.has_official_uri() {
+ infos.push(APTRepositoryInfo {
+ path: self.path.clone(),
+ index: n,
+ kind: "badge".to_string(),
+ property: Some("URIs".to_string()),
+ message: "official host name".to_string(),
+ });
+ }
+ }
+
+ infos
+ }
}
diff --git a/src/repositories/mod.rs b/src/repositories/mod.rs
index d540bcb..fc54857 100644
--- a/src/repositories/mod.rs
+++ b/src/repositories/mod.rs
@@ -9,7 +9,9 @@ pub use repository::{
};
mod file;
-pub use file::{APTRepositoryFile, APTRepositoryFileError};
+pub use file::{APTRepositoryFile, APTRepositoryFileError, APTRepositoryInfo};
+
+mod release;
const APT_SOURCES_LIST_FILENAME: &str = "/etc/apt/sources.list";
const APT_SOURCES_LIST_DIRECTORY: &str = "/etc/apt/sources.list.d/";
@@ -37,6 +39,23 @@ fn common_digest(files: &[APTRepositoryFile]) -> [u8; 32] {
openssl::sha::sha256(&common_raw[..])
}
+/// Provides additional information about the repositories.
+///
+/// The kind of information can be:
+/// `warnings` for bad suites.
+/// `ignore-pre-upgrade-warning` when the next stable suite is configured.
+/// `badge` for official URIs.
+pub fn check_repositories(files: &[APTRepositoryFile]) -> Result<Vec<APTRepositoryInfo>, Error> {
+ let mut infos = vec![];
+
+ for file in files.iter() {
+ infos.append(&mut file.check_suites()?);
+ infos.append(&mut file.check_uris());
+ }
+
+ Ok(infos)
+}
+
/// Returns all APT repositories configured in `/etc/apt/sources.list` and
/// in `/etc/apt/sources.list.d` including disabled repositories.
///
diff --git a/src/repositories/release.rs b/src/repositories/release.rs
new file mode 100644
index 0000000..688f038
--- /dev/null
+++ b/src/repositories/release.rs
@@ -0,0 +1,42 @@
+use anyhow::{bail, format_err, Error};
+
+use std::io::{BufRead, BufReader};
+
+/// The suites of Debian releases, ordered chronologically, with variable releases
+/// like 'oldstable' and 'testing' ordered at the extremes. Does not include 'stable'.
+pub const DEBIAN_SUITES: [&str; 15] = [
+ "oldoldstable",
+ "oldstable",
+ "lenny",
+ "squeeze",
+ "wheezy",
+ "jessie",
+ "stretch",
+ "buster",
+ "bullseye",
+ "bookworm",
+ "trixie",
+ "sid",
+ "testing",
+ "unstable",
+ "experimental",
+];
+
+/// Read the `VERSION_CODENAME` from `/etc/os-release`.
+pub fn get_current_release_codename() -> Result<String, Error> {
+ let raw = std::fs::read("/etc/os-release")
+ .map_err(|err| format_err!("unable to read '/etc/os-release' - {}", err))?;
+
+ let reader = BufReader::new(&*raw);
+
+ for line in reader.lines() {
+ let line = line.map_err(|err| format_err!("unable to read '/etc/os-release' - {}", err))?;
+
+ if let Some(codename) = line.strip_prefix("VERSION_CODENAME=") {
+ let codename = codename.trim_matches(&['"', '\''][..]);
+ return Ok(codename.to_string());
+ }
+ }
+
+ bail!("unable to parse codename from '/etc/os-release'");
+}
diff --git a/src/repositories/repository.rs b/src/repositories/repository.rs
index 9ee7df9..875e4ee 100644
--- a/src/repositories/repository.rs
+++ b/src/repositories/repository.rs
@@ -266,6 +266,30 @@ impl APTRepository {
Ok(())
}
+ /// Check if a variant of the given suite is configured in this repository
+ pub fn has_suite_variant(&self, base_suite: &str) -> bool {
+ self.suites
+ .iter()
+ .any(|suite| suite_variant(suite).0 == base_suite)
+ }
+
+ /// Checks if an official host is configured in the repository.
+ pub fn has_official_uri(&self) -> bool {
+ for uri in self.uris.iter() {
+ if let Some(host) = host_from_uri(uri) {
+ if host == "proxmox.com"
+ || host.ends_with(".proxmox.com")
+ || host == "debian.org"
+ || host.ends_with(".debian.org")
+ {
+ return true;
+ }
+ }
+ }
+
+ false
+ }
+
/// Writes a repository in the corresponding format followed by a blank.
///
/// Expects that `basic_check()` for the repository was successful.
@@ -277,6 +301,40 @@ impl APTRepository {
}
}
+/// Get the host part from a given URI.
+fn host_from_uri(uri: &str) -> Option<&str> {
+ let host = uri.strip_prefix("http")?;
+ let host = host.strip_prefix("s").unwrap_or(host);
+ let mut host = host.strip_prefix("://")?;
+
+ if let Some(end) = host.find('/') {
+ host = &host[..end];
+ }
+
+ if let Some(begin) = host.find('@') {
+ host = &host[(begin + 1)..];
+ }
+
+ if let Some(end) = host.find(':') {
+ host = &host[..end];
+ }
+
+ Some(host)
+}
+
+/// Splits the suite into its base part and variant.
+fn suite_variant(suite: &str) -> (&str, &str) {
+ let variants = ["-backports-sloppy", "-backports", "-updates", "/updates"];
+
+ for variant in variants.iter() {
+ if let Some(base) = suite.strip_suffix(variant) {
+ return (base, variant);
+ }
+ }
+
+ (suite, "")
+}
+
/// Writes a repository in one-line format followed by a blank line.
///
/// Expects that `repo.file_type == APTRepositoryFileType::List`.
diff --git a/tests/repositories.rs b/tests/repositories.rs
index 477d718..58f1322 100644
--- a/tests/repositories.rs
+++ b/tests/repositories.rs
@@ -2,7 +2,7 @@ use std::path::PathBuf;
use anyhow::{bail, format_err, Error};
-use proxmox_apt::repositories::APTRepositoryFile;
+use proxmox_apt::repositories::{check_repositories, APTRepositoryFile, APTRepositoryInfo};
#[test]
fn test_parse_write() -> Result<(), Error> {
@@ -160,3 +160,107 @@ fn test_empty_write() -> Result<(), Error> {
Ok(())
}
+
+#[test]
+fn test_check_repositories() -> Result<(), Error> {
+ let test_dir = std::env::current_dir()?.join("tests");
+ let read_dir = test_dir.join("sources.list.d");
+
+ let absolute_suite_list = read_dir.join("absolute_suite.list");
+ let mut file = APTRepositoryFile::new(&absolute_suite_list)?.unwrap();
+ file.parse()?;
+
+ let infos = check_repositories(&vec![file])?;
+
+ assert_eq!(infos.is_empty(), true);
+ let pve_list = read_dir.join("pve.list");
+ let mut file = APTRepositoryFile::new(&pve_list)?.unwrap();
+ file.parse()?;
+
+ let path_string = pve_list.into_os_string().into_string().unwrap();
+
+ let mut expected_infos = vec![];
+ for n in 0..=5 {
+ expected_infos.push(APTRepositoryInfo {
+ path: path_string.clone(),
+ index: n,
+ property: Some("URIs".to_string()),
+ kind: "badge".to_string(),
+ message: "official host name".to_string(),
+ });
+ }
+ expected_infos.sort();
+
+ let mut infos = check_repositories(&vec![file])?;
+ infos.sort();
+
+ assert_eq!(infos, expected_infos);
+
+ let bad_sources = read_dir.join("bad.sources");
+ let mut file = APTRepositoryFile::new(&bad_sources)?.unwrap();
+ file.parse()?;
+
+ let path_string = bad_sources.into_os_string().into_string().unwrap();
+
+ let mut expected_infos = vec![
+ APTRepositoryInfo {
+ path: path_string.clone(),
+ index: 0,
+ property: Some("Suites".to_string()),
+ kind: "warning".to_string(),
+ message: "suite 'sid' should not be used in production!".to_string(),
+ },
+ APTRepositoryInfo {
+ path: path_string.clone(),
+ index: 1,
+ property: Some("Suites".to_string()),
+ kind: "warning".to_string(),
+ message: "old suite 'lenny' configured!".to_string(),
+ },
+ APTRepositoryInfo {
+ path: path_string.clone(),
+ index: 2,
+ property: Some("Suites".to_string()),
+ kind: "warning".to_string(),
+ message: "old suite 'stretch' configured!".to_string(),
+ },
+ APTRepositoryInfo {
+ path: path_string.clone(),
+ index: 3,
+ property: Some("Suites".to_string()),
+ kind: "warning".to_string(),
+ message: "use the name of the stable distribution instead of 'stable'!".to_string(),
+ },
+ APTRepositoryInfo {
+ path: path_string.clone(),
+ index: 4,
+ property: Some("Suites".to_string()),
+ kind: "ignore-pre-upgrade-warning".to_string(),
+ message: "suite 'bookworm' should not be used in production!".to_string(),
+ },
+ APTRepositoryInfo {
+ path: path_string.clone(),
+ index: 5,
+ property: Some("Suites".to_string()),
+ kind: "warning".to_string(),
+ message: "suite 'testing' should not be used in production!".to_string(),
+ },
+ ];
+ for n in 0..=5 {
+ expected_infos.push(APTRepositoryInfo {
+ path: path_string.clone(),
+ index: n,
+ property: Some("URIs".to_string()),
+ kind: "badge".to_string(),
+ message: "official host name".to_string(),
+ });
+ }
+ expected_infos.sort();
+
+ let mut infos = check_repositories(&vec![file])?;
+ infos.sort();
+
+ assert_eq!(infos, expected_infos);
+
+ Ok(())
+}
diff --git a/tests/sources.list.d.expected/bad.sources b/tests/sources.list.d.expected/bad.sources
new file mode 100644
index 0000000..7eff6a5
--- /dev/null
+++ b/tests/sources.list.d.expected/bad.sources
@@ -0,0 +1,30 @@
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: sid
+Components: main contrib
+
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: lenny-backports
+Components: contrib
+
+Types: deb
+URIs: http://security.debian.org:80
+Suites: stretch/updates
+Components: main contrib
+
+Types: deb
+URIs: http://ftp.at.debian.org:80/debian
+Suites: stable
+Components: main
+
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: bookworm
+Components: main
+
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: testing
+Components: main
+
diff --git a/tests/sources.list.d.expected/pve.list b/tests/sources.list.d.expected/pve.list
index 95725ab..c801261 100644
--- a/tests/sources.list.d.expected/pve.list
+++ b/tests/sources.list.d.expected/pve.list
@@ -8,6 +8,8 @@ deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
# deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise
+deb-src https://enterprise.proxmox.com/debian/pve buster pve-enterprise
+
# security updates
deb http://security.debian.org/debian-security bullseye-security main contrib
diff --git a/tests/sources.list.d/bad.sources b/tests/sources.list.d/bad.sources
new file mode 100644
index 0000000..46eb82a
--- /dev/null
+++ b/tests/sources.list.d/bad.sources
@@ -0,0 +1,29 @@
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: sid
+Components: main contrib
+
+Types: deb
+URIs: http://ftp.at.debian.org/debian
+Suites: lenny-backports
+Components: contrib
+
+Types: deb
+URIs: http://security.debian.org:80
+Suites: stretch/updates
+Components: main contrib
+
+Suites: stable
+URIs: http://ftp.at.debian.org:80/debian
+Components: main
+Types: deb
+
+Suites: bookworm
+URIs: http://ftp.at.debian.org/debian
+Components: main
+Types: deb
+
+Suites: testing
+URIs: http://ftp.at.debian.org/debian
+Components: main
+Types: deb
diff --git a/tests/sources.list.d/pve.list b/tests/sources.list.d/pve.list
index c6d451a..4d36d3d 100644
--- a/tests/sources.list.d/pve.list
+++ b/tests/sources.list.d/pve.list
@@ -6,5 +6,7 @@ deb http://ftp.debian.org/debian bullseye-updates main contrib
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
# deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise
+deb-src https://enterprise.proxmox.com/debian/pve buster pve-enterprise
+
# security updates
deb http://security.debian.org/debian-security bullseye-security main contrib
--
2.30.2
^ permalink raw reply [flat|nested] 15+ messages in thread* [pve-devel] [PATCH v7 proxmox-apt 4/5] add handling of Proxmox standard repositories
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
` (2 preceding siblings ...)
2021-06-23 13:38 ` [pve-devel] [PATCH v7 proxmox-apt 3/5] add more functions to check repositories Fabian Ebner
@ 2021-06-23 13:38 ` Fabian Ebner
2021-06-23 13:38 ` [pve-devel] [PATCH v7 proxmox-apt 5/5] bump version to 0.2.0-1 Fabian Ebner
` (8 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Fabian Ebner @ 2021-06-23 13:38 UTC (permalink / raw)
To: pve-devel
Get handles for the available repositories along with their current
configuration status and make it possible to add them.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
New in v7.
This also generalizes the Proxmox repository detection logic that was present in
v6 and replaces the is_enterprise_enabled and is_nosubscription_enabled
functions.
src/repositories/mod.rs | 83 +++++++++++++++
src/repositories/repository.rs | 14 +++
src/repositories/standard.rs | 180 +++++++++++++++++++++++++++++++++
tests/repositories.rs | 82 ++++++++++++++-
4 files changed, 358 insertions(+), 1 deletion(-)
create mode 100644 src/repositories/standard.rs
diff --git a/src/repositories/mod.rs b/src/repositories/mod.rs
index fc54857..eba65f4 100644
--- a/src/repositories/mod.rs
+++ b/src/repositories/mod.rs
@@ -12,6 +12,10 @@ mod file;
pub use file::{APTRepositoryFile, APTRepositoryFileError, APTRepositoryInfo};
mod release;
+use release::get_current_release_codename;
+
+mod standard;
+pub use standard::{APTRepositoryHandle, APTStandardRepository};
const APT_SOURCES_LIST_FILENAME: &str = "/etc/apt/sources.list";
const APT_SOURCES_LIST_DIRECTORY: &str = "/etc/apt/sources.list.d/";
@@ -56,6 +60,85 @@ pub fn check_repositories(files: &[APTRepositoryFile]) -> Result<Vec<APTReposito
Ok(infos)
}
+/// Get the repository associated to the handle and the path where its usually configured.
+pub fn get_standard_repository(
+ handle: APTRepositoryHandle,
+ product: &str,
+) -> Result<(APTRepository, String), Error> {
+ let suite = get_current_release_codename()?;
+
+ let repo = handle.to_repository(product, &suite);
+ let path = handle.path(product);
+
+ Ok((repo, path))
+}
+
+/// Return handles for standard Proxmox repositories and whether their status, where
+/// None means not configured, and Some(bool) indicates enabled or disabled
+pub fn standard_repositories(
+ product: &str,
+ files: &[APTRepositoryFile],
+) -> Vec<APTStandardRepository> {
+ let mut result = vec![
+ APTStandardRepository {
+ handle: APTRepositoryHandle::Enterprise,
+ status: None,
+ name: APTRepositoryHandle::Enterprise.name(product),
+ },
+ APTStandardRepository {
+ handle: APTRepositoryHandle::NoSubscription,
+ status: None,
+ name: APTRepositoryHandle::NoSubscription.name(product),
+ },
+ APTStandardRepository {
+ handle: APTRepositoryHandle::Test,
+ status: None,
+ name: APTRepositoryHandle::Test.name(product),
+ },
+ ];
+
+ if product == "pve" {
+ result.append(&mut vec![
+ APTStandardRepository {
+ handle: APTRepositoryHandle::CephPacific,
+ status: None,
+ name: APTRepositoryHandle::CephPacific.name(product),
+ },
+ APTStandardRepository {
+ handle: APTRepositoryHandle::CephPacificTest,
+ status: None,
+ name: APTRepositoryHandle::CephPacificTest.name(product),
+ },
+ APTStandardRepository {
+ handle: APTRepositoryHandle::CephOctopus,
+ status: None,
+ name: APTRepositoryHandle::CephOctopus.name(product),
+ },
+ APTStandardRepository {
+ handle: APTRepositoryHandle::CephOctopusTest,
+ status: None,
+ name: APTRepositoryHandle::CephOctopusTest.name(product),
+ },
+ ]);
+ }
+
+ for file in files.iter() {
+ for repo in file.repositories.iter() {
+ for entry in result.iter_mut() {
+ if entry.status == Some(true) {
+ continue;
+ }
+
+ if repo.is_referenced_repository(entry.handle, product) {
+ entry.status = Some(repo.enabled);
+ }
+ }
+ }
+ }
+
+ result
+}
+
/// Returns all APT repositories configured in `/etc/apt/sources.list` and
/// in `/etc/apt/sources.list.d` including disabled repositories.
///
diff --git a/src/repositories/repository.rs b/src/repositories/repository.rs
index 875e4ee..d0a2b81 100644
--- a/src/repositories/repository.rs
+++ b/src/repositories/repository.rs
@@ -7,6 +7,8 @@ use serde::{Deserialize, Serialize};
use proxmox::api::api;
+use crate::repositories::standard::APTRepositoryHandle;
+
#[api]
#[derive(Debug, Copy, Clone, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "lowercase")]
@@ -266,6 +268,18 @@ impl APTRepository {
Ok(())
}
+ /// Checks if the repository is the one referenced by the handle.
+ pub fn is_referenced_repository(&self, handle: APTRepositoryHandle, product: &str) -> bool {
+ let (package_type, uri, component) = handle.info(product);
+
+ self.types.contains(&package_type)
+ && self
+ .uris
+ .iter()
+ .any(|self_uri| self_uri.trim_end_matches('/') == uri)
+ && self.components.contains(&component)
+ }
+
/// Check if a variant of the given suite is configured in this repository
pub fn has_suite_variant(&self, base_suite: &str) -> bool {
self.suites
diff --git a/src/repositories/standard.rs b/src/repositories/standard.rs
new file mode 100644
index 0000000..2a29852
--- /dev/null
+++ b/src/repositories/standard.rs
@@ -0,0 +1,180 @@
+use std::convert::TryFrom;
+use std::fmt::Display;
+
+use anyhow::{bail, Error};
+use serde::{Deserialize, Serialize};
+
+use crate::repositories::repository::{
+ APTRepository, APTRepositoryFileType, APTRepositoryPackageType,
+};
+
+use proxmox::api::api;
+
+#[api(
+ properties: {
+ handle: {
+ description: "Handle referencing a standard repository.",
+ type: String,
+ },
+ },
+)]
+#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// Reference to a standard repository and configuration status.
+pub struct APTStandardRepository {
+ /// Handle referencing a standard repository.
+ pub handle: APTRepositoryHandle,
+
+ /// Configuration status of the associated repository.
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub status: Option<bool>,
+
+ /// Full name of the repository.
+ pub name: String,
+}
+
+#[api]
+#[derive(Debug, Copy, Clone, Serialize, Deserialize, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// Handles for Proxmox repositories.
+pub enum APTRepositoryHandle {
+ /// The enterprise repository for production use.
+ Enterprise,
+ /// The repository that can be used without subscription.
+ NoSubscription,
+ /// The test repository.
+ Test,
+ /// Ceph Pacific repository.
+ CephPacific,
+ /// Ceph Pacific test repository.
+ CephPacificTest,
+ /// Ceph Octoput repository.
+ CephOctopus,
+ /// Ceph Octoput test repository.
+ CephOctopusTest,
+}
+
+impl TryFrom<&str> for APTRepositoryHandle {
+ type Error = Error;
+
+ fn try_from(string: &str) -> Result<Self, Error> {
+ match string {
+ "enterprise" => Ok(APTRepositoryHandle::Enterprise),
+ "no-subscription" => Ok(APTRepositoryHandle::NoSubscription),
+ "test" => Ok(APTRepositoryHandle::Test),
+ "ceph-pacific" => Ok(APTRepositoryHandle::CephPacific),
+ "ceph-pacific-test" => Ok(APTRepositoryHandle::CephPacificTest),
+ "ceph-octopus" => Ok(APTRepositoryHandle::CephOctopus),
+ "ceph-octopus-test" => Ok(APTRepositoryHandle::CephOctopusTest),
+ _ => bail!("unknown repository handle '{}'", string),
+ }
+ }
+}
+
+impl Display for APTRepositoryHandle {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match self {
+ APTRepositoryHandle::Enterprise => write!(f, "enterprise"),
+ APTRepositoryHandle::NoSubscription => write!(f, "no-subscription"),
+ APTRepositoryHandle::Test => write!(f, "test"),
+ APTRepositoryHandle::CephPacific => write!(f, "ceph-pacific"),
+ APTRepositoryHandle::CephPacificTest => write!(f, "ceph-pacific-test"),
+ APTRepositoryHandle::CephOctopus => write!(f, "ceph-octopus"),
+ APTRepositoryHandle::CephOctopusTest => write!(f, "ceph-octopus-test"),
+ }
+ }
+}
+
+impl APTRepositoryHandle {
+ /// Get the full name of the repository.
+ pub fn name(self, product: &str) -> String {
+ match self {
+ APTRepositoryHandle::Enterprise => {
+ format!("{} Enterprise Repository", product.to_uppercase())
+ }
+ APTRepositoryHandle::NoSubscription => {
+ format!("{} No-Subscription Repository", product.to_uppercase())
+ }
+ APTRepositoryHandle::Test => format!("{} Test Repository", product.to_uppercase()),
+ APTRepositoryHandle::CephPacific => "PVE Ceph Pacific Repository".to_string(),
+ APTRepositoryHandle::CephPacificTest => "PVE Ceph Pacific Test Repository".to_string(),
+ APTRepositoryHandle::CephOctopus => "PVE Ceph Octopus Repository".to_string(),
+ APTRepositoryHandle::CephOctopusTest => "PVE Ceph Octopus Test Repository".to_string(),
+ }
+ }
+
+ /// Get the standard file path for the repository referenced by the handle.
+ pub fn path(self, product: &str) -> String {
+ match self {
+ APTRepositoryHandle::Enterprise => {
+ format!("/etc/apt/sources.list.d/{}-enterprise.list", product)
+ }
+ APTRepositoryHandle::NoSubscription => "/etc/apt/sources.list".to_string(),
+ APTRepositoryHandle::Test => "/etc/apt/sources.list".to_string(),
+ APTRepositoryHandle::CephPacific => "/etc/apt/sources.list.d/ceph.list".to_string(),
+ APTRepositoryHandle::CephPacificTest => "/etc/apt/sources.list.d/ceph.list".to_string(),
+ APTRepositoryHandle::CephOctopus => "/etc/apt/sources.list.d/ceph.list".to_string(),
+ APTRepositoryHandle::CephOctopusTest => "/etc/apt/sources.list.d/ceph.list".to_string(),
+ }
+ }
+
+ /// Get package type, URI and the component associated with the handle.
+ pub fn info(self, product: &str) -> (APTRepositoryPackageType, String, String) {
+ match self {
+ APTRepositoryHandle::Enterprise => (
+ APTRepositoryPackageType::Deb,
+ format!("https://enterprise.proxmox.com/debian/{}", product),
+ format!("{}-enterprise", product),
+ ),
+ APTRepositoryHandle::NoSubscription => (
+ APTRepositoryPackageType::Deb,
+ format!("http://download.proxmox.com/debian/{}", product),
+ format!("{}-no-subscription", product),
+ ),
+ APTRepositoryHandle::Test => (
+ APTRepositoryPackageType::Deb,
+ format!("http://download.proxmox.com/debian/{}", product),
+ format!("{}test", product),
+ ),
+ APTRepositoryHandle::CephPacific => (
+ APTRepositoryPackageType::Deb,
+ "http://download.proxmox.com/debian/ceph-pacific".to_string(),
+ "main".to_string(),
+ ),
+ APTRepositoryHandle::CephPacificTest => (
+ APTRepositoryPackageType::Deb,
+ "http://download.proxmox.com/debian/ceph-pacific".to_string(),
+ "test".to_string(),
+ ),
+ APTRepositoryHandle::CephOctopus => (
+ APTRepositoryPackageType::Deb,
+ "http://download.proxmox.com/debian/ceph-octopus".to_string(),
+ "main".to_string(),
+ ),
+ APTRepositoryHandle::CephOctopusTest => (
+ APTRepositoryPackageType::Deb,
+ "http://download.proxmox.com/debian/ceph-octopus".to_string(),
+ "test".to_string(),
+ ),
+ }
+ }
+
+ /// Get the standard repository referenced by the handle.
+ ///
+ /// An URI in the result is not '/'-terminated (under the assumption that no valid
+ /// product name is).
+ pub fn to_repository(self, product: &str, suite: &str) -> APTRepository {
+ let (package_type, uri, component) = self.info(product);
+
+ APTRepository {
+ types: vec![package_type],
+ uris: vec![uri],
+ suites: vec![suite.to_string()],
+ components: vec![component],
+ options: vec![],
+ comment: String::new(),
+ file_type: APTRepositoryFileType::List,
+ enabled: true,
+ }
+ }
+}
diff --git a/tests/repositories.rs b/tests/repositories.rs
index 58f1322..d0e9329 100644
--- a/tests/repositories.rs
+++ b/tests/repositories.rs
@@ -2,7 +2,10 @@ use std::path::PathBuf;
use anyhow::{bail, format_err, Error};
-use proxmox_apt::repositories::{check_repositories, APTRepositoryFile, APTRepositoryInfo};
+use proxmox_apt::repositories::{
+ check_repositories, standard_repositories, APTRepositoryFile, APTRepositoryHandle,
+ APTRepositoryInfo, APTStandardRepository,
+};
#[test]
fn test_parse_write() -> Result<(), Error> {
@@ -264,3 +267,80 @@ fn test_check_repositories() -> Result<(), Error> {
Ok(())
}
+#[test]
+fn test_standard_repositories() -> Result<(), Error> {
+ let test_dir = std::env::current_dir()?.join("tests");
+ let read_dir = test_dir.join("sources.list.d");
+
+ let mut expected = vec![
+ APTStandardRepository {
+ handle: APTRepositoryHandle::Enterprise,
+ status: None,
+ name: APTRepositoryHandle::Enterprise.name("pve"),
+ },
+ APTStandardRepository {
+ handle: APTRepositoryHandle::NoSubscription,
+ status: None,
+ name: APTRepositoryHandle::NoSubscription.name("pve"),
+ },
+ APTStandardRepository {
+ handle: APTRepositoryHandle::Test,
+ status: None,
+ name: APTRepositoryHandle::Test.name("pve"),
+ },
+ APTStandardRepository {
+ handle: APTRepositoryHandle::CephPacific,
+ status: None,
+ name: APTRepositoryHandle::CephPacific.name("pve"),
+ },
+ APTStandardRepository {
+ handle: APTRepositoryHandle::CephPacificTest,
+ status: None,
+ name: APTRepositoryHandle::CephPacificTest.name("pve"),
+ },
+ APTStandardRepository {
+ handle: APTRepositoryHandle::CephOctopus,
+ status: None,
+ name: APTRepositoryHandle::CephOctopus.name("pve"),
+ },
+ APTStandardRepository {
+ handle: APTRepositoryHandle::CephOctopusTest,
+ status: None,
+ name: APTRepositoryHandle::CephOctopusTest.name("pve"),
+ },
+ ];
+
+ let absolute_suite_list = read_dir.join("absolute_suite.list");
+ let mut file = APTRepositoryFile::new(&absolute_suite_list)?.unwrap();
+ file.parse()?;
+
+ let std_repos = standard_repositories("pve", &vec![file]);
+
+ assert_eq!(std_repos, expected);
+
+ let pve_list = read_dir.join("pve.list");
+ let mut file = APTRepositoryFile::new(&pve_list)?.unwrap();
+ file.parse()?;
+
+ let file_vec = vec![file];
+
+ let std_repos = standard_repositories("pbs", &file_vec);
+
+ expected[0].name = APTRepositoryHandle::Enterprise.name("pbs");
+ expected[1].name = APTRepositoryHandle::NoSubscription.name("pbs");
+ expected[2].name = APTRepositoryHandle::Test.name("pbs");
+
+ assert_eq!(&std_repos, &expected[0..=2]);
+
+ expected[0].status = Some(false);
+ expected[1].status = Some(true);
+ expected[0].name = APTRepositoryHandle::Enterprise.name("pve");
+ expected[1].name = APTRepositoryHandle::NoSubscription.name("pve");
+ expected[2].name = APTRepositoryHandle::Test.name("pve");
+
+ let std_repos = standard_repositories("pve", &file_vec);
+
+ assert_eq!(std_repos, expected);
+
+ Ok(())
+}
--
2.30.2
^ permalink raw reply [flat|nested] 15+ messages in thread* [pve-devel] [PATCH v7 proxmox-apt 5/5] bump version to 0.2.0-1
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
` (3 preceding siblings ...)
2021-06-23 13:38 ` [pve-devel] [PATCH v7 proxmox-apt 4/5] add handling of Proxmox standard repositories Fabian Ebner
@ 2021-06-23 13:38 ` Fabian Ebner
2021-06-23 13:38 ` [pve-devel] [PATCH v7 proxmox-widget-toolkit 1/2] add UI for APT repositories Fabian Ebner
` (7 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Fabian Ebner @ 2021-06-23 13:38 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v6:
* different changes in the changelog
Cargo.toml | 2 +-
debian/changelog | 8 ++++++++
debian/control | 8 ++++----
3 files changed, 13 insertions(+), 5 deletions(-)
diff --git a/Cargo.toml b/Cargo.toml
index 24f734b..9bed970 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,6 +1,6 @@
[package]
name = "proxmox-apt"
-version = "0.1.0"
+version = "0.2.0"
authors = [
"Fabian Ebner <f.ebner@proxmox.com>",
"Proxmox Support Team <support@proxmox.com>",
diff --git a/debian/changelog b/debian/changelog
index 11e26ed..21a429e 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+rust-proxmox-apt (0.2.0-1) unstable; urgency=medium
+
+ * Add functions to check repositories.
+
+ * Add handling of standard Proxmox repositories.
+
+ -- Proxmox Support Team <support@proxmox.com> Wed, 23 Jun 2021 14:57:52 +0200
+
rust-proxmox-apt (0.1.0-1) unstable; urgency=medium
* Initial release.
diff --git a/debian/control b/debian/control
index e0938bf..99ac284 100644
--- a/debian/control
+++ b/debian/control
@@ -34,10 +34,10 @@ Provides:
librust-proxmox-apt+default-dev (= ${binary:Version}),
librust-proxmox-apt-0-dev (= ${binary:Version}),
librust-proxmox-apt-0+default-dev (= ${binary:Version}),
- librust-proxmox-apt-0.1-dev (= ${binary:Version}),
- librust-proxmox-apt-0.1+default-dev (= ${binary:Version}),
- librust-proxmox-apt-0.1.0-dev (= ${binary:Version}),
- librust-proxmox-apt-0.1.0+default-dev (= ${binary:Version})
+ librust-proxmox-apt-0.2-dev (= ${binary:Version}),
+ librust-proxmox-apt-0.2+default-dev (= ${binary:Version}),
+ librust-proxmox-apt-0.2.0-dev (= ${binary:Version}),
+ librust-proxmox-apt-0.2.0+default-dev (= ${binary:Version})
Description: Proxmox library for APT - Rust source code
This package contains the source for the Rust proxmox-apt crate, packaged by
debcargo for use with cargo and dh-cargo.
--
2.30.2
^ permalink raw reply [flat|nested] 15+ messages in thread* [pve-devel] [PATCH v7 proxmox-widget-toolkit 1/2] add UI for APT repositories
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
` (4 preceding siblings ...)
2021-06-23 13:38 ` [pve-devel] [PATCH v7 proxmox-apt 5/5] bump version to 0.2.0-1 Fabian Ebner
@ 2021-06-23 13:38 ` Fabian Ebner
2021-06-23 13:39 ` [pve-devel] [PATCH v7 proxmox-widget-toolkit 2/2] add buttons for add/enable/disable Fabian Ebner
` (6 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Fabian Ebner @ 2021-06-23 13:38 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v6:
* adapt to new API
* squashed patch adding warnings/checks into this one
* say 'enabled' instead of 'configured' in warnings
* move tbar to grid component (selection model is needed for future buttons)
* use greyed-out icon if repository is disabled
src/Makefile | 1 +
src/node/APTRepositories.js | 423 ++++++++++++++++++++++++++++++++++++
2 files changed, 424 insertions(+)
create mode 100644 src/node/APTRepositories.js
diff --git a/src/Makefile b/src/Makefile
index 37da480..23f2360 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -72,6 +72,7 @@ JSSRC= \
window/ACMEDomains.js \
window/FileBrowser.js \
node/APT.js \
+ node/APTRepositories.js \
node/NetworkEdit.js \
node/NetworkView.js \
node/DNSEdit.js \
diff --git a/src/node/APTRepositories.js b/src/node/APTRepositories.js
new file mode 100644
index 0000000..30c31ec
--- /dev/null
+++ b/src/node/APTRepositories.js
@@ -0,0 +1,423 @@
+Ext.define('apt-repolist', {
+ extend: 'Ext.data.Model',
+ fields: [
+ 'Path',
+ 'Index',
+ 'OfficialHost',
+ 'FileType',
+ 'Enabled',
+ 'Comment',
+ 'Types',
+ 'URIs',
+ 'Suites',
+ 'Components',
+ 'Options',
+ ],
+});
+
+Ext.define('Proxmox.node.APTRepositoriesErrors', {
+ extend: 'Ext.grid.GridPanel',
+
+ xtype: 'proxmoxNodeAPTRepositoriesErrors',
+
+ title: gettext('Errors'),
+
+ store: {},
+
+ viewConfig: {
+ stripeRows: false,
+ getRowClass: () => 'proxmox-invalid-row',
+ },
+
+ columns: [
+ {
+ header: gettext('File'),
+ dataIndex: 'path',
+ renderer: function(value, cell, record) {
+ return "<i class='pve-grid-fa fa fa-fw " +
+ "fa-exclamation-triangle'></i>" + value;
+ },
+ width: 350,
+ },
+ {
+ header: gettext('Error'),
+ dataIndex: 'error',
+ flex: 1,
+ },
+ ],
+});
+
+Ext.define('Proxmox.node.APTRepositoriesGrid', {
+ extend: 'Ext.grid.GridPanel',
+
+ xtype: 'proxmoxNodeAPTRepositoriesGrid',
+
+ title: gettext('APT Repositories'),
+
+ tbar: [
+ {
+ text: gettext('Reload'),
+ iconCls: 'fa fa-refresh',
+ handler: function() {
+ let me = this;
+ me.up('proxmoxNodeAPTRepositories').reload();
+ },
+ },
+ ],
+
+ sortableColumns: false,
+
+ columns: [
+ {
+ header: gettext('Official'),
+ dataIndex: 'OfficialHost',
+ renderer: function(value, cell, record) {
+ let icon = (cls) => `<i class="fa fa-fw ${cls}"></i>`;
+
+ const enabled = record.data.Enabled;
+
+ if (value === undefined || value === null) {
+ return icon('fa-question-circle-o');
+ }
+ if (!value) {
+ return icon('fa-times ' + (enabled ? 'critical' : 'faded'));
+ }
+ return icon('fa-check ' + (enabled ? 'good' : 'faded'));
+ },
+ width: 70,
+ },
+ {
+ header: gettext('Enabled'),
+ dataIndex: 'Enabled',
+ renderer: Proxmox.Utils.format_enabled_toggle,
+ width: 90,
+ },
+ {
+ header: gettext('Types'),
+ dataIndex: 'Types',
+ renderer: function(types, cell, record) {
+ return types.join(' ');
+ },
+ width: 100,
+ },
+ {
+ header: gettext('URIs'),
+ dataIndex: 'URIs',
+ renderer: function(uris, cell, record) {
+ return uris.join(' ');
+ },
+ width: 350,
+ },
+ {
+ header: gettext('Suites'),
+ dataIndex: 'Suites',
+ renderer: function(suites, cell, record) {
+ return suites.join(' ');
+ },
+ width: 130,
+ },
+ {
+ header: gettext('Components'),
+ dataIndex: 'Components',
+ renderer: function(components, cell, record) {
+ return components.join(' ');
+ },
+ width: 170,
+ },
+ {
+ header: gettext('Options'),
+ dataIndex: 'Options',
+ renderer: function(options, cell, record) {
+ if (!options) {
+ return '';
+ }
+
+ let filetype = record.data.FileType;
+ let text = '';
+
+ options.forEach(function(option) {
+ let key = option.Key;
+ if (filetype === 'list') {
+ let values = option.Values.join(',');
+ text += `${key}=${values} `;
+ } else if (filetype === 'sources') {
+ let values = option.Values.join(' ');
+ text += `${key}: ${values}<br>`;
+ } else {
+ throw "unkown file type";
+ }
+ });
+ return text;
+ },
+ flex: 1,
+ },
+ {
+ header: gettext('Comment'),
+ dataIndex: 'Comment',
+ flex: 2,
+ },
+ ],
+
+ addAdditionalInfos: function(gridData, infos) {
+ let me = this;
+
+ let warnings = {};
+ let officialHosts = {};
+
+ let addLine = function(obj, key, line) {
+ if (obj[key]) {
+ obj[key] += "\n";
+ obj[key] += line;
+ } else {
+ obj[key] = line;
+ }
+ };
+
+ for (const info of infos) {
+ const key = `${info.path}:${info.index}`;
+ if (info.kind === 'warning' ||
+ (info.kind === 'ignore-pre-upgrade-warning' && !me.majorUpgradeAllowed)) {
+ addLine(warnings, key, gettext('Warning') + ": " + info.message);
+ } else if (info.kind === 'badge' && info.message === 'official host name') {
+ officialHosts[key] = true;
+ }
+ }
+
+ gridData.forEach(function(record) {
+ const key = `${record.Path}:${record.Index}`;
+ record.OfficialHost = !!officialHosts[key];
+ });
+
+ me.rowBodyFeature.getAdditionalData = function(innerData, rowIndex, record, orig) {
+ let headerCt = this.view.headerCt;
+ let colspan = headerCt.getColumnCount();
+
+ const key = `${innerData.Path}:${innerData.Index}`;
+ const warning_text = warnings[key];
+
+ return {
+ rowBody: '<div style="color: red; white-space: pre-line">' +
+ Ext.String.htmlEncode(warning_text) + '</div>',
+ rowBodyCls: warning_text ? '' : Ext.baseCSSPrefix + 'grid-row-body-hidden',
+ rowBodyColspan: colspan,
+ };
+ };
+ },
+
+ initComponent: function() {
+ let me = this;
+
+ if (!me.nodename) {
+ throw "no node name specified";
+ }
+
+ let store = Ext.create('Ext.data.Store', {
+ model: 'apt-repolist',
+ groupField: 'Path',
+ sorters: [
+ {
+ property: 'Index',
+ direction: 'ASC',
+ },
+ ],
+ });
+
+ let rowBodyFeature = Ext.create('Ext.grid.feature.RowBody', {});
+
+ let groupingFeature = Ext.create('Ext.grid.feature.Grouping', {
+ groupHeaderTpl: '{[ "File: " + values.name ]} ({rows.length} ' +
+ 'repositor{[values.rows.length > 1 ? "ies" : "y"]})',
+ enableGroupingMenu: false,
+ });
+
+ let sm = Ext.create('Ext.selection.RowModel', {});
+
+ Ext.apply(me, {
+ store: store,
+ selModel: sm,
+ rowBodyFeature: rowBodyFeature,
+ features: [groupingFeature, rowBodyFeature],
+ });
+
+ me.callParent();
+ },
+});
+
+Ext.define('Proxmox.node.APTRepositories', {
+ extend: 'Ext.panel.Panel',
+
+ xtype: 'proxmoxNodeAPTRepositories',
+ mixins: ['Proxmox.Mixin.CBind'],
+
+ digest: undefined,
+
+ viewModel: {
+ data: {
+ errorCount: 0,
+ subscriptionActive: '',
+ noSubscriptionRepo: '',
+ enterpriseRepo: '',
+ },
+ formulas: {
+ noErrors: (get) => get('errorCount') === 0,
+ mainWarning: function(get) {
+ // Not yet initialized
+ if (get('subscriptionActive') === '' ||
+ get('enterpriseRepo') === '') {
+ return '';
+ }
+
+ let withStyle = (msg) => "<div style='color:red;'><i class='fa fa-fw " +
+ "fa-exclamation-triangle'></i>" + gettext('Warning') + ': ' + msg + "</div>";
+
+ if (!get('subscriptionActive') && get('enterpriseRepo')) {
+ return withStyle(gettext('The enterprise repository is ' +
+ 'enabled, but there is no active subscription!'));
+ }
+
+ if (get('noSubscriptionRepo')) {
+ return withStyle(gettext('The no-subscription repository is ' +
+ 'not recommended for production use!'));
+ }
+
+ if (!get('enterpriseRepo') && !get('noSubscriptionRepo')) {
+ return withStyle(gettext('No Proxmox repository is enabled!'));
+ }
+
+ return '';
+ },
+ },
+ },
+
+ items: [
+ {
+ title: gettext('Warning'),
+ name: 'repositoriesMainWarning',
+ xtype: 'panel',
+ bind: {
+ title: '{mainWarning}',
+ hidden: '{!mainWarning}',
+ },
+ },
+ {
+ xtype: 'proxmoxNodeAPTRepositoriesErrors',
+ name: 'repositoriesErrors',
+ hidden: true,
+ bind: {
+ hidden: '{noErrors}',
+ },
+ },
+ {
+ xtype: 'proxmoxNodeAPTRepositoriesGrid',
+ name: 'repositoriesGrid',
+ cbind: {
+ nodename: '{nodename}',
+ },
+ majorUpgradeAllowed: false, // TODO get release information from an API call?
+ },
+ ],
+
+ check_subscription: function() {
+ let me = this;
+ let vm = me.getViewModel();
+
+ Proxmox.Utils.API2Request({
+ url: `/nodes/${me.nodename}/subscription`,
+ method: 'GET',
+ failure: function(response, opts) {
+ Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+ },
+ success: function(response, opts) {
+ const res = response.result;
+ const subscription = !(res === null || res === undefined ||
+ !res || res.data.status.toLowerCase() !== 'active');
+ vm.set('subscriptionActive', subscription);
+ },
+ });
+ },
+
+ updateStandardRepos: function(standardRepos) {
+ let me = this;
+ let vm = me.getViewModel();
+
+ for (const standardRepo of standardRepos) {
+ const handle = standardRepo.handle;
+ const status = standardRepo.status;
+
+ if (handle === "enterprise") {
+ vm.set('enterpriseRepo', status);
+ } else if (handle === "no-subscription") {
+ vm.set('noSubscriptionRepo', status);
+ }
+ }
+ },
+
+ reload: function() {
+ let me = this;
+ let vm = me.getViewModel();
+ let repoGrid = me.down('proxmoxNodeAPTRepositoriesGrid');
+ let errorGrid = me.down('proxmoxNodeAPTRepositoriesErrors');
+
+ me.store.load(function(records, operation, success) {
+ let gridData = [];
+ let errors = [];
+ let digest;
+
+ if (success && records.length > 0) {
+ let data = records[0].data;
+ let files = data.files;
+ errors = data.errors;
+ digest = data.digest;
+
+ files.forEach(function(file) {
+ for (let n = 0; n < file.repositories.length; n++) {
+ let repo = file.repositories[n];
+ repo.Path = file.path;
+ repo.Index = n;
+ gridData.push(repo);
+ }
+ });
+
+ repoGrid.addAdditionalInfos(gridData, data.infos);
+ repoGrid.store.loadData(gridData);
+
+ me.updateStandardRepos(data['standard-repos']);
+ }
+
+ me.digest = digest;
+
+ vm.set('errorCount', errors.length);
+ errorGrid.store.loadData(errors);
+ });
+
+ me.check_subscription();
+ },
+
+ listeners: {
+ activate: function() {
+ let me = this;
+ me.reload();
+ },
+ },
+
+ initComponent: function() {
+ let me = this;
+
+ if (!me.nodename) {
+ throw "no node name specified";
+ }
+
+ let store = Ext.create('Ext.data.Store', {
+ proxy: {
+ type: 'proxmox',
+ url: `/api2/json/nodes/${me.nodename}/apt/repositories`,
+ },
+ });
+
+ Ext.apply(me, { store: store });
+
+ Proxmox.Utils.monStoreErrors(me, me.store, true);
+
+ me.callParent();
+ },
+});
--
2.30.2
^ permalink raw reply [flat|nested] 15+ messages in thread* [pve-devel] [PATCH v7 proxmox-widget-toolkit 2/2] add buttons for add/enable/disable
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
` (5 preceding siblings ...)
2021-06-23 13:38 ` [pve-devel] [PATCH v7 proxmox-widget-toolkit 1/2] add UI for APT repositories Fabian Ebner
@ 2021-06-23 13:39 ` Fabian Ebner
2021-06-23 13:39 ` [pve-devel] [PATCH v7 pve-rs 1/1] add bindings for proxmox-apt Fabian Ebner
` (5 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Fabian Ebner @ 2021-06-23 13:39 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
New in v7.
src/node/APTRepositories.js | 80 +++++++++++++++++++++++++++++++++++++
1 file changed, 80 insertions(+)
diff --git a/src/node/APTRepositories.js b/src/node/APTRepositories.js
index 30c31ec..2121a0f 100644
--- a/src/node/APTRepositories.js
+++ b/src/node/APTRepositories.js
@@ -63,6 +63,46 @@ Ext.define('Proxmox.node.APTRepositoriesGrid', {
me.up('proxmoxNodeAPTRepositories').reload();
},
},
+ {
+ text: gettext('Add'),
+ menu: {
+ plain: true,
+ itemId: "addMenu",
+ items: [],
+ },
+ },
+ {
+ xtype: 'proxmoxButton',
+ text: gettext('Enable') + '/' + gettext('Disable'),
+ disabled: true,
+ handler: function(button, event, record) {
+ let me = this;
+ let panel = me.up('proxmoxNodeAPTRepositories');
+
+ let params = {
+ path: record.data.Path,
+ index: record.data.Index,
+ enabled: record.data.Enabled ? 0 : 1, // invert
+ };
+
+ if (panel.digest !== undefined) {
+ params.digest = panel.digest;
+ }
+
+ Proxmox.Utils.API2Request({
+ url: `/nodes/${panel.nodename}/apt/repositories`,
+ method: 'POST',
+ params: params,
+ failure: function(response, opts) {
+ Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+ panel.reload();
+ },
+ success: function(response, opts) {
+ panel.reload();
+ },
+ });
+ },
+ },
],
sortableColumns: false,
@@ -340,6 +380,9 @@ Ext.define('Proxmox.node.APTRepositories', {
let me = this;
let vm = me.getViewModel();
+ let menu = me.down('#addMenu');
+ menu.removeAll();
+
for (const standardRepo of standardRepos) {
const handle = standardRepo.handle;
const status = standardRepo.status;
@@ -349,6 +392,43 @@ Ext.define('Proxmox.node.APTRepositories', {
} else if (handle === "no-subscription") {
vm.set('noSubscriptionRepo', status);
}
+
+ let status_text = '';
+ if (status !== undefined && status !== null) {
+ status_text = Ext.String.format(
+ ' ({0}, {1})',
+ gettext('configured'),
+ status ? gettext('enabled') : gettext('disabled'),
+ );
+ }
+
+ menu.add({
+ text: standardRepo.name + status_text,
+ disabled: status !== undefined && status !== null,
+ repoHandle: handle,
+ handler: function(menuItem) {
+ let params = {
+ handle: menuItem.repoHandle,
+ };
+
+ if (me.digest !== undefined) {
+ params.digest = me.digest;
+ }
+
+ Proxmox.Utils.API2Request({
+ url: `/nodes/${me.nodename}/apt/repositories`,
+ method: 'PUT',
+ params: params,
+ failure: function(response, opts) {
+ Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+ me.reload();
+ },
+ success: function(response, opts) {
+ me.reload();
+ },
+ });
+ },
+ });
}
},
--
2.30.2
^ permalink raw reply [flat|nested] 15+ messages in thread* [pve-devel] [PATCH v7 pve-rs 1/1] add bindings for proxmox-apt
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
` (6 preceding siblings ...)
2021-06-23 13:39 ` [pve-devel] [PATCH v7 proxmox-widget-toolkit 2/2] add buttons for add/enable/disable Fabian Ebner
@ 2021-06-23 13:39 ` Fabian Ebner
2021-06-30 19:17 ` [pve-devel] applied: " Thomas Lamprecht
2021-06-23 13:39 ` [pve-devel] [PATCH v7 pve-manager 1/3] api: apt: add call for repository information Fabian Ebner
` (4 subsequent siblings)
12 siblings, 1 reply; 15+ messages in thread
From: Fabian Ebner @ 2021-06-23 13:39 UTC (permalink / raw)
To: pve-devel
which contains includes logic for repository handling.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Changes from v6:
* have repositories() return everything at once
* base on now existing pve-rs
* don't use raw_return
* add bindings for add/change
Cargo.toml | 2 +
Makefile | 6 +-
src/apt/mod.rs | 1 +
src/apt/repositories.rs | 128 ++++++++++++++++++++++++++++++++++++++++
src/lib.rs | 4 +-
5 files changed, 136 insertions(+), 5 deletions(-)
create mode 100644 src/apt/mod.rs
create mode 100644 src/apt/repositories.rs
diff --git a/Cargo.toml b/Cargo.toml
index 9c274ff..d39c1ad 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -19,4 +19,6 @@ crate-type = [ "cdylib" ]
anyhow = "1.0"
proxmox = { version = "0.11.5", default-features = false }
perlmod = { version = "0.5.1", features = [ "exporter" ] }
+proxmox-apt = "0.2.0"
proxmox-openid = "0.5.0"
+serde = "1.0"
diff --git a/Makefile b/Makefile
index e340623..9701c8e 100644
--- a/Makefile
+++ b/Makefile
@@ -14,10 +14,12 @@ DEBS=$(MAIN_DEB) $(DBGSYM_DEB)
DESTDIR=
-PM_DIRS :=
+PM_DIRS := \
+ PVE/RS/APT
PM_FILES := \
- PVE/RS/OpenId.pm
+ PVE/RS/OpenId.pm \
+ PVE/RS/APT/Repositories.pm
ifeq ($(BUILD_MODE), release)
CARGO_BUILD_ARGS += --release
diff --git a/src/apt/mod.rs b/src/apt/mod.rs
new file mode 100644
index 0000000..574c1a7
--- /dev/null
+++ b/src/apt/mod.rs
@@ -0,0 +1 @@
+mod repositories;
diff --git a/src/apt/repositories.rs b/src/apt/repositories.rs
new file mode 100644
index 0000000..3a421f0
--- /dev/null
+++ b/src/apt/repositories.rs
@@ -0,0 +1,128 @@
+#[perlmod::package(name = "PVE::RS::APT::Repositories", lib = "pve_rs")]
+mod export {
+ use std::convert::TryInto;
+
+ use anyhow::{bail, Error};
+ use serde::{Deserialize, Serialize};
+
+ use proxmox_apt::repositories::{
+ APTRepositoryFile, APTRepositoryFileError, APTRepositoryInfo, APTStandardRepository,
+ };
+
+ #[derive(Deserialize, Serialize)]
+ #[serde(rename_all = "kebab-case")]
+ /// Result for the repositories() function
+ pub struct RepositoriesResult {
+ /// Successfully parsed files.
+ pub files: Vec<APTRepositoryFile>,
+
+ /// Errors for files that could not be parsed or read.
+ pub errors: Vec<APTRepositoryFileError>,
+
+ /// Common digest for successfully parsed files.
+ pub digest: String,
+
+ /// Additional information/warnings about repositories.
+ pub infos: Vec<APTRepositoryInfo>,
+
+ /// Standard repositories and their configuration status.
+ pub standard_repos: Vec<APTStandardRepository>,
+ }
+
+ #[derive(Deserialize, Serialize)]
+ #[serde(rename_all = "kebab-case")]
+ /// For changing an existing repository.
+ pub struct ChangeProperties {
+ /// Whether the repository should be enabled or not.
+ pub enabled: Option<bool>,
+ }
+
+ /// Get information about configured and standard repositories.
+ #[export]
+ pub fn repositories() -> Result<RepositoriesResult, Error> {
+ let (files, errors, digest) = proxmox_apt::repositories::repositories()?;
+ let digest = proxmox::tools::digest_to_hex(&digest);
+
+ let infos = proxmox_apt::repositories::check_repositories(&files)?;
+ let standard_repos = proxmox_apt::repositories::standard_repositories("pve", &files);
+
+ Ok(RepositoriesResult {
+ files,
+ errors,
+ digest,
+ infos,
+ standard_repos,
+ })
+ }
+
+ /// Add the repository identified by the `handle`.
+ ///
+ /// The `digest` parameter asserts that the configuration has not been modified.
+ #[export]
+ pub fn add_repository(handle: &str, digest: Option<&str>) -> Result<(), Error> {
+ let (mut files, _errors, current_digest) = proxmox_apt::repositories::repositories()?;
+
+ if let Some(digest) = digest {
+ let expected_digest = proxmox::tools::hex_to_digest(digest)?;
+ if expected_digest != current_digest {
+ bail!("detected modified configuration - file changed by other user? Try again.");
+ }
+ }
+
+ let (repo, path) =
+ proxmox_apt::repositories::get_standard_repository(handle.try_into()?, "pve")?;
+
+ if let Some(file) = files.iter_mut().find(|file| file.path == path) {
+ file.repositories.push(repo);
+
+ file.write()?;
+ } else {
+ let mut file = match APTRepositoryFile::new(&path)? {
+ Some(file) => file,
+ None => bail!("invalid path - {}", path),
+ };
+
+ file.repositories.push(repo);
+
+ file.write()?;
+ }
+
+ Ok(())
+ }
+
+ /// Change the properties of the specified repository.
+ ///
+ /// The `digest` parameter asserts that the configuration has not been modified.
+ #[export]
+ pub fn change_repository(
+ path: &str,
+ index: usize,
+ options: ChangeProperties,
+ digest: Option<&str>,
+ ) -> Result<(), Error> {
+ let (mut files, _errors, current_digest) = proxmox_apt::repositories::repositories()?;
+
+ if let Some(digest) = digest {
+ let expected_digest = proxmox::tools::hex_to_digest(digest)?;
+ if expected_digest != current_digest {
+ bail!("detected modified configuration - file changed by other user? Try again.");
+ }
+ }
+
+ if let Some(file) = files.iter_mut().find(|file| file.path == path) {
+ if let Some(repo) = file.repositories.get_mut(index) {
+ if let Some(enabled) = options.enabled {
+ repo.set_enabled(enabled);
+ }
+
+ file.write()?;
+ } else {
+ bail!("invalid index - {}", index);
+ }
+ } else {
+ bail!("invalid path - {}", path);
+ }
+
+ Ok(())
+ }
+}
diff --git a/src/lib.rs b/src/lib.rs
index ec61052..cad331d 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -1,4 +1,2 @@
-
-// TODO: add submodule here
-//pub mod apt;
+pub mod apt;
pub mod openid;
--
2.30.2
^ permalink raw reply [flat|nested] 15+ messages in thread* [pve-devel] applied: [PATCH v7 pve-rs 1/1] add bindings for proxmox-apt
2021-06-23 13:39 ` [pve-devel] [PATCH v7 pve-rs 1/1] add bindings for proxmox-apt Fabian Ebner
@ 2021-06-30 19:17 ` Thomas Lamprecht
0 siblings, 0 replies; 15+ messages in thread
From: Thomas Lamprecht @ 2021-06-30 19:17 UTC (permalink / raw)
To: Proxmox VE development discussion, Fabian Ebner
On 23.06.21 15:39, Fabian Ebner wrote:
> which contains includes logic for repository handling.
>
> Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
> ---
>
> Changes from v6:
> * have repositories() return everything at once
> * base on now existing pve-rs
> * don't use raw_return
> * add bindings for add/change
>
> Cargo.toml | 2 +
> Makefile | 6 +-
> src/apt/mod.rs | 1 +
> src/apt/repositories.rs | 128 ++++++++++++++++++++++++++++++++++++++++
> src/lib.rs | 4 +-
> 5 files changed, 136 insertions(+), 5 deletions(-)
> create mode 100644 src/apt/mod.rs
> create mode 100644 src/apt/repositories.rs
>
>
applied, thanks!
^ permalink raw reply [flat|nested] 15+ messages in thread
* [pve-devel] [PATCH v7 pve-manager 1/3] api: apt: add call for repository information
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
` (7 preceding siblings ...)
2021-06-23 13:39 ` [pve-devel] [PATCH v7 pve-rs 1/1] add bindings for proxmox-apt Fabian Ebner
@ 2021-06-23 13:39 ` Fabian Ebner
2021-06-23 13:39 ` [pve-devel] [PATCH v7 pve-manager 2/3] api: apt: add PUT and POST handler for repositories Fabian Ebner
` (3 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Fabian Ebner @ 2021-06-23 13:39 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Dependency bump for pve-rs needed.
Changes from v6:
* adapt to backend API changes
* merged the checkrepositories call into this one
* the call now also gives a list of standard repositories and their
configuration status.
PVE/API2/APT.pm | 200 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 200 insertions(+)
diff --git a/PVE/API2/APT.pm b/PVE/API2/APT.pm
index fb4954e7..36d0e67a 100644
--- a/PVE/API2/APT.pm
+++ b/PVE/API2/APT.pm
@@ -19,6 +19,7 @@ use PVE::INotify;
use PVE::Exception;
use PVE::RESTHandler;
use PVE::RPCEnvironment;
+use PVE::RS::APT::Repositories;
use PVE::API2Tools;
use JSON;
@@ -66,6 +67,7 @@ __PACKAGE__->register_method({
my $res = [
{ id => 'changelog' },
+ { id => 'repositories' },
{ id => 'update' },
{ id => 'versions' },
];
@@ -478,6 +480,204 @@ __PACKAGE__->register_method({
return $data;
}});
+__PACKAGE__->register_method({
+ name => 'repositories',
+ path => 'repositories',
+ method => 'GET',
+ proxyto => 'node',
+ description => "Get APT repository information.",
+ permissions => {
+ check => ['perm', '/nodes/{node}', [ 'Sys.Audit' ]],
+ },
+ parameters => {
+ additionalProperties => 0,
+ properties => {
+ node => get_standard_option('pve-node'),
+ },
+ },
+ returns => {
+ type => "object",
+ description => "Result from parsing the APT repository files in /etc/apt/.",
+ properties => {
+ files => {
+ type => "array",
+ description => "List of parsed repository files.",
+ items => {
+ type => "object",
+ properties => {
+ path => {
+ type => "string",
+ description => "Path to the problematic file.",
+ },
+ 'file-type' => {
+ type => "string",
+ enum => [ 'list', 'sources' ],
+ description => "Format of the file.",
+ },
+ repositories => {
+ type => "array",
+ description => "The parsed repositories.",
+ items => {
+ type => "object",
+ properties => {
+ Types => {
+ type => "array",
+ description => "List of package types.",
+ items => {
+ type => "string",
+ enum => [ 'deb', 'deb-src' ],
+ },
+ },
+ URIs => {
+ description => "List of repository URIs.",
+ type => "array",
+ items => {
+ type => "string",
+ },
+ },
+ Suites => {
+ type => "array",
+ description => "List of package distribuitions",
+ items => {
+ type => "string",
+ },
+ },
+ Components => {
+ type => "array",
+ description => "List of repository components",
+ optional => 1, # not present if suite is absolute
+ items => {
+ type => "string",
+ },
+ },
+ Options => {
+ type => "array",
+ description => "Additional options",
+ optional => 1,
+ items => {
+ type => "object",
+ properties => {
+ Key => {
+ type => "string",
+ },
+ Values => {
+ type => "array",
+ items => {
+ type => "string",
+ },
+ },
+ },
+ },
+ },
+ Comment => {
+ type => "string",
+ description => "Associated comment",
+ optional => 1,
+ },
+ FileType => {
+ type => "string",
+ enum => [ 'list', 'sources' ],
+ description => "Format of the defining file.",
+ },
+ Enabled => {
+ type => "boolean",
+ description => "Whether the repository is enabled or not",
+ },
+ },
+ },
+ },
+ digest => {
+ type => "array",
+ description => "Digest of the file as bytes.",
+ items => {
+ type => "integer",
+ },
+ },
+ },
+ },
+ },
+ errors => {
+ type => "array",
+ description => "List of problematic repository files.",
+ items => {
+ type => "object",
+ properties => {
+ path => {
+ type => "string",
+ description => "Path to the problematic file.",
+ },
+ error => {
+ type => "string",
+ description => "The error message",
+ },
+ },
+ },
+ },
+ digest => {
+ type => "string",
+ description => "Common digest of all files.",
+ },
+ infos => {
+ type => "array",
+ description => "Additional information/warnings for APT repositories.",
+ items => {
+ type => "object",
+ properties => {
+ path => {
+ type => "string",
+ description => "Path to the associated file.",
+ },
+ index => {
+ type => "string",
+ description => "Index of the associated repository within the file.",
+ },
+ property => {
+ type => "string",
+ description => "Property from which the info originates.",
+ optional => 1,
+ },
+ kind => {
+ type => "string",
+ description => "Kind of the information (e.g. warning).",
+ },
+ message => {
+ type => "string",
+ description => "Information message.",
+ }
+ },
+ },
+ },
+ 'standard-repos' => {
+ type => "array",
+ description => "List of standard repositories and their configuration status",
+ items => {
+ type => "object",
+ properties => {
+ handle => {
+ type => "string",
+ description => "Handle to identify the repository.",
+ },
+ name => {
+ type => "string",
+ description => "Full name of the repository.",
+ },
+ status => {
+ type => "boolean",
+ optional => 1,
+ description => "Indicating enabled/disabled status, if the " .
+ "repository is configured.",
+ },
+ },
+ },
+ },
+ },
+ },
+ code => sub {
+ my ($param) = @_;
+
+ return PVE::RS::APT::Repositories::repositories();
+ }});
+
__PACKAGE__->register_method({
name => 'versions',
path => 'versions',
--
2.30.2
^ permalink raw reply [flat|nested] 15+ messages in thread* [pve-devel] [PATCH v7 pve-manager 2/3] api: apt: add PUT and POST handler for repositories
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
` (8 preceding siblings ...)
2021-06-23 13:39 ` [pve-devel] [PATCH v7 pve-manager 1/3] api: apt: add call for repository information Fabian Ebner
@ 2021-06-23 13:39 ` Fabian Ebner
2021-06-23 13:39 ` [pve-devel] [PATCH v7 pve-manager 3/3] ui: add panel for listing APT repositories Fabian Ebner
` (2 subsequent siblings)
12 siblings, 0 replies; 15+ messages in thread
From: Fabian Ebner @ 2021-06-23 13:39 UTC (permalink / raw)
To: pve-devel
To allow adding/modifying them. Currently the only possible modification is
enable/disable.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
New in v7.
PVE/API2/APT.pm | 88 +++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 88 insertions(+)
diff --git a/PVE/API2/APT.pm b/PVE/API2/APT.pm
index 36d0e67a..c58203a7 100644
--- a/PVE/API2/APT.pm
+++ b/PVE/API2/APT.pm
@@ -678,6 +678,94 @@ __PACKAGE__->register_method({
return PVE::RS::APT::Repositories::repositories();
}});
+__PACKAGE__->register_method({
+ name => 'add_repository',
+ path => 'repositories',
+ method => 'PUT',
+ description => "Add a standard repository to the configuration",
+ permissions => {
+ check => ['perm', '/nodes/{node}', [ 'Sys.Modify' ]],
+ },
+ protected => 1,
+ proxyto => 'node',
+ parameters => {
+ additionalProperties => 0,
+ properties => {
+ node => get_standard_option('pve-node'),
+ handle => {
+ type => 'string',
+ description => "Handle that identifies a repository.",
+ },
+ digest => {
+ type => "string",
+ description => "Digest to detect modifications.",
+ maxLength => 80,
+ optional => 1,
+ },
+ },
+ },
+ returns => {
+ type => 'null',
+ },
+ code => sub {
+ my ($param) = @_;
+
+ PVE::RS::APT::Repositories::add_repository($param->{handle}, $param->{digest});
+ }});
+
+__PACKAGE__->register_method({
+ name => 'change_repository',
+ path => 'repositories',
+ method => 'POST',
+ description => "Change the properties of a repository. Currently only allows enabling/disabling.",
+ permissions => {
+ check => ['perm', '/nodes/{node}', [ 'Sys.Modify' ]],
+ },
+ protected => 1,
+ proxyto => 'node',
+ parameters => {
+ additionalProperties => 0,
+ properties => {
+ node => get_standard_option('pve-node'),
+ path => {
+ type => 'string',
+ description => "Path to the containing file.",
+ },
+ index => {
+ type => 'integer',
+ description => "Index within the file (starting from 0).",
+ },
+ enabled => {
+ type => 'boolean',
+ description => "Whether the repository should be enabled or not.",
+ optional => 1,
+ },
+ digest => {
+ type => "string",
+ description => "Digest to detect modifications.",
+ maxLength => 80,
+ optional => 1,
+ },
+ },
+ },
+ returns => {
+ type => 'null',
+ },
+ code => sub {
+ my ($param) = @_;
+
+ my $options = {
+ enabled => $param->{enabled},
+ };
+
+ PVE::RS::APT::Repositories::change_repository(
+ $param->{path},
+ $param->{index},
+ $options,
+ $param->{digest}
+ );
+ }});
+
__PACKAGE__->register_method({
name => 'versions',
path => 'versions',
--
2.30.2
^ permalink raw reply [flat|nested] 15+ messages in thread* [pve-devel] [PATCH v7 pve-manager 3/3] ui: add panel for listing APT repositories
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
` (9 preceding siblings ...)
2021-06-23 13:39 ` [pve-devel] [PATCH v7 pve-manager 2/3] api: apt: add PUT and POST handler for repositories Fabian Ebner
@ 2021-06-23 13:39 ` Fabian Ebner
2021-06-23 18:02 ` [pve-devel] partially-applied: [PATCH-SERIES v7] APT repositories API/UI Thomas Lamprecht
2021-07-05 6:51 ` [pve-devel] applied-series: " Thomas Lamprecht
12 siblings, 0 replies; 15+ messages in thread
From: Fabian Ebner @ 2021-06-23 13:39 UTC (permalink / raw)
To: pve-devel
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
---
Dependency bump for proxmox-widget-toolkit needed.
Changes from v6:
* remove majorUpgradeAllowed parameter (the plan is to have an API call
returning release information and use that in proxmox-widget-toolkit)
* Have 'Updates' be our parent in the config panel/tree.
www/manager6/node/Config.js | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/www/manager6/node/Config.js b/www/manager6/node/Config.js
index 235a7480..0d2c690c 100644
--- a/www/manager6/node/Config.js
+++ b/www/manager6/node/Config.js
@@ -242,6 +242,15 @@ Ext.define('PVE.node.Config', {
},
nodename: nodename,
});
+
+ me.items.push({
+ xtype: 'proxmoxNodeAPTRepositories',
+ title: gettext('APT Repositories'),
+ iconCls: 'fa fa-files-o',
+ itemId: 'aptrepositories',
+ nodename: nodename,
+ groups: ['apt'],
+ });
}
}
--
2.30.2
^ permalink raw reply [flat|nested] 15+ messages in thread* [pve-devel] partially-applied: [PATCH-SERIES v7] APT repositories API/UI
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
` (10 preceding siblings ...)
2021-06-23 13:39 ` [pve-devel] [PATCH v7 pve-manager 3/3] ui: add panel for listing APT repositories Fabian Ebner
@ 2021-06-23 18:02 ` Thomas Lamprecht
2021-07-05 6:51 ` [pve-devel] applied-series: " Thomas Lamprecht
12 siblings, 0 replies; 15+ messages in thread
From: Thomas Lamprecht @ 2021-06-23 18:02 UTC (permalink / raw)
To: Proxmox VE development discussion, Fabian Ebner
On 23.06.21 15:38, Fabian Ebner wrote:
> List the configured repositories and have some basic checks for them.
> Allow adding standard Proxmox repositories and enabling/disabling repositories.
>
>
> Changes from v6:
> * do not include the upgrade functionality for now, focus on being
> able to add standard repositories and enable/disable them
> * put impl blocks and struct declarations together in proxmox-apt
> * merge repositories and check_repositories get calls
> * see individual patches for more
>
> Older changes can be found here:
> https://lists.proxmox.com/pipermail/pve-devel/2021-June/048598.html
>
>
> The dependency for pve-rs to proxmox-apt is already in the patches.
> Additionally, pve-manager depends on pve-rs and proxmox-widget-toolkit.
>
>
> proxmox-apt:
>
> Fabian Ebner (5):
> initial commit
> add files for Debian packaging
> add more functions to check repositories
> add handling of Proxmox standard repositories
> bump version to 0.2.0-1
> proxmox-widget-toolkit:
>
> Fabian Ebner (2):
> add UI for APT repositories
> add buttons for add/enable/disable
>
> src/Makefile | 1 +
> src/node/APTRepositories.js | 503 ++++++++++++++++++++++++++++++++++++
> 2 files changed, 504 insertions(+)
> create mode 100644 src/node/APTRepositories.js
>
applied above, i.e., proxmox-apt and widget-toolkit patches, as those itself do not show up
anywhere and that should make it easier to adapt and test in collaboration - thanks!
FYI: The widget-toolkit got some followups on top.
^ permalink raw reply [flat|nested] 15+ messages in thread* [pve-devel] applied-series: [PATCH-SERIES v7] APT repositories API/UI
2021-06-23 13:38 [pve-devel] [PATCH-SERIES v7] APT repositories API/UI Fabian Ebner
` (11 preceding siblings ...)
2021-06-23 18:02 ` [pve-devel] partially-applied: [PATCH-SERIES v7] APT repositories API/UI Thomas Lamprecht
@ 2021-07-05 6:51 ` Thomas Lamprecht
12 siblings, 0 replies; 15+ messages in thread
From: Thomas Lamprecht @ 2021-07-05 6:51 UTC (permalink / raw)
To: Proxmox VE development discussion, Fabian Ebner
On 23.06.21 15:38, Fabian Ebner wrote:
> pve-manager:
>
> Fabian Ebner (3):
> api: apt: add call for repository information
> api: apt: add PUT and POST handler for repositories
> ui: add panel for listing APT repositories
>
> PVE/API2/APT.pm | 288 ++++++++++++++++++++++++++++++++++++
> www/manager6/node/Config.js | 9 ++
> 2 files changed, 297 insertions(+)
>
applied the remaining (above) ones, thanks!
^ permalink raw reply [flat|nested] 15+ messages in thread