Re: [pbs-devel] [PATCH v2 backup 14/27] add acme config

Re: [pbs-devel] [PATCH v2 backup 14/27] add acme config

[Dietmar Maurer]

> On 04/29/2021 1:36 PM Wolfgang Bumiller <w.bumiller@proxmox.com> wrote:
> 
>  
> On Thu, Apr 29, 2021 at 12:48:52PM +0200, Dietmar Maurer wrote:
> > What is the purpose of this AccountName wrapper type?
> > 
> > I would prefer to simply use String...
> 
> If you want to... On the other hand I was thinking about adding a macro
> for the boilerplate string stuff.

I simply do not see any advantage in this case.

Re: [pbs-devel] [PATCH v2 backup 14/27] add acme config

[Wolfgang Bumiller]

On Thu, Apr 29, 2021 at 02:34:28PM +0200, Dietmar Maurer wrote:
> > > 12 | use crate::acme::AcmeClient;
> > >    |            ^^^^
> > >    |            |
> > >    |            unresolved import
> > >    |            help: a similar path exists: `crate::config::acme`
> > > 
> > > error: aborting due to 2 previous errors
> > 
> > looks like I mis-ordered the patches, this will be in 17/27
> 
> I think there are also cyclic dependencies. Please can you resend the
> patches in correct order, so that each step compiles?

Done

Re: [pbs-devel] [PATCH v2 backup 14/27] add acme config

[Dietmar Maurer]

> > 12 | use crate::acme::AcmeClient;
> >    |            ^^^^
> >    |            |
> >    |            unresolved import
> >    |            help: a similar path exists: `crate::config::acme`
> > 
> > error: aborting due to 2 previous errors
> 
> looks like I mis-ordered the patches, this will be in 17/27

I think there are also cyclic dependencies. Please can you resend the
patches in correct order, so that each step compiles?

Re: [pbs-devel] [PATCH v2 backup 14/27] add acme config

[Wolfgang Bumiller]

On Thu, Apr 29, 2021 at 12:48:52PM +0200, Dietmar Maurer wrote:
> What is the purpose of this AccountName wrapper type?
> 
> I would prefer to simply use String...

If you want to... On the other hand I was thinking about adding a macro
for the boilerplate string stuff.

I mean, I write it once, then use `{ type: AccountName }` in the api
code and it's easy to see what it means and more difficult to get
wrong...

Re: [pbs-devel] [PATCH v2 backup 14/27] add acme config

[Wolfgang Bumiller]

On Thu, Apr 29, 2021 at 12:53:01PM +0200, Dietmar Maurer wrote:
> I get a compile error:
> 
> make
> cargo build --release
>    Compiling proxmox-backup v1.1.5 (/home/dietmar/rust/proxmox-backup)
> error[E0432]: unresolved import `crate::acme`
>   --> src/config/acme/plugin.rs:22:12
>    |
> 22 | use crate::acme::AcmeClient;
>    |            ^^^^
>    |            |
>    |            unresolved import
>    |            help: a similar path exists: `crate::config::acme`
> 
> error[E0432]: unresolved import `crate::acme`
>   --> src/config/node.rs:12:12
>    |
> 12 | use crate::acme::AcmeClient;
>    |            ^^^^
>    |            |
>    |            unresolved import
>    |            help: a similar path exists: `crate::config::acme`
> 
> error: aborting due to 2 previous errors

looks like I mis-ordered the patches, this will be in 17/27

Re: [pbs-devel] [PATCH v2 backup 14/27] add acme config

[Dietmar Maurer]

I get a compile error:

make
cargo build --release
    Compiling proxmox-backup v1.1.5 (/home/dietmar/rust/proxmox-backup)
error[E0432]: unresolved import `crate::acme`
   --> src/config/acme/plugin.rs:22:12
    |
22 | use crate::acme::AcmeClient;
    |            ^^^^
    |            |
    |            unresolved import
    |            help: a similar path exists: `crate::config::acme`

error[E0432]: unresolved import `crate::acme`
   --> src/config/node.rs:12:12
    |
12 | use crate::acme::AcmeClient;
    |            ^^^^
    |            |
    |            unresolved import
    |            help: a similar path exists: `crate::config::acme`

error: aborting due to 2 previous errors


On 4/22/21 4:02 PM, Wolfgang Bumiller wrote:
> Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
> ---
>   src/config.rs             |   1 +
>   src/config/acme/mod.rs    | 198 ++++++++++++++++++++
>   src/config/acme/plugin.rs | 380 ++++++++++++++++++++++++++++++++++++++
>   3 files changed, 579 insertions(+)
>   create mode 100644 src/config/acme/mod.rs
>   create mode 100644 src/config/acme/plugin.rs
>
> diff --git a/src/config.rs b/src/config.rs
> index 717829e2..94b7fb6c 100644
> --- a/src/config.rs
> +++ b/src/config.rs
> @@ -16,6 +16,7 @@ use proxmox::try_block;
>   use crate::buildcfg;
>   
>   pub mod acl;
> +pub mod acme;
>   pub mod cached_user_info;
>   pub mod datastore;
>   pub mod network;
> diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
> new file mode 100644
> index 00000000..ac409c20
> --- /dev/null
> +++ b/src/config/acme/mod.rs
> @@ -0,0 +1,198 @@
> +use std::collections::HashMap;
> +use std::fmt;
> +use std::path::Path;
> +
> +use anyhow::{bail, format_err, Error};
> +use serde::{Deserialize, Serialize};
> +
> +use proxmox::api::api;
> +use proxmox::sys::error::SysError;
> +
> +use crate::api2::types::{PROXMOX_SAFE_ID_FORMAT, PROXMOX_SAFE_ID_REGEX};
> +use crate::tools::ControlFlow;
> +
> +pub(crate) const ACME_ACCOUNT_DIR: &str = configdir!("/acme/accounts");
> +
> +pub mod plugin;
> +
> +#[api(
> +    properties: {
> +        name: { type: String },
> +        url: { type: String },
> +    },
> +)]
> +/// An ACME directory endpoint with a name and URL.
> +#[derive(Serialize)]
> +pub struct KnownAcmeDirectory {
> +    /// The ACME directory's name.
> +    pub name: &'static str,
> +
> +    /// The ACME directory's endpoint URL.
> +    pub url: &'static str,
> +}
> +
> +pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
> +    KnownAcmeDirectory {
> +        name: "Let's Encrypt V2",
> +        url: "https://acme-v02.api.letsencrypt.org/directory",
> +    },
> +    KnownAcmeDirectory {
> +        name: "Let's Encrypt V2 Staging",
> +        url: "https://acme-staging-v02.api.letsencrypt.org/directory",
> +    },
> +];
> +
> +pub const DEFAULT_ACME_DIRECTORY_ENTRY: &KnownAcmeDirectory = &KNOWN_ACME_DIRECTORIES[0];
> +
> +pub fn account_path(name: &str) -> String {
> +    format!("{}/{}", ACME_ACCOUNT_DIR, name)
> +}
> +
> +#[api(format: &PROXMOX_SAFE_ID_FORMAT)]
> +/// ACME account name.
> +#[derive(Clone, Eq, PartialEq, Hash, Deserialize, Serialize)]
> +#[serde(transparent)]
> +pub struct AccountName(String);
> +
> +impl AccountName {
> +    pub fn into_string(self) -> String {
> +        self.0
> +    }
> +}
> +
> +impl std::ops::Deref for AccountName {
> +    type Target = str;
> +
> +    #[inline]
> +    fn deref(&self) -> &str {
> +        &self.0
> +    }
> +}
> +
> +impl std::ops::DerefMut for AccountName {
> +    #[inline]
> +    fn deref_mut(&mut self) -> &mut str {
> +        &mut self.0
> +    }
> +}
> +
> +impl AsRef<str> for AccountName {
> +    #[inline]
> +    fn as_ref(&self) -> &str {
> +        self.0.as_ref()
> +    }
> +}
> +
> +impl fmt::Debug for AccountName {
> +    #[inline]
> +    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
> +        fmt::Debug::fmt(&self.0, f)
> +    }
> +}
> +
> +impl fmt::Display for AccountName {
> +    #[inline]
> +    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
> +        fmt::Display::fmt(&self.0, f)
> +    }
> +}
> +
> +pub fn foreach_acme_account<F>(mut func: F) -> Result<(), Error>
> +where
> +    F: FnMut(AccountName) -> ControlFlow<Result<(), Error>>,
> +{
> +    match crate::tools::fs::scan_subdir(-1, ACME_ACCOUNT_DIR, &PROXMOX_SAFE_ID_REGEX) {
> +        Ok(files) => {
> +            for file in files {
> +                let file = file?;
> +                let file_name = unsafe { file.file_name_utf8_unchecked() };
> +
> +                if file_name.starts_with('_') {
> +                    continue;
> +                }
> +
> +                let account_name = AccountName(file_name.to_owned());
> +
> +                if let ControlFlow::Break(result) = func(account_name) {
> +                    return result;
> +                }
> +            }
> +            Ok(())
> +        }
> +        Err(err) if err.not_found() => Ok(()),
> +        Err(err) => Err(err.into()),
> +    }
> +}
> +
> +/// Run a function for each DNS plugin ID.
> +pub fn foreach_dns_plugin<F>(mut func: F) -> Result<(), Error>
> +where
> +    F: FnMut(&str) -> ControlFlow<Result<(), Error>>,
> +{
> +    match crate::tools::fs::read_subdir(-1, "/usr/share/proxmox-acme/dnsapi") {
> +        Ok(files) => {
> +            for file in files.filter_map(Result::ok) {
> +                if let Some(id) = file
> +                    .file_name()
> +                    .to_str()
> +                    .ok()
> +                    .and_then(|name| name.strip_prefix("dns_"))
> +                    .and_then(|name| name.strip_suffix(".sh"))
> +                {
> +                    if let ControlFlow::Break(result) = func(id) {
> +                        return result;
> +                    }
> +                }
> +            }
> +
> +            Ok(())
> +        }
> +        Err(err) if err.not_found() => Ok(()),
> +        Err(err) => Err(err.into()),
> +    }
> +}
> +
> +pub fn mark_account_deactivated(name: &str) -> Result<(), Error> {
> +    let from = account_path(name);
> +    for i in 0..100 {
> +        let to = account_path(&format!("_deactivated_{}_{}", name, i));
> +        if !Path::new(&to).exists() {
> +            return std::fs::rename(&from, &to).map_err(|err| {
> +                format_err!(
> +                    "failed to move account path {:?} to {:?} - {}",
> +                    from,
> +                    to,
> +                    err
> +                )
> +            });
> +        }
> +    }
> +    bail!(
> +        "No free slot to rename deactivated account {:?}, please cleanup {:?}",
> +        from,
> +        ACME_ACCOUNT_DIR
> +    );
> +}
> +
> +pub fn complete_acme_account(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
> +    let mut out = Vec::new();
> +    let _ = foreach_acme_account(|name| {
> +        out.push(name.into_string());
> +        ControlFlow::CONTINUE
> +    });
> +    out
> +}
> +
> +pub fn complete_acme_plugin(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
> +    match plugin::config() {
> +        Ok((config, _digest)) => config
> +            .iter()
> +            .map(|(id, (_type, _cfg))| id.clone())
> +            .collect(),
> +        Err(_) => Vec::new(),
> +    }
> +}
> +
> +pub fn complete_acme_plugin_type(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
> +    vec!["dns".to_string(), "http".to_string()]
> +}
> diff --git a/src/config/acme/plugin.rs b/src/config/acme/plugin.rs
> new file mode 100644
> index 00000000..b0c655c0
> --- /dev/null
> +++ b/src/config/acme/plugin.rs
> @@ -0,0 +1,380 @@
> +use std::future::Future;
> +use std::pin::Pin;
> +use std::process::Stdio;
> +
> +use anyhow::{bail, format_err, Error};
> +use lazy_static::lazy_static;
> +use serde::{Deserialize, Serialize};
> +use serde_json::Value;
> +use tokio::io::AsyncWriteExt;
> +use tokio::process::Command;
> +
> +use proxmox::api::{
> +    api,
> +    schema::*,
> +    section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin},
> +};
> +
> +use proxmox::tools::{fs::replace_file, fs::CreateOptions};
> +
> +use proxmox_acme_rs::{Authorization, Challenge};
> +
> +use crate::acme::AcmeClient;
> +use crate::api2::types::PROXMOX_SAFE_ID_FORMAT;
> +use crate::config::node::AcmeDomain;
> +
> +const ACME_PATH: &str = "/usr/share/proxmox-acme/proxmox-acme";
> +
> +pub const PLUGIN_ID_SCHEMA: Schema = StringSchema::new("ACME Challenge Plugin ID.")
> +    .format(&PROXMOX_SAFE_ID_FORMAT)
> +    .schema();
> +
> +lazy_static! {
> +    pub static ref CONFIG: SectionConfig = init();
> +}
> +
> +#[api(
> +    properties: {
> +        id: { schema: PLUGIN_ID_SCHEMA },
> +    },
> +)]
> +#[derive(Deserialize, Serialize)]
> +/// Standalone ACME Plugin for the http-1 challenge.
> +pub struct StandalonePlugin {
> +    /// Plugin ID.
> +    id: String,
> +}
> +
> +impl Default for StandalonePlugin {
> +    fn default() -> Self {
> +        Self {
> +            id: "standalone".to_string(),
> +        }
> +    }
> +}
> +
> +/// In PVE/PMG we store the plugin's "data" member as base64url encoded string. The UI sends
> +/// regular base64 encoded data. We need to "fix" this up.
> +
> +#[api(
> +    properties: {
> +        id: { schema: PLUGIN_ID_SCHEMA },
> +        disable: {
> +            optional: true,
> +            default: false,
> +        },
> +        "validation-delay": {
> +            default: 30,
> +            optional: true,
> +            minimum: 0,
> +            maximum: 2 * 24 * 60 * 60,
> +        },
> +    },
> +)]
> +/// DNS ACME Challenge Plugin core data.
> +#[derive(Deserialize, Serialize, Updater)]
> +#[serde(rename_all = "kebab-case")]
> +pub struct DnsPluginCore {
> +    /// Plugin ID.
> +    pub(crate) id: String,
> +
> +    /// DNS API Plugin Id.
> +    api: String,
> +
> +    /// Extra delay in seconds to wait before requesting validation.
> +    ///
> +    /// Allows to cope with long TTL of DNS records.
> +    #[serde(skip_serializing_if = "Option::is_none", default)]
> +    validation_delay: Option<u32>,
> +
> +    /// Flag to disable the config.
> +    #[serde(skip_serializing_if = "Option::is_none", default)]
> +    disable: Option<bool>,
> +}
> +
> +#[api(
> +    properties: {
> +        core: { type: DnsPluginCore },
> +    },
> +)]
> +/// DNS ACME Challenge Plugin.
> +#[derive(Deserialize, Serialize)]
> +#[serde(rename_all = "kebab-case")]
> +pub struct DnsPlugin {
> +    #[serde(flatten)]
> +    pub(crate) core: DnsPluginCore,
> +
> +    // FIXME: The `Updater` should allow:
> +    //   * having different descriptions for this and the Updater version
> +    //   * having different `#[serde]` attributes for the Updater
> +    //   * or, well, leaving fields out completely in teh Updater but this means we may need to
> +    //     separate Updater and Builder deriving.
> +    // We handle this property separately in the API calls.
> +    /// DNS plugin data (base64url encoded without padding).
> +    #[serde(with = "proxmox::tools::serde::string_as_base64url_nopad")]
> +    pub(crate) data: String,
> +}
> +
> +impl DnsPlugin {
> +    pub fn decode_data(&self, output: &mut Vec<u8>) -> Result<(), Error> {
> +        Ok(base64::decode_config_buf(&self.data, base64::URL_SAFE_NO_PAD, output)?)
> +    }
> +}
> +
> +//impl DnsPluginUpdater {
> +//    // The UI passes regular base64 data, we need base64url data. In PVE/PMG this happens magically
> +//    // since perl parses both on decode...
> +//    pub fn api_fixup(&mut self) -> Result<(), Error> {
> +//        if let Some(data) = self.data.as_mut() {
> +//            let new = base64::encode_config(&base64::decode(&data)?, base64::URL_SAFE_NO_PAD);
> +//            *data = new;
> +//        }
> +//        Ok(())
> +//    }
> +//}
> +
> +fn init() -> SectionConfig {
> +    let mut config = SectionConfig::new(&PLUGIN_ID_SCHEMA);
> +
> +    let standalone_schema = match &StandalonePlugin::API_SCHEMA {
> +        Schema::Object(schema) => schema,
> +        _ => unreachable!(),
> +    };
> +    let standalone_plugin = SectionConfigPlugin::new(
> +        "standalone".to_string(),
> +        Some("id".to_string()),
> +        standalone_schema,
> +    );
> +    config.register_plugin(standalone_plugin);
> +
> +    let dns_challenge_schema = match DnsPlugin::API_SCHEMA {
> +        Schema::AllOf(ref schema) => schema,
> +        _ => unreachable!(),
> +    };
> +    let dns_challenge_plugin = SectionConfigPlugin::new(
> +        "dns".to_string(),
> +        Some("id".to_string()),
> +        dns_challenge_schema,
> +    );
> +    config.register_plugin(dns_challenge_plugin);
> +
> +    config
> +}
> +
> +pub const ACME_PLUGIN_CFG_FILENAME: &str = "/etc/proxmox-backup/acme/plugins.cfg";
> +pub const ACME_PLUGIN_CFG_LOCKFILE: &str = "/etc/proxmox-backup/acme/.plugins.lck";
> +const LOCK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
> +
> +pub fn read_lock() -> Result<std::fs::File, Error> {
> +    proxmox::tools::fs::open_file_locked(ACME_PLUGIN_CFG_LOCKFILE, LOCK_TIMEOUT, false)
> +}
> +
> +pub fn write_lock() -> Result<std::fs::File, Error> {
> +    proxmox::tools::fs::open_file_locked(ACME_PLUGIN_CFG_LOCKFILE, LOCK_TIMEOUT, true)
> +}
> +
> +pub fn config() -> Result<(PluginData, [u8; 32]), Error> {
> +    let content = proxmox::tools::fs::file_read_optional_string(ACME_PLUGIN_CFG_FILENAME)?
> +        .unwrap_or_else(|| "".to_string());
> +
> +    let digest = openssl::sha::sha256(content.as_bytes());
> +    let mut data = CONFIG.parse(ACME_PLUGIN_CFG_FILENAME, &content)?;
> +
> +    if data.sections.get("standalone").is_none() {
> +        let standalone = StandalonePlugin::default();
> +        data.set_data("standalone", "standalone", &standalone)
> +            .unwrap();
> +    }
> +
> +    Ok((PluginData { data }, digest))
> +}
> +
> +pub fn save_config(config: &PluginData) -> Result<(), Error> {
> +    let raw = CONFIG.write(ACME_PLUGIN_CFG_FILENAME, &config.data)?;
> +
> +    let backup_user = crate::backup::backup_user()?;
> +    let mode = nix::sys::stat::Mode::from_bits_truncate(0o0640);
> +    // set the correct owner/group/permissions while saving file
> +    // owner(rw) = root, group(r)= backup
> +    let options = CreateOptions::new()
> +        .perm(mode)
> +        .owner(nix::unistd::ROOT)
> +        .group(backup_user.gid);
> +
> +    replace_file(ACME_PLUGIN_CFG_FILENAME, raw.as_bytes(), options)?;
> +
> +    Ok(())
> +}
> +
> +pub struct PluginData {
> +    data: SectionConfigData,
> +}
> +
> +impl PluginData {
> +    #[inline]
> +    pub fn remove(&mut self, name: &str) -> Option<(String, Value)> {
> +        self.data.sections.remove(name)
> +    }
> +
> +    #[inline]
> +    pub fn contains_key(&mut self, name: &str) -> bool {
> +        self.data.sections.contains_key(name)
> +    }
> +
> +    #[inline]
> +    pub fn get(&self, name: &str) -> Option<&(String, Value)> {
> +        self.data.sections.get(name)
> +    }
> +
> +    #[inline]
> +    pub fn get_mut(&mut self, name: &str) -> Option<&mut (String, Value)> {
> +        self.data.sections.get_mut(name)
> +    }
> +
> +    // FIXME: Verify the plugin type *exists* and check its config schema...
> +    pub fn insert(&mut self, id: String, ty: String, plugin: Value) {
> +        self.data.sections.insert(id, (ty, plugin));
> +    }
> +
> +    pub fn get_plugin(
> +        &self,
> +        name: &str,
> +    ) -> Result<Option<Box<dyn AcmePlugin + Send + Sync + 'static>>, Error> {
> +        let (ty, data) = match self.get(name) {
> +            Some(plugin) => plugin,
> +            None => return Ok(None),
> +        };
> +
> +        Ok(Some(match ty.as_str() {
> +            "dns" => {
> +                let plugin: DnsPlugin = serde_json::from_value(data.clone())?;
> +                Box::new(plugin)
> +            }
> +            // "standalone" => todo!("standalone plugin"),
> +            other => bail!("missing implementation for plugin type '{}'", other),
> +        }))
> +    }
> +
> +    pub fn iter(&self) -> impl Iterator<Item = (&String, &(String, Value))> + Send {
> +        self.data.sections.iter()
> +    }
> +}
> +
> +pub trait AcmePlugin {
> +    /// Setup everything required to trigger the validation and return the corresponding validation
> +    /// URL.
> +    fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
> +        &'a self,
> +        client: &'b mut AcmeClient,
> +        authorization: &'c Authorization,
> +        domain: &'d AcmeDomain,
> +    ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>>;
> +
> +    fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
> +        &'a self,
> +        client: &'b mut AcmeClient,
> +        authorization: &'c Authorization,
> +        domain: &'d AcmeDomain,
> +    ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>>;
> +}
> +
> +impl DnsPlugin {
> +    fn extract_challenge(authorization: &Authorization) -> Result<&Challenge, Error> {
> +        authorization
> +            .challenges
> +            .iter()
> +            .find(|ch| ch.ty == "dns-01")
> +            .ok_or_else(|| format_err!("no supported challenge type (dns-01) found"))
> +    }
> +
> +    async fn action<'a>(
> +        &self,
> +        client: &mut AcmeClient,
> +        authorization: &'a Authorization,
> +        domain: &AcmeDomain,
> +        action: &str,
> +    ) -> Result<&'a str, Error> {
> +        let challenge = Self::extract_challenge(authorization)?;
> +        let mut stdin_data = client
> +            .dns_01_txt_value(
> +                challenge
> +                    .token()
> +                    .ok_or_else(|| format_err!("missing token in challenge"))?,
> +            )?
> +            .into_bytes();
> +        stdin_data.push(b'\n');
> +        stdin_data.extend(self.data.as_bytes());
> +        if stdin_data.last() != Some(&b'\n') {
> +            stdin_data.push(b'\n');
> +        }
> +
> +        let mut command = Command::new("/usr/bin/setpriv");
> +
> +        #[rustfmt::skip]
> +        command.args(&[
> +            "--reuid", "nobody",
> +            "--regid", "nogroup",
> +            "--clear-groups",
> +            "--reset-env",
> +            "--",
> +            "/bin/bash",
> +                ACME_PATH,
> +                action,
> +                &self.core.api,
> +                domain.alias.as_deref().unwrap_or(&domain.domain),
> +        ]);
> +
> +        let mut child = command.stdin(Stdio::piped()).spawn()?;
> +
> +        let mut stdin = child.stdin.take().expect("Stdio::piped()");
> +        match async move {
> +            stdin.write_all(&stdin_data).await?;
> +            stdin.flush().await?;
> +            Ok::<_, std::io::Error>(())
> +        }.await {
> +            Ok(()) => (),
> +            Err(err) => {
> +                if let Err(err) = child.kill().await {
> +                    eprintln!("failed to kill '{} {}' command: {}", ACME_PATH, action, err);
> +                }
> +                bail!("'{}' failed: {}", ACME_PATH, err);
> +            }
> +        }
> +
> +        let status = child.wait().await?;
> +        if !status.success() {
> +            bail!(
> +                "'{} {}' exited with error ({})",
> +                ACME_PATH,
> +                action,
> +                status.code().unwrap_or(-1)
> +            );
> +        }
> +
> +        Ok(&challenge.url)
> +    }
> +}
> +
> +impl AcmePlugin for DnsPlugin {
> +    fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
> +        &'a self,
> +        client: &'b mut AcmeClient,
> +        authorization: &'c Authorization,
> +        domain: &'d AcmeDomain,
> +    ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>> {
> +        Box::pin(self.action(client, authorization, domain, "setup"))
> +    }
> +
> +    fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
> +        &'a self,
> +        client: &'b mut AcmeClient,
> +        authorization: &'c Authorization,
> +        domain: &'d AcmeDomain,
> +    ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>> {
> +        Box::pin(async move {
> +            self.action(client, authorization, domain, "teardown")
> +                .await
> +                .map(drop)
> +        })
> +    }
> +}

Re: [pbs-devel] [PATCH v2 backup 14/27] add acme config

[Dietmar Maurer]

What is the purpose of this AccountName wrapper type?

I would prefer to simply use String...

On 4/22/21 4:02 PM, Wolfgang Bumiller wrote:
> +#[api(format: &PROXMOX_SAFE_ID_FORMAT)]
> +/// ACME account name.
> +#[derive(Clone, Eq, PartialEq, Hash, Deserialize, Serialize)]
> +#[serde(transparent)]
> +pub struct AccountName(String);
> +
> +impl AccountName {
> +    pub fn into_string(self) -> String {
> +        self.0
> +    }
> +}
> +
> +impl std::ops::Deref for AccountName {
> +    type Target = str;
> +
> +    #[inline]
> +    fn deref(&self) -> &str {
> +        &self.0
> +    }
> +}
> +
> +impl std::ops::DerefMut for AccountName {
> +    #[inline]
> +    fn deref_mut(&mut self) -> &mut str {
> +        &mut self.0
> +    }
> +}
> +
> +impl AsRef<str> for AccountName {
> +    #[inline]
> +    fn as_ref(&self) -> &str {
> +        self.0.as_ref()
> +    }
> +}
> +
> +impl fmt::Debug for AccountName {
> +    #[inline]
> +    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
> +        fmt::Debug::fmt(&self.0, f)
> +    }
> +}
> +
> +impl fmt::Display for AccountName {
> +    #[inline]
> +    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
> +        fmt::Display::fmt(&self.0, f)
> +    }
> +}
> +

[pbs-devel] [PATCH v2 backup 14/27] add acme config

[Wolfgang Bumiller]

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
 src/config.rs             |   1 +
 src/config/acme/mod.rs    | 198 ++++++++++++++++++++
 src/config/acme/plugin.rs | 380 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 579 insertions(+)
 create mode 100644 src/config/acme/mod.rs
 create mode 100644 src/config/acme/plugin.rs

diff --git a/src/config.rs b/src/config.rs
index 717829e2..94b7fb6c 100644
--- a/src/config.rs
+++ b/src/config.rs
@@ -16,6 +16,7 @@ use proxmox::try_block;
 use crate::buildcfg;
 
 pub mod acl;
+pub mod acme;
 pub mod cached_user_info;
 pub mod datastore;
 pub mod network;
diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
new file mode 100644
index 00000000..ac409c20
--- /dev/null
+++ b/src/config/acme/mod.rs
@@ -0,0 +1,198 @@
+use std::collections::HashMap;
+use std::fmt;
+use std::path::Path;
+
+use anyhow::{bail, format_err, Error};
+use serde::{Deserialize, Serialize};
+
+use proxmox::api::api;
+use proxmox::sys::error::SysError;
+
+use crate::api2::types::{PROXMOX_SAFE_ID_FORMAT, PROXMOX_SAFE_ID_REGEX};
+use crate::tools::ControlFlow;
+
+pub(crate) const ACME_ACCOUNT_DIR: &str = configdir!("/acme/accounts");
+
+pub mod plugin;
+
+#[api(
+    properties: {
+        name: { type: String },
+        url: { type: String },
+    },
+)]
+/// An ACME directory endpoint with a name and URL.
+#[derive(Serialize)]
+pub struct KnownAcmeDirectory {
+    /// The ACME directory's name.
+    pub name: &'static str,
+
+    /// The ACME directory's endpoint URL.
+    pub url: &'static str,
+}
+
+pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
+    KnownAcmeDirectory {
+        name: "Let's Encrypt V2",
+        url: "https://acme-v02.api.letsencrypt.org/directory",
+    },
+    KnownAcmeDirectory {
+        name: "Let's Encrypt V2 Staging",
+        url: "https://acme-staging-v02.api.letsencrypt.org/directory",
+    },
+];
+
+pub const DEFAULT_ACME_DIRECTORY_ENTRY: &KnownAcmeDirectory = &KNOWN_ACME_DIRECTORIES[0];
+
+pub fn account_path(name: &str) -> String {
+    format!("{}/{}", ACME_ACCOUNT_DIR, name)
+}
+
+#[api(format: &PROXMOX_SAFE_ID_FORMAT)]
+/// ACME account name.
+#[derive(Clone, Eq, PartialEq, Hash, Deserialize, Serialize)]
+#[serde(transparent)]
+pub struct AccountName(String);
+
+impl AccountName {
+    pub fn into_string(self) -> String {
+        self.0
+    }
+}
+
+impl std::ops::Deref for AccountName {
+    type Target = str;
+
+    #[inline]
+    fn deref(&self) -> &str {
+        &self.0
+    }
+}
+
+impl std::ops::DerefMut for AccountName {
+    #[inline]
+    fn deref_mut(&mut self) -> &mut str {
+        &mut self.0
+    }
+}
+
+impl AsRef<str> for AccountName {
+    #[inline]
+    fn as_ref(&self) -> &str {
+        self.0.as_ref()
+    }
+}
+
+impl fmt::Debug for AccountName {
+    #[inline]
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        fmt::Debug::fmt(&self.0, f)
+    }
+}
+
+impl fmt::Display for AccountName {
+    #[inline]
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        fmt::Display::fmt(&self.0, f)
+    }
+}
+
+pub fn foreach_acme_account<F>(mut func: F) -> Result<(), Error>
+where
+    F: FnMut(AccountName) -> ControlFlow<Result<(), Error>>,
+{
+    match crate::tools::fs::scan_subdir(-1, ACME_ACCOUNT_DIR, &PROXMOX_SAFE_ID_REGEX) {
+        Ok(files) => {
+            for file in files {
+                let file = file?;
+                let file_name = unsafe { file.file_name_utf8_unchecked() };
+
+                if file_name.starts_with('_') {
+                    continue;
+                }
+
+                let account_name = AccountName(file_name.to_owned());
+
+                if let ControlFlow::Break(result) = func(account_name) {
+                    return result;
+                }
+            }
+            Ok(())
+        }
+        Err(err) if err.not_found() => Ok(()),
+        Err(err) => Err(err.into()),
+    }
+}
+
+/// Run a function for each DNS plugin ID.
+pub fn foreach_dns_plugin<F>(mut func: F) -> Result<(), Error>
+where
+    F: FnMut(&str) -> ControlFlow<Result<(), Error>>,
+{
+    match crate::tools::fs::read_subdir(-1, "/usr/share/proxmox-acme/dnsapi") {
+        Ok(files) => {
+            for file in files.filter_map(Result::ok) {
+                if let Some(id) = file
+                    .file_name()
+                    .to_str()
+                    .ok()
+                    .and_then(|name| name.strip_prefix("dns_"))
+                    .and_then(|name| name.strip_suffix(".sh"))
+                {
+                    if let ControlFlow::Break(result) = func(id) {
+                        return result;
+                    }
+                }
+            }
+
+            Ok(())
+        }
+        Err(err) if err.not_found() => Ok(()),
+        Err(err) => Err(err.into()),
+    }
+}
+
+pub fn mark_account_deactivated(name: &str) -> Result<(), Error> {
+    let from = account_path(name);
+    for i in 0..100 {
+        let to = account_path(&format!("_deactivated_{}_{}", name, i));
+        if !Path::new(&to).exists() {
+            return std::fs::rename(&from, &to).map_err(|err| {
+                format_err!(
+                    "failed to move account path {:?} to {:?} - {}",
+                    from,
+                    to,
+                    err
+                )
+            });
+        }
+    }
+    bail!(
+        "No free slot to rename deactivated account {:?}, please cleanup {:?}",
+        from,
+        ACME_ACCOUNT_DIR
+    );
+}
+
+pub fn complete_acme_account(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
+    let mut out = Vec::new();
+    let _ = foreach_acme_account(|name| {
+        out.push(name.into_string());
+        ControlFlow::CONTINUE
+    });
+    out
+}
+
+pub fn complete_acme_plugin(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
+    match plugin::config() {
+        Ok((config, _digest)) => config
+            .iter()
+            .map(|(id, (_type, _cfg))| id.clone())
+            .collect(),
+        Err(_) => Vec::new(),
+    }
+}
+
+pub fn complete_acme_plugin_type(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
+    vec!["dns".to_string(), "http".to_string()]
+}
diff --git a/src/config/acme/plugin.rs b/src/config/acme/plugin.rs
new file mode 100644
index 00000000..b0c655c0
--- /dev/null
+++ b/src/config/acme/plugin.rs
@@ -0,0 +1,380 @@
+use std::future::Future;
+use std::pin::Pin;
+use std::process::Stdio;
+
+use anyhow::{bail, format_err, Error};
+use lazy_static::lazy_static;
+use serde::{Deserialize, Serialize};
+use serde_json::Value;
+use tokio::io::AsyncWriteExt;
+use tokio::process::Command;
+
+use proxmox::api::{
+    api,
+    schema::*,
+    section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin},
+};
+
+use proxmox::tools::{fs::replace_file, fs::CreateOptions};
+
+use proxmox_acme_rs::{Authorization, Challenge};
+
+use crate::acme::AcmeClient;
+use crate::api2::types::PROXMOX_SAFE_ID_FORMAT;
+use crate::config::node::AcmeDomain;
+
+const ACME_PATH: &str = "/usr/share/proxmox-acme/proxmox-acme";
+
+pub const PLUGIN_ID_SCHEMA: Schema = StringSchema::new("ACME Challenge Plugin ID.")
+    .format(&PROXMOX_SAFE_ID_FORMAT)
+    .schema();
+
+lazy_static! {
+    pub static ref CONFIG: SectionConfig = init();
+}
+
+#[api(
+    properties: {
+        id: { schema: PLUGIN_ID_SCHEMA },
+    },
+)]
+#[derive(Deserialize, Serialize)]
+/// Standalone ACME Plugin for the http-1 challenge.
+pub struct StandalonePlugin {
+    /// Plugin ID.
+    id: String,
+}
+
+impl Default for StandalonePlugin {
+    fn default() -> Self {
+        Self {
+            id: "standalone".to_string(),
+        }
+    }
+}
+
+/// In PVE/PMG we store the plugin's "data" member as base64url encoded string. The UI sends
+/// regular base64 encoded data. We need to "fix" this up.
+
+#[api(
+    properties: {
+        id: { schema: PLUGIN_ID_SCHEMA },
+        disable: {
+            optional: true,
+            default: false,
+        },
+        "validation-delay": {
+            default: 30,
+            optional: true,
+            minimum: 0,
+            maximum: 2 * 24 * 60 * 60,
+        },
+    },
+)]
+/// DNS ACME Challenge Plugin core data.
+#[derive(Deserialize, Serialize, Updater)]
+#[serde(rename_all = "kebab-case")]
+pub struct DnsPluginCore {
+    /// Plugin ID.
+    pub(crate) id: String,
+
+    /// DNS API Plugin Id.
+    api: String,
+
+    /// Extra delay in seconds to wait before requesting validation.
+    ///
+    /// Allows to cope with long TTL of DNS records.
+    #[serde(skip_serializing_if = "Option::is_none", default)]
+    validation_delay: Option<u32>,
+
+    /// Flag to disable the config.
+    #[serde(skip_serializing_if = "Option::is_none", default)]
+    disable: Option<bool>,
+}
+
+#[api(
+    properties: {
+        core: { type: DnsPluginCore },
+    },
+)]
+/// DNS ACME Challenge Plugin.
+#[derive(Deserialize, Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct DnsPlugin {
+    #[serde(flatten)]
+    pub(crate) core: DnsPluginCore,
+
+    // FIXME: The `Updater` should allow:
+    //   * having different descriptions for this and the Updater version
+    //   * having different `#[serde]` attributes for the Updater
+    //   * or, well, leaving fields out completely in teh Updater but this means we may need to
+    //     separate Updater and Builder deriving.
+    // We handle this property separately in the API calls.
+    /// DNS plugin data (base64url encoded without padding).
+    #[serde(with = "proxmox::tools::serde::string_as_base64url_nopad")]
+    pub(crate) data: String,
+}
+
+impl DnsPlugin {
+    pub fn decode_data(&self, output: &mut Vec<u8>) -> Result<(), Error> {
+        Ok(base64::decode_config_buf(&self.data, base64::URL_SAFE_NO_PAD, output)?)
+    }
+}
+
+//impl DnsPluginUpdater {
+//    // The UI passes regular base64 data, we need base64url data. In PVE/PMG this happens magically
+//    // since perl parses both on decode...
+//    pub fn api_fixup(&mut self) -> Result<(), Error> {
+//        if let Some(data) = self.data.as_mut() {
+//            let new = base64::encode_config(&base64::decode(&data)?, base64::URL_SAFE_NO_PAD);
+//            *data = new;
+//        }
+//        Ok(())
+//    }
+//}
+
+fn init() -> SectionConfig {
+    let mut config = SectionConfig::new(&PLUGIN_ID_SCHEMA);
+
+    let standalone_schema = match &StandalonePlugin::API_SCHEMA {
+        Schema::Object(schema) => schema,
+        _ => unreachable!(),
+    };
+    let standalone_plugin = SectionConfigPlugin::new(
+        "standalone".to_string(),
+        Some("id".to_string()),
+        standalone_schema,
+    );
+    config.register_plugin(standalone_plugin);
+
+    let dns_challenge_schema = match DnsPlugin::API_SCHEMA {
+        Schema::AllOf(ref schema) => schema,
+        _ => unreachable!(),
+    };
+    let dns_challenge_plugin = SectionConfigPlugin::new(
+        "dns".to_string(),
+        Some("id".to_string()),
+        dns_challenge_schema,
+    );
+    config.register_plugin(dns_challenge_plugin);
+
+    config
+}
+
+pub const ACME_PLUGIN_CFG_FILENAME: &str = "/etc/proxmox-backup/acme/plugins.cfg";
+pub const ACME_PLUGIN_CFG_LOCKFILE: &str = "/etc/proxmox-backup/acme/.plugins.lck";
+const LOCK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(5);
+
+pub fn read_lock() -> Result<std::fs::File, Error> {
+    proxmox::tools::fs::open_file_locked(ACME_PLUGIN_CFG_LOCKFILE, LOCK_TIMEOUT, false)
+}
+
+pub fn write_lock() -> Result<std::fs::File, Error> {
+    proxmox::tools::fs::open_file_locked(ACME_PLUGIN_CFG_LOCKFILE, LOCK_TIMEOUT, true)
+}
+
+pub fn config() -> Result<(PluginData, [u8; 32]), Error> {
+    let content = proxmox::tools::fs::file_read_optional_string(ACME_PLUGIN_CFG_FILENAME)?
+        .unwrap_or_else(|| "".to_string());
+
+    let digest = openssl::sha::sha256(content.as_bytes());
+    let mut data = CONFIG.parse(ACME_PLUGIN_CFG_FILENAME, &content)?;
+
+    if data.sections.get("standalone").is_none() {
+        let standalone = StandalonePlugin::default();
+        data.set_data("standalone", "standalone", &standalone)
+            .unwrap();
+    }
+
+    Ok((PluginData { data }, digest))
+}
+
+pub fn save_config(config: &PluginData) -> Result<(), Error> {
+    let raw = CONFIG.write(ACME_PLUGIN_CFG_FILENAME, &config.data)?;
+
+    let backup_user = crate::backup::backup_user()?;
+    let mode = nix::sys::stat::Mode::from_bits_truncate(0o0640);
+    // set the correct owner/group/permissions while saving file
+    // owner(rw) = root, group(r)= backup
+    let options = CreateOptions::new()
+        .perm(mode)
+        .owner(nix::unistd::ROOT)
+        .group(backup_user.gid);
+
+    replace_file(ACME_PLUGIN_CFG_FILENAME, raw.as_bytes(), options)?;
+
+    Ok(())
+}
+
+pub struct PluginData {
+    data: SectionConfigData,
+}
+
+impl PluginData {
+    #[inline]
+    pub fn remove(&mut self, name: &str) -> Option<(String, Value)> {
+        self.data.sections.remove(name)
+    }
+
+    #[inline]
+    pub fn contains_key(&mut self, name: &str) -> bool {
+        self.data.sections.contains_key(name)
+    }
+
+    #[inline]
+    pub fn get(&self, name: &str) -> Option<&(String, Value)> {
+        self.data.sections.get(name)
+    }
+
+    #[inline]
+    pub fn get_mut(&mut self, name: &str) -> Option<&mut (String, Value)> {
+        self.data.sections.get_mut(name)
+    }
+
+    // FIXME: Verify the plugin type *exists* and check its config schema...
+    pub fn insert(&mut self, id: String, ty: String, plugin: Value) {
+        self.data.sections.insert(id, (ty, plugin));
+    }
+
+    pub fn get_plugin(
+        &self,
+        name: &str,
+    ) -> Result<Option<Box<dyn AcmePlugin + Send + Sync + 'static>>, Error> {
+        let (ty, data) = match self.get(name) {
+            Some(plugin) => plugin,
+            None => return Ok(None),
+        };
+
+        Ok(Some(match ty.as_str() {
+            "dns" => {
+                let plugin: DnsPlugin = serde_json::from_value(data.clone())?;
+                Box::new(plugin)
+            }
+            // "standalone" => todo!("standalone plugin"),
+            other => bail!("missing implementation for plugin type '{}'", other),
+        }))
+    }
+
+    pub fn iter(&self) -> impl Iterator<Item = (&String, &(String, Value))> + Send {
+        self.data.sections.iter()
+    }
+}
+
+pub trait AcmePlugin {
+    /// Setup everything required to trigger the validation and return the corresponding validation
+    /// URL.
+    fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
+        &'a self,
+        client: &'b mut AcmeClient,
+        authorization: &'c Authorization,
+        domain: &'d AcmeDomain,
+    ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>>;
+
+    fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
+        &'a self,
+        client: &'b mut AcmeClient,
+        authorization: &'c Authorization,
+        domain: &'d AcmeDomain,
+    ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>>;
+}
+
+impl DnsPlugin {
+    fn extract_challenge(authorization: &Authorization) -> Result<&Challenge, Error> {
+        authorization
+            .challenges
+            .iter()
+            .find(|ch| ch.ty == "dns-01")
+            .ok_or_else(|| format_err!("no supported challenge type (dns-01) found"))
+    }
+
+    async fn action<'a>(
+        &self,
+        client: &mut AcmeClient,
+        authorization: &'a Authorization,
+        domain: &AcmeDomain,
+        action: &str,
+    ) -> Result<&'a str, Error> {
+        let challenge = Self::extract_challenge(authorization)?;
+        let mut stdin_data = client
+            .dns_01_txt_value(
+                challenge
+                    .token()
+                    .ok_or_else(|| format_err!("missing token in challenge"))?,
+            )?
+            .into_bytes();
+        stdin_data.push(b'\n');
+        stdin_data.extend(self.data.as_bytes());
+        if stdin_data.last() != Some(&b'\n') {
+            stdin_data.push(b'\n');
+        }
+
+        let mut command = Command::new("/usr/bin/setpriv");
+
+        #[rustfmt::skip]
+        command.args(&[
+            "--reuid", "nobody",
+            "--regid", "nogroup",
+            "--clear-groups",
+            "--reset-env",
+            "--",
+            "/bin/bash",
+                ACME_PATH,
+                action,
+                &self.core.api,
+                domain.alias.as_deref().unwrap_or(&domain.domain),
+        ]);
+
+        let mut child = command.stdin(Stdio::piped()).spawn()?;
+
+        let mut stdin = child.stdin.take().expect("Stdio::piped()");
+        match async move {
+            stdin.write_all(&stdin_data).await?;
+            stdin.flush().await?;
+            Ok::<_, std::io::Error>(())
+        }.await {
+            Ok(()) => (),
+            Err(err) => {
+                if let Err(err) = child.kill().await {
+                    eprintln!("failed to kill '{} {}' command: {}", ACME_PATH, action, err);
+                }
+                bail!("'{}' failed: {}", ACME_PATH, err);
+            }
+        }
+
+        let status = child.wait().await?;
+        if !status.success() {
+            bail!(
+                "'{} {}' exited with error ({})",
+                ACME_PATH,
+                action,
+                status.code().unwrap_or(-1)
+            );
+        }
+
+        Ok(&challenge.url)
+    }
+}
+
+impl AcmePlugin for DnsPlugin {
+    fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
+        &'a self,
+        client: &'b mut AcmeClient,
+        authorization: &'c Authorization,
+        domain: &'d AcmeDomain,
+    ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>> {
+        Box::pin(self.action(client, authorization, domain, "setup"))
+    }
+
+    fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
+        &'a self,
+        client: &'b mut AcmeClient,
+        authorization: &'c Authorization,
+        domain: &'d AcmeDomain,
+    ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>> {
+        Box::pin(async move {
+            self.action(client, authorization, domain, "teardown")
+                .await
+                .map(drop)
+        })
+    }
+}
-- 
2.20.1