Re: [pbs-devel] [PATCH v2 backup 02/27] add dns alias schema

Re: [pbs-devel] [PATCH v2 backup 02/27] add dns alias schema

[Dietmar Maurer]

So we should call it SCOPED_DNS_NAME, and require the underscore at start?

> On 04/28/2021 6:34 PM Thomas Lamprecht <t.lamprecht@proxmox.com> wrote:
> 
>  
> On 28.04.21 18:10, Dietmar Maurer wrote:
> > Seems DNS names in general are totally unrestricted and
> > may contain arbitrary binary data:
> > 
> > https://tools.ietf.org/html/rfc2181#section-11
> > 
> > Only host names and url are restricted.
> > 
> > ?!
> 
> FYI, from a off-list message from Wolfgang:
> 
> >> and why now only allow as first one?
> > mostly because that's what we did in PVE and because using underscores
> > anywhere else is silly ;-)
> > it's used for scoping, there was even an RFC draft but it's expired and only
> > marked as "best current practice" whatever that's worth:
> > https://tools.ietf.org/id/draft-ietf-dnsop-attrleaf-07.html#rfc.section.1.1
> > so basically: we used to not allow underscores, people do use leading
> > underscores for scoping, and we use it particularly for ACME aliases...
> 
> There was some confusion with which ALIAS you meant, as there's a not really
> used DNS record type named "ALIAS" too (which is unrelated to that one here).

Re: [pbs-devel] [PATCH v2 backup 02/27] add dns alias schema

[Wolfgang Bumiller]

On Wed, Apr 28, 2021 at 08:18:21PM +0200, Dietmar Maurer wrote:
> So we should call it SCOPED_DNS_NAME, and require the underscore at start?

No, the scoping is optional.

Re: [pbs-devel] [PATCH v2 backup 02/27] add dns alias schema

[Dietmar Maurer]

Seems DNS names in general are totally unrestricted and
may contain arbitrary binary data:

https://tools.ietf.org/html/rfc2181#section-11

Only host names and url are restricted.

?!

> On 04/28/2021 5:55 PM Dietmar Maurer <dietmar@proxmox.com> wrote:
> 
>  
> Is this the same syntax used for DNS SRV records?
> 
> https://en.wikipedia.org/wiki/SRV_record
> 
> > On 04/28/2021 1:15 PM Dietmar Maurer <dietmar@proxmox.com> wrote:
> > 
> >  
> > > On 04/28/2021 1:07 PM Wolfgang Bumiller <w.bumiller@proxmox.com> wrote:
> > > 
> > >  
> > > On Wed, Apr 28, 2021 at 12:26:11PM +0200, Dietmar Maurer wrote:
> > > > Sorry, I don't get this. Why is DNS_LABEL and DNS_ALIAS_LABEL different?
> > > 
> > > One allows underscores at the beginning, the other doesn't, as for
> > 
> > But where is it defined that an ALIAS may contain underscores?
> > Do you have a link to the corresponding RFC?
> > 
> > 
> > _______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH v2 backup 02/27] add dns alias schema

[Thomas Lamprecht]

On 28.04.21 18:10, Dietmar Maurer wrote:
> Seems DNS names in general are totally unrestricted and
> may contain arbitrary binary data:
> 
> https://tools.ietf.org/html/rfc2181#section-11
> 
> Only host names and url are restricted.
> 
> ?!

FYI, from a off-list message from Wolfgang:

>> and why now only allow as first one?
> mostly because that's what we did in PVE and because using underscores
> anywhere else is silly ;-)
> it's used for scoping, there was even an RFC draft but it's expired and only
> marked as "best current practice" whatever that's worth:
> https://tools.ietf.org/id/draft-ietf-dnsop-attrleaf-07.html#rfc.section.1.1
> so basically: we used to not allow underscores, people do use leading
> underscores for scoping, and we use it particularly for ACME aliases...

There was some confusion with which ALIAS you meant, as there's a not really
used DNS record type named "ALIAS" too (which is unrelated to that one here).

Re: [pbs-devel] [PATCH v2 backup 02/27] add dns alias schema

[Dietmar Maurer]

Is this the same syntax used for DNS SRV records?

https://en.wikipedia.org/wiki/SRV_record

> On 04/28/2021 1:15 PM Dietmar Maurer <dietmar@proxmox.com> wrote:
> 
>  
> > On 04/28/2021 1:07 PM Wolfgang Bumiller <w.bumiller@proxmox.com> wrote:
> > 
> >  
> > On Wed, Apr 28, 2021 at 12:26:11PM +0200, Dietmar Maurer wrote:
> > > Sorry, I don't get this. Why is DNS_LABEL and DNS_ALIAS_LABEL different?
> > 
> > One allows underscores at the beginning, the other doesn't, as for
> 
> But where is it defined that an ALIAS may contain underscores?
> Do you have a link to the corresponding RFC?
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH v2 backup 02/27] add dns alias schema

[Wolfgang Bumiller]

On Wed, Apr 28, 2021 at 05:55:27PM +0200, Dietmar Maurer wrote:
> Is this the same syntax used for DNS SRV records?
> 
> https://en.wikipedia.org/wiki/SRV_record

Disclaimer: My main motivation was, I just followed along with what
we're doing in PVE (and did the same in PMG as well).

TBH I don't know if any ACME implementation worries about that at all.
The main idea is this:

  * you want to get a certificate for foo.bar.com
  * giving pbs direct access to managing the `foo.bar.com` or `bar.com`
    zones is inconvenient or impossible
  * you setup a `CNAME` for `_acme-challenge.foo.bar.com` to point to
    X.Y.Z
  * you configure the domain foo.bar.com and set the alias to X.Y.Z, so
    that our DNS plugins will set the TXT entry for X.Y.Z instead of
    `_acme-challenge.foo.bar.comm`
  * the ACME provider's DNS resolver will decide which values for X, Y
    and Z they're willing to accept while resolving the TXT entry.
      Most likely they can be completely arbitrary. We know that due to
    common practice, they'll most likely allow at least hostnames with
    the addition of leading underscores, but as far as formal
    definitions go, the DNS RFC is the only "real source of
    what-should-be-the-truth", while in practice you'll just have to try
    and see if it works...

And sure, *technically* we could just relax the DNS schema in general,
but then user's may run into issues when they configure something that
should be legal as per the DNS RFC but is not accepted by their browsers
or some other tool. Of course we could still relax it and just keep the
"normal" restrictions purely in the GUI... I don't know.

Re: [pbs-devel] [PATCH v2 backup 02/27] add dns alias schema

[Dietmar Maurer]

> On 04/28/2021 1:07 PM Wolfgang Bumiller <w.bumiller@proxmox.com> wrote:
> 
>  
> On Wed, Apr 28, 2021 at 12:26:11PM +0200, Dietmar Maurer wrote:
> > Sorry, I don't get this. Why is DNS_LABEL and DNS_ALIAS_LABEL different?
> 
> One allows underscores at the beginning, the other doesn't, as for

But where is it defined that an ALIAS may contain underscores?
Do you have a link to the corresponding RFC?

Re: [pbs-devel] [PATCH v2 backup 02/27] add dns alias schema

[Wolfgang Bumiller]

On Wed, Apr 28, 2021 at 01:15:01PM +0200, Dietmar Maurer wrote:
> 
> > On 04/28/2021 1:07 PM Wolfgang Bumiller <w.bumiller@proxmox.com> wrote:
> > 
> >  
> > On Wed, Apr 28, 2021 at 12:26:11PM +0200, Dietmar Maurer wrote:
> > > Sorry, I don't get this. Why is DNS_LABEL and DNS_ALIAS_LABEL different?
> > 
> > One allows underscores at the beginning, the other doesn't, as for
> 
> But where is it defined that an ALIAS may contain underscores?
> Do you have a link to the corresponding RFC?

I suppose it makes more sense to rename it to DNS_ACME_ALIAS...

[pbs-devel] [PATCH v2 backup 00/27] Implements ACME support for PBS

[Wolfgang Bumiller]

Version 2 of this addresses a few raised issues:

NOTE: The widget-toolkit patch from v1 is still required. I just did not
re-send it now.

* The config file format parser does not use serde anymore as we
  a) don't need it as we have a lot of ready-to-go parsing code in the
     proxmox crate that is now being reused.
  b) is harder to read and the benefits are mostly performance related,
     while more useful things such as using structs for property strings
     in the format really would instead need more formal support on the
     schema side...

* Rebased the acme client to use the new `SimpleHttp` client.
  * and ported the changes to add the user agent string to the new api

* Fixes a few issues found by Dominic:
  * create the acme related directories if they do not exist yet
  * pipe dns plugin command output to the task log
  * made the account name optional in the register api call (since
  * pve/pmg do it too)

* Fixed a warning about a missing semicolon in the ui code.

The original patch 4 (tools/http helper) was dropped and is replaced by
patches 15 & 16. I added the main changes outlined above as separate
patches and only merged minor cleanup/style fixups into the existing
patches.

-- Original cover letter:

Reusing the ACME UI elements from the widget toolkit and therefore
providing a compatible API and pretty much the same config file layout.

Contains the async version of the acme client directly in the tree here,
though it may also be an option to move it to proxmox-acme-rs w/ a
feature-gate. (The code is also very similar to the sync version so
there's a possibility that the implementation could be wrapped in a
macro...)

The series starts out with some helpers & refactoring, followed by a
serde-driven config file format read/writer (meant to be (or become)
compatible to what we have in perl via PVE::JSONSchema::parse_config,
but without the json::Value intermediate step), followed by the config,
client & api call implementation.

(Wildcard support like stoiko just added to PMG still needs to be added,
though...)

Wolfgang Bumiller (27):
  systemd: add reload_unit
  add dns alias schema
  tools::fs::scan_subdir: use nix::Error instead of anyhow
  config: factor out certificate writing
  CertInfo: add not_{after,before}_unix
  CertInfo: add is_expired_after_epoch
  tools: add ControlFlow type
  catalog shell: replace LoopState with ControlFlow
  Cargo.toml: depend on proxmox-acme-rs
  bump d/control
  config::acl: make /system/certificates a valid path
  add 'config file format' to tools::config
  add node config
  add acme config
  tools/http: dedup user agent string
  tools/http: add request_with_agent helper
  add async acme client implementation
  add config/acme api path
  add node/{node}/certificates api call
  add node/{node}/config api path
  add acme commands to proxmox-backup-manager
  implement standalone acme validation
  ui: add certificate & acme view
  daily-update: check acme certificates
  acme: create directories as needed
  acme: pipe plugin output to task log
  api: acme: make account name optional in register call

 Cargo.toml                             |   3 +
 debian/control                         |   2 +
 src/acme/client.rs                     | 672 +++++++++++++++++++++++
 src/acme/mod.rs                        |   2 +
 src/api2/config.rs                     |   2 +
 src/api2/config/acme.rs                | 725 +++++++++++++++++++++++++
 src/api2/node.rs                       |   4 +
 src/api2/node/certificates.rs          | 577 ++++++++++++++++++++
 src/api2/node/config.rs                |  81 +++
 src/api2/types/mod.rs                  |  10 +
 src/backup/catalog_shell.rs            |  18 +-
 src/bin/proxmox-backup-manager.rs      |   1 +
 src/bin/proxmox-daily-update.rs        |  30 +-
 src/bin/proxmox_backup_manager/acme.rs | 415 ++++++++++++++
 src/bin/proxmox_backup_manager/mod.rs  |   2 +
 src/config.rs                          |  55 +-
 src/config/acl.rs                      |   2 +-
 src/config/acme/mod.rs                 | 237 ++++++++
 src/config/acme/plugin.rs              | 532 ++++++++++++++++++
 src/config/node.rs                     | 225 ++++++++
 src/lib.rs                             |   2 +
 src/tools.rs                           |  12 +
 src/tools/cert.rs                      |  41 +-
 src/tools/config.rs                    | 171 ++++++
 src/tools/fs.rs                        |   2 +-
 src/tools/http.rs                      |  15 +-
 src/tools/systemd.rs                   |  11 +
 www/Makefile                           |   1 +
 www/NavigationTree.js                  |   6 +
 www/config/CertificateView.js          |  80 +++
 30 files changed, 3897 insertions(+), 39 deletions(-)
 create mode 100644 src/acme/client.rs
 create mode 100644 src/acme/mod.rs
 create mode 100644 src/api2/config/acme.rs
 create mode 100644 src/api2/node/certificates.rs
 create mode 100644 src/api2/node/config.rs
 create mode 100644 src/bin/proxmox_backup_manager/acme.rs
 create mode 100644 src/config/acme/mod.rs
 create mode 100644 src/config/acme/plugin.rs
 create mode 100644 src/config/node.rs
 create mode 100644 src/tools/config.rs
 create mode 100644 www/config/CertificateView.js

-- 
2.20.1

[pbs-devel] [PATCH v2 backup 02/27] add dns alias schema

[Wolfgang Bumiller]

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
---
 src/api2/types/mod.rs | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/src/api2/types/mod.rs b/src/api2/types/mod.rs
index 9d1bd301..eee91dfd 100644
--- a/src/api2/types/mod.rs
+++ b/src/api2/types/mod.rs
@@ -51,6 +51,11 @@ pub const FILENAME_FORMAT: ApiStringFormat = ApiStringFormat::VerifyFn(|name| {
 macro_rules! DNS_LABEL { () => (r"(?:[a-zA-Z0-9](?:[a-zA-Z0-9\-]*[a-zA-Z0-9])?)") }
 macro_rules! DNS_NAME { () => (concat!(r"(?:(?:", DNS_LABEL!() , r"\.)*", DNS_LABEL!(), ")")) }
 
+macro_rules! DNS_ALIAS_LABEL { () => (r"(?:[a-zA-Z0-9_](?:[a-zA-Z0-9\-]*[a-zA-Z0-9])?)") }
+macro_rules! DNS_ALIAS_NAME {
+    () => (concat!(r"(?:(?:", DNS_ALIAS_LABEL!() , r"\.)*", DNS_ALIAS_LABEL!(), ")"))
+}
+
 macro_rules! CIDR_V4_REGEX_STR { () => (concat!(r"(?:", IPV4RE!(), r"/\d{1,2})$")) }
 macro_rules! CIDR_V6_REGEX_STR { () => (concat!(r"(?:", IPV6RE!(), r"/\d{1,3})$")) }
 
@@ -87,6 +92,8 @@ const_regex!{
 
     pub DNS_NAME_REGEX =  concat!(r"^", DNS_NAME!(), r"$");
 
+    pub DNS_ALIAS_REGEX =  concat!(r"^", DNS_ALIAS_NAME!(), r"$");
+
     pub DNS_NAME_OR_IP_REGEX = concat!(r"^(?:", DNS_NAME!(), "|",  IPRE!(), r")$");
 
     pub BACKUP_REPO_URL_REGEX = concat!(r"^^(?:(?:(", USER_ID_REGEX_STR!(), "|", APITOKEN_ID_REGEX_STR!(), ")@)?(", DNS_NAME!(), "|",  IPRE_BRACKET!() ,"):)?(?:([0-9]{1,5}):)?(", PROXMOX_SAFE_ID_REGEX_STR!(), r")$");
@@ -142,6 +149,9 @@ pub const HOSTNAME_FORMAT: ApiStringFormat =
 pub const DNS_NAME_FORMAT: ApiStringFormat =
     ApiStringFormat::Pattern(&DNS_NAME_REGEX);
 
+pub const DNS_ALIAS_FORMAT: ApiStringFormat =
+    ApiStringFormat::Pattern(&DNS_ALIAS_REGEX);
+
 pub const DNS_NAME_OR_IP_FORMAT: ApiStringFormat =
     ApiStringFormat::Pattern(&DNS_NAME_OR_IP_REGEX);
 
-- 
2.20.1

Re: [pbs-devel] [PATCH v2 backup 02/27] add dns alias schema

[Dietmar Maurer]

Sorry, I don't get this. Why is DNS_LABEL and DNS_ALIAS_LABEL different?


On 4/22/21 4:01 PM, Wolfgang Bumiller wrote:
> Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
> ---
>   src/api2/types/mod.rs | 10 ++++++++++
>   1 file changed, 10 insertions(+)
>
> diff --git a/src/api2/types/mod.rs b/src/api2/types/mod.rs
> index 9d1bd301..eee91dfd 100644
> --- a/src/api2/types/mod.rs
> +++ b/src/api2/types/mod.rs
> @@ -51,6 +51,11 @@ pub const FILENAME_FORMAT: ApiStringFormat = ApiStringFormat::VerifyFn(|name| {
>   macro_rules! DNS_LABEL { () => (r"(?:[a-zA-Z0-9](?:[a-zA-Z0-9\-]*[a-zA-Z0-9])?)") }
>   macro_rules! DNS_NAME { () => (concat!(r"(?:(?:", DNS_LABEL!() , r"\.)*", DNS_LABEL!(), ")")) }
>   
> +macro_rules! DNS_ALIAS_LABEL { () => (r"(?:[a-zA-Z0-9_](?:[a-zA-Z0-9\-]*[a-zA-Z0-9])?)") }
> +macro_rules! DNS_ALIAS_NAME {
> +    () => (concat!(r"(?:(?:", DNS_ALIAS_LABEL!() , r"\.)*", DNS_ALIAS_LABEL!(), ")"))
> +}
> +
>   macro_rules! CIDR_V4_REGEX_STR { () => (concat!(r"(?:", IPV4RE!(), r"/\d{1,2})$")) }
>   macro_rules! CIDR_V6_REGEX_STR { () => (concat!(r"(?:", IPV6RE!(), r"/\d{1,3})$")) }
>   
> @@ -87,6 +92,8 @@ const_regex!{
>   
>       pub DNS_NAME_REGEX =  concat!(r"^", DNS_NAME!(), r"$");
>   
> +    pub DNS_ALIAS_REGEX =  concat!(r"^", DNS_ALIAS_NAME!(), r"$");
> +
>       pub DNS_NAME_OR_IP_REGEX = concat!(r"^(?:", DNS_NAME!(), "|",  IPRE!(), r")$");
>   
>       pub BACKUP_REPO_URL_REGEX = concat!(r"^^(?:(?:(", USER_ID_REGEX_STR!(), "|", APITOKEN_ID_REGEX_STR!(), ")@)?(", DNS_NAME!(), "|",  IPRE_BRACKET!() ,"):)?(?:([0-9]{1,5}):)?(", PROXMOX_SAFE_ID_REGEX_STR!(), r")$");
> @@ -142,6 +149,9 @@ pub const HOSTNAME_FORMAT: ApiStringFormat =
>   pub const DNS_NAME_FORMAT: ApiStringFormat =
>       ApiStringFormat::Pattern(&DNS_NAME_REGEX);
>   
> +pub const DNS_ALIAS_FORMAT: ApiStringFormat =
> +    ApiStringFormat::Pattern(&DNS_ALIAS_REGEX);
> +
>   pub const DNS_NAME_OR_IP_FORMAT: ApiStringFormat =
>       ApiStringFormat::Pattern(&DNS_NAME_OR_IP_REGEX);
>   

Re: [pbs-devel] [PATCH v2 backup 02/27] add dns alias schema

[Wolfgang Bumiller]

On Wed, Apr 28, 2021 at 12:26:11PM +0200, Dietmar Maurer wrote:
> Sorry, I don't get this. Why is DNS_LABEL and DNS_ALIAS_LABEL different?

One allows underscores at the beginning, the other doesn't, as for
"regular" domains that's not allowed, and the acme challenge domain uses
`_acme_challenge` as a prefix and it makes sense to allow users to use
this in their aliases as well, as that's just the domain where the
challenge data ultimately ends up at.

> 
> 
> On 4/22/21 4:01 PM, Wolfgang Bumiller wrote:
> > Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
> > ---
> >   src/api2/types/mod.rs | 10 ++++++++++
> >   1 file changed, 10 insertions(+)
> > 
> > diff --git a/src/api2/types/mod.rs b/src/api2/types/mod.rs
> > index 9d1bd301..eee91dfd 100644
> > --- a/src/api2/types/mod.rs
> > +++ b/src/api2/types/mod.rs
> > @@ -51,6 +51,11 @@ pub const FILENAME_FORMAT: ApiStringFormat = ApiStringFormat::VerifyFn(|name| {
> >   macro_rules! DNS_LABEL { () => (r"(?:[a-zA-Z0-9](?:[a-zA-Z0-9\-]*[a-zA-Z0-9])?)") }
> >   macro_rules! DNS_NAME { () => (concat!(r"(?:(?:", DNS_LABEL!() , r"\.)*", DNS_LABEL!(), ")")) }
> > +macro_rules! DNS_ALIAS_LABEL { () => (r"(?:[a-zA-Z0-9_](?:[a-zA-Z0-9\-]*[a-zA-Z0-9])?)") }
> > +macro_rules! DNS_ALIAS_NAME {
> > +    () => (concat!(r"(?:(?:", DNS_ALIAS_LABEL!() , r"\.)*", DNS_ALIAS_LABEL!(), ")"))
> > +}
> > +
> >   macro_rules! CIDR_V4_REGEX_STR { () => (concat!(r"(?:", IPV4RE!(), r"/\d{1,2})$")) }
> >   macro_rules! CIDR_V6_REGEX_STR { () => (concat!(r"(?:", IPV6RE!(), r"/\d{1,3})$")) }
> > @@ -87,6 +92,8 @@ const_regex!{
> >       pub DNS_NAME_REGEX =  concat!(r"^", DNS_NAME!(), r"$");
> > +    pub DNS_ALIAS_REGEX =  concat!(r"^", DNS_ALIAS_NAME!(), r"$");
> > +
> >       pub DNS_NAME_OR_IP_REGEX = concat!(r"^(?:", DNS_NAME!(), "|",  IPRE!(), r")$");
> >       pub BACKUP_REPO_URL_REGEX = concat!(r"^^(?:(?:(", USER_ID_REGEX_STR!(), "|", APITOKEN_ID_REGEX_STR!(), ")@)?(", DNS_NAME!(), "|",  IPRE_BRACKET!() ,"):)?(?:([0-9]{1,5}):)?(", PROXMOX_SAFE_ID_REGEX_STR!(), r")$");
> > @@ -142,6 +149,9 @@ pub const HOSTNAME_FORMAT: ApiStringFormat =
> >   pub const DNS_NAME_FORMAT: ApiStringFormat =
> >       ApiStringFormat::Pattern(&DNS_NAME_REGEX);
> > +pub const DNS_ALIAS_FORMAT: ApiStringFormat =
> > +    ApiStringFormat::Pattern(&DNS_ALIAS_REGEX);
> > +
> >   pub const DNS_NAME_OR_IP_FORMAT: ApiStringFormat =
> >       ApiStringFormat::Pattern(&DNS_NAME_OR_IP_REGEX);