* [pbs-devel] [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests
@ 2025-10-28 15:21 14% Samuel Rufinatscha
2025-10-28 15:22 15% ` [pbs-devel] [PATCH proxmox 1/1] " Samuel Rufinatscha
` (3 more replies)
0 siblings, 4 replies; 200+ results
From: Samuel Rufinatscha @ 2025-10-28 15:21 UTC (permalink / raw)
To: pbs-devel
Hi,
this series proposes a change to ACME account registration in Proxmox
Backup Server (PBS), so that it also works with ACME servers that return
HTTP 204 No Content to the HEAD request for newNonce.
This behaviour was observed against a specific ACME deployment and
reported as bug #6939 [1]. Currently, PBS cannot register an ACME
account for this CA.
## Problem
During ACME account registration, PBS first fetches an anti-replay nonce
by sending a HEAD request to the CA’s newNonce URL. RFC 8555 7.2 [2]
says:
* the server MUST include a Replay-Nonce header with a fresh nonce,
* the server SHOULD use status 200 OK for the HEAD request,
* the server MUST also handle GET on the same resource with status 204 No
Content and an empty body [2].
Currently, our Rust ACME clients only accept 200 OK. PBS inherits that
strictness and aborts with:
*ACME server responded with unexpected status code: 204*
This is also stricter than the PVE Perl ACME client, which tolerates any
2xx success codes [3]. The author mentions, the issue did not appear
with PVE9 [1].
## Ideas to solve the problem
To support ACME providers which return 204 No Content, the underlying
ACME clients need to tolerate both 200 OK and 204 No Content as valid
responses for the nonce HEAD request, as long as the Replay-Nonce is
provided.
I considered following solutions:
1. Change the `expected` field of the `AcmeRequest` type from `u16` to
`Vec<u16>`, to support multiple success codes
2. Keep `expected: u16` and add a second field e.g. `expected_other:
Vec<u16>` for "also allowed" codes.
3. Support any 2xx success codes, and remove the `expected` check
I thought (1) might be reasonable, because:
* It stays explicit and makes it clear which statuses are considered
success.
* We don’t create two parallel concepts ("expected" vs
"expected_other") which introduces additional complexity
* Can be extend later if we meet yet another harmless but not 200
variant.
* We don’t allow arbitrary 2xx.
What do you think? Do you maybe have any other solution in mind that
would fit better?
## Testing
To prove the proposed fix, I reproduced the scenario:
Pebble (release 2.8.0) from Let's Encrypt [5] running on a Debian 9 VM
as the ACME server. nginx in front of Pebble, to intercept the
`newNonce` request in order to return 204 No Content instead of 200 OK,
all other requests are unchanged and forwarded to Pebble. Trust the
Pebble and ngix CAs via `/usr/local/share/ca-certificates` +
`update-ca-certificates` on the PBS VM.
Then I ran following command against nginx:
```
proxmox-backup-manager acme account register proxytest root@backup.local
--directory 'https://nginx-address/dir
Attempting to fetch Terms of
Service from "https://acme-vm/dir"
Terms of Service:
data:text/plain,Do%20what%20thou%20wilt
Do you agree to the above terms?
[y|N]: y
Do you want to use external account binding? [y|N]: N
Attempting
to register account with "https://acme-vm/dir"...
Registration
successful, account URL: https://acme-vm/my-account/160e58b66bdd72da
```
When adjusting the nginx configuration to return any other non-expected
success status code, e.g. 205, PBS expectely rejects with `API
misbehaved: ACME server responded with unexpected status code: 205`.
## Maintainer notes:
The patch series involves the following components:
proxmox-acme: Apply PATCH 1 to change `expected` from `u16` to
`Vec<u16>`. This results in a breaking change, as it changes the public
API of the `AcmeRequest` type that is used by other components.
proxmox-acme-api: Needs to depend on the new proxmox-acme; patch bump
proxmox-backup: Apply PATCH 2 to use the new API changes; no breaking
change as of only internal changes; patch bump
proxmox-perl-rs / proxmox-datacenter-manager: Will need to use the
dependency version bumps to follow the new proxmox-acme.
## Patch summary
[PATCH 1/2] fix #6939: support providers returning 204 for nonce
requests
* Make the expected-status logic accept multiple allowed codes.
* Treat both 200 OK and 204 No Content as valid for HEAD /newNonce,
provided Replay-Nonce is present.
* Keep rejecting other codes.
[PATCH 2/2] acme: accept HTTP 204 from newNonce endpoint
* Use the updated proxmox-acme behavior in PBS.
* PBS can now register an ACME account against servers that return 204
for the nonce HEAD request.
* Still rejects unexpected codes.
Thanks for considering this patch series, I look forward to your
feedback.
Best,
Samuel Rufinatscha
[1] Bugzilla report #6939:
[https://bugzilla.proxmox.com/show_bug.cgi?id=6939](https://bugzilla.proxmox.com/show_bug.cgi?id=6939)
[2] RFC 8555 (ACME):
[https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2](https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2)
[3] PVE’s Perl ACME client:
[https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597)
[4] Pebble ACME server:
[https://github.com/letsencrypt/pebble](https://github.com/letsencrypt/pebble)
proxmox:
Samuel Rufinatscha (1):
fix #6939: acme: support servers returning 204 for nonce requests
proxmox-acme/src/account.rs | 10 +++++-----
proxmox-acme/src/async_client.rs | 6 +++---
proxmox-acme/src/client.rs | 2 +-
proxmox-acme/src/request.rs | 4 ++--
4 files changed, 11 insertions(+), 11 deletions(-)
proxmox-backup:
Samuel Rufinatscha (1):
fix #6939: acme: accept HTTP 204 from newNonce endpoint
src/acme/client.rs | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
Summary over all repositories:
5 files changed, 14 insertions(+), 14 deletions(-)
--
Generated by git-murpp 0.8.1
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 14%]
* [pbs-devel] [PATCH proxmox 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-28 15:21 14% [pbs-devel] [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
@ 2025-10-28 15:22 15% ` Samuel Rufinatscha
2025-10-29 7:23 5% ` Christian Ebner
2025-10-29 10:38 5% ` Wolfgang Bumiller
2025-10-28 15:22 16% ` [pbs-devel] [PATCH proxmox-backup 1/1] fix #6939: acme: accept HTTP 204 from newNonce endpoint Samuel Rufinatscha
` (2 subsequent siblings)
3 siblings, 2 replies; 200+ results
From: Samuel Rufinatscha @ 2025-10-28 15:22 UTC (permalink / raw)
To: pbs-devel
Some ACME servers (notably custom or legacy implementations) respond
to HEAD /newNonce with a 204 No Content instead of the
RFC 8555-recommended 200 OK [1]. While this behavior is technically
off-spec, it is functionally harmless. This issue was reported on our
bug tracker [2].
The previous implementation treated any non-200 response as an error,
causing account registration to fail against such servers. Relax the
status-code check to accept both 200 and 204 responses (and potentially
support other 2xx codes) to improve interoperability.
This aligns behavior with PVE’s more tolerant Perl ACME client and
avoids regressions.
[1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
Fixes: #6939
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme/src/account.rs | 10 +++++-----
proxmox-acme/src/async_client.rs | 6 +++---
proxmox-acme/src/client.rs | 2 +-
proxmox-acme/src/request.rs | 4 ++--
4 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
index 73d786b8..60719865 100644
--- a/proxmox-acme/src/account.rs
+++ b/proxmox-acme/src/account.rs
@@ -85,7 +85,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::request::CREATED,
+ expected: vec![crate::request::CREATED],
};
Ok(NewOrder::new(request))
@@ -107,7 +107,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: vec![200],
})
}
@@ -132,7 +132,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: vec![200],
})
}
@@ -157,7 +157,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: vec![200],
})
}
@@ -405,7 +405,7 @@ impl AccountCreator {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::request::CREATED,
+ expected: vec![crate::request::CREATED],
})
}
diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
index 60e1f359..0901aa8d 100644
--- a/proxmox-acme/src/async_client.rs
+++ b/proxmox-acme/src/async_client.rs
@@ -421,7 +421,7 @@ impl AcmeClient {
};
if parts.status.is_success() {
- if status != request.expected {
+ if !request.expected.contains(&status) {
return Err(Error::InvalidApi(format!(
"ACME server responded with unexpected status code: {:?}",
parts.status
@@ -501,7 +501,7 @@ impl AcmeClient {
method: "GET",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: vec![200],
},
nonce,
)
@@ -553,7 +553,7 @@ impl AcmeClient {
method: "HEAD",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: vec![200, 204],
},
nonce,
)
diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
index d8a62081..ea8a8655 100644
--- a/proxmox-acme/src/client.rs
+++ b/proxmox-acme/src/client.rs
@@ -203,7 +203,7 @@ impl Inner {
let got_nonce = self.update_nonce(&mut response)?;
if response.is_success() {
- if response.status != request.expected {
+ if !request.expected.contains(&response.status) {
return Err(Error::InvalidApi(format!(
"API server responded with unexpected status code: {:?}",
response.status
diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
index 78a90913..38e825d6 100644
--- a/proxmox-acme/src/request.rs
+++ b/proxmox-acme/src/request.rs
@@ -17,8 +17,8 @@ pub struct Request {
/// The body to pass along with request, or an empty string.
pub body: String,
- /// The expected status code a compliant ACME provider will return on success.
- pub expected: u16,
+ /// The set of HTTP status codes that indicate a successful response from an ACME provider.
+ pub expected: Vec<u16>,
}
/// An ACME error response contains a specially formatted type string, and can optionally
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox-backup 1/1] fix #6939: acme: accept HTTP 204 from newNonce endpoint
2025-10-28 15:21 14% [pbs-devel] [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2025-10-28 15:22 15% ` [pbs-devel] [PATCH proxmox 1/1] " Samuel Rufinatscha
@ 2025-10-28 15:22 16% ` Samuel Rufinatscha
2025-10-29 7:51 5% ` [pbs-devel] [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests Thomas Lamprecht
2025-10-29 16:49 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
3 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-10-28 15:22 UTC (permalink / raw)
To: pbs-devel
When registering an ACME account, PBS fetches a fresh nonce by issuing a
HEAD request to the server's newNonce URL. Until now we assumed this
request would return HTTP 200 OK.
In practice, some ACME servers respond with HTTP 204 No Content for this
HEAD request while still providing a valid Replay-Nonce header. This
causes PBS to abort registration with "ACME server responded with
unexpected status code: 204", even though the server would otherwise
issue certificates correctly.
Adjust the ACME client code in PBS to accept both 200 OK and 204 No
Content as successful results for the newNonce step. We continue to
reject other status codes so we don't silently accept arbitrary 2xx
responses.
This restores interoperability with ACME servers that send 204 for
newNonce, and aligns PBS' behavior with the updated proxmox-acme library
as well as PVE's more tolerant ACME client.
Fixes: #6939
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/acme/client.rs | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/src/acme/client.rs b/src/acme/client.rs
index 1c12a4b9..0dabf676 100644
--- a/src/acme/client.rs
+++ b/src/acme/client.rs
@@ -530,7 +530,7 @@ impl AcmeClient {
};
if parts.status.is_success() {
- if status != request.expected {
+ if !request.expected.contains(&status) {
return Err(Error::InvalidApi(format!(
"ACME server responded with unexpected status code: {:?}",
parts.status
@@ -609,7 +609,7 @@ impl AcmeClient {
method: "GET",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: vec![200],
},
nonce,
)
@@ -657,7 +657,7 @@ impl AcmeClient {
method: "HEAD",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: vec![200, 204],
},
nonce,
)
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 16%]
* Re: [pbs-devel] [PATCH proxmox 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-28 15:22 15% ` [pbs-devel] [PATCH proxmox 1/1] " Samuel Rufinatscha
@ 2025-10-29 7:23 5% ` Christian Ebner
2025-10-29 7:53 0% ` Thomas Lamprecht
2025-10-29 10:38 5% ` Wolfgang Bumiller
1 sibling, 1 reply; 200+ results
From: Christian Ebner @ 2025-10-29 7:23 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Samuel Rufinatscha
Cc: Wolfgang Bumiller
Hi, thanks for the patches!
comments inline
On 10/28/25 8:34 PM, Samuel Rufinatscha wrote:
> Some ACME servers (notably custom or legacy implementations) respond
> to HEAD /newNonce with a 204 No Content instead of the
> RFC 8555-recommended 200 OK [1]. While this behavior is technically
> off-spec, it is functionally harmless. This issue was reported on our
> bug tracker [2].
>
> The previous implementation treated any non-200 response as an error,
> causing account registration to fail against such servers. Relax the
> status-code check to accept both 200 and 204 responses (and potentially
> support other 2xx codes) to improve interoperability.
>
> This aligns behavior with PVE’s more tolerant Perl ACME client and
> avoids regressions.
>
> [1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
> [2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
>
> Fixes: #6939
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> proxmox-acme/src/account.rs | 10 +++++-----
> proxmox-acme/src/async_client.rs | 6 +++---
> proxmox-acme/src/client.rs | 2 +-
> proxmox-acme/src/request.rs | 4 ++--
> 4 files changed, 11 insertions(+), 11 deletions(-)
>
> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
> index 73d786b8..60719865 100644
> --- a/proxmox-acme/src/account.rs
> +++ b/proxmox-acme/src/account.rs
> @@ -85,7 +85,7 @@ impl Account {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: crate::request::CREATED,
> + expected: vec![crate::request::CREATED],
while this is defined as dedicated constant...
> };
>
> Ok(NewOrder::new(request))
> @@ -107,7 +107,7 @@ impl Account {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: 200,
> + expected: vec![200],
... these and the others below are not. Same for the 204 status code you
are about to add.
So in preparation for adding the new status code, these should probably
be defined as, either:
- as dedicated status code constants as well, or
- all moved over to directly use
https://docs.rs/http/1.3.1/http/status/struct.StatusCode.html
I feel like the latter is not done here intentionally to avoid the
dependency on hyper or http (re-exported by hyper) for the api types only.
@wolfgang, comments on that?
> })
> }
>
> @@ -132,7 +132,7 @@ impl Account {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: 200,
> + expected: vec![200],
> })
> }
>
> @@ -157,7 +157,7 @@ impl Account {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: 200,
> + expected: vec![200],
> })
> }
>
> @@ -405,7 +405,7 @@ impl AccountCreator {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: crate::request::CREATED,
> + expected: vec![crate::request::CREATED],
> })
> }
>
> diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
> index 60e1f359..0901aa8d 100644
> --- a/proxmox-acme/src/async_client.rs
> +++ b/proxmox-acme/src/async_client.rs
> @@ -421,7 +421,7 @@ impl AcmeClient {
> };
>
> if parts.status.is_success() {
> - if status != request.expected {
> + if !request.expected.contains(&status) {
> return Err(Error::InvalidApi(format!(
> "ACME server responded with unexpected status code: {:?}",
> parts.status
> @@ -501,7 +501,7 @@ impl AcmeClient {
> method: "GET",
> content_type: "",
> body: String::new(),
> - expected: 200,
> + expected: vec![200],
> },
> nonce,
> )
> @@ -553,7 +553,7 @@ impl AcmeClient {
> method: "HEAD",
> content_type: "",
> body: String::new(),
> - expected: 200,
> + expected: vec![200, 204],
> },
> nonce,
> )
> diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
> index d8a62081..ea8a8655 100644
> --- a/proxmox-acme/src/client.rs
> +++ b/proxmox-acme/src/client.rs
> @@ -203,7 +203,7 @@ impl Inner {
> let got_nonce = self.update_nonce(&mut response)?;
>
> if response.is_success() {
> - if response.status != request.expected {
> + if !request.expected.contains(&response.status) {
> return Err(Error::InvalidApi(format!(
> "API server responded with unexpected status code: {:?}",
> response.status
> diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
> index 78a90913..38e825d6 100644
> --- a/proxmox-acme/src/request.rs
> +++ b/proxmox-acme/src/request.rs
> @@ -17,8 +17,8 @@ pub struct Request {
> /// The body to pass along with request, or an empty string.
> pub body: String,
>
> - /// The expected status code a compliant ACME provider will return on success.
> - pub expected: u16,
> + /// The set of HTTP status codes that indicate a successful response from an ACME provider.
> + pub expected: Vec<u16>,
> }
>
> /// An ACME error response contains a specially formatted type string, and can optionally
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-28 15:21 14% [pbs-devel] [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2025-10-28 15:22 15% ` [pbs-devel] [PATCH proxmox 1/1] " Samuel Rufinatscha
2025-10-28 15:22 16% ` [pbs-devel] [PATCH proxmox-backup 1/1] fix #6939: acme: accept HTTP 204 from newNonce endpoint Samuel Rufinatscha
@ 2025-10-29 7:51 5% ` Thomas Lamprecht
2025-10-29 16:02 6% ` Samuel Rufinatscha
2025-10-29 16:49 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
3 siblings, 1 reply; 200+ results
From: Thomas Lamprecht @ 2025-10-29 7:51 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Samuel Rufinatscha
Thanks for the patch and actually also checking if PVE is affected, and why
it isn't.
Am 28.10.25 um 20:34 schrieb Samuel Rufinatscha:
> This is also stricter than the PVE Perl ACME client, which tolerates any
> 2xx success codes [3]. The author mentions, the issue did not appear
> with PVE9 [1].
>
> ## Ideas to solve the problem
>
> To support ACME providers which return 204 No Content, the underlying
> ACME clients need to tolerate both 200 OK and 204 No Content as valid
> responses for the nonce HEAD request, as long as the Replay-Nonce is
> provided.
>
> I considered following solutions:
>
> 1. Change the `expected` field of the `AcmeRequest` type from `u16` to
> `Vec<u16>`, to support multiple success codes
>
> 2. Keep `expected: u16` and add a second field e.g. `expected_other:
> Vec<u16>` for "also allowed" codes.
>
> 3. Support any 2xx success codes, and remove the `expected` check
>
> I thought (1) might be reasonable, because:
>
> * It stays explicit and makes it clear which statuses are considered
> success.
> * We don’t create two parallel concepts ("expected" vs
> "expected_other") which introduces additional complexity
> * Can be extend later if we meet yet another harmless but not 200
> variant.
> * We don’t allow arbitrary 2xx.
>
> What do you think? Do you maybe have any other solution in mind that
> would fit better?
There probably isn't answer that strictly right in all cases, but in
general it's good to have similar behavior across implementations,
especially given that we do not know of any report where the PVE behavior
of accepting all 2xx response codes caused any problems, from that 3.
would be best, or does the RFC forbid the server to accept other status
codes?
That said, in practice only 201 (Created) might make sense for ACME in
addition to the referenced 200 (OK) and 204 (No Content), and following
the RFC is fine, so 1. is IMO also a good solution here.
Please note the difference to PVE in the commit message, there you write
that behavior is now aligned with PVE, but it's rather "closer aligned"
to, not fully.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-29 7:23 5% ` Christian Ebner
@ 2025-10-29 7:53 0% ` Thomas Lamprecht
2025-10-29 8:07 0% ` Christian Ebner
2025-10-29 10:36 0% ` Wolfgang Bumiller
0 siblings, 2 replies; 200+ results
From: Thomas Lamprecht @ 2025-10-29 7:53 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Christian Ebner,
Samuel Rufinatscha
Cc: Wolfgang Bumiller
Am 29.10.25 um 08:23 schrieb Christian Ebner:
> Hi, thanks for the patches!
>
> comments inline
>
> On 10/28/25 8:34 PM, Samuel Rufinatscha wrote:
>> Some ACME servers (notably custom or legacy implementations) respond
>> to HEAD /newNonce with a 204 No Content instead of the
>> RFC 8555-recommended 200 OK [1]. While this behavior is technically
>> off-spec, it is functionally harmless. This issue was reported on our
>> bug tracker [2].
>>
>> The previous implementation treated any non-200 response as an error,
>> causing account registration to fail against such servers. Relax the
>> status-code check to accept both 200 and 204 responses (and potentially
>> support other 2xx codes) to improve interoperability.
>>
>> This aligns behavior with PVE’s more tolerant Perl ACME client and
>> avoids regressions.
>>
>> [1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
>> [2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
>>
>> Fixes: #6939
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> proxmox-acme/src/account.rs | 10 +++++-----
>> proxmox-acme/src/async_client.rs | 6 +++---
>> proxmox-acme/src/client.rs | 2 +-
>> proxmox-acme/src/request.rs | 4 ++--
>> 4 files changed, 11 insertions(+), 11 deletions(-)
>>
>> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
>> index 73d786b8..60719865 100644
>> --- a/proxmox-acme/src/account.rs
>> +++ b/proxmox-acme/src/account.rs
>> @@ -85,7 +85,7 @@ impl Account {
>> method: "POST",
>> content_type: crate::request::JSON_CONTENT_TYPE,
>> body,
>> - expected: crate::request::CREATED,
>> + expected: vec![crate::request::CREATED],
>
> while this is defined as dedicated constant...
>
>> };
>> Ok(NewOrder::new(request))
>> @@ -107,7 +107,7 @@ impl Account {
>> method: "POST",
>> content_type: crate::request::JSON_CONTENT_TYPE,
>> body,
>> - expected: 200,
>> + expected: vec![200],
>
> ... these and the others below are not. Same for the 204 status code you are about to add.
>
> So in preparation for adding the new status code, these should probably be defined as, either:
> - as dedicated status code constants as well, or
> - all moved over to directly use https://docs.rs/http/1.3.1/http/status/struct.StatusCode.html
>
> I feel like the latter is not done here intentionally to avoid the dependency on hyper or http (re-exported by hyper) for the api types only.
While you are right that constants are generally nicer, IMO HTTP codes are
very stable and universal to be fine to be used directly as numbers in the few
limited instances here.
If we already (even just transitively) would get them from a dependency we still
should switch to that, but I'd not introduce a new dependency just for that; IMO
to high of a cost.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 0%]
* Re: [pbs-devel] [PATCH proxmox 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-29 7:53 0% ` Thomas Lamprecht
@ 2025-10-29 8:07 0% ` Christian Ebner
2025-10-29 10:36 0% ` Wolfgang Bumiller
1 sibling, 0 replies; 200+ results
From: Christian Ebner @ 2025-10-29 8:07 UTC (permalink / raw)
To: Thomas Lamprecht, Proxmox Backup Server development discussion,
Samuel Rufinatscha
Cc: Wolfgang Bumiller
On 10/29/25 8:53 AM, Thomas Lamprecht wrote:
> Am 29.10.25 um 08:23 schrieb Christian Ebner:
>> Hi, thanks for the patches!
>>
>> comments inline
>>
>> On 10/28/25 8:34 PM, Samuel Rufinatscha wrote:
>>> Some ACME servers (notably custom or legacy implementations) respond
>>> to HEAD /newNonce with a 204 No Content instead of the
>>> RFC 8555-recommended 200 OK [1]. While this behavior is technically
>>> off-spec, it is functionally harmless. This issue was reported on our
>>> bug tracker [2].
>>>
>>> The previous implementation treated any non-200 response as an error,
>>> causing account registration to fail against such servers. Relax the
>>> status-code check to accept both 200 and 204 responses (and potentially
>>> support other 2xx codes) to improve interoperability.
>>>
>>> This aligns behavior with PVE’s more tolerant Perl ACME client and
>>> avoids regressions.
>>>
>>> [1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
>>> [2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
>>>
>>> Fixes: #6939
>>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>>> ---
>>> proxmox-acme/src/account.rs | 10 +++++-----
>>> proxmox-acme/src/async_client.rs | 6 +++---
>>> proxmox-acme/src/client.rs | 2 +-
>>> proxmox-acme/src/request.rs | 4 ++--
>>> 4 files changed, 11 insertions(+), 11 deletions(-)
>>>
>>> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
>>> index 73d786b8..60719865 100644
>>> --- a/proxmox-acme/src/account.rs
>>> +++ b/proxmox-acme/src/account.rs
>>> @@ -85,7 +85,7 @@ impl Account {
>>> method: "POST",
>>> content_type: crate::request::JSON_CONTENT_TYPE,
>>> body,
>>> - expected: crate::request::CREATED,
>>> + expected: vec![crate::request::CREATED],
>>
>> while this is defined as dedicated constant...
>>
>>> };
>>> Ok(NewOrder::new(request))
>>> @@ -107,7 +107,7 @@ impl Account {
>>> method: "POST",
>>> content_type: crate::request::JSON_CONTENT_TYPE,
>>> body,
>>> - expected: 200,
>>> + expected: vec![200],
>>
>> ... these and the others below are not. Same for the 204 status code you are about to add.
>>
>> So in preparation for adding the new status code, these should probably be defined as, either:
>> - as dedicated status code constants as well, or
>> - all moved over to directly use https://docs.rs/http/1.3.1/http/status/struct.StatusCode.html
>>
>> I feel like the latter is not done here intentionally to avoid the dependency on hyper or http (re-exported by hyper) for the api types only.
>
> While you are right that constants are generally nicer, IMO HTTP codes are
> very stable and universal to be fine to be used directly as numbers in the few
> limited instances here.
>
> If we already (even just transitively) would get them from a dependency we still
> should switch to that, but I'd not introduce a new dependency just for that; IMO
> to high of a cost.
Agreed, given that the `create::request::CREATED` constant should be
inlined and dropped then for consistency.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 0%]
* Re: [pbs-devel] [PATCH proxmox 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-29 7:53 0% ` Thomas Lamprecht
2025-10-29 8:07 0% ` Christian Ebner
@ 2025-10-29 10:36 0% ` Wolfgang Bumiller
2025-10-29 15:50 6% ` Samuel Rufinatscha
1 sibling, 1 reply; 200+ results
From: Wolfgang Bumiller @ 2025-10-29 10:36 UTC (permalink / raw)
To: Thomas Lamprecht; +Cc: Proxmox Backup Server development discussion
On Wed, Oct 29, 2025 at 08:53:34AM +0100, Thomas Lamprecht wrote:
> Am 29.10.25 um 08:23 schrieb Christian Ebner:
> > Hi, thanks for the patches!
> >
> > comments inline
> >
> > On 10/28/25 8:34 PM, Samuel Rufinatscha wrote:
> >> Some ACME servers (notably custom or legacy implementations) respond
> >> to HEAD /newNonce with a 204 No Content instead of the
> >> RFC 8555-recommended 200 OK [1]. While this behavior is technically
> >> off-spec, it is functionally harmless. This issue was reported on our
> >> bug tracker [2].
> >>
> >> The previous implementation treated any non-200 response as an error,
> >> causing account registration to fail against such servers. Relax the
> >> status-code check to accept both 200 and 204 responses (and potentially
> >> support other 2xx codes) to improve interoperability.
> >>
> >> This aligns behavior with PVE’s more tolerant Perl ACME client and
> >> avoids regressions.
> >>
> >> [1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
> >> [2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
> >>
> >> Fixes: #6939
> >> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> >> ---
> >> proxmox-acme/src/account.rs | 10 +++++-----
> >> proxmox-acme/src/async_client.rs | 6 +++---
> >> proxmox-acme/src/client.rs | 2 +-
> >> proxmox-acme/src/request.rs | 4 ++--
> >> 4 files changed, 11 insertions(+), 11 deletions(-)
> >>
> >> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
> >> index 73d786b8..60719865 100644
> >> --- a/proxmox-acme/src/account.rs
> >> +++ b/proxmox-acme/src/account.rs
> >> @@ -85,7 +85,7 @@ impl Account {
> >> method: "POST",
> >> content_type: crate::request::JSON_CONTENT_TYPE,
> >> body,
> >> - expected: crate::request::CREATED,
> >> + expected: vec![crate::request::CREATED],
> >
> > while this is defined as dedicated constant...
> >
> >> };
> >> Ok(NewOrder::new(request))
> >> @@ -107,7 +107,7 @@ impl Account {
> >> method: "POST",
> >> content_type: crate::request::JSON_CONTENT_TYPE,
> >> body,
> >> - expected: 200,
> >> + expected: vec![200],
> >
> > ... these and the others below are not. Same for the 204 status code you are about to add.
> >
> > So in preparation for adding the new status code, these should probably be defined as, either:
> > - as dedicated status code constants as well, or
> > - all moved over to directly use https://docs.rs/http/1.3.1/http/status/struct.StatusCode.html
> >
> > I feel like the latter is not done here intentionally to avoid the dependency on hyper or http (re-exported by hyper) for the api types only.
>
> While you are right that constants are generally nicer, IMO HTTP codes are
> very stable and universal to be fine to be used directly as numbers in the few
> limited instances here.
Mostly this, but we can also just add internal constants as well. 200
just seemed common enough...
>
> If we already (even just transitively) would get them from a dependency we still
> should switch to that, but I'd not introduce a new dependency just for that; IMO
> to high of a cost.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 0%]
* Re: [pbs-devel] [PATCH proxmox 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-28 15:22 15% ` [pbs-devel] [PATCH proxmox 1/1] " Samuel Rufinatscha
2025-10-29 7:23 5% ` Christian Ebner
@ 2025-10-29 10:38 5% ` Wolfgang Bumiller
2025-10-29 15:56 6% ` Samuel Rufinatscha
1 sibling, 1 reply; 200+ results
From: Wolfgang Bumiller @ 2025-10-29 10:38 UTC (permalink / raw)
To: Samuel Rufinatscha; +Cc: pbs-devel
On Tue, Oct 28, 2025 at 04:22:00PM +0100, Samuel Rufinatscha wrote:
> Some ACME servers (notably custom or legacy implementations) respond
> to HEAD /newNonce with a 204 No Content instead of the
> RFC 8555-recommended 200 OK [1]. While this behavior is technically
> off-spec, it is functionally harmless. This issue was reported on our
> bug tracker [2].
>
> The previous implementation treated any non-200 response as an error,
> causing account registration to fail against such servers. Relax the
> status-code check to accept both 200 and 204 responses (and potentially
> support other 2xx codes) to improve interoperability.
>
> This aligns behavior with PVE’s more tolerant Perl ACME client and
> avoids regressions.
>
> [1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
> [2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
>
> Fixes: #6939
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> proxmox-acme/src/account.rs | 10 +++++-----
> proxmox-acme/src/async_client.rs | 6 +++---
> proxmox-acme/src/client.rs | 2 +-
> proxmox-acme/src/request.rs | 4 ++--
> 4 files changed, 11 insertions(+), 11 deletions(-)
>
> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
> index 73d786b8..60719865 100644
> --- a/proxmox-acme/src/account.rs
> +++ b/proxmox-acme/src/account.rs
> @@ -85,7 +85,7 @@ impl Account {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: crate::request::CREATED,
> + expected: vec![crate::request::CREATED],
> };
>
> Ok(NewOrder::new(request))
> @@ -107,7 +107,7 @@ impl Account {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: 200,
> + expected: vec![200],
> })
> }
>
> @@ -132,7 +132,7 @@ impl Account {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: 200,
> + expected: vec![200],
> })
> }
>
> @@ -157,7 +157,7 @@ impl Account {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: 200,
> + expected: vec![200],
> })
> }
>
> @@ -405,7 +405,7 @@ impl AccountCreator {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: crate::request::CREATED,
> + expected: vec![crate::request::CREATED],
> })
> }
>
> diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
> index 60e1f359..0901aa8d 100644
> --- a/proxmox-acme/src/async_client.rs
> +++ b/proxmox-acme/src/async_client.rs
> @@ -421,7 +421,7 @@ impl AcmeClient {
> };
>
> if parts.status.is_success() {
> - if status != request.expected {
> + if !request.expected.contains(&status) {
> return Err(Error::InvalidApi(format!(
> "ACME server responded with unexpected status code: {:?}",
> parts.status
> @@ -501,7 +501,7 @@ impl AcmeClient {
> method: "GET",
> content_type: "",
> body: String::new(),
> - expected: 200,
> + expected: vec![200],
> },
> nonce,
> )
> @@ -553,7 +553,7 @@ impl AcmeClient {
> method: "HEAD",
> content_type: "",
> body: String::new(),
> - expected: 200,
> + expected: vec![200, 204],
> },
> nonce,
> )
> diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
> index d8a62081..ea8a8655 100644
> --- a/proxmox-acme/src/client.rs
> +++ b/proxmox-acme/src/client.rs
> @@ -203,7 +203,7 @@ impl Inner {
> let got_nonce = self.update_nonce(&mut response)?;
>
> if response.is_success() {
> - if response.status != request.expected {
> + if !request.expected.contains(&response.status) {
> return Err(Error::InvalidApi(format!(
> "API server responded with unexpected status code: {:?}",
> response.status
> diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
> index 78a90913..38e825d6 100644
> --- a/proxmox-acme/src/request.rs
> +++ b/proxmox-acme/src/request.rs
> @@ -17,8 +17,8 @@ pub struct Request {
> /// The body to pass along with request, or an empty string.
> pub body: String,
>
> - /// The expected status code a compliant ACME provider will return on success.
> - pub expected: u16,
> + /// The set of HTTP status codes that indicate a successful response from an ACME provider.
> + pub expected: Vec<u16>,
We always have a static set, so I'd rather use `&'static [u16]` here.
There's no need to allocate usually-single-element vectors everywhere.
> }
>
> /// An ACME error response contains a specially formatted type string, and can optionally
> --
> 2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-29 10:36 0% ` Wolfgang Bumiller
@ 2025-10-29 15:50 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-10-29 15:50 UTC (permalink / raw)
To: Wolfgang Bumiller, Thomas Lamprecht
Cc: Proxmox Backup Server development discussion
On 10/29/25 11:35 AM, Wolfgang Bumiller wrote:
> On Wed, Oct 29, 2025 at 08:53:34AM +0100, Thomas Lamprecht wrote:
>> Am 29.10.25 um 08:23 schrieb Christian Ebner:
>>> Hi, thanks for the patches!
>>>
>>> comments inline
>>>
>>> On 10/28/25 8:34 PM, Samuel Rufinatscha wrote:
>>>> Some ACME servers (notably custom or legacy implementations) respond
>>>> to HEAD /newNonce with a 204 No Content instead of the
>>>> RFC 8555-recommended 200 OK [1]. While this behavior is technically
>>>> off-spec, it is functionally harmless. This issue was reported on our
>>>> bug tracker [2].
>>>>
>>>> The previous implementation treated any non-200 response as an error,
>>>> causing account registration to fail against such servers. Relax the
>>>> status-code check to accept both 200 and 204 responses (and potentially
>>>> support other 2xx codes) to improve interoperability.
>>>>
>>>> This aligns behavior with PVE’s more tolerant Perl ACME client and
>>>> avoids regressions.
>>>>
>>>> [1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
>>>> [2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
>>>>
>>>> Fixes: #6939
>>>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>>>> ---
>>>> proxmox-acme/src/account.rs | 10 +++++-----
>>>> proxmox-acme/src/async_client.rs | 6 +++---
>>>> proxmox-acme/src/client.rs | 2 +-
>>>> proxmox-acme/src/request.rs | 4 ++--
>>>> 4 files changed, 11 insertions(+), 11 deletions(-)
>>>>
>>>> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
>>>> index 73d786b8..60719865 100644
>>>> --- a/proxmox-acme/src/account.rs
>>>> +++ b/proxmox-acme/src/account.rs
>>>> @@ -85,7 +85,7 @@ impl Account {
>>>> method: "POST",
>>>> content_type: crate::request::JSON_CONTENT_TYPE,
>>>> body,
>>>> - expected: crate::request::CREATED,
>>>> + expected: vec![crate::request::CREATED],
>>>
>>> while this is defined as dedicated constant...
>>>
>>>> };
>>>> Ok(NewOrder::new(request))
>>>> @@ -107,7 +107,7 @@ impl Account {
>>>> method: "POST",
>>>> content_type: crate::request::JSON_CONTENT_TYPE,
>>>> body,
>>>> - expected: 200,
>>>> + expected: vec![200],
>>>
>>> ... these and the others below are not. Same for the 204 status code you are about to add.
>>>
>>> So in preparation for adding the new status code, these should probably be defined as, either:
>>> - as dedicated status code constants as well, or
>>> - all moved over to directly use https://docs.rs/http/1.3.1/http/status/struct.StatusCode.html
>>>
>>> I feel like the latter is not done here intentionally to avoid the dependency on hyper or http (re-exported by hyper) for the api types only.
>>
>> While you are right that constants are generally nicer, IMO HTTP codes are
>> very stable and universal to be fine to be used directly as numbers in the few
>> limited instances here.
>
> Mostly this, but we can also just add internal constants as well. 200
> just seemed common enough...
>>
>> If we already (even just transitively) would get them from a dependency we still
>> should switch to that, but I'd not introduce a new dependency just for that; IMO
>> to high of a cost.
First thanks for the review Christian, Thomas, Wolfgang - agree!
I checked the option of using `StatusCode`, but as you mentioned, that
would require adding the `http` or `hyper` dependency, which we
currently don’t include in the core types. I would therefore drop the
constant as suggested and introduced a small, dedicated internal module
to hold the ACME HTTP success constants. This would keep them together
and would make imports handy.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-29 10:38 5% ` Wolfgang Bumiller
@ 2025-10-29 15:56 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-10-29 15:56 UTC (permalink / raw)
To: Wolfgang Bumiller; +Cc: pbs-devel
On 10/29/25 11:38 AM, Wolfgang Bumiller wrote:
> On Tue, Oct 28, 2025 at 04:22:00PM +0100, Samuel Rufinatscha wrote:
>> Some ACME servers (notably custom or legacy implementations) respond
>> to HEAD /newNonce with a 204 No Content instead of the
>> RFC 8555-recommended 200 OK [1]. While this behavior is technically
>> off-spec, it is functionally harmless. This issue was reported on our
>> bug tracker [2].
>>
>> The previous implementation treated any non-200 response as an error,
>> causing account registration to fail against such servers. Relax the
>> status-code check to accept both 200 and 204 responses (and potentially
>> support other 2xx codes) to improve interoperability.
>>
>> This aligns behavior with PVE’s more tolerant Perl ACME client and
>> avoids regressions.
>>
>> [1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
>> [2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
>>
>> Fixes: #6939
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> proxmox-acme/src/account.rs | 10 +++++-----
>> proxmox-acme/src/async_client.rs | 6 +++---
>> proxmox-acme/src/client.rs | 2 +-
>> proxmox-acme/src/request.rs | 4 ++--
>> 4 files changed, 11 insertions(+), 11 deletions(-)
>>
>> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
>> index 73d786b8..60719865 100644
>> --- a/proxmox-acme/src/account.rs
>> +++ b/proxmox-acme/src/account.rs
>> @@ -85,7 +85,7 @@ impl Account {
>> method: "POST",
>> content_type: crate::request::JSON_CONTENT_TYPE,
>> body,
>> - expected: crate::request::CREATED,
>> + expected: vec![crate::request::CREATED],
>> };
>>
>> Ok(NewOrder::new(request))
>> @@ -107,7 +107,7 @@ impl Account {
>> method: "POST",
>> content_type: crate::request::JSON_CONTENT_TYPE,
>> body,
>> - expected: 200,
>> + expected: vec![200],
>> })
>> }
>>
>> @@ -132,7 +132,7 @@ impl Account {
>> method: "POST",
>> content_type: crate::request::JSON_CONTENT_TYPE,
>> body,
>> - expected: 200,
>> + expected: vec![200],
>> })
>> }
>>
>> @@ -157,7 +157,7 @@ impl Account {
>> method: "POST",
>> content_type: crate::request::JSON_CONTENT_TYPE,
>> body,
>> - expected: 200,
>> + expected: vec![200],
>> })
>> }
>>
>> @@ -405,7 +405,7 @@ impl AccountCreator {
>> method: "POST",
>> content_type: crate::request::JSON_CONTENT_TYPE,
>> body,
>> - expected: crate::request::CREATED,
>> + expected: vec![crate::request::CREATED],
>> })
>> }
>>
>> diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
>> index 60e1f359..0901aa8d 100644
>> --- a/proxmox-acme/src/async_client.rs
>> +++ b/proxmox-acme/src/async_client.rs
>> @@ -421,7 +421,7 @@ impl AcmeClient {
>> };
>>
>> if parts.status.is_success() {
>> - if status != request.expected {
>> + if !request.expected.contains(&status) {
>> return Err(Error::InvalidApi(format!(
>> "ACME server responded with unexpected status code: {:?}",
>> parts.status
>> @@ -501,7 +501,7 @@ impl AcmeClient {
>> method: "GET",
>> content_type: "",
>> body: String::new(),
>> - expected: 200,
>> + expected: vec![200],
>> },
>> nonce,
>> )
>> @@ -553,7 +553,7 @@ impl AcmeClient {
>> method: "HEAD",
>> content_type: "",
>> body: String::new(),
>> - expected: 200,
>> + expected: vec![200, 204],
>> },
>> nonce,
>> )
>> diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
>> index d8a62081..ea8a8655 100644
>> --- a/proxmox-acme/src/client.rs
>> +++ b/proxmox-acme/src/client.rs
>> @@ -203,7 +203,7 @@ impl Inner {
>> let got_nonce = self.update_nonce(&mut response)?;
>>
>> if response.is_success() {
>> - if response.status != request.expected {
>> + if !request.expected.contains(&response.status) {
>> return Err(Error::InvalidApi(format!(
>> "API server responded with unexpected status code: {:?}",
>> response.status
>> diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
>> index 78a90913..38e825d6 100644
>> --- a/proxmox-acme/src/request.rs
>> +++ b/proxmox-acme/src/request.rs
>> @@ -17,8 +17,8 @@ pub struct Request {
>> /// The body to pass along with request, or an empty string.
>> pub body: String,
>>
>> - /// The expected status code a compliant ACME provider will return on success.
>> - pub expected: u16,
>> + /// The set of HTTP status codes that indicate a successful response from an ACME provider.
>> + pub expected: Vec<u16>,
>
> We always have a static set, so I'd rather use `&'static [u16]` here.
> There's no need to allocate usually-single-element vectors everywhere.
Agree, will replace the `Vec` with `&'static [u16]`.
>
>> }
>>
>> /// An ACME error response contains a specially formatted type string, and can optionally
>> --
>> 2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-29 7:51 5% ` [pbs-devel] [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests Thomas Lamprecht
@ 2025-10-29 16:02 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-10-29 16:02 UTC (permalink / raw)
To: Thomas Lamprecht, Proxmox Backup Server development discussion
On 10/29/25 8:50 AM, Thomas Lamprecht wrote:
> Thanks for the patch and actually also checking if PVE is affected, and why
> it isn't.
>
> Am 28.10.25 um 20:34 schrieb Samuel Rufinatscha:
>> This is also stricter than the PVE Perl ACME client, which tolerates any
>> 2xx success codes [3]. The author mentions, the issue did not appear
>> with PVE9 [1].
>>
>> ## Ideas to solve the problem
>>
>> To support ACME providers which return 204 No Content, the underlying
>> ACME clients need to tolerate both 200 OK and 204 No Content as valid
>> responses for the nonce HEAD request, as long as the Replay-Nonce is
>> provided.
>>
>> I considered following solutions:
>>
>> 1. Change the `expected` field of the `AcmeRequest` type from `u16` to
>> `Vec<u16>`, to support multiple success codes
>>
>> 2. Keep `expected: u16` and add a second field e.g. `expected_other:
>> Vec<u16>` for "also allowed" codes.
>>
>> 3. Support any 2xx success codes, and remove the `expected` check
>>
>> I thought (1) might be reasonable, because:
>>
>> * It stays explicit and makes it clear which statuses are considered
>> success.
>> * We don’t create two parallel concepts ("expected" vs
>> "expected_other") which introduces additional complexity
>> * Can be extend later if we meet yet another harmless but not 200
>> variant.
>> * We don’t allow arbitrary 2xx.
>>
>> What do you think? Do you maybe have any other solution in mind that
>> would fit better?
>
> There probably isn't answer that strictly right in all cases, but in
> general it's good to have similar behavior across implementations,
> especially given that we do not know of any report where the PVE behavior
> of accepting all 2xx response codes caused any problems, from that 3.
> would be best, or does the RFC forbid the server to accept other status
> codes?
>
> That said, in practice only 201 (Created) might make sense for ACME in
> addition to the referenced 200 (OK) and 204 (No Content), and following
> the RFC is fine, so 1. is IMO also a good solution here.
> Please note the difference to PVE in the commit message, there you write
> that behavior is now aligned with PVE, but it's rather "closer aligned"
> to, not fully.
According to RFC 8555, most ACME endpoints define exact expected status
codes. The only case where multiple 2xx codes are explicitly allowed is the
newNonce endpoint, where the RFC states that the server SHOULD return
200 for
HEAD and MUST return 204 for GET.
I further looked into the Perl client and found that it enforces specific
codes for most endpoints as well, but performs a GET request [1] for nonce
retrieval (instead of HEAD) and accepts any 2xx success [2] code in that
case. So the behavior isn’t directly comparable to the Rust client, but it’s
worth noting. Functionally, this shouldn’t be an issue, as ACME
providers are
required to support both methods, with HEAD being the preferred one.
I will update the commit messages and cover letter accordingly.
[1]
https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219
[2]
https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* [pbs-devel] [PATCH proxmox{, -backup} v2 0/2] fix #6939: acme: support servers returning 204 for nonce requests
@ 2025-10-29 16:45 13% Samuel Rufinatscha
2025-10-29 16:45 13% ` [pbs-devel] [PATCH proxmox v2 1/1] " Samuel Rufinatscha
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Samuel Rufinatscha @ 2025-10-29 16:45 UTC (permalink / raw)
To: pbs-devel
Hi,
this series proposes a change to ACME account registration in Proxmox
Backup Server (PBS), so that it also works with ACME servers that return
HTTP 204 No Content to the HEAD request for newNonce.
This behaviour was observed against a specific ACME deployment and
reported as bug #6939 [1]. Currently, PBS cannot register an ACME
account for this CA.
## Problem
During ACME account registration, PBS first fetches an anti-replay nonce
by sending a HEAD request to the CA’s newNonce URL. RFC 8555 7.2 [2]
says:
* the server MUST include a Replay-Nonce header with a fresh nonce,
* the server SHOULD use status 200 OK for the HEAD request,
* the server MUST also handle GET on the same resource with status 204 No
Content and an empty body [2].
Currently, our Rust ACME clients only accept 200 OK. PBS inherits that
strictness and aborts with:
*ACME server responded with unexpected status code: 204*
The author mentions, the issue did not appear with PVE9 [1].
After looking into PVE’s Perl ACME client [3] it appears it uses a GET
request instead of a HEAD request and accepts any 2xx success code
when retrieving the nonce [5]. This difference in behavior does not
affect functionality but is worth noting for consistency across
implementations.
## Ideas to solve the problem
To support ACME providers which return 204 No Content, the underlying
ACME clients need to tolerate both 200 OK and 204 No Content as valid
responses for the nonce HEAD request, as long as the Replay-Nonce is
provided.
I considered following solutions:
1. Change the `expected` field of the `AcmeRequest` type from `u16` to
`Vec<u16>`, to support multiple success codes
2. Keep `expected: u16` and add a second field e.g. `expected_other:
Vec<u16>` for "also allowed" codes.
3. Support any 2xx success codes, and remove the `expected` check
I thought (1) might be reasonable, because:
* It stays explicit and makes it clear which statuses are considered
success.
* We don’t create two parallel concepts ("expected" vs
"expected_other") which introduces additional complexity
* Can be extend later if we meet yet another harmless but not 200
variant.
* We don’t allow arbitrary 2xx.
What do you think? Do you maybe have any other solution in mind that
would fit better?
## Testing
To prove the proposed fix, I reproduced the scenario:
Pebble (release 2.8.0) from Let's Encrypt [5] running on a Debian 9 VM
as the ACME server. nginx in front of Pebble, to intercept the
`newNonce` request in order to return 204 No Content instead of 200 OK,
all other requests are unchanged and forwarded to Pebble. Trust the
Pebble and ngix CAs via `/usr/local/share/ca-certificates` +
`update-ca-certificates` on the PBS VM.
Then I ran following command against nginx:
```
proxmox-backup-manager acme account register proxytest root@backup.local
--directory 'https://nginx-address/dir
Attempting to fetch Terms of
Service from "https://acme-vm/dir"
Terms of Service:
data:text/plain,Do%20what%20thou%20wilt
Do you agree to the above terms?
[y|N]: y
Do you want to use external account binding? [y|N]: N
Attempting
to register account with "https://acme-vm/dir"...
Registration
successful, account URL: https://acme-vm/my-account/160e58b66bdd72da
```
When adjusting the nginx configuration to return any other non-expected
success status code, e.g. 205, PBS expectely rejects with `API
misbehaved: ACME server responded with unexpected status code: 205`.
## Maintainer notes:
The patch series involves the following components:
proxmox-acme: Apply PATCH 1 to change `expected` from `u16` to
`Vec<u16>`. This results in a breaking change, as it changes the public
API of the `AcmeRequest` type that is used by other components.
proxmox-acme-api: Needs to depend on the new proxmox-acme; patch bump
proxmox-backup: Apply PATCH 2 to use the new API changes; no breaking
change as of only internal changes; patch bump
proxmox-perl-rs / proxmox-datacenter-manager: Will need to use the
dependency version bumps to follow the new proxmox-acme.
## Patch summary
[PATCH 1/2] fix #6939: support providers returning 204 for nonce
requests
* Make the expected-status logic accept multiple allowed codes.
* Treat both 200 OK and 204 No Content as valid for HEAD /newNonce,
provided Replay-Nonce is present.
* Keep rejecting other codes.
[PATCH 2/2] acme: accept HTTP 204 from newNonce endpoint
* Use the updated proxmox-acme behavior in PBS.
* PBS can now register an ACME account against servers that return 204
for the nonce HEAD request.
* Still rejects unexpected codes.
Thanks for considering this patch series, I look forward to your
feedback.
Best,
Samuel Rufinatscha
## Changes from v1:
[PATCH 1/2] fix #6939: support providers returning 204 for nonce
requests
* Introduced `http_success` module to contain the http success codes
* Replaced `Vec<u16>` with `&[u16]` for expected codes to avoid
allocations.
* Clarified the PVEs Perl ACME client behaviour in the commit message.
[PATCH 2/2] acme: accept HTTP 204 from newNonce endpoint
* Integrated the `http_success` module, replacing `Vec<u16>` with `&[u16]`
* Clarified the PVEs Perl ACME client behaviour in the commit message.
[1] Bugzilla report #6939:
[https://bugzilla.proxmox.com/show_bug.cgi?id=6939](https://bugzilla.proxmox.com/show_bug.cgi?id=6939)
[2] RFC 8555 (ACME):
[https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2](https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2)
[3] PVE’s Perl ACME client (allow 2xx codes for nonce requests):
[https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597)
[4] Pebble ACME server:
[https://github.com/letsencrypt/pebble](https://github.com/letsencrypt/pebble)
[5] Pebble ACME server (perform GET request:
[https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219)
proxmox:
Samuel Rufinatscha (1):
fix #6939: acme: support servers returning 204 for nonce requests
proxmox-acme/src/account.rs | 10 +++++-----
proxmox-acme/src/async_client.rs | 6 +++---
proxmox-acme/src/client.rs | 2 +-
proxmox-acme/src/lib.rs | 4 ++++
proxmox-acme/src/request.rs | 15 ++++++++++++---
5 files changed, 25 insertions(+), 12 deletions(-)
proxmox-backup:
Samuel Rufinatscha (1):
fix #6939: acme: accept HTTP 204 from newNonce endpoint
src/acme/client.rs | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
Summary over all repositories:
6 files changed, 29 insertions(+), 16 deletions(-)
--
Generated by git-murpp 0.8.1
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox-backup v2 1/1] fix #6939: acme: accept HTTP 204 from newNonce endpoint
2025-10-29 16:45 13% [pbs-devel] [PATCH proxmox{, -backup} v2 " Samuel Rufinatscha
2025-10-29 16:45 13% ` [pbs-devel] [PATCH proxmox v2 1/1] " Samuel Rufinatscha
@ 2025-10-29 16:45 15% ` Samuel Rufinatscha
2025-11-03 10:21 13% ` [pbs-devel] superseded: [PATCH proxmox{, -backup} v2 0/2] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-10-29 16:45 UTC (permalink / raw)
To: pbs-devel
When registering an ACME account, PBS fetches a fresh nonce by issuing a
HEAD request to the server's newNonce URL. Until now we assumed this
request would return HTTP 200 OK.
In practice, some ACME servers [1] respond with HTTP 204 No Content for
this HEAD request while still providing a valid Replay-Nonce header.
This causes PBS to abort registration with "ACME server responded with
unexpected status code: 204", even though the server would otherwise
issue certificates correctly. This behavior is technically
off-spec [2], however not forbidden.
Adjust the ACME client code in PBS to accept both 200 OK and 204 No
Content as successful results for the newNonce step. We continue to
reject other status codes so we don't silently accept arbitrary 2xx
responses.
Note: In comparison, PVE’s Perl ACME client performs a GET request [3]
instead of a HEAD request and accepts any 2xx success code when
retrieving the nonce [4]. This difference in behavior does not affect
functionality but is worth noting for consistency across
implementations.
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
[2] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
[3] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219
[4] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597
Fixes: #6939
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/acme/client.rs | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/acme/client.rs b/src/acme/client.rs
index 1c12a4b9..7c32940b 100644
--- a/src/acme/client.rs
+++ b/src/acme/client.rs
@@ -15,7 +15,7 @@ use serde::{Deserialize, Serialize};
use proxmox_acme::account::AccountCreator;
use proxmox_acme::order::{Order, OrderData};
use proxmox_acme::types::AccountData as AcmeAccountData;
-use proxmox_acme::Request as AcmeRequest;
+use proxmox_acme::{http_success, Request as AcmeRequest};
use proxmox_acme::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
use proxmox_http::client::Client;
use proxmox_sys::fs::{replace_file, CreateOptions};
@@ -530,7 +530,7 @@ impl AcmeClient {
};
if parts.status.is_success() {
- if status != request.expected {
+ if !request.expected.contains(&status) {
return Err(Error::InvalidApi(format!(
"ACME server responded with unexpected status code: {:?}",
parts.status
@@ -609,7 +609,7 @@ impl AcmeClient {
method: "GET",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: &[http_success::OK],
},
nonce,
)
@@ -657,7 +657,7 @@ impl AcmeClient {
method: "HEAD",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: &[http_success::OK, http_success::NO_CONTENT],
},
nonce,
)
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox v2 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-29 16:45 13% [pbs-devel] [PATCH proxmox{, -backup} v2 " Samuel Rufinatscha
@ 2025-10-29 16:45 13% ` Samuel Rufinatscha
2025-10-31 16:21 5% ` Thomas Lamprecht
2025-10-29 16:45 15% ` [pbs-devel] [PATCH proxmox-backup v2 1/1] fix #6939: acme: accept HTTP 204 from newNonce endpoint Samuel Rufinatscha
2025-11-03 10:21 13% ` [pbs-devel] superseded: [PATCH proxmox{, -backup} v2 0/2] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-10-29 16:45 UTC (permalink / raw)
To: pbs-devel
Some ACME servers (notably custom or legacy implementations) respond
to HEAD /newNonce with a 204 No Content instead of the
RFC 8555-recommended 200 OK [1]. While this behavior is technically
off-spec, it is not illegal. This issue was reported on our bug
tracker [2].
The previous implementation treated any non-200 response as an error,
causing account registration to fail against such servers. Relax the
status-code check to accept both 200 and 204 responses (and potentially
support other 2xx codes) to improve interoperability.
Note: In comparison, PVE’s Perl ACME client performs a GET request [3]
instead of a HEAD request and accepts any 2xx success code when
retrieving the nonce [4]. This difference in behavior does not affect
functionality but is worth noting for consistency across
implementations.
[1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
[3] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219
[4] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597
Fixes: #6939
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme/src/account.rs | 10 +++++-----
proxmox-acme/src/async_client.rs | 6 +++---
proxmox-acme/src/client.rs | 2 +-
proxmox-acme/src/lib.rs | 4 ++++
proxmox-acme/src/request.rs | 15 ++++++++++++---
5 files changed, 25 insertions(+), 12 deletions(-)
diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
index 73d786b8..44f9383f 100644
--- a/proxmox-acme/src/account.rs
+++ b/proxmox-acme/src/account.rs
@@ -85,7 +85,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::request::CREATED,
+ expected: &[crate::http_success::CREATED],
};
Ok(NewOrder::new(request))
@@ -107,7 +107,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: &[crate::http_success::OK],
})
}
@@ -132,7 +132,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: &[crate::http_success::OK],
})
}
@@ -157,7 +157,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: &[crate::http_success::OK],
})
}
@@ -405,7 +405,7 @@ impl AccountCreator {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::request::CREATED,
+ expected: &[crate::http_success::CREATED],
})
}
diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
index 60e1f359..b9df0f55 100644
--- a/proxmox-acme/src/async_client.rs
+++ b/proxmox-acme/src/async_client.rs
@@ -421,7 +421,7 @@ impl AcmeClient {
};
if parts.status.is_success() {
- if status != request.expected {
+ if !request.expected.contains(&status) {
return Err(Error::InvalidApi(format!(
"ACME server responded with unexpected status code: {:?}",
parts.status
@@ -501,7 +501,7 @@ impl AcmeClient {
method: "GET",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: &[crate::http_success::OK],
},
nonce,
)
@@ -553,7 +553,7 @@ impl AcmeClient {
method: "HEAD",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: &[crate::http_success::OK, crate::http_success::NO_CONTENT],
},
nonce,
)
diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
index d8a62081..ea8a8655 100644
--- a/proxmox-acme/src/client.rs
+++ b/proxmox-acme/src/client.rs
@@ -203,7 +203,7 @@ impl Inner {
let got_nonce = self.update_nonce(&mut response)?;
if response.is_success() {
- if response.status != request.expected {
+ if !request.expected.contains(&response.status) {
return Err(Error::InvalidApi(format!(
"API server responded with unexpected status code: {:?}",
response.status
diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
index df722629..ec586ec4 100644
--- a/proxmox-acme/src/lib.rs
+++ b/proxmox-acme/src/lib.rs
@@ -70,6 +70,10 @@ pub use order::Order;
#[doc(inline)]
pub use request::Request;
+#[cfg(feature = "impl")]
+#[doc(inline)]
+pub use request::http_success;
+
// we don't inline these:
#[cfg(feature = "impl")]
pub use order::NewOrder;
diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
index 78a90913..0532528e 100644
--- a/proxmox-acme/src/request.rs
+++ b/proxmox-acme/src/request.rs
@@ -1,7 +1,6 @@
use serde::Deserialize;
pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
-pub(crate) const CREATED: u16 = 201;
/// A request which should be performed on the ACME provider.
pub struct Request {
@@ -17,8 +16,18 @@ pub struct Request {
/// The body to pass along with request, or an empty string.
pub body: String,
- /// The expected status code a compliant ACME provider will return on success.
- pub expected: u16,
+ /// The set of HTTP status codes that indicate a successful response from an ACME provider.
+ pub expected: &'static [u16],
+}
+
+/// Common HTTP success status codes used in ACME responses.
+pub mod http_success {
+ /// 200 OK
+ pub const OK: u16 = 200;
+ /// 201 Created
+ pub const CREATED: u16 = 201;
+ /// 204 No Content
+ pub const NO_CONTENT: u16 = 204;
}
/// An ACME error response contains a specially formatted type string, and can optionally
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] superseded: [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-28 15:21 14% [pbs-devel] [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (2 preceding siblings ...)
2025-10-29 7:51 5% ` [pbs-devel] [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests Thomas Lamprecht
@ 2025-10-29 16:49 13% ` Samuel Rufinatscha
3 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-10-29 16:49 UTC (permalink / raw)
To: pbs-devel
https://lore.proxmox.com/pbs-devel/20251029164520.263926-1-s.rufinatscha@proxmox.com/T/#t
On 10/28/25 4:21 PM, Samuel Rufinatscha wrote:
> Hi,
>
> this series proposes a change to ACME account registration in Proxmox
> Backup Server (PBS), so that it also works with ACME servers that return
> HTTP 204 No Content to the HEAD request for newNonce.
>
> This behaviour was observed against a specific ACME deployment and
> reported as bug #6939 [1]. Currently, PBS cannot register an ACME
> account for this CA.
>
> ## Problem
>
> During ACME account registration, PBS first fetches an anti-replay nonce
> by sending a HEAD request to the CA’s newNonce URL. RFC 8555 7.2 [2]
> says:
>
> * the server MUST include a Replay-Nonce header with a fresh nonce,
> * the server SHOULD use status 200 OK for the HEAD request,
> * the server MUST also handle GET on the same resource with status 204 No
> Content and an empty body [2].
>
> Currently, our Rust ACME clients only accept 200 OK. PBS inherits that
> strictness and aborts with:
>
> *ACME server responded with unexpected status code: 204*
>
> This is also stricter than the PVE Perl ACME client, which tolerates any
> 2xx success codes [3]. The author mentions, the issue did not appear
> with PVE9 [1].
>
> ## Ideas to solve the problem
>
> To support ACME providers which return 204 No Content, the underlying
> ACME clients need to tolerate both 200 OK and 204 No Content as valid
> responses for the nonce HEAD request, as long as the Replay-Nonce is
> provided.
>
> I considered following solutions:
>
> 1. Change the `expected` field of the `AcmeRequest` type from `u16` to
> `Vec<u16>`, to support multiple success codes
>
> 2. Keep `expected: u16` and add a second field e.g. `expected_other:
> Vec<u16>` for "also allowed" codes.
>
> 3. Support any 2xx success codes, and remove the `expected` check
>
> I thought (1) might be reasonable, because:
>
> * It stays explicit and makes it clear which statuses are considered
> success.
> * We don’t create two parallel concepts ("expected" vs
> "expected_other") which introduces additional complexity
> * Can be extend later if we meet yet another harmless but not 200
> variant.
> * We don’t allow arbitrary 2xx.
>
> What do you think? Do you maybe have any other solution in mind that
> would fit better?
>
> ## Testing
>
> To prove the proposed fix, I reproduced the scenario:
>
> Pebble (release 2.8.0) from Let's Encrypt [5] running on a Debian 9 VM
> as the ACME server. nginx in front of Pebble, to intercept the
> `newNonce` request in order to return 204 No Content instead of 200 OK,
> all other requests are unchanged and forwarded to Pebble. Trust the
> Pebble and ngix CAs via `/usr/local/share/ca-certificates` +
> `update-ca-certificates` on the PBS VM.
>
> Then I ran following command against nginx:
>
> ```
> proxmox-backup-manager acme account register proxytest root@backup.local
> --directory 'https://nginx-address/dir
>
> Attempting to fetch Terms of
> Service from "https://acme-vm/dir"
> Terms of Service:
> data:text/plain,Do%20what%20thou%20wilt
> Do you agree to the above terms?
> [y|N]: y
> Do you want to use external account binding? [y|N]: N
> Attempting
> to register account with "https://acme-vm/dir"...
> Registration
> successful, account URL: https://acme-vm/my-account/160e58b66bdd72da
> ```
>
> When adjusting the nginx configuration to return any other non-expected
> success status code, e.g. 205, PBS expectely rejects with `API
> misbehaved: ACME server responded with unexpected status code: 205`.
>
> ## Maintainer notes:
>
> The patch series involves the following components:
>
> proxmox-acme: Apply PATCH 1 to change `expected` from `u16` to
> `Vec<u16>`. This results in a breaking change, as it changes the public
> API of the `AcmeRequest` type that is used by other components.
>
> proxmox-acme-api: Needs to depend on the new proxmox-acme; patch bump
>
> proxmox-backup: Apply PATCH 2 to use the new API changes; no breaking
> change as of only internal changes; patch bump
>
> proxmox-perl-rs / proxmox-datacenter-manager: Will need to use the
> dependency version bumps to follow the new proxmox-acme.
>
> ## Patch summary
>
> [PATCH 1/2] fix #6939: support providers returning 204 for nonce
> requests
>
> * Make the expected-status logic accept multiple allowed codes.
> * Treat both 200 OK and 204 No Content as valid for HEAD /newNonce,
> provided Replay-Nonce is present.
> * Keep rejecting other codes.
>
> [PATCH 2/2] acme: accept HTTP 204 from newNonce endpoint
>
> * Use the updated proxmox-acme behavior in PBS.
> * PBS can now register an ACME account against servers that return 204
> for the nonce HEAD request.
> * Still rejects unexpected codes.
>
> Thanks for considering this patch series, I look forward to your
> feedback.
>
> Best,
> Samuel Rufinatscha
>
> [1] Bugzilla report #6939:
> [https://bugzilla.proxmox.com/show_bug.cgi?id=6939](https://bugzilla.proxmox.com/show_bug.cgi?id=6939)
> [2] RFC 8555 (ACME):
> [https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2](https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2)
> [3] PVE’s Perl ACME client:
> [https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597)
> [4] Pebble ACME server:
> [https://github.com/letsencrypt/pebble](https://github.com/letsencrypt/pebble)
>
> proxmox:
>
> Samuel Rufinatscha (1):
> fix #6939: acme: support servers returning 204 for nonce requests
>
> proxmox-acme/src/account.rs | 10 +++++-----
> proxmox-acme/src/async_client.rs | 6 +++---
> proxmox-acme/src/client.rs | 2 +-
> proxmox-acme/src/request.rs | 4 ++--
> 4 files changed, 11 insertions(+), 11 deletions(-)
>
>
> proxmox-backup:
>
> Samuel Rufinatscha (1):
> fix #6939: acme: accept HTTP 204 from newNonce endpoint
>
> src/acme/client.rs | 6 +++---
> 1 file changed, 3 insertions(+), 3 deletions(-)
>
>
> Summary over all repositories:
> 5 files changed, 14 insertions(+), 14 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* Re: [pbs-devel] [PATCH proxmox v2 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-29 16:45 13% ` [pbs-devel] [PATCH proxmox v2 1/1] " Samuel Rufinatscha
@ 2025-10-31 16:21 5% ` Thomas Lamprecht
2025-11-03 8:51 9% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Thomas Lamprecht @ 2025-10-31 16:21 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Samuel Rufinatscha
Am 29.10.25 um 17:45 schrieb Samuel Rufinatscha:
> Some ACME servers (notably custom or legacy implementations) respond
> to HEAD /newNonce with a 204 No Content instead of the
> RFC 8555-recommended 200 OK [1]. While this behavior is technically
> off-spec, it is not illegal. This issue was reported on our bug
> tracker [2].
>
> The previous implementation treated any non-200 response as an error,
> causing account registration to fail against such servers. Relax the
> status-code check to accept both 200 and 204 responses (and potentially
> support other 2xx codes) to improve interoperability.
>
> Note: In comparison, PVE’s Perl ACME client performs a GET request [3]
> instead of a HEAD request and accepts any 2xx success code when
> retrieving the nonce [4]. This difference in behavior does not affect
> functionality but is worth noting for consistency across
> implementations.
>
> [1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
> [2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
> [3] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219
> [4] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597
>
> Fixes: #6939
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
Thanks for the v2, looks OK in general, one naming issue – that irked
my a tiny bit to much to just ignore it – inline.
> proxmox-acme/src/account.rs | 10 +++++-----
> proxmox-acme/src/async_client.rs | 6 +++---
> proxmox-acme/src/client.rs | 2 +-
> proxmox-acme/src/lib.rs | 4 ++++
> proxmox-acme/src/request.rs | 15 ++++++++++++---
> 5 files changed, 25 insertions(+), 12 deletions(-)
> diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
> index 78a90913..0532528e 100644
> --- a/proxmox-acme/src/request.rs
> +++ b/proxmox-acme/src/request.rs
> @@ -1,7 +1,6 @@
> use serde::Deserialize;
>
> pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
> -pub(crate) const CREATED: u16 = 201;
>
> /// A request which should be performed on the ACME provider.
> pub struct Request {
> @@ -17,8 +16,18 @@ pub struct Request {
> /// The body to pass along with request, or an empty string.
> pub body: String,
>
> - /// The expected status code a compliant ACME provider will return on success.
> - pub expected: u16,
> + /// The set of HTTP status codes that indicate a successful response from an ACME provider.
> + pub expected: &'static [u16],
> +}
> +
> +/// Common HTTP success status codes used in ACME responses.
> +pub mod http_success {
It's not wrong, but reads a bit odd to me; is it really relevant to differ
between success and (in the future, e.g. if this gets copied+adapted somewhere
else) get a http_client_error or http_server_error module?
I'd probably rather just use one of http_status or http_status_code, as the
codes themselves are uniquely named already anyway and with "status" included
on the name it would be IMO slightly better describe what this refers to without
having to read the doc comment.
> + /// 200 OK
> + pub const OK: u16 = 200;
> + /// 201 Created
> + pub const CREATED: u16 = 201;
> + /// 204 No Content
> + pub const NO_CONTENT: u16 = 204;
> }
>
> /// An ACME error response contains a specially formatted type string, and can optionally
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox v2 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-31 16:21 5% ` Thomas Lamprecht
@ 2025-11-03 8:51 9% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-03 8:51 UTC (permalink / raw)
To: Thomas Lamprecht, Proxmox Backup Server development discussion
On 10/31/25 5:21 PM, Thomas Lamprecht wrote:
> Am 29.10.25 um 17:45 schrieb Samuel Rufinatscha:
>> Some ACME servers (notably custom or legacy implementations) respond
>> to HEAD /newNonce with a 204 No Content instead of the
>> RFC 8555-recommended 200 OK [1]. While this behavior is technically
>> off-spec, it is not illegal. This issue was reported on our bug
>> tracker [2].
>>
>> The previous implementation treated any non-200 response as an error,
>> causing account registration to fail against such servers. Relax the
>> status-code check to accept both 200 and 204 responses (and potentially
>> support other 2xx codes) to improve interoperability.
>>
>> Note: In comparison, PVE’s Perl ACME client performs a GET request [3]
>> instead of a HEAD request and accepts any 2xx success code when
>> retrieving the nonce [4]. This difference in behavior does not affect
>> functionality but is worth noting for consistency across
>> implementations.
>>
>> [1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
>> [2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
>> [3] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219
>> [4] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597
>>
>> Fixes: #6939
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>
> Thanks for the v2, looks OK in general, one naming issue – that irked
> my a tiny bit to much to just ignore it – inline.
>
>> proxmox-acme/src/account.rs | 10 +++++-----
>> proxmox-acme/src/async_client.rs | 6 +++---
>> proxmox-acme/src/client.rs | 2 +-
>> proxmox-acme/src/lib.rs | 4 ++++
>> proxmox-acme/src/request.rs | 15 ++++++++++++---
>> 5 files changed, 25 insertions(+), 12 deletions(-)
>
>> diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
>> index 78a90913..0532528e 100644
>> --- a/proxmox-acme/src/request.rs
>> +++ b/proxmox-acme/src/request.rs
>> @@ -1,7 +1,6 @@
>> use serde::Deserialize;
>>
>> pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
>> -pub(crate) const CREATED: u16 = 201;
>>
>> /// A request which should be performed on the ACME provider.
>> pub struct Request {
>> @@ -17,8 +16,18 @@ pub struct Request {
>> /// The body to pass along with request, or an empty string.
>> pub body: String,
>>
>> - /// The expected status code a compliant ACME provider will return on success.
>> - pub expected: u16,
>> + /// The set of HTTP status codes that indicate a successful response from an ACME provider.
>> + pub expected: &'static [u16],
>> +}
>> +
>> +/// Common HTTP success status codes used in ACME responses.
>> +pub mod http_success {
>
> It's not wrong, but reads a bit odd to me; is it really relevant to differ
> between success and (in the future, e.g. if this gets copied+adapted somewhere
> else) get a http_client_error or http_server_error module?
>
> I'd probably rather just use one of http_status or http_status_code, as the
> codes themselves are uniquely named already anyway and with "status" included
> on the name it would be IMO slightly better describe what this refers to without
> having to read the doc comment.
>
>> + /// 200 OK
>> + pub const OK: u16 = 200;
>> + /// 201 Created
>> + pub const CREATED: u16 = 201;
>> + /// 204 No Content
>> + pub const NO_CONTENT: u16 = 204;
>> }
>>
>> /// An ACME error response contains a specially formatted type string, and can optionally
>
Thanks, Thomas! Agreed, http_status sounds clearer. I’ll update it and
send a v3.
Best,
Samuel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 9%]
* [pbs-devel] [PATCH proxmox{, -backup} v3 0/2] fix #6939: acme: support servers returning 204 for nonce requests
@ 2025-11-03 10:13 13% Samuel Rufinatscha
2025-11-03 10:13 13% ` [pbs-devel] [PATCH proxmox v3 1/1] " Samuel Rufinatscha
` (2 more replies)
0 siblings, 3 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-03 10:13 UTC (permalink / raw)
To: pbs-devel
Hi,
this series proposes a change to ACME account registration in Proxmox
Backup Server (PBS), so that it also works with ACME servers that return
HTTP 204 No Content to the HEAD request for newNonce.
This behaviour was observed against a specific ACME deployment and
reported as bug #6939 [1]. Currently, PBS cannot register an ACME
account for this CA.
## Problem
During ACME account registration, PBS first fetches an anti-replay nonce
by sending a HEAD request to the CA’s newNonce URL. RFC 8555 7.2 [2]
says:
* the server MUST include a Replay-Nonce header with a fresh nonce,
* the server SHOULD use status 200 OK for the HEAD request,
* the server MUST also handle GET on the same resource with status 204 No
Content and an empty body [2].
Currently, our Rust ACME clients only accept 200 OK. PBS inherits that
strictness and aborts with:
*ACME server responded with unexpected status code: 204*
The author mentions, the issue did not appear with PVE9 [1].
After looking into PVE’s Perl ACME client [3] it appears it uses a GET
request instead of a HEAD request and accepts any 2xx success code
when retrieving the nonce [5]. This difference in behavior does not
affect functionality but is worth noting for consistency across
implementations.
## Ideas to solve the problem
To support ACME providers which return 204 No Content, the underlying
ACME clients need to tolerate both 200 OK and 204 No Content as valid
responses for the nonce HEAD request, as long as the Replay-Nonce is
provided.
I considered following solutions:
1. Change the `expected` field of the `AcmeRequest` type from `u16` to
`Vec<u16>`, to support multiple success codes
2. Keep `expected: u16` and add a second field e.g. `expected_other:
Vec<u16>` for "also allowed" codes.
3. Support any 2xx success codes, and remove the `expected` check
I thought (1) might be reasonable, because:
* It stays explicit and makes it clear which statuses are considered
success.
* We don’t create two parallel concepts ("expected" vs
"expected_other") which introduces additional complexity
* Can be extend later if we meet yet another harmless but not 200
variant.
* We don’t allow arbitrary 2xx.
What do you think? Do you maybe have any other solution in mind that
would fit better?
## Testing
To prove the proposed fix, I reproduced the scenario:
Pebble (release 2.8.0) from Let's Encrypt [5] running on a Debian 9 VM
as the ACME server. nginx in front of Pebble, to intercept the
`newNonce` request in order to return 204 No Content instead of 200 OK,
all other requests are unchanged and forwarded to Pebble. Trust the
Pebble and ngix CAs via `/usr/local/share/ca-certificates` +
`update-ca-certificates` on the PBS VM.
Then I ran following command against nginx:
```
proxmox-backup-manager acme account register proxytest root@backup.local
--directory 'https://nginx-address/dir
Attempting to fetch Terms of
Service from "https://acme-vm/dir"
Terms of Service:
data:text/plain,Do%20what%20thou%20wilt
Do you agree to the above terms?
[y|N]: y
Do you want to use external account binding? [y|N]: N
Attempting
to register account with "https://acme-vm/dir"...
Registration
successful, account URL: https://acme-vm/my-account/160e58b66bdd72da
```
When adjusting the nginx configuration to return any other non-expected
success status code, e.g. 205, PBS expectely rejects with `API
misbehaved: ACME server responded with unexpected status code: 205`.
## Maintainer notes:
The patch series involves the following components:
proxmox-acme: Apply PATCH 1 to change `expected` from `u16` to
`Vec<u16>`. This results in a breaking change, as it changes the public
API of the `AcmeRequest` type that is used by other components.
proxmox-acme-api: Needs to depend on the new proxmox-acme; patch bump
proxmox-backup: Apply PATCH 2 to use the new API changes; no breaking
change as of only internal changes; patch bump
proxmox-perl-rs / proxmox-datacenter-manager: Will need to use the
dependency version bumps to follow the new proxmox-acme.
## Patch summary
[PATCH 1/2] fix #6939: support providers returning 204 for nonce
requests
* Make the expected-status logic accept multiple allowed codes.
* Treat both 200 OK and 204 No Content as valid for HEAD /newNonce,
provided Replay-Nonce is present.
* Keep rejecting other codes.
[PATCH 2/2] acme: accept HTTP 204 from newNonce endpoint
* Use the updated proxmox-acme behavior in PBS.
* PBS can now register an ACME account against servers that return 204
for the nonce HEAD request.
* Still rejects unexpected codes.
Thanks for considering this patch series, I look forward to your
feedback.
Best,
Samuel Rufinatscha
## Changes from v1:
[PATCH 1/2] fix #6939: support providers returning 204 for nonce
requests
* Introduced `http_success` module to contain the http success codes
* Replaced `Vec<u16>` with `&[u16]` for expected codes to avoid
allocations.
* Clarified the PVEs Perl ACME client behaviour in the commit message.
[PATCH 2/2] acme: accept HTTP 204 from newNonce endpoint
* Integrated the `http_success` module, replacing `Vec<u16>` with `&[u16]`
* Clarified the PVEs Perl ACME client behaviour in the commit message.
## Changes from v2:
[PATCH 1/2] fix #6939: support providers returning 204 for nonce
requests
* Rename `http_success` module to `http_status`
[PATCH 2/2] acme: accept HTTP 204 from newNonce endpoint
* Replace `http_success` usage
[1] Bugzilla report #6939:
[https://bugzilla.proxmox.com/show_bug.cgi?id=6939](https://bugzilla.proxmox.com/show_bug.cgi?id=6939)
[2] RFC 8555 (ACME):
[https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2](https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2)
[3] PVE’s Perl ACME client (allow 2xx codes for nonce requests):
[https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597)
[4] Pebble ACME server:
[https://github.com/letsencrypt/pebble](https://github.com/letsencrypt/pebble)
[5] Pebble ACME server (perform GET request:
[https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219)
proxmox:
Samuel Rufinatscha (1):
fix #6939: acme: support servers returning 204 for nonce requests
proxmox-acme/src/account.rs | 10 +++++-----
proxmox-acme/src/async_client.rs | 6 +++---
proxmox-acme/src/client.rs | 2 +-
proxmox-acme/src/lib.rs | 4 ++++
proxmox-acme/src/request.rs | 15 ++++++++++++---
5 files changed, 25 insertions(+), 12 deletions(-)
proxmox-backup:
Samuel Rufinatscha (1):
fix #6939: acme: accept HTTP 204 from newNonce endpoint
src/acme/client.rs | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
Summary over all repositories:
6 files changed, 29 insertions(+), 16 deletions(-)
--
Generated by git-murpp 0.8.1
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox-backup v3 1/1] fix #6939: acme: accept HTTP 204 from newNonce endpoint
2025-11-03 10:13 13% [pbs-devel] [PATCH proxmox{, -backup} v3 " Samuel Rufinatscha
2025-11-03 10:13 13% ` [pbs-devel] [PATCH proxmox v3 1/1] " Samuel Rufinatscha
@ 2025-11-03 10:13 15% ` Samuel Rufinatscha
2025-12-03 10:23 13% ` [pbs-devel] superseded: [PATCH proxmox{, -backup} v3 0/2] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-03 10:13 UTC (permalink / raw)
To: pbs-devel
When registering an ACME account, PBS fetches a fresh nonce by issuing a
HEAD request to the server's newNonce URL. Until now we assumed this
request would return HTTP 200 OK.
In practice, some ACME servers [1] respond with HTTP 204 No Content for
this HEAD request while still providing a valid Replay-Nonce header.
This causes PBS to abort registration with "ACME server responded with
unexpected status code: 204", even though the server would otherwise
issue certificates correctly. This behavior is technically
off-spec [2], however not forbidden.
Adjust the ACME client code in PBS to accept both 200 OK and 204 No
Content as successful results for the newNonce step. We continue to
reject other status codes so we don't silently accept arbitrary 2xx
responses.
Note: In comparison, PVE’s Perl ACME client performs a GET request [3]
instead of a HEAD request and accepts any 2xx success code when
retrieving the nonce [4]. This difference in behavior does not affect
functionality but is worth noting for consistency across
implementations.
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
[2] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
[3] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219
[4] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597
Fixes: #6939
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/acme/client.rs | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/acme/client.rs b/src/acme/client.rs
index 9fb6ad55..1962529a 100644
--- a/src/acme/client.rs
+++ b/src/acme/client.rs
@@ -15,7 +15,7 @@ use serde::{Deserialize, Serialize};
use proxmox_acme::account::AccountCreator;
use proxmox_acme::order::{Order, OrderData};
use proxmox_acme::types::AccountData as AcmeAccountData;
-use proxmox_acme::Request as AcmeRequest;
+use proxmox_acme::{http_status, Request as AcmeRequest};
use proxmox_acme::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
use proxmox_http::client::Client;
use proxmox_sys::fs::{replace_file, CreateOptions};
@@ -529,7 +529,7 @@ impl AcmeClient {
};
if parts.status.is_success() {
- if status != request.expected {
+ if !request.expected.contains(&status) {
return Err(Error::InvalidApi(format!(
"ACME server responded with unexpected status code: {:?}",
parts.status
@@ -606,7 +606,7 @@ impl AcmeClient {
method: "GET",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: &[http_status::OK],
},
nonce,
)
@@ -654,7 +654,7 @@ impl AcmeClient {
method: "HEAD",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: &[http_status::OK, http_status::NO_CONTENT],
},
nonce,
)
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox v3 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-11-03 10:13 13% [pbs-devel] [PATCH proxmox{, -backup} v3 " Samuel Rufinatscha
@ 2025-11-03 10:13 13% ` Samuel Rufinatscha
2025-11-04 14:11 4% ` Thomas Lamprecht
2025-11-03 10:13 15% ` [pbs-devel] [PATCH proxmox-backup v3 1/1] fix #6939: acme: accept HTTP 204 from newNonce endpoint Samuel Rufinatscha
2025-12-03 10:23 13% ` [pbs-devel] superseded: [PATCH proxmox{, -backup} v3 0/2] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-11-03 10:13 UTC (permalink / raw)
To: pbs-devel
Some ACME servers (notably custom or legacy implementations) respond
to HEAD /newNonce with a 204 No Content instead of the
RFC 8555-recommended 200 OK [1]. While this behavior is technically
off-spec, it is not illegal. This issue was reported on our bug
tracker [2].
The previous implementation treated any non-200 response as an error,
causing account registration to fail against such servers. Relax the
status-code check to accept both 200 and 204 responses (and potentially
support other 2xx codes) to improve interoperability.
Note: In comparison, PVE’s Perl ACME client performs a GET request [3]
instead of a HEAD request and accepts any 2xx success code when
retrieving the nonce [4]. This difference in behavior does not affect
functionality but is worth noting for consistency across
implementations.
[1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
[3] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219
[4] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597
Fixes: #6939
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme/src/account.rs | 10 +++++-----
proxmox-acme/src/async_client.rs | 6 +++---
proxmox-acme/src/client.rs | 2 +-
proxmox-acme/src/lib.rs | 4 ++++
proxmox-acme/src/request.rs | 15 ++++++++++++---
5 files changed, 25 insertions(+), 12 deletions(-)
diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
index 0bbf0027..1ad485a2 100644
--- a/proxmox-acme/src/account.rs
+++ b/proxmox-acme/src/account.rs
@@ -85,7 +85,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::request::CREATED,
+ expected: &[crate::http_status::CREATED],
};
Ok(NewOrder::new(request))
@@ -107,7 +107,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: &[crate::http_status::OK],
})
}
@@ -132,7 +132,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: &[crate::http_status::OK],
})
}
@@ -157,7 +157,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: &[crate::http_status::OK],
})
}
@@ -405,7 +405,7 @@ impl AccountCreator {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::request::CREATED,
+ expected: &[crate::http_status::CREATED],
})
}
diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
index dc755fb9..66ec6024 100644
--- a/proxmox-acme/src/async_client.rs
+++ b/proxmox-acme/src/async_client.rs
@@ -420,7 +420,7 @@ impl AcmeClient {
};
if parts.status.is_success() {
- if status != request.expected {
+ if !request.expected.contains(&status) {
return Err(Error::InvalidApi(format!(
"ACME server responded with unexpected status code: {:?}",
parts.status
@@ -498,7 +498,7 @@ impl AcmeClient {
method: "GET",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: &[crate::http_status::OK],
},
nonce,
)
@@ -550,7 +550,7 @@ impl AcmeClient {
method: "HEAD",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: &[crate::http_status::OK, crate::http_status::NO_CONTENT],
},
nonce,
)
diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
index 931f7245..881ee83d 100644
--- a/proxmox-acme/src/client.rs
+++ b/proxmox-acme/src/client.rs
@@ -203,7 +203,7 @@ impl Inner {
let got_nonce = self.update_nonce(&mut response)?;
if response.is_success() {
- if response.status != request.expected {
+ if !request.expected.contains(&response.status) {
return Err(Error::InvalidApi(format!(
"API server responded with unexpected status code: {:?}",
response.status
diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
index df722629..cf1bc68d 100644
--- a/proxmox-acme/src/lib.rs
+++ b/proxmox-acme/src/lib.rs
@@ -70,6 +70,10 @@ pub use order::Order;
#[doc(inline)]
pub use request::Request;
+#[cfg(feature = "impl")]
+#[doc(inline)]
+pub use request::http_status;
+
// we don't inline these:
#[cfg(feature = "impl")]
pub use order::NewOrder;
diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
index 78a90913..24e669c5 100644
--- a/proxmox-acme/src/request.rs
+++ b/proxmox-acme/src/request.rs
@@ -1,7 +1,6 @@
use serde::Deserialize;
pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
-pub(crate) const CREATED: u16 = 201;
/// A request which should be performed on the ACME provider.
pub struct Request {
@@ -17,8 +16,18 @@ pub struct Request {
/// The body to pass along with request, or an empty string.
pub body: String,
- /// The expected status code a compliant ACME provider will return on success.
- pub expected: u16,
+ /// The set of HTTP status codes that indicate a successful response from an ACME provider.
+ pub expected: &'static [u16],
+}
+
+/// Common HTTP status codes used in ACME responses.
+pub mod http_status {
+ /// 200 OK
+ pub const OK: u16 = 200;
+ /// 201 Created
+ pub const CREATED: u16 = 201;
+ /// 204 No Content
+ pub const NO_CONTENT: u16 = 204;
}
/// An ACME error response contains a specially formatted type string, and can optionally
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] superseded: [PATCH proxmox{, -backup} v2 0/2] fix #6939: acme: support servers returning 204 for nonce requests
2025-10-29 16:45 13% [pbs-devel] [PATCH proxmox{, -backup} v2 " Samuel Rufinatscha
2025-10-29 16:45 13% ` [pbs-devel] [PATCH proxmox v2 1/1] " Samuel Rufinatscha
2025-10-29 16:45 15% ` [pbs-devel] [PATCH proxmox-backup v2 1/1] fix #6939: acme: accept HTTP 204 from newNonce endpoint Samuel Rufinatscha
@ 2025-11-03 10:21 13% ` Samuel Rufinatscha
2 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-03 10:21 UTC (permalink / raw)
To: pbs-devel
https://lore.proxmox.com/pbs-devel/20251103101322.100392-1-s.rufinatscha@proxmox.com/T/#t
On 10/29/25 5:45 PM, Samuel Rufinatscha wrote:
> Hi,
>
> this series proposes a change to ACME account registration in Proxmox
> Backup Server (PBS), so that it also works with ACME servers that return
> HTTP 204 No Content to the HEAD request for newNonce.
>
> This behaviour was observed against a specific ACME deployment and
> reported as bug #6939 [1]. Currently, PBS cannot register an ACME
> account for this CA.
>
> ## Problem
>
> During ACME account registration, PBS first fetches an anti-replay nonce
> by sending a HEAD request to the CA’s newNonce URL. RFC 8555 7.2 [2]
> says:
>
> * the server MUST include a Replay-Nonce header with a fresh nonce,
> * the server SHOULD use status 200 OK for the HEAD request,
> * the server MUST also handle GET on the same resource with status 204 No
> Content and an empty body [2].
>
> Currently, our Rust ACME clients only accept 200 OK. PBS inherits that
> strictness and aborts with:
>
> *ACME server responded with unexpected status code: 204*
>
> The author mentions, the issue did not appear with PVE9 [1].
> After looking into PVE’s Perl ACME client [3] it appears it uses a GET
> request instead of a HEAD request and accepts any 2xx success code
> when retrieving the nonce [5]. This difference in behavior does not
> affect functionality but is worth noting for consistency across
> implementations.
>
> ## Ideas to solve the problem
>
> To support ACME providers which return 204 No Content, the underlying
> ACME clients need to tolerate both 200 OK and 204 No Content as valid
> responses for the nonce HEAD request, as long as the Replay-Nonce is
> provided.
>
> I considered following solutions:
>
> 1. Change the `expected` field of the `AcmeRequest` type from `u16` to
> `Vec<u16>`, to support multiple success codes
>
> 2. Keep `expected: u16` and add a second field e.g. `expected_other:
> Vec<u16>` for "also allowed" codes.
>
> 3. Support any 2xx success codes, and remove the `expected` check
>
> I thought (1) might be reasonable, because:
>
> * It stays explicit and makes it clear which statuses are considered
> success.
> * We don’t create two parallel concepts ("expected" vs
> "expected_other") which introduces additional complexity
> * Can be extend later if we meet yet another harmless but not 200
> variant.
> * We don’t allow arbitrary 2xx.
>
> What do you think? Do you maybe have any other solution in mind that
> would fit better?
>
> ## Testing
>
> To prove the proposed fix, I reproduced the scenario:
>
> Pebble (release 2.8.0) from Let's Encrypt [5] running on a Debian 9 VM
> as the ACME server. nginx in front of Pebble, to intercept the
> `newNonce` request in order to return 204 No Content instead of 200 OK,
> all other requests are unchanged and forwarded to Pebble. Trust the
> Pebble and ngix CAs via `/usr/local/share/ca-certificates` +
> `update-ca-certificates` on the PBS VM.
>
> Then I ran following command against nginx:
>
> ```
> proxmox-backup-manager acme account register proxytest root@backup.local
> --directory 'https://nginx-address/dir
>
> Attempting to fetch Terms of
> Service from "https://acme-vm/dir"
> Terms of Service:
> data:text/plain,Do%20what%20thou%20wilt
> Do you agree to the above terms?
> [y|N]: y
> Do you want to use external account binding? [y|N]: N
> Attempting
> to register account with "https://acme-vm/dir"...
> Registration
> successful, account URL: https://acme-vm/my-account/160e58b66bdd72da
> ```
>
> When adjusting the nginx configuration to return any other non-expected
> success status code, e.g. 205, PBS expectely rejects with `API
> misbehaved: ACME server responded with unexpected status code: 205`.
>
> ## Maintainer notes:
>
> The patch series involves the following components:
>
> proxmox-acme: Apply PATCH 1 to change `expected` from `u16` to
> `Vec<u16>`. This results in a breaking change, as it changes the public
> API of the `AcmeRequest` type that is used by other components.
>
> proxmox-acme-api: Needs to depend on the new proxmox-acme; patch bump
>
> proxmox-backup: Apply PATCH 2 to use the new API changes; no breaking
> change as of only internal changes; patch bump
>
> proxmox-perl-rs / proxmox-datacenter-manager: Will need to use the
> dependency version bumps to follow the new proxmox-acme.
>
> ## Patch summary
>
> [PATCH 1/2] fix #6939: support providers returning 204 for nonce
> requests
>
> * Make the expected-status logic accept multiple allowed codes.
> * Treat both 200 OK and 204 No Content as valid for HEAD /newNonce,
> provided Replay-Nonce is present.
> * Keep rejecting other codes.
>
> [PATCH 2/2] acme: accept HTTP 204 from newNonce endpoint
>
> * Use the updated proxmox-acme behavior in PBS.
> * PBS can now register an ACME account against servers that return 204
> for the nonce HEAD request.
> * Still rejects unexpected codes.
>
> Thanks for considering this patch series, I look forward to your
> feedback.
>
> Best,
> Samuel Rufinatscha
>
> ## Changes from v1:
>
> [PATCH 1/2] fix #6939: support providers returning 204 for nonce
> requests
> * Introduced `http_success` module to contain the http success codes
> * Replaced `Vec<u16>` with `&[u16]` for expected codes to avoid
> allocations.
> * Clarified the PVEs Perl ACME client behaviour in the commit message.
>
> [PATCH 2/2] acme: accept HTTP 204 from newNonce endpoint
> * Integrated the `http_success` module, replacing `Vec<u16>` with `&[u16]`
> * Clarified the PVEs Perl ACME client behaviour in the commit message.
>
> [1] Bugzilla report #6939:
> [https://bugzilla.proxmox.com/show_bug.cgi?id=6939](https://bugzilla.proxmox.com/show_bug.cgi?id=6939)
> [2] RFC 8555 (ACME):
> [https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2](https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2)
> [3] PVE’s Perl ACME client (allow 2xx codes for nonce requests):
> [https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597)
> [4] Pebble ACME server:
> [https://github.com/letsencrypt/pebble](https://github.com/letsencrypt/pebble)
> [5] Pebble ACME server (perform GET request:
> [https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219)
>
> proxmox:
>
> Samuel Rufinatscha (1):
> fix #6939: acme: support servers returning 204 for nonce requests
>
> proxmox-acme/src/account.rs | 10 +++++-----
> proxmox-acme/src/async_client.rs | 6 +++---
> proxmox-acme/src/client.rs | 2 +-
> proxmox-acme/src/lib.rs | 4 ++++
> proxmox-acme/src/request.rs | 15 ++++++++++++---
> 5 files changed, 25 insertions(+), 12 deletions(-)
>
>
> proxmox-backup:
>
> Samuel Rufinatscha (1):
> fix #6939: acme: accept HTTP 204 from newNonce endpoint
>
> src/acme/client.rs | 8 ++++----
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
>
> Summary over all repositories:
> 6 files changed, 29 insertions(+), 16 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox] fix #6913: auth-api: fix user ID parsing for 2-character realms
@ 2025-11-03 16:26 15% Samuel Rufinatscha
2025-11-11 10:40 5% ` Fabian Grünbichler
2025-11-14 10:34 5% ` [pbs-devel] applied: " Fabian Grünbichler
0 siblings, 2 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-03 16:26 UTC (permalink / raw)
To: pbs-devel
PVE and PBS both allow creating realms with names of length ≥ 2.
However, when creating a user, PBS rejected realms with 2 characters
(e.g. `test@aa`), while PVE accepted them. This issue was reported
in our bug tracker [1]. Since the issue appears in the underlying
`proxmox/proxmox-auth-api` crate, also PDM userid handling is
affected.
The issue is caused by a mismatch between realm creation and parsing
rules in `proxmox/proxmox-auth-api`. `REALM_ID_SCHEMA` allows
min_length(2), but `PROXMOX_AUTH_REALM_STRING_SCHEMA` enforced
min_length(3).
This patch lowers the minimum realm length in
`PROXMOX_AUTH_REALM_STRING_SCHEMA` from 3 to 2 to align PBS and PMG
with PVE.
## Testing
Please see the attached unit tests.
The changes were further verified using a rebuilt PBS .deb
deployment. PDM was tested using a non-package binary through the
provided client CLI.
## Maintainer notes:
Bump the `proxmox-auth-api` dependency, no breaking change.
PBS and PDM to use the new dependency.
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6913
Fixes: #6913
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-auth-api/src/types.rs | 68 ++++++++++++++++++++++++++++++++++-
1 file changed, 67 insertions(+), 1 deletion(-)
diff --git a/proxmox-auth-api/src/types.rs b/proxmox-auth-api/src/types.rs
index 9bde661c..aa09fb93 100644
--- a/proxmox-auth-api/src/types.rs
+++ b/proxmox-auth-api/src/types.rs
@@ -95,7 +95,7 @@ pub const PROXMOX_GROUP_ID_SCHEMA: Schema = StringSchema::new("Group ID")
pub const PROXMOX_AUTH_REALM_STRING_SCHEMA: StringSchema =
StringSchema::new("Authentication domain ID")
.format(&proxmox_schema::api_types::SAFE_ID_FORMAT)
- .min_length(3)
+ .min_length(2)
.max_length(32);
pub const PROXMOX_AUTH_REALM_SCHEMA: Schema = PROXMOX_AUTH_REALM_STRING_SCHEMA.schema();
@@ -769,6 +769,72 @@ fn test_token_id() {
assert_eq!(auth_id.to_string(), "test@pam!bar".to_string());
}
+#[test]
+fn test_realm_validation() {
+ let empty_realm: Result<Realm, _> = "".to_string().try_into();
+ let one_char_realm: Result<Realm, _> = "a".to_string().try_into();
+ let two_char_realm: Result<Realm, _> = "aa".to_string().try_into();
+ let long_realm: Result<Realm, _> = "a".repeat(33).try_into();
+ let valid_realm: Result<Realm, _> = "pam".to_string().try_into();
+
+ assert!(empty_realm.is_err(), "Empty realm should fail validation");
+ assert!(
+ one_char_realm.is_err(),
+ "1-char realm should fail validation"
+ );
+ assert!(
+ two_char_realm.is_ok(),
+ "2-char realm should pass validation"
+ );
+ assert!(valid_realm.is_ok(), "Typical realm should pass validation");
+ assert!(
+ long_realm.is_err(),
+ "Realm >32 chars should fail validation"
+ );
+}
+
+#[test]
+fn test_userid_validation() {
+ let empty_str: Result<Userid, _> = "".parse();
+ let invalid_no_realm: Result<Userid, _> = "user".parse();
+ let invalid_empty_realm: Result<Userid, _> = "user@".parse();
+ let invalid_one_char_realm: Result<Userid, _> = "user@a".parse();
+ let valid_two_char_realm: Result<Userid, _> = "user@aa".parse();
+ let valid_long_realm: Result<Userid, _> = "user@pam".parse();
+ let invalid_long_realm: Result<Userid, _> = format!("user@{}", "a".repeat(33)).parse();
+ let invalid_empty_username: Result<Userid, _> = "@aa".parse();
+
+ assert!(empty_str.is_err(), "Empty userid should fail");
+ assert!(
+ invalid_no_realm.is_err(),
+ "Userid without realm should fail"
+ );
+ assert!(
+ invalid_empty_realm.is_err(),
+ "Userid with empty realm should fail"
+ );
+ assert!(
+ invalid_one_char_realm.is_err(),
+ "Userid with 1-char realm should fail"
+ );
+ assert!(
+ valid_two_char_realm.is_ok(),
+ "Userid with 2-char realm should pass"
+ );
+ assert!(
+ valid_long_realm.is_ok(),
+ "Userid with normal realm should pass"
+ );
+ assert!(
+ invalid_long_realm.is_err(),
+ "Userid with realm >32 chars should fail"
+ );
+ assert!(
+ invalid_empty_username.is_err(),
+ "Userid with empty username should fail"
+ );
+}
+
serde_plain::derive_deserialize_from_fromstr!(Userid, "valid user id");
serde_plain::derive_serialize_from_display!(Userid);
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* Re: [pbs-devel] [PATCH proxmox v3 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-11-03 10:13 13% ` [pbs-devel] [PATCH proxmox v3 1/1] " Samuel Rufinatscha
@ 2025-11-04 14:11 4% ` Thomas Lamprecht
2025-11-05 10:22 9% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Thomas Lamprecht @ 2025-11-04 14:11 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Samuel Rufinatscha
Am 03.11.25 um 11:13 schrieb Samuel Rufinatscha:
> Some ACME servers (notably custom or legacy implementations) respond
> to HEAD /newNonce with a 204 No Content instead of the
> RFC 8555-recommended 200 OK [1]. While this behavior is technically
> off-spec, it is not illegal. This issue was reported on our bug
> tracker [2].
>
> The previous implementation treated any non-200 response as an error,
> causing account registration to fail against such servers. Relax the
> status-code check to accept both 200 and 204 responses (and potentially
> support other 2xx codes) to improve interoperability.
>
> Note: In comparison, PVE’s Perl ACME client performs a GET request [3]
> instead of a HEAD request and accepts any 2xx success code when
> retrieving the nonce [4]. This difference in behavior does not affect
> functionality but is worth noting for consistency across
> implementations.
The resulting code now looks mostly OK, but when doing a final review
before pushing this out I noticed two things that I should have actually
caught earlier:
1. Introducing + using the new http_status module should be a separate
patch upfront, it has nothing to do with the actual bug fix but is
rather an independent cleanup.
2. The Request struct and it's expected member are pub and is still
used directly in PBS - where it was originally factored out from
but not yet replaced. So the type change from `u16` to `&'static [u16]`
causes a breaking ABI change. That change itself can be manageable,
albeit if it can be easily avoided it's always nicer to do so; using
a constructor (potentially with builder pattern) and changing the
visibility of the members to pub(crate) or even making them private
to the module, can often be a good option to reduce friction for
any future change.
But here it would be nicer to clear the tech debt and switch PBS fully
over to the factored out impl, like e.g. PDM uses already. This should
then also allow us to reduce the visibility of the struct members and
the http_status module, which as of now I'd also rather see as
proxmox-acme specific and thus not exposing it to the public would be
better.
Do you want to give switching PBS over to the factored-out impl a try?
It adds a bit to the scope, but we have to clean this up (or accumulate
tech debt interest) sooner or later anyway, and if we do already a breaking
change I'd prefer sooner.
For the acme side of your changes nothing should change – besides maybe
already reducing visibility of structs and/or their members in a separate
patch.
In summary, a good order/split for the resulting patches could look
something like:
1. change PBS over to proxmox-acme, at least the client part.
2. reduce the visibility of types that now are only used in proxmox-acme
internally
3. introduce and use http_status mod in proxmox-acme
4. fix #6939
What do you think? If moving PBS to proxmox-acme turns out to be tricky,
or even more work than expected, we can also go this route here as stop
gap; but IMO it would be favorable to spend a bit more time in actually
cleaning this up and reduce code duplication than doing so.
Again, I should have caught that early, but FWIW, almost all of the work
you done so far will still be relevant, so no real harm done FWICT.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 4%]
* Re: [pbs-devel] [PATCH proxmox v3 1/1] fix #6939: acme: support servers returning 204 for nonce requests
2025-11-04 14:11 4% ` Thomas Lamprecht
@ 2025-11-05 10:22 9% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-05 10:22 UTC (permalink / raw)
To: Thomas Lamprecht, Proxmox Backup Server development discussion
On 11/4/25 3:11 PM, Thomas Lamprecht wrote:
> Am 03.11.25 um 11:13 schrieb Samuel Rufinatscha:
>> Some ACME servers (notably custom or legacy implementations) respond
>> to HEAD /newNonce with a 204 No Content instead of the
>> RFC 8555-recommended 200 OK [1]. While this behavior is technically
>> off-spec, it is not illegal. This issue was reported on our bug
>> tracker [2].
>>
>> The previous implementation treated any non-200 response as an error,
>> causing account registration to fail against such servers. Relax the
>> status-code check to accept both 200 and 204 responses (and potentially
>> support other 2xx codes) to improve interoperability.
>>
>> Note: In comparison, PVE’s Perl ACME client performs a GET request [3]
>> instead of a HEAD request and accepts any 2xx success code when
>> retrieving the nonce [4]. This difference in behavior does not affect
>> functionality but is worth noting for consistency across
>> implementations.
>
> The resulting code now looks mostly OK, but when doing a final review
> before pushing this out I noticed two things that I should have actually
> caught earlier:
>
> 1. Introducing + using the new http_status module should be a separate
> patch upfront, it has nothing to do with the actual bug fix but is
> rather an independent cleanup.
>
> 2. The Request struct and it's expected member are pub and is still
> used directly in PBS - where it was originally factored out from
> but not yet replaced. So the type change from `u16` to `&'static [u16]`
> causes a breaking ABI change. That change itself can be manageable,
> albeit if it can be easily avoided it's always nicer to do so; using
> a constructor (potentially with builder pattern) and changing the
> visibility of the members to pub(crate) or even making them private
> to the module, can often be a good option to reduce friction for
> any future change.
> But here it would be nicer to clear the tech debt and switch PBS fully
> over to the factored out impl, like e.g. PDM uses already. This should
> then also allow us to reduce the visibility of the struct members and
> the http_status module, which as of now I'd also rather see as
> proxmox-acme specific and thus not exposing it to the public would be
> better.
>
> Do you want to give switching PBS over to the factored-out impl a try?
> It adds a bit to the scope, but we have to clean this up (or accumulate
> tech debt interest) sooner or later anyway, and if we do already a breaking
> change I'd prefer sooner.
> For the acme side of your changes nothing should change – besides maybe
> already reducing visibility of structs and/or their members in a separate
> patch.
>
> In summary, a good order/split for the resulting patches could look
> something like:
>
> 1. change PBS over to proxmox-acme, at least the client part.
> 2. reduce the visibility of types that now are only used in proxmox-acme
> internally
> 3. introduce and use http_status mod in proxmox-acme
> 4. fix #6939
>
> What do you think? If moving PBS to proxmox-acme turns out to be tricky,
> or even more work than expected, we can also go this route here as stop
> gap; but IMO it would be favorable to spend a bit more time in actually
> cleaning this up and reduce code duplication than doing so.
> Again, I should have caught that early, but FWIW, almost all of the work
> you done so far will still be relevant, so no real harm done FWICT.
Hi Thomas,
thanks a lot for having a second look and the detailed feedback -
sounds good to me! I agree that it would be better to split off the
fix-independent changes, and it would be a good opportunity now to
switch PBS over to the factored-out proxmox-acme implementation (get
at least rid of the duplicate client impl), and follow PDM. I think
this is manageable and will start by switching PBS to the shared
client, adjust visibility and add the http_status module before
applying the fix.
I’ll go ahead with the refactoring and will send a v4 with the
suggested patch structure.
Samuel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 9%]
* Re: [pbs-devel] [PATCH proxmox] fix #6913: auth-api: fix user ID parsing for 2-character realms
2025-11-03 16:26 15% [pbs-devel] [PATCH proxmox] fix #6913: auth-api: fix user ID parsing for 2-character realms Samuel Rufinatscha
@ 2025-11-11 10:40 5% ` Fabian Grünbichler
2025-11-11 13:49 6% ` Samuel Rufinatscha
2025-11-14 10:34 5% ` [pbs-devel] applied: " Fabian Grünbichler
1 sibling, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-11-11 10:40 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On November 3, 2025 5:26 pm, Samuel Rufinatscha wrote:
> PVE and PBS both allow creating realms with names of length ≥ 2.
> However, when creating a user, PBS rejected realms with 2 characters
> (e.g. `test@aa`), while PVE accepted them. This issue was reported
> in our bug tracker [1]. Since the issue appears in the underlying
> `proxmox/proxmox-auth-api` crate, also PDM userid handling is
> affected.
>
> The issue is caused by a mismatch between realm creation and parsing
> rules in `proxmox/proxmox-auth-api`. `REALM_ID_SCHEMA` allows
> min_length(2), but `PROXMOX_AUTH_REALM_STRING_SCHEMA` enforced
> min_length(3).
>
> This patch lowers the minimum realm length in
> `PROXMOX_AUTH_REALM_STRING_SCHEMA` from 3 to 2 to align PBS and PMG
> with PVE.
>
> ## Testing
>
> Please see the attached unit tests.
> The changes were further verified using a rebuilt PBS .deb
> deployment. PDM was tested using a non-package binary through the
> provided client CLI.
>
> ## Maintainer notes:
>
> Bump the `proxmox-auth-api` dependency, no breaking change.
> PBS and PDM to use the new dependency.
this part here we'd usually put into the patch notes (the part below the
`---`), which doesn't show up in git history. you can manage those notes
using `git notes ..`, including (if you set your config accordingly),
preserving/merging them across rebases.
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
>
> [1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6913
>
> Fixes: #6913
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> proxmox-auth-api/src/types.rs | 68 ++++++++++++++++++++++++++++++++++-
> 1 file changed, 67 insertions(+), 1 deletion(-)
>
> diff --git a/proxmox-auth-api/src/types.rs b/proxmox-auth-api/src/types.rs
> index 9bde661c..aa09fb93 100644
> --- a/proxmox-auth-api/src/types.rs
> +++ b/proxmox-auth-api/src/types.rs
> @@ -95,7 +95,7 @@ pub const PROXMOX_GROUP_ID_SCHEMA: Schema = StringSchema::new("Group ID")
> pub const PROXMOX_AUTH_REALM_STRING_SCHEMA: StringSchema =
> StringSchema::new("Authentication domain ID")
> .format(&proxmox_schema::api_types::SAFE_ID_FORMAT)
> - .min_length(3)
> + .min_length(2)
> .max_length(32);
> pub const PROXMOX_AUTH_REALM_SCHEMA: Schema = PROXMOX_AUTH_REALM_STRING_SCHEMA.schema();
>
> @@ -769,6 +769,72 @@ fn test_token_id() {
> assert_eq!(auth_id.to_string(), "test@pam!bar".to_string());
> }
>
> +#[test]
> +fn test_realm_validation() {
> + let empty_realm: Result<Realm, _> = "".to_string().try_into();
> + let one_char_realm: Result<Realm, _> = "a".to_string().try_into();
> + let two_char_realm: Result<Realm, _> = "aa".to_string().try_into();
> + let long_realm: Result<Realm, _> = "a".repeat(33).try_into();
> + let valid_realm: Result<Realm, _> = "pam".to_string().try_into();
> +
> + assert!(empty_realm.is_err(), "Empty realm should fail validation");
> + assert!(
> + one_char_realm.is_err(),
> + "1-char realm should fail validation"
> + );
> + assert!(
> + two_char_realm.is_ok(),
> + "2-char realm should pass validation"
> + );
> + assert!(valid_realm.is_ok(), "Typical realm should pass validation");
> + assert!(
> + long_realm.is_err(),
> + "Realm >32 chars should fail validation"
> + );
> +}
> +
> +#[test]
> +fn test_userid_validation() {
> + let empty_str: Result<Userid, _> = "".parse();
> + let invalid_no_realm: Result<Userid, _> = "user".parse();
> + let invalid_empty_realm: Result<Userid, _> = "user@".parse();
> + let invalid_one_char_realm: Result<Userid, _> = "user@a".parse();
> + let valid_two_char_realm: Result<Userid, _> = "user@aa".parse();
> + let valid_long_realm: Result<Userid, _> = "user@pam".parse();
> + let invalid_long_realm: Result<Userid, _> = format!("user@{}", "a".repeat(33)).parse();
> + let invalid_empty_username: Result<Userid, _> = "@aa".parse();
> +
> + assert!(empty_str.is_err(), "Empty userid should fail");
> + assert!(
> + invalid_no_realm.is_err(),
> + "Userid without realm should fail"
> + );
> + assert!(
> + invalid_empty_realm.is_err(),
> + "Userid with empty realm should fail"
> + );
> + assert!(
> + invalid_one_char_realm.is_err(),
> + "Userid with 1-char realm should fail"
> + );
> + assert!(
> + valid_two_char_realm.is_ok(),
> + "Userid with 2-char realm should pass"
> + );
> + assert!(
> + valid_long_realm.is_ok(),
> + "Userid with normal realm should pass"
> + );
> + assert!(
> + invalid_long_realm.is_err(),
> + "Userid with realm >32 chars should fail"
> + );
> + assert!(
> + invalid_empty_username.is_err(),
> + "Userid with empty username should fail"
> + );
> +}
these two are more or less tests validating our schema deserializer, but
as the types are rather core types they also don't hurt.
AFAICT we don't have in-depth tests in proxmox-schema that verify that
the schema constraints validation actually works as expected, there's
just some basic tests for query parameter handling and schema types
themselves - might be an area worth improving ;)
> +
> serde_plain::derive_deserialize_from_fromstr!(Userid, "valid user id");
> serde_plain::derive_serialize_from_display!(Userid);
>
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* [pbs-devel] [PATCH proxmox-backup 2/3] partial fix #6049: datastore: use config fast-path in Drop
2025-11-11 12:29 11% [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path Samuel Rufinatscha
2025-11-11 12:29 12% ` [pbs-devel] [PATCH proxmox-backup 1/3] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
@ 2025-11-11 12:29 15% ` Samuel Rufinatscha
2025-11-12 11:24 5% ` Fabian Grünbichler
2025-11-11 12:29 15% ` [pbs-devel] [PATCH proxmox-backup 3/3] datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
` (2 subsequent siblings)
4 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-11-11 12:29 UTC (permalink / raw)
To: pbs-devel
The Drop impl of DataStore re-read datastore.cfg to decide whether
the entry should be evicted from the in-process cache (based on
maintenance mode’s clear_from_cache). During the investigation of
issue #6049 [1], a flamegraph [2] showed that the config reload in Drop
accounted for a measurable share of CPU time under load.
This patch makes Drop O(1) on the fast path by reusing the maintenance-
mode decision captured at lookup time and stored with the cached
datastore entry. When the last reference goes away we:
- decrement active-operation counters, and
- evict only if the cached decision mandates eviction.
If the cache tag is absent or not fresh, a subsequent slow-path lookup
will be performed.
Testing
Compared flamegraphs before and after: prior to this change
(on top of patch 1), stacks originating from Drop included
pbs_config::datastore::config(). After the change, those vanish from
the drop path.
An end-to-end benchmark using `/status?verbose=0` with 1000 datastores,
5 requests per store, and 16-way parallelism shows a further
improvement:
| Metric | After commit 1 | After commit 2 | Δ (abs) | Δ (%) |
|-------------------------|:--------------:|:--------------:|:-------:|:-------:|
| Total time | 11s | 10s | −1s | −9.09% |
| Throughput (all rounds) | 454.55 | 500.00 | +45.45 | +10.00% |
| Cold RPS (round #1) | 90.91 | 100.00 | +9.09 | +10.00% |
| Warm RPS (rounds 2..N) | 363.64 | 400.00 | +36.36 | +10.00% |
Optimizing Drop improves overall throughput by ~10%. The gain appears
in both cold and warm rounds, and the flamegraph confirms the config
reload no longer sits on the hot path.
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-datastore/src/datastore.rs | 31 +++++++++++++++++++++++++++----
1 file changed, 27 insertions(+), 4 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 18eebb58..da80416a 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -200,15 +200,38 @@ impl Drop for DataStore {
// remove datastore from cache iff
// - last task finished, and
// - datastore is in a maintenance mode that mandates it
- let remove_from_cache = last_task
- && pbs_config::datastore::config()
+
+ // first check: check if last task finished
+ if !last_task {
+ return;
+ }
+
+ let cached_tag = self.inner.cached_config_tag.as_ref();
+ let last_gen_num = cached_tag.and_then(|c| c.last_generation);
+ let gen_num = ConfigVersionCache::new()
+ .ok()
+ .map(|c| c.datastore_generation());
+
+ let cache_is_fresh = match (last_gen_num, gen_num) {
+ (Some(a), Some(b)) => a == b,
+ _ => false,
+ };
+
+ let mm_mandate = if cache_is_fresh {
+ cached_tag
+ .and_then(|c| c.last_maintenance_mode.as_ref())
+ .is_some_and(|m| m.clear_from_cache())
+ } else {
+ pbs_config::datastore::config()
.and_then(|(s, _)| s.lookup::<DataStoreConfig>("datastore", self.name()))
.is_ok_and(|c| {
c.get_maintenance_mode()
.is_some_and(|m| m.clear_from_cache())
- });
+ })
+ };
- if remove_from_cache {
+ // second check: check maintenance mode mandate
+ if mm_mandate {
DATASTORE_MAP.lock().unwrap().remove(self.name());
}
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox-backup 3/3] datastore: add TTL fallback to catch manual config edits
2025-11-11 12:29 11% [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path Samuel Rufinatscha
2025-11-11 12:29 12% ` [pbs-devel] [PATCH proxmox-backup 1/3] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
2025-11-11 12:29 15% ` [pbs-devel] [PATCH proxmox-backup 2/3] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
@ 2025-11-11 12:29 15% ` Samuel Rufinatscha
2025-11-12 11:27 5% ` [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path Fabian Grünbichler
2025-11-14 15:08 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-11 12:29 UTC (permalink / raw)
To: pbs-devel
The lookup fast path reacts to API-driven config changes because
save_config() bumps the generation. Manual edits of datastore.cfg do
not bump the counter. To keep the system robust against such edits
without reintroducing config reading and hashing on the hot path, this
patch adds a TTL to the cache entry.
If the datastore’s cached tag is older than
DATASTORE_CONFIG_CACHE_TTL_SECS (set to 60s), the next lookup takes
the slow path (re-read/parse) and refreshes the cached entry. Within
the TTL window, unchanged generations still use the fast path.
Note: Manual edits may remain unseen until the TTL elapses or any API
config write occurs.
Testing
With the TTL enabled, flamegraphs for hot status requests remain flat. A
0.1 second interval test confirmed periodic latency spikes at TTL expiry.
Maintainer notes
No dependency bumps or breaking changes.
Links
[1] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Refs: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-datastore/src/datastore.rs | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index da80416a..5eaae49b 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -22,7 +22,7 @@ use proxmox_sys::error::SysError;
use proxmox_sys::fs::{file_read_optional_string, replace_file, CreateOptions};
use proxmox_sys::linux::procfs::MountInfo;
use proxmox_sys::process_locker::{ProcessLockExclusiveGuard, ProcessLockSharedGuard};
-use proxmox_time::TimeSpan;
+use proxmox_time::{epoch_i64, TimeSpan};
use proxmox_worker_task::WorkerTaskContext;
use pbs_api_types::{
@@ -56,6 +56,9 @@ pub const GROUP_OWNER_FILE_NAME: &str = "owner";
/// Filename for in-use marker stored on S3 object store backend
pub const S3_DATASTORE_IN_USE_MARKER: &str = ".in-use";
const NAMESPACE_MARKER_FILENAME: &str = ".namespace";
+/// Max age in seconds to reuse the datastore lookup fast path
+/// before forcing a slow-path config read.
+const DATASTORE_CONFIG_CACHE_TTL_SECS: i64 = 60;
/// checks if auth_id is owner, or, if owner is a token, if
/// auth_id is the user of the token
@@ -254,6 +257,8 @@ struct CachedDatastoreConfigTag {
last_maintenance_mode: Option<MaintenanceMode>,
/// Datastore generation number from `ConfigVersionCache`; `None` when the cache wasn't available.
last_generation: Option<usize>,
+ /// Epoch seconds when this lookup hint was created.
+ last_update: i64,
}
impl DataStore {
@@ -335,13 +340,16 @@ impl DataStore {
let gen_num = ConfigVersionCache::new()
.ok()
.map(|c| c.datastore_generation());
+ let now = epoch_i64();
// Fast-path: if we have a cached entry created under the same datastore.cfg generation number, reuse it.
if let (Some(gen_num), Some(ds)) =
(gen_num, DATASTORE_MAP.lock().unwrap().get(name).cloned())
{
if let Some(cached_tag) = &ds.cached_config_tag {
- if cached_tag.last_generation == Some(gen_num) {
+ if cached_tag.last_generation == Some(gen_num)
+ && (now - cached_tag.last_update) < DATASTORE_CONFIG_CACHE_TTL_SECS
+ {
if let Some(mm) = &cached_tag.last_maintenance_mode {
if let Err(error) = mm.check(operation) {
bail!("datastore '{name}' is unavailable: {error}");
@@ -397,6 +405,7 @@ impl DataStore {
datastore.cached_config_tag = Some(CachedDatastoreConfigTag {
last_maintenance_mode: maintenance_mode,
last_generation: gen_num,
+ last_update: now,
});
let datastore = Arc::new(datastore);
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox-backup 1/3] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
2025-11-11 12:29 11% [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path Samuel Rufinatscha
@ 2025-11-11 12:29 12% ` Samuel Rufinatscha
2025-11-12 13:24 4% ` Fabian Grünbichler
2025-11-11 12:29 15% ` [pbs-devel] [PATCH proxmox-backup 2/3] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
` (3 subsequent siblings)
4 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-11-11 12:29 UTC (permalink / raw)
To: pbs-devel
Repeated /status requests caused lookup_datastore() to re-read and
parse datastore.cfg on every call. The issue was mentioned in report
#6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
dominated by pbs_config::datastore::config() (config parsing).
This patch adds a fast path to lookup_datastore() using the shared-
memory ConfigVersionCache generation counter for datastore.cfg. It
stores the last seen generation number alongside the cached
DataStoreImpl and, when a subsequent lookup sees the same generation,
we reuse the cached instance without re-reading the on-disk config. If
the generation differs (or the cache is unavailable), it falls back to
the existing slow path with no behavioral changes.
Behavioral notes
- The generation is bumped via the existing save_config() path, so
API-driven config changes are detected immediately.
- Manual edits to datastore.cfg are not detected by this commit; a TTL
guard is introduced in a later patch to cover that case.
- DataStore::drop still performs a config read on the common path; this
is addressed in the next patch.
Testing
cargo-flamegraph confirms the config-parse hotspot in
lookup_datastore() disappears from the hot path.
Additionally, an end-to-end benchmark was performed using the
`/status?verbose=0` API with 1000 datastores, 5 requests per store,
and 16-way parallelism:
| Metric | Before | After | Δ (abs) | Δ (%) |
|--------------------------|:------:|:------:|:-------:|:-------:|
| Total time | 13s | 11s | −2s | −15.38% |
| Throughput (all rounds) | 384.62 | 454.55 | +69.93 | +18.18% |
| Cold RPS (round #1) | 76.92 | 90.91 | +13.99 | +18.19% |
| Warm RPS (rounds 2..N) | 307.69 | 363.64 | +55.95 | +18.18% |
Throughput improved by ~18% overall, with total time reduced by ~15%.
Warm lookups now reuse cached datastore instances and skip redundant
config parsing entirely. The first-access round also shows a similar
improvement, likely due to reduced contention and I/O when many stores
are accessed in parallel.
Note: A second hotspot remains in Drop where a config read occurs; that
is addressed in the next commit.
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-config/src/config_version_cache.rs | 10 +++-
pbs-datastore/src/datastore.rs | 77 +++++++++++++++++---------
2 files changed, 59 insertions(+), 28 deletions(-)
diff --git a/pbs-config/src/config_version_cache.rs b/pbs-config/src/config_version_cache.rs
index e8fb994f..b875f7e0 100644
--- a/pbs-config/src/config_version_cache.rs
+++ b/pbs-config/src/config_version_cache.rs
@@ -26,7 +26,6 @@ struct ConfigVersionCacheDataInner {
// Traffic control (traffic-control.cfg) generation/version.
traffic_control_generation: AtomicUsize,
// datastore (datastore.cfg) generation/version
- // FIXME: remove with PBS 3.0
datastore_generation: AtomicUsize,
// Add further atomics here
}
@@ -145,8 +144,15 @@ impl ConfigVersionCache {
.fetch_add(1, Ordering::AcqRel);
}
+ /// Returns the datastore generation number.
+ pub fn datastore_generation(&self) -> usize {
+ self.shmem
+ .data()
+ .datastore_generation
+ .load(Ordering::Acquire)
+ }
+
/// Increase the datastore generation number.
- // FIXME: remove with PBS 3.0 or make actually useful again in datastore lookup
pub fn increase_datastore_generation(&self) -> usize {
self.shmem
.data()
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 70af94d8..18eebb58 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -32,7 +32,7 @@ use pbs_api_types::{
MaintenanceType, Operation, UPID,
};
use pbs_config::s3::S3_CFG_TYPE_ID;
-use pbs_config::BackupLockGuard;
+use pbs_config::{BackupLockGuard, ConfigVersionCache};
use crate::backup_info::{
BackupDir, BackupGroup, BackupInfo, OLD_LOCKING, PROTECTED_MARKER_FILENAME,
@@ -140,10 +140,10 @@ pub struct DataStoreImpl {
last_gc_status: Mutex<GarbageCollectionStatus>,
verify_new: bool,
chunk_order: ChunkOrder,
- last_digest: Option<[u8; 32]>,
sync_level: DatastoreFSyncLevel,
backend_config: DatastoreBackendConfig,
lru_store_caching: Option<LocalDatastoreLruCache>,
+ cached_config_tag: Option<CachedDatastoreConfigTag>,
}
impl DataStoreImpl {
@@ -156,10 +156,10 @@ impl DataStoreImpl {
last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
verify_new: false,
chunk_order: Default::default(),
- last_digest: None,
sync_level: Default::default(),
backend_config: Default::default(),
lru_store_caching: None,
+ cached_config_tag: None,
})
}
}
@@ -224,6 +224,15 @@ pub enum DatastoreBackend {
S3(Arc<S3Client>),
}
+/// Used to determine whether a cached datastore instance is still valid
+/// or needs to be reloaded after a config change.
+struct CachedDatastoreConfigTag {
+ /// Maintenance mode at the time the lookup hint was captured, if any.
+ last_maintenance_mode: Option<MaintenanceMode>,
+ /// Datastore generation number from `ConfigVersionCache`; `None` when the cache wasn't available.
+ last_generation: Option<usize>,
+}
+
impl DataStore {
// This one just panics on everything
#[doc(hidden)]
@@ -299,13 +308,40 @@ impl DataStore {
// we use it to decide whether it is okay to delete the datastore.
let _config_lock = pbs_config::datastore::lock_config()?;
- // we could use the ConfigVersionCache's generation for staleness detection, but we load
- // the config anyway -> just use digest, additional benefit: manual changes get detected
- let (config, digest) = pbs_config::datastore::config()?;
+ // Get the current datastore.cfg generation number
+ let gen_num = ConfigVersionCache::new()
+ .ok()
+ .map(|c| c.datastore_generation());
+
+ // Fast-path: if we have a cached entry created under the same datastore.cfg generation number, reuse it.
+ if let (Some(gen_num), Some(ds)) =
+ (gen_num, DATASTORE_MAP.lock().unwrap().get(name).cloned())
+ {
+ if let Some(cached_tag) = &ds.cached_config_tag {
+ if cached_tag.last_generation == Some(gen_num) {
+ if let Some(mm) = &cached_tag.last_maintenance_mode {
+ if let Err(error) = mm.check(operation) {
+ bail!("datastore '{name}' is unavailable: {error}");
+ }
+ }
+ if let Some(operation) = operation {
+ update_active_operations(name, operation, 1)?;
+ }
+ return Ok(Arc::new(Self {
+ inner: ds,
+ operation,
+ }));
+ }
+ }
+ }
+
+ // Slow path: (re)load config
+ let (config, _digest) = pbs_config::datastore::config()?;
let config: DataStoreConfig = config.lookup("datastore", name)?;
- if let Some(maintenance_mode) = config.get_maintenance_mode() {
- if let Err(error) = maintenance_mode.check(operation) {
+ let maintenance_mode = config.get_maintenance_mode();
+ if let Some(mm) = &maintenance_mode {
+ if let Err(error) = mm.check(operation) {
bail!("datastore '{name}' is unavailable: {error}");
}
}
@@ -321,16 +357,6 @@ impl DataStore {
// reuse chunk store so that we keep using the same process locker instance!
let chunk_store = if let Some(datastore) = &entry {
- let last_digest = datastore.last_digest.as_ref();
- if let Some(true) = last_digest.map(|last_digest| last_digest == &digest) {
- if let Some(operation) = operation {
- update_active_operations(name, operation, 1)?;
- }
- return Ok(Arc::new(Self {
- inner: Arc::clone(datastore),
- operation,
- }));
- }
Arc::clone(&datastore.chunk_store)
} else {
let tuning: DatastoreTuning = serde_json::from_value(
@@ -344,7 +370,11 @@ impl DataStore {
)?)
};
- let datastore = DataStore::with_store_and_config(chunk_store, config, Some(digest))?;
+ let mut datastore = DataStore::with_store_and_config(chunk_store, config)?;
+ datastore.cached_config_tag = Some(CachedDatastoreConfigTag {
+ last_maintenance_mode: maintenance_mode,
+ last_generation: gen_num,
+ });
let datastore = Arc::new(datastore);
datastore_cache.insert(name.to_string(), datastore.clone());
@@ -430,11 +460,7 @@ impl DataStore {
config.absolute_path(),
tuning.sync_level.unwrap_or_default(),
)?;
- let inner = Arc::new(Self::with_store_and_config(
- Arc::new(chunk_store),
- config,
- None,
- )?);
+ let inner = Arc::new(Self::with_store_and_config(Arc::new(chunk_store), config)?);
if let Some(operation) = operation {
update_active_operations(&name, operation, 1)?;
@@ -446,7 +472,6 @@ impl DataStore {
fn with_store_and_config(
chunk_store: Arc<ChunkStore>,
config: DataStoreConfig,
- last_digest: Option<[u8; 32]>,
) -> Result<DataStoreImpl, Error> {
let mut gc_status_path = chunk_store.base_path();
gc_status_path.push(".gc-status");
@@ -506,10 +531,10 @@ impl DataStore {
last_gc_status: Mutex::new(gc_status),
verify_new: config.verify_new.unwrap_or(false),
chunk_order: tuning.chunk_order.unwrap_or_default(),
- last_digest,
sync_level: tuning.sync_level.unwrap_or_default(),
backend_config,
lru_store_caching,
+ cached_config_tag: None,
})
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path
@ 2025-11-11 12:29 11% Samuel Rufinatscha
2025-11-11 12:29 12% ` [pbs-devel] [PATCH proxmox-backup 1/3] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
` (4 more replies)
0 siblings, 5 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-11 12:29 UTC (permalink / raw)
To: pbs-devel
Hi,
this series reduces CPU time in datastore lookups by avoiding repeated
datastore.cfg reads/parses in both `lookup_datastore()` and
`DataStore::Drop`. It also adds a TTL so manual config edits are
noticed without reintroducing hashing on every request.
While investigating #6049 [1], cargo-flamegraph [2] showed hotspots during
repeated `/status` calls in `lookup_datastore()` and in `Drop`,
dominated by `pbs_config::datastore::config()` (config parse).
The parsing cost itself should eventually be investigated in a future
effort. Furthermore, cargo-flamegraph showed that when using a
token-based auth method to access the API, a significant amount of time
is spent in validation on every request, likely related to bcrypt.
Also this should be eventually revisited in a future effort.
## Approach
[PATCH 1/3] Fast path for datastore lookups
Use the shared-memory `ConfigVersionCache` generation for `datastore.cfg`.
Tag each cached `DataStoreImpl` with the last seen generation; when it
matches, reuse the cached instance. Fall back to the existing slow path
on mismatch or when the cache is unavailable.
[PATCH 2/3] Fast path for `Drop`
Reuse the maintenance mode eviction decision captured at lookup time,
removing the config reload from `Drop`.
[PATCH 3/3] TTL to catch manual edits
If a cached entry is older than `DATASTORE_CONFIG_CACHE_TTL_SECS`
(default 60s), the next lookup refreshes it via the slow path. This
detects manual file edits without hashing on every request.
## Results
End-to-end `/status?verbose=0` (1000 stores, 5 req/store, parallel=16):
Metric Baseline [1/3] [2/3]
------------------------------------------------
Total time 13s 11s 10s
Throughput (all) 384.62 454.55 500.00
Cold RPS (round #1) 76.92 90.91 100.00
Warm RPS (2..N) 307.69 363.64 400.00
Patch 1 improves overall throughput by ~18% (−15% total time). Patch 2
adds ~10% on top. Patch 3 is a robustness feature; a 0.1 s probe shows
periodic latency spikes at TTL expiry and flat latencies otherwise.
## Reproduction steps
VM: 4 vCPU, ~8 GiB RAM, VirtIO-SCSI; disks:
- scsi0 32G (OS)
- scsi1 1000G (datastores)
Install PBS from ISO on the VM.
Set up ZFS on /dev/sdb (adjust if different):
zpool create -f -o ashift=12 pbsbench /dev/sdb
zfs set mountpoint=/pbsbench pbsbench
zfs create pbsbench/pbs-bench
Raise file-descriptor limit:
sudo systemctl edit proxmox-backup-proxy.service
Add the following lines:
[Service]
LimitNOFILE=1048576
Reload systemd and restart the proxy:
sudo systemctl daemon-reload
sudo systemctl restart proxmox-backup-proxy.service
Verify the limit:
systemctl show proxmox-backup-proxy.service | grep LimitNOFILE
Create 1000 ZFS-backed datastores (as used in #6049 [1]):
seq -w 001 1000 | xargs -n1 -P1 bash -c '
id=$0
name="ds${id}"
dataset="pbsbench/pbs-bench/${name}"
path="/pbsbench/pbs-bench/${name}"
zfs create -o mountpoint="$path" "$dataset"
proxmox-backup-manager datastore create "$name" "$path" \
--comment "ZFS dataset-based datastore"
'
Build PBS from this series, then run the server under manually
under flamegraph:
systemctl stop proxmox-backup-proxy
cargo flamegraph --release --bin proxmox-backup-proxy
Benchmark script (`bench.sh`) used for the numbers above:
#!/usr/bin/env bash
set -euo pipefail
# --- Config ---------------------------------------------------------------
HOST='https://localhost:8007'
USER='root@pam'
PASS="$(cat passfile)"
DATASTORE_PATH="/pbsbench/pbs-bench"
MAX_STORES=1000 # how many stores to include
PARALLEL=16 # concurrent workers
REPEAT=5 # requests per store (1 cold + REPEAT-1 warm)
PRINT_FIRST=false # true => log first request's HTTP code per store
# --- Helpers --------------------------------------------------------------
fmt_rps () {
local n="$1" t="$2"
awk -v n="$n" -v t="$t" 'BEGIN { if (t > 0) printf("%.2f\n", n/t); else print "0.00" }'
}
# --- Login ---------------------------------------------------------------
auth=$(curl -ks -X POST "$HOST/api2/json/access/ticket" \
-d "username=$USER" -d "password=$PASS")
ticket=$(echo "$auth" | jq -r '.data.ticket')
if [[ -z "${ticket:-}" || "$ticket" == "null" ]]; then
echo "[ERROR] Login failed (no ticket)"
exit 1
fi
# --- Collect stores (deterministic order) --------------------------------
mapfile -t STORES < <(
find "$DATASTORE_PATH" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' \
| sort | head -n "$MAX_STORES"
)
USED_STORES=${#STORES[@]}
if (( USED_STORES == 0 )); then
echo "[ERROR] No datastore dirs under $DATASTORE_PATH"
exit 1
fi
echo "[INFO] Running with stores=$USED_STORES, repeat=$REPEAT, parallel=$PARALLEL"
# --- Temp counters --------------------------------------------------------
SUCCESS_ALL="$(mktemp)"
FAIL_ALL="$(mktemp)"
COLD_OK="$(mktemp)"
WARM_OK="$(mktemp)"
trap 'rm -f "$SUCCESS_ALL" "$FAIL_ALL" "$COLD_OK" "$WARM_OK"' EXIT
export HOST ticket REPEAT SUCCESS_ALL FAIL_ALL COLD_OK WARM_OK PRINT_FIRST
SECONDS=0
# --- Fire requests --------------------------------------------------------
printf "%s\n" "${STORES[@]}" \
| xargs -P"$PARALLEL" -I{} bash -c '
store="$1"
url="$HOST/api2/json/admin/datastore/$store/status?verbose=0"
for ((i=1;i<=REPEAT;i++)); do
code=$(curl -ks -o /dev/null -w "%{http_code}" -b "PBSAuthCookie=$ticket" "$url" || echo 000)
if [[ "$code" == "200" ]]; then
echo 1 >> "$SUCCESS_ALL"
if (( i == 1 )); then
echo 1 >> "$COLD_OK"
else
echo 1 >> "$WARM_OK"
fi
if [[ "$PRINT_FIRST" == "true" && $i -eq 1 ]]; then
ts=$(date +%H:%M:%S)
echo "[$ts] $store #$i HTTP:200"
fi
else
echo 1 >> "$FAIL_ALL"
if [[ "$PRINT_FIRST" == "true" && $i -eq 1 ]]; then
ts=$(date +%H:%M:%S)
echo "[$ts] $store #$i HTTP:$code (FAIL)"
fi
fi
done
' _ {}
# --- Summary --------------------------------------------------------------
elapsed=$SECONDS
ok=$(wc -l < "$SUCCESS_ALL" 2>/dev/null || echo 0)
fail=$(wc -l < "$FAIL_ALL" 2>/dev/null || echo 0)
cold_ok=$(wc -l < "$COLD_OK" 2>/dev/null || echo 0)
warm_ok=$(wc -l < "$WARM_OK" 2>/dev/null || echo 0)
expected=$(( USED_STORES * REPEAT ))
total=$(( ok + fail ))
rps_all=$(fmt_rps "$ok" "$elapsed")
rps_cold=$(fmt_rps "$cold_ok" "$elapsed")
rps_warm=$(fmt_rps "$warm_ok" "$elapsed")
echo "===== Summary ====="
echo "Stores used: $USED_STORES"
echo "Expected requests: $expected"
echo "Executed requests: $total"
echo "OK (HTTP 200): $ok"
echo "Failed: $fail"
printf "Total time: %dm %ds\n" $((elapsed/60)) $((elapsed%60))
echo "Throughput all RPS: $rps_all"
echo "Cold RPS (round #1): $rps_cold"
echo "Warm RPS (#2..N): $rps_warm"
## Maintainer notes
- No dependency bumps, no API changes, no breaking changes in this
series.
## Patch summary
[PATCH 1/3] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
[PATCH 2/3] partial fix #6049: datastore: use config fast-path in Drop
[PATCH 3/3] datastore: add TTL fallback to catch manual config edits
Thanks for reviewing!
[1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Samuel Rufinatscha (3):
partial fix #6049: datastore: impl ConfigVersionCache fast path for
lookups
partial fix #6049: datastore: use config fast-path in Drop
datastore: add TTL fallback to catch manual config edits
pbs-config/src/config_version_cache.rs | 10 ++-
pbs-datastore/src/datastore.rs | 119 ++++++++++++++++++-------
2 files changed, 96 insertions(+), 33 deletions(-)
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 11%]
* Re: [pbs-devel] [PATCH proxmox] fix #6913: auth-api: fix user ID parsing for 2-character realms
2025-11-11 10:40 5% ` Fabian Grünbichler
@ 2025-11-11 13:49 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-11 13:49 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 11/11/25 11:40 AM, Fabian Grünbichler wrote:
> On November 3, 2025 5:26 pm, Samuel Rufinatscha wrote:
>> PVE and PBS both allow creating realms with names of length ≥ 2.
>> However, when creating a user, PBS rejected realms with 2 characters
>> (e.g. `test@aa`), while PVE accepted them. This issue was reported
>> in our bug tracker [1]. Since the issue appears in the underlying
>> `proxmox/proxmox-auth-api` crate, also PDM userid handling is
>> affected.
>>
>> The issue is caused by a mismatch between realm creation and parsing
>> rules in `proxmox/proxmox-auth-api`. `REALM_ID_SCHEMA` allows
>> min_length(2), but `PROXMOX_AUTH_REALM_STRING_SCHEMA` enforced
>> min_length(3).
>>
>> This patch lowers the minimum realm length in
>> `PROXMOX_AUTH_REALM_STRING_SCHEMA` from 3 to 2 to align PBS and PMG
>> with PVE.
>>
>> ## Testing
>>
>> Please see the attached unit tests.
>> The changes were further verified using a rebuilt PBS .deb
>> deployment. PDM was tested using a non-package binary through the
>> provided client CLI.
>>
>> ## Maintainer notes:
>>
>> Bump the `proxmox-auth-api` dependency, no breaking change.
>> PBS and PDM to use the new dependency.
>
> this part here we'd usually put into the patch notes (the part below the
> `---`), which doesn't show up in git history. you can manage those notes
> using `git notes ..`, including (if you set your config accordingly),
> preserving/merging them across rebases.
>
> Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
>
Thanks for the review Fabian - makes absolutely sense! I will keep this
in mind for my future patches.
>>
>> [1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6913
>>
>> Fixes: #6913
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> proxmox-auth-api/src/types.rs | 68 ++++++++++++++++++++++++++++++++++-
>> 1 file changed, 67 insertions(+), 1 deletion(-)
>>
>> diff --git a/proxmox-auth-api/src/types.rs b/proxmox-auth-api/src/types.rs
>> index 9bde661c..aa09fb93 100644
>> --- a/proxmox-auth-api/src/types.rs
>> +++ b/proxmox-auth-api/src/types.rs
>> @@ -95,7 +95,7 @@ pub const PROXMOX_GROUP_ID_SCHEMA: Schema = StringSchema::new("Group ID")
>> pub const PROXMOX_AUTH_REALM_STRING_SCHEMA: StringSchema =
>> StringSchema::new("Authentication domain ID")
>> .format(&proxmox_schema::api_types::SAFE_ID_FORMAT)
>> - .min_length(3)
>> + .min_length(2)
>> .max_length(32);
>> pub const PROXMOX_AUTH_REALM_SCHEMA: Schema = PROXMOX_AUTH_REALM_STRING_SCHEMA.schema();
>>
>> @@ -769,6 +769,72 @@ fn test_token_id() {
>> assert_eq!(auth_id.to_string(), "test@pam!bar".to_string());
>> }
>>
>> +#[test]
>> +fn test_realm_validation() {
>> + let empty_realm: Result<Realm, _> = "".to_string().try_into();
>> + let one_char_realm: Result<Realm, _> = "a".to_string().try_into();
>> + let two_char_realm: Result<Realm, _> = "aa".to_string().try_into();
>> + let long_realm: Result<Realm, _> = "a".repeat(33).try_into();
>> + let valid_realm: Result<Realm, _> = "pam".to_string().try_into();
>> +
>> + assert!(empty_realm.is_err(), "Empty realm should fail validation");
>> + assert!(
>> + one_char_realm.is_err(),
>> + "1-char realm should fail validation"
>> + );
>> + assert!(
>> + two_char_realm.is_ok(),
>> + "2-char realm should pass validation"
>> + );
>> + assert!(valid_realm.is_ok(), "Typical realm should pass validation");
>> + assert!(
>> + long_realm.is_err(),
>> + "Realm >32 chars should fail validation"
>> + );
>> +}
>> +
>> +#[test]
>> +fn test_userid_validation() {
>> + let empty_str: Result<Userid, _> = "".parse();
>> + let invalid_no_realm: Result<Userid, _> = "user".parse();
>> + let invalid_empty_realm: Result<Userid, _> = "user@".parse();
>> + let invalid_one_char_realm: Result<Userid, _> = "user@a".parse();
>> + let valid_two_char_realm: Result<Userid, _> = "user@aa".parse();
>> + let valid_long_realm: Result<Userid, _> = "user@pam".parse();
>> + let invalid_long_realm: Result<Userid, _> = format!("user@{}", "a".repeat(33)).parse();
>> + let invalid_empty_username: Result<Userid, _> = "@aa".parse();
>> +
>> + assert!(empty_str.is_err(), "Empty userid should fail");
>> + assert!(
>> + invalid_no_realm.is_err(),
>> + "Userid without realm should fail"
>> + );
>> + assert!(
>> + invalid_empty_realm.is_err(),
>> + "Userid with empty realm should fail"
>> + );
>> + assert!(
>> + invalid_one_char_realm.is_err(),
>> + "Userid with 1-char realm should fail"
>> + );
>> + assert!(
>> + valid_two_char_realm.is_ok(),
>> + "Userid with 2-char realm should pass"
>> + );
>> + assert!(
>> + valid_long_realm.is_ok(),
>> + "Userid with normal realm should pass"
>> + );
>> + assert!(
>> + invalid_long_realm.is_err(),
>> + "Userid with realm >32 chars should fail"
>> + );
>> + assert!(
>> + invalid_empty_username.is_err(),
>> + "Userid with empty username should fail"
>> + );
>> +}
>
> these two are more or less tests validating our schema deserializer, but
> as the types are rather core types they also don't hurt.
>
> AFAICT we don't have in-depth tests in proxmox-schema that verify that
> the schema constraints validation actually works as expected, there's
> just some basic tests for query parameter handling and schema types
> themselves - might be an area worth improving ;)
>
Good point! Having tests for the schema constraints would be a great
follow-up and probably good-to-have, also we could move these tests
then.
>> +
>> serde_plain::derive_deserialize_from_fromstr!(Userid, "valid user id");
>> serde_plain::derive_serialize_from_display!(Userid);
>>
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup 2/3] partial fix #6049: datastore: use config fast-path in Drop
2025-11-11 12:29 15% ` [pbs-devel] [PATCH proxmox-backup 2/3] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
@ 2025-11-12 11:24 5% ` Fabian Grünbichler
2025-11-12 15:20 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-11-12 11:24 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On November 11, 2025 1:29 pm, Samuel Rufinatscha wrote:
> The Drop impl of DataStore re-read datastore.cfg to decide whether
> the entry should be evicted from the in-process cache (based on
> maintenance mode’s clear_from_cache). During the investigation of
> issue #6049 [1], a flamegraph [2] showed that the config reload in Drop
> accounted for a measurable share of CPU time under load.
>
> This patch makes Drop O(1) on the fast path by reusing the maintenance-
I am not sure what the O(1) is refering to? This patch implements a
faster cache lookup in front of the (slow) config parsing variant, but
that doesn't really align well with what the "Big O" notation tries to
express ;)
The parsing below still scales with the number of datastores in the
config, after all. It can just be skipped sometimes :)
> mode decision captured at lookup time and stored with the cached
> datastore entry. When the last reference goes away we:
> - decrement active-operation counters, and
> - evict only if the cached decision mandates eviction.
>
> If the cache tag is absent or not fresh, a subsequent slow-path lookup
> will be performed.
>
> Testing
>
> Compared flamegraphs before and after: prior to this change
> (on top of patch 1), stacks originating from Drop included
> pbs_config::datastore::config(). After the change, those vanish from
> the drop path.
>
> An end-to-end benchmark using `/status?verbose=0` with 1000 datastores,
> 5 requests per store, and 16-way parallelism shows a further
> improvement:
>
> | Metric | After commit 1 | After commit 2 | Δ (abs) | Δ (%) |
> |-------------------------|:--------------:|:--------------:|:-------:|:-------:|
> | Total time | 11s | 10s | −1s | −9.09% |
> | Throughput (all rounds) | 454.55 | 500.00 | +45.45 | +10.00% |
> | Cold RPS (round #1) | 90.91 | 100.00 | +9.09 | +10.00% |
> | Warm RPS (rounds 2..N) | 363.64 | 400.00 | +36.36 | +10.00% |
>
> Optimizing Drop improves overall throughput by ~10%. The gain appears
> in both cold and warm rounds, and the flamegraph confirms the config
> reload no longer sits on the hot path.
>
> Links
>
> [1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>
> Fixes: #6049
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> pbs-datastore/src/datastore.rs | 31 +++++++++++++++++++++++++++----
> 1 file changed, 27 insertions(+), 4 deletions(-)
>
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 18eebb58..da80416a 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -200,15 +200,38 @@ impl Drop for DataStore {
> // remove datastore from cache iff
> // - last task finished, and
> // - datastore is in a maintenance mode that mandates it
> - let remove_from_cache = last_task
> - && pbs_config::datastore::config()
> +
> + // first check: check if last task finished
> + if !last_task {
> + return;
> + }
> +
> + let cached_tag = self.inner.cached_config_tag.as_ref();
> + let last_gen_num = cached_tag.and_then(|c| c.last_generation);
> + let gen_num = ConfigVersionCache::new()
> + .ok()
> + .map(|c| c.datastore_generation());
> +
> + let cache_is_fresh = match (last_gen_num, gen_num) {
> + (Some(a), Some(b)) => a == b,
> + _ => false,
> + };
this is just last_gen_num == gen_num and checking that either is Some.
if we make the tag always contain a generation instead of an option, we
can simplify this code ;)
> +
> + let mm_mandate = if cache_is_fresh {
> + cached_tag
> + .and_then(|c| c.last_maintenance_mode.as_ref())
> + .is_some_and(|m| m.clear_from_cache())
> + } else {
> + pbs_config::datastore::config()
> .and_then(|(s, _)| s.lookup::<DataStoreConfig>("datastore", self.name()))
> .is_ok_and(|c| {
> c.get_maintenance_mode()
> .is_some_and(|m| m.clear_from_cache())
> - });
> + })
> + };
>
> - if remove_from_cache {
> + // second check: check maintenance mode mandate
> + if mm_mandate {
> DATASTORE_MAP.lock().unwrap().remove(self.name());
> }
> }
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path
2025-11-11 12:29 11% [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path Samuel Rufinatscha
` (2 preceding siblings ...)
2025-11-11 12:29 15% ` [pbs-devel] [PATCH proxmox-backup 3/3] datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
@ 2025-11-12 11:27 5% ` Fabian Grünbichler
2025-11-12 17:27 6% ` Samuel Rufinatscha
2025-11-14 15:08 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
4 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-11-12 11:27 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On November 11, 2025 1:29 pm, Samuel Rufinatscha wrote:
> Hi,
>
> this series reduces CPU time in datastore lookups by avoiding repeated
> datastore.cfg reads/parses in both `lookup_datastore()` and
> `DataStore::Drop`. It also adds a TTL so manual config edits are
> noticed without reintroducing hashing on every request.
>
> While investigating #6049 [1], cargo-flamegraph [2] showed hotspots during
> repeated `/status` calls in `lookup_datastore()` and in `Drop`,
> dominated by `pbs_config::datastore::config()` (config parse).
>
> The parsing cost itself should eventually be investigated in a future
> effort. Furthermore, cargo-flamegraph showed that when using a
> token-based auth method to access the API, a significant amount of time
> is spent in validation on every request, likely related to bcrypt.
> Also this should be eventually revisited in a future effort.
please file a bug for the token part, if there isn't already one!
thanks for diving into this, it already looks promising, even though the
effect on more "normal" systems with more reasonable numbers of
datastores and clients will be less pronounced ;)
the big TL;DR would be that we trade faster datastore lookups (which
happen quite frequently, in particular if there are many datastores with
clients checking their status) against slightly delayed reload of the
configuration in case of manual, behind-our-backs edits, with one
particular corner case that is slightly problematic, but also a bit
contrived:
- datastore is looked up
- config is edited (manually) to set maintenance mode to one that
requires removing from the datastore map once the last task exits
- last task drops the datastore struct
- no regular edits happened in the meantime
- the Drop handler doesn't know it needs to remove the datastore from
the map
- open FD is held by proxy, datastore fails to be unmounted/..
we could solve this issue by not only bumping the generation on save,
but also when we reload the config (in particular if we cache the whole
config!). that would make the Drop handler efficient enough for idle
systems that have mostly lookups but no long running tasks. as soon as a
datastore has long running tasks, the last such task will likely exit
long after the TTL for its config lookup has expired, so will need to do
a refresh - although that refresh could again be from the global cache,
instead of from disk? still wouldn't close the window entirely, but make
it pretty unlikely to be hit in practice..
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup 1/3] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
2025-11-11 12:29 12% ` [pbs-devel] [PATCH proxmox-backup 1/3] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
@ 2025-11-12 13:24 4% ` Fabian Grünbichler
2025-11-13 12:59 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-11-12 13:24 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On November 11, 2025 1:29 pm, Samuel Rufinatscha wrote:
> Repeated /status requests caused lookup_datastore() to re-read and
> parse datastore.cfg on every call. The issue was mentioned in report
> #6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
> dominated by pbs_config::datastore::config() (config parsing).
>
> This patch adds a fast path to lookup_datastore() using the shared-
> memory ConfigVersionCache generation counter for datastore.cfg. It
> stores the last seen generation number alongside the cached
> DataStoreImpl and, when a subsequent lookup sees the same generation,
> we reuse the cached instance without re-reading the on-disk config. If
> the generation differs (or the cache is unavailable), it falls back to
> the existing slow path with no behavioral changes.
>
> Behavioral notes
>
> - The generation is bumped via the existing save_config() path, so
> API-driven config changes are detected immediately.
> - Manual edits to datastore.cfg are not detected by this commit; a TTL
> guard is introduced in a later patch to cover that case.
> - DataStore::drop still performs a config read on the common path; this
> is addressed in the next patch.
>
> Testing
>
> cargo-flamegraph confirms the config-parse hotspot in
> lookup_datastore() disappears from the hot path.
>
> Additionally, an end-to-end benchmark was performed using the
> `/status?verbose=0` API with 1000 datastores, 5 requests per store,
> and 16-way parallelism:
>
> | Metric | Before | After | Δ (abs) | Δ (%) |
> |--------------------------|:------:|:------:|:-------:|:-------:|
> | Total time | 13s | 11s | −2s | −15.38% |
> | Throughput (all rounds) | 384.62 | 454.55 | +69.93 | +18.18% |
> | Cold RPS (round #1) | 76.92 | 90.91 | +13.99 | +18.19% |
> | Warm RPS (rounds 2..N) | 307.69 | 363.64 | +55.95 | +18.18% |
>
> Throughput improved by ~18% overall, with total time reduced by ~15%.
> Warm lookups now reuse cached datastore instances and skip redundant
> config parsing entirely. The first-access round also shows a similar
> improvement, likely due to reduced contention and I/O when many stores
> are accessed in parallel.
>
> Note: A second hotspot remains in Drop where a config read occurs; that
> is addressed in the next commit.
it would be interesting to also have numbers for just lookup+Drop,
without all the HTTP and TLS overhead to really isolate it. that should
also make it easier to reliably benchmark with something like hyperfine
;)
for my simple config (5 datastores) single-threaded lookup+drop of a
single datastore 100k times gives around 1.31 speedup for the whole
series. the slightly modified version from below (which basically runs
the most expensive part of Drop only once) for the same test setup still
gives a speedup of 1.17
re-running the same benchmark with a config with 1k datastores, querying
M datastores N times gives the following speedups:
M = 1, N = 1000: 15.6x faster
M = 10, N = 100: 14.5x
M = 100, N = 10: 8.8x
M = 1000, N = 1: 1.8x (so this is basically showing the speedup of the
improved Drop handling alone!)
and then a slightly modified version, that keeps the DataStore instance
around until all N lookups for that datastore are done, then dropping
them in bulk (which mimics having lots of lookups while a task is
running, making the Drop handler only do the expensive part every once
in a while when the last task for a given datastore exits):
use anyhow::Error;
use pbs_api_types::Operation;
use pbs_datastore::DataStore;
fn main() -> Result<(), Error> {
let mut args = std::env::args();
args.next();
let datastores = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
let iterations = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
for d in 1..=datastores {
let name = format!("pbs_bench_{d}");
let mut stores = Vec::with_capacity(iterations);
for i in 1..=iterations {
stores.push(DataStore::lookup_datastore(&name, Some(Operation::Write))?);
}
}
Ok(())
}
M = 1, N = 1000: 8.5x faster
M = 10, N = 100: 7.9x
M = 100, N = 10: 5.2x
M = 1000, N = 1: 1.9x
looking at the flamegraph of this isolated benchmark it's now obvious
that the remaining overhead is lockfiles and tracking the operations
(both in lookup and when dropping)..
side-note: I actually found a bug in our operations tracking while
benchmarking this series that gave me wildly different numbers because
the "drop last task" part never got executed as a result:
https://lore.proxmox.com/pbs-devel/20251112131525.645971-1-f.gruenbichler@proxmox.com/T/#u
>
> Links
>
> [1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>
> Fixes: #6049
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> pbs-config/src/config_version_cache.rs | 10 +++-
> pbs-datastore/src/datastore.rs | 77 +++++++++++++++++---------
> 2 files changed, 59 insertions(+), 28 deletions(-)
>
> diff --git a/pbs-config/src/config_version_cache.rs b/pbs-config/src/config_version_cache.rs
> index e8fb994f..b875f7e0 100644
> --- a/pbs-config/src/config_version_cache.rs
> +++ b/pbs-config/src/config_version_cache.rs
> @@ -26,7 +26,6 @@ struct ConfigVersionCacheDataInner {
> // Traffic control (traffic-control.cfg) generation/version.
> traffic_control_generation: AtomicUsize,
> // datastore (datastore.cfg) generation/version
> - // FIXME: remove with PBS 3.0
> datastore_generation: AtomicUsize,
> // Add further atomics here
> }
> @@ -145,8 +144,15 @@ impl ConfigVersionCache {
> .fetch_add(1, Ordering::AcqRel);
> }
>
> + /// Returns the datastore generation number.
> + pub fn datastore_generation(&self) -> usize {
> + self.shmem
> + .data()
> + .datastore_generation
> + .load(Ordering::Acquire)
> + }
> +
> /// Increase the datastore generation number.
> - // FIXME: remove with PBS 3.0 or make actually useful again in datastore lookup
> pub fn increase_datastore_generation(&self) -> usize {
> self.shmem
> .data()
this part could be split out into its own patch
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 70af94d8..18eebb58 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -32,7 +32,7 @@ use pbs_api_types::{
> MaintenanceType, Operation, UPID,
> };
> use pbs_config::s3::S3_CFG_TYPE_ID;
> -use pbs_config::BackupLockGuard;
> +use pbs_config::{BackupLockGuard, ConfigVersionCache};
>
> use crate::backup_info::{
> BackupDir, BackupGroup, BackupInfo, OLD_LOCKING, PROTECTED_MARKER_FILENAME,
> @@ -140,10 +140,10 @@ pub struct DataStoreImpl {
> last_gc_status: Mutex<GarbageCollectionStatus>,
> verify_new: bool,
> chunk_order: ChunkOrder,
> - last_digest: Option<[u8; 32]>,
> sync_level: DatastoreFSyncLevel,
> backend_config: DatastoreBackendConfig,
> lru_store_caching: Option<LocalDatastoreLruCache>,
> + cached_config_tag: Option<CachedDatastoreConfigTag>,
> }
>
> impl DataStoreImpl {
> @@ -156,10 +156,10 @@ impl DataStoreImpl {
> last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
> verify_new: false,
> chunk_order: Default::default(),
> - last_digest: None,
> sync_level: Default::default(),
> backend_config: Default::default(),
> lru_store_caching: None,
> + cached_config_tag: None,
> })
> }
> }
> @@ -224,6 +224,15 @@ pub enum DatastoreBackend {
> S3(Arc<S3Client>),
> }
>
> +/// Used to determine whether a cached datastore instance is still valid
> +/// or needs to be reloaded after a config change.
> +struct CachedDatastoreConfigTag {
> + /// Maintenance mode at the time the lookup hint was captured, if any.
> + last_maintenance_mode: Option<MaintenanceMode>,
> + /// Datastore generation number from `ConfigVersionCache`; `None` when the cache wasn't available.
> + last_generation: Option<usize>,
if the whole tag is an option, do we really need to make the generation
an option as well?
> +}
> +
> impl DataStore {
> // This one just panics on everything
> #[doc(hidden)]
> @@ -299,13 +308,40 @@ impl DataStore {
> // we use it to decide whether it is okay to delete the datastore.
> let _config_lock = pbs_config::datastore::lock_config()?;
>
> - // we could use the ConfigVersionCache's generation for staleness detection, but we load
> - // the config anyway -> just use digest, additional benefit: manual changes get detected
> - let (config, digest) = pbs_config::datastore::config()?;
> + // Get the current datastore.cfg generation number
> + let gen_num = ConfigVersionCache::new()
> + .ok()
> + .map(|c| c.datastore_generation());
> +
> + // Fast-path: if we have a cached entry created under the same datastore.cfg generation number, reuse it.
> + if let (Some(gen_num), Some(ds)) =
> + (gen_num, DATASTORE_MAP.lock().unwrap().get(name).cloned())
> + {
> + if let Some(cached_tag) = &ds.cached_config_tag {
> + if cached_tag.last_generation == Some(gen_num) {
> + if let Some(mm) = &cached_tag.last_maintenance_mode {
> + if let Err(error) = mm.check(operation) {
> + bail!("datastore '{name}' is unavailable: {error}");
> + }
> + }
> + if let Some(operation) = operation {
> + update_active_operations(name, operation, 1)?;
> + }
> + return Ok(Arc::new(Self {
> + inner: ds,
> + operation,
> + }));
> + }
> + }
> + }
> +
> + // Slow path: (re)load config
> + let (config, _digest) = pbs_config::datastore::config()?;
> let config: DataStoreConfig = config.lookup("datastore", name)?;
>
> - if let Some(maintenance_mode) = config.get_maintenance_mode() {
> - if let Err(error) = maintenance_mode.check(operation) {
> + let maintenance_mode = config.get_maintenance_mode();
> + if let Some(mm) = &maintenance_mode {
> + if let Err(error) = mm.check(operation) {
> bail!("datastore '{name}' is unavailable: {error}");
> }
> }
after this here we have a check for the mount status in the old hot
path, that is missing in the new hot path. the mount status can change
even if the config doesn't, so we should probably add this back to the
hot path and re-run the numbers?
that check again needs more parts of the config, so maybe we could
explore caching the full config here? e.g., add a new static
Mutex<Option<(DataStoreConfig, usize)>> (extended by the timestamp in
the last patch) and adapt the patches here to use it? depending on
whether we make the cached config available outside of lookup_datastore,
we could then even not add the maintenance mode to the cache tag, and
just store the generation number there, and retrieve the maintenance
mode from the cached config in the Drop implementation..
there's a little more duplication here, e.g. we lock the map and check
for an entry in both the fast and slow paths, we could do it once up
front (nobody can change the map while we have it locked anyway), the
checks are written twice and could probably be extracted into a helper
so that future similar checks are added to both paths and not to only
one by accident, ..
> @@ -321,16 +357,6 @@ impl DataStore {
>
> // reuse chunk store so that we keep using the same process locker instance!
> let chunk_store = if let Some(datastore) = &entry {
> - let last_digest = datastore.last_digest.as_ref();
> - if let Some(true) = last_digest.map(|last_digest| last_digest == &digest) {
> - if let Some(operation) = operation {
> - update_active_operations(name, operation, 1)?;
> - }
> - return Ok(Arc::new(Self {
> - inner: Arc::clone(datastore),
> - operation,
> - }));
> - }
> Arc::clone(&datastore.chunk_store)
> } else {
> let tuning: DatastoreTuning = serde_json::from_value(
> @@ -344,7 +370,11 @@ impl DataStore {
> )?)
> };
>
> - let datastore = DataStore::with_store_and_config(chunk_store, config, Some(digest))?;
> + let mut datastore = DataStore::with_store_and_config(chunk_store, config)?;
> + datastore.cached_config_tag = Some(CachedDatastoreConfigTag {
> + last_maintenance_mode: maintenance_mode,
> + last_generation: gen_num,
> + });
this part should be in with_store_and_config, which should get the
last_generation (and later last_update) as parameter(s), just like it
had the digest before this patch..
>
> let datastore = Arc::new(datastore);
> datastore_cache.insert(name.to_string(), datastore.clone());
> @@ -430,11 +460,7 @@ impl DataStore {
> config.absolute_path(),
> tuning.sync_level.unwrap_or_default(),
> )?;
> - let inner = Arc::new(Self::with_store_and_config(
> - Arc::new(chunk_store),
> - config,
> - None,
> - )?);
> + let inner = Arc::new(Self::with_store_and_config(Arc::new(chunk_store), config)?);
>
> if let Some(operation) = operation {
> update_active_operations(&name, operation, 1)?;
> @@ -446,7 +472,6 @@ impl DataStore {
> fn with_store_and_config(
> chunk_store: Arc<ChunkStore>,
> config: DataStoreConfig,
> - last_digest: Option<[u8; 32]>,
> ) -> Result<DataStoreImpl, Error> {
> let mut gc_status_path = chunk_store.base_path();
> gc_status_path.push(".gc-status");
> @@ -506,10 +531,10 @@ impl DataStore {
> last_gc_status: Mutex::new(gc_status),
> verify_new: config.verify_new.unwrap_or(false),
> chunk_order: tuning.chunk_order.unwrap_or_default(),
> - last_digest,
> sync_level: tuning.sync_level.unwrap_or_default(),
> backend_config,
> lru_store_caching,
> + cached_config_tag: None,
> })
> }
>
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 4%]
* Re: [pbs-devel] [PATCH proxmox-backup 2/3] partial fix #6049: datastore: use config fast-path in Drop
2025-11-12 11:24 5% ` Fabian Grünbichler
@ 2025-11-12 15:20 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-12 15:20 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 11/12/25 12:25 PM, Fabian Grünbichler wrote:
> On November 11, 2025 1:29 pm, Samuel Rufinatscha wrote:
>> The Drop impl of DataStore re-read datastore.cfg to decide whether
>> the entry should be evicted from the in-process cache (based on
>> maintenance mode’s clear_from_cache). During the investigation of
>> issue #6049 [1], a flamegraph [2] showed that the config reload in Drop
>> accounted for a measurable share of CPU time under load.
>>
>> This patch makes Drop O(1) on the fast path by reusing the maintenance-
>
> I am not sure what the O(1) is refering to? This patch implements a
> faster cache lookup in front of the (slow) config parsing variant, but
> that doesn't really align well with what the "Big O" notation tries to
> express ;)
>
> The parsing below still scales with the number of datastores in the
> config, after all. It can just be skipped sometimes :)
>
Good point — the O(1) reference is a rather misleading. I’ll rephrase it
in v2 :)
>> mode decision captured at lookup time and stored with the cached
>> datastore entry. When the last reference goes away we:
>> - decrement active-operation counters, and
>> - evict only if the cached decision mandates eviction.
>>
>> If the cache tag is absent or not fresh, a subsequent slow-path lookup
>> will be performed.
>>
>> Testing
>>
>> Compared flamegraphs before and after: prior to this change
>> (on top of patch 1), stacks originating from Drop included
>> pbs_config::datastore::config(). After the change, those vanish from
>> the drop path.
>>
>> An end-to-end benchmark using `/status?verbose=0` with 1000 datastores,
>> 5 requests per store, and 16-way parallelism shows a further
>> improvement:
>>
>> | Metric | After commit 1 | After commit 2 | Δ (abs) | Δ (%) |
>> |-------------------------|:--------------:|:--------------:|:-------:|:-------:|
>> | Total time | 11s | 10s | −1s | −9.09% |
>> | Throughput (all rounds) | 454.55 | 500.00 | +45.45 | +10.00% |
>> | Cold RPS (round #1) | 90.91 | 100.00 | +9.09 | +10.00% |
>> | Warm RPS (rounds 2..N) | 363.64 | 400.00 | +36.36 | +10.00% |
>>
>> Optimizing Drop improves overall throughput by ~10%. The gain appears
>> in both cold and warm rounds, and the flamegraph confirms the config
>> reload no longer sits on the hot path.
>>
>> Links
>>
>> [1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
>> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>>
>> Fixes: #6049
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> pbs-datastore/src/datastore.rs | 31 +++++++++++++++++++++++++++----
>> 1 file changed, 27 insertions(+), 4 deletions(-)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index 18eebb58..da80416a 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -200,15 +200,38 @@ impl Drop for DataStore {
>> // remove datastore from cache iff
>> // - last task finished, and
>> // - datastore is in a maintenance mode that mandates it
>> - let remove_from_cache = last_task
>> - && pbs_config::datastore::config()
>> +
>> + // first check: check if last task finished
>> + if !last_task {
>> + return;
>> + }
>> +
>> + let cached_tag = self.inner.cached_config_tag.as_ref();
>> + let last_gen_num = cached_tag.and_then(|c| c.last_generation);
>> + let gen_num = ConfigVersionCache::new()
>> + .ok()
>> + .map(|c| c.datastore_generation());
>> +
>> + let cache_is_fresh = match (last_gen_num, gen_num) {
>> + (Some(a), Some(b)) => a == b,
>> + _ => false,
>> + };
>
> this is just last_gen_num == gen_num and checking that either is Some.
> if we make the tag always contain a generation instead of an option, we
> can simplify this code ;)
>
Good point, will adjust this. I think we could keep
`ConfigVersionCache::new().ok()` and create the optional cache tag only
if the generation number is `Some`. This way, the lookup would still be
able to perform a slow path read if the cache isn’t available for any
reason.
>> +
>> + let mm_mandate = if cache_is_fresh {
>> + cached_tag
>> + .and_then(|c| c.last_maintenance_mode.as_ref())
>> + .is_some_and(|m| m.clear_from_cache())
>> + } else {
>> + pbs_config::datastore::config()
>> .and_then(|(s, _)| s.lookup::<DataStoreConfig>("datastore", self.name()))
>> .is_ok_and(|c| {
>> c.get_maintenance_mode()
>> .is_some_and(|m| m.clear_from_cache())
>> - });
>> + })
>> + };
>>
>> - if remove_from_cache {
>> + // second check: check maintenance mode mandate
>> + if mm_mandate {
>> DATASTORE_MAP.lock().unwrap().remove(self.name());
>> }
>> }
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path
2025-11-12 11:27 5% ` [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path Fabian Grünbichler
@ 2025-11-12 17:27 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-12 17:27 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 11/12/25 12:27 PM, Fabian Grünbichler wrote:
> On November 11, 2025 1:29 pm, Samuel Rufinatscha wrote:
>> Hi,
>>
>> this series reduces CPU time in datastore lookups by avoiding repeated
>> datastore.cfg reads/parses in both `lookup_datastore()` and
>> `DataStore::Drop`. It also adds a TTL so manual config edits are
>> noticed without reintroducing hashing on every request.
>>
>> While investigating #6049 [1], cargo-flamegraph [2] showed hotspots during
>> repeated `/status` calls in `lookup_datastore()` and in `Drop`,
>> dominated by `pbs_config::datastore::config()` (config parse).
>>
>> The parsing cost itself should eventually be investigated in a future
>> effort. Furthermore, cargo-flamegraph showed that when using a
>> token-based auth method to access the API, a significant amount of time
>> is spent in validation on every request, likely related to bcrypt.
>> Also this should be eventually revisited in a future effort.
>
> please file a bug for the token part, if there isn't already one!
>
Thanks for the in-depth review Fabian! I created a bug report for the
token part and added the relevant flamegraph - this should help narrow
down the issue: https://bugzilla.proxmox.com/show_bug.cgi?id=7017
> thanks for diving into this, it already looks promising, even though the
> effect on more "normal" systems with more reasonable numbers of
> datastores and clients will be less pronounced ;)
>
> the big TL;DR would be that we trade faster datastore lookups (which
> happen quite frequently, in particular if there are many datastores with
> clients checking their status) against slightly delayed reload of the
> configuration in case of manual, behind-our-backs edits, with one
> particular corner case that is slightly problematic, but also a bit
> contrived:
> - datastore is looked up
> - config is edited (manually) to set maintenance mode to one that
> requires removing from the datastore map once the last task exits
> - last task drops the datastore struct
> - no regular edits happened in the meantime
> - the Drop handler doesn't know it needs to remove the datastore from
> the map
> - open FD is held by proxy, datastore fails to be unmounted/..
>
> we could solve this issue by not only bumping the generation on save,
> but also when we reload the config (in particular if we cache the whole
> config!). that would make the Drop handler efficient enough for idle
> systems that have mostly lookups but no long running tasks. as soon as a
> datastore has long running tasks, the last such task will likely exit
> long after the TTL for its config lookup has expired, so will need to do
> a refresh - although that refresh could again be from the global cache,
> instead of from disk? still wouldn't close the window entirely, but make
> it pretty unlikely to be hit in practice..
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
Good idea! I will add the bump in the lookup_datastore() slow path
directly after (config, digest) is read and increment the generation if
the digest changed but generation hasn’t - this should also help avoid
unnecessary cache invalidations.
In Drop we then either check if the shared gen differs from the cached
tag gen or the tag is TTL expired, otherwise use the cached decision.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup 1/3] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
2025-11-12 13:24 4% ` Fabian Grünbichler
@ 2025-11-13 12:59 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-13 12:59 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 11/12/25 2:24 PM, Fabian Grünbichler wrote:
> On November 11, 2025 1:29 pm, Samuel Rufinatscha wrote:
>> Repeated /status requests caused lookup_datastore() to re-read and
>> parse datastore.cfg on every call. The issue was mentioned in report
>> #6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
>> dominated by pbs_config::datastore::config() (config parsing).
>>
>> This patch adds a fast path to lookup_datastore() using the shared-
>> memory ConfigVersionCache generation counter for datastore.cfg. It
>> stores the last seen generation number alongside the cached
>> DataStoreImpl and, when a subsequent lookup sees the same generation,
>> we reuse the cached instance without re-reading the on-disk config. If
>> the generation differs (or the cache is unavailable), it falls back to
>> the existing slow path with no behavioral changes.
>>
>> Behavioral notes
>>
>> - The generation is bumped via the existing save_config() path, so
>> API-driven config changes are detected immediately.
>> - Manual edits to datastore.cfg are not detected by this commit; a TTL
>> guard is introduced in a later patch to cover that case.
>> - DataStore::drop still performs a config read on the common path; this
>> is addressed in the next patch.
>>
>> Testing
>>
>> cargo-flamegraph confirms the config-parse hotspot in
>> lookup_datastore() disappears from the hot path.
>>
>> Additionally, an end-to-end benchmark was performed using the
>> `/status?verbose=0` API with 1000 datastores, 5 requests per store,
>> and 16-way parallelism:
>>
>> | Metric | Before | After | Δ (abs) | Δ (%) |
>> |--------------------------|:------:|:------:|:-------:|:-------:|
>> | Total time | 13s | 11s | −2s | −15.38% |
>> | Throughput (all rounds) | 384.62 | 454.55 | +69.93 | +18.18% |
>> | Cold RPS (round #1) | 76.92 | 90.91 | +13.99 | +18.19% |
>> | Warm RPS (rounds 2..N) | 307.69 | 363.64 | +55.95 | +18.18% |
>>
>> Throughput improved by ~18% overall, with total time reduced by ~15%.
>> Warm lookups now reuse cached datastore instances and skip redundant
>> config parsing entirely. The first-access round also shows a similar
>> improvement, likely due to reduced contention and I/O when many stores
>> are accessed in parallel.
>>
>> Note: A second hotspot remains in Drop where a config read occurs; that
>> is addressed in the next commit.
>
> it would be interesting to also have numbers for just lookup+Drop,
> without all the HTTP and TLS overhead to really isolate it. that should
> also make it easier to reliably benchmark with something like hyperfine
> ;)
>
Good point, I will isolate the numbers - seems like the TLS overhead is
quite huge. Thanks for the hyperfine reference!
> for my simple config (5 datastores) single-threaded lookup+drop of a
> single datastore 100k times gives around 1.31 speedup for the whole
> series. the slightly modified version from below (which basically runs
> the most expensive part of Drop only once) for the same test setup still
> gives a speedup of 1.17
>
So the lookup optimization dominates the speedup if we hold longer to
the datastores, great to see.
> re-running the same benchmark with a config with 1k datastores, querying> M datastores N times gives the following speedups:
>
> M = 1, N = 1000: 15.6x faster
> M = 10, N = 100: 14.5x
> M = 100, N = 10: 8.8x
> M = 1000, N = 1: 1.8x (so this is basically showing the speedup of the
> improved Drop handling alone!)
>
> and then a slightly modified version, that keeps the DataStore instance
> around until all N lookups for that datastore are done, then dropping
> them in bulk (which mimics having lots of lookups while a task is
> running, making the Drop handler only do the expensive part every once
> in a while when the last task for a given datastore exits):
>
> use anyhow::Error;
>
> use pbs_api_types::Operation;
> use pbs_datastore::DataStore;
>
> fn main() -> Result<(), Error> {
> let mut args = std::env::args();
> args.next();
>
> let datastores = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> let iterations = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> for d in 1..=datastores {
> let name = format!("pbs_bench_{d}");
> let mut stores = Vec::with_capacity(iterations);
> for i in 1..=iterations {
> stores.push(DataStore::lookup_datastore(&name, Some(Operation::Write))?);
> }
> }
>
> Ok(())
> }
>
> M = 1, N = 1000: 8.5x faster
> M = 10, N = 100: 7.9x
> M = 100, N = 10: 5.2x
> M = 1000, N = 1: 1.9x
>
> looking at the flamegraph of this isolated benchmark it's now obvious
> that the remaining overhead is lockfiles and tracking the operations
> (both in lookup and when dropping)..
>
Thanks a lot also for setting up the test environment and providing your
numbers, which are helpful to compare against!
Regarding the overhead in lockfiles and tracking operations, it is a
good confirmation that everything else on the hot path is optimized - I
think the locks and tracking operations could maybe be revisited in a
future effort!
> side-note: I actually found a bug in our operations tracking while
> benchmarking this series that gave me wildly different numbers because
> the "drop last task" part never got executed as a result:
>
> https://lore.proxmox.com/pbs-devel/20251112131525.645971-1-f.gruenbichler@proxmox.com/T/#u
>
Nice catch!
>>
>> Links
>>
>> [1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
>> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>>
>> Fixes: #6049
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> pbs-config/src/config_version_cache.rs | 10 +++-
>> pbs-datastore/src/datastore.rs | 77 +++++++++++++++++---------
>> 2 files changed, 59 insertions(+), 28 deletions(-)
>>
>> diff --git a/pbs-config/src/config_version_cache.rs b/pbs-config/src/config_version_cache.rs
>> index e8fb994f..b875f7e0 100644
>> --- a/pbs-config/src/config_version_cache.rs
>> +++ b/pbs-config/src/config_version_cache.rs
>> @@ -26,7 +26,6 @@ struct ConfigVersionCacheDataInner {
>> // Traffic control (traffic-control.cfg) generation/version.
>> traffic_control_generation: AtomicUsize,
>> // datastore (datastore.cfg) generation/version
>> - // FIXME: remove with PBS 3.0
>> datastore_generation: AtomicUsize,
>> // Add further atomics here
>> }
>> @@ -145,8 +144,15 @@ impl ConfigVersionCache {
>> .fetch_add(1, Ordering::AcqRel);
>> }
>>
>> + /// Returns the datastore generation number.
>> + pub fn datastore_generation(&self) -> usize {
>> + self.shmem
>> + .data()
>> + .datastore_generation
>> + .load(Ordering::Acquire)
>> + }
>> +
>> /// Increase the datastore generation number.
>> - // FIXME: remove with PBS 3.0 or make actually useful again in datastore lookup
>> pub fn increase_datastore_generation(&self) -> usize {
>> self.shmem
>> .data()
>
> this part could be split out into its own patch
>
Will factor this out into a separate patch.
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index 70af94d8..18eebb58 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -32,7 +32,7 @@ use pbs_api_types::{
>> MaintenanceType, Operation, UPID,
>> };
>> use pbs_config::s3::S3_CFG_TYPE_ID;
>> -use pbs_config::BackupLockGuard;
>> +use pbs_config::{BackupLockGuard, ConfigVersionCache};
>>
>> use crate::backup_info::{
>> BackupDir, BackupGroup, BackupInfo, OLD_LOCKING, PROTECTED_MARKER_FILENAME,
>> @@ -140,10 +140,10 @@ pub struct DataStoreImpl {
>> last_gc_status: Mutex<GarbageCollectionStatus>,
>> verify_new: bool,
>> chunk_order: ChunkOrder,
>> - last_digest: Option<[u8; 32]>,
>> sync_level: DatastoreFSyncLevel,
>> backend_config: DatastoreBackendConfig,
>> lru_store_caching: Option<LocalDatastoreLruCache>,
>> + cached_config_tag: Option<CachedDatastoreConfigTag>,
>> }
>>
>> impl DataStoreImpl {
>> @@ -156,10 +156,10 @@ impl DataStoreImpl {
>> last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
>> verify_new: false,
>> chunk_order: Default::default(),
>> - last_digest: None,
>> sync_level: Default::default(),
>> backend_config: Default::default(),
>> lru_store_caching: None,
>> + cached_config_tag: None,
>> })
>> }
>> }
>> @@ -224,6 +224,15 @@ pub enum DatastoreBackend {
>> S3(Arc<S3Client>),
>> }
>>
>> +/// Used to determine whether a cached datastore instance is still valid
>> +/// or needs to be reloaded after a config change.
>> +struct CachedDatastoreConfigTag {
>> + /// Maintenance mode at the time the lookup hint was captured, if any.
>> + last_maintenance_mode: Option<MaintenanceMode>,
>> + /// Datastore generation number from `ConfigVersionCache`; `None` when the cache wasn't available.
>> + last_generation: Option<usize>,
>
> if the whole tag is an option, do we really need to make the generation
> an option as well?
>
Good point, keeping the tag optional only is enough and will simplify
generation checks.
>> +}
>> +
>> impl DataStore {
>> // This one just panics on everything
>> #[doc(hidden)]
>> @@ -299,13 +308,40 @@ impl DataStore {
>> // we use it to decide whether it is okay to delete the datastore.
>> let _config_lock = pbs_config::datastore::lock_config()?;
>>
>> - // we could use the ConfigVersionCache's generation for staleness detection, but we load
>> - // the config anyway -> just use digest, additional benefit: manual changes get detected
>> - let (config, digest) = pbs_config::datastore::config()?;
>> + // Get the current datastore.cfg generation number
>> + let gen_num = ConfigVersionCache::new()
>> + .ok()
>> + .map(|c| c.datastore_generation());
>> +
>> + // Fast-path: if we have a cached entry created under the same datastore.cfg generation number, reuse it.
>> + if let (Some(gen_num), Some(ds)) =
>> + (gen_num, DATASTORE_MAP.lock().unwrap().get(name).cloned())
>> + {
>> + if let Some(cached_tag) = &ds.cached_config_tag {
>> + if cached_tag.last_generation == Some(gen_num) {
>> + if let Some(mm) = &cached_tag.last_maintenance_mode {
>> + if let Err(error) = mm.check(operation) {
>> + bail!("datastore '{name}' is unavailable: {error}");
>> + }
>> + }
>> + if let Some(operation) = operation {
>> + update_active_operations(name, operation, 1)?;
>> + }
>> + return Ok(Arc::new(Self {
>> + inner: ds,
>> + operation,
>> + }));
>> + }
>> + }
>> + }
>> +
>> + // Slow path: (re)load config
>> + let (config, _digest) = pbs_config::datastore::config()?;
>> let config: DataStoreConfig = config.lookup("datastore", name)?;
>>
>> - if let Some(maintenance_mode) = config.get_maintenance_mode() {
>> - if let Err(error) = maintenance_mode.check(operation) {
>> + let maintenance_mode = config.get_maintenance_mode();
>> + if let Some(mm) = &maintenance_mode {
>> + if let Err(error) = mm.check(operation) {
>> bail!("datastore '{name}' is unavailable: {error}");
>> }
>> }
>
> after this here we have a check for the mount status in the old hot
> path, that is missing in the new hot path. the mount status can change
> even if the config doesn't, so we should probably add this back to the
> hot path and re-run the numbers?
>
> that check again needs more parts of the config, so maybe we could
> explore caching the full config here? e.g., add a new static
> Mutex<Option<(DataStoreConfig, usize)>> (extended by the timestamp in
> the last patch) and adapt the patches here to use it? depending on
> whether we make the cached config available outside of lookup_datastore,
> we could then even not add the maintenance mode to the cache tag, and
> just store the generation number there, and retrieve the maintenance
> mode from the cached config in the Drop implementation..
>
For the mount check we will need device_uuid and datastore mount dir
path which we could add to the cached entry, however I think I would
also rather explore caching of the full global config here. This should
result in further performance gains - this would/should allow for
eventual fast-lookups to any other datastore. Will try to integrate it
in v2.
> there's a little more duplication here, e.g. we lock the map and check
> for an entry in both the fast and slow paths, we could do it once up
> front (nobody can change the map while we have it locked anyway), the
> checks are written twice and could probably be extracted into a helper
> so that future similar checks are added to both paths and not to only
> one by accident, ..
>
Good point, I will factor it out (e.g. small closure).
>> @@ -321,16 +357,6 @@ impl DataStore {
>>
>> // reuse chunk store so that we keep using the same process locker instance!
>> let chunk_store = if let Some(datastore) = &entry {
>> - let last_digest = datastore.last_digest.as_ref();
>> - if let Some(true) = last_digest.map(|last_digest| last_digest == &digest) {
>> - if let Some(operation) = operation {
>> - update_active_operations(name, operation, 1)?;
>> - }
>> - return Ok(Arc::new(Self {
>> - inner: Arc::clone(datastore),
>> - operation,
>> - }));
>> - }
>> Arc::clone(&datastore.chunk_store)
>> } else {
>> let tuning: DatastoreTuning = serde_json::from_value(
>> @@ -344,7 +370,11 @@ impl DataStore {
>> )?)
>> };
>>
>> - let datastore = DataStore::with_store_and_config(chunk_store, config, Some(digest))?;
>> + let mut datastore = DataStore::with_store_and_config(chunk_store, config)?;
>> + datastore.cached_config_tag = Some(CachedDatastoreConfigTag {
>> + last_maintenance_mode: maintenance_mode,
>> + last_generation: gen_num,
>> + });
>
> this part should be in with_store_and_config, which should get the
> last_generation (and later last_update) as parameter(s), just like it
> had the digest before this patch..
>
Agree, I will move this part.
>>
>> let datastore = Arc::new(datastore);
>> datastore_cache.insert(name.to_string(), datastore.clone());
>> @@ -430,11 +460,7 @@ impl DataStore {
>> config.absolute_path(),
>> tuning.sync_level.unwrap_or_default(),
>> )?;
>> - let inner = Arc::new(Self::with_store_and_config(
>> - Arc::new(chunk_store),
>> - config,
>> - None,
>> - )?);
>> + let inner = Arc::new(Self::with_store_and_config(Arc::new(chunk_store), config)?);
>>
>> if let Some(operation) = operation {
>> update_active_operations(&name, operation, 1)?;
>> @@ -446,7 +472,6 @@ impl DataStore {
>> fn with_store_and_config(
>> chunk_store: Arc<ChunkStore>,
>> config: DataStoreConfig,
>> - last_digest: Option<[u8; 32]>,
>> ) -> Result<DataStoreImpl, Error> {
>> let mut gc_status_path = chunk_store.base_path();
>> gc_status_path.push(".gc-status");
>> @@ -506,10 +531,10 @@ impl DataStore {
>> last_gc_status: Mutex::new(gc_status),
>> verify_new: config.verify_new.unwrap_or(false),
>> chunk_order: tuning.chunk_order.unwrap_or_default(),
>> - last_digest,
>> sync_level: tuning.sync_level.unwrap_or_default(),
>> backend_config,
>> lru_store_caching,
>> + cached_config_tag: None,
>> })
>> }
>>
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* [pbs-devel] applied: [PATCH proxmox] fix #6913: auth-api: fix user ID parsing for 2-character realms
2025-11-03 16:26 15% [pbs-devel] [PATCH proxmox] fix #6913: auth-api: fix user ID parsing for 2-character realms Samuel Rufinatscha
2025-11-11 10:40 5% ` Fabian Grünbichler
@ 2025-11-14 10:34 5% ` Fabian Grünbichler
1 sibling, 0 replies; 200+ results
From: Fabian Grünbichler @ 2025-11-14 10:34 UTC (permalink / raw)
To: pbs-devel, Samuel Rufinatscha
On Mon, 03 Nov 2025 17:26:05 +0100, Samuel Rufinatscha wrote:
> PVE and PBS both allow creating realms with names of length ≥ 2.
> However, when creating a user, PBS rejected realms with 2 characters
> (e.g. `test@aa`), while PVE accepted them. This issue was reported
> in our bug tracker [1]. Since the issue appears in the underlying
> `proxmox/proxmox-auth-api` crate, also PDM userid handling is
> affected.
>
> [...]
Applied, thanks!
[1/1] fix #6913: auth-api: fix user ID parsing for 2-character realms
commit: 2ac004ec5516beb93000544bb709dccd8c48f116
Best regards,
--
Fabian Grünbichler <f.gruenbichler@proxmox.com>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* [pbs-devel] [PATCH proxmox-backup v2 3/4] partial fix #6049: datastore: use config fast-path in Drop
2025-11-14 15:05 10% [pbs-devel] [PATCH proxmox-backup v2 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
2025-11-14 15:05 17% ` [pbs-devel] [PATCH proxmox-backup v2 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
2025-11-14 15:05 12% ` [pbs-devel] [PATCH proxmox-backup v2 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
@ 2025-11-14 15:05 16% ` Samuel Rufinatscha
2025-11-14 15:05 15% ` [pbs-devel] [PATCH proxmox-backup v2 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
2025-11-20 13:07 13% ` [pbs-devel] superseded: [PATCH proxmox-backup v2 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-14 15:05 UTC (permalink / raw)
To: pbs-devel
The Drop impl of DataStore re-read datastore.cfg to decide whether
the entry should be evicted from the in-process cache (based on
maintenance mode’s clear_from_cache). During the investigation of
issue #6049 [1], a flamegraph [2] showed that the config reload in Drop
accounted for a measurable share of CPU time under load.
This patch adds the datastore config fast path to the Drop impl to
eventually avoid an expensive config reload from disk to capture
the maintenance mandate.
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-datastore/src/datastore.rs | 43 +++++++++++++++++++++++++++-------
1 file changed, 34 insertions(+), 9 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index e7748872..0fabf592 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -214,15 +214,40 @@ impl Drop for DataStore {
// remove datastore from cache iff
// - last task finished, and
// - datastore is in a maintenance mode that mandates it
- let remove_from_cache = last_task
- && pbs_config::datastore::config()
- .and_then(|(s, _)| s.lookup::<DataStoreConfig>("datastore", self.name()))
- .is_ok_and(|c| {
- c.get_maintenance_mode()
- .is_some_and(|m| m.clear_from_cache())
- });
-
- if remove_from_cache {
+
+ // first check: check if last task finished
+ if !last_task {
+ return;
+ }
+
+ let (section_config, _gen) = match datastore_section_config_cached() {
+ Ok(v) => v,
+ Err(err) => {
+ log::error!(
+ "failed to load datastore config in Drop for {} - {err}",
+ self.name()
+ );
+ return;
+ }
+ };
+
+ let datastore_cfg: DataStoreConfig =
+ match section_config.lookup("datastore", self.name()) {
+ Ok(cfg) => cfg,
+ Err(err) => {
+ log::error!(
+ "failed to look up datastore '{}' in Drop - {err}",
+ self.name()
+ );
+ return;
+ }
+ };
+
+ // second check: check maintenance mode mandate
+ if datastore_cfg
+ .get_maintenance_mode()
+ .is_some_and(|m| m.clear_from_cache())
+ {
DATASTORE_MAP.lock().unwrap().remove(self.name());
}
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 16%]
* [pbs-devel] [PATCH proxmox-backup v2 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
2025-11-14 15:05 10% [pbs-devel] [PATCH proxmox-backup v2 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
` (2 preceding siblings ...)
2025-11-14 15:05 16% ` [pbs-devel] [PATCH proxmox-backup v2 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
@ 2025-11-14 15:05 15% ` Samuel Rufinatscha
2025-11-19 13:24 5% ` Fabian Grünbichler
2025-11-20 13:07 13% ` [pbs-devel] superseded: [PATCH proxmox-backup v2 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
4 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-11-14 15:05 UTC (permalink / raw)
To: pbs-devel
The lookup fast path reacts to API-driven config changes because
save_config() bumps the generation. Manual edits of datastore.cfg do
not bump the counter. To keep the system robust against such edits
without reintroducing config reading and hashing on the hot path, this
patch adds a TTL to the cache entry.
If the cached config is older than
DATASTORE_CONFIG_CACHE_TTL_SECS (set to 60s), the next lookup takes
the slow path and refreshes the cached entry. Within
the TTL window, unchanged generations still use the fast path.
Links
[1] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Refs: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-datastore/src/datastore.rs | 46 +++++++++++++++++++++++++---------
1 file changed, 34 insertions(+), 12 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 0fabf592..7a18435c 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -22,7 +22,7 @@ use proxmox_sys::error::SysError;
use proxmox_sys::fs::{file_read_optional_string, replace_file, CreateOptions};
use proxmox_sys::linux::procfs::MountInfo;
use proxmox_sys::process_locker::{ProcessLockExclusiveGuard, ProcessLockSharedGuard};
-use proxmox_time::TimeSpan;
+use proxmox_time::{epoch_i64, TimeSpan};
use proxmox_worker_task::WorkerTaskContext;
use pbs_api_types::{
@@ -53,6 +53,8 @@ struct DatastoreConfigCache {
config: Arc<SectionConfigData>,
// Generation number from ConfigVersionCache
last_generation: usize,
+ // Last update time (epoch seconds)
+ last_update: i64,
}
static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
@@ -61,6 +63,8 @@ static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
+/// Max age in seconds to reuse the cached datastore config.
+const DATASTORE_CONFIG_CACHE_TTL_SECS: i64 = 60;
/// Filename to store backup group notes
pub const GROUP_NOTES_FILE_NAME: &str = "notes";
/// Filename to store backup group owner
@@ -295,16 +299,22 @@ impl DatastoreBackend {
/// Return the cached datastore SectionConfig and its generation.
fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
- let gen = ConfigVersionCache::new()
- .ok()
- .map(|c| c.datastore_generation());
+ let now = epoch_i64();
+ let version_cache = ConfigVersionCache::new().ok();
+ let current_gen = version_cache.as_ref().map(|c| c.datastore_generation());
let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
- // Fast path: re-use cached datastore.cfg
- if let (Some(gen), Some(cache)) = (gen, guard.as_ref()) {
- if cache.last_generation == gen {
- return Ok((cache.config.clone(), Some(gen)));
+ // Fast path: re-use cached datastore.cfg if cache is available, generation matches and TTL not expired
+ if let (Some(current_gen), Some(config_cache)) = (current_gen, guard.as_ref()) {
+ let gen_matches = config_cache.last_generation == current_gen;
+ let ttl_ok = (now - config_cache.last_update) < DATASTORE_CONFIG_CACHE_TTL_SECS;
+
+ if gen_matches && ttl_ok {
+ return Ok((
+ config_cache.config.clone(),
+ Some(config_cache.last_generation),
+ ));
}
}
@@ -312,16 +322,28 @@ fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<u
let (config_raw, _digest) = pbs_config::datastore::config()?;
let config = Arc::new(config_raw);
- if let Some(gen_val) = gen {
+ // Update cache
+ let new_gen = if let Some(handle) = version_cache {
+ // Bump datastore generation whenever we reload the config.
+ // This ensures that Drop handlers will detect that a newer config exists
+ // and will not rely on a stale cached entry for maintenance mandate.
+ let prev_gen = handle.increase_datastore_generation();
+ let new_gen = prev_gen + 1;
+
*guard = Some(DatastoreConfigCache {
config: config.clone(),
- last_generation: gen_val,
+ last_generation: new_gen,
+ last_update: now,
});
+
+ Some(new_gen)
} else {
+ // if the cache was not available, use again the slow path next time
*guard = None;
- }
+ None
+ };
- Ok((config, gen))
+ Ok((config, new_gen))
}
impl DataStore {
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox-backup v2 1/4] partial fix #6049: config: enable config version cache for datastore
2025-11-14 15:05 10% [pbs-devel] [PATCH proxmox-backup v2 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
@ 2025-11-14 15:05 17% ` Samuel Rufinatscha
2025-11-14 15:05 12% ` [pbs-devel] [PATCH proxmox-backup v2 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
` (3 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-14 15:05 UTC (permalink / raw)
To: pbs-devel
Repeated /status requests caused lookup_datastore() to re-read and
parse datastore.cfg on every call. The issue was mentioned in report
#6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
dominated by pbs_config::datastore::config() (config parsing).
To solve the issue, this patch prepares the config version cache,
so that datastore config caching can be built on top of it.
This patch specifically:
(1) implements increment function in order to invalidate generations
(2) removes obsolete comments
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-config/src/config_version_cache.rs | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/pbs-config/src/config_version_cache.rs b/pbs-config/src/config_version_cache.rs
index e8fb994f..b875f7e0 100644
--- a/pbs-config/src/config_version_cache.rs
+++ b/pbs-config/src/config_version_cache.rs
@@ -26,7 +26,6 @@ struct ConfigVersionCacheDataInner {
// Traffic control (traffic-control.cfg) generation/version.
traffic_control_generation: AtomicUsize,
// datastore (datastore.cfg) generation/version
- // FIXME: remove with PBS 3.0
datastore_generation: AtomicUsize,
// Add further atomics here
}
@@ -145,8 +144,15 @@ impl ConfigVersionCache {
.fetch_add(1, Ordering::AcqRel);
}
+ /// Returns the datastore generation number.
+ pub fn datastore_generation(&self) -> usize {
+ self.shmem
+ .data()
+ .datastore_generation
+ .load(Ordering::Acquire)
+ }
+
/// Increase the datastore generation number.
- // FIXME: remove with PBS 3.0 or make actually useful again in datastore lookup
pub fn increase_datastore_generation(&self) -> usize {
self.shmem
.data()
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 17%]
* [pbs-devel] [PATCH proxmox-backup v2 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
2025-11-14 15:05 10% [pbs-devel] [PATCH proxmox-backup v2 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
2025-11-14 15:05 17% ` [pbs-devel] [PATCH proxmox-backup v2 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
@ 2025-11-14 15:05 12% ` Samuel Rufinatscha
2025-11-19 13:24 5% ` Fabian Grünbichler
2025-11-14 15:05 16% ` [pbs-devel] [PATCH proxmox-backup v2 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
` (2 subsequent siblings)
4 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-11-14 15:05 UTC (permalink / raw)
To: pbs-devel
Repeated /status requests caused lookup_datastore() to re-read and
parse datastore.cfg on every call. The issue was mentioned in report
#6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
dominated by pbs_config::datastore::config() (config parsing).
This patch implements caching of the global datastore.cfg using the
generation numbers from the shared config version cache. It caches the
datastore.cfg along with the generation number and, when a subsequent
lookup sees the same generation, it reuses the cached config without
re-reading it from disk. If the generation differs
(or the cache is unavailable), it falls back to the existing slow path
with no behavioral changes.
Behavioral notes
- The generation is bumped via the existing save_config() path, so
API-driven config changes are detected immediately.
- Manual edits to datastore.cfg are not detected; a TTL
guard is introduced in a dedicated patch in this series.
- DataStore::drop still performs a config read on the common path,
this is covered in a dedicated patch in this series.
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-datastore/Cargo.toml | 1 +
pbs-datastore/src/datastore.rs | 120 +++++++++++++++++++++++----------
2 files changed, 87 insertions(+), 34 deletions(-)
diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index 8ce930a9..42f49a7b 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -40,6 +40,7 @@ proxmox-io.workspace = true
proxmox-lang.workspace=true
proxmox-s3-client = { workspace = true, features = [ "impl" ] }
proxmox-schema = { workspace = true, features = [ "api-macro" ] }
+proxmox-section-config.workspace = true
proxmox-serde = { workspace = true, features = [ "serde_json" ] }
proxmox-sys.workspace = true
proxmox-systemd.workspace = true
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 031fa958..e7748872 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -32,7 +32,8 @@ use pbs_api_types::{
MaintenanceType, Operation, UPID,
};
use pbs_config::s3::S3_CFG_TYPE_ID;
-use pbs_config::BackupLockGuard;
+use pbs_config::{BackupLockGuard, ConfigVersionCache};
+use proxmox_section_config::SectionConfigData;
use crate::backup_info::{
BackupDir, BackupGroup, BackupInfo, OLD_LOCKING, PROTECTED_MARKER_FILENAME,
@@ -46,6 +47,17 @@ use crate::s3::S3_CONTENT_PREFIX;
use crate::task_tracking::{self, update_active_operations};
use crate::{DataBlob, LocalDatastoreLruCache};
+// Cache for fully parsed datastore.cfg
+struct DatastoreConfigCache {
+ // Parsed datastore.cfg file
+ config: Arc<SectionConfigData>,
+ // Generation number from ConfigVersionCache
+ last_generation: usize,
+}
+
+static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
+ LazyLock::new(|| Mutex::new(None));
+
static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
@@ -140,10 +152,12 @@ pub struct DataStoreImpl {
last_gc_status: Mutex<GarbageCollectionStatus>,
verify_new: bool,
chunk_order: ChunkOrder,
- last_digest: Option<[u8; 32]>,
sync_level: DatastoreFSyncLevel,
backend_config: DatastoreBackendConfig,
lru_store_caching: Option<LocalDatastoreLruCache>,
+ /// Datastore generation number from `ConfigVersionCache` at creation time, used to
+ /// validate reuse of this cached `DataStoreImpl`.
+ config_generation: Option<usize>,
}
impl DataStoreImpl {
@@ -156,10 +170,10 @@ impl DataStoreImpl {
last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
verify_new: false,
chunk_order: Default::default(),
- last_digest: None,
sync_level: Default::default(),
backend_config: Default::default(),
lru_store_caching: None,
+ config_generation: None,
})
}
}
@@ -254,6 +268,37 @@ impl DatastoreBackend {
}
}
+/// Return the cached datastore SectionConfig and its generation.
+fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
+ let gen = ConfigVersionCache::new()
+ .ok()
+ .map(|c| c.datastore_generation());
+
+ let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
+
+ // Fast path: re-use cached datastore.cfg
+ if let (Some(gen), Some(cache)) = (gen, guard.as_ref()) {
+ if cache.last_generation == gen {
+ return Ok((cache.config.clone(), Some(gen)));
+ }
+ }
+
+ // Slow path: re-read datastore.cfg
+ let (config_raw, _digest) = pbs_config::datastore::config()?;
+ let config = Arc::new(config_raw);
+
+ if let Some(gen_val) = gen {
+ *guard = Some(DatastoreConfigCache {
+ config: config.clone(),
+ last_generation: gen_val,
+ });
+ } else {
+ *guard = None;
+ }
+
+ Ok((config, gen))
+}
+
impl DataStore {
// This one just panics on everything
#[doc(hidden)]
@@ -325,56 +370,63 @@ impl DataStore {
name: &str,
operation: Option<Operation>,
) -> Result<Arc<DataStore>, Error> {
- // Avoid TOCTOU between checking maintenance mode and updating active operation counter, as
- // we use it to decide whether it is okay to delete the datastore.
+ // Avoid TOCTOU between checking maintenance mode and updating active operations.
let _config_lock = pbs_config::datastore::lock_config()?;
- // we could use the ConfigVersionCache's generation for staleness detection, but we load
- // the config anyway -> just use digest, additional benefit: manual changes get detected
- let (config, digest) = pbs_config::datastore::config()?;
- let config: DataStoreConfig = config.lookup("datastore", name)?;
+ // Get the current datastore.cfg generation number and cached config
+ let (section_config, gen_num) = datastore_section_config_cached()?;
+
+ let datastore_cfg: DataStoreConfig = section_config.lookup("datastore", name)?;
+ let maintenance_mode = datastore_cfg.get_maintenance_mode();
+ let mount_status = get_datastore_mount_status(&datastore_cfg);
- if let Some(maintenance_mode) = config.get_maintenance_mode() {
- if let Err(error) = maintenance_mode.check(operation) {
+ if let Some(mm) = &maintenance_mode {
+ if let Err(error) = mm.check(operation.clone()) {
bail!("datastore '{name}' is unavailable: {error}");
}
}
- if get_datastore_mount_status(&config) == Some(false) {
- let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
- datastore_cache.remove(&config.name);
- bail!("datastore '{}' is not mounted", config.name);
+ let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
+
+ if mount_status == Some(false) {
+ datastore_cache.remove(&datastore_cfg.name);
+ bail!("datastore '{}' is not mounted", datastore_cfg.name);
}
- let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
- let entry = datastore_cache.get(name);
-
- // reuse chunk store so that we keep using the same process locker instance!
- let chunk_store = if let Some(datastore) = &entry {
- let last_digest = datastore.last_digest.as_ref();
- if let Some(true) = last_digest.map(|last_digest| last_digest == &digest) {
- if let Some(operation) = operation {
- update_active_operations(name, operation, 1)?;
+ // Re-use DataStoreImpl
+ if let Some(existing) = datastore_cache.get(name).cloned() {
+ if let (Some(last_generation), Some(gen_num)) = (existing.config_generation, gen_num) {
+ if last_generation == gen_num {
+ if let Some(op) = operation {
+ update_active_operations(name, op, 1)?;
+ }
+
+ return Ok(Arc::new(Self {
+ inner: existing,
+ operation,
+ }));
}
- return Ok(Arc::new(Self {
- inner: Arc::clone(datastore),
- operation,
- }));
}
- Arc::clone(&datastore.chunk_store)
+ }
+
+ // (Re)build DataStoreImpl
+
+ // Reuse chunk store so that we keep using the same process locker instance!
+ let chunk_store = if let Some(existing) = datastore_cache.get(name) {
+ Arc::clone(&existing.chunk_store)
} else {
let tuning: DatastoreTuning = serde_json::from_value(
DatastoreTuning::API_SCHEMA
- .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
+ .parse_property_string(datastore_cfg.tuning.as_deref().unwrap_or(""))?,
)?;
Arc::new(ChunkStore::open(
name,
- config.absolute_path(),
+ datastore_cfg.absolute_path(),
tuning.sync_level.unwrap_or_default(),
)?)
};
- let datastore = DataStore::with_store_and_config(chunk_store, config, Some(digest))?;
+ let datastore = DataStore::with_store_and_config(chunk_store, datastore_cfg, gen_num)?;
let datastore = Arc::new(datastore);
datastore_cache.insert(name.to_string(), datastore.clone());
@@ -476,7 +528,7 @@ impl DataStore {
fn with_store_and_config(
chunk_store: Arc<ChunkStore>,
config: DataStoreConfig,
- last_digest: Option<[u8; 32]>,
+ generation: Option<usize>,
) -> Result<DataStoreImpl, Error> {
let mut gc_status_path = chunk_store.base_path();
gc_status_path.push(".gc-status");
@@ -536,10 +588,10 @@ impl DataStore {
last_gc_status: Mutex::new(gc_status),
verify_new: config.verify_new.unwrap_or(false),
chunk_order: tuning.chunk_order.unwrap_or_default(),
- last_digest,
sync_level: tuning.sync_level.unwrap_or_default(),
backend_config,
lru_store_caching,
+ config_generation: generation,
})
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] [PATCH proxmox-backup v2 0/4] datastore: remove config reload on hot path
@ 2025-11-14 15:05 10% Samuel Rufinatscha
2025-11-14 15:05 17% ` [pbs-devel] [PATCH proxmox-backup v2 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
` (4 more replies)
0 siblings, 5 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-14 15:05 UTC (permalink / raw)
To: pbs-devel
Hi,
this series reduces CPU time in datastore lookups by avoiding repeated
datastore.cfg reads/parses in both `lookup_datastore()` and
`DataStore::Drop`. It also adds a TTL so manual config edits are
noticed without reintroducing hashing on every request.
While investigating #6049 [1], cargo-flamegraph [2] showed hotspots during
repeated `/status` calls in `lookup_datastore()` and in `Drop`,
dominated by `pbs_config::datastore::config()` (config parse).
The parsing cost itself should eventually be investigated in a future
effort. Furthermore, cargo-flamegraph showed that when using a
token-based auth method to access the API, a significant amount of time
is spent in validation on every request request [3].
## Approach
[PATCH 1/4] Extend ConfigVersionCache for datastore generation
Expose a dedicated datastore generation counter and an increment
helper so callers can cheaply track datastore.cfg versions.
[PATCH 2/4] Fast path for datastore lookups
Cache the parsed datastore.cfg keyed by the shared datastore
generation. lookup_datastore() reuses both the cached config and an
existing DataStoreImpl when the generation matches, and falls back
to the old slow path otherwise.
[PATCH 3/4] Fast path for Drop
Make DataStore::Drop use the cached config if possible instead of
rereading datastore.cfg from disk.
[PATCH 4/4] TTL to catch manual edits
Add a small TTL around the cached config and bump the datastore
generation whenever the config is reloaded. This catches manual
edits to datastore.cfg without reintroducing hashing or
config parsing on every request.
## Benchmark results
All the following benchmarks are based on top of
https://lore.proxmox.com/pbs-devel/20251112131525.645971-1-f.gruenbichler@proxmox.com/T/#u
### End-to-end
Testing `/status?verbose=0` end-to-end with 1000 stores, 5 req/store
and parallel=16 before/after the series:
Metric Before After
----------------------------------------
Total time 12s 9s
Throughput (all) 416.67 555.56
Cold RPS (round #1) 83.33 111.11
Warm RPS (#2..N) 333.33 444.44
Running under flamegraph [2], TLS appears to consume a significant
amount of CPU time and blur the results. Still, a ~33% higher overall
throughput and ~25% less end-to-end time for this workload.
### Isolated benchmarks (hyperfine)
In addition to the end-to-end tests, I measured two standalone benchmarks
with hyperfine, each using a config with 1000
datastores. `M` is the number of distinct datastores looked up and
`N` is the number of lookups per datastore.
Drop-direct variant:
Drops the `DataStore` after every lookup, so the `Drop` path runs on
every iteration:
use anyhow::Error;
use pbs_api_types::Operation;
use pbs_datastore::DataStore;
fn main() -> Result<(), Error> {
let mut args = std::env::args();
args.next();
let datastores = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
let iterations = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
for d in 1..=datastores {
let name = format!("ds{:04}", d);
for i in 1..=iterations {
DataStore::lookup_datastore(&name, Some(Operation::Write))?;
}
}
Ok(())
}
+----+------+-----------+-----------+---------+
| M | N | Baseline | Patched | Speedup |
+----+------+-----------+-----------+---------+
| 1 | 1000 | 1.670 s | 34.3 ms | 48.7x |
| 10 | 100 | 1.672 s | 34.5 ms | 48.4x |
| 100| 10 | 1.679 s | 35.1 ms | 47.8x |
|1000| 1 | 1.787 s | 38.2 ms | 46.8x |
+----+------+-----------+-----------+---------+
Bulk-drop variant:
Keeps the `DataStore` instances alive for
all `N` lookups of a given datastore and then drops them in bulk,
mimicking a task that performs many lookups while it is running and
only triggers the expensive `Drop` logic when the last user exits.
use anyhow::Error;
use pbs_api_types::Operation;
use pbs_datastore::DataStore;
fn main() -> Result<(), Error> {
let mut args = std::env::args();
args.next();
let datastores = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
let iterations = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
for d in 1..=datastores {
let name = format!("ds{:04}", d);
let mut stores = Vec::with_capacity(iterations);
for i in 1..=iterations {
stores.push(DataStore::lookup_datastore(&name, Some(Operation::Write))?);
}
}
Ok(())
}
+------+------+---------------+--------------+---------+
| M | N | Baseline mean | Patched mean | Speedup |
+------+------+---------------+--------------+---------+
| 1 | 1000 | 884.0 ms | 33.9 ms | 26.1x |
| 10 | 100 | 881.8 ms | 35.3 ms | 25.0x |
| 100 | 10 | 969.3 ms | 35.9 ms | 27.0x |
| 1000 | 1 | 1827.0 ms | 40.7 ms | 44.9x |
+------+------+---------------+--------------+---------+
Both variants show that the combination of the cached config lookups
and the cheaper `Drop` handling reduces the hot-path cost from ~1.7 s
per run to a few tens of milliseconds in these benchmarks.
## Reproduction steps
VM: 4 vCPU, ~8 GiB RAM, VirtIO-SCSI; disks:
- scsi0 32G (OS)
- scsi1 1000G (datastores)
Install PBS from ISO on the VM.
Set up ZFS on /dev/sdb (adjust if different):
zpool create -f -o ashift=12 pbsbench /dev/sdb
zfs set mountpoint=/pbsbench pbsbench
zfs create pbsbench/pbs-bench
Raise file-descriptor limit:
sudo systemctl edit proxmox-backup-proxy.service
Add the following lines:
[Service]
LimitNOFILE=1048576
Reload systemd and restart the proxy:
sudo systemctl daemon-reload
sudo systemctl restart proxmox-backup-proxy.service
Verify the limit:
systemctl show proxmox-backup-proxy.service | grep LimitNOFILE
Create 1000 ZFS-backed datastores (as used in #6049 [1]):
seq -w 001 1000 | xargs -n1 -P1 bash -c '
id=$0
name="ds${id}"
dataset="pbsbench/pbs-bench/${name}"
path="/pbsbench/pbs-bench/${name}"
zfs create -o mountpoint="$path" "$dataset"
proxmox-backup-manager datastore create "$name" "$path" \
--comment "ZFS dataset-based datastore"
'
Build PBS from this series, then run the server under manually
under flamegraph:
systemctl stop proxmox-backup-proxy
cargo flamegraph --release --bin proxmox-backup-proxy
## Other resources:
### E2E benchmark script:
#!/usr/bin/env bash
set -euo pipefail
# --- Config ---------------------------------------------------------------
HOST='https://localhost:8007'
USER='root@pam'
PASS="$(cat passfile)"
DATASTORE_PATH="/pbsbench/pbs-bench"
MAX_STORES=1000 # how many stores to include
PARALLEL=16 # concurrent workers
REPEAT=5 # requests per store (1 cold + REPEAT-1 warm)
PRINT_FIRST=false # true => log first request's HTTP code per store
# --- Helpers --------------------------------------------------------------
fmt_rps () {
local n="$1" t="$2"
awk -v n="$n" -v t="$t" 'BEGIN { if (t > 0) printf("%.2f\n", n/t); else print "0.00" }'
}
# --- Login ---------------------------------------------------------------
auth=$(curl -ks -X POST "$HOST/api2/json/access/ticket" \
-d "username=$USER" -d "password=$PASS")
ticket=$(echo "$auth" | jq -r '.data.ticket')
if [[ -z "${ticket:-}" || "$ticket" == "null" ]]; then
echo "[ERROR] Login failed (no ticket)"
exit 1
fi
# --- Collect stores (deterministic order) --------------------------------
mapfile -t STORES < <(
find "$DATASTORE_PATH" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' \
| sort | head -n "$MAX_STORES"
)
USED_STORES=${#STORES[@]}
if (( USED_STORES == 0 )); then
echo "[ERROR] No datastore dirs under $DATASTORE_PATH"
exit 1
fi
echo "[INFO] Running with stores=$USED_STORES, repeat=$REPEAT, parallel=$PARALLEL"
# --- Temp counters --------------------------------------------------------
SUCCESS_ALL="$(mktemp)"
FAIL_ALL="$(mktemp)"
COLD_OK="$(mktemp)"
WARM_OK="$(mktemp)"
trap 'rm -f "$SUCCESS_ALL" "$FAIL_ALL" "$COLD_OK" "$WARM_OK"' EXIT
export HOST ticket REPEAT SUCCESS_ALL FAIL_ALL COLD_OK WARM_OK PRINT_FIRST
SECONDS=0
# --- Fire requests --------------------------------------------------------
printf "%s\n" "${STORES[@]}" \
| xargs -P"$PARALLEL" -I{} bash -c '
store="$1"
url="$HOST/api2/json/admin/datastore/$store/status?verbose=0"
for ((i=1;i<=REPEAT;i++)); do
code=$(curl -ks -o /dev/null -w "%{http_code}" -b "PBSAuthCookie=$ticket" "$url" || echo 000)
if [[ "$code" == "200" ]]; then
echo 1 >> "$SUCCESS_ALL"
if (( i == 1 )); then
echo 1 >> "$COLD_OK"
else
echo 1 >> "$WARM_OK"
fi
if [[ "$PRINT_FIRST" == "true" && $i -eq 1 ]]; then
ts=$(date +%H:%M:%S)
echo "[$ts] $store #$i HTTP:200"
fi
else
echo 1 >> "$FAIL_ALL"
if [[ "$PRINT_FIRST" == "true" && $i -eq 1 ]]; then
ts=$(date +%H:%M:%S)
echo "[$ts] $store #$i HTTP:$code (FAIL)"
fi
fi
done
' _ {}
# --- Summary --------------------------------------------------------------
elapsed=$SECONDS
ok=$(wc -l < "$SUCCESS_ALL" 2>/dev/null || echo 0)
fail=$(wc -l < "$FAIL_ALL" 2>/dev/null || echo 0)
cold_ok=$(wc -l < "$COLD_OK" 2>/dev/null || echo 0)
warm_ok=$(wc -l < "$WARM_OK" 2>/dev/null || echo 0)
expected=$(( USED_STORES * REPEAT ))
total=$(( ok + fail ))
rps_all=$(fmt_rps "$ok" "$elapsed")
rps_cold=$(fmt_rps "$cold_ok" "$elapsed")
rps_warm=$(fmt_rps "$warm_ok" "$elapsed")
echo "===== Summary ====="
echo "Stores used: $USED_STORES"
echo "Expected requests: $expected"
echo "Executed requests: $total"
echo "OK (HTTP 200): $ok"
echo "Failed: $fail"
printf "Total time: %dm %ds\n" $((elapsed/60)) $((elapsed%60))
echo "Throughput all RPS: $rps_all"
echo "Cold RPS (round #1): $rps_cold"
echo "Warm RPS (#2..N): $rps_warm"
## Maintainer notes
No dependency bumps, no API changes and no breaking changes.
## Patch summary
[PATCH 1/4] partial fix #6049: config: enable config version cache for datastore
[PATCH 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
[PATCH 3/4] partial fix #6049: datastore: use config fast-path in Drop
[PATCH 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
Thanks,
Samuel
[1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
[3] Bugzilla #7017: https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Samuel Rufinatscha (4):
partial fix #6049: config: enable config version cache for datastore
partial fix #6049: datastore: impl ConfigVersionCache fast path for
lookups
partial fix #6049: datastore: use config fast-path in Drop
partial fix #6049: datastore: add TTL fallback to catch manual config
edits
pbs-config/src/config_version_cache.rs | 10 +-
pbs-datastore/Cargo.toml | 1 +
pbs-datastore/src/datastore.rs | 187 +++++++++++++++++++------
3 files changed, 152 insertions(+), 46 deletions(-)
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 10%]
* [pbs-devel] superseded: [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path
2025-11-11 12:29 11% [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path Samuel Rufinatscha
` (3 preceding siblings ...)
2025-11-12 11:27 5% ` [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path Fabian Grünbichler
@ 2025-11-14 15:08 13% ` Samuel Rufinatscha
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-14 15:08 UTC (permalink / raw)
To: pbs-devel
https://lore.proxmox.com/pbs-devel/20251114150544.224839-1-s.rufinatscha@proxmox.com/T/#t
On 11/11/25 1:29 PM, Samuel Rufinatscha wrote:
> Hi,
>
> this series reduces CPU time in datastore lookups by avoiding repeated
> datastore.cfg reads/parses in both `lookup_datastore()` and
> `DataStore::Drop`. It also adds a TTL so manual config edits are
> noticed without reintroducing hashing on every request.
>
> While investigating #6049 [1], cargo-flamegraph [2] showed hotspots during
> repeated `/status` calls in `lookup_datastore()` and in `Drop`,
> dominated by `pbs_config::datastore::config()` (config parse).
>
> The parsing cost itself should eventually be investigated in a future
> effort. Furthermore, cargo-flamegraph showed that when using a
> token-based auth method to access the API, a significant amount of time
> is spent in validation on every request, likely related to bcrypt.
> Also this should be eventually revisited in a future effort.
>
> ## Approach
>
> [PATCH 1/3] Fast path for datastore lookups
> Use the shared-memory `ConfigVersionCache` generation for `datastore.cfg`.
> Tag each cached `DataStoreImpl` with the last seen generation; when it
> matches, reuse the cached instance. Fall back to the existing slow path
> on mismatch or when the cache is unavailable.
>
> [PATCH 2/3] Fast path for `Drop`
> Reuse the maintenance mode eviction decision captured at lookup time,
> removing the config reload from `Drop`.
>
> [PATCH 3/3] TTL to catch manual edits
> If a cached entry is older than `DATASTORE_CONFIG_CACHE_TTL_SECS`
> (default 60s), the next lookup refreshes it via the slow path. This
> detects manual file edits without hashing on every request.
>
> ## Results
>
> End-to-end `/status?verbose=0` (1000 stores, 5 req/store, parallel=16):
>
> Metric Baseline [1/3] [2/3]
> ------------------------------------------------
> Total time 13s 11s 10s
> Throughput (all) 384.62 454.55 500.00
> Cold RPS (round #1) 76.92 90.91 100.00
> Warm RPS (2..N) 307.69 363.64 400.00
>
> Patch 1 improves overall throughput by ~18% (−15% total time). Patch 2
> adds ~10% on top. Patch 3 is a robustness feature; a 0.1 s probe shows
> periodic latency spikes at TTL expiry and flat latencies otherwise.
>
> ## Reproduction steps
>
> VM: 4 vCPU, ~8 GiB RAM, VirtIO-SCSI; disks:
> - scsi0 32G (OS)
> - scsi1 1000G (datastores)
>
> Install PBS from ISO on the VM.
>
> Set up ZFS on /dev/sdb (adjust if different):
>
> zpool create -f -o ashift=12 pbsbench /dev/sdb
> zfs set mountpoint=/pbsbench pbsbench
> zfs create pbsbench/pbs-bench
>
> Raise file-descriptor limit:
>
> sudo systemctl edit proxmox-backup-proxy.service
>
> Add the following lines:
>
> [Service]
> LimitNOFILE=1048576
>
> Reload systemd and restart the proxy:
>
> sudo systemctl daemon-reload
> sudo systemctl restart proxmox-backup-proxy.service
>
> Verify the limit:
>
> systemctl show proxmox-backup-proxy.service | grep LimitNOFILE
>
> Create 1000 ZFS-backed datastores (as used in #6049 [1]):
>
> seq -w 001 1000 | xargs -n1 -P1 bash -c '
> id=$0
> name="ds${id}"
> dataset="pbsbench/pbs-bench/${name}"
> path="/pbsbench/pbs-bench/${name}"
> zfs create -o mountpoint="$path" "$dataset"
> proxmox-backup-manager datastore create "$name" "$path" \
> --comment "ZFS dataset-based datastore"
> '
>
> Build PBS from this series, then run the server under manually
> under flamegraph:
>
> systemctl stop proxmox-backup-proxy
> cargo flamegraph --release --bin proxmox-backup-proxy
>
> Benchmark script (`bench.sh`) used for the numbers above:
>
> #!/usr/bin/env bash
> set -euo pipefail
>
> # --- Config ---------------------------------------------------------------
> HOST='https://localhost:8007'
> USER='root@pam'
> PASS="$(cat passfile)"
>
> DATASTORE_PATH="/pbsbench/pbs-bench"
> MAX_STORES=1000 # how many stores to include
> PARALLEL=16 # concurrent workers
> REPEAT=5 # requests per store (1 cold + REPEAT-1 warm)
>
> PRINT_FIRST=false # true => log first request's HTTP code per store
>
> # --- Helpers --------------------------------------------------------------
> fmt_rps () {
> local n="$1" t="$2"
> awk -v n="$n" -v t="$t" 'BEGIN { if (t > 0) printf("%.2f\n", n/t); else print "0.00" }'
> }
>
> # --- Login ---------------------------------------------------------------
> auth=$(curl -ks -X POST "$HOST/api2/json/access/ticket" \
> -d "username=$USER" -d "password=$PASS")
> ticket=$(echo "$auth" | jq -r '.data.ticket')
>
> if [[ -z "${ticket:-}" || "$ticket" == "null" ]]; then
> echo "[ERROR] Login failed (no ticket)"
> exit 1
> fi
>
> # --- Collect stores (deterministic order) --------------------------------
> mapfile -t STORES < <(
> find "$DATASTORE_PATH" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' \
> | sort | head -n "$MAX_STORES"
> )
>
> USED_STORES=${#STORES[@]}
> if (( USED_STORES == 0 )); then
> echo "[ERROR] No datastore dirs under $DATASTORE_PATH"
> exit 1
> fi
>
> echo "[INFO] Running with stores=$USED_STORES, repeat=$REPEAT, parallel=$PARALLEL"
>
> # --- Temp counters --------------------------------------------------------
> SUCCESS_ALL="$(mktemp)"
> FAIL_ALL="$(mktemp)"
> COLD_OK="$(mktemp)"
> WARM_OK="$(mktemp)"
> trap 'rm -f "$SUCCESS_ALL" "$FAIL_ALL" "$COLD_OK" "$WARM_OK"' EXIT
>
> export HOST ticket REPEAT SUCCESS_ALL FAIL_ALL COLD_OK WARM_OK PRINT_FIRST
>
> SECONDS=0
>
> # --- Fire requests --------------------------------------------------------
> printf "%s\n" "${STORES[@]}" \
> | xargs -P"$PARALLEL" -I{} bash -c '
> store="$1"
> url="$HOST/api2/json/admin/datastore/$store/status?verbose=0"
>
> for ((i=1;i<=REPEAT;i++)); do
> code=$(curl -ks -o /dev/null -w "%{http_code}" -b "PBSAuthCookie=$ticket" "$url" || echo 000)
>
> if [[ "$code" == "200" ]]; then
> echo 1 >> "$SUCCESS_ALL"
> if (( i == 1 )); then
> echo 1 >> "$COLD_OK"
> else
> echo 1 >> "$WARM_OK"
> fi
> if [[ "$PRINT_FIRST" == "true" && $i -eq 1 ]]; then
> ts=$(date +%H:%M:%S)
> echo "[$ts] $store #$i HTTP:200"
> fi
> else
> echo 1 >> "$FAIL_ALL"
> if [[ "$PRINT_FIRST" == "true" && $i -eq 1 ]]; then
> ts=$(date +%H:%M:%S)
> echo "[$ts] $store #$i HTTP:$code (FAIL)"
> fi
> fi
> done
> ' _ {}
>
> # --- Summary --------------------------------------------------------------
> elapsed=$SECONDS
> ok=$(wc -l < "$SUCCESS_ALL" 2>/dev/null || echo 0)
> fail=$(wc -l < "$FAIL_ALL" 2>/dev/null || echo 0)
> cold_ok=$(wc -l < "$COLD_OK" 2>/dev/null || echo 0)
> warm_ok=$(wc -l < "$WARM_OK" 2>/dev/null || echo 0)
>
> expected=$(( USED_STORES * REPEAT ))
> total=$(( ok + fail ))
>
> rps_all=$(fmt_rps "$ok" "$elapsed")
> rps_cold=$(fmt_rps "$cold_ok" "$elapsed")
> rps_warm=$(fmt_rps "$warm_ok" "$elapsed")
>
> echo "===== Summary ====="
> echo "Stores used: $USED_STORES"
> echo "Expected requests: $expected"
> echo "Executed requests: $total"
> echo "OK (HTTP 200): $ok"
> echo "Failed: $fail"
> printf "Total time: %dm %ds\n" $((elapsed/60)) $((elapsed%60))
> echo "Throughput all RPS: $rps_all"
> echo "Cold RPS (round #1): $rps_cold"
> echo "Warm RPS (#2..N): $rps_warm"
>
> ## Maintainer notes
>
> - No dependency bumps, no API changes, no breaking changes in this
> series.
>
> ## Patch summary
>
> [PATCH 1/3] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
> [PATCH 2/3] partial fix #6049: datastore: use config fast-path in Drop
> [PATCH 3/3] datastore: add TTL fallback to catch manual config edits
>
> Thanks for reviewing!
>
> [1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>
> Samuel Rufinatscha (3):
> partial fix #6049: datastore: impl ConfigVersionCache fast path for
> lookups
> partial fix #6049: datastore: use config fast-path in Drop
> datastore: add TTL fallback to catch manual config edits
>
> pbs-config/src/config_version_cache.rs | 10 ++-
> pbs-datastore/src/datastore.rs | 119 ++++++++++++++++++-------
> 2 files changed, 96 insertions(+), 33 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* Re: [pbs-devel] [PATCH proxmox-backup] task tracking: fix adding new entry if other PID is tracked
@ 2025-11-17 8:41 13% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-17 8:41 UTC (permalink / raw)
To: pbs-devel
Looks good to me!
Reviewed-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
On 11/12/25 2:15 PM, Fabian Grünbichler wrote:
> if the tracking file contains an entry for another, still running PID, that
> entry must be preserved, but a new entry for the current PID should still be
> inserted..
>
> Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
> ---
> found while benchmarking Samuel's datastore lookup caching series..
>
> pbs-datastore/src/task_tracking.rs | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/pbs-datastore/src/task_tracking.rs b/pbs-datastore/src/task_tracking.rs
> index 77851cab6..44a4522dc 100644
> --- a/pbs-datastore/src/task_tracking.rs
> +++ b/pbs-datastore/src/task_tracking.rs
> @@ -108,6 +108,7 @@ pub fn update_active_operations(
> Operation::Write => ActiveOperationStats { read: 0, write: 1 },
> Operation::Lookup => ActiveOperationStats { read: 0, write: 0 },
> };
> + let mut found_entry = false;
> let mut updated_tasks: Vec<TaskOperations> = match file_read_optional_string(&path)? {
> Some(data) => serde_json::from_str::<Vec<TaskOperations>>(&data)?
> .iter_mut()
> @@ -116,6 +117,7 @@ pub fn update_active_operations(
> Some(stat) if pid == task.pid && stat.starttime != task.starttime => None,
> Some(_) => {
> if pid == task.pid {
> + found_entry = true;
> match operation {
> Operation::Read => task.active_operations.read += count,
> Operation::Write => task.active_operations.write += count,
> @@ -132,7 +134,7 @@ pub fn update_active_operations(
> None => Vec::new(),
> };
>
> - if updated_tasks.is_empty() {
> + if !found_entry {
> updated_tasks.push(TaskOperations {
> pid,
> starttime,
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* Re: [pbs-devel] [PATCH proxmox-backup v2 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
2025-11-14 15:05 15% ` [pbs-devel] [PATCH proxmox-backup v2 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
@ 2025-11-19 13:24 5% ` Fabian Grünbichler
2025-11-19 17:25 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-11-19 13:24 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On November 14, 2025 4:05 pm, Samuel Rufinatscha wrote:
> The lookup fast path reacts to API-driven config changes because
> save_config() bumps the generation. Manual edits of datastore.cfg do
> not bump the counter. To keep the system robust against such edits
> without reintroducing config reading and hashing on the hot path, this
> patch adds a TTL to the cache entry.
>
> If the cached config is older than
> DATASTORE_CONFIG_CACHE_TTL_SECS (set to 60s), the next lookup takes
> the slow path and refreshes the cached entry. Within
> the TTL window, unchanged generations still use the fast path.
>
> Links
>
> [1] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>
> Refs: #6049
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> pbs-datastore/src/datastore.rs | 46 +++++++++++++++++++++++++---------
> 1 file changed, 34 insertions(+), 12 deletions(-)
>
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 0fabf592..7a18435c 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -22,7 +22,7 @@ use proxmox_sys::error::SysError;
> use proxmox_sys::fs::{file_read_optional_string, replace_file, CreateOptions};
> use proxmox_sys::linux::procfs::MountInfo;
> use proxmox_sys::process_locker::{ProcessLockExclusiveGuard, ProcessLockSharedGuard};
> -use proxmox_time::TimeSpan;
> +use proxmox_time::{epoch_i64, TimeSpan};
> use proxmox_worker_task::WorkerTaskContext;
>
> use pbs_api_types::{
> @@ -53,6 +53,8 @@ struct DatastoreConfigCache {
> config: Arc<SectionConfigData>,
> // Generation number from ConfigVersionCache
> last_generation: usize,
> + // Last update time (epoch seconds)
> + last_update: i64,
> }
>
> static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
> @@ -61,6 +63,8 @@ static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
> static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
> LazyLock::new(|| Mutex::new(HashMap::new()));
>
> +/// Max age in seconds to reuse the cached datastore config.
> +const DATASTORE_CONFIG_CACHE_TTL_SECS: i64 = 60;
> /// Filename to store backup group notes
> pub const GROUP_NOTES_FILE_NAME: &str = "notes";
> /// Filename to store backup group owner
> @@ -295,16 +299,22 @@ impl DatastoreBackend {
>
> /// Return the cached datastore SectionConfig and its generation.
> fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
> - let gen = ConfigVersionCache::new()
> - .ok()
> - .map(|c| c.datastore_generation());
> + let now = epoch_i64();
> + let version_cache = ConfigVersionCache::new().ok();
> + let current_gen = version_cache.as_ref().map(|c| c.datastore_generation());
>
> let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
>
> - // Fast path: re-use cached datastore.cfg
> - if let (Some(gen), Some(cache)) = (gen, guard.as_ref()) {
> - if cache.last_generation == gen {
> - return Ok((cache.config.clone(), Some(gen)));
> + // Fast path: re-use cached datastore.cfg if cache is available, generation matches and TTL not expired
> + if let (Some(current_gen), Some(config_cache)) = (current_gen, guard.as_ref()) {
> + let gen_matches = config_cache.last_generation == current_gen;
> + let ttl_ok = (now - config_cache.last_update) < DATASTORE_CONFIG_CACHE_TTL_SECS;
> +
> + if gen_matches && ttl_ok {
> + return Ok((
> + config_cache.config.clone(),
> + Some(config_cache.last_generation),
> + ));
> }
> }
>
> @@ -312,16 +322,28 @@ fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<u
> let (config_raw, _digest) = pbs_config::datastore::config()?;
> let config = Arc::new(config_raw);
>
> - if let Some(gen_val) = gen {
> + // Update cache
> + let new_gen = if let Some(handle) = version_cache {
> + // Bump datastore generation whenever we reload the config.
> + // This ensures that Drop handlers will detect that a newer config exists
> + // and will not rely on a stale cached entry for maintenance mandate.
> + let prev_gen = handle.increase_datastore_generation();
this could be optimized (further) if we keep the digest when we
load+parse the config above, because we only need to bump the generation
if the digest changed. we need to bump the timestamp always of course ;)
also we only want to bump if we previously had a generation saved, if we
didn't, then this is the first load and bumping is meaningless anyway..
but there is another issue here - this is now called in the Drop
handler, where we don't hold the config lock, so we have no guard
against a parallel config change API call that also bumps the generation
between us reloading and us bumping here.. which means we could have a
mismatch between the value in new_gen and the actual config we loaded..
I think we need to extend this helper here with a bool flag that
determines whether we want to reload if the TTL expired, or return
potentially outdated information? *every* lookup will handle the TTL
anyway (by setting that parameter), so I think just fetching the
"freshest" info we can get without reloading (by not setting it) is fine
for the Drop handler..
> + let new_gen = prev_gen + 1;
> +
> *guard = Some(DatastoreConfigCache {
> config: config.clone(),
> - last_generation: gen_val,
> + last_generation: new_gen,
> + last_update: now,
> });
> +
> + Some(new_gen)
> } else {
> + // if the cache was not available, use again the slow path next time
> *guard = None;
> - }
> + None
> + };
>
> - Ok((config, gen))
> + Ok((config, new_gen))
> }
>
> impl DataStore {
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v2 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
2025-11-14 15:05 12% ` [pbs-devel] [PATCH proxmox-backup v2 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
@ 2025-11-19 13:24 5% ` Fabian Grünbichler
0 siblings, 0 replies; 200+ results
From: Fabian Grünbichler @ 2025-11-19 13:24 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On November 14, 2025 4:05 pm, Samuel Rufinatscha wrote:
> Repeated /status requests caused lookup_datastore() to re-read and
> parse datastore.cfg on every call. The issue was mentioned in report
> #6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
> dominated by pbs_config::datastore::config() (config parsing).
>
> This patch implements caching of the global datastore.cfg using the
> generation numbers from the shared config version cache. It caches the
> datastore.cfg along with the generation number and, when a subsequent
> lookup sees the same generation, it reuses the cached config without
> re-reading it from disk. If the generation differs
> (or the cache is unavailable), it falls back to the existing slow path
> with no behavioral changes.
>
> Behavioral notes
>
> - The generation is bumped via the existing save_config() path, so
> API-driven config changes are detected immediately.
> - Manual edits to datastore.cfg are not detected; a TTL
> guard is introduced in a dedicated patch in this series.
> - DataStore::drop still performs a config read on the common path,
> this is covered in a dedicated patch in this series.
>
> Links
>
> [1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>
> Fixes: #6049
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> pbs-datastore/Cargo.toml | 1 +
> pbs-datastore/src/datastore.rs | 120 +++++++++++++++++++++++----------
> 2 files changed, 87 insertions(+), 34 deletions(-)
>
> diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
> index 8ce930a9..42f49a7b 100644
> --- a/pbs-datastore/Cargo.toml
> +++ b/pbs-datastore/Cargo.toml
> @@ -40,6 +40,7 @@ proxmox-io.workspace = true
> proxmox-lang.workspace=true
> proxmox-s3-client = { workspace = true, features = [ "impl" ] }
> proxmox-schema = { workspace = true, features = [ "api-macro" ] }
> +proxmox-section-config.workspace = true
> proxmox-serde = { workspace = true, features = [ "serde_json" ] }
> proxmox-sys.workspace = true
> proxmox-systemd.workspace = true
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 031fa958..e7748872 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -32,7 +32,8 @@ use pbs_api_types::{
> MaintenanceType, Operation, UPID,
> };
> use pbs_config::s3::S3_CFG_TYPE_ID;
> -use pbs_config::BackupLockGuard;
> +use pbs_config::{BackupLockGuard, ConfigVersionCache};
> +use proxmox_section_config::SectionConfigData;
>
> use crate::backup_info::{
> BackupDir, BackupGroup, BackupInfo, OLD_LOCKING, PROTECTED_MARKER_FILENAME,
> @@ -46,6 +47,17 @@ use crate::s3::S3_CONTENT_PREFIX;
> use crate::task_tracking::{self, update_active_operations};
> use crate::{DataBlob, LocalDatastoreLruCache};
>
> +// Cache for fully parsed datastore.cfg
> +struct DatastoreConfigCache {
> + // Parsed datastore.cfg file
> + config: Arc<SectionConfigData>,
> + // Generation number from ConfigVersionCache
> + last_generation: usize,
> +}
> +
> +static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
> + LazyLock::new(|| Mutex::new(None));
> +
> static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
> LazyLock::new(|| Mutex::new(HashMap::new()));
>
> @@ -140,10 +152,12 @@ pub struct DataStoreImpl {
> last_gc_status: Mutex<GarbageCollectionStatus>,
> verify_new: bool,
> chunk_order: ChunkOrder,
> - last_digest: Option<[u8; 32]>,
> sync_level: DatastoreFSyncLevel,
> backend_config: DatastoreBackendConfig,
> lru_store_caching: Option<LocalDatastoreLruCache>,
> + /// Datastore generation number from `ConfigVersionCache` at creation time, used to
> + /// validate reuse of this cached `DataStoreImpl`.
> + config_generation: Option<usize>,
> }
>
> impl DataStoreImpl {
> @@ -156,10 +170,10 @@ impl DataStoreImpl {
> last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
> verify_new: false,
> chunk_order: Default::default(),
> - last_digest: None,
> sync_level: Default::default(),
> backend_config: Default::default(),
> lru_store_caching: None,
> + config_generation: None,
> })
> }
> }
> @@ -254,6 +268,37 @@ impl DatastoreBackend {
> }
> }
>
> +/// Return the cached datastore SectionConfig and its generation.
> +fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
> + let gen = ConfigVersionCache::new()
> + .ok()
> + .map(|c| c.datastore_generation());
> +
> + let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
> +
> + // Fast path: re-use cached datastore.cfg
> + if let (Some(gen), Some(cache)) = (gen, guard.as_ref()) {
> + if cache.last_generation == gen {
> + return Ok((cache.config.clone(), Some(gen)));
> + }
> + }
> +
> + // Slow path: re-read datastore.cfg
> + let (config_raw, _digest) = pbs_config::datastore::config()?;
> + let config = Arc::new(config_raw);
> +
> + if let Some(gen_val) = gen {
> + *guard = Some(DatastoreConfigCache {
> + config: config.clone(),
> + last_generation: gen_val,
> + });
> + } else {
> + *guard = None;
> + }
> +
> + Ok((config, gen))
I think this would be more readable (especially with the extensions
coming later, and with what I propose in my reply to patch #4) if
ordered like this:
let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
if let Ok(version_cache) = ConfigVersionCache::new() {
let gen = version_cache.datastore_generation();
if let Some(cached) = guard.as_ref() {
// Fast path: re-use cached datastore.cfg
if gen == cached.last_generation {
return Ok((cached.config.clone(), Some(gen)));
}
}
// Slow path: re-read datastore.cfg
let (config_raw, _digest) = pbs_config::datastore::config()?;
let config = Arc::new(config_raw);
*guard = Some(DatastoreConfigCache {
config: config.clone(),
last_generation: gen,
});
Ok((config, Some(gen)))
} else {
// Fallback path, no config version cache: read datastore.cfg
*guard = None;
let (config_raw, _digest) = pbs_config::datastore::config()?;
Ok((Arc::new(config_raw), None))
}
with the later changes it would then look like this (but this still has
the issues I mentioned in my comment to patch #4 ;)):
let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
if let Some(version_cache) = ConfigVersionCache::new().ok() {
let now = epoch_i64();
let current_gen = version_cache.datastore_generation();
if let Some(cached) = guard.as_ref() {
// Fast path: re-use cached datastore.cfg if cache is available, generation matches and TTL not expired
if cached.last_generation == current_gen
&& now - cached.last_update < DATASTORE_CONFIG_CACHE_TTL_SECS
{
return Ok((cached.config.clone(), Some(cached.last_generation)));
}
}
// Slow path: re-read datastore.cfg
let (config_raw, _digest) = pbs_config::datastore::config()?;
let config = Arc::new(config_raw);
// Bump datastore generation whenever we reload the config.
// This ensures that Drop handlers will detect that a newer config exists
// and will not rely on a stale cached entry for maintenance mandate.
let prev_gen = version_cache.increase_datastore_generation();
let new_gen = prev_gen + 1;
// Update cache
*guard = Some(DatastoreConfigCache {
config: config.clone(),
last_generation: new_gen,
last_update: now,
});
Ok((config, Some(new_gen)))
} else {
// Fallback path, no config version cache: read datastore.cfg
*guard = None;
let (config_raw, _digest) = pbs_config::datastore::config()?;
Ok((Arc::new(config_raw), None))
}
technically setting the guard to None in the else branch is not needed,
since if we ever get an Ok result back it has been initialized and
subsequent calls cannot fail..
> +}
> +
> impl DataStore {
> // This one just panics on everything
> #[doc(hidden)]
> @@ -325,56 +370,63 @@ impl DataStore {
> name: &str,
> operation: Option<Operation>,
> ) -> Result<Arc<DataStore>, Error> {
> - // Avoid TOCTOU between checking maintenance mode and updating active operation counter, as
> - // we use it to decide whether it is okay to delete the datastore.
> + // Avoid TOCTOU between checking maintenance mode and updating active operations.
> let _config_lock = pbs_config::datastore::lock_config()?;
>
> - // we could use the ConfigVersionCache's generation for staleness detection, but we load
> - // the config anyway -> just use digest, additional benefit: manual changes get detected
> - let (config, digest) = pbs_config::datastore::config()?;
> - let config: DataStoreConfig = config.lookup("datastore", name)?;
> + // Get the current datastore.cfg generation number and cached config
> + let (section_config, gen_num) = datastore_section_config_cached()?;
> +
> + let datastore_cfg: DataStoreConfig = section_config.lookup("datastore", name)?;
> + let maintenance_mode = datastore_cfg.get_maintenance_mode();
> + let mount_status = get_datastore_mount_status(&datastore_cfg);
>
> - if let Some(maintenance_mode) = config.get_maintenance_mode() {
> - if let Err(error) = maintenance_mode.check(operation) {
> + if let Some(mm) = &maintenance_mode {
> + if let Err(error) = mm.check(operation.clone()) {
> bail!("datastore '{name}' is unavailable: {error}");
> }
> }
>
> - if get_datastore_mount_status(&config) == Some(false) {
> - let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
> - datastore_cache.remove(&config.name);
> - bail!("datastore '{}' is not mounted", config.name);
> + let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
> +
> + if mount_status == Some(false) {
> + datastore_cache.remove(&datastore_cfg.name);
> + bail!("datastore '{}' is not mounted", datastore_cfg.name);
> }
>
> - let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
> - let entry = datastore_cache.get(name);
> -
> - // reuse chunk store so that we keep using the same process locker instance!
> - let chunk_store = if let Some(datastore) = &entry {
> - let last_digest = datastore.last_digest.as_ref();
> - if let Some(true) = last_digest.map(|last_digest| last_digest == &digest) {
> - if let Some(operation) = operation {
> - update_active_operations(name, operation, 1)?;
> + // Re-use DataStoreImpl
> + if let Some(existing) = datastore_cache.get(name).cloned() {
> + if let (Some(last_generation), Some(gen_num)) = (existing.config_generation, gen_num) {
> + if last_generation == gen_num {
> + if let Some(op) = operation {
> + update_active_operations(name, op, 1)?;
> + }
> +
> + return Ok(Arc::new(Self {
> + inner: existing,
> + operation,
> + }));
> }
> - return Ok(Arc::new(Self {
> - inner: Arc::clone(datastore),
> - operation,
> - }));
> }
> - Arc::clone(&datastore.chunk_store)
> + }
> +
> + // (Re)build DataStoreImpl
> +
> + // Reuse chunk store so that we keep using the same process locker instance!
> + let chunk_store = if let Some(existing) = datastore_cache.get(name) {
> + Arc::clone(&existing.chunk_store)
> } else {
> let tuning: DatastoreTuning = serde_json::from_value(
> DatastoreTuning::API_SCHEMA
> - .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
> + .parse_property_string(datastore_cfg.tuning.as_deref().unwrap_or(""))?,
> )?;
> Arc::new(ChunkStore::open(
> name,
> - config.absolute_path(),
> + datastore_cfg.absolute_path(),
> tuning.sync_level.unwrap_or_default(),
> )?)
> };
>
> - let datastore = DataStore::with_store_and_config(chunk_store, config, Some(digest))?;
> + let datastore = DataStore::with_store_and_config(chunk_store, datastore_cfg, gen_num)?;
>
> let datastore = Arc::new(datastore);
> datastore_cache.insert(name.to_string(), datastore.clone());
> @@ -476,7 +528,7 @@ impl DataStore {
> fn with_store_and_config(
> chunk_store: Arc<ChunkStore>,
> config: DataStoreConfig,
> - last_digest: Option<[u8; 32]>,
> + generation: Option<usize>,
> ) -> Result<DataStoreImpl, Error> {
> let mut gc_status_path = chunk_store.base_path();
> gc_status_path.push(".gc-status");
> @@ -536,10 +588,10 @@ impl DataStore {
> last_gc_status: Mutex::new(gc_status),
> verify_new: config.verify_new.unwrap_or(false),
> chunk_order: tuning.chunk_order.unwrap_or_default(),
> - last_digest,
> sync_level: tuning.sync_level.unwrap_or_default(),
> backend_config,
> lru_store_caching,
> + config_generation: generation,
> })
> }
>
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v2 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
2025-11-19 13:24 5% ` Fabian Grünbichler
@ 2025-11-19 17:25 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-19 17:25 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
comments inline
On 11/19/25 2:24 PM, Fabian Grünbichler wrote:
> On November 14, 2025 4:05 pm, Samuel Rufinatscha wrote:
>> The lookup fast path reacts to API-driven config changes because
>> save_config() bumps the generation. Manual edits of datastore.cfg do
>> not bump the counter. To keep the system robust against such edits
>> without reintroducing config reading and hashing on the hot path, this
>> patch adds a TTL to the cache entry.
>>
>> If the cached config is older than
>> DATASTORE_CONFIG_CACHE_TTL_SECS (set to 60s), the next lookup takes
>> the slow path and refreshes the cached entry. Within
>> the TTL window, unchanged generations still use the fast path.
>>
>> Links
>>
>> [1] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>>
>> Refs: #6049
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> pbs-datastore/src/datastore.rs | 46 +++++++++++++++++++++++++---------
>> 1 file changed, 34 insertions(+), 12 deletions(-)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index 0fabf592..7a18435c 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -22,7 +22,7 @@ use proxmox_sys::error::SysError;
>> use proxmox_sys::fs::{file_read_optional_string, replace_file, CreateOptions};
>> use proxmox_sys::linux::procfs::MountInfo;
>> use proxmox_sys::process_locker::{ProcessLockExclusiveGuard, ProcessLockSharedGuard};
>> -use proxmox_time::TimeSpan;
>> +use proxmox_time::{epoch_i64, TimeSpan};
>> use proxmox_worker_task::WorkerTaskContext;
>>
>> use pbs_api_types::{
>> @@ -53,6 +53,8 @@ struct DatastoreConfigCache {
>> config: Arc<SectionConfigData>,
>> // Generation number from ConfigVersionCache
>> last_generation: usize,
>> + // Last update time (epoch seconds)
>> + last_update: i64,
>> }
>>
>> static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
>> @@ -61,6 +63,8 @@ static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
>> static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
>> LazyLock::new(|| Mutex::new(HashMap::new()));
>>
>> +/// Max age in seconds to reuse the cached datastore config.
>> +const DATASTORE_CONFIG_CACHE_TTL_SECS: i64 = 60;
>> /// Filename to store backup group notes
>> pub const GROUP_NOTES_FILE_NAME: &str = "notes";
>> /// Filename to store backup group owner
>> @@ -295,16 +299,22 @@ impl DatastoreBackend {
>>
>> /// Return the cached datastore SectionConfig and its generation.
>> fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
>> - let gen = ConfigVersionCache::new()
>> - .ok()
>> - .map(|c| c.datastore_generation());
>> + let now = epoch_i64();
>> + let version_cache = ConfigVersionCache::new().ok();
>> + let current_gen = version_cache.as_ref().map(|c| c.datastore_generation());
>>
>> let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
>>
>> - // Fast path: re-use cached datastore.cfg
>> - if let (Some(gen), Some(cache)) = (gen, guard.as_ref()) {
>> - if cache.last_generation == gen {
>> - return Ok((cache.config.clone(), Some(gen)));
>> + // Fast path: re-use cached datastore.cfg if cache is available, generation matches and TTL not expired
>> + if let (Some(current_gen), Some(config_cache)) = (current_gen, guard.as_ref()) {
>> + let gen_matches = config_cache.last_generation == current_gen;
>> + let ttl_ok = (now - config_cache.last_update) < DATASTORE_CONFIG_CACHE_TTL_SECS;
>> +
>> + if gen_matches && ttl_ok {
>> + return Ok((
>> + config_cache.config.clone(),
>> + Some(config_cache.last_generation),
>> + ));
>> }
>> }
>>
>> @@ -312,16 +322,28 @@ fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<u
>> let (config_raw, _digest) = pbs_config::datastore::config()?;
>> let config = Arc::new(config_raw);
>>
>> - if let Some(gen_val) = gen {
>> + // Update cache
>> + let new_gen = if let Some(handle) = version_cache {
>> + // Bump datastore generation whenever we reload the config.
>> + // This ensures that Drop handlers will detect that a newer config exists
>> + // and will not rely on a stale cached entry for maintenance mandate.
>> + let prev_gen = handle.increase_datastore_generation();
>
> this could be optimized (further) if we keep the digest when we
> load+parse the config above, because we only need to bump the generation
> if the digest changed. we need to bump the timestamp always of course ;)
> also we only want to bump if we previously had a generation saved, if we
> didn't, then this is the first load and bumping is meaningless anyway..
>
Good point, I think this would be a great optimization - TTL would only
eventually invalidate cached DataStoreImpls (if the config did change
manually). Will add!
> but there is another issue here - this is now called in the Drop
> handler, where we don't hold the config lock, so we have no guard
> against a parallel config change API call that also bumps the generation
> between us reloading and us bumping here.. which means we could have a
> mismatch between the value in new_gen and the actual config we loaded..
>
> I think we need to extend this helper here with a bool flag that
> determines whether we want to reload if the TTL expired, or return
> potentially outdated information? *every* lookup will handle the TTL
> anyway (by setting that parameter), so I think just fetching the
> "freshest" info we can get without reloading (by not setting it) is fine
> for the Drop handler..
>
Good point, will add the flag!
>> + let new_gen = prev_gen + 1;
>> +
>> *guard = Some(DatastoreConfigCache {
>> config: config.clone(),
>> - last_generation: gen_val,
>> + last_generation: new_gen,
>> + last_update: now,
>> });
>> +
>> + Some(new_gen)
>> } else {
>> + // if the cache was not available, use again the slow path next time
>> *guard = None;
>> - }
>> + None
>> + };
>>
>> - Ok((config, gen))
>> + Ok((config, new_gen))
>> }
>>
>> impl DataStore {
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* [pbs-devel] [PATCH proxmox-backup v3 3/6] partial fix #6049: datastore: use config fast-path in Drop
2025-11-20 13:03 10% [pbs-devel] [PATCH proxmox-backup v3 0/6] " Samuel Rufinatscha
2025-11-20 13:03 17% ` [pbs-devel] [PATCH proxmox-backup v3 1/6] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
2025-11-20 13:03 12% ` [pbs-devel] [PATCH proxmox-backup v3 2/6] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
@ 2025-11-20 13:03 16% ` Samuel Rufinatscha
2025-11-20 13:03 15% ` [pbs-devel] [PATCH proxmox-backup v3 4/6] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
` (4 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-20 13:03 UTC (permalink / raw)
To: pbs-devel
The Drop impl of DataStore re-read datastore.cfg to decide whether
the entry should be evicted from the in-process cache (based on
maintenance mode’s clear_from_cache). During the investigation of
issue #6049 [1], a flamegraph [2] showed that the config reload in Drop
accounted for a measurable share of CPU time under load.
This patch adds the datastore config fast path to the Drop impl to
eventually avoid an expensive config reload from disk to capture
the maintenance mandate.
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-datastore/src/datastore.rs | 43 +++++++++++++++++++++++++++-------
1 file changed, 34 insertions(+), 9 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 8c687097..1494521c 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -216,15 +216,40 @@ impl Drop for DataStore {
// remove datastore from cache iff
// - last task finished, and
// - datastore is in a maintenance mode that mandates it
- let remove_from_cache = last_task
- && pbs_config::datastore::config()
- .and_then(|(s, _)| s.lookup::<DataStoreConfig>("datastore", self.name()))
- .is_ok_and(|c| {
- c.get_maintenance_mode()
- .is_some_and(|m| m.clear_from_cache())
- });
-
- if remove_from_cache {
+
+ // first check: check if last task finished
+ if !last_task {
+ return;
+ }
+
+ let (section_config, _gen) = match datastore_section_config_cached() {
+ Ok(v) => v,
+ Err(err) => {
+ log::error!(
+ "failed to load datastore config in Drop for {} - {err}",
+ self.name()
+ );
+ return;
+ }
+ };
+
+ let datastore_cfg: DataStoreConfig =
+ match section_config.lookup("datastore", self.name()) {
+ Ok(cfg) => cfg,
+ Err(err) => {
+ log::error!(
+ "failed to look up datastore '{}' in Drop - {err}",
+ self.name()
+ );
+ return;
+ }
+ };
+
+ // second check: check maintenance mode mandate
+ if datastore_cfg
+ .get_maintenance_mode()
+ .is_some_and(|m| m.clear_from_cache())
+ {
DATASTORE_MAP.lock().unwrap().remove(self.name());
}
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 16%]
* [pbs-devel] [PATCH proxmox-backup v3 1/6] partial fix #6049: config: enable config version cache for datastore
2025-11-20 13:03 10% [pbs-devel] [PATCH proxmox-backup v3 0/6] " Samuel Rufinatscha
@ 2025-11-20 13:03 17% ` Samuel Rufinatscha
2025-11-20 13:03 12% ` [pbs-devel] [PATCH proxmox-backup v3 2/6] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
` (6 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-20 13:03 UTC (permalink / raw)
To: pbs-devel
Repeated /status requests caused lookup_datastore() to re-read and
parse datastore.cfg on every call. The issue was mentioned in report
#6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
dominated by pbs_config::datastore::config() (config parsing).
To solve the issue, this patch prepares the config version cache,
so that datastore config caching can be built on top of it.
This patch specifically:
(1) implements increment function in order to invalidate generations
(2) removes obsolete comments
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-config/src/config_version_cache.rs | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/pbs-config/src/config_version_cache.rs b/pbs-config/src/config_version_cache.rs
index e8fb994f..b875f7e0 100644
--- a/pbs-config/src/config_version_cache.rs
+++ b/pbs-config/src/config_version_cache.rs
@@ -26,7 +26,6 @@ struct ConfigVersionCacheDataInner {
// Traffic control (traffic-control.cfg) generation/version.
traffic_control_generation: AtomicUsize,
// datastore (datastore.cfg) generation/version
- // FIXME: remove with PBS 3.0
datastore_generation: AtomicUsize,
// Add further atomics here
}
@@ -145,8 +144,15 @@ impl ConfigVersionCache {
.fetch_add(1, Ordering::AcqRel);
}
+ /// Returns the datastore generation number.
+ pub fn datastore_generation(&self) -> usize {
+ self.shmem
+ .data()
+ .datastore_generation
+ .load(Ordering::Acquire)
+ }
+
/// Increase the datastore generation number.
- // FIXME: remove with PBS 3.0 or make actually useful again in datastore lookup
pub fn increase_datastore_generation(&self) -> usize {
self.shmem
.data()
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 17%]
* [pbs-devel] [PATCH proxmox-backup v3 5/6] partial fix #6049: datastore: add reload flag to config cache helper
2025-11-20 13:03 10% [pbs-devel] [PATCH proxmox-backup v3 0/6] " Samuel Rufinatscha
` (3 preceding siblings ...)
2025-11-20 13:03 15% ` [pbs-devel] [PATCH proxmox-backup v3 4/6] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
@ 2025-11-20 13:03 15% ` Samuel Rufinatscha
2025-11-20 14:50 5% ` Fabian Grünbichler
2025-11-20 13:03 15% ` [pbs-devel] [PATCH proxmox-backup v3 6/6] datastore: only bump generation when config digest changes Samuel Rufinatscha
` (2 subsequent siblings)
7 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-11-20 13:03 UTC (permalink / raw)
To: pbs-devel
Extend datastore_section_config_cached() with an `allow_reload` flag to
separate two use cases:
1) lookup_datastore() passes `true` and is allowed to reload
datastore.cfg from disk when the cache is missing, the generation
changed or the TTL expired. The helper may bump the datastore
generation if the digest changed.
2) DataStore::drop() passes `false` and only consumes the most recent
cached entry without touching the disk, TTL or generation. If the
cache was never initialised, it returns an error.
This avoids races between Drop and concurrent config changes.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-datastore/src/datastore.rs | 36 ++++++++++++++++++++++++++++++----
1 file changed, 32 insertions(+), 4 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 1711c753..12076f31 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -226,7 +226,7 @@ impl Drop for DataStore {
return;
}
- let (section_config, _gen) = match datastore_section_config_cached() {
+ let (section_config, _gen) = match datastore_section_config_cached(false) {
Ok(v) => v,
Err(err) => {
log::error!(
@@ -299,14 +299,42 @@ impl DatastoreBackend {
}
}
-/// Return the cached datastore SectionConfig and its generation.
-fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
+/// Returns the cached `datastore.cfg` and its generation.
+///
+/// When `allow_reload` is `true`, callers are expected to hold the datastore config. It may:
+/// - Reload `datastore.cfg` from disk if either
+/// - no cache exists yet, or cache is unavailable
+/// - the cached generation does not match the shared generation
+/// - the cache entry is older than `DATASTORE_CONFIG_CACHE_TTL_SECS`
+/// - Updates the cache with the new config, timestamp and digest.
+/// - Bumps the datastore generation in `ConfigVersionCache` only if
+/// there was a previous cached entry and the digest changed (manual edit or
+/// API write). If the digest is unchanged, the timestamp is refreshed but the
+/// generation is kept to avoid unnecessary invalidations.
+///
+/// When `allow_reload` is `false`:
+/// - Never touches the disk or the shared generation.
+/// - Ignores TTL and simply returns the most recent cached entry if available.
+/// - Returns an error if the cache has not been initialised yet.
+///
+/// Intended for use with `Datastore::drop` where no config lock is held
+/// and eventual stale data is acceptable.
+fn datastore_section_config_cached(
+ allow_reload: bool,
+) -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
let now = epoch_i64();
let version_cache = ConfigVersionCache::new().ok();
let current_gen = version_cache.as_ref().map(|c| c.datastore_generation());
let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
+ if !allow_reload {
+ if let Some(cache) = guard.as_ref() {
+ return Ok((cache.config.clone(), Some(cache.last_generation)));
+ }
+ bail!("datastore config cache not initialized");
+ }
+
// Fast path: re-use cached datastore.cfg if cache is available, generation matches and TTL not expired
if let (Some(current_gen), Some(config_cache)) = (current_gen, guard.as_ref()) {
let gen_matches = config_cache.last_generation == current_gen;
@@ -423,7 +451,7 @@ impl DataStore {
let _config_lock = pbs_config::datastore::lock_config()?;
// Get the current datastore.cfg generation number and cached config
- let (section_config, gen_num) = datastore_section_config_cached()?;
+ let (section_config, gen_num) = datastore_section_config_cached(true)?;
let datastore_cfg: DataStoreConfig = section_config.lookup("datastore", name)?;
let maintenance_mode = datastore_cfg.get_maintenance_mode();
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox-backup v3 0/6] datastore: remove config reload on hot path
@ 2025-11-20 13:03 10% Samuel Rufinatscha
2025-11-20 13:03 17% ` [pbs-devel] [PATCH proxmox-backup v3 1/6] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
` (7 more replies)
0 siblings, 8 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-20 13:03 UTC (permalink / raw)
To: pbs-devel
Hi,
this series reduces CPU time in datastore lookups by avoiding repeated
datastore.cfg reads/parses in both `lookup_datastore()` and
`DataStore::Drop`. It also adds a TTL so manual config edits are
noticed without reintroducing hashing on every request.
While investigating #6049 [1], cargo-flamegraph [2] showed hotspots during
repeated `/status` calls in `lookup_datastore()` and in `Drop`,
dominated by `pbs_config::datastore::config()` (config parse).
The parsing cost itself should eventually be investigated in a future
effort. Furthermore, cargo-flamegraph showed that when using a
token-based auth method to access the API, a significant amount of time
is spent in validation on every request request [3].
## Approach
[PATCH 1/6] Extend ConfigVersionCache for datastore generation
Expose a dedicated datastore generation counter and an increment
helper so callers can cheaply track datastore.cfg versions.
[PATCH 2/6] Fast path for datastore lookups
Cache the parsed datastore.cfg keyed by the shared datastore
generation. lookup_datastore() reuses both the cached config and an
existing DataStoreImpl when the generation matches, and falls back
to the old slow path otherwise.
[PATCH 3/6] Fast path for Drop
Make DataStore::Drop use the cached config if possible instead of
rereading datastore.cfg from disk.
[PATCH 4/6] TTL to catch manual edits
Add a small TTL around the cached config and bump the datastore
generation whenever the config is reloaded. This catches manual
edits to datastore.cfg without reintroducing hashing or
config parsing on every request.
[PATCH 5/6] Add reload flag to config cache helper
Add a flag to the config cache helper to indicate whether a
config reload is acceptable.
[PATCH 6/6] Only bump generation on config digest change
Avoid unnecessary generation bumps when the config is reloaded
but the digest did not change.
## Benchmark results
All the following benchmarks are based on top of
https://lore.proxmox.com/pbs-devel/20251112131525.645971-1-f.gruenbichler@proxmox.com/T/#u
### End-to-end
Testing `/status?verbose=0` end-to-end with 1000 stores, 5 req/store
and parallel=16 before/after the series:
Metric Before After
----------------------------------------
Total time 12s 9s
Throughput (all) 416.67 555.56
Cold RPS (round #1) 83.33 111.11
Warm RPS (#2..N) 333.33 444.44
Running under flamegraph [2], TLS appears to consume a significant
amount of CPU time and blur the results. Still, a ~33% higher overall
throughput and ~25% less end-to-end time for this workload.
### Isolated benchmarks (hyperfine)
In addition to the end-to-end tests, I measured two standalone benchmarks
with hyperfine, each using a config with 1000
datastores. `M` is the number of distinct datastores looked up and
`N` is the number of lookups per datastore.
Drop-direct variant:
Drops the `DataStore` after every lookup, so the `Drop` path runs on
every iteration:
use anyhow::Error;
use pbs_api_types::Operation;
use pbs_datastore::DataStore;
fn main() -> Result<(), Error> {
let mut args = std::env::args();
args.next();
let datastores = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
let iterations = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
for d in 1..=datastores {
let name = format!("ds{:04}", d);
for i in 1..=iterations {
DataStore::lookup_datastore(&name, Some(Operation::Write))?;
}
}
Ok(())
}
+------+-------+------------+------------+----------+
| M | N | Baseline | Patched | Speedup |
+------+-------+------------+------------+----------+
| 1 | 1000 | 1.699 s | 37.3 ms | 45.5x |
| 10 | 100 | 1.710 s | 35.8 ms | 47.7x |
| 100 | 10 | 1.787 s | 36.6 ms | 48.9x |
| 1000 | 1 | 1.899 s | 46.0 ms | 41.3x |
+------+-------+------------+------------+----------+
Bulk-drop variant:
Keeps the `DataStore` instances alive for
all `N` lookups of a given datastore and then drops them in bulk,
mimicking a task that performs many lookups while it is running and
only triggers the expensive `Drop` logic when the last user exits.
use anyhow::Error;
use pbs_api_types::Operation;
use pbs_datastore::DataStore;
fn main() -> Result<(), Error> {
let mut args = std::env::args();
args.next();
let datastores = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
let iterations = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
for d in 1..=datastores {
let name = format!("ds{:04}", d);
let mut stores = Vec::with_capacity(iterations);
for i in 1..=iterations {
stores.push(DataStore::lookup_datastore(&name, Some(Operation::Write))?);
}
}
Ok(())
}
+------+-------+--------------+-------------+----------+
| M | N | Baseline | Patched | Speedup |
+------+-------+--------------+-------------+----------+
| 1 | 1000 | 888.8 ms | 39.3 ms | 22.6x |
| 10 | 100 | 890.8 ms | 35.3 ms | 25.3x |
| 100 | 10 | 974.5 ms | 36.3 ms | 26.8x |
| 1000 | 1 | 1.848 s | 39.9 ms | 46.3x |
+------+-------+--------------+-------------+----------+
Both variants show that the combination of the cached config lookups
and the cheaper `Drop` handling reduces the hot-path cost from ~1.7 s
per run to a few tens of milliseconds in these benchmarks.
## Reproduction steps
VM: 4 vCPU, ~8 GiB RAM, VirtIO-SCSI; disks:
- scsi0 32G (OS)
- scsi1 1000G (datastores)
Install PBS from ISO on the VM.
Set up ZFS on /dev/sdb (adjust if different):
zpool create -f -o ashift=12 pbsbench /dev/sdb
zfs set mountpoint=/pbsbench pbsbench
zfs create pbsbench/pbs-bench
Raise file-descriptor limit:
sudo systemctl edit proxmox-backup-proxy.service
Add the following lines:
[Service]
LimitNOFILE=1048576
Reload systemd and restart the proxy:
sudo systemctl daemon-reload
sudo systemctl restart proxmox-backup-proxy.service
Verify the limit:
systemctl show proxmox-backup-proxy.service | grep LimitNOFILE
Create 1000 ZFS-backed datastores (as used in #6049 [1]):
seq -w 001 1000 | xargs -n1 -P1 bash -c '
id=$0
name="ds${id}"
dataset="pbsbench/pbs-bench/${name}"
path="/pbsbench/pbs-bench/${name}"
zfs create -o mountpoint="$path" "$dataset"
proxmox-backup-manager datastore create "$name" "$path" \
--comment "ZFS dataset-based datastore"
'
Build PBS from this series, then run the server under manually
under flamegraph:
systemctl stop proxmox-backup-proxy
cargo flamegraph --release --bin proxmox-backup-proxy
## Other resources:
### E2E benchmark script:
#!/usr/bin/env bash
set -euo pipefail
# --- Config ---------------------------------------------------------------
HOST='https://localhost:8007'
USER='root@pam'
PASS="$(cat passfile)"
DATASTORE_PATH="/pbsbench/pbs-bench"
MAX_STORES=1000 # how many stores to include
PARALLEL=16 # concurrent workers
REPEAT=5 # requests per store (1 cold + REPEAT-1 warm)
PRINT_FIRST=false # true => log first request's HTTP code per store
# --- Helpers --------------------------------------------------------------
fmt_rps () {
local n="$1" t="$2"
awk -v n="$n" -v t="$t" 'BEGIN { if (t > 0) printf("%.2f\n", n/t); else print "0.00" }'
}
# --- Login ---------------------------------------------------------------
auth=$(curl -ks -X POST "$HOST/api2/json/access/ticket" \
-d "username=$USER" -d "password=$PASS")
ticket=$(echo "$auth" | jq -r '.data.ticket')
if [[ -z "${ticket:-}" || "$ticket" == "null" ]]; then
echo "[ERROR] Login failed (no ticket)"
exit 1
fi
# --- Collect stores (deterministic order) --------------------------------
mapfile -t STORES < <(
find "$DATASTORE_PATH" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' \
| sort | head -n "$MAX_STORES"
)
USED_STORES=${#STORES[@]}
if (( USED_STORES == 0 )); then
echo "[ERROR] No datastore dirs under $DATASTORE_PATH"
exit 1
fi
echo "[INFO] Running with stores=$USED_STORES, repeat=$REPEAT, parallel=$PARALLEL"
# --- Temp counters --------------------------------------------------------
SUCCESS_ALL="$(mktemp)"
FAIL_ALL="$(mktemp)"
COLD_OK="$(mktemp)"
WARM_OK="$(mktemp)"
trap 'rm -f "$SUCCESS_ALL" "$FAIL_ALL" "$COLD_OK" "$WARM_OK"' EXIT
export HOST ticket REPEAT SUCCESS_ALL FAIL_ALL COLD_OK WARM_OK PRINT_FIRST
SECONDS=0
# --- Fire requests --------------------------------------------------------
printf "%s\n" "${STORES[@]}" \
| xargs -P"$PARALLEL" -I{} bash -c '
store="$1"
url="$HOST/api2/json/admin/datastore/$store/status?verbose=0"
for ((i=1;i<=REPEAT;i++)); do
code=$(curl -ks -o /dev/null -w "%{http_code}" -b "PBSAuthCookie=$ticket" "$url" || echo 000)
if [[ "$code" == "200" ]]; then
echo 1 >> "$SUCCESS_ALL"
if (( i == 1 )); then
echo 1 >> "$COLD_OK"
else
echo 1 >> "$WARM_OK"
fi
if [[ "$PRINT_FIRST" == "true" && $i -eq 1 ]]; then
ts=$(date +%H:%M:%S)
echo "[$ts] $store #$i HTTP:200"
fi
else
echo 1 >> "$FAIL_ALL"
if [[ "$PRINT_FIRST" == "true" && $i -eq 1 ]]; then
ts=$(date +%H:%M:%S)
echo "[$ts] $store #$i HTTP:$code (FAIL)"
fi
fi
done
' _ {}
# --- Summary --------------------------------------------------------------
elapsed=$SECONDS
ok=$(wc -l < "$SUCCESS_ALL" 2>/dev/null || echo 0)
fail=$(wc -l < "$FAIL_ALL" 2>/dev/null || echo 0)
cold_ok=$(wc -l < "$COLD_OK" 2>/dev/null || echo 0)
warm_ok=$(wc -l < "$WARM_OK" 2>/dev/null || echo 0)
expected=$(( USED_STORES * REPEAT ))
total=$(( ok + fail ))
rps_all=$(fmt_rps "$ok" "$elapsed")
rps_cold=$(fmt_rps "$cold_ok" "$elapsed")
rps_warm=$(fmt_rps "$warm_ok" "$elapsed")
echo "===== Summary ====="
echo "Stores used: $USED_STORES"
echo "Expected requests: $expected"
echo "Executed requests: $total"
echo "OK (HTTP 200): $ok"
echo "Failed: $fail"
printf "Total time: %dm %ds\n" $((elapsed/60)) $((elapsed%60))
echo "Throughput all RPS: $rps_all"
echo "Cold RPS (round #1): $rps_cold"
echo "Warm RPS (#2..N): $rps_warm"
## Patch summary
[PATCH 1/6] partial fix #6049: config: enable config version cache for datastore
[PATCH 2/6] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
[PATCH 3/6] partial fix #6049: datastore: use config fast-path in Drop
[PATCH 4/6] partial fix #6049: datastore: add TTL fallback to catch manual config edits
[PATCH 5/6] to add a reload flag to the config cache helper.
[PATCH 6/6] to only bump generation when the config digest changes.
## Changes from v2:
Added:
- [PATCH 5/6]: Add a reload flag to the config cache helper.
- [PATCH 6/6]: Only bump generation when the config digest changes.
## Maintainer notes
No dependency bumps, no API changes and no breaking changes.
Thanks,
Samuel
[1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
[3] Bugzilla #7017: https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Samuel Rufinatscha (6):
partial fix #6049: config: enable config version cache for datastore
partial fix #6049: datastore: impl ConfigVersionCache fast path for
lookups
partial fix #6049: datastore: use config fast-path in Drop
partial fix #6049: datastore: add TTL fallback to catch manual config
edits
partial fix #6049: datastore: add reload flag to config cache helper
datastore: only bump generation when config digest changes
pbs-config/src/config_version_cache.rs | 10 +-
pbs-datastore/Cargo.toml | 1 +
pbs-datastore/src/datastore.rs | 232 ++++++++++++++++++++-----
3 files changed, 197 insertions(+), 46 deletions(-)
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 10%]
* [pbs-devel] [PATCH proxmox-backup v3 6/6] datastore: only bump generation when config digest changes
2025-11-20 13:03 10% [pbs-devel] [PATCH proxmox-backup v3 0/6] " Samuel Rufinatscha
` (4 preceding siblings ...)
2025-11-20 13:03 15% ` [pbs-devel] [PATCH proxmox-backup v3 5/6] partial fix #6049: datastore: add reload flag to config cache helper Samuel Rufinatscha
@ 2025-11-20 13:03 15% ` Samuel Rufinatscha
2025-11-20 14:50 5% ` Fabian Grünbichler
2025-11-20 14:50 5% ` [pbs-devel] [PATCH proxmox-backup v3 0/6] datastore: remove config reload on hot path Fabian Grünbichler
2025-11-24 15:35 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
7 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-11-20 13:03 UTC (permalink / raw)
To: pbs-devel
When reloading datastore.cfg in datastore_section_config_cached(),
we currently bump the datastore generation unconditionally. This is
only necessary when the on disk content actually changed and when
we already had a previous cached entry.
This patch extends the DatastoreConfigCache to store the last digest of
datastore.cfg and track the previously cached generation and digest.
Only when the digest differs from the cached one. On first load, it
reuses the existing datastore_generation without bumping.
This avoids unnecessary cache invalidations if the config did not
change.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-datastore/src/datastore.rs | 43 ++++++++++++++++++++++++----------
1 file changed, 30 insertions(+), 13 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 12076f31..bf04332e 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -51,6 +51,8 @@ use crate::{DataBlob, LocalDatastoreLruCache};
struct DatastoreConfigCache {
// Parsed datastore.cfg file
config: Arc<SectionConfigData>,
+ // Digest of the datastore.cfg file
+ last_digest: [u8; 32],
// Generation number from ConfigVersionCache
last_generation: usize,
// Last update time (epoch seconds)
@@ -349,29 +351,44 @@ fn datastore_section_config_cached(
}
// Slow path: re-read datastore.cfg
- let (config_raw, _digest) = pbs_config::datastore::config()?;
+ let (config_raw, digest) = pbs_config::datastore::config()?;
let config = Arc::new(config_raw);
- // Update cache
+ // Decide whether to bump the shared generation.
+ // Only bump if we already had a cached generation and the digest changed (manual edit or API write)
+ let (prev_gen, prev_digest) = guard
+ .as_ref()
+ .map(|c| (Some(c.last_generation), Some(c.last_digest)))
+ .unwrap_or((None, None));
+
let new_gen = if let Some(handle) = version_cache {
- // Bump datastore generation whenever we reload the config.
- // This ensures that Drop handlers will detect that a newer config exists
- // and will not rely on a stale cached entry for maintenance mandate.
- let prev_gen = handle.increase_datastore_generation();
- let new_gen = prev_gen + 1;
+ match (prev_gen, prev_digest) {
+ // We had a previous generation and the digest changed => bump generation.
+ (Some(_prev_gen), Some(prev_digest)) if prev_digest != digest => {
+ let old = handle.increase_datastore_generation();
+ Some(old + 1)
+ }
+ // We had a previous generation but the digest stayed the same:
+ // keep the existing generation, just refresh the timestamp.
+ (Some(prev_gen), _) => Some(prev_gen),
+ // We didn't have a previous generation, just use the current one.
+ (None, _) => Some(handle.datastore_generation()),
+ }
+ } else {
+ None
+ };
+ if let Some(gen_val) = new_gen {
*guard = Some(DatastoreConfigCache {
config: config.clone(),
- last_generation: new_gen,
+ last_digest: digest,
+ last_generation: gen_val,
last_update: now,
});
-
- Some(new_gen)
} else {
- // if the cache was not available, use again the slow path next time
+ // If the shared version cache is not available, don't cache.
*guard = None;
- None
- };
+ }
Ok((config, new_gen))
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox-backup v3 4/6] partial fix #6049: datastore: add TTL fallback to catch manual config edits
2025-11-20 13:03 10% [pbs-devel] [PATCH proxmox-backup v3 0/6] " Samuel Rufinatscha
` (2 preceding siblings ...)
2025-11-20 13:03 16% ` [pbs-devel] [PATCH proxmox-backup v3 3/6] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
@ 2025-11-20 13:03 15% ` Samuel Rufinatscha
2025-11-20 13:03 15% ` [pbs-devel] [PATCH proxmox-backup v3 5/6] partial fix #6049: datastore: add reload flag to config cache helper Samuel Rufinatscha
` (3 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-20 13:03 UTC (permalink / raw)
To: pbs-devel
The lookup fast path reacts to API-driven config changes because
save_config() bumps the generation. Manual edits of datastore.cfg do
not bump the counter. To keep the system robust against such edits
without reintroducing config reading and hashing on the hot path, this
patch adds a TTL to the cache entry.
If the cached config is older than
DATASTORE_CONFIG_CACHE_TTL_SECS (set to 60s), the next lookup takes
the slow path and refreshes the cached entry. Within
the TTL window, unchanged generations still use the fast path.
Links
[1] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-datastore/src/datastore.rs | 46 +++++++++++++++++++++++++---------
1 file changed, 34 insertions(+), 12 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 1494521c..1711c753 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -22,7 +22,7 @@ use proxmox_sys::error::SysError;
use proxmox_sys::fs::{file_read_optional_string, replace_file, CreateOptions};
use proxmox_sys::linux::procfs::MountInfo;
use proxmox_sys::process_locker::{ProcessLockExclusiveGuard, ProcessLockSharedGuard};
-use proxmox_time::TimeSpan;
+use proxmox_time::{epoch_i64, TimeSpan};
use proxmox_worker_task::WorkerTaskContext;
use pbs_api_types::{
@@ -53,6 +53,8 @@ struct DatastoreConfigCache {
config: Arc<SectionConfigData>,
// Generation number from ConfigVersionCache
last_generation: usize,
+ // Last update time (epoch seconds)
+ last_update: i64,
}
static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
@@ -61,6 +63,8 @@ static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
+/// Max age in seconds to reuse the cached datastore config.
+const DATASTORE_CONFIG_CACHE_TTL_SECS: i64 = 60;
/// Filename to store backup group notes
pub const GROUP_NOTES_FILE_NAME: &str = "notes";
/// Filename to store backup group owner
@@ -297,16 +301,22 @@ impl DatastoreBackend {
/// Return the cached datastore SectionConfig and its generation.
fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
- let gen = ConfigVersionCache::new()
- .ok()
- .map(|c| c.datastore_generation());
+ let now = epoch_i64();
+ let version_cache = ConfigVersionCache::new().ok();
+ let current_gen = version_cache.as_ref().map(|c| c.datastore_generation());
let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
- // Fast path: re-use cached datastore.cfg
- if let (Some(gen), Some(cache)) = (gen, guard.as_ref()) {
- if cache.last_generation == gen {
- return Ok((cache.config.clone(), Some(gen)));
+ // Fast path: re-use cached datastore.cfg if cache is available, generation matches and TTL not expired
+ if let (Some(current_gen), Some(config_cache)) = (current_gen, guard.as_ref()) {
+ let gen_matches = config_cache.last_generation == current_gen;
+ let ttl_ok = (now - config_cache.last_update) < DATASTORE_CONFIG_CACHE_TTL_SECS;
+
+ if gen_matches && ttl_ok {
+ return Ok((
+ config_cache.config.clone(),
+ Some(config_cache.last_generation),
+ ));
}
}
@@ -314,16 +324,28 @@ fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<u
let (config_raw, _digest) = pbs_config::datastore::config()?;
let config = Arc::new(config_raw);
- if let Some(gen_val) = gen {
+ // Update cache
+ let new_gen = if let Some(handle) = version_cache {
+ // Bump datastore generation whenever we reload the config.
+ // This ensures that Drop handlers will detect that a newer config exists
+ // and will not rely on a stale cached entry for maintenance mandate.
+ let prev_gen = handle.increase_datastore_generation();
+ let new_gen = prev_gen + 1;
+
*guard = Some(DatastoreConfigCache {
config: config.clone(),
- last_generation: gen_val,
+ last_generation: new_gen,
+ last_update: now,
});
+
+ Some(new_gen)
} else {
+ // if the cache was not available, use again the slow path next time
*guard = None;
- }
+ None
+ };
- Ok((config, gen))
+ Ok((config, new_gen))
}
impl DataStore {
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox-backup v3 2/6] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
2025-11-20 13:03 10% [pbs-devel] [PATCH proxmox-backup v3 0/6] " Samuel Rufinatscha
2025-11-20 13:03 17% ` [pbs-devel] [PATCH proxmox-backup v3 1/6] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
@ 2025-11-20 13:03 12% ` Samuel Rufinatscha
2025-11-20 13:03 16% ` [pbs-devel] [PATCH proxmox-backup v3 3/6] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
` (5 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-20 13:03 UTC (permalink / raw)
To: pbs-devel
Repeated /status requests caused lookup_datastore() to re-read and
parse datastore.cfg on every call. The issue was mentioned in report
#6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
dominated by pbs_config::datastore::config() (config parsing).
This patch implements caching of the global datastore.cfg using the
generation numbers from the shared config version cache. It caches the
datastore.cfg along with the generation number and, when a subsequent
lookup sees the same generation, it reuses the cached config without
re-reading it from disk. If the generation differs
(or the cache is unavailable), it falls back to the existing slow path
with no behavioral changes.
Behavioral notes
- The generation is bumped via the existing save_config() path, so
API-driven config changes are detected immediately.
- Manual edits to datastore.cfg are not detected; a TTL
guard is introduced in a dedicated patch in this series.
- DataStore::drop still performs a config read on the common path,
this is covered in a dedicated patch in this series.
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-datastore/Cargo.toml | 1 +
pbs-datastore/src/datastore.rs | 120 +++++++++++++++++++++++----------
2 files changed, 87 insertions(+), 34 deletions(-)
diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index 8ce930a9..42f49a7b 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -40,6 +40,7 @@ proxmox-io.workspace = true
proxmox-lang.workspace=true
proxmox-s3-client = { workspace = true, features = [ "impl" ] }
proxmox-schema = { workspace = true, features = [ "api-macro" ] }
+proxmox-section-config.workspace = true
proxmox-serde = { workspace = true, features = [ "serde_json" ] }
proxmox-sys.workspace = true
proxmox-systemd.workspace = true
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 0a517923..8c687097 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -32,7 +32,8 @@ use pbs_api_types::{
MaintenanceType, Operation, UPID,
};
use pbs_config::s3::S3_CFG_TYPE_ID;
-use pbs_config::BackupLockGuard;
+use pbs_config::{BackupLockGuard, ConfigVersionCache};
+use proxmox_section_config::SectionConfigData;
use crate::backup_info::{
BackupDir, BackupGroup, BackupInfo, OLD_LOCKING, PROTECTED_MARKER_FILENAME,
@@ -46,6 +47,17 @@ use crate::s3::S3_CONTENT_PREFIX;
use crate::task_tracking::{self, update_active_operations};
use crate::{DataBlob, LocalDatastoreLruCache};
+// Cache for fully parsed datastore.cfg
+struct DatastoreConfigCache {
+ // Parsed datastore.cfg file
+ config: Arc<SectionConfigData>,
+ // Generation number from ConfigVersionCache
+ last_generation: usize,
+}
+
+static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
+ LazyLock::new(|| Mutex::new(None));
+
static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
@@ -142,10 +154,12 @@ pub struct DataStoreImpl {
last_gc_status: Mutex<GarbageCollectionStatus>,
verify_new: bool,
chunk_order: ChunkOrder,
- last_digest: Option<[u8; 32]>,
sync_level: DatastoreFSyncLevel,
backend_config: DatastoreBackendConfig,
lru_store_caching: Option<LocalDatastoreLruCache>,
+ /// Datastore generation number from `ConfigVersionCache` at creation time, used to
+ /// validate reuse of this cached `DataStoreImpl`.
+ config_generation: Option<usize>,
}
impl DataStoreImpl {
@@ -158,10 +172,10 @@ impl DataStoreImpl {
last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
verify_new: false,
chunk_order: Default::default(),
- last_digest: None,
sync_level: Default::default(),
backend_config: Default::default(),
lru_store_caching: None,
+ config_generation: None,
})
}
}
@@ -256,6 +270,37 @@ impl DatastoreBackend {
}
}
+/// Return the cached datastore SectionConfig and its generation.
+fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
+ let gen = ConfigVersionCache::new()
+ .ok()
+ .map(|c| c.datastore_generation());
+
+ let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
+
+ // Fast path: re-use cached datastore.cfg
+ if let (Some(gen), Some(cache)) = (gen, guard.as_ref()) {
+ if cache.last_generation == gen {
+ return Ok((cache.config.clone(), Some(gen)));
+ }
+ }
+
+ // Slow path: re-read datastore.cfg
+ let (config_raw, _digest) = pbs_config::datastore::config()?;
+ let config = Arc::new(config_raw);
+
+ if let Some(gen_val) = gen {
+ *guard = Some(DatastoreConfigCache {
+ config: config.clone(),
+ last_generation: gen_val,
+ });
+ } else {
+ *guard = None;
+ }
+
+ Ok((config, gen))
+}
+
impl DataStore {
// This one just panics on everything
#[doc(hidden)]
@@ -327,56 +372,63 @@ impl DataStore {
name: &str,
operation: Option<Operation>,
) -> Result<Arc<DataStore>, Error> {
- // Avoid TOCTOU between checking maintenance mode and updating active operation counter, as
- // we use it to decide whether it is okay to delete the datastore.
+ // Avoid TOCTOU between checking maintenance mode and updating active operations.
let _config_lock = pbs_config::datastore::lock_config()?;
- // we could use the ConfigVersionCache's generation for staleness detection, but we load
- // the config anyway -> just use digest, additional benefit: manual changes get detected
- let (config, digest) = pbs_config::datastore::config()?;
- let config: DataStoreConfig = config.lookup("datastore", name)?;
+ // Get the current datastore.cfg generation number and cached config
+ let (section_config, gen_num) = datastore_section_config_cached()?;
+
+ let datastore_cfg: DataStoreConfig = section_config.lookup("datastore", name)?;
+ let maintenance_mode = datastore_cfg.get_maintenance_mode();
+ let mount_status = get_datastore_mount_status(&datastore_cfg);
- if let Some(maintenance_mode) = config.get_maintenance_mode() {
- if let Err(error) = maintenance_mode.check(operation) {
+ if let Some(mm) = &maintenance_mode {
+ if let Err(error) = mm.check(operation.clone()) {
bail!("datastore '{name}' is unavailable: {error}");
}
}
- if get_datastore_mount_status(&config) == Some(false) {
- let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
- datastore_cache.remove(&config.name);
- bail!("datastore '{}' is not mounted", config.name);
+ let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
+
+ if mount_status == Some(false) {
+ datastore_cache.remove(&datastore_cfg.name);
+ bail!("datastore '{}' is not mounted", datastore_cfg.name);
}
- let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
- let entry = datastore_cache.get(name);
-
- // reuse chunk store so that we keep using the same process locker instance!
- let chunk_store = if let Some(datastore) = &entry {
- let last_digest = datastore.last_digest.as_ref();
- if let Some(true) = last_digest.map(|last_digest| last_digest == &digest) {
- if let Some(operation) = operation {
- update_active_operations(name, operation, 1)?;
+ // Re-use DataStoreImpl
+ if let Some(existing) = datastore_cache.get(name).cloned() {
+ if let (Some(last_generation), Some(gen_num)) = (existing.config_generation, gen_num) {
+ if last_generation == gen_num {
+ if let Some(op) = operation {
+ update_active_operations(name, op, 1)?;
+ }
+
+ return Ok(Arc::new(Self {
+ inner: existing,
+ operation,
+ }));
}
- return Ok(Arc::new(Self {
- inner: Arc::clone(datastore),
- operation,
- }));
}
- Arc::clone(&datastore.chunk_store)
+ }
+
+ // (Re)build DataStoreImpl
+
+ // Reuse chunk store so that we keep using the same process locker instance!
+ let chunk_store = if let Some(existing) = datastore_cache.get(name) {
+ Arc::clone(&existing.chunk_store)
} else {
let tuning: DatastoreTuning = serde_json::from_value(
DatastoreTuning::API_SCHEMA
- .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
+ .parse_property_string(datastore_cfg.tuning.as_deref().unwrap_or(""))?,
)?;
Arc::new(ChunkStore::open(
name,
- config.absolute_path(),
+ datastore_cfg.absolute_path(),
tuning.sync_level.unwrap_or_default(),
)?)
};
- let datastore = DataStore::with_store_and_config(chunk_store, config, Some(digest))?;
+ let datastore = DataStore::with_store_and_config(chunk_store, datastore_cfg, gen_num)?;
let datastore = Arc::new(datastore);
datastore_cache.insert(name.to_string(), datastore.clone());
@@ -478,7 +530,7 @@ impl DataStore {
fn with_store_and_config(
chunk_store: Arc<ChunkStore>,
config: DataStoreConfig,
- last_digest: Option<[u8; 32]>,
+ generation: Option<usize>,
) -> Result<DataStoreImpl, Error> {
let mut gc_status_path = chunk_store.base_path();
gc_status_path.push(".gc-status");
@@ -538,10 +590,10 @@ impl DataStore {
last_gc_status: Mutex::new(gc_status),
verify_new: config.verify_new.unwrap_or(false),
chunk_order: tuning.chunk_order.unwrap_or_default(),
- last_digest,
sync_level: tuning.sync_level.unwrap_or_default(),
backend_config,
lru_store_caching,
+ config_generation: generation,
})
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] superseded: [PATCH proxmox-backup v2 0/4] datastore: remove config reload on hot path
2025-11-14 15:05 10% [pbs-devel] [PATCH proxmox-backup v2 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
` (3 preceding siblings ...)
2025-11-14 15:05 15% ` [pbs-devel] [PATCH proxmox-backup v2 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
@ 2025-11-20 13:07 13% ` Samuel Rufinatscha
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-20 13:07 UTC (permalink / raw)
To: pbs-devel
https://lore.proxmox.com/pbs-devel/20251120130342.248815-1-s.rufinatscha@proxmox.com/T/#t
On 11/14/25 4:05 PM, Samuel Rufinatscha wrote:
> Hi,
>
> this series reduces CPU time in datastore lookups by avoiding repeated
> datastore.cfg reads/parses in both `lookup_datastore()` and
> `DataStore::Drop`. It also adds a TTL so manual config edits are
> noticed without reintroducing hashing on every request.
>
> While investigating #6049 [1], cargo-flamegraph [2] showed hotspots during
> repeated `/status` calls in `lookup_datastore()` and in `Drop`,
> dominated by `pbs_config::datastore::config()` (config parse).
>
> The parsing cost itself should eventually be investigated in a future
> effort. Furthermore, cargo-flamegraph showed that when using a
> token-based auth method to access the API, a significant amount of time
> is spent in validation on every request request [3].
>
> ## Approach
>
> [PATCH 1/4] Extend ConfigVersionCache for datastore generation
> Expose a dedicated datastore generation counter and an increment
> helper so callers can cheaply track datastore.cfg versions.
>
> [PATCH 2/4] Fast path for datastore lookups
> Cache the parsed datastore.cfg keyed by the shared datastore
> generation. lookup_datastore() reuses both the cached config and an
> existing DataStoreImpl when the generation matches, and falls back
> to the old slow path otherwise.
>
> [PATCH 3/4] Fast path for Drop
> Make DataStore::Drop use the cached config if possible instead of
> rereading datastore.cfg from disk.
>
> [PATCH 4/4] TTL to catch manual edits
> Add a small TTL around the cached config and bump the datastore
> generation whenever the config is reloaded. This catches manual
> edits to datastore.cfg without reintroducing hashing or
> config parsing on every request.
>
> ## Benchmark results
>
> All the following benchmarks are based on top of
> https://lore.proxmox.com/pbs-devel/20251112131525.645971-1-f.gruenbichler@proxmox.com/T/#u
>
> ### End-to-end
>
> Testing `/status?verbose=0` end-to-end with 1000 stores, 5 req/store
> and parallel=16 before/after the series:
>
> Metric Before After
> ----------------------------------------
> Total time 12s 9s
> Throughput (all) 416.67 555.56
> Cold RPS (round #1) 83.33 111.11
> Warm RPS (#2..N) 333.33 444.44
>
> Running under flamegraph [2], TLS appears to consume a significant
> amount of CPU time and blur the results. Still, a ~33% higher overall
> throughput and ~25% less end-to-end time for this workload.
>
> ### Isolated benchmarks (hyperfine)
>
> In addition to the end-to-end tests, I measured two standalone benchmarks
> with hyperfine, each using a config with 1000
> datastores. `M` is the number of distinct datastores looked up and
> `N` is the number of lookups per datastore.
>
> Drop-direct variant:
>
> Drops the `DataStore` after every lookup, so the `Drop` path runs on
> every iteration:
>
> use anyhow::Error;
>
> use pbs_api_types::Operation;
> use pbs_datastore::DataStore;
>
> fn main() -> Result<(), Error> {
> let mut args = std::env::args();
> args.next();
>
> let datastores = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> let iterations = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> for d in 1..=datastores {
> let name = format!("ds{:04}", d);
>
> for i in 1..=iterations {
> DataStore::lookup_datastore(&name, Some(Operation::Write))?;
> }
> }
>
> Ok(())
> }
>
> +----+------+-----------+-----------+---------+
> | M | N | Baseline | Patched | Speedup |
> +----+------+-----------+-----------+---------+
> | 1 | 1000 | 1.670 s | 34.3 ms | 48.7x |
> | 10 | 100 | 1.672 s | 34.5 ms | 48.4x |
> | 100| 10 | 1.679 s | 35.1 ms | 47.8x |
> |1000| 1 | 1.787 s | 38.2 ms | 46.8x |
> +----+------+-----------+-----------+---------+
>
> Bulk-drop variant:
>
> Keeps the `DataStore` instances alive for
> all `N` lookups of a given datastore and then drops them in bulk,
> mimicking a task that performs many lookups while it is running and
> only triggers the expensive `Drop` logic when the last user exits.
>
> use anyhow::Error;
>
> use pbs_api_types::Operation;
> use pbs_datastore::DataStore;
>
> fn main() -> Result<(), Error> {
> let mut args = std::env::args();
> args.next();
>
> let datastores = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> let iterations = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> for d in 1..=datastores {
> let name = format!("ds{:04}", d);
>
> let mut stores = Vec::with_capacity(iterations);
> for i in 1..=iterations {
> stores.push(DataStore::lookup_datastore(&name, Some(Operation::Write))?);
> }
> }
>
> Ok(())
> }
>
> +------+------+---------------+--------------+---------+
> | M | N | Baseline mean | Patched mean | Speedup |
> +------+------+---------------+--------------+---------+
> | 1 | 1000 | 884.0 ms | 33.9 ms | 26.1x |
> | 10 | 100 | 881.8 ms | 35.3 ms | 25.0x |
> | 100 | 10 | 969.3 ms | 35.9 ms | 27.0x |
> | 1000 | 1 | 1827.0 ms | 40.7 ms | 44.9x |
> +------+------+---------------+--------------+---------+
>
> Both variants show that the combination of the cached config lookups
> and the cheaper `Drop` handling reduces the hot-path cost from ~1.7 s
> per run to a few tens of milliseconds in these benchmarks.
>
> ## Reproduction steps
>
> VM: 4 vCPU, ~8 GiB RAM, VirtIO-SCSI; disks:
> - scsi0 32G (OS)
> - scsi1 1000G (datastores)
>
> Install PBS from ISO on the VM.
>
> Set up ZFS on /dev/sdb (adjust if different):
>
> zpool create -f -o ashift=12 pbsbench /dev/sdb
> zfs set mountpoint=/pbsbench pbsbench
> zfs create pbsbench/pbs-bench
>
> Raise file-descriptor limit:
>
> sudo systemctl edit proxmox-backup-proxy.service
>
> Add the following lines:
>
> [Service]
> LimitNOFILE=1048576
>
> Reload systemd and restart the proxy:
>
> sudo systemctl daemon-reload
> sudo systemctl restart proxmox-backup-proxy.service
>
> Verify the limit:
>
> systemctl show proxmox-backup-proxy.service | grep LimitNOFILE
>
> Create 1000 ZFS-backed datastores (as used in #6049 [1]):
>
> seq -w 001 1000 | xargs -n1 -P1 bash -c '
> id=$0
> name="ds${id}"
> dataset="pbsbench/pbs-bench/${name}"
> path="/pbsbench/pbs-bench/${name}"
> zfs create -o mountpoint="$path" "$dataset"
> proxmox-backup-manager datastore create "$name" "$path" \
> --comment "ZFS dataset-based datastore"
> '
>
> Build PBS from this series, then run the server under manually
> under flamegraph:
>
> systemctl stop proxmox-backup-proxy
> cargo flamegraph --release --bin proxmox-backup-proxy
>
> ## Other resources:
>
> ### E2E benchmark script:
>
> #!/usr/bin/env bash
> set -euo pipefail
>
> # --- Config ---------------------------------------------------------------
> HOST='https://localhost:8007'
> USER='root@pam'
> PASS="$(cat passfile)"
>
> DATASTORE_PATH="/pbsbench/pbs-bench"
> MAX_STORES=1000 # how many stores to include
> PARALLEL=16 # concurrent workers
> REPEAT=5 # requests per store (1 cold + REPEAT-1 warm)
>
> PRINT_FIRST=false # true => log first request's HTTP code per store
>
> # --- Helpers --------------------------------------------------------------
> fmt_rps () {
> local n="$1" t="$2"
> awk -v n="$n" -v t="$t" 'BEGIN { if (t > 0) printf("%.2f\n", n/t); else print "0.00" }'
> }
>
> # --- Login ---------------------------------------------------------------
> auth=$(curl -ks -X POST "$HOST/api2/json/access/ticket" \
> -d "username=$USER" -d "password=$PASS")
> ticket=$(echo "$auth" | jq -r '.data.ticket')
>
> if [[ -z "${ticket:-}" || "$ticket" == "null" ]]; then
> echo "[ERROR] Login failed (no ticket)"
> exit 1
> fi
>
> # --- Collect stores (deterministic order) --------------------------------
> mapfile -t STORES < <(
> find "$DATASTORE_PATH" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' \
> | sort | head -n "$MAX_STORES"
> )
>
> USED_STORES=${#STORES[@]}
> if (( USED_STORES == 0 )); then
> echo "[ERROR] No datastore dirs under $DATASTORE_PATH"
> exit 1
> fi
>
> echo "[INFO] Running with stores=$USED_STORES, repeat=$REPEAT, parallel=$PARALLEL"
>
> # --- Temp counters --------------------------------------------------------
> SUCCESS_ALL="$(mktemp)"
> FAIL_ALL="$(mktemp)"
> COLD_OK="$(mktemp)"
> WARM_OK="$(mktemp)"
> trap 'rm -f "$SUCCESS_ALL" "$FAIL_ALL" "$COLD_OK" "$WARM_OK"' EXIT
>
> export HOST ticket REPEAT SUCCESS_ALL FAIL_ALL COLD_OK WARM_OK PRINT_FIRST
>
> SECONDS=0
>
> # --- Fire requests --------------------------------------------------------
> printf "%s\n" "${STORES[@]}" \
> | xargs -P"$PARALLEL" -I{} bash -c '
> store="$1"
> url="$HOST/api2/json/admin/datastore/$store/status?verbose=0"
>
> for ((i=1;i<=REPEAT;i++)); do
> code=$(curl -ks -o /dev/null -w "%{http_code}" -b "PBSAuthCookie=$ticket" "$url" || echo 000)
>
> if [[ "$code" == "200" ]]; then
> echo 1 >> "$SUCCESS_ALL"
> if (( i == 1 )); then
> echo 1 >> "$COLD_OK"
> else
> echo 1 >> "$WARM_OK"
> fi
> if [[ "$PRINT_FIRST" == "true" && $i -eq 1 ]]; then
> ts=$(date +%H:%M:%S)
> echo "[$ts] $store #$i HTTP:200"
> fi
> else
> echo 1 >> "$FAIL_ALL"
> if [[ "$PRINT_FIRST" == "true" && $i -eq 1 ]]; then
> ts=$(date +%H:%M:%S)
> echo "[$ts] $store #$i HTTP:$code (FAIL)"
> fi
> fi
> done
> ' _ {}
>
> # --- Summary --------------------------------------------------------------
> elapsed=$SECONDS
> ok=$(wc -l < "$SUCCESS_ALL" 2>/dev/null || echo 0)
> fail=$(wc -l < "$FAIL_ALL" 2>/dev/null || echo 0)
> cold_ok=$(wc -l < "$COLD_OK" 2>/dev/null || echo 0)
> warm_ok=$(wc -l < "$WARM_OK" 2>/dev/null || echo 0)
>
> expected=$(( USED_STORES * REPEAT ))
> total=$(( ok + fail ))
>
> rps_all=$(fmt_rps "$ok" "$elapsed")
> rps_cold=$(fmt_rps "$cold_ok" "$elapsed")
> rps_warm=$(fmt_rps "$warm_ok" "$elapsed")
>
> echo "===== Summary ====="
> echo "Stores used: $USED_STORES"
> echo "Expected requests: $expected"
> echo "Executed requests: $total"
> echo "OK (HTTP 200): $ok"
> echo "Failed: $fail"
> printf "Total time: %dm %ds\n" $((elapsed/60)) $((elapsed%60))
> echo "Throughput all RPS: $rps_all"
> echo "Cold RPS (round #1): $rps_cold"
> echo "Warm RPS (#2..N): $rps_warm"
>
> ## Maintainer notes
>
> No dependency bumps, no API changes and no breaking changes.
>
> ## Patch summary
>
> [PATCH 1/4] partial fix #6049: config: enable config version cache for datastore
> [PATCH 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
> [PATCH 3/4] partial fix #6049: datastore: use config fast-path in Drop
> [PATCH 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
>
> Thanks,
> Samuel
>
> [1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
> [3] Bugzilla #7017: https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>
> Samuel Rufinatscha (4):
> partial fix #6049: config: enable config version cache for datastore
> partial fix #6049: datastore: impl ConfigVersionCache fast path for
> lookups
> partial fix #6049: datastore: use config fast-path in Drop
> partial fix #6049: datastore: add TTL fallback to catch manual config
> edits
>
> pbs-config/src/config_version_cache.rs | 10 +-
> pbs-datastore/Cargo.toml | 1 +
> pbs-datastore/src/datastore.rs | 187 +++++++++++++++++++------
> 3 files changed, 152 insertions(+), 46 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* Re: [pbs-devel] [PATCH proxmox-backup v3 6/6] datastore: only bump generation when config digest changes
2025-11-20 13:03 15% ` [pbs-devel] [PATCH proxmox-backup v3 6/6] datastore: only bump generation when config digest changes Samuel Rufinatscha
@ 2025-11-20 14:50 5% ` Fabian Grünbichler
2025-11-21 8:37 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-11-20 14:50 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On November 20, 2025 2:03 pm, Samuel Rufinatscha wrote:
> When reloading datastore.cfg in datastore_section_config_cached(),
> we currently bump the datastore generation unconditionally. This is
> only necessary when the on disk content actually changed and when
> we already had a previous cached entry.
>
> This patch extends the DatastoreConfigCache to store the last digest of
> datastore.cfg and track the previously cached generation and digest.
> Only when the digest differs from the cached one. On first load, it
> reuses the existing datastore_generation without bumping.
>
> This avoids unnecessary cache invalidations if the config did not
> change.
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> pbs-datastore/src/datastore.rs | 43 ++++++++++++++++++++++++----------
> 1 file changed, 30 insertions(+), 13 deletions(-)
>
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 12076f31..bf04332e 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -51,6 +51,8 @@ use crate::{DataBlob, LocalDatastoreLruCache};
> struct DatastoreConfigCache {
> // Parsed datastore.cfg file
> config: Arc<SectionConfigData>,
> + // Digest of the datastore.cfg file
> + last_digest: [u8; 32],
> // Generation number from ConfigVersionCache
> last_generation: usize,
> // Last update time (epoch seconds)
> @@ -349,29 +351,44 @@ fn datastore_section_config_cached(
> }
>
> // Slow path: re-read datastore.cfg
> - let (config_raw, _digest) = pbs_config::datastore::config()?;
> + let (config_raw, digest) = pbs_config::datastore::config()?;
> let config = Arc::new(config_raw);
>
> - // Update cache
> + // Decide whether to bump the shared generation.
> + // Only bump if we already had a cached generation and the digest changed (manual edit or API write)
> + let (prev_gen, prev_digest) = guard
> + .as_ref()
> + .map(|c| (Some(c.last_generation), Some(c.last_digest)))
> + .unwrap_or((None, None));
> +
> let new_gen = if let Some(handle) = version_cache {
> - // Bump datastore generation whenever we reload the config.
> - // This ensures that Drop handlers will detect that a newer config exists
> - // and will not rely on a stale cached entry for maintenance mandate.
> - let prev_gen = handle.increase_datastore_generation();
> - let new_gen = prev_gen + 1;
> + match (prev_gen, prev_digest) {
> + // We had a previous generation and the digest changed => bump generation.
> + (Some(_prev_gen), Some(prev_digest)) if prev_digest != digest => {
this is not quite the correct logic - I think.
we only need to bump *iff* the digest doesn't match, but the generation
does - that implies somebody changed the config behind our back.
if the generation is different, we should *expect* the digest to also
not be identical, but we don't have to care in that case, since the
generation was already bumped (compared to the last cached state with
the different digest), and that invalidates all the old cache references
anyway..
again, I think this would be easier to follow along if the structure of
the ifs is changed ;)
> + let old = handle.increase_datastore_generation();
> + Some(old + 1)
> + }
> + // We had a previous generation but the digest stayed the same:
> + // keep the existing generation, just refresh the timestamp.
> + (Some(prev_gen), _) => Some(prev_gen),
> + // We didn't have a previous generation, just use the current one.
> + (None, _) => Some(handle.datastore_generation()),
> + }
> + } else {
> + None
> + };
>
> + if let Some(gen_val) = new_gen {
> *guard = Some(DatastoreConfigCache {
> config: config.clone(),
> - last_generation: new_gen,
> + last_digest: digest,
> + last_generation: gen_val,
> last_update: now,
> });
> -
> - Some(new_gen)
> } else {
> - // if the cache was not available, use again the slow path next time
> + // If the shared version cache is not available, don't cache.
> *guard = None;
> - None
> - };
> + }
>
> Ok((config, new_gen))
> }
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v3 5/6] partial fix #6049: datastore: add reload flag to config cache helper
2025-11-20 13:03 15% ` [pbs-devel] [PATCH proxmox-backup v3 5/6] partial fix #6049: datastore: add reload flag to config cache helper Samuel Rufinatscha
@ 2025-11-20 14:50 5% ` Fabian Grünbichler
2025-11-20 18:15 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-11-20 14:50 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On November 20, 2025 2:03 pm, Samuel Rufinatscha wrote:
> Extend datastore_section_config_cached() with an `allow_reload` flag to
> separate two use cases:
>
> 1) lookup_datastore() passes `true` and is allowed to reload
> datastore.cfg from disk when the cache is missing, the generation
> changed or the TTL expired. The helper may bump the datastore
> generation if the digest changed.
>
> 2) DataStore::drop() passes `false` and only consumes the most recent
> cached entry without touching the disk, TTL or generation. If the
> cache was never initialised, it returns an error.
>
> This avoids races between Drop and concurrent config changes.
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> pbs-datastore/src/datastore.rs | 36 ++++++++++++++++++++++++++++++----
> 1 file changed, 32 insertions(+), 4 deletions(-)
>
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 1711c753..12076f31 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -226,7 +226,7 @@ impl Drop for DataStore {
> return;
> }
>
> - let (section_config, _gen) = match datastore_section_config_cached() {
> + let (section_config, _gen) = match datastore_section_config_cached(false) {
> Ok(v) => v,
> Err(err) => {
> log::error!(
> @@ -299,14 +299,42 @@ impl DatastoreBackend {
> }
> }
>
> -/// Return the cached datastore SectionConfig and its generation.
> -fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
> +/// Returns the cached `datastore.cfg` and its generation.
> +///
> +/// When `allow_reload` is `true`, callers are expected to hold the datastore config. It may:
> +/// - Reload `datastore.cfg` from disk if either
> +/// - no cache exists yet, or cache is unavailable
> +/// - the cached generation does not match the shared generation
> +/// - the cache entry is older than `DATASTORE_CONFIG_CACHE_TTL_SECS`
> +/// - Updates the cache with the new config, timestamp and digest.
> +/// - Bumps the datastore generation in `ConfigVersionCache` only if
> +/// there was a previous cached entry and the digest changed (manual edit or
> +/// API write). If the digest is unchanged, the timestamp is refreshed but the
> +/// generation is kept to avoid unnecessary invalidations.
> +///
> +/// When `allow_reload` is `false`:
> +/// - Never touches the disk or the shared generation.
> +/// - Ignores TTL and simply returns the most recent cached entry if available.
> +/// - Returns an error if the cache has not been initialised yet.
> +///
> +/// Intended for use with `Datastore::drop` where no config lock is held
> +/// and eventual stale data is acceptable.
> +fn datastore_section_config_cached(
> + allow_reload: bool,
> +) -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
> let now = epoch_i64();
> let version_cache = ConfigVersionCache::new().ok();
> let current_gen = version_cache.as_ref().map(|c| c.datastore_generation());
>
> let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
>
> + if !allow_reload {
> + if let Some(cache) = guard.as_ref() {
> + return Ok((cache.config.clone(), Some(cache.last_generation)));
> + }
> + bail!("datastore config cache not initialized");
> + }
this is not quite what I intended, we are actually allowed to reload,
just not bump the generation number and store the result ;) the
difference is basically whether we
- hold the lock and can be sure that nothing modifies the
config/generation number while we do the lookup and bump
- don't hold the lock and can just compare and reload, but not bump and
persist
if the code is restructured then this is should boil down to an if
wrapping the generation bump and cache update, leaving the rest as it
was..
> +
> // Fast path: re-use cached datastore.cfg if cache is available, generation matches and TTL not expired
> if let (Some(current_gen), Some(config_cache)) = (current_gen, guard.as_ref()) {
> let gen_matches = config_cache.last_generation == current_gen;
> @@ -423,7 +451,7 @@ impl DataStore {
> let _config_lock = pbs_config::datastore::lock_config()?;
>
> // Get the current datastore.cfg generation number and cached config
> - let (section_config, gen_num) = datastore_section_config_cached()?;
> + let (section_config, gen_num) = datastore_section_config_cached(true)?;
>
> let datastore_cfg: DataStoreConfig = section_config.lookup("datastore", name)?;
> let maintenance_mode = datastore_cfg.get_maintenance_mode();
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v3 0/6] datastore: remove config reload on hot path
2025-11-20 13:03 10% [pbs-devel] [PATCH proxmox-backup v3 0/6] " Samuel Rufinatscha
` (5 preceding siblings ...)
2025-11-20 13:03 15% ` [pbs-devel] [PATCH proxmox-backup v3 6/6] datastore: only bump generation when config digest changes Samuel Rufinatscha
@ 2025-11-20 14:50 5% ` Fabian Grünbichler
2025-11-20 15:17 6% ` Samuel Rufinatscha
2025-11-24 15:35 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
7 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-11-20 14:50 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On November 20, 2025 2:03 pm, Samuel Rufinatscha wrote:
> Hi,
>
> [..]
nit: this is getting a bit long ;)
>
> ## Patch summary
>
> [PATCH 1/6] partial fix #6049: config: enable config version cache for datastore
> [PATCH 2/6] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
> [PATCH 3/6] partial fix #6049: datastore: use config fast-path in Drop
> [PATCH 4/6] partial fix #6049: datastore: add TTL fallback to catch manual config edits
> [PATCH 5/6] to add a reload flag to the config cache helper.
> [PATCH 6/6] to only bump generation when the config digest changes.
>
> ## Changes from v2:
>
> Added:
> - [PATCH 5/6]: Add a reload flag to the config cache helper.
> - [PATCH 6/6]: Only bump generation when the config digest changes.
please fold those into the existing version where they make sense, and
include a per-patch changelog to know *what* changed ;)
e.g., the digest part can already go into the first patch (if the
generation bumping is also moved thre from patch #4), or into patch #4.
the structural changes I suggested are missing, and I think the
readability got worse as a result since v2, we now have six instances of
checking whether there is some cache we are operating on or not..
I'll give more detailed feedback on the two new patches..
>
> ## Maintainer notes
>
> No dependency bumps, no API changes and no breaking changes.
>
> Thanks,
> Samuel
>
> [1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
> [3] Bugzilla #7017: https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>
> Samuel Rufinatscha (6):
> partial fix #6049: config: enable config version cache for datastore
> partial fix #6049: datastore: impl ConfigVersionCache fast path for
> lookups
> partial fix #6049: datastore: use config fast-path in Drop
> partial fix #6049: datastore: add TTL fallback to catch manual config
> edits
> partial fix #6049: datastore: add reload flag to config cache helper
> datastore: only bump generation when config digest changes
>
> pbs-config/src/config_version_cache.rs | 10 +-
> pbs-datastore/Cargo.toml | 1 +
> pbs-datastore/src/datastore.rs | 232 ++++++++++++++++++++-----
> 3 files changed, 197 insertions(+), 46 deletions(-)
>
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v3 0/6] datastore: remove config reload on hot path
2025-11-20 14:50 5% ` [pbs-devel] [PATCH proxmox-backup v3 0/6] datastore: remove config reload on hot path Fabian Grünbichler
@ 2025-11-20 15:17 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-20 15:17 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 11/20/25 3:50 PM, Fabian Grünbichler wrote:
> On November 20, 2025 2:03 pm, Samuel Rufinatscha wrote:
>> Hi,
>>
>> [..]
>
> nit: this is getting a bit long ;)
>
>>
>> ## Patch summary
>>
>> [PATCH 1/6] partial fix #6049: config: enable config version cache for datastore
>> [PATCH 2/6] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
>> [PATCH 3/6] partial fix #6049: datastore: use config fast-path in Drop
>> [PATCH 4/6] partial fix #6049: datastore: add TTL fallback to catch manual config edits
>> [PATCH 5/6] to add a reload flag to the config cache helper.
>> [PATCH 6/6] to only bump generation when the config digest changes.
>>
>> ## Changes from v2:
>>
>> Added:
>> - [PATCH 5/6]: Add a reload flag to the config cache helper.
>> - [PATCH 6/6]: Only bump generation when the config digest changes.
>
> please fold those into the existing version where they make sense, and
> include a per-patch changelog to know *what* changed ;)
>
> e.g., the digest part can already go into the first patch (if the
> generation bumping is also moved thre from patch #4), or into patch #4.
>
> the structural changes I suggested are missing, and I think the
> readability got worse as a result since v2, we now have six instances of
> checking whether there is some cache we are operating on or not..
>
> I'll give more detailed feedback on the two new patches..
>
Thanks for the review Fabian! I actually somehow missed your comment on
PATCH 2/4 v2, sorry for that! Will make sure its included in the new
version for sure.
>>
>> ## Maintainer notes
>>
>> No dependency bumps, no API changes and no breaking changes.
>>
>> Thanks,
>> Samuel
>>
>> [1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
>> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>> [3] Bugzilla #7017: https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>>
>> Samuel Rufinatscha (6):
>> partial fix #6049: config: enable config version cache for datastore
>> partial fix #6049: datastore: impl ConfigVersionCache fast path for
>> lookups
>> partial fix #6049: datastore: use config fast-path in Drop
>> partial fix #6049: datastore: add TTL fallback to catch manual config
>> edits
>> partial fix #6049: datastore: add reload flag to config cache helper
>> datastore: only bump generation when config digest changes
>>
>> pbs-config/src/config_version_cache.rs | 10 +-
>> pbs-datastore/Cargo.toml | 1 +
>> pbs-datastore/src/datastore.rs | 232 ++++++++++++++++++++-----
>> 3 files changed, 197 insertions(+), 46 deletions(-)
>>
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup v3 5/6] partial fix #6049: datastore: add reload flag to config cache helper
2025-11-20 14:50 5% ` Fabian Grünbichler
@ 2025-11-20 18:15 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-20 18:15 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 11/20/25 3:50 PM, Fabian Grünbichler wrote:
> On November 20, 2025 2:03 pm, Samuel Rufinatscha wrote:
>> Extend datastore_section_config_cached() with an `allow_reload` flag to
>> separate two use cases:
>>
>> 1) lookup_datastore() passes `true` and is allowed to reload
>> datastore.cfg from disk when the cache is missing, the generation
>> changed or the TTL expired. The helper may bump the datastore
>> generation if the digest changed.
>>
>> 2) DataStore::drop() passes `false` and only consumes the most recent
>> cached entry without touching the disk, TTL or generation. If the
>> cache was never initialised, it returns an error.
>>
>> This avoids races between Drop and concurrent config changes.
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> pbs-datastore/src/datastore.rs | 36 ++++++++++++++++++++++++++++++----
>> 1 file changed, 32 insertions(+), 4 deletions(-)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index 1711c753..12076f31 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -226,7 +226,7 @@ impl Drop for DataStore {
>> return;
>> }
>>
>> - let (section_config, _gen) = match datastore_section_config_cached() {
>> + let (section_config, _gen) = match datastore_section_config_cached(false) {
>> Ok(v) => v,
>> Err(err) => {
>> log::error!(
>> @@ -299,14 +299,42 @@ impl DatastoreBackend {
>> }
>> }
>>
>> -/// Return the cached datastore SectionConfig and its generation.
>> -fn datastore_section_config_cached() -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
>> +/// Returns the cached `datastore.cfg` and its generation.
>> +///
>> +/// When `allow_reload` is `true`, callers are expected to hold the datastore config. It may:
>> +/// - Reload `datastore.cfg` from disk if either
>> +/// - no cache exists yet, or cache is unavailable
>> +/// - the cached generation does not match the shared generation
>> +/// - the cache entry is older than `DATASTORE_CONFIG_CACHE_TTL_SECS`
>> +/// - Updates the cache with the new config, timestamp and digest.
>> +/// - Bumps the datastore generation in `ConfigVersionCache` only if
>> +/// there was a previous cached entry and the digest changed (manual edit or
>> +/// API write). If the digest is unchanged, the timestamp is refreshed but the
>> +/// generation is kept to avoid unnecessary invalidations.
>> +///
>> +/// When `allow_reload` is `false`:
>> +/// - Never touches the disk or the shared generation.
>> +/// - Ignores TTL and simply returns the most recent cached entry if available.
>> +/// - Returns an error if the cache has not been initialised yet.
>> +///
>> +/// Intended for use with `Datastore::drop` where no config lock is held
>> +/// and eventual stale data is acceptable.
>> +fn datastore_section_config_cached(
>> + allow_reload: bool,
>> +) -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
>> let now = epoch_i64();
>> let version_cache = ConfigVersionCache::new().ok();
>> let current_gen = version_cache.as_ref().map(|c| c.datastore_generation());
>>
>> let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
>>
>> + if !allow_reload {
>> + if let Some(cache) = guard.as_ref() {
>> + return Ok((cache.config.clone(), Some(cache.last_generation)));
>> + }
>> + bail!("datastore config cache not initialized");
>> + }
>
> this is not quite what I intended, we are actually allowed to reload,
> just not bump the generation number and store the result ;) the
> difference is basically whether we
> - hold the lock and can be sure that nothing modifies the
> config/generation number while we do the lookup and bump
> - don't hold the lock and can just compare and reload, but not bump and
> persist
>
> if the code is restructured then this is should boil down to an if
> wrapping the generation bump and cache update, leaving the rest as it
> was..
>
Makes sense, thanks Fabian! I will restructure it and fix the flag
check. The check should then wrap only bump and update as you
suggested. I think it could look like this:
fn datastore_section_config_cached(
update_cache_and_generation: bool,
) -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
let mut guard = DATASTORE_CONFIG_CACHE.lock().unwrap();
if let Some(version_cache) = ConfigVersionCache::new().ok() {
let now = epoch_i64();
let current_gen = version_cache.datastore_generation();
if let Some(cached) = guard.as_ref() {
// Fast path: re-use cached datastore.cfg if cache is
available, generation matches and TTL not expired
if cached.last_generation == current_gen
&& now - cached.last_update <
DATASTORE_CONFIG_CACHE_TTL_SECS
{
return Ok((cached.config.clone(),
Some(cached.last_generation)));
}
}
// Slow path: re-read datastore.cfg
let (config_raw, digest) = pbs_config::datastore::config()?;
let config = Arc::new(config_raw);
let mut effective_gen = current_gen;
if update_cache_and_generation {
let (prev_gen, prev_digest) = guard
.as_ref()
.map(|c| (Some(c.last_generation), Some(c.digest)))
.unwrap_or((None, None));
let manual_edit = match (prev_gen, prev_digest) {
(Some(prev_g), Some(prev_d)) => prev_g == current_gen
&& prev_d != digest,
_ => false,
};
if manual_edit {
let old = version_cache.increase_datastore_generation();
effective_gen = old + 1;
}
// Update cache
*guard = Some(DatastoreConfigCache {
config: config.clone(),
digest,
last_generation: effective_gen,
last_update: now,
});
}
Ok((config, Some(effective_gen)))
} else {
// Fallback path, no config version cache: read datastore.cfg
*guard = None;
let (config_raw, _digest) = pbs_config::datastore::config()?;
Ok((Arc::new(config_raw), None))
}
}
>> +
>> // Fast path: re-use cached datastore.cfg if cache is available, generation matches and TTL not expired
>> if let (Some(current_gen), Some(config_cache)) = (current_gen, guard.as_ref()) {
>> let gen_matches = config_cache.last_generation == current_gen;
>> @@ -423,7 +451,7 @@ impl DataStore {
>> let _config_lock = pbs_config::datastore::lock_config()?;
>>
>> // Get the current datastore.cfg generation number and cached config
>> - let (section_config, gen_num) = datastore_section_config_cached()?;
>> + let (section_config, gen_num) = datastore_section_config_cached(true)?;
>>
>> let datastore_cfg: DataStoreConfig = section_config.lookup("datastore", name)?;
>> let maintenance_mode = datastore_cfg.get_maintenance_mode();
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup v3 6/6] datastore: only bump generation when config digest changes
2025-11-20 14:50 5% ` Fabian Grünbichler
@ 2025-11-21 8:37 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-21 8:37 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 11/20/25 3:50 PM, Fabian Grünbichler wrote:
> On November 20, 2025 2:03 pm, Samuel Rufinatscha wrote:
>> When reloading datastore.cfg in datastore_section_config_cached(),
>> we currently bump the datastore generation unconditionally. This is
>> only necessary when the on disk content actually changed and when
>> we already had a previous cached entry.
>>
>> This patch extends the DatastoreConfigCache to store the last digest of
>> datastore.cfg and track the previously cached generation and digest.
>> Only when the digest differs from the cached one. On first load, it
>> reuses the existing datastore_generation without bumping.
>>
>> This avoids unnecessary cache invalidations if the config did not
>> change.
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> pbs-datastore/src/datastore.rs | 43 ++++++++++++++++++++++++----------
>> 1 file changed, 30 insertions(+), 13 deletions(-)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index 12076f31..bf04332e 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -51,6 +51,8 @@ use crate::{DataBlob, LocalDatastoreLruCache};
>> struct DatastoreConfigCache {
>> // Parsed datastore.cfg file
>> config: Arc<SectionConfigData>,
>> + // Digest of the datastore.cfg file
>> + last_digest: [u8; 32],
>> // Generation number from ConfigVersionCache
>> last_generation: usize,
>> // Last update time (epoch seconds)
>> @@ -349,29 +351,44 @@ fn datastore_section_config_cached(
>> }
>>
>> // Slow path: re-read datastore.cfg
>> - let (config_raw, _digest) = pbs_config::datastore::config()?;
>> + let (config_raw, digest) = pbs_config::datastore::config()?;
>> let config = Arc::new(config_raw);
>>
>> - // Update cache
>> + // Decide whether to bump the shared generation.
>> + // Only bump if we already had a cached generation and the digest changed (manual edit or API write)
>> + let (prev_gen, prev_digest) = guard
>> + .as_ref()
>> + .map(|c| (Some(c.last_generation), Some(c.last_digest)))
>> + .unwrap_or((None, None));
>> +
>> let new_gen = if let Some(handle) = version_cache {
>> - // Bump datastore generation whenever we reload the config.
>> - // This ensures that Drop handlers will detect that a newer config exists
>> - // and will not rely on a stale cached entry for maintenance mandate.
>> - let prev_gen = handle.increase_datastore_generation();
>> - let new_gen = prev_gen + 1;
>> + match (prev_gen, prev_digest) {
>> + // We had a previous generation and the digest changed => bump generation.
>> + (Some(_prev_gen), Some(prev_digest)) if prev_digest != digest => {
>
> this is not quite the correct logic - I think.
>
> we only need to bump *iff* the digest doesn't match, but the generation
> does - that implies somebody changed the config behind our back.
>
> if the generation is different, we should *expect* the digest to also
> not be identical, but we don't have to care in that case, since the
> generation was already bumped (compared to the last cached state with
> the different digest), and that invalidates all the old cache references
> anyway..
>
Makes sense and good point! I will restrict bumping here for the case
you mentioned (*iff* the digest doesn't match, but the generation does).
So in the case the generation is different, we can rely on the current gen.
> again, I think this would be easier to follow along if the structure of
> the ifs is changed ;)
>
I agree, changing :-)
>> + let old = handle.increase_datastore_generation();
>> + Some(old + 1)
>> + }
>> + // We had a previous generation but the digest stayed the same:
>> + // keep the existing generation, just refresh the timestamp.
>> + (Some(prev_gen), _) => Some(prev_gen),
>> + // We didn't have a previous generation, just use the current one.
>> + (None, _) => Some(handle.datastore_generation()),
>> + }
>> + } else {
>> + None
>> + };
>>
>> + if let Some(gen_val) = new_gen {
>> *guard = Some(DatastoreConfigCache {
>> config: config.clone(),
>> - last_generation: new_gen,
>> + last_digest: digest,
>> + last_generation: gen_val,
>> last_update: now,
>> });
>> -
>> - Some(new_gen)
>> } else {
>> - // if the cache was not available, use again the slow path next time
>> + // If the shared version cache is not available, don't cache.
>> *guard = None;
>> - None
>> - };
>> + }
>>
>> Ok((config, new_gen))
>> }
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* [pbs-devel] [PATCH proxmox-backup v4 0/4] datastore: remove config reload on hot path
@ 2025-11-24 15:33 12% Samuel Rufinatscha
2025-11-24 15:33 16% ` [pbs-devel] [PATCH proxmox-backup v4 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
` (4 more replies)
0 siblings, 5 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-24 15:33 UTC (permalink / raw)
To: pbs-devel
Hi,
this series reduces CPU time in datastore lookups by avoiding repeated
datastore.cfg reads/parses in both `lookup_datastore()` and
`DataStore::Drop`. It also adds a TTL so manual config edits are
noticed without reintroducing hashing on every request.
While investigating #6049 [1], cargo-flamegraph [2] showed hotspots
during repeated `/status` calls in `lookup_datastore()` and in `Drop`,
dominated by `pbs_config::datastore::config()` (config parse).
The parsing cost itself should eventually be investigated in a future
effort. Furthermore, cargo-flamegraph showed that when using a
token-based auth method to access the API, a significant amount of time
is spent in validation on every request request [3].
## Approach
[PATCH 1/4] Support datastore generation in ConfigVersionCache
[PATCH 2/4] Fast path for datastore lookups
Cache the parsed datastore.cfg keyed by the shared datastore
generation. lookup_datastore() reuses both the cached config and an
existing DataStoreImpl when the generation matches, and falls back
to the old slow path otherwise. The caching logic is implemented
using the datastore_section_config_cached(update_cache: bool) helper.
[PATCH 3/4] Fast path for Drop
Make DataStore::Drop use the datastore_section_config_cached()
helper to avoid re-reading/parsing datastore.cfg on every Drop.
Bump generation not only on API config saves, but also on slow-path
lookups (if update_cache is true), to enable Drop handlers see
eventual newer configs.
[PATCH 4/4] TTL to catch manual edits
Add a TTL to the cached config and bump the datastore generation iff
the digest changed but generation stays the same. This catches manual
edits to datastore.cfg without reintroducing hashing or config
parsing on every request.
## Benchmark results
### End-to-end
Testing `/status?verbose=0` end-to-end with 1000 stores, 5 req/store
and parallel=16 before/after the series:
Metric Before After
----------------------------------------
Total time 12s 9s
Throughput (all) 416.67 555.56
Cold RPS (round #1) 83.33 111.11
Warm RPS (#2..N) 333.33 444.44
Running under flamegraph [2], TLS appears to consume a significant
amount of CPU time and blur the results. Still, a ~33% higher overall
throughput and ~25% less end-to-end time for this workload.
### Isolated benchmarks (hyperfine)
In addition to the end-to-end tests, I measured two standalone
benchmarks with hyperfine, each using a config with 1000 datastores.
`M` is the number of distinct datastores looked up and
`N` is the number of lookups per datastore.
Drop-direct variant:
Drops the `DataStore` after every lookup, so the `Drop` path runs on
every iteration:
use anyhow::Error;
use pbs_api_types::Operation;
use pbs_datastore::DataStore;
fn main() -> Result<(), Error> {
let mut args = std::env::args();
args.next();
let datastores = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
let iterations = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
for d in 1..=datastores {
let name = format!("ds{:04}", d);
for i in 1..=iterations {
DataStore::lookup_datastore(&name, Some(Operation::Write))?;
}
}
Ok(())
}
+----+------+-----------+-----------+---------+
| M | N | Baseline | Patched | Speedup |
+----+------+-----------+-----------+---------+
| 1 | 1000 | 1.684 s | 35.3 ms | 47.7x |
| 10 | 100 | 1.689 s | 35.0 ms | 48.3x |
| 100| 10 | 1.709 s | 35.8 ms | 47.7x |
|1000| 1 | 1.809 s | 39.0 ms | 46.4x |
+----+------+-----------+-----------+---------+
Bulk-drop variant:
Keeps the `DataStore` instances alive for
all `N` lookups of a given datastore and then drops them in bulk,
mimicking a task that performs many lookups while it is running and
only triggers the expensive `Drop` logic when the last user exits.
use anyhow::Error;
use pbs_api_types::Operation;
use pbs_datastore::DataStore;
fn main() -> Result<(), Error> {
let mut args = std::env::args();
args.next();
let datastores = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
let iterations = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
for d in 1..=datastores {
let name = format!("ds{:04}", d);
let mut stores = Vec::with_capacity(iterations);
for i in 1..=iterations {
stores.push(DataStore::lookup_datastore(&name, Some(Operation::Write))?);
}
}
Ok(())
}
+------+------+---------------+--------------+---------+
| M | N | Baseline mean | Patched mean | Speedup |
+------+------+---------------+--------------+---------+
| 1 | 1000 | 890.6 ms | 35.5 ms | 25.1x |
| 10 | 100 | 891.3 ms | 35.1 ms | 25.4x |
| 100 | 10 | 983.9 ms | 35.6 ms | 27.6x |
| 1000 | 1 | 1829.0 ms | 45.2 ms | 40.5x |
+------+------+---------------+--------------+---------+
Both variants show that the combination of the cached config lookups
and the cheaper `Drop` handling reduces the hot-path cost from ~1.8 s
per run to a few tens of milliseconds in these benchmarks.
## Reproduction steps
VM: 4 vCPU, ~8 GiB RAM, VirtIO-SCSI; disks:
- scsi0 32G (OS)
- scsi1 1000G (datastores)
Install PBS from ISO on the VM.
Set up ZFS on /dev/sdb (adjust if different):
zpool create -f -o ashift=12 pbsbench /dev/sdb
zfs set mountpoint=/pbsbench pbsbench
zfs create pbsbench/pbs-bench
Raise file-descriptor limit:
sudo systemctl edit proxmox-backup-proxy.service
Add the following lines:
[Service]
LimitNOFILE=1048576
Reload systemd and restart the proxy:
sudo systemctl daemon-reload
sudo systemctl restart proxmox-backup-proxy.service
Verify the limit:
systemctl show proxmox-backup-proxy.service | grep LimitNOFILE
Create 1000 ZFS-backed datastores (as used in #6049 [1]):
seq -w 001 1000 | xargs -n1 -P1 bash -c '
id=$0
name="ds${id}"
dataset="pbsbench/pbs-bench/${name}"
path="/pbsbench/pbs-bench/${name}"
zfs create -o mountpoint="$path" "$dataset"
proxmox-backup-manager datastore create "$name" "$path" \
--comment "ZFS dataset-based datastore"
'
Build PBS from this series, then run the server under manually
under flamegraph:
systemctl stop proxmox-backup-proxy
cargo flamegraph --release --bin proxmox-backup-proxy
## Patch summary
[PATCH 1/4] partial fix #6049: config: enable config version cache for datastore
[PATCH 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
[PATCH 3/4] partial fix #6049: datastore: use config fast-path in Drop
[PATCH 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
## Maintainer notes
No dependency bumps, no API changes and no breaking changes.
Thanks,
Samuel
Links
[1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
[3] Bugzilla #7017: https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Samuel Rufinatscha (4):
partial fix #6049: config: enable config version cache for datastore
partial fix #6049: datastore: impl ConfigVersionCache fast path for
lookups
partial fix #6049: datastore: use config fast-path in Drop
partial fix #6049: datastore: add TTL fallback to catch manual config
edits
pbs-config/src/config_version_cache.rs | 10 +-
pbs-datastore/Cargo.toml | 1 +
pbs-datastore/src/datastore.rs | 215 ++++++++++++++++++++-----
3 files changed, 180 insertions(+), 46 deletions(-)
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] [PATCH proxmox-backup v4 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
2025-11-24 15:33 12% [pbs-devel] [PATCH proxmox-backup v4 0/4] " Samuel Rufinatscha
` (2 preceding siblings ...)
2025-11-24 15:33 14% ` [pbs-devel] [PATCH proxmox-backup v4 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
@ 2025-11-24 15:33 13% ` Samuel Rufinatscha
2025-11-24 17:06 13% ` [pbs-devel] superseded: [PATCH proxmox-backup v4 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-24 15:33 UTC (permalink / raw)
To: pbs-devel
The lookup fast path reacts to API-driven config changes because
save_config() bumps the generation. Manual edits of datastore.cfg do
not bump the counter. To keep the system robust against such edits
without reintroducing config reading and hashing on the hot path, this
patch adds a TTL to the cache entry.
If the cached config is older than
DATASTORE_CONFIG_CACHE_TTL_SECS (set to 60s), the next lookup takes
the slow path and refreshes the entry. As an optimization, a check to
catch manual edits was added (if the digest changed but generation
stayed the same), so that the generation is only bumped when needed.
Links
[1] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes:
From v1 → v2
- Store last_update timestamp in DatastoreConfigCache type.
From v2 → v3
No changes
From v3 → v4
- Fix digest generation bump logic in update_cache, thanks @Fabian.
pbs-datastore/src/datastore.rs | 55 ++++++++++++++++++++++++----------
1 file changed, 39 insertions(+), 16 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 942656e6..a5c450d0 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -22,7 +22,7 @@ use proxmox_sys::error::SysError;
use proxmox_sys::fs::{file_read_optional_string, replace_file, CreateOptions};
use proxmox_sys::linux::procfs::MountInfo;
use proxmox_sys::process_locker::{ProcessLockExclusiveGuard, ProcessLockSharedGuard};
-use proxmox_time::TimeSpan;
+use proxmox_time::{epoch_i64, TimeSpan};
use proxmox_worker_task::WorkerTaskContext;
use pbs_api_types::{
@@ -51,8 +51,12 @@ use crate::{DataBlob, LocalDatastoreLruCache};
struct DatastoreConfigCache {
// Parsed datastore.cfg file
config: Arc<SectionConfigData>,
+ // Digest of the datastore.cfg file
+ digest: [u8; 32],
// Generation number from ConfigVersionCache
last_generation: usize,
+ // Last update time (epoch seconds)
+ last_update: i64,
}
static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
@@ -61,6 +65,8 @@ static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
+/// Max age in seconds to reuse the cached datastore config.
+const DATASTORE_CONFIG_CACHE_TTL_SECS: i64 = 60;
/// Filename to store backup group notes
pub const GROUP_NOTES_FILE_NAME: &str = "notes";
/// Filename to store backup group owner
@@ -299,13 +305,14 @@ impl DatastoreBackend {
/// generation.
///
/// Uses `ConfigVersionCache` to detect stale entries:
-/// - If the cached generation matches the current generation, the
-/// cached config is returned.
+/// - If the cached generation matches the current generation and TTL is
+/// OK, the cached config is returned.
/// - Otherwise the config is re-read from disk. If `update_cache` is
-/// `true`, the new config and bumped generation are stored in the
-/// cache. Callers that set `update_cache = true` must hold the
-/// datastore config lock to avoid racing with concurrent config
-/// changes.
+/// `true` and a previous cached entry exists with the same generation
+/// but a different digest, this indicates the config has changed
+/// (e.g. manual edit) and the generation must be bumped. Callers
+/// that set `update_cache = true` must hold the datastore config lock
+/// to avoid racing with concurrent config changes.
/// - If `update_cache` is `false`, the freshly read config is returned
/// but the cache and generation are left unchanged.
///
@@ -317,30 +324,46 @@ fn datastore_section_config_cached(
let mut config_cache = DATASTORE_CONFIG_CACHE.lock().unwrap();
if let Ok(version_cache) = ConfigVersionCache::new() {
+ let now = epoch_i64();
let current_gen = version_cache.datastore_generation();
if let Some(cached) = config_cache.as_ref() {
- // Fast path: re-use cached datastore.cfg
- if cached.last_generation == current_gen {
+ // Fast path: re-use cached datastore.cfg if generation matches and TTL not expired
+ if cached.last_generation == current_gen
+ && now - cached.last_update < DATASTORE_CONFIG_CACHE_TTL_SECS
+ {
return Ok((cached.config.clone(), Some(cached.last_generation)));
}
}
// Slow path: re-read datastore.cfg
- let (config_raw, _digest) = pbs_config::datastore::config()?;
+ let (config_raw, digest) = pbs_config::datastore::config()?;
let config = Arc::new(config_raw);
let mut effective_gen = current_gen;
if update_cache {
- // Bump the generation. This ensures that Drop
- // handlers will detect that a newer config exists
- // and will not rely on a stale cached entry for
- // maintenance mandate.
- let prev_gen = version_cache.increase_datastore_generation();
- effective_gen = prev_gen + 1;
+ // Bump the generation if the config has been changed manually.
+ // This ensures that Drop handlers will detect that a newer config exists
+ // and will not rely on a stale cached entry for maintenance mandate.
+ let (prev_gen, prev_digest) = config_cache
+ .as_ref()
+ .map(|c| (Some(c.last_generation), Some(c.digest)))
+ .unwrap_or((None, None));
+
+ let manual_edit = match (prev_gen, prev_digest) {
+ (Some(prev_g), Some(prev_d)) => prev_g == current_gen && prev_d != digest,
+ _ => false,
+ };
+
+ if manual_edit {
+ let prev_gen = version_cache.increase_datastore_generation();
+ effective_gen = prev_gen + 1;
+ }
// Persist
*config_cache = Some(DatastoreConfigCache {
config: config.clone(),
+ digest,
last_generation: effective_gen,
+ last_update: now,
});
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox-backup v4 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
2025-11-24 15:33 12% [pbs-devel] [PATCH proxmox-backup v4 0/4] " Samuel Rufinatscha
2025-11-24 15:33 16% ` [pbs-devel] [PATCH proxmox-backup v4 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
@ 2025-11-24 15:33 11% ` Samuel Rufinatscha
2025-11-24 15:33 14% ` [pbs-devel] [PATCH proxmox-backup v4 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
` (2 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-24 15:33 UTC (permalink / raw)
To: pbs-devel
Repeated /status requests caused lookup_datastore() to re-read and
parse datastore.cfg on every call. The issue was mentioned in report
#6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
dominated by pbs_config::datastore::config() (config parsing).
This patch implements caching of the global datastore.cfg using the
generation numbers from the shared config version cache. It caches the
datastore.cfg along with the generation number and, when a subsequent
lookup sees the same generation, it reuses the cached config without
re-reading it from disk. If the generation differs
(or the cache is unavailable), the config is re-read from disk.
If `update_cache = true`, the new config and current generation are
persisted in the cache. In this case, callers must hold the datastore
config lock to avoid racing with concurrent config changes.
If `update_cache` is `false` and generation did not match, the freshly
read config is returned but the cache is left unchanged. If
`ConfigVersionCache` is not available, the config is always read from
disk and `None` is returned as generation.
Behavioral notes
- The generation is bumped via the existing save_config() path, so
API-driven config changes are detected immediately.
- Manual edits to datastore.cfg are not detected; this is covered in a
dedicated patch in this series.
- DataStore::drop still performs a config read on the common path;
also covered in a dedicated patch in this series.
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes:
From v1 → v2, thanks @Fabian
- Moved the ConfigVersionCache changes into its own patch.
- Introduced the global static DATASTORE_CONFIG_CACHE to store the
fully parsed datastore.cfg instead, along with its generation number.
Introduced DatastoreConfigCache struct to hold both.
- Removed and replaced the CachedDatastoreConfigTag field of
DataStoreImpl with a generation number field only (Option<usize>)
to validate DataStoreImpl reuse.
- Added DataStore::datastore_section_config_cached() helper function
to encapsulate the caching logic and simplify reuse.
- Modified DataStore::lookup_datastore() to use the new helper.
From v2 → v3
No changes
From v3 → v4, thanks @Fabian
- Restructured the version cache checks in
datastore_section_config_cached(), to simplify the logic.
- Added update_cache parameter to datastore_section_config_cached() to
control cache updates.
pbs-datastore/Cargo.toml | 1 +
pbs-datastore/src/datastore.rs | 138 +++++++++++++++++++++++++--------
2 files changed, 105 insertions(+), 34 deletions(-)
diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index 8ce930a9..42f49a7b 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -40,6 +40,7 @@ proxmox-io.workspace = true
proxmox-lang.workspace=true
proxmox-s3-client = { workspace = true, features = [ "impl" ] }
proxmox-schema = { workspace = true, features = [ "api-macro" ] }
+proxmox-section-config.workspace = true
proxmox-serde = { workspace = true, features = [ "serde_json" ] }
proxmox-sys.workspace = true
proxmox-systemd.workspace = true
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 0a517923..11e16eaf 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -32,7 +32,8 @@ use pbs_api_types::{
MaintenanceType, Operation, UPID,
};
use pbs_config::s3::S3_CFG_TYPE_ID;
-use pbs_config::BackupLockGuard;
+use pbs_config::{BackupLockGuard, ConfigVersionCache};
+use proxmox_section_config::SectionConfigData;
use crate::backup_info::{
BackupDir, BackupGroup, BackupInfo, OLD_LOCKING, PROTECTED_MARKER_FILENAME,
@@ -46,6 +47,17 @@ use crate::s3::S3_CONTENT_PREFIX;
use crate::task_tracking::{self, update_active_operations};
use crate::{DataBlob, LocalDatastoreLruCache};
+// Cache for fully parsed datastore.cfg
+struct DatastoreConfigCache {
+ // Parsed datastore.cfg file
+ config: Arc<SectionConfigData>,
+ // Generation number from ConfigVersionCache
+ last_generation: usize,
+}
+
+static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
+ LazyLock::new(|| Mutex::new(None));
+
static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
@@ -142,10 +154,12 @@ pub struct DataStoreImpl {
last_gc_status: Mutex<GarbageCollectionStatus>,
verify_new: bool,
chunk_order: ChunkOrder,
- last_digest: Option<[u8; 32]>,
sync_level: DatastoreFSyncLevel,
backend_config: DatastoreBackendConfig,
lru_store_caching: Option<LocalDatastoreLruCache>,
+ /// Datastore generation number from `ConfigVersionCache` at creation time, used to
+ /// validate reuse of this cached `DataStoreImpl`.
+ config_generation: Option<usize>,
}
impl DataStoreImpl {
@@ -158,10 +172,10 @@ impl DataStoreImpl {
last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
verify_new: false,
chunk_order: Default::default(),
- last_digest: None,
sync_level: Default::default(),
backend_config: Default::default(),
lru_store_caching: None,
+ config_generation: None,
})
}
}
@@ -256,6 +270,55 @@ impl DatastoreBackend {
}
}
+/// Returns the parsed datastore config (`datastore.cfg`) and its
+/// generation.
+///
+/// Uses `ConfigVersionCache` to detect stale entries:
+/// - If the cached generation matches the current generation, the
+/// cached config is returned.
+/// - Otherwise the config is re-read from disk. If `update_cache` is
+/// `true`, the new config and current generation are stored in the
+/// cache. Callers that set `update_cache = true` must hold the
+/// datastore config lock to avoid racing with concurrent config
+/// changes.
+/// - If `update_cache` is `false`, the freshly read config is returned
+/// but the cache is left unchanged.
+///
+/// If `ConfigVersionCache` is not available, the config is always read
+/// from disk and `None` is returned as the generation.
+fn datastore_section_config_cached(
+ update_cache: bool,
+) -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
+ let mut config_cache = DATASTORE_CONFIG_CACHE.lock().unwrap();
+
+ if let Ok(version_cache) = ConfigVersionCache::new() {
+ let current_gen = version_cache.datastore_generation();
+ if let Some(cached) = config_cache.as_ref() {
+ // Fast path: re-use cached datastore.cfg
+ if cached.last_generation == current_gen {
+ return Ok((cached.config.clone(), Some(cached.last_generation)));
+ }
+ }
+ // Slow path: re-read datastore.cfg
+ let (config_raw, _digest) = pbs_config::datastore::config()?;
+ let config = Arc::new(config_raw);
+
+ if update_cache {
+ *config_cache = Some(DatastoreConfigCache {
+ config: config.clone(),
+ last_generation: current_gen,
+ });
+ }
+
+ Ok((config, Some(current_gen)))
+ } else {
+ // Fallback path, no config version cache: read datastore.cfg and return None as generation
+ *config_cache = None;
+ let (config_raw, _digest) = pbs_config::datastore::config()?;
+ Ok((Arc::new(config_raw), None))
+ }
+}
+
impl DataStore {
// This one just panics on everything
#[doc(hidden)]
@@ -327,56 +390,63 @@ impl DataStore {
name: &str,
operation: Option<Operation>,
) -> Result<Arc<DataStore>, Error> {
- // Avoid TOCTOU between checking maintenance mode and updating active operation counter, as
- // we use it to decide whether it is okay to delete the datastore.
+ // Avoid TOCTOU between checking maintenance mode and updating active operations.
let _config_lock = pbs_config::datastore::lock_config()?;
- // we could use the ConfigVersionCache's generation for staleness detection, but we load
- // the config anyway -> just use digest, additional benefit: manual changes get detected
- let (config, digest) = pbs_config::datastore::config()?;
- let config: DataStoreConfig = config.lookup("datastore", name)?;
+ // Get the current datastore.cfg generation number and cached config
+ let (section_config, gen_num) = datastore_section_config_cached(true)?;
+
+ let datastore_cfg: DataStoreConfig = section_config.lookup("datastore", name)?;
+ let maintenance_mode = datastore_cfg.get_maintenance_mode();
+ let mount_status = get_datastore_mount_status(&datastore_cfg);
- if let Some(maintenance_mode) = config.get_maintenance_mode() {
- if let Err(error) = maintenance_mode.check(operation) {
+ if let Some(mm) = &maintenance_mode {
+ if let Err(error) = mm.check(operation.clone()) {
bail!("datastore '{name}' is unavailable: {error}");
}
}
- if get_datastore_mount_status(&config) == Some(false) {
- let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
- datastore_cache.remove(&config.name);
- bail!("datastore '{}' is not mounted", config.name);
+ let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
+
+ if mount_status == Some(false) {
+ datastore_cache.remove(&datastore_cfg.name);
+ bail!("datastore '{}' is not mounted", datastore_cfg.name);
}
- let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
- let entry = datastore_cache.get(name);
-
- // reuse chunk store so that we keep using the same process locker instance!
- let chunk_store = if let Some(datastore) = &entry {
- let last_digest = datastore.last_digest.as_ref();
- if let Some(true) = last_digest.map(|last_digest| last_digest == &digest) {
- if let Some(operation) = operation {
- update_active_operations(name, operation, 1)?;
+ // Re-use DataStoreImpl
+ if let Some(existing) = datastore_cache.get(name).cloned() {
+ if let (Some(last_generation), Some(gen_num)) = (existing.config_generation, gen_num) {
+ if last_generation == gen_num {
+ if let Some(op) = operation {
+ update_active_operations(name, op, 1)?;
+ }
+
+ return Ok(Arc::new(Self {
+ inner: existing,
+ operation,
+ }));
}
- return Ok(Arc::new(Self {
- inner: Arc::clone(datastore),
- operation,
- }));
}
- Arc::clone(&datastore.chunk_store)
+ }
+
+ // (Re)build DataStoreImpl
+
+ // Reuse chunk store so that we keep using the same process locker instance!
+ let chunk_store = if let Some(existing) = datastore_cache.get(name) {
+ Arc::clone(&existing.chunk_store)
} else {
let tuning: DatastoreTuning = serde_json::from_value(
DatastoreTuning::API_SCHEMA
- .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
+ .parse_property_string(datastore_cfg.tuning.as_deref().unwrap_or(""))?,
)?;
Arc::new(ChunkStore::open(
name,
- config.absolute_path(),
+ datastore_cfg.absolute_path(),
tuning.sync_level.unwrap_or_default(),
)?)
};
- let datastore = DataStore::with_store_and_config(chunk_store, config, Some(digest))?;
+ let datastore = DataStore::with_store_and_config(chunk_store, datastore_cfg, gen_num)?;
let datastore = Arc::new(datastore);
datastore_cache.insert(name.to_string(), datastore.clone());
@@ -478,7 +548,7 @@ impl DataStore {
fn with_store_and_config(
chunk_store: Arc<ChunkStore>,
config: DataStoreConfig,
- last_digest: Option<[u8; 32]>,
+ generation: Option<usize>,
) -> Result<DataStoreImpl, Error> {
let mut gc_status_path = chunk_store.base_path();
gc_status_path.push(".gc-status");
@@ -538,10 +608,10 @@ impl DataStore {
last_gc_status: Mutex::new(gc_status),
verify_new: config.verify_new.unwrap_or(false),
chunk_order: tuning.chunk_order.unwrap_or_default(),
- last_digest,
sync_level: tuning.sync_level.unwrap_or_default(),
backend_config,
lru_store_caching,
+ config_generation: generation,
})
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 11%]
* [pbs-devel] [PATCH proxmox-backup v4 3/4] partial fix #6049: datastore: use config fast-path in Drop
2025-11-24 15:33 12% [pbs-devel] [PATCH proxmox-backup v4 0/4] " Samuel Rufinatscha
2025-11-24 15:33 16% ` [pbs-devel] [PATCH proxmox-backup v4 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
2025-11-24 15:33 11% ` [pbs-devel] [PATCH proxmox-backup v4 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
@ 2025-11-24 15:33 14% ` Samuel Rufinatscha
2025-11-24 15:33 13% ` [pbs-devel] [PATCH proxmox-backup v4 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
2025-11-24 17:06 13% ` [pbs-devel] superseded: [PATCH proxmox-backup v4 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-24 15:33 UTC (permalink / raw)
To: pbs-devel
The Drop impl of DataStore re-read datastore.cfg to decide whether
the entry should be evicted from the in-process cache (based on
maintenance mode’s clear_from_cache). During the investigation of
issue #6049 [1], a flamegraph [2] showed that the config reload in Drop
accounted for a measurable share of CPU time under load.
This patch wires the datastore config fast path to the Drop
impl to eventually avoid an expensive config reload from disk to capture
the maintenance mandate. Also, to ensure the Drop handlers will detect
that a newer config exists / to mitigate usage of an eventually stale
cached entry, generation will not only be bumped on config save, but also
on re-read of the config file (slow path), if `update_cache = true`.
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes:
From v1 → v2
- Replace caching logic with the datastore_section_config_cached()
helper.
From v2 → v3
No changes
From v3 → v4, thanks @Fabian
- Pass datastore_section_config_cached(false) in Drop to avoid
concurrent cache updates.
pbs-datastore/src/datastore.rs | 60 ++++++++++++++++++++++++++--------
1 file changed, 47 insertions(+), 13 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 11e16eaf..942656e6 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -216,15 +216,40 @@ impl Drop for DataStore {
// remove datastore from cache iff
// - last task finished, and
// - datastore is in a maintenance mode that mandates it
- let remove_from_cache = last_task
- && pbs_config::datastore::config()
- .and_then(|(s, _)| s.lookup::<DataStoreConfig>("datastore", self.name()))
- .is_ok_and(|c| {
- c.get_maintenance_mode()
- .is_some_and(|m| m.clear_from_cache())
- });
-
- if remove_from_cache {
+
+ // first check: check if last task finished
+ if !last_task {
+ return;
+ }
+
+ let (section_config, _gen) = match datastore_section_config_cached(false) {
+ Ok(v) => v,
+ Err(err) => {
+ log::error!(
+ "failed to load datastore config in Drop for {} - {err}",
+ self.name()
+ );
+ return;
+ }
+ };
+
+ let datastore_cfg: DataStoreConfig =
+ match section_config.lookup("datastore", self.name()) {
+ Ok(cfg) => cfg,
+ Err(err) => {
+ log::error!(
+ "failed to look up datastore '{}' in Drop - {err}",
+ self.name()
+ );
+ return;
+ }
+ };
+
+ // second check: check maintenance mode mandate
+ if datastore_cfg
+ .get_maintenance_mode()
+ .is_some_and(|m| m.clear_from_cache())
+ {
DATASTORE_MAP.lock().unwrap().remove(self.name());
}
}
@@ -277,12 +302,12 @@ impl DatastoreBackend {
/// - If the cached generation matches the current generation, the
/// cached config is returned.
/// - Otherwise the config is re-read from disk. If `update_cache` is
-/// `true`, the new config and current generation are stored in the
+/// `true`, the new config and bumped generation are stored in the
/// cache. Callers that set `update_cache = true` must hold the
/// datastore config lock to avoid racing with concurrent config
/// changes.
/// - If `update_cache` is `false`, the freshly read config is returned
-/// but the cache is left unchanged.
+/// but the cache and generation are left unchanged.
///
/// If `ConfigVersionCache` is not available, the config is always read
/// from disk and `None` is returned as the generation.
@@ -303,14 +328,23 @@ fn datastore_section_config_cached(
let (config_raw, _digest) = pbs_config::datastore::config()?;
let config = Arc::new(config_raw);
+ let mut effective_gen = current_gen;
if update_cache {
+ // Bump the generation. This ensures that Drop
+ // handlers will detect that a newer config exists
+ // and will not rely on a stale cached entry for
+ // maintenance mandate.
+ let prev_gen = version_cache.increase_datastore_generation();
+ effective_gen = prev_gen + 1;
+
+ // Persist
*config_cache = Some(DatastoreConfigCache {
config: config.clone(),
- last_generation: current_gen,
+ last_generation: effective_gen,
});
}
- Ok((config, Some(current_gen)))
+ Ok((config, Some(effective_gen)))
} else {
// Fallback path, no config version cache: read datastore.cfg and return None as generation
*config_cache = None;
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 14%]
* [pbs-devel] [PATCH proxmox-backup v4 1/4] partial fix #6049: config: enable config version cache for datastore
2025-11-24 15:33 12% [pbs-devel] [PATCH proxmox-backup v4 0/4] " Samuel Rufinatscha
@ 2025-11-24 15:33 16% ` Samuel Rufinatscha
2025-11-24 15:33 11% ` [pbs-devel] [PATCH proxmox-backup v4 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
` (3 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-24 15:33 UTC (permalink / raw)
To: pbs-devel
Repeated /status requests caused lookup_datastore() to re-read and
parse datastore.cfg on every call. The issue was mentioned in report
#6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
dominated by pbs_config::datastore::config() (config parsing).
To solve the issue, this patch prepares the config version cache,
so that datastore config caching can be built on top of it.
This patch specifically:
(1) implements increment function in order to invalidate generations
(2) removes obsolete comments
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes:
From v1 → v2 (original introduction), thanks @Fabian
- Split the ConfigVersionCache changes out of the large datastore patch
into their own config-only patch.
* removed the obsolete // FIXME comment on datastore_generation.
* added ConfigVersionCache::datastore_generation() as getter.
From v2 → v3
No changes
From v3 → v4
No changes
pbs-config/src/config_version_cache.rs | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/pbs-config/src/config_version_cache.rs b/pbs-config/src/config_version_cache.rs
index e8fb994f..b875f7e0 100644
--- a/pbs-config/src/config_version_cache.rs
+++ b/pbs-config/src/config_version_cache.rs
@@ -26,7 +26,6 @@ struct ConfigVersionCacheDataInner {
// Traffic control (traffic-control.cfg) generation/version.
traffic_control_generation: AtomicUsize,
// datastore (datastore.cfg) generation/version
- // FIXME: remove with PBS 3.0
datastore_generation: AtomicUsize,
// Add further atomics here
}
@@ -145,8 +144,15 @@ impl ConfigVersionCache {
.fetch_add(1, Ordering::AcqRel);
}
+ /// Returns the datastore generation number.
+ pub fn datastore_generation(&self) -> usize {
+ self.shmem
+ .data()
+ .datastore_generation
+ .load(Ordering::Acquire)
+ }
+
/// Increase the datastore generation number.
- // FIXME: remove with PBS 3.0 or make actually useful again in datastore lookup
pub fn increase_datastore_generation(&self) -> usize {
self.shmem
.data()
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 16%]
* [pbs-devel] superseded: [PATCH proxmox-backup v3 0/6] datastore: remove config reload on hot path
2025-11-20 13:03 10% [pbs-devel] [PATCH proxmox-backup v3 0/6] " Samuel Rufinatscha
` (6 preceding siblings ...)
2025-11-20 14:50 5% ` [pbs-devel] [PATCH proxmox-backup v3 0/6] datastore: remove config reload on hot path Fabian Grünbichler
@ 2025-11-24 15:35 13% ` Samuel Rufinatscha
7 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-24 15:35 UTC (permalink / raw)
To: pbs-devel
https://lore.proxmox.com/pbs-devel/20251124153328.239666-1-s.rufinatscha@proxmox.com/T/#t
On 11/20/25 2:03 PM, Samuel Rufinatscha wrote:
> Hi,
>
> this series reduces CPU time in datastore lookups by avoiding repeated
> datastore.cfg reads/parses in both `lookup_datastore()` and
> `DataStore::Drop`. It also adds a TTL so manual config edits are
> noticed without reintroducing hashing on every request.
>
> While investigating #6049 [1], cargo-flamegraph [2] showed hotspots during
> repeated `/status` calls in `lookup_datastore()` and in `Drop`,
> dominated by `pbs_config::datastore::config()` (config parse).
>
> The parsing cost itself should eventually be investigated in a future
> effort. Furthermore, cargo-flamegraph showed that when using a
> token-based auth method to access the API, a significant amount of time
> is spent in validation on every request request [3].
>
> ## Approach
>
> [PATCH 1/6] Extend ConfigVersionCache for datastore generation
> Expose a dedicated datastore generation counter and an increment
> helper so callers can cheaply track datastore.cfg versions.
>
> [PATCH 2/6] Fast path for datastore lookups
> Cache the parsed datastore.cfg keyed by the shared datastore
> generation. lookup_datastore() reuses both the cached config and an
> existing DataStoreImpl when the generation matches, and falls back
> to the old slow path otherwise.
>
> [PATCH 3/6] Fast path for Drop
> Make DataStore::Drop use the cached config if possible instead of
> rereading datastore.cfg from disk.
>
> [PATCH 4/6] TTL to catch manual edits
> Add a small TTL around the cached config and bump the datastore
> generation whenever the config is reloaded. This catches manual
> edits to datastore.cfg without reintroducing hashing or
> config parsing on every request.
>
> [PATCH 5/6] Add reload flag to config cache helper
> Add a flag to the config cache helper to indicate whether a
> config reload is acceptable.
>
> [PATCH 6/6] Only bump generation on config digest change
> Avoid unnecessary generation bumps when the config is reloaded
> but the digest did not change.
>
> ## Benchmark results
>
> All the following benchmarks are based on top of
> https://lore.proxmox.com/pbs-devel/20251112131525.645971-1-f.gruenbichler@proxmox.com/T/#u
>
> ### End-to-end
>
> Testing `/status?verbose=0` end-to-end with 1000 stores, 5 req/store
> and parallel=16 before/after the series:
>
> Metric Before After
> ----------------------------------------
> Total time 12s 9s
> Throughput (all) 416.67 555.56
> Cold RPS (round #1) 83.33 111.11
> Warm RPS (#2..N) 333.33 444.44
>
> Running under flamegraph [2], TLS appears to consume a significant
> amount of CPU time and blur the results. Still, a ~33% higher overall
> throughput and ~25% less end-to-end time for this workload.
>
> ### Isolated benchmarks (hyperfine)
>
> In addition to the end-to-end tests, I measured two standalone benchmarks
> with hyperfine, each using a config with 1000
> datastores. `M` is the number of distinct datastores looked up and
> `N` is the number of lookups per datastore.
>
> Drop-direct variant:
>
> Drops the `DataStore` after every lookup, so the `Drop` path runs on
> every iteration:
>
> use anyhow::Error;
>
> use pbs_api_types::Operation;
> use pbs_datastore::DataStore;
>
> fn main() -> Result<(), Error> {
> let mut args = std::env::args();
> args.next();
>
> let datastores = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> let iterations = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> for d in 1..=datastores {
> let name = format!("ds{:04}", d);
>
> for i in 1..=iterations {
> DataStore::lookup_datastore(&name, Some(Operation::Write))?;
> }
> }
>
> Ok(())
> }
>
> +------+-------+------------+------------+----------+
> | M | N | Baseline | Patched | Speedup |
> +------+-------+------------+------------+----------+
> | 1 | 1000 | 1.699 s | 37.3 ms | 45.5x |
> | 10 | 100 | 1.710 s | 35.8 ms | 47.7x |
> | 100 | 10 | 1.787 s | 36.6 ms | 48.9x |
> | 1000 | 1 | 1.899 s | 46.0 ms | 41.3x |
> +------+-------+------------+------------+----------+
>
>
> Bulk-drop variant:
>
> Keeps the `DataStore` instances alive for
> all `N` lookups of a given datastore and then drops them in bulk,
> mimicking a task that performs many lookups while it is running and
> only triggers the expensive `Drop` logic when the last user exits.
>
> use anyhow::Error;
>
> use pbs_api_types::Operation;
> use pbs_datastore::DataStore;
>
> fn main() -> Result<(), Error> {
> let mut args = std::env::args();
> args.next();
>
> let datastores = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> let iterations = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> for d in 1..=datastores {
> let name = format!("ds{:04}", d);
>
> let mut stores = Vec::with_capacity(iterations);
> for i in 1..=iterations {
> stores.push(DataStore::lookup_datastore(&name, Some(Operation::Write))?);
> }
> }
>
> Ok(())
> }
>
> +------+-------+--------------+-------------+----------+
> | M | N | Baseline | Patched | Speedup |
> +------+-------+--------------+-------------+----------+
> | 1 | 1000 | 888.8 ms | 39.3 ms | 22.6x |
> | 10 | 100 | 890.8 ms | 35.3 ms | 25.3x |
> | 100 | 10 | 974.5 ms | 36.3 ms | 26.8x |
> | 1000 | 1 | 1.848 s | 39.9 ms | 46.3x |
> +------+-------+--------------+-------------+----------+
>
>
> Both variants show that the combination of the cached config lookups
> and the cheaper `Drop` handling reduces the hot-path cost from ~1.7 s
> per run to a few tens of milliseconds in these benchmarks.
>
> ## Reproduction steps
>
> VM: 4 vCPU, ~8 GiB RAM, VirtIO-SCSI; disks:
> - scsi0 32G (OS)
> - scsi1 1000G (datastores)
>
> Install PBS from ISO on the VM.
>
> Set up ZFS on /dev/sdb (adjust if different):
>
> zpool create -f -o ashift=12 pbsbench /dev/sdb
> zfs set mountpoint=/pbsbench pbsbench
> zfs create pbsbench/pbs-bench
>
> Raise file-descriptor limit:
>
> sudo systemctl edit proxmox-backup-proxy.service
>
> Add the following lines:
>
> [Service]
> LimitNOFILE=1048576
>
> Reload systemd and restart the proxy:
>
> sudo systemctl daemon-reload
> sudo systemctl restart proxmox-backup-proxy.service
>
> Verify the limit:
>
> systemctl show proxmox-backup-proxy.service | grep LimitNOFILE
>
> Create 1000 ZFS-backed datastores (as used in #6049 [1]):
>
> seq -w 001 1000 | xargs -n1 -P1 bash -c '
> id=$0
> name="ds${id}"
> dataset="pbsbench/pbs-bench/${name}"
> path="/pbsbench/pbs-bench/${name}"
> zfs create -o mountpoint="$path" "$dataset"
> proxmox-backup-manager datastore create "$name" "$path" \
> --comment "ZFS dataset-based datastore"
> '
>
> Build PBS from this series, then run the server under manually
> under flamegraph:
>
> systemctl stop proxmox-backup-proxy
> cargo flamegraph --release --bin proxmox-backup-proxy
>
> ## Other resources:
>
> ### E2E benchmark script:
>
> #!/usr/bin/env bash
> set -euo pipefail
>
> # --- Config ---------------------------------------------------------------
> HOST='https://localhost:8007'
> USER='root@pam'
> PASS="$(cat passfile)"
>
> DATASTORE_PATH="/pbsbench/pbs-bench"
> MAX_STORES=1000 # how many stores to include
> PARALLEL=16 # concurrent workers
> REPEAT=5 # requests per store (1 cold + REPEAT-1 warm)
>
> PRINT_FIRST=false # true => log first request's HTTP code per store
>
> # --- Helpers --------------------------------------------------------------
> fmt_rps () {
> local n="$1" t="$2"
> awk -v n="$n" -v t="$t" 'BEGIN { if (t > 0) printf("%.2f\n", n/t); else print "0.00" }'
> }
>
> # --- Login ---------------------------------------------------------------
> auth=$(curl -ks -X POST "$HOST/api2/json/access/ticket" \
> -d "username=$USER" -d "password=$PASS")
> ticket=$(echo "$auth" | jq -r '.data.ticket')
>
> if [[ -z "${ticket:-}" || "$ticket" == "null" ]]; then
> echo "[ERROR] Login failed (no ticket)"
> exit 1
> fi
>
> # --- Collect stores (deterministic order) --------------------------------
> mapfile -t STORES < <(
> find "$DATASTORE_PATH" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' \
> | sort | head -n "$MAX_STORES"
> )
>
> USED_STORES=${#STORES[@]}
> if (( USED_STORES == 0 )); then
> echo "[ERROR] No datastore dirs under $DATASTORE_PATH"
> exit 1
> fi
>
> echo "[INFO] Running with stores=$USED_STORES, repeat=$REPEAT, parallel=$PARALLEL"
>
> # --- Temp counters --------------------------------------------------------
> SUCCESS_ALL="$(mktemp)"
> FAIL_ALL="$(mktemp)"
> COLD_OK="$(mktemp)"
> WARM_OK="$(mktemp)"
> trap 'rm -f "$SUCCESS_ALL" "$FAIL_ALL" "$COLD_OK" "$WARM_OK"' EXIT
>
> export HOST ticket REPEAT SUCCESS_ALL FAIL_ALL COLD_OK WARM_OK PRINT_FIRST
>
> SECONDS=0
>
> # --- Fire requests --------------------------------------------------------
> printf "%s\n" "${STORES[@]}" \
> | xargs -P"$PARALLEL" -I{} bash -c '
> store="$1"
> url="$HOST/api2/json/admin/datastore/$store/status?verbose=0"
>
> for ((i=1;i<=REPEAT;i++)); do
> code=$(curl -ks -o /dev/null -w "%{http_code}" -b "PBSAuthCookie=$ticket" "$url" || echo 000)
>
> if [[ "$code" == "200" ]]; then
> echo 1 >> "$SUCCESS_ALL"
> if (( i == 1 )); then
> echo 1 >> "$COLD_OK"
> else
> echo 1 >> "$WARM_OK"
> fi
> if [[ "$PRINT_FIRST" == "true" && $i -eq 1 ]]; then
> ts=$(date +%H:%M:%S)
> echo "[$ts] $store #$i HTTP:200"
> fi
> else
> echo 1 >> "$FAIL_ALL"
> if [[ "$PRINT_FIRST" == "true" && $i -eq 1 ]]; then
> ts=$(date +%H:%M:%S)
> echo "[$ts] $store #$i HTTP:$code (FAIL)"
> fi
> fi
> done
> ' _ {}
>
> # --- Summary --------------------------------------------------------------
> elapsed=$SECONDS
> ok=$(wc -l < "$SUCCESS_ALL" 2>/dev/null || echo 0)
> fail=$(wc -l < "$FAIL_ALL" 2>/dev/null || echo 0)
> cold_ok=$(wc -l < "$COLD_OK" 2>/dev/null || echo 0)
> warm_ok=$(wc -l < "$WARM_OK" 2>/dev/null || echo 0)
>
> expected=$(( USED_STORES * REPEAT ))
> total=$(( ok + fail ))
>
> rps_all=$(fmt_rps "$ok" "$elapsed")
> rps_cold=$(fmt_rps "$cold_ok" "$elapsed")
> rps_warm=$(fmt_rps "$warm_ok" "$elapsed")
>
> echo "===== Summary ====="
> echo "Stores used: $USED_STORES"
> echo "Expected requests: $expected"
> echo "Executed requests: $total"
> echo "OK (HTTP 200): $ok"
> echo "Failed: $fail"
> printf "Total time: %dm %ds\n" $((elapsed/60)) $((elapsed%60))
> echo "Throughput all RPS: $rps_all"
> echo "Cold RPS (round #1): $rps_cold"
> echo "Warm RPS (#2..N): $rps_warm"
>
> ## Patch summary
>
> [PATCH 1/6] partial fix #6049: config: enable config version cache for datastore
> [PATCH 2/6] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
> [PATCH 3/6] partial fix #6049: datastore: use config fast-path in Drop
> [PATCH 4/6] partial fix #6049: datastore: add TTL fallback to catch manual config edits
> [PATCH 5/6] to add a reload flag to the config cache helper.
> [PATCH 6/6] to only bump generation when the config digest changes.
>
> ## Changes from v2:
>
> Added:
> - [PATCH 5/6]: Add a reload flag to the config cache helper.
> - [PATCH 6/6]: Only bump generation when the config digest changes.
>
> ## Maintainer notes
>
> No dependency bumps, no API changes and no breaking changes.
>
> Thanks,
> Samuel
>
> [1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
> [3] Bugzilla #7017: https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>
> Samuel Rufinatscha (6):
> partial fix #6049: config: enable config version cache for datastore
> partial fix #6049: datastore: impl ConfigVersionCache fast path for
> lookups
> partial fix #6049: datastore: use config fast-path in Drop
> partial fix #6049: datastore: add TTL fallback to catch manual config
> edits
> partial fix #6049: datastore: add reload flag to config cache helper
> datastore: only bump generation when config digest changes
>
> pbs-config/src/config_version_cache.rs | 10 +-
> pbs-datastore/Cargo.toml | 1 +
> pbs-datastore/src/datastore.rs | 232 ++++++++++++++++++++-----
> 3 files changed, 197 insertions(+), 46 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox-backup v5 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
2025-11-24 17:04 12% [pbs-devel] [PATCH proxmox-backup v5 " Samuel Rufinatscha
` (2 preceding siblings ...)
2025-11-24 17:04 14% ` [pbs-devel] [PATCH proxmox-backup v5 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
@ 2025-11-24 17:04 14% ` Samuel Rufinatscha
2025-11-26 15:15 5% ` Fabian Grünbichler
2025-11-26 15:16 5% ` [pbs-devel] [PATCH proxmox-backup v5 0/4] datastore: remove config reload on hot path Fabian Grünbichler
2026-01-05 14:21 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
5 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-11-24 17:04 UTC (permalink / raw)
To: pbs-devel
The lookup fast path reacts to API-driven config changes because
save_config() bumps the generation. Manual edits of datastore.cfg do
not bump the counter. To keep the system robust against such edits
without reintroducing config reading and hashing on the hot path, this
patch adds a TTL to the cache entry.
If the cached config is older than
DATASTORE_CONFIG_CACHE_TTL_SECS (set to 60s), the next lookup takes
the slow path and refreshes the entry. As an optimization, a check to
catch manual edits was added (if the digest changed but generation
stayed the same), so that the generation is only bumped when needed.
Links
[1] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes:
From v1 → v2
- Store last_update timestamp in DatastoreConfigCache type.
From v2 → v3
No changes
From v3 → v4
- Fix digest generation bump logic in update_cache, thanks @Fabian.
From v4 → v5
- Rebased only, no changes
pbs-datastore/src/datastore.rs | 53 ++++++++++++++++++++++++----------
1 file changed, 38 insertions(+), 15 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 7638a899..0fc3fbf2 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -53,8 +53,12 @@ use crate::{DataBlob, LocalDatastoreLruCache};
struct DatastoreConfigCache {
// Parsed datastore.cfg file
config: Arc<SectionConfigData>,
+ // Digest of the datastore.cfg file
+ digest: [u8; 32],
// Generation number from ConfigVersionCache
last_generation: usize,
+ // Last update time (epoch seconds)
+ last_update: i64,
}
static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
@@ -63,6 +67,8 @@ static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
+/// Max age in seconds to reuse the cached datastore config.
+const DATASTORE_CONFIG_CACHE_TTL_SECS: i64 = 60;
/// Filename to store backup group notes
pub const GROUP_NOTES_FILE_NAME: &str = "notes";
/// Filename to store backup group owner
@@ -329,13 +335,14 @@ impl DatastoreThreadSettings {
/// generation.
///
/// Uses `ConfigVersionCache` to detect stale entries:
-/// - If the cached generation matches the current generation, the
-/// cached config is returned.
+/// - If the cached generation matches the current generation and TTL is
+/// OK, the cached config is returned.
/// - Otherwise the config is re-read from disk. If `update_cache` is
-/// `true`, the new config and bumped generation are stored in the
-/// cache. Callers that set `update_cache = true` must hold the
-/// datastore config lock to avoid racing with concurrent config
-/// changes.
+/// `true` and a previous cached entry exists with the same generation
+/// but a different digest, this indicates the config has changed
+/// (e.g. manual edit) and the generation must be bumped. Callers
+/// that set `update_cache = true` must hold the datastore config lock
+/// to avoid racing with concurrent config changes.
/// - If `update_cache` is `false`, the freshly read config is returned
/// but the cache and generation are left unchanged.
///
@@ -347,30 +354,46 @@ fn datastore_section_config_cached(
let mut config_cache = DATASTORE_CONFIG_CACHE.lock().unwrap();
if let Ok(version_cache) = ConfigVersionCache::new() {
+ let now = epoch_i64();
let current_gen = version_cache.datastore_generation();
if let Some(cached) = config_cache.as_ref() {
- // Fast path: re-use cached datastore.cfg
- if cached.last_generation == current_gen {
+ // Fast path: re-use cached datastore.cfg if generation matches and TTL not expired
+ if cached.last_generation == current_gen
+ && now - cached.last_update < DATASTORE_CONFIG_CACHE_TTL_SECS
+ {
return Ok((cached.config.clone(), Some(cached.last_generation)));
}
}
// Slow path: re-read datastore.cfg
- let (config_raw, _digest) = pbs_config::datastore::config()?;
+ let (config_raw, digest) = pbs_config::datastore::config()?;
let config = Arc::new(config_raw);
let mut effective_gen = current_gen;
if update_cache {
- // Bump the generation. This ensures that Drop
- // handlers will detect that a newer config exists
- // and will not rely on a stale cached entry for
- // maintenance mandate.
- let prev_gen = version_cache.increase_datastore_generation();
- effective_gen = prev_gen + 1;
+ // Bump the generation if the config has been changed manually.
+ // This ensures that Drop handlers will detect that a newer config exists
+ // and will not rely on a stale cached entry for maintenance mandate.
+ let (prev_gen, prev_digest) = config_cache
+ .as_ref()
+ .map(|c| (Some(c.last_generation), Some(c.digest)))
+ .unwrap_or((None, None));
+
+ let manual_edit = match (prev_gen, prev_digest) {
+ (Some(prev_g), Some(prev_d)) => prev_g == current_gen && prev_d != digest,
+ _ => false,
+ };
+
+ if manual_edit {
+ let prev_gen = version_cache.increase_datastore_generation();
+ effective_gen = prev_gen + 1;
+ }
// Persist
*config_cache = Some(DatastoreConfigCache {
config: config.clone(),
+ digest,
last_generation: effective_gen,
+ last_update: now,
});
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 14%]
* [pbs-devel] [PATCH proxmox-backup v5 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
2025-11-24 17:04 12% [pbs-devel] [PATCH proxmox-backup v5 " Samuel Rufinatscha
2025-11-24 17:04 16% ` [pbs-devel] [PATCH proxmox-backup v5 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
@ 2025-11-24 17:04 11% ` Samuel Rufinatscha
2025-11-26 15:15 5% ` Fabian Grünbichler
2025-11-24 17:04 14% ` [pbs-devel] [PATCH proxmox-backup v5 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
` (3 subsequent siblings)
5 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-11-24 17:04 UTC (permalink / raw)
To: pbs-devel
Repeated /status requests caused lookup_datastore() to re-read and
parse datastore.cfg on every call. The issue was mentioned in report
#6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
dominated by pbs_config::datastore::config() (config parsing).
This patch implements caching of the global datastore.cfg using the
generation numbers from the shared config version cache. It caches the
datastore.cfg along with the generation number and, when a subsequent
lookup sees the same generation, it reuses the cached config without
re-reading it from disk. If the generation differs
(or the cache is unavailable), the config is re-read from disk.
If `update_cache = true`, the new config and current generation are
persisted in the cache. In this case, callers must hold the datastore
config lock to avoid racing with concurrent config changes.
If `update_cache` is `false` and generation did not match, the freshly
read config is returned but the cache is left unchanged. If
`ConfigVersionCache` is not available, the config is always read from
disk and `None` is returned as generation.
Behavioral notes
- The generation is bumped via the existing save_config() path, so
API-driven config changes are detected immediately.
- Manual edits to datastore.cfg are not detected; this is covered in a
dedicated patch in this series.
- DataStore::drop still performs a config read on the common path;
also covered in a dedicated patch in this series.
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes:
From v1 → v2, thanks @Fabian
- Moved the ConfigVersionCache changes into its own patch.
- Introduced the global static DATASTORE_CONFIG_CACHE to store the
fully parsed datastore.cfg instead, along with its generation number.
Introduced DatastoreConfigCache struct to hold both.
- Removed and replaced the CachedDatastoreConfigTag field of
DataStoreImpl with a generation number field only (Option<usize>)
to validate DataStoreImpl reuse.
- Added DataStore::datastore_section_config_cached() helper function
to encapsulate the caching logic and simplify reuse.
- Modified DataStore::lookup_datastore() to use the new helper.
From v2 → v3
No changes
From v3 → v4, thanks @Fabian
- Restructured the version cache checks in
datastore_section_config_cached(), to simplify the logic.
- Added update_cache parameter to datastore_section_config_cached() to
control cache updates.
From v4 → v5
- Rebased only, no changes
pbs-datastore/Cargo.toml | 1 +
pbs-datastore/src/datastore.rs | 138 +++++++++++++++++++++++++--------
2 files changed, 105 insertions(+), 34 deletions(-)
diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index 8ce930a9..42f49a7b 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -40,6 +40,7 @@ proxmox-io.workspace = true
proxmox-lang.workspace=true
proxmox-s3-client = { workspace = true, features = [ "impl" ] }
proxmox-schema = { workspace = true, features = [ "api-macro" ] }
+proxmox-section-config.workspace = true
proxmox-serde = { workspace = true, features = [ "serde_json" ] }
proxmox-sys.workspace = true
proxmox-systemd.workspace = true
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 36550ff6..c9cb5d65 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -34,7 +34,8 @@ use pbs_api_types::{
MaintenanceType, Operation, UPID,
};
use pbs_config::s3::S3_CFG_TYPE_ID;
-use pbs_config::BackupLockGuard;
+use pbs_config::{BackupLockGuard, ConfigVersionCache};
+use proxmox_section_config::SectionConfigData;
use crate::backup_info::{
BackupDir, BackupGroup, BackupInfo, OLD_LOCKING, PROTECTED_MARKER_FILENAME,
@@ -48,6 +49,17 @@ use crate::s3::S3_CONTENT_PREFIX;
use crate::task_tracking::{self, update_active_operations};
use crate::{DataBlob, LocalDatastoreLruCache};
+// Cache for fully parsed datastore.cfg
+struct DatastoreConfigCache {
+ // Parsed datastore.cfg file
+ config: Arc<SectionConfigData>,
+ // Generation number from ConfigVersionCache
+ last_generation: usize,
+}
+
+static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
+ LazyLock::new(|| Mutex::new(None));
+
static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
@@ -149,11 +161,13 @@ pub struct DataStoreImpl {
last_gc_status: Mutex<GarbageCollectionStatus>,
verify_new: bool,
chunk_order: ChunkOrder,
- last_digest: Option<[u8; 32]>,
sync_level: DatastoreFSyncLevel,
backend_config: DatastoreBackendConfig,
lru_store_caching: Option<LocalDatastoreLruCache>,
thread_settings: DatastoreThreadSettings,
+ /// Datastore generation number from `ConfigVersionCache` at creation time, used to
+ /// validate reuse of this cached `DataStoreImpl`.
+ config_generation: Option<usize>,
}
impl DataStoreImpl {
@@ -166,11 +180,11 @@ impl DataStoreImpl {
last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
verify_new: false,
chunk_order: Default::default(),
- last_digest: None,
sync_level: Default::default(),
backend_config: Default::default(),
lru_store_caching: None,
thread_settings: Default::default(),
+ config_generation: None,
})
}
}
@@ -286,6 +300,55 @@ impl DatastoreThreadSettings {
}
}
+/// Returns the parsed datastore config (`datastore.cfg`) and its
+/// generation.
+///
+/// Uses `ConfigVersionCache` to detect stale entries:
+/// - If the cached generation matches the current generation, the
+/// cached config is returned.
+/// - Otherwise the config is re-read from disk. If `update_cache` is
+/// `true`, the new config and current generation are stored in the
+/// cache. Callers that set `update_cache = true` must hold the
+/// datastore config lock to avoid racing with concurrent config
+/// changes.
+/// - If `update_cache` is `false`, the freshly read config is returned
+/// but the cache is left unchanged.
+///
+/// If `ConfigVersionCache` is not available, the config is always read
+/// from disk and `None` is returned as the generation.
+fn datastore_section_config_cached(
+ update_cache: bool,
+) -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
+ let mut config_cache = DATASTORE_CONFIG_CACHE.lock().unwrap();
+
+ if let Ok(version_cache) = ConfigVersionCache::new() {
+ let current_gen = version_cache.datastore_generation();
+ if let Some(cached) = config_cache.as_ref() {
+ // Fast path: re-use cached datastore.cfg
+ if cached.last_generation == current_gen {
+ return Ok((cached.config.clone(), Some(cached.last_generation)));
+ }
+ }
+ // Slow path: re-read datastore.cfg
+ let (config_raw, _digest) = pbs_config::datastore::config()?;
+ let config = Arc::new(config_raw);
+
+ if update_cache {
+ *config_cache = Some(DatastoreConfigCache {
+ config: config.clone(),
+ last_generation: current_gen,
+ });
+ }
+
+ Ok((config, Some(current_gen)))
+ } else {
+ // Fallback path, no config version cache: read datastore.cfg and return None as generation
+ *config_cache = None;
+ let (config_raw, _digest) = pbs_config::datastore::config()?;
+ Ok((Arc::new(config_raw), None))
+ }
+}
+
impl DataStore {
// This one just panics on everything
#[doc(hidden)]
@@ -363,56 +426,63 @@ impl DataStore {
name: &str,
operation: Option<Operation>,
) -> Result<Arc<DataStore>, Error> {
- // Avoid TOCTOU between checking maintenance mode and updating active operation counter, as
- // we use it to decide whether it is okay to delete the datastore.
+ // Avoid TOCTOU between checking maintenance mode and updating active operations.
let _config_lock = pbs_config::datastore::lock_config()?;
- // we could use the ConfigVersionCache's generation for staleness detection, but we load
- // the config anyway -> just use digest, additional benefit: manual changes get detected
- let (config, digest) = pbs_config::datastore::config()?;
- let config: DataStoreConfig = config.lookup("datastore", name)?;
+ // Get the current datastore.cfg generation number and cached config
+ let (section_config, gen_num) = datastore_section_config_cached(true)?;
+
+ let datastore_cfg: DataStoreConfig = section_config.lookup("datastore", name)?;
+ let maintenance_mode = datastore_cfg.get_maintenance_mode();
+ let mount_status = get_datastore_mount_status(&datastore_cfg);
- if let Some(maintenance_mode) = config.get_maintenance_mode() {
- if let Err(error) = maintenance_mode.check(operation) {
+ if let Some(mm) = &maintenance_mode {
+ if let Err(error) = mm.check(operation.clone()) {
bail!("datastore '{name}' is unavailable: {error}");
}
}
- if get_datastore_mount_status(&config) == Some(false) {
- let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
- datastore_cache.remove(&config.name);
- bail!("datastore '{}' is not mounted", config.name);
+ let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
+
+ if mount_status == Some(false) {
+ datastore_cache.remove(&datastore_cfg.name);
+ bail!("datastore '{}' is not mounted", datastore_cfg.name);
}
- let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
- let entry = datastore_cache.get(name);
-
- // reuse chunk store so that we keep using the same process locker instance!
- let chunk_store = if let Some(datastore) = &entry {
- let last_digest = datastore.last_digest.as_ref();
- if let Some(true) = last_digest.map(|last_digest| last_digest == &digest) {
- if let Some(operation) = operation {
- update_active_operations(name, operation, 1)?;
+ // Re-use DataStoreImpl
+ if let Some(existing) = datastore_cache.get(name).cloned() {
+ if let (Some(last_generation), Some(gen_num)) = (existing.config_generation, gen_num) {
+ if last_generation == gen_num {
+ if let Some(op) = operation {
+ update_active_operations(name, op, 1)?;
+ }
+
+ return Ok(Arc::new(Self {
+ inner: existing,
+ operation,
+ }));
}
- return Ok(Arc::new(Self {
- inner: Arc::clone(datastore),
- operation,
- }));
}
- Arc::clone(&datastore.chunk_store)
+ }
+
+ // (Re)build DataStoreImpl
+
+ // Reuse chunk store so that we keep using the same process locker instance!
+ let chunk_store = if let Some(existing) = datastore_cache.get(name) {
+ Arc::clone(&existing.chunk_store)
} else {
let tuning: DatastoreTuning = serde_json::from_value(
DatastoreTuning::API_SCHEMA
- .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
+ .parse_property_string(datastore_cfg.tuning.as_deref().unwrap_or(""))?,
)?;
Arc::new(ChunkStore::open(
name,
- config.absolute_path(),
+ datastore_cfg.absolute_path(),
tuning.sync_level.unwrap_or_default(),
)?)
};
- let datastore = DataStore::with_store_and_config(chunk_store, config, Some(digest))?;
+ let datastore = DataStore::with_store_and_config(chunk_store, datastore_cfg, gen_num)?;
let datastore = Arc::new(datastore);
datastore_cache.insert(name.to_string(), datastore.clone());
@@ -514,7 +584,7 @@ impl DataStore {
fn with_store_and_config(
chunk_store: Arc<ChunkStore>,
config: DataStoreConfig,
- last_digest: Option<[u8; 32]>,
+ generation: Option<usize>,
) -> Result<DataStoreImpl, Error> {
let mut gc_status_path = chunk_store.base_path();
gc_status_path.push(".gc-status");
@@ -579,11 +649,11 @@ impl DataStore {
last_gc_status: Mutex::new(gc_status),
verify_new: config.verify_new.unwrap_or(false),
chunk_order: tuning.chunk_order.unwrap_or_default(),
- last_digest,
sync_level: tuning.sync_level.unwrap_or_default(),
backend_config,
lru_store_caching,
thread_settings,
+ config_generation: generation,
})
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 11%]
* [pbs-devel] [PATCH proxmox-backup v5 1/4] partial fix #6049: config: enable config version cache for datastore
2025-11-24 17:04 12% [pbs-devel] [PATCH proxmox-backup v5 " Samuel Rufinatscha
@ 2025-11-24 17:04 16% ` Samuel Rufinatscha
2025-11-26 15:15 5% ` Fabian Grünbichler
2025-11-24 17:04 11% ` [pbs-devel] [PATCH proxmox-backup v5 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
` (4 subsequent siblings)
5 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-11-24 17:04 UTC (permalink / raw)
To: pbs-devel
Repeated /status requests caused lookup_datastore() to re-read and
parse datastore.cfg on every call. The issue was mentioned in report
#6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
dominated by pbs_config::datastore::config() (config parsing).
To solve the issue, this patch prepares the config version cache,
so that datastore config caching can be built on top of it.
This patch specifically:
(1) implements increment function in order to invalidate generations
(2) removes obsolete comments
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes:
From v1 → v2 (original introduction), thanks @Fabian
- Split the ConfigVersionCache changes out of the large datastore patch
into their own config-only patch
* removed the obsolete // FIXME comment on datastore_generation
* added ConfigVersionCache::datastore_generation() as getter
From v2 → v3
No changes
From v3 → v4
No changes
From v4 → v5
- Rebased only, no changes
pbs-config/src/config_version_cache.rs | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/pbs-config/src/config_version_cache.rs b/pbs-config/src/config_version_cache.rs
index e8fb994f..b875f7e0 100644
--- a/pbs-config/src/config_version_cache.rs
+++ b/pbs-config/src/config_version_cache.rs
@@ -26,7 +26,6 @@ struct ConfigVersionCacheDataInner {
// Traffic control (traffic-control.cfg) generation/version.
traffic_control_generation: AtomicUsize,
// datastore (datastore.cfg) generation/version
- // FIXME: remove with PBS 3.0
datastore_generation: AtomicUsize,
// Add further atomics here
}
@@ -145,8 +144,15 @@ impl ConfigVersionCache {
.fetch_add(1, Ordering::AcqRel);
}
+ /// Returns the datastore generation number.
+ pub fn datastore_generation(&self) -> usize {
+ self.shmem
+ .data()
+ .datastore_generation
+ .load(Ordering::Acquire)
+ }
+
/// Increase the datastore generation number.
- // FIXME: remove with PBS 3.0 or make actually useful again in datastore lookup
pub fn increase_datastore_generation(&self) -> usize {
self.shmem
.data()
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 16%]
* [pbs-devel] [PATCH proxmox-backup v5 0/4] datastore: remove config reload on hot path
@ 2025-11-24 17:04 12% Samuel Rufinatscha
2025-11-24 17:04 16% ` [pbs-devel] [PATCH proxmox-backup v5 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
` (5 more replies)
0 siblings, 6 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-24 17:04 UTC (permalink / raw)
To: pbs-devel
Hi,
this series reduces CPU time in datastore lookups by avoiding repeated
datastore.cfg reads/parses in both `lookup_datastore()` and
`DataStore::Drop`. It also adds a TTL so manual config edits are
noticed without reintroducing hashing on every request.
While investigating #6049 [1], cargo-flamegraph [2] showed hotspots
during repeated `/status` calls in `lookup_datastore()` and in `Drop`,
dominated by `pbs_config::datastore::config()` (config parse).
The parsing cost itself should eventually be investigated in a future
effort. Furthermore, cargo-flamegraph showed that when using a
token-based auth method to access the API, a significant amount of time
is spent in validation on every request request [3].
## Approach
[PATCH 1/4] Support datastore generation in ConfigVersionCache
[PATCH 2/4] Fast path for datastore lookups
Cache the parsed datastore.cfg keyed by the shared datastore
generation. lookup_datastore() reuses both the cached config and an
existing DataStoreImpl when the generation matches, and falls back
to the old slow path otherwise. The caching logic is implemented
using the datastore_section_config_cached(update_cache: bool) helper.
[PATCH 3/4] Fast path for Drop
Make DataStore::Drop use the datastore_section_config_cached()
helper to avoid re-reading/parsing datastore.cfg on every Drop.
Bump generation not only on API config saves, but also on slow-path
lookups (if update_cache is true), to enable Drop handlers see
eventual newer configs.
[PATCH 4/4] TTL to catch manual edits
Add a TTL to the cached config and bump the datastore generation iff
the digest changed but generation stays the same. This catches manual
edits to datastore.cfg without reintroducing hashing or config
parsing on every request.
## Benchmark results
### End-to-end
Testing `/status?verbose=0` end-to-end with 1000 stores, 5 req/store
and parallel=16 before/after the series:
Metric Before After
----------------------------------------
Total time 12s 9s
Throughput (all) 416.67 555.56
Cold RPS (round #1) 83.33 111.11
Warm RPS (#2..N) 333.33 444.44
Running under flamegraph [2], TLS appears to consume a significant
amount of CPU time and blur the results. Still, a ~33% higher overall
throughput and ~25% less end-to-end time for this workload.
### Isolated benchmarks (hyperfine)
In addition to the end-to-end tests, I measured two standalone
benchmarks with hyperfine, each using a config with 1000 datastores.
`M` is the number of distinct datastores looked up and
`N` is the number of lookups per datastore.
Drop-direct variant:
Drops the `DataStore` after every lookup, so the `Drop` path runs on
every iteration:
use anyhow::Error;
use pbs_api_types::Operation;
use pbs_datastore::DataStore;
fn main() -> Result<(), Error> {
let mut args = std::env::args();
args.next();
let datastores = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
let iterations = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
for d in 1..=datastores {
let name = format!("ds{:04}", d);
for i in 1..=iterations {
DataStore::lookup_datastore(&name, Some(Operation::Write))?;
}
}
Ok(())
}
+----+------+-----------+-----------+---------+
| M | N | Baseline | Patched | Speedup |
+----+------+-----------+-----------+---------+
| 1 | 1000 | 1.684 s | 35.3 ms | 47.7x |
| 10 | 100 | 1.689 s | 35.0 ms | 48.3x |
| 100| 10 | 1.709 s | 35.8 ms | 47.7x |
|1000| 1 | 1.809 s | 39.0 ms | 46.4x |
+----+------+-----------+-----------+---------+
Bulk-drop variant:
Keeps the `DataStore` instances alive for
all `N` lookups of a given datastore and then drops them in bulk,
mimicking a task that performs many lookups while it is running and
only triggers the expensive `Drop` logic when the last user exits.
use anyhow::Error;
use pbs_api_types::Operation;
use pbs_datastore::DataStore;
fn main() -> Result<(), Error> {
let mut args = std::env::args();
args.next();
let datastores = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
let iterations = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
for d in 1..=datastores {
let name = format!("ds{:04}", d);
let mut stores = Vec::with_capacity(iterations);
for i in 1..=iterations {
stores.push(DataStore::lookup_datastore(&name, Some(Operation::Write))?);
}
}
Ok(())
}
+------+------+---------------+--------------+---------+
| M | N | Baseline mean | Patched mean | Speedup |
+------+------+---------------+--------------+---------+
| 1 | 1000 | 890.6 ms | 35.5 ms | 25.1x |
| 10 | 100 | 891.3 ms | 35.1 ms | 25.4x |
| 100 | 10 | 983.9 ms | 35.6 ms | 27.6x |
| 1000 | 1 | 1829.0 ms | 45.2 ms | 40.5x |
+------+------+---------------+--------------+---------+
Both variants show that the combination of the cached config lookups
and the cheaper `Drop` handling reduces the hot-path cost from ~1.8 s
per run to a few tens of milliseconds in these benchmarks.
## Reproduction steps
VM: 4 vCPU, ~8 GiB RAM, VirtIO-SCSI; disks:
- scsi0 32G (OS)
- scsi1 1000G (datastores)
Install PBS from ISO on the VM.
Set up ZFS on /dev/sdb (adjust if different):
zpool create -f -o ashift=12 pbsbench /dev/sdb
zfs set mountpoint=/pbsbench pbsbench
zfs create pbsbench/pbs-bench
Raise file-descriptor limit:
sudo systemctl edit proxmox-backup-proxy.service
Add the following lines:
[Service]
LimitNOFILE=1048576
Reload systemd and restart the proxy:
sudo systemctl daemon-reload
sudo systemctl restart proxmox-backup-proxy.service
Verify the limit:
systemctl show proxmox-backup-proxy.service | grep LimitNOFILE
Create 1000 ZFS-backed datastores (as used in #6049 [1]):
seq -w 001 1000 | xargs -n1 -P1 bash -c '
id=$0
name="ds${id}"
dataset="pbsbench/pbs-bench/${name}"
path="/pbsbench/pbs-bench/${name}"
zfs create -o mountpoint="$path" "$dataset"
proxmox-backup-manager datastore create "$name" "$path" \
--comment "ZFS dataset-based datastore"
'
Build PBS from this series, then run the server under manually
under flamegraph:
systemctl stop proxmox-backup-proxy
cargo flamegraph --release --bin proxmox-backup-proxy
## Patch summary
[PATCH 1/4] partial fix #6049: config: enable config version cache for datastore
[PATCH 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
[PATCH 3/4] partial fix #6049: datastore: use config fast-path in Drop
[PATCH 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
## Maintainer notes
No dependency bumps, no API changes and no breaking changes.
Thanks,
Samuel
Links
[1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
[3] Bugzilla #7017: https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Samuel Rufinatscha (4):
partial fix #6049: config: enable config version cache for datastore
partial fix #6049: datastore: impl ConfigVersionCache fast path for
lookups
partial fix #6049: datastore: use config fast-path in Drop
partial fix #6049: datastore: add TTL fallback to catch manual config
edits
pbs-config/src/config_version_cache.rs | 10 +-
pbs-datastore/Cargo.toml | 1 +
pbs-datastore/src/datastore.rs | 213 ++++++++++++++++++++-----
3 files changed, 179 insertions(+), 45 deletions(-)
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] [PATCH proxmox-backup v5 3/4] partial fix #6049: datastore: use config fast-path in Drop
2025-11-24 17:04 12% [pbs-devel] [PATCH proxmox-backup v5 " Samuel Rufinatscha
2025-11-24 17:04 16% ` [pbs-devel] [PATCH proxmox-backup v5 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
2025-11-24 17:04 11% ` [pbs-devel] [PATCH proxmox-backup v5 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
@ 2025-11-24 17:04 14% ` Samuel Rufinatscha
2025-11-26 15:15 5% ` Fabian Grünbichler
2025-11-24 17:04 14% ` [pbs-devel] [PATCH proxmox-backup v5 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
` (2 subsequent siblings)
5 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-11-24 17:04 UTC (permalink / raw)
To: pbs-devel
The Drop impl of DataStore re-read datastore.cfg to decide whether
the entry should be evicted from the in-process cache (based on
maintenance mode’s clear_from_cache). During the investigation of
issue #6049 [1], a flamegraph [2] showed that the config reload in Drop
accounted for a measurable share of CPU time under load.
This patch wires the datastore config fast path to the Drop
impl to eventually avoid an expensive config reload from disk to capture
the maintenance mandate. Also, to ensure the Drop handlers will detect
that a newer config exists / to mitigate usage of an eventually stale
cached entry, generation will not only be bumped on config save, but also
on re-read of the config file (slow path), if `update_cache = true`.
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes:
From v1 → v2
- Replace caching logic with the datastore_section_config_cached()
helper.
From v2 → v3
No changes
From v3 → v4, thanks @Fabian
- Pass datastore_section_config_cached(false) in Drop to avoid
concurrent cache updates.
From v4 → v5
- Rebased only, no changes
pbs-datastore/src/datastore.rs | 60 ++++++++++++++++++++++++++--------
1 file changed, 47 insertions(+), 13 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index c9cb5d65..7638a899 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -225,15 +225,40 @@ impl Drop for DataStore {
// remove datastore from cache iff
// - last task finished, and
// - datastore is in a maintenance mode that mandates it
- let remove_from_cache = last_task
- && pbs_config::datastore::config()
- .and_then(|(s, _)| s.lookup::<DataStoreConfig>("datastore", self.name()))
- .is_ok_and(|c| {
- c.get_maintenance_mode()
- .is_some_and(|m| m.clear_from_cache())
- });
-
- if remove_from_cache {
+
+ // first check: check if last task finished
+ if !last_task {
+ return;
+ }
+
+ let (section_config, _gen) = match datastore_section_config_cached(false) {
+ Ok(v) => v,
+ Err(err) => {
+ log::error!(
+ "failed to load datastore config in Drop for {} - {err}",
+ self.name()
+ );
+ return;
+ }
+ };
+
+ let datastore_cfg: DataStoreConfig =
+ match section_config.lookup("datastore", self.name()) {
+ Ok(cfg) => cfg,
+ Err(err) => {
+ log::error!(
+ "failed to look up datastore '{}' in Drop - {err}",
+ self.name()
+ );
+ return;
+ }
+ };
+
+ // second check: check maintenance mode mandate
+ if datastore_cfg
+ .get_maintenance_mode()
+ .is_some_and(|m| m.clear_from_cache())
+ {
DATASTORE_MAP.lock().unwrap().remove(self.name());
}
}
@@ -307,12 +332,12 @@ impl DatastoreThreadSettings {
/// - If the cached generation matches the current generation, the
/// cached config is returned.
/// - Otherwise the config is re-read from disk. If `update_cache` is
-/// `true`, the new config and current generation are stored in the
+/// `true`, the new config and bumped generation are stored in the
/// cache. Callers that set `update_cache = true` must hold the
/// datastore config lock to avoid racing with concurrent config
/// changes.
/// - If `update_cache` is `false`, the freshly read config is returned
-/// but the cache is left unchanged.
+/// but the cache and generation are left unchanged.
///
/// If `ConfigVersionCache` is not available, the config is always read
/// from disk and `None` is returned as the generation.
@@ -333,14 +358,23 @@ fn datastore_section_config_cached(
let (config_raw, _digest) = pbs_config::datastore::config()?;
let config = Arc::new(config_raw);
+ let mut effective_gen = current_gen;
if update_cache {
+ // Bump the generation. This ensures that Drop
+ // handlers will detect that a newer config exists
+ // and will not rely on a stale cached entry for
+ // maintenance mandate.
+ let prev_gen = version_cache.increase_datastore_generation();
+ effective_gen = prev_gen + 1;
+
+ // Persist
*config_cache = Some(DatastoreConfigCache {
config: config.clone(),
- last_generation: current_gen,
+ last_generation: effective_gen,
});
}
- Ok((config, Some(current_gen)))
+ Ok((config, Some(effective_gen)))
} else {
// Fallback path, no config version cache: read datastore.cfg and return None as generation
*config_cache = None;
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 14%]
* [pbs-devel] superseded: [PATCH proxmox-backup v4 0/4] datastore: remove config reload on hot path
2025-11-24 15:33 12% [pbs-devel] [PATCH proxmox-backup v4 0/4] " Samuel Rufinatscha
` (3 preceding siblings ...)
2025-11-24 15:33 13% ` [pbs-devel] [PATCH proxmox-backup v4 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
@ 2025-11-24 17:06 13% ` Samuel Rufinatscha
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-24 17:06 UTC (permalink / raw)
To: pbs-devel
https://lore.proxmox.com/pbs-devel/20251124170423.303300-1-s.rufinatscha@proxmox.com/T/#t
On 11/24/25 4:32 PM, Samuel Rufinatscha wrote:
> Hi,
>
> this series reduces CPU time in datastore lookups by avoiding repeated
> datastore.cfg reads/parses in both `lookup_datastore()` and
> `DataStore::Drop`. It also adds a TTL so manual config edits are
> noticed without reintroducing hashing on every request.
>
> While investigating #6049 [1], cargo-flamegraph [2] showed hotspots
> during repeated `/status` calls in `lookup_datastore()` and in `Drop`,
> dominated by `pbs_config::datastore::config()` (config parse).
>
> The parsing cost itself should eventually be investigated in a future
> effort. Furthermore, cargo-flamegraph showed that when using a
> token-based auth method to access the API, a significant amount of time
> is spent in validation on every request request [3].
>
> ## Approach
>
> [PATCH 1/4] Support datastore generation in ConfigVersionCache
>
> [PATCH 2/4] Fast path for datastore lookups
> Cache the parsed datastore.cfg keyed by the shared datastore
> generation. lookup_datastore() reuses both the cached config and an
> existing DataStoreImpl when the generation matches, and falls back
> to the old slow path otherwise. The caching logic is implemented
> using the datastore_section_config_cached(update_cache: bool) helper.
>
> [PATCH 3/4] Fast path for Drop
> Make DataStore::Drop use the datastore_section_config_cached()
> helper to avoid re-reading/parsing datastore.cfg on every Drop.
> Bump generation not only on API config saves, but also on slow-path
> lookups (if update_cache is true), to enable Drop handlers see
> eventual newer configs.
>
> [PATCH 4/4] TTL to catch manual edits
> Add a TTL to the cached config and bump the datastore generation iff
> the digest changed but generation stays the same. This catches manual
> edits to datastore.cfg without reintroducing hashing or config
> parsing on every request.
>
> ## Benchmark results
>
> ### End-to-end
>
> Testing `/status?verbose=0` end-to-end with 1000 stores, 5 req/store
> and parallel=16 before/after the series:
>
> Metric Before After
> ----------------------------------------
> Total time 12s 9s
> Throughput (all) 416.67 555.56
> Cold RPS (round #1) 83.33 111.11
> Warm RPS (#2..N) 333.33 444.44
>
> Running under flamegraph [2], TLS appears to consume a significant
> amount of CPU time and blur the results. Still, a ~33% higher overall
> throughput and ~25% less end-to-end time for this workload.
>
> ### Isolated benchmarks (hyperfine)
>
> In addition to the end-to-end tests, I measured two standalone
> benchmarks with hyperfine, each using a config with 1000 datastores.
> `M` is the number of distinct datastores looked up and
> `N` is the number of lookups per datastore.
>
> Drop-direct variant:
>
> Drops the `DataStore` after every lookup, so the `Drop` path runs on
> every iteration:
>
> use anyhow::Error;
>
> use pbs_api_types::Operation;
> use pbs_datastore::DataStore;
>
> fn main() -> Result<(), Error> {
> let mut args = std::env::args();
> args.next();
>
> let datastores = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> let iterations = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> for d in 1..=datastores {
> let name = format!("ds{:04}", d);
>
> for i in 1..=iterations {
> DataStore::lookup_datastore(&name, Some(Operation::Write))?;
> }
> }
>
> Ok(())
> }
>
> +----+------+-----------+-----------+---------+
> | M | N | Baseline | Patched | Speedup |
> +----+------+-----------+-----------+---------+
> | 1 | 1000 | 1.684 s | 35.3 ms | 47.7x |
> | 10 | 100 | 1.689 s | 35.0 ms | 48.3x |
> | 100| 10 | 1.709 s | 35.8 ms | 47.7x |
> |1000| 1 | 1.809 s | 39.0 ms | 46.4x |
> +----+------+-----------+-----------+---------+
>
> Bulk-drop variant:
>
> Keeps the `DataStore` instances alive for
> all `N` lookups of a given datastore and then drops them in bulk,
> mimicking a task that performs many lookups while it is running and
> only triggers the expensive `Drop` logic when the last user exits.
>
> use anyhow::Error;
>
> use pbs_api_types::Operation;
> use pbs_datastore::DataStore;
>
> fn main() -> Result<(), Error> {
> let mut args = std::env::args();
> args.next();
>
> let datastores = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> let iterations = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> for d in 1..=datastores {
> let name = format!("ds{:04}", d);
>
> let mut stores = Vec::with_capacity(iterations);
> for i in 1..=iterations {
> stores.push(DataStore::lookup_datastore(&name, Some(Operation::Write))?);
> }
> }
>
> Ok(())
> }
>
> +------+------+---------------+--------------+---------+
> | M | N | Baseline mean | Patched mean | Speedup |
> +------+------+---------------+--------------+---------+
> | 1 | 1000 | 890.6 ms | 35.5 ms | 25.1x |
> | 10 | 100 | 891.3 ms | 35.1 ms | 25.4x |
> | 100 | 10 | 983.9 ms | 35.6 ms | 27.6x |
> | 1000 | 1 | 1829.0 ms | 45.2 ms | 40.5x |
> +------+------+---------------+--------------+---------+
>
>
> Both variants show that the combination of the cached config lookups
> and the cheaper `Drop` handling reduces the hot-path cost from ~1.8 s
> per run to a few tens of milliseconds in these benchmarks.
>
> ## Reproduction steps
>
> VM: 4 vCPU, ~8 GiB RAM, VirtIO-SCSI; disks:
> - scsi0 32G (OS)
> - scsi1 1000G (datastores)
>
> Install PBS from ISO on the VM.
>
> Set up ZFS on /dev/sdb (adjust if different):
>
> zpool create -f -o ashift=12 pbsbench /dev/sdb
> zfs set mountpoint=/pbsbench pbsbench
> zfs create pbsbench/pbs-bench
>
> Raise file-descriptor limit:
>
> sudo systemctl edit proxmox-backup-proxy.service
>
> Add the following lines:
>
> [Service]
> LimitNOFILE=1048576
>
> Reload systemd and restart the proxy:
>
> sudo systemctl daemon-reload
> sudo systemctl restart proxmox-backup-proxy.service
>
> Verify the limit:
>
> systemctl show proxmox-backup-proxy.service | grep LimitNOFILE
>
> Create 1000 ZFS-backed datastores (as used in #6049 [1]):
>
> seq -w 001 1000 | xargs -n1 -P1 bash -c '
> id=$0
> name="ds${id}"
> dataset="pbsbench/pbs-bench/${name}"
> path="/pbsbench/pbs-bench/${name}"
> zfs create -o mountpoint="$path" "$dataset"
> proxmox-backup-manager datastore create "$name" "$path" \
> --comment "ZFS dataset-based datastore"
> '
>
> Build PBS from this series, then run the server under manually
> under flamegraph:
>
> systemctl stop proxmox-backup-proxy
> cargo flamegraph --release --bin proxmox-backup-proxy
>
> ## Patch summary
>
> [PATCH 1/4] partial fix #6049: config: enable config version cache for datastore
> [PATCH 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
> [PATCH 3/4] partial fix #6049: datastore: use config fast-path in Drop
> [PATCH 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
>
> ## Maintainer notes
>
> No dependency bumps, no API changes and no breaking changes.
>
> Thanks,
> Samuel
>
> Links
>
> [1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
> [3] Bugzilla #7017: https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>
> Samuel Rufinatscha (4):
> partial fix #6049: config: enable config version cache for datastore
> partial fix #6049: datastore: impl ConfigVersionCache fast path for
> lookups
> partial fix #6049: datastore: use config fast-path in Drop
> partial fix #6049: datastore: add TTL fallback to catch manual config
> edits
>
> pbs-config/src/config_version_cache.rs | 10 +-
> pbs-datastore/Cargo.toml | 1 +
> pbs-datastore/src/datastore.rs | 215 ++++++++++++++++++++-----
> 3 files changed, 180 insertions(+), 46 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 1/4] partial fix #6049: config: enable config version cache for datastore
2025-11-24 17:04 16% ` [pbs-devel] [PATCH proxmox-backup v5 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
@ 2025-11-26 15:15 5% ` Fabian Grünbichler
0 siblings, 0 replies; 200+ results
From: Fabian Grünbichler @ 2025-11-26 15:15 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
nit for the subject: this doesn't yet partially fix anything..
On November 24, 2025 6:04 pm, Samuel Rufinatscha wrote:
> Repeated /status requests caused lookup_datastore() to re-read and
> parse datastore.cfg on every call. The issue was mentioned in report
> #6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
> dominated by pbs_config::datastore::config() (config parsing).
>
> To solve the issue, this patch prepares the config version cache,
> so that datastore config caching can be built on top of it.
> This patch specifically:
> (1) implements increment function in order to invalidate generations
> (2) removes obsolete comments
>
> Links
>
> [1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>
> Fixes: #6049
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
> ---
> Changes:
>
> From v1 → v2 (original introduction), thanks @Fabian
> - Split the ConfigVersionCache changes out of the large datastore patch
> into their own config-only patch
> * removed the obsolete // FIXME comment on datastore_generation
> * added ConfigVersionCache::datastore_generation() as getter
>
> From v2 → v3
> No changes
>
> From v3 → v4
> No changes
>
> From v4 → v5
> - Rebased only, no changes
>
> pbs-config/src/config_version_cache.rs | 10 ++++++++--
> 1 file changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/pbs-config/src/config_version_cache.rs b/pbs-config/src/config_version_cache.rs
> index e8fb994f..b875f7e0 100644
> --- a/pbs-config/src/config_version_cache.rs
> +++ b/pbs-config/src/config_version_cache.rs
> @@ -26,7 +26,6 @@ struct ConfigVersionCacheDataInner {
> // Traffic control (traffic-control.cfg) generation/version.
> traffic_control_generation: AtomicUsize,
> // datastore (datastore.cfg) generation/version
> - // FIXME: remove with PBS 3.0
> datastore_generation: AtomicUsize,
> // Add further atomics here
> }
> @@ -145,8 +144,15 @@ impl ConfigVersionCache {
> .fetch_add(1, Ordering::AcqRel);
> }
>
> + /// Returns the datastore generation number.
> + pub fn datastore_generation(&self) -> usize {
> + self.shmem
> + .data()
> + .datastore_generation
> + .load(Ordering::Acquire)
> + }
> +
> /// Increase the datastore generation number.
> - // FIXME: remove with PBS 3.0 or make actually useful again in datastore lookup
> pub fn increase_datastore_generation(&self) -> usize {
> self.shmem
> .data()
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
2025-11-24 17:04 14% ` [pbs-devel] [PATCH proxmox-backup v5 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
@ 2025-11-26 15:15 5% ` Fabian Grünbichler
0 siblings, 0 replies; 200+ results
From: Fabian Grünbichler @ 2025-11-26 15:15 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
nit for the subject: this doesn't fix the reported issue, it just
improves the fix further, so please drop that part and maybe instead add
"lookup" somewhere..
On November 24, 2025 6:04 pm, Samuel Rufinatscha wrote:
> The lookup fast path reacts to API-driven config changes because
> save_config() bumps the generation. Manual edits of datastore.cfg do
> not bump the counter. To keep the system robust against such edits
> without reintroducing config reading and hashing on the hot path, this
> patch adds a TTL to the cache entry.
>
> If the cached config is older than
> DATASTORE_CONFIG_CACHE_TTL_SECS (set to 60s), the next lookup takes
> the slow path and refreshes the entry. As an optimization, a check to
> catch manual edits was added (if the digest changed but generation
> stayed the same), so that the generation is only bumped when needed.
>
> Links
>
> [1] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>
> Fixes: #6049
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
one style nit below, otherwise:
Reviewed-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
> ---
> Changes:
>
> From v1 → v2
> - Store last_update timestamp in DatastoreConfigCache type.
>
> From v2 → v3
> No changes
>
> From v3 → v4
> - Fix digest generation bump logic in update_cache, thanks @Fabian.
>
> From v4 → v5
> - Rebased only, no changes
>
> pbs-datastore/src/datastore.rs | 53 ++++++++++++++++++++++++----------
> 1 file changed, 38 insertions(+), 15 deletions(-)
>
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 7638a899..0fc3fbf2 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -53,8 +53,12 @@ use crate::{DataBlob, LocalDatastoreLruCache};
> struct DatastoreConfigCache {
> // Parsed datastore.cfg file
> config: Arc<SectionConfigData>,
> + // Digest of the datastore.cfg file
> + digest: [u8; 32],
> // Generation number from ConfigVersionCache
> last_generation: usize,
> + // Last update time (epoch seconds)
> + last_update: i64,
> }
>
> static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
> @@ -63,6 +67,8 @@ static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
> static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
> LazyLock::new(|| Mutex::new(HashMap::new()));
>
> +/// Max age in seconds to reuse the cached datastore config.
> +const DATASTORE_CONFIG_CACHE_TTL_SECS: i64 = 60;
> /// Filename to store backup group notes
> pub const GROUP_NOTES_FILE_NAME: &str = "notes";
> /// Filename to store backup group owner
> @@ -329,13 +335,14 @@ impl DatastoreThreadSettings {
> /// generation.
> ///
> /// Uses `ConfigVersionCache` to detect stale entries:
> -/// - If the cached generation matches the current generation, the
> -/// cached config is returned.
> +/// - If the cached generation matches the current generation and TTL is
> +/// OK, the cached config is returned.
> /// - Otherwise the config is re-read from disk. If `update_cache` is
> -/// `true`, the new config and bumped generation are stored in the
> -/// cache. Callers that set `update_cache = true` must hold the
> -/// datastore config lock to avoid racing with concurrent config
> -/// changes.
> +/// `true` and a previous cached entry exists with the same generation
> +/// but a different digest, this indicates the config has changed
> +/// (e.g. manual edit) and the generation must be bumped. Callers
> +/// that set `update_cache = true` must hold the datastore config lock
> +/// to avoid racing with concurrent config changes.
> /// - If `update_cache` is `false`, the freshly read config is returned
> /// but the cache and generation are left unchanged.
> ///
> @@ -347,30 +354,46 @@ fn datastore_section_config_cached(
> let mut config_cache = DATASTORE_CONFIG_CACHE.lock().unwrap();
>
> if let Ok(version_cache) = ConfigVersionCache::new() {
> + let now = epoch_i64();
> let current_gen = version_cache.datastore_generation();
> if let Some(cached) = config_cache.as_ref() {
> - // Fast path: re-use cached datastore.cfg
> - if cached.last_generation == current_gen {
> + // Fast path: re-use cached datastore.cfg if generation matches and TTL not expired
> + if cached.last_generation == current_gen
> + && now - cached.last_update < DATASTORE_CONFIG_CACHE_TTL_SECS
> + {
> return Ok((cached.config.clone(), Some(cached.last_generation)));
> }
> }
> // Slow path: re-read datastore.cfg
> - let (config_raw, _digest) = pbs_config::datastore::config()?;
> + let (config_raw, digest) = pbs_config::datastore::config()?;
> let config = Arc::new(config_raw);
>
> let mut effective_gen = current_gen;
> if update_cache {
> - // Bump the generation. This ensures that Drop
> - // handlers will detect that a newer config exists
> - // and will not rely on a stale cached entry for
> - // maintenance mandate.
> - let prev_gen = version_cache.increase_datastore_generation();
> - effective_gen = prev_gen + 1;
> + // Bump the generation if the config has been changed manually.
> + // This ensures that Drop handlers will detect that a newer config exists
> + // and will not rely on a stale cached entry for maintenance mandate.
> + let (prev_gen, prev_digest) = config_cache
> + .as_ref()
> + .map(|c| (Some(c.last_generation), Some(c.digest)))
> + .unwrap_or((None, None));
so here we map an option to a tuple of options and unwrap it
> +
> + let manual_edit = match (prev_gen, prev_digest) {
only to then match and convert it to a boolean again here
> + (Some(prev_g), Some(prev_d)) => prev_g == current_gen && prev_d != digest,
> + _ => false,
> + };
> +
> + if manual_edit {
> + let prev_gen = version_cache.increase_datastore_generation();
> + effective_gen = prev_gen + 1;
to then do some code here, if the boolean is true ;)
this can all just be a single block of code instead:
if let Some(cached) = config_cache.as_ref() {
if cached.last_generation == current_gen && cached.digest != digest {
effective_gen = version_cache.increase_datastore_generation() + 1;
}
}
which also matches the first block higher up in the helper..
> + }
>
> // Persist
> *config_cache = Some(DatastoreConfigCache {
> config: config.clone(),
> + digest,
> last_generation: effective_gen,
> + last_update: now,
> });
> }
>
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 3/4] partial fix #6049: datastore: use config fast-path in Drop
2025-11-24 17:04 14% ` [pbs-devel] [PATCH proxmox-backup v5 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
@ 2025-11-26 15:15 5% ` Fabian Grünbichler
2025-11-28 9:03 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-11-26 15:15 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On November 24, 2025 6:04 pm, Samuel Rufinatscha wrote:
> The Drop impl of DataStore re-read datastore.cfg to decide whether
> the entry should be evicted from the in-process cache (based on
> maintenance mode’s clear_from_cache). During the investigation of
> issue #6049 [1], a flamegraph [2] showed that the config reload in Drop
> accounted for a measurable share of CPU time under load.
>
> This patch wires the datastore config fast path to the Drop
> impl to eventually avoid an expensive config reload from disk to capture
> the maintenance mandate. Also, to ensure the Drop handlers will detect
> that a newer config exists / to mitigate usage of an eventually stale
> cached entry, generation will not only be bumped on config save, but also
> on re-read of the config file (slow path), if `update_cache = true`.
>
> Links
>
> [1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>
> Fixes: #6049
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> Changes:
>
> From v1 → v2
> - Replace caching logic with the datastore_section_config_cached()
> helper.
>
> From v2 → v3
> No changes
>
> From v3 → v4, thanks @Fabian
> - Pass datastore_section_config_cached(false) in Drop to avoid
> concurrent cache updates.
>
> From v4 → v5
> - Rebased only, no changes
>
> pbs-datastore/src/datastore.rs | 60 ++++++++++++++++++++++++++--------
> 1 file changed, 47 insertions(+), 13 deletions(-)
>
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index c9cb5d65..7638a899 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -225,15 +225,40 @@ impl Drop for DataStore {
> // remove datastore from cache iff
> // - last task finished, and
> // - datastore is in a maintenance mode that mandates it
> - let remove_from_cache = last_task
> - && pbs_config::datastore::config()
> - .and_then(|(s, _)| s.lookup::<DataStoreConfig>("datastore", self.name()))
> - .is_ok_and(|c| {
> - c.get_maintenance_mode()
> - .is_some_and(|m| m.clear_from_cache())
> - });
old code here ignored parsing/locking/.. issues and just assumed if no
config can be obtained nothing should be done..
> -
> - if remove_from_cache {
> +
> + // first check: check if last task finished
> + if !last_task {
> + return;
> + }
> +
> + let (section_config, _gen) = match datastore_section_config_cached(false) {
> + Ok(v) => v,
> + Err(err) => {
> + log::error!(
> + "failed to load datastore config in Drop for {} - {err}",
> + self.name()
> + );
> + return;
> + }
> + };
> +
> + let datastore_cfg: DataStoreConfig =
> + match section_config.lookup("datastore", self.name()) {
> + Ok(cfg) => cfg,
> + Err(err) => {
> + log::error!(
> + "failed to look up datastore '{}' in Drop - {err}",
> + self.name()
> + );
> + return;
here we now have fancy error logging ;) which can be fine, but if we go
from silently ignoring errors to logging them at error level that should
be mentioned to make it clear that it is intentional.
besides that, the second error here means that the datastore was removed
from the config in the meantime.. in which case we should probably
remove it from the map as well, if is still there, even though we can't
check the maintenance mode..
> + }
> + };
> +
> + // second check: check maintenance mode mandate
what is a "maintenance mode mandate"? ;)
keeping it simple, why not just
// check if maintenance mode requires closing FDs
> + if datastore_cfg
> + .get_maintenance_mode()
> + .is_some_and(|m| m.clear_from_cache())
> + {
> DATASTORE_MAP.lock().unwrap().remove(self.name());
> }
> }
> @@ -307,12 +332,12 @@ impl DatastoreThreadSettings {
> /// - If the cached generation matches the current generation, the
> /// cached config is returned.
> /// - Otherwise the config is re-read from disk. If `update_cache` is
> -/// `true`, the new config and current generation are stored in the
> +/// `true`, the new config and bumped generation are stored in the
> /// cache. Callers that set `update_cache = true` must hold the
> /// datastore config lock to avoid racing with concurrent config
> /// changes.
> /// - If `update_cache` is `false`, the freshly read config is returned
> -/// but the cache is left unchanged.
> +/// but the cache and generation are left unchanged.
> ///
> /// If `ConfigVersionCache` is not available, the config is always read
> /// from disk and `None` is returned as the generation.
> @@ -333,14 +358,23 @@ fn datastore_section_config_cached(
does this part here make any sense in this patch?
we don't check the generation in the Drop handler anyway, so it will get
the latest cached version, no matter what?
we'd only end up in this part of the code via lookup_datastore, and only
if:
- the previous cached entry and the current one have a different
generation -> no need to bump again, the cache is already invalidated
- there is no previous cached entry -> nothing to invalidate
I think this part should move to the next patch..
> let (config_raw, _digest) = pbs_config::datastore::config()?;
> let config = Arc::new(config_raw);
>
> + let mut effective_gen = current_gen;
> if update_cache {
> + // Bump the generation. This ensures that Drop
> + // handlers will detect that a newer config exists
> + // and will not rely on a stale cached entry for
> + // maintenance mandate.
> + let prev_gen = version_cache.increase_datastore_generation();
> + effective_gen = prev_gen + 1;
> +
> + // Persist
> *config_cache = Some(DatastoreConfigCache {
> config: config.clone(),
> - last_generation: current_gen,
> + last_generation: effective_gen,
> });
> }
>
> - Ok((config, Some(current_gen)))
> + Ok((config, Some(effective_gen)))
> } else {
> // Fallback path, no config version cache: read datastore.cfg and return None as generation
> *config_cache = None;
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
2025-11-24 17:04 11% ` [pbs-devel] [PATCH proxmox-backup v5 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
@ 2025-11-26 15:15 5% ` Fabian Grünbichler
2025-11-26 17:21 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-11-26 15:15 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On November 24, 2025 6:04 pm, Samuel Rufinatscha wrote:
> Repeated /status requests caused lookup_datastore() to re-read and
> parse datastore.cfg on every call. The issue was mentioned in report
> #6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
> dominated by pbs_config::datastore::config() (config parsing).
>
> This patch implements caching of the global datastore.cfg using the
> generation numbers from the shared config version cache. It caches the
> datastore.cfg along with the generation number and, when a subsequent
> lookup sees the same generation, it reuses the cached config without
> re-reading it from disk. If the generation differs
> (or the cache is unavailable), the config is re-read from disk.
> If `update_cache = true`, the new config and current generation are
> persisted in the cache. In this case, callers must hold the datastore
> config lock to avoid racing with concurrent config changes.
> If `update_cache` is `false` and generation did not match, the freshly
> read config is returned but the cache is left unchanged. If
> `ConfigVersionCache` is not available, the config is always read from
> disk and `None` is returned as generation.
>
> Behavioral notes
>
> - The generation is bumped via the existing save_config() path, so
> API-driven config changes are detected immediately.
> - Manual edits to datastore.cfg are not detected; this is covered in a
> dedicated patch in this series.
> - DataStore::drop still performs a config read on the common path;
> also covered in a dedicated patch in this series.
>
> Links
>
> [1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>
> Fixes: #6049
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
style nits below, but otherwise
Reviewed-by: Fabian Grünbicher <f.gruenbichler@proxmox.com>
> ---
> Changes:
>
> From v1 → v2, thanks @Fabian
> - Moved the ConfigVersionCache changes into its own patch.
> - Introduced the global static DATASTORE_CONFIG_CACHE to store the
> fully parsed datastore.cfg instead, along with its generation number.
> Introduced DatastoreConfigCache struct to hold both.
> - Removed and replaced the CachedDatastoreConfigTag field of
> DataStoreImpl with a generation number field only (Option<usize>)
> to validate DataStoreImpl reuse.
> - Added DataStore::datastore_section_config_cached() helper function
> to encapsulate the caching logic and simplify reuse.
> - Modified DataStore::lookup_datastore() to use the new helper.
>
> From v2 → v3
> No changes
>
> From v3 → v4, thanks @Fabian
> - Restructured the version cache checks in
> datastore_section_config_cached(), to simplify the logic.
> - Added update_cache parameter to datastore_section_config_cached() to
> control cache updates.
>
> From v4 → v5
> - Rebased only, no changes
>
> pbs-datastore/Cargo.toml | 1 +
> pbs-datastore/src/datastore.rs | 138 +++++++++++++++++++++++++--------
> 2 files changed, 105 insertions(+), 34 deletions(-)
this could be
2 files changed, 80 insertions(+), 17 deletions(-)
see below. this might sound nit-picky, but keeping diffs concise makes
reviewing a lot easier because of the higher signal to noise ratio..
>
> diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
> index 8ce930a9..42f49a7b 100644
> --- a/pbs-datastore/Cargo.toml
> +++ b/pbs-datastore/Cargo.toml
> @@ -40,6 +40,7 @@ proxmox-io.workspace = true
> proxmox-lang.workspace=true
> proxmox-s3-client = { workspace = true, features = [ "impl" ] }
> proxmox-schema = { workspace = true, features = [ "api-macro" ] }
> +proxmox-section-config.workspace = true
> proxmox-serde = { workspace = true, features = [ "serde_json" ] }
> proxmox-sys.workspace = true
> proxmox-systemd.workspace = true
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 36550ff6..c9cb5d65 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -34,7 +34,8 @@ use pbs_api_types::{
> MaintenanceType, Operation, UPID,
> };
> use pbs_config::s3::S3_CFG_TYPE_ID;
> -use pbs_config::BackupLockGuard;
> +use pbs_config::{BackupLockGuard, ConfigVersionCache};
> +use proxmox_section_config::SectionConfigData;
>
> use crate::backup_info::{
> BackupDir, BackupGroup, BackupInfo, OLD_LOCKING, PROTECTED_MARKER_FILENAME,
> @@ -48,6 +49,17 @@ use crate::s3::S3_CONTENT_PREFIX;
> use crate::task_tracking::{self, update_active_operations};
> use crate::{DataBlob, LocalDatastoreLruCache};
>
> +// Cache for fully parsed datastore.cfg
> +struct DatastoreConfigCache {
> + // Parsed datastore.cfg file
> + config: Arc<SectionConfigData>,
> + // Generation number from ConfigVersionCache
> + last_generation: usize,
> +}
> +
> +static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
> + LazyLock::new(|| Mutex::new(None));
> +
> static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
> LazyLock::new(|| Mutex::new(HashMap::new()));
>
> @@ -149,11 +161,13 @@ pub struct DataStoreImpl {
> last_gc_status: Mutex<GarbageCollectionStatus>,
> verify_new: bool,
> chunk_order: ChunkOrder,
> - last_digest: Option<[u8; 32]>,
> sync_level: DatastoreFSyncLevel,
> backend_config: DatastoreBackendConfig,
> lru_store_caching: Option<LocalDatastoreLruCache>,
> thread_settings: DatastoreThreadSettings,
> + /// Datastore generation number from `ConfigVersionCache` at creation time, used to
datastore.cfg cache generation number at lookup time, used to invalidate
this cached `DataStoreImpl`
creation time could also refer to the datastore creation time..
> + /// validate reuse of this cached `DataStoreImpl`.
> + config_generation: Option<usize>,
> }
>
> impl DataStoreImpl {
> @@ -166,11 +180,11 @@ impl DataStoreImpl {
> last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
> verify_new: false,
> chunk_order: Default::default(),
> - last_digest: None,
> sync_level: Default::default(),
> backend_config: Default::default(),
> lru_store_caching: None,
> thread_settings: Default::default(),
> + config_generation: None,
> })
> }
> }
> @@ -286,6 +300,55 @@ impl DatastoreThreadSettings {
so this new helper is already +49 lines, which means it's the bulk of
the change if the noise below is removed..
> }
> }
>
> +/// Returns the parsed datastore config (`datastore.cfg`) and its
> +/// generation.
> +///
> +/// Uses `ConfigVersionCache` to detect stale entries:
> +/// - If the cached generation matches the current generation, the
> +/// cached config is returned.
> +/// - Otherwise the config is re-read from disk. If `update_cache` is
> +/// `true`, the new config and current generation are stored in the
> +/// cache. Callers that set `update_cache = true` must hold the
> +/// datastore config lock to avoid racing with concurrent config
> +/// changes.
> +/// - If `update_cache` is `false`, the freshly read config is returned
> +/// but the cache is left unchanged.
> +///
> +/// If `ConfigVersionCache` is not available, the config is always read
> +/// from disk and `None` is returned as the generation.
> +fn datastore_section_config_cached(
> + update_cache: bool,
> +) -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
> + let mut config_cache = DATASTORE_CONFIG_CACHE.lock().unwrap();
> +
> + if let Ok(version_cache) = ConfigVersionCache::new() {
> + let current_gen = version_cache.datastore_generation();
> + if let Some(cached) = config_cache.as_ref() {
> + // Fast path: re-use cached datastore.cfg
> + if cached.last_generation == current_gen {
> + return Ok((cached.config.clone(), Some(cached.last_generation)));
> + }
> + }
> + // Slow path: re-read datastore.cfg
> + let (config_raw, _digest) = pbs_config::datastore::config()?;
> + let config = Arc::new(config_raw);
> +
> + if update_cache {
> + *config_cache = Some(DatastoreConfigCache {
> + config: config.clone(),
> + last_generation: current_gen,
> + });
> + }
> +
> + Ok((config, Some(current_gen)))
> + } else {
> + // Fallback path, no config version cache: read datastore.cfg and return None as generation
> + *config_cache = None;
> + let (config_raw, _digest) = pbs_config::datastore::config()?;
> + Ok((Arc::new(config_raw), None))
> + }
> +}
> +
> impl DataStore {
> // This one just panics on everything
> #[doc(hidden)]
> @@ -363,56 +426,63 @@ impl DataStore {
> name: &str,
> operation: Option<Operation>,
> ) -> Result<Arc<DataStore>, Error> {
the changes here contain a lot of churn that is not needed for the
actual change that is done - e.g., there's lots of variables being
needless renamed..
> - // Avoid TOCTOU between checking maintenance mode and updating active operation counter, as
> - // we use it to decide whether it is okay to delete the datastore.
> + // Avoid TOCTOU between checking maintenance mode and updating active operations.
> let _config_lock = pbs_config::datastore::lock_config()?;
>
> - // we could use the ConfigVersionCache's generation for staleness detection, but we load
> - // the config anyway -> just use digest, additional benefit: manual changes get detected
> - let (config, digest) = pbs_config::datastore::config()?;
> - let config: DataStoreConfig = config.lookup("datastore", name)?;
> + // Get the current datastore.cfg generation number and cached config
> + let (section_config, gen_num) = datastore_section_config_cached(true)?;
> +
> + let datastore_cfg: DataStoreConfig = section_config.lookup("datastore", name)?;
renaming this variable here already causes a lot of noise - if you
really want to do that, do it up-front or at the end as cleanup commit
(depending on how sure you are the change is agree-able - if it's almost
certain it is accepted, put it up front, then it can get applied
already. if not, put it at the end -> then it can be skipped without
affecting the other patches ;))
but going from config to datastore_cfg doesn't make the code any clearer
- the latter can still refer to the whole config (datastore.cfg) or the
individual datastore's section within.. since we have no futher business
with the whole section config here, I think keeping this as `config` is
fine..
> + let maintenance_mode = datastore_cfg.get_maintenance_mode();
> + let mount_status = get_datastore_mount_status(&datastore_cfg);
and things extracted into variables here despite being used only once
(this also doesn't change during the rest of the series)
>
> - if let Some(maintenance_mode) = config.get_maintenance_mode() {
> - if let Err(error) = maintenance_mode.check(operation) {
> + if let Some(mm) = &maintenance_mode {
binding name changes
> + if let Err(error) = mm.check(operation.clone()) {
new clone() is introduced but not needed
> bail!("datastore '{name}' is unavailable: {error}");
> }
> }
>
> - if get_datastore_mount_status(&config) == Some(false) {
> - let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
> - datastore_cache.remove(&config.name);
> - bail!("datastore '{}' is not mounted", config.name);
> + let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
moving this is fine!
> +
> + if mount_status == Some(false) {
> + datastore_cache.remove(&datastore_cfg.name);
> + bail!("datastore '{}' is not mounted", datastore_cfg.name);
> }
>
> - let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
> - let entry = datastore_cache.get(name);
this is changed to doing it twice and cloning..
> -
> - // reuse chunk store so that we keep using the same process locker instance!
> - let chunk_store = if let Some(datastore) = &entry {
structure changed here and binding renamed, even though the old one works as-is
> - let last_digest = datastore.last_digest.as_ref();
> - if let Some(true) = last_digest.map(|last_digest| last_digest == &digest) {
> - if let Some(operation) = operation {
> - update_active_operations(name, operation, 1)?;
> + // Re-use DataStoreImpl
> + if let Some(existing) = datastore_cache.get(name).cloned() {
> + if let (Some(last_generation), Some(gen_num)) = (existing.config_generation, gen_num) {
> + if last_generation == gen_num {
these two ifs can be collapsed into
if datastore.config_generation == gen_num && gen_num.is_some() {
since we only want to reuse the entry if the current gen num is the same
as the last one.
> + if let Some(op) = operation {
binding needlessly renamed
> + update_active_operations(name, op, 1)?;
> + }
> +
> + return Ok(Arc::new(Self {
> + inner: existing,
> + operation,
> + }));
> }
> - return Ok(Arc::new(Self {
> - inner: Arc::clone(datastore),
> - operation,
> - }));
> }
> - Arc::clone(&datastore.chunk_store)
> + }
> +
> + // (Re)build DataStoreImpl
> +
> + // Reuse chunk store so that we keep using the same process locker instance!
> + let chunk_store = if let Some(existing) = datastore_cache.get(name) {
> + Arc::clone(&existing.chunk_store)
this one is not needed at all, can just be combined with the previous if
as before
> } else {
> let tuning: DatastoreTuning = serde_json::from_value(
> DatastoreTuning::API_SCHEMA
> - .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
> + .parse_property_string(datastore_cfg.tuning.as_deref().unwrap_or(""))?,
> )?;
> Arc::new(ChunkStore::open(
> name,
> - config.absolute_path(),
> + datastore_cfg.absolute_path(),
> tuning.sync_level.unwrap_or_default(),
> )?)
> };
>
> - let datastore = DataStore::with_store_and_config(chunk_store, config, Some(digest))?;
> + let datastore = DataStore::with_store_and_config(chunk_store, datastore_cfg, gen_num)?;
>
> let datastore = Arc::new(datastore);
> datastore_cache.insert(name.to_string(), datastore.clone());
> @@ -514,7 +584,7 @@ impl DataStore {
> fn with_store_and_config(
> chunk_store: Arc<ChunkStore>,
> config: DataStoreConfig,
> - last_digest: Option<[u8; 32]>,
> + generation: Option<usize>,
> ) -> Result<DataStoreImpl, Error> {
> let mut gc_status_path = chunk_store.base_path();
> gc_status_path.push(".gc-status");
> @@ -579,11 +649,11 @@ impl DataStore {
> last_gc_status: Mutex::new(gc_status),
> verify_new: config.verify_new.unwrap_or(false),
> chunk_order: tuning.chunk_order.unwrap_or_default(),
> - last_digest,
> sync_level: tuning.sync_level.unwrap_or_default(),
> backend_config,
> lru_store_caching,
> thread_settings,
> + config_generation: generation,
> })
> }
>
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 0/4] datastore: remove config reload on hot path
2025-11-24 17:04 12% [pbs-devel] [PATCH proxmox-backup v5 " Samuel Rufinatscha
` (3 preceding siblings ...)
2025-11-24 17:04 14% ` [pbs-devel] [PATCH proxmox-backup v5 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
@ 2025-11-26 15:16 5% ` Fabian Grünbichler
2025-11-26 16:10 6% ` Samuel Rufinatscha
2026-01-05 14:21 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
5 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-11-26 15:16 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On November 24, 2025 6:04 pm, Samuel Rufinatscha wrote:
> Hi,
>
> this series reduces CPU time in datastore lookups by avoiding repeated
> datastore.cfg reads/parses in both `lookup_datastore()` and
> `DataStore::Drop`. It also adds a TTL so manual config edits are
> noticed without reintroducing hashing on every request.
>
> While investigating #6049 [1], cargo-flamegraph [2] showed hotspots
> during repeated `/status` calls in `lookup_datastore()` and in `Drop`,
> dominated by `pbs_config::datastore::config()` (config parse).
>
> The parsing cost itself should eventually be investigated in a future
> effort. Furthermore, cargo-flamegraph showed that when using a
> token-based auth method to access the API, a significant amount of time
> is spent in validation on every request request [3].
>
> ## Approach
>
> [PATCH 1/4] Support datastore generation in ConfigVersionCache
>
> [PATCH 2/4] Fast path for datastore lookups
> Cache the parsed datastore.cfg keyed by the shared datastore
> generation. lookup_datastore() reuses both the cached config and an
> existing DataStoreImpl when the generation matches, and falls back
> to the old slow path otherwise. The caching logic is implemented
> using the datastore_section_config_cached(update_cache: bool) helper.
>
> [PATCH 3/4] Fast path for Drop
> Make DataStore::Drop use the datastore_section_config_cached()
> helper to avoid re-reading/parsing datastore.cfg on every Drop.
> Bump generation not only on API config saves, but also on slow-path
> lookups (if update_cache is true), to enable Drop handlers see
> eventual newer configs.
>
> [PATCH 4/4] TTL to catch manual edits
> Add a TTL to the cached config and bump the datastore generation iff
> the digest changed but generation stays the same. This catches manual
> edits to datastore.cfg without reintroducing hashing or config
> parsing on every request.
semantics wise this looks mostly good to me now, sent a few style
remarks for the individual patches. let's wait for feedback from the
reporter, and then wrap this up hopefully :)
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 0/4] datastore: remove config reload on hot path
2025-11-26 15:16 5% ` [pbs-devel] [PATCH proxmox-backup v5 0/4] datastore: remove config reload on hot path Fabian Grünbichler
@ 2025-11-26 16:10 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-26 16:10 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 11/26/25 4:16 PM, Fabian Grünbichler wrote:
> On November 24, 2025 6:04 pm, Samuel Rufinatscha wrote:
>> Hi,
>>
>> this series reduces CPU time in datastore lookups by avoiding repeated
>> datastore.cfg reads/parses in both `lookup_datastore()` and
>> `DataStore::Drop`. It also adds a TTL so manual config edits are
>> noticed without reintroducing hashing on every request.
>>
>> While investigating #6049 [1], cargo-flamegraph [2] showed hotspots
>> during repeated `/status` calls in `lookup_datastore()` and in `Drop`,
>> dominated by `pbs_config::datastore::config()` (config parse).
>>
>> The parsing cost itself should eventually be investigated in a future
>> effort. Furthermore, cargo-flamegraph showed that when using a
>> token-based auth method to access the API, a significant amount of time
>> is spent in validation on every request request [3].
>>
>> ## Approach
>>
>> [PATCH 1/4] Support datastore generation in ConfigVersionCache
>>
>> [PATCH 2/4] Fast path for datastore lookups
>> Cache the parsed datastore.cfg keyed by the shared datastore
>> generation. lookup_datastore() reuses both the cached config and an
>> existing DataStoreImpl when the generation matches, and falls back
>> to the old slow path otherwise. The caching logic is implemented
>> using the datastore_section_config_cached(update_cache: bool) helper.
>>
>> [PATCH 3/4] Fast path for Drop
>> Make DataStore::Drop use the datastore_section_config_cached()
>> helper to avoid re-reading/parsing datastore.cfg on every Drop.
>> Bump generation not only on API config saves, but also on slow-path
>> lookups (if update_cache is true), to enable Drop handlers see
>> eventual newer configs.
>>
>> [PATCH 4/4] TTL to catch manual edits
>> Add a TTL to the cached config and bump the datastore generation iff
>> the digest changed but generation stays the same. This catches manual
>> edits to datastore.cfg without reintroducing hashing or config
>> parsing on every request.
>
> semantics wise this looks mostly good to me now, sent a few style
> remarks for the individual patches. let's wait for feedback from the
> reporter, and then wrap this up hopefully :)
Thanks for the great review Fabian! I agree, will make sure to integrate
the style remarks and avoid more of the diff noise :)
Thanks!
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
2025-11-26 15:15 5% ` Fabian Grünbichler
@ 2025-11-26 17:21 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-26 17:21 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 11/26/25 4:15 PM, Fabian Grünbichler wrote:
> On November 24, 2025 6:04 pm, Samuel Rufinatscha wrote:
>> Repeated /status requests caused lookup_datastore() to re-read and
>> parse datastore.cfg on every call. The issue was mentioned in report
>> #6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
>> dominated by pbs_config::datastore::config() (config parsing).
>>
>> This patch implements caching of the global datastore.cfg using the
>> generation numbers from the shared config version cache. It caches the
>> datastore.cfg along with the generation number and, when a subsequent
>> lookup sees the same generation, it reuses the cached config without
>> re-reading it from disk. If the generation differs
>> (or the cache is unavailable), the config is re-read from disk.
>> If `update_cache = true`, the new config and current generation are
>> persisted in the cache. In this case, callers must hold the datastore
>> config lock to avoid racing with concurrent config changes.
>> If `update_cache` is `false` and generation did not match, the freshly
>> read config is returned but the cache is left unchanged. If
>> `ConfigVersionCache` is not available, the config is always read from
>> disk and `None` is returned as generation.
>>
>> Behavioral notes
>>
>> - The generation is bumped via the existing save_config() path, so
>> API-driven config changes are detected immediately.
>> - Manual edits to datastore.cfg are not detected; this is covered in a
>> dedicated patch in this series.
>> - DataStore::drop still performs a config read on the common path;
>> also covered in a dedicated patch in this series.
>>
>> Links
>>
>> [1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
>> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>>
>> Fixes: #6049
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>
> style nits below, but otherwise
>
> Reviewed-by: Fabian Grünbicher <f.gruenbichler@proxmox.com>
>
>> ---
>> Changes:
>>
>> From v1 → v2, thanks @Fabian
>> - Moved the ConfigVersionCache changes into its own patch.
>> - Introduced the global static DATASTORE_CONFIG_CACHE to store the
>> fully parsed datastore.cfg instead, along with its generation number.
>> Introduced DatastoreConfigCache struct to hold both.
>> - Removed and replaced the CachedDatastoreConfigTag field of
>> DataStoreImpl with a generation number field only (Option<usize>)
>> to validate DataStoreImpl reuse.
>> - Added DataStore::datastore_section_config_cached() helper function
>> to encapsulate the caching logic and simplify reuse.
>> - Modified DataStore::lookup_datastore() to use the new helper.
>>
>> From v2 → v3
>> No changes
>>
>> From v3 → v4, thanks @Fabian
>> - Restructured the version cache checks in
>> datastore_section_config_cached(), to simplify the logic.
>> - Added update_cache parameter to datastore_section_config_cached() to
>> control cache updates.
>>
>> From v4 → v5
>> - Rebased only, no changes
>>
>> pbs-datastore/Cargo.toml | 1 +
>> pbs-datastore/src/datastore.rs | 138 +++++++++++++++++++++++++--------
>> 2 files changed, 105 insertions(+), 34 deletions(-)
>
> this could be
>
> 2 files changed, 80 insertions(+), 17 deletions(-)
>
> see below. this might sound nit-picky, but keeping diffs concise makes
> reviewing a lot easier because of the higher signal to noise ratio..
>
>>
>> diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
>> index 8ce930a9..42f49a7b 100644
>> --- a/pbs-datastore/Cargo.toml
>> +++ b/pbs-datastore/Cargo.toml
>> @@ -40,6 +40,7 @@ proxmox-io.workspace = true
>> proxmox-lang.workspace=true
>> proxmox-s3-client = { workspace = true, features = [ "impl" ] }
>> proxmox-schema = { workspace = true, features = [ "api-macro" ] }
>> +proxmox-section-config.workspace = true
>> proxmox-serde = { workspace = true, features = [ "serde_json" ] }
>> proxmox-sys.workspace = true
>> proxmox-systemd.workspace = true
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index 36550ff6..c9cb5d65 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -34,7 +34,8 @@ use pbs_api_types::{
>> MaintenanceType, Operation, UPID,
>> };
>> use pbs_config::s3::S3_CFG_TYPE_ID;
>> -use pbs_config::BackupLockGuard;
>> +use pbs_config::{BackupLockGuard, ConfigVersionCache};
>> +use proxmox_section_config::SectionConfigData;
>>
>> use crate::backup_info::{
>> BackupDir, BackupGroup, BackupInfo, OLD_LOCKING, PROTECTED_MARKER_FILENAME,
>> @@ -48,6 +49,17 @@ use crate::s3::S3_CONTENT_PREFIX;
>> use crate::task_tracking::{self, update_active_operations};
>> use crate::{DataBlob, LocalDatastoreLruCache};
>>
>> +// Cache for fully parsed datastore.cfg
>> +struct DatastoreConfigCache {
>> + // Parsed datastore.cfg file
>> + config: Arc<SectionConfigData>,
>> + // Generation number from ConfigVersionCache
>> + last_generation: usize,
>> +}
>> +
>> +static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
>> + LazyLock::new(|| Mutex::new(None));
>> +
>> static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
>> LazyLock::new(|| Mutex::new(HashMap::new()));
>>
>> @@ -149,11 +161,13 @@ pub struct DataStoreImpl {
>> last_gc_status: Mutex<GarbageCollectionStatus>,
>> verify_new: bool,
>> chunk_order: ChunkOrder,
>> - last_digest: Option<[u8; 32]>,
>> sync_level: DatastoreFSyncLevel,
>> backend_config: DatastoreBackendConfig,
>> lru_store_caching: Option<LocalDatastoreLruCache>,
>> thread_settings: DatastoreThreadSettings,
>> + /// Datastore generation number from `ConfigVersionCache` at creation time, used to
>
> datastore.cfg cache generation number at lookup time, used to invalidate
> this cached `DataStoreImpl`
>
> creation time could also refer to the datastore creation time..
>
Good point, agree! Will apply your suggestion, I like it more :)
>> + /// validate reuse of this cached `DataStoreImpl`.
>> + config_generation: Option<usize>,
>> }
>>
>> impl DataStoreImpl {
>> @@ -166,11 +180,11 @@ impl DataStoreImpl {
>> last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
>> verify_new: false,
>> chunk_order: Default::default(),
>> - last_digest: None,
>> sync_level: Default::default(),
>> backend_config: Default::default(),
>> lru_store_caching: None,
>> thread_settings: Default::default(),
>> + config_generation: None,
>> })
>> }
>> }
>> @@ -286,6 +300,55 @@ impl DatastoreThreadSettings {
>
> so this new helper is already +49 lines, which means it's the bulk of
> the change if the noise below is removed..
>
>> }
>> }
>>
>> +/// Returns the parsed datastore config (`datastore.cfg`) and its
>> +/// generation.
>> +///
>> +/// Uses `ConfigVersionCache` to detect stale entries:
>> +/// - If the cached generation matches the current generation, the
>> +/// cached config is returned.
>> +/// - Otherwise the config is re-read from disk. If `update_cache` is
>> +/// `true`, the new config and current generation are stored in the
>> +/// cache. Callers that set `update_cache = true` must hold the
>> +/// datastore config lock to avoid racing with concurrent config
>> +/// changes.
>> +/// - If `update_cache` is `false`, the freshly read config is returned
>> +/// but the cache is left unchanged.
>> +///
>> +/// If `ConfigVersionCache` is not available, the config is always read
>> +/// from disk and `None` is returned as the generation.
>> +fn datastore_section_config_cached(
>> + update_cache: bool,
>> +) -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
>> + let mut config_cache = DATASTORE_CONFIG_CACHE.lock().unwrap();
>> +
>> + if let Ok(version_cache) = ConfigVersionCache::new() {
>> + let current_gen = version_cache.datastore_generation();
>> + if let Some(cached) = config_cache.as_ref() {
>> + // Fast path: re-use cached datastore.cfg
>> + if cached.last_generation == current_gen {
>> + return Ok((cached.config.clone(), Some(cached.last_generation)));
>> + }
>> + }
>> + // Slow path: re-read datastore.cfg
>> + let (config_raw, _digest) = pbs_config::datastore::config()?;
>> + let config = Arc::new(config_raw);
>> +
>> + if update_cache {
>> + *config_cache = Some(DatastoreConfigCache {
>> + config: config.clone(),
>> + last_generation: current_gen,
>> + });
>> + }
>> +
>> + Ok((config, Some(current_gen)))
>> + } else {
>> + // Fallback path, no config version cache: read datastore.cfg and return None as generation
>> + *config_cache = None;
>> + let (config_raw, _digest) = pbs_config::datastore::config()?;
>> + Ok((Arc::new(config_raw), None))
>> + }
>> +}
>> +
>> impl DataStore {
>> // This one just panics on everything
>> #[doc(hidden)]
>> @@ -363,56 +426,63 @@ impl DataStore {
>> name: &str,
>> operation: Option<Operation>,
>> ) -> Result<Arc<DataStore>, Error> {
>
> the changes here contain a lot of churn that is not needed for the
> actual change that is done - e.g., there's lots of variables being
> needless renamed..
>
>> - // Avoid TOCTOU between checking maintenance mode and updating active operation counter, as
>> - // we use it to decide whether it is okay to delete the datastore.
>> + // Avoid TOCTOU between checking maintenance mode and updating active operations.
>> let _config_lock = pbs_config::datastore::lock_config()?;
>>
>> - // we could use the ConfigVersionCache's generation for staleness detection, but we load
>> - // the config anyway -> just use digest, additional benefit: manual changes get detected
>> - let (config, digest) = pbs_config::datastore::config()?;
>> - let config: DataStoreConfig = config.lookup("datastore", name)?;
>> + // Get the current datastore.cfg generation number and cached config
>> + let (section_config, gen_num) = datastore_section_config_cached(true)?;
>> +
>> + let datastore_cfg: DataStoreConfig = section_config.lookup("datastore", name)?;
>
> renaming this variable here already causes a lot of noise - if you
> really want to do that, do it up-front or at the end as cleanup commit
> (depending on how sure you are the change is agree-able - if it's almost
> certain it is accepted, put it up front, then it can get applied
> already. if not, put it at the end -> then it can be skipped without
> affecting the other patches ;))
>
> but going from config to datastore_cfg doesn't make the code any clearer
> - the latter can still refer to the whole config (datastore.cfg) or the
> individual datastore's section within.. since we have no futher business
> with the whole section config here, I think keeping this as `config` is
> fine..
>
Agree, will change back!
As you mentioned, the idea of the rename was to somehow better
differentiate between the global datastore.cfg and the individual
datastore config.
As we have 2 configs here in the same block, I tried to avoid to rely on
"config". As you said datastore_cfg
does not make it much better, or could be misinterpreted. Ultimatively,
I directly aligend with the type names (datastore_cfg for
DataStoreConfig).
I see your point and agree:
since we have no futher business
> with the whole section config here, I think keeping this as `config` is
> fine..
This is the best solution here.
>> + let maintenance_mode = datastore_cfg.get_maintenance_mode();
>> + let mount_status = get_datastore_mount_status(&datastore_cfg);
>
> and things extracted into variables here despite being used only once
> (this also doesn't change during the rest of the series)
>
Good point, will remove!
>>
>> - if let Some(maintenance_mode) = config.get_maintenance_mode() {
>> - if let Err(error) = maintenance_mode.check(operation) {
>> + if let Some(mm) = &maintenance_mode {
>
> binding name changes
>
>> + if let Err(error) = mm.check(operation.clone()) {
>
> new clone() is introduced but not needed
Nice catch!
>
>> bail!("datastore '{name}' is unavailable: {error}");
>> }
>> }
>>
>> - if get_datastore_mount_status(&config) == Some(false) {
>> - let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
>> - datastore_cache.remove(&config.name);
>> - bail!("datastore '{}' is not mounted", config.name);
>> + let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
>
> moving this is fine!
>
Helps to avoid the duplicate checks :)
>> +
>> + if mount_status == Some(false) {
>> + datastore_cache.remove(&datastore_cfg.name);
>> + bail!("datastore '{}' is not mounted", datastore_cfg.name);
>> }
>>
>> - let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
>> - let entry = datastore_cache.get(name);
>
> this is changed to doing it twice and cloning..
>
Refactoring this!
>> -
>> - // reuse chunk store so that we keep using the same process locker instance!
>> - let chunk_store = if let Some(datastore) = &entry {
>
> structure changed here and binding renamed, even though the old one works as-is
>
Agree, will revert!
>> - let last_digest = datastore.last_digest.as_ref();
>> - if let Some(true) = last_digest.map(|last_digest| last_digest == &digest) {
>> - if let Some(operation) = operation {
>> - update_active_operations(name, operation, 1)?;
>> + // Re-use DataStoreImpl
>> + if let Some(existing) = datastore_cache.get(name).cloned() {
>> + if let (Some(last_generation), Some(gen_num)) = (existing.config_generation, gen_num) {
>> + if last_generation == gen_num {
>
> these two ifs can be collapsed into
>
> if datastore.config_generation == gen_num && gen_num.is_some() {
>
> since we only want to reuse the entry if the current gen num is the same
> as the last one.
>
Will collapse the two ifs, good catch!
>> + if let Some(op) = operation {
>
> binding needlessly renamed
>
Agree, will revert!
>> + update_active_operations(name, op, 1)?;
>> + }
>> +
>> + return Ok(Arc::new(Self {
>> + inner: existing,
>> + operation,
>> + }));
>> }
>> - return Ok(Arc::new(Self {
>> - inner: Arc::clone(datastore),
>> - operation,
>> - }));
>> }
>> - Arc::clone(&datastore.chunk_store)
>> + }
>> +
>> + // (Re)build DataStoreImpl
>> +
>> + // Reuse chunk store so that we keep using the same process locker instance!
>> + let chunk_store = if let Some(existing) = datastore_cache.get(name) {
>> + Arc::clone(&existing.chunk_store)
>
> this one is not needed at all, can just be combined with the previous if
> as before
>
Will refactor!
>> } else {
>> let tuning: DatastoreTuning = serde_json::from_value(
>> DatastoreTuning::API_SCHEMA
>> - .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
>> + .parse_property_string(datastore_cfg.tuning.as_deref().unwrap_or(""))?,
>> )?;
>> Arc::new(ChunkStore::open(
>> name,
>> - config.absolute_path(),
>> + datastore_cfg.absolute_path(),
>> tuning.sync_level.unwrap_or_default(),
>> )?)
>> };
>>
>> - let datastore = DataStore::with_store_and_config(chunk_store, config, Some(digest))?;
>> + let datastore = DataStore::with_store_and_config(chunk_store, datastore_cfg, gen_num)?;
>>
>> let datastore = Arc::new(datastore);
>> datastore_cache.insert(name.to_string(), datastore.clone());
>> @@ -514,7 +584,7 @@ impl DataStore {
>> fn with_store_and_config(
>> chunk_store: Arc<ChunkStore>,
>> config: DataStoreConfig,
>> - last_digest: Option<[u8; 32]>,
>> + generation: Option<usize>,
>> ) -> Result<DataStoreImpl, Error> {
>> let mut gc_status_path = chunk_store.base_path();
>> gc_status_path.push(".gc-status");
>> @@ -579,11 +649,11 @@ impl DataStore {
>> last_gc_status: Mutex::new(gc_status),
>> verify_new: config.verify_new.unwrap_or(false),
>> chunk_order: tuning.chunk_order.unwrap_or_default(),
>> - last_digest,
>> sync_level: tuning.sync_level.unwrap_or_default(),
>> backend_config,
>> lru_store_caching,
>> thread_settings,
>> + config_generation: generation,
>> })
>> }
>>
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 3/4] partial fix #6049: datastore: use config fast-path in Drop
2025-11-26 15:15 5% ` Fabian Grünbichler
@ 2025-11-28 9:03 6% ` Samuel Rufinatscha
2025-11-28 10:46 5% ` Fabian Grünbichler
0 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-11-28 9:03 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 11/26/25 4:15 PM, Fabian Grünbichler wrote:
> On November 24, 2025 6:04 pm, Samuel Rufinatscha wrote:
>> The Drop impl of DataStore re-read datastore.cfg to decide whether
>> the entry should be evicted from the in-process cache (based on
>> maintenance mode’s clear_from_cache). During the investigation of
>> issue #6049 [1], a flamegraph [2] showed that the config reload in Drop
>> accounted for a measurable share of CPU time under load.
>>
>> This patch wires the datastore config fast path to the Drop
>> impl to eventually avoid an expensive config reload from disk to capture
>> the maintenance mandate. Also, to ensure the Drop handlers will detect
>> that a newer config exists / to mitigate usage of an eventually stale
>> cached entry, generation will not only be bumped on config save, but also
>> on re-read of the config file (slow path), if `update_cache = true`.
>>
>> Links
>>
>> [1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
>> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
>>
>> Fixes: #6049
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> Changes:
>>
>> From v1 → v2
>> - Replace caching logic with the datastore_section_config_cached()
>> helper.
>>
>> From v2 → v3
>> No changes
>>
>> From v3 → v4, thanks @Fabian
>> - Pass datastore_section_config_cached(false) in Drop to avoid
>> concurrent cache updates.
>>
>> From v4 → v5
>> - Rebased only, no changes
>>
>> pbs-datastore/src/datastore.rs | 60 ++++++++++++++++++++++++++--------
>> 1 file changed, 47 insertions(+), 13 deletions(-)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index c9cb5d65..7638a899 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -225,15 +225,40 @@ impl Drop for DataStore {
>> // remove datastore from cache iff
>> // - last task finished, and
>> // - datastore is in a maintenance mode that mandates it
>> - let remove_from_cache = last_task
>> - && pbs_config::datastore::config()
>> - .and_then(|(s, _)| s.lookup::<DataStoreConfig>("datastore", self.name()))
>> - .is_ok_and(|c| {
>> - c.get_maintenance_mode()
>> - .is_some_and(|m| m.clear_from_cache())
>> - });
>
> old code here ignored parsing/locking/.. issues and just assumed if no
> config can be obtained nothing should be done..
>
>> -
>> - if remove_from_cache {
>> +
>> + // first check: check if last task finished
>> + if !last_task {
>> + return;
>> + }
>> +
>> + let (section_config, _gen) = match datastore_section_config_cached(false) {
>> + Ok(v) => v,
>> + Err(err) => {
>> + log::error!(
>> + "failed to load datastore config in Drop for {} - {err}",
>> + self.name()
>> + );
>> + return;
>> + }
>> + };
>> +
>> + let datastore_cfg: DataStoreConfig =
>> + match section_config.lookup("datastore", self.name()) {
>> + Ok(cfg) => cfg,
>> + Err(err) => {
>> + log::error!(
>> + "failed to look up datastore '{}' in Drop - {err}",
>> + self.name()
>> + );
>> + return;
>
> here we now have fancy error logging ;) which can be fine, but if we go
> from silently ignoring errors to logging them at error level that should
> be mentioned to make it clear that it is intentional.
>
Makes sense, will mention that change in the commit message.
> besides that, the second error here means that the datastore was removed
> from the config in the meantime.. in which case we should probably
> remove it from the map as well, if is still there, even though we can't
> check the maintenance mode..
>
>> + }
>> + };
>> +
>> + // second check: check maintenance mode mandate
>
> what is a "maintenance mode mandate"? ;)
>
> keeping it simple, why not just
>
> // check if maintenance mode requires closing FDs
>
I see, will rephrase this, thanks!
>> + if datastore_cfg
>> + .get_maintenance_mode()
>> + .is_some_and(|m| m.clear_from_cache())
>> + {
>> DATASTORE_MAP.lock().unwrap().remove(self.name());
>> }
>> }
>> @@ -307,12 +332,12 @@ impl DatastoreThreadSettings {
>> /// - If the cached generation matches the current generation, the
>> /// cached config is returned.
>> /// - Otherwise the config is re-read from disk. If `update_cache` is
>> -/// `true`, the new config and current generation are stored in the
>> +/// `true`, the new config and bumped generation are stored in the
>> /// cache. Callers that set `update_cache = true` must hold the
>> /// datastore config lock to avoid racing with concurrent config
>> /// changes.
>> /// - If `update_cache` is `false`, the freshly read config is returned
>> -/// but the cache is left unchanged.
>> +/// but the cache and generation are left unchanged.
>> ///
>> /// If `ConfigVersionCache` is not available, the config is always read
>> /// from disk and `None` is returned as the generation.
>> @@ -333,14 +358,23 @@ fn datastore_section_config_cached(
>
> does this part here make any sense in this patch?
>
> we don't check the generation in the Drop handler anyway, so it will get
> the latest cached version, no matter what?
>
we don't check the generation in the Drop handler, but the drop handler
depends on this to potentially get a most fresh cached version?
> we'd only end up in this part of the code via lookup_datastore, and only
> if:
> - the previous cached entry and the current one have a different
> generation -> no need to bump again, the cache is already invalidated
> - there is no previous cached entry -> nothing to invalidate
>
> I think this part should move to the next patch..
Shouldn't it be rather in PATCH 2 then, instead part of the TTL feature
Also I would adjust the comment below then, so that it doesn't
necessarily just benefit the drop handler that calls
datastore_section_config_cached(false) but would in general future uses
of datastore_section_config_cached(false)?
>
>> let (config_raw, _digest) = pbs_config::datastore::config()?;
>> let config = Arc::new(config_raw);
>>
>> + let mut effective_gen = current_gen;
>> if update_cache {
>> + // Bump the generation. This ensures that Drop
>> + // handlers will detect that a newer config exists
>> + // and will not rely on a stale cached entry for
>> + // maintenance mandate.
>> + let prev_gen = version_cache.increase_datastore_generation();
>> + effective_gen = prev_gen + 1;
>> +
>> + // Persist
>> *config_cache = Some(DatastoreConfigCache {
>> config: config.clone(),
>> - last_generation: current_gen,
>> + last_generation: effective_gen,
>> });
>> }
>>
>> - Ok((config, Some(current_gen)))
>> + Ok((config, Some(effective_gen)))
>> } else {
>> // Fallback path, no config version cache: read datastore.cfg and return None as generation
>> *config_cache = None;
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 3/4] partial fix #6049: datastore: use config fast-path in Drop
2025-11-28 9:03 6% ` Samuel Rufinatscha
@ 2025-11-28 10:46 5% ` Fabian Grünbichler
2025-11-28 11:10 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-11-28 10:46 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Samuel Rufinatscha
On November 28, 2025 10:03 am, Samuel Rufinatscha wrote:
> On 11/26/25 4:15 PM, Fabian Grünbichler wrote:
>> On November 24, 2025 6:04 pm, Samuel Rufinatscha wrote:
>>> @@ -307,12 +332,12 @@ impl DatastoreThreadSettings {
>>> /// - If the cached generation matches the current generation, the
>>> /// cached config is returned.
>>> /// - Otherwise the config is re-read from disk. If `update_cache` is
>>> -/// `true`, the new config and current generation are stored in the
>>> +/// `true`, the new config and bumped generation are stored in the
>>> /// cache. Callers that set `update_cache = true` must hold the
>>> /// datastore config lock to avoid racing with concurrent config
>>> /// changes.
>>> /// - If `update_cache` is `false`, the freshly read config is returned
>>> -/// but the cache is left unchanged.
>>> +/// but the cache and generation are left unchanged.
>>> ///
>>> /// If `ConfigVersionCache` is not available, the config is always read
>>> /// from disk and `None` is returned as the generation.
>>> @@ -333,14 +358,23 @@ fn datastore_section_config_cached(
>>
>> does this part here make any sense in this patch?
>>
>> we don't check the generation in the Drop handler anyway, so it will get
>> the latest cached version, no matter what?
>>
>
> we don't check the generation in the Drop handler, but the drop handler
> depends on this to potentially get a most fresh cached version?
datastore_section_config_cached will only reload the config if it was
changed over our API and the generation in the cached entry does no
longer match the current generation number. in that case there is no
need to bump the generation number, since that was already done by
whichever call saved the config and caused the generation number
mismatch in the first place - this already invalidated all previously
cached entries..
bumping the generation number only makes sense once we introduce the
force-reload mechanism in patch #4.
>
>> we'd only end up in this part of the code via lookup_datastore, and only
>> if:
>> - the previous cached entry and the current one have a different
>> generation -> no need to bump again, the cache is already invalidated
>> - there is no previous cached entry -> nothing to invalidate
>>
>> I think this part should move to the next patch..
>
> Shouldn't it be rather in PATCH 2 then, instead part of the TTL feature
> Also I would adjust the comment below then, so that it doesn't
> necessarily just benefit the drop handler that calls
> datastore_section_config_cached(false) but would in general future uses
> of datastore_section_config_cached(false)?
it has no benefit at this point in the series (or after/at patch #2),
see above. bumping only makes sense if we detect the generation number
is not valid, which we can only do via the digest check from patch#4.
and the digest check only makes sense with the TTL force-reload, because
else we can never end up in the code path where we read the config
without the cache already being invalid anyway.
>
>>
>>> let (config_raw, _digest) = pbs_config::datastore::config()?;
>>> let config = Arc::new(config_raw);
>>>
>>> + let mut effective_gen = current_gen;
>>> if update_cache {
>>> + // Bump the generation. This ensures that Drop
>>> + // handlers will detect that a newer config exists
>>> + // and will not rely on a stale cached entry for
>>> + // maintenance mandate.
>>> + let prev_gen = version_cache.increase_datastore_generation();
>>> + effective_gen = prev_gen + 1;
>>> +
>>> + // Persist
>>> *config_cache = Some(DatastoreConfigCache {
>>> config: config.clone(),
>>> - last_generation: current_gen,
>>> + last_generation: effective_gen,
>>> });
>>> }
>>>
>>> - Ok((config, Some(current_gen)))
>>> + Ok((config, Some(effective_gen)))
>>> } else {
>>> // Fallback path, no config version cache: read datastore.cfg and return None as generation
>>> *config_cache = None;
>>> --
>>> 2.47.3
>>>
>>>
>>>
>>> _______________________________________________
>>> pbs-devel mailing list
>>> pbs-devel@lists.proxmox.com
>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 3/4] partial fix #6049: datastore: use config fast-path in Drop
2025-11-28 10:46 5% ` Fabian Grünbichler
@ 2025-11-28 11:10 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-11-28 11:10 UTC (permalink / raw)
To: Fabian Grünbichler, Proxmox Backup Server development discussion
On 11/28/25 11:46 AM, Fabian Grünbichler wrote:
> On November 28, 2025 10:03 am, Samuel Rufinatscha wrote:
>> On 11/26/25 4:15 PM, Fabian Grünbichler wrote:
>>> On November 24, 2025 6:04 pm, Samuel Rufinatscha wrote:
>>>> @@ -307,12 +332,12 @@ impl DatastoreThreadSettings {
>>>> /// - If the cached generation matches the current generation, the
>>>> /// cached config is returned.
>>>> /// - Otherwise the config is re-read from disk. If `update_cache` is
>>>> -/// `true`, the new config and current generation are stored in the
>>>> +/// `true`, the new config and bumped generation are stored in the
>>>> /// cache. Callers that set `update_cache = true` must hold the
>>>> /// datastore config lock to avoid racing with concurrent config
>>>> /// changes.
>>>> /// - If `update_cache` is `false`, the freshly read config is returned
>>>> -/// but the cache is left unchanged.
>>>> +/// but the cache and generation are left unchanged.
>>>> ///
>>>> /// If `ConfigVersionCache` is not available, the config is always read
>>>> /// from disk and `None` is returned as the generation.
>>>> @@ -333,14 +358,23 @@ fn datastore_section_config_cached(
>>>
>>> does this part here make any sense in this patch?
>>>
>>> we don't check the generation in the Drop handler anyway, so it will get
>>> the latest cached version, no matter what?
>>>
>>
>> we don't check the generation in the Drop handler, but the drop handler
>> depends on this to potentially get a most fresh cached version?
>
> datastore_section_config_cached will only reload the config if it was
> changed over our API and the generation in the cached entry does no
> longer match the current generation number. in that case there is no
> need to bump the generation number, since that was already done by
> whichever call saved the config and caused the generation number
> mismatch in the first place - this already invalidated all previously
> cached entries..
>
> bumping the generation number only makes sense once we introduce the
> force-reload mechanism in patch #4.
>
>>
>>> we'd only end up in this part of the code via lookup_datastore, and only
>>> if:
>>> - the previous cached entry and the current one have a different
>>> generation -> no need to bump again, the cache is already invalidated
>>> - there is no previous cached entry -> nothing to invalidate
>>>
>>> I think this part should move to the next patch..
>>
>> Shouldn't it be rather in PATCH 2 then, instead part of the TTL feature
>> Also I would adjust the comment below then, so that it doesn't
>> necessarily just benefit the drop handler that calls
>> datastore_section_config_cached(false) but would in general future uses
>> of datastore_section_config_cached(false)?
>
> it has no benefit at this point in the series (or after/at patch #2),
> see above. bumping only makes sense if we detect the generation number
> is not valid, which we can only do via the digest check from patch#4.
> and the digest check only makes sense with the TTL force-reload, because
> else we can never end up in the code path where we read the config
> without the cache already being invalid anyway.
>
Makes sense, I see. Thanks for clarifying Fabian!
Will add it to patch 4.
>>
>>>
>>>> let (config_raw, _digest) = pbs_config::datastore::config()?;
>>>> let config = Arc::new(config_raw);
>>>>
>>>> + let mut effective_gen = current_gen;
>>>> if update_cache {
>>>> + // Bump the generation. This ensures that Drop
>>>> + // handlers will detect that a newer config exists
>>>> + // and will not rely on a stale cached entry for
>>>> + // maintenance mandate.
>>>> + let prev_gen = version_cache.increase_datastore_generation();
>>>> + effective_gen = prev_gen + 1;
>>>> +
>>>> + // Persist
>>>> *config_cache = Some(DatastoreConfigCache {
>>>> config: config.clone(),
>>>> - last_generation: current_gen,
>>>> + last_generation: effective_gen,
>>>> });
>>>> }
>>>>
>>>> - Ok((config, Some(current_gen)))
>>>> + Ok((config, Some(effective_gen)))
>>>> } else {
>>>> // Fallback path, no config version cache: read datastore.cfg and return None as generation
>>>> *config_cache = None;
>>>> --
>>>> 2.47.3
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> pbs-devel mailing list
>>>> pbs-devel@lists.proxmox.com
>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>>>
>>>
>>>
>>> _______________________________________________
>>> pbs-devel mailing list
>>> pbs-devel@lists.proxmox.com
>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* [pbs-devel] [PATCH proxmox-backup 1/4] acme: include proxmox-acme-api dependency
2025-12-02 15:56 12% [pbs-devel] [PATCH proxmox{-backup, } 0/8] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
@ 2025-12-02 15:56 15% ` Samuel Rufinatscha
2025-12-02 15:56 6% ` [pbs-devel] [PATCH proxmox-backup 2/4] acme: drop local AcmeClient Samuel Rufinatscha
` (7 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-02 15:56 UTC (permalink / raw)
To: pbs-devel
PBS currently uses its own ACME client and API logic, while PDM uses the
factored out proxmox-acme and proxmox-acme-api crates. This duplication
risks differences in behaviour and requires ACME maintenance in two
places. This patch is part of a series to move PBS over to the shared
ACME stack.
Changes:
- Add proxmox-acme-api with the "impl" feature as a dependency.
- Initialize proxmox_acme_api in proxmox-backup- api, manager and proxy.
* Inits PBS config dir /acme as proxmox ACME directory
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Cargo.toml | 3 +++
src/bin/proxmox-backup-api.rs | 2 ++
src/bin/proxmox-backup-manager.rs | 2 ++
src/bin/proxmox-backup-proxy.rs | 1 +
4 files changed, 8 insertions(+)
diff --git a/Cargo.toml b/Cargo.toml
index ff143932..bdaf7d85 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -101,6 +101,7 @@ pbs-api-types = "1.0.8"
# other proxmox crates
pathpatterns = "1"
proxmox-acme = "1"
+proxmox-acme-api = { version = "1", features = [ "impl" ] }
pxar = "1"
# PBS workspace
@@ -251,6 +252,7 @@ pbs-api-types.workspace = true
# in their respective repo
proxmox-acme.workspace = true
+proxmox-acme-api.workspace = true
pxar.workspace = true
# proxmox-backup workspace/internal crates
@@ -269,6 +271,7 @@ proxmox-rrd-api-types.workspace = true
[patch.crates-io]
#pbs-api-types = { path = "../proxmox/pbs-api-types" }
#proxmox-acme = { path = "../proxmox/proxmox-acme" }
+#proxmox-acme-api = { path = "../proxmox/proxmox-acme-api" }
#proxmox-api-macro = { path = "../proxmox/proxmox-api-macro" }
#proxmox-apt = { path = "../proxmox/proxmox-apt" }
#proxmox-apt-api-types = { path = "../proxmox/proxmox-apt-api-types" }
diff --git a/src/bin/proxmox-backup-api.rs b/src/bin/proxmox-backup-api.rs
index 417e9e97..48f10092 100644
--- a/src/bin/proxmox-backup-api.rs
+++ b/src/bin/proxmox-backup-api.rs
@@ -8,6 +8,7 @@ use hyper_util::server::graceful::GracefulShutdown;
use tokio::net::TcpListener;
use tracing::level_filters::LevelFilter;
+use pbs_buildcfg::configdir;
use proxmox_http::Body;
use proxmox_lang::try_block;
use proxmox_rest_server::{ApiConfig, RestServer};
@@ -78,6 +79,7 @@ async fn run() -> Result<(), Error> {
let mut command_sock = proxmox_daemon::command_socket::CommandSocket::new(backup_user.gid);
proxmox_product_config::init(backup_user.clone(), pbs_config::priv_user()?);
+ proxmox_acme_api::init(configdir!("/acme"), true)?;
let dir_opts = CreateOptions::new()
.owner(backup_user.uid)
diff --git a/src/bin/proxmox-backup-manager.rs b/src/bin/proxmox-backup-manager.rs
index d9f41353..0facb76c 100644
--- a/src/bin/proxmox-backup-manager.rs
+++ b/src/bin/proxmox-backup-manager.rs
@@ -18,6 +18,7 @@ use pbs_api_types::{
VERIFICATION_OUTDATED_AFTER_SCHEMA, VERIFY_JOB_READ_THREADS_SCHEMA,
VERIFY_JOB_VERIFY_THREADS_SCHEMA,
};
+use pbs_buildcfg::configdir;
use pbs_client::{display_task_log, view_task_result};
use pbs_config::sync;
use pbs_tools::json::required_string_param;
@@ -669,6 +670,7 @@ async fn run() -> Result<(), Error> {
.init()?;
proxmox_backup::server::notifications::init()?;
proxmox_product_config::init(pbs_config::backup_user()?, pbs_config::priv_user()?);
+ proxmox_acme_api::init(configdir!("/acme"), false)?;
let cmd_def = CliCommandMap::new()
.insert("acl", acl_commands())
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 92a8cb3c..0bab18ec 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -190,6 +190,7 @@ async fn run() -> Result<(), Error> {
proxmox_backup::server::notifications::init()?;
metric_collection::init()?;
proxmox_product_config::init(pbs_config::backup_user()?, pbs_config::priv_user()?);
+ proxmox_acme_api::init(configdir!("/acme"), false)?;
let mut indexpath = PathBuf::from(pbs_buildcfg::JS_DIR);
indexpath.push("index.hbs");
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox-backup 3/4] acme: change API impls to use proxmox-acme-api handlers
2025-12-02 15:56 12% [pbs-devel] [PATCH proxmox{-backup, } 0/8] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2025-12-02 15:56 15% ` [pbs-devel] [PATCH proxmox-backup 1/4] acme: include proxmox-acme-api dependency Samuel Rufinatscha
2025-12-02 15:56 6% ` [pbs-devel] [PATCH proxmox-backup 2/4] acme: drop local AcmeClient Samuel Rufinatscha
@ 2025-12-02 15:56 8% ` Samuel Rufinatscha
2025-12-02 15:56 7% ` [pbs-devel] [PATCH proxmox-backup 4/4] acme: certificate ordering through proxmox-acme-api Samuel Rufinatscha
` (5 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-02 15:56 UTC (permalink / raw)
To: pbs-devel
PBS currently uses its own ACME client and API logic, while PDM uses the
factored out proxmox-acme and proxmox-acme-api crates. This duplication
risks differences in behaviour and requires ACME maintenance in two
places. This patch is part of a series to move PBS over to the shared
ACME stack.
Changes:
- Replace api2/config/acme.rs API logic with proxmox-acme-api handlers.
- Drop local caching and helper types that duplicate proxmox-acme-api.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/api2/config/acme.rs | 385 ++-----------------------
src/api2/types/acme.rs | 16 -
src/bin/proxmox_backup_manager/acme.rs | 6 +-
src/config/acme/mod.rs | 44 +--
4 files changed, 35 insertions(+), 416 deletions(-)
diff --git a/src/api2/config/acme.rs b/src/api2/config/acme.rs
index 02f88e2e..a112c8ee 100644
--- a/src/api2/config/acme.rs
+++ b/src/api2/config/acme.rs
@@ -1,31 +1,17 @@
-use std::fs;
-use std::ops::ControlFlow;
-use std::path::Path;
-use std::sync::{Arc, LazyLock, Mutex};
-use std::time::SystemTime;
-
-use anyhow::{bail, format_err, Error};
-use hex::FromHex;
-use serde::{Deserialize, Serialize};
-use serde_json::{json, Value};
-use tracing::{info, warn};
-
-use proxmox_router::{
- http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
-};
-use proxmox_schema::{api, param_bail};
-
-use proxmox_acme::types::AccountData as AcmeAccountData;
-
+use anyhow::Error;
use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
-
-use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
-use crate::config::acme::plugin::{
- self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
+use proxmox_acme_api::{
+ AccountEntry, AccountInfo, AcmeAccountName, AcmeChallengeSchema, ChallengeSchemaWrapper,
+ DeletablePluginProperty, DnsPluginCore, DnsPluginCoreUpdater, KnownAcmeDirectory, PluginConfig,
+ DEFAULT_ACME_DIRECTORY_ENTRY, PLUGIN_ID_SCHEMA,
};
-use proxmox_acme::async_client::AcmeClient;
-use proxmox_acme_api::AcmeAccountName;
+use proxmox_config_digest::ConfigDigest;
use proxmox_rest_server::WorkerTask;
+use proxmox_router::{
+ http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
+};
+use proxmox_schema::api;
+use tracing::info;
pub(crate) const ROUTER: Router = Router::new()
.get(&list_subdirs_api_method!(SUBDIRS))
@@ -67,19 +53,6 @@ const PLUGIN_ITEM_ROUTER: Router = Router::new()
.put(&API_METHOD_UPDATE_PLUGIN)
.delete(&API_METHOD_DELETE_PLUGIN);
-#[api(
- properties: {
- name: { type: AcmeAccountName },
- },
-)]
-/// An ACME Account entry.
-///
-/// Currently only contains a 'name' property.
-#[derive(Serialize)]
-pub struct AccountEntry {
- name: AcmeAccountName,
-}
-
#[api(
access: {
permission: &Permission::Privilege(&["system", "certificates"], PRIV_SYS_MODIFY, false),
@@ -93,40 +66,7 @@ pub struct AccountEntry {
)]
/// List ACME accounts.
pub fn list_accounts() -> Result<Vec<AccountEntry>, Error> {
- let mut entries = Vec::new();
- crate::config::acme::foreach_acme_account(|name| {
- entries.push(AccountEntry { name });
- ControlFlow::Continue(())
- })?;
- Ok(entries)
-}
-
-#[api(
- properties: {
- account: { type: Object, properties: {}, additional_properties: true },
- tos: {
- type: String,
- optional: true,
- },
- },
-)]
-/// ACME Account information.
-///
-/// This is what we return via the API.
-#[derive(Serialize)]
-pub struct AccountInfo {
- /// Raw account data.
- account: AcmeAccountData,
-
- /// The ACME directory URL the account was created at.
- directory: String,
-
- /// The account's own URL within the ACME directory.
- location: String,
-
- /// The ToS URL, if the user agreed to one.
- #[serde(skip_serializing_if = "Option::is_none")]
- tos: Option<String>,
+ proxmox_acme_api::list_accounts()
}
#[api(
@@ -143,23 +83,7 @@ pub struct AccountInfo {
)]
/// Return existing ACME account information.
pub async fn get_account(name: AcmeAccountName) -> Result<AccountInfo, Error> {
- let account_info = proxmox_acme_api::get_account(name).await?;
-
- Ok(AccountInfo {
- location: account_info.location,
- tos: account_info.tos,
- directory: account_info.directory,
- account: AcmeAccountData {
- only_return_existing: false, // don't actually write this out in case it's set
- ..account_info.account
- },
- })
-}
-
-fn account_contact_from_string(s: &str) -> Vec<String> {
- s.split(&[' ', ';', ',', '\0'][..])
- .map(|s| format!("mailto:{s}"))
- .collect()
+ proxmox_acme_api::get_account(name).await
}
#[api(
@@ -224,15 +148,11 @@ fn register_account(
);
}
- if Path::new(&crate::config::acme::account_path(&name)).exists() {
+ if std::path::Path::new(&proxmox_acme_api::account_config_filename(&name)).exists() {
http_bail!(BAD_REQUEST, "account {} already exists", name);
}
- let directory = directory.unwrap_or_else(|| {
- crate::config::acme::DEFAULT_ACME_DIRECTORY_ENTRY
- .url
- .to_owned()
- });
+ let directory = directory.unwrap_or_else(|| DEFAULT_ACME_DIRECTORY_ENTRY.url.to_string());
WorkerTask::spawn(
"acme-register",
@@ -288,17 +208,7 @@ pub fn update_account(
auth_id.to_string(),
true,
move |_worker| async move {
- let data = match contact {
- Some(data) => json!({
- "contact": account_contact_from_string(&data),
- }),
- None => json!({}),
- };
-
- proxmox_acme_api::load_client_with_account(&name)
- .await?
- .update_account(&data)
- .await?;
+ proxmox_acme_api::update_account(&name, contact).await?;
Ok(())
},
@@ -336,18 +246,8 @@ pub fn deactivate_account(
auth_id.to_string(),
true,
move |_worker| async move {
- match proxmox_acme_api::load_client_with_account(&name)
- .await?
- .update_account(&json!({"status": "deactivated"}))
- .await
- {
- Ok(_account) => (),
- Err(err) if !force => return Err(err),
- Err(err) => {
- warn!("error deactivating account {name}, proceeding anyway - {err}");
- }
- }
- crate::config::acme::mark_account_deactivated(&name)?;
+ proxmox_acme_api::deactivate_account(&name, force).await?;
+
Ok(())
},
)
@@ -374,15 +274,7 @@ pub fn deactivate_account(
)]
/// Get the Terms of Service URL for an ACME directory.
async fn get_tos(directory: Option<String>) -> Result<Option<String>, Error> {
- let directory = directory.unwrap_or_else(|| {
- crate::config::acme::DEFAULT_ACME_DIRECTORY_ENTRY
- .url
- .to_owned()
- });
- Ok(AcmeClient::new(directory)
- .terms_of_service_url()
- .await?
- .map(str::to_owned))
+ proxmox_acme_api::get_tos(directory).await
}
#[api(
@@ -397,52 +289,7 @@ async fn get_tos(directory: Option<String>) -> Result<Option<String>, Error> {
)]
/// Get named known ACME directory endpoints.
fn get_directories() -> Result<&'static [KnownAcmeDirectory], Error> {
- Ok(crate::config::acme::KNOWN_ACME_DIRECTORIES)
-}
-
-/// Wrapper for efficient Arc use when returning the ACME challenge-plugin schema for serializing
-struct ChallengeSchemaWrapper {
- inner: Arc<Vec<AcmeChallengeSchema>>,
-}
-
-impl Serialize for ChallengeSchemaWrapper {
- fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
- where
- S: serde::Serializer,
- {
- self.inner.serialize(serializer)
- }
-}
-
-struct CachedSchema {
- schema: Arc<Vec<AcmeChallengeSchema>>,
- cached_mtime: SystemTime,
-}
-
-fn get_cached_challenge_schemas() -> Result<ChallengeSchemaWrapper, Error> {
- static CACHE: LazyLock<Mutex<Option<CachedSchema>>> = LazyLock::new(|| Mutex::new(None));
-
- // the actual loading code
- let mut last = CACHE.lock().unwrap();
-
- let actual_mtime = fs::metadata(crate::config::acme::ACME_DNS_SCHEMA_FN)?.modified()?;
-
- let schema = match &*last {
- Some(CachedSchema {
- schema,
- cached_mtime,
- }) if *cached_mtime >= actual_mtime => schema.clone(),
- _ => {
- let new_schema = Arc::new(crate::config::acme::load_dns_challenge_schema()?);
- *last = Some(CachedSchema {
- schema: Arc::clone(&new_schema),
- cached_mtime: actual_mtime,
- });
- new_schema
- }
- };
-
- Ok(ChallengeSchemaWrapper { inner: schema })
+ Ok(proxmox_acme_api::KNOWN_ACME_DIRECTORIES)
}
#[api(
@@ -457,69 +304,7 @@ fn get_cached_challenge_schemas() -> Result<ChallengeSchemaWrapper, Error> {
)]
/// Get named known ACME directory endpoints.
fn get_challenge_schema() -> Result<ChallengeSchemaWrapper, Error> {
- get_cached_challenge_schemas()
-}
-
-#[api]
-#[derive(Default, Deserialize, Serialize)]
-#[serde(rename_all = "kebab-case")]
-/// The API's format is inherited from PVE/PMG:
-pub struct PluginConfig {
- /// Plugin ID.
- plugin: String,
-
- /// Plugin type.
- #[serde(rename = "type")]
- ty: String,
-
- /// DNS Api name.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- api: Option<String>,
-
- /// Plugin configuration data.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- data: Option<String>,
-
- /// Extra delay in seconds to wait before requesting validation.
- ///
- /// Allows to cope with long TTL of DNS records.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- validation_delay: Option<u32>,
-
- /// Flag to disable the config.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- disable: Option<bool>,
-}
-
-// See PMG/PVE's $modify_cfg_for_api sub
-fn modify_cfg_for_api(id: &str, ty: &str, data: &Value) -> PluginConfig {
- let mut entry = data.clone();
-
- let obj = entry.as_object_mut().unwrap();
- obj.remove("id");
- obj.insert("plugin".to_string(), Value::String(id.to_owned()));
- obj.insert("type".to_string(), Value::String(ty.to_owned()));
-
- // FIXME: This needs to go once the `Updater` is fixed.
- // None of these should be able to fail unless the user changed the files by hand, in which
- // case we leave the unmodified string in the Value for now. This will be handled with an error
- // later.
- if let Some(Value::String(ref mut data)) = obj.get_mut("data") {
- if let Ok(new) = proxmox_base64::url::decode_no_pad(&data) {
- if let Ok(utf8) = String::from_utf8(new) {
- *data = utf8;
- }
- }
- }
-
- // PVE/PMG do this explicitly for ACME plugins...
- // obj.insert("digest".to_string(), Value::String(digest.clone()));
-
- serde_json::from_value(entry).unwrap_or_else(|_| PluginConfig {
- plugin: "*Error*".to_string(),
- ty: "*Error*".to_string(),
- ..Default::default()
- })
+ proxmox_acme_api::get_cached_challenge_schemas()
}
#[api(
@@ -535,12 +320,7 @@ fn modify_cfg_for_api(id: &str, ty: &str, data: &Value) -> PluginConfig {
)]
/// List ACME challenge plugins.
pub fn list_plugins(rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<PluginConfig>, Error> {
- let (plugins, digest) = plugin::config()?;
- rpcenv["digest"] = hex::encode(digest).into();
- Ok(plugins
- .iter()
- .map(|(id, (ty, data))| modify_cfg_for_api(id, ty, data))
- .collect())
+ proxmox_acme_api::list_plugins(rpcenv)
}
#[api(
@@ -557,13 +337,7 @@ pub fn list_plugins(rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<PluginConfig>
)]
/// List ACME challenge plugins.
pub fn get_plugin(id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<PluginConfig, Error> {
- let (plugins, digest) = plugin::config()?;
- rpcenv["digest"] = hex::encode(digest).into();
-
- match plugins.get(&id) {
- Some((ty, data)) => Ok(modify_cfg_for_api(&id, ty, data)),
- None => http_bail!(NOT_FOUND, "no such plugin"),
- }
+ proxmox_acme_api::get_plugin(id, rpcenv)
}
// Currently we only have "the" standalone plugin and DNS plugins so we can just flatten a
@@ -595,30 +369,7 @@ pub fn get_plugin(id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<PluginC
)]
/// Add ACME plugin configuration.
pub fn add_plugin(r#type: String, core: DnsPluginCore, data: String) -> Result<(), Error> {
- // Currently we only support DNS plugins and the standalone plugin is "fixed":
- if r#type != "dns" {
- param_bail!("type", "invalid ACME plugin type: {:?}", r#type);
- }
-
- let data = String::from_utf8(proxmox_base64::decode(data)?)
- .map_err(|_| format_err!("data must be valid UTF-8"))?;
-
- let id = core.id.clone();
-
- let _lock = plugin::lock()?;
-
- let (mut plugins, _digest) = plugin::config()?;
- if plugins.contains_key(&id) {
- param_bail!("id", "ACME plugin ID {:?} already exists", id);
- }
-
- let plugin = serde_json::to_value(DnsPlugin { core, data })?;
-
- plugins.insert(id, r#type, plugin);
-
- plugin::save_config(&plugins)?;
-
- Ok(())
+ proxmox_acme_api::add_plugin(r#type, core, data)
}
#[api(
@@ -634,26 +385,7 @@ pub fn add_plugin(r#type: String, core: DnsPluginCore, data: String) -> Result<(
)]
/// Delete an ACME plugin configuration.
pub fn delete_plugin(id: String) -> Result<(), Error> {
- let _lock = plugin::lock()?;
-
- let (mut plugins, _digest) = plugin::config()?;
- if plugins.remove(&id).is_none() {
- http_bail!(NOT_FOUND, "no such plugin");
- }
- plugin::save_config(&plugins)?;
-
- Ok(())
-}
-
-#[api()]
-#[derive(Serialize, Deserialize)]
-#[serde(rename_all = "kebab-case")]
-/// Deletable property name
-pub enum DeletableProperty {
- /// Delete the disable property
- Disable,
- /// Delete the validation-delay property
- ValidationDelay,
+ proxmox_acme_api::delete_plugin(id)
}
#[api(
@@ -675,12 +407,12 @@ pub enum DeletableProperty {
type: Array,
optional: true,
items: {
- type: DeletableProperty,
+ type: DeletablePluginProperty,
}
},
digest: {
- description: "Digest to protect against concurrent updates",
optional: true,
+ type: ConfigDigest,
},
},
},
@@ -694,65 +426,8 @@ pub fn update_plugin(
id: String,
update: DnsPluginCoreUpdater,
data: Option<String>,
- delete: Option<Vec<DeletableProperty>>,
- digest: Option<String>,
+ delete: Option<Vec<DeletablePluginProperty>>,
+ digest: Option<ConfigDigest>,
) -> Result<(), Error> {
- let data = data
- .as_deref()
- .map(proxmox_base64::decode)
- .transpose()?
- .map(String::from_utf8)
- .transpose()
- .map_err(|_| format_err!("data must be valid UTF-8"))?;
-
- let _lock = plugin::lock()?;
-
- let (mut plugins, expected_digest) = plugin::config()?;
-
- if let Some(digest) = digest {
- let digest = <[u8; 32]>::from_hex(digest)?;
- crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
- }
-
- match plugins.get_mut(&id) {
- Some((ty, ref mut entry)) => {
- if ty != "dns" {
- bail!("cannot update plugin of type {:?}", ty);
- }
-
- let mut plugin = DnsPlugin::deserialize(&*entry)?;
-
- if let Some(delete) = delete {
- for delete_prop in delete {
- match delete_prop {
- DeletableProperty::ValidationDelay => {
- plugin.core.validation_delay = None;
- }
- DeletableProperty::Disable => {
- plugin.core.disable = None;
- }
- }
- }
- }
- if let Some(data) = data {
- plugin.data = data;
- }
- if let Some(api) = update.api {
- plugin.core.api = api;
- }
- if update.validation_delay.is_some() {
- plugin.core.validation_delay = update.validation_delay;
- }
- if update.disable.is_some() {
- plugin.core.disable = update.disable;
- }
-
- *entry = serde_json::to_value(plugin)?;
- }
- None => http_bail!(NOT_FOUND, "no such plugin"),
- }
-
- plugin::save_config(&plugins)?;
-
- Ok(())
+ proxmox_acme_api::update_plugin(id, update, data, delete, digest)
}
diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
index 7c9063c0..2905b41b 100644
--- a/src/api2/types/acme.rs
+++ b/src/api2/types/acme.rs
@@ -44,22 +44,6 @@ pub const ACME_DOMAIN_PROPERTY_SCHEMA: Schema =
.format(&ApiStringFormat::PropertyString(&AcmeDomain::API_SCHEMA))
.schema();
-#[api(
- properties: {
- name: { type: String },
- url: { type: String },
- },
-)]
-/// An ACME directory endpoint with a name and URL.
-#[derive(Serialize)]
-pub struct KnownAcmeDirectory {
- /// The ACME directory's name.
- pub name: &'static str,
-
- /// The ACME directory's endpoint URL.
- pub url: &'static str,
-}
-
#[api(
properties: {
schema: {
diff --git a/src/bin/proxmox_backup_manager/acme.rs b/src/bin/proxmox_backup_manager/acme.rs
index bb987b26..e7bd67af 100644
--- a/src/bin/proxmox_backup_manager/acme.rs
+++ b/src/bin/proxmox_backup_manager/acme.rs
@@ -8,10 +8,8 @@ use proxmox_schema::api;
use proxmox_sys::fs::file_get_contents;
use proxmox_acme::async_client::AcmeClient;
-use proxmox_acme_api::AcmeAccountName;
+use proxmox_acme_api::{AcmeAccountName, DnsPluginCore, KNOWN_ACME_DIRECTORIES};
use proxmox_backup::api2;
-use proxmox_backup::config::acme::plugin::DnsPluginCore;
-use proxmox_backup::config::acme::KNOWN_ACME_DIRECTORIES;
pub fn acme_mgmt_cli() -> CommandLineInterface {
let cmd_def = CliCommandMap::new()
@@ -122,7 +120,7 @@ async fn register_account(
match input.trim().parse::<usize>() {
Ok(n) if n < KNOWN_ACME_DIRECTORIES.len() => {
- break (KNOWN_ACME_DIRECTORIES[n].url.to_owned(), false);
+ break (KNOWN_ACME_DIRECTORIES[n].url.to_string(), false);
}
Ok(n) if n == KNOWN_ACME_DIRECTORIES.len() => {
input.clear();
diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
index d31b2bc9..35cda50b 100644
--- a/src/config/acme/mod.rs
+++ b/src/config/acme/mod.rs
@@ -1,8 +1,7 @@
use std::collections::HashMap;
use std::ops::ControlFlow;
-use std::path::Path;
-use anyhow::{bail, format_err, Error};
+use anyhow::Error;
use serde_json::Value;
use proxmox_sys::error::SysError;
@@ -10,8 +9,8 @@ use proxmox_sys::fs::{file_read_string, CreateOptions};
use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
-use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
-use proxmox_acme_api::AcmeAccountName;
+use crate::api2::types::AcmeChallengeSchema;
+use proxmox_acme_api::{AcmeAccountName, KnownAcmeDirectory, KNOWN_ACME_DIRECTORIES};
pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
@@ -36,23 +35,8 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
create_acme_subdir(ACME_DIR)
}
-pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
- KnownAcmeDirectory {
- name: "Let's Encrypt V2",
- url: "https://acme-v02.api.letsencrypt.org/directory",
- },
- KnownAcmeDirectory {
- name: "Let's Encrypt V2 Staging",
- url: "https://acme-staging-v02.api.letsencrypt.org/directory",
- },
-];
-
pub const DEFAULT_ACME_DIRECTORY_ENTRY: &KnownAcmeDirectory = &KNOWN_ACME_DIRECTORIES[0];
-pub fn account_path(name: &str) -> String {
- format!("{ACME_ACCOUNT_DIR}/{name}")
-}
-
pub fn foreach_acme_account<F>(mut func: F) -> Result<(), Error>
where
F: FnMut(AcmeAccountName) -> ControlFlow<Result<(), Error>>,
@@ -83,28 +67,6 @@ where
}
}
-pub fn mark_account_deactivated(name: &str) -> Result<(), Error> {
- let from = account_path(name);
- for i in 0..100 {
- let to = account_path(&format!("_deactivated_{name}_{i}"));
- if !Path::new(&to).exists() {
- return std::fs::rename(&from, &to).map_err(|err| {
- format_err!(
- "failed to move account path {:?} to {:?} - {}",
- from,
- to,
- err
- )
- });
- }
- }
- bail!(
- "No free slot to rename deactivated account {:?}, please cleanup {:?}",
- from,
- ACME_ACCOUNT_DIR
- );
-}
-
pub fn load_dns_challenge_schema() -> Result<Vec<AcmeChallengeSchema>, Error> {
let raw = file_read_string(ACME_DNS_SCHEMA_FN)?;
let schemas: serde_json::Map<String, Value> = serde_json::from_str(&raw)?;
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 8%]
* [pbs-devel] [PATCH proxmox{-backup, } 0/8] fix #6939: acme: support servers returning 204 for nonce requests
@ 2025-12-02 15:56 12% Samuel Rufinatscha
2025-12-02 15:56 15% ` [pbs-devel] [PATCH proxmox-backup 1/4] acme: include proxmox-acme-api dependency Samuel Rufinatscha
` (8 more replies)
0 siblings, 9 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-02 15:56 UTC (permalink / raw)
To: pbs-devel
Hi,
this series fixes account registration for ACME providers that return
HTTP 204 No Content to the newNonce request. Currently, both the PBS
ACME client and the shared ACME client in proxmox-acme only accept
HTTP 200 OK for this request. The issue was observed in PBS against a
custom ACME deployment and reported as bug #6939 [1].
## Problem
During ACME account registration, PBS first fetches an anti-replay
nonce by sending a HEAD request to the CA’s newNonce URL.
RFC 8555 §7.2 [2] states that:
* the server MUST include a Replay-Nonce header with a fresh nonce,
* the server SHOULD use status 200 OK for the HEAD request,
* the server MUST also handle GET on the same resource and may return
204 No Content with an empty body.
The reporter observed the following error message:
*ACME server responded with unexpected status code: 204*
and mentioned that the issue did not appear with PVE 9 [1]. Looking at
PVE’s Perl ACME client [3], it uses a GET request instead of HEAD and
accepts any 2xx success code when retrieving the nonce. This difference
in behavior does not affect functionality but is worth noting for
consistency across implementations.
## Approach
To support ACME providers which return 204 No Content, the Rust ACME
clients in proxmox-backup and proxmox need to treat both 200 OK and 204
No Content as valid responses for the nonce request, as long as a
Replay-Nonce header is present.
This series changes the expected field of the internal Request type
from a single u16 to a list of allowed status codes
(e.g. &'static [u16]), so one request can explicitly accept multiple
success codes.
To avoid fixing the issue twice (once in PBS’ own ACME client and once
in the shared Rust client), this series first refactors PBS to use the
shared AcmeClient from proxmox-acme / proxmox-acme-api, similar to PDM,
and then applies the bug fix in that shared implementation so that all
consumers benefit from the more tolerant behavior.
## Testing
*Testing the refactor*
To test the refactor, I
(1) installed latest stable PBS on a VM
(2) created .deb package from latest PBS (master), containing the
refactor
(3) installed created .deb package
(4) installed Pebble from Let's Encrypt from Let's Encrypt [5] on the
same VM
(5) created an ACME account and ordered the new certificate for the
host domain.
Steps to reproduce:
(1) install latest stable PBS on a VM, created .deb package from latest
PBS (master) containing the refactor, install created .deb package
(2) install Pebble from Let's Encrypt from Let's Encrypt [5] on the
same VM:
cd
apt update
apt install -y golang git
git clone https://github.com/letsencrypt/pebble
cd pebble
go build ./cmd/pebble
then, downloaded and trusted the Pebble cert:
wget https://raw.githubusercontent.com/letsencrypt/pebble/main/test/certs/pebble.minica.pem
cp pebble.minica.pem /usr/local/share/ca-certificates/pebble.minica.crt
update-ca-certificates
We want Pebble to perform HTTP-01 validation against port 80, because
PBS’s standalone plugin will bind port 80. Set httpPort to 80.
nano ./test/config/pebble-config.json
Started the Pebble server in the background:
./pebble -config ./test/config/pebble-config.json &
Created a Pebble ACME account:
proxmox-backup-manager acme account register default admin@example.com \
--directory 'https://127.0.0.1:14000/dir'
To verify persistence of the account I checked
ls /etc/proxmox-backup/acme/accounts
Verified if update-account works
proxmox-backup-manager acme account update default --contact "a@example.com,b@example.com"
proxmox-backup-manager acme account info default
In the PBS GUI, you can create a new domain. You can use your host
domain name (see /etc/hosts). Select the created account and order the
certificate.
After a page reload, you might need to accept the new certificate in the browser.
In the PBS dashboard, you should then see the new Pebble certificate.
*Note: on reboot, the created Pebble ACME account will be gone and you
will need to create a new one. Pebble does not persist account info.
In that case remove your previously created account in
/etc/proxmox-backup/acme/accounts.
*Testing the newNonce fix*
To prove the ACME newNonce fix, I put nginx in front of Pebble, to
intercept the newNonce request in order to return 204 No Content
instead of 200 OK, all other requests are unchanged and forwarded to
Pebble. Requires trusting the nginx CAs via
/usr/local/share/ca-certificates + update-ca-certificates on the VM.
Then I ran following command against nginx:
proxmox-backup-manager acme account register proxytest root@backup.local --directory 'https://nginx-address/dir
The account could be created successfully. When adjusting the nginx
configuration to return any other non-expected success status code,
PBS expectely rejects.
## Patch summary
[PATCH proxmox-backup v4 1/4] acme: include proxmox-acme-api dependency
[PATCH proxmox-backup v4 2/4] acme: drop local AcmeClient
[PATCH proxmox-backup v4 3/4] acme: change API impls to use proxmox-acme-api handlers
[PATCH proxmox-backup v4 4/4] acme: certificate ordering through proxmox-acme-api
[PATCH proxmox v4 1/4] acme: reduce visibility of Request type
[PATCH proxmox v4 2/4] acme: introduce http_status module
[PATCH proxmox v4 3/4] acme-api: add helper to load client for an account
[PATCH proxmox v4 4/4] fix #6939: support servers returning 204 for newNonce
Thanks for considering this patch series, I look forward to your
feedback.
Best,
Samuel Rufinatscha
## Changes from v1:
[PATCH proxmox v2 1/1] fix #6939: support providers returning 204 for nonce
requests
* Introduced `http_success` module to contain the http success codes
* Replaced `Vec<u16>` with `&[u16]` for expected codes to avoid
allocations.
* Clarified the PVEs Perl ACME client behaviour in the commit message.
[PATCH proxmox-backup v2 1/1] acme: accept HTTP 204 from newNonce endpoint
* Integrated the `http_success` module, replacing `Vec<u16>` with `&[u16]`
* Clarified the PVEs Perl ACME client behaviour in the commit message.
## Changes from v2:
[PATCH proxmox v3 1/1] fix #6939: support providers returning 204 for nonce
requests
* Rename `http_success` module to `http_status`
[PATCH proxmox-backup v3 1/1] acme: accept HTTP 204 from newNonce endpoint
* Replace `http_success` usage
## Changes from v3:
Removed: [PATCH proxmox-backup v3 1/1].
Added:
[PATCH proxmox-backup v4 1/4] acme: include proxmox-acme-api dependency
* New: add proxmox-acme-api as a dependency and initialize it in
PBS so PBS can use the shared ACME API instead.
[PATCH proxmox-backup v4 2/4] acme: drop local AcmeClient
* New: remove the PBS-local AcmeClient implementation and switch PBS
over to the shared proxmox-acme async client.
[PATCH proxmox-backup v4 3/4] acme: change API impls to use proxmox-acme-api
handlers
* New: rework PBS’ ACME API endpoints to delegate to
proxmox-acme-api handlers instead of duplicating logic locally.
[PATCH proxmox-backup v4 4/4] acme: certificate ordering through
proxmox-acme-api
* New: move PBS’ ACME certificate ordering logic over to
proxmox-acme-api, keeping only certificate installation/reload in
PBS.
[PATCH proxmox v4 1/4] acme: reduce visibility of Request type
* New: hide the low-level Request type and its fields behind
constructors / reduced visibility so changes to “expected” no longer
affect the public API as they did in v3.
[PATCH proxmox v4 2/4] acme: introduce http_status module
* New: split out the HTTP status constants into an internal
http_status module as a separate preparatory cleanup before the bug
fix, instead of doing this inline like in v3.
[PATCH proxmox v4 3/4] acme-api: add helper to load client for an account
* New: add a load_client_with_account helper in proxmox-acme-api so
PBS (and others) can construct an AcmeClient for a configured account
without duplicating boilerplate.
Changed:
[PATCH proxmox v3 1/1] -> [PATCH proxmox v4 4/4]
fix #6939: acme: support server returning 204 for nonce requests
* Rebased on top of the refactor: keep the same behavioural fix as in v3
(accept 204 for newNonce with Replay-Nonce present), but implement it
on top of the http_status module that is part of the refactor.
proxmox-backup:
Samuel Rufinatscha (4):
acme: include proxmox-acme-api dependency
acme: drop local AcmeClient
acme: change API impls to use proxmox-acme-api handlers
acme: certificate ordering through proxmox-acme-api
Cargo.toml | 3 +
src/acme/client.rs | 691 -------------------------
src/acme/mod.rs | 5 -
src/acme/plugin.rs | 336 ------------
src/api2/config/acme.rs | 407 ++-------------
src/api2/node/certificates.rs | 240 ++-------
src/api2/types/acme.rs | 98 ----
src/api2/types/mod.rs | 3 -
src/bin/proxmox-backup-api.rs | 2 +
src/bin/proxmox-backup-manager.rs | 2 +
src/bin/proxmox-backup-proxy.rs | 1 +
src/bin/proxmox_backup_manager/acme.rs | 21 +-
src/config/acme/mod.rs | 51 +-
src/config/acme/plugin.rs | 99 +---
src/config/node.rs | 29 +-
src/lib.rs | 2 -
16 files changed, 103 insertions(+), 1887 deletions(-)
delete mode 100644 src/acme/client.rs
delete mode 100644 src/acme/mod.rs
delete mode 100644 src/acme/plugin.rs
delete mode 100644 src/api2/types/acme.rs
proxmox:
Samuel Rufinatscha (4):
acme: reduce visibility of Request type
acme: introduce http_status module
acme-api: add helper to load client for an account
fix #6939: acme: support servers returning 204 for nonce requests
proxmox-acme-api/src/account_api_impl.rs | 5 +++++
proxmox-acme-api/src/lib.rs | 3 ++-
proxmox-acme/src/account.rs | 27 +++++++++++++-----------
proxmox-acme/src/async_client.rs | 8 +++----
proxmox-acme/src/authorization.rs | 2 +-
proxmox-acme/src/client.rs | 8 +++----
proxmox-acme/src/lib.rs | 6 ++----
proxmox-acme/src/order.rs | 2 +-
proxmox-acme/src/request.rs | 25 +++++++++++++++-------
9 files changed, 51 insertions(+), 35 deletions(-)
Summary over all repositories:
25 files changed, 154 insertions(+), 1922 deletions(-)
--
Generated by git-murpp 0.8.1
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] [PATCH proxmox-backup 4/4] acme: certificate ordering through proxmox-acme-api
2025-12-02 15:56 12% [pbs-devel] [PATCH proxmox{-backup, } 0/8] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (2 preceding siblings ...)
2025-12-02 15:56 8% ` [pbs-devel] [PATCH proxmox-backup 3/4] acme: change API impls to use proxmox-acme-api handlers Samuel Rufinatscha
@ 2025-12-02 15:56 7% ` Samuel Rufinatscha
2025-12-02 15:56 12% ` [pbs-devel] [PATCH proxmox 1/4] acme: reduce visibility of Request type Samuel Rufinatscha
` (4 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-02 15:56 UTC (permalink / raw)
To: pbs-devel
PBS currently uses its own ACME client and API logic, while PDM uses the
factored out proxmox-acme and proxmox-acme-api crates. This duplication
risks differences in behaviour and requires ACME maintenance in two
places. This patch is part of a series to move PBS over to the shared
ACME stack.
Changes:
- Replace the custom ACME order/authorization loop in node certificates
with a call to proxmox_acme_api::order_certificate.
- Build domain + config data as proxmox-acme-api types
- Remove obsolete local ACME ordering and plugin glue code.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/acme/mod.rs | 2 -
src/acme/plugin.rs | 336 ----------------------------------
src/api2/node/certificates.rs | 240 ++++--------------------
src/api2/types/acme.rs | 74 --------
src/api2/types/mod.rs | 3 -
src/config/acme/mod.rs | 7 +-
src/config/acme/plugin.rs | 99 +---------
src/config/node.rs | 22 +--
src/lib.rs | 2 -
9 files changed, 46 insertions(+), 739 deletions(-)
delete mode 100644 src/acme/mod.rs
delete mode 100644 src/acme/plugin.rs
delete mode 100644 src/api2/types/acme.rs
diff --git a/src/acme/mod.rs b/src/acme/mod.rs
deleted file mode 100644
index cc561f9a..00000000
--- a/src/acme/mod.rs
+++ /dev/null
@@ -1,2 +0,0 @@
-pub(crate) mod plugin;
-pub(crate) use plugin::get_acme_plugin;
diff --git a/src/acme/plugin.rs b/src/acme/plugin.rs
deleted file mode 100644
index 5bc09e1f..00000000
--- a/src/acme/plugin.rs
+++ /dev/null
@@ -1,336 +0,0 @@
-use std::future::Future;
-use std::net::{IpAddr, SocketAddr};
-use std::pin::Pin;
-use std::process::Stdio;
-use std::sync::Arc;
-use std::time::Duration;
-
-use anyhow::{bail, format_err, Error};
-use bytes::Bytes;
-use futures::TryFutureExt;
-use http_body_util::Full;
-use hyper::body::Incoming;
-use hyper::server::conn::http1;
-use hyper::service::service_fn;
-use hyper::{Request, Response};
-use hyper_util::rt::TokioIo;
-use tokio::io::{AsyncBufReadExt, AsyncRead, AsyncWriteExt, BufReader};
-use tokio::net::TcpListener;
-use tokio::process::Command;
-
-use proxmox_acme::{Authorization, Challenge};
-
-use crate::api2::types::AcmeDomain;
-use proxmox_acme::async_client::AcmeClient;
-use proxmox_rest_server::WorkerTask;
-
-use crate::config::acme::plugin::{DnsPlugin, PluginData};
-
-const PROXMOX_ACME_SH_PATH: &str = "/usr/share/proxmox-acme/proxmox-acme";
-
-pub(crate) fn get_acme_plugin(
- plugin_data: &PluginData,
- name: &str,
-) -> Result<Option<Box<dyn AcmePlugin + Send + Sync + 'static>>, Error> {
- let (ty, data) = match plugin_data.get(name) {
- Some(plugin) => plugin,
- None => return Ok(None),
- };
-
- Ok(Some(match ty.as_str() {
- "dns" => {
- let plugin: DnsPlugin = serde::Deserialize::deserialize(data)?;
- Box::new(plugin)
- }
- "standalone" => {
- // this one has no config
- Box::<StandaloneServer>::default()
- }
- other => bail!("missing implementation for plugin type '{}'", other),
- }))
-}
-
-pub(crate) trait AcmePlugin {
- /// Setup everything required to trigger the validation and return the corresponding validation
- /// URL.
- fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- domain: &'d AcmeDomain,
- task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>>;
-
- fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- domain: &'d AcmeDomain,
- task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>>;
-}
-
-fn extract_challenge<'a>(
- authorization: &'a Authorization,
- ty: &str,
-) -> Result<&'a Challenge, Error> {
- authorization
- .challenges
- .iter()
- .find(|ch| ch.ty == ty)
- .ok_or_else(|| format_err!("no supported challenge type ({}) found", ty))
-}
-
-async fn pipe_to_tasklog<T: AsyncRead + Unpin>(
- pipe: T,
- task: Arc<WorkerTask>,
-) -> Result<(), std::io::Error> {
- let mut pipe = BufReader::new(pipe);
- let mut line = String::new();
- loop {
- line.clear();
- match pipe.read_line(&mut line).await {
- Ok(0) => return Ok(()),
- Ok(_) => task.log_message(line.as_str()),
- Err(err) => return Err(err),
- }
- }
-}
-
-impl DnsPlugin {
- async fn action<'a>(
- &self,
- client: &mut AcmeClient,
- authorization: &'a Authorization,
- domain: &AcmeDomain,
- task: Arc<WorkerTask>,
- action: &str,
- ) -> Result<&'a str, Error> {
- let challenge = extract_challenge(authorization, "dns-01")?;
- let mut stdin_data = client
- .dns_01_txt_value(
- challenge
- .token()
- .ok_or_else(|| format_err!("missing token in challenge"))?,
- )?
- .into_bytes();
- stdin_data.push(b'\n');
- stdin_data.extend(self.data.as_bytes());
- if stdin_data.last() != Some(&b'\n') {
- stdin_data.push(b'\n');
- }
-
- let mut command = Command::new("/usr/bin/setpriv");
-
- #[rustfmt::skip]
- command.args([
- "--reuid", "nobody",
- "--regid", "nogroup",
- "--clear-groups",
- "--reset-env",
- "--",
- "/bin/bash",
- PROXMOX_ACME_SH_PATH,
- action,
- &self.core.api,
- domain.alias.as_deref().unwrap_or(&domain.domain),
- ]);
-
- // We could use 1 socketpair, but tokio wraps them all in `File` internally causing `close`
- // to be called separately on all of them without exception, so we need 3 pipes :-(
-
- let mut child = command
- .stdin(Stdio::piped())
- .stdout(Stdio::piped())
- .stderr(Stdio::piped())
- .spawn()?;
-
- let mut stdin = child.stdin.take().expect("Stdio::piped()");
- let stdout = child.stdout.take().expect("Stdio::piped() failed?");
- let stdout = pipe_to_tasklog(stdout, Arc::clone(&task));
- let stderr = child.stderr.take().expect("Stdio::piped() failed?");
- let stderr = pipe_to_tasklog(stderr, Arc::clone(&task));
- let stdin = async move {
- stdin.write_all(&stdin_data).await?;
- stdin.flush().await?;
- Ok::<_, std::io::Error>(())
- };
- match futures::try_join!(stdin, stdout, stderr) {
- Ok(((), (), ())) => (),
- Err(err) => {
- if let Err(err) = child.kill().await {
- task.log_message(format!(
- "failed to kill '{PROXMOX_ACME_SH_PATH} {action}' command: {err}"
- ));
- }
- bail!("'{}' failed: {}", PROXMOX_ACME_SH_PATH, err);
- }
- }
-
- let status = child.wait().await?;
- if !status.success() {
- bail!(
- "'{} {}' exited with error ({})",
- PROXMOX_ACME_SH_PATH,
- action,
- status.code().unwrap_or(-1)
- );
- }
-
- Ok(&challenge.url)
- }
-}
-
-impl AcmePlugin for DnsPlugin {
- fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- domain: &'d AcmeDomain,
- task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>> {
- Box::pin(async move {
- let result = self
- .action(client, authorization, domain, task.clone(), "setup")
- .await;
-
- let validation_delay = self.core.validation_delay.unwrap_or(30) as u64;
- if validation_delay > 0 {
- task.log_message(format!(
- "Sleeping {validation_delay} seconds to wait for TXT record propagation"
- ));
- tokio::time::sleep(Duration::from_secs(validation_delay)).await;
- }
- result
- })
- }
-
- fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- domain: &'d AcmeDomain,
- task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>> {
- Box::pin(async move {
- self.action(client, authorization, domain, task, "teardown")
- .await
- .map(drop)
- })
- }
-}
-
-#[derive(Default)]
-struct StandaloneServer {
- abort_handle: Option<futures::future::AbortHandle>,
-}
-
-// In case the "order_certificates" future gets dropped between setup & teardown, let's also cancel
-// the HTTP listener on Drop:
-impl Drop for StandaloneServer {
- fn drop(&mut self) {
- self.stop();
- }
-}
-
-impl StandaloneServer {
- fn stop(&mut self) {
- if let Some(abort) = self.abort_handle.take() {
- abort.abort();
- }
- }
-}
-
-async fn standalone_respond(
- req: Request<Incoming>,
- path: Arc<String>,
- key_auth: Arc<String>,
-) -> Result<Response<Full<Bytes>>, hyper::Error> {
- if req.method() == hyper::Method::GET && req.uri().path() == path.as_str() {
- Ok(Response::builder()
- .status(hyper::http::StatusCode::OK)
- .body(key_auth.as_bytes().to_vec().into())
- .unwrap())
- } else {
- Ok(Response::builder()
- .status(hyper::http::StatusCode::NOT_FOUND)
- .body("Not found.".into())
- .unwrap())
- }
-}
-
-impl AcmePlugin for StandaloneServer {
- fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- _domain: &'d AcmeDomain,
- _task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>> {
- Box::pin(async move {
- self.stop();
-
- let challenge = extract_challenge(authorization, "http-01")?;
- let token = challenge
- .token()
- .ok_or_else(|| format_err!("missing token in challenge"))?;
- let key_auth = Arc::new(client.key_authorization(token)?);
- let path = Arc::new(format!("/.well-known/acme-challenge/{token}"));
-
- // `[::]:80` first, then `*:80`
- let dual = SocketAddr::new(IpAddr::from([0u16; 8]), 80);
- let ipv4 = SocketAddr::new(IpAddr::from([0u8; 4]), 80);
- let incoming = TcpListener::bind(dual)
- .or_else(|_| TcpListener::bind(ipv4))
- .await?;
-
- let server = async move {
- loop {
- let key_auth = Arc::clone(&key_auth);
- let path = Arc::clone(&path);
- match incoming.accept().await {
- Ok((tcp, _)) => {
- let io = TokioIo::new(tcp);
- let service = service_fn(move |request| {
- standalone_respond(
- request,
- Arc::clone(&path),
- Arc::clone(&key_auth),
- )
- });
-
- tokio::task::spawn(async move {
- if let Err(err) =
- http1::Builder::new().serve_connection(io, service).await
- {
- println!("Error serving connection: {err:?}");
- }
- });
- }
- Err(err) => println!("Error accepting connection: {err:?}"),
- }
- }
- };
- let (future, abort) = futures::future::abortable(server);
- self.abort_handle = Some(abort);
- tokio::spawn(future);
-
- Ok(challenge.url.as_str())
- })
- }
-
- fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- _client: &'b mut AcmeClient,
- _authorization: &'c Authorization,
- _domain: &'d AcmeDomain,
- _task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>> {
- Box::pin(async move {
- if let Some(abort) = self.abort_handle.take() {
- abort.abort();
- }
- Ok(())
- })
- }
-}
diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.rs
index 31196715..2a645b4a 100644
--- a/src/api2/node/certificates.rs
+++ b/src/api2/node/certificates.rs
@@ -1,27 +1,19 @@
-use std::sync::Arc;
-use std::time::Duration;
-
use anyhow::{bail, format_err, Error};
use openssl::pkey::PKey;
use openssl::x509::X509;
use serde::{Deserialize, Serialize};
use tracing::info;
-use proxmox_router::list_subdirs_api_method;
-use proxmox_router::SubdirMap;
-use proxmox_router::{Permission, Router, RpcEnvironment};
-use proxmox_schema::api;
-
+use crate::server::send_certificate_renewal_mail;
use pbs_api_types::{NODE_SCHEMA, PRIV_SYS_MODIFY};
use pbs_buildcfg::configdir;
use pbs_tools::cert;
-use tracing::warn;
-
-use crate::api2::types::AcmeDomain;
-use crate::config::node::NodeConfig;
-use crate::server::send_certificate_renewal_mail;
-use proxmox_acme::async_client::AcmeClient;
+use proxmox_acme_api::AcmeDomain;
use proxmox_rest_server::WorkerTask;
+use proxmox_router::list_subdirs_api_method;
+use proxmox_router::SubdirMap;
+use proxmox_router::{Permission, Router, RpcEnvironment};
+use proxmox_schema::api;
pub const ROUTER: Router = Router::new()
.get(&list_subdirs_api_method!(SUBDIRS))
@@ -269,193 +261,6 @@ pub async fn delete_custom_certificate() -> Result<(), Error> {
Ok(())
}
-struct OrderedCertificate {
- certificate: hyper::body::Bytes,
- private_key_pem: Vec<u8>,
-}
-
-async fn order_certificate(
- worker: Arc<WorkerTask>,
- node_config: &NodeConfig,
-) -> Result<Option<OrderedCertificate>, Error> {
- use proxmox_acme::authorization::Status;
- use proxmox_acme::order::Identifier;
-
- let domains = node_config.acme_domains().try_fold(
- Vec::<AcmeDomain>::new(),
- |mut acc, domain| -> Result<_, Error> {
- let mut domain = domain?;
- domain.domain.make_ascii_lowercase();
- if let Some(alias) = &mut domain.alias {
- alias.make_ascii_lowercase();
- }
- acc.push(domain);
- Ok(acc)
- },
- )?;
-
- let get_domain_config = |domain: &str| {
- domains
- .iter()
- .find(|d| d.domain == domain)
- .ok_or_else(|| format_err!("no config for domain '{}'", domain))
- };
-
- if domains.is_empty() {
- info!("No domains configured to be ordered from an ACME server.");
- return Ok(None);
- }
-
- let (plugins, _) = crate::config::acme::plugin::config()?;
-
- let mut acme = node_config.acme_client().await?;
-
- info!("Placing ACME order");
- let order = acme
- .new_order(domains.iter().map(|d| d.domain.to_ascii_lowercase()))
- .await?;
- info!("Order URL: {}", order.location);
-
- let identifiers: Vec<String> = order
- .data
- .identifiers
- .iter()
- .map(|identifier| match identifier {
- Identifier::Dns(domain) => domain.clone(),
- })
- .collect();
-
- for auth_url in &order.data.authorizations {
- info!("Getting authorization details from '{auth_url}'");
- let mut auth = acme.get_authorization(auth_url).await?;
-
- let domain = match &mut auth.identifier {
- Identifier::Dns(domain) => domain.to_ascii_lowercase(),
- };
-
- if auth.status == Status::Valid {
- info!("{domain} is already validated!");
- continue;
- }
-
- info!("The validation for {domain} is pending");
- let domain_config: &AcmeDomain = get_domain_config(&domain)?;
- let plugin_id = domain_config.plugin.as_deref().unwrap_or("standalone");
- let mut plugin_cfg = crate::acme::get_acme_plugin(&plugins, plugin_id)?
- .ok_or_else(|| format_err!("plugin '{plugin_id}' for domain '{domain}' not found!"))?;
-
- info!("Setting up validation plugin");
- let validation_url = plugin_cfg
- .setup(&mut acme, &auth, domain_config, Arc::clone(&worker))
- .await?;
-
- let result = request_validation(&mut acme, auth_url, validation_url).await;
-
- if let Err(err) = plugin_cfg
- .teardown(&mut acme, &auth, domain_config, Arc::clone(&worker))
- .await
- {
- warn!("Failed to teardown plugin '{plugin_id}' for domain '{domain}' - {err}");
- }
-
- result?;
- }
-
- info!("All domains validated");
- info!("Creating CSR");
-
- let csr = proxmox_acme::util::Csr::generate(&identifiers, &Default::default())?;
- let mut finalize_error_cnt = 0u8;
- let order_url = &order.location;
- let mut order;
- loop {
- use proxmox_acme::order::Status;
-
- order = acme.get_order(order_url).await?;
-
- match order.status {
- Status::Pending => {
- info!("still pending, trying to finalize anyway");
- let finalize = order
- .finalize
- .as_deref()
- .ok_or_else(|| format_err!("missing 'finalize' URL in order"))?;
- if let Err(err) = acme.finalize(finalize, &csr.data).await {
- if finalize_error_cnt >= 5 {
- return Err(err);
- }
-
- finalize_error_cnt += 1;
- }
- tokio::time::sleep(Duration::from_secs(5)).await;
- }
- Status::Ready => {
- info!("order is ready, finalizing");
- let finalize = order
- .finalize
- .as_deref()
- .ok_or_else(|| format_err!("missing 'finalize' URL in order"))?;
- acme.finalize(finalize, &csr.data).await?;
- tokio::time::sleep(Duration::from_secs(5)).await;
- }
- Status::Processing => {
- info!("still processing, trying again in 30 seconds");
- tokio::time::sleep(Duration::from_secs(30)).await;
- }
- Status::Valid => {
- info!("valid");
- break;
- }
- other => bail!("order status: {:?}", other),
- }
- }
-
- info!("Downloading certificate");
- let certificate = acme
- .get_certificate(
- order
- .certificate
- .as_deref()
- .ok_or_else(|| format_err!("missing certificate url in finalized order"))?,
- )
- .await?;
-
- Ok(Some(OrderedCertificate {
- certificate,
- private_key_pem: csr.private_key_pem,
- }))
-}
-
-async fn request_validation(
- acme: &mut AcmeClient,
- auth_url: &str,
- validation_url: &str,
-) -> Result<(), Error> {
- info!("Triggering validation");
- acme.request_challenge_validation(validation_url).await?;
-
- info!("Sleeping for 5 seconds");
- tokio::time::sleep(Duration::from_secs(5)).await;
-
- loop {
- use proxmox_acme::authorization::Status;
-
- let auth = acme.get_authorization(auth_url).await?;
- match auth.status {
- Status::Pending => {
- info!("Status is still 'pending', trying again in 10 seconds");
- tokio::time::sleep(Duration::from_secs(10)).await;
- }
- Status::Valid => return Ok(()),
- other => bail!(
- "validating challenge '{}' failed - status: {:?}",
- validation_url,
- other
- ),
- }
- }
-}
-
#[api(
input: {
properties: {
@@ -525,9 +330,30 @@ fn spawn_certificate_worker(
let auth_id = rpcenv.get_auth_id().unwrap();
+ let acme_config = if let Some(cfg) = node_config.acme_config().transpose()? {
+ cfg
+ } else {
+ proxmox_acme_api::parse_acme_config_string("account=default")?
+ };
+
+ let domains = node_config.acme_domains().try_fold(
+ Vec::<AcmeDomain>::new(),
+ |mut acc, domain| -> Result<_, Error> {
+ let mut domain = domain?;
+ domain.domain.make_ascii_lowercase();
+ if let Some(alias) = &mut domain.alias {
+ alias.make_ascii_lowercase();
+ }
+ acc.push(domain);
+ Ok(acc)
+ },
+ )?;
+
WorkerTask::spawn(name, None, auth_id, true, move |worker| async move {
let work = || async {
- if let Some(cert) = order_certificate(worker, &node_config).await? {
+ if let Some(cert) =
+ proxmox_acme_api::order_certificate(worker, &acme_config, &domains).await?
+ {
crate::config::set_proxy_certificate(&cert.certificate, &cert.private_key_pem)?;
crate::server::reload_proxy_certificate().await?;
}
@@ -563,16 +389,20 @@ pub fn revoke_acme_cert(rpcenv: &mut dyn RpcEnvironment) -> Result<String, Error
let auth_id = rpcenv.get_auth_id().unwrap();
+ let acme_config = if let Some(cfg) = node_config.acme_config().transpose()? {
+ cfg
+ } else {
+ proxmox_acme_api::parse_acme_config_string("account=default")?
+ };
+
WorkerTask::spawn(
"acme-revoke-cert",
None,
auth_id,
true,
move |_worker| async move {
- info!("Loading ACME account");
- let mut acme = node_config.acme_client().await?;
info!("Revoking old certificate");
- acme.revoke_certificate(cert_pem.as_bytes(), None).await?;
+ proxmox_acme_api::revoke_certificate(&acme_config, &cert_pem.as_bytes()).await?;
info!("Deleting certificate and regenerating a self-signed one");
delete_custom_certificate().await?;
Ok(())
diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
deleted file mode 100644
index 2905b41b..00000000
--- a/src/api2/types/acme.rs
+++ /dev/null
@@ -1,74 +0,0 @@
-use serde::{Deserialize, Serialize};
-use serde_json::Value;
-
-use proxmox_schema::{api, ApiStringFormat, ApiType, Schema, StringSchema};
-
-use pbs_api_types::{DNS_ALIAS_FORMAT, DNS_NAME_FORMAT, PROXMOX_SAFE_ID_FORMAT};
-
-#[api(
- properties: {
- "domain": { format: &DNS_NAME_FORMAT },
- "alias": {
- optional: true,
- format: &DNS_ALIAS_FORMAT,
- },
- "plugin": {
- optional: true,
- format: &PROXMOX_SAFE_ID_FORMAT,
- },
- },
- default_key: "domain",
-)]
-#[derive(Deserialize, Serialize)]
-/// A domain entry for an ACME certificate.
-pub struct AcmeDomain {
- /// The domain to certify for.
- pub domain: String,
-
- /// The domain to use for challenges instead of the default acme challenge domain.
- ///
- /// This is useful if you use CNAME entries to redirect `_acme-challenge.*` domains to a
- /// different DNS server.
- #[serde(skip_serializing_if = "Option::is_none")]
- pub alias: Option<String>,
-
- /// The plugin to use to validate this domain.
- ///
- /// Empty means standalone HTTP validation is used.
- #[serde(skip_serializing_if = "Option::is_none")]
- pub plugin: Option<String>,
-}
-
-pub const ACME_DOMAIN_PROPERTY_SCHEMA: Schema =
- StringSchema::new("ACME domain configuration string")
- .format(&ApiStringFormat::PropertyString(&AcmeDomain::API_SCHEMA))
- .schema();
-
-#[api(
- properties: {
- schema: {
- type: Object,
- additional_properties: true,
- properties: {},
- },
- type: {
- type: String,
- },
- },
-)]
-#[derive(Serialize)]
-/// Schema for an ACME challenge plugin.
-pub struct AcmeChallengeSchema {
- /// Plugin ID.
- pub id: String,
-
- /// Human readable name, falls back to id.
- pub name: String,
-
- /// Plugin Type.
- #[serde(rename = "type")]
- pub ty: &'static str,
-
- /// The plugin's parameter schema.
- pub schema: Value,
-}
diff --git a/src/api2/types/mod.rs b/src/api2/types/mod.rs
index afc34b30..34193685 100644
--- a/src/api2/types/mod.rs
+++ b/src/api2/types/mod.rs
@@ -4,9 +4,6 @@ use anyhow::bail;
use proxmox_schema::*;
-mod acme;
-pub use acme::*;
-
// File names: may not contain slashes, may not start with "."
pub const FILENAME_FORMAT: ApiStringFormat = ApiStringFormat::VerifyFn(|name| {
if name.starts_with('.') {
diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
index 35cda50b..afd7abf8 100644
--- a/src/config/acme/mod.rs
+++ b/src/config/acme/mod.rs
@@ -9,8 +9,7 @@ use proxmox_sys::fs::{file_read_string, CreateOptions};
use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
-use crate::api2::types::AcmeChallengeSchema;
-use proxmox_acme_api::{AcmeAccountName, KnownAcmeDirectory, KNOWN_ACME_DIRECTORIES};
+use proxmox_acme_api::{AcmeAccountName, AcmeChallengeSchema};
pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
@@ -35,8 +34,6 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
create_acme_subdir(ACME_DIR)
}
-pub const DEFAULT_ACME_DIRECTORY_ENTRY: &KnownAcmeDirectory = &KNOWN_ACME_DIRECTORIES[0];
-
pub fn foreach_acme_account<F>(mut func: F) -> Result<(), Error>
where
F: FnMut(AcmeAccountName) -> ControlFlow<Result<(), Error>>,
@@ -80,7 +77,7 @@ pub fn load_dns_challenge_schema() -> Result<Vec<AcmeChallengeSchema>, Error> {
.and_then(Value::as_str)
.unwrap_or(id)
.to_owned(),
- ty: "dns",
+ ty: "dns".into(),
schema: schema.to_owned(),
})
.collect())
diff --git a/src/config/acme/plugin.rs b/src/config/acme/plugin.rs
index 18e71199..2e979ffe 100644
--- a/src/config/acme/plugin.rs
+++ b/src/config/acme/plugin.rs
@@ -1,104 +1,15 @@
use std::sync::LazyLock;
use anyhow::Error;
-use serde::{Deserialize, Serialize};
-use serde_json::Value;
-
-use proxmox_schema::{api, ApiType, Schema, StringSchema, Updater};
-use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
-
-use pbs_api_types::PROXMOX_SAFE_ID_FORMAT;
use pbs_config::{open_backup_lockfile, BackupLockGuard};
-
-pub const PLUGIN_ID_SCHEMA: Schema = StringSchema::new("ACME Challenge Plugin ID.")
- .format(&PROXMOX_SAFE_ID_FORMAT)
- .min_length(1)
- .max_length(32)
- .schema();
+use proxmox_acme_api::PLUGIN_ID_SCHEMA;
+use proxmox_acme_api::{DnsPlugin, StandalonePlugin};
+use proxmox_schema::{ApiType, Schema};
+use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
+use serde_json::Value;
pub static CONFIG: LazyLock<SectionConfig> = LazyLock::new(init);
-#[api(
- properties: {
- id: { schema: PLUGIN_ID_SCHEMA },
- },
-)]
-#[derive(Deserialize, Serialize)]
-/// Standalone ACME Plugin for the http-1 challenge.
-pub struct StandalonePlugin {
- /// Plugin ID.
- id: String,
-}
-
-impl Default for StandalonePlugin {
- fn default() -> Self {
- Self {
- id: "standalone".to_string(),
- }
- }
-}
-
-#[api(
- properties: {
- id: { schema: PLUGIN_ID_SCHEMA },
- disable: {
- optional: true,
- default: false,
- },
- "validation-delay": {
- default: 30,
- optional: true,
- minimum: 0,
- maximum: 2 * 24 * 60 * 60,
- },
- },
-)]
-/// DNS ACME Challenge Plugin core data.
-#[derive(Deserialize, Serialize, Updater)]
-#[serde(rename_all = "kebab-case")]
-pub struct DnsPluginCore {
- /// Plugin ID.
- #[updater(skip)]
- pub id: String,
-
- /// DNS API Plugin Id.
- pub api: String,
-
- /// Extra delay in seconds to wait before requesting validation.
- ///
- /// Allows to cope with long TTL of DNS records.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- pub validation_delay: Option<u32>,
-
- /// Flag to disable the config.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- pub disable: Option<bool>,
-}
-
-#[api(
- properties: {
- core: { type: DnsPluginCore },
- },
-)]
-/// DNS ACME Challenge Plugin.
-#[derive(Deserialize, Serialize)]
-#[serde(rename_all = "kebab-case")]
-pub struct DnsPlugin {
- #[serde(flatten)]
- pub core: DnsPluginCore,
-
- // We handle this property separately in the API calls.
- /// DNS plugin data (base64url encoded without padding).
- #[serde(with = "proxmox_serde::string_as_base64url_nopad")]
- pub data: String,
-}
-
-impl DnsPlugin {
- pub fn decode_data(&self, output: &mut Vec<u8>) -> Result<(), Error> {
- Ok(proxmox_base64::url::decode_to_vec(&self.data, output)?)
- }
-}
-
fn init() -> SectionConfig {
let mut config = SectionConfig::new(&PLUGIN_ID_SCHEMA);
diff --git a/src/config/node.rs b/src/config/node.rs
index d2a17a49..b9257adf 100644
--- a/src/config/node.rs
+++ b/src/config/node.rs
@@ -6,17 +6,17 @@ use serde::{Deserialize, Serialize};
use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
-use proxmox_http::ProxyConfig;
-
use pbs_api_types::{
EMAIL_SCHEMA, MULTI_LINE_COMMENT_SCHEMA, OPENSSL_CIPHERS_TLS_1_2_SCHEMA,
OPENSSL_CIPHERS_TLS_1_3_SCHEMA,
};
+use proxmox_acme_api::{AcmeConfig, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA};
+use proxmox_http::ProxyConfig;
use pbs_buildcfg::configdir;
use pbs_config::{open_backup_lockfile, BackupLockGuard};
-use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
+use crate::api2::types::HTTP_PROXY_SCHEMA;
use proxmox_acme::async_client::AcmeClient;
use proxmox_acme_api::AcmeAccountName;
@@ -45,20 +45,6 @@ pub fn save_config(config: &NodeConfig) -> Result<(), Error> {
pbs_config::replace_backup_config(CONF_FILE, &raw)
}
-#[api(
- properties: {
- account: { type: AcmeAccountName },
- }
-)]
-#[derive(Deserialize, Serialize)]
-/// The ACME configuration.
-///
-/// Currently only contains the name of the account use.
-pub struct AcmeConfig {
- /// Account to use to acquire ACME certificates.
- account: AcmeAccountName,
-}
-
/// All available languages in Proxmox. Taken from proxmox-i18n repository.
/// pt_BR, zh_CN, and zh_TW use the same case in the translation files.
// TODO: auto-generate from available translations
@@ -244,7 +230,7 @@ impl NodeConfig {
pub async fn acme_client(&self) -> Result<AcmeClient, Error> {
let account = if let Some(cfg) = self.acme_config().transpose()? {
- cfg.account
+ AcmeAccountName::from_string(cfg.account)?
} else {
AcmeAccountName::from_string("default".to_string())? // should really not happen
};
diff --git a/src/lib.rs b/src/lib.rs
index 8633378c..828f5842 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -27,8 +27,6 @@ pub(crate) mod auth;
pub mod tape;
-pub mod acme;
-
pub mod client_helpers;
pub mod traffic_control_cache;
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 7%]
* [pbs-devel] [PATCH proxmox-backup 2/4] acme: drop local AcmeClient
2025-12-02 15:56 12% [pbs-devel] [PATCH proxmox{-backup, } 0/8] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2025-12-02 15:56 15% ` [pbs-devel] [PATCH proxmox-backup 1/4] acme: include proxmox-acme-api dependency Samuel Rufinatscha
@ 2025-12-02 15:56 6% ` Samuel Rufinatscha
2025-12-02 15:56 8% ` [pbs-devel] [PATCH proxmox-backup 3/4] acme: change API impls to use proxmox-acme-api handlers Samuel Rufinatscha
` (6 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-02 15:56 UTC (permalink / raw)
To: pbs-devel
PBS currently uses its own ACME client and API logic, while PDM uses the
factored out proxmox-acme and proxmox-acme-api crates. This duplication
risks differences in behaviour and requires ACME maintenance in two
places. This patch is part of a series to move PBS over to the shared
ACME stack.
Changes:
- Remove the local src/acme/client.rs and switch to
proxmox_acme::async_client::AcmeClient where needed.
- Use proxmox_acme_api::load_client_with_account to the custom
AcmeClient::load() function
- Replace the local do_register() logic with
proxmox_acme_api::register_account, to further ensure accounts are persisted
- Replace the local AcmeAccountName type, required for
proxmox_acme_api::register_account
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/acme/client.rs | 691 -------------------------
src/acme/mod.rs | 3 -
src/acme/plugin.rs | 2 +-
src/api2/config/acme.rs | 50 +-
src/api2/node/certificates.rs | 2 +-
src/api2/types/acme.rs | 8 -
src/bin/proxmox_backup_manager/acme.rs | 17 +-
src/config/acme/mod.rs | 8 +-
src/config/node.rs | 9 +-
9 files changed, 36 insertions(+), 754 deletions(-)
delete mode 100644 src/acme/client.rs
diff --git a/src/acme/client.rs b/src/acme/client.rs
deleted file mode 100644
index 9fb6ad55..00000000
--- a/src/acme/client.rs
+++ /dev/null
@@ -1,691 +0,0 @@
-//! HTTP Client for the ACME protocol.
-
-use std::fs::OpenOptions;
-use std::io;
-use std::os::unix::fs::OpenOptionsExt;
-
-use anyhow::{bail, format_err};
-use bytes::Bytes;
-use http_body_util::BodyExt;
-use hyper::Request;
-use nix::sys::stat::Mode;
-use proxmox_http::Body;
-use serde::{Deserialize, Serialize};
-
-use proxmox_acme::account::AccountCreator;
-use proxmox_acme::order::{Order, OrderData};
-use proxmox_acme::types::AccountData as AcmeAccountData;
-use proxmox_acme::Request as AcmeRequest;
-use proxmox_acme::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
-use proxmox_http::client::Client;
-use proxmox_sys::fs::{replace_file, CreateOptions};
-
-use crate::api2::types::AcmeAccountName;
-use crate::config::acme::account_path;
-use crate::tools::pbs_simple_http;
-
-/// Our on-disk format inherited from PVE's proxmox-acme code.
-#[derive(Deserialize, Serialize)]
-#[serde(rename_all = "camelCase")]
-pub struct AccountData {
- /// The account's location URL.
- location: String,
-
- /// The account data.
- account: AcmeAccountData,
-
- /// The private key as PEM formatted string.
- key: String,
-
- /// ToS URL the user agreed to.
- #[serde(skip_serializing_if = "Option::is_none")]
- tos: Option<String>,
-
- #[serde(skip_serializing_if = "is_false", default)]
- debug: bool,
-
- /// The directory's URL.
- directory_url: String,
-}
-
-#[inline]
-fn is_false(b: &bool) -> bool {
- !*b
-}
-
-pub struct AcmeClient {
- directory_url: String,
- debug: bool,
- account_path: Option<String>,
- tos: Option<String>,
- account: Option<Account>,
- directory: Option<Directory>,
- nonce: Option<String>,
- http_client: Client,
-}
-
-impl AcmeClient {
- /// Create a new ACME client for a given ACME directory URL.
- pub fn new(directory_url: String) -> Self {
- Self {
- directory_url,
- debug: false,
- account_path: None,
- tos: None,
- account: None,
- directory: None,
- nonce: None,
- http_client: pbs_simple_http(None),
- }
- }
-
- /// Load an existing ACME account by name.
- pub async fn load(account_name: &AcmeAccountName) -> Result<Self, anyhow::Error> {
- let account_path = account_path(account_name.as_ref());
- let data = match tokio::fs::read(&account_path).await {
- Ok(data) => data,
- Err(err) if err.kind() == io::ErrorKind::NotFound => {
- bail!("acme account '{}' does not exist", account_name)
- }
- Err(err) => bail!(
- "failed to load acme account from '{}' - {}",
- account_path,
- err
- ),
- };
- let data: AccountData = serde_json::from_slice(&data).map_err(|err| {
- format_err!(
- "failed to parse acme account from '{}' - {}",
- account_path,
- err
- )
- })?;
-
- let account = Account::from_parts(data.location, data.key, data.account);
-
- let mut me = Self::new(data.directory_url);
- me.debug = data.debug;
- me.account_path = Some(account_path);
- me.tos = data.tos;
- me.account = Some(account);
-
- Ok(me)
- }
-
- pub async fn new_account<'a>(
- &'a mut self,
- account_name: &AcmeAccountName,
- tos_agreed: bool,
- contact: Vec<String>,
- rsa_bits: Option<u32>,
- eab_creds: Option<(String, String)>,
- ) -> Result<&'a Account, anyhow::Error> {
- self.tos = if tos_agreed {
- self.terms_of_service_url().await?.map(str::to_owned)
- } else {
- None
- };
-
- let mut account = Account::creator()
- .set_contacts(contact)
- .agree_to_tos(tos_agreed);
-
- if let Some((eab_kid, eab_hmac_key)) = eab_creds {
- account = account.set_eab_credentials(eab_kid, eab_hmac_key)?;
- }
-
- let account = if let Some(bits) = rsa_bits {
- account.generate_rsa_key(bits)?
- } else {
- account.generate_ec_key()?
- };
-
- let _ = self.register_account(account).await?;
-
- crate::config::acme::make_acme_account_dir()?;
- let account_path = account_path(account_name.as_ref());
- let file = OpenOptions::new()
- .write(true)
- .create_new(true)
- .mode(0o600)
- .open(&account_path)
- .map_err(|err| format_err!("failed to open {:?} for writing: {}", account_path, err))?;
- self.write_to(file).map_err(|err| {
- format_err!(
- "failed to write acme account to {:?}: {}",
- account_path,
- err
- )
- })?;
- self.account_path = Some(account_path);
-
- // unwrap: Setting `self.account` is literally this function's job, we just can't keep
- // the borrow from from `self.register_account()` active due to clashes.
- Ok(self.account.as_ref().unwrap())
- }
-
- fn save(&self) -> Result<(), anyhow::Error> {
- let mut data = Vec::<u8>::new();
- self.write_to(&mut data)?;
- let account_path = self.account_path.as_ref().ok_or_else(|| {
- format_err!("no account path set, cannot save updated account information")
- })?;
- crate::config::acme::make_acme_account_dir()?;
- replace_file(
- account_path,
- &data,
- CreateOptions::new()
- .perm(Mode::from_bits_truncate(0o600))
- .owner(nix::unistd::ROOT)
- .group(nix::unistd::Gid::from_raw(0)),
- true,
- )
- }
-
- /// Shortcut to `account().ok_or_else(...).key_authorization()`.
- pub fn key_authorization(&self, token: &str) -> Result<String, anyhow::Error> {
- Ok(Self::need_account(&self.account)?.key_authorization(token)?)
- }
-
- /// Shortcut to `account().ok_or_else(...).dns_01_txt_value()`.
- /// the key authorization value.
- pub fn dns_01_txt_value(&self, token: &str) -> Result<String, anyhow::Error> {
- Ok(Self::need_account(&self.account)?.dns_01_txt_value(token)?)
- }
-
- async fn register_account(
- &mut self,
- account: AccountCreator,
- ) -> Result<&Account, anyhow::Error> {
- let mut retry = retry();
- let mut response = loop {
- retry.tick()?;
-
- let (directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
- let request = account.request(directory, nonce)?;
- match self.run_request(request).await {
- Ok(response) => break response,
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- };
-
- let account = account.response(response.location_required()?, &response.body)?;
-
- self.account = Some(account);
- Ok(self.account.as_ref().unwrap())
- }
-
- pub async fn update_account<T: Serialize>(
- &mut self,
- data: &T,
- ) -> Result<&Account, anyhow::Error> {
- let account = Self::need_account(&self.account)?;
-
- let mut retry = retry();
- let response = loop {
- retry.tick()?;
-
- let (_directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let request = account.post_request(&account.location, nonce, data)?;
- match Self::execute(&mut self.http_client, request, &mut self.nonce).await {
- Ok(response) => break response,
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- };
-
- // unwrap: we've been keeping an immutable reference to it from the top of the method
- let _ = account;
- self.account.as_mut().unwrap().data = response.json()?;
- self.save()?;
- Ok(self.account.as_ref().unwrap())
- }
-
- pub async fn new_order<I>(&mut self, domains: I) -> Result<Order, anyhow::Error>
- where
- I: IntoIterator<Item = String>,
- {
- let account = Self::need_account(&self.account)?;
-
- let order = domains
- .into_iter()
- .fold(OrderData::new(), |order, domain| order.domain(domain));
-
- let mut retry = retry();
- loop {
- retry.tick()?;
-
- let (directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let mut new_order = account.new_order(&order, directory, nonce)?;
- let mut response = match Self::execute(
- &mut self.http_client,
- new_order.request.take().unwrap(),
- &mut self.nonce,
- )
- .await
- {
- Ok(response) => response,
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- };
-
- return Ok(
- new_order.response(response.location_required()?, response.bytes().as_ref())?
- );
- }
- }
-
- /// Low level "POST-as-GET" request.
- async fn post_as_get(&mut self, url: &str) -> Result<AcmeResponse, anyhow::Error> {
- let account = Self::need_account(&self.account)?;
-
- let mut retry = retry();
- loop {
- retry.tick()?;
-
- let (_directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let request = account.get_request(url, nonce)?;
- match Self::execute(&mut self.http_client, request, &mut self.nonce).await {
- Ok(response) => return Ok(response),
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- }
- }
-
- /// Low level POST request.
- async fn post<T: Serialize>(
- &mut self,
- url: &str,
- data: &T,
- ) -> Result<AcmeResponse, anyhow::Error> {
- let account = Self::need_account(&self.account)?;
-
- let mut retry = retry();
- loop {
- retry.tick()?;
-
- let (_directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let request = account.post_request(url, nonce, data)?;
- match Self::execute(&mut self.http_client, request, &mut self.nonce).await {
- Ok(response) => return Ok(response),
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- }
- }
-
- /// Request challenge validation. Afterwards, the challenge should be polled.
- pub async fn request_challenge_validation(
- &mut self,
- url: &str,
- ) -> Result<Challenge, anyhow::Error> {
- Ok(self
- .post(url, &serde_json::Value::Object(Default::default()))
- .await?
- .json()?)
- }
-
- /// Assuming the provided URL is an 'Authorization' URL, get and deserialize it.
- pub async fn get_authorization(&mut self, url: &str) -> Result<Authorization, anyhow::Error> {
- Ok(self.post_as_get(url).await?.json()?)
- }
-
- /// Assuming the provided URL is an 'Order' URL, get and deserialize it.
- pub async fn get_order(&mut self, url: &str) -> Result<OrderData, anyhow::Error> {
- Ok(self.post_as_get(url).await?.json()?)
- }
-
- /// Finalize an Order via its `finalize` URL property and the DER encoded CSR.
- pub async fn finalize(&mut self, url: &str, csr: &[u8]) -> Result<(), anyhow::Error> {
- let csr = proxmox_base64::url::encode_no_pad(csr);
- let data = serde_json::json!({ "csr": csr });
- self.post(url, &data).await?;
- Ok(())
- }
-
- /// Download a certificate via its 'certificate' URL property.
- ///
- /// The certificate will be a PEM certificate chain.
- pub async fn get_certificate(&mut self, url: &str) -> Result<Bytes, anyhow::Error> {
- Ok(self.post_as_get(url).await?.body)
- }
-
- /// Revoke an existing certificate (PEM or DER formatted).
- pub async fn revoke_certificate(
- &mut self,
- certificate: &[u8],
- reason: Option<u32>,
- ) -> Result<(), anyhow::Error> {
- // TODO: This can also work without an account.
- let account = Self::need_account(&self.account)?;
-
- let revocation = account.revoke_certificate(certificate, reason)?;
-
- let mut retry = retry();
- loop {
- retry.tick()?;
-
- let (directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let request = revocation.request(directory, nonce)?;
- match Self::execute(&mut self.http_client, request, &mut self.nonce).await {
- Ok(_response) => return Ok(()),
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- }
- }
-
- fn need_account(account: &Option<Account>) -> Result<&Account, anyhow::Error> {
- account
- .as_ref()
- .ok_or_else(|| format_err!("cannot use client without an account"))
- }
-
- pub(crate) fn account(&self) -> Result<&Account, anyhow::Error> {
- Self::need_account(&self.account)
- }
-
- pub fn tos(&self) -> Option<&str> {
- self.tos.as_deref()
- }
-
- pub fn directory_url(&self) -> &str {
- &self.directory_url
- }
-
- fn to_account_data(&self) -> Result<AccountData, anyhow::Error> {
- let account = self.account()?;
-
- Ok(AccountData {
- location: account.location.clone(),
- key: account.private_key.clone(),
- account: AcmeAccountData {
- only_return_existing: false, // don't actually write this out in case it's set
- ..account.data.clone()
- },
- tos: self.tos.clone(),
- debug: self.debug,
- directory_url: self.directory_url.clone(),
- })
- }
-
- fn write_to<T: io::Write>(&self, out: T) -> Result<(), anyhow::Error> {
- let data = self.to_account_data()?;
-
- Ok(serde_json::to_writer_pretty(out, &data)?)
- }
-}
-
-struct AcmeResponse {
- body: Bytes,
- location: Option<String>,
- got_nonce: bool,
-}
-
-impl AcmeResponse {
- /// Convenience helper to assert that a location header was part of the response.
- fn location_required(&mut self) -> Result<String, anyhow::Error> {
- self.location
- .take()
- .ok_or_else(|| format_err!("missing Location header"))
- }
-
- /// Convenience shortcut to perform json deserialization of the returned body.
- fn json<T: for<'a> Deserialize<'a>>(&self) -> Result<T, Error> {
- Ok(serde_json::from_slice(&self.body)?)
- }
-
- /// Convenience shortcut to get the body as bytes.
- fn bytes(&self) -> &[u8] {
- &self.body
- }
-}
-
-impl AcmeClient {
- /// Non-self-borrowing run_request version for borrow workarounds.
- async fn execute(
- http_client: &mut Client,
- request: AcmeRequest,
- nonce: &mut Option<String>,
- ) -> Result<AcmeResponse, Error> {
- let req_builder = Request::builder().method(request.method).uri(&request.url);
-
- let http_request = if !request.content_type.is_empty() {
- req_builder
- .header("Content-Type", request.content_type)
- .header("Content-Length", request.body.len())
- .body(request.body.into())
- } else {
- req_builder.body(Body::empty())
- }
- .map_err(|err| Error::Custom(format!("failed to create http request: {err}")))?;
-
- let response = http_client
- .request(http_request)
- .await
- .map_err(|err| Error::Custom(err.to_string()))?;
- let (parts, body) = response.into_parts();
-
- let status = parts.status.as_u16();
- let body = body
- .collect()
- .await
- .map_err(|err| Error::Custom(format!("failed to retrieve response body: {err}")))?
- .to_bytes();
-
- let got_nonce = if let Some(new_nonce) = parts.headers.get(proxmox_acme::REPLAY_NONCE) {
- let new_nonce = new_nonce.to_str().map_err(|err| {
- Error::Client(format!(
- "received invalid replay-nonce header from ACME server: {err}"
- ))
- })?;
- *nonce = Some(new_nonce.to_owned());
- true
- } else {
- false
- };
-
- if parts.status.is_success() {
- if status != request.expected {
- return Err(Error::InvalidApi(format!(
- "ACME server responded with unexpected status code: {:?}",
- parts.status
- )));
- }
-
- let location = parts
- .headers
- .get("Location")
- .map(|header| {
- header.to_str().map(str::to_owned).map_err(|err| {
- Error::Client(format!(
- "received invalid location header from ACME server: {err}"
- ))
- })
- })
- .transpose()?;
-
- return Ok(AcmeResponse {
- body,
- location,
- got_nonce,
- });
- }
-
- let error: ErrorResponse = serde_json::from_slice(&body).map_err(|err| {
- Error::Client(format!(
- "error status with improper error ACME response: {err}"
- ))
- })?;
-
- if error.ty == proxmox_acme::error::BAD_NONCE {
- if !got_nonce {
- return Err(Error::InvalidApi(
- "badNonce without a new Replay-Nonce header".to_string(),
- ));
- }
- return Err(Error::BadNonce);
- }
-
- Err(Error::Api(error))
- }
-
- /// Low-level API to run an n API request. This automatically updates the current nonce!
- async fn run_request(&mut self, request: AcmeRequest) -> Result<AcmeResponse, Error> {
- Self::execute(&mut self.http_client, request, &mut self.nonce).await
- }
-
- pub async fn directory(&mut self) -> Result<&Directory, Error> {
- Ok(Self::get_directory(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?
- .0)
- }
-
- async fn get_directory<'a, 'b>(
- http_client: &mut Client,
- directory_url: &str,
- directory: &'a mut Option<Directory>,
- nonce: &'b mut Option<String>,
- ) -> Result<(&'a Directory, Option<&'b str>), Error> {
- if let Some(d) = directory {
- return Ok((d, nonce.as_deref()));
- }
-
- let response = Self::execute(
- http_client,
- AcmeRequest {
- url: directory_url.to_string(),
- method: "GET",
- content_type: "",
- body: String::new(),
- expected: 200,
- },
- nonce,
- )
- .await?;
-
- *directory = Some(Directory::from_parts(
- directory_url.to_string(),
- response.json()?,
- ));
-
- Ok((directory.as_mut().unwrap(), nonce.as_deref()))
- }
-
- /// Like `get_directory`, but if the directory provides no nonce, also performs a `HEAD`
- /// request on the new nonce URL.
- async fn get_dir_nonce<'a, 'b>(
- http_client: &mut Client,
- directory_url: &str,
- directory: &'a mut Option<Directory>,
- nonce: &'b mut Option<String>,
- ) -> Result<(&'a Directory, &'b str), Error> {
- // this let construct is a lifetime workaround:
- let _ = Self::get_directory(http_client, directory_url, directory, nonce).await?;
- let dir = directory.as_ref().unwrap(); // the above fails if it couldn't fill this option
- if nonce.is_none() {
- // this is also a lifetime issue...
- let _ = Self::get_nonce(http_client, nonce, dir.new_nonce_url()).await?;
- };
- Ok((dir, nonce.as_deref().unwrap()))
- }
-
- pub async fn terms_of_service_url(&mut self) -> Result<Option<&str>, Error> {
- Ok(self.directory().await?.terms_of_service_url())
- }
-
- async fn get_nonce<'a>(
- http_client: &mut Client,
- nonce: &'a mut Option<String>,
- new_nonce_url: &str,
- ) -> Result<&'a str, Error> {
- let response = Self::execute(
- http_client,
- AcmeRequest {
- url: new_nonce_url.to_owned(),
- method: "HEAD",
- content_type: "",
- body: String::new(),
- expected: 200,
- },
- nonce,
- )
- .await?;
-
- if !response.got_nonce {
- return Err(Error::InvalidApi(
- "no new nonce received from new nonce URL".to_string(),
- ));
- }
-
- nonce
- .as_deref()
- .ok_or_else(|| Error::Client("failed to update nonce".to_string()))
- }
-}
-
-/// bad nonce retry count helper
-struct Retry(usize);
-
-const fn retry() -> Retry {
- Retry(0)
-}
-
-impl Retry {
- fn tick(&mut self) -> Result<(), Error> {
- if self.0 >= 3 {
- Err(Error::Client("kept getting a badNonce error!".to_string()))
- } else {
- self.0 += 1;
- Ok(())
- }
- }
-}
diff --git a/src/acme/mod.rs b/src/acme/mod.rs
index bf61811c..cc561f9a 100644
--- a/src/acme/mod.rs
+++ b/src/acme/mod.rs
@@ -1,5 +1,2 @@
-mod client;
-pub use client::AcmeClient;
-
pub(crate) mod plugin;
pub(crate) use plugin::get_acme_plugin;
diff --git a/src/acme/plugin.rs b/src/acme/plugin.rs
index f756e9b5..5bc09e1f 100644
--- a/src/acme/plugin.rs
+++ b/src/acme/plugin.rs
@@ -20,8 +20,8 @@ use tokio::process::Command;
use proxmox_acme::{Authorization, Challenge};
-use crate::acme::AcmeClient;
use crate::api2::types::AcmeDomain;
+use proxmox_acme::async_client::AcmeClient;
use proxmox_rest_server::WorkerTask;
use crate::config::acme::plugin::{DnsPlugin, PluginData};
diff --git a/src/api2/config/acme.rs b/src/api2/config/acme.rs
index 35c3fb77..02f88e2e 100644
--- a/src/api2/config/acme.rs
+++ b/src/api2/config/acme.rs
@@ -16,15 +16,15 @@ use proxmox_router::{
use proxmox_schema::{api, param_bail};
use proxmox_acme::types::AccountData as AcmeAccountData;
-use proxmox_acme::Account;
use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
-use crate::acme::AcmeClient;
-use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
+use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
use crate::config::acme::plugin::{
self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
};
+use proxmox_acme::async_client::AcmeClient;
+use proxmox_acme_api::AcmeAccountName;
use proxmox_rest_server::WorkerTask;
pub(crate) const ROUTER: Router = Router::new()
@@ -143,15 +143,15 @@ pub struct AccountInfo {
)]
/// Return existing ACME account information.
pub async fn get_account(name: AcmeAccountName) -> Result<AccountInfo, Error> {
- let client = AcmeClient::load(&name).await?;
- let account = client.account()?;
+ let account_info = proxmox_acme_api::get_account(name).await?;
+
Ok(AccountInfo {
- location: account.location.clone(),
- tos: client.tos().map(str::to_owned),
- directory: client.directory_url().to_owned(),
+ location: account_info.location,
+ tos: account_info.tos,
+ directory: account_info.directory,
account: AcmeAccountData {
only_return_existing: false, // don't actually write this out in case it's set
- ..account.data.clone()
+ ..account_info.account
},
})
}
@@ -240,41 +240,24 @@ fn register_account(
auth_id.to_string(),
true,
move |_worker| async move {
- let mut client = AcmeClient::new(directory);
-
info!("Registering ACME account '{}'...", &name);
- let account = do_register_account(
- &mut client,
+ let location = proxmox_acme_api::register_account(
&name,
- tos_url.is_some(),
contact,
- None,
+ tos_url,
+ Some(directory),
eab_kid.zip(eab_hmac_key),
)
.await?;
- info!("Registration successful, account URL: {}", account.location);
+ info!("Registration successful, account URL: {}", location);
Ok(())
},
)
}
-pub async fn do_register_account<'a>(
- client: &'a mut AcmeClient,
- name: &AcmeAccountName,
- agree_to_tos: bool,
- contact: String,
- rsa_bits: Option<u32>,
- eab_creds: Option<(String, String)>,
-) -> Result<&'a Account, Error> {
- let contact = account_contact_from_string(&contact);
- client
- .new_account(name, agree_to_tos, contact, rsa_bits, eab_creds)
- .await
-}
-
#[api(
input: {
properties: {
@@ -312,7 +295,10 @@ pub fn update_account(
None => json!({}),
};
- AcmeClient::load(&name).await?.update_account(&data).await?;
+ proxmox_acme_api::load_client_with_account(&name)
+ .await?
+ .update_account(&data)
+ .await?;
Ok(())
},
@@ -350,7 +336,7 @@ pub fn deactivate_account(
auth_id.to_string(),
true,
move |_worker| async move {
- match AcmeClient::load(&name)
+ match proxmox_acme_api::load_client_with_account(&name)
.await?
.update_account(&json!({"status": "deactivated"}))
.await
diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.rs
index 61ef910e..31196715 100644
--- a/src/api2/node/certificates.rs
+++ b/src/api2/node/certificates.rs
@@ -17,10 +17,10 @@ use pbs_buildcfg::configdir;
use pbs_tools::cert;
use tracing::warn;
-use crate::acme::AcmeClient;
use crate::api2::types::AcmeDomain;
use crate::config::node::NodeConfig;
use crate::server::send_certificate_renewal_mail;
+use proxmox_acme::async_client::AcmeClient;
use proxmox_rest_server::WorkerTask;
pub const ROUTER: Router = Router::new()
diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
index 210ebdbc..7c9063c0 100644
--- a/src/api2/types/acme.rs
+++ b/src/api2/types/acme.rs
@@ -60,14 +60,6 @@ pub struct KnownAcmeDirectory {
pub url: &'static str,
}
-proxmox_schema::api_string_type! {
- #[api(format: &PROXMOX_SAFE_ID_FORMAT)]
- /// ACME account name.
- #[derive(Clone, Eq, PartialEq, Hash, Deserialize, Serialize)]
- #[serde(transparent)]
- pub struct AcmeAccountName(String);
-}
-
#[api(
properties: {
schema: {
diff --git a/src/bin/proxmox_backup_manager/acme.rs b/src/bin/proxmox_backup_manager/acme.rs
index 0f0eafea..bb987b26 100644
--- a/src/bin/proxmox_backup_manager/acme.rs
+++ b/src/bin/proxmox_backup_manager/acme.rs
@@ -7,9 +7,9 @@ use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
use proxmox_schema::api;
use proxmox_sys::fs::file_get_contents;
-use proxmox_backup::acme::AcmeClient;
+use proxmox_acme::async_client::AcmeClient;
+use proxmox_acme_api::AcmeAccountName;
use proxmox_backup::api2;
-use proxmox_backup::api2::types::AcmeAccountName;
use proxmox_backup::config::acme::plugin::DnsPluginCore;
use proxmox_backup::config::acme::KNOWN_ACME_DIRECTORIES;
@@ -188,17 +188,20 @@ async fn register_account(
println!("Attempting to register account with {directory_url:?}...");
- let account = api2::config::acme::do_register_account(
- &mut client,
+ let tos_agreed = tos_agreed
+ .then(|| directory.terms_of_service_url().map(str::to_owned))
+ .flatten();
+
+ let location = proxmox_acme_api::register_account(
&name,
- tos_agreed,
contact,
- None,
+ tos_agreed,
+ Some(directory_url),
eab_creds,
)
.await?;
- println!("Registration successful, account URL: {}", account.location);
+ println!("Registration successful, account URL: {}", location);
Ok(())
}
diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
index 274a23fd..d31b2bc9 100644
--- a/src/config/acme/mod.rs
+++ b/src/config/acme/mod.rs
@@ -10,7 +10,8 @@ use proxmox_sys::fs::{file_read_string, CreateOptions};
use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
-use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
+use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
+use proxmox_acme_api::AcmeAccountName;
pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
@@ -35,11 +36,6 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
create_acme_subdir(ACME_DIR)
}
-pub(crate) fn make_acme_account_dir() -> Result<(), Error> {
- make_acme_dir()?;
- create_acme_subdir(ACME_ACCOUNT_DIR)
-}
-
pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
KnownAcmeDirectory {
name: "Let's Encrypt V2",
diff --git a/src/config/node.rs b/src/config/node.rs
index d2d6e383..d2a17a49 100644
--- a/src/config/node.rs
+++ b/src/config/node.rs
@@ -16,10 +16,9 @@ use pbs_api_types::{
use pbs_buildcfg::configdir;
use pbs_config::{open_backup_lockfile, BackupLockGuard};
-use crate::acme::AcmeClient;
-use crate::api2::types::{
- AcmeAccountName, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA,
-};
+use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
+use proxmox_acme::async_client::AcmeClient;
+use proxmox_acme_api::AcmeAccountName;
const CONF_FILE: &str = configdir!("/node.cfg");
const LOCK_FILE: &str = configdir!("/.node.lck");
@@ -249,7 +248,7 @@ impl NodeConfig {
} else {
AcmeAccountName::from_string("default".to_string())? // should really not happen
};
- AcmeClient::load(&account).await
+ proxmox_acme_api::load_client_with_account(&account).await
}
pub fn acme_domains(&'_ self) -> AcmeDomainIter<'_> {
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* [pbs-devel] [PATCH proxmox 3/4] acme-api: add helper to load client for an account
2025-12-02 15:56 12% [pbs-devel] [PATCH proxmox{-backup, } 0/8] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (5 preceding siblings ...)
2025-12-02 15:56 15% ` [pbs-devel] [PATCH proxmox 2/4] acme: introduce http_status module Samuel Rufinatscha
@ 2025-12-02 15:56 17% ` Samuel Rufinatscha
2025-12-02 15:56 14% ` [pbs-devel] [PATCH proxmox 4/4] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2025-12-02 16:02 6% ` [pbs-devel] [PATCH proxmox{-backup, } 0/8] " Samuel Rufinatscha
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-02 15:56 UTC (permalink / raw)
To: pbs-devel
The PBS ACME refactoring needs a simple way to obtain an AcmeClient for
a given configured account without duplicating config wiring. This patch
adds a load_client_with_account helper in proxmox-acme-api that loads
the account and constructs a matching client, similarly as PBS previous
own AcmeClient::load() function.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme-api/src/account_api_impl.rs | 5 +++++
proxmox-acme-api/src/lib.rs | 3 ++-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/proxmox-acme-api/src/account_api_impl.rs b/proxmox-acme-api/src/account_api_impl.rs
index ef195908..ca8c8655 100644
--- a/proxmox-acme-api/src/account_api_impl.rs
+++ b/proxmox-acme-api/src/account_api_impl.rs
@@ -116,3 +116,8 @@ pub async fn update_account(name: &AcmeAccountName, contact: Option<String>) ->
Ok(())
}
+
+pub async fn load_client_with_account(account_name: &AcmeAccountName) -> Result<AcmeClient, Error> {
+ let account_data = super::account_config::load_account_config(&account_name).await?;
+ Ok(account_data.client())
+}
diff --git a/proxmox-acme-api/src/lib.rs b/proxmox-acme-api/src/lib.rs
index 623e9e23..96f88ae2 100644
--- a/proxmox-acme-api/src/lib.rs
+++ b/proxmox-acme-api/src/lib.rs
@@ -31,7 +31,8 @@ mod plugin_config;
mod account_api_impl;
#[cfg(feature = "impl")]
pub use account_api_impl::{
- deactivate_account, get_account, get_tos, list_accounts, register_account, update_account,
+ deactivate_account, get_account, get_tos, list_accounts, load_client_with_account,
+ register_account, update_account,
};
#[cfg(feature = "impl")]
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 17%]
* [pbs-devel] [PATCH proxmox 2/4] acme: introduce http_status module
2025-12-02 15:56 12% [pbs-devel] [PATCH proxmox{-backup, } 0/8] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (4 preceding siblings ...)
2025-12-02 15:56 12% ` [pbs-devel] [PATCH proxmox 1/4] acme: reduce visibility of Request type Samuel Rufinatscha
@ 2025-12-02 15:56 15% ` Samuel Rufinatscha
2025-12-02 15:56 17% ` [pbs-devel] [PATCH proxmox 3/4] acme-api: add helper to load client for an account Samuel Rufinatscha
` (2 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-02 15:56 UTC (permalink / raw)
To: pbs-devel
Introduce an internal http_status module with the common ACME HTTP
response codes, and replace use of crate::request::CREATED as well as
direct numeric status code usages.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme/src/account.rs | 10 +++++-----
proxmox-acme/src/async_client.rs | 4 ++--
proxmox-acme/src/lib.rs | 2 ++
proxmox-acme/src/request.rs | 11 ++++++++++-
4 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
index 081ca986..350c78d4 100644
--- a/proxmox-acme/src/account.rs
+++ b/proxmox-acme/src/account.rs
@@ -85,7 +85,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::request::CREATED,
+ expected: crate::http_status::CREATED,
};
Ok(NewOrder::new(request))
@@ -107,7 +107,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: crate::http_status::OK,
})
}
@@ -132,7 +132,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: crate::http_status::OK,
})
}
@@ -157,7 +157,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: crate::http_status::OK,
})
}
@@ -408,7 +408,7 @@ impl AccountCreator {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::request::CREATED,
+ expected: crate::http_status::CREATED,
})
}
diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
index 2ff3ba22..043648bb 100644
--- a/proxmox-acme/src/async_client.rs
+++ b/proxmox-acme/src/async_client.rs
@@ -498,7 +498,7 @@ impl AcmeClient {
method: "GET",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: crate::http_status::OK,
},
nonce,
)
@@ -550,7 +550,7 @@ impl AcmeClient {
method: "HEAD",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: crate::http_status::OK,
},
nonce,
)
diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
index 6722030c..6051a025 100644
--- a/proxmox-acme/src/lib.rs
+++ b/proxmox-acme/src/lib.rs
@@ -70,6 +70,8 @@ pub use order::Order;
#[cfg(feature = "impl")]
pub use order::NewOrder;
#[cfg(feature = "impl")]
+pub(crate) use request::http_status;
+#[cfg(feature = "impl")]
pub use request::ErrorResponse;
/// Header name for nonces.
diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
index dadfc5af..341ce53e 100644
--- a/proxmox-acme/src/request.rs
+++ b/proxmox-acme/src/request.rs
@@ -1,7 +1,6 @@
use serde::Deserialize;
pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
-pub(crate) const CREATED: u16 = 201;
/// A request which should be performed on the ACME provider.
pub(crate) struct Request {
@@ -21,6 +20,16 @@ pub(crate) struct Request {
pub(crate) expected: u16,
}
+/// Common HTTP status codes used in ACME responses.
+pub(crate) mod http_status {
+ /// 200 OK
+ pub(crate) const OK: u16 = 200;
+ /// 201 Created
+ pub(crate) const CREATED: u16 = 201;
+ /// 204 No Content
+ pub(crate) const NO_CONTENT: u16 = 204;
+}
+
/// An ACME error response contains a specially formatted type string, and can optionally
/// contain textual details and a set of sub problems.
#[derive(Clone, Debug, Deserialize)]
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox 4/4] fix #6939: acme: support servers returning 204 for nonce requests
2025-12-02 15:56 12% [pbs-devel] [PATCH proxmox{-backup, } 0/8] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (6 preceding siblings ...)
2025-12-02 15:56 17% ` [pbs-devel] [PATCH proxmox 3/4] acme-api: add helper to load client for an account Samuel Rufinatscha
@ 2025-12-02 15:56 14% ` Samuel Rufinatscha
2025-12-02 16:02 6% ` [pbs-devel] [PATCH proxmox{-backup, } 0/8] " Samuel Rufinatscha
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-02 15:56 UTC (permalink / raw)
To: pbs-devel
Some ACME servers (notably custom or legacy implementations) respond
to HEAD /newNonce with a 204 No Content instead of the
RFC 8555-recommended 200 OK [1]. While this behavior is technically
off-spec, it is not illegal. This issue was reported on our bug
tracker [2].
The previous implementation treated any non-200 response as an error,
causing account registration to fail against such servers. Relax the
status-code check to accept both 200 and 204 responses (and potentially
support other 2xx codes) to improve interoperability.
Note: In comparison, PVE’s Perl ACME client performs a GET request [3]
instead of a HEAD request and accepts any 2xx success code when
retrieving the nonce [4]. This difference in behavior does not affect
functionality but is worth noting for consistency across
implementations.
[1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
[3] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219
[4] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597
Fixes: #6939
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme/src/account.rs | 10 +++++-----
proxmox-acme/src/async_client.rs | 6 +++---
proxmox-acme/src/client.rs | 2 +-
proxmox-acme/src/request.rs | 4 ++--
4 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
index 350c78d4..820b209d 100644
--- a/proxmox-acme/src/account.rs
+++ b/proxmox-acme/src/account.rs
@@ -85,7 +85,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::CREATED,
+ expected: &[crate::http_status::CREATED],
};
Ok(NewOrder::new(request))
@@ -107,7 +107,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK],
})
}
@@ -132,7 +132,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK],
})
}
@@ -157,7 +157,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK],
})
}
@@ -408,7 +408,7 @@ impl AccountCreator {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::CREATED,
+ expected: &[crate::http_status::CREATED],
})
}
diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
index 043648bb..07da842c 100644
--- a/proxmox-acme/src/async_client.rs
+++ b/proxmox-acme/src/async_client.rs
@@ -420,7 +420,7 @@ impl AcmeClient {
};
if parts.status.is_success() {
- if status != request.expected {
+ if !request.expected.contains(&status) {
return Err(Error::InvalidApi(format!(
"ACME server responded with unexpected status code: {:?}",
parts.status
@@ -498,7 +498,7 @@ impl AcmeClient {
method: "GET",
content_type: "",
body: String::new(),
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK],
},
nonce,
)
@@ -550,7 +550,7 @@ impl AcmeClient {
method: "HEAD",
content_type: "",
body: String::new(),
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK, crate::http_status::NO_CONTENT],
},
nonce,
)
diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
index 5c812567..af250fb8 100644
--- a/proxmox-acme/src/client.rs
+++ b/proxmox-acme/src/client.rs
@@ -203,7 +203,7 @@ impl Inner {
let got_nonce = self.update_nonce(&mut response)?;
if response.is_success() {
- if response.status != request.expected {
+ if !request.expected.contains(&response.status) {
return Err(Error::InvalidApi(format!(
"API server responded with unexpected status code: {:?}",
response.status
diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
index 341ce53e..d782a7de 100644
--- a/proxmox-acme/src/request.rs
+++ b/proxmox-acme/src/request.rs
@@ -16,8 +16,8 @@ pub(crate) struct Request {
/// The body to pass along with request, or an empty string.
pub(crate) body: String,
- /// The expected status code a compliant ACME provider will return on success.
- pub(crate) expected: u16,
+ /// The set of HTTP status codes that indicate a successful response from an ACME provider.
+ pub(crate) expected: &'static [u16],
}
/// Common HTTP status codes used in ACME responses.
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 14%]
* [pbs-devel] [PATCH proxmox 1/4] acme: reduce visibility of Request type
2025-12-02 15:56 12% [pbs-devel] [PATCH proxmox{-backup, } 0/8] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (3 preceding siblings ...)
2025-12-02 15:56 7% ` [pbs-devel] [PATCH proxmox-backup 4/4] acme: certificate ordering through proxmox-acme-api Samuel Rufinatscha
@ 2025-12-02 15:56 12% ` Samuel Rufinatscha
2025-12-02 15:56 15% ` [pbs-devel] [PATCH proxmox 2/4] acme: introduce http_status module Samuel Rufinatscha
` (3 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-02 15:56 UTC (permalink / raw)
To: pbs-devel
Currently, the low-level ACME Request type is publicly exposed, even
though users are expected to go through AcmeClient and
proxmox-acme-api handlers. This patch reduces visibility so that
the Request type and related fields/methods are crate-internal only.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme/src/account.rs | 17 ++++++++++-------
proxmox-acme/src/async_client.rs | 2 +-
proxmox-acme/src/authorization.rs | 2 +-
proxmox-acme/src/client.rs | 6 +++---
proxmox-acme/src/lib.rs | 4 ----
proxmox-acme/src/order.rs | 2 +-
proxmox-acme/src/request.rs | 12 ++++++------
7 files changed, 22 insertions(+), 23 deletions(-)
diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
index 0bbf0027..081ca986 100644
--- a/proxmox-acme/src/account.rs
+++ b/proxmox-acme/src/account.rs
@@ -92,7 +92,7 @@ impl Account {
}
/// Prepare a "POST-as-GET" request to fetch data. Low level helper.
- pub fn get_request(&self, url: &str, nonce: &str) -> Result<Request, Error> {
+ pub(crate) fn get_request(&self, url: &str, nonce: &str) -> Result<Request, Error> {
let key = PKey::private_key_from_pem(self.private_key.as_bytes())?;
let body = serde_json::to_string(&Jws::new_full(
&key,
@@ -112,7 +112,7 @@ impl Account {
}
/// Prepare a JSON POST request. Low level helper.
- pub fn post_request<T: Serialize>(
+ pub(crate) fn post_request<T: Serialize>(
&self,
url: &str,
nonce: &str,
@@ -179,7 +179,7 @@ impl Account {
/// Prepare a request to update account data.
///
/// This is a rather low level interface. You should know what you're doing.
- pub fn update_account_request<T: Serialize>(
+ pub(crate) fn update_account_request<T: Serialize>(
&self,
nonce: &str,
data: &T,
@@ -188,7 +188,10 @@ impl Account {
}
/// Prepare a request to deactivate this account.
- pub fn deactivate_account_request<T: Serialize>(&self, nonce: &str) -> Result<Request, Error> {
+ pub(crate) fn deactivate_account_request<T: Serialize>(
+ &self,
+ nonce: &str,
+ ) -> Result<Request, Error> {
self.post_request_raw_payload(
&self.location,
nonce,
@@ -220,7 +223,7 @@ impl Account {
///
/// This returns a raw `Request` since validation takes some time and the `Authorization`
/// object has to be re-queried and its `status` inspected.
- pub fn validate_challenge(
+ pub(crate) fn validate_challenge(
&self,
authorization: &Authorization,
challenge_index: usize,
@@ -274,7 +277,7 @@ pub struct CertificateRevocation<'a> {
impl CertificateRevocation<'_> {
/// Create the revocation request using the specified nonce for the given directory.
- pub fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
+ pub(crate) fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
let revoke_cert = directory.data.revoke_cert.as_ref().ok_or_else(|| {
Error::Custom("no 'revokeCert' URL specified by provider".to_string())
})?;
@@ -364,7 +367,7 @@ impl AccountCreator {
/// the resulting request.
/// Changing the private key between using the request and passing the response to
/// [`response`](AccountCreator::response()) will render the account unusable!
- pub fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
+ pub(crate) fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
let key = self.key.as_deref().ok_or(Error::MissingKey)?;
let url = directory.new_account_url().ok_or_else(|| {
Error::Custom("no 'newAccount' URL specified by provider".to_string())
diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
index dc755fb9..2ff3ba22 100644
--- a/proxmox-acme/src/async_client.rs
+++ b/proxmox-acme/src/async_client.rs
@@ -10,7 +10,7 @@ use proxmox_http::{client::Client, Body};
use crate::account::AccountCreator;
use crate::order::{Order, OrderData};
-use crate::Request as AcmeRequest;
+use crate::request::Request as AcmeRequest;
use crate::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
/// A non-blocking Acme client using tokio/hyper.
diff --git a/proxmox-acme/src/authorization.rs b/proxmox-acme/src/authorization.rs
index 28bc1b4b..765714fc 100644
--- a/proxmox-acme/src/authorization.rs
+++ b/proxmox-acme/src/authorization.rs
@@ -145,7 +145,7 @@ pub struct GetAuthorization {
/// this is guaranteed to be `Some`.
///
/// The response should be passed to the the [`response`](GetAuthorization::response()) method.
- pub request: Option<Request>,
+ pub(crate) request: Option<Request>,
}
impl GetAuthorization {
diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
index 931f7245..5c812567 100644
--- a/proxmox-acme/src/client.rs
+++ b/proxmox-acme/src/client.rs
@@ -7,8 +7,8 @@ use serde::{Deserialize, Serialize};
use crate::b64u;
use crate::error;
use crate::order::OrderData;
-use crate::request::ErrorResponse;
-use crate::{Account, Authorization, Challenge, Directory, Error, Order, Request};
+use crate::request::{ErrorResponse, Request};
+use crate::{Account, Authorization, Challenge, Directory, Error, Order};
macro_rules! format_err {
($($fmt:tt)*) => { Error::Client(format!($($fmt)*)) };
@@ -564,7 +564,7 @@ impl Client {
}
/// Low-level API to run an n API request. This automatically updates the current nonce!
- pub fn run_request(&mut self, request: Request) -> Result<HttpResponse, Error> {
+ pub(crate) fn run_request(&mut self, request: Request) -> Result<HttpResponse, Error> {
self.inner.run_request(request)
}
diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
index df722629..6722030c 100644
--- a/proxmox-acme/src/lib.rs
+++ b/proxmox-acme/src/lib.rs
@@ -66,10 +66,6 @@ pub use error::Error;
#[doc(inline)]
pub use order::Order;
-#[cfg(feature = "impl")]
-#[doc(inline)]
-pub use request::Request;
-
// we don't inline these:
#[cfg(feature = "impl")]
pub use order::NewOrder;
diff --git a/proxmox-acme/src/order.rs b/proxmox-acme/src/order.rs
index b6551004..432a81a4 100644
--- a/proxmox-acme/src/order.rs
+++ b/proxmox-acme/src/order.rs
@@ -153,7 +153,7 @@ pub struct NewOrder {
//order: OrderData,
/// The request to execute to place the order. When creating a [`NewOrder`] via
/// [`Account::new_order`](crate::Account::new_order) this is guaranteed to be `Some`.
- pub request: Option<Request>,
+ pub(crate) request: Option<Request>,
}
impl NewOrder {
diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
index 78a90913..dadfc5af 100644
--- a/proxmox-acme/src/request.rs
+++ b/proxmox-acme/src/request.rs
@@ -4,21 +4,21 @@ pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
pub(crate) const CREATED: u16 = 201;
/// A request which should be performed on the ACME provider.
-pub struct Request {
+pub(crate) struct Request {
/// The complete URL to send the request to.
- pub url: String,
+ pub(crate) url: String,
/// The HTTP method name to use.
- pub method: &'static str,
+ pub(crate) method: &'static str,
/// The `Content-Type` header to pass along.
- pub content_type: &'static str,
+ pub(crate) content_type: &'static str,
/// The body to pass along with request, or an empty string.
- pub body: String,
+ pub(crate) body: String,
/// The expected status code a compliant ACME provider will return on success.
- pub expected: u16,
+ pub(crate) expected: u16,
}
/// An ACME error response contains a specially formatted type string, and can optionally
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* Re: [pbs-devel] [PATCH proxmox{-backup, } 0/8] fix #6939: acme: support servers returning 204 for nonce requests
2025-12-02 15:56 12% [pbs-devel] [PATCH proxmox{-backup, } 0/8] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (7 preceding siblings ...)
2025-12-02 15:56 14% ` [pbs-devel] [PATCH proxmox 4/4] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
@ 2025-12-02 16:02 6% ` Samuel Rufinatscha
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-02 16:02 UTC (permalink / raw)
To: pbs-devel
Ignore this please, forgot to add the version in the subject.
Will send a new one.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* [pbs-devel] [PATCH proxmox v4 2/4] acme: reduce visibility of Request type
2025-12-03 10:22 11% [pbs-devel] [PATCH proxmox{-backup, } v4 " Samuel Rufinatscha
` (4 preceding siblings ...)
2025-12-03 10:22 17% ` [pbs-devel] [PATCH proxmox v4 1/4] acme-api: add helper to load client for an account Samuel Rufinatscha
@ 2025-12-03 10:22 12% ` Samuel Rufinatscha
2025-12-09 16:51 5% ` Max R. Carrara
2025-12-03 10:22 15% ` [pbs-devel] [PATCH proxmox v4 3/4] acme: introduce http_status module Samuel Rufinatscha
` (3 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-12-03 10:22 UTC (permalink / raw)
To: pbs-devel
Currently, the low-level ACME Request type is publicly exposed, even
though users are expected to go through AcmeClient and
proxmox-acme-api handlers. This patch reduces visibility so that
the Request type and related fields/methods are crate-internal only.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme/src/account.rs | 17 ++++++++++-------
proxmox-acme/src/async_client.rs | 2 +-
proxmox-acme/src/authorization.rs | 2 +-
proxmox-acme/src/client.rs | 6 +++---
proxmox-acme/src/lib.rs | 4 ----
proxmox-acme/src/order.rs | 2 +-
proxmox-acme/src/request.rs | 12 ++++++------
7 files changed, 22 insertions(+), 23 deletions(-)
diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
index 0bbf0027..081ca986 100644
--- a/proxmox-acme/src/account.rs
+++ b/proxmox-acme/src/account.rs
@@ -92,7 +92,7 @@ impl Account {
}
/// Prepare a "POST-as-GET" request to fetch data. Low level helper.
- pub fn get_request(&self, url: &str, nonce: &str) -> Result<Request, Error> {
+ pub(crate) fn get_request(&self, url: &str, nonce: &str) -> Result<Request, Error> {
let key = PKey::private_key_from_pem(self.private_key.as_bytes())?;
let body = serde_json::to_string(&Jws::new_full(
&key,
@@ -112,7 +112,7 @@ impl Account {
}
/// Prepare a JSON POST request. Low level helper.
- pub fn post_request<T: Serialize>(
+ pub(crate) fn post_request<T: Serialize>(
&self,
url: &str,
nonce: &str,
@@ -179,7 +179,7 @@ impl Account {
/// Prepare a request to update account data.
///
/// This is a rather low level interface. You should know what you're doing.
- pub fn update_account_request<T: Serialize>(
+ pub(crate) fn update_account_request<T: Serialize>(
&self,
nonce: &str,
data: &T,
@@ -188,7 +188,10 @@ impl Account {
}
/// Prepare a request to deactivate this account.
- pub fn deactivate_account_request<T: Serialize>(&self, nonce: &str) -> Result<Request, Error> {
+ pub(crate) fn deactivate_account_request<T: Serialize>(
+ &self,
+ nonce: &str,
+ ) -> Result<Request, Error> {
self.post_request_raw_payload(
&self.location,
nonce,
@@ -220,7 +223,7 @@ impl Account {
///
/// This returns a raw `Request` since validation takes some time and the `Authorization`
/// object has to be re-queried and its `status` inspected.
- pub fn validate_challenge(
+ pub(crate) fn validate_challenge(
&self,
authorization: &Authorization,
challenge_index: usize,
@@ -274,7 +277,7 @@ pub struct CertificateRevocation<'a> {
impl CertificateRevocation<'_> {
/// Create the revocation request using the specified nonce for the given directory.
- pub fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
+ pub(crate) fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
let revoke_cert = directory.data.revoke_cert.as_ref().ok_or_else(|| {
Error::Custom("no 'revokeCert' URL specified by provider".to_string())
})?;
@@ -364,7 +367,7 @@ impl AccountCreator {
/// the resulting request.
/// Changing the private key between using the request and passing the response to
/// [`response`](AccountCreator::response()) will render the account unusable!
- pub fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
+ pub(crate) fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
let key = self.key.as_deref().ok_or(Error::MissingKey)?;
let url = directory.new_account_url().ok_or_else(|| {
Error::Custom("no 'newAccount' URL specified by provider".to_string())
diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
index dc755fb9..2ff3ba22 100644
--- a/proxmox-acme/src/async_client.rs
+++ b/proxmox-acme/src/async_client.rs
@@ -10,7 +10,7 @@ use proxmox_http::{client::Client, Body};
use crate::account::AccountCreator;
use crate::order::{Order, OrderData};
-use crate::Request as AcmeRequest;
+use crate::request::Request as AcmeRequest;
use crate::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
/// A non-blocking Acme client using tokio/hyper.
diff --git a/proxmox-acme/src/authorization.rs b/proxmox-acme/src/authorization.rs
index 28bc1b4b..765714fc 100644
--- a/proxmox-acme/src/authorization.rs
+++ b/proxmox-acme/src/authorization.rs
@@ -145,7 +145,7 @@ pub struct GetAuthorization {
/// this is guaranteed to be `Some`.
///
/// The response should be passed to the the [`response`](GetAuthorization::response()) method.
- pub request: Option<Request>,
+ pub(crate) request: Option<Request>,
}
impl GetAuthorization {
diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
index 931f7245..5c812567 100644
--- a/proxmox-acme/src/client.rs
+++ b/proxmox-acme/src/client.rs
@@ -7,8 +7,8 @@ use serde::{Deserialize, Serialize};
use crate::b64u;
use crate::error;
use crate::order::OrderData;
-use crate::request::ErrorResponse;
-use crate::{Account, Authorization, Challenge, Directory, Error, Order, Request};
+use crate::request::{ErrorResponse, Request};
+use crate::{Account, Authorization, Challenge, Directory, Error, Order};
macro_rules! format_err {
($($fmt:tt)*) => { Error::Client(format!($($fmt)*)) };
@@ -564,7 +564,7 @@ impl Client {
}
/// Low-level API to run an n API request. This automatically updates the current nonce!
- pub fn run_request(&mut self, request: Request) -> Result<HttpResponse, Error> {
+ pub(crate) fn run_request(&mut self, request: Request) -> Result<HttpResponse, Error> {
self.inner.run_request(request)
}
diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
index df722629..6722030c 100644
--- a/proxmox-acme/src/lib.rs
+++ b/proxmox-acme/src/lib.rs
@@ -66,10 +66,6 @@ pub use error::Error;
#[doc(inline)]
pub use order::Order;
-#[cfg(feature = "impl")]
-#[doc(inline)]
-pub use request::Request;
-
// we don't inline these:
#[cfg(feature = "impl")]
pub use order::NewOrder;
diff --git a/proxmox-acme/src/order.rs b/proxmox-acme/src/order.rs
index b6551004..432a81a4 100644
--- a/proxmox-acme/src/order.rs
+++ b/proxmox-acme/src/order.rs
@@ -153,7 +153,7 @@ pub struct NewOrder {
//order: OrderData,
/// The request to execute to place the order. When creating a [`NewOrder`] via
/// [`Account::new_order`](crate::Account::new_order) this is guaranteed to be `Some`.
- pub request: Option<Request>,
+ pub(crate) request: Option<Request>,
}
impl NewOrder {
diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
index 78a90913..dadfc5af 100644
--- a/proxmox-acme/src/request.rs
+++ b/proxmox-acme/src/request.rs
@@ -4,21 +4,21 @@ pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
pub(crate) const CREATED: u16 = 201;
/// A request which should be performed on the ACME provider.
-pub struct Request {
+pub(crate) struct Request {
/// The complete URL to send the request to.
- pub url: String,
+ pub(crate) url: String,
/// The HTTP method name to use.
- pub method: &'static str,
+ pub(crate) method: &'static str,
/// The `Content-Type` header to pass along.
- pub content_type: &'static str,
+ pub(crate) content_type: &'static str,
/// The body to pass along with request, or an empty string.
- pub body: String,
+ pub(crate) body: String,
/// The expected status code a compliant ACME provider will return on success.
- pub expected: u16,
+ pub(crate) expected: u16,
}
/// An ACME error response contains a specially formatted type string, and can optionally
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] [PATCH proxmox v4 3/4] acme: introduce http_status module
2025-12-03 10:22 11% [pbs-devel] [PATCH proxmox{-backup, } v4 " Samuel Rufinatscha
` (5 preceding siblings ...)
2025-12-03 10:22 12% ` [pbs-devel] [PATCH proxmox v4 2/4] acme: reduce visibility of Request type Samuel Rufinatscha
@ 2025-12-03 10:22 15% ` Samuel Rufinatscha
2025-12-03 10:22 14% ` [pbs-devel] [PATCH proxmox v4 4/4] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (2 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-03 10:22 UTC (permalink / raw)
To: pbs-devel
Introduce an internal http_status module with the common ACME HTTP
response codes, and replace use of crate::request::CREATED as well as
direct numeric status code usages.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme/src/account.rs | 10 +++++-----
proxmox-acme/src/async_client.rs | 4 ++--
proxmox-acme/src/lib.rs | 2 ++
proxmox-acme/src/request.rs | 11 ++++++++++-
4 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
index 081ca986..350c78d4 100644
--- a/proxmox-acme/src/account.rs
+++ b/proxmox-acme/src/account.rs
@@ -85,7 +85,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::request::CREATED,
+ expected: crate::http_status::CREATED,
};
Ok(NewOrder::new(request))
@@ -107,7 +107,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: crate::http_status::OK,
})
}
@@ -132,7 +132,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: crate::http_status::OK,
})
}
@@ -157,7 +157,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: crate::http_status::OK,
})
}
@@ -408,7 +408,7 @@ impl AccountCreator {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::request::CREATED,
+ expected: crate::http_status::CREATED,
})
}
diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
index 2ff3ba22..043648bb 100644
--- a/proxmox-acme/src/async_client.rs
+++ b/proxmox-acme/src/async_client.rs
@@ -498,7 +498,7 @@ impl AcmeClient {
method: "GET",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: crate::http_status::OK,
},
nonce,
)
@@ -550,7 +550,7 @@ impl AcmeClient {
method: "HEAD",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: crate::http_status::OK,
},
nonce,
)
diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
index 6722030c..6051a025 100644
--- a/proxmox-acme/src/lib.rs
+++ b/proxmox-acme/src/lib.rs
@@ -70,6 +70,8 @@ pub use order::Order;
#[cfg(feature = "impl")]
pub use order::NewOrder;
#[cfg(feature = "impl")]
+pub(crate) use request::http_status;
+#[cfg(feature = "impl")]
pub use request::ErrorResponse;
/// Header name for nonces.
diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
index dadfc5af..341ce53e 100644
--- a/proxmox-acme/src/request.rs
+++ b/proxmox-acme/src/request.rs
@@ -1,7 +1,6 @@
use serde::Deserialize;
pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
-pub(crate) const CREATED: u16 = 201;
/// A request which should be performed on the ACME provider.
pub(crate) struct Request {
@@ -21,6 +20,16 @@ pub(crate) struct Request {
pub(crate) expected: u16,
}
+/// Common HTTP status codes used in ACME responses.
+pub(crate) mod http_status {
+ /// 200 OK
+ pub(crate) const OK: u16 = 200;
+ /// 201 Created
+ pub(crate) const CREATED: u16 = 201;
+ /// 204 No Content
+ pub(crate) const NO_CONTENT: u16 = 204;
+}
+
/// An ACME error response contains a specially formatted type string, and can optionally
/// contain textual details and a set of sub problems.
#[derive(Clone, Debug, Deserialize)]
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox{-backup, } v4 0/8] fix #6939: acme: support servers returning 204 for nonce requests
@ 2025-12-03 10:22 11% Samuel Rufinatscha
2025-12-03 10:22 15% ` [pbs-devel] [PATCH proxmox-backup v4 1/4] acme: include proxmox-acme-api dependency Samuel Rufinatscha
` (9 more replies)
0 siblings, 10 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-03 10:22 UTC (permalink / raw)
To: pbs-devel
Hi,
this series fixes account registration for ACME providers that return
HTTP 204 No Content to the newNonce request. Currently, both the PBS
ACME client and the shared ACME client in proxmox-acme only accept
HTTP 200 OK for this request. The issue was observed in PBS against a
custom ACME deployment and reported as bug #6939 [1].
## Problem
During ACME account registration, PBS first fetches an anti-replay
nonce by sending a HEAD request to the CA’s newNonce URL.
RFC 8555 §7.2 [2] states that:
* the server MUST include a Replay-Nonce header with a fresh nonce,
* the server SHOULD use status 200 OK for the HEAD request,
* the server MUST also handle GET on the same resource and may return
204 No Content with an empty body.
The reporter observed the following error message:
*ACME server responded with unexpected status code: 204*
and mentioned that the issue did not appear with PVE 9 [1]. Looking at
PVE’s Perl ACME client [3], it uses a GET request instead of HEAD and
accepts any 2xx success code when retrieving the nonce. This difference
in behavior does not affect functionality but is worth noting for
consistency across implementations.
## Approach
To support ACME providers which return 204 No Content, the Rust ACME
clients in proxmox-backup and proxmox need to treat both 200 OK and 204
No Content as valid responses for the nonce request, as long as a
Replay-Nonce header is present.
This series changes the expected field of the internal Request type
from a single u16 to a list of allowed status codes
(e.g. &'static [u16]), so one request can explicitly accept multiple
success codes.
To avoid fixing the issue twice (once in PBS’ own ACME client and once
in the shared Rust client), this series first refactors PBS to use the
shared AcmeClient from proxmox-acme / proxmox-acme-api, similar to PDM,
and then applies the bug fix in that shared implementation so that all
consumers benefit from the more tolerant behavior.
## Testing
*Testing the refactor*
To test the refactor, I
(1) installed latest stable PBS on a VM
(2) created .deb package from latest PBS (master), containing the
refactor
(3) installed created .deb package
(4) installed Pebble from Let's Encrypt [5] on the same VM
(5) created an ACME account and ordered the new certificate for the
host domain.
Steps to reproduce:
(1) install latest stable PBS on a VM, create .deb package from latest
PBS (master) containing the refactor, install created .deb package
(2) install Pebble from Let's Encrypt [5] on the same VM:
cd
apt update
apt install -y golang git
git clone https://github.com/letsencrypt/pebble
cd pebble
go build ./cmd/pebble
then, download and trust the Pebble cert:
wget https://raw.githubusercontent.com/letsencrypt/pebble/main/test/certs/pebble.minica.pem
cp pebble.minica.pem /usr/local/share/ca-certificates/pebble.minica.crt
update-ca-certificates
We want Pebble to perform HTTP-01 validation against port 80, because
PBS’s standalone plugin will bind port 80. Set httpPort to 80.
nano ./test/config/pebble-config.json
Start the Pebble server in the background:
./pebble -config ./test/config/pebble-config.json &
Create a Pebble ACME account:
proxmox-backup-manager acme account register default admin@example.com --directory 'https://127.0.0.1:14000/dir'
To verify persistence of the account I checked
ls /etc/proxmox-backup/acme/accounts
Verified if update-account works
proxmox-backup-manager acme account update default --contact "a@example.com,b@example.com"
proxmox-backup-manager acme account info default
In the PBS GUI, you can create a new domain. You can use your host
domain name (see /etc/hosts). Select the created account and order the
certificate.
After a page reload, you might need to accept the new certificate in the browser.
In the PBS dashboard, you should see the new Pebble certificate.
*Note: on reboot, the created Pebble ACME account will be gone and you
will need to create a new one. Pebble does not persist account info.
In that case remove the previously created account in
/etc/proxmox-backup/acme/accounts.
*Testing the newNonce fix*
To prove the ACME newNonce fix, I put nginx in front of Pebble, to
intercept the newNonce request in order to return 204 No Content
instead of 200 OK, all other requests are unchanged and forwarded to
Pebble. Requires trusting the nginx CAs via
/usr/local/share/ca-certificates + update-ca-certificates on the VM.
Then I ran following command against nginx:
proxmox-backup-manager acme account register proxytest root@backup.local --directory 'https://nginx-address/dir
The account could be created successfully. When adjusting the nginx
configuration to return any other non-expected success status code,
PBS rejects as expected.
## Patch summary
0001 – acme: include proxmox-acme-api dependency
Adds proxmox-acme-api as a new dependency for the ACME code. This
prepares the codebase to use the shared ACME API instead of local
implementations.
0002 – acme: drop local AcmeClient
Removes the local AcmeClient implementation. Minimal changes
required to support the removal.
0003 – acme: change API impls to use proxmox-acme-api handler
Updates existing ACME API implementations to use the handlers provided
by proxmox-acme-api.
0004 – acme: certificate ordering through proxmox-acme-api
Perform certificate ordering through proxmox-acme-api instead of local
logic.
0005 – acme api: add helper to load client for an account
Introduces a helper function to load an ACME client instance for a
given account. Required for the PBS refactor.
0006 – acme: reduce visibility of Request type
Restricts the visibility of the internal Request type.
0007 – acme: introduce http_status module
Adds a dedicated http_status module for handling common HTTP status
codes.
0008 – fix #6939: acme: support servers returning 204 for nonce
Adjusts nonce handling to support ACME servers that return HTTP 204
(No Content) for new-nonce requests.
Thanks for considering this patch series, I look forward to your
feedback.
Best,
Samuel Rufinatscha
## Changelog
Changes from v3 to v4:
Removed: [PATCH proxmox-backup v3 1/1].
Added:
[PATCH proxmox-backup v4 1/4] acme: include proxmox-acme-api dependency
* New: add proxmox-acme-api as a dependency and initialize it in
PBS so PBS can use the shared ACME API instead.
[PATCH proxmox-backup v4 2/4] acme: drop local AcmeClient
* New: remove the PBS-local AcmeClient implementation and switch PBS
over to the shared proxmox-acme async client.
[PATCH proxmox-backup v4 3/4] acme: change API impls to use proxmox-acme-api
handlers
* New: rework PBS’ ACME API endpoints to delegate to
proxmox-acme-api handlers instead of duplicating logic locally.
[PATCH proxmox-backup v4 4/4] acme: certificate ordering through
proxmox-acme-api
* New: move PBS’ ACME certificate ordering logic over to
proxmox-acme-api, keeping only certificate installation/reload in
PBS.
[PATCH proxmox v4 1/4] acme-api: add helper to load client for an account
* New: add a load_client_with_account helper in proxmox-acme-api so
PBS (and others) can construct an AcmeClient for a configured account
without duplicating boilerplate.
[PATCH proxmox v4 2/4] acme: reduce visibility of Request type
* New: hide the low-level Request type and its fields behind
constructors / reduced visibility so changes to “expected” no longer
affect the public API as they did in v3.
[PATCH proxmox v4 3/4] acme: introduce http_status module
* New: split out the HTTP status constants into an internal
http_status module as a separate preparatory cleanup before the bug
fix, instead of doing this inline like in v3.
Changed:
[PATCH proxmox v3 1/1] -> [PATCH proxmox v4 4/4]
fix #6939: acme: support server returning 204 for nonce requests
* Rebased on top of the refactor: keep the same behavioural fix as in v3
(accept 204 for newNonce with Replay-Nonce present), but implement it
on top of the http_status module that is part of the refactor.
Changes from v2 to v3:
[PATCH proxmox v3 1/1] fix #6939: support providers returning 204 for nonce
requests
* Rename `http_success` module to `http_status`
[PATCH proxmox-backup v3 1/1] acme: accept HTTP 204 from newNonce endpoint
* Replace `http_success` usage
Changes from v1 to v2:
[PATCH proxmox v2 1/1] fix #6939: support providers returning 204 for nonce
requests
* Introduced `http_success` module to contain the http success codes
* Replaced `Vec<u16>` with `&[u16]` for expected codes to avoid
allocations.
* Clarified the PVEs Perl ACME client behaviour in the commit message.
[PATCH proxmox-backup v2 1/1] acme: accept HTTP 204 from newNonce endpoint
* Integrated the `http_success` module, replacing `Vec<u16>` with `&[u16]`
* Clarified the PVEs Perl ACME client behaviour in the commit message.
[1] Bugzilla report #6939:
[https://bugzilla.proxmox.com/show_bug.cgi?id=6939](https://bugzilla.proxmox.com/show_bug.cgi?id=6939)
[2] RFC 8555 (ACME):
[https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2](https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2)
[3] PVE’s Perl ACME client (allow 2xx codes for nonce requests):
[https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597)
[4] Pebble ACME server:
[https://github.com/letsencrypt/pebble](https://github.com/letsencrypt/pebble)
[5] Pebble ACME server (perform GET request:
[https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219)
proxmox-backup:
Samuel Rufinatscha (4):
acme: include proxmox-acme-api dependency
acme: drop local AcmeClient
acme: change API impls to use proxmox-acme-api handlers
acme: certificate ordering through proxmox-acme-api
Cargo.toml | 3 +
src/acme/client.rs | 691 -------------------------
src/acme/mod.rs | 5 -
src/acme/plugin.rs | 336 ------------
src/api2/config/acme.rs | 407 ++-------------
src/api2/node/certificates.rs | 240 ++-------
src/api2/types/acme.rs | 98 ----
src/api2/types/mod.rs | 3 -
src/bin/proxmox-backup-api.rs | 2 +
src/bin/proxmox-backup-manager.rs | 2 +
src/bin/proxmox-backup-proxy.rs | 1 +
src/bin/proxmox_backup_manager/acme.rs | 21 +-
src/config/acme/mod.rs | 51 +-
src/config/acme/plugin.rs | 99 +---
src/config/node.rs | 29 +-
src/lib.rs | 2 -
16 files changed, 103 insertions(+), 1887 deletions(-)
delete mode 100644 src/acme/client.rs
delete mode 100644 src/acme/mod.rs
delete mode 100644 src/acme/plugin.rs
delete mode 100644 src/api2/types/acme.rs
proxmox:
Samuel Rufinatscha (4):
acme-api: add helper to load client for an account
acme: reduce visibility of Request type
acme: introduce http_status module
fix #6939: acme: support servers returning 204 for nonce requests
proxmox-acme-api/src/account_api_impl.rs | 5 +++++
proxmox-acme-api/src/lib.rs | 3 ++-
proxmox-acme/src/account.rs | 27 +++++++++++++-----------
proxmox-acme/src/async_client.rs | 8 +++----
proxmox-acme/src/authorization.rs | 2 +-
proxmox-acme/src/client.rs | 8 +++----
proxmox-acme/src/lib.rs | 6 ++----
proxmox-acme/src/order.rs | 2 +-
proxmox-acme/src/request.rs | 25 +++++++++++++++-------
9 files changed, 51 insertions(+), 35 deletions(-)
Summary over all repositories:
25 files changed, 154 insertions(+), 1922 deletions(-)
--
Generated by git-murpp 0.8.1
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 11%]
* [pbs-devel] [PATCH proxmox-backup v4 4/4] acme: certificate ordering through proxmox-acme-api
2025-12-03 10:22 11% [pbs-devel] [PATCH proxmox{-backup, } v4 " Samuel Rufinatscha
` (2 preceding siblings ...)
2025-12-03 10:22 8% ` [pbs-devel] [PATCH proxmox-backup v4 3/4] acme: change API impls to use proxmox-acme-api handlers Samuel Rufinatscha
@ 2025-12-03 10:22 7% ` Samuel Rufinatscha
2025-12-09 16:50 5% ` Max R. Carrara
2025-12-03 10:22 17% ` [pbs-devel] [PATCH proxmox v4 1/4] acme-api: add helper to load client for an account Samuel Rufinatscha
` (5 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-12-03 10:22 UTC (permalink / raw)
To: pbs-devel
PBS currently uses its own ACME client and API logic, while PDM uses the
factored out proxmox-acme and proxmox-acme-api crates. This duplication
risks differences in behaviour and requires ACME maintenance in two
places. This patch is part of a series to move PBS over to the shared
ACME stack.
Changes:
- Replace the custom ACME order/authorization loop in node certificates
with a call to proxmox_acme_api::order_certificate.
- Build domain + config data as proxmox-acme-api types
- Remove obsolete local ACME ordering and plugin glue code.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/acme/mod.rs | 2 -
src/acme/plugin.rs | 336 ----------------------------------
src/api2/node/certificates.rs | 240 ++++--------------------
src/api2/types/acme.rs | 74 --------
src/api2/types/mod.rs | 3 -
src/config/acme/mod.rs | 7 +-
src/config/acme/plugin.rs | 99 +---------
src/config/node.rs | 22 +--
src/lib.rs | 2 -
9 files changed, 46 insertions(+), 739 deletions(-)
delete mode 100644 src/acme/mod.rs
delete mode 100644 src/acme/plugin.rs
delete mode 100644 src/api2/types/acme.rs
diff --git a/src/acme/mod.rs b/src/acme/mod.rs
deleted file mode 100644
index cc561f9a..00000000
--- a/src/acme/mod.rs
+++ /dev/null
@@ -1,2 +0,0 @@
-pub(crate) mod plugin;
-pub(crate) use plugin::get_acme_plugin;
diff --git a/src/acme/plugin.rs b/src/acme/plugin.rs
deleted file mode 100644
index 5bc09e1f..00000000
--- a/src/acme/plugin.rs
+++ /dev/null
@@ -1,336 +0,0 @@
-use std::future::Future;
-use std::net::{IpAddr, SocketAddr};
-use std::pin::Pin;
-use std::process::Stdio;
-use std::sync::Arc;
-use std::time::Duration;
-
-use anyhow::{bail, format_err, Error};
-use bytes::Bytes;
-use futures::TryFutureExt;
-use http_body_util::Full;
-use hyper::body::Incoming;
-use hyper::server::conn::http1;
-use hyper::service::service_fn;
-use hyper::{Request, Response};
-use hyper_util::rt::TokioIo;
-use tokio::io::{AsyncBufReadExt, AsyncRead, AsyncWriteExt, BufReader};
-use tokio::net::TcpListener;
-use tokio::process::Command;
-
-use proxmox_acme::{Authorization, Challenge};
-
-use crate::api2::types::AcmeDomain;
-use proxmox_acme::async_client::AcmeClient;
-use proxmox_rest_server::WorkerTask;
-
-use crate::config::acme::plugin::{DnsPlugin, PluginData};
-
-const PROXMOX_ACME_SH_PATH: &str = "/usr/share/proxmox-acme/proxmox-acme";
-
-pub(crate) fn get_acme_plugin(
- plugin_data: &PluginData,
- name: &str,
-) -> Result<Option<Box<dyn AcmePlugin + Send + Sync + 'static>>, Error> {
- let (ty, data) = match plugin_data.get(name) {
- Some(plugin) => plugin,
- None => return Ok(None),
- };
-
- Ok(Some(match ty.as_str() {
- "dns" => {
- let plugin: DnsPlugin = serde::Deserialize::deserialize(data)?;
- Box::new(plugin)
- }
- "standalone" => {
- // this one has no config
- Box::<StandaloneServer>::default()
- }
- other => bail!("missing implementation for plugin type '{}'", other),
- }))
-}
-
-pub(crate) trait AcmePlugin {
- /// Setup everything required to trigger the validation and return the corresponding validation
- /// URL.
- fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- domain: &'d AcmeDomain,
- task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>>;
-
- fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- domain: &'d AcmeDomain,
- task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>>;
-}
-
-fn extract_challenge<'a>(
- authorization: &'a Authorization,
- ty: &str,
-) -> Result<&'a Challenge, Error> {
- authorization
- .challenges
- .iter()
- .find(|ch| ch.ty == ty)
- .ok_or_else(|| format_err!("no supported challenge type ({}) found", ty))
-}
-
-async fn pipe_to_tasklog<T: AsyncRead + Unpin>(
- pipe: T,
- task: Arc<WorkerTask>,
-) -> Result<(), std::io::Error> {
- let mut pipe = BufReader::new(pipe);
- let mut line = String::new();
- loop {
- line.clear();
- match pipe.read_line(&mut line).await {
- Ok(0) => return Ok(()),
- Ok(_) => task.log_message(line.as_str()),
- Err(err) => return Err(err),
- }
- }
-}
-
-impl DnsPlugin {
- async fn action<'a>(
- &self,
- client: &mut AcmeClient,
- authorization: &'a Authorization,
- domain: &AcmeDomain,
- task: Arc<WorkerTask>,
- action: &str,
- ) -> Result<&'a str, Error> {
- let challenge = extract_challenge(authorization, "dns-01")?;
- let mut stdin_data = client
- .dns_01_txt_value(
- challenge
- .token()
- .ok_or_else(|| format_err!("missing token in challenge"))?,
- )?
- .into_bytes();
- stdin_data.push(b'\n');
- stdin_data.extend(self.data.as_bytes());
- if stdin_data.last() != Some(&b'\n') {
- stdin_data.push(b'\n');
- }
-
- let mut command = Command::new("/usr/bin/setpriv");
-
- #[rustfmt::skip]
- command.args([
- "--reuid", "nobody",
- "--regid", "nogroup",
- "--clear-groups",
- "--reset-env",
- "--",
- "/bin/bash",
- PROXMOX_ACME_SH_PATH,
- action,
- &self.core.api,
- domain.alias.as_deref().unwrap_or(&domain.domain),
- ]);
-
- // We could use 1 socketpair, but tokio wraps them all in `File` internally causing `close`
- // to be called separately on all of them without exception, so we need 3 pipes :-(
-
- let mut child = command
- .stdin(Stdio::piped())
- .stdout(Stdio::piped())
- .stderr(Stdio::piped())
- .spawn()?;
-
- let mut stdin = child.stdin.take().expect("Stdio::piped()");
- let stdout = child.stdout.take().expect("Stdio::piped() failed?");
- let stdout = pipe_to_tasklog(stdout, Arc::clone(&task));
- let stderr = child.stderr.take().expect("Stdio::piped() failed?");
- let stderr = pipe_to_tasklog(stderr, Arc::clone(&task));
- let stdin = async move {
- stdin.write_all(&stdin_data).await?;
- stdin.flush().await?;
- Ok::<_, std::io::Error>(())
- };
- match futures::try_join!(stdin, stdout, stderr) {
- Ok(((), (), ())) => (),
- Err(err) => {
- if let Err(err) = child.kill().await {
- task.log_message(format!(
- "failed to kill '{PROXMOX_ACME_SH_PATH} {action}' command: {err}"
- ));
- }
- bail!("'{}' failed: {}", PROXMOX_ACME_SH_PATH, err);
- }
- }
-
- let status = child.wait().await?;
- if !status.success() {
- bail!(
- "'{} {}' exited with error ({})",
- PROXMOX_ACME_SH_PATH,
- action,
- status.code().unwrap_or(-1)
- );
- }
-
- Ok(&challenge.url)
- }
-}
-
-impl AcmePlugin for DnsPlugin {
- fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- domain: &'d AcmeDomain,
- task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>> {
- Box::pin(async move {
- let result = self
- .action(client, authorization, domain, task.clone(), "setup")
- .await;
-
- let validation_delay = self.core.validation_delay.unwrap_or(30) as u64;
- if validation_delay > 0 {
- task.log_message(format!(
- "Sleeping {validation_delay} seconds to wait for TXT record propagation"
- ));
- tokio::time::sleep(Duration::from_secs(validation_delay)).await;
- }
- result
- })
- }
-
- fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- domain: &'d AcmeDomain,
- task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>> {
- Box::pin(async move {
- self.action(client, authorization, domain, task, "teardown")
- .await
- .map(drop)
- })
- }
-}
-
-#[derive(Default)]
-struct StandaloneServer {
- abort_handle: Option<futures::future::AbortHandle>,
-}
-
-// In case the "order_certificates" future gets dropped between setup & teardown, let's also cancel
-// the HTTP listener on Drop:
-impl Drop for StandaloneServer {
- fn drop(&mut self) {
- self.stop();
- }
-}
-
-impl StandaloneServer {
- fn stop(&mut self) {
- if let Some(abort) = self.abort_handle.take() {
- abort.abort();
- }
- }
-}
-
-async fn standalone_respond(
- req: Request<Incoming>,
- path: Arc<String>,
- key_auth: Arc<String>,
-) -> Result<Response<Full<Bytes>>, hyper::Error> {
- if req.method() == hyper::Method::GET && req.uri().path() == path.as_str() {
- Ok(Response::builder()
- .status(hyper::http::StatusCode::OK)
- .body(key_auth.as_bytes().to_vec().into())
- .unwrap())
- } else {
- Ok(Response::builder()
- .status(hyper::http::StatusCode::NOT_FOUND)
- .body("Not found.".into())
- .unwrap())
- }
-}
-
-impl AcmePlugin for StandaloneServer {
- fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- _domain: &'d AcmeDomain,
- _task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>> {
- Box::pin(async move {
- self.stop();
-
- let challenge = extract_challenge(authorization, "http-01")?;
- let token = challenge
- .token()
- .ok_or_else(|| format_err!("missing token in challenge"))?;
- let key_auth = Arc::new(client.key_authorization(token)?);
- let path = Arc::new(format!("/.well-known/acme-challenge/{token}"));
-
- // `[::]:80` first, then `*:80`
- let dual = SocketAddr::new(IpAddr::from([0u16; 8]), 80);
- let ipv4 = SocketAddr::new(IpAddr::from([0u8; 4]), 80);
- let incoming = TcpListener::bind(dual)
- .or_else(|_| TcpListener::bind(ipv4))
- .await?;
-
- let server = async move {
- loop {
- let key_auth = Arc::clone(&key_auth);
- let path = Arc::clone(&path);
- match incoming.accept().await {
- Ok((tcp, _)) => {
- let io = TokioIo::new(tcp);
- let service = service_fn(move |request| {
- standalone_respond(
- request,
- Arc::clone(&path),
- Arc::clone(&key_auth),
- )
- });
-
- tokio::task::spawn(async move {
- if let Err(err) =
- http1::Builder::new().serve_connection(io, service).await
- {
- println!("Error serving connection: {err:?}");
- }
- });
- }
- Err(err) => println!("Error accepting connection: {err:?}"),
- }
- }
- };
- let (future, abort) = futures::future::abortable(server);
- self.abort_handle = Some(abort);
- tokio::spawn(future);
-
- Ok(challenge.url.as_str())
- })
- }
-
- fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- _client: &'b mut AcmeClient,
- _authorization: &'c Authorization,
- _domain: &'d AcmeDomain,
- _task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>> {
- Box::pin(async move {
- if let Some(abort) = self.abort_handle.take() {
- abort.abort();
- }
- Ok(())
- })
- }
-}
diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.rs
index 31196715..2a645b4a 100644
--- a/src/api2/node/certificates.rs
+++ b/src/api2/node/certificates.rs
@@ -1,27 +1,19 @@
-use std::sync::Arc;
-use std::time::Duration;
-
use anyhow::{bail, format_err, Error};
use openssl::pkey::PKey;
use openssl::x509::X509;
use serde::{Deserialize, Serialize};
use tracing::info;
-use proxmox_router::list_subdirs_api_method;
-use proxmox_router::SubdirMap;
-use proxmox_router::{Permission, Router, RpcEnvironment};
-use proxmox_schema::api;
-
+use crate::server::send_certificate_renewal_mail;
use pbs_api_types::{NODE_SCHEMA, PRIV_SYS_MODIFY};
use pbs_buildcfg::configdir;
use pbs_tools::cert;
-use tracing::warn;
-
-use crate::api2::types::AcmeDomain;
-use crate::config::node::NodeConfig;
-use crate::server::send_certificate_renewal_mail;
-use proxmox_acme::async_client::AcmeClient;
+use proxmox_acme_api::AcmeDomain;
use proxmox_rest_server::WorkerTask;
+use proxmox_router::list_subdirs_api_method;
+use proxmox_router::SubdirMap;
+use proxmox_router::{Permission, Router, RpcEnvironment};
+use proxmox_schema::api;
pub const ROUTER: Router = Router::new()
.get(&list_subdirs_api_method!(SUBDIRS))
@@ -269,193 +261,6 @@ pub async fn delete_custom_certificate() -> Result<(), Error> {
Ok(())
}
-struct OrderedCertificate {
- certificate: hyper::body::Bytes,
- private_key_pem: Vec<u8>,
-}
-
-async fn order_certificate(
- worker: Arc<WorkerTask>,
- node_config: &NodeConfig,
-) -> Result<Option<OrderedCertificate>, Error> {
- use proxmox_acme::authorization::Status;
- use proxmox_acme::order::Identifier;
-
- let domains = node_config.acme_domains().try_fold(
- Vec::<AcmeDomain>::new(),
- |mut acc, domain| -> Result<_, Error> {
- let mut domain = domain?;
- domain.domain.make_ascii_lowercase();
- if let Some(alias) = &mut domain.alias {
- alias.make_ascii_lowercase();
- }
- acc.push(domain);
- Ok(acc)
- },
- )?;
-
- let get_domain_config = |domain: &str| {
- domains
- .iter()
- .find(|d| d.domain == domain)
- .ok_or_else(|| format_err!("no config for domain '{}'", domain))
- };
-
- if domains.is_empty() {
- info!("No domains configured to be ordered from an ACME server.");
- return Ok(None);
- }
-
- let (plugins, _) = crate::config::acme::plugin::config()?;
-
- let mut acme = node_config.acme_client().await?;
-
- info!("Placing ACME order");
- let order = acme
- .new_order(domains.iter().map(|d| d.domain.to_ascii_lowercase()))
- .await?;
- info!("Order URL: {}", order.location);
-
- let identifiers: Vec<String> = order
- .data
- .identifiers
- .iter()
- .map(|identifier| match identifier {
- Identifier::Dns(domain) => domain.clone(),
- })
- .collect();
-
- for auth_url in &order.data.authorizations {
- info!("Getting authorization details from '{auth_url}'");
- let mut auth = acme.get_authorization(auth_url).await?;
-
- let domain = match &mut auth.identifier {
- Identifier::Dns(domain) => domain.to_ascii_lowercase(),
- };
-
- if auth.status == Status::Valid {
- info!("{domain} is already validated!");
- continue;
- }
-
- info!("The validation for {domain} is pending");
- let domain_config: &AcmeDomain = get_domain_config(&domain)?;
- let plugin_id = domain_config.plugin.as_deref().unwrap_or("standalone");
- let mut plugin_cfg = crate::acme::get_acme_plugin(&plugins, plugin_id)?
- .ok_or_else(|| format_err!("plugin '{plugin_id}' for domain '{domain}' not found!"))?;
-
- info!("Setting up validation plugin");
- let validation_url = plugin_cfg
- .setup(&mut acme, &auth, domain_config, Arc::clone(&worker))
- .await?;
-
- let result = request_validation(&mut acme, auth_url, validation_url).await;
-
- if let Err(err) = plugin_cfg
- .teardown(&mut acme, &auth, domain_config, Arc::clone(&worker))
- .await
- {
- warn!("Failed to teardown plugin '{plugin_id}' for domain '{domain}' - {err}");
- }
-
- result?;
- }
-
- info!("All domains validated");
- info!("Creating CSR");
-
- let csr = proxmox_acme::util::Csr::generate(&identifiers, &Default::default())?;
- let mut finalize_error_cnt = 0u8;
- let order_url = &order.location;
- let mut order;
- loop {
- use proxmox_acme::order::Status;
-
- order = acme.get_order(order_url).await?;
-
- match order.status {
- Status::Pending => {
- info!("still pending, trying to finalize anyway");
- let finalize = order
- .finalize
- .as_deref()
- .ok_or_else(|| format_err!("missing 'finalize' URL in order"))?;
- if let Err(err) = acme.finalize(finalize, &csr.data).await {
- if finalize_error_cnt >= 5 {
- return Err(err);
- }
-
- finalize_error_cnt += 1;
- }
- tokio::time::sleep(Duration::from_secs(5)).await;
- }
- Status::Ready => {
- info!("order is ready, finalizing");
- let finalize = order
- .finalize
- .as_deref()
- .ok_or_else(|| format_err!("missing 'finalize' URL in order"))?;
- acme.finalize(finalize, &csr.data).await?;
- tokio::time::sleep(Duration::from_secs(5)).await;
- }
- Status::Processing => {
- info!("still processing, trying again in 30 seconds");
- tokio::time::sleep(Duration::from_secs(30)).await;
- }
- Status::Valid => {
- info!("valid");
- break;
- }
- other => bail!("order status: {:?}", other),
- }
- }
-
- info!("Downloading certificate");
- let certificate = acme
- .get_certificate(
- order
- .certificate
- .as_deref()
- .ok_or_else(|| format_err!("missing certificate url in finalized order"))?,
- )
- .await?;
-
- Ok(Some(OrderedCertificate {
- certificate,
- private_key_pem: csr.private_key_pem,
- }))
-}
-
-async fn request_validation(
- acme: &mut AcmeClient,
- auth_url: &str,
- validation_url: &str,
-) -> Result<(), Error> {
- info!("Triggering validation");
- acme.request_challenge_validation(validation_url).await?;
-
- info!("Sleeping for 5 seconds");
- tokio::time::sleep(Duration::from_secs(5)).await;
-
- loop {
- use proxmox_acme::authorization::Status;
-
- let auth = acme.get_authorization(auth_url).await?;
- match auth.status {
- Status::Pending => {
- info!("Status is still 'pending', trying again in 10 seconds");
- tokio::time::sleep(Duration::from_secs(10)).await;
- }
- Status::Valid => return Ok(()),
- other => bail!(
- "validating challenge '{}' failed - status: {:?}",
- validation_url,
- other
- ),
- }
- }
-}
-
#[api(
input: {
properties: {
@@ -525,9 +330,30 @@ fn spawn_certificate_worker(
let auth_id = rpcenv.get_auth_id().unwrap();
+ let acme_config = if let Some(cfg) = node_config.acme_config().transpose()? {
+ cfg
+ } else {
+ proxmox_acme_api::parse_acme_config_string("account=default")?
+ };
+
+ let domains = node_config.acme_domains().try_fold(
+ Vec::<AcmeDomain>::new(),
+ |mut acc, domain| -> Result<_, Error> {
+ let mut domain = domain?;
+ domain.domain.make_ascii_lowercase();
+ if let Some(alias) = &mut domain.alias {
+ alias.make_ascii_lowercase();
+ }
+ acc.push(domain);
+ Ok(acc)
+ },
+ )?;
+
WorkerTask::spawn(name, None, auth_id, true, move |worker| async move {
let work = || async {
- if let Some(cert) = order_certificate(worker, &node_config).await? {
+ if let Some(cert) =
+ proxmox_acme_api::order_certificate(worker, &acme_config, &domains).await?
+ {
crate::config::set_proxy_certificate(&cert.certificate, &cert.private_key_pem)?;
crate::server::reload_proxy_certificate().await?;
}
@@ -563,16 +389,20 @@ pub fn revoke_acme_cert(rpcenv: &mut dyn RpcEnvironment) -> Result<String, Error
let auth_id = rpcenv.get_auth_id().unwrap();
+ let acme_config = if let Some(cfg) = node_config.acme_config().transpose()? {
+ cfg
+ } else {
+ proxmox_acme_api::parse_acme_config_string("account=default")?
+ };
+
WorkerTask::spawn(
"acme-revoke-cert",
None,
auth_id,
true,
move |_worker| async move {
- info!("Loading ACME account");
- let mut acme = node_config.acme_client().await?;
info!("Revoking old certificate");
- acme.revoke_certificate(cert_pem.as_bytes(), None).await?;
+ proxmox_acme_api::revoke_certificate(&acme_config, &cert_pem.as_bytes()).await?;
info!("Deleting certificate and regenerating a self-signed one");
delete_custom_certificate().await?;
Ok(())
diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
deleted file mode 100644
index 2905b41b..00000000
--- a/src/api2/types/acme.rs
+++ /dev/null
@@ -1,74 +0,0 @@
-use serde::{Deserialize, Serialize};
-use serde_json::Value;
-
-use proxmox_schema::{api, ApiStringFormat, ApiType, Schema, StringSchema};
-
-use pbs_api_types::{DNS_ALIAS_FORMAT, DNS_NAME_FORMAT, PROXMOX_SAFE_ID_FORMAT};
-
-#[api(
- properties: {
- "domain": { format: &DNS_NAME_FORMAT },
- "alias": {
- optional: true,
- format: &DNS_ALIAS_FORMAT,
- },
- "plugin": {
- optional: true,
- format: &PROXMOX_SAFE_ID_FORMAT,
- },
- },
- default_key: "domain",
-)]
-#[derive(Deserialize, Serialize)]
-/// A domain entry for an ACME certificate.
-pub struct AcmeDomain {
- /// The domain to certify for.
- pub domain: String,
-
- /// The domain to use for challenges instead of the default acme challenge domain.
- ///
- /// This is useful if you use CNAME entries to redirect `_acme-challenge.*` domains to a
- /// different DNS server.
- #[serde(skip_serializing_if = "Option::is_none")]
- pub alias: Option<String>,
-
- /// The plugin to use to validate this domain.
- ///
- /// Empty means standalone HTTP validation is used.
- #[serde(skip_serializing_if = "Option::is_none")]
- pub plugin: Option<String>,
-}
-
-pub const ACME_DOMAIN_PROPERTY_SCHEMA: Schema =
- StringSchema::new("ACME domain configuration string")
- .format(&ApiStringFormat::PropertyString(&AcmeDomain::API_SCHEMA))
- .schema();
-
-#[api(
- properties: {
- schema: {
- type: Object,
- additional_properties: true,
- properties: {},
- },
- type: {
- type: String,
- },
- },
-)]
-#[derive(Serialize)]
-/// Schema for an ACME challenge plugin.
-pub struct AcmeChallengeSchema {
- /// Plugin ID.
- pub id: String,
-
- /// Human readable name, falls back to id.
- pub name: String,
-
- /// Plugin Type.
- #[serde(rename = "type")]
- pub ty: &'static str,
-
- /// The plugin's parameter schema.
- pub schema: Value,
-}
diff --git a/src/api2/types/mod.rs b/src/api2/types/mod.rs
index afc34b30..34193685 100644
--- a/src/api2/types/mod.rs
+++ b/src/api2/types/mod.rs
@@ -4,9 +4,6 @@ use anyhow::bail;
use proxmox_schema::*;
-mod acme;
-pub use acme::*;
-
// File names: may not contain slashes, may not start with "."
pub const FILENAME_FORMAT: ApiStringFormat = ApiStringFormat::VerifyFn(|name| {
if name.starts_with('.') {
diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
index 35cda50b..afd7abf8 100644
--- a/src/config/acme/mod.rs
+++ b/src/config/acme/mod.rs
@@ -9,8 +9,7 @@ use proxmox_sys::fs::{file_read_string, CreateOptions};
use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
-use crate::api2::types::AcmeChallengeSchema;
-use proxmox_acme_api::{AcmeAccountName, KnownAcmeDirectory, KNOWN_ACME_DIRECTORIES};
+use proxmox_acme_api::{AcmeAccountName, AcmeChallengeSchema};
pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
@@ -35,8 +34,6 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
create_acme_subdir(ACME_DIR)
}
-pub const DEFAULT_ACME_DIRECTORY_ENTRY: &KnownAcmeDirectory = &KNOWN_ACME_DIRECTORIES[0];
-
pub fn foreach_acme_account<F>(mut func: F) -> Result<(), Error>
where
F: FnMut(AcmeAccountName) -> ControlFlow<Result<(), Error>>,
@@ -80,7 +77,7 @@ pub fn load_dns_challenge_schema() -> Result<Vec<AcmeChallengeSchema>, Error> {
.and_then(Value::as_str)
.unwrap_or(id)
.to_owned(),
- ty: "dns",
+ ty: "dns".into(),
schema: schema.to_owned(),
})
.collect())
diff --git a/src/config/acme/plugin.rs b/src/config/acme/plugin.rs
index 18e71199..2e979ffe 100644
--- a/src/config/acme/plugin.rs
+++ b/src/config/acme/plugin.rs
@@ -1,104 +1,15 @@
use std::sync::LazyLock;
use anyhow::Error;
-use serde::{Deserialize, Serialize};
-use serde_json::Value;
-
-use proxmox_schema::{api, ApiType, Schema, StringSchema, Updater};
-use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
-
-use pbs_api_types::PROXMOX_SAFE_ID_FORMAT;
use pbs_config::{open_backup_lockfile, BackupLockGuard};
-
-pub const PLUGIN_ID_SCHEMA: Schema = StringSchema::new("ACME Challenge Plugin ID.")
- .format(&PROXMOX_SAFE_ID_FORMAT)
- .min_length(1)
- .max_length(32)
- .schema();
+use proxmox_acme_api::PLUGIN_ID_SCHEMA;
+use proxmox_acme_api::{DnsPlugin, StandalonePlugin};
+use proxmox_schema::{ApiType, Schema};
+use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
+use serde_json::Value;
pub static CONFIG: LazyLock<SectionConfig> = LazyLock::new(init);
-#[api(
- properties: {
- id: { schema: PLUGIN_ID_SCHEMA },
- },
-)]
-#[derive(Deserialize, Serialize)]
-/// Standalone ACME Plugin for the http-1 challenge.
-pub struct StandalonePlugin {
- /// Plugin ID.
- id: String,
-}
-
-impl Default for StandalonePlugin {
- fn default() -> Self {
- Self {
- id: "standalone".to_string(),
- }
- }
-}
-
-#[api(
- properties: {
- id: { schema: PLUGIN_ID_SCHEMA },
- disable: {
- optional: true,
- default: false,
- },
- "validation-delay": {
- default: 30,
- optional: true,
- minimum: 0,
- maximum: 2 * 24 * 60 * 60,
- },
- },
-)]
-/// DNS ACME Challenge Plugin core data.
-#[derive(Deserialize, Serialize, Updater)]
-#[serde(rename_all = "kebab-case")]
-pub struct DnsPluginCore {
- /// Plugin ID.
- #[updater(skip)]
- pub id: String,
-
- /// DNS API Plugin Id.
- pub api: String,
-
- /// Extra delay in seconds to wait before requesting validation.
- ///
- /// Allows to cope with long TTL of DNS records.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- pub validation_delay: Option<u32>,
-
- /// Flag to disable the config.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- pub disable: Option<bool>,
-}
-
-#[api(
- properties: {
- core: { type: DnsPluginCore },
- },
-)]
-/// DNS ACME Challenge Plugin.
-#[derive(Deserialize, Serialize)]
-#[serde(rename_all = "kebab-case")]
-pub struct DnsPlugin {
- #[serde(flatten)]
- pub core: DnsPluginCore,
-
- // We handle this property separately in the API calls.
- /// DNS plugin data (base64url encoded without padding).
- #[serde(with = "proxmox_serde::string_as_base64url_nopad")]
- pub data: String,
-}
-
-impl DnsPlugin {
- pub fn decode_data(&self, output: &mut Vec<u8>) -> Result<(), Error> {
- Ok(proxmox_base64::url::decode_to_vec(&self.data, output)?)
- }
-}
-
fn init() -> SectionConfig {
let mut config = SectionConfig::new(&PLUGIN_ID_SCHEMA);
diff --git a/src/config/node.rs b/src/config/node.rs
index d2a17a49..b9257adf 100644
--- a/src/config/node.rs
+++ b/src/config/node.rs
@@ -6,17 +6,17 @@ use serde::{Deserialize, Serialize};
use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
-use proxmox_http::ProxyConfig;
-
use pbs_api_types::{
EMAIL_SCHEMA, MULTI_LINE_COMMENT_SCHEMA, OPENSSL_CIPHERS_TLS_1_2_SCHEMA,
OPENSSL_CIPHERS_TLS_1_3_SCHEMA,
};
+use proxmox_acme_api::{AcmeConfig, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA};
+use proxmox_http::ProxyConfig;
use pbs_buildcfg::configdir;
use pbs_config::{open_backup_lockfile, BackupLockGuard};
-use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
+use crate::api2::types::HTTP_PROXY_SCHEMA;
use proxmox_acme::async_client::AcmeClient;
use proxmox_acme_api::AcmeAccountName;
@@ -45,20 +45,6 @@ pub fn save_config(config: &NodeConfig) -> Result<(), Error> {
pbs_config::replace_backup_config(CONF_FILE, &raw)
}
-#[api(
- properties: {
- account: { type: AcmeAccountName },
- }
-)]
-#[derive(Deserialize, Serialize)]
-/// The ACME configuration.
-///
-/// Currently only contains the name of the account use.
-pub struct AcmeConfig {
- /// Account to use to acquire ACME certificates.
- account: AcmeAccountName,
-}
-
/// All available languages in Proxmox. Taken from proxmox-i18n repository.
/// pt_BR, zh_CN, and zh_TW use the same case in the translation files.
// TODO: auto-generate from available translations
@@ -244,7 +230,7 @@ impl NodeConfig {
pub async fn acme_client(&self) -> Result<AcmeClient, Error> {
let account = if let Some(cfg) = self.acme_config().transpose()? {
- cfg.account
+ AcmeAccountName::from_string(cfg.account)?
} else {
AcmeAccountName::from_string("default".to_string())? // should really not happen
};
diff --git a/src/lib.rs b/src/lib.rs
index 8633378c..828f5842 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -27,8 +27,6 @@ pub(crate) mod auth;
pub mod tape;
-pub mod acme;
-
pub mod client_helpers;
pub mod traffic_control_cache;
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 7%]
* [pbs-devel] [PATCH proxmox-backup v4 1/4] acme: include proxmox-acme-api dependency
2025-12-03 10:22 11% [pbs-devel] [PATCH proxmox{-backup, } v4 " Samuel Rufinatscha
@ 2025-12-03 10:22 15% ` Samuel Rufinatscha
2025-12-03 10:22 6% ` [pbs-devel] [PATCH proxmox-backup v4 2/4] acme: drop local AcmeClient Samuel Rufinatscha
` (8 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-03 10:22 UTC (permalink / raw)
To: pbs-devel
PBS currently uses its own ACME client and API logic, while PDM uses the
factored out proxmox-acme and proxmox-acme-api crates. This duplication
risks differences in behaviour and requires ACME maintenance in two
places. This patch is part of a series to move PBS over to the shared
ACME stack.
Changes:
- Add proxmox-acme-api with the "impl" feature as a dependency.
- Initialize proxmox_acme_api in proxmox-backup- api, manager and proxy.
* Inits PBS config dir /acme as proxmox ACME directory
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Cargo.toml | 3 +++
src/bin/proxmox-backup-api.rs | 2 ++
src/bin/proxmox-backup-manager.rs | 2 ++
src/bin/proxmox-backup-proxy.rs | 1 +
4 files changed, 8 insertions(+)
diff --git a/Cargo.toml b/Cargo.toml
index ff143932..bdaf7d85 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -101,6 +101,7 @@ pbs-api-types = "1.0.8"
# other proxmox crates
pathpatterns = "1"
proxmox-acme = "1"
+proxmox-acme-api = { version = "1", features = [ "impl" ] }
pxar = "1"
# PBS workspace
@@ -251,6 +252,7 @@ pbs-api-types.workspace = true
# in their respective repo
proxmox-acme.workspace = true
+proxmox-acme-api.workspace = true
pxar.workspace = true
# proxmox-backup workspace/internal crates
@@ -269,6 +271,7 @@ proxmox-rrd-api-types.workspace = true
[patch.crates-io]
#pbs-api-types = { path = "../proxmox/pbs-api-types" }
#proxmox-acme = { path = "../proxmox/proxmox-acme" }
+#proxmox-acme-api = { path = "../proxmox/proxmox-acme-api" }
#proxmox-api-macro = { path = "../proxmox/proxmox-api-macro" }
#proxmox-apt = { path = "../proxmox/proxmox-apt" }
#proxmox-apt-api-types = { path = "../proxmox/proxmox-apt-api-types" }
diff --git a/src/bin/proxmox-backup-api.rs b/src/bin/proxmox-backup-api.rs
index 417e9e97..48f10092 100644
--- a/src/bin/proxmox-backup-api.rs
+++ b/src/bin/proxmox-backup-api.rs
@@ -8,6 +8,7 @@ use hyper_util::server::graceful::GracefulShutdown;
use tokio::net::TcpListener;
use tracing::level_filters::LevelFilter;
+use pbs_buildcfg::configdir;
use proxmox_http::Body;
use proxmox_lang::try_block;
use proxmox_rest_server::{ApiConfig, RestServer};
@@ -78,6 +79,7 @@ async fn run() -> Result<(), Error> {
let mut command_sock = proxmox_daemon::command_socket::CommandSocket::new(backup_user.gid);
proxmox_product_config::init(backup_user.clone(), pbs_config::priv_user()?);
+ proxmox_acme_api::init(configdir!("/acme"), true)?;
let dir_opts = CreateOptions::new()
.owner(backup_user.uid)
diff --git a/src/bin/proxmox-backup-manager.rs b/src/bin/proxmox-backup-manager.rs
index d9f41353..0facb76c 100644
--- a/src/bin/proxmox-backup-manager.rs
+++ b/src/bin/proxmox-backup-manager.rs
@@ -18,6 +18,7 @@ use pbs_api_types::{
VERIFICATION_OUTDATED_AFTER_SCHEMA, VERIFY_JOB_READ_THREADS_SCHEMA,
VERIFY_JOB_VERIFY_THREADS_SCHEMA,
};
+use pbs_buildcfg::configdir;
use pbs_client::{display_task_log, view_task_result};
use pbs_config::sync;
use pbs_tools::json::required_string_param;
@@ -669,6 +670,7 @@ async fn run() -> Result<(), Error> {
.init()?;
proxmox_backup::server::notifications::init()?;
proxmox_product_config::init(pbs_config::backup_user()?, pbs_config::priv_user()?);
+ proxmox_acme_api::init(configdir!("/acme"), false)?;
let cmd_def = CliCommandMap::new()
.insert("acl", acl_commands())
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 92a8cb3c..0bab18ec 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -190,6 +190,7 @@ async fn run() -> Result<(), Error> {
proxmox_backup::server::notifications::init()?;
metric_collection::init()?;
proxmox_product_config::init(pbs_config::backup_user()?, pbs_config::priv_user()?);
+ proxmox_acme_api::init(configdir!("/acme"), false)?;
let mut indexpath = PathBuf::from(pbs_buildcfg::JS_DIR);
indexpath.push("index.hbs");
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox v4 1/4] acme-api: add helper to load client for an account
2025-12-03 10:22 11% [pbs-devel] [PATCH proxmox{-backup, } v4 " Samuel Rufinatscha
` (3 preceding siblings ...)
2025-12-03 10:22 7% ` [pbs-devel] [PATCH proxmox-backup v4 4/4] acme: certificate ordering through proxmox-acme-api Samuel Rufinatscha
@ 2025-12-03 10:22 17% ` Samuel Rufinatscha
2025-12-09 16:51 5% ` Max R. Carrara
2025-12-03 10:22 12% ` [pbs-devel] [PATCH proxmox v4 2/4] acme: reduce visibility of Request type Samuel Rufinatscha
` (4 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-12-03 10:22 UTC (permalink / raw)
To: pbs-devel
The PBS ACME refactoring needs a simple way to obtain an AcmeClient for
a given configured account without duplicating config wiring. This patch
adds a load_client_with_account helper in proxmox-acme-api that loads
the account and constructs a matching client, similarly as PBS previous
own AcmeClient::load() function.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme-api/src/account_api_impl.rs | 5 +++++
proxmox-acme-api/src/lib.rs | 3 ++-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/proxmox-acme-api/src/account_api_impl.rs b/proxmox-acme-api/src/account_api_impl.rs
index ef195908..ca8c8655 100644
--- a/proxmox-acme-api/src/account_api_impl.rs
+++ b/proxmox-acme-api/src/account_api_impl.rs
@@ -116,3 +116,8 @@ pub async fn update_account(name: &AcmeAccountName, contact: Option<String>) ->
Ok(())
}
+
+pub async fn load_client_with_account(account_name: &AcmeAccountName) -> Result<AcmeClient, Error> {
+ let account_data = super::account_config::load_account_config(&account_name).await?;
+ Ok(account_data.client())
+}
diff --git a/proxmox-acme-api/src/lib.rs b/proxmox-acme-api/src/lib.rs
index 623e9e23..96f88ae2 100644
--- a/proxmox-acme-api/src/lib.rs
+++ b/proxmox-acme-api/src/lib.rs
@@ -31,7 +31,8 @@ mod plugin_config;
mod account_api_impl;
#[cfg(feature = "impl")]
pub use account_api_impl::{
- deactivate_account, get_account, get_tos, list_accounts, register_account, update_account,
+ deactivate_account, get_account, get_tos, list_accounts, load_client_with_account,
+ register_account, update_account,
};
#[cfg(feature = "impl")]
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 17%]
* [pbs-devel] [PATCH proxmox v4 4/4] fix #6939: acme: support servers returning 204 for nonce requests
2025-12-03 10:22 11% [pbs-devel] [PATCH proxmox{-backup, } v4 " Samuel Rufinatscha
` (6 preceding siblings ...)
2025-12-03 10:22 15% ` [pbs-devel] [PATCH proxmox v4 3/4] acme: introduce http_status module Samuel Rufinatscha
@ 2025-12-03 10:22 14% ` Samuel Rufinatscha
2025-12-09 16:50 5% ` [pbs-devel] [PATCH proxmox{-backup, } v4 0/8] " Max R. Carrara
2026-01-08 11:48 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
9 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-03 10:22 UTC (permalink / raw)
To: pbs-devel
Some ACME servers (notably custom or legacy implementations) respond
to HEAD /newNonce with a 204 No Content instead of the
RFC 8555-recommended 200 OK [1]. While this behavior is technically
off-spec, it is not illegal. This issue was reported on our bug
tracker [2].
The previous implementation treated any non-200 response as an error,
causing account registration to fail against such servers. Relax the
status-code check to accept both 200 and 204 responses (and potentially
support other 2xx codes) to improve interoperability.
Note: In comparison, PVE’s Perl ACME client performs a GET request [3]
instead of a HEAD request and accepts any 2xx success code when
retrieving the nonce [4]. This difference in behavior does not affect
functionality but is worth noting for consistency across
implementations.
[1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
[3] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219
[4] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597
Fixes: #6939
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme/src/account.rs | 10 +++++-----
proxmox-acme/src/async_client.rs | 6 +++---
proxmox-acme/src/client.rs | 2 +-
proxmox-acme/src/request.rs | 4 ++--
4 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
index 350c78d4..820b209d 100644
--- a/proxmox-acme/src/account.rs
+++ b/proxmox-acme/src/account.rs
@@ -85,7 +85,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::CREATED,
+ expected: &[crate::http_status::CREATED],
};
Ok(NewOrder::new(request))
@@ -107,7 +107,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK],
})
}
@@ -132,7 +132,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK],
})
}
@@ -157,7 +157,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK],
})
}
@@ -408,7 +408,7 @@ impl AccountCreator {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::CREATED,
+ expected: &[crate::http_status::CREATED],
})
}
diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
index 043648bb..07da842c 100644
--- a/proxmox-acme/src/async_client.rs
+++ b/proxmox-acme/src/async_client.rs
@@ -420,7 +420,7 @@ impl AcmeClient {
};
if parts.status.is_success() {
- if status != request.expected {
+ if !request.expected.contains(&status) {
return Err(Error::InvalidApi(format!(
"ACME server responded with unexpected status code: {:?}",
parts.status
@@ -498,7 +498,7 @@ impl AcmeClient {
method: "GET",
content_type: "",
body: String::new(),
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK],
},
nonce,
)
@@ -550,7 +550,7 @@ impl AcmeClient {
method: "HEAD",
content_type: "",
body: String::new(),
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK, crate::http_status::NO_CONTENT],
},
nonce,
)
diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
index 5c812567..af250fb8 100644
--- a/proxmox-acme/src/client.rs
+++ b/proxmox-acme/src/client.rs
@@ -203,7 +203,7 @@ impl Inner {
let got_nonce = self.update_nonce(&mut response)?;
if response.is_success() {
- if response.status != request.expected {
+ if !request.expected.contains(&response.status) {
return Err(Error::InvalidApi(format!(
"API server responded with unexpected status code: {:?}",
response.status
diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
index 341ce53e..d782a7de 100644
--- a/proxmox-acme/src/request.rs
+++ b/proxmox-acme/src/request.rs
@@ -16,8 +16,8 @@ pub(crate) struct Request {
/// The body to pass along with request, or an empty string.
pub(crate) body: String,
- /// The expected status code a compliant ACME provider will return on success.
- pub(crate) expected: u16,
+ /// The set of HTTP status codes that indicate a successful response from an ACME provider.
+ pub(crate) expected: &'static [u16],
}
/// Common HTTP status codes used in ACME responses.
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 14%]
* [pbs-devel] [PATCH proxmox-backup v4 3/4] acme: change API impls to use proxmox-acme-api handlers
2025-12-03 10:22 11% [pbs-devel] [PATCH proxmox{-backup, } v4 " Samuel Rufinatscha
2025-12-03 10:22 15% ` [pbs-devel] [PATCH proxmox-backup v4 1/4] acme: include proxmox-acme-api dependency Samuel Rufinatscha
2025-12-03 10:22 6% ` [pbs-devel] [PATCH proxmox-backup v4 2/4] acme: drop local AcmeClient Samuel Rufinatscha
@ 2025-12-03 10:22 8% ` Samuel Rufinatscha
2025-12-09 16:50 5% ` Max R. Carrara
2025-12-03 10:22 7% ` [pbs-devel] [PATCH proxmox-backup v4 4/4] acme: certificate ordering through proxmox-acme-api Samuel Rufinatscha
` (6 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-12-03 10:22 UTC (permalink / raw)
To: pbs-devel
PBS currently uses its own ACME client and API logic, while PDM uses the
factored out proxmox-acme and proxmox-acme-api crates. This duplication
risks differences in behaviour and requires ACME maintenance in two
places. This patch is part of a series to move PBS over to the shared
ACME stack.
Changes:
- Replace api2/config/acme.rs API logic with proxmox-acme-api handlers.
- Drop local caching and helper types that duplicate proxmox-acme-api.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/api2/config/acme.rs | 385 ++-----------------------
src/api2/types/acme.rs | 16 -
src/bin/proxmox_backup_manager/acme.rs | 6 +-
src/config/acme/mod.rs | 44 +--
4 files changed, 35 insertions(+), 416 deletions(-)
diff --git a/src/api2/config/acme.rs b/src/api2/config/acme.rs
index 02f88e2e..a112c8ee 100644
--- a/src/api2/config/acme.rs
+++ b/src/api2/config/acme.rs
@@ -1,31 +1,17 @@
-use std::fs;
-use std::ops::ControlFlow;
-use std::path::Path;
-use std::sync::{Arc, LazyLock, Mutex};
-use std::time::SystemTime;
-
-use anyhow::{bail, format_err, Error};
-use hex::FromHex;
-use serde::{Deserialize, Serialize};
-use serde_json::{json, Value};
-use tracing::{info, warn};
-
-use proxmox_router::{
- http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
-};
-use proxmox_schema::{api, param_bail};
-
-use proxmox_acme::types::AccountData as AcmeAccountData;
-
+use anyhow::Error;
use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
-
-use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
-use crate::config::acme::plugin::{
- self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
+use proxmox_acme_api::{
+ AccountEntry, AccountInfo, AcmeAccountName, AcmeChallengeSchema, ChallengeSchemaWrapper,
+ DeletablePluginProperty, DnsPluginCore, DnsPluginCoreUpdater, KnownAcmeDirectory, PluginConfig,
+ DEFAULT_ACME_DIRECTORY_ENTRY, PLUGIN_ID_SCHEMA,
};
-use proxmox_acme::async_client::AcmeClient;
-use proxmox_acme_api::AcmeAccountName;
+use proxmox_config_digest::ConfigDigest;
use proxmox_rest_server::WorkerTask;
+use proxmox_router::{
+ http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
+};
+use proxmox_schema::api;
+use tracing::info;
pub(crate) const ROUTER: Router = Router::new()
.get(&list_subdirs_api_method!(SUBDIRS))
@@ -67,19 +53,6 @@ const PLUGIN_ITEM_ROUTER: Router = Router::new()
.put(&API_METHOD_UPDATE_PLUGIN)
.delete(&API_METHOD_DELETE_PLUGIN);
-#[api(
- properties: {
- name: { type: AcmeAccountName },
- },
-)]
-/// An ACME Account entry.
-///
-/// Currently only contains a 'name' property.
-#[derive(Serialize)]
-pub struct AccountEntry {
- name: AcmeAccountName,
-}
-
#[api(
access: {
permission: &Permission::Privilege(&["system", "certificates"], PRIV_SYS_MODIFY, false),
@@ -93,40 +66,7 @@ pub struct AccountEntry {
)]
/// List ACME accounts.
pub fn list_accounts() -> Result<Vec<AccountEntry>, Error> {
- let mut entries = Vec::new();
- crate::config::acme::foreach_acme_account(|name| {
- entries.push(AccountEntry { name });
- ControlFlow::Continue(())
- })?;
- Ok(entries)
-}
-
-#[api(
- properties: {
- account: { type: Object, properties: {}, additional_properties: true },
- tos: {
- type: String,
- optional: true,
- },
- },
-)]
-/// ACME Account information.
-///
-/// This is what we return via the API.
-#[derive(Serialize)]
-pub struct AccountInfo {
- /// Raw account data.
- account: AcmeAccountData,
-
- /// The ACME directory URL the account was created at.
- directory: String,
-
- /// The account's own URL within the ACME directory.
- location: String,
-
- /// The ToS URL, if the user agreed to one.
- #[serde(skip_serializing_if = "Option::is_none")]
- tos: Option<String>,
+ proxmox_acme_api::list_accounts()
}
#[api(
@@ -143,23 +83,7 @@ pub struct AccountInfo {
)]
/// Return existing ACME account information.
pub async fn get_account(name: AcmeAccountName) -> Result<AccountInfo, Error> {
- let account_info = proxmox_acme_api::get_account(name).await?;
-
- Ok(AccountInfo {
- location: account_info.location,
- tos: account_info.tos,
- directory: account_info.directory,
- account: AcmeAccountData {
- only_return_existing: false, // don't actually write this out in case it's set
- ..account_info.account
- },
- })
-}
-
-fn account_contact_from_string(s: &str) -> Vec<String> {
- s.split(&[' ', ';', ',', '\0'][..])
- .map(|s| format!("mailto:{s}"))
- .collect()
+ proxmox_acme_api::get_account(name).await
}
#[api(
@@ -224,15 +148,11 @@ fn register_account(
);
}
- if Path::new(&crate::config::acme::account_path(&name)).exists() {
+ if std::path::Path::new(&proxmox_acme_api::account_config_filename(&name)).exists() {
http_bail!(BAD_REQUEST, "account {} already exists", name);
}
- let directory = directory.unwrap_or_else(|| {
- crate::config::acme::DEFAULT_ACME_DIRECTORY_ENTRY
- .url
- .to_owned()
- });
+ let directory = directory.unwrap_or_else(|| DEFAULT_ACME_DIRECTORY_ENTRY.url.to_string());
WorkerTask::spawn(
"acme-register",
@@ -288,17 +208,7 @@ pub fn update_account(
auth_id.to_string(),
true,
move |_worker| async move {
- let data = match contact {
- Some(data) => json!({
- "contact": account_contact_from_string(&data),
- }),
- None => json!({}),
- };
-
- proxmox_acme_api::load_client_with_account(&name)
- .await?
- .update_account(&data)
- .await?;
+ proxmox_acme_api::update_account(&name, contact).await?;
Ok(())
},
@@ -336,18 +246,8 @@ pub fn deactivate_account(
auth_id.to_string(),
true,
move |_worker| async move {
- match proxmox_acme_api::load_client_with_account(&name)
- .await?
- .update_account(&json!({"status": "deactivated"}))
- .await
- {
- Ok(_account) => (),
- Err(err) if !force => return Err(err),
- Err(err) => {
- warn!("error deactivating account {name}, proceeding anyway - {err}");
- }
- }
- crate::config::acme::mark_account_deactivated(&name)?;
+ proxmox_acme_api::deactivate_account(&name, force).await?;
+
Ok(())
},
)
@@ -374,15 +274,7 @@ pub fn deactivate_account(
)]
/// Get the Terms of Service URL for an ACME directory.
async fn get_tos(directory: Option<String>) -> Result<Option<String>, Error> {
- let directory = directory.unwrap_or_else(|| {
- crate::config::acme::DEFAULT_ACME_DIRECTORY_ENTRY
- .url
- .to_owned()
- });
- Ok(AcmeClient::new(directory)
- .terms_of_service_url()
- .await?
- .map(str::to_owned))
+ proxmox_acme_api::get_tos(directory).await
}
#[api(
@@ -397,52 +289,7 @@ async fn get_tos(directory: Option<String>) -> Result<Option<String>, Error> {
)]
/// Get named known ACME directory endpoints.
fn get_directories() -> Result<&'static [KnownAcmeDirectory], Error> {
- Ok(crate::config::acme::KNOWN_ACME_DIRECTORIES)
-}
-
-/// Wrapper for efficient Arc use when returning the ACME challenge-plugin schema for serializing
-struct ChallengeSchemaWrapper {
- inner: Arc<Vec<AcmeChallengeSchema>>,
-}
-
-impl Serialize for ChallengeSchemaWrapper {
- fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
- where
- S: serde::Serializer,
- {
- self.inner.serialize(serializer)
- }
-}
-
-struct CachedSchema {
- schema: Arc<Vec<AcmeChallengeSchema>>,
- cached_mtime: SystemTime,
-}
-
-fn get_cached_challenge_schemas() -> Result<ChallengeSchemaWrapper, Error> {
- static CACHE: LazyLock<Mutex<Option<CachedSchema>>> = LazyLock::new(|| Mutex::new(None));
-
- // the actual loading code
- let mut last = CACHE.lock().unwrap();
-
- let actual_mtime = fs::metadata(crate::config::acme::ACME_DNS_SCHEMA_FN)?.modified()?;
-
- let schema = match &*last {
- Some(CachedSchema {
- schema,
- cached_mtime,
- }) if *cached_mtime >= actual_mtime => schema.clone(),
- _ => {
- let new_schema = Arc::new(crate::config::acme::load_dns_challenge_schema()?);
- *last = Some(CachedSchema {
- schema: Arc::clone(&new_schema),
- cached_mtime: actual_mtime,
- });
- new_schema
- }
- };
-
- Ok(ChallengeSchemaWrapper { inner: schema })
+ Ok(proxmox_acme_api::KNOWN_ACME_DIRECTORIES)
}
#[api(
@@ -457,69 +304,7 @@ fn get_cached_challenge_schemas() -> Result<ChallengeSchemaWrapper, Error> {
)]
/// Get named known ACME directory endpoints.
fn get_challenge_schema() -> Result<ChallengeSchemaWrapper, Error> {
- get_cached_challenge_schemas()
-}
-
-#[api]
-#[derive(Default, Deserialize, Serialize)]
-#[serde(rename_all = "kebab-case")]
-/// The API's format is inherited from PVE/PMG:
-pub struct PluginConfig {
- /// Plugin ID.
- plugin: String,
-
- /// Plugin type.
- #[serde(rename = "type")]
- ty: String,
-
- /// DNS Api name.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- api: Option<String>,
-
- /// Plugin configuration data.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- data: Option<String>,
-
- /// Extra delay in seconds to wait before requesting validation.
- ///
- /// Allows to cope with long TTL of DNS records.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- validation_delay: Option<u32>,
-
- /// Flag to disable the config.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- disable: Option<bool>,
-}
-
-// See PMG/PVE's $modify_cfg_for_api sub
-fn modify_cfg_for_api(id: &str, ty: &str, data: &Value) -> PluginConfig {
- let mut entry = data.clone();
-
- let obj = entry.as_object_mut().unwrap();
- obj.remove("id");
- obj.insert("plugin".to_string(), Value::String(id.to_owned()));
- obj.insert("type".to_string(), Value::String(ty.to_owned()));
-
- // FIXME: This needs to go once the `Updater` is fixed.
- // None of these should be able to fail unless the user changed the files by hand, in which
- // case we leave the unmodified string in the Value for now. This will be handled with an error
- // later.
- if let Some(Value::String(ref mut data)) = obj.get_mut("data") {
- if let Ok(new) = proxmox_base64::url::decode_no_pad(&data) {
- if let Ok(utf8) = String::from_utf8(new) {
- *data = utf8;
- }
- }
- }
-
- // PVE/PMG do this explicitly for ACME plugins...
- // obj.insert("digest".to_string(), Value::String(digest.clone()));
-
- serde_json::from_value(entry).unwrap_or_else(|_| PluginConfig {
- plugin: "*Error*".to_string(),
- ty: "*Error*".to_string(),
- ..Default::default()
- })
+ proxmox_acme_api::get_cached_challenge_schemas()
}
#[api(
@@ -535,12 +320,7 @@ fn modify_cfg_for_api(id: &str, ty: &str, data: &Value) -> PluginConfig {
)]
/// List ACME challenge plugins.
pub fn list_plugins(rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<PluginConfig>, Error> {
- let (plugins, digest) = plugin::config()?;
- rpcenv["digest"] = hex::encode(digest).into();
- Ok(plugins
- .iter()
- .map(|(id, (ty, data))| modify_cfg_for_api(id, ty, data))
- .collect())
+ proxmox_acme_api::list_plugins(rpcenv)
}
#[api(
@@ -557,13 +337,7 @@ pub fn list_plugins(rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<PluginConfig>
)]
/// List ACME challenge plugins.
pub fn get_plugin(id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<PluginConfig, Error> {
- let (plugins, digest) = plugin::config()?;
- rpcenv["digest"] = hex::encode(digest).into();
-
- match plugins.get(&id) {
- Some((ty, data)) => Ok(modify_cfg_for_api(&id, ty, data)),
- None => http_bail!(NOT_FOUND, "no such plugin"),
- }
+ proxmox_acme_api::get_plugin(id, rpcenv)
}
// Currently we only have "the" standalone plugin and DNS plugins so we can just flatten a
@@ -595,30 +369,7 @@ pub fn get_plugin(id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<PluginC
)]
/// Add ACME plugin configuration.
pub fn add_plugin(r#type: String, core: DnsPluginCore, data: String) -> Result<(), Error> {
- // Currently we only support DNS plugins and the standalone plugin is "fixed":
- if r#type != "dns" {
- param_bail!("type", "invalid ACME plugin type: {:?}", r#type);
- }
-
- let data = String::from_utf8(proxmox_base64::decode(data)?)
- .map_err(|_| format_err!("data must be valid UTF-8"))?;
-
- let id = core.id.clone();
-
- let _lock = plugin::lock()?;
-
- let (mut plugins, _digest) = plugin::config()?;
- if plugins.contains_key(&id) {
- param_bail!("id", "ACME plugin ID {:?} already exists", id);
- }
-
- let plugin = serde_json::to_value(DnsPlugin { core, data })?;
-
- plugins.insert(id, r#type, plugin);
-
- plugin::save_config(&plugins)?;
-
- Ok(())
+ proxmox_acme_api::add_plugin(r#type, core, data)
}
#[api(
@@ -634,26 +385,7 @@ pub fn add_plugin(r#type: String, core: DnsPluginCore, data: String) -> Result<(
)]
/// Delete an ACME plugin configuration.
pub fn delete_plugin(id: String) -> Result<(), Error> {
- let _lock = plugin::lock()?;
-
- let (mut plugins, _digest) = plugin::config()?;
- if plugins.remove(&id).is_none() {
- http_bail!(NOT_FOUND, "no such plugin");
- }
- plugin::save_config(&plugins)?;
-
- Ok(())
-}
-
-#[api()]
-#[derive(Serialize, Deserialize)]
-#[serde(rename_all = "kebab-case")]
-/// Deletable property name
-pub enum DeletableProperty {
- /// Delete the disable property
- Disable,
- /// Delete the validation-delay property
- ValidationDelay,
+ proxmox_acme_api::delete_plugin(id)
}
#[api(
@@ -675,12 +407,12 @@ pub enum DeletableProperty {
type: Array,
optional: true,
items: {
- type: DeletableProperty,
+ type: DeletablePluginProperty,
}
},
digest: {
- description: "Digest to protect against concurrent updates",
optional: true,
+ type: ConfigDigest,
},
},
},
@@ -694,65 +426,8 @@ pub fn update_plugin(
id: String,
update: DnsPluginCoreUpdater,
data: Option<String>,
- delete: Option<Vec<DeletableProperty>>,
- digest: Option<String>,
+ delete: Option<Vec<DeletablePluginProperty>>,
+ digest: Option<ConfigDigest>,
) -> Result<(), Error> {
- let data = data
- .as_deref()
- .map(proxmox_base64::decode)
- .transpose()?
- .map(String::from_utf8)
- .transpose()
- .map_err(|_| format_err!("data must be valid UTF-8"))?;
-
- let _lock = plugin::lock()?;
-
- let (mut plugins, expected_digest) = plugin::config()?;
-
- if let Some(digest) = digest {
- let digest = <[u8; 32]>::from_hex(digest)?;
- crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
- }
-
- match plugins.get_mut(&id) {
- Some((ty, ref mut entry)) => {
- if ty != "dns" {
- bail!("cannot update plugin of type {:?}", ty);
- }
-
- let mut plugin = DnsPlugin::deserialize(&*entry)?;
-
- if let Some(delete) = delete {
- for delete_prop in delete {
- match delete_prop {
- DeletableProperty::ValidationDelay => {
- plugin.core.validation_delay = None;
- }
- DeletableProperty::Disable => {
- plugin.core.disable = None;
- }
- }
- }
- }
- if let Some(data) = data {
- plugin.data = data;
- }
- if let Some(api) = update.api {
- plugin.core.api = api;
- }
- if update.validation_delay.is_some() {
- plugin.core.validation_delay = update.validation_delay;
- }
- if update.disable.is_some() {
- plugin.core.disable = update.disable;
- }
-
- *entry = serde_json::to_value(plugin)?;
- }
- None => http_bail!(NOT_FOUND, "no such plugin"),
- }
-
- plugin::save_config(&plugins)?;
-
- Ok(())
+ proxmox_acme_api::update_plugin(id, update, data, delete, digest)
}
diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
index 7c9063c0..2905b41b 100644
--- a/src/api2/types/acme.rs
+++ b/src/api2/types/acme.rs
@@ -44,22 +44,6 @@ pub const ACME_DOMAIN_PROPERTY_SCHEMA: Schema =
.format(&ApiStringFormat::PropertyString(&AcmeDomain::API_SCHEMA))
.schema();
-#[api(
- properties: {
- name: { type: String },
- url: { type: String },
- },
-)]
-/// An ACME directory endpoint with a name and URL.
-#[derive(Serialize)]
-pub struct KnownAcmeDirectory {
- /// The ACME directory's name.
- pub name: &'static str,
-
- /// The ACME directory's endpoint URL.
- pub url: &'static str,
-}
-
#[api(
properties: {
schema: {
diff --git a/src/bin/proxmox_backup_manager/acme.rs b/src/bin/proxmox_backup_manager/acme.rs
index bb987b26..e7bd67af 100644
--- a/src/bin/proxmox_backup_manager/acme.rs
+++ b/src/bin/proxmox_backup_manager/acme.rs
@@ -8,10 +8,8 @@ use proxmox_schema::api;
use proxmox_sys::fs::file_get_contents;
use proxmox_acme::async_client::AcmeClient;
-use proxmox_acme_api::AcmeAccountName;
+use proxmox_acme_api::{AcmeAccountName, DnsPluginCore, KNOWN_ACME_DIRECTORIES};
use proxmox_backup::api2;
-use proxmox_backup::config::acme::plugin::DnsPluginCore;
-use proxmox_backup::config::acme::KNOWN_ACME_DIRECTORIES;
pub fn acme_mgmt_cli() -> CommandLineInterface {
let cmd_def = CliCommandMap::new()
@@ -122,7 +120,7 @@ async fn register_account(
match input.trim().parse::<usize>() {
Ok(n) if n < KNOWN_ACME_DIRECTORIES.len() => {
- break (KNOWN_ACME_DIRECTORIES[n].url.to_owned(), false);
+ break (KNOWN_ACME_DIRECTORIES[n].url.to_string(), false);
}
Ok(n) if n == KNOWN_ACME_DIRECTORIES.len() => {
input.clear();
diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
index d31b2bc9..35cda50b 100644
--- a/src/config/acme/mod.rs
+++ b/src/config/acme/mod.rs
@@ -1,8 +1,7 @@
use std::collections::HashMap;
use std::ops::ControlFlow;
-use std::path::Path;
-use anyhow::{bail, format_err, Error};
+use anyhow::Error;
use serde_json::Value;
use proxmox_sys::error::SysError;
@@ -10,8 +9,8 @@ use proxmox_sys::fs::{file_read_string, CreateOptions};
use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
-use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
-use proxmox_acme_api::AcmeAccountName;
+use crate::api2::types::AcmeChallengeSchema;
+use proxmox_acme_api::{AcmeAccountName, KnownAcmeDirectory, KNOWN_ACME_DIRECTORIES};
pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
@@ -36,23 +35,8 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
create_acme_subdir(ACME_DIR)
}
-pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
- KnownAcmeDirectory {
- name: "Let's Encrypt V2",
- url: "https://acme-v02.api.letsencrypt.org/directory",
- },
- KnownAcmeDirectory {
- name: "Let's Encrypt V2 Staging",
- url: "https://acme-staging-v02.api.letsencrypt.org/directory",
- },
-];
-
pub const DEFAULT_ACME_DIRECTORY_ENTRY: &KnownAcmeDirectory = &KNOWN_ACME_DIRECTORIES[0];
-pub fn account_path(name: &str) -> String {
- format!("{ACME_ACCOUNT_DIR}/{name}")
-}
-
pub fn foreach_acme_account<F>(mut func: F) -> Result<(), Error>
where
F: FnMut(AcmeAccountName) -> ControlFlow<Result<(), Error>>,
@@ -83,28 +67,6 @@ where
}
}
-pub fn mark_account_deactivated(name: &str) -> Result<(), Error> {
- let from = account_path(name);
- for i in 0..100 {
- let to = account_path(&format!("_deactivated_{name}_{i}"));
- if !Path::new(&to).exists() {
- return std::fs::rename(&from, &to).map_err(|err| {
- format_err!(
- "failed to move account path {:?} to {:?} - {}",
- from,
- to,
- err
- )
- });
- }
- }
- bail!(
- "No free slot to rename deactivated account {:?}, please cleanup {:?}",
- from,
- ACME_ACCOUNT_DIR
- );
-}
-
pub fn load_dns_challenge_schema() -> Result<Vec<AcmeChallengeSchema>, Error> {
let raw = file_read_string(ACME_DNS_SCHEMA_FN)?;
let schemas: serde_json::Map<String, Value> = serde_json::from_str(&raw)?;
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 8%]
* [pbs-devel] [PATCH proxmox-backup v4 2/4] acme: drop local AcmeClient
2025-12-03 10:22 11% [pbs-devel] [PATCH proxmox{-backup, } v4 " Samuel Rufinatscha
2025-12-03 10:22 15% ` [pbs-devel] [PATCH proxmox-backup v4 1/4] acme: include proxmox-acme-api dependency Samuel Rufinatscha
@ 2025-12-03 10:22 6% ` Samuel Rufinatscha
2025-12-09 16:50 4% ` Max R. Carrara
2025-12-03 10:22 8% ` [pbs-devel] [PATCH proxmox-backup v4 3/4] acme: change API impls to use proxmox-acme-api handlers Samuel Rufinatscha
` (7 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-12-03 10:22 UTC (permalink / raw)
To: pbs-devel
PBS currently uses its own ACME client and API logic, while PDM uses the
factored out proxmox-acme and proxmox-acme-api crates. This duplication
risks differences in behaviour and requires ACME maintenance in two
places. This patch is part of a series to move PBS over to the shared
ACME stack.
Changes:
- Remove the local src/acme/client.rs and switch to
proxmox_acme::async_client::AcmeClient where needed.
- Use proxmox_acme_api::load_client_with_account to the custom
AcmeClient::load() function
- Replace the local do_register() logic with
proxmox_acme_api::register_account, to further ensure accounts are persisted
- Replace the local AcmeAccountName type, required for
proxmox_acme_api::register_account
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/acme/client.rs | 691 -------------------------
src/acme/mod.rs | 3 -
src/acme/plugin.rs | 2 +-
src/api2/config/acme.rs | 50 +-
src/api2/node/certificates.rs | 2 +-
src/api2/types/acme.rs | 8 -
src/bin/proxmox_backup_manager/acme.rs | 17 +-
src/config/acme/mod.rs | 8 +-
src/config/node.rs | 9 +-
9 files changed, 36 insertions(+), 754 deletions(-)
delete mode 100644 src/acme/client.rs
diff --git a/src/acme/client.rs b/src/acme/client.rs
deleted file mode 100644
index 9fb6ad55..00000000
--- a/src/acme/client.rs
+++ /dev/null
@@ -1,691 +0,0 @@
-//! HTTP Client for the ACME protocol.
-
-use std::fs::OpenOptions;
-use std::io;
-use std::os::unix::fs::OpenOptionsExt;
-
-use anyhow::{bail, format_err};
-use bytes::Bytes;
-use http_body_util::BodyExt;
-use hyper::Request;
-use nix::sys::stat::Mode;
-use proxmox_http::Body;
-use serde::{Deserialize, Serialize};
-
-use proxmox_acme::account::AccountCreator;
-use proxmox_acme::order::{Order, OrderData};
-use proxmox_acme::types::AccountData as AcmeAccountData;
-use proxmox_acme::Request as AcmeRequest;
-use proxmox_acme::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
-use proxmox_http::client::Client;
-use proxmox_sys::fs::{replace_file, CreateOptions};
-
-use crate::api2::types::AcmeAccountName;
-use crate::config::acme::account_path;
-use crate::tools::pbs_simple_http;
-
-/// Our on-disk format inherited from PVE's proxmox-acme code.
-#[derive(Deserialize, Serialize)]
-#[serde(rename_all = "camelCase")]
-pub struct AccountData {
- /// The account's location URL.
- location: String,
-
- /// The account data.
- account: AcmeAccountData,
-
- /// The private key as PEM formatted string.
- key: String,
-
- /// ToS URL the user agreed to.
- #[serde(skip_serializing_if = "Option::is_none")]
- tos: Option<String>,
-
- #[serde(skip_serializing_if = "is_false", default)]
- debug: bool,
-
- /// The directory's URL.
- directory_url: String,
-}
-
-#[inline]
-fn is_false(b: &bool) -> bool {
- !*b
-}
-
-pub struct AcmeClient {
- directory_url: String,
- debug: bool,
- account_path: Option<String>,
- tos: Option<String>,
- account: Option<Account>,
- directory: Option<Directory>,
- nonce: Option<String>,
- http_client: Client,
-}
-
-impl AcmeClient {
- /// Create a new ACME client for a given ACME directory URL.
- pub fn new(directory_url: String) -> Self {
- Self {
- directory_url,
- debug: false,
- account_path: None,
- tos: None,
- account: None,
- directory: None,
- nonce: None,
- http_client: pbs_simple_http(None),
- }
- }
-
- /// Load an existing ACME account by name.
- pub async fn load(account_name: &AcmeAccountName) -> Result<Self, anyhow::Error> {
- let account_path = account_path(account_name.as_ref());
- let data = match tokio::fs::read(&account_path).await {
- Ok(data) => data,
- Err(err) if err.kind() == io::ErrorKind::NotFound => {
- bail!("acme account '{}' does not exist", account_name)
- }
- Err(err) => bail!(
- "failed to load acme account from '{}' - {}",
- account_path,
- err
- ),
- };
- let data: AccountData = serde_json::from_slice(&data).map_err(|err| {
- format_err!(
- "failed to parse acme account from '{}' - {}",
- account_path,
- err
- )
- })?;
-
- let account = Account::from_parts(data.location, data.key, data.account);
-
- let mut me = Self::new(data.directory_url);
- me.debug = data.debug;
- me.account_path = Some(account_path);
- me.tos = data.tos;
- me.account = Some(account);
-
- Ok(me)
- }
-
- pub async fn new_account<'a>(
- &'a mut self,
- account_name: &AcmeAccountName,
- tos_agreed: bool,
- contact: Vec<String>,
- rsa_bits: Option<u32>,
- eab_creds: Option<(String, String)>,
- ) -> Result<&'a Account, anyhow::Error> {
- self.tos = if tos_agreed {
- self.terms_of_service_url().await?.map(str::to_owned)
- } else {
- None
- };
-
- let mut account = Account::creator()
- .set_contacts(contact)
- .agree_to_tos(tos_agreed);
-
- if let Some((eab_kid, eab_hmac_key)) = eab_creds {
- account = account.set_eab_credentials(eab_kid, eab_hmac_key)?;
- }
-
- let account = if let Some(bits) = rsa_bits {
- account.generate_rsa_key(bits)?
- } else {
- account.generate_ec_key()?
- };
-
- let _ = self.register_account(account).await?;
-
- crate::config::acme::make_acme_account_dir()?;
- let account_path = account_path(account_name.as_ref());
- let file = OpenOptions::new()
- .write(true)
- .create_new(true)
- .mode(0o600)
- .open(&account_path)
- .map_err(|err| format_err!("failed to open {:?} for writing: {}", account_path, err))?;
- self.write_to(file).map_err(|err| {
- format_err!(
- "failed to write acme account to {:?}: {}",
- account_path,
- err
- )
- })?;
- self.account_path = Some(account_path);
-
- // unwrap: Setting `self.account` is literally this function's job, we just can't keep
- // the borrow from from `self.register_account()` active due to clashes.
- Ok(self.account.as_ref().unwrap())
- }
-
- fn save(&self) -> Result<(), anyhow::Error> {
- let mut data = Vec::<u8>::new();
- self.write_to(&mut data)?;
- let account_path = self.account_path.as_ref().ok_or_else(|| {
- format_err!("no account path set, cannot save updated account information")
- })?;
- crate::config::acme::make_acme_account_dir()?;
- replace_file(
- account_path,
- &data,
- CreateOptions::new()
- .perm(Mode::from_bits_truncate(0o600))
- .owner(nix::unistd::ROOT)
- .group(nix::unistd::Gid::from_raw(0)),
- true,
- )
- }
-
- /// Shortcut to `account().ok_or_else(...).key_authorization()`.
- pub fn key_authorization(&self, token: &str) -> Result<String, anyhow::Error> {
- Ok(Self::need_account(&self.account)?.key_authorization(token)?)
- }
-
- /// Shortcut to `account().ok_or_else(...).dns_01_txt_value()`.
- /// the key authorization value.
- pub fn dns_01_txt_value(&self, token: &str) -> Result<String, anyhow::Error> {
- Ok(Self::need_account(&self.account)?.dns_01_txt_value(token)?)
- }
-
- async fn register_account(
- &mut self,
- account: AccountCreator,
- ) -> Result<&Account, anyhow::Error> {
- let mut retry = retry();
- let mut response = loop {
- retry.tick()?;
-
- let (directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
- let request = account.request(directory, nonce)?;
- match self.run_request(request).await {
- Ok(response) => break response,
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- };
-
- let account = account.response(response.location_required()?, &response.body)?;
-
- self.account = Some(account);
- Ok(self.account.as_ref().unwrap())
- }
-
- pub async fn update_account<T: Serialize>(
- &mut self,
- data: &T,
- ) -> Result<&Account, anyhow::Error> {
- let account = Self::need_account(&self.account)?;
-
- let mut retry = retry();
- let response = loop {
- retry.tick()?;
-
- let (_directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let request = account.post_request(&account.location, nonce, data)?;
- match Self::execute(&mut self.http_client, request, &mut self.nonce).await {
- Ok(response) => break response,
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- };
-
- // unwrap: we've been keeping an immutable reference to it from the top of the method
- let _ = account;
- self.account.as_mut().unwrap().data = response.json()?;
- self.save()?;
- Ok(self.account.as_ref().unwrap())
- }
-
- pub async fn new_order<I>(&mut self, domains: I) -> Result<Order, anyhow::Error>
- where
- I: IntoIterator<Item = String>,
- {
- let account = Self::need_account(&self.account)?;
-
- let order = domains
- .into_iter()
- .fold(OrderData::new(), |order, domain| order.domain(domain));
-
- let mut retry = retry();
- loop {
- retry.tick()?;
-
- let (directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let mut new_order = account.new_order(&order, directory, nonce)?;
- let mut response = match Self::execute(
- &mut self.http_client,
- new_order.request.take().unwrap(),
- &mut self.nonce,
- )
- .await
- {
- Ok(response) => response,
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- };
-
- return Ok(
- new_order.response(response.location_required()?, response.bytes().as_ref())?
- );
- }
- }
-
- /// Low level "POST-as-GET" request.
- async fn post_as_get(&mut self, url: &str) -> Result<AcmeResponse, anyhow::Error> {
- let account = Self::need_account(&self.account)?;
-
- let mut retry = retry();
- loop {
- retry.tick()?;
-
- let (_directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let request = account.get_request(url, nonce)?;
- match Self::execute(&mut self.http_client, request, &mut self.nonce).await {
- Ok(response) => return Ok(response),
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- }
- }
-
- /// Low level POST request.
- async fn post<T: Serialize>(
- &mut self,
- url: &str,
- data: &T,
- ) -> Result<AcmeResponse, anyhow::Error> {
- let account = Self::need_account(&self.account)?;
-
- let mut retry = retry();
- loop {
- retry.tick()?;
-
- let (_directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let request = account.post_request(url, nonce, data)?;
- match Self::execute(&mut self.http_client, request, &mut self.nonce).await {
- Ok(response) => return Ok(response),
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- }
- }
-
- /// Request challenge validation. Afterwards, the challenge should be polled.
- pub async fn request_challenge_validation(
- &mut self,
- url: &str,
- ) -> Result<Challenge, anyhow::Error> {
- Ok(self
- .post(url, &serde_json::Value::Object(Default::default()))
- .await?
- .json()?)
- }
-
- /// Assuming the provided URL is an 'Authorization' URL, get and deserialize it.
- pub async fn get_authorization(&mut self, url: &str) -> Result<Authorization, anyhow::Error> {
- Ok(self.post_as_get(url).await?.json()?)
- }
-
- /// Assuming the provided URL is an 'Order' URL, get and deserialize it.
- pub async fn get_order(&mut self, url: &str) -> Result<OrderData, anyhow::Error> {
- Ok(self.post_as_get(url).await?.json()?)
- }
-
- /// Finalize an Order via its `finalize` URL property and the DER encoded CSR.
- pub async fn finalize(&mut self, url: &str, csr: &[u8]) -> Result<(), anyhow::Error> {
- let csr = proxmox_base64::url::encode_no_pad(csr);
- let data = serde_json::json!({ "csr": csr });
- self.post(url, &data).await?;
- Ok(())
- }
-
- /// Download a certificate via its 'certificate' URL property.
- ///
- /// The certificate will be a PEM certificate chain.
- pub async fn get_certificate(&mut self, url: &str) -> Result<Bytes, anyhow::Error> {
- Ok(self.post_as_get(url).await?.body)
- }
-
- /// Revoke an existing certificate (PEM or DER formatted).
- pub async fn revoke_certificate(
- &mut self,
- certificate: &[u8],
- reason: Option<u32>,
- ) -> Result<(), anyhow::Error> {
- // TODO: This can also work without an account.
- let account = Self::need_account(&self.account)?;
-
- let revocation = account.revoke_certificate(certificate, reason)?;
-
- let mut retry = retry();
- loop {
- retry.tick()?;
-
- let (directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let request = revocation.request(directory, nonce)?;
- match Self::execute(&mut self.http_client, request, &mut self.nonce).await {
- Ok(_response) => return Ok(()),
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- }
- }
-
- fn need_account(account: &Option<Account>) -> Result<&Account, anyhow::Error> {
- account
- .as_ref()
- .ok_or_else(|| format_err!("cannot use client without an account"))
- }
-
- pub(crate) fn account(&self) -> Result<&Account, anyhow::Error> {
- Self::need_account(&self.account)
- }
-
- pub fn tos(&self) -> Option<&str> {
- self.tos.as_deref()
- }
-
- pub fn directory_url(&self) -> &str {
- &self.directory_url
- }
-
- fn to_account_data(&self) -> Result<AccountData, anyhow::Error> {
- let account = self.account()?;
-
- Ok(AccountData {
- location: account.location.clone(),
- key: account.private_key.clone(),
- account: AcmeAccountData {
- only_return_existing: false, // don't actually write this out in case it's set
- ..account.data.clone()
- },
- tos: self.tos.clone(),
- debug: self.debug,
- directory_url: self.directory_url.clone(),
- })
- }
-
- fn write_to<T: io::Write>(&self, out: T) -> Result<(), anyhow::Error> {
- let data = self.to_account_data()?;
-
- Ok(serde_json::to_writer_pretty(out, &data)?)
- }
-}
-
-struct AcmeResponse {
- body: Bytes,
- location: Option<String>,
- got_nonce: bool,
-}
-
-impl AcmeResponse {
- /// Convenience helper to assert that a location header was part of the response.
- fn location_required(&mut self) -> Result<String, anyhow::Error> {
- self.location
- .take()
- .ok_or_else(|| format_err!("missing Location header"))
- }
-
- /// Convenience shortcut to perform json deserialization of the returned body.
- fn json<T: for<'a> Deserialize<'a>>(&self) -> Result<T, Error> {
- Ok(serde_json::from_slice(&self.body)?)
- }
-
- /// Convenience shortcut to get the body as bytes.
- fn bytes(&self) -> &[u8] {
- &self.body
- }
-}
-
-impl AcmeClient {
- /// Non-self-borrowing run_request version for borrow workarounds.
- async fn execute(
- http_client: &mut Client,
- request: AcmeRequest,
- nonce: &mut Option<String>,
- ) -> Result<AcmeResponse, Error> {
- let req_builder = Request::builder().method(request.method).uri(&request.url);
-
- let http_request = if !request.content_type.is_empty() {
- req_builder
- .header("Content-Type", request.content_type)
- .header("Content-Length", request.body.len())
- .body(request.body.into())
- } else {
- req_builder.body(Body::empty())
- }
- .map_err(|err| Error::Custom(format!("failed to create http request: {err}")))?;
-
- let response = http_client
- .request(http_request)
- .await
- .map_err(|err| Error::Custom(err.to_string()))?;
- let (parts, body) = response.into_parts();
-
- let status = parts.status.as_u16();
- let body = body
- .collect()
- .await
- .map_err(|err| Error::Custom(format!("failed to retrieve response body: {err}")))?
- .to_bytes();
-
- let got_nonce = if let Some(new_nonce) = parts.headers.get(proxmox_acme::REPLAY_NONCE) {
- let new_nonce = new_nonce.to_str().map_err(|err| {
- Error::Client(format!(
- "received invalid replay-nonce header from ACME server: {err}"
- ))
- })?;
- *nonce = Some(new_nonce.to_owned());
- true
- } else {
- false
- };
-
- if parts.status.is_success() {
- if status != request.expected {
- return Err(Error::InvalidApi(format!(
- "ACME server responded with unexpected status code: {:?}",
- parts.status
- )));
- }
-
- let location = parts
- .headers
- .get("Location")
- .map(|header| {
- header.to_str().map(str::to_owned).map_err(|err| {
- Error::Client(format!(
- "received invalid location header from ACME server: {err}"
- ))
- })
- })
- .transpose()?;
-
- return Ok(AcmeResponse {
- body,
- location,
- got_nonce,
- });
- }
-
- let error: ErrorResponse = serde_json::from_slice(&body).map_err(|err| {
- Error::Client(format!(
- "error status with improper error ACME response: {err}"
- ))
- })?;
-
- if error.ty == proxmox_acme::error::BAD_NONCE {
- if !got_nonce {
- return Err(Error::InvalidApi(
- "badNonce without a new Replay-Nonce header".to_string(),
- ));
- }
- return Err(Error::BadNonce);
- }
-
- Err(Error::Api(error))
- }
-
- /// Low-level API to run an n API request. This automatically updates the current nonce!
- async fn run_request(&mut self, request: AcmeRequest) -> Result<AcmeResponse, Error> {
- Self::execute(&mut self.http_client, request, &mut self.nonce).await
- }
-
- pub async fn directory(&mut self) -> Result<&Directory, Error> {
- Ok(Self::get_directory(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?
- .0)
- }
-
- async fn get_directory<'a, 'b>(
- http_client: &mut Client,
- directory_url: &str,
- directory: &'a mut Option<Directory>,
- nonce: &'b mut Option<String>,
- ) -> Result<(&'a Directory, Option<&'b str>), Error> {
- if let Some(d) = directory {
- return Ok((d, nonce.as_deref()));
- }
-
- let response = Self::execute(
- http_client,
- AcmeRequest {
- url: directory_url.to_string(),
- method: "GET",
- content_type: "",
- body: String::new(),
- expected: 200,
- },
- nonce,
- )
- .await?;
-
- *directory = Some(Directory::from_parts(
- directory_url.to_string(),
- response.json()?,
- ));
-
- Ok((directory.as_mut().unwrap(), nonce.as_deref()))
- }
-
- /// Like `get_directory`, but if the directory provides no nonce, also performs a `HEAD`
- /// request on the new nonce URL.
- async fn get_dir_nonce<'a, 'b>(
- http_client: &mut Client,
- directory_url: &str,
- directory: &'a mut Option<Directory>,
- nonce: &'b mut Option<String>,
- ) -> Result<(&'a Directory, &'b str), Error> {
- // this let construct is a lifetime workaround:
- let _ = Self::get_directory(http_client, directory_url, directory, nonce).await?;
- let dir = directory.as_ref().unwrap(); // the above fails if it couldn't fill this option
- if nonce.is_none() {
- // this is also a lifetime issue...
- let _ = Self::get_nonce(http_client, nonce, dir.new_nonce_url()).await?;
- };
- Ok((dir, nonce.as_deref().unwrap()))
- }
-
- pub async fn terms_of_service_url(&mut self) -> Result<Option<&str>, Error> {
- Ok(self.directory().await?.terms_of_service_url())
- }
-
- async fn get_nonce<'a>(
- http_client: &mut Client,
- nonce: &'a mut Option<String>,
- new_nonce_url: &str,
- ) -> Result<&'a str, Error> {
- let response = Self::execute(
- http_client,
- AcmeRequest {
- url: new_nonce_url.to_owned(),
- method: "HEAD",
- content_type: "",
- body: String::new(),
- expected: 200,
- },
- nonce,
- )
- .await?;
-
- if !response.got_nonce {
- return Err(Error::InvalidApi(
- "no new nonce received from new nonce URL".to_string(),
- ));
- }
-
- nonce
- .as_deref()
- .ok_or_else(|| Error::Client("failed to update nonce".to_string()))
- }
-}
-
-/// bad nonce retry count helper
-struct Retry(usize);
-
-const fn retry() -> Retry {
- Retry(0)
-}
-
-impl Retry {
- fn tick(&mut self) -> Result<(), Error> {
- if self.0 >= 3 {
- Err(Error::Client("kept getting a badNonce error!".to_string()))
- } else {
- self.0 += 1;
- Ok(())
- }
- }
-}
diff --git a/src/acme/mod.rs b/src/acme/mod.rs
index bf61811c..cc561f9a 100644
--- a/src/acme/mod.rs
+++ b/src/acme/mod.rs
@@ -1,5 +1,2 @@
-mod client;
-pub use client::AcmeClient;
-
pub(crate) mod plugin;
pub(crate) use plugin::get_acme_plugin;
diff --git a/src/acme/plugin.rs b/src/acme/plugin.rs
index f756e9b5..5bc09e1f 100644
--- a/src/acme/plugin.rs
+++ b/src/acme/plugin.rs
@@ -20,8 +20,8 @@ use tokio::process::Command;
use proxmox_acme::{Authorization, Challenge};
-use crate::acme::AcmeClient;
use crate::api2::types::AcmeDomain;
+use proxmox_acme::async_client::AcmeClient;
use proxmox_rest_server::WorkerTask;
use crate::config::acme::plugin::{DnsPlugin, PluginData};
diff --git a/src/api2/config/acme.rs b/src/api2/config/acme.rs
index 35c3fb77..02f88e2e 100644
--- a/src/api2/config/acme.rs
+++ b/src/api2/config/acme.rs
@@ -16,15 +16,15 @@ use proxmox_router::{
use proxmox_schema::{api, param_bail};
use proxmox_acme::types::AccountData as AcmeAccountData;
-use proxmox_acme::Account;
use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
-use crate::acme::AcmeClient;
-use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
+use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
use crate::config::acme::plugin::{
self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
};
+use proxmox_acme::async_client::AcmeClient;
+use proxmox_acme_api::AcmeAccountName;
use proxmox_rest_server::WorkerTask;
pub(crate) const ROUTER: Router = Router::new()
@@ -143,15 +143,15 @@ pub struct AccountInfo {
)]
/// Return existing ACME account information.
pub async fn get_account(name: AcmeAccountName) -> Result<AccountInfo, Error> {
- let client = AcmeClient::load(&name).await?;
- let account = client.account()?;
+ let account_info = proxmox_acme_api::get_account(name).await?;
+
Ok(AccountInfo {
- location: account.location.clone(),
- tos: client.tos().map(str::to_owned),
- directory: client.directory_url().to_owned(),
+ location: account_info.location,
+ tos: account_info.tos,
+ directory: account_info.directory,
account: AcmeAccountData {
only_return_existing: false, // don't actually write this out in case it's set
- ..account.data.clone()
+ ..account_info.account
},
})
}
@@ -240,41 +240,24 @@ fn register_account(
auth_id.to_string(),
true,
move |_worker| async move {
- let mut client = AcmeClient::new(directory);
-
info!("Registering ACME account '{}'...", &name);
- let account = do_register_account(
- &mut client,
+ let location = proxmox_acme_api::register_account(
&name,
- tos_url.is_some(),
contact,
- None,
+ tos_url,
+ Some(directory),
eab_kid.zip(eab_hmac_key),
)
.await?;
- info!("Registration successful, account URL: {}", account.location);
+ info!("Registration successful, account URL: {}", location);
Ok(())
},
)
}
-pub async fn do_register_account<'a>(
- client: &'a mut AcmeClient,
- name: &AcmeAccountName,
- agree_to_tos: bool,
- contact: String,
- rsa_bits: Option<u32>,
- eab_creds: Option<(String, String)>,
-) -> Result<&'a Account, Error> {
- let contact = account_contact_from_string(&contact);
- client
- .new_account(name, agree_to_tos, contact, rsa_bits, eab_creds)
- .await
-}
-
#[api(
input: {
properties: {
@@ -312,7 +295,10 @@ pub fn update_account(
None => json!({}),
};
- AcmeClient::load(&name).await?.update_account(&data).await?;
+ proxmox_acme_api::load_client_with_account(&name)
+ .await?
+ .update_account(&data)
+ .await?;
Ok(())
},
@@ -350,7 +336,7 @@ pub fn deactivate_account(
auth_id.to_string(),
true,
move |_worker| async move {
- match AcmeClient::load(&name)
+ match proxmox_acme_api::load_client_with_account(&name)
.await?
.update_account(&json!({"status": "deactivated"}))
.await
diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.rs
index 61ef910e..31196715 100644
--- a/src/api2/node/certificates.rs
+++ b/src/api2/node/certificates.rs
@@ -17,10 +17,10 @@ use pbs_buildcfg::configdir;
use pbs_tools::cert;
use tracing::warn;
-use crate::acme::AcmeClient;
use crate::api2::types::AcmeDomain;
use crate::config::node::NodeConfig;
use crate::server::send_certificate_renewal_mail;
+use proxmox_acme::async_client::AcmeClient;
use proxmox_rest_server::WorkerTask;
pub const ROUTER: Router = Router::new()
diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
index 210ebdbc..7c9063c0 100644
--- a/src/api2/types/acme.rs
+++ b/src/api2/types/acme.rs
@@ -60,14 +60,6 @@ pub struct KnownAcmeDirectory {
pub url: &'static str,
}
-proxmox_schema::api_string_type! {
- #[api(format: &PROXMOX_SAFE_ID_FORMAT)]
- /// ACME account name.
- #[derive(Clone, Eq, PartialEq, Hash, Deserialize, Serialize)]
- #[serde(transparent)]
- pub struct AcmeAccountName(String);
-}
-
#[api(
properties: {
schema: {
diff --git a/src/bin/proxmox_backup_manager/acme.rs b/src/bin/proxmox_backup_manager/acme.rs
index 0f0eafea..bb987b26 100644
--- a/src/bin/proxmox_backup_manager/acme.rs
+++ b/src/bin/proxmox_backup_manager/acme.rs
@@ -7,9 +7,9 @@ use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
use proxmox_schema::api;
use proxmox_sys::fs::file_get_contents;
-use proxmox_backup::acme::AcmeClient;
+use proxmox_acme::async_client::AcmeClient;
+use proxmox_acme_api::AcmeAccountName;
use proxmox_backup::api2;
-use proxmox_backup::api2::types::AcmeAccountName;
use proxmox_backup::config::acme::plugin::DnsPluginCore;
use proxmox_backup::config::acme::KNOWN_ACME_DIRECTORIES;
@@ -188,17 +188,20 @@ async fn register_account(
println!("Attempting to register account with {directory_url:?}...");
- let account = api2::config::acme::do_register_account(
- &mut client,
+ let tos_agreed = tos_agreed
+ .then(|| directory.terms_of_service_url().map(str::to_owned))
+ .flatten();
+
+ let location = proxmox_acme_api::register_account(
&name,
- tos_agreed,
contact,
- None,
+ tos_agreed,
+ Some(directory_url),
eab_creds,
)
.await?;
- println!("Registration successful, account URL: {}", account.location);
+ println!("Registration successful, account URL: {}", location);
Ok(())
}
diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
index 274a23fd..d31b2bc9 100644
--- a/src/config/acme/mod.rs
+++ b/src/config/acme/mod.rs
@@ -10,7 +10,8 @@ use proxmox_sys::fs::{file_read_string, CreateOptions};
use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
-use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
+use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
+use proxmox_acme_api::AcmeAccountName;
pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
@@ -35,11 +36,6 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
create_acme_subdir(ACME_DIR)
}
-pub(crate) fn make_acme_account_dir() -> Result<(), Error> {
- make_acme_dir()?;
- create_acme_subdir(ACME_ACCOUNT_DIR)
-}
-
pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
KnownAcmeDirectory {
name: "Let's Encrypt V2",
diff --git a/src/config/node.rs b/src/config/node.rs
index d2d6e383..d2a17a49 100644
--- a/src/config/node.rs
+++ b/src/config/node.rs
@@ -16,10 +16,9 @@ use pbs_api_types::{
use pbs_buildcfg::configdir;
use pbs_config::{open_backup_lockfile, BackupLockGuard};
-use crate::acme::AcmeClient;
-use crate::api2::types::{
- AcmeAccountName, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA,
-};
+use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
+use proxmox_acme::async_client::AcmeClient;
+use proxmox_acme_api::AcmeAccountName;
const CONF_FILE: &str = configdir!("/node.cfg");
const LOCK_FILE: &str = configdir!("/.node.lck");
@@ -249,7 +248,7 @@ impl NodeConfig {
} else {
AcmeAccountName::from_string("default".to_string())? // should really not happen
};
- AcmeClient::load(&account).await
+ proxmox_acme_api::load_client_with_account(&account).await
}
pub fn acme_domains(&'_ self) -> AcmeDomainIter<'_> {
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* [pbs-devel] superseded: [PATCH proxmox{, -backup} v3 0/2] fix #6939: acme: support servers returning 204 for nonce requests
2025-11-03 10:13 13% [pbs-devel] [PATCH proxmox{, -backup} v3 " Samuel Rufinatscha
2025-11-03 10:13 13% ` [pbs-devel] [PATCH proxmox v3 1/1] " Samuel Rufinatscha
2025-11-03 10:13 15% ` [pbs-devel] [PATCH proxmox-backup v3 1/1] fix #6939: acme: accept HTTP 204 from newNonce endpoint Samuel Rufinatscha
@ 2025-12-03 10:23 13% ` Samuel Rufinatscha
2 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-03 10:23 UTC (permalink / raw)
To: pbs-devel
https://lore.proxmox.com/pbs-devel/20251203102217.59923-1-s.rufinatscha@proxmox.com/T/#t
On 11/3/25 11:13 AM, Samuel Rufinatscha wrote:
> Hi,
>
> this series proposes a change to ACME account registration in Proxmox
> Backup Server (PBS), so that it also works with ACME servers that return
> HTTP 204 No Content to the HEAD request for newNonce.
>
> This behaviour was observed against a specific ACME deployment and
> reported as bug #6939 [1]. Currently, PBS cannot register an ACME
> account for this CA.
>
> ## Problem
>
> During ACME account registration, PBS first fetches an anti-replay nonce
> by sending a HEAD request to the CA’s newNonce URL. RFC 8555 7.2 [2]
> says:
>
> * the server MUST include a Replay-Nonce header with a fresh nonce,
> * the server SHOULD use status 200 OK for the HEAD request,
> * the server MUST also handle GET on the same resource with status 204 No
> Content and an empty body [2].
>
> Currently, our Rust ACME clients only accept 200 OK. PBS inherits that
> strictness and aborts with:
>
> *ACME server responded with unexpected status code: 204*
>
> The author mentions, the issue did not appear with PVE9 [1].
> After looking into PVE’s Perl ACME client [3] it appears it uses a GET
> request instead of a HEAD request and accepts any 2xx success code
> when retrieving the nonce [5]. This difference in behavior does not
> affect functionality but is worth noting for consistency across
> implementations.
>
> ## Ideas to solve the problem
>
> To support ACME providers which return 204 No Content, the underlying
> ACME clients need to tolerate both 200 OK and 204 No Content as valid
> responses for the nonce HEAD request, as long as the Replay-Nonce is
> provided.
>
> I considered following solutions:
>
> 1. Change the `expected` field of the `AcmeRequest` type from `u16` to
> `Vec<u16>`, to support multiple success codes
>
> 2. Keep `expected: u16` and add a second field e.g. `expected_other:
> Vec<u16>` for "also allowed" codes.
>
> 3. Support any 2xx success codes, and remove the `expected` check
>
> I thought (1) might be reasonable, because:
>
> * It stays explicit and makes it clear which statuses are considered
> success.
> * We don’t create two parallel concepts ("expected" vs
> "expected_other") which introduces additional complexity
> * Can be extend later if we meet yet another harmless but not 200
> variant.
> * We don’t allow arbitrary 2xx.
>
> What do you think? Do you maybe have any other solution in mind that
> would fit better?
>
> ## Testing
>
> To prove the proposed fix, I reproduced the scenario:
>
> Pebble (release 2.8.0) from Let's Encrypt [5] running on a Debian 9 VM
> as the ACME server. nginx in front of Pebble, to intercept the
> `newNonce` request in order to return 204 No Content instead of 200 OK,
> all other requests are unchanged and forwarded to Pebble. Trust the
> Pebble and ngix CAs via `/usr/local/share/ca-certificates` +
> `update-ca-certificates` on the PBS VM.
>
> Then I ran following command against nginx:
>
> ```
> proxmox-backup-manager acme account register proxytest root@backup.local
> --directory 'https://nginx-address/dir
>
> Attempting to fetch Terms of
> Service from "https://acme-vm/dir"
> Terms of Service:
> data:text/plain,Do%20what%20thou%20wilt
> Do you agree to the above terms?
> [y|N]: y
> Do you want to use external account binding? [y|N]: N
> Attempting
> to register account with "https://acme-vm/dir"...
> Registration
> successful, account URL: https://acme-vm/my-account/160e58b66bdd72da
> ```
>
> When adjusting the nginx configuration to return any other non-expected
> success status code, e.g. 205, PBS expectely rejects with `API
> misbehaved: ACME server responded with unexpected status code: 205`.
>
> ## Maintainer notes:
>
> The patch series involves the following components:
>
> proxmox-acme: Apply PATCH 1 to change `expected` from `u16` to
> `Vec<u16>`. This results in a breaking change, as it changes the public
> API of the `AcmeRequest` type that is used by other components.
>
> proxmox-acme-api: Needs to depend on the new proxmox-acme; patch bump
>
> proxmox-backup: Apply PATCH 2 to use the new API changes; no breaking
> change as of only internal changes; patch bump
>
> proxmox-perl-rs / proxmox-datacenter-manager: Will need to use the
> dependency version bumps to follow the new proxmox-acme.
>
> ## Patch summary
>
> [PATCH 1/2] fix #6939: support providers returning 204 for nonce
> requests
>
> * Make the expected-status logic accept multiple allowed codes.
> * Treat both 200 OK and 204 No Content as valid for HEAD /newNonce,
> provided Replay-Nonce is present.
> * Keep rejecting other codes.
>
> [PATCH 2/2] acme: accept HTTP 204 from newNonce endpoint
>
> * Use the updated proxmox-acme behavior in PBS.
> * PBS can now register an ACME account against servers that return 204
> for the nonce HEAD request.
> * Still rejects unexpected codes.
>
> Thanks for considering this patch series, I look forward to your
> feedback.
>
> Best,
> Samuel Rufinatscha
>
> ## Changes from v1:
>
> [PATCH 1/2] fix #6939: support providers returning 204 for nonce
> requests
> * Introduced `http_success` module to contain the http success codes
> * Replaced `Vec<u16>` with `&[u16]` for expected codes to avoid
> allocations.
> * Clarified the PVEs Perl ACME client behaviour in the commit message.
>
> [PATCH 2/2] acme: accept HTTP 204 from newNonce endpoint
> * Integrated the `http_success` module, replacing `Vec<u16>` with `&[u16]`
> * Clarified the PVEs Perl ACME client behaviour in the commit message.
>
> ## Changes from v2:
>
> [PATCH 1/2] fix #6939: support providers returning 204 for nonce
> requests
> * Rename `http_success` module to `http_status`
>
> [PATCH 2/2] acme: accept HTTP 204 from newNonce endpoint
> * Replace `http_success` usage
>
> [1] Bugzilla report #6939:
> [https://bugzilla.proxmox.com/show_bug.cgi?id=6939](https://bugzilla.proxmox.com/show_bug.cgi?id=6939)
> [2] RFC 8555 (ACME):
> [https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2](https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2)
> [3] PVE’s Perl ACME client (allow 2xx codes for nonce requests):
> [https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597)
> [4] Pebble ACME server:
> [https://github.com/letsencrypt/pebble](https://github.com/letsencrypt/pebble)
> [5] Pebble ACME server (perform GET request:
> [https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219)
>
> proxmox:
>
> Samuel Rufinatscha (1):
> fix #6939: acme: support servers returning 204 for nonce requests
>
> proxmox-acme/src/account.rs | 10 +++++-----
> proxmox-acme/src/async_client.rs | 6 +++---
> proxmox-acme/src/client.rs | 2 +-
> proxmox-acme/src/lib.rs | 4 ++++
> proxmox-acme/src/request.rs | 15 ++++++++++++---
> 5 files changed, 25 insertions(+), 12 deletions(-)
>
>
> proxmox-backup:
>
> Samuel Rufinatscha (1):
> fix #6939: acme: accept HTTP 204 from newNonce endpoint
>
> src/acme/client.rs | 8 ++++----
> 1 file changed, 4 insertions(+), 4 deletions(-)
>
>
> Summary over all repositories:
> 6 files changed, 29 insertions(+), 16 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox 1/3] proxmox-access-control: cache verified API token secrets
2025-12-05 13:25 15% [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Samuel Rufinatscha
` (2 preceding siblings ...)
2025-12-05 13:25 16% ` [pbs-devel] [PATCH proxmox-backup 3/3] pbs-config: add TTL window to token secret cache Samuel Rufinatscha
@ 2025-12-05 13:25 14% ` Samuel Rufinatscha
2025-12-05 13:25 15% ` [pbs-devel] [PATCH proxmox 2/3] proxmox-access-control: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
` (3 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-05 13:25 UTC (permalink / raw)
To: pbs-devel
Currently, every token-based API request reads the token.shadow file and
runs the expensive password hash verification for the given token
secret. This issue was first observed as part of profiling the PBS
/status endpoint (see bug #6049 [1]) and is required for the factored
out proxmox_access_control token_shadow implementation too.
This patch introduces an in-memory cache of successfully verified token
secrets. Subsequent requests for the same token+secret combination only
perform a comparison using openssl::memcmp::eq and avoid re-running the
password hash. The cache is updated when a token secret is set and
cleared when a token is deleted. Note, this does NOT include manual
config changes, which will be covered in a subsequent patch.
This patch is a partly-fix.
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-access-control/src/token_shadow.rs | 57 +++++++++++++++++++++-
1 file changed, 56 insertions(+), 1 deletion(-)
diff --git a/proxmox-access-control/src/token_shadow.rs b/proxmox-access-control/src/token_shadow.rs
index c586d834..2dcd117d 100644
--- a/proxmox-access-control/src/token_shadow.rs
+++ b/proxmox-access-control/src/token_shadow.rs
@@ -1,4 +1,5 @@
use std::collections::HashMap;
+use std::sync::{OnceLock, RwLock};
use anyhow::{bail, format_err, Error};
use serde_json::{from_value, Value};
@@ -8,6 +9,13 @@ use proxmox_product_config::{open_api_lockfile, replace_config, ApiLockGuard};
use crate::init::impl_feature::{token_shadow, token_shadow_lock};
+/// Global in-memory cache for successfully verified API token secrets.
+/// The cache stores plain text secrets for token Authids that have already been
+/// verified against the hashed values in `token.shadow`. This allows for cheap
+/// subsequent authentications for the same token+secret combination, avoiding
+/// recomputing the password hash on every request.
+static TOKEN_SECRET_CACHE: OnceLock<RwLock<ApiTokenSecretCache>> = OnceLock::new();
+
// Get exclusive lock
fn lock_config() -> Result<ApiLockGuard, Error> {
open_api_lockfile(token_shadow_lock(), None, true)
@@ -36,9 +44,25 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
bail!("not an API token ID");
}
+ // Fast path
+ if let Some(cached) = token_secret_cache().read().unwrap().secrets.get(tokenid) {
+ // Compare cached secret with provided one using constant time comparison
+ if openssl::memcmp::eq(cached.as_bytes(), secret.as_bytes()) {
+ // Already verified before
+ return Ok(());
+ }
+ // Fall through to slow path if secret doesn't match cached one
+ }
+
+ // Slow path: read file + verify hash
let data = read_file()?;
match data.get(tokenid) {
- Some(hashed_secret) => proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
+ Some(hashed_secret) => {
+ proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
+ // Cache the plain secret for future requests
+ cache_insert_secret(tokenid.clone(), secret.to_owned());
+ Ok(())
+ }
None => bail!("invalid API token"),
}
}
@@ -56,6 +80,8 @@ pub fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
data.insert(tokenid.clone(), hashed_secret);
write_file(data)?;
+ cache_insert_secret(tokenid.clone(), secret.to_owned());
+
Ok(())
}
@@ -71,6 +97,8 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
data.remove(tokenid);
write_file(data)?;
+ cache_remove_secret(tokenid);
+
Ok(())
}
@@ -81,3 +109,30 @@ pub fn generate_and_set_secret(tokenid: &Authid) -> Result<String, Error> {
set_secret(tokenid, &secret)?;
Ok(secret)
}
+
+struct ApiTokenSecretCache {
+ /// Keys are token Authids, values are the corresponding plain text secrets.
+ /// Entries are added after a successful on-disk verification in
+ /// `verify_secret` or when a new token secret is generated by
+ /// `generate_and_set_secret`. Used to avoid repeated
+ /// password-hash computation on subsequent authentications.
+ secrets: HashMap<Authid, String>,
+}
+
+fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
+ TOKEN_SECRET_CACHE.get_or_init(|| {
+ RwLock::new(ApiTokenSecretCache {
+ secrets: HashMap::new(),
+ })
+ })
+}
+
+fn cache_insert_secret(tokenid: Authid, secret: String) {
+ let mut cache = token_secret_cache().write().unwrap();
+ cache.secrets.insert(tokenid, secret);
+}
+
+fn cache_remove_secret(tokenid: &Authid) {
+ let mut cache = token_secret_cache().write().unwrap();
+ cache.secrets.remove(tokenid);
+}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 14%]
* [pbs-devel] [PATCH proxmox-backup 3/3] pbs-config: add TTL window to token secret cache
2025-12-05 13:25 15% [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Samuel Rufinatscha
2025-12-05 13:25 14% ` [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets Samuel Rufinatscha
2025-12-05 13:25 15% ` [pbs-devel] [PATCH proxmox-backup 2/3] pbs-config: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
@ 2025-12-05 13:25 16% ` Samuel Rufinatscha
2025-12-05 13:25 14% ` [pbs-devel] [PATCH proxmox 1/3] proxmox-access-control: cache verified API token secrets Samuel Rufinatscha
` (4 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-05 13:25 UTC (permalink / raw)
To: pbs-devel
Verify_secret() currently calls refresh_cache_if_file_changed() on every
request, which performs a metadata() call on token.shadow each time.
Under load this adds unnecessary overhead, considering also the file
usually should rarely change.
This patch introduces a TTL boundary, controlled by
TOKEN_SECRET_CACHE_TTL_SECS. File metadata is only re-loaded once the
TTL has expired.
This patch partly fixes bug #6049 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-config/src/token_shadow.rs | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
index ed54cdfa..23837c60 100644
--- a/pbs-config/src/token_shadow.rs
+++ b/pbs-config/src/token_shadow.rs
@@ -10,6 +10,7 @@ use serde::{Deserialize, Serialize};
use serde_json::{from_value, Value};
use proxmox_sys::fs::CreateOptions;
+use proxmox_time::epoch_i64;
use pbs_api_types::Authid;
//use crate::auth;
@@ -24,6 +25,8 @@ const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
/// subsequent authentications for the same token+secret combination, avoiding
/// recomputing the password hash on every request.
static TOKEN_SECRET_CACHE: OnceCell<RwLock<ApiTokenSecretCache>> = OnceCell::new();
+/// Max age in seconds of the token secret cache before checking for file changes.
+const TOKEN_SECRET_CACHE_TTL_SECS: i64 = 60;
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
@@ -63,6 +66,15 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
fn refresh_cache_if_file_changed() -> Result<(), Error> {
let mut cache = token_secret_cache().write().unwrap();
+ let now = epoch_i64();
+
+ // Fast path: Within TTL boundary
+ if let Some(last) = cache.last_checked {
+ if now - last < TOKEN_SECRET_CACHE_TTL_SECS {
+ return Ok(());
+ }
+ }
+
// Fetch the current token.shadow metadata
let (new_mtime, new_len) = match fs::metadata(CONF_FILE) {
Ok(meta) => (meta.modified().ok(), Some(meta.len())),
@@ -79,6 +91,7 @@ fn refresh_cache_if_file_changed() -> Result<(), Error> {
cache.secrets.clear();
cache.file_mtime = new_mtime;
cache.file_len = new_len;
+ cache.last_checked = Some(now);
Ok(())
}
@@ -169,6 +182,8 @@ struct ApiTokenSecretCache {
file_mtime: Option<SystemTime>,
// shadow file length to detect changes
file_len: Option<u64>,
+ // last time the file metadata was checked
+ last_checked: Option<i64>,
}
fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
@@ -177,6 +192,7 @@ fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
secrets: HashMap::new(),
file_mtime: None,
file_len: None,
+ last_checked: None,
})
})
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 16%]
* [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets
2025-12-05 13:25 15% [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Samuel Rufinatscha
@ 2025-12-05 13:25 14% ` Samuel Rufinatscha
2025-12-05 14:04 5% ` Shannon Sterz
2025-12-10 11:47 5% ` Fabian Grünbichler
2025-12-05 13:25 15% ` [pbs-devel] [PATCH proxmox-backup 2/3] pbs-config: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
` (6 subsequent siblings)
7 siblings, 2 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-05 13:25 UTC (permalink / raw)
To: pbs-devel
Currently, every token-based API request reads the token.shadow file and
runs the expensive password hash verification for the given token
secret. This shows up as a hotspot in /status profiling (see
bug #6049 [1]).
This patch introduces an in-memory cache of successfully verified token
secrets. Subsequent requests for the same token+secret combination only
perform a comparison using openssl::memcmp::eq and avoid re-running the
password hash. The cache is updated when a token secret is set and
cleared when a token is deleted. Note, this does NOT include manual
config changes, which will be covered in a subsequent patch.
This patch partly fixes bug #6049 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-config/src/token_shadow.rs | 58 +++++++++++++++++++++++++++++++++-
1 file changed, 57 insertions(+), 1 deletion(-)
diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
index 640fabbf..47aa2fc2 100644
--- a/pbs-config/src/token_shadow.rs
+++ b/pbs-config/src/token_shadow.rs
@@ -1,6 +1,8 @@
use std::collections::HashMap;
+use std::sync::RwLock;
use anyhow::{bail, format_err, Error};
+use once_cell::sync::OnceCell;
use serde::{Deserialize, Serialize};
use serde_json::{from_value, Value};
@@ -13,6 +15,13 @@ use crate::{open_backup_lockfile, BackupLockGuard};
const LOCK_FILE: &str = pbs_buildcfg::configdir!("/token.shadow.lock");
const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
+/// Global in-memory cache for successfully verified API token secrets.
+/// The cache stores plain text secrets for token Authids that have already been
+/// verified against the hashed values in `token.shadow`. This allows for cheap
+/// subsequent authentications for the same token+secret combination, avoiding
+/// recomputing the password hash on every request.
+static TOKEN_SECRET_CACHE: OnceCell<RwLock<ApiTokenSecretCache>> = OnceCell::new();
+
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
/// ApiToken id / secret pair
@@ -54,9 +63,25 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
bail!("not an API token ID");
}
+ // Fast path
+ if let Some(cached) = token_secret_cache().read().unwrap().secrets.get(tokenid) {
+ // Compare cached secret with provided one using constant time comparison
+ if openssl::memcmp::eq(cached.as_bytes(), secret.as_bytes()) {
+ // Already verified before
+ return Ok(());
+ }
+ // Fall through to slow path if secret doesn't match cached one
+ }
+
+ // Slow path: read file + verify hash
let data = read_file()?;
match data.get(tokenid) {
- Some(hashed_secret) => proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
+ Some(hashed_secret) => {
+ proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
+ // Cache the plain secret for future requests
+ cache_insert_secret(tokenid.clone(), secret.to_owned());
+ Ok(())
+ }
None => bail!("invalid API token"),
}
}
@@ -82,6 +107,8 @@ fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
data.insert(tokenid.clone(), hashed_secret);
write_file(data)?;
+ cache_insert_secret(tokenid.clone(), secret.to_owned());
+
Ok(())
}
@@ -97,5 +124,34 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
data.remove(tokenid);
write_file(data)?;
+ cache_remove_secret(tokenid);
+
Ok(())
}
+
+struct ApiTokenSecretCache {
+ /// Keys are token Authids, values are the corresponding plain text secrets.
+ /// Entries are added after a successful on-disk verification in
+ /// `verify_secret` or when a new token secret is generated by
+ /// `generate_and_set_secret`. Used to avoid repeated
+ /// password-hash computation on subsequent authentications.
+ secrets: HashMap<Authid, String>,
+}
+
+fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
+ TOKEN_SECRET_CACHE.get_or_init(|| {
+ RwLock::new(ApiTokenSecretCache {
+ secrets: HashMap::new(),
+ })
+ })
+}
+
+fn cache_insert_secret(tokenid: Authid, secret: String) {
+ let mut cache = token_secret_cache().write().unwrap();
+ cache.secrets.insert(tokenid, secret);
+}
+
+fn cache_remove_secret(tokenid: &Authid) {
+ let mut cache = token_secret_cache().write().unwrap();
+ cache.secrets.remove(tokenid);
+}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 14%]
* [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead
@ 2025-12-05 13:25 15% Samuel Rufinatscha
2025-12-05 13:25 14% ` [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets Samuel Rufinatscha
` (7 more replies)
0 siblings, 8 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-05 13:25 UTC (permalink / raw)
To: pbs-devel
Hi,
this series improves the performance of token-based API authentication
in PBS (pbs-config) and in PDM (underlying proxmox-access-control
crate), addressing the API token verification hotspot reported in our
bugtracker #6049 [1].
When profiling PBS /status endpoint with cargo flamegraph [2],
token-based authentication showed up as a dominant hotspot via
proxmox_sys::crypt::verify_crypt_pw. Applying this series removes that
path from the hot section of the flamegraph. The same performance issue
was measured [3] for PDM. PDM uses the underlying shared
proxmox-access-control library for token handling, which is a
factored out version of the token.shadow handling code from PBS.
While this series fixes the immediate performance issue both in PBS
(pbs-config) and in the shared proxmox-access-control crate used by
PDM, PBS should eventually, ideally be refactored, in a separate
effort, to use proxmox-access-control for token handling instead of its
local implementation.
Problem
For token-based API requests, both PBS’s pbs-config token.shadow
handling and PDM proxmox-access-control’s token.shadow handling
currently:
1. read the token.shadow file on each request
2. deserialize it into a HashMap<Authid, String>
3. run password hash verification via
proxmox_sys::crypt::verify_crypt_pw for the provided token secret
Under load, this results in significant CPU usage spent in repeated
password hash computations for the same token+secret pairs. The
attached flamegraphs for PBS [2] and PDM [3] show
proxmox_sys::crypt::verify_crypt_pw dominating the hot path.
Approach
The goal is to reduce the cost of token-based authentication preserving
the existing token handling semantics (including detecting manual edits
to token.shadow) and be consistent between PBS (pbs-config) and
PDM (proxmox-access-control). For both sites, the series proposes
following approach:
1. Introduce an in-memory cache for verified token secrets
2. Invalidate the cache when token.shadow changes (detect manual edits)
3. Control metadata checks with a TTL window
Testing
*PBS (pbs-config)*
To verify the effect in PBS, I:
1. Set up test environment based on latest PBS ISO, installed Rust
toolchain, cloned proxmox-backup repository to use with cargo
flamegraph. Reproduced bug #6049 [1] by profiling the /status
endpoint with token-based authentication using cargo flamegraph [2].
The flamegraph showed proxmox_sys::crypt::verify_crypt_pw is the
hotspot.
2. Built PBS with pbs-config patches and re-ran the same workload and
profiling setup.
3. Confirmed that the proxmox_sys::crypt::verify_crypt_pw path no
longer appears in the hot section of the flamegraph. CPU usage is
now dominated by TLS overhead.
4. Functionally verified that:
* token-based API authentication still works for valid tokens
* invalid secrets are rejected as before
* generating a new token secret via dashboard works and
authenticates correctly
*PDM (proxmox-access-control)*
To verify the effect in PDM, I followed a similar testing approach.
Instead of /status, I profiled the /version endpoint with cargo
flamegraph [3] and verified that the token hashing path disappears
from the hot section after applying the proxmox-access-control patches.
Functionally I verified that:
* token-based API authentication still works for valid tokens
* invalid secrets are rejected as before
* generating a new token secret via dashboard works and
authenticates correctly
Patch summary
pbs-config:
0001 – pbs-config: cache verified API token secrets
Adds an in-memory cache keyed by Authid that stores plain text token
secrets after a successful verification or generation and uses
openssl’s memcmp constant-time for comparison.
0002 – pbs-config: invalidate token-secret cache on token.shadow changes
Tracks token.shadow mtime and length and clears the in-memory cache
when the file changes.
0003 – pbs-config: add TTL window to token-secret cache
Introduces a TTL (TOKEN_SECRET_CACHE_TTL_SECS, default 60) for metadata checks so
that fs::metadata is only called periodically.
proxmox-access-control:
0004 – access-control: cache verified API token secrets
Mirrors PBS patch 0001.
0005 – access-control: invalidate token-secret cache on token.shadow changes
Mirrors PBS patch 0002.
0006 – access-control: add TTL window to token-secret cache
Mirrors PBS patch 0003.
Thanks for considering this patch series, I look forward to your
feedback.
Best,
Samuel Rufinatscha
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] Flamegraph illustrating the`proxmox_sys::crypt::verify_crypt_pw
hotspot before this series (attached to [1])
proxmox-backup:
Samuel Rufinatscha (3):
pbs-config: cache verified API token secrets
pbs-config: invalidate token-secret cache on token.shadow changes
pbs-config: add TTL window to token secret cache
pbs-config/src/token_shadow.rs | 109 ++++++++++++++++++++++++++++++++-
1 file changed, 108 insertions(+), 1 deletion(-)
proxmox:
Samuel Rufinatscha (3):
proxmox-access-control: cache verified API token secrets
proxmox-access-control: invalidate token-secret cache on token.shadow
changes
proxmox-access-control: add TTL window to token secret cache
proxmox-access-control/src/token_shadow.rs | 108 ++++++++++++++++++++-
1 file changed, 107 insertions(+), 1 deletion(-)
Summary over all repositories:
2 files changed, 215 insertions(+), 2 deletions(-)
--
Generated by git-murpp 0.8.1
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox 2/3] proxmox-access-control: invalidate token-secret cache on token.shadow changes
2025-12-05 13:25 15% [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Samuel Rufinatscha
` (3 preceding siblings ...)
2025-12-05 13:25 14% ` [pbs-devel] [PATCH proxmox 1/3] proxmox-access-control: cache verified API token secrets Samuel Rufinatscha
@ 2025-12-05 13:25 15% ` Samuel Rufinatscha
2025-12-05 13:25 16% ` [pbs-devel] [PATCH proxmox 3/3] proxmox-access-control: add TTL window to token secret cache Samuel Rufinatscha
` (2 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-05 13:25 UTC (permalink / raw)
To: pbs-devel
Previously the in-memory token-secret cache was only updated via
set_secret() and delete_secret(), so manual edits to token.shadow were
not reflected.
This patch adds file change detection to the cache. It tracks the mtime
and length of token.shadow and clears the in-memory token secret cache
whenever these values change.
Note, this patch fetches file stats on every request. An TTL-based
optimization will be covered in a subsequent patch of the series.
This patch is a partly-fix.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-access-control/src/token_shadow.rs | 35 ++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/proxmox-access-control/src/token_shadow.rs b/proxmox-access-control/src/token_shadow.rs
index 2dcd117d..d08fb06a 100644
--- a/proxmox-access-control/src/token_shadow.rs
+++ b/proxmox-access-control/src/token_shadow.rs
@@ -1,5 +1,8 @@
use std::collections::HashMap;
+use std::fs;
+use std::io::ErrorKind;
use std::sync::{OnceLock, RwLock};
+use std::time::SystemTime;
use anyhow::{bail, format_err, Error};
use serde_json::{from_value, Value};
@@ -38,12 +41,38 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
replace_config(token_shadow(), &json)
}
+fn refresh_cache_if_file_changed() -> Result<(), Error> {
+ let mut cache = token_secret_cache().write().unwrap();
+
+ // Fetch the current token.shadow metadata
+ let (new_mtime, new_len) = match fs::metadata(token_shadow().as_path()) {
+ Ok(meta) => (meta.modified().ok(), Some(meta.len())),
+ Err(e) if e.kind() == ErrorKind::NotFound => (None, None),
+ Err(e) => return Err(e.into()),
+ };
+
+ // Fast path: file did not change, keep the cache
+ if cache.file_mtime == new_mtime && cache.file_len == new_len {
+ return Ok(());
+ }
+
+ // File changed, drop all cached secrets
+ cache.secrets.clear();
+ cache.file_mtime = new_mtime;
+ cache.file_len = new_len;
+
+ Ok(())
+}
+
/// Verifies that an entry for given tokenid / API token secret exists
pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
if !tokenid.is_token() {
bail!("not an API token ID");
}
+ // Ensure cache is in sync with on-disk token.shadow file
+ refresh_cache_if_file_changed()?;
+
// Fast path
if let Some(cached) = token_secret_cache().read().unwrap().secrets.get(tokenid) {
// Compare cached secret with provided one using constant time comparison
@@ -117,12 +146,18 @@ struct ApiTokenSecretCache {
/// `generate_and_set_secret`. Used to avoid repeated
/// password-hash computation on subsequent authentications.
secrets: HashMap<Authid, String>,
+ // shadow file mtime to detect changes
+ file_mtime: Option<SystemTime>,
+ // shadow file length to detect changes
+ file_len: Option<u64>,
}
fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
TOKEN_SECRET_CACHE.get_or_init(|| {
RwLock::new(ApiTokenSecretCache {
secrets: HashMap::new(),
+ file_mtime: None,
+ file_len: None,
})
})
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox-backup 2/3] pbs-config: invalidate token-secret cache on token.shadow changes
2025-12-05 13:25 15% [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Samuel Rufinatscha
2025-12-05 13:25 14% ` [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets Samuel Rufinatscha
@ 2025-12-05 13:25 15% ` Samuel Rufinatscha
2025-12-05 13:25 16% ` [pbs-devel] [PATCH proxmox-backup 3/3] pbs-config: add TTL window to token secret cache Samuel Rufinatscha
` (5 subsequent siblings)
7 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-05 13:25 UTC (permalink / raw)
To: pbs-devel
Previously the in-memory token-secret cache was only updated via
set_secret() and delete_secret(), so manual edits to token.shadow were
not reflected.
This patch adds file change detection to the cache. It tracks the mtime
and length of token.shadow and clears the in-memory token secret cache
whenever these values change.
Note, this patch fetches file stats on every request. An TTL-based
optimization will be covered in a subsequent patch of the series.
This patch is a partly-fix.
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-config/src/token_shadow.rs | 35 ++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
index 47aa2fc2..ed54cdfa 100644
--- a/pbs-config/src/token_shadow.rs
+++ b/pbs-config/src/token_shadow.rs
@@ -1,5 +1,8 @@
use std::collections::HashMap;
+use std::fs;
+use std::io::ErrorKind;
use std::sync::RwLock;
+use std::time::SystemTime;
use anyhow::{bail, format_err, Error};
use once_cell::sync::OnceCell;
@@ -57,12 +60,38 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
proxmox_sys::fs::replace_file(CONF_FILE, &json, options, true)
}
+fn refresh_cache_if_file_changed() -> Result<(), Error> {
+ let mut cache = token_secret_cache().write().unwrap();
+
+ // Fetch the current token.shadow metadata
+ let (new_mtime, new_len) = match fs::metadata(CONF_FILE) {
+ Ok(meta) => (meta.modified().ok(), Some(meta.len())),
+ Err(e) if e.kind() == ErrorKind::NotFound => (None, None),
+ Err(e) => return Err(e.into()),
+ };
+
+ // Fast path: file did not change, keep the cache
+ if cache.file_mtime == new_mtime && cache.file_len == new_len {
+ return Ok(());
+ }
+
+ // File changed, drop all cached secrets
+ cache.secrets.clear();
+ cache.file_mtime = new_mtime;
+ cache.file_len = new_len;
+
+ Ok(())
+}
+
/// Verifies that an entry for given tokenid / API token secret exists
pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
if !tokenid.is_token() {
bail!("not an API token ID");
}
+ // Ensure cache is in sync with on-disk token.shadow file
+ refresh_cache_if_file_changed()?;
+
// Fast path
if let Some(cached) = token_secret_cache().read().unwrap().secrets.get(tokenid) {
// Compare cached secret with provided one using constant time comparison
@@ -136,12 +165,18 @@ struct ApiTokenSecretCache {
/// `generate_and_set_secret`. Used to avoid repeated
/// password-hash computation on subsequent authentications.
secrets: HashMap<Authid, String>,
+ // shadow file mtime to detect changes
+ file_mtime: Option<SystemTime>,
+ // shadow file length to detect changes
+ file_len: Option<u64>,
}
fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
TOKEN_SECRET_CACHE.get_or_init(|| {
RwLock::new(ApiTokenSecretCache {
secrets: HashMap::new(),
+ file_mtime: None,
+ file_len: None,
})
})
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox 3/3] proxmox-access-control: add TTL window to token secret cache
2025-12-05 13:25 15% [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Samuel Rufinatscha
` (4 preceding siblings ...)
2025-12-05 13:25 15% ` [pbs-devel] [PATCH proxmox 2/3] proxmox-access-control: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
@ 2025-12-05 13:25 16% ` Samuel Rufinatscha
2025-12-05 14:06 5% ` [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Shannon Sterz
2025-12-17 16:27 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
7 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-05 13:25 UTC (permalink / raw)
To: pbs-devel
Verify_secret() currently calls refresh_cache_if_file_changed() on every
request, which performs a metadata() call on token.shadow each time.
Under load this adds unnecessary overhead, considering also the file
should rarely change.
This patch introduces a TTL boundary, controlled by
TOKEN_SECRET_CACHE_TTL_SECS. File metadata is only re-loaded once the
TTL has expired.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-access-control/src/token_shadow.rs | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/proxmox-access-control/src/token_shadow.rs b/proxmox-access-control/src/token_shadow.rs
index d08fb06a..885e629d 100644
--- a/proxmox-access-control/src/token_shadow.rs
+++ b/proxmox-access-control/src/token_shadow.rs
@@ -9,6 +9,7 @@ use serde_json::{from_value, Value};
use proxmox_auth_api::types::Authid;
use proxmox_product_config::{open_api_lockfile, replace_config, ApiLockGuard};
+use proxmox_time::epoch_i64;
use crate::init::impl_feature::{token_shadow, token_shadow_lock};
@@ -18,6 +19,8 @@ use crate::init::impl_feature::{token_shadow, token_shadow_lock};
/// subsequent authentications for the same token+secret combination, avoiding
/// recomputing the password hash on every request.
static TOKEN_SECRET_CACHE: OnceLock<RwLock<ApiTokenSecretCache>> = OnceLock::new();
+/// Max age in seconds of the token secret cache before checking for file changes.
+const TOKEN_SECRET_CACHE_TTL_SECS: i64 = 60;
// Get exclusive lock
fn lock_config() -> Result<ApiLockGuard, Error> {
@@ -44,6 +47,15 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
fn refresh_cache_if_file_changed() -> Result<(), Error> {
let mut cache = token_secret_cache().write().unwrap();
+ let now = epoch_i64();
+
+ // Fast path: Within TTL boundary
+ if let Some(last) = cache.last_checked {
+ if now - last < TOKEN_SECRET_CACHE_TTL_SECS {
+ return Ok(());
+ }
+ }
+
// Fetch the current token.shadow metadata
let (new_mtime, new_len) = match fs::metadata(token_shadow().as_path()) {
Ok(meta) => (meta.modified().ok(), Some(meta.len())),
@@ -60,6 +72,7 @@ fn refresh_cache_if_file_changed() -> Result<(), Error> {
cache.secrets.clear();
cache.file_mtime = new_mtime;
cache.file_len = new_len;
+ cache.last_checked = Some(now);
Ok(())
}
@@ -150,6 +163,8 @@ struct ApiTokenSecretCache {
file_mtime: Option<SystemTime>,
// shadow file length to detect changes
file_len: Option<u64>,
+ // last time the file metadata was checked
+ last_checked: Option<i64>,
}
fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
@@ -158,6 +173,7 @@ fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
secrets: HashMap::new(),
file_mtime: None,
file_len: None,
+ last_checked: None,
})
})
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 16%]
* Re: [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets
2025-12-05 13:25 14% ` [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets Samuel Rufinatscha
@ 2025-12-05 14:04 5% ` Shannon Sterz
2025-12-09 13:29 6% ` Samuel Rufinatscha
2025-12-10 11:47 5% ` Fabian Grünbichler
1 sibling, 1 reply; 200+ results
From: Shannon Sterz @ 2025-12-05 14:04 UTC (permalink / raw)
To: Samuel Rufinatscha; +Cc: Proxmox Backup Server development discussion
On Fri Dec 5, 2025 at 2:25 PM CET, Samuel Rufinatscha wrote:
> Currently, every token-based API request reads the token.shadow file and
> runs the expensive password hash verification for the given token
> secret. This shows up as a hotspot in /status profiling (see
> bug #6049 [1]).
>
> This patch introduces an in-memory cache of successfully verified token
> secrets. Subsequent requests for the same token+secret combination only
> perform a comparison using openssl::memcmp::eq and avoid re-running the
> password hash. The cache is updated when a token secret is set and
> cleared when a token is deleted. Note, this does NOT include manual
> config changes, which will be covered in a subsequent patch.
>
> This patch partly fixes bug #6049 [1].
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> pbs-config/src/token_shadow.rs | 58 +++++++++++++++++++++++++++++++++-
> 1 file changed, 57 insertions(+), 1 deletion(-)
>
> diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
> index 640fabbf..47aa2fc2 100644
> --- a/pbs-config/src/token_shadow.rs
> +++ b/pbs-config/src/token_shadow.rs
> @@ -1,6 +1,8 @@
> use std::collections::HashMap;
> +use std::sync::RwLock;
>
> use anyhow::{bail, format_err, Error};
> +use once_cell::sync::OnceCell;
> use serde::{Deserialize, Serialize};
> use serde_json::{from_value, Value};
>
> @@ -13,6 +15,13 @@ use crate::{open_backup_lockfile, BackupLockGuard};
> const LOCK_FILE: &str = pbs_buildcfg::configdir!("/token.shadow.lock");
> const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
>
> +/// Global in-memory cache for successfully verified API token secrets.
> +/// The cache stores plain text secrets for token Authids that have already been
> +/// verified against the hashed values in `token.shadow`. This allows for cheap
> +/// subsequent authentications for the same token+secret combination, avoiding
> +/// recomputing the password hash on every request.
> +static TOKEN_SECRET_CACHE: OnceCell<RwLock<ApiTokenSecretCache>> = OnceCell::new();
any reason you are using a once cell with a cutom get_or_init function
instead of a simple `LazyCell` [1] here? seems to me that this would be
the more appropriate type here? similar question for the
proxmox-access-control portion of this series.
[1]: https://doc.rust-lang.org/std/cell/struct.LazyCell.html
> +
> #[derive(Serialize, Deserialize)]
> #[serde(rename_all = "kebab-case")]
> /// ApiToken id / secret pair
> @@ -54,9 +63,25 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
> bail!("not an API token ID");
> }
>
> + // Fast path
> + if let Some(cached) = token_secret_cache().read().unwrap().secrets.get(tokenid) {
> + // Compare cached secret with provided one using constant time comparison
> + if openssl::memcmp::eq(cached.as_bytes(), secret.as_bytes()) {
> + // Already verified before
> + return Ok(());
> + }
> + // Fall through to slow path if secret doesn't match cached one
> + }
> +
> + // Slow path: read file + verify hash
> let data = read_file()?;
> match data.get(tokenid) {
> - Some(hashed_secret) => proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
> + Some(hashed_secret) => {
> + proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
> + // Cache the plain secret for future requests
> + cache_insert_secret(tokenid.clone(), secret.to_owned());
> + Ok(())
> + }
> None => bail!("invalid API token"),
> }
> }
> @@ -82,6 +107,8 @@ fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
> data.insert(tokenid.clone(), hashed_secret);
> write_file(data)?;
>
> + cache_insert_secret(tokenid.clone(), secret.to_owned());
> +
> Ok(())
> }
>
> @@ -97,5 +124,34 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
> data.remove(tokenid);
> write_file(data)?;
>
> + cache_remove_secret(tokenid);
> +
> Ok(())
> }
> +
> +struct ApiTokenSecretCache {
> + /// Keys are token Authids, values are the corresponding plain text secrets.
> + /// Entries are added after a successful on-disk verification in
> + /// `verify_secret` or when a new token secret is generated by
> + /// `generate_and_set_secret`. Used to avoid repeated
> + /// password-hash computation on subsequent authentications.
> + secrets: HashMap<Authid, String>,
> +}
> +
> +fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
> + TOKEN_SECRET_CACHE.get_or_init(|| {
> + RwLock::new(ApiTokenSecretCache {
> + secrets: HashMap::new(),
> + })
> + })
> +}
> +
> +fn cache_insert_secret(tokenid: Authid, secret: String) {
> + let mut cache = token_secret_cache().write().unwrap();
unwrap here could panic if another thread is holding a guard, any reason
to not return a result here and bubble up the error instead?
> + cache.secrets.insert(tokenid, secret);
> +}
> +
> +fn cache_remove_secret(tokenid: &Authid) {
> + let mut cache = token_secret_cache().write().unwrap();
same here and in the following patches (i won't comment on each
occurrence there separately.)
> + cache.secrets.remove(tokenid);
> +}
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead
2025-12-05 13:25 15% [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Samuel Rufinatscha
` (5 preceding siblings ...)
2025-12-05 13:25 16% ` [pbs-devel] [PATCH proxmox 3/3] proxmox-access-control: add TTL window to token secret cache Samuel Rufinatscha
@ 2025-12-05 14:06 5% ` Shannon Sterz
2025-12-09 13:58 6% ` Samuel Rufinatscha
2025-12-17 16:27 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
7 siblings, 1 reply; 200+ results
From: Shannon Sterz @ 2025-12-05 14:06 UTC (permalink / raw)
To: Samuel Rufinatscha; +Cc: Proxmox Backup Server development discussion
thank you for this series and the extensive documentation in it. it was
very easy to follow. the changes look good to me for the most part, see
the comments on the first patch. one top level question, though:
should we publicly document that manually editing the token.shadow will
now not instantly make requests by tokens invalid, but changes will take
up to one minute to take effect? i don't think that this is necessarily
and issue, but imo we shouldn't make such a change without informing
users.
other than this and the comments in-line, consider this:
Reviewed-by: Shannon Sterz <s.sterz@proxmox.com>
On Fri Dec 5, 2025 at 2:25 PM CET, Samuel Rufinatscha wrote:
> Hi,
>
> this series improves the performance of token-based API authentication
> in PBS (pbs-config) and in PDM (underlying proxmox-access-control
> crate), addressing the API token verification hotspot reported in our
> bugtracker #6049 [1].
>
> When profiling PBS /status endpoint with cargo flamegraph [2],
> token-based authentication showed up as a dominant hotspot via
> proxmox_sys::crypt::verify_crypt_pw. Applying this series removes that
> path from the hot section of the flamegraph. The same performance issue
> was measured [3] for PDM. PDM uses the underlying shared
> proxmox-access-control library for token handling, which is a
> factored out version of the token.shadow handling code from PBS.
>
> While this series fixes the immediate performance issue both in PBS
> (pbs-config) and in the shared proxmox-access-control crate used by
> PDM, PBS should eventually, ideally be refactored, in a separate
> effort, to use proxmox-access-control for token handling instead of its
> local implementation.
>
> Problem
>
> For token-based API requests, both PBS’s pbs-config token.shadow
> handling and PDM proxmox-access-control’s token.shadow handling
> currently:
>
> 1. read the token.shadow file on each request
> 2. deserialize it into a HashMap<Authid, String>
> 3. run password hash verification via
> proxmox_sys::crypt::verify_crypt_pw for the provided token secret
>
> Under load, this results in significant CPU usage spent in repeated
> password hash computations for the same token+secret pairs. The
> attached flamegraphs for PBS [2] and PDM [3] show
> proxmox_sys::crypt::verify_crypt_pw dominating the hot path.
>
> Approach
>
> The goal is to reduce the cost of token-based authentication preserving
> the existing token handling semantics (including detecting manual edits
> to token.shadow) and be consistent between PBS (pbs-config) and
> PDM (proxmox-access-control). For both sites, the series proposes
> following approach:
>
> 1. Introduce an in-memory cache for verified token secrets
> 2. Invalidate the cache when token.shadow changes (detect manual edits)
> 3. Control metadata checks with a TTL window
>
> Testing
>
> *PBS (pbs-config)*
>
> To verify the effect in PBS, I:
> 1. Set up test environment based on latest PBS ISO, installed Rust
> toolchain, cloned proxmox-backup repository to use with cargo
> flamegraph. Reproduced bug #6049 [1] by profiling the /status
> endpoint with token-based authentication using cargo flamegraph [2].
> The flamegraph showed proxmox_sys::crypt::verify_crypt_pw is the
> hotspot.
> 2. Built PBS with pbs-config patches and re-ran the same workload and
> profiling setup.
> 3. Confirmed that the proxmox_sys::crypt::verify_crypt_pw path no
> longer appears in the hot section of the flamegraph. CPU usage is
> now dominated by TLS overhead.
> 4. Functionally verified that:
> * token-based API authentication still works for valid tokens
> * invalid secrets are rejected as before
> * generating a new token secret via dashboard works and
> authenticates correctly
>
> *PDM (proxmox-access-control)*
>
> To verify the effect in PDM, I followed a similar testing approach.
> Instead of /status, I profiled the /version endpoint with cargo
> flamegraph [3] and verified that the token hashing path disappears
> from the hot section after applying the proxmox-access-control patches.
>
> Functionally I verified that:
> * token-based API authentication still works for valid tokens
> * invalid secrets are rejected as before
> * generating a new token secret via dashboard works and
> authenticates correctly
>
> Patch summary
>
> pbs-config:
>
> 0001 – pbs-config: cache verified API token secrets
> Adds an in-memory cache keyed by Authid that stores plain text token
> secrets after a successful verification or generation and uses
> openssl’s memcmp constant-time for comparison.
>
> 0002 – pbs-config: invalidate token-secret cache on token.shadow changes
> Tracks token.shadow mtime and length and clears the in-memory cache
> when the file changes.
>
> 0003 – pbs-config: add TTL window to token-secret cache
> Introduces a TTL (TOKEN_SECRET_CACHE_TTL_SECS, default 60) for metadata checks so
> that fs::metadata is only called periodically.
>
> proxmox-access-control:
>
> 0004 – access-control: cache verified API token secrets
> Mirrors PBS patch 0001.
>
> 0005 – access-control: invalidate token-secret cache on token.shadow changes
> Mirrors PBS patch 0002.
>
> 0006 – access-control: add TTL window to token-secret cache
> Mirrors PBS patch 0003.
>
> Thanks for considering this patch series, I look forward to your
> feedback.
>
> Best,
> Samuel Rufinatscha
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] Flamegraph illustrating the`proxmox_sys::crypt::verify_crypt_pw
> hotspot before this series (attached to [1])
>
> proxmox-backup:
>
> Samuel Rufinatscha (3):
> pbs-config: cache verified API token secrets
> pbs-config: invalidate token-secret cache on token.shadow changes
> pbs-config: add TTL window to token secret cache
>
> pbs-config/src/token_shadow.rs | 109 ++++++++++++++++++++++++++++++++-
> 1 file changed, 108 insertions(+), 1 deletion(-)
>
>
> proxmox:
>
> Samuel Rufinatscha (3):
> proxmox-access-control: cache verified API token secrets
> proxmox-access-control: invalidate token-secret cache on token.shadow
> changes
> proxmox-access-control: add TTL window to token secret cache
>
> proxmox-access-control/src/token_shadow.rs | 108 ++++++++++++++++++++-
> 1 file changed, 107 insertions(+), 1 deletion(-)
>
>
> Summary over all repositories:
> 2 files changed, 215 insertions(+), 2 deletions(-)
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets
2025-12-05 14:04 5% ` Shannon Sterz
@ 2025-12-09 13:29 6% ` Samuel Rufinatscha
2025-12-17 11:16 5% ` Christian Ebner
0 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-12-09 13:29 UTC (permalink / raw)
To: Shannon Sterz; +Cc: Proxmox Backup Server development discussion
On 12/5/25 3:03 PM, Shannon Sterz wrote:
> On Fri Dec 5, 2025 at 2:25 PM CET, Samuel Rufinatscha wrote:
>> Currently, every token-based API request reads the token.shadow file and
>> runs the expensive password hash verification for the given token
>> secret. This shows up as a hotspot in /status profiling (see
>> bug #6049 [1]).
>>
>> This patch introduces an in-memory cache of successfully verified token
>> secrets. Subsequent requests for the same token+secret combination only
>> perform a comparison using openssl::memcmp::eq and avoid re-running the
>> password hash. The cache is updated when a token secret is set and
>> cleared when a token is deleted. Note, this does NOT include manual
>> config changes, which will be covered in a subsequent patch.
>>
>> This patch partly fixes bug #6049 [1].
>>
>> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> pbs-config/src/token_shadow.rs | 58 +++++++++++++++++++++++++++++++++-
>> 1 file changed, 57 insertions(+), 1 deletion(-)
>>
>> diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
>> index 640fabbf..47aa2fc2 100644
>> --- a/pbs-config/src/token_shadow.rs
>> +++ b/pbs-config/src/token_shadow.rs
>> @@ -1,6 +1,8 @@
>> use std::collections::HashMap;
>> +use std::sync::RwLock;
>>
>> use anyhow::{bail, format_err, Error};
>> +use once_cell::sync::OnceCell;
>> use serde::{Deserialize, Serialize};
>> use serde_json::{from_value, Value};
>>
>> @@ -13,6 +15,13 @@ use crate::{open_backup_lockfile, BackupLockGuard};
>> const LOCK_FILE: &str = pbs_buildcfg::configdir!("/token.shadow.lock");
>> const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
>>
>> +/// Global in-memory cache for successfully verified API token secrets.
>> +/// The cache stores plain text secrets for token Authids that have already been
>> +/// verified against the hashed values in `token.shadow`. This allows for cheap
>> +/// subsequent authentications for the same token+secret combination, avoiding
>> +/// recomputing the password hash on every request.
>> +static TOKEN_SECRET_CACHE: OnceCell<RwLock<ApiTokenSecretCache>> = OnceCell::new();
>
> any reason you are using a once cell with a cutom get_or_init function
> instead of a simple `LazyCell` [1] here? seems to me that this would be
> the more appropriate type here? similar question for the
> proxmox-access-control portion of this series.
>
> [1]: https://doc.rust-lang.org/std/cell/struct.LazyCell.html
>
Good point, we should / can directly initialize it! Will change
to LazyCell. Thanks!
>> +
>> #[derive(Serialize, Deserialize)]
>> #[serde(rename_all = "kebab-case")]
>> /// ApiToken id / secret pair
>> @@ -54,9 +63,25 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
>> bail!("not an API token ID");
>> }
>>
>> + // Fast path
>> + if let Some(cached) = token_secret_cache().read().unwrap().secrets.get(tokenid) {
>> + // Compare cached secret with provided one using constant time comparison
>> + if openssl::memcmp::eq(cached.as_bytes(), secret.as_bytes()) {
>> + // Already verified before
>> + return Ok(());
>> + }
>> + // Fall through to slow path if secret doesn't match cached one
>> + }
>> +
>> + // Slow path: read file + verify hash
>> let data = read_file()?;
>> match data.get(tokenid) {
>> - Some(hashed_secret) => proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
>> + Some(hashed_secret) => {
>> + proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
>> + // Cache the plain secret for future requests
>> + cache_insert_secret(tokenid.clone(), secret.to_owned());
>> + Ok(())
>> + }
>> None => bail!("invalid API token"),
>> }
>> }
>> @@ -82,6 +107,8 @@ fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
>> data.insert(tokenid.clone(), hashed_secret);
>> write_file(data)?;
>>
>> + cache_insert_secret(tokenid.clone(), secret.to_owned());
>> +
>> Ok(())
>> }
>>
>> @@ -97,5 +124,34 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
>> data.remove(tokenid);
>> write_file(data)?;
>>
>> + cache_remove_secret(tokenid);
>> +
>> Ok(())
>> }
>> +
>> +struct ApiTokenSecretCache {
>> + /// Keys are token Authids, values are the corresponding plain text secrets.
>> + /// Entries are added after a successful on-disk verification in
>> + /// `verify_secret` or when a new token secret is generated by
>> + /// `generate_and_set_secret`. Used to avoid repeated
>> + /// password-hash computation on subsequent authentications.
>> + secrets: HashMap<Authid, String>,
>> +}
>> +
>> +fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
>> + TOKEN_SECRET_CACHE.get_or_init(|| {
>> + RwLock::new(ApiTokenSecretCache {
>> + secrets: HashMap::new(),
>> + })
>> + })
>> +}
>> +
>> +fn cache_insert_secret(tokenid: Authid, secret: String) {
>> + let mut cache = token_secret_cache().write().unwrap();
>
> unwrap here could panic if another thread is holding a guard, any reason
> to not return a result here and bubble up the error instead?
>
Unwrap only panics here if another thread panicked while holding the
write lock. If that happens the cache might be in an inconsistent
state and future read() / write() will also return PoisonError. If we
return an error here we return the poison error to every subsequent
request.
I think we can:
– treat this as a hard bug and let the process panic on PoisonError; so
keep write().unwrap()
- catch the error, clear the cache and access the data via .into_inner().
but still forces every future read/write call to handle the poison logic
correctly
I think it makes sense to fail hard here. If the lock is poisoned the
state is likely broken and it seems better to let the process restart
>> + cache.secrets.insert(tokenid, secret);
>> +}
>> +
>> +fn cache_remove_secret(tokenid: &Authid) {
>> + let mut cache = token_secret_cache().write().unwrap();
>
> same here and in the following patches (i won't comment on each
> occurrence there separately.)
>
>> + cache.secrets.remove(tokenid);
>> +}
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead
2025-12-05 14:06 5% ` [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Shannon Sterz
@ 2025-12-09 13:58 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-09 13:58 UTC (permalink / raw)
To: Shannon Sterz; +Cc: Proxmox Backup Server development discussion
Thank you for your great review, Shannon, and for your feedback.
I agree, it would be a good to publicly document the TTL window.
Where would you add this best?
Thanks!
On 12/5/25 3:05 PM, Shannon Sterz wrote:
> thank you for this series and the extensive documentation in it. it was
> very easy to follow. the changes look good to me for the most part, see
> the comments on the first patch. one top level question, though:
>
> should we publicly document that manually editing the token.shadow will
> now not instantly make requests by tokens invalid, but changes will take
> up to one minute to take effect? i don't think that this is necessarily
> and issue, but imo we shouldn't make such a change without informing
> users.
>
> other than this and the comments in-line, consider this:
>
> Reviewed-by: Shannon Sterz <s.sterz@proxmox.com>
>
> On Fri Dec 5, 2025 at 2:25 PM CET, Samuel Rufinatscha wrote:
>> Hi,
>>
>> this series improves the performance of token-based API authentication
>> in PBS (pbs-config) and in PDM (underlying proxmox-access-control
>> crate), addressing the API token verification hotspot reported in our
>> bugtracker #6049 [1].
>>
>> When profiling PBS /status endpoint with cargo flamegraph [2],
>> token-based authentication showed up as a dominant hotspot via
>> proxmox_sys::crypt::verify_crypt_pw. Applying this series removes that
>> path from the hot section of the flamegraph. The same performance issue
>> was measured [3] for PDM. PDM uses the underlying shared
>> proxmox-access-control library for token handling, which is a
>> factored out version of the token.shadow handling code from PBS.
>>
>> While this series fixes the immediate performance issue both in PBS
>> (pbs-config) and in the shared proxmox-access-control crate used by
>> PDM, PBS should eventually, ideally be refactored, in a separate
>> effort, to use proxmox-access-control for token handling instead of its
>> local implementation.
>>
>> Problem
>>
>> For token-based API requests, both PBS’s pbs-config token.shadow
>> handling and PDM proxmox-access-control’s token.shadow handling
>> currently:
>>
>> 1. read the token.shadow file on each request
>> 2. deserialize it into a HashMap<Authid, String>
>> 3. run password hash verification via
>> proxmox_sys::crypt::verify_crypt_pw for the provided token secret
>>
>> Under load, this results in significant CPU usage spent in repeated
>> password hash computations for the same token+secret pairs. The
>> attached flamegraphs for PBS [2] and PDM [3] show
>> proxmox_sys::crypt::verify_crypt_pw dominating the hot path.
>>
>> Approach
>>
>> The goal is to reduce the cost of token-based authentication preserving
>> the existing token handling semantics (including detecting manual edits
>> to token.shadow) and be consistent between PBS (pbs-config) and
>> PDM (proxmox-access-control). For both sites, the series proposes
>> following approach:
>>
>> 1. Introduce an in-memory cache for verified token secrets
>> 2. Invalidate the cache when token.shadow changes (detect manual edits)
>> 3. Control metadata checks with a TTL window
>>
>> Testing
>>
>> *PBS (pbs-config)*
>>
>> To verify the effect in PBS, I:
>> 1. Set up test environment based on latest PBS ISO, installed Rust
>> toolchain, cloned proxmox-backup repository to use with cargo
>> flamegraph. Reproduced bug #6049 [1] by profiling the /status
>> endpoint with token-based authentication using cargo flamegraph [2].
>> The flamegraph showed proxmox_sys::crypt::verify_crypt_pw is the
>> hotspot.
>> 2. Built PBS with pbs-config patches and re-ran the same workload and
>> profiling setup.
>> 3. Confirmed that the proxmox_sys::crypt::verify_crypt_pw path no
>> longer appears in the hot section of the flamegraph. CPU usage is
>> now dominated by TLS overhead.
>> 4. Functionally verified that:
>> * token-based API authentication still works for valid tokens
>> * invalid secrets are rejected as before
>> * generating a new token secret via dashboard works and
>> authenticates correctly
>>
>> *PDM (proxmox-access-control)*
>>
>> To verify the effect in PDM, I followed a similar testing approach.
>> Instead of /status, I profiled the /version endpoint with cargo
>> flamegraph [3] and verified that the token hashing path disappears
>> from the hot section after applying the proxmox-access-control patches.
>>
>> Functionally I verified that:
>> * token-based API authentication still works for valid tokens
>> * invalid secrets are rejected as before
>> * generating a new token secret via dashboard works and
>> authenticates correctly
>>
>> Patch summary
>>
>> pbs-config:
>>
>> 0001 – pbs-config: cache verified API token secrets
>> Adds an in-memory cache keyed by Authid that stores plain text token
>> secrets after a successful verification or generation and uses
>> openssl’s memcmp constant-time for comparison.
>>
>> 0002 – pbs-config: invalidate token-secret cache on token.shadow changes
>> Tracks token.shadow mtime and length and clears the in-memory cache
>> when the file changes.
>>
>> 0003 – pbs-config: add TTL window to token-secret cache
>> Introduces a TTL (TOKEN_SECRET_CACHE_TTL_SECS, default 60) for metadata checks so
>> that fs::metadata is only called periodically.
>>
>> proxmox-access-control:
>>
>> 0004 – access-control: cache verified API token secrets
>> Mirrors PBS patch 0001.
>>
>> 0005 – access-control: invalidate token-secret cache on token.shadow changes
>> Mirrors PBS patch 0002.
>>
>> 0006 – access-control: add TTL window to token-secret cache
>> Mirrors PBS patch 0003.
>>
>> Thanks for considering this patch series, I look forward to your
>> feedback.
>>
>> Best,
>> Samuel Rufinatscha
>>
>> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=6049
>> [2] Flamegraph illustrating the`proxmox_sys::crypt::verify_crypt_pw
>> hotspot before this series (attached to [1])
>>
>> proxmox-backup:
>>
>> Samuel Rufinatscha (3):
>> pbs-config: cache verified API token secrets
>> pbs-config: invalidate token-secret cache on token.shadow changes
>> pbs-config: add TTL window to token secret cache
>>
>> pbs-config/src/token_shadow.rs | 109 ++++++++++++++++++++++++++++++++-
>> 1 file changed, 108 insertions(+), 1 deletion(-)
>>
>>
>> proxmox:
>>
>> Samuel Rufinatscha (3):
>> proxmox-access-control: cache verified API token secrets
>> proxmox-access-control: invalidate token-secret cache on token.shadow
>> changes
>> proxmox-access-control: add TTL window to token secret cache
>>
>> proxmox-access-control/src/token_shadow.rs | 108 ++++++++++++++++++++-
>> 1 file changed, 107 insertions(+), 1 deletion(-)
>>
>>
>> Summary over all repositories:
>> 2 files changed, 215 insertions(+), 2 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox{-backup, } v4 0/8] fix #6939: acme: support servers returning 204 for nonce requests
2025-12-03 10:22 11% [pbs-devel] [PATCH proxmox{-backup, } v4 " Samuel Rufinatscha
` (7 preceding siblings ...)
2025-12-03 10:22 14% ` [pbs-devel] [PATCH proxmox v4 4/4] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
@ 2025-12-09 16:50 5% ` Max R. Carrara
2025-12-10 9:44 6% ` Samuel Rufinatscha
2026-01-08 11:48 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
9 siblings, 1 reply; 200+ results
From: Max R. Carrara @ 2025-12-09 16:50 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Wed Dec 3, 2025 at 11:22 AM CET, Samuel Rufinatscha wrote:
> Hi,
>
> this series fixes account registration for ACME providers that return
> HTTP 204 No Content to the newNonce request. Currently, both the PBS
> ACME client and the shared ACME client in proxmox-acme only accept
> HTTP 200 OK for this request. The issue was observed in PBS against a
> custom ACME deployment and reported as bug #6939 [1].
>
> [...]
Testing
-------
Tested this on my local PBS development instance with the DNS-01
challenge using one of my domains on OVH and Let's Encrypt Staging.
The cert was ordered without any problems. Everything worked just as
before.
Comments Regarding the Changes Made
-----------------------------------
Overall, looks pretty good! I only found a few minor things, see my
comments inline.
What I would recommend overall is to make the changes in `proxmox`
first, and then use the new `async fn` you introduced in patch #4
(proxmox) in `proxmox-backup` instead of doing things the other way
around. That way you could perhaps use the function you introduced,
since I'm assuming you added it for good reason.
Conclusion
----------
LGTM—needs a teeny tiny bit more polish (see comments inline), but
otherwise works great already! :D Good to see a lot of redundant code
being removed.
The few things I mentioned inline aren't *strict* blockers IMO and can
maybe be addressed in a couple follow-up patches, if this gets merged as
is. Otherwise, should you release a v5 of this series, I'll do another
review.
Anyhow, should the maintainer decide to merge this series, please
consider:
Reviewed-by: Max R. Carrara <m.carrara@proxmox.com>
Tested-by: Max R. Carrara <m.carrara@proxmox.com>
>
> proxmox-backup:
>
> Samuel Rufinatscha (4):
> acme: include proxmox-acme-api dependency
> acme: drop local AcmeClient
> acme: change API impls to use proxmox-acme-api handlers
> acme: certificate ordering through proxmox-acme-api
>
> Cargo.toml | 3 +
> src/acme/client.rs | 691 -------------------------
> src/acme/mod.rs | 5 -
> src/acme/plugin.rs | 336 ------------
> src/api2/config/acme.rs | 407 ++-------------
> src/api2/node/certificates.rs | 240 ++-------
> src/api2/types/acme.rs | 98 ----
> src/api2/types/mod.rs | 3 -
> src/bin/proxmox-backup-api.rs | 2 +
> src/bin/proxmox-backup-manager.rs | 2 +
> src/bin/proxmox-backup-proxy.rs | 1 +
> src/bin/proxmox_backup_manager/acme.rs | 21 +-
> src/config/acme/mod.rs | 51 +-
> src/config/acme/plugin.rs | 99 +---
> src/config/node.rs | 29 +-
> src/lib.rs | 2 -
> 16 files changed, 103 insertions(+), 1887 deletions(-)
> delete mode 100644 src/acme/client.rs
> delete mode 100644 src/acme/mod.rs
> delete mode 100644 src/acme/plugin.rs
> delete mode 100644 src/api2/types/acme.rs
>
>
> proxmox:
>
> Samuel Rufinatscha (4):
> acme-api: add helper to load client for an account
> acme: reduce visibility of Request type
> acme: introduce http_status module
> fix #6939: acme: support servers returning 204 for nonce requests
>
> proxmox-acme-api/src/account_api_impl.rs | 5 +++++
> proxmox-acme-api/src/lib.rs | 3 ++-
> proxmox-acme/src/account.rs | 27 +++++++++++++-----------
> proxmox-acme/src/async_client.rs | 8 +++----
> proxmox-acme/src/authorization.rs | 2 +-
> proxmox-acme/src/client.rs | 8 +++----
> proxmox-acme/src/lib.rs | 6 ++----
> proxmox-acme/src/order.rs | 2 +-
> proxmox-acme/src/request.rs | 25 +++++++++++++++-------
> 9 files changed, 51 insertions(+), 35 deletions(-)
>
>
> Summary over all repositories:
> 25 files changed, 154 insertions(+), 1922 deletions(-)
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox v4 2/4] acme: reduce visibility of Request type
2025-12-03 10:22 12% ` [pbs-devel] [PATCH proxmox v4 2/4] acme: reduce visibility of Request type Samuel Rufinatscha
@ 2025-12-09 16:51 5% ` Max R. Carrara
0 siblings, 0 replies; 200+ results
From: Max R. Carrara @ 2025-12-09 16:51 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Wed Dec 3, 2025 at 11:22 AM CET, Samuel Rufinatscha wrote:
> Currently, the low-level ACME Request type is publicly exposed, even
> though users are expected to go through AcmeClient and
> proxmox-acme-api handlers. This patch reduces visibility so that
> the Request type and related fields/methods are crate-internal only.
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> proxmox-acme/src/account.rs | 17 ++++++++++-------
> proxmox-acme/src/async_client.rs | 2 +-
> proxmox-acme/src/authorization.rs | 2 +-
> proxmox-acme/src/client.rs | 6 +++---
> proxmox-acme/src/lib.rs | 4 ----
> proxmox-acme/src/order.rs | 2 +-
> proxmox-acme/src/request.rs | 12 ++++++------
> 7 files changed, 22 insertions(+), 23 deletions(-)
>
> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
> index 0bbf0027..081ca986 100644
> --- a/proxmox-acme/src/account.rs
> +++ b/proxmox-acme/src/account.rs
> @@ -92,7 +92,7 @@ impl Account {
> }
>
> /// Prepare a "POST-as-GET" request to fetch data. Low level helper.
> - pub fn get_request(&self, url: &str, nonce: &str) -> Result<Request, Error> {
> + pub(crate) fn get_request(&self, url: &str, nonce: &str) -> Result<Request, Error> {
> let key = PKey::private_key_from_pem(self.private_key.as_bytes())?;
> let body = serde_json::to_string(&Jws::new_full(
> &key,
> @@ -112,7 +112,7 @@ impl Account {
> }
>
> /// Prepare a JSON POST request. Low level helper.
> - pub fn post_request<T: Serialize>(
> + pub(crate) fn post_request<T: Serialize>(
> &self,
> url: &str,
> nonce: &str,
> @@ -179,7 +179,7 @@ impl Account {
> /// Prepare a request to update account data.
> ///
> /// This is a rather low level interface. You should know what you're doing.
> - pub fn update_account_request<T: Serialize>(
> + pub(crate) fn update_account_request<T: Serialize>(
^ Regarding this function ...
> &self,
> nonce: &str,
> data: &T,
> @@ -188,7 +188,10 @@ impl Account {
> }
>
> /// Prepare a request to deactivate this account.
> - pub fn deactivate_account_request<T: Serialize>(&self, nonce: &str) -> Result<Request, Error> {
> + pub(crate) fn deactivate_account_request<T: Serialize>(
> + &self,
> + nonce: &str,
> + ) -> Result<Request, Error> {
^ and this one ...
> self.post_request_raw_payload(
> &self.location,
> nonce,
> @@ -220,7 +223,7 @@ impl Account {
> ///
> /// This returns a raw `Request` since validation takes some time and the `Authorization`
> /// object has to be re-queried and its `status` inspected.
> - pub fn validate_challenge(
> + pub(crate) fn validate_challenge(
^ as well as this one here, I noticed that they aren't used anywhere in
our code, at least I couldn't find any references to them by grepping
through our sources. Since they're not used at all, we could just remove
them entirely here, IMO. If it's not used, there's not really any point
in keeping those methods around—and as you mentioned, users should be
using `AcmeClient` and `proxmox-acme-api` handlers anyway.
Note that `post_request_raw_payload()` then also becomes redundant,
since its used in `validate_challenge()` and
`deactivate_account_request()`.
> &self,
> authorization: &Authorization,
> challenge_index: usize,
> @@ -274,7 +277,7 @@ pub struct CertificateRevocation<'a> {
>
> impl CertificateRevocation<'_> {
> /// Create the revocation request using the specified nonce for the given directory.
> - pub fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
> + pub(crate) fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
> let revoke_cert = directory.data.revoke_cert.as_ref().ok_or_else(|| {
> Error::Custom("no 'revokeCert' URL specified by provider".to_string())
> })?;
> @@ -364,7 +367,7 @@ impl AccountCreator {
> /// the resulting request.
> /// Changing the private key between using the request and passing the response to
> /// [`response`](AccountCreator::response()) will render the account unusable!
> - pub fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
> + pub(crate) fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
> let key = self.key.as_deref().ok_or(Error::MissingKey)?;
> let url = directory.new_account_url().ok_or_else(|| {
> Error::Custom("no 'newAccount' URL specified by provider".to_string())
> diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
> index dc755fb9..2ff3ba22 100644
> --- a/proxmox-acme/src/async_client.rs
> +++ b/proxmox-acme/src/async_client.rs
> @@ -10,7 +10,7 @@ use proxmox_http::{client::Client, Body};
>
> use crate::account::AccountCreator;
> use crate::order::{Order, OrderData};
> -use crate::Request as AcmeRequest;
> +use crate::request::Request as AcmeRequest;
> use crate::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
>
> /// A non-blocking Acme client using tokio/hyper.
> diff --git a/proxmox-acme/src/authorization.rs b/proxmox-acme/src/authorization.rs
> index 28bc1b4b..765714fc 100644
> --- a/proxmox-acme/src/authorization.rs
> +++ b/proxmox-acme/src/authorization.rs
> @@ -145,7 +145,7 @@ pub struct GetAuthorization {
> /// this is guaranteed to be `Some`.
> ///
> /// The response should be passed to the the [`response`](GetAuthorization::response()) method.
> - pub request: Option<Request>,
> + pub(crate) request: Option<Request>,
> }
>
> impl GetAuthorization {
> diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
> index 931f7245..5c812567 100644
> --- a/proxmox-acme/src/client.rs
> +++ b/proxmox-acme/src/client.rs
> @@ -7,8 +7,8 @@ use serde::{Deserialize, Serialize};
> use crate::b64u;
> use crate::error;
> use crate::order::OrderData;
> -use crate::request::ErrorResponse;
> -use crate::{Account, Authorization, Challenge, Directory, Error, Order, Request};
> +use crate::request::{ErrorResponse, Request};
> +use crate::{Account, Authorization, Challenge, Directory, Error, Order};
>
> macro_rules! format_err {
> ($($fmt:tt)*) => { Error::Client(format!($($fmt)*)) };
> @@ -564,7 +564,7 @@ impl Client {
> }
>
> /// Low-level API to run an n API request. This automatically updates the current nonce!
> - pub fn run_request(&mut self, request: Request) -> Result<HttpResponse, Error> {
> + pub(crate) fn run_request(&mut self, request: Request) -> Result<HttpResponse, Error> {
> self.inner.run_request(request)
> }
>
> diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
> index df722629..6722030c 100644
> --- a/proxmox-acme/src/lib.rs
> +++ b/proxmox-acme/src/lib.rs
> @@ -66,10 +66,6 @@ pub use error::Error;
> #[doc(inline)]
> pub use order::Order;
>
> -#[cfg(feature = "impl")]
> -#[doc(inline)]
> -pub use request::Request;
> -
> // we don't inline these:
> #[cfg(feature = "impl")]
> pub use order::NewOrder;
> diff --git a/proxmox-acme/src/order.rs b/proxmox-acme/src/order.rs
> index b6551004..432a81a4 100644
> --- a/proxmox-acme/src/order.rs
> +++ b/proxmox-acme/src/order.rs
> @@ -153,7 +153,7 @@ pub struct NewOrder {
> //order: OrderData,
> /// The request to execute to place the order. When creating a [`NewOrder`] via
> /// [`Account::new_order`](crate::Account::new_order) this is guaranteed to be `Some`.
> - pub request: Option<Request>,
> + pub(crate) request: Option<Request>,
> }
>
> impl NewOrder {
> diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
> index 78a90913..dadfc5af 100644
> --- a/proxmox-acme/src/request.rs
> +++ b/proxmox-acme/src/request.rs
> @@ -4,21 +4,21 @@ pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
> pub(crate) const CREATED: u16 = 201;
>
> /// A request which should be performed on the ACME provider.
> -pub struct Request {
> +pub(crate) struct Request {
> /// The complete URL to send the request to.
> - pub url: String,
> + pub(crate) url: String,
>
> /// The HTTP method name to use.
> - pub method: &'static str,
> + pub(crate) method: &'static str,
>
> /// The `Content-Type` header to pass along.
> - pub content_type: &'static str,
> + pub(crate) content_type: &'static str,
>
> /// The body to pass along with request, or an empty string.
> - pub body: String,
> + pub(crate) body: String,
>
> /// The expected status code a compliant ACME provider will return on success.
> - pub expected: u16,
> + pub(crate) expected: u16,
> }
>
> /// An ACME error response contains a specially formatted type string, and can optionally
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v4 2/4] acme: drop local AcmeClient
2025-12-03 10:22 6% ` [pbs-devel] [PATCH proxmox-backup v4 2/4] acme: drop local AcmeClient Samuel Rufinatscha
@ 2025-12-09 16:50 4% ` Max R. Carrara
0 siblings, 0 replies; 200+ results
From: Max R. Carrara @ 2025-12-09 16:50 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Wed Dec 3, 2025 at 11:22 AM CET, Samuel Rufinatscha wrote:
> PBS currently uses its own ACME client and API logic, while PDM uses the
> factored out proxmox-acme and proxmox-acme-api crates. This duplication
> risks differences in behaviour and requires ACME maintenance in two
> places. This patch is part of a series to move PBS over to the shared
> ACME stack.
>
> Changes:
> - Remove the local src/acme/client.rs and switch to
> proxmox_acme::async_client::AcmeClient where needed.
> - Use proxmox_acme_api::load_client_with_account to the custom
> AcmeClient::load() function
> - Replace the local do_register() logic with
> proxmox_acme_api::register_account, to further ensure accounts are persisted
> - Replace the local AcmeAccountName type, required for
> proxmox_acme_api::register_account
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
Since you changed a lot of the imported types and traits in this patch
(and later ones), note that we have a particular ordering regarding
imports:
(At least as far as I'm aware at least; otherwise, someone please
correct me if I'm wrong)
1. imports from the stdlib
2. imports from external dependencies
3. imports from internal dependencies (so, mostly stuff from proxmox/)
4. imports from crates local to the repository
5. imports from the current crate
All of these groups are then separated by a blank line. The `use`
statements within those groups are (usually) ordered alphabetically. For
some examples, just browse around PBS a little bit.
Note that we're not suuuper strict about it, since we seem to not follow
that all too precisely in some isolated cases, but nevertheless, it's
good to stick to that format in order to keep things neat.
Unfortunately this isn't something we've automated yet due to it not
(completely?) supported in `rustfmt` / `cargo fmt` AFAIK. `cargo fmt`
should at least sort the individual groups, though.
Also, one apparently common exception to that format is the placement of
`pbs_api_types`—sometimes its part of 3., sometimes it's thrown in with
the crates of 4. In my suggestions in this patch (and the following
ones), I've added it to 3. for consistency's sake.
I would say that overall when you add new `use` statements, just make
sure they're added to the corresponding group if it exists already;
otherwise, add the group using the ordering above. It's not worth to
change the ordering of existing groups, at least not as part of the same
patch.
> src/acme/client.rs | 691 -------------------------
> src/acme/mod.rs | 3 -
> src/acme/plugin.rs | 2 +-
> src/api2/config/acme.rs | 50 +-
> src/api2/node/certificates.rs | 2 +-
> src/api2/types/acme.rs | 8 -
> src/bin/proxmox_backup_manager/acme.rs | 17 +-
> src/config/acme/mod.rs | 8 +-
> src/config/node.rs | 9 +-
> 9 files changed, 36 insertions(+), 754 deletions(-)
> delete mode 100644 src/acme/client.rs
>
> diff --git a/src/acme/client.rs b/src/acme/client.rs
> deleted file mode 100644
> index 9fb6ad55..00000000
> --- a/src/acme/client.rs
> +++ /dev/null
> @@ -1,691 +0,0 @@
snip 8<---------
> diff --git a/src/acme/mod.rs b/src/acme/mod.rs
> index bf61811c..cc561f9a 100644
> --- a/src/acme/mod.rs
> +++ b/src/acme/mod.rs
> @@ -1,5 +1,2 @@
> -mod client;
> -pub use client::AcmeClient;
> -
> pub(crate) mod plugin;
> pub(crate) use plugin::get_acme_plugin;
> diff --git a/src/acme/plugin.rs b/src/acme/plugin.rs
> index f756e9b5..5bc09e1f 100644
> --- a/src/acme/plugin.rs
> +++ b/src/acme/plugin.rs
> @@ -20,8 +20,8 @@ use tokio::process::Command;
>
> use proxmox_acme::{Authorization, Challenge};
>
> -use crate::acme::AcmeClient;
> use crate::api2::types::AcmeDomain;
> +use proxmox_acme::async_client::AcmeClient;
> use proxmox_rest_server::WorkerTask;
use proxmox_acme::{Authorization, Challenge};
use proxmox_acme::async_client::AcmeClient;
use proxmox_rest_server::WorkerTask;
use crate::api2::types::AcmeDomain;
>
> use crate::config::acme::plugin::{DnsPlugin, PluginData};
> diff --git a/src/api2/config/acme.rs b/src/api2/config/acme.rs
> index 35c3fb77..02f88e2e 100644
> --- a/src/api2/config/acme.rs
> +++ b/src/api2/config/acme.rs
> @@ -16,15 +16,15 @@ use proxmox_router::{
> use proxmox_schema::{api, param_bail};
>
> use proxmox_acme::types::AccountData as AcmeAccountData;
> -use proxmox_acme::Account;
>
> use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
>
> -use crate::acme::AcmeClient;
> -use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
> +use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
> use crate::config::acme::plugin::{
> self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
> };
> +use proxmox_acme::async_client::AcmeClient;
> +use proxmox_acme_api::AcmeAccountName;
> use proxmox_rest_server::WorkerTask;
This file is a good example where we weren't strictly following that
format yet ...
use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
use proxmox_acme::async_client::AcmeClient;
use proxmox_acme::types::AccountData as AcmeAccountData;
use proxmox_acme_api::AcmeAccountName;
use proxmox_rest_server::WorkerTask;
use proxmox_schema::{api, param_bail};
use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
use crate::config::acme::plugin::{
self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
};
>
> pub(crate) const ROUTER: Router = Router::new()
> @@ -143,15 +143,15 @@ pub struct AccountInfo {
> )]
> /// Return existing ACME account information.
> pub async fn get_account(name: AcmeAccountName) -> Result<AccountInfo, Error> {
> - let client = AcmeClient::load(&name).await?;
> - let account = client.account()?;
> + let account_info = proxmox_acme_api::get_account(name).await?;
> +
> Ok(AccountInfo {
> - location: account.location.clone(),
> - tos: client.tos().map(str::to_owned),
> - directory: client.directory_url().to_owned(),
> + location: account_info.location,
> + tos: account_info.tos,
> + directory: account_info.directory,
> account: AcmeAccountData {
> only_return_existing: false, // don't actually write this out in case it's set
> - ..account.data.clone()
> + ..account_info.account
> },
> })
> }
> @@ -240,41 +240,24 @@ fn register_account(
> auth_id.to_string(),
> true,
> move |_worker| async move {
> - let mut client = AcmeClient::new(directory);
> -
> info!("Registering ACME account '{}'...", &name);
>
> - let account = do_register_account(
> - &mut client,
> + let location = proxmox_acme_api::register_account(
> &name,
> - tos_url.is_some(),
> contact,
> - None,
> + tos_url,
> + Some(directory),
> eab_kid.zip(eab_hmac_key),
> )
> .await?;
>
> - info!("Registration successful, account URL: {}", account.location);
> + info!("Registration successful, account URL: {}", location);
>
> Ok(())
> },
> )
> }
>
> -pub async fn do_register_account<'a>(
> - client: &'a mut AcmeClient,
> - name: &AcmeAccountName,
> - agree_to_tos: bool,
> - contact: String,
> - rsa_bits: Option<u32>,
> - eab_creds: Option<(String, String)>,
> -) -> Result<&'a Account, Error> {
> - let contact = account_contact_from_string(&contact);
> - client
> - .new_account(name, agree_to_tos, contact, rsa_bits, eab_creds)
> - .await
> -}
> -
> #[api(
> input: {
> properties: {
> @@ -312,7 +295,10 @@ pub fn update_account(
> None => json!({}),
> };
>
> - AcmeClient::load(&name).await?.update_account(&data).await?;
> + proxmox_acme_api::load_client_with_account(&name)
> + .await?
> + .update_account(&data)
> + .await?;
>
> Ok(())
> },
> @@ -350,7 +336,7 @@ pub fn deactivate_account(
> auth_id.to_string(),
> true,
> move |_worker| async move {
> - match AcmeClient::load(&name)
> + match proxmox_acme_api::load_client_with_account(&name)
> .await?
> .update_account(&json!({"status": "deactivated"}))
> .await
> diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.rs
> index 61ef910e..31196715 100644
> --- a/src/api2/node/certificates.rs
> +++ b/src/api2/node/certificates.rs
> @@ -17,10 +17,10 @@ use pbs_buildcfg::configdir;
> use pbs_tools::cert;
> use tracing::warn;
>
> -use crate::acme::AcmeClient;
> use crate::api2::types::AcmeDomain;
> use crate::config::node::NodeConfig;
> use crate::server::send_certificate_renewal_mail;
> +use proxmox_acme::async_client::AcmeClient;
> use proxmox_rest_server::WorkerTask;
use tracing::warn;
use proxmox_acme::async_client::AcmeClient;
use proxmox_rest_server::WorkerTask;
use pbs_tools::cert;
use crate::api2::types::AcmeDomain;
use crate::config::node::NodeConfig;
use crate::server::send_certificate_renewal_mail;
>
> pub const ROUTER: Router = Router::new()
> diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
> index 210ebdbc..7c9063c0 100644
> --- a/src/api2/types/acme.rs
> +++ b/src/api2/types/acme.rs
> @@ -60,14 +60,6 @@ pub struct KnownAcmeDirectory {
> pub url: &'static str,
> }
>
> -proxmox_schema::api_string_type! {
> - #[api(format: &PROXMOX_SAFE_ID_FORMAT)]
> - /// ACME account name.
> - #[derive(Clone, Eq, PartialEq, Hash, Deserialize, Serialize)]
> - #[serde(transparent)]
> - pub struct AcmeAccountName(String);
> -}
> -
> #[api(
> properties: {
> schema: {
> diff --git a/src/bin/proxmox_backup_manager/acme.rs b/src/bin/proxmox_backup_manager/acme.rs
> index 0f0eafea..bb987b26 100644
> --- a/src/bin/proxmox_backup_manager/acme.rs
> +++ b/src/bin/proxmox_backup_manager/acme.rs
> @@ -7,9 +7,9 @@ use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
> use proxmox_schema::api;
> use proxmox_sys::fs::file_get_contents;
>
> -use proxmox_backup::acme::AcmeClient;
> +use proxmox_acme::async_client::AcmeClient;
> +use proxmox_acme_api::AcmeAccountName;
> use proxmox_backup::api2;
> -use proxmox_backup::api2::types::AcmeAccountName;
> use proxmox_backup::config::acme::plugin::DnsPluginCore;
> use proxmox_backup::config::acme::KNOWN_ACME_DIRECTORIES;
use proxmox_acme::async_client::AcmeClient;
use proxmox_acme_api::AcmeAccountName;
use proxmox_schema::api;
use proxmox_sys::fs::file_get_contents;
use proxmox_backup::acme::AcmeClient;
use proxmox_backup::api2;
use proxmox_backup::api2::types::AcmeAccountName;
use proxmox_backup::config::acme::plugin::DnsPluginCore;
use proxmox_backup::config::acme::KNOWN_ACME_DIRECTORIES;
>
> @@ -188,17 +188,20 @@ async fn register_account(
>
> println!("Attempting to register account with {directory_url:?}...");
>
> - let account = api2::config::acme::do_register_account(
> - &mut client,
> + let tos_agreed = tos_agreed
> + .then(|| directory.terms_of_service_url().map(str::to_owned))
> + .flatten();
> +
> + let location = proxmox_acme_api::register_account(
> &name,
> - tos_agreed,
> contact,
> - None,
> + tos_agreed,
> + Some(directory_url),
> eab_creds,
> )
> .await?;
>
> - println!("Registration successful, account URL: {}", account.location);
> + println!("Registration successful, account URL: {}", location);
>
> Ok(())
> }
> diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
> index 274a23fd..d31b2bc9 100644
> --- a/src/config/acme/mod.rs
> +++ b/src/config/acme/mod.rs
> @@ -10,7 +10,8 @@ use proxmox_sys::fs::{file_read_string, CreateOptions};
>
> use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
>
> -use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
> +use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
> +use proxmox_acme_api::AcmeAccountName;
use proxmox_acme_api::AcmeAccountName;
[...]
use proxmox_sys::fs::{file_read_string, CreateOptions};
use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
>
> pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
> pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
> @@ -35,11 +36,6 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
> create_acme_subdir(ACME_DIR)
> }
>
> -pub(crate) fn make_acme_account_dir() -> Result<(), Error> {
> - make_acme_dir()?;
> - create_acme_subdir(ACME_ACCOUNT_DIR)
> -}
> -
> pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
> KnownAcmeDirectory {
> name: "Let's Encrypt V2",
> diff --git a/src/config/node.rs b/src/config/node.rs
> index d2d6e383..d2a17a49 100644
> --- a/src/config/node.rs
> +++ b/src/config/node.rs
> @@ -16,10 +16,9 @@ use pbs_api_types::{
> use pbs_buildcfg::configdir;
> use pbs_config::{open_backup_lockfile, BackupLockGuard};
>
> -use crate::acme::AcmeClient;
> -use crate::api2::types::{
> - AcmeAccountName, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA,
> -};
> +use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
> +use proxmox_acme::async_client::AcmeClient;
> +use proxmox_acme_api::AcmeAccountName;
use proxmox_acme::async_client::AcmeClient;
use proxmox_acme_api::AcmeAccountName;
use pbs_buildcfg::configdir;
use pbs_config::{open_backup_lockfile, BackupLockGuard};
use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
>
> const CONF_FILE: &str = configdir!("/node.cfg");
> const LOCK_FILE: &str = configdir!("/.node.lck");
> @@ -249,7 +248,7 @@ impl NodeConfig {
> } else {
> AcmeAccountName::from_string("default".to_string())? // should really not happen
> };
> - AcmeClient::load(&account).await
> + proxmox_acme_api::load_client_with_account(&account).await
> }
>
> pub fn acme_domains(&'_ self) -> AcmeDomainIter<'_> {
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 4%]
* Re: [pbs-devel] [PATCH proxmox-backup v4 3/4] acme: change API impls to use proxmox-acme-api handlers
2025-12-03 10:22 8% ` [pbs-devel] [PATCH proxmox-backup v4 3/4] acme: change API impls to use proxmox-acme-api handlers Samuel Rufinatscha
@ 2025-12-09 16:50 5% ` Max R. Carrara
0 siblings, 0 replies; 200+ results
From: Max R. Carrara @ 2025-12-09 16:50 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Wed Dec 3, 2025 at 11:22 AM CET, Samuel Rufinatscha wrote:
> PBS currently uses its own ACME client and API logic, while PDM uses the
> factored out proxmox-acme and proxmox-acme-api crates. This duplication
> risks differences in behaviour and requires ACME maintenance in two
> places. This patch is part of a series to move PBS over to the shared
> ACME stack.
>
> Changes:
> - Replace api2/config/acme.rs API logic with proxmox-acme-api handlers.
> - Drop local caching and helper types that duplicate proxmox-acme-api.
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> src/api2/config/acme.rs | 385 ++-----------------------
> src/api2/types/acme.rs | 16 -
> src/bin/proxmox_backup_manager/acme.rs | 6 +-
> src/config/acme/mod.rs | 44 +--
> 4 files changed, 35 insertions(+), 416 deletions(-)
>
> diff --git a/src/api2/config/acme.rs b/src/api2/config/acme.rs
> index 02f88e2e..a112c8ee 100644
> --- a/src/api2/config/acme.rs
> +++ b/src/api2/config/acme.rs
> @@ -1,31 +1,17 @@
> -use std::fs;
> -use std::ops::ControlFlow;
> -use std::path::Path;
> -use std::sync::{Arc, LazyLock, Mutex};
> -use std::time::SystemTime;
> -
> -use anyhow::{bail, format_err, Error};
> -use hex::FromHex;
> -use serde::{Deserialize, Serialize};
> -use serde_json::{json, Value};
> -use tracing::{info, warn};
> -
> -use proxmox_router::{
> - http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
> -};
> -use proxmox_schema::{api, param_bail};
> -
> -use proxmox_acme::types::AccountData as AcmeAccountData;
> -
> +use anyhow::Error;
> use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
> -
> -use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
> -use crate::config::acme::plugin::{
> - self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
> +use proxmox_acme_api::{
> + AccountEntry, AccountInfo, AcmeAccountName, AcmeChallengeSchema, ChallengeSchemaWrapper,
> + DeletablePluginProperty, DnsPluginCore, DnsPluginCoreUpdater, KnownAcmeDirectory, PluginConfig,
> + DEFAULT_ACME_DIRECTORY_ENTRY, PLUGIN_ID_SCHEMA,
> };
> -use proxmox_acme::async_client::AcmeClient;
> -use proxmox_acme_api::AcmeAccountName;
> +use proxmox_config_digest::ConfigDigest;
> use proxmox_rest_server::WorkerTask;
> +use proxmox_router::{
> + http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
> +};
> +use proxmox_schema::api;
> +use tracing::info;
(See my comment to patch 2/4 for an explanation)
use anyhow::Error;
use tracing::info;
use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
use proxmox_acme_api::{
AccountEntry, AccountInfo, AcmeAccountName, AcmeChallengeSchema, ChallengeSchemaWrapper,
DeletablePluginProperty, DnsPluginCore, DnsPluginCoreUpdater, KnownAcmeDirectory, PluginConfig,
DEFAULT_ACME_DIRECTORY_ENTRY, PLUGIN_ID_SCHEMA,
};
use proxmox_config_digest::ConfigDigest;
use proxmox_rest_server::WorkerTask;
use proxmox_router::{
http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
};
use proxmox_schema::api;
>
> pub(crate) const ROUTER: Router = Router::new()
> .get(&list_subdirs_api_method!(SUBDIRS))
> @@ -67,19 +53,6 @@ const PLUGIN_ITEM_ROUTER: Router = Router::new()
> .put(&API_METHOD_UPDATE_PLUGIN)
> .delete(&API_METHOD_DELETE_PLUGIN);
>
> -#[api(
> - properties: {
> - name: { type: AcmeAccountName },
> - },
> -)]
> -/// An ACME Account entry.
> -///
> -/// Currently only contains a 'name' property.
> -#[derive(Serialize)]
> -pub struct AccountEntry {
> - name: AcmeAccountName,
> -}
> -
> #[api(
> access: {
> permission: &Permission::Privilege(&["system", "certificates"], PRIV_SYS_MODIFY, false),
> @@ -93,40 +66,7 @@ pub struct AccountEntry {
> )]
> /// List ACME accounts.
> pub fn list_accounts() -> Result<Vec<AccountEntry>, Error> {
> - let mut entries = Vec::new();
> - crate::config::acme::foreach_acme_account(|name| {
> - entries.push(AccountEntry { name });
> - ControlFlow::Continue(())
> - })?;
> - Ok(entries)
> -}
> -
> -#[api(
> - properties: {
> - account: { type: Object, properties: {}, additional_properties: true },
> - tos: {
> - type: String,
> - optional: true,
> - },
> - },
> -)]
> -/// ACME Account information.
> -///
> -/// This is what we return via the API.
> -#[derive(Serialize)]
> -pub struct AccountInfo {
> - /// Raw account data.
> - account: AcmeAccountData,
> -
> - /// The ACME directory URL the account was created at.
> - directory: String,
> -
> - /// The account's own URL within the ACME directory.
> - location: String,
> -
> - /// The ToS URL, if the user agreed to one.
> - #[serde(skip_serializing_if = "Option::is_none")]
> - tos: Option<String>,
> + proxmox_acme_api::list_accounts()
> }
>
> #[api(
> @@ -143,23 +83,7 @@ pub struct AccountInfo {
> )]
> /// Return existing ACME account information.
> pub async fn get_account(name: AcmeAccountName) -> Result<AccountInfo, Error> {
> - let account_info = proxmox_acme_api::get_account(name).await?;
> -
> - Ok(AccountInfo {
> - location: account_info.location,
> - tos: account_info.tos,
> - directory: account_info.directory,
> - account: AcmeAccountData {
> - only_return_existing: false, // don't actually write this out in case it's set
> - ..account_info.account
> - },
> - })
> -}
> -
> -fn account_contact_from_string(s: &str) -> Vec<String> {
> - s.split(&[' ', ';', ',', '\0'][..])
> - .map(|s| format!("mailto:{s}"))
> - .collect()
> + proxmox_acme_api::get_account(name).await
> }
>
> #[api(
> @@ -224,15 +148,11 @@ fn register_account(
> );
> }
>
> - if Path::new(&crate::config::acme::account_path(&name)).exists() {
> + if std::path::Path::new(&proxmox_acme_api::account_config_filename(&name)).exists() {
> http_bail!(BAD_REQUEST, "account {} already exists", name);
> }
>
> - let directory = directory.unwrap_or_else(|| {
> - crate::config::acme::DEFAULT_ACME_DIRECTORY_ENTRY
> - .url
> - .to_owned()
> - });
> + let directory = directory.unwrap_or_else(|| DEFAULT_ACME_DIRECTORY_ENTRY.url.to_string());
>
> WorkerTask::spawn(
> "acme-register",
> @@ -288,17 +208,7 @@ pub fn update_account(
> auth_id.to_string(),
> true,
> move |_worker| async move {
> - let data = match contact {
> - Some(data) => json!({
> - "contact": account_contact_from_string(&data),
> - }),
> - None => json!({}),
> - };
> -
> - proxmox_acme_api::load_client_with_account(&name)
> - .await?
> - .update_account(&data)
> - .await?;
> + proxmox_acme_api::update_account(&name, contact).await?;
>
> Ok(())
> },
> @@ -336,18 +246,8 @@ pub fn deactivate_account(
> auth_id.to_string(),
> true,
> move |_worker| async move {
> - match proxmox_acme_api::load_client_with_account(&name)
> - .await?
> - .update_account(&json!({"status": "deactivated"}))
> - .await
> - {
> - Ok(_account) => (),
> - Err(err) if !force => return Err(err),
> - Err(err) => {
> - warn!("error deactivating account {name}, proceeding anyway - {err}");
> - }
> - }
> - crate::config::acme::mark_account_deactivated(&name)?;
> + proxmox_acme_api::deactivate_account(&name, force).await?;
> +
> Ok(())
> },
> )
> @@ -374,15 +274,7 @@ pub fn deactivate_account(
> )]
> /// Get the Terms of Service URL for an ACME directory.
> async fn get_tos(directory: Option<String>) -> Result<Option<String>, Error> {
> - let directory = directory.unwrap_or_else(|| {
> - crate::config::acme::DEFAULT_ACME_DIRECTORY_ENTRY
> - .url
> - .to_owned()
> - });
> - Ok(AcmeClient::new(directory)
> - .terms_of_service_url()
> - .await?
> - .map(str::to_owned))
> + proxmox_acme_api::get_tos(directory).await
> }
>
> #[api(
> @@ -397,52 +289,7 @@ async fn get_tos(directory: Option<String>) -> Result<Option<String>, Error> {
> )]
> /// Get named known ACME directory endpoints.
> fn get_directories() -> Result<&'static [KnownAcmeDirectory], Error> {
> - Ok(crate::config::acme::KNOWN_ACME_DIRECTORIES)
> -}
> -
> -/// Wrapper for efficient Arc use when returning the ACME challenge-plugin schema for serializing
> -struct ChallengeSchemaWrapper {
> - inner: Arc<Vec<AcmeChallengeSchema>>,
> -}
> -
> -impl Serialize for ChallengeSchemaWrapper {
> - fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
> - where
> - S: serde::Serializer,
> - {
> - self.inner.serialize(serializer)
> - }
> -}
> -
> -struct CachedSchema {
> - schema: Arc<Vec<AcmeChallengeSchema>>,
> - cached_mtime: SystemTime,
> -}
> -
> -fn get_cached_challenge_schemas() -> Result<ChallengeSchemaWrapper, Error> {
> - static CACHE: LazyLock<Mutex<Option<CachedSchema>>> = LazyLock::new(|| Mutex::new(None));
> -
> - // the actual loading code
> - let mut last = CACHE.lock().unwrap();
> -
> - let actual_mtime = fs::metadata(crate::config::acme::ACME_DNS_SCHEMA_FN)?.modified()?;
> -
> - let schema = match &*last {
> - Some(CachedSchema {
> - schema,
> - cached_mtime,
> - }) if *cached_mtime >= actual_mtime => schema.clone(),
> - _ => {
> - let new_schema = Arc::new(crate::config::acme::load_dns_challenge_schema()?);
> - *last = Some(CachedSchema {
> - schema: Arc::clone(&new_schema),
> - cached_mtime: actual_mtime,
> - });
> - new_schema
> - }
> - };
> -
> - Ok(ChallengeSchemaWrapper { inner: schema })
> + Ok(proxmox_acme_api::KNOWN_ACME_DIRECTORIES)
> }
>
> #[api(
> @@ -457,69 +304,7 @@ fn get_cached_challenge_schemas() -> Result<ChallengeSchemaWrapper, Error> {
> )]
> /// Get named known ACME directory endpoints.
> fn get_challenge_schema() -> Result<ChallengeSchemaWrapper, Error> {
> - get_cached_challenge_schemas()
> -}
> -
> -#[api]
> -#[derive(Default, Deserialize, Serialize)]
> -#[serde(rename_all = "kebab-case")]
> -/// The API's format is inherited from PVE/PMG:
> -pub struct PluginConfig {
> - /// Plugin ID.
> - plugin: String,
> -
> - /// Plugin type.
> - #[serde(rename = "type")]
> - ty: String,
> -
> - /// DNS Api name.
> - #[serde(skip_serializing_if = "Option::is_none", default)]
> - api: Option<String>,
> -
> - /// Plugin configuration data.
> - #[serde(skip_serializing_if = "Option::is_none", default)]
> - data: Option<String>,
> -
> - /// Extra delay in seconds to wait before requesting validation.
> - ///
> - /// Allows to cope with long TTL of DNS records.
> - #[serde(skip_serializing_if = "Option::is_none", default)]
> - validation_delay: Option<u32>,
> -
> - /// Flag to disable the config.
> - #[serde(skip_serializing_if = "Option::is_none", default)]
> - disable: Option<bool>,
> -}
> -
> -// See PMG/PVE's $modify_cfg_for_api sub
> -fn modify_cfg_for_api(id: &str, ty: &str, data: &Value) -> PluginConfig {
> - let mut entry = data.clone();
> -
> - let obj = entry.as_object_mut().unwrap();
> - obj.remove("id");
> - obj.insert("plugin".to_string(), Value::String(id.to_owned()));
> - obj.insert("type".to_string(), Value::String(ty.to_owned()));
> -
> - // FIXME: This needs to go once the `Updater` is fixed.
> - // None of these should be able to fail unless the user changed the files by hand, in which
> - // case we leave the unmodified string in the Value for now. This will be handled with an error
> - // later.
> - if let Some(Value::String(ref mut data)) = obj.get_mut("data") {
> - if let Ok(new) = proxmox_base64::url::decode_no_pad(&data) {
> - if let Ok(utf8) = String::from_utf8(new) {
> - *data = utf8;
> - }
> - }
> - }
> -
> - // PVE/PMG do this explicitly for ACME plugins...
> - // obj.insert("digest".to_string(), Value::String(digest.clone()));
> -
> - serde_json::from_value(entry).unwrap_or_else(|_| PluginConfig {
> - plugin: "*Error*".to_string(),
> - ty: "*Error*".to_string(),
> - ..Default::default()
> - })
> + proxmox_acme_api::get_cached_challenge_schemas()
> }
>
> #[api(
> @@ -535,12 +320,7 @@ fn modify_cfg_for_api(id: &str, ty: &str, data: &Value) -> PluginConfig {
> )]
> /// List ACME challenge plugins.
> pub fn list_plugins(rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<PluginConfig>, Error> {
> - let (plugins, digest) = plugin::config()?;
> - rpcenv["digest"] = hex::encode(digest).into();
> - Ok(plugins
> - .iter()
> - .map(|(id, (ty, data))| modify_cfg_for_api(id, ty, data))
> - .collect())
> + proxmox_acme_api::list_plugins(rpcenv)
> }
>
> #[api(
> @@ -557,13 +337,7 @@ pub fn list_plugins(rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<PluginConfig>
> )]
> /// List ACME challenge plugins.
> pub fn get_plugin(id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<PluginConfig, Error> {
> - let (plugins, digest) = plugin::config()?;
> - rpcenv["digest"] = hex::encode(digest).into();
> -
> - match plugins.get(&id) {
> - Some((ty, data)) => Ok(modify_cfg_for_api(&id, ty, data)),
> - None => http_bail!(NOT_FOUND, "no such plugin"),
> - }
> + proxmox_acme_api::get_plugin(id, rpcenv)
> }
>
> // Currently we only have "the" standalone plugin and DNS plugins so we can just flatten a
> @@ -595,30 +369,7 @@ pub fn get_plugin(id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<PluginC
> )]
> /// Add ACME plugin configuration.
> pub fn add_plugin(r#type: String, core: DnsPluginCore, data: String) -> Result<(), Error> {
> - // Currently we only support DNS plugins and the standalone plugin is "fixed":
> - if r#type != "dns" {
> - param_bail!("type", "invalid ACME plugin type: {:?}", r#type);
> - }
> -
> - let data = String::from_utf8(proxmox_base64::decode(data)?)
> - .map_err(|_| format_err!("data must be valid UTF-8"))?;
> -
> - let id = core.id.clone();
> -
> - let _lock = plugin::lock()?;
> -
> - let (mut plugins, _digest) = plugin::config()?;
> - if plugins.contains_key(&id) {
> - param_bail!("id", "ACME plugin ID {:?} already exists", id);
> - }
> -
> - let plugin = serde_json::to_value(DnsPlugin { core, data })?;
> -
> - plugins.insert(id, r#type, plugin);
> -
> - plugin::save_config(&plugins)?;
> -
> - Ok(())
> + proxmox_acme_api::add_plugin(r#type, core, data)
> }
>
> #[api(
> @@ -634,26 +385,7 @@ pub fn add_plugin(r#type: String, core: DnsPluginCore, data: String) -> Result<(
> )]
> /// Delete an ACME plugin configuration.
> pub fn delete_plugin(id: String) -> Result<(), Error> {
> - let _lock = plugin::lock()?;
> -
> - let (mut plugins, _digest) = plugin::config()?;
> - if plugins.remove(&id).is_none() {
> - http_bail!(NOT_FOUND, "no such plugin");
> - }
> - plugin::save_config(&plugins)?;
> -
> - Ok(())
> -}
> -
> -#[api()]
> -#[derive(Serialize, Deserialize)]
> -#[serde(rename_all = "kebab-case")]
> -/// Deletable property name
> -pub enum DeletableProperty {
> - /// Delete the disable property
> - Disable,
> - /// Delete the validation-delay property
> - ValidationDelay,
> + proxmox_acme_api::delete_plugin(id)
> }
>
> #[api(
> @@ -675,12 +407,12 @@ pub enum DeletableProperty {
> type: Array,
> optional: true,
> items: {
> - type: DeletableProperty,
> + type: DeletablePluginProperty,
> }
> },
> digest: {
> - description: "Digest to protect against concurrent updates",
> optional: true,
> + type: ConfigDigest,
> },
> },
> },
> @@ -694,65 +426,8 @@ pub fn update_plugin(
> id: String,
> update: DnsPluginCoreUpdater,
> data: Option<String>,
> - delete: Option<Vec<DeletableProperty>>,
> - digest: Option<String>,
> + delete: Option<Vec<DeletablePluginProperty>>,
> + digest: Option<ConfigDigest>,
> ) -> Result<(), Error> {
> - let data = data
> - .as_deref()
> - .map(proxmox_base64::decode)
> - .transpose()?
> - .map(String::from_utf8)
> - .transpose()
> - .map_err(|_| format_err!("data must be valid UTF-8"))?;
> -
> - let _lock = plugin::lock()?;
> -
> - let (mut plugins, expected_digest) = plugin::config()?;
> -
> - if let Some(digest) = digest {
> - let digest = <[u8; 32]>::from_hex(digest)?;
> - crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
> - }
> -
> - match plugins.get_mut(&id) {
> - Some((ty, ref mut entry)) => {
> - if ty != "dns" {
> - bail!("cannot update plugin of type {:?}", ty);
> - }
> -
> - let mut plugin = DnsPlugin::deserialize(&*entry)?;
> -
> - if let Some(delete) = delete {
> - for delete_prop in delete {
> - match delete_prop {
> - DeletableProperty::ValidationDelay => {
> - plugin.core.validation_delay = None;
> - }
> - DeletableProperty::Disable => {
> - plugin.core.disable = None;
> - }
> - }
> - }
> - }
> - if let Some(data) = data {
> - plugin.data = data;
> - }
> - if let Some(api) = update.api {
> - plugin.core.api = api;
> - }
> - if update.validation_delay.is_some() {
> - plugin.core.validation_delay = update.validation_delay;
> - }
> - if update.disable.is_some() {
> - plugin.core.disable = update.disable;
> - }
> -
> - *entry = serde_json::to_value(plugin)?;
> - }
> - None => http_bail!(NOT_FOUND, "no such plugin"),
> - }
> -
> - plugin::save_config(&plugins)?;
> -
> - Ok(())
> + proxmox_acme_api::update_plugin(id, update, data, delete, digest)
> }
> diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
> index 7c9063c0..2905b41b 100644
> --- a/src/api2/types/acme.rs
> +++ b/src/api2/types/acme.rs
> @@ -44,22 +44,6 @@ pub const ACME_DOMAIN_PROPERTY_SCHEMA: Schema =
> .format(&ApiStringFormat::PropertyString(&AcmeDomain::API_SCHEMA))
> .schema();
>
> -#[api(
> - properties: {
> - name: { type: String },
> - url: { type: String },
> - },
> -)]
> -/// An ACME directory endpoint with a name and URL.
> -#[derive(Serialize)]
> -pub struct KnownAcmeDirectory {
> - /// The ACME directory's name.
> - pub name: &'static str,
> -
> - /// The ACME directory's endpoint URL.
> - pub url: &'static str,
> -}
> -
> #[api(
> properties: {
> schema: {
> diff --git a/src/bin/proxmox_backup_manager/acme.rs b/src/bin/proxmox_backup_manager/acme.rs
> index bb987b26..e7bd67af 100644
> --- a/src/bin/proxmox_backup_manager/acme.rs
> +++ b/src/bin/proxmox_backup_manager/acme.rs
> @@ -8,10 +8,8 @@ use proxmox_schema::api;
> use proxmox_sys::fs::file_get_contents;
>
> use proxmox_acme::async_client::AcmeClient;
> -use proxmox_acme_api::AcmeAccountName;
> +use proxmox_acme_api::{AcmeAccountName, DnsPluginCore, KNOWN_ACME_DIRECTORIES};
> use proxmox_backup::api2;
> -use proxmox_backup::config::acme::plugin::DnsPluginCore;
> -use proxmox_backup::config::acme::KNOWN_ACME_DIRECTORIES;
use proxmox_acme::async_client::AcmeClient;
use proxmox_acme_api::{AcmeAccountName, DnsPluginCore, KNOWN_ACME_DIRECTORIES};
[...]
use proxmox_sys::fs::file_get_contents;
use proxmox_backup::api2;
>
> pub fn acme_mgmt_cli() -> CommandLineInterface {
> let cmd_def = CliCommandMap::new()
> @@ -122,7 +120,7 @@ async fn register_account(
>
> match input.trim().parse::<usize>() {
> Ok(n) if n < KNOWN_ACME_DIRECTORIES.len() => {
> - break (KNOWN_ACME_DIRECTORIES[n].url.to_owned(), false);
> + break (KNOWN_ACME_DIRECTORIES[n].url.to_string(), false);
> }
> Ok(n) if n == KNOWN_ACME_DIRECTORIES.len() => {
> input.clear();
> diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
> index d31b2bc9..35cda50b 100644
> --- a/src/config/acme/mod.rs
> +++ b/src/config/acme/mod.rs
> @@ -1,8 +1,7 @@
> use std::collections::HashMap;
> use std::ops::ControlFlow;
> -use std::path::Path;
>
> -use anyhow::{bail, format_err, Error};
> +use anyhow::Error;
> use serde_json::Value;
>
> use proxmox_sys::error::SysError;
This here is alright 🎉
> @@ -10,8 +9,8 @@ use proxmox_sys::fs::{file_read_string, CreateOptions};
>
> use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
>
> -use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
> -use proxmox_acme_api::AcmeAccountName;
> +use crate::api2::types::AcmeChallengeSchema;
> +use proxmox_acme_api::{AcmeAccountName, KnownAcmeDirectory, KNOWN_ACME_DIRECTORIES};
use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
use proxmox_acme_api::{AcmeAccountName, KnownAcmeDirectory, KNOWN_ACME_DIRECTORIES};
use crate::api2::types::AcmeChallengeSchema;
>
> pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
> pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
> @@ -36,23 +35,8 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
> create_acme_subdir(ACME_DIR)
> }
>
> -pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
> - KnownAcmeDirectory {
> - name: "Let's Encrypt V2",
> - url: "https://acme-v02.api.letsencrypt.org/directory",
> - },
> - KnownAcmeDirectory {
> - name: "Let's Encrypt V2 Staging",
> - url: "https://acme-staging-v02.api.letsencrypt.org/directory",
> - },
> -];
> -
> pub const DEFAULT_ACME_DIRECTORY_ENTRY: &KnownAcmeDirectory = &KNOWN_ACME_DIRECTORIES[0];
>
> -pub fn account_path(name: &str) -> String {
> - format!("{ACME_ACCOUNT_DIR}/{name}")
> -}
> -
> pub fn foreach_acme_account<F>(mut func: F) -> Result<(), Error>
> where
> F: FnMut(AcmeAccountName) -> ControlFlow<Result<(), Error>>,
> @@ -83,28 +67,6 @@ where
> }
> }
>
> -pub fn mark_account_deactivated(name: &str) -> Result<(), Error> {
> - let from = account_path(name);
> - for i in 0..100 {
> - let to = account_path(&format!("_deactivated_{name}_{i}"));
> - if !Path::new(&to).exists() {
> - return std::fs::rename(&from, &to).map_err(|err| {
> - format_err!(
> - "failed to move account path {:?} to {:?} - {}",
> - from,
> - to,
> - err
> - )
> - });
> - }
> - }
> - bail!(
> - "No free slot to rename deactivated account {:?}, please cleanup {:?}",
> - from,
> - ACME_ACCOUNT_DIR
> - );
> -}
> -
> pub fn load_dns_challenge_schema() -> Result<Vec<AcmeChallengeSchema>, Error> {
> let raw = file_read_string(ACME_DNS_SCHEMA_FN)?;
> let schemas: serde_json::Map<String, Value> = serde_json::from_str(&raw)?;
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox v4 1/4] acme-api: add helper to load client for an account
2025-12-03 10:22 17% ` [pbs-devel] [PATCH proxmox v4 1/4] acme-api: add helper to load client for an account Samuel Rufinatscha
@ 2025-12-09 16:51 5% ` Max R. Carrara
2025-12-10 10:08 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Max R. Carrara @ 2025-12-09 16:51 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Wed Dec 3, 2025 at 11:22 AM CET, Samuel Rufinatscha wrote:
> The PBS ACME refactoring needs a simple way to obtain an AcmeClient for
> a given configured account without duplicating config wiring. This patch
> adds a load_client_with_account helper in proxmox-acme-api that loads
> the account and constructs a matching client, similarly as PBS previous
> own AcmeClient::load() function.
Hmm, you say *needs* here, but the function added here isn't actually
used in this series ...
I personally don't mind keeping this one around for future cases, but
are there any cases among this series (in PBS or otherwise) where we
could've used this function instead already?
If not, then it's probably not worth keeping it around. I assume you
added it for good reason though, so I suggest to double-check where it's
useful ;)
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> proxmox-acme-api/src/account_api_impl.rs | 5 +++++
> proxmox-acme-api/src/lib.rs | 3 ++-
> 2 files changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/proxmox-acme-api/src/account_api_impl.rs b/proxmox-acme-api/src/account_api_impl.rs
> index ef195908..ca8c8655 100644
> --- a/proxmox-acme-api/src/account_api_impl.rs
> +++ b/proxmox-acme-api/src/account_api_impl.rs
> @@ -116,3 +116,8 @@ pub async fn update_account(name: &AcmeAccountName, contact: Option<String>) ->
>
> Ok(())
> }
> +
> +pub async fn load_client_with_account(account_name: &AcmeAccountName) -> Result<AcmeClient, Error> {
> + let account_data = super::account_config::load_account_config(&account_name).await?;
> + Ok(account_data.client())
> +}
> diff --git a/proxmox-acme-api/src/lib.rs b/proxmox-acme-api/src/lib.rs
> index 623e9e23..96f88ae2 100644
> --- a/proxmox-acme-api/src/lib.rs
> +++ b/proxmox-acme-api/src/lib.rs
> @@ -31,7 +31,8 @@ mod plugin_config;
> mod account_api_impl;
> #[cfg(feature = "impl")]
> pub use account_api_impl::{
> - deactivate_account, get_account, get_tos, list_accounts, register_account, update_account,
> + deactivate_account, get_account, get_tos, list_accounts, load_client_with_account,
> + register_account, update_account,
> };
>
> #[cfg(feature = "impl")]
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v4 4/4] acme: certificate ordering through proxmox-acme-api
2025-12-03 10:22 7% ` [pbs-devel] [PATCH proxmox-backup v4 4/4] acme: certificate ordering through proxmox-acme-api Samuel Rufinatscha
@ 2025-12-09 16:50 5% ` Max R. Carrara
0 siblings, 0 replies; 200+ results
From: Max R. Carrara @ 2025-12-09 16:50 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Wed Dec 3, 2025 at 11:22 AM CET, Samuel Rufinatscha wrote:
> PBS currently uses its own ACME client and API logic, while PDM uses the
> factored out proxmox-acme and proxmox-acme-api crates. This duplication
> risks differences in behaviour and requires ACME maintenance in two
> places. This patch is part of a series to move PBS over to the shared
> ACME stack.
>
> Changes:
> - Replace the custom ACME order/authorization loop in node certificates
> with a call to proxmox_acme_api::order_certificate.
> - Build domain + config data as proxmox-acme-api types
> - Remove obsolete local ACME ordering and plugin glue code.
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> src/acme/mod.rs | 2 -
> src/acme/plugin.rs | 336 ----------------------------------
> src/api2/node/certificates.rs | 240 ++++--------------------
> src/api2/types/acme.rs | 74 --------
> src/api2/types/mod.rs | 3 -
> src/config/acme/mod.rs | 7 +-
> src/config/acme/plugin.rs | 99 +---------
> src/config/node.rs | 22 +--
> src/lib.rs | 2 -
> 9 files changed, 46 insertions(+), 739 deletions(-)
> delete mode 100644 src/acme/mod.rs
> delete mode 100644 src/acme/plugin.rs
> delete mode 100644 src/api2/types/acme.rs
>
> diff --git a/src/acme/mod.rs b/src/acme/mod.rs
> deleted file mode 100644
> index cc561f9a..00000000
> --- a/src/acme/mod.rs
> +++ /dev/null
> @@ -1,2 +0,0 @@
> -pub(crate) mod plugin;
> -pub(crate) use plugin::get_acme_plugin;
> diff --git a/src/acme/plugin.rs b/src/acme/plugin.rs
> deleted file mode 100644
> index 5bc09e1f..00000000
> --- a/src/acme/plugin.rs
> +++ /dev/null
> @@ -1,336 +0,0 @@
snip 8<-------------
> diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.rs
> index 31196715..2a645b4a 100644
> --- a/src/api2/node/certificates.rs
> +++ b/src/api2/node/certificates.rs
> @@ -1,27 +1,19 @@
> -use std::sync::Arc;
> -use std::time::Duration;
> -
> use anyhow::{bail, format_err, Error};
> use openssl::pkey::PKey;
> use openssl::x509::X509;
> use serde::{Deserialize, Serialize};
> use tracing::info;
>
> -use proxmox_router::list_subdirs_api_method;
> -use proxmox_router::SubdirMap;
> -use proxmox_router::{Permission, Router, RpcEnvironment};
> -use proxmox_schema::api;
> -
> +use crate::server::send_certificate_renewal_mail;
> use pbs_api_types::{NODE_SCHEMA, PRIV_SYS_MODIFY};
> use pbs_buildcfg::configdir;
> use pbs_tools::cert;
> -use tracing::warn;
> -
> -use crate::api2::types::AcmeDomain;
> -use crate::config::node::NodeConfig;
> -use crate::server::send_certificate_renewal_mail;
> -use proxmox_acme::async_client::AcmeClient;
> +use proxmox_acme_api::AcmeDomain;
> use proxmox_rest_server::WorkerTask;
> +use proxmox_router::list_subdirs_api_method;
> +use proxmox_router::SubdirMap;
> +use proxmox_router::{Permission, Router, RpcEnvironment};
> +use proxmox_schema::api;
(See my comment on patch 2/4 for more information / context)
use anyhow::{bail, format_err, Error};
use openssl::pkey::PKey;
use openssl::x509::X509;
use serde::{Deserialize, Serialize};
use tracing::info;
use pbs_api_types::{NODE_SCHEMA, PRIV_SYS_MODIFY};
use proxmox_acme_api::AcmeDomain;
use proxmox_rest_server::WorkerTask;
use proxmox_router::SubdirMap;
use proxmox_router::list_subdirs_api_method;
use proxmox_router::{Permission, Router, RpcEnvironment};
use proxmox_schema::api;
use pbs_buildcfg::configdir;
use pbs_tools::cert;
use crate::server::send_certificate_renewal_mail;
>
> pub const ROUTER: Router = Router::new()
> .get(&list_subdirs_api_method!(SUBDIRS))
> @@ -269,193 +261,6 @@ pub async fn delete_custom_certificate() -> Result<(), Error> {
> Ok(())
> }
>
> -struct OrderedCertificate {
> - certificate: hyper::body::Bytes,
> - private_key_pem: Vec<u8>,
> -}
> -
> -async fn order_certificate(
> - worker: Arc<WorkerTask>,
> - node_config: &NodeConfig,
> -) -> Result<Option<OrderedCertificate>, Error> {
> - use proxmox_acme::authorization::Status;
> - use proxmox_acme::order::Identifier;
> -
> - let domains = node_config.acme_domains().try_fold(
> - Vec::<AcmeDomain>::new(),
> - |mut acc, domain| -> Result<_, Error> {
> - let mut domain = domain?;
> - domain.domain.make_ascii_lowercase();
> - if let Some(alias) = &mut domain.alias {
> - alias.make_ascii_lowercase();
> - }
> - acc.push(domain);
> - Ok(acc)
> - },
> - )?;
> -
> - let get_domain_config = |domain: &str| {
> - domains
> - .iter()
> - .find(|d| d.domain == domain)
> - .ok_or_else(|| format_err!("no config for domain '{}'", domain))
> - };
> -
> - if domains.is_empty() {
> - info!("No domains configured to be ordered from an ACME server.");
> - return Ok(None);
> - }
> -
> - let (plugins, _) = crate::config::acme::plugin::config()?;
> -
> - let mut acme = node_config.acme_client().await?;
> -
> - info!("Placing ACME order");
> - let order = acme
> - .new_order(domains.iter().map(|d| d.domain.to_ascii_lowercase()))
> - .await?;
> - info!("Order URL: {}", order.location);
> -
> - let identifiers: Vec<String> = order
> - .data
> - .identifiers
> - .iter()
> - .map(|identifier| match identifier {
> - Identifier::Dns(domain) => domain.clone(),
> - })
> - .collect();
> -
> - for auth_url in &order.data.authorizations {
> - info!("Getting authorization details from '{auth_url}'");
> - let mut auth = acme.get_authorization(auth_url).await?;
> -
> - let domain = match &mut auth.identifier {
> - Identifier::Dns(domain) => domain.to_ascii_lowercase(),
> - };
> -
> - if auth.status == Status::Valid {
> - info!("{domain} is already validated!");
> - continue;
> - }
> -
> - info!("The validation for {domain} is pending");
> - let domain_config: &AcmeDomain = get_domain_config(&domain)?;
> - let plugin_id = domain_config.plugin.as_deref().unwrap_or("standalone");
> - let mut plugin_cfg = crate::acme::get_acme_plugin(&plugins, plugin_id)?
> - .ok_or_else(|| format_err!("plugin '{plugin_id}' for domain '{domain}' not found!"))?;
> -
> - info!("Setting up validation plugin");
> - let validation_url = plugin_cfg
> - .setup(&mut acme, &auth, domain_config, Arc::clone(&worker))
> - .await?;
> -
> - let result = request_validation(&mut acme, auth_url, validation_url).await;
> -
> - if let Err(err) = plugin_cfg
> - .teardown(&mut acme, &auth, domain_config, Arc::clone(&worker))
> - .await
> - {
> - warn!("Failed to teardown plugin '{plugin_id}' for domain '{domain}' - {err}");
> - }
> -
> - result?;
> - }
> -
> - info!("All domains validated");
> - info!("Creating CSR");
> -
> - let csr = proxmox_acme::util::Csr::generate(&identifiers, &Default::default())?;
> - let mut finalize_error_cnt = 0u8;
> - let order_url = &order.location;
> - let mut order;
> - loop {
> - use proxmox_acme::order::Status;
> -
> - order = acme.get_order(order_url).await?;
> -
> - match order.status {
> - Status::Pending => {
> - info!("still pending, trying to finalize anyway");
> - let finalize = order
> - .finalize
> - .as_deref()
> - .ok_or_else(|| format_err!("missing 'finalize' URL in order"))?;
> - if let Err(err) = acme.finalize(finalize, &csr.data).await {
> - if finalize_error_cnt >= 5 {
> - return Err(err);
> - }
> -
> - finalize_error_cnt += 1;
> - }
> - tokio::time::sleep(Duration::from_secs(5)).await;
> - }
> - Status::Ready => {
> - info!("order is ready, finalizing");
> - let finalize = order
> - .finalize
> - .as_deref()
> - .ok_or_else(|| format_err!("missing 'finalize' URL in order"))?;
> - acme.finalize(finalize, &csr.data).await?;
> - tokio::time::sleep(Duration::from_secs(5)).await;
> - }
> - Status::Processing => {
> - info!("still processing, trying again in 30 seconds");
> - tokio::time::sleep(Duration::from_secs(30)).await;
> - }
> - Status::Valid => {
> - info!("valid");
> - break;
> - }
> - other => bail!("order status: {:?}", other),
> - }
> - }
> -
> - info!("Downloading certificate");
> - let certificate = acme
> - .get_certificate(
> - order
> - .certificate
> - .as_deref()
> - .ok_or_else(|| format_err!("missing certificate url in finalized order"))?,
> - )
> - .await?;
> -
> - Ok(Some(OrderedCertificate {
> - certificate,
> - private_key_pem: csr.private_key_pem,
> - }))
> -}
> -
> -async fn request_validation(
> - acme: &mut AcmeClient,
> - auth_url: &str,
> - validation_url: &str,
> -) -> Result<(), Error> {
> - info!("Triggering validation");
> - acme.request_challenge_validation(validation_url).await?;
> -
> - info!("Sleeping for 5 seconds");
> - tokio::time::sleep(Duration::from_secs(5)).await;
> -
> - loop {
> - use proxmox_acme::authorization::Status;
> -
> - let auth = acme.get_authorization(auth_url).await?;
> - match auth.status {
> - Status::Pending => {
> - info!("Status is still 'pending', trying again in 10 seconds");
> - tokio::time::sleep(Duration::from_secs(10)).await;
> - }
> - Status::Valid => return Ok(()),
> - other => bail!(
> - "validating challenge '{}' failed - status: {:?}",
> - validation_url,
> - other
> - ),
> - }
> - }
> -}
> -
> #[api(
> input: {
> properties: {
> @@ -525,9 +330,30 @@ fn spawn_certificate_worker(
>
> let auth_id = rpcenv.get_auth_id().unwrap();
>
> + let acme_config = if let Some(cfg) = node_config.acme_config().transpose()? {
> + cfg
> + } else {
> + proxmox_acme_api::parse_acme_config_string("account=default")?
> + };
> +
> + let domains = node_config.acme_domains().try_fold(
> + Vec::<AcmeDomain>::new(),
> + |mut acc, domain| -> Result<_, Error> {
> + let mut domain = domain?;
> + domain.domain.make_ascii_lowercase();
> + if let Some(alias) = &mut domain.alias {
> + alias.make_ascii_lowercase();
> + }
> + acc.push(domain);
> + Ok(acc)
> + },
> + )?;
> +
> WorkerTask::spawn(name, None, auth_id, true, move |worker| async move {
> let work = || async {
> - if let Some(cert) = order_certificate(worker, &node_config).await? {
> + if let Some(cert) =
> + proxmox_acme_api::order_certificate(worker, &acme_config, &domains).await?
> + {
> crate::config::set_proxy_certificate(&cert.certificate, &cert.private_key_pem)?;
> crate::server::reload_proxy_certificate().await?;
> }
> @@ -563,16 +389,20 @@ pub fn revoke_acme_cert(rpcenv: &mut dyn RpcEnvironment) -> Result<String, Error
>
> let auth_id = rpcenv.get_auth_id().unwrap();
>
> + let acme_config = if let Some(cfg) = node_config.acme_config().transpose()? {
> + cfg
> + } else {
> + proxmox_acme_api::parse_acme_config_string("account=default")?
> + };
> +
> WorkerTask::spawn(
> "acme-revoke-cert",
> None,
> auth_id,
> true,
> move |_worker| async move {
> - info!("Loading ACME account");
> - let mut acme = node_config.acme_client().await?;
> info!("Revoking old certificate");
> - acme.revoke_certificate(cert_pem.as_bytes(), None).await?;
> + proxmox_acme_api::revoke_certificate(&acme_config, &cert_pem.as_bytes()).await?;
> info!("Deleting certificate and regenerating a self-signed one");
> delete_custom_certificate().await?;
> Ok(())
> diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
> deleted file mode 100644
> index 2905b41b..00000000
> --- a/src/api2/types/acme.rs
> +++ /dev/null
> @@ -1,74 +0,0 @@
> -use serde::{Deserialize, Serialize};
> -use serde_json::Value;
> -
> -use proxmox_schema::{api, ApiStringFormat, ApiType, Schema, StringSchema};
> -
> -use pbs_api_types::{DNS_ALIAS_FORMAT, DNS_NAME_FORMAT, PROXMOX_SAFE_ID_FORMAT};
> -
> -#[api(
> - properties: {
> - "domain": { format: &DNS_NAME_FORMAT },
> - "alias": {
> - optional: true,
> - format: &DNS_ALIAS_FORMAT,
> - },
> - "plugin": {
> - optional: true,
> - format: &PROXMOX_SAFE_ID_FORMAT,
> - },
> - },
> - default_key: "domain",
> -)]
> -#[derive(Deserialize, Serialize)]
> -/// A domain entry for an ACME certificate.
> -pub struct AcmeDomain {
> - /// The domain to certify for.
> - pub domain: String,
> -
> - /// The domain to use for challenges instead of the default acme challenge domain.
> - ///
> - /// This is useful if you use CNAME entries to redirect `_acme-challenge.*` domains to a
> - /// different DNS server.
> - #[serde(skip_serializing_if = "Option::is_none")]
> - pub alias: Option<String>,
> -
> - /// The plugin to use to validate this domain.
> - ///
> - /// Empty means standalone HTTP validation is used.
> - #[serde(skip_serializing_if = "Option::is_none")]
> - pub plugin: Option<String>,
> -}
> -
> -pub const ACME_DOMAIN_PROPERTY_SCHEMA: Schema =
> - StringSchema::new("ACME domain configuration string")
> - .format(&ApiStringFormat::PropertyString(&AcmeDomain::API_SCHEMA))
> - .schema();
> -
> -#[api(
> - properties: {
> - schema: {
> - type: Object,
> - additional_properties: true,
> - properties: {},
> - },
> - type: {
> - type: String,
> - },
> - },
> -)]
> -#[derive(Serialize)]
> -/// Schema for an ACME challenge plugin.
> -pub struct AcmeChallengeSchema {
> - /// Plugin ID.
> - pub id: String,
> -
> - /// Human readable name, falls back to id.
> - pub name: String,
> -
> - /// Plugin Type.
> - #[serde(rename = "type")]
> - pub ty: &'static str,
> -
> - /// The plugin's parameter schema.
> - pub schema: Value,
> -}
> diff --git a/src/api2/types/mod.rs b/src/api2/types/mod.rs
> index afc34b30..34193685 100644
> --- a/src/api2/types/mod.rs
> +++ b/src/api2/types/mod.rs
> @@ -4,9 +4,6 @@ use anyhow::bail;
>
> use proxmox_schema::*;
>
> -mod acme;
> -pub use acme::*;
> -
> // File names: may not contain slashes, may not start with "."
> pub const FILENAME_FORMAT: ApiStringFormat = ApiStringFormat::VerifyFn(|name| {
> if name.starts_with('.') {
> diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
> index 35cda50b..afd7abf8 100644
> --- a/src/config/acme/mod.rs
> +++ b/src/config/acme/mod.rs
> @@ -9,8 +9,7 @@ use proxmox_sys::fs::{file_read_string, CreateOptions};
>
> use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
>
> -use crate::api2::types::AcmeChallengeSchema;
> -use proxmox_acme_api::{AcmeAccountName, KnownAcmeDirectory, KNOWN_ACME_DIRECTORIES};
> +use proxmox_acme_api::{AcmeAccountName, AcmeChallengeSchema};
use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
use proxmox_acme_api::{AcmeAccountName, AcmeChallengeSchema};
>
> pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
> pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
> @@ -35,8 +34,6 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
> create_acme_subdir(ACME_DIR)
> }
>
> -pub const DEFAULT_ACME_DIRECTORY_ENTRY: &KnownAcmeDirectory = &KNOWN_ACME_DIRECTORIES[0];
> -
> pub fn foreach_acme_account<F>(mut func: F) -> Result<(), Error>
> where
> F: FnMut(AcmeAccountName) -> ControlFlow<Result<(), Error>>,
> @@ -80,7 +77,7 @@ pub fn load_dns_challenge_schema() -> Result<Vec<AcmeChallengeSchema>, Error> {
> .and_then(Value::as_str)
> .unwrap_or(id)
> .to_owned(),
> - ty: "dns",
> + ty: "dns".into(),
> schema: schema.to_owned(),
> })
> .collect())
> diff --git a/src/config/acme/plugin.rs b/src/config/acme/plugin.rs
> index 18e71199..2e979ffe 100644
> --- a/src/config/acme/plugin.rs
> +++ b/src/config/acme/plugin.rs
> @@ -1,104 +1,15 @@
> use std::sync::LazyLock;
>
> use anyhow::Error;
> -use serde::{Deserialize, Serialize};
> -use serde_json::Value;
> -
> -use proxmox_schema::{api, ApiType, Schema, StringSchema, Updater};
> -use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
> -
> -use pbs_api_types::PROXMOX_SAFE_ID_FORMAT;
> use pbs_config::{open_backup_lockfile, BackupLockGuard};
> -
> -pub const PLUGIN_ID_SCHEMA: Schema = StringSchema::new("ACME Challenge Plugin ID.")
> - .format(&PROXMOX_SAFE_ID_FORMAT)
> - .min_length(1)
> - .max_length(32)
> - .schema();
> +use proxmox_acme_api::PLUGIN_ID_SCHEMA;
> +use proxmox_acme_api::{DnsPlugin, StandalonePlugin};
> +use proxmox_schema::{ApiType, Schema};
> +use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
> +use serde_json::Value;
use std::sync::LazyLock;
use anyhow::Error;
use serde::{Deserialize, Serialize};
use serde_json::Value;
use proxmox_acme_api::PLUGIN_ID_SCHEMA;
use proxmox_acme_api::{DnsPlugin, StandalonePlugin};
use proxmox_schema::{ApiType, Schema};
use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
use pbs_config::{open_backup_lockfile, BackupLockGuard};
>
> pub static CONFIG: LazyLock<SectionConfig> = LazyLock::new(init);
>
> -#[api(
> - properties: {
> - id: { schema: PLUGIN_ID_SCHEMA },
> - },
> -)]
> -#[derive(Deserialize, Serialize)]
> -/// Standalone ACME Plugin for the http-1 challenge.
> -pub struct StandalonePlugin {
> - /// Plugin ID.
> - id: String,
> -}
> -
> -impl Default for StandalonePlugin {
> - fn default() -> Self {
> - Self {
> - id: "standalone".to_string(),
> - }
> - }
> -}
> -
> -#[api(
> - properties: {
> - id: { schema: PLUGIN_ID_SCHEMA },
> - disable: {
> - optional: true,
> - default: false,
> - },
> - "validation-delay": {
> - default: 30,
> - optional: true,
> - minimum: 0,
> - maximum: 2 * 24 * 60 * 60,
> - },
> - },
> -)]
> -/// DNS ACME Challenge Plugin core data.
> -#[derive(Deserialize, Serialize, Updater)]
> -#[serde(rename_all = "kebab-case")]
> -pub struct DnsPluginCore {
> - /// Plugin ID.
> - #[updater(skip)]
> - pub id: String,
> -
> - /// DNS API Plugin Id.
> - pub api: String,
> -
> - /// Extra delay in seconds to wait before requesting validation.
> - ///
> - /// Allows to cope with long TTL of DNS records.
> - #[serde(skip_serializing_if = "Option::is_none", default)]
> - pub validation_delay: Option<u32>,
> -
> - /// Flag to disable the config.
> - #[serde(skip_serializing_if = "Option::is_none", default)]
> - pub disable: Option<bool>,
> -}
> -
> -#[api(
> - properties: {
> - core: { type: DnsPluginCore },
> - },
> -)]
> -/// DNS ACME Challenge Plugin.
> -#[derive(Deserialize, Serialize)]
> -#[serde(rename_all = "kebab-case")]
> -pub struct DnsPlugin {
> - #[serde(flatten)]
> - pub core: DnsPluginCore,
> -
> - // We handle this property separately in the API calls.
> - /// DNS plugin data (base64url encoded without padding).
> - #[serde(with = "proxmox_serde::string_as_base64url_nopad")]
> - pub data: String,
> -}
> -
> -impl DnsPlugin {
> - pub fn decode_data(&self, output: &mut Vec<u8>) -> Result<(), Error> {
> - Ok(proxmox_base64::url::decode_to_vec(&self.data, output)?)
> - }
> -}
> -
> fn init() -> SectionConfig {
> let mut config = SectionConfig::new(&PLUGIN_ID_SCHEMA);
>
> diff --git a/src/config/node.rs b/src/config/node.rs
> index d2a17a49..b9257adf 100644
> --- a/src/config/node.rs
> +++ b/src/config/node.rs
> @@ -6,17 +6,17 @@ use serde::{Deserialize, Serialize};
>
> use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
>
> -use proxmox_http::ProxyConfig;
> -
> use pbs_api_types::{
> EMAIL_SCHEMA, MULTI_LINE_COMMENT_SCHEMA, OPENSSL_CIPHERS_TLS_1_2_SCHEMA,
> OPENSSL_CIPHERS_TLS_1_3_SCHEMA,
> };
> +use proxmox_acme_api::{AcmeConfig, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA};
> +use proxmox_http::ProxyConfig;
>
> use pbs_buildcfg::configdir;
> use pbs_config::{open_backup_lockfile, BackupLockGuard};
>
> -use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
> +use crate::api2::types::HTTP_PROXY_SCHEMA;
> use proxmox_acme::async_client::AcmeClient;
> use proxmox_acme_api::AcmeAccountName;
use serde::{Deserialize, Serialize};
use pbs_api_types::{
EMAIL_SCHEMA, MULTI_LINE_COMMENT_SCHEMA, OPENSSL_CIPHERS_TLS_1_2_SCHEMA,
OPENSSL_CIPHERS_TLS_1_3_SCHEMA,
};
use proxmox_acme::async_client::AcmeClient;
use proxmox_acme_api::AcmeAccountName;
use proxmox_acme_api::{AcmeConfig, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA};
use proxmox_http::ProxyConfig;
use proxmox_http::ProxyConfig;
use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
pbs_buildcfg::configdir;
pbs_config::{open_backup_lockfile, BackupLockGuard};
use crate::api2::types::HTTP_PROXY_SCHEMA;
>
> @@ -45,20 +45,6 @@ pub fn save_config(config: &NodeConfig) -> Result<(), Error> {
> pbs_config::replace_backup_config(CONF_FILE, &raw)
> }
>
> -#[api(
> - properties: {
> - account: { type: AcmeAccountName },
> - }
> -)]
> -#[derive(Deserialize, Serialize)]
> -/// The ACME configuration.
> -///
> -/// Currently only contains the name of the account use.
> -pub struct AcmeConfig {
> - /// Account to use to acquire ACME certificates.
> - account: AcmeAccountName,
> -}
> -
> /// All available languages in Proxmox. Taken from proxmox-i18n repository.
> /// pt_BR, zh_CN, and zh_TW use the same case in the translation files.
> // TODO: auto-generate from available translations
> @@ -244,7 +230,7 @@ impl NodeConfig {
>
> pub async fn acme_client(&self) -> Result<AcmeClient, Error> {
> let account = if let Some(cfg) = self.acme_config().transpose()? {
> - cfg.account
> + AcmeAccountName::from_string(cfg.account)?
> } else {
> AcmeAccountName::from_string("default".to_string())? // should really not happen
> };
> diff --git a/src/lib.rs b/src/lib.rs
> index 8633378c..828f5842 100644
> --- a/src/lib.rs
> +++ b/src/lib.rs
> @@ -27,8 +27,6 @@ pub(crate) mod auth;
>
> pub mod tape;
>
> -pub mod acme;
> -
> pub mod client_helpers;
>
> pub mod traffic_control_cache;
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox{-backup, } v4 0/8] fix #6939: acme: support servers returning 204 for nonce requests
2025-12-09 16:50 5% ` [pbs-devel] [PATCH proxmox{-backup, } v4 0/8] " Max R. Carrara
@ 2025-12-10 9:44 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-10 9:44 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Max R. Carrara
On 12/9/25 5:51 PM, Max R. Carrara wrote:
> On Wed Dec 3, 2025 at 11:22 AM CET, Samuel Rufinatscha wrote:
>> Hi,
>>
>> this series fixes account registration for ACME providers that return
>> HTTP 204 No Content to the newNonce request. Currently, both the PBS
>> ACME client and the shared ACME client in proxmox-acme only accept
>> HTTP 200 OK for this request. The issue was observed in PBS against a
>> custom ACME deployment and reported as bug #6939 [1].
>>
>> [...]
>
> Testing
> -------
>
> Tested this on my local PBS development instance with the DNS-01
> challenge using one of my domains on OVH and Let's Encrypt Staging.
>
> The cert was ordered without any problems. Everything worked just as
> before.
>
> Comments Regarding the Changes Made
> -----------------------------------
>
> Overall, looks pretty good! I only found a few minor things, see my
> comments inline.
>
> What I would recommend overall is to make the changes in `proxmox`
> first, and then use the new `async fn` you introduced in patch #4
> (proxmox) in `proxmox-backup` instead of doing things the other way
> around. That way you could perhaps use the function you introduced,
> since I'm assuming you added it for good reason.
>
> Conclusion
> ----------
>
> LGTM—needs a teeny tiny bit more polish (see comments inline), but
> otherwise works great already! :D Good to see a lot of redundant code
> being removed.
>
> The few things I mentioned inline aren't *strict* blockers IMO and can
> maybe be addressed in a couple follow-up patches, if this gets merged as
> is. Otherwise, should you release a v5 of this series, I'll do another
> review.
>
> Anyhow, should the maintainer decide to merge this series, please
> consider:
>
> Reviewed-by: Max R. Carrara <m.carrara@proxmox.com>
> Tested-by: Max R. Carrara <m.carrara@proxmox.com>
>
Thank you Max for the detailed review and for testing! It's great to
hear that this refactor behaves as expected - for you too.
I will make sure to polish the imports as suggested (thanks for
providing them). Also, I will re-order the patches so that PBS can
depend on the newly introduced proxmox-acme-api function.
Regarding the unused Account functions you mentioned in
[PATCH proxmox v4 2/4] acme: reduce visibility of Request type, I agree,
we could probably remove them, especially now that their visibility
changed (pub -> pub(crate)) - will do that!
>>
>> proxmox-backup:
>>
>> Samuel Rufinatscha (4):
>> acme: include proxmox-acme-api dependency
>> acme: drop local AcmeClient
>> acme: change API impls to use proxmox-acme-api handlers
>> acme: certificate ordering through proxmox-acme-api
>>
>> Cargo.toml | 3 +
>> src/acme/client.rs | 691 -------------------------
>> src/acme/mod.rs | 5 -
>> src/acme/plugin.rs | 336 ------------
>> src/api2/config/acme.rs | 407 ++-------------
>> src/api2/node/certificates.rs | 240 ++-------
>> src/api2/types/acme.rs | 98 ----
>> src/api2/types/mod.rs | 3 -
>> src/bin/proxmox-backup-api.rs | 2 +
>> src/bin/proxmox-backup-manager.rs | 2 +
>> src/bin/proxmox-backup-proxy.rs | 1 +
>> src/bin/proxmox_backup_manager/acme.rs | 21 +-
>> src/config/acme/mod.rs | 51 +-
>> src/config/acme/plugin.rs | 99 +---
>> src/config/node.rs | 29 +-
>> src/lib.rs | 2 -
>> 16 files changed, 103 insertions(+), 1887 deletions(-)
>> delete mode 100644 src/acme/client.rs
>> delete mode 100644 src/acme/mod.rs
>> delete mode 100644 src/acme/plugin.rs
>> delete mode 100644 src/api2/types/acme.rs
>>
>>
>> proxmox:
>>
>> Samuel Rufinatscha (4):
>> acme-api: add helper to load client for an account
>> acme: reduce visibility of Request type
>> acme: introduce http_status module
>> fix #6939: acme: support servers returning 204 for nonce requests
>>
>> proxmox-acme-api/src/account_api_impl.rs | 5 +++++
>> proxmox-acme-api/src/lib.rs | 3 ++-
>> proxmox-acme/src/account.rs | 27 +++++++++++++-----------
>> proxmox-acme/src/async_client.rs | 8 +++----
>> proxmox-acme/src/authorization.rs | 2 +-
>> proxmox-acme/src/client.rs | 8 +++----
>> proxmox-acme/src/lib.rs | 6 ++----
>> proxmox-acme/src/order.rs | 2 +-
>> proxmox-acme/src/request.rs | 25 +++++++++++++++-------
>> 9 files changed, 51 insertions(+), 35 deletions(-)
>>
>>
>> Summary over all repositories:
>> 25 files changed, 154 insertions(+), 1922 deletions(-)
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox v4 1/4] acme-api: add helper to load client for an account
2025-12-09 16:51 5% ` Max R. Carrara
@ 2025-12-10 10:08 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-10 10:08 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Max R. Carrara
On 12/9/25 5:51 PM, Max R. Carrara wrote:
> On Wed Dec 3, 2025 at 11:22 AM CET, Samuel Rufinatscha wrote:
>> The PBS ACME refactoring needs a simple way to obtain an AcmeClient for
>> a given configured account without duplicating config wiring. This patch
>> adds a load_client_with_account helper in proxmox-acme-api that loads
>> the account and constructs a matching client, similarly as PBS previous
>> own AcmeClient::load() function.
>
> Hmm, you say *needs* here, but the function added here isn't actually
> used in this series ...
>
> I personally don't mind keeping this one around for future cases, but
> are there any cases among this series (in PBS or otherwise) where we
> could've used this function instead already?
>
> If not, then it's probably not worth keeping it around. I assume you
> added it for good reason though, so I suggest to double-check where it's
> useful ;)
>
Good point about this function! :)
It was originally introduced to support the minimal client-swap
refactor:
[PATCH proxmox-backup v4 2/4] acme: drop local AcmeClient.
Some of its usages could be removed as part of the API implementation
changes in:
[PATCH proxmox-backup v4 3/4] acme: change API impls to use
proxmox-acme-api handlers.
However, it is still required for NodeConfig::acme_client() currently.
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> proxmox-acme-api/src/account_api_impl.rs | 5 +++++
>> proxmox-acme-api/src/lib.rs | 3 ++-
>> 2 files changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/proxmox-acme-api/src/account_api_impl.rs b/proxmox-acme-api/src/account_api_impl.rs
>> index ef195908..ca8c8655 100644
>> --- a/proxmox-acme-api/src/account_api_impl.rs
>> +++ b/proxmox-acme-api/src/account_api_impl.rs
>> @@ -116,3 +116,8 @@ pub async fn update_account(name: &AcmeAccountName, contact: Option<String>) ->
>>
>> Ok(())
>> }
>> +
>> +pub async fn load_client_with_account(account_name: &AcmeAccountName) -> Result<AcmeClient, Error> {
>> + let account_data = super::account_config::load_account_config(&account_name).await?;
>> + Ok(account_data.client())
>> +}
>> diff --git a/proxmox-acme-api/src/lib.rs b/proxmox-acme-api/src/lib.rs
>> index 623e9e23..96f88ae2 100644
>> --- a/proxmox-acme-api/src/lib.rs
>> +++ b/proxmox-acme-api/src/lib.rs
>> @@ -31,7 +31,8 @@ mod plugin_config;
>> mod account_api_impl;
>> #[cfg(feature = "impl")]
>> pub use account_api_impl::{
>> - deactivate_account, get_account, get_tos, list_accounts, register_account, update_account,
>> + deactivate_account, get_account, get_tos, list_accounts, load_client_with_account,
>> + register_account, update_account,
>> };
>>
>> #[cfg(feature = "impl")]
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets
2025-12-05 13:25 14% ` [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets Samuel Rufinatscha
2025-12-05 14:04 5% ` Shannon Sterz
@ 2025-12-10 11:47 5% ` Fabian Grünbichler
2025-12-10 15:35 6% ` Samuel Rufinatscha
1 sibling, 1 reply; 200+ results
From: Fabian Grünbichler @ 2025-12-10 11:47 UTC (permalink / raw)
To: Samuel Rufinatscha, pbs-devel
Quoting Samuel Rufinatscha (2025-12-05 14:25:54)
> Currently, every token-based API request reads the token.shadow file and
> runs the expensive password hash verification for the given token
> secret. This shows up as a hotspot in /status profiling (see
> bug #6049 [1]).
>
> This patch introduces an in-memory cache of successfully verified token
> secrets. Subsequent requests for the same token+secret combination only
> perform a comparison using openssl::memcmp::eq and avoid re-running the
> password hash. The cache is updated when a token secret is set and
> cleared when a token is deleted. Note, this does NOT include manual
> config changes, which will be covered in a subsequent patch.
>
> This patch partly fixes bug #6049 [1].
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> pbs-config/src/token_shadow.rs | 58 +++++++++++++++++++++++++++++++++-
> 1 file changed, 57 insertions(+), 1 deletion(-)
>
> diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
> index 640fabbf..47aa2fc2 100644
> --- a/pbs-config/src/token_shadow.rs
> +++ b/pbs-config/src/token_shadow.rs
> @@ -1,6 +1,8 @@
> use std::collections::HashMap;
> +use std::sync::RwLock;
>
> use anyhow::{bail, format_err, Error};
> +use once_cell::sync::OnceCell;
> use serde::{Deserialize, Serialize};
> use serde_json::{from_value, Value};
>
> @@ -13,6 +15,13 @@ use crate::{open_backup_lockfile, BackupLockGuard};
> const LOCK_FILE: &str = pbs_buildcfg::configdir!("/token.shadow.lock");
> const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
>
> +/// Global in-memory cache for successfully verified API token secrets.
> +/// The cache stores plain text secrets for token Authids that have already been
> +/// verified against the hashed values in `token.shadow`. This allows for cheap
> +/// subsequent authentications for the same token+secret combination, avoiding
> +/// recomputing the password hash on every request.
> +static TOKEN_SECRET_CACHE: OnceCell<RwLock<ApiTokenSecretCache>> = OnceCell::new();
> +
> #[derive(Serialize, Deserialize)]
> #[serde(rename_all = "kebab-case")]
> /// ApiToken id / secret pair
> @@ -54,9 +63,25 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
> bail!("not an API token ID");
> }
>
> + // Fast path
> + if let Some(cached) = token_secret_cache().read().unwrap().secrets.get(tokenid) {
did you benchmark this with a lot of parallel token requests? a plain RwLock
gives no guarantees at all w.r.t. ordering or fairness, so a lot of token-based
requests could effectively prevent token removal AFAICT (or vice-versa,
spamming token creation could lock out all tokens?)
since we don't actually require the cache here to proceed, we could also make this a try_read
or a read with timeout, and fallback to the slow path if there is too much
contention? alternatively, comparing with parking_lot would also be
interesting, since that implementation does have fairness guarantees.
note that token-based requests are basically doable by anyone being able to
reach PBS, whereas token creation/deletion is available to every authenticaed
user.
> + // Compare cached secret with provided one using constant time comparison
> + if openssl::memcmp::eq(cached.as_bytes(), secret.as_bytes()) {
> + // Already verified before
> + return Ok(());
> + }
> + // Fall through to slow path if secret doesn't match cached one
> + }
this could also be a helper, like the rest. then it would consume (a reference
to) the user-provided secret value, instead of giving access to all cached
ones. doesn't make a real difference now other than consistence, but the cache
is (more) cleanly encapsulated then.
> +
> + // Slow path: read file + verify hash
> let data = read_file()?;
> match data.get(tokenid) {
> - Some(hashed_secret) => proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
> + Some(hashed_secret) => {
> + proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
> + // Cache the plain secret for future requests
> + cache_insert_secret(tokenid.clone(), secret.to_owned());
same applies here - storing the value in the cache is optional (and good if it
works), but we don't want to stall forever waiting for the cache insertion to
go through..
> + Ok(())
> + }
> None => bail!("invalid API token"),
> }
> }
> @@ -82,6 +107,8 @@ fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
> data.insert(tokenid.clone(), hashed_secret);
> write_file(data)?;
>
> + cache_insert_secret(tokenid.clone(), secret.to_owned());
this
> +
> Ok(())
> }
>
> @@ -97,5 +124,34 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
> data.remove(tokenid);
> write_file(data)?;
>
> + cache_remove_secret(tokenid);
and this need to block of course and can't be skipped, because otherwise the
read above might operate on wrong data..
> +
> Ok(())
> }
> +
> +struct ApiTokenSecretCache {
> + /// Keys are token Authids, values are the corresponding plain text secrets.
> + /// Entries are added after a successful on-disk verification in
> + /// `verify_secret` or when a new token secret is generated by
> + /// `generate_and_set_secret`. Used to avoid repeated
> + /// password-hash computation on subsequent authentications.
> + secrets: HashMap<Authid, String>,
> +}
> +
> +fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
> + TOKEN_SECRET_CACHE.get_or_init(|| {
> + RwLock::new(ApiTokenSecretCache {
> + secrets: HashMap::new(),
> + })
> + })
> +}
> +
> +fn cache_insert_secret(tokenid: Authid, secret: String) {
> + let mut cache = token_secret_cache().write().unwrap();
> + cache.secrets.insert(tokenid, secret);
> +}
> +
> +fn cache_remove_secret(tokenid: &Authid) {
> + let mut cache = token_secret_cache().write().unwrap();
> + cache.secrets.remove(tokenid);
> +}
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets
2025-12-10 11:47 5% ` Fabian Grünbichler
@ 2025-12-10 15:35 6% ` Samuel Rufinatscha
2025-12-15 15:05 12% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-12-10 15:35 UTC (permalink / raw)
To: Fabian Grünbichler, pbs-devel
On 12/10/25 12:47 PM, Fabian Grünbichler wrote:
> Quoting Samuel Rufinatscha (2025-12-05 14:25:54)
>> Currently, every token-based API request reads the token.shadow file and
>> runs the expensive password hash verification for the given token
>> secret. This shows up as a hotspot in /status profiling (see
>> bug #6049 [1]).
>>
>> This patch introduces an in-memory cache of successfully verified token
>> secrets. Subsequent requests for the same token+secret combination only
>> perform a comparison using openssl::memcmp::eq and avoid re-running the
>> password hash. The cache is updated when a token secret is set and
>> cleared when a token is deleted. Note, this does NOT include manual
>> config changes, which will be covered in a subsequent patch.
>>
>> This patch partly fixes bug #6049 [1].
>>
>> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> pbs-config/src/token_shadow.rs | 58 +++++++++++++++++++++++++++++++++-
>> 1 file changed, 57 insertions(+), 1 deletion(-)
>>
>> diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
>> index 640fabbf..47aa2fc2 100644
>> --- a/pbs-config/src/token_shadow.rs
>> +++ b/pbs-config/src/token_shadow.rs
>> @@ -1,6 +1,8 @@
>> use std::collections::HashMap;
>> +use std::sync::RwLock;
>>
>> use anyhow::{bail, format_err, Error};
>> +use once_cell::sync::OnceCell;
>> use serde::{Deserialize, Serialize};
>> use serde_json::{from_value, Value};
>>
>> @@ -13,6 +15,13 @@ use crate::{open_backup_lockfile, BackupLockGuard};
>> const LOCK_FILE: &str = pbs_buildcfg::configdir!("/token.shadow.lock");
>> const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
>>
>> +/// Global in-memory cache for successfully verified API token secrets.
>> +/// The cache stores plain text secrets for token Authids that have already been
>> +/// verified against the hashed values in `token.shadow`. This allows for cheap
>> +/// subsequent authentications for the same token+secret combination, avoiding
>> +/// recomputing the password hash on every request.
>> +static TOKEN_SECRET_CACHE: OnceCell<RwLock<ApiTokenSecretCache>> = OnceCell::new();
>> +
>> #[derive(Serialize, Deserialize)]
>> #[serde(rename_all = "kebab-case")]
>> /// ApiToken id / secret pair
>> @@ -54,9 +63,25 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
>> bail!("not an API token ID");
>> }
>>
>> + // Fast path
>> + if let Some(cached) = token_secret_cache().read().unwrap().secrets.get(tokenid) {
>
> did you benchmark this with a lot of parallel token requests? a plain RwLock
> gives no guarantees at all w.r.t. ordering or fairness, so a lot of token-based
> requests could effectively prevent token removal AFAICT (or vice-versa,
> spamming token creation could lock out all tokens?)
>
> since we don't actually require the cache here to proceed, we could also make this a try_read
> or a read with timeout, and fallback to the slow path if there is too much
> contention? alternatively, comparing with parking_lot would also be
> interesting, since that implementation does have fairness guarantees.
>
> note that token-based requests are basically doable by anyone being able to
> reach PBS, whereas token creation/deletion is available to every authenticaed
> user.
>
Thanks for the review Fabian and the valuable comments!
I did not benchmark the RwLock itself under load. Your point about
contention/fairness for RwLock makes perfect sense, and we should
consider this. So for v2, I will integrate try_read() /
try_write() as mentioned to avoid possible contention / DoS issues.
I’ll also consider parking_lot::RwLock, thanks for the hint!
>> + // Compare cached secret with provided one using constant time comparison
>> + if openssl::memcmp::eq(cached.as_bytes(), secret.as_bytes()) {
>> + // Already verified before
>> + return Ok(());
>> + }
>> + // Fall through to slow path if secret doesn't match cached one
>> + }
>
> this could also be a helper, like the rest. then it would consume (a reference
> to) the user-provided secret value, instead of giving access to all cached
> ones. doesn't make a real difference now other than consistence, but the cache
> is (more) cleanly encapsulated then.
>
>> +
>> + // Slow path: read file + verify hash
>> let data = read_file()?;
>> match data.get(tokenid) {
>> - Some(hashed_secret) => proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
>> + Some(hashed_secret) => {
>> + proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
>> + // Cache the plain secret for future requests
>> + cache_insert_secret(tokenid.clone(), secret.to_owned());
>
> same applies here - storing the value in the cache is optional (and good if it
> works), but we don't want to stall forever waiting for the cache insertion to
> go through..
>
>> + Ok(())
>> + }
>> None => bail!("invalid API token"),
>> }
>> }
>> @@ -82,6 +107,8 @@ fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
>> data.insert(tokenid.clone(), hashed_secret);
>> write_file(data)?;
>>
>> + cache_insert_secret(tokenid.clone(), secret.to_owned());
>
> this
>
>> +
>> Ok(())
>> }
>>
>> @@ -97,5 +124,34 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
>> data.remove(tokenid);
>> write_file(data)?;
>>
>> + cache_remove_secret(tokenid);
>
> and this need to block of course and can't be skipped, because otherwise the
> read above might operate on wrong data..
>
>> +
>> Ok(())
>> }
>> +
>> +struct ApiTokenSecretCache {
>> + /// Keys are token Authids, values are the corresponding plain text secrets.
>> + /// Entries are added after a successful on-disk verification in
>> + /// `verify_secret` or when a new token secret is generated by
>> + /// `generate_and_set_secret`. Used to avoid repeated
>> + /// password-hash computation on subsequent authentications.
>> + secrets: HashMap<Authid, String>,
>> +}
>> +
>> +fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
>> + TOKEN_SECRET_CACHE.get_or_init(|| {
>> + RwLock::new(ApiTokenSecretCache {
>> + secrets: HashMap::new(),
>> + })
>> + })
>> +}
>> +
>> +fn cache_insert_secret(tokenid: Authid, secret: String) {
>> + let mut cache = token_secret_cache().write().unwrap();
>> + cache.secrets.insert(tokenid, secret);
>> +}
>> +
>> +fn cache_remove_secret(tokenid: &Authid) {
>> + let mut cache = token_secret_cache().write().unwrap();
>> + cache.secrets.remove(tokenid);
>> +}
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets
2025-12-10 15:35 6% ` Samuel Rufinatscha
@ 2025-12-15 15:05 12% ` Samuel Rufinatscha
2025-12-15 19:00 12% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-12-15 15:05 UTC (permalink / raw)
To: Fabian Grünbichler, pbs-devel
On 12/10/25 4:35 PM, Samuel Rufinatscha wrote:
> On 12/10/25 12:47 PM, Fabian Grünbichler wrote:
>> Quoting Samuel Rufinatscha (2025-12-05 14:25:54)
>>> Currently, every token-based API request reads the token.shadow file and
>>> runs the expensive password hash verification for the given token
>>> secret. This shows up as a hotspot in /status profiling (see
>>> bug #6049 [1]).
>>>
>>> This patch introduces an in-memory cache of successfully verified token
>>> secrets. Subsequent requests for the same token+secret combination only
>>> perform a comparison using openssl::memcmp::eq and avoid re-running the
>>> password hash. The cache is updated when a token secret is set and
>>> cleared when a token is deleted. Note, this does NOT include manual
>>> config changes, which will be covered in a subsequent patch.
>>>
>>> This patch partly fixes bug #6049 [1].
>>>
>>> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>>>
>>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>>> ---
>>> pbs-config/src/token_shadow.rs | 58 +++++++++++++++++++++++++++++++++-
>>> 1 file changed, 57 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/
>>> token_shadow.rs
>>> index 640fabbf..47aa2fc2 100644
>>> --- a/pbs-config/src/token_shadow.rs
>>> +++ b/pbs-config/src/token_shadow.rs
>>> @@ -1,6 +1,8 @@
>>> use std::collections::HashMap;
>>> +use std::sync::RwLock;
>>> use anyhow::{bail, format_err, Error};
>>> +use once_cell::sync::OnceCell;
>>> use serde::{Deserialize, Serialize};
>>> use serde_json::{from_value, Value};
>>> @@ -13,6 +15,13 @@ use crate::{open_backup_lockfile, BackupLockGuard};
>>> const LOCK_FILE: &str = pbs_buildcfg::configdir!("/
>>> token.shadow.lock");
>>> const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
>>> +/// Global in-memory cache for successfully verified API token secrets.
>>> +/// The cache stores plain text secrets for token Authids that have
>>> already been
>>> +/// verified against the hashed values in `token.shadow`. This
>>> allows for cheap
>>> +/// subsequent authentications for the same token+secret
>>> combination, avoiding
>>> +/// recomputing the password hash on every request.
>>> +static TOKEN_SECRET_CACHE: OnceCell<RwLock<ApiTokenSecretCache>> =
>>> OnceCell::new();
>>> +
>>> #[derive(Serialize, Deserialize)]
>>> #[serde(rename_all = "kebab-case")]
>>> /// ApiToken id / secret pair
>>> @@ -54,9 +63,25 @@ pub fn verify_secret(tokenid: &Authid, secret:
>>> &str) -> Result<(), Error> {
>>> bail!("not an API token ID");
>>> }
>>> + // Fast path
>>> + if let Some(cached) =
>>> token_secret_cache().read().unwrap().secrets.get(tokenid) {
>>
>> did you benchmark this with a lot of parallel token requests? a plain
>> RwLock
>> gives no guarantees at all w.r.t. ordering or fairness, so a lot of
>> token-based
>> requests could effectively prevent token removal AFAICT (or vice-versa,
>> spamming token creation could lock out all tokens?)
>>
>> since we don't actually require the cache here to proceed, we could
>> also make this a try_read
>> or a read with timeout, and fallback to the slow path if there is too
>> much
>> contention? alternatively, comparing with parking_lot would also be
>> interesting, since that implementation does have fairness guarantees.
>>
>> note that token-based requests are basically doable by anyone being
>> able to
>> reach PBS, whereas token creation/deletion is available to every
>> authenticaed
>> user.
>>
>
> Thanks for the review Fabian and the valuable comments!
>
> I did not benchmark the RwLock itself under load. Your point about
> contention/fairness for RwLock makes perfect sense, and we should
> consider this. So for v2, I will integrate try_read() /
> try_write() as mentioned to avoid possible contention / DoS issues.
>
> I’ll also consider parking_lot::RwLock, thanks for the hint!
>
I benchmarked the "writer under heavy parallel readers" scenario by
running a 64-parallel token-auth flood against
/admin/datastore/ds0001/status?verbose=0 (≈ 44-48k successful
requests total) while executing 50 token create + 50 token delete
operations.
With the suggested best-effort approach (cache lookups/inserts via
try_read/try_write) I saw the following e2e API latencies:
delete: p95 ~39ms, max ~44ms
create: p95 ~50ms, max ~56ms
I also compared against parking_lot::RwLock under the same setup,
results were in the same range (delete p95 ~39–43ms, max ~43–64ms)
so I didn’t see a clear benefit there for this workload.
For v2 I will keep std::sync::RwLock with read/insert best-effort, while
delete/removal blocking.
>>> + // Compare cached secret with provided one using constant
>>> time comparison
>>> + if openssl::memcmp::eq(cached.as_bytes(), secret.as_bytes()) {
>>> + // Already verified before
>>> + return Ok(());
>>> + }
>>> + // Fall through to slow path if secret doesn't match cached one
>>> + }
>>
>> this could also be a helper, like the rest. then it would consume (a
>> reference
>> to) the user-provided secret value, instead of giving access to all
>> cached
>> ones. doesn't make a real difference now other than consistence, but
>> the cache
>> is (more) cleanly encapsulated then.
>>
>>> +
>>> + // Slow path: read file + verify hash
>>> let data = read_file()?;
>>> match data.get(tokenid) {
>>> - Some(hashed_secret) =>
>>> proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
>>> + Some(hashed_secret) => {
>>> + proxmox_sys::crypt::verify_crypt_pw(secret,
>>> hashed_secret)?;
>>> + // Cache the plain secret for future requests
>>> + cache_insert_secret(tokenid.clone(), secret.to_owned());
>>
>> same applies here - storing the value in the cache is optional (and
>> good if it
>> works), but we don't want to stall forever waiting for the cache
>> insertion to
>> go through..
>>
>>> + Ok(())
>>> + }
>>> None => bail!("invalid API token"),
>>> }
>>> }
>>> @@ -82,6 +107,8 @@ fn set_secret(tokenid: &Authid, secret: &str) ->
>>> Result<(), Error> {
>>> data.insert(tokenid.clone(), hashed_secret);
>>> write_file(data)?;
>>> + cache_insert_secret(tokenid.clone(), secret.to_owned());
>>
>> this
>>
>>> +
>>> Ok(())
>>> }
>>> @@ -97,5 +124,34 @@ pub fn delete_secret(tokenid: &Authid) ->
>>> Result<(), Error> {
>>> data.remove(tokenid);
>>> write_file(data)?;
>>> + cache_remove_secret(tokenid);
>>
>> and this need to block of course and can't be skipped, because
>> otherwise the
>> read above might operate on wrong data..
>>
>>> +
>>> Ok(())
>>> }
>>> +
>>> +struct ApiTokenSecretCache {
>>> + /// Keys are token Authids, values are the corresponding plain
>>> text secrets.
>>> + /// Entries are added after a successful on-disk verification in
>>> + /// `verify_secret` or when a new token secret is generated by
>>> + /// `generate_and_set_secret`. Used to avoid repeated
>>> + /// password-hash computation on subsequent authentications.
>>> + secrets: HashMap<Authid, String>,
>>> +}
>>> +
>>> +fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
>>> + TOKEN_SECRET_CACHE.get_or_init(|| {
>>> + RwLock::new(ApiTokenSecretCache {
>>> + secrets: HashMap::new(),
>>> + })
>>> + })
>>> +}
>>> +
>>> +fn cache_insert_secret(tokenid: Authid, secret: String) {
>>> + let mut cache = token_secret_cache().write().unwrap();
>>> + cache.secrets.insert(tokenid, secret);
>>> +}
>>> +
>>> +fn cache_remove_secret(tokenid: &Authid) {
>>> + let mut cache = token_secret_cache().write().unwrap();
>>> + cache.secrets.remove(tokenid);
>>> +}
>>> --
>>> 2.47.3
>>>
>>>
>>>
>>> _______________________________________________
>>> pbs-devel mailing list
>>> pbs-devel@lists.proxmox.com
>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>>
>>>
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* Re: [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets
2025-12-15 15:05 12% ` Samuel Rufinatscha
@ 2025-12-15 19:00 12% ` Samuel Rufinatscha
2025-12-16 8:16 5% ` Fabian Grünbichler
0 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2025-12-15 19:00 UTC (permalink / raw)
To: Fabian Grünbichler, pbs-devel
On 12/15/25 4:06 PM, Samuel Rufinatscha wrote:
> On 12/10/25 4:35 PM, Samuel Rufinatscha wrote:
>> On 12/10/25 12:47 PM, Fabian Grünbichler wrote:
>>> Quoting Samuel Rufinatscha (2025-12-05 14:25:54)
>>>> Currently, every token-based API request reads the token.shadow file
>>>> and
>>>> runs the expensive password hash verification for the given token
>>>> secret. This shows up as a hotspot in /status profiling (see
>>>> bug #6049 [1]).
>>>>
>>>> This patch introduces an in-memory cache of successfully verified token
>>>> secrets. Subsequent requests for the same token+secret combination only
>>>> perform a comparison using openssl::memcmp::eq and avoid re-running the
>>>> password hash. The cache is updated when a token secret is set and
>>>> cleared when a token is deleted. Note, this does NOT include manual
>>>> config changes, which will be covered in a subsequent patch.
>>>>
>>>> This patch partly fixes bug #6049 [1].
>>>>
>>>> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>>>>
>>>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>>>> ---
>>>> pbs-config/src/token_shadow.rs | 58 ++++++++++++++++++++++++++++++
>>>> +++-
>>>> 1 file changed, 57 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/
>>>> token_shadow.rs
>>>> index 640fabbf..47aa2fc2 100644
>>>> --- a/pbs-config/src/token_shadow.rs
>>>> +++ b/pbs-config/src/token_shadow.rs
>>>> @@ -1,6 +1,8 @@
>>>> use std::collections::HashMap;
>>>> +use std::sync::RwLock;
>>>> use anyhow::{bail, format_err, Error};
>>>> +use once_cell::sync::OnceCell;
>>>> use serde::{Deserialize, Serialize};
>>>> use serde_json::{from_value, Value};
>>>> @@ -13,6 +15,13 @@ use crate::{open_backup_lockfile, BackupLockGuard};
>>>> const LOCK_FILE: &str = pbs_buildcfg::configdir!("/
>>>> token.shadow.lock");
>>>> const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
>>>> +/// Global in-memory cache for successfully verified API token
>>>> secrets.
>>>> +/// The cache stores plain text secrets for token Authids that have
>>>> already been
>>>> +/// verified against the hashed values in `token.shadow`. This
>>>> allows for cheap
>>>> +/// subsequent authentications for the same token+secret
>>>> combination, avoiding
>>>> +/// recomputing the password hash on every request.
>>>> +static TOKEN_SECRET_CACHE: OnceCell<RwLock<ApiTokenSecretCache>> =
>>>> OnceCell::new();
>>>> +
>>>> #[derive(Serialize, Deserialize)]
>>>> #[serde(rename_all = "kebab-case")]
>>>> /// ApiToken id / secret pair
>>>> @@ -54,9 +63,25 @@ pub fn verify_secret(tokenid: &Authid, secret:
>>>> &str) -> Result<(), Error> {
>>>> bail!("not an API token ID");
>>>> }
>>>> + // Fast path
>>>> + if let Some(cached) =
>>>> token_secret_cache().read().unwrap().secrets.get(tokenid) {
>>>
>>> did you benchmark this with a lot of parallel token requests? a plain
>>> RwLock
>>> gives no guarantees at all w.r.t. ordering or fairness, so a lot of
>>> token-based
>>> requests could effectively prevent token removal AFAICT (or vice-versa,
>>> spamming token creation could lock out all tokens?)
>>>
>>> since we don't actually require the cache here to proceed, we could
>>> also make this a try_read
>>> or a read with timeout, and fallback to the slow path if there is too
>>> much
>>> contention? alternatively, comparing with parking_lot would also be
>>> interesting, since that implementation does have fairness guarantees.
>>>
>>> note that token-based requests are basically doable by anyone being
>>> able to
>>> reach PBS, whereas token creation/deletion is available to every
>>> authenticaed
>>> user.
>>>
>>
>> Thanks for the review Fabian and the valuable comments!
>>
>> I did not benchmark the RwLock itself under load. Your point about
>> contention/fairness for RwLock makes perfect sense, and we should
>> consider this. So for v2, I will integrate try_read() /
>> try_write() as mentioned to avoid possible contention / DoS issues.
>>
>> I’ll also consider parking_lot::RwLock, thanks for the hint!
>>
>
>
> I benchmarked the "writer under heavy parallel readers" scenario by
> running a 64-parallel token-auth flood against
> /admin/datastore/ds0001/status?verbose=0 (≈ 44-48k successful
> requests total) while executing 50 token create + 50 token delete
> operations.
>
> With the suggested best-effort approach (cache lookups/inserts via
> try_read/try_write) I saw the following e2e API latencies:
>
> delete: p95 ~39ms, max ~44ms
> create: p95 ~50ms, max ~56ms
>
> I also compared against parking_lot::RwLock under the same setup,
> results were in the same range (delete p95 ~39–43ms, max ~43–64ms)
> so I didn’t see a clear benefit there for this workload.
>
> For v2 I will keep std::sync::RwLock with read/insert best-effort, while
> delete/removal blocking.
>
>
Fabian,
one clarification/follow-up: the comparison against parking_lot::RwLock
was focused on end-to-end latency, and under the benchmarked
workload we didn’t observe starvation effects. Still, std::sync::RwLock
does not provide ordering or fairness guarantees, so under sustained
token-auth read load cache invalidation could theoretically be delayed.
Given that, I think switching to parking_lot::RwLock for v2 to get clear
fairness semantics, while keeping the try_read/try_insert approach, is
the better solution here.
>>>> + // Compare cached secret with provided one using constant
>>>> time comparison
>>>> + if openssl::memcmp::eq(cached.as_bytes(), secret.as_bytes()) {
>>>> + // Already verified before
>>>> + return Ok(());
>>>> + }
>>>> + // Fall through to slow path if secret doesn't match cached
>>>> one
>>>> + }
>>>
>>> this could also be a helper, like the rest. then it would consume (a
>>> reference
>>> to) the user-provided secret value, instead of giving access to all
>>> cached
>>> ones. doesn't make a real difference now other than consistence, but
>>> the cache
>>> is (more) cleanly encapsulated then.
>>>
>>>> +
>>>> + // Slow path: read file + verify hash
>>>> let data = read_file()?;
>>>> match data.get(tokenid) {
>>>> - Some(hashed_secret) =>
>>>> proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
>>>> + Some(hashed_secret) => {
>>>> + proxmox_sys::crypt::verify_crypt_pw(secret,
>>>> hashed_secret)?;
>>>> + // Cache the plain secret for future requests
>>>> + cache_insert_secret(tokenid.clone(), secret.to_owned());
>>>
>>> same applies here - storing the value in the cache is optional (and
>>> good if it
>>> works), but we don't want to stall forever waiting for the cache
>>> insertion to
>>> go through..
>>>
>>>> + Ok(())
>>>> + }
>>>> None => bail!("invalid API token"),
>>>> }
>>>> }
>>>> @@ -82,6 +107,8 @@ fn set_secret(tokenid: &Authid, secret: &str) ->
>>>> Result<(), Error> {
>>>> data.insert(tokenid.clone(), hashed_secret);
>>>> write_file(data)?;
>>>> + cache_insert_secret(tokenid.clone(), secret.to_owned());
>>>
>>> this
>>>
>>>> +
>>>> Ok(())
>>>> }
>>>> @@ -97,5 +124,34 @@ pub fn delete_secret(tokenid: &Authid) ->
>>>> Result<(), Error> {
>>>> data.remove(tokenid);
>>>> write_file(data)?;
>>>> + cache_remove_secret(tokenid);
>>>
>>> and this need to block of course and can't be skipped, because
>>> otherwise the
>>> read above might operate on wrong data..
>>>
>>>> +
>>>> Ok(())
>>>> }
>>>> +
>>>> +struct ApiTokenSecretCache {
>>>> + /// Keys are token Authids, values are the corresponding plain
>>>> text secrets.
>>>> + /// Entries are added after a successful on-disk verification in
>>>> + /// `verify_secret` or when a new token secret is generated by
>>>> + /// `generate_and_set_secret`. Used to avoid repeated
>>>> + /// password-hash computation on subsequent authentications.
>>>> + secrets: HashMap<Authid, String>,
>>>> +}
>>>> +
>>>> +fn token_secret_cache() -> &'static RwLock<ApiTokenSecretCache> {
>>>> + TOKEN_SECRET_CACHE.get_or_init(|| {
>>>> + RwLock::new(ApiTokenSecretCache {
>>>> + secrets: HashMap::new(),
>>>> + })
>>>> + })
>>>> +}
>>>> +
>>>> +fn cache_insert_secret(tokenid: Authid, secret: String) {
>>>> + let mut cache = token_secret_cache().write().unwrap();
>>>> + cache.secrets.insert(tokenid, secret);
>>>> +}
>>>> +
>>>> +fn cache_remove_secret(tokenid: &Authid) {
>>>> + let mut cache = token_secret_cache().write().unwrap();
>>>> + cache.secrets.remove(tokenid);
>>>> +}
>>>> --
>>>> 2.47.3
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> pbs-devel mailing list
>>>> pbs-devel@lists.proxmox.com
>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>>>
>>>>
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* Re: [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets
2025-12-15 19:00 12% ` Samuel Rufinatscha
@ 2025-12-16 8:16 5% ` Fabian Grünbichler
0 siblings, 0 replies; 200+ results
From: Fabian Grünbichler @ 2025-12-16 8:16 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On December 15, 2025 8:00 pm, Samuel Rufinatscha wrote:
> On 12/15/25 4:06 PM, Samuel Rufinatscha wrote:
>> On 12/10/25 4:35 PM, Samuel Rufinatscha wrote:
>>> On 12/10/25 12:47 PM, Fabian Grünbichler wrote:
>>>> Quoting Samuel Rufinatscha (2025-12-05 14:25:54)
>>>>> Currently, every token-based API request reads the token.shadow file
>>>>> and
>>>>> runs the expensive password hash verification for the given token
>>>>> secret. This shows up as a hotspot in /status profiling (see
>>>>> bug #6049 [1]).
>>>>>
>>>>> This patch introduces an in-memory cache of successfully verified token
>>>>> secrets. Subsequent requests for the same token+secret combination only
>>>>> perform a comparison using openssl::memcmp::eq and avoid re-running the
>>>>> password hash. The cache is updated when a token secret is set and
>>>>> cleared when a token is deleted. Note, this does NOT include manual
>>>>> config changes, which will be covered in a subsequent patch.
>>>>>
>>>>> This patch partly fixes bug #6049 [1].
>>>>>
>>>>> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>>>>>
>>>>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>>>>> ---
>>>>> pbs-config/src/token_shadow.rs | 58 ++++++++++++++++++++++++++++++
>>>>> +++-
>>>>> 1 file changed, 57 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/
>>>>> token_shadow.rs
>>>>> index 640fabbf..47aa2fc2 100644
>>>>> --- a/pbs-config/src/token_shadow.rs
>>>>> +++ b/pbs-config/src/token_shadow.rs
>>>>> @@ -1,6 +1,8 @@
>>>>> use std::collections::HashMap;
>>>>> +use std::sync::RwLock;
>>>>> use anyhow::{bail, format_err, Error};
>>>>> +use once_cell::sync::OnceCell;
>>>>> use serde::{Deserialize, Serialize};
>>>>> use serde_json::{from_value, Value};
>>>>> @@ -13,6 +15,13 @@ use crate::{open_backup_lockfile, BackupLockGuard};
>>>>> const LOCK_FILE: &str = pbs_buildcfg::configdir!("/
>>>>> token.shadow.lock");
>>>>> const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
>>>>> +/// Global in-memory cache for successfully verified API token
>>>>> secrets.
>>>>> +/// The cache stores plain text secrets for token Authids that have
>>>>> already been
>>>>> +/// verified against the hashed values in `token.shadow`. This
>>>>> allows for cheap
>>>>> +/// subsequent authentications for the same token+secret
>>>>> combination, avoiding
>>>>> +/// recomputing the password hash on every request.
>>>>> +static TOKEN_SECRET_CACHE: OnceCell<RwLock<ApiTokenSecretCache>> =
>>>>> OnceCell::new();
>>>>> +
>>>>> #[derive(Serialize, Deserialize)]
>>>>> #[serde(rename_all = "kebab-case")]
>>>>> /// ApiToken id / secret pair
>>>>> @@ -54,9 +63,25 @@ pub fn verify_secret(tokenid: &Authid, secret:
>>>>> &str) -> Result<(), Error> {
>>>>> bail!("not an API token ID");
>>>>> }
>>>>> + // Fast path
>>>>> + if let Some(cached) =
>>>>> token_secret_cache().read().unwrap().secrets.get(tokenid) {
>>>>
>>>> did you benchmark this with a lot of parallel token requests? a plain
>>>> RwLock
>>>> gives no guarantees at all w.r.t. ordering or fairness, so a lot of
>>>> token-based
>>>> requests could effectively prevent token removal AFAICT (or vice-versa,
>>>> spamming token creation could lock out all tokens?)
>>>>
>>>> since we don't actually require the cache here to proceed, we could
>>>> also make this a try_read
>>>> or a read with timeout, and fallback to the slow path if there is too
>>>> much
>>>> contention? alternatively, comparing with parking_lot would also be
>>>> interesting, since that implementation does have fairness guarantees.
>>>>
>>>> note that token-based requests are basically doable by anyone being
>>>> able to
>>>> reach PBS, whereas token creation/deletion is available to every
>>>> authenticaed
>>>> user.
>>>>
>>>
>>> Thanks for the review Fabian and the valuable comments!
>>>
>>> I did not benchmark the RwLock itself under load. Your point about
>>> contention/fairness for RwLock makes perfect sense, and we should
>>> consider this. So for v2, I will integrate try_read() /
>>> try_write() as mentioned to avoid possible contention / DoS issues.
>>>
>>> I’ll also consider parking_lot::RwLock, thanks for the hint!
>>>
>>
>>
>> I benchmarked the "writer under heavy parallel readers" scenario by
>> running a 64-parallel token-auth flood against
>> /admin/datastore/ds0001/status?verbose=0 (≈ 44-48k successful
>> requests total) while executing 50 token create + 50 token delete
>> operations.
>>
>> With the suggested best-effort approach (cache lookups/inserts via
>> try_read/try_write) I saw the following e2e API latencies:
>>
>> delete: p95 ~39ms, max ~44ms
>> create: p95 ~50ms, max ~56ms
>>
>> I also compared against parking_lot::RwLock under the same setup,
>> results were in the same range (delete p95 ~39–43ms, max ~43–64ms)
>> so I didn’t see a clear benefit there for this workload.
>>
>> For v2 I will keep std::sync::RwLock with read/insert best-effort, while
>> delete/removal blocking.
>>
>>
>
> Fabian,
>
> one clarification/follow-up: the comparison against parking_lot::RwLock
> was focused on end-to-end latency, and under the benchmarked
> workload we didn’t observe starvation effects. Still, std::sync::RwLock
> does not provide ordering or fairness guarantees, so under sustained
> token-auth read load cache invalidation could theoretically be delayed.
>
> Given that, I think switching to parking_lot::RwLock for v2 to get clear
> fairness semantics, while keeping the try_read/try_insert approach, is
> the better solution here.
I think going with parking_lot is okay here (it's already a dependency
of tokio anyway..). If we go with the std one, we should keep it in mind
in case we ever see signs of this being a problem.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets
2025-12-09 13:29 6% ` Samuel Rufinatscha
@ 2025-12-17 11:16 5% ` Christian Ebner
2025-12-17 11:25 0% ` Shannon Sterz
0 siblings, 1 reply; 200+ results
From: Christian Ebner @ 2025-12-17 11:16 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Samuel Rufinatscha,
Shannon Sterz
On 12/9/25 2:29 PM, Samuel Rufinatscha wrote:
> On 12/5/25 3:03 PM, Shannon Sterz wrote:
>> On Fri Dec 5, 2025 at 2:25 PM CET, Samuel Rufinatscha wrote:
>>> Currently, every token-based API request reads the token.shadow file and
>>> runs the expensive password hash verification for the given token
>>> secret. This shows up as a hotspot in /status profiling (see
>>> bug #6049 [1]).
>>>
>>> This patch introduces an in-memory cache of successfully verified token
>>> secrets. Subsequent requests for the same token+secret combination only
>>> perform a comparison using openssl::memcmp::eq and avoid re-running the
>>> password hash. The cache is updated when a token secret is set and
>>> cleared when a token is deleted. Note, this does NOT include manual
>>> config changes, which will be covered in a subsequent patch.
>>>
>>> This patch partly fixes bug #6049 [1].
>>>
>>> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>>>
>>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>>> ---
>>> pbs-config/src/token_shadow.rs | 58 +++++++++++++++++++++++++++++++++-
>>> 1 file changed, 57 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/
>>> token_shadow.rs
>>> index 640fabbf..47aa2fc2 100644
>>> --- a/pbs-config/src/token_shadow.rs
>>> +++ b/pbs-config/src/token_shadow.rs
>>> @@ -1,6 +1,8 @@
>>> use std::collections::HashMap;
>>> +use std::sync::RwLock;
>>>
>>> use anyhow::{bail, format_err, Error};
>>> +use once_cell::sync::OnceCell;
>>> use serde::{Deserialize, Serialize};
>>> use serde_json::{from_value, Value};
>>>
>>> @@ -13,6 +15,13 @@ use crate::{open_backup_lockfile, BackupLockGuard};
>>> const LOCK_FILE: &str = pbs_buildcfg::configdir!("/
>>> token.shadow.lock");
>>> const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
>>>
>>> +/// Global in-memory cache for successfully verified API token secrets.
>>> +/// The cache stores plain text secrets for token Authids that have
>>> already been
>>> +/// verified against the hashed values in `token.shadow`. This
>>> allows for cheap
>>> +/// subsequent authentications for the same token+secret
>>> combination, avoiding
>>> +/// recomputing the password hash on every request.
>>> +static TOKEN_SECRET_CACHE: OnceCell<RwLock<ApiTokenSecretCache>> =
>>> OnceCell::new();
>>
>> any reason you are using a once cell with a cutom get_or_init function
>> instead of a simple `LazyCell` [1] here? seems to me that this would be
>> the more appropriate type here? similar question for the
>> proxmox-access-control portion of this series.
>>
>> [1]: https://doc.rust-lang.org/std/cell/struct.LazyCell.html
>>
>
> Good point, we should / can directly initialize it! Will change
> to LazyCell. Thanks!
LazyCell is however not thread safe, so could cause issues with
concurrent inits from different threads. IMO std::sync::LazyLock [0] is
a better fit here and follows along the line of what we do for other
caches in PBS, e.g. in pbs-config::user.
[0] https://doc.rust-lang.org/std/sync/struct.LazyLock.html
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets
2025-12-17 11:16 5% ` Christian Ebner
@ 2025-12-17 11:25 0% ` Shannon Sterz
0 siblings, 0 replies; 200+ results
From: Shannon Sterz @ 2025-12-17 11:25 UTC (permalink / raw)
To: Christian Ebner; +Cc: Proxmox Backup Server development discussion
On Wed Dec 17, 2025 at 12:16 PM CET, Christian Ebner wrote:
> On 12/9/25 2:29 PM, Samuel Rufinatscha wrote:
>> On 12/5/25 3:03 PM, Shannon Sterz wrote:
>>> On Fri Dec 5, 2025 at 2:25 PM CET, Samuel Rufinatscha wrote:
>>>> Currently, every token-based API request reads the token.shadow file and
>>>> runs the expensive password hash verification for the given token
>>>> secret. This shows up as a hotspot in /status profiling (see
>>>> bug #6049 [1]).
>>>>
>>>> This patch introduces an in-memory cache of successfully verified token
>>>> secrets. Subsequent requests for the same token+secret combination only
>>>> perform a comparison using openssl::memcmp::eq and avoid re-running the
>>>> password hash. The cache is updated when a token secret is set and
>>>> cleared when a token is deleted. Note, this does NOT include manual
>>>> config changes, which will be covered in a subsequent patch.
>>>>
>>>> This patch partly fixes bug #6049 [1].
>>>>
>>>> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>>>>
>>>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>>>> ---
>>>> pbs-config/src/token_shadow.rs | 58 +++++++++++++++++++++++++++++++++-
>>>> 1 file changed, 57 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/
>>>> token_shadow.rs
>>>> index 640fabbf..47aa2fc2 100644
>>>> --- a/pbs-config/src/token_shadow.rs
>>>> +++ b/pbs-config/src/token_shadow.rs
>>>> @@ -1,6 +1,8 @@
>>>> use std::collections::HashMap;
>>>> +use std::sync::RwLock;
>>>>
>>>> use anyhow::{bail, format_err, Error};
>>>> +use once_cell::sync::OnceCell;
>>>> use serde::{Deserialize, Serialize};
>>>> use serde_json::{from_value, Value};
>>>>
>>>> @@ -13,6 +15,13 @@ use crate::{open_backup_lockfile, BackupLockGuard};
>>>> const LOCK_FILE: &str = pbs_buildcfg::configdir!("/
>>>> token.shadow.lock");
>>>> const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
>>>>
>>>> +/// Global in-memory cache for successfully verified API token secrets.
>>>> +/// The cache stores plain text secrets for token Authids that have
>>>> already been
>>>> +/// verified against the hashed values in `token.shadow`. This
>>>> allows for cheap
>>>> +/// subsequent authentications for the same token+secret
>>>> combination, avoiding
>>>> +/// recomputing the password hash on every request.
>>>> +static TOKEN_SECRET_CACHE: OnceCell<RwLock<ApiTokenSecretCache>> =
>>>> OnceCell::new();
>>>
>>> any reason you are using a once cell with a cutom get_or_init function
>>> instead of a simple `LazyCell` [1] here? seems to me that this would be
>>> the more appropriate type here? similar question for the
>>> proxmox-access-control portion of this series.
>>>
>>> [1]: https://doc.rust-lang.org/std/cell/struct.LazyCell.html
>>>
>>
>> Good point, we should / can directly initialize it! Will change
>> to LazyCell. Thanks!
>
> LazyCell is however not thread safe, so could cause issues with
> concurrent inits from different threads. IMO std::sync::LazyLock [0] is
> a better fit here and follows along the line of what we do for other
> caches in PBS, e.g. in pbs-config::user.
>
> [0] https://doc.rust-lang.org/std/sync/struct.LazyLock.html
ah right, yes that makes a lot of sense, thanks for catching that.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 0%]
* [pbs-devel] [PATCH proxmox-datacenter-manager v2 1/1] docs: document API token-cache TTL effects
2025-12-17 16:25 14% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token " Samuel Rufinatscha
` (5 preceding siblings ...)
2025-12-17 16:25 15% ` [pbs-devel] [PATCH proxmox v2 3/3] proxmox-access-control: add TTL window to token secret cache Samuel Rufinatscha
@ 2025-12-17 16:25 17% ` Samuel Rufinatscha
2025-12-18 11:03 12% ` [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token verification overhead Samuel Rufinatscha
2026-01-02 16:09 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-17 16:25 UTC (permalink / raw)
To: pbs-devel
Documents the effects of the added API token-cache in the
proxmox-access-control crate. This patch is part of the
series that fixes bug #7017 [1].
This patch partly fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
docs/access-control.rst | 3 +++
1 file changed, 3 insertions(+)
diff --git a/docs/access-control.rst b/docs/access-control.rst
index adf26cd..f4f26f2 100644
--- a/docs/access-control.rst
+++ b/docs/access-control.rst
@@ -47,6 +47,9 @@ place of the user ID (``user@realm``) and the user password, respectively.
The API token is passed from the client to the server by setting the ``Authorization`` HTTP header
with method ``PDMAPIToken`` to the value ``TOKENID:TOKENSECRET``.
+.. WARNING:: If you manually remove a generated API token from the token secrets file (token.shadow),
+ it can take up to one minute before the token is rejected. This is due to caching.
+
.. _access_control:
Access Control
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 17%]
* [pbs-devel] [PATCH proxmox-backup v2 2/3] pbs-config: invalidate token-secret cache on token.shadow changes
2025-12-17 16:25 14% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token " Samuel Rufinatscha
2025-12-17 16:25 13% ` [pbs-devel] [PATCH proxmox-backup v2 1/3] pbs-config: cache verified API token secrets Samuel Rufinatscha
@ 2025-12-17 16:25 12% ` Samuel Rufinatscha
2025-12-17 16:25 14% ` [pbs-devel] [PATCH proxmox-backup v2 3/3] pbs-config: add TTL window to token secret cache Samuel Rufinatscha
` (6 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-17 16:25 UTC (permalink / raw)
To: pbs-devel
Previously the in-memory token-secret cache was only updated via
set_secret() and delete_secret(), so manual edits to token.shadow were
not reflected.
This patch adds file change detection to the cache. It tracks the mtime
and length of token.shadow and clears the in-memory token secret cache
whenever these values change.
Note, this patch fetches file stats on every request. An TTL-based
optimization will be covered in a subsequent patch of the series.
This patch partly fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v1 to v2:
- Add file metadata tracking (file_mtime, file_len) and
FILE_GENERATION.
- Store file_gen in CachedSecret and verify it against the current
FILE_GENERATION to ensure cached entries belong to the current file
state.
- Add shadow_mtime_len() helper and convert refresh to best-effort
(try_write, returns bool).
- Pass a pre-write metadata snapshot into apply_api_mutation and
clear/bump generation if the cache metadata indicates missed external
edits.
pbs-config/src/token_shadow.rs | 128 +++++++++++++++++++++++++++++----
1 file changed, 116 insertions(+), 12 deletions(-)
diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
index ce845e8d..71553aae 100644
--- a/pbs-config/src/token_shadow.rs
+++ b/pbs-config/src/token_shadow.rs
@@ -1,6 +1,9 @@
use std::collections::HashMap;
+use std::fs;
+use std::io::ErrorKind;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::LazyLock;
+use std::time::SystemTime;
use anyhow::{bail, format_err, Error};
use parking_lot::RwLock;
@@ -24,10 +27,14 @@ const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new(|| {
RwLock::new(ApiTokenSecretCache {
secrets: HashMap::new(),
+ file_mtime: None,
+ file_len: None,
})
});
/// API mutation generation (set/delete)
static API_MUTATION_GENERATION: AtomicU64 = AtomicU64::new(0);
+/// External/manual edits generation for the token.shadow file
+static FILE_GENERATION: AtomicU64 = AtomicU64::new(0);
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
@@ -64,6 +71,29 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
proxmox_sys::fs::replace_file(CONF_FILE, &json, options, true)
}
+/// Refreshes the in-memory cache if the on-disk token.shadow file changed.
+/// Returns true if the cache is valid to use, false if not.
+fn refresh_cache_if_file_changed() -> bool {
+ let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
+ return false; // cannot validate external changes -> don't trust cache
+ };
+
+ let Ok((new_mtime, new_len)) = shadow_mtime_len() else {
+ return false; // cannot validate external changes -> don't trust cache
+ };
+
+ if cache.file_mtime == new_mtime && cache.file_len == new_len {
+ return true;
+ }
+
+ cache.secrets.clear();
+ cache.file_mtime = new_mtime;
+ cache.file_len = new_len;
+ FILE_GENERATION.fetch_add(1, Ordering::AcqRel);
+
+ true
+}
+
/// Verifies that an entry for given tokenid / API token secret exists
pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
if !tokenid.is_token() {
@@ -71,12 +101,13 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
}
// Fast path
- if cache_try_secret_matches(tokenid, secret) {
+ if refresh_cache_if_file_changed() && cache_try_secret_matches(tokenid, secret) {
return Ok(());
}
// Slow path snapshot (before expensive work)
let api_gen_before = API_MUTATION_GENERATION.load(Ordering::Acquire);
+ let file_gen_before = FILE_GENERATION.load(Ordering::Acquire);
let data = read_file()?;
match data.get(tokenid) {
@@ -84,7 +115,12 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
// Try to cache only if nothing changed while we verified
- cache_try_insert_secret(tokenid.clone(), secret.to_owned(), api_gen_before);
+ cache_try_insert_secret(
+ tokenid.clone(),
+ secret.to_owned(),
+ api_gen_before,
+ file_gen_before,
+ );
Ok(())
}
@@ -108,12 +144,15 @@ fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
let _guard = lock_config()?;
+ // Capture state BEFORE we write to detect external edits.
+ let pre_meta = shadow_mtime_len().unwrap_or((None, None));
+
let mut data = read_file()?;
let hashed_secret = proxmox_sys::crypt::encrypt_pw(secret)?;
data.insert(tokenid.clone(), hashed_secret);
write_file(data)?;
- apply_api_mutation(tokenid, Some(secret));
+ apply_api_mutation(tokenid, Some(secret), pre_meta);
Ok(())
}
@@ -126,11 +165,14 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
let _guard = lock_config()?;
+ // Capture state BEFORE we write to detect external edits.
+ let pre_meta = shadow_mtime_len().unwrap_or((None, None));
+
let mut data = read_file()?;
data.remove(tokenid);
write_file(data)?;
- apply_api_mutation(tokenid, None);
+ apply_api_mutation(tokenid, None, pre_meta);
Ok(())
}
@@ -142,20 +184,40 @@ struct ApiTokenSecretCache {
/// `generate_and_set_secret`. Used to avoid repeated
/// password-hash computation on subsequent authentications.
secrets: HashMap<Authid, CachedSecret>,
+ // shadow file mtime to detect changes
+ file_mtime: Option<SystemTime>,
+ // shadow file length to detect changes
+ file_len: Option<u64>,
}
-/// Cached secret.
+/// Cached secret and the file generation it was cached at.
struct CachedSecret {
secret: String,
+ file_gen: u64,
}
-fn cache_try_insert_secret(tokenid: Authid, secret: String, api_gen_snapshot: u64) {
+fn cache_try_insert_secret(
+ tokenid: Authid,
+ secret: String,
+ api_gen_snapshot: u64,
+ file_gen_snapshot: u64,
+) {
let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
return;
};
- if API_MUTATION_GENERATION.load(Ordering::Acquire) == api_gen_snapshot {
- cache.secrets.insert(tokenid, CachedSecret { secret });
+ // Check generations to avoid zombie-inserts
+ let cur_file_gen = FILE_GENERATION.load(Ordering::Acquire);
+ let cur_api_gen = API_MUTATION_GENERATION.load(Ordering::Acquire);
+
+ if cur_file_gen == file_gen_snapshot && cur_api_gen == api_gen_snapshot {
+ cache.secrets.insert(
+ tokenid,
+ CachedSecret {
+ secret,
+ file_gen: cur_file_gen,
+ },
+ );
}
}
@@ -167,22 +229,44 @@ fn cache_try_secret_matches(tokenid: &Authid, secret: &str) -> bool {
return false;
};
- openssl::memcmp::eq(entry.secret.as_bytes(), secret.as_bytes())
+ let gen1 = FILE_GENERATION.load(Ordering::Acquire);
+ if entry.file_gen != gen1 {
+ return false;
+ }
+
+ let eq = openssl::memcmp::eq(entry.secret.as_bytes(), secret.as_bytes());
+
+ let gen2 = FILE_GENERATION.load(Ordering::Acquire);
+ eq && gen1 == gen2
}
-fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
- // Prevent in-flight verify_secret() from caching results across a mutation.
+fn apply_api_mutation(
+ tokenid: &Authid,
+ new_secret: Option<&str>,
+ pre_write_meta: (Option<SystemTime>, Option<u64>),
+) {
API_MUTATION_GENERATION.fetch_add(1, Ordering::AcqRel);
- // Mutations must be reflected immediately once set/delete returns.
let mut cache = TOKEN_SECRET_CACHE.write();
+ // If the cache meta doesn't match the file state before the on-disk write,
+ // external/manual edits happened -> drop everything and bump FILE_GENERATION.
+ let (pre_mtime, pre_len) = pre_write_meta;
+ if cache.file_mtime != pre_mtime || cache.file_len != pre_len {
+ cache.secrets.clear();
+ FILE_GENERATION.fetch_add(1, Ordering::AcqRel);
+ }
+
+ let file_gen = FILE_GENERATION.load(Ordering::Acquire);
+
+ // Apply the API mutation to the cache.
match new_secret {
Some(secret) => {
cache.secrets.insert(
tokenid.clone(),
CachedSecret {
secret: secret.to_owned(),
+ file_gen,
},
);
}
@@ -190,4 +274,24 @@ fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
cache.secrets.remove(tokenid);
}
}
+
+ // Keep cache metadata aligned if possible.
+ match shadow_mtime_len() {
+ Ok((mtime, len)) => {
+ cache.file_mtime = mtime;
+ cache.file_len = len;
+ }
+ Err(_) => {
+ cache.file_mtime = None;
+ cache.file_len = None;
+ }
+ }
+}
+
+fn shadow_mtime_len() -> Result<(Option<SystemTime>, Option<u64>), Error> {
+ match fs::metadata(CONF_FILE) {
+ Ok(meta) => Ok((meta.modified().ok(), Some(meta.len()))),
+ Err(e) if e.kind() == ErrorKind::NotFound => Ok((None, None)),
+ Err(e) => Err(e.into()),
+ }
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] [PATCH proxmox v2 3/3] proxmox-access-control: add TTL window to token secret cache
2025-12-17 16:25 14% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token " Samuel Rufinatscha
` (4 preceding siblings ...)
2025-12-17 16:25 12% ` [pbs-devel] [PATCH proxmox v2 2/3] proxmox-access-control: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
@ 2025-12-17 16:25 15% ` Samuel Rufinatscha
2025-12-17 16:25 17% ` [pbs-devel] [PATCH proxmox-datacenter-manager v2 1/1] docs: document API token-cache TTL effects Samuel Rufinatscha
` (2 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-17 16:25 UTC (permalink / raw)
To: pbs-devel
Verify_secret() currently calls refresh_cache_if_file_changed() on every
request, which performs a metadata() call on token.shadow each time.
Under load this adds unnecessary overhead, considering also the file
should rarely change.
This patch introduces a TTL boundary, controlled by
TOKEN_SECRET_CACHE_TTL_SECS. File metadata is only re-loaded once the
TTL has expired.
This patch partly fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v1 to v2:
- Add TOKEN_SECRET_CACHE_TTL_SECS and last_checked.
- Implement double-checked TTL: check with try_read first; only attempt
refresh with try_write if expired/unknown.
- Fix TTL bookkeeping: update last_checked on the “file unchanged” path
and after API mutations.
proxmox-access-control/src/token_shadow.rs | 42 +++++++++++++++++++++-
1 file changed, 41 insertions(+), 1 deletion(-)
diff --git a/proxmox-access-control/src/token_shadow.rs b/proxmox-access-control/src/token_shadow.rs
index efadce94..4ca56de9 100644
--- a/proxmox-access-control/src/token_shadow.rs
+++ b/proxmox-access-control/src/token_shadow.rs
@@ -11,6 +11,7 @@ use serde_json::{from_value, Value};
use proxmox_auth_api::types::Authid;
use proxmox_product_config::{open_api_lockfile, replace_config, ApiLockGuard};
+use proxmox_time::epoch_i64;
use crate::init::impl_feature::{token_shadow, token_shadow_lock};
@@ -24,12 +25,15 @@ static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new
secrets: HashMap::new(),
file_mtime: None,
file_len: None,
+ last_checked: None,
})
});
/// API mutation generation (set/delete)
static API_MUTATION_GENERATION: AtomicU64 = AtomicU64::new(0);
/// External/manual edits generation for the token.shadow file
static FILE_GENERATION: AtomicU64 = AtomicU64::new(0);
+/// Max age in seconds of the token secret cache before checking for file changes.
+const TOKEN_SECRET_CACHE_TTL_SECS: i64 = 60;
// Get exclusive lock
fn lock_config() -> Result<ApiLockGuard, Error> {
@@ -56,22 +60,54 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
/// Refreshes the in-memory cache if the on-disk token.shadow file changed.
/// Returns true if the cache is valid to use, false if not.
fn refresh_cache_if_file_changed() -> bool {
+ let now = epoch_i64();
+
+ // Check TTL (best-effort)
+ let Some(cache) = TOKEN_SECRET_CACHE.try_read() else {
+ return false; // cannot validate external changes -> don't trust cache
+ };
+
+ let ttl_ok = cache
+ .last_checked
+ .is_some_and(|last| now.saturating_sub(last) < TOKEN_SECRET_CACHE_TTL_SECS);
+
+ drop(cache);
+
+ if ttl_ok {
+ return true;
+ }
+
+ // TTL expired/unknown at this point -> do best-effort refresh.
let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
return false; // cannot validate external changes -> don't trust cache
};
+ // Check TTL after acquiring write lock.
+ if let Some(last) = cache.last_checked {
+ if now.saturating_sub(last) < TOKEN_SECRET_CACHE_TTL_SECS {
+ return true;
+ }
+ }
+
+ let had_prior_state = cache.last_checked.is_some();
+
let Ok((new_mtime, new_len)) = shadow_mtime_len() else {
return false; // cannot validate external changes -> don't trust cache
};
if cache.file_mtime == new_mtime && cache.file_len == new_len {
+ cache.last_checked = Some(now);
return true;
}
cache.secrets.clear();
cache.file_mtime = new_mtime;
cache.file_len = new_len;
- FILE_GENERATION.fetch_add(1, Ordering::AcqRel);
+ cache.last_checked = Some(now);
+
+ if had_prior_state {
+ FILE_GENERATION.fetch_add(1, Ordering::AcqRel);
+ }
true
}
@@ -170,6 +206,8 @@ struct ApiTokenSecretCache {
file_mtime: Option<SystemTime>,
// shadow file length to detect changes
file_len: Option<u64>,
+ // last time the file metadata was checked
+ last_checked: Option<i64>,
}
/// Cached secret and the file generation it was cached at.
@@ -262,10 +300,12 @@ fn apply_api_mutation(
Ok((mtime, len)) => {
cache.file_mtime = mtime;
cache.file_len = len;
+ cache.last_checked = Some(epoch_i64());
}
Err(_) => {
cache.file_mtime = None;
cache.file_len = None;
+ cache.last_checked = None; // to force refresh next time
}
}
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox v2 2/3] proxmox-access-control: invalidate token-secret cache on token.shadow changes
2025-12-17 16:25 14% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token " Samuel Rufinatscha
` (3 preceding siblings ...)
2025-12-17 16:25 13% ` [pbs-devel] [PATCH proxmox v2 1/3] proxmox-access-control: cache verified API token secrets Samuel Rufinatscha
@ 2025-12-17 16:25 12% ` Samuel Rufinatscha
2025-12-17 16:25 15% ` [pbs-devel] [PATCH proxmox v2 3/3] proxmox-access-control: add TTL window to token secret cache Samuel Rufinatscha
` (3 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-17 16:25 UTC (permalink / raw)
To: pbs-devel
Previously the in-memory token-secret cache was only updated via
set_secret() and delete_secret(), so manual edits to token.shadow were
not reflected.
This patch adds file change detection to the cache. It tracks the mtime
and length of token.shadow and clears the in-memory token secret cache
whenever these values change.
Note, this patch fetches file stats on every request. An TTL-based
optimization will be covered in a subsequent patch of the series.
This patch partly fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v1 to v2:
- Add file metadata tracking (file_mtime, file_len) and
FILE_GENERATION.
- Store file_gen in CachedSecret and verify it against the current
FILE_GENERATION to ensure cached entries belong to the current file
state.
- Add shadow_mtime_len() helper and convert refresh to best-effort
(try_write, returns bool).
- Pass a pre-write metadata snapshot into apply_api_mutation and
clear/bump generation if the cache metadata indicates missed external
edits.
proxmox-access-control/src/token_shadow.rs | 128 +++++++++++++++++++--
1 file changed, 116 insertions(+), 12 deletions(-)
diff --git a/proxmox-access-control/src/token_shadow.rs b/proxmox-access-control/src/token_shadow.rs
index c0285b62..efadce94 100644
--- a/proxmox-access-control/src/token_shadow.rs
+++ b/proxmox-access-control/src/token_shadow.rs
@@ -1,6 +1,9 @@
use std::collections::HashMap;
+use std::fs;
+use std::io::ErrorKind;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::LazyLock;
+use std::time::SystemTime;
use anyhow::{bail, format_err, Error};
use parking_lot::RwLock;
@@ -19,10 +22,14 @@ use crate::init::impl_feature::{token_shadow, token_shadow_lock};
static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new(|| {
RwLock::new(ApiTokenSecretCache {
secrets: HashMap::new(),
+ file_mtime: None,
+ file_len: None,
})
});
/// API mutation generation (set/delete)
static API_MUTATION_GENERATION: AtomicU64 = AtomicU64::new(0);
+/// External/manual edits generation for the token.shadow file
+static FILE_GENERATION: AtomicU64 = AtomicU64::new(0);
// Get exclusive lock
fn lock_config() -> Result<ApiLockGuard, Error> {
@@ -46,6 +53,29 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
replace_config(token_shadow(), &json)
}
+/// Refreshes the in-memory cache if the on-disk token.shadow file changed.
+/// Returns true if the cache is valid to use, false if not.
+fn refresh_cache_if_file_changed() -> bool {
+ let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
+ return false; // cannot validate external changes -> don't trust cache
+ };
+
+ let Ok((new_mtime, new_len)) = shadow_mtime_len() else {
+ return false; // cannot validate external changes -> don't trust cache
+ };
+
+ if cache.file_mtime == new_mtime && cache.file_len == new_len {
+ return true;
+ }
+
+ cache.secrets.clear();
+ cache.file_mtime = new_mtime;
+ cache.file_len = new_len;
+ FILE_GENERATION.fetch_add(1, Ordering::AcqRel);
+
+ true
+}
+
/// Verifies that an entry for given tokenid / API token secret exists
pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
if !tokenid.is_token() {
@@ -53,12 +83,13 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
}
// Fast path
- if cache_try_secret_matches(tokenid, secret) {
+ if refresh_cache_if_file_changed() && cache_try_secret_matches(tokenid, secret) {
return Ok(());
}
// Slow path snapshot (before expensive work)
let api_gen_before = API_MUTATION_GENERATION.load(Ordering::Acquire);
+ let file_gen_before = FILE_GENERATION.load(Ordering::Acquire);
let data = read_file()?;
match data.get(tokenid) {
@@ -66,7 +97,12 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
// Try to cache only if nothing changed while we verified
- cache_try_insert_secret(tokenid.clone(), secret.to_owned(), api_gen_before);
+ cache_try_insert_secret(
+ tokenid.clone(),
+ secret.to_owned(),
+ api_gen_before,
+ file_gen_before,
+ );
Ok(())
}
@@ -82,12 +118,15 @@ pub fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
let _guard = lock_config()?;
+ // Capture state BEFORE we write to detect external edits.
+ let pre_meta = shadow_mtime_len().unwrap_or((None, None));
+
let mut data = read_file()?;
let hashed_secret = proxmox_sys::crypt::encrypt_pw(secret)?;
data.insert(tokenid.clone(), hashed_secret);
write_file(data)?;
- apply_api_mutation(tokenid, Some(secret));
+ apply_api_mutation(tokenid, Some(secret), pre_meta);
Ok(())
}
@@ -100,11 +139,14 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
let _guard = lock_config()?;
+ // Capture state BEFORE we write to detect external edits.
+ let pre_meta = shadow_mtime_len().unwrap_or((None, None));
+
let mut data = read_file()?;
data.remove(tokenid);
write_file(data)?;
- apply_api_mutation(tokenid, None);
+ apply_api_mutation(tokenid, None, pre_meta);
Ok(())
}
@@ -124,20 +166,40 @@ struct ApiTokenSecretCache {
/// `generate_and_set_secret`. Used to avoid repeated
/// password-hash computation on subsequent authentications.
secrets: HashMap<Authid, CachedSecret>,
+ // shadow file mtime to detect changes
+ file_mtime: Option<SystemTime>,
+ // shadow file length to detect changes
+ file_len: Option<u64>,
}
-/// Cached secret.
+/// Cached secret and the file generation it was cached at.
struct CachedSecret {
secret: String,
+ file_gen: u64,
}
-fn cache_try_insert_secret(tokenid: Authid, secret: String, api_gen_snapshot: u64) {
+fn cache_try_insert_secret(
+ tokenid: Authid,
+ secret: String,
+ api_gen_snapshot: u64,
+ file_gen_snapshot: u64,
+) {
let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
return;
};
- if API_MUTATION_GENERATION.load(Ordering::Acquire) == api_gen_snapshot {
- cache.secrets.insert(tokenid, CachedSecret { secret });
+ // Check generations to avoid zombie-inserts
+ let cur_file_gen = FILE_GENERATION.load(Ordering::Acquire);
+ let cur_api_gen = API_MUTATION_GENERATION.load(Ordering::Acquire);
+
+ if cur_file_gen == file_gen_snapshot && cur_api_gen == api_gen_snapshot {
+ cache.secrets.insert(
+ tokenid,
+ CachedSecret {
+ secret,
+ file_gen: cur_file_gen,
+ },
+ );
}
}
@@ -149,22 +211,44 @@ fn cache_try_secret_matches(tokenid: &Authid, secret: &str) -> bool {
return false;
};
- openssl::memcmp::eq(entry.secret.as_bytes(), secret.as_bytes())
+ let gen1 = FILE_GENERATION.load(Ordering::Acquire);
+ if entry.file_gen != gen1 {
+ return false;
+ }
+
+ let eq = openssl::memcmp::eq(entry.secret.as_bytes(), secret.as_bytes());
+
+ let gen2 = FILE_GENERATION.load(Ordering::Acquire);
+ eq && gen1 == gen2
}
-fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
- // Prevent in-flight verify_secret() from caching results across a mutation.
+fn apply_api_mutation(
+ tokenid: &Authid,
+ new_secret: Option<&str>,
+ pre_write_meta: (Option<SystemTime>, Option<u64>),
+) {
API_MUTATION_GENERATION.fetch_add(1, Ordering::AcqRel);
- // Mutations must be reflected immediately once set/delete returns.
let mut cache = TOKEN_SECRET_CACHE.write();
+ // If the cache meta doesn't match the file state before the on-disk write,
+ // external/manual edits happened -> drop everything and bump FILE_GENERATION.
+ let (pre_mtime, pre_len) = pre_write_meta;
+ if cache.file_mtime != pre_mtime || cache.file_len != pre_len {
+ cache.secrets.clear();
+ FILE_GENERATION.fetch_add(1, Ordering::AcqRel);
+ }
+
+ let file_gen = FILE_GENERATION.load(Ordering::Acquire);
+
+ // Apply the API mutation to the cache.
match new_secret {
Some(secret) => {
cache.secrets.insert(
tokenid.clone(),
CachedSecret {
secret: secret.to_owned(),
+ file_gen,
},
);
}
@@ -172,4 +256,24 @@ fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
cache.secrets.remove(tokenid);
}
}
+
+ // Keep cache metadata aligned if possible.
+ match shadow_mtime_len() {
+ Ok((mtime, len)) => {
+ cache.file_mtime = mtime;
+ cache.file_len = len;
+ }
+ Err(_) => {
+ cache.file_mtime = None;
+ cache.file_len = None;
+ }
+ }
+}
+
+fn shadow_mtime_len() -> Result<(Option<SystemTime>, Option<u64>), Error> {
+ match fs::metadata(token_shadow().as_path()) {
+ Ok(meta) => Ok((meta.modified().ok(), Some(meta.len()))),
+ Err(e) if e.kind() == ErrorKind::NotFound => Ok((None, None)),
+ Err(e) => Err(e.into()),
+ }
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] [PATCH proxmox-backup v2 3/3] pbs-config: add TTL window to token secret cache
2025-12-17 16:25 14% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token " Samuel Rufinatscha
2025-12-17 16:25 13% ` [pbs-devel] [PATCH proxmox-backup v2 1/3] pbs-config: cache verified API token secrets Samuel Rufinatscha
2025-12-17 16:25 12% ` [pbs-devel] [PATCH proxmox-backup v2 2/3] pbs-config: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
@ 2025-12-17 16:25 14% ` Samuel Rufinatscha
2025-12-17 16:25 13% ` [pbs-devel] [PATCH proxmox v2 1/3] proxmox-access-control: cache verified API token secrets Samuel Rufinatscha
` (5 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-17 16:25 UTC (permalink / raw)
To: pbs-devel
Verify_secret() currently calls refresh_cache_if_file_changed() on every
request, which performs a metadata() call on token.shadow each time.
Under load this adds unnecessary overhead, considering also the file
usually should rarely change.
This patch introduces a TTL boundary, controlled by
TOKEN_SECRET_CACHE_TTL_SECS. File metadata is only re-loaded once the
TTL has expired. Documents TTL effects.
This patch partly fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v1 to v2:
- Add TOKEN_SECRET_CACHE_TTL_SECS and last_checked.
- Implement double-checked TTL: check with try_read first; only attempt
refresh with try_write if expired/unknown.
- Fix TTL bookkeeping: update last_checked on the “file unchanged” path
and after API mutations.
- Add documentation warning about TTL-delayed effect of manual
token.shadow edits.
docs/user-management.rst | 4 ++++
pbs-config/src/token_shadow.rs | 42 +++++++++++++++++++++++++++++++++-
2 files changed, 45 insertions(+), 1 deletion(-)
diff --git a/docs/user-management.rst b/docs/user-management.rst
index 41b43d60..32a9ec29 100644
--- a/docs/user-management.rst
+++ b/docs/user-management.rst
@@ -156,6 +156,10 @@ metadata:
Similarly, the ``user delete-token`` subcommand can be used to delete a token
again.
+.. WARNING:: If you manually remove a generated API token from the token secrets
+ file (token.shadow), it can take up to one minute before the token is
+ rejected. This is due to caching.
+
Newly generated API tokens don't have any permissions. Please read the next
section to learn how to set access permissions.
diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
index 71553aae..79940fd5 100644
--- a/pbs-config/src/token_shadow.rs
+++ b/pbs-config/src/token_shadow.rs
@@ -11,6 +11,7 @@ use serde::{Deserialize, Serialize};
use serde_json::{from_value, Value};
use proxmox_sys::fs::CreateOptions;
+use proxmox_time::epoch_i64;
use pbs_api_types::Authid;
//use crate::auth;
@@ -29,12 +30,15 @@ static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new
secrets: HashMap::new(),
file_mtime: None,
file_len: None,
+ last_checked: None,
})
});
/// API mutation generation (set/delete)
static API_MUTATION_GENERATION: AtomicU64 = AtomicU64::new(0);
/// External/manual edits generation for the token.shadow file
static FILE_GENERATION: AtomicU64 = AtomicU64::new(0);
+/// Max age in seconds of the token secret cache before checking for file changes.
+const TOKEN_SECRET_CACHE_TTL_SECS: i64 = 60;
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
@@ -74,22 +78,54 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
/// Refreshes the in-memory cache if the on-disk token.shadow file changed.
/// Returns true if the cache is valid to use, false if not.
fn refresh_cache_if_file_changed() -> bool {
+ let now = epoch_i64();
+
+ // Check TTL (best-effort)
+ let Some(cache) = TOKEN_SECRET_CACHE.try_read() else {
+ return false; // cannot validate external changes -> don't trust cache
+ };
+
+ let ttl_ok = cache
+ .last_checked
+ .is_some_and(|last| now.saturating_sub(last) < TOKEN_SECRET_CACHE_TTL_SECS);
+
+ drop(cache);
+
+ if ttl_ok {
+ return true;
+ }
+
+ // TTL expired/unknown at this point -> do best-effort refresh.
let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
return false; // cannot validate external changes -> don't trust cache
};
+ // Check TTL after acquiring write lock.
+ if let Some(last) = cache.last_checked {
+ if now.saturating_sub(last) < TOKEN_SECRET_CACHE_TTL_SECS {
+ return true;
+ }
+ }
+
+ let had_prior_state = cache.last_checked.is_some();
+
let Ok((new_mtime, new_len)) = shadow_mtime_len() else {
return false; // cannot validate external changes -> don't trust cache
};
if cache.file_mtime == new_mtime && cache.file_len == new_len {
+ cache.last_checked = Some(now);
return true;
}
cache.secrets.clear();
cache.file_mtime = new_mtime;
cache.file_len = new_len;
- FILE_GENERATION.fetch_add(1, Ordering::AcqRel);
+ cache.last_checked = Some(now);
+
+ if had_prior_state {
+ FILE_GENERATION.fetch_add(1, Ordering::AcqRel);
+ }
true
}
@@ -188,6 +224,8 @@ struct ApiTokenSecretCache {
file_mtime: Option<SystemTime>,
// shadow file length to detect changes
file_len: Option<u64>,
+ // last time the file metadata was checked
+ last_checked: Option<i64>,
}
/// Cached secret and the file generation it was cached at.
@@ -280,10 +318,12 @@ fn apply_api_mutation(
Ok((mtime, len)) => {
cache.file_mtime = mtime;
cache.file_len = len;
+ cache.last_checked = Some(epoch_i64());
}
Err(_) => {
cache.file_mtime = None;
cache.file_len = None;
+ cache.last_checked = None; // to force refresh next time
}
}
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 14%]
* [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token verification overhead
@ 2025-12-17 16:25 14% Samuel Rufinatscha
2025-12-17 16:25 13% ` [pbs-devel] [PATCH proxmox-backup v2 1/3] pbs-config: cache verified API token secrets Samuel Rufinatscha
` (8 more replies)
0 siblings, 9 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-17 16:25 UTC (permalink / raw)
To: pbs-devel
Hi,
this series improves the performance of token-based API authentication
in PBS (pbs-config) and in PDM (underlying proxmox-access-control
crate), addressing the API token verification hotspot reported in our
bugtracker #6049 [1].
When profiling PBS /status endpoint with cargo flamegraph [2],
token-based authentication showed up as a dominant hotspot via
proxmox_sys::crypt::verify_crypt_pw. Applying this series removes that
path from the hot section of the flamegraph. The same performance issue
was measured [2] for PDM. PDM uses the underlying shared
proxmox-access-control library for token handling, which is a
factored out version of the token.shadow handling code from PBS.
While this series fixes the immediate performance issue both in PBS
(pbs-config) and in the shared proxmox-access-control crate used by
PDM, PBS should eventually, ideally be refactored, in a separate
effort, to use proxmox-access-control for token handling instead of its
local implementation.
Problem
For token-based API requests, both PBS’s pbs-config token.shadow
handling and PDM proxmox-access-control’s token.shadow handling
currently:
1. read the token.shadow file on each request
2. deserialize it into a HashMap<Authid, String>
3. run password hash verification via
proxmox_sys::crypt::verify_crypt_pw for the provided token secret
Under load, this results in significant CPU usage spent in repeated
password hash computations for the same token+secret pairs. The
attached flamegraphs for PBS [2] and PDM [3] show
proxmox_sys::crypt::verify_crypt_pw dominating the hot path.
Approach
The goal is to reduce the cost of token-based authentication preserving
the existing token handling semantics (including detecting manual edits
to token.shadow) and be consistent between PBS (pbs-config) and
PDM (proxmox-access-control). For both sites, the series proposes
following approach:
1. Introduce an in-memory cache for verified token secrets
2. Invalidate the cache when token.shadow changes (detect manual edits)
3. Control metadata checks with a TTL window
Testing
*PBS (pbs-config)*
To verify the effect in PBS, I:
1. Set up test environment based on latest PBS ISO, installed Rust
toolchain, cloned proxmox-backup repository to use with cargo
flamegraph. Reproduced bug #6049 [1] by profiling the /status
endpoint with token-based authentication using cargo flamegraph [2].
The flamegraph showed proxmox_sys::crypt::verify_crypt_pw is the
hotspot.
2. Built PBS with pbs-config patches and re-ran the same workload and
profiling setup.
3. Confirmed that the proxmox_sys::crypt::verify_crypt_pw path no
longer appears in the hot section of the flamegraph. CPU usage is
now dominated by TLS overhead.
4. Functionally verified that:
* token-based API authentication still works for valid tokens
* invalid secrets are rejected as before
* generating a new token secret via dashboard works and
authenticates correctly
*PDM (proxmox-access-control)*
To verify the effect in PDM, I followed a similar testing approach.
Instead of /status, I profiled the /version endpoint with cargo
flamegraph [2] and verified that the token hashing path disappears [4]
from the hot section after applying the proxmox-access-control patches.
Functionally I verified that:
* token-based API authentication still works for valid tokens
* invalid secrets are rejected as before
* generating a new token secret via dashboard works and
authenticates correctly
Benchmarks:
Two different benchmarks have been run to measure caching effects
and RwLock contention:
(1) Requests per second for PBS /status endpoint (E2E)
(2) RwLock contention for token create/delete under
heavy parallel token-authenticated readers; compared
std::sync::RwLock and parking_lot::RwLock.
(1) benchmarked parallel token auth requests for
/status?verbose=0 on top of the datastore lookup cache series [5]
to check throughput impact. With datastores=1, repeat=5000, parallel=16
this series gives ~179 req/s compared to ~65 req/s without it.
This is a ~2.75x improvement.
(2) benchmarked token create/delete operations under heavy load of
token-authenticated requests on top of the datastore lookup cache [5]
series. This benchmark was done using against a 64-parallel
token-auth flood (200k requests) against
/admin/datastore/ds0001/status?verbose=0 while executing 50 token
create + 50 token delete operations. After the series I saw the
following e2e API latencies:
parking_lot::RwLock
- create avg ~27ms (p95 ~28ms) vs ~46ms (p95 ~50ms) baseline
- delete avg ~17ms (p95 ~19ms) vs ~33ms (p95 ~35ms) baseline
std::sync::RwLock
- create avg ~27ms (p95 ~28ms)
- create avg ~17ms (p95 ~19ms)
It appears that the both RwLock implementations perform similarly
for this workload. The parking_lot version has been chosen for the
added fairness guarantees.
Patch summary
pbs-config:
0001 – pbs-config: cache verified API token secrets
Adds an in-memory cache keyed by Authid that stores plain text token
secrets after a successful verification or generation and uses
openssl’s memcmp constant-time for comparison.
0002 – pbs-config: invalidate token-secret cache on token.shadow
changes
Tracks token.shadow mtime and length and clears the in-memory
cache when the file changes.
0003 – pbs-config: add TTL window to token-secret cache
Introduces a TTL (TOKEN_SECRET_CACHE_TTL_SECS, default 60) for metadata
checks so that fs::metadata is only called periodically.
proxmox-access-control:
0004 – access-control: cache verified API token secrets
Mirrors PBS PATCH 0001.
0005 – access-control: invalidate token-secret cache on token.shadow changes
Mirrors PBS PATCH 0002.
0006 – access-control: add TTL window to token-secret cache
Mirrors PBS PATCH 0003.
proxmox-datacenter-manager:
0007 – docs: document API token-cache TTL effects
Documents the effects of the TTL window on token.shadow edits
Changes since v1
- (refactor) Switched cache initialization to LazyLock
- (perf) Use parking_lot::RwLock and best-effort cache access on the
read/refresh path (try_read/try_write) to avoid lock contention
- (doc) Document TTL-delayed effect of manual token.shadow edits
- (fix) Add generation guards (API_MUTATION_GENERATION +
FILE_GENERATION) to prevent caching across concurrent set/delete and
external edits
Please see the patch specific changelogs for more details.
Thanks for considering this patch series, I look forward to your
feedback.
Best,
Samuel Rufinatscha
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
[2] attachment 1767 [1]: Flamegraph showing the proxmox_sys::crypt::verify_crypt_pw stack
[3] attachment 1794 [1]: Flamegraph PDM baseline
[4] attachment 1795 [1]: Flamegraph PDM patched
[5] https://bugzilla.proxmox.com/show_bug.cgi?id=6049
proxmox-backup:
Samuel Rufinatscha (3):
pbs-config: cache verified API token secrets
pbs-config: invalidate token-secret cache on token.shadow changes
pbs-config: add TTL window to token secret cache
Cargo.toml | 1 +
docs/user-management.rst | 4 +
pbs-config/Cargo.toml | 1 +
pbs-config/src/token_shadow.rs | 238 ++++++++++++++++++++++++++++++++-
4 files changed, 243 insertions(+), 1 deletion(-)
proxmox:
Samuel Rufinatscha (3):
proxmox-access-control: cache verified API token secrets
proxmox-access-control: invalidate token-secret cache on token.shadow
changes
proxmox-access-control: add TTL window to token secret cache
Cargo.toml | 1 +
proxmox-access-control/Cargo.toml | 1 +
proxmox-access-control/src/token_shadow.rs | 238 ++++++++++++++++++++-
3 files changed, 239 insertions(+), 1 deletion(-)
proxmox-datacenter-manager:
Samuel Rufinatscha (1):
docs: document API token-cache TTL effects
docs/access-control.rst | 3 +++
1 file changed, 3 insertions(+)
Summary over all repositories:
8 files changed, 485 insertions(+), 2 deletions(-)
--
Generated by git-murpp 0.8.1
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 14%]
* [pbs-devel] [PATCH proxmox-backup v2 1/3] pbs-config: cache verified API token secrets
2025-12-17 16:25 14% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token " Samuel Rufinatscha
@ 2025-12-17 16:25 13% ` Samuel Rufinatscha
2025-12-17 16:25 12% ` [pbs-devel] [PATCH proxmox-backup v2 2/3] pbs-config: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
` (7 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-17 16:25 UTC (permalink / raw)
To: pbs-devel
Currently, every token-based API request reads the token.shadow file and
runs the expensive password hash verification for the given token
secret. This shows up as a hotspot in /status profiling (see
bug #7017 [1]).
This patch introduces an in-memory cache of successfully verified token
secrets. Subsequent requests for the same token+secret combination only
perform a comparison using openssl::memcmp::eq and avoid re-running the
password hash. The cache is updated when a token secret is set and
cleared when a token is deleted. Note, this does NOT include manual
config changes, which will be covered in a subsequent patch.
This patch partly fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v1 to v2:
- Replace OnceCell with LazyLock, and std::sync::RwLock with
parking_lot::RwLock.
- Add API_MUTATION_GENERATION and guard cache inserts
to prevent “zombie inserts” across concurrent set/delete.
- Refactor cache operations into cache_try_secret_matches,
cache_try_insert_secret, and centralize write-side behavior in
apply_api_mutation.
- Switch fast-path cache access to try_read/try_write (best-effort).
Cargo.toml | 1 +
pbs-config/Cargo.toml | 1 +
pbs-config/src/token_shadow.rs | 94 +++++++++++++++++++++++++++++++++-
3 files changed, 95 insertions(+), 1 deletion(-)
diff --git a/Cargo.toml b/Cargo.toml
index ff143932..231cdca8 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -143,6 +143,7 @@ nom = "7"
num-traits = "0.2"
once_cell = "1.3.1"
openssl = "0.10.40"
+parking_lot = "0.12"
percent-encoding = "2.1"
pin-project-lite = "0.2"
regex = "1.5.5"
diff --git a/pbs-config/Cargo.toml b/pbs-config/Cargo.toml
index 74afb3c6..eb81ce00 100644
--- a/pbs-config/Cargo.toml
+++ b/pbs-config/Cargo.toml
@@ -13,6 +13,7 @@ libc.workspace = true
nix.workspace = true
once_cell.workspace = true
openssl.workspace = true
+parking_lot.workspace = true
regex.workspace = true
serde.workspace = true
serde_json.workspace = true
diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
index 640fabbf..ce845e8d 100644
--- a/pbs-config/src/token_shadow.rs
+++ b/pbs-config/src/token_shadow.rs
@@ -1,6 +1,9 @@
use std::collections::HashMap;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::LazyLock;
use anyhow::{bail, format_err, Error};
+use parking_lot::RwLock;
use serde::{Deserialize, Serialize};
use serde_json::{from_value, Value};
@@ -13,6 +16,19 @@ use crate::{open_backup_lockfile, BackupLockGuard};
const LOCK_FILE: &str = pbs_buildcfg::configdir!("/token.shadow.lock");
const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
+/// Global in-memory cache for successfully verified API token secrets.
+/// The cache stores plain text secrets for token Authids that have already been
+/// verified against the hashed values in `token.shadow`. This allows for cheap
+/// subsequent authentications for the same token+secret combination, avoiding
+/// recomputing the password hash on every request.
+static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new(|| {
+ RwLock::new(ApiTokenSecretCache {
+ secrets: HashMap::new(),
+ })
+});
+/// API mutation generation (set/delete)
+static API_MUTATION_GENERATION: AtomicU64 = AtomicU64::new(0);
+
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
/// ApiToken id / secret pair
@@ -54,9 +70,24 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
bail!("not an API token ID");
}
+ // Fast path
+ if cache_try_secret_matches(tokenid, secret) {
+ return Ok(());
+ }
+
+ // Slow path snapshot (before expensive work)
+ let api_gen_before = API_MUTATION_GENERATION.load(Ordering::Acquire);
+
let data = read_file()?;
match data.get(tokenid) {
- Some(hashed_secret) => proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
+ Some(hashed_secret) => {
+ proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
+
+ // Try to cache only if nothing changed while we verified
+ cache_try_insert_secret(tokenid.clone(), secret.to_owned(), api_gen_before);
+
+ Ok(())
+ }
None => bail!("invalid API token"),
}
}
@@ -82,6 +113,8 @@ fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
data.insert(tokenid.clone(), hashed_secret);
write_file(data)?;
+ apply_api_mutation(tokenid, Some(secret));
+
Ok(())
}
@@ -97,5 +130,64 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
data.remove(tokenid);
write_file(data)?;
+ apply_api_mutation(tokenid, None);
+
Ok(())
}
+
+struct ApiTokenSecretCache {
+ /// Keys are token Authids, values are the corresponding plain text secrets.
+ /// Entries are added after a successful on-disk verification in
+ /// `verify_secret` or when a new token secret is generated by
+ /// `generate_and_set_secret`. Used to avoid repeated
+ /// password-hash computation on subsequent authentications.
+ secrets: HashMap<Authid, CachedSecret>,
+}
+
+/// Cached secret.
+struct CachedSecret {
+ secret: String,
+}
+
+fn cache_try_insert_secret(tokenid: Authid, secret: String, api_gen_snapshot: u64) {
+ let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
+ return;
+ };
+
+ if API_MUTATION_GENERATION.load(Ordering::Acquire) == api_gen_snapshot {
+ cache.secrets.insert(tokenid, CachedSecret { secret });
+ }
+}
+
+fn cache_try_secret_matches(tokenid: &Authid, secret: &str) -> bool {
+ let Some(cache) = TOKEN_SECRET_CACHE.try_read() else {
+ return false;
+ };
+ let Some(entry) = cache.secrets.get(tokenid) else {
+ return false;
+ };
+
+ openssl::memcmp::eq(entry.secret.as_bytes(), secret.as_bytes())
+}
+
+fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
+ // Prevent in-flight verify_secret() from caching results across a mutation.
+ API_MUTATION_GENERATION.fetch_add(1, Ordering::AcqRel);
+
+ // Mutations must be reflected immediately once set/delete returns.
+ let mut cache = TOKEN_SECRET_CACHE.write();
+
+ match new_secret {
+ Some(secret) => {
+ cache.secrets.insert(
+ tokenid.clone(),
+ CachedSecret {
+ secret: secret.to_owned(),
+ },
+ );
+ }
+ None => {
+ cache.secrets.remove(tokenid);
+ }
+ }
+}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox v2 1/3] proxmox-access-control: cache verified API token secrets
2025-12-17 16:25 14% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token " Samuel Rufinatscha
` (2 preceding siblings ...)
2025-12-17 16:25 14% ` [pbs-devel] [PATCH proxmox-backup v2 3/3] pbs-config: add TTL window to token secret cache Samuel Rufinatscha
@ 2025-12-17 16:25 13% ` Samuel Rufinatscha
2025-12-17 16:25 12% ` [pbs-devel] [PATCH proxmox v2 2/3] proxmox-access-control: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
` (4 subsequent siblings)
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-17 16:25 UTC (permalink / raw)
To: pbs-devel
Currently, every token-based API request reads the token.shadow file and
runs the expensive password hash verification for the given token
secret. This issue was first observed as part of profiling the PBS
/status endpoint (see bug #7017 [1]) and is required for the factored
out proxmox_access_control token_shadow implementation too.
This patch introduces an in-memory cache of successfully verified token
secrets. Subsequent requests for the same token+secret combination only
perform a comparison using openssl::memcmp::eq and avoid re-running the
password hash. The cache is updated when a token secret is set and
cleared when a token is deleted. Note, this does NOT include manual
config changes, which will be covered in a subsequent patch.
This patch partly fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v1 to v2:
- Replace OnceCell with LazyLock, and std::sync::RwLock with
parking_lot::RwLock.
- Add API_MUTATION_GENERATION and guard cache inserts
to prevent “zombie inserts” across concurrent set/delete.
- Refactor cache operations into cache_try_secret_matches,
cache_try_insert_secret, and centralize write-side behavior in
apply_api_mutation.
- Switch fast-path cache access to try_read/try_write (best-effort).
Cargo.toml | 1 +
proxmox-access-control/Cargo.toml | 1 +
proxmox-access-control/src/token_shadow.rs | 94 +++++++++++++++++++++-
3 files changed, 95 insertions(+), 1 deletion(-)
diff --git a/Cargo.toml b/Cargo.toml
index 27a69afa..59a2ec93 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -112,6 +112,7 @@ native-tls = "0.2"
nix = "0.29"
openssl = "0.10"
pam-sys = "0.5"
+parking_lot = "0.12"
percent-encoding = "2.1"
pin-utils = "0.1.0"
proc-macro2 = "1.0"
diff --git a/proxmox-access-control/Cargo.toml b/proxmox-access-control/Cargo.toml
index ec189664..1de2842c 100644
--- a/proxmox-access-control/Cargo.toml
+++ b/proxmox-access-control/Cargo.toml
@@ -16,6 +16,7 @@ anyhow.workspace = true
const_format.workspace = true
nix = { workspace = true, optional = true }
openssl = { workspace = true, optional = true }
+parking_lot.workspace = true
regex.workspace = true
hex = { workspace = true, optional = true }
serde.workspace = true
diff --git a/proxmox-access-control/src/token_shadow.rs b/proxmox-access-control/src/token_shadow.rs
index c586d834..c0285b62 100644
--- a/proxmox-access-control/src/token_shadow.rs
+++ b/proxmox-access-control/src/token_shadow.rs
@@ -1,6 +1,9 @@
use std::collections::HashMap;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::LazyLock;
use anyhow::{bail, format_err, Error};
+use parking_lot::RwLock;
use serde_json::{from_value, Value};
use proxmox_auth_api::types::Authid;
@@ -8,6 +11,19 @@ use proxmox_product_config::{open_api_lockfile, replace_config, ApiLockGuard};
use crate::init::impl_feature::{token_shadow, token_shadow_lock};
+/// Global in-memory cache for successfully verified API token secrets.
+/// The cache stores plain text secrets for token Authids that have already been
+/// verified against the hashed values in `token.shadow`. This allows for cheap
+/// subsequent authentications for the same token+secret combination, avoiding
+/// recomputing the password hash on every request.
+static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new(|| {
+ RwLock::new(ApiTokenSecretCache {
+ secrets: HashMap::new(),
+ })
+});
+/// API mutation generation (set/delete)
+static API_MUTATION_GENERATION: AtomicU64 = AtomicU64::new(0);
+
// Get exclusive lock
fn lock_config() -> Result<ApiLockGuard, Error> {
open_api_lockfile(token_shadow_lock(), None, true)
@@ -36,9 +52,24 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
bail!("not an API token ID");
}
+ // Fast path
+ if cache_try_secret_matches(tokenid, secret) {
+ return Ok(());
+ }
+
+ // Slow path snapshot (before expensive work)
+ let api_gen_before = API_MUTATION_GENERATION.load(Ordering::Acquire);
+
let data = read_file()?;
match data.get(tokenid) {
- Some(hashed_secret) => proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
+ Some(hashed_secret) => {
+ proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
+
+ // Try to cache only if nothing changed while we verified
+ cache_try_insert_secret(tokenid.clone(), secret.to_owned(), api_gen_before);
+
+ Ok(())
+ }
None => bail!("invalid API token"),
}
}
@@ -56,6 +87,8 @@ pub fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
data.insert(tokenid.clone(), hashed_secret);
write_file(data)?;
+ apply_api_mutation(tokenid, Some(secret));
+
Ok(())
}
@@ -71,6 +104,8 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
data.remove(tokenid);
write_file(data)?;
+ apply_api_mutation(tokenid, None);
+
Ok(())
}
@@ -81,3 +116,60 @@ pub fn generate_and_set_secret(tokenid: &Authid) -> Result<String, Error> {
set_secret(tokenid, &secret)?;
Ok(secret)
}
+
+struct ApiTokenSecretCache {
+ /// Keys are token Authids, values are the corresponding plain text secrets.
+ /// Entries are added after a successful on-disk verification in
+ /// `verify_secret` or when a new token secret is generated by
+ /// `generate_and_set_secret`. Used to avoid repeated
+ /// password-hash computation on subsequent authentications.
+ secrets: HashMap<Authid, CachedSecret>,
+}
+
+/// Cached secret.
+struct CachedSecret {
+ secret: String,
+}
+
+fn cache_try_insert_secret(tokenid: Authid, secret: String, api_gen_snapshot: u64) {
+ let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
+ return;
+ };
+
+ if API_MUTATION_GENERATION.load(Ordering::Acquire) == api_gen_snapshot {
+ cache.secrets.insert(tokenid, CachedSecret { secret });
+ }
+}
+
+fn cache_try_secret_matches(tokenid: &Authid, secret: &str) -> bool {
+ let Some(cache) = TOKEN_SECRET_CACHE.try_read() else {
+ return false;
+ };
+ let Some(entry) = cache.secrets.get(tokenid) else {
+ return false;
+ };
+
+ openssl::memcmp::eq(entry.secret.as_bytes(), secret.as_bytes())
+}
+
+fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
+ // Prevent in-flight verify_secret() from caching results across a mutation.
+ API_MUTATION_GENERATION.fetch_add(1, Ordering::AcqRel);
+
+ // Mutations must be reflected immediately once set/delete returns.
+ let mut cache = TOKEN_SECRET_CACHE.write();
+
+ match new_secret {
+ Some(secret) => {
+ cache.secrets.insert(
+ tokenid.clone(),
+ CachedSecret {
+ secret: secret.to_owned(),
+ },
+ );
+ }
+ None => {
+ cache.secrets.remove(tokenid);
+ }
+ }
+}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] superseded: [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead
2025-12-05 13:25 15% [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Samuel Rufinatscha
` (6 preceding siblings ...)
2025-12-05 14:06 5% ` [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Shannon Sterz
@ 2025-12-17 16:27 13% ` Samuel Rufinatscha
7 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-17 16:27 UTC (permalink / raw)
To: pbs-devel
https://lore.proxmox.com/pbs-devel/20251217162520.486520-1-s.rufinatscha@proxmox.com/T/#t
On 12/5/25 2:25 PM, Samuel Rufinatscha wrote:
> Hi,
>
> this series improves the performance of token-based API authentication
> in PBS (pbs-config) and in PDM (underlying proxmox-access-control
> crate), addressing the API token verification hotspot reported in our
> bugtracker #6049 [1].
>
> When profiling PBS /status endpoint with cargo flamegraph [2],
> token-based authentication showed up as a dominant hotspot via
> proxmox_sys::crypt::verify_crypt_pw. Applying this series removes that
> path from the hot section of the flamegraph. The same performance issue
> was measured [3] for PDM. PDM uses the underlying shared
> proxmox-access-control library for token handling, which is a
> factored out version of the token.shadow handling code from PBS.
>
> While this series fixes the immediate performance issue both in PBS
> (pbs-config) and in the shared proxmox-access-control crate used by
> PDM, PBS should eventually, ideally be refactored, in a separate
> effort, to use proxmox-access-control for token handling instead of its
> local implementation.
>
> Problem
>
> For token-based API requests, both PBS’s pbs-config token.shadow
> handling and PDM proxmox-access-control’s token.shadow handling
> currently:
>
> 1. read the token.shadow file on each request
> 2. deserialize it into a HashMap<Authid, String>
> 3. run password hash verification via
> proxmox_sys::crypt::verify_crypt_pw for the provided token secret
>
> Under load, this results in significant CPU usage spent in repeated
> password hash computations for the same token+secret pairs. The
> attached flamegraphs for PBS [2] and PDM [3] show
> proxmox_sys::crypt::verify_crypt_pw dominating the hot path.
>
> Approach
>
> The goal is to reduce the cost of token-based authentication preserving
> the existing token handling semantics (including detecting manual edits
> to token.shadow) and be consistent between PBS (pbs-config) and
> PDM (proxmox-access-control). For both sites, the series proposes
> following approach:
>
> 1. Introduce an in-memory cache for verified token secrets
> 2. Invalidate the cache when token.shadow changes (detect manual edits)
> 3. Control metadata checks with a TTL window
>
> Testing
>
> *PBS (pbs-config)*
>
> To verify the effect in PBS, I:
> 1. Set up test environment based on latest PBS ISO, installed Rust
> toolchain, cloned proxmox-backup repository to use with cargo
> flamegraph. Reproduced bug #6049 [1] by profiling the /status
> endpoint with token-based authentication using cargo flamegraph [2].
> The flamegraph showed proxmox_sys::crypt::verify_crypt_pw is the
> hotspot.
> 2. Built PBS with pbs-config patches and re-ran the same workload and
> profiling setup.
> 3. Confirmed that the proxmox_sys::crypt::verify_crypt_pw path no
> longer appears in the hot section of the flamegraph. CPU usage is
> now dominated by TLS overhead.
> 4. Functionally verified that:
> * token-based API authentication still works for valid tokens
> * invalid secrets are rejected as before
> * generating a new token secret via dashboard works and
> authenticates correctly
>
> *PDM (proxmox-access-control)*
>
> To verify the effect in PDM, I followed a similar testing approach.
> Instead of /status, I profiled the /version endpoint with cargo
> flamegraph [3] and verified that the token hashing path disappears
> from the hot section after applying the proxmox-access-control patches.
>
> Functionally I verified that:
> * token-based API authentication still works for valid tokens
> * invalid secrets are rejected as before
> * generating a new token secret via dashboard works and
> authenticates correctly
>
> Patch summary
>
> pbs-config:
>
> 0001 – pbs-config: cache verified API token secrets
> Adds an in-memory cache keyed by Authid that stores plain text token
> secrets after a successful verification or generation and uses
> openssl’s memcmp constant-time for comparison.
>
> 0002 – pbs-config: invalidate token-secret cache on token.shadow changes
> Tracks token.shadow mtime and length and clears the in-memory cache
> when the file changes.
>
> 0003 – pbs-config: add TTL window to token-secret cache
> Introduces a TTL (TOKEN_SECRET_CACHE_TTL_SECS, default 60) for metadata checks so
> that fs::metadata is only called periodically.
>
> proxmox-access-control:
>
> 0004 – access-control: cache verified API token secrets
> Mirrors PBS patch 0001.
>
> 0005 – access-control: invalidate token-secret cache on token.shadow changes
> Mirrors PBS patch 0002.
>
> 0006 – access-control: add TTL window to token-secret cache
> Mirrors PBS patch 0003.
>
> Thanks for considering this patch series, I look forward to your
> feedback.
>
> Best,
> Samuel Rufinatscha
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] Flamegraph illustrating the`proxmox_sys::crypt::verify_crypt_pw
> hotspot before this series (attached to [1])
>
> proxmox-backup:
>
> Samuel Rufinatscha (3):
> pbs-config: cache verified API token secrets
> pbs-config: invalidate token-secret cache on token.shadow changes
> pbs-config: add TTL window to token secret cache
>
> pbs-config/src/token_shadow.rs | 109 ++++++++++++++++++++++++++++++++-
> 1 file changed, 108 insertions(+), 1 deletion(-)
>
>
> proxmox:
>
> Samuel Rufinatscha (3):
> proxmox-access-control: cache verified API token secrets
> proxmox-access-control: invalidate token-secret cache on token.shadow
> changes
> proxmox-access-control: add TTL window to token secret cache
>
> proxmox-access-control/src/token_shadow.rs | 108 ++++++++++++++++++++-
> 1 file changed, 107 insertions(+), 1 deletion(-)
>
>
> Summary over all repositories:
> 2 files changed, 215 insertions(+), 2 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* Re: [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token verification overhead
2025-12-17 16:25 14% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token " Samuel Rufinatscha
` (6 preceding siblings ...)
2025-12-17 16:25 17% ` [pbs-devel] [PATCH proxmox-datacenter-manager v2 1/1] docs: document API token-cache TTL effects Samuel Rufinatscha
@ 2025-12-18 11:03 12% ` Samuel Rufinatscha
2026-01-02 16:09 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2025-12-18 11:03 UTC (permalink / raw)
To: pbs-devel
It appears we need to switch the cache to a SharedMemory
implementation. I ran additional token invalidation tests, which showed
that regenerating a secret for a given token in the dashboard can take
up to one minute (TTL) to propagate, even though it calls the set_secret
API, which should directly update the cache.
I discussed this with Fabian, and it appears the root cause seems to be
that token modifications happen in the privileged daemon, while most
regular API requests are handled by the non-privileged one.
Moving the cache to a SharedMemory implementation should resolve
this. No other changes should be required.
I will send a v3 incorporating this change.
On 12/17/25 5:25 PM, Samuel Rufinatscha wrote:
> Hi,
>
> this series improves the performance of token-based API authentication
> in PBS (pbs-config) and in PDM (underlying proxmox-access-control
> crate), addressing the API token verification hotspot reported in our
> bugtracker #6049 [1].
>
> When profiling PBS /status endpoint with cargo flamegraph [2],
> token-based authentication showed up as a dominant hotspot via
> proxmox_sys::crypt::verify_crypt_pw. Applying this series removes that
> path from the hot section of the flamegraph. The same performance issue
> was measured [2] for PDM. PDM uses the underlying shared
> proxmox-access-control library for token handling, which is a
> factored out version of the token.shadow handling code from PBS.
>
> While this series fixes the immediate performance issue both in PBS
> (pbs-config) and in the shared proxmox-access-control crate used by
> PDM, PBS should eventually, ideally be refactored, in a separate
> effort, to use proxmox-access-control for token handling instead of its
> local implementation.
>
> Problem
>
> For token-based API requests, both PBS’s pbs-config token.shadow
> handling and PDM proxmox-access-control’s token.shadow handling
> currently:
>
> 1. read the token.shadow file on each request
> 2. deserialize it into a HashMap<Authid, String>
> 3. run password hash verification via
> proxmox_sys::crypt::verify_crypt_pw for the provided token secret
>
> Under load, this results in significant CPU usage spent in repeated
> password hash computations for the same token+secret pairs. The
> attached flamegraphs for PBS [2] and PDM [3] show
> proxmox_sys::crypt::verify_crypt_pw dominating the hot path.
>
> Approach
>
> The goal is to reduce the cost of token-based authentication preserving
> the existing token handling semantics (including detecting manual edits
> to token.shadow) and be consistent between PBS (pbs-config) and
> PDM (proxmox-access-control). For both sites, the series proposes
> following approach:
>
> 1. Introduce an in-memory cache for verified token secrets
> 2. Invalidate the cache when token.shadow changes (detect manual edits)
> 3. Control metadata checks with a TTL window
>
> Testing
>
> *PBS (pbs-config)*
>
> To verify the effect in PBS, I:
> 1. Set up test environment based on latest PBS ISO, installed Rust
> toolchain, cloned proxmox-backup repository to use with cargo
> flamegraph. Reproduced bug #6049 [1] by profiling the /status
> endpoint with token-based authentication using cargo flamegraph [2].
> The flamegraph showed proxmox_sys::crypt::verify_crypt_pw is the
> hotspot.
> 2. Built PBS with pbs-config patches and re-ran the same workload and
> profiling setup.
> 3. Confirmed that the proxmox_sys::crypt::verify_crypt_pw path no
> longer appears in the hot section of the flamegraph. CPU usage is
> now dominated by TLS overhead.
> 4. Functionally verified that:
> * token-based API authentication still works for valid tokens
> * invalid secrets are rejected as before
> * generating a new token secret via dashboard works and
> authenticates correctly
>
> *PDM (proxmox-access-control)*
>
> To verify the effect in PDM, I followed a similar testing approach.
> Instead of /status, I profiled the /version endpoint with cargo
> flamegraph [2] and verified that the token hashing path disappears [4]
> from the hot section after applying the proxmox-access-control patches.
>
> Functionally I verified that:
> * token-based API authentication still works for valid tokens
> * invalid secrets are rejected as before
> * generating a new token secret via dashboard works and
> authenticates correctly
>
> Benchmarks:
>
> Two different benchmarks have been run to measure caching effects
> and RwLock contention:
>
> (1) Requests per second for PBS /status endpoint (E2E)
> (2) RwLock contention for token create/delete under
> heavy parallel token-authenticated readers; compared
> std::sync::RwLock and parking_lot::RwLock.
>
> (1) benchmarked parallel token auth requests for
> /status?verbose=0 on top of the datastore lookup cache series [5]
> to check throughput impact. With datastores=1, repeat=5000, parallel=16
> this series gives ~179 req/s compared to ~65 req/s without it.
> This is a ~2.75x improvement.
>
> (2) benchmarked token create/delete operations under heavy load of
> token-authenticated requests on top of the datastore lookup cache [5]
> series. This benchmark was done using against a 64-parallel
> token-auth flood (200k requests) against
> /admin/datastore/ds0001/status?verbose=0 while executing 50 token
> create + 50 token delete operations. After the series I saw the
> following e2e API latencies:
>
> parking_lot::RwLock
> - create avg ~27ms (p95 ~28ms) vs ~46ms (p95 ~50ms) baseline
> - delete avg ~17ms (p95 ~19ms) vs ~33ms (p95 ~35ms) baseline
>
> std::sync::RwLock
> - create avg ~27ms (p95 ~28ms)
> - create avg ~17ms (p95 ~19ms)
>
> It appears that the both RwLock implementations perform similarly
> for this workload. The parking_lot version has been chosen for the
> added fairness guarantees.
>
> Patch summary
>
> pbs-config:
>
> 0001 – pbs-config: cache verified API token secrets
> Adds an in-memory cache keyed by Authid that stores plain text token
> secrets after a successful verification or generation and uses
> openssl’s memcmp constant-time for comparison.
>
> 0002 – pbs-config: invalidate token-secret cache on token.shadow
> changes
> Tracks token.shadow mtime and length and clears the in-memory
> cache when the file changes.
>
> 0003 – pbs-config: add TTL window to token-secret cache
> Introduces a TTL (TOKEN_SECRET_CACHE_TTL_SECS, default 60) for metadata
> checks so that fs::metadata is only called periodically.
>
> proxmox-access-control:
>
> 0004 – access-control: cache verified API token secrets
> Mirrors PBS PATCH 0001.
>
> 0005 – access-control: invalidate token-secret cache on token.shadow changes
> Mirrors PBS PATCH 0002.
>
> 0006 – access-control: add TTL window to token-secret cache
> Mirrors PBS PATCH 0003.
>
> proxmox-datacenter-manager:
>
> 0007 – docs: document API token-cache TTL effects
> Documents the effects of the TTL window on token.shadow edits
>
> Changes since v1
>
> - (refactor) Switched cache initialization to LazyLock
> - (perf) Use parking_lot::RwLock and best-effort cache access on the
> read/refresh path (try_read/try_write) to avoid lock contention
> - (doc) Document TTL-delayed effect of manual token.shadow edits
> - (fix) Add generation guards (API_MUTATION_GENERATION +
> FILE_GENERATION) to prevent caching across concurrent set/delete and
> external edits
>
> Please see the patch specific changelogs for more details.
>
> Thanks for considering this patch series, I look forward to your
> feedback.
>
> Best,
> Samuel Rufinatscha
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
> [2] attachment 1767 [1]: Flamegraph showing the proxmox_sys::crypt::verify_crypt_pw stack
> [3] attachment 1794 [1]: Flamegraph PDM baseline
> [4] attachment 1795 [1]: Flamegraph PDM patched
> [5] https://bugzilla.proxmox.com/show_bug.cgi?id=6049
>
> proxmox-backup:
>
> Samuel Rufinatscha (3):
> pbs-config: cache verified API token secrets
> pbs-config: invalidate token-secret cache on token.shadow changes
> pbs-config: add TTL window to token secret cache
>
> Cargo.toml | 1 +
> docs/user-management.rst | 4 +
> pbs-config/Cargo.toml | 1 +
> pbs-config/src/token_shadow.rs | 238 ++++++++++++++++++++++++++++++++-
> 4 files changed, 243 insertions(+), 1 deletion(-)
>
>
> proxmox:
>
> Samuel Rufinatscha (3):
> proxmox-access-control: cache verified API token secrets
> proxmox-access-control: invalidate token-secret cache on token.shadow
> changes
> proxmox-access-control: add TTL window to token secret cache
>
> Cargo.toml | 1 +
> proxmox-access-control/Cargo.toml | 1 +
> proxmox-access-control/src/token_shadow.rs | 238 ++++++++++++++++++++-
> 3 files changed, 239 insertions(+), 1 deletion(-)
>
>
> proxmox-datacenter-manager:
>
> Samuel Rufinatscha (1):
> docs: document API token-cache TTL effects
>
> docs/access-control.rst | 3 +++
> 1 file changed, 3 insertions(+)
>
>
> Summary over all repositories:
> 8 files changed, 485 insertions(+), 2 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] [PATCH proxmox-backup v3 4/4] pbs-config: add TTL window to token secret cache
2026-01-02 16:07 13% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v3 00/10] " Samuel Rufinatscha
` (2 preceding siblings ...)
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox-backup v3 3/4] pbs-config: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
@ 2026-01-02 16:07 15% ` Samuel Rufinatscha
2026-01-02 16:07 17% ` [pbs-devel] [PATCH proxmox v3 1/4] proxmox-access-control: extend AccessControlConfig for token.shadow invalidation Samuel Rufinatscha
` (5 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-02 16:07 UTC (permalink / raw)
To: pbs-devel
Verify_secret() currently calls refresh_cache_if_file_changed() on every
request, which performs a metadata() call on token.shadow each time.
Under load this adds unnecessary overhead, considering also the file
usually should rarely change.
This patch introduces a TTL boundary, controlled by
TOKEN_SECRET_CACHE_TTL_SECS. File metadata is only re-loaded once the
TTL has expired. Documents TTL effects.
This patch is part of the series which fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v1 to v2:
* Add TOKEN_SECRET_CACHE_TTL_SECS and last_checked.
* Implement double-checked TTL: check with try_read first; only attempt
refresh with try_write if expired/unknown.
* Fix TTL bookkeeping: update last_checked on the “file unchanged” path
and after API mutations.
* Add documentation warning about TTL-delayed effect of manual
token.shadow edits.
Changes from v2 to v3:
* Refactored refresh_cache_if_file_changed TTL logic.
* Remove had_prior_state check (replaced by last_checked logic).
* Improve TTL bound checks.
* Reword documentation warning for clarity.
docs/user-management.rst | 4 ++++
pbs-config/src/token_shadow.rs | 29 ++++++++++++++++++++++++++++-
2 files changed, 32 insertions(+), 1 deletion(-)
diff --git a/docs/user-management.rst b/docs/user-management.rst
index 41b43d60..8dfae528 100644
--- a/docs/user-management.rst
+++ b/docs/user-management.rst
@@ -156,6 +156,10 @@ metadata:
Similarly, the ``user delete-token`` subcommand can be used to delete a token
again.
+.. WARNING:: Direct/manual edits to ``token.shadow`` may take up to 60 seconds (or
+ longer in edge cases) to take effect due to caching. Restart services for
+ immediate effect of manual edits.
+
Newly generated API tokens don't have any permissions. Please read the next
section to learn how to set access permissions.
diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
index 02fb191b..e3529b40 100644
--- a/pbs-config/src/token_shadow.rs
+++ b/pbs-config/src/token_shadow.rs
@@ -33,6 +33,8 @@ static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new
last_checked: None,
})
});
+/// Max age in seconds of the token secret cache before checking for file changes.
+const TOKEN_SECRET_CACHE_TTL_SECS: i64 = 60;
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
@@ -74,11 +76,28 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
fn refresh_cache_if_file_changed() -> bool {
let now = epoch_i64();
- // Best-effort refresh under write lock.
+ // Fast path: cache is fresh if shared-gen matches and TTL not expired.
+ if let (Some(cache), Some(shared_gen_read)) =
+ (TOKEN_SECRET_CACHE.try_read(), token_shadow_shared_gen())
+ {
+ if cache.shared_gen == shared_gen_read
+ && cache
+ .last_checked
+ .is_some_and(|last| now >= last && (now - last) < TOKEN_SECRET_CACHE_TTL_SECS)
+ {
+ return true;
+ }
+ // read lock drops here
+ } else {
+ return false;
+ }
+
+ // Slow path: best-effort refresh under write lock.
let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
return false;
};
+ // Re-read generation after acquiring the lock (may have changed meanwhile).
let Some(shared_gen_now) = token_shadow_shared_gen() else {
return false;
};
@@ -89,6 +108,14 @@ fn refresh_cache_if_file_changed() -> bool {
cache.shared_gen = shared_gen_now;
}
+ // TTL check again after acquiring the lock
+ if cache
+ .last_checked
+ .is_some_and(|last| now >= last && (now - last) < TOKEN_SECRET_CACHE_TTL_SECS)
+ {
+ return true;
+ }
+
// Stat the file to detect manual edits.
let Ok((new_mtime, new_len)) = shadow_mtime_len() else {
return false;
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox-backup v3 2/4] pbs-config: cache verified API token secrets
2026-01-02 16:07 13% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v3 00/10] " Samuel Rufinatscha
2026-01-02 16:07 17% ` [pbs-devel] [PATCH proxmox-backup v3 1/4] pbs-config: add token.shadow generation to ConfigVersionCache Samuel Rufinatscha
@ 2026-01-02 16:07 12% ` Samuel Rufinatscha
2026-01-14 10:44 5% ` Fabian Grünbichler
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox-backup v3 3/4] pbs-config: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
` (7 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-02 16:07 UTC (permalink / raw)
To: pbs-devel
Currently, every token-based API request reads the token.shadow file and
runs the expensive password hash verification for the given token
secret. This shows up as a hotspot in /status profiling (see
bug #7017 [1]).
This patch introduces an in-memory cache of successfully verified token
secrets. Subsequent requests for the same token+secret combination only
perform a comparison using openssl::memcmp::eq and avoid re-running the
password hash. The cache is updated when a token secret is set and
cleared when a token is deleted. Note, this does NOT include manual
config changes, which will be covered in a subsequent patch.
This patch is part of the series which fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v1 to v2:
* Replace OnceCell with LazyLock, and std::sync::RwLock with
parking_lot::RwLock.
* Add API_MUTATION_GENERATION and guard cache inserts
to prevent “zombie inserts” across concurrent set/delete.
* Refactor cache operations into cache_try_secret_matches,
cache_try_insert_secret, and centralize write-side behavior in
apply_api_mutation.
* Switch fast-path cache access to try_read/try_write (best-effort).
Changes from v2 to v3:
* Replaced process-local cache invalidation (AtomicU64
API_MUTATION_GENERATION) with a cross-process shared generation via
ConfigVersionCache.
* Validate shared generation before/after the constant-time secret
compare; only insert into cache if the generation is unchanged.
* invalidate_cache_state() on insert if shared generation changed.
Cargo.toml | 1 +
pbs-config/Cargo.toml | 1 +
pbs-config/src/token_shadow.rs | 157 ++++++++++++++++++++++++++++++++-
3 files changed, 158 insertions(+), 1 deletion(-)
diff --git a/Cargo.toml b/Cargo.toml
index 1aa57ae5..821b63b7 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -143,6 +143,7 @@ nom = "7"
num-traits = "0.2"
once_cell = "1.3.1"
openssl = "0.10.40"
+parking_lot = "0.12"
percent-encoding = "2.1"
pin-project-lite = "0.2"
regex = "1.5.5"
diff --git a/pbs-config/Cargo.toml b/pbs-config/Cargo.toml
index 74afb3c6..eb81ce00 100644
--- a/pbs-config/Cargo.toml
+++ b/pbs-config/Cargo.toml
@@ -13,6 +13,7 @@ libc.workspace = true
nix.workspace = true
once_cell.workspace = true
openssl.workspace = true
+parking_lot.workspace = true
regex.workspace = true
serde.workspace = true
serde_json.workspace = true
diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
index 640fabbf..fa84aee5 100644
--- a/pbs-config/src/token_shadow.rs
+++ b/pbs-config/src/token_shadow.rs
@@ -1,6 +1,8 @@
use std::collections::HashMap;
+use std::sync::LazyLock;
use anyhow::{bail, format_err, Error};
+use parking_lot::RwLock;
use serde::{Deserialize, Serialize};
use serde_json::{from_value, Value};
@@ -13,6 +15,18 @@ use crate::{open_backup_lockfile, BackupLockGuard};
const LOCK_FILE: &str = pbs_buildcfg::configdir!("/token.shadow.lock");
const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
+/// Global in-memory cache for successfully verified API token secrets.
+/// The cache stores plain text secrets for token Authids that have already been
+/// verified against the hashed values in `token.shadow`. This allows for cheap
+/// subsequent authentications for the same token+secret combination, avoiding
+/// recomputing the password hash on every request.
+static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new(|| {
+ RwLock::new(ApiTokenSecretCache {
+ secrets: HashMap::new(),
+ shared_gen: 0,
+ })
+});
+
#[derive(Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
/// ApiToken id / secret pair
@@ -54,9 +68,27 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
bail!("not an API token ID");
}
+ // Fast path
+ if cache_try_secret_matches(tokenid, secret) {
+ return Ok(());
+ }
+
+ // Slow path
+ // First, capture the shared generation before doing the hash verification.
+ let gen_before = token_shadow_shared_gen();
+
let data = read_file()?;
match data.get(tokenid) {
- Some(hashed_secret) => proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
+ Some(hashed_secret) => {
+ proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
+
+ // Try to cache only if nothing changed while verifying the secret.
+ if let Some(gen) = gen_before {
+ cache_try_insert_secret(tokenid.clone(), secret.to_owned(), gen);
+ }
+
+ Ok(())
+ }
None => bail!("invalid API token"),
}
}
@@ -82,6 +114,8 @@ fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
data.insert(tokenid.clone(), hashed_secret);
write_file(data)?;
+ apply_api_mutation(tokenid, Some(secret));
+
Ok(())
}
@@ -97,5 +131,126 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
data.remove(tokenid);
write_file(data)?;
+ apply_api_mutation(tokenid, None);
+
Ok(())
}
+
+struct ApiTokenSecretCache {
+ /// Keys are token Authids, values are the corresponding plain text secrets.
+ /// Entries are added after a successful on-disk verification in
+ /// `verify_secret` or when a new token secret is generated by
+ /// `generate_and_set_secret`. Used to avoid repeated
+ /// password-hash computation on subsequent authentications.
+ secrets: HashMap<Authid, CachedSecret>,
+ /// Shared generation to detect mutations of the underlying token.shadow file.
+ shared_gen: usize,
+}
+
+/// Cached secret.
+struct CachedSecret {
+ secret: String,
+}
+
+fn cache_try_insert_secret(tokenid: Authid, secret: String, shared_gen_before: usize) {
+ let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
+ return;
+ };
+
+ let Some(shared_gen_now) = token_shadow_shared_gen() else {
+ return;
+ };
+
+ // If this process missed a generation bump, its cache is stale.
+ if cache.shared_gen != shared_gen_now {
+ invalidate_cache_state(&mut cache);
+ cache.shared_gen = shared_gen_now;
+ }
+
+ // If a mutation happened while we were verifying the secret, do not insert.
+ if shared_gen_now == shared_gen_before {
+ cache.secrets.insert(tokenid, CachedSecret { secret });
+ }
+}
+
+// Tries to match the given token secret against the cached secret.
+// Checks the generation before and after the constant-time compare to avoid a
+// TOCTOU window. If another process rotates/deletes a token while we're validating
+// the cached secret, the generation will change, and we
+// must not trust the cache for this request.
+fn cache_try_secret_matches(tokenid: &Authid, secret: &str) -> bool {
+ let Some(cache) = TOKEN_SECRET_CACHE.try_read() else {
+ return false;
+ };
+ let Some(entry) = cache.secrets.get(tokenid) else {
+ return false;
+ };
+
+ let cache_gen = cache.shared_gen;
+
+ let Some(gen1) = token_shadow_shared_gen() else {
+ return false;
+ };
+ if gen1 != cache_gen {
+ return false;
+ }
+
+ let eq = openssl::memcmp::eq(entry.secret.as_bytes(), secret.as_bytes());
+
+ let Some(gen2) = token_shadow_shared_gen() else {
+ return false;
+ };
+
+ eq && gen2 == cache_gen
+}
+
+fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
+ // Signal cache invalidation to other processes (best-effort).
+ let new_shared_gen = bump_token_shadow_shared_gen();
+
+ let mut cache = TOKEN_SECRET_CACHE.write();
+
+ // If we cannot read/bump the shared generation, we cannot safely trust the cache.
+ let Some(gen) = new_shared_gen else {
+ invalidate_cache_state(&mut cache);
+ cache.shared_gen = 0;
+ return;
+ };
+
+ // Update to the post-mutation generation.
+ cache.shared_gen = gen;
+
+ // Apply the new mutation.
+ match new_secret {
+ Some(secret) => {
+ cache.secrets.insert(
+ tokenid.clone(),
+ CachedSecret {
+ secret: secret.to_owned(),
+ },
+ );
+ }
+ None => {
+ cache.secrets.remove(tokenid);
+ }
+ }
+}
+
+/// Get the current shared generation.
+fn token_shadow_shared_gen() -> Option<usize> {
+ crate::ConfigVersionCache::new()
+ .ok()
+ .map(|cvc| cvc.token_shadow_generation())
+}
+
+/// Bump and return the new shared generation.
+fn bump_token_shadow_shared_gen() -> Option<usize> {
+ crate::ConfigVersionCache::new()
+ .ok()
+ .map(|cvc| cvc.increase_token_shadow_generation() + 1)
+}
+
+/// Invalidates the cache state and only keeps the shared generation.
+fn invalidate_cache_state(cache: &mut ApiTokenSecretCache) {
+ cache.secrets.clear();
+}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] [PATCH proxmox-backup v3 3/4] pbs-config: invalidate token-secret cache on token.shadow changes
2026-01-02 16:07 13% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v3 00/10] " Samuel Rufinatscha
2026-01-02 16:07 17% ` [pbs-devel] [PATCH proxmox-backup v3 1/4] pbs-config: add token.shadow generation to ConfigVersionCache Samuel Rufinatscha
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox-backup v3 2/4] pbs-config: cache verified API token secrets Samuel Rufinatscha
@ 2026-01-02 16:07 12% ` Samuel Rufinatscha
2026-01-14 10:44 5% ` Fabian Grünbichler
2026-01-02 16:07 15% ` [pbs-devel] [PATCH proxmox-backup v3 4/4] pbs-config: add TTL window to token secret cache Samuel Rufinatscha
` (6 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-02 16:07 UTC (permalink / raw)
To: pbs-devel
Previously the in-memory token-secret cache was only updated via
set_secret() and delete_secret(), so manual edits to token.shadow were
not reflected.
This patch adds file change detection to the cache. It tracks the mtime
and length of token.shadow and clears the in-memory token secret cache
whenever these values change.
Note, this patch fetches file stats on every request. An TTL-based
optimization will be covered in a subsequent patch of the series.
This patch is part of the series which fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v1 to v2:
* Add file metadata tracking (file_mtime, file_len) and
FILE_GENERATION.
* Store file_gen in CachedSecret and verify it against the current
FILE_GENERATION to ensure cached entries belong to the current file
state.
* Add shadow_mtime_len() helper and convert refresh to best-effort
(try_write, returns bool).
* Pass a pre-write metadata snapshot into apply_api_mutation and
clear/bump generation if the cache metadata indicates missed external
edits.
Changes from v2 to v3:
* Cache now tracks last_checked (epoch seconds).
* Simplified refresh_cache_if_file_changed, removed
FILE_GENERATION logic
* On first load, initializes file metadata and keeps empty cache.
pbs-config/src/token_shadow.rs | 122 +++++++++++++++++++++++++++++++--
1 file changed, 118 insertions(+), 4 deletions(-)
diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
index fa84aee5..02fb191b 100644
--- a/pbs-config/src/token_shadow.rs
+++ b/pbs-config/src/token_shadow.rs
@@ -1,5 +1,8 @@
use std::collections::HashMap;
+use std::fs;
+use std::io::ErrorKind;
use std::sync::LazyLock;
+use std::time::SystemTime;
use anyhow::{bail, format_err, Error};
use parking_lot::RwLock;
@@ -7,6 +10,7 @@ use serde::{Deserialize, Serialize};
use serde_json::{from_value, Value};
use proxmox_sys::fs::CreateOptions;
+use proxmox_time::epoch_i64;
use pbs_api_types::Authid;
//use crate::auth;
@@ -24,6 +28,9 @@ static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new
RwLock::new(ApiTokenSecretCache {
secrets: HashMap::new(),
shared_gen: 0,
+ file_mtime: None,
+ file_len: None,
+ last_checked: None,
})
});
@@ -62,6 +69,63 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
proxmox_sys::fs::replace_file(CONF_FILE, &json, options, true)
}
+/// Refreshes the in-memory cache if the on-disk token.shadow file changed.
+/// Returns true if the cache is valid to use, false if not.
+fn refresh_cache_if_file_changed() -> bool {
+ let now = epoch_i64();
+
+ // Best-effort refresh under write lock.
+ let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
+ return false;
+ };
+
+ let Some(shared_gen_now) = token_shadow_shared_gen() else {
+ return false;
+ };
+
+ // If another process bumped the generation, we don't know what changed -> clear cache
+ if cache.shared_gen != shared_gen_now {
+ invalidate_cache_state(&mut cache);
+ cache.shared_gen = shared_gen_now;
+ }
+
+ // Stat the file to detect manual edits.
+ let Ok((new_mtime, new_len)) = shadow_mtime_len() else {
+ return false;
+ };
+
+ // Initialize file stats if we have no prior state.
+ if cache.last_checked.is_none() {
+ cache.secrets.clear(); // ensure cache is empty on first load
+ cache.file_mtime = new_mtime;
+ cache.file_len = new_len;
+ cache.last_checked = Some(now);
+ return true;
+ }
+
+ // No change detected.
+ if cache.file_mtime == new_mtime && cache.file_len == new_len {
+ cache.last_checked = Some(now);
+ return true;
+ }
+
+ // Manual edit detected -> invalidate cache and update stat.
+ cache.secrets.clear();
+ cache.file_mtime = new_mtime;
+ cache.file_len = new_len;
+ cache.last_checked = Some(now);
+
+ // Best-effort propagation to other processes + update local view.
+ if let Some(shared_gen_new) = bump_token_shadow_shared_gen() {
+ cache.shared_gen = shared_gen_new;
+ } else {
+ // Do not fail: local cache is already safe as we cleared it above.
+ // Keep local shared_gen as-is to avoid repeated failed attempts.
+ }
+
+ true
+}
+
/// Verifies that an entry for given tokenid / API token secret exists
pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
if !tokenid.is_token() {
@@ -69,7 +133,7 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
}
// Fast path
- if cache_try_secret_matches(tokenid, secret) {
+ if refresh_cache_if_file_changed() && cache_try_secret_matches(tokenid, secret) {
return Ok(());
}
@@ -109,12 +173,15 @@ fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
let _guard = lock_config()?;
+ // Capture state before we write to detect external edits.
+ let pre_meta = shadow_mtime_len().unwrap_or((None, None));
+
let mut data = read_file()?;
let hashed_secret = proxmox_sys::crypt::encrypt_pw(secret)?;
data.insert(tokenid.clone(), hashed_secret);
write_file(data)?;
- apply_api_mutation(tokenid, Some(secret));
+ apply_api_mutation(tokenid, Some(secret), pre_meta);
Ok(())
}
@@ -127,11 +194,14 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
let _guard = lock_config()?;
+ // Capture state before we write to detect external edits.
+ let pre_meta = shadow_mtime_len().unwrap_or((None, None));
+
let mut data = read_file()?;
data.remove(tokenid);
write_file(data)?;
- apply_api_mutation(tokenid, None);
+ apply_api_mutation(tokenid, None, pre_meta);
Ok(())
}
@@ -145,6 +215,12 @@ struct ApiTokenSecretCache {
secrets: HashMap<Authid, CachedSecret>,
/// Shared generation to detect mutations of the underlying token.shadow file.
shared_gen: usize,
+ // shadow file mtime to detect changes
+ file_mtime: Option<SystemTime>,
+ // shadow file length to detect changes
+ file_len: Option<u64>,
+ // last time the file metadata was checked
+ last_checked: Option<i64>,
}
/// Cached secret.
@@ -204,7 +280,13 @@ fn cache_try_secret_matches(tokenid: &Authid, secret: &str) -> bool {
eq && gen2 == cache_gen
}
-fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
+fn apply_api_mutation(
+ tokenid: &Authid,
+ new_secret: Option<&str>,
+ pre_write_meta: (Option<SystemTime>, Option<u64>),
+) {
+ let now = epoch_i64();
+
// Signal cache invalidation to other processes (best-effort).
let new_shared_gen = bump_token_shadow_shared_gen();
@@ -220,6 +302,13 @@ fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
// Update to the post-mutation generation.
cache.shared_gen = gen;
+ // If our cached file metadata does not match the on-disk state before our write,
+ // we likely missed an external/manual edit. We can no longer trust any cached secrets.
+ let (pre_mtime, pre_len) = pre_write_meta;
+ if cache.file_mtime != pre_mtime || cache.file_len != pre_len {
+ cache.secrets.clear();
+ }
+
// Apply the new mutation.
match new_secret {
Some(secret) => {
@@ -234,6 +323,20 @@ fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
cache.secrets.remove(tokenid);
}
}
+
+ // Update our view of the file metadata to the post-write state (best-effort).
+ // (If this fails, drop local cache so callers fall back to slow path until refreshed.)
+ match shadow_mtime_len() {
+ Ok((mtime, len)) => {
+ cache.file_mtime = mtime;
+ cache.file_len = len;
+ cache.last_checked = Some(now);
+ }
+ Err(_) => {
+ // If we cannot validate state, do not trust cache.
+ invalidate_cache_state(&mut cache);
+ }
+ }
}
/// Get the current shared generation.
@@ -253,4 +356,15 @@ fn bump_token_shadow_shared_gen() -> Option<usize> {
/// Invalidates the cache state and only keeps the shared generation.
fn invalidate_cache_state(cache: &mut ApiTokenSecretCache) {
cache.secrets.clear();
+ cache.file_mtime = None;
+ cache.file_len = None;
+ cache.last_checked = None;
+}
+
+fn shadow_mtime_len() -> Result<(Option<SystemTime>, Option<u64>), Error> {
+ match fs::metadata(CONF_FILE) {
+ Ok(meta) => Ok((meta.modified().ok(), Some(meta.len()))),
+ Err(e) if e.kind() == ErrorKind::NotFound => Ok((None, None)),
+ Err(e) => Err(e.into()),
+ }
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v3 00/10] token-shadow: reduce api token verification overhead
@ 2026-01-02 16:07 13% Samuel Rufinatscha
2026-01-02 16:07 17% ` [pbs-devel] [PATCH proxmox-backup v3 1/4] pbs-config: add token.shadow generation to ConfigVersionCache Samuel Rufinatscha
` (9 more replies)
0 siblings, 10 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-02 16:07 UTC (permalink / raw)
To: pbs-devel
Hi,
this series improves the performance of token-based API authentication
in PBS (pbs-config) and in PDM (underlying proxmox-access-control
crate), addressing the API token verification hotspot reported in our
bugtracker #7017 [1].
When profiling PBS /status endpoint with cargo flamegraph [2],
token-based authentication showed up as a dominant hotspot via
proxmox_sys::crypt::verify_crypt_pw. Applying this series removes that
path from the hot section of the flamegraph. The same performance issue
was measured [2] for PDM. PDM uses the underlying shared
proxmox-access-control library for token handling, which is a
factored out version of the token.shadow handling code from PBS.
While this series fixes the immediate performance issue both in PBS
(pbs-config) and in the shared proxmox-access-control crate used by
PDM, PBS should eventually, ideally be refactored, in a separate
effort, to use proxmox-access-control for token handling instead of its
local implementation.
Problem
For token-based API requests, both PBS’s pbs-config token.shadow
handling and PDM proxmox-access-control’s token.shadow handling
currently:
1. read the token.shadow file on each request
2. deserialize it into a HashMap<Authid, String>
3. run password hash verification via
proxmox_sys::crypt::verify_crypt_pw for the provided token secret
Under load, this results in significant CPU usage spent in repeated
password hashing for the same token+secret pairs. The attached
flamegraphs for PBS [2] and PDM [3] show
proxmox_sys::crypt::verify_crypt_pw dominating the hot path.
Approach
The goal is to reduce the cost of token-based authentication preserving
the existing token handling semantics (including detecting manual edits
to token.shadow) and be consistent between PBS (pbs-config) and
PDM (proxmox-access-control). For both sites, this series proposes to:
1. Introduce an in-memory cache for verified token secrets and
invalidate it through a shared ConfigVersionCache generation. Note, a
shared generation is required to keep privileged and unprivileged
daemon in sync to avoid caching inconsistencies across processes.
2. Invalidate on token.shadow file API changes (set_secret,
delete_secret)
3. Invalidate on direct/manual token.shadow file changes (mtime +
length)
4. Avoid per-request file stat calls using a TTL window
Testing
*PBS (pbs-config)*
To verify the effect in PBS, I:
1. Set up test environment based on latest PBS ISO, installed Rust
toolchain, cloned proxmox-backup repository to use with cargo
flamegraph. Reproduced bug #7017 [1] by profiling the /status
endpoint with token-based authentication using cargo flamegraph [2].
2. Built PBS with pbs-config patches and re-ran the same workload and
profiling setup. Confirmed that
proxmox_sys::crypt::verify_crypt_pw path no longer appears in the
hot section of the flamegraph. CPU usage is now dominated by TLS
overhead.
3. Functionally-wise, I verified that:
* valid tokens authenticate correctly when used in API requests
* invalid secrets are rejected as before
* generating a new token secret via dashboard (create token for user,
regenerate existing secret) works and authenticates correctly
*PDM (proxmox-access-control)*
To verify the effect in PDM, I followed a similar testing approach.
Instead of PBS’ /status, I profiled the /version endpoint with cargo
flamegraph [2] and verified that the expensive hashing path disappears
from the hot section after introducing caching.
Functionally-wise, I verified that:
* valid tokens authenticate correctly when used in API requests
* invalid secrets are rejected as before
* generating a new token secret via dashboard (create token for user,
regenerate existing secret) works and authenticates correctly
Benchmarks:
Two different benchmarks have been run to measure caching effects
and RwLock contention:
(1) Requests per second for PBS /status endpoint (E2E)
Benchmarked parallel token auth requests for
/status?verbose=0 on top of the datastore lookup cache series [4]
to check throughput impact. With datastores=1, repeat=5000, parallel=16
this series gives ~172 req/s compared to ~65 req/s without it.
This is a ~2.6x improvement (and aligns with the ~179 req/s from the
previous series, which used per-process cache invalidation).
(2) RwLock contention for token create/delete under heavy load of
token-authenticated requests
The previous version of the series compared std::sync::RwLock and
parking_lot::RwLock contention for token create/delete under heavy
parallel token-authenticated readers. parking_lot::RwLock has been
chosen for the added fairness guarantees.
Patch summary
pbs-config:
0001 – pbs-config: add token.shadow generation to ConfigVersionCache
Extends ConfigVersionCache to provide a process-shared generation
number for token.shadow changes.
0002 – pbs-config: cache verified API token secrets
Adds an in-memory cache to cache verified, plain-text API token secrets.
Cache is invalidated through the process-shared ConfigVersionCache
generation number. Uses openssl’s memcmp constant-time for matching
secrets.
0003 – pbs-config: invalidate token-secret cache on token.shadow
changes
Stats token.shadow mtime and length and clears the cache when the
file changes, on each token verification request.
0004 – pbs-config: add TTL window to token-secret cache
Introduces a TTL (TOKEN_SECRET_CACHE_TTL_SECS, default 60) for metadata
checks so that fs::metadata calls are not performed on each request.
proxmox-access-control:
0005 – access-control: extend AccessControlConfig for token.shadow invalidation
Extends the AccessControlConfig trait with
token_shadow_cache_generation() and
increment_token_shadow_cache_generation() for
proxmox-access-control to get the shared token.shadow generation number
and bump it on token shadow changes.
0006 – access-control: cache verified API token secrets
Mirrors PBS PATCH 0002.
0007 – access-control: invalidate token-secret cache on token.shadow changes
Mirrors PBS PATCH 0003.
0008 – access-control: add TTL window to token-secret cache
Mirrors PBS PATCH 0004.
proxmox-datacenter-manager:
0009 – pdm-config: add token.shadow generation to ConfigVersionCache
Extends PDM ConfigVersionCache and implements
token_shadow_cache_generation() and
increment_token_shadow_cache_generation() from AccessControlConfig for
PDM.
0010 – docs: document API token-cache TTL effects
Documents the effects of the TTL window on token.shadow edits
Changes from v1 to v2:
* (refactor) Switched cache initialization to LazyLock
* (perf) Use parking_lot::RwLock and best-effort cache access on the
read/refresh path (try_read/try_write) to avoid lock contention
* (doc) Document TTL-delayed effect of manual token.shadow edits
* (fix) Add generation guards (API_MUTATION_GENERATION +
FILE_GENERATION) to prevent caching across concurrent set/delete and
external edits
Changes from v2 to v3:
* (refactor) Replace PBS per-process cache invalidation with a
cross-process token.shadow generation based on PBS
ConfigVersionCache, ensuring cache consistency between privileged
and unprivileged daemons.
* (refactor) Decoupling generation source from the
proxmox/proxmox-access-control cache implementation: extend
AccessControlConfig hooks so that products can provide the shared
token.shadow generation source.
* (refactor) Extend PDM's ConfigVersionCache with
token_shadow_generation
and introduce a pdm_config::AccessControlConfig wrapper implementing
the new proxmox-access-control trait hooks. Switch server and CLI
initialization to use pdm_config::AccessControlConfig instead of
pdm_api_types::AccessControlConfig.
* (refactor) Adapt generation checks around cached-secret comparison to
use the new shared generation source.
* (fix/logic) cache_try_insert_secret: Update the local cache
generation if stale, allowing the new secret to be inserted
immediately
* (refactor) Extract cache invalidation logic into a
invalidate_cache_state helper to reduce duplication and ensure
consistent state resets
* (refactor) Simplify refresh_cache_if_file_changed: handle the
un-initialized/reset state and adjust the generation mismatch
path to ensure file metadata is always re-read.
* (doc) Clarify TTL-delayed effects of manual token.shadow edits.
Please see the patch specific changelogs for more details.
Thanks for considering this patch series, I look forward to your
feedback.
Best,
Samuel Rufinatscha
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
[2] attachment 1767 [1]: Flamegraph showing the proxmox_sys::crypt::verify_crypt_pw stack
[3] attachment 1794 [1]: Flamegraph PDM baseline
[4] https://bugzilla.proxmox.com/show_bug.cgi?id=6049
proxmox-backup:
Samuel Rufinatscha (4):
pbs-config: add token.shadow generation to ConfigVersionCache
pbs-config: cache verified API token secrets
pbs-config: invalidate token-secret cache on token.shadow changes
pbs-config: add TTL window to token secret cache
Cargo.toml | 1 +
docs/user-management.rst | 4 +
pbs-config/Cargo.toml | 1 +
pbs-config/src/config_version_cache.rs | 18 ++
pbs-config/src/token_shadow.rs | 298 ++++++++++++++++++++++++-
5 files changed, 321 insertions(+), 1 deletion(-)
proxmox:
Samuel Rufinatscha (4):
proxmox-access-control: extend AccessControlConfig for token.shadow
invalidation
proxmox-access-control: cache verified API token secrets
proxmox-access-control: invalidate token-secret cache on token.shadow
changes
proxmox-access-control: add TTL window to token secret cache
Cargo.toml | 1 +
proxmox-access-control/Cargo.toml | 1 +
proxmox-access-control/src/init.rs | 17 ++
proxmox-access-control/src/token_shadow.rs | 299 ++++++++++++++++++++-
4 files changed, 317 insertions(+), 1 deletion(-)
proxmox-datacenter-manager:
Samuel Rufinatscha (2):
pdm-config: implement token.shadow generation
docs: document API token-cache TTL effects
cli/admin/src/main.rs | 2 +-
docs/access-control.rst | 4 ++
lib/pdm-config/Cargo.toml | 1 +
lib/pdm-config/src/access_control_config.rs | 73 +++++++++++++++++++++
lib/pdm-config/src/config_version_cache.rs | 18 +++++
lib/pdm-config/src/lib.rs | 2 +
server/src/acl.rs | 3 +-
7 files changed, 100 insertions(+), 3 deletions(-)
create mode 100644 lib/pdm-config/src/access_control_config.rs
Summary over all repositories:
16 files changed, 738 insertions(+), 5 deletions(-)
--
Generated by git-murpp 0.8.1
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox-datacenter-manager v3 2/2] docs: document API token-cache TTL effects
2026-01-02 16:07 13% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v3 00/10] " Samuel Rufinatscha
` (8 preceding siblings ...)
2026-01-02 16:07 13% ` [pbs-devel] [PATCH proxmox-datacenter-manager v3 1/2] pdm-config: implement token.shadow generation Samuel Rufinatscha
@ 2026-01-02 16:07 17% ` Samuel Rufinatscha
2026-01-14 10:45 5% ` Fabian Grünbichler
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-02 16:07 UTC (permalink / raw)
To: pbs-devel
Documents the effects of the added API token-cache in the
proxmox-access-control crate. This patch is part of the
series that fixes bug #7017 [1].
This patch is part of the series which fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v2 to v3:
* Reword documentation warning for clarity.
docs/access-control.rst | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/docs/access-control.rst b/docs/access-control.rst
index adf26cd..18e57a2 100644
--- a/docs/access-control.rst
+++ b/docs/access-control.rst
@@ -47,6 +47,10 @@ place of the user ID (``user@realm``) and the user password, respectively.
The API token is passed from the client to the server by setting the ``Authorization`` HTTP header
with method ``PDMAPIToken`` to the value ``TOKENID:TOKENSECRET``.
+.. WARNING:: Direct/manual edits to ``token.shadow`` may take up to 60 seconds (or
+ longer in edge cases) to take effect due to caching. Restart services for
+ immediate effect of manual edits.
+
.. _access_control:
Access Control
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 17%]
* [pbs-devel] [PATCH proxmox-datacenter-manager v3 1/2] pdm-config: implement token.shadow generation
2026-01-02 16:07 13% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v3 00/10] " Samuel Rufinatscha
` (7 preceding siblings ...)
2026-01-02 16:07 15% ` [pbs-devel] [PATCH proxmox v3 4/4] proxmox-access-control: add TTL window to token secret cache Samuel Rufinatscha
@ 2026-01-02 16:07 13% ` Samuel Rufinatscha
2026-01-14 10:45 5% ` Fabian Grünbichler
2026-01-02 16:07 17% ` [pbs-devel] [PATCH proxmox-datacenter-manager v3 2/2] docs: document API token-cache TTL effects Samuel Rufinatscha
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-02 16:07 UTC (permalink / raw)
To: pbs-devel
PDM depends on the shared proxmox/proxmox-access-control crate for
token.shadow handling, which expects the product to provide a
cross-process invalidation signal so it can safely cache verified API
token secrets and invalidate them when token.shadow is changed.
This patch
* adds a token_shadow_generation to PDM’s shared-memory
ConfigVersionCache
* implements proxmox_access_control::init::AccessControlConfig
for pdm_config::AccessControlConfig, which
- delegates roles/privs/path checks to the existing
pdm_api_types::AccessControlConfig implementation
- implements the shadow cache generation trait functions
* switches the AccessControlConfig init paths (server + CLI) to use
pdm_config::AccessControlConfig instead of
pdm_api_types::AccessControlConfig
This patch is part of the series which fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
cli/admin/src/main.rs | 2 +-
lib/pdm-config/Cargo.toml | 1 +
lib/pdm-config/src/access_control_config.rs | 73 +++++++++++++++++++++
lib/pdm-config/src/config_version_cache.rs | 18 +++++
lib/pdm-config/src/lib.rs | 2 +
server/src/acl.rs | 3 +-
6 files changed, 96 insertions(+), 3 deletions(-)
create mode 100644 lib/pdm-config/src/access_control_config.rs
diff --git a/cli/admin/src/main.rs b/cli/admin/src/main.rs
index f698fa2..916c633 100644
--- a/cli/admin/src/main.rs
+++ b/cli/admin/src/main.rs
@@ -19,7 +19,7 @@ fn main() {
proxmox_product_config::init(api_user, priv_user);
proxmox_access_control::init::init(
- &pdm_api_types::AccessControlConfig,
+ &pdm_config::AccessControlConfig,
pdm_buildcfg::configdir!("/access"),
)
.expect("failed to setup access control config");
diff --git a/lib/pdm-config/Cargo.toml b/lib/pdm-config/Cargo.toml
index d39c2ad..19781d2 100644
--- a/lib/pdm-config/Cargo.toml
+++ b/lib/pdm-config/Cargo.toml
@@ -13,6 +13,7 @@ once_cell.workspace = true
openssl.workspace = true
serde.workspace = true
+proxmox-access-control.workspace = true
proxmox-config-digest = { workspace = true, features = [ "openssl" ] }
proxmox-http = { workspace = true, features = [ "http-helpers" ] }
proxmox-ldap = { workspace = true, features = [ "types" ]}
diff --git a/lib/pdm-config/src/access_control_config.rs b/lib/pdm-config/src/access_control_config.rs
new file mode 100644
index 0000000..6f2e6b3
--- /dev/null
+++ b/lib/pdm-config/src/access_control_config.rs
@@ -0,0 +1,73 @@
+// e.g. in src/main.rs or server::context mod, wherever convenient
+
+use anyhow::Error;
+use pdm_api_types::{Authid, Userid};
+use proxmox_section_config::SectionConfigData;
+use std::collections::HashMap;
+
+pub struct AccessControlConfig;
+
+impl proxmox_access_control::init::AccessControlConfig for AccessControlConfig {
+ fn privileges(&self) -> &HashMap<&str, u64> {
+ pdm_api_types::AccessControlConfig.privileges()
+ }
+
+ fn roles(&self) -> &HashMap<&str, (u64, &str)> {
+ pdm_api_types::AccessControlConfig.roles()
+ }
+
+ fn is_superuser(&self, auth_id: &Authid) -> bool {
+ pdm_api_types::AccessControlConfig.is_superuser(auth_id)
+ }
+
+ fn is_group_member(&self, user_id: &Userid, group: &str) -> bool {
+ pdm_api_types::AccessControlConfig.is_group_member(user_id, group)
+ }
+
+ fn role_admin(&self) -> Option<&str> {
+ pdm_api_types::AccessControlConfig.role_admin()
+ }
+
+ fn role_no_access(&self) -> Option<&str> {
+ pdm_api_types::AccessControlConfig.role_no_access()
+ }
+
+ fn init_user_config(&self, config: &mut SectionConfigData) -> Result<(), Error> {
+ pdm_api_types::AccessControlConfig.init_user_config(config)
+ }
+
+ fn acl_audit_privileges(&self) -> u64 {
+ pdm_api_types::AccessControlConfig.acl_audit_privileges()
+ }
+
+ fn acl_modify_privileges(&self) -> u64 {
+ pdm_api_types::AccessControlConfig.acl_modify_privileges()
+ }
+
+ fn check_acl_path(&self, path: &str) -> Result<(), Error> {
+ pdm_api_types::AccessControlConfig.check_acl_path(path)
+ }
+
+ fn allow_partial_permission_match(&self) -> bool {
+ pdm_api_types::AccessControlConfig.allow_partial_permission_match()
+ }
+
+ fn cache_generation(&self) -> Option<usize> {
+ pdm_api_types::AccessControlConfig.cache_generation()
+ }
+
+ fn increment_cache_generation(&self) -> Result<(), Error> {
+ pdm_api_types::AccessControlConfig.increment_cache_generation()
+ }
+
+ fn token_shadow_cache_generation(&self) -> Option<usize> {
+ crate::ConfigVersionCache::new()
+ .ok()
+ .map(|c| c.token_shadow_generation())
+ }
+
+ fn increment_token_shadow_cache_generation(&self) -> Result<usize, Error> {
+ let c = crate::ConfigVersionCache::new()?;
+ Ok(c.increase_token_shadow_generation())
+ }
+}
diff --git a/lib/pdm-config/src/config_version_cache.rs b/lib/pdm-config/src/config_version_cache.rs
index 36a6a77..933140c 100644
--- a/lib/pdm-config/src/config_version_cache.rs
+++ b/lib/pdm-config/src/config_version_cache.rs
@@ -27,6 +27,8 @@ struct ConfigVersionCacheDataInner {
traffic_control_generation: AtomicUsize,
// Tracks updates to the remote/hostname/nodename mapping cache.
remote_mapping_cache: AtomicUsize,
+ // Token shadow (token.shadow) generation/version.
+ token_shadow_generation: AtomicUsize,
// Add further atomics here
}
@@ -172,4 +174,20 @@ impl ConfigVersionCache {
.fetch_add(1, Ordering::Relaxed)
+ 1
}
+
+ /// Returns the token shadow generation number.
+ pub fn token_shadow_generation(&self) -> usize {
+ self.shmem
+ .data()
+ .token_shadow_generation
+ .load(Ordering::Acquire)
+ }
+
+ /// Increase the token shadow generation number.
+ pub fn increase_token_shadow_generation(&self) -> usize {
+ self.shmem
+ .data()
+ .token_shadow_generation
+ .fetch_add(1, Ordering::AcqRel)
+ }
}
diff --git a/lib/pdm-config/src/lib.rs b/lib/pdm-config/src/lib.rs
index 4c49054..a15a006 100644
--- a/lib/pdm-config/src/lib.rs
+++ b/lib/pdm-config/src/lib.rs
@@ -9,6 +9,8 @@ pub mod remotes;
pub mod setup;
pub mod views;
+mod access_control_config;
+pub use access_control_config::AccessControlConfig;
mod config_version_cache;
pub use config_version_cache::ConfigVersionCache;
diff --git a/server/src/acl.rs b/server/src/acl.rs
index f421814..e6e007b 100644
--- a/server/src/acl.rs
+++ b/server/src/acl.rs
@@ -1,6 +1,5 @@
pub(crate) fn init() {
- static ACCESS_CONTROL_CONFIG: pdm_api_types::AccessControlConfig =
- pdm_api_types::AccessControlConfig;
+ static ACCESS_CONTROL_CONFIG: pdm_config::AccessControlConfig = pdm_config::AccessControlConfig;
proxmox_access_control::init::init(&ACCESS_CONTROL_CONFIG, pdm_buildcfg::configdir!("/access"))
.expect("failed to setup access control config");
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox-backup v3 1/4] pbs-config: add token.shadow generation to ConfigVersionCache
2026-01-02 16:07 13% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v3 00/10] " Samuel Rufinatscha
@ 2026-01-02 16:07 17% ` Samuel Rufinatscha
2026-01-14 10:44 5% ` Fabian Grünbichler
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox-backup v3 2/4] pbs-config: cache verified API token secrets Samuel Rufinatscha
` (8 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-02 16:07 UTC (permalink / raw)
To: pbs-devel
Currently, every token-based API request reads the token.shadow file and
runs the expensive password hash verification for the given token
secret. This shows up as a hotspot in /status profiling (see
bug #7017 [1]).
To solve the issue, this patch prepares the config version cache,
so that token_shadow_generation config caching can be built on
top of it.
This patch specifically:
(1) implements increment function in order to invalidate generations
This patch is part of the series which fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
pbs-config/src/config_version_cache.rs | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/pbs-config/src/config_version_cache.rs b/pbs-config/src/config_version_cache.rs
index e8fb994f..1376b11d 100644
--- a/pbs-config/src/config_version_cache.rs
+++ b/pbs-config/src/config_version_cache.rs
@@ -28,6 +28,8 @@ struct ConfigVersionCacheDataInner {
// datastore (datastore.cfg) generation/version
// FIXME: remove with PBS 3.0
datastore_generation: AtomicUsize,
+ // Token shadow (token.shadow) generation/version.
+ token_shadow_generation: AtomicUsize,
// Add further atomics here
}
@@ -153,4 +155,20 @@ impl ConfigVersionCache {
.datastore_generation
.fetch_add(1, Ordering::AcqRel)
}
+
+ /// Returns the token shadow generation number.
+ pub fn token_shadow_generation(&self) -> usize {
+ self.shmem
+ .data()
+ .token_shadow_generation
+ .load(Ordering::Acquire)
+ }
+
+ /// Increase the token shadow generation number.
+ pub fn increase_token_shadow_generation(&self) -> usize {
+ self.shmem
+ .data()
+ .token_shadow_generation
+ .fetch_add(1, Ordering::AcqRel)
+ }
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 17%]
* [pbs-devel] [PATCH proxmox v3 1/4] proxmox-access-control: extend AccessControlConfig for token.shadow invalidation
2026-01-02 16:07 13% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v3 00/10] " Samuel Rufinatscha
` (3 preceding siblings ...)
2026-01-02 16:07 15% ` [pbs-devel] [PATCH proxmox-backup v3 4/4] pbs-config: add TTL window to token secret cache Samuel Rufinatscha
@ 2026-01-02 16:07 17% ` Samuel Rufinatscha
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox v3 2/4] proxmox-access-control: cache verified API token secrets Samuel Rufinatscha
` (4 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-02 16:07 UTC (permalink / raw)
To: pbs-devel
Add token_shadow_cache_generation() and
increment_token_shadow_cache_generation()
hooks to AccessControlConfig. This lets products provide a cross-process
invalidation signal for token.shadow so proxmox-access-control can cache
verified API token secrets and invalidate that cache on token
rotation/deletion.
This patch is part of the series which fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-access-control/src/init.rs | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/proxmox-access-control/src/init.rs b/proxmox-access-control/src/init.rs
index e64398e8..0ba1a526 100644
--- a/proxmox-access-control/src/init.rs
+++ b/proxmox-access-control/src/init.rs
@@ -51,6 +51,23 @@ pub trait AccessControlConfig: Send + Sync {
Ok(())
}
+ /// Returns the current cache generation of the token shadow cache. If the generation was
+ /// incremented since the last time the cache was queried, the token shadow cache is reloaded
+ /// from disk.
+ ///
+ /// Default: Always returns `None`.
+ fn token_shadow_cache_generation(&self) -> Option<usize> {
+ None
+ }
+
+ /// Increment the cache generation of the token shadow cache. This indicates that it was
+ /// changed on disk.
+ ///
+ /// Default: Returns an error as token shadow generation is not supported.
+ fn increment_token_shadow_cache_generation(&self) -> Result<usize, Error> {
+ anyhow::bail!("token shadow generation not supported");
+ }
+
/// Optionally returns a role that has no access to any resource.
///
/// Default: Returns `None`.
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 17%]
* [pbs-devel] [PATCH proxmox v3 4/4] proxmox-access-control: add TTL window to token secret cache
2026-01-02 16:07 13% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v3 00/10] " Samuel Rufinatscha
` (6 preceding siblings ...)
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox v3 3/4] proxmox-access-control: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
@ 2026-01-02 16:07 15% ` Samuel Rufinatscha
2026-01-02 16:07 13% ` [pbs-devel] [PATCH proxmox-datacenter-manager v3 1/2] pdm-config: implement token.shadow generation Samuel Rufinatscha
2026-01-02 16:07 17% ` [pbs-devel] [PATCH proxmox-datacenter-manager v3 2/2] docs: document API token-cache TTL effects Samuel Rufinatscha
9 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-02 16:07 UTC (permalink / raw)
To: pbs-devel
Verify_secret() currently calls refresh_cache_if_file_changed() on every
request, which performs a metadata() call on token.shadow each time.
Under load this adds unnecessary overhead, considering also the file
should rarely change.
This patch introduces a TTL boundary, controlled by
TOKEN_SECRET_CACHE_TTL_SECS. File metadata is only re-loaded once the
TTL has expired.
This patch is part of the series which fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v1 to v2:
* Add TOKEN_SECRET_CACHE_TTL_SECS and last_checked.
* Implement double-checked TTL: check with try_read first; only attempt
refresh with try_write if expired/unknown.
* Fix TTL bookkeeping: update last_checked on the “file unchanged” path
and after API mutations.
* Add documentation warning about TTL-delayed effect of manual
token.shadow edits.
Changes from v2 to v3:
* Refactored refresh_cache_if_file_changed TTL logic.
* Remove had_prior_state check (replaced by last_checked logic).
* Improve TTL bound checks.
* Reword documentation warning for clarity.
proxmox-access-control/src/token_shadow.rs | 30 +++++++++++++++++++++-
1 file changed, 29 insertions(+), 1 deletion(-)
diff --git a/proxmox-access-control/src/token_shadow.rs b/proxmox-access-control/src/token_shadow.rs
index f30c8ed5..14eea560 100644
--- a/proxmox-access-control/src/token_shadow.rs
+++ b/proxmox-access-control/src/token_shadow.rs
@@ -30,6 +30,9 @@ static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new
})
});
+/// Max age in seconds of the token secret cache before checking for file changes.
+const TOKEN_SECRET_CACHE_TTL_SECS: i64 = 60;
+
// Get exclusive lock
fn lock_config() -> Result<ApiLockGuard, Error> {
open_api_lockfile(token_shadow_lock(), None, true)
@@ -57,11 +60,28 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
fn refresh_cache_if_file_changed() -> bool {
let now = epoch_i64();
- // Best-effort refresh under write lock.
+ // Fast path: cache is fresh if shared-gen matches and TTL not expired.
+ if let (Some(cache), Some(shared_gen_read)) =
+ (TOKEN_SECRET_CACHE.try_read(), token_shadow_shared_gen())
+ {
+ if cache.shared_gen == shared_gen_read
+ && cache
+ .last_checked
+ .is_some_and(|last| now >= last && (now - last) < TOKEN_SECRET_CACHE_TTL_SECS)
+ {
+ return true;
+ }
+ // read lock drops here
+ } else {
+ return false;
+ }
+
+ // Slow path: best-effort refresh under write lock.
let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
return false;
};
+ // Re-read generation after acquiring the lock (may have changed meanwhile).
let Some(shared_gen_now) = token_shadow_shared_gen() else {
return false;
};
@@ -72,6 +92,14 @@ fn refresh_cache_if_file_changed() -> bool {
cache.shared_gen = shared_gen_now;
}
+ // TTL check again after acquiring the lock
+ if cache
+ .last_checked
+ .is_some_and(|last| now >= last && (now - last) < TOKEN_SECRET_CACHE_TTL_SECS)
+ {
+ return true;
+ }
+
// Stat the file to detect manual edits.
let Ok((new_mtime, new_len)) = shadow_mtime_len() else {
return false;
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox v3 2/4] proxmox-access-control: cache verified API token secrets
2026-01-02 16:07 13% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v3 00/10] " Samuel Rufinatscha
` (4 preceding siblings ...)
2026-01-02 16:07 17% ` [pbs-devel] [PATCH proxmox v3 1/4] proxmox-access-control: extend AccessControlConfig for token.shadow invalidation Samuel Rufinatscha
@ 2026-01-02 16:07 12% ` Samuel Rufinatscha
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox v3 3/4] proxmox-access-control: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
` (3 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-02 16:07 UTC (permalink / raw)
To: pbs-devel
Currently, every token-based API request reads the token.shadow file and
runs the expensive password hash verification for the given token
secret. This issue was first observed as part of profiling the PBS
/status endpoint (see bug #7017 [1]) and is required for the factored
out proxmox_access_control token_shadow implementation too.
This patch introduces an in-memory cache of successfully verified token
secrets. Subsequent requests for the same token+secret combination only
perform a comparison using openssl::memcmp::eq and avoid re-running the
password hash. The cache is updated when a token secret is set and
cleared when a token is deleted. Note, this does NOT include manual
config changes, which will be covered in a subsequent patch.
This patch is part of the series which fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v1 to v2:
* Replace OnceCell with LazyLock, and std::sync::RwLock with
parking_lot::RwLock.
* Add API_MUTATION_GENERATION and guard cache inserts
to prevent “zombie inserts” across concurrent set/delete.
* Refactor cache operations into cache_try_secret_matches,
cache_try_insert_secret, and centralize write-side behavior in
apply_api_mutation.
* Switch fast-path cache access to try_read/try_write (best-effort).
Changes from v2 to v3:
* Replaced process-local cache invalidation (AtomicU64
API_MUTATION_GENERATION) with a cross-process shared generation via
ConfigVersionCache.
* Validate shared generation before/after the constant-time secret
compare; only insert into cache if the generation is unchanged.
* invalidate_cache_state() on insert if shared generation changed.
Cargo.toml | 1 +
proxmox-access-control/Cargo.toml | 1 +
proxmox-access-control/src/token_shadow.rs | 154 ++++++++++++++++++++-
3 files changed, 155 insertions(+), 1 deletion(-)
diff --git a/Cargo.toml b/Cargo.toml
index 27a69afa..59a2ec93 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -112,6 +112,7 @@ native-tls = "0.2"
nix = "0.29"
openssl = "0.10"
pam-sys = "0.5"
+parking_lot = "0.12"
percent-encoding = "2.1"
pin-utils = "0.1.0"
proc-macro2 = "1.0"
diff --git a/proxmox-access-control/Cargo.toml b/proxmox-access-control/Cargo.toml
index ec189664..1de2842c 100644
--- a/proxmox-access-control/Cargo.toml
+++ b/proxmox-access-control/Cargo.toml
@@ -16,6 +16,7 @@ anyhow.workspace = true
const_format.workspace = true
nix = { workspace = true, optional = true }
openssl = { workspace = true, optional = true }
+parking_lot.workspace = true
regex.workspace = true
hex = { workspace = true, optional = true }
serde.workspace = true
diff --git a/proxmox-access-control/src/token_shadow.rs b/proxmox-access-control/src/token_shadow.rs
index c586d834..895309d2 100644
--- a/proxmox-access-control/src/token_shadow.rs
+++ b/proxmox-access-control/src/token_shadow.rs
@@ -1,13 +1,28 @@
use std::collections::HashMap;
+use std::sync::LazyLock;
use anyhow::{bail, format_err, Error};
+use parking_lot::RwLock;
use serde_json::{from_value, Value};
use proxmox_auth_api::types::Authid;
use proxmox_product_config::{open_api_lockfile, replace_config, ApiLockGuard};
+use crate::init::access_conf;
use crate::init::impl_feature::{token_shadow, token_shadow_lock};
+/// Global in-memory cache for successfully verified API token secrets.
+/// The cache stores plain text secrets for token Authids that have already been
+/// verified against the hashed values in `token.shadow`. This allows for cheap
+/// subsequent authentications for the same token+secret combination, avoiding
+/// recomputing the password hash on every request.
+static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new(|| {
+ RwLock::new(ApiTokenSecretCache {
+ secrets: HashMap::new(),
+ shared_gen: 0,
+ })
+});
+
// Get exclusive lock
fn lock_config() -> Result<ApiLockGuard, Error> {
open_api_lockfile(token_shadow_lock(), None, true)
@@ -36,9 +51,27 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
bail!("not an API token ID");
}
+ // Fast path
+ if cache_try_secret_matches(tokenid, secret) {
+ return Ok(());
+ }
+
+ // Slow path
+ // First, capture the shared generation before doing the hash verification.
+ let gen_before = token_shadow_shared_gen();
+
let data = read_file()?;
match data.get(tokenid) {
- Some(hashed_secret) => proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
+ Some(hashed_secret) => {
+ proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
+
+ // Try to cache only if nothing changed while verifying the secret.
+ if let Some(gen) = gen_before {
+ cache_try_insert_secret(tokenid.clone(), secret.to_owned(), gen);
+ }
+
+ Ok(())
+ }
None => bail!("invalid API token"),
}
}
@@ -56,6 +89,8 @@ pub fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
data.insert(tokenid.clone(), hashed_secret);
write_file(data)?;
+ apply_api_mutation(tokenid, Some(secret));
+
Ok(())
}
@@ -71,6 +106,8 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
data.remove(tokenid);
write_file(data)?;
+ apply_api_mutation(tokenid, None);
+
Ok(())
}
@@ -81,3 +118,118 @@ pub fn generate_and_set_secret(tokenid: &Authid) -> Result<String, Error> {
set_secret(tokenid, &secret)?;
Ok(secret)
}
+
+struct ApiTokenSecretCache {
+ /// Keys are token Authids, values are the corresponding plain text secrets.
+ /// Entries are added after a successful on-disk verification in
+ /// `verify_secret` or when a new token secret is generated by
+ /// `generate_and_set_secret`. Used to avoid repeated
+ /// password-hash computation on subsequent authentications.
+ secrets: HashMap<Authid, CachedSecret>,
+ /// Shared generation to detect mutations of the underlying token.shadow file.
+ shared_gen: usize,
+}
+
+/// Cached secret.
+struct CachedSecret {
+ secret: String,
+}
+
+fn cache_try_insert_secret(tokenid: Authid, secret: String, shared_gen_before: usize) {
+ let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
+ return;
+ };
+
+ let Some(shared_gen_now) = token_shadow_shared_gen() else {
+ return;
+ };
+
+ // If this process missed a generation bump, its cache is stale.
+ if cache.shared_gen != shared_gen_now {
+ invalidate_cache_state(&mut cache);
+ cache.shared_gen = shared_gen_now;
+ }
+
+ // If a mutation happened while we were verifying the secret, do not insert.
+ if shared_gen_now == shared_gen_before {
+ cache.secrets.insert(tokenid, CachedSecret { secret });
+ }
+}
+
+// Tries to match the given token secret against the cached secret.
+// Checks the generation before and after the constant-time compare to avoid a
+// TOCTOU window. If another process rotates/deletes a token while we're validating
+// the cached secret, the generation will change, and we
+// must not trust the cache for this request.
+fn cache_try_secret_matches(tokenid: &Authid, secret: &str) -> bool {
+ let Some(cache) = TOKEN_SECRET_CACHE.try_read() else {
+ return false;
+ };
+ let Some(entry) = cache.secrets.get(tokenid) else {
+ return false;
+ };
+
+ let cache_gen = cache.shared_gen;
+
+ let Some(gen1) = token_shadow_shared_gen() else {
+ return false;
+ };
+ if gen1 != cache_gen {
+ return false;
+ }
+
+ let eq = openssl::memcmp::eq(entry.secret.as_bytes(), secret.as_bytes());
+
+ let Some(gen2) = token_shadow_shared_gen() else {
+ return false;
+ };
+
+ eq && gen2 == cache_gen
+}
+
+fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
+ // Signal cache invalidation to other processes (best-effort).
+ let new_shared_gen = bump_token_shadow_shared_gen();
+
+ let mut cache = TOKEN_SECRET_CACHE.write();
+
+ // If we cannot read/bump the shared generation, we cannot safely trust the cache.
+ let Some(gen) = new_shared_gen else {
+ invalidate_cache_state(&mut cache);
+ cache.shared_gen = 0;
+ return;
+ };
+
+ // Update to the post-mutation generation.
+ cache.shared_gen = gen;
+
+ // Apply the new mutation.
+ match new_secret {
+ Some(secret) => {
+ cache.secrets.insert(
+ tokenid.clone(),
+ CachedSecret {
+ secret: secret.to_owned(),
+ },
+ );
+ }
+ None => {
+ cache.secrets.remove(tokenid);
+ }
+ }
+}
+
+/// Get the current shared generation.
+fn token_shadow_shared_gen() -> Option<usize> {
+ access_conf().token_shadow_cache_generation()
+}
+
+/// Bump and return the new shared generation.
+fn bump_token_shadow_shared_gen() -> Option<usize> {
+ access_conf().increment_token_shadow_cache_generation().ok().map(|prev| prev + 1)
+}
+
+/// Invalidates the cache state and only keeps the shared generation.
+fn invalidate_cache_state(cache: &mut ApiTokenSecretCache) {
+ cache.secrets.clear();
+}
\ No newline at end of file
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] [PATCH proxmox v3 3/4] proxmox-access-control: invalidate token-secret cache on token.shadow changes
2026-01-02 16:07 13% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v3 00/10] " Samuel Rufinatscha
` (5 preceding siblings ...)
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox v3 2/4] proxmox-access-control: cache verified API token secrets Samuel Rufinatscha
@ 2026-01-02 16:07 12% ` Samuel Rufinatscha
2026-01-02 16:07 15% ` [pbs-devel] [PATCH proxmox v3 4/4] proxmox-access-control: add TTL window to token secret cache Samuel Rufinatscha
` (2 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-02 16:07 UTC (permalink / raw)
To: pbs-devel
Previously the in-memory token-secret cache was only updated via
set_secret() and delete_secret(), so manual edits to token.shadow were
not reflected.
This patch adds file change detection to the cache. It tracks the mtime
and length of token.shadow and clears the in-memory token secret cache
whenever these values change.
Note, this patch fetches file stats on every request. An TTL-based
optimization will be covered in a subsequent patch of the series.
This patch is part of the series which fixes bug #7017 [1].
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes from v1 to v2:
* Add file metadata tracking (file_mtime, file_len) and
FILE_GENERATION.
* Store file_gen in CachedSecret and verify it against the current
FILE_GENERATION to ensure cached entries belong to the current file
state.
* Add shadow_mtime_len() helper and convert refresh to best-effort
(try_write, returns bool).
* Pass a pre-write metadata snapshot into apply_api_mutation and
clear/bump generation if the cache metadata indicates missed external
edits.
Changes from v2 to v3:
* Cache now tracks last_checked (epoch seconds).
* Simplified refresh_cache_if_file_changed, removed
FILE_GENERATION logic
* On first load, initializes file metadata and keeps empty cache.
proxmox-access-control/src/token_shadow.rs | 129 ++++++++++++++++++++-
1 file changed, 123 insertions(+), 6 deletions(-)
diff --git a/proxmox-access-control/src/token_shadow.rs b/proxmox-access-control/src/token_shadow.rs
index 895309d2..f30c8ed5 100644
--- a/proxmox-access-control/src/token_shadow.rs
+++ b/proxmox-access-control/src/token_shadow.rs
@@ -1,5 +1,8 @@
use std::collections::HashMap;
+use std::fs;
+use std::io::ErrorKind;
use std::sync::LazyLock;
+use std::time::SystemTime;
use anyhow::{bail, format_err, Error};
use parking_lot::RwLock;
@@ -7,6 +10,7 @@ use serde_json::{from_value, Value};
use proxmox_auth_api::types::Authid;
use proxmox_product_config::{open_api_lockfile, replace_config, ApiLockGuard};
+use proxmox_time::epoch_i64;
use crate::init::access_conf;
use crate::init::impl_feature::{token_shadow, token_shadow_lock};
@@ -20,6 +24,9 @@ static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new
RwLock::new(ApiTokenSecretCache {
secrets: HashMap::new(),
shared_gen: 0,
+ file_mtime: None,
+ file_len: None,
+ last_checked: None,
})
});
@@ -45,6 +52,63 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
replace_config(token_shadow(), &json)
}
+/// Refreshes the in-memory cache if the on-disk token.shadow file changed.
+/// Returns true if the cache is valid to use, false if not.
+fn refresh_cache_if_file_changed() -> bool {
+ let now = epoch_i64();
+
+ // Best-effort refresh under write lock.
+ let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
+ return false;
+ };
+
+ let Some(shared_gen_now) = token_shadow_shared_gen() else {
+ return false;
+ };
+
+ // If another process bumped the generation, we don't know what changed -> clear cache
+ if cache.shared_gen != shared_gen_now {
+ invalidate_cache_state(&mut cache);
+ cache.shared_gen = shared_gen_now;
+ }
+
+ // Stat the file to detect manual edits.
+ let Ok((new_mtime, new_len)) = shadow_mtime_len() else {
+ return false;
+ };
+
+ // Initialize file stats if we have no prior state.
+ if cache.last_checked.is_none() {
+ cache.secrets.clear(); // ensure cache is empty on first load
+ cache.file_mtime = new_mtime;
+ cache.file_len = new_len;
+ cache.last_checked = Some(now);
+ return true;
+ }
+
+ // No change detected.
+ if cache.file_mtime == new_mtime && cache.file_len == new_len {
+ cache.last_checked = Some(now);
+ return true;
+ }
+
+ // Manual edit detected -> invalidate cache and update stat.
+ cache.secrets.clear();
+ cache.file_mtime = new_mtime;
+ cache.file_len = new_len;
+ cache.last_checked = Some(now);
+
+ // Best-effort propagation to other processes + update local view.
+ if let Some(shared_gen_new) = bump_token_shadow_shared_gen() {
+ cache.shared_gen = shared_gen_new;
+ } else {
+ // Do not fail: local cache is already safe as we cleared it above.
+ // Keep local shared_gen as-is to avoid repeated failed attempts.
+ }
+
+ true
+}
+
/// Verifies that an entry for given tokenid / API token secret exists
pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
if !tokenid.is_token() {
@@ -52,7 +116,7 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
}
// Fast path
- if cache_try_secret_matches(tokenid, secret) {
+ if refresh_cache_if_file_changed() && cache_try_secret_matches(tokenid, secret) {
return Ok(());
}
@@ -84,12 +148,15 @@ pub fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
let _guard = lock_config()?;
+ // Capture state before we write to detect external edits.
+ let pre_meta = shadow_mtime_len().unwrap_or((None, None));
+
let mut data = read_file()?;
let hashed_secret = proxmox_sys::crypt::encrypt_pw(secret)?;
data.insert(tokenid.clone(), hashed_secret);
write_file(data)?;
- apply_api_mutation(tokenid, Some(secret));
+ apply_api_mutation(tokenid, Some(secret), pre_meta);
Ok(())
}
@@ -102,11 +169,14 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
let _guard = lock_config()?;
+ // Capture state before we write to detect external edits.
+ let pre_meta = shadow_mtime_len().unwrap_or((None, None));
+
let mut data = read_file()?;
data.remove(tokenid);
write_file(data)?;
- apply_api_mutation(tokenid, None);
+ apply_api_mutation(tokenid, None, pre_meta);
Ok(())
}
@@ -128,6 +198,12 @@ struct ApiTokenSecretCache {
secrets: HashMap<Authid, CachedSecret>,
/// Shared generation to detect mutations of the underlying token.shadow file.
shared_gen: usize,
+ // shadow file mtime to detect changes
+ file_mtime: Option<SystemTime>,
+ // shadow file length to detect changes
+ file_len: Option<u64>,
+ // last time the file metadata was checked
+ last_checked: Option<i64>,
}
/// Cached secret.
@@ -187,7 +263,13 @@ fn cache_try_secret_matches(tokenid: &Authid, secret: &str) -> bool {
eq && gen2 == cache_gen
}
-fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
+fn apply_api_mutation(
+ tokenid: &Authid,
+ new_secret: Option<&str>,
+ pre_write_meta: (Option<SystemTime>, Option<u64>),
+) {
+ let now = epoch_i64();
+
// Signal cache invalidation to other processes (best-effort).
let new_shared_gen = bump_token_shadow_shared_gen();
@@ -203,6 +285,13 @@ fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
// Update to the post-mutation generation.
cache.shared_gen = gen;
+ // If our cached file metadata does not match the on-disk state before our write,
+ // we likely missed an external/manual edit. We can no longer trust any cached secrets.
+ let (pre_mtime, pre_len) = pre_write_meta;
+ if cache.file_mtime != pre_mtime || cache.file_len != pre_len {
+ cache.secrets.clear();
+ }
+
// Apply the new mutation.
match new_secret {
Some(secret) => {
@@ -217,6 +306,20 @@ fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
cache.secrets.remove(tokenid);
}
}
+
+ // Update our view of the file metadata to the post-write state (best-effort).
+ // (If this fails, drop local cache so callers fall back to slow path until refreshed.)
+ match shadow_mtime_len() {
+ Ok((mtime, len)) => {
+ cache.file_mtime = mtime;
+ cache.file_len = len;
+ cache.last_checked = Some(now);
+ }
+ Err(_) => {
+ // If we cannot validate state, do not trust cache.
+ invalidate_cache_state(&mut cache);
+ }
+ }
}
/// Get the current shared generation.
@@ -226,10 +329,24 @@ fn token_shadow_shared_gen() -> Option<usize> {
/// Bump and return the new shared generation.
fn bump_token_shadow_shared_gen() -> Option<usize> {
- access_conf().increment_token_shadow_cache_generation().ok().map(|prev| prev + 1)
+ access_conf()
+ .increment_token_shadow_cache_generation()
+ .ok()
+ .map(|prev| prev + 1)
}
/// Invalidates the cache state and only keeps the shared generation.
fn invalidate_cache_state(cache: &mut ApiTokenSecretCache) {
cache.secrets.clear();
-}
\ No newline at end of file
+ cache.file_mtime = None;
+ cache.file_len = None;
+ cache.last_checked = None;
+}
+
+fn shadow_mtime_len() -> Result<(Option<SystemTime>, Option<u64>), Error> {
+ match fs::metadata(token_shadow()) {
+ Ok(meta) => Ok((meta.modified().ok(), Some(meta.len()))),
+ Err(e) if e.kind() == ErrorKind::NotFound => Ok((None, None)),
+ Err(e) => Err(e.into()),
+ }
+}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] superseded: [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token verification overhead
2025-12-17 16:25 14% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token " Samuel Rufinatscha
` (7 preceding siblings ...)
2025-12-18 11:03 12% ` [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token verification overhead Samuel Rufinatscha
@ 2026-01-02 16:09 13% ` Samuel Rufinatscha
8 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-02 16:09 UTC (permalink / raw)
To: pbs-devel
https://lore.proxmox.com/pbs-devel/20260102160750.285157-1-s.rufinatscha@proxmox.com/T/#t
On 12/17/25 5:25 PM, Samuel Rufinatscha wrote:
> Hi,
>
> this series improves the performance of token-based API authentication
> in PBS (pbs-config) and in PDM (underlying proxmox-access-control
> crate), addressing the API token verification hotspot reported in our
> bugtracker #6049 [1].
>
> When profiling PBS /status endpoint with cargo flamegraph [2],
> token-based authentication showed up as a dominant hotspot via
> proxmox_sys::crypt::verify_crypt_pw. Applying this series removes that
> path from the hot section of the flamegraph. The same performance issue
> was measured [2] for PDM. PDM uses the underlying shared
> proxmox-access-control library for token handling, which is a
> factored out version of the token.shadow handling code from PBS.
>
> While this series fixes the immediate performance issue both in PBS
> (pbs-config) and in the shared proxmox-access-control crate used by
> PDM, PBS should eventually, ideally be refactored, in a separate
> effort, to use proxmox-access-control for token handling instead of its
> local implementation.
>
> Problem
>
> For token-based API requests, both PBS’s pbs-config token.shadow
> handling and PDM proxmox-access-control’s token.shadow handling
> currently:
>
> 1. read the token.shadow file on each request
> 2. deserialize it into a HashMap<Authid, String>
> 3. run password hash verification via
> proxmox_sys::crypt::verify_crypt_pw for the provided token secret
>
> Under load, this results in significant CPU usage spent in repeated
> password hash computations for the same token+secret pairs. The
> attached flamegraphs for PBS [2] and PDM [3] show
> proxmox_sys::crypt::verify_crypt_pw dominating the hot path.
>
> Approach
>
> The goal is to reduce the cost of token-based authentication preserving
> the existing token handling semantics (including detecting manual edits
> to token.shadow) and be consistent between PBS (pbs-config) and
> PDM (proxmox-access-control). For both sites, the series proposes
> following approach:
>
> 1. Introduce an in-memory cache for verified token secrets
> 2. Invalidate the cache when token.shadow changes (detect manual edits)
> 3. Control metadata checks with a TTL window
>
> Testing
>
> *PBS (pbs-config)*
>
> To verify the effect in PBS, I:
> 1. Set up test environment based on latest PBS ISO, installed Rust
> toolchain, cloned proxmox-backup repository to use with cargo
> flamegraph. Reproduced bug #6049 [1] by profiling the /status
> endpoint with token-based authentication using cargo flamegraph [2].
> The flamegraph showed proxmox_sys::crypt::verify_crypt_pw is the
> hotspot.
> 2. Built PBS with pbs-config patches and re-ran the same workload and
> profiling setup.
> 3. Confirmed that the proxmox_sys::crypt::verify_crypt_pw path no
> longer appears in the hot section of the flamegraph. CPU usage is
> now dominated by TLS overhead.
> 4. Functionally verified that:
> * token-based API authentication still works for valid tokens
> * invalid secrets are rejected as before
> * generating a new token secret via dashboard works and
> authenticates correctly
>
> *PDM (proxmox-access-control)*
>
> To verify the effect in PDM, I followed a similar testing approach.
> Instead of /status, I profiled the /version endpoint with cargo
> flamegraph [2] and verified that the token hashing path disappears [4]
> from the hot section after applying the proxmox-access-control patches.
>
> Functionally I verified that:
> * token-based API authentication still works for valid tokens
> * invalid secrets are rejected as before
> * generating a new token secret via dashboard works and
> authenticates correctly
>
> Benchmarks:
>
> Two different benchmarks have been run to measure caching effects
> and RwLock contention:
>
> (1) Requests per second for PBS /status endpoint (E2E)
> (2) RwLock contention for token create/delete under
> heavy parallel token-authenticated readers; compared
> std::sync::RwLock and parking_lot::RwLock.
>
> (1) benchmarked parallel token auth requests for
> /status?verbose=0 on top of the datastore lookup cache series [5]
> to check throughput impact. With datastores=1, repeat=5000, parallel=16
> this series gives ~179 req/s compared to ~65 req/s without it.
> This is a ~2.75x improvement.
>
> (2) benchmarked token create/delete operations under heavy load of
> token-authenticated requests on top of the datastore lookup cache [5]
> series. This benchmark was done using against a 64-parallel
> token-auth flood (200k requests) against
> /admin/datastore/ds0001/status?verbose=0 while executing 50 token
> create + 50 token delete operations. After the series I saw the
> following e2e API latencies:
>
> parking_lot::RwLock
> - create avg ~27ms (p95 ~28ms) vs ~46ms (p95 ~50ms) baseline
> - delete avg ~17ms (p95 ~19ms) vs ~33ms (p95 ~35ms) baseline
>
> std::sync::RwLock
> - create avg ~27ms (p95 ~28ms)
> - create avg ~17ms (p95 ~19ms)
>
> It appears that the both RwLock implementations perform similarly
> for this workload. The parking_lot version has been chosen for the
> added fairness guarantees.
>
> Patch summary
>
> pbs-config:
>
> 0001 – pbs-config: cache verified API token secrets
> Adds an in-memory cache keyed by Authid that stores plain text token
> secrets after a successful verification or generation and uses
> openssl’s memcmp constant-time for comparison.
>
> 0002 – pbs-config: invalidate token-secret cache on token.shadow
> changes
> Tracks token.shadow mtime and length and clears the in-memory
> cache when the file changes.
>
> 0003 – pbs-config: add TTL window to token-secret cache
> Introduces a TTL (TOKEN_SECRET_CACHE_TTL_SECS, default 60) for metadata
> checks so that fs::metadata is only called periodically.
>
> proxmox-access-control:
>
> 0004 – access-control: cache verified API token secrets
> Mirrors PBS PATCH 0001.
>
> 0005 – access-control: invalidate token-secret cache on token.shadow changes
> Mirrors PBS PATCH 0002.
>
> 0006 – access-control: add TTL window to token-secret cache
> Mirrors PBS PATCH 0003.
>
> proxmox-datacenter-manager:
>
> 0007 – docs: document API token-cache TTL effects
> Documents the effects of the TTL window on token.shadow edits
>
> Changes since v1
>
> - (refactor) Switched cache initialization to LazyLock
> - (perf) Use parking_lot::RwLock and best-effort cache access on the
> read/refresh path (try_read/try_write) to avoid lock contention
> - (doc) Document TTL-delayed effect of manual token.shadow edits
> - (fix) Add generation guards (API_MUTATION_GENERATION +
> FILE_GENERATION) to prevent caching across concurrent set/delete and
> external edits
>
> Please see the patch specific changelogs for more details.
>
> Thanks for considering this patch series, I look forward to your
> feedback.
>
> Best,
> Samuel Rufinatscha
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
> [2] attachment 1767 [1]: Flamegraph showing the proxmox_sys::crypt::verify_crypt_pw stack
> [3] attachment 1794 [1]: Flamegraph PDM baseline
> [4] attachment 1795 [1]: Flamegraph PDM patched
> [5] https://bugzilla.proxmox.com/show_bug.cgi?id=6049
>
> proxmox-backup:
>
> Samuel Rufinatscha (3):
> pbs-config: cache verified API token secrets
> pbs-config: invalidate token-secret cache on token.shadow changes
> pbs-config: add TTL window to token secret cache
>
> Cargo.toml | 1 +
> docs/user-management.rst | 4 +
> pbs-config/Cargo.toml | 1 +
> pbs-config/src/token_shadow.rs | 238 ++++++++++++++++++++++++++++++++-
> 4 files changed, 243 insertions(+), 1 deletion(-)
>
>
> proxmox:
>
> Samuel Rufinatscha (3):
> proxmox-access-control: cache verified API token secrets
> proxmox-access-control: invalidate token-secret cache on token.shadow
> changes
> proxmox-access-control: add TTL window to token secret cache
>
> Cargo.toml | 1 +
> proxmox-access-control/Cargo.toml | 1 +
> proxmox-access-control/src/token_shadow.rs | 238 ++++++++++++++++++++-
> 3 files changed, 239 insertions(+), 1 deletion(-)
>
>
> proxmox-datacenter-manager:
>
> Samuel Rufinatscha (1):
> docs: document API token-cache TTL effects
>
> docs/access-control.rst | 3 +++
> 1 file changed, 3 insertions(+)
>
>
> Summary over all repositories:
> 8 files changed, 485 insertions(+), 2 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox-backup v6 3/4] partial fix #6049: datastore: use config fast-path in Drop
2026-01-05 14:16 12% [pbs-devel] [PATCH proxmox-backup v6 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
2026-01-05 14:16 16% ` [pbs-devel] [PATCH proxmox-backup v6 1/4] config: enable config version cache for datastore Samuel Rufinatscha
2026-01-05 14:16 11% ` [pbs-devel] [PATCH proxmox-backup v6 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
@ 2026-01-05 14:16 15% ` Samuel Rufinatscha
2026-01-05 14:16 13% ` [pbs-devel] [PATCH proxmox-backup v6 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
2026-01-14 9:54 5% ` [pbs-devel] applied-series: [PATCH proxmox-backup v6 0/4] datastore: remove config reload on hot path Fabian Grünbichler
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-05 14:16 UTC (permalink / raw)
To: pbs-devel
The Drop impl of DataStore re-read datastore.cfg to decide whether
the entry should be evicted from the in-process cache (based on
maintenance mode’s clear_from_cache). During the investigation of
issue #6049 [1], a flamegraph [2] showed that the config reload in Drop
accounted for a measurable share of CPU time under load.
This patch wires the datastore config fast path to the Drop
impl to eventually avoid an expensive config reload from disk to capture
the maintenance mandate.
Behavioral notes
- Drop no longer silently ignores config/lookup failures: failures to
load/parse datastore.cfg are logged at WARN level
- If the datastore no longer exists in datastore.cfg when the last
handle is dropped, the cached instance is evicted from DATASTORE_MAP
if available (without checking maintenance mode).
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes:
From v1 → v2
- Replace caching logic with the datastore_section_config_cached()
helper.
From v2 → v3
No changes
From v3 → v4, thanks @Fabian
- Pass datastore_section_config_cached(false) in Drop to avoid
concurrent cache updates.
From v4 → v5
- Rebased only, no changes
From v5 → v6
- Rebased
- Styling: restructured cache eviction condition
- Drop impl: log cache-related failures to load/parse datastore.cfg at
WARN level instead of ERROR
- Note logging change in the patch message, thanks @Fabian
- Remove cached entry from DATASTORE_MAP (if available) if datastore no
longer exists in datastore.cfg when the last handle is dropped,
thanks @Fabian
- Removed slow-path generation bumping in
datastore_section_config_cached, since API changes already
bump the generation on config save. Moved to subsequent patch,
relevant for TTL-based mechanism to bump on non-API edits, thanks @Fabian
pbs-datastore/src/datastore.rs | 35 ++++++++++++++++++++++++++--------
1 file changed, 27 insertions(+), 8 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index aa366826..8adb0e3b 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -224,14 +224,33 @@ impl Drop for DataStore {
// remove datastore from cache iff
// - last task finished, and
- // - datastore is in a maintenance mode that mandates it
- let remove_from_cache = last_task
- && pbs_config::datastore::config()
- .and_then(|(s, _)| s.lookup::<DataStoreConfig>("datastore", self.name()))
- .is_ok_and(|c| {
- c.get_maintenance_mode()
- .is_some_and(|m| m.clear_from_cache())
- });
+ // - datastore is in a maintenance mode that mandates it, or the datastore was removed from datastore.cfg
+
+ // first check: check if last task finished
+ if !last_task {
+ return;
+ }
+
+ // determine whether we should evict from DATASTORE_MAP.
+ let remove_from_cache = match datastore_section_config_cached(false) {
+ Ok((section_config, _gen)) => {
+ match section_config.lookup::<DataStoreConfig>("datastore", self.name()) {
+ // second check: check if maintenance mode requires closing FDs
+ Ok(config) => config
+ .get_maintenance_mode()
+ .is_some_and(|m| m.clear_from_cache()),
+ Err(err) => {
+ // datastore removed from config; evict cached entry if available (without checking maintenance mode)
+ log::warn!("DataStore::drop: datastore '{}' missing from datastore.cfg; evicting cached instance: {err}", self.name());
+ true
+ }
+ }
+ }
+ Err(err) => {
+ log::warn!("DataStore::drop: failed to load datastore.cfg for '{}'; skipping cache-eviction: {err}", self.name());
+ false
+ }
+ };
if remove_from_cache {
DATASTORE_MAP.lock().unwrap().remove(self.name());
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox-backup v6 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
2026-01-05 14:16 12% [pbs-devel] [PATCH proxmox-backup v6 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
2026-01-05 14:16 16% ` [pbs-devel] [PATCH proxmox-backup v6 1/4] config: enable config version cache for datastore Samuel Rufinatscha
@ 2026-01-05 14:16 11% ` Samuel Rufinatscha
2026-01-05 14:16 15% ` [pbs-devel] [PATCH proxmox-backup v6 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
` (2 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-05 14:16 UTC (permalink / raw)
To: pbs-devel
Repeated /status requests caused lookup_datastore() to re-read and
parse datastore.cfg on every call. The issue was mentioned in report
#6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
dominated by pbs_config::datastore::config() (config parsing).
This patch implements caching of the global datastore.cfg using the
generation numbers from the shared config version cache. It caches the
datastore.cfg along with the generation number and, when a subsequent
lookup sees the same generation, it reuses the cached config without
re-reading it from disk. If the generation differs
(or the cache is unavailable), the config is re-read from disk.
If `update_cache = true`, the new config and current generation are
persisted in the cache. In this case, callers must hold the datastore
config lock to avoid racing with concurrent config changes.
If `update_cache` is `false` and generation did not match, the freshly
read config is returned but the cache is left unchanged. If
`ConfigVersionCache` is not available, the config is always read from
disk and `None` is returned as generation.
Behavioral notes
- The generation is bumped via the existing save_config() path, so
API-driven config changes are detected immediately.
- Manual edits to datastore.cfg are not detected; this is covered in a
dedicated patch in this series.
- DataStore::drop still performs a config read on the common path;
also covered in a dedicated patch in this series.
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes:
From v1 → v2
- Moved the ConfigVersionCache changes into its own patch,
thanks @Fabian
- Introduced the global static DATASTORE_CONFIG_CACHE to store the
fully parsed datastore.cfg instead, along with its generation number.
thanks @Fabian
- Introduced DatastoreConfigCache struct to hold cache values
- Removed and replaced the CachedDatastoreConfigTag field of
DataStoreImpl with a generation number field only (Option<usize>)
to validate DataStoreImpl reuse.
- Added DataStore::datastore_section_config_cached() helper function
to encapsulate the caching logic and simplify reuse.
- Modified DataStore::lookup_datastore() to use the new helper.
From v2 → v3
No changes
From v3 → v4, thanks @Fabian
- Restructured the version cache checks in
datastore_section_config_cached(), to simplify the logic.
- Added update_cache parameter to datastore_section_config_cached() to
control cache updates.
From v4 → v5
- Rebased only, no changes
From v5 → v6
- Rebased
- Styling: minimize/avoid diff noise, thanks @Fabian
pbs-datastore/Cargo.toml | 1 +
pbs-datastore/src/datastore.rs | 90 ++++++++++++++++++++++++++++------
2 files changed, 77 insertions(+), 14 deletions(-)
diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index 8ce930a9..42f49a7b 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -40,6 +40,7 @@ proxmox-io.workspace = true
proxmox-lang.workspace=true
proxmox-s3-client = { workspace = true, features = [ "impl" ] }
proxmox-schema = { workspace = true, features = [ "api-macro" ] }
+proxmox-section-config.workspace = true
proxmox-serde = { workspace = true, features = [ "serde_json" ] }
proxmox-sys.workspace = true
proxmox-systemd.workspace = true
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 9c57aaac..aa366826 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -34,7 +34,8 @@ use pbs_api_types::{
MaintenanceType, Operation, UPID,
};
use pbs_config::s3::S3_CFG_TYPE_ID;
-use pbs_config::BackupLockGuard;
+use pbs_config::{BackupLockGuard, ConfigVersionCache};
+use proxmox_section_config::SectionConfigData;
use crate::backup_info::{
BackupDir, BackupGroup, BackupInfo, OLD_LOCKING, PROTECTED_MARKER_FILENAME,
@@ -48,6 +49,17 @@ use crate::s3::S3_CONTENT_PREFIX;
use crate::task_tracking::{self, update_active_operations};
use crate::{DataBlob, LocalDatastoreLruCache};
+// Cache for fully parsed datastore.cfg
+struct DatastoreConfigCache {
+ // Parsed datastore.cfg file
+ config: Arc<SectionConfigData>,
+ // Generation number from ConfigVersionCache
+ last_generation: usize,
+}
+
+static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
+ LazyLock::new(|| Mutex::new(None));
+
static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
@@ -149,11 +161,13 @@ pub struct DataStoreImpl {
last_gc_status: Mutex<GarbageCollectionStatus>,
verify_new: bool,
chunk_order: ChunkOrder,
- last_digest: Option<[u8; 32]>,
sync_level: DatastoreFSyncLevel,
backend_config: DatastoreBackendConfig,
lru_store_caching: Option<LocalDatastoreLruCache>,
thread_settings: DatastoreThreadSettings,
+ /// datastore.cfg cache generation number at lookup time, used to
+ /// invalidate this cached `DataStoreImpl`
+ config_generation: Option<usize>,
}
impl DataStoreImpl {
@@ -166,11 +180,11 @@ impl DataStoreImpl {
last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
verify_new: false,
chunk_order: Default::default(),
- last_digest: None,
sync_level: Default::default(),
backend_config: Default::default(),
lru_store_caching: None,
thread_settings: Default::default(),
+ config_generation: None,
})
}
}
@@ -286,6 +300,55 @@ impl DatastoreThreadSettings {
}
}
+/// Returns the parsed datastore config (`datastore.cfg`) and its
+/// generation.
+///
+/// Uses `ConfigVersionCache` to detect stale entries:
+/// - If the cached generation matches the current generation, the
+/// cached config is returned.
+/// - Otherwise the config is re-read from disk. If `update_cache` is
+/// `true`, the new config and current generation are stored in the
+/// cache. Callers that set `update_cache = true` must hold the
+/// datastore config lock to avoid racing with concurrent config
+/// changes.
+/// - If `update_cache` is `false`, the freshly read config is returned
+/// but the cache is left unchanged.
+///
+/// If `ConfigVersionCache` is not available, the config is always read
+/// from disk and `None` is returned as the generation.
+fn datastore_section_config_cached(
+ update_cache: bool,
+) -> Result<(Arc<SectionConfigData>, Option<usize>), Error> {
+ let mut config_cache = DATASTORE_CONFIG_CACHE.lock().unwrap();
+
+ if let Ok(version_cache) = ConfigVersionCache::new() {
+ let current_gen = version_cache.datastore_generation();
+ if let Some(cached) = config_cache.as_ref() {
+ // Fast path: re-use cached datastore.cfg
+ if cached.last_generation == current_gen {
+ return Ok((cached.config.clone(), Some(cached.last_generation)));
+ }
+ }
+ // Slow path: re-read datastore.cfg
+ let (config_raw, _digest) = pbs_config::datastore::config()?;
+ let config = Arc::new(config_raw);
+
+ if update_cache {
+ *config_cache = Some(DatastoreConfigCache {
+ config: config.clone(),
+ last_generation: current_gen,
+ });
+ }
+
+ Ok((config, Some(current_gen)))
+ } else {
+ // Fallback path, no config version cache: read datastore.cfg and return None as generation
+ *config_cache = None;
+ let (config_raw, _digest) = pbs_config::datastore::config()?;
+ Ok((Arc::new(config_raw), None))
+ }
+}
+
impl DataStore {
// This one just panics on everything
#[doc(hidden)]
@@ -367,10 +430,9 @@ impl DataStore {
// we use it to decide whether it is okay to delete the datastore.
let _config_lock = pbs_config::datastore::lock_config()?;
- // we could use the ConfigVersionCache's generation for staleness detection, but we load
- // the config anyway -> just use digest, additional benefit: manual changes get detected
- let (config, digest) = pbs_config::datastore::config()?;
- let config: DataStoreConfig = config.lookup("datastore", name)?;
+ // Get the current datastore.cfg generation number and cached config
+ let (section_config, gen_num) = datastore_section_config_cached(true)?;
+ let config: DataStoreConfig = section_config.lookup("datastore", name)?;
if let Some(maintenance_mode) = config.get_maintenance_mode() {
if let Err(error) = maintenance_mode.check(operation) {
@@ -378,19 +440,19 @@ impl DataStore {
}
}
+ let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
+
if get_datastore_mount_status(&config) == Some(false) {
- let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
datastore_cache.remove(&config.name);
bail!("datastore '{}' is not mounted", config.name);
}
- let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
let entry = datastore_cache.get(name);
// reuse chunk store so that we keep using the same process locker instance!
let chunk_store = if let Some(datastore) = &entry {
- let last_digest = datastore.last_digest.as_ref();
- if let Some(true) = last_digest.map(|last_digest| last_digest == &digest) {
+ // Re-use DataStoreImpl
+ if datastore.config_generation == gen_num && gen_num.is_some() {
if let Some(operation) = operation {
update_active_operations(name, operation, 1)?;
}
@@ -412,7 +474,7 @@ impl DataStore {
)?)
};
- let datastore = DataStore::with_store_and_config(chunk_store, config, Some(digest))?;
+ let datastore = DataStore::with_store_and_config(chunk_store, config, gen_num)?;
let datastore = Arc::new(datastore);
datastore_cache.insert(name.to_string(), datastore.clone());
@@ -514,7 +576,7 @@ impl DataStore {
fn with_store_and_config(
chunk_store: Arc<ChunkStore>,
config: DataStoreConfig,
- last_digest: Option<[u8; 32]>,
+ generation: Option<usize>,
) -> Result<DataStoreImpl, Error> {
let mut gc_status_path = chunk_store.base_path();
gc_status_path.push(".gc-status");
@@ -579,11 +641,11 @@ impl DataStore {
last_gc_status: Mutex::new(gc_status),
verify_new: config.verify_new.unwrap_or(false),
chunk_order: tuning.chunk_order.unwrap_or_default(),
- last_digest,
sync_level: tuning.sync_level.unwrap_or_default(),
backend_config,
lru_store_caching,
thread_settings,
+ config_generation: generation,
})
}
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 11%]
* [pbs-devel] [PATCH proxmox-backup v6 1/4] config: enable config version cache for datastore
2026-01-05 14:16 12% [pbs-devel] [PATCH proxmox-backup v6 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
@ 2026-01-05 14:16 16% ` Samuel Rufinatscha
2026-01-05 14:16 11% ` [pbs-devel] [PATCH proxmox-backup v6 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
` (3 subsequent siblings)
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-05 14:16 UTC (permalink / raw)
To: pbs-devel
Repeated /status requests caused lookup_datastore() to re-read and
parse datastore.cfg on every call. The issue was mentioned in report
#6049 [1]. cargo-flamegraph [2] confirmed that the hot path is
dominated by pbs_config::datastore::config() (config parsing).
To solve the issue, this patch prepares the config version cache,
so that datastore config caching can be built on top of it.
This patch specifically:
(1) implements increment function in order to invalidate generations
(2) removes obsolete comments
Links
[1] Bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes:
From v1 → v2 (original introduction), thanks @Fabian
- Split the ConfigVersionCache changes out of the large datastore patch
into their own config-only patch
From v2 → v3
No changes
From v3 → v4
No changes
From v4 → v5
- Rebased only, no changes
From v5 → v6
- Rebased
- Removed "partial-fix" prefix from subject, thanks @Fabian
pbs-config/src/config_version_cache.rs | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/pbs-config/src/config_version_cache.rs b/pbs-config/src/config_version_cache.rs
index e8fb994f..b875f7e0 100644
--- a/pbs-config/src/config_version_cache.rs
+++ b/pbs-config/src/config_version_cache.rs
@@ -26,7 +26,6 @@ struct ConfigVersionCacheDataInner {
// Traffic control (traffic-control.cfg) generation/version.
traffic_control_generation: AtomicUsize,
// datastore (datastore.cfg) generation/version
- // FIXME: remove with PBS 3.0
datastore_generation: AtomicUsize,
// Add further atomics here
}
@@ -145,8 +144,15 @@ impl ConfigVersionCache {
.fetch_add(1, Ordering::AcqRel);
}
+ /// Returns the datastore generation number.
+ pub fn datastore_generation(&self) -> usize {
+ self.shmem
+ .data()
+ .datastore_generation
+ .load(Ordering::Acquire)
+ }
+
/// Increase the datastore generation number.
- // FIXME: remove with PBS 3.0 or make actually useful again in datastore lookup
pub fn increase_datastore_generation(&self) -> usize {
self.shmem
.data()
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 16%]
* [pbs-devel] [PATCH proxmox-backup v6 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
2026-01-05 14:16 12% [pbs-devel] [PATCH proxmox-backup v6 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
` (2 preceding siblings ...)
2026-01-05 14:16 15% ` [pbs-devel] [PATCH proxmox-backup v6 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
@ 2026-01-05 14:16 13% ` Samuel Rufinatscha
2026-01-14 9:54 5% ` [pbs-devel] applied-series: [PATCH proxmox-backup v6 0/4] datastore: remove config reload on hot path Fabian Grünbichler
4 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-05 14:16 UTC (permalink / raw)
To: pbs-devel
The lookup fast path reacts to API-driven config changes because
save_config() bumps the generation. Manual edits of datastore.cfg do
not bump the counter. To keep the system robust against such edits
without reintroducing config reading and hashing on the hot path, this
patch adds a TTL to the cache entry.
If the cached config is older than
DATASTORE_CONFIG_CACHE_TTL_SECS (set to 60s), the next lookup takes
the slow path and refreshes the entry. As an optimization, a check to
catch manual edits was added (if the digest changed but generation
stayed the same). If a manual edit was detected, the generation will be
bumped.
Links
[1] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
Fixes: #6049
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Changes:
From v1 → v2
- Store last_update timestamp in DatastoreConfigCache type.
From v2 → v3
No changes
From v3 → v4
- Fix digest generation bump logic in update_cache, thanks @Fabian.
From v4 → v5
- Rebased only, no changes
From v5 → v6
- Rebased
- Styling: simplified digest-matching, thanks @Fabian
pbs-datastore/src/datastore.rs | 47 +++++++++++++++++++++++++---------
1 file changed, 35 insertions(+), 12 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 8adb0e3b..c4be55ad 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -53,8 +53,12 @@ use crate::{DataBlob, LocalDatastoreLruCache};
struct DatastoreConfigCache {
// Parsed datastore.cfg file
config: Arc<SectionConfigData>,
+ // Digest of the datastore.cfg file
+ digest: [u8; 32],
// Generation number from ConfigVersionCache
last_generation: usize,
+ // Last update time (epoch seconds)
+ last_update: i64,
}
static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
@@ -63,6 +67,8 @@ static DATASTORE_CONFIG_CACHE: LazyLock<Mutex<Option<DatastoreConfigCache>>> =
static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
+/// Max age in seconds to reuse the cached datastore config.
+const DATASTORE_CONFIG_CACHE_TTL_SECS: i64 = 60;
/// Filename to store backup group notes
pub const GROUP_NOTES_FILE_NAME: &str = "notes";
/// Filename to store backup group owner
@@ -323,15 +329,16 @@ impl DatastoreThreadSettings {
/// generation.
///
/// Uses `ConfigVersionCache` to detect stale entries:
-/// - If the cached generation matches the current generation, the
-/// cached config is returned.
+/// - If the cached generation matches the current generation and TTL is
+/// OK, the cached config is returned.
/// - Otherwise the config is re-read from disk. If `update_cache` is
-/// `true`, the new config and current generation are stored in the
-/// cache. Callers that set `update_cache = true` must hold the
-/// datastore config lock to avoid racing with concurrent config
-/// changes.
+/// `true` and a previous cached entry exists with the same generation
+/// but a different digest, this indicates the config has changed
+/// (e.g. manual edit) and the generation must be bumped. Callers
+/// that set `update_cache = true` must hold the datastore config lock
+/// to avoid racing with concurrent config changes.
/// - If `update_cache` is `false`, the freshly read config is returned
-/// but the cache is left unchanged.
+/// but the cache and generation are left unchanged.
///
/// If `ConfigVersionCache` is not available, the config is always read
/// from disk and `None` is returned as the generation.
@@ -341,25 +348,41 @@ fn datastore_section_config_cached(
let mut config_cache = DATASTORE_CONFIG_CACHE.lock().unwrap();
if let Ok(version_cache) = ConfigVersionCache::new() {
+ let now = epoch_i64();
let current_gen = version_cache.datastore_generation();
if let Some(cached) = config_cache.as_ref() {
- // Fast path: re-use cached datastore.cfg
- if cached.last_generation == current_gen {
+ // Fast path: re-use cached datastore.cfg if generation matches and TTL not expired
+ if cached.last_generation == current_gen
+ && now - cached.last_update < DATASTORE_CONFIG_CACHE_TTL_SECS
+ {
return Ok((cached.config.clone(), Some(cached.last_generation)));
}
}
// Slow path: re-read datastore.cfg
- let (config_raw, _digest) = pbs_config::datastore::config()?;
+ let (config_raw, digest) = pbs_config::datastore::config()?;
let config = Arc::new(config_raw);
+ let mut effective_gen = current_gen;
if update_cache {
+ // Bump the generation if the config has been changed manually.
+ // This ensures that Drop handlers will detect that a newer config exists
+ // and will not rely on a stale cached entry for maintenance mandate.
+ if let Some(cached) = config_cache.as_ref() {
+ if cached.last_generation == current_gen && cached.digest != digest {
+ effective_gen = version_cache.increase_datastore_generation() + 1;
+ }
+ }
+
+ // Persist
*config_cache = Some(DatastoreConfigCache {
config: config.clone(),
- last_generation: current_gen,
+ digest,
+ last_generation: effective_gen,
+ last_update: now,
});
}
- Ok((config, Some(current_gen)))
+ Ok((config, Some(effective_gen)))
} else {
// Fallback path, no config version cache: read datastore.cfg and return None as generation
*config_cache = None;
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox-backup v6 0/4] datastore: remove config reload on hot path
@ 2026-01-05 14:16 12% Samuel Rufinatscha
2026-01-05 14:16 16% ` [pbs-devel] [PATCH proxmox-backup v6 1/4] config: enable config version cache for datastore Samuel Rufinatscha
` (4 more replies)
0 siblings, 5 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-05 14:16 UTC (permalink / raw)
To: pbs-devel
Hi,
this series reduces CPU time in datastore lookups by avoiding repeated
datastore.cfg reads/parses in both `lookup_datastore()` and
`DataStore::Drop`. It also adds a TTL so manual config edits are
noticed without reintroducing hashing on every request.
While investigating #6049 [1], cargo-flamegraph [2] showed hotspots
during repeated `/status` calls in `lookup_datastore()` and in `Drop`,
dominated by `pbs_config::datastore::config()` (config parse).
The parsing cost itself should eventually be investigated in a future
effort. Furthermore, cargo-flamegraph showed that when using a
token-based auth method to access the API, a significant amount of time
is spent in validation on every request request [3].
## Approach
[PATCH 1/4] Support datastore generation in ConfigVersionCache
[PATCH 2/4] Fast path for datastore lookups
Cache the parsed datastore.cfg keyed by the shared datastore
generation. lookup_datastore() reuses both the cached config and an
existing DataStoreImpl when the generation matches, and falls back
to the old slow path otherwise. The caching logic is implemented
using the datastore_section_config_cached(update_cache: bool) helper.
[PATCH 3/4] Fast path for Drop
Make DataStore::Drop use the datastore_section_config_cached()
helper to avoid re-reading/parsing datastore.cfg on every Drop.
[PATCH 4/4] TTL to catch manual edits
Add a TTL to the cached config and bump the datastore generation iff
the digest changed but generation stays the same. This catches manual
edits to datastore.cfg without reintroducing hashing or config
parsing on every request.
## Benchmark results
### End-to-end
Testing `/status?verbose=0` end-to-end with 1000 stores, 5 req/store
and parallel=16 before/after the series:
Metric Before After
----------------------------------------
Total time 12s 9s
Throughput (all) 416.67 555.56
Cold RPS (round #1) 83.33 111.11
Warm RPS (#2..N) 333.33 444.44
Running under flamegraph [2], TLS appears to consume a significant
amount of CPU time and blur the results. Still, a ~33% higher overall
throughput and ~25% less end-to-end time for this workload.
### Isolated benchmarks (hyperfine)
In addition to the end-to-end tests, I measured two standalone
benchmarks with hyperfine, each using a config with 1000 datastores.
`M` is the number of distinct datastores looked up and
`N` is the number of lookups per datastore.
Drop-direct variant:
Drops the `DataStore` after every lookup, so the `Drop` path runs on
every iteration:
use anyhow::Error;
use pbs_api_types::Operation;
use pbs_datastore::DataStore;
fn main() -> Result<(), Error> {
let mut args = std::env::args();
args.next();
let datastores = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
let iterations = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
for d in 1..=datastores {
let name = format!("ds{:04}", d);
for i in 1..=iterations {
DataStore::lookup_datastore(&name, Some(Operation::Write))?;
}
}
Ok(())
}
+----+------+-----------+-----------+---------+
| M | N | Baseline | Patched | Speedup |
+----+------+-----------+-----------+---------+
| 1 | 1000 | 1.684 s | 35.3 ms | 47.7x |
| 10 | 100 | 1.689 s | 35.0 ms | 48.3x |
| 100| 10 | 1.709 s | 35.8 ms | 47.7x |
|1000| 1 | 1.809 s | 39.0 ms | 46.4x |
+----+------+-----------+-----------+---------+
Bulk-drop variant:
Keeps the `DataStore` instances alive for
all `N` lookups of a given datastore and then drops them in bulk,
mimicking a task that performs many lookups while it is running and
only triggers the expensive `Drop` logic when the last user exits.
use anyhow::Error;
use pbs_api_types::Operation;
use pbs_datastore::DataStore;
fn main() -> Result<(), Error> {
let mut args = std::env::args();
args.next();
let datastores = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
let iterations = if let Some(n) = args.next() {
n.parse::<usize>()?
} else {
1000
};
for d in 1..=datastores {
let name = format!("ds{:04}", d);
let mut stores = Vec::with_capacity(iterations);
for i in 1..=iterations {
stores.push(DataStore::lookup_datastore(&name, Some(Operation::Write))?);
}
}
Ok(())
}
+------+------+---------------+--------------+---------+
| M | N | Baseline mean | Patched mean | Speedup |
+------+------+---------------+--------------+---------+
| 1 | 1000 | 890.6 ms | 35.5 ms | 25.1x |
| 10 | 100 | 891.3 ms | 35.1 ms | 25.4x |
| 100 | 10 | 983.9 ms | 35.6 ms | 27.6x |
| 1000 | 1 | 1829.0 ms | 45.2 ms | 40.5x |
+------+------+---------------+--------------+---------+
Both variants show that the combination of the cached config lookups
and the cheaper `Drop` handling reduces the hot-path cost from ~1.8 s
per run to a few tens of milliseconds in these benchmarks.
## Reproduction steps
VM: 4 vCPU, ~8 GiB RAM, VirtIO-SCSI; disks:
- scsi0 32G (OS)
- scsi1 1000G (datastores)
Install PBS from ISO on the VM.
Set up ZFS on /dev/sdb (adjust if different):
zpool create -f -o ashift=12 pbsbench /dev/sdb
zfs set mountpoint=/pbsbench pbsbench
zfs create pbsbench/pbs-bench
Raise file-descriptor limit:
sudo systemctl edit proxmox-backup-proxy.service
Add the following lines:
[Service]
LimitNOFILE=1048576
Reload systemd and restart the proxy:
sudo systemctl daemon-reload
sudo systemctl restart proxmox-backup-proxy.service
Verify the limit:
systemctl show proxmox-backup-proxy.service | grep LimitNOFILE
Create 1000 ZFS-backed datastores (as used in #6049 [1]):
seq -w 001 1000 | xargs -n1 -P1 bash -c '
id=$0
name="ds${id}"
dataset="pbsbench/pbs-bench/${name}"
path="/pbsbench/pbs-bench/${name}"
zfs create -o mountpoint="$path" "$dataset"
proxmox-backup-manager datastore create "$name" "$path" \
--comment "ZFS dataset-based datastore"
'
Build PBS from this series, then run the server under manually
under flamegraph:
systemctl stop proxmox-backup-proxy
cargo flamegraph --release --bin proxmox-backup-proxy
## Patch summary
[PATCH 1/4] config: enable config version cache for datastore
[PATCH 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
[PATCH 3/4] partial fix #6049: datastore: use config fast-path in Drop
[PATCH 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
## Changes
Please refer to the per-patch changelogs.
## Maintainer notes
No dependency bumps, no API changes and no breaking changes.
Kind regards,
Samuel
Links
[1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
[2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
[3] Bugzilla #7017: https://bugzilla.proxmox.com/show_bug.cgi?id=7017
Samuel Rufinatscha (4):
config: enable config version cache for datastore
partial fix #6049: datastore: impl ConfigVersionCache fast path for
lookups
partial fix #6049: datastore: use config fast-path in Drop
partial fix #6049: datastore: add TTL fallback to catch manual config
edits
pbs-config/src/config_version_cache.rs | 10 +-
pbs-datastore/Cargo.toml | 1 +
pbs-datastore/src/datastore.rs | 148 +++++++++++++++++++++----
3 files changed, 135 insertions(+), 24 deletions(-)
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* [pbs-devel] superseded: [PATCH proxmox-backup v5 0/4] datastore: remove config reload on hot path
2025-11-24 17:04 12% [pbs-devel] [PATCH proxmox-backup v5 " Samuel Rufinatscha
` (4 preceding siblings ...)
2025-11-26 15:16 5% ` [pbs-devel] [PATCH proxmox-backup v5 0/4] datastore: remove config reload on hot path Fabian Grünbichler
@ 2026-01-05 14:21 13% ` Samuel Rufinatscha
5 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-05 14:21 UTC (permalink / raw)
To: pbs-devel
https://lore.proxmox.com/pbs-devel/20260105141615.242463-1-s.rufinatscha@proxmox.com/T/#t
On 11/24/25 6:03 PM, Samuel Rufinatscha wrote:
> Hi,
>
> this series reduces CPU time in datastore lookups by avoiding repeated
> datastore.cfg reads/parses in both `lookup_datastore()` and
> `DataStore::Drop`. It also adds a TTL so manual config edits are
> noticed without reintroducing hashing on every request.
>
> While investigating #6049 [1], cargo-flamegraph [2] showed hotspots
> during repeated `/status` calls in `lookup_datastore()` and in `Drop`,
> dominated by `pbs_config::datastore::config()` (config parse).
>
> The parsing cost itself should eventually be investigated in a future
> effort. Furthermore, cargo-flamegraph showed that when using a
> token-based auth method to access the API, a significant amount of time
> is spent in validation on every request request [3].
>
> ## Approach
>
> [PATCH 1/4] Support datastore generation in ConfigVersionCache
>
> [PATCH 2/4] Fast path for datastore lookups
> Cache the parsed datastore.cfg keyed by the shared datastore
> generation. lookup_datastore() reuses both the cached config and an
> existing DataStoreImpl when the generation matches, and falls back
> to the old slow path otherwise. The caching logic is implemented
> using the datastore_section_config_cached(update_cache: bool) helper.
>
> [PATCH 3/4] Fast path for Drop
> Make DataStore::Drop use the datastore_section_config_cached()
> helper to avoid re-reading/parsing datastore.cfg on every Drop.
> Bump generation not only on API config saves, but also on slow-path
> lookups (if update_cache is true), to enable Drop handlers see
> eventual newer configs.
>
> [PATCH 4/4] TTL to catch manual edits
> Add a TTL to the cached config and bump the datastore generation iff
> the digest changed but generation stays the same. This catches manual
> edits to datastore.cfg without reintroducing hashing or config
> parsing on every request.
>
> ## Benchmark results
>
> ### End-to-end
>
> Testing `/status?verbose=0` end-to-end with 1000 stores, 5 req/store
> and parallel=16 before/after the series:
>
> Metric Before After
> ----------------------------------------
> Total time 12s 9s
> Throughput (all) 416.67 555.56
> Cold RPS (round #1) 83.33 111.11
> Warm RPS (#2..N) 333.33 444.44
>
> Running under flamegraph [2], TLS appears to consume a significant
> amount of CPU time and blur the results. Still, a ~33% higher overall
> throughput and ~25% less end-to-end time for this workload.
>
> ### Isolated benchmarks (hyperfine)
>
> In addition to the end-to-end tests, I measured two standalone
> benchmarks with hyperfine, each using a config with 1000 datastores.
> `M` is the number of distinct datastores looked up and
> `N` is the number of lookups per datastore.
>
> Drop-direct variant:
>
> Drops the `DataStore` after every lookup, so the `Drop` path runs on
> every iteration:
>
> use anyhow::Error;
>
> use pbs_api_types::Operation;
> use pbs_datastore::DataStore;
>
> fn main() -> Result<(), Error> {
> let mut args = std::env::args();
> args.next();
>
> let datastores = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> let iterations = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> for d in 1..=datastores {
> let name = format!("ds{:04}", d);
>
> for i in 1..=iterations {
> DataStore::lookup_datastore(&name, Some(Operation::Write))?;
> }
> }
>
> Ok(())
> }
>
> +----+------+-----------+-----------+---------+
> | M | N | Baseline | Patched | Speedup |
> +----+------+-----------+-----------+---------+
> | 1 | 1000 | 1.684 s | 35.3 ms | 47.7x |
> | 10 | 100 | 1.689 s | 35.0 ms | 48.3x |
> | 100| 10 | 1.709 s | 35.8 ms | 47.7x |
> |1000| 1 | 1.809 s | 39.0 ms | 46.4x |
> +----+------+-----------+-----------+---------+
>
> Bulk-drop variant:
>
> Keeps the `DataStore` instances alive for
> all `N` lookups of a given datastore and then drops them in bulk,
> mimicking a task that performs many lookups while it is running and
> only triggers the expensive `Drop` logic when the last user exits.
>
> use anyhow::Error;
>
> use pbs_api_types::Operation;
> use pbs_datastore::DataStore;
>
> fn main() -> Result<(), Error> {
> let mut args = std::env::args();
> args.next();
>
> let datastores = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> let iterations = if let Some(n) = args.next() {
> n.parse::<usize>()?
> } else {
> 1000
> };
>
> for d in 1..=datastores {
> let name = format!("ds{:04}", d);
>
> let mut stores = Vec::with_capacity(iterations);
> for i in 1..=iterations {
> stores.push(DataStore::lookup_datastore(&name, Some(Operation::Write))?);
> }
> }
>
> Ok(())
> }
>
> +------+------+---------------+--------------+---------+
> | M | N | Baseline mean | Patched mean | Speedup |
> +------+------+---------------+--------------+---------+
> | 1 | 1000 | 890.6 ms | 35.5 ms | 25.1x |
> | 10 | 100 | 891.3 ms | 35.1 ms | 25.4x |
> | 100 | 10 | 983.9 ms | 35.6 ms | 27.6x |
> | 1000 | 1 | 1829.0 ms | 45.2 ms | 40.5x |
> +------+------+---------------+--------------+---------+
>
>
> Both variants show that the combination of the cached config lookups
> and the cheaper `Drop` handling reduces the hot-path cost from ~1.8 s
> per run to a few tens of milliseconds in these benchmarks.
>
> ## Reproduction steps
>
> VM: 4 vCPU, ~8 GiB RAM, VirtIO-SCSI; disks:
> - scsi0 32G (OS)
> - scsi1 1000G (datastores)
>
> Install PBS from ISO on the VM.
>
> Set up ZFS on /dev/sdb (adjust if different):
>
> zpool create -f -o ashift=12 pbsbench /dev/sdb
> zfs set mountpoint=/pbsbench pbsbench
> zfs create pbsbench/pbs-bench
>
> Raise file-descriptor limit:
>
> sudo systemctl edit proxmox-backup-proxy.service
>
> Add the following lines:
>
> [Service]
> LimitNOFILE=1048576
>
> Reload systemd and restart the proxy:
>
> sudo systemctl daemon-reload
> sudo systemctl restart proxmox-backup-proxy.service
>
> Verify the limit:
>
> systemctl show proxmox-backup-proxy.service | grep LimitNOFILE
>
> Create 1000 ZFS-backed datastores (as used in #6049 [1]):
>
> seq -w 001 1000 | xargs -n1 -P1 bash -c '
> id=$0
> name="ds${id}"
> dataset="pbsbench/pbs-bench/${name}"
> path="/pbsbench/pbs-bench/${name}"
> zfs create -o mountpoint="$path" "$dataset"
> proxmox-backup-manager datastore create "$name" "$path" \
> --comment "ZFS dataset-based datastore"
> '
>
> Build PBS from this series, then run the server under manually
> under flamegraph:
>
> systemctl stop proxmox-backup-proxy
> cargo flamegraph --release --bin proxmox-backup-proxy
>
> ## Patch summary
>
> [PATCH 1/4] partial fix #6049: config: enable config version cache for datastore
> [PATCH 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
> [PATCH 3/4] partial fix #6049: datastore: use config fast-path in Drop
> [PATCH 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
>
> ## Maintainer notes
>
> No dependency bumps, no API changes and no breaking changes.
>
> Thanks,
> Samuel
>
> Links
>
> [1] Bugzilla #6049: https://bugzilla.proxmox.com/show_bug.cgi?id=6049
> [2] cargo-flamegraph: https://github.com/flamegraph-rs/flamegraph
> [3] Bugzilla #7017: https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>
> Samuel Rufinatscha (4):
> partial fix #6049: config: enable config version cache for datastore
> partial fix #6049: datastore: impl ConfigVersionCache fast path for
> lookups
> partial fix #6049: datastore: use config fast-path in Drop
> partial fix #6049: datastore: add TTL fallback to catch manual config
> edits
>
> pbs-config/src/config_version_cache.rs | 10 +-
> pbs-datastore/Cargo.toml | 1 +
> pbs-datastore/src/datastore.rs | 213 ++++++++++++++++++++-----
> 3 files changed, 179 insertions(+), 45 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* Re: [pbs-devel] [PATCH proxmox-backup 1/1] fix: s3: make s3_refresh apihandler sync
@ 2026-01-05 15:22 13% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-05 15:22 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Nicolas Frey
Thanks, this makes sense - the ApiHandler mismatch explains the panic.
Reviewed-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
On 1/5/26 11:34 AM, Nicolas Frey wrote:
> fixes regression from 524cf1e that made `datastore::s3_refresh` sync
> but did not change the ApiHandler matching part here
>
> This would result in a panic every time an s3-refresh was initiated
>
> Fixes: https://forum.proxmox.com/threads/178655
> Signed-off-by: Nicolas Frey <n.frey@proxmox.com>
> ---
> src/bin/proxmox_backup_manager/datastore.rs | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
> index 57b4ca29..5c65c5ec 100644
> --- a/src/bin/proxmox_backup_manager/datastore.rs
> +++ b/src/bin/proxmox_backup_manager/datastore.rs
> @@ -339,7 +339,7 @@ async fn s3_refresh(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result
>
> let info = &api2::admin::datastore::API_METHOD_S3_REFRESH;
> let result = match info.handler {
> - ApiHandler::Async(handler) => (handler)(param, info, rpcenv).await?,
> + ApiHandler::Sync(handler) => (handler)(param, info, rpcenv)?,
> _ => unreachable!(),
> };
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox-backup v2 1/1] fix: s3: make s3_refresh apihandler sync
@ 2026-01-07 12:46 6% Nicolas Frey
0 siblings, 0 replies; 200+ results
From: Nicolas Frey @ 2026-01-07 12:46 UTC (permalink / raw)
To: pbs-devel
fixes regression from 524cf1e7 that made `datastore::s3_refresh` sync
but did not change the ApiHandler matching part here
This would result in a panic every time an s3-refresh was initiated
Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Tested-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
Fixes: 524cf1e7 ("api: admin: make s3 refresh handler sync")
Fixes: https://forum.proxmox.com/threads/178655
Signed-off-by: Nicolas Frey <n.frey@proxmox.com>
---
added Fixes trailer to reference blamed commit
src/bin/proxmox_backup_manager/datastore.rs | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
index 57b4ca29..5c65c5ec 100644
--- a/src/bin/proxmox_backup_manager/datastore.rs
+++ b/src/bin/proxmox_backup_manager/datastore.rs
@@ -339,7 +339,7 @@ async fn s3_refresh(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result
let info = &api2::admin::datastore::API_METHOD_S3_REFRESH;
let result = match info.handler {
- ApiHandler::Async(handler) => (handler)(param, info, rpcenv).await?,
+ ApiHandler::Sync(handler) => (handler)(param, info, rpcenv)?,
_ => unreachable!(),
};
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* [pbs-devel] [PATCH proxmox v5 4/4] acme-api: add helper to load client for an account
2026-01-08 11:26 11% [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (2 preceding siblings ...)
2026-01-08 11:26 14% ` [pbs-devel] [PATCH proxmox v5 3/4] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
@ 2026-01-08 11:26 17% ` Samuel Rufinatscha
2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-08 11:26 13% ` [pbs-devel] [PATCH proxmox-backup v5 1/5] acme: clean up ACME-related imports Samuel Rufinatscha
` (5 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-08 11:26 UTC (permalink / raw)
To: pbs-devel
The PBS ACME refactoring needs a simple way to obtain an AcmeClient for
a given configured account without duplicating config wiring. This patch
adds a load_client_with_account helper in proxmox-acme-api that loads
the account and constructs a matching client, similarly as PBS previous
own AcmeClient::load() function.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme-api/src/account_api_impl.rs | 5 +++++
proxmox-acme-api/src/lib.rs | 3 ++-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/proxmox-acme-api/src/account_api_impl.rs b/proxmox-acme-api/src/account_api_impl.rs
index ef195908..ca8c8655 100644
--- a/proxmox-acme-api/src/account_api_impl.rs
+++ b/proxmox-acme-api/src/account_api_impl.rs
@@ -116,3 +116,8 @@ pub async fn update_account(name: &AcmeAccountName, contact: Option<String>) ->
Ok(())
}
+
+pub async fn load_client_with_account(account_name: &AcmeAccountName) -> Result<AcmeClient, Error> {
+ let account_data = super::account_config::load_account_config(&account_name).await?;
+ Ok(account_data.client())
+}
diff --git a/proxmox-acme-api/src/lib.rs b/proxmox-acme-api/src/lib.rs
index 623e9e23..96f88ae2 100644
--- a/proxmox-acme-api/src/lib.rs
+++ b/proxmox-acme-api/src/lib.rs
@@ -31,7 +31,8 @@ mod plugin_config;
mod account_api_impl;
#[cfg(feature = "impl")]
pub use account_api_impl::{
- deactivate_account, get_account, get_tos, list_accounts, register_account, update_account,
+ deactivate_account, get_account, get_tos, list_accounts, load_client_with_account,
+ register_account, update_account,
};
#[cfg(feature = "impl")]
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 17%]
* [pbs-devel] [PATCH proxmox v5 3/4] fix #6939: acme: support servers returning 204 for nonce requests
2026-01-08 11:26 11% [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2026-01-08 11:26 10% ` [pbs-devel] [PATCH proxmox v5 1/4] acme: reduce visibility of Request type Samuel Rufinatscha
2026-01-08 11:26 15% ` [pbs-devel] [PATCH proxmox v5 2/4] acme: introduce http_status module Samuel Rufinatscha
@ 2026-01-08 11:26 14% ` Samuel Rufinatscha
2026-01-08 11:26 17% ` [pbs-devel] [PATCH proxmox v5 4/4] acme-api: add helper to load client for an account Samuel Rufinatscha
` (6 subsequent siblings)
9 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-08 11:26 UTC (permalink / raw)
To: pbs-devel
Some ACME servers (notably custom or legacy implementations) respond
to HEAD /newNonce with a 204 No Content instead of the
RFC 8555-recommended 200 OK [1]. While this behavior is technically
off-spec, it is not illegal. This issue was reported on our bug
tracker [2].
The previous implementation treated any non-200 response as an error,
causing account registration to fail against such servers. Relax the
status-code check to accept both 200 and 204 responses (and potentially
support other 2xx codes) to improve interoperability.
Note: In comparison, PVE’s Perl ACME client performs a GET request [3]
instead of a HEAD request and accepts any 2xx success code when
retrieving the nonce [4]. This difference in behavior does not affect
functionality but is worth noting for consistency across
implementations.
[1] https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=6939
[3] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219
[4] https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597
Fixes: #6939
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme/src/account.rs | 8 ++++----
proxmox-acme/src/async_client.rs | 6 +++---
proxmox-acme/src/client.rs | 2 +-
proxmox-acme/src/request.rs | 4 ++--
4 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
index ea1a3c60..84610bf3 100644
--- a/proxmox-acme/src/account.rs
+++ b/proxmox-acme/src/account.rs
@@ -84,7 +84,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::CREATED,
+ expected: &[crate::http_status::CREATED],
};
Ok(NewOrder::new(request))
@@ -106,7 +106,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK],
})
}
@@ -131,7 +131,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK],
})
}
@@ -321,7 +321,7 @@ impl AccountCreator {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::http_status::CREATED,
+ expected: &[crate::http_status::CREATED],
})
}
diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
index 043648bb..07da842c 100644
--- a/proxmox-acme/src/async_client.rs
+++ b/proxmox-acme/src/async_client.rs
@@ -420,7 +420,7 @@ impl AcmeClient {
};
if parts.status.is_success() {
- if status != request.expected {
+ if !request.expected.contains(&status) {
return Err(Error::InvalidApi(format!(
"ACME server responded with unexpected status code: {:?}",
parts.status
@@ -498,7 +498,7 @@ impl AcmeClient {
method: "GET",
content_type: "",
body: String::new(),
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK],
},
nonce,
)
@@ -550,7 +550,7 @@ impl AcmeClient {
method: "HEAD",
content_type: "",
body: String::new(),
- expected: crate::http_status::OK,
+ expected: &[crate::http_status::OK, crate::http_status::NO_CONTENT],
},
nonce,
)
diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
index 5c812567..af250fb8 100644
--- a/proxmox-acme/src/client.rs
+++ b/proxmox-acme/src/client.rs
@@ -203,7 +203,7 @@ impl Inner {
let got_nonce = self.update_nonce(&mut response)?;
if response.is_success() {
- if response.status != request.expected {
+ if !request.expected.contains(&response.status) {
return Err(Error::InvalidApi(format!(
"API server responded with unexpected status code: {:?}",
response.status
diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
index 341ce53e..d782a7de 100644
--- a/proxmox-acme/src/request.rs
+++ b/proxmox-acme/src/request.rs
@@ -16,8 +16,8 @@ pub(crate) struct Request {
/// The body to pass along with request, or an empty string.
pub(crate) body: String,
- /// The expected status code a compliant ACME provider will return on success.
- pub(crate) expected: u16,
+ /// The set of HTTP status codes that indicate a successful response from an ACME provider.
+ pub(crate) expected: &'static [u16],
}
/// Common HTTP status codes used in ACME responses.
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 14%]
* [pbs-devel] [PATCH proxmox-backup v5 4/5] acme: change API impls to use proxmox-acme-api handlers
2026-01-08 11:26 11% [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (6 preceding siblings ...)
2026-01-08 11:26 6% ` [pbs-devel] [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient Samuel Rufinatscha
@ 2026-01-08 11:26 8% ` Samuel Rufinatscha
2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-08 11:26 7% ` [pbs-devel] [PATCH proxmox-backup v5 5/5] acme: certificate ordering through proxmox-acme-api Samuel Rufinatscha
2026-01-13 13:48 5% ` [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Fabian Grünbichler
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-08 11:26 UTC (permalink / raw)
To: pbs-devel
PBS currently uses its own ACME client and API logic, while PDM uses the
factored out proxmox-acme and proxmox-acme-api crates. This duplication
risks differences in behaviour and requires ACME maintenance in two
places. This patch is part of a series to move PBS over to the shared
ACME stack.
Changes:
- Replace api2/config/acme.rs API logic with proxmox-acme-api handlers.
- Drop local caching and helper types that duplicate proxmox-acme-api.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/api2/config/acme.rs | 378 ++-----------------------
src/api2/types/acme.rs | 16 --
src/bin/proxmox_backup_manager/acme.rs | 6 +-
src/config/acme/mod.rs | 44 +--
4 files changed, 33 insertions(+), 411 deletions(-)
diff --git a/src/api2/config/acme.rs b/src/api2/config/acme.rs
index 898f06dd..3314430c 100644
--- a/src/api2/config/acme.rs
+++ b/src/api2/config/acme.rs
@@ -1,29 +1,18 @@
-use std::fs;
-use std::ops::ControlFlow;
-use std::path::Path;
-use std::sync::{Arc, LazyLock, Mutex};
-use std::time::SystemTime;
-
-use anyhow::{bail, format_err, Error};
-use hex::FromHex;
-use serde::{Deserialize, Serialize};
-use serde_json::{json, Value};
-use tracing::{info, warn};
+use anyhow::Error;
+use tracing::info;
use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
-use proxmox_acme::async_client::AcmeClient;
-use proxmox_acme::types::AccountData as AcmeAccountData;
-use proxmox_acme_api::AcmeAccountName;
+use proxmox_acme_api::{
+ AccountEntry, AccountInfo, AcmeAccountName, AcmeChallengeSchema, ChallengeSchemaWrapper,
+ DeletablePluginProperty, DnsPluginCore, DnsPluginCoreUpdater, KnownAcmeDirectory, PluginConfig,
+ DEFAULT_ACME_DIRECTORY_ENTRY, PLUGIN_ID_SCHEMA,
+};
+use proxmox_config_digest::ConfigDigest;
use proxmox_rest_server::WorkerTask;
use proxmox_router::{
http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
};
-use proxmox_schema::{api, param_bail};
-
-use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
-use crate::config::acme::plugin::{
- self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
-};
+use proxmox_schema::api;
pub(crate) const ROUTER: Router = Router::new()
.get(&list_subdirs_api_method!(SUBDIRS))
@@ -65,19 +54,6 @@ const PLUGIN_ITEM_ROUTER: Router = Router::new()
.put(&API_METHOD_UPDATE_PLUGIN)
.delete(&API_METHOD_DELETE_PLUGIN);
-#[api(
- properties: {
- name: { type: AcmeAccountName },
- },
-)]
-/// An ACME Account entry.
-///
-/// Currently only contains a 'name' property.
-#[derive(Serialize)]
-pub struct AccountEntry {
- name: AcmeAccountName,
-}
-
#[api(
access: {
permission: &Permission::Privilege(&["system", "certificates"], PRIV_SYS_MODIFY, false),
@@ -91,40 +67,7 @@ pub struct AccountEntry {
)]
/// List ACME accounts.
pub fn list_accounts() -> Result<Vec<AccountEntry>, Error> {
- let mut entries = Vec::new();
- crate::config::acme::foreach_acme_account(|name| {
- entries.push(AccountEntry { name });
- ControlFlow::Continue(())
- })?;
- Ok(entries)
-}
-
-#[api(
- properties: {
- account: { type: Object, properties: {}, additional_properties: true },
- tos: {
- type: String,
- optional: true,
- },
- },
-)]
-/// ACME Account information.
-///
-/// This is what we return via the API.
-#[derive(Serialize)]
-pub struct AccountInfo {
- /// Raw account data.
- account: AcmeAccountData,
-
- /// The ACME directory URL the account was created at.
- directory: String,
-
- /// The account's own URL within the ACME directory.
- location: String,
-
- /// The ToS URL, if the user agreed to one.
- #[serde(skip_serializing_if = "Option::is_none")]
- tos: Option<String>,
+ proxmox_acme_api::list_accounts()
}
#[api(
@@ -141,23 +84,7 @@ pub struct AccountInfo {
)]
/// Return existing ACME account information.
pub async fn get_account(name: AcmeAccountName) -> Result<AccountInfo, Error> {
- let account_info = proxmox_acme_api::get_account(name).await?;
-
- Ok(AccountInfo {
- location: account_info.location,
- tos: account_info.tos,
- directory: account_info.directory,
- account: AcmeAccountData {
- only_return_existing: false, // don't actually write this out in case it's set
- ..account_info.account
- },
- })
-}
-
-fn account_contact_from_string(s: &str) -> Vec<String> {
- s.split(&[' ', ';', ',', '\0'][..])
- .map(|s| format!("mailto:{s}"))
- .collect()
+ proxmox_acme_api::get_account(name).await
}
#[api(
@@ -222,15 +149,11 @@ fn register_account(
);
}
- if Path::new(&crate::config::acme::account_path(&name)).exists() {
+ if std::path::Path::new(&proxmox_acme_api::account_config_filename(&name)).exists() {
http_bail!(BAD_REQUEST, "account {} already exists", name);
}
- let directory = directory.unwrap_or_else(|| {
- crate::config::acme::DEFAULT_ACME_DIRECTORY_ENTRY
- .url
- .to_owned()
- });
+ let directory = directory.unwrap_or_else(|| DEFAULT_ACME_DIRECTORY_ENTRY.url.to_string());
WorkerTask::spawn(
"acme-register",
@@ -286,17 +209,7 @@ pub fn update_account(
auth_id.to_string(),
true,
move |_worker| async move {
- let data = match contact {
- Some(data) => json!({
- "contact": account_contact_from_string(&data),
- }),
- None => json!({}),
- };
-
- proxmox_acme_api::load_client_with_account(&name)
- .await?
- .update_account(&data)
- .await?;
+ proxmox_acme_api::update_account(&name, contact).await?;
Ok(())
},
@@ -334,18 +247,8 @@ pub fn deactivate_account(
auth_id.to_string(),
true,
move |_worker| async move {
- match proxmox_acme_api::load_client_with_account(&name)
- .await?
- .update_account(&json!({"status": "deactivated"}))
- .await
- {
- Ok(_account) => (),
- Err(err) if !force => return Err(err),
- Err(err) => {
- warn!("error deactivating account {name}, proceeding anyway - {err}");
- }
- }
- crate::config::acme::mark_account_deactivated(&name)?;
+ proxmox_acme_api::deactivate_account(&name, force).await?;
+
Ok(())
},
)
@@ -372,15 +275,7 @@ pub fn deactivate_account(
)]
/// Get the Terms of Service URL for an ACME directory.
async fn get_tos(directory: Option<String>) -> Result<Option<String>, Error> {
- let directory = directory.unwrap_or_else(|| {
- crate::config::acme::DEFAULT_ACME_DIRECTORY_ENTRY
- .url
- .to_owned()
- });
- Ok(AcmeClient::new(directory)
- .terms_of_service_url()
- .await?
- .map(str::to_owned))
+ proxmox_acme_api::get_tos(directory).await
}
#[api(
@@ -395,52 +290,7 @@ async fn get_tos(directory: Option<String>) -> Result<Option<String>, Error> {
)]
/// Get named known ACME directory endpoints.
fn get_directories() -> Result<&'static [KnownAcmeDirectory], Error> {
- Ok(crate::config::acme::KNOWN_ACME_DIRECTORIES)
-}
-
-/// Wrapper for efficient Arc use when returning the ACME challenge-plugin schema for serializing
-struct ChallengeSchemaWrapper {
- inner: Arc<Vec<AcmeChallengeSchema>>,
-}
-
-impl Serialize for ChallengeSchemaWrapper {
- fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
- where
- S: serde::Serializer,
- {
- self.inner.serialize(serializer)
- }
-}
-
-struct CachedSchema {
- schema: Arc<Vec<AcmeChallengeSchema>>,
- cached_mtime: SystemTime,
-}
-
-fn get_cached_challenge_schemas() -> Result<ChallengeSchemaWrapper, Error> {
- static CACHE: LazyLock<Mutex<Option<CachedSchema>>> = LazyLock::new(|| Mutex::new(None));
-
- // the actual loading code
- let mut last = CACHE.lock().unwrap();
-
- let actual_mtime = fs::metadata(crate::config::acme::ACME_DNS_SCHEMA_FN)?.modified()?;
-
- let schema = match &*last {
- Some(CachedSchema {
- schema,
- cached_mtime,
- }) if *cached_mtime >= actual_mtime => schema.clone(),
- _ => {
- let new_schema = Arc::new(crate::config::acme::load_dns_challenge_schema()?);
- *last = Some(CachedSchema {
- schema: Arc::clone(&new_schema),
- cached_mtime: actual_mtime,
- });
- new_schema
- }
- };
-
- Ok(ChallengeSchemaWrapper { inner: schema })
+ Ok(proxmox_acme_api::KNOWN_ACME_DIRECTORIES)
}
#[api(
@@ -455,69 +305,7 @@ fn get_cached_challenge_schemas() -> Result<ChallengeSchemaWrapper, Error> {
)]
/// Get named known ACME directory endpoints.
fn get_challenge_schema() -> Result<ChallengeSchemaWrapper, Error> {
- get_cached_challenge_schemas()
-}
-
-#[api]
-#[derive(Default, Deserialize, Serialize)]
-#[serde(rename_all = "kebab-case")]
-/// The API's format is inherited from PVE/PMG:
-pub struct PluginConfig {
- /// Plugin ID.
- plugin: String,
-
- /// Plugin type.
- #[serde(rename = "type")]
- ty: String,
-
- /// DNS Api name.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- api: Option<String>,
-
- /// Plugin configuration data.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- data: Option<String>,
-
- /// Extra delay in seconds to wait before requesting validation.
- ///
- /// Allows to cope with long TTL of DNS records.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- validation_delay: Option<u32>,
-
- /// Flag to disable the config.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- disable: Option<bool>,
-}
-
-// See PMG/PVE's $modify_cfg_for_api sub
-fn modify_cfg_for_api(id: &str, ty: &str, data: &Value) -> PluginConfig {
- let mut entry = data.clone();
-
- let obj = entry.as_object_mut().unwrap();
- obj.remove("id");
- obj.insert("plugin".to_string(), Value::String(id.to_owned()));
- obj.insert("type".to_string(), Value::String(ty.to_owned()));
-
- // FIXME: This needs to go once the `Updater` is fixed.
- // None of these should be able to fail unless the user changed the files by hand, in which
- // case we leave the unmodified string in the Value for now. This will be handled with an error
- // later.
- if let Some(Value::String(ref mut data)) = obj.get_mut("data") {
- if let Ok(new) = proxmox_base64::url::decode_no_pad(&data) {
- if let Ok(utf8) = String::from_utf8(new) {
- *data = utf8;
- }
- }
- }
-
- // PVE/PMG do this explicitly for ACME plugins...
- // obj.insert("digest".to_string(), Value::String(digest.clone()));
-
- serde_json::from_value(entry).unwrap_or_else(|_| PluginConfig {
- plugin: "*Error*".to_string(),
- ty: "*Error*".to_string(),
- ..Default::default()
- })
+ proxmox_acme_api::get_cached_challenge_schemas()
}
#[api(
@@ -533,12 +321,7 @@ fn modify_cfg_for_api(id: &str, ty: &str, data: &Value) -> PluginConfig {
)]
/// List ACME challenge plugins.
pub fn list_plugins(rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<PluginConfig>, Error> {
- let (plugins, digest) = plugin::config()?;
- rpcenv["digest"] = hex::encode(digest).into();
- Ok(plugins
- .iter()
- .map(|(id, (ty, data))| modify_cfg_for_api(id, ty, data))
- .collect())
+ proxmox_acme_api::list_plugins(rpcenv)
}
#[api(
@@ -555,13 +338,7 @@ pub fn list_plugins(rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<PluginConfig>
)]
/// List ACME challenge plugins.
pub fn get_plugin(id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<PluginConfig, Error> {
- let (plugins, digest) = plugin::config()?;
- rpcenv["digest"] = hex::encode(digest).into();
-
- match plugins.get(&id) {
- Some((ty, data)) => Ok(modify_cfg_for_api(&id, ty, data)),
- None => http_bail!(NOT_FOUND, "no such plugin"),
- }
+ proxmox_acme_api::get_plugin(id, rpcenv)
}
// Currently we only have "the" standalone plugin and DNS plugins so we can just flatten a
@@ -593,30 +370,7 @@ pub fn get_plugin(id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<PluginC
)]
/// Add ACME plugin configuration.
pub fn add_plugin(r#type: String, core: DnsPluginCore, data: String) -> Result<(), Error> {
- // Currently we only support DNS plugins and the standalone plugin is "fixed":
- if r#type != "dns" {
- param_bail!("type", "invalid ACME plugin type: {:?}", r#type);
- }
-
- let data = String::from_utf8(proxmox_base64::decode(data)?)
- .map_err(|_| format_err!("data must be valid UTF-8"))?;
-
- let id = core.id.clone();
-
- let _lock = plugin::lock()?;
-
- let (mut plugins, _digest) = plugin::config()?;
- if plugins.contains_key(&id) {
- param_bail!("id", "ACME plugin ID {:?} already exists", id);
- }
-
- let plugin = serde_json::to_value(DnsPlugin { core, data })?;
-
- plugins.insert(id, r#type, plugin);
-
- plugin::save_config(&plugins)?;
-
- Ok(())
+ proxmox_acme_api::add_plugin(r#type, core, data)
}
#[api(
@@ -632,26 +386,7 @@ pub fn add_plugin(r#type: String, core: DnsPluginCore, data: String) -> Result<(
)]
/// Delete an ACME plugin configuration.
pub fn delete_plugin(id: String) -> Result<(), Error> {
- let _lock = plugin::lock()?;
-
- let (mut plugins, _digest) = plugin::config()?;
- if plugins.remove(&id).is_none() {
- http_bail!(NOT_FOUND, "no such plugin");
- }
- plugin::save_config(&plugins)?;
-
- Ok(())
-}
-
-#[api()]
-#[derive(Serialize, Deserialize)]
-#[serde(rename_all = "kebab-case")]
-/// Deletable property name
-pub enum DeletableProperty {
- /// Delete the disable property
- Disable,
- /// Delete the validation-delay property
- ValidationDelay,
+ proxmox_acme_api::delete_plugin(id)
}
#[api(
@@ -673,12 +408,12 @@ pub enum DeletableProperty {
type: Array,
optional: true,
items: {
- type: DeletableProperty,
+ type: DeletablePluginProperty,
}
},
digest: {
- description: "Digest to protect against concurrent updates",
optional: true,
+ type: ConfigDigest,
},
},
},
@@ -692,65 +427,8 @@ pub fn update_plugin(
id: String,
update: DnsPluginCoreUpdater,
data: Option<String>,
- delete: Option<Vec<DeletableProperty>>,
- digest: Option<String>,
+ delete: Option<Vec<DeletablePluginProperty>>,
+ digest: Option<ConfigDigest>,
) -> Result<(), Error> {
- let data = data
- .as_deref()
- .map(proxmox_base64::decode)
- .transpose()?
- .map(String::from_utf8)
- .transpose()
- .map_err(|_| format_err!("data must be valid UTF-8"))?;
-
- let _lock = plugin::lock()?;
-
- let (mut plugins, expected_digest) = plugin::config()?;
-
- if let Some(digest) = digest {
- let digest = <[u8; 32]>::from_hex(digest)?;
- crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
- }
-
- match plugins.get_mut(&id) {
- Some((ty, ref mut entry)) => {
- if ty != "dns" {
- bail!("cannot update plugin of type {:?}", ty);
- }
-
- let mut plugin = DnsPlugin::deserialize(&*entry)?;
-
- if let Some(delete) = delete {
- for delete_prop in delete {
- match delete_prop {
- DeletableProperty::ValidationDelay => {
- plugin.core.validation_delay = None;
- }
- DeletableProperty::Disable => {
- plugin.core.disable = None;
- }
- }
- }
- }
- if let Some(data) = data {
- plugin.data = data;
- }
- if let Some(api) = update.api {
- plugin.core.api = api;
- }
- if update.validation_delay.is_some() {
- plugin.core.validation_delay = update.validation_delay;
- }
- if update.disable.is_some() {
- plugin.core.disable = update.disable;
- }
-
- *entry = serde_json::to_value(plugin)?;
- }
- None => http_bail!(NOT_FOUND, "no such plugin"),
- }
-
- plugin::save_config(&plugins)?;
-
- Ok(())
+ proxmox_acme_api::update_plugin(id, update, data, delete, digest)
}
diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
index 64175aff..0ff496b6 100644
--- a/src/api2/types/acme.rs
+++ b/src/api2/types/acme.rs
@@ -43,22 +43,6 @@ pub const ACME_DOMAIN_PROPERTY_SCHEMA: Schema =
.format(&ApiStringFormat::PropertyString(&AcmeDomain::API_SCHEMA))
.schema();
-#[api(
- properties: {
- name: { type: String },
- url: { type: String },
- },
-)]
-/// An ACME directory endpoint with a name and URL.
-#[derive(Serialize)]
-pub struct KnownAcmeDirectory {
- /// The ACME directory's name.
- pub name: &'static str,
-
- /// The ACME directory's endpoint URL.
- pub url: &'static str,
-}
-
#[api(
properties: {
schema: {
diff --git a/src/bin/proxmox_backup_manager/acme.rs b/src/bin/proxmox_backup_manager/acme.rs
index 6ed61560..d11d7498 100644
--- a/src/bin/proxmox_backup_manager/acme.rs
+++ b/src/bin/proxmox_backup_manager/acme.rs
@@ -4,14 +4,12 @@ use anyhow::{bail, Error};
use serde_json::Value;
use proxmox_acme::async_client::AcmeClient;
-use proxmox_acme_api::AcmeAccountName;
+use proxmox_acme_api::{AcmeAccountName, DnsPluginCore, KNOWN_ACME_DIRECTORIES};
use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
use proxmox_schema::api;
use proxmox_sys::fs::file_get_contents;
use proxmox_backup::api2;
-use proxmox_backup::config::acme::plugin::DnsPluginCore;
-use proxmox_backup::config::acme::KNOWN_ACME_DIRECTORIES;
pub fn acme_mgmt_cli() -> CommandLineInterface {
let cmd_def = CliCommandMap::new()
@@ -122,7 +120,7 @@ async fn register_account(
match input.trim().parse::<usize>() {
Ok(n) if n < KNOWN_ACME_DIRECTORIES.len() => {
- break (KNOWN_ACME_DIRECTORIES[n].url.to_owned(), false);
+ break (KNOWN_ACME_DIRECTORIES[n].url.to_string(), false);
}
Ok(n) if n == KNOWN_ACME_DIRECTORIES.len() => {
input.clear();
diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
index e4639c53..01ab6223 100644
--- a/src/config/acme/mod.rs
+++ b/src/config/acme/mod.rs
@@ -1,16 +1,15 @@
use std::collections::HashMap;
use std::ops::ControlFlow;
-use std::path::Path;
-use anyhow::{bail, format_err, Error};
+use anyhow::Error;
use serde_json::Value;
use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
-use proxmox_acme_api::AcmeAccountName;
+use proxmox_acme_api::{AcmeAccountName, KnownAcmeDirectory, KNOWN_ACME_DIRECTORIES};
use proxmox_sys::error::SysError;
use proxmox_sys::fs::{file_read_string, CreateOptions};
-use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
+use crate::api2::types::AcmeChallengeSchema;
pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
@@ -35,23 +34,8 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
create_acme_subdir(ACME_DIR)
}
-pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
- KnownAcmeDirectory {
- name: "Let's Encrypt V2",
- url: "https://acme-v02.api.letsencrypt.org/directory",
- },
- KnownAcmeDirectory {
- name: "Let's Encrypt V2 Staging",
- url: "https://acme-staging-v02.api.letsencrypt.org/directory",
- },
-];
-
pub const DEFAULT_ACME_DIRECTORY_ENTRY: &KnownAcmeDirectory = &KNOWN_ACME_DIRECTORIES[0];
-pub fn account_path(name: &str) -> String {
- format!("{ACME_ACCOUNT_DIR}/{name}")
-}
-
pub fn foreach_acme_account<F>(mut func: F) -> Result<(), Error>
where
F: FnMut(AcmeAccountName) -> ControlFlow<Result<(), Error>>,
@@ -82,28 +66,6 @@ where
}
}
-pub fn mark_account_deactivated(name: &str) -> Result<(), Error> {
- let from = account_path(name);
- for i in 0..100 {
- let to = account_path(&format!("_deactivated_{name}_{i}"));
- if !Path::new(&to).exists() {
- return std::fs::rename(&from, &to).map_err(|err| {
- format_err!(
- "failed to move account path {:?} to {:?} - {}",
- from,
- to,
- err
- )
- });
- }
- }
- bail!(
- "No free slot to rename deactivated account {:?}, please cleanup {:?}",
- from,
- ACME_ACCOUNT_DIR
- );
-}
-
pub fn load_dns_challenge_schema() -> Result<Vec<AcmeChallengeSchema>, Error> {
let raw = file_read_string(ACME_DNS_SCHEMA_FN)?;
let schemas: serde_json::Map<String, Value> = serde_json::from_str(&raw)?;
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 8%]
* [pbs-devel] [PATCH proxmox-backup v5 5/5] acme: certificate ordering through proxmox-acme-api
2026-01-08 11:26 11% [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (7 preceding siblings ...)
2026-01-08 11:26 8% ` [pbs-devel] [PATCH proxmox-backup v5 4/5] acme: change API impls to use proxmox-acme-api handlers Samuel Rufinatscha
@ 2026-01-08 11:26 7% ` Samuel Rufinatscha
2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-13 13:48 5% ` [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Fabian Grünbichler
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-08 11:26 UTC (permalink / raw)
To: pbs-devel
PBS currently uses its own ACME client and API logic, while PDM uses the
factored out proxmox-acme and proxmox-acme-api crates. This duplication
risks differences in behaviour and requires ACME maintenance in two
places. This patch is part of a series to move PBS over to the shared
ACME stack.
Changes:
- Replace the custom ACME order/authorization loop in node certificates
with a call to proxmox_acme_api::order_certificate.
- Build domain + config data as proxmox-acme-api types
- Remove obsolete local ACME ordering and plugin glue code.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/acme/mod.rs | 2 -
src/acme/plugin.rs | 335 ----------------------------------
src/api2/node/certificates.rs | 229 ++++-------------------
src/api2/types/acme.rs | 73 --------
src/api2/types/mod.rs | 3 -
src/config/acme/mod.rs | 8 +-
src/config/acme/plugin.rs | 92 +---------
src/config/node.rs | 20 +-
src/lib.rs | 2 -
9 files changed, 38 insertions(+), 726 deletions(-)
delete mode 100644 src/acme/mod.rs
delete mode 100644 src/acme/plugin.rs
delete mode 100644 src/api2/types/acme.rs
diff --git a/src/acme/mod.rs b/src/acme/mod.rs
deleted file mode 100644
index cc561f9a..00000000
--- a/src/acme/mod.rs
+++ /dev/null
@@ -1,2 +0,0 @@
-pub(crate) mod plugin;
-pub(crate) use plugin::get_acme_plugin;
diff --git a/src/acme/plugin.rs b/src/acme/plugin.rs
deleted file mode 100644
index 6804243c..00000000
--- a/src/acme/plugin.rs
+++ /dev/null
@@ -1,335 +0,0 @@
-use std::future::Future;
-use std::net::{IpAddr, SocketAddr};
-use std::pin::Pin;
-use std::process::Stdio;
-use std::sync::Arc;
-use std::time::Duration;
-
-use anyhow::{bail, format_err, Error};
-use bytes::Bytes;
-use futures::TryFutureExt;
-use http_body_util::Full;
-use hyper::body::Incoming;
-use hyper::server::conn::http1;
-use hyper::service::service_fn;
-use hyper::{Request, Response};
-use hyper_util::rt::TokioIo;
-use tokio::io::{AsyncBufReadExt, AsyncRead, AsyncWriteExt, BufReader};
-use tokio::net::TcpListener;
-use tokio::process::Command;
-
-use proxmox_acme::async_client::AcmeClient;
-use proxmox_acme::{Authorization, Challenge};
-use proxmox_rest_server::WorkerTask;
-
-use crate::api2::types::AcmeDomain;
-use crate::config::acme::plugin::{DnsPlugin, PluginData};
-
-const PROXMOX_ACME_SH_PATH: &str = "/usr/share/proxmox-acme/proxmox-acme";
-
-pub(crate) fn get_acme_plugin(
- plugin_data: &PluginData,
- name: &str,
-) -> Result<Option<Box<dyn AcmePlugin + Send + Sync + 'static>>, Error> {
- let (ty, data) = match plugin_data.get(name) {
- Some(plugin) => plugin,
- None => return Ok(None),
- };
-
- Ok(Some(match ty.as_str() {
- "dns" => {
- let plugin: DnsPlugin = serde::Deserialize::deserialize(data)?;
- Box::new(plugin)
- }
- "standalone" => {
- // this one has no config
- Box::<StandaloneServer>::default()
- }
- other => bail!("missing implementation for plugin type '{}'", other),
- }))
-}
-
-pub(crate) trait AcmePlugin {
- /// Setup everything required to trigger the validation and return the corresponding validation
- /// URL.
- fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- domain: &'d AcmeDomain,
- task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>>;
-
- fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- domain: &'d AcmeDomain,
- task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>>;
-}
-
-fn extract_challenge<'a>(
- authorization: &'a Authorization,
- ty: &str,
-) -> Result<&'a Challenge, Error> {
- authorization
- .challenges
- .iter()
- .find(|ch| ch.ty == ty)
- .ok_or_else(|| format_err!("no supported challenge type ({}) found", ty))
-}
-
-async fn pipe_to_tasklog<T: AsyncRead + Unpin>(
- pipe: T,
- task: Arc<WorkerTask>,
-) -> Result<(), std::io::Error> {
- let mut pipe = BufReader::new(pipe);
- let mut line = String::new();
- loop {
- line.clear();
- match pipe.read_line(&mut line).await {
- Ok(0) => return Ok(()),
- Ok(_) => task.log_message(line.as_str()),
- Err(err) => return Err(err),
- }
- }
-}
-
-impl DnsPlugin {
- async fn action<'a>(
- &self,
- client: &mut AcmeClient,
- authorization: &'a Authorization,
- domain: &AcmeDomain,
- task: Arc<WorkerTask>,
- action: &str,
- ) -> Result<&'a str, Error> {
- let challenge = extract_challenge(authorization, "dns-01")?;
- let mut stdin_data = client
- .dns_01_txt_value(
- challenge
- .token()
- .ok_or_else(|| format_err!("missing token in challenge"))?,
- )?
- .into_bytes();
- stdin_data.push(b'\n');
- stdin_data.extend(self.data.as_bytes());
- if stdin_data.last() != Some(&b'\n') {
- stdin_data.push(b'\n');
- }
-
- let mut command = Command::new("/usr/bin/setpriv");
-
- #[rustfmt::skip]
- command.args([
- "--reuid", "nobody",
- "--regid", "nogroup",
- "--clear-groups",
- "--reset-env",
- "--",
- "/bin/bash",
- PROXMOX_ACME_SH_PATH,
- action,
- &self.core.api,
- domain.alias.as_deref().unwrap_or(&domain.domain),
- ]);
-
- // We could use 1 socketpair, but tokio wraps them all in `File` internally causing `close`
- // to be called separately on all of them without exception, so we need 3 pipes :-(
-
- let mut child = command
- .stdin(Stdio::piped())
- .stdout(Stdio::piped())
- .stderr(Stdio::piped())
- .spawn()?;
-
- let mut stdin = child.stdin.take().expect("Stdio::piped()");
- let stdout = child.stdout.take().expect("Stdio::piped() failed?");
- let stdout = pipe_to_tasklog(stdout, Arc::clone(&task));
- let stderr = child.stderr.take().expect("Stdio::piped() failed?");
- let stderr = pipe_to_tasklog(stderr, Arc::clone(&task));
- let stdin = async move {
- stdin.write_all(&stdin_data).await?;
- stdin.flush().await?;
- Ok::<_, std::io::Error>(())
- };
- match futures::try_join!(stdin, stdout, stderr) {
- Ok(((), (), ())) => (),
- Err(err) => {
- if let Err(err) = child.kill().await {
- task.log_message(format!(
- "failed to kill '{PROXMOX_ACME_SH_PATH} {action}' command: {err}"
- ));
- }
- bail!("'{}' failed: {}", PROXMOX_ACME_SH_PATH, err);
- }
- }
-
- let status = child.wait().await?;
- if !status.success() {
- bail!(
- "'{} {}' exited with error ({})",
- PROXMOX_ACME_SH_PATH,
- action,
- status.code().unwrap_or(-1)
- );
- }
-
- Ok(&challenge.url)
- }
-}
-
-impl AcmePlugin for DnsPlugin {
- fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- domain: &'d AcmeDomain,
- task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>> {
- Box::pin(async move {
- let result = self
- .action(client, authorization, domain, task.clone(), "setup")
- .await;
-
- let validation_delay = self.core.validation_delay.unwrap_or(30) as u64;
- if validation_delay > 0 {
- task.log_message(format!(
- "Sleeping {validation_delay} seconds to wait for TXT record propagation"
- ));
- tokio::time::sleep(Duration::from_secs(validation_delay)).await;
- }
- result
- })
- }
-
- fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- domain: &'d AcmeDomain,
- task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>> {
- Box::pin(async move {
- self.action(client, authorization, domain, task, "teardown")
- .await
- .map(drop)
- })
- }
-}
-
-#[derive(Default)]
-struct StandaloneServer {
- abort_handle: Option<futures::future::AbortHandle>,
-}
-
-// In case the "order_certificates" future gets dropped between setup & teardown, let's also cancel
-// the HTTP listener on Drop:
-impl Drop for StandaloneServer {
- fn drop(&mut self) {
- self.stop();
- }
-}
-
-impl StandaloneServer {
- fn stop(&mut self) {
- if let Some(abort) = self.abort_handle.take() {
- abort.abort();
- }
- }
-}
-
-async fn standalone_respond(
- req: Request<Incoming>,
- path: Arc<String>,
- key_auth: Arc<String>,
-) -> Result<Response<Full<Bytes>>, hyper::Error> {
- if req.method() == hyper::Method::GET && req.uri().path() == path.as_str() {
- Ok(Response::builder()
- .status(hyper::http::StatusCode::OK)
- .body(key_auth.as_bytes().to_vec().into())
- .unwrap())
- } else {
- Ok(Response::builder()
- .status(hyper::http::StatusCode::NOT_FOUND)
- .body("Not found.".into())
- .unwrap())
- }
-}
-
-impl AcmePlugin for StandaloneServer {
- fn setup<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- client: &'b mut AcmeClient,
- authorization: &'c Authorization,
- _domain: &'d AcmeDomain,
- _task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<&'c str, Error>> + Send + 'fut>> {
- Box::pin(async move {
- self.stop();
-
- let challenge = extract_challenge(authorization, "http-01")?;
- let token = challenge
- .token()
- .ok_or_else(|| format_err!("missing token in challenge"))?;
- let key_auth = Arc::new(client.key_authorization(token)?);
- let path = Arc::new(format!("/.well-known/acme-challenge/{token}"));
-
- // `[::]:80` first, then `*:80`
- let dual = SocketAddr::new(IpAddr::from([0u16; 8]), 80);
- let ipv4 = SocketAddr::new(IpAddr::from([0u8; 4]), 80);
- let incoming = TcpListener::bind(dual)
- .or_else(|_| TcpListener::bind(ipv4))
- .await?;
-
- let server = async move {
- loop {
- let key_auth = Arc::clone(&key_auth);
- let path = Arc::clone(&path);
- match incoming.accept().await {
- Ok((tcp, _)) => {
- let io = TokioIo::new(tcp);
- let service = service_fn(move |request| {
- standalone_respond(
- request,
- Arc::clone(&path),
- Arc::clone(&key_auth),
- )
- });
-
- tokio::task::spawn(async move {
- if let Err(err) =
- http1::Builder::new().serve_connection(io, service).await
- {
- println!("Error serving connection: {err:?}");
- }
- });
- }
- Err(err) => println!("Error accepting connection: {err:?}"),
- }
- }
- };
- let (future, abort) = futures::future::abortable(server);
- self.abort_handle = Some(abort);
- tokio::spawn(future);
-
- Ok(challenge.url.as_str())
- })
- }
-
- fn teardown<'fut, 'a: 'fut, 'b: 'fut, 'c: 'fut, 'd: 'fut>(
- &'a mut self,
- _client: &'b mut AcmeClient,
- _authorization: &'c Authorization,
- _domain: &'d AcmeDomain,
- _task: Arc<WorkerTask>,
- ) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send + 'fut>> {
- Box::pin(async move {
- if let Some(abort) = self.abort_handle.take() {
- abort.abort();
- }
- Ok(())
- })
- }
-}
diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.rs
index 47ff8de5..73401c41 100644
--- a/src/api2/node/certificates.rs
+++ b/src/api2/node/certificates.rs
@@ -1,14 +1,11 @@
-use std::sync::Arc;
-use std::time::Duration;
-
use anyhow::{bail, format_err, Error};
use openssl::pkey::PKey;
use openssl::x509::X509;
use serde::{Deserialize, Serialize};
-use tracing::{info, warn};
+use tracing::info;
use pbs_api_types::{NODE_SCHEMA, PRIV_SYS_MODIFY};
-use proxmox_acme::async_client::AcmeClient;
+use proxmox_acme_api::AcmeDomain;
use proxmox_rest_server::WorkerTask;
use proxmox_router::list_subdirs_api_method;
use proxmox_router::SubdirMap;
@@ -18,8 +15,6 @@ use proxmox_schema::api;
use pbs_buildcfg::configdir;
use pbs_tools::cert;
-use crate::api2::types::AcmeDomain;
-use crate::config::node::NodeConfig;
use crate::server::send_certificate_renewal_mail;
pub const ROUTER: Router = Router::new()
@@ -268,193 +263,6 @@ pub async fn delete_custom_certificate() -> Result<(), Error> {
Ok(())
}
-struct OrderedCertificate {
- certificate: hyper::body::Bytes,
- private_key_pem: Vec<u8>,
-}
-
-async fn order_certificate(
- worker: Arc<WorkerTask>,
- node_config: &NodeConfig,
-) -> Result<Option<OrderedCertificate>, Error> {
- use proxmox_acme::authorization::Status;
- use proxmox_acme::order::Identifier;
-
- let domains = node_config.acme_domains().try_fold(
- Vec::<AcmeDomain>::new(),
- |mut acc, domain| -> Result<_, Error> {
- let mut domain = domain?;
- domain.domain.make_ascii_lowercase();
- if let Some(alias) = &mut domain.alias {
- alias.make_ascii_lowercase();
- }
- acc.push(domain);
- Ok(acc)
- },
- )?;
-
- let get_domain_config = |domain: &str| {
- domains
- .iter()
- .find(|d| d.domain == domain)
- .ok_or_else(|| format_err!("no config for domain '{}'", domain))
- };
-
- if domains.is_empty() {
- info!("No domains configured to be ordered from an ACME server.");
- return Ok(None);
- }
-
- let (plugins, _) = crate::config::acme::plugin::config()?;
-
- let mut acme = node_config.acme_client().await?;
-
- info!("Placing ACME order");
- let order = acme
- .new_order(domains.iter().map(|d| d.domain.to_ascii_lowercase()))
- .await?;
- info!("Order URL: {}", order.location);
-
- let identifiers: Vec<String> = order
- .data
- .identifiers
- .iter()
- .map(|identifier| match identifier {
- Identifier::Dns(domain) => domain.clone(),
- })
- .collect();
-
- for auth_url in &order.data.authorizations {
- info!("Getting authorization details from '{auth_url}'");
- let mut auth = acme.get_authorization(auth_url).await?;
-
- let domain = match &mut auth.identifier {
- Identifier::Dns(domain) => domain.to_ascii_lowercase(),
- };
-
- if auth.status == Status::Valid {
- info!("{domain} is already validated!");
- continue;
- }
-
- info!("The validation for {domain} is pending");
- let domain_config: &AcmeDomain = get_domain_config(&domain)?;
- let plugin_id = domain_config.plugin.as_deref().unwrap_or("standalone");
- let mut plugin_cfg = crate::acme::get_acme_plugin(&plugins, plugin_id)?
- .ok_or_else(|| format_err!("plugin '{plugin_id}' for domain '{domain}' not found!"))?;
-
- info!("Setting up validation plugin");
- let validation_url = plugin_cfg
- .setup(&mut acme, &auth, domain_config, Arc::clone(&worker))
- .await?;
-
- let result = request_validation(&mut acme, auth_url, validation_url).await;
-
- if let Err(err) = plugin_cfg
- .teardown(&mut acme, &auth, domain_config, Arc::clone(&worker))
- .await
- {
- warn!("Failed to teardown plugin '{plugin_id}' for domain '{domain}' - {err}");
- }
-
- result?;
- }
-
- info!("All domains validated");
- info!("Creating CSR");
-
- let csr = proxmox_acme::util::Csr::generate(&identifiers, &Default::default())?;
- let mut finalize_error_cnt = 0u8;
- let order_url = &order.location;
- let mut order;
- loop {
- use proxmox_acme::order::Status;
-
- order = acme.get_order(order_url).await?;
-
- match order.status {
- Status::Pending => {
- info!("still pending, trying to finalize anyway");
- let finalize = order
- .finalize
- .as_deref()
- .ok_or_else(|| format_err!("missing 'finalize' URL in order"))?;
- if let Err(err) = acme.finalize(finalize, &csr.data).await {
- if finalize_error_cnt >= 5 {
- return Err(err);
- }
-
- finalize_error_cnt += 1;
- }
- tokio::time::sleep(Duration::from_secs(5)).await;
- }
- Status::Ready => {
- info!("order is ready, finalizing");
- let finalize = order
- .finalize
- .as_deref()
- .ok_or_else(|| format_err!("missing 'finalize' URL in order"))?;
- acme.finalize(finalize, &csr.data).await?;
- tokio::time::sleep(Duration::from_secs(5)).await;
- }
- Status::Processing => {
- info!("still processing, trying again in 30 seconds");
- tokio::time::sleep(Duration::from_secs(30)).await;
- }
- Status::Valid => {
- info!("valid");
- break;
- }
- other => bail!("order status: {:?}", other),
- }
- }
-
- info!("Downloading certificate");
- let certificate = acme
- .get_certificate(
- order
- .certificate
- .as_deref()
- .ok_or_else(|| format_err!("missing certificate url in finalized order"))?,
- )
- .await?;
-
- Ok(Some(OrderedCertificate {
- certificate,
- private_key_pem: csr.private_key_pem,
- }))
-}
-
-async fn request_validation(
- acme: &mut AcmeClient,
- auth_url: &str,
- validation_url: &str,
-) -> Result<(), Error> {
- info!("Triggering validation");
- acme.request_challenge_validation(validation_url).await?;
-
- info!("Sleeping for 5 seconds");
- tokio::time::sleep(Duration::from_secs(5)).await;
-
- loop {
- use proxmox_acme::authorization::Status;
-
- let auth = acme.get_authorization(auth_url).await?;
- match auth.status {
- Status::Pending => {
- info!("Status is still 'pending', trying again in 10 seconds");
- tokio::time::sleep(Duration::from_secs(10)).await;
- }
- Status::Valid => return Ok(()),
- other => bail!(
- "validating challenge '{}' failed - status: {:?}",
- validation_url,
- other
- ),
- }
- }
-}
-
#[api(
input: {
properties: {
@@ -524,9 +332,30 @@ fn spawn_certificate_worker(
let auth_id = rpcenv.get_auth_id().unwrap();
+ let acme_config = if let Some(cfg) = node_config.acme_config().transpose()? {
+ cfg
+ } else {
+ proxmox_acme_api::parse_acme_config_string("account=default")?
+ };
+
+ let domains = node_config.acme_domains().try_fold(
+ Vec::<AcmeDomain>::new(),
+ |mut acc, domain| -> Result<_, Error> {
+ let mut domain = domain?;
+ domain.domain.make_ascii_lowercase();
+ if let Some(alias) = &mut domain.alias {
+ alias.make_ascii_lowercase();
+ }
+ acc.push(domain);
+ Ok(acc)
+ },
+ )?;
+
WorkerTask::spawn(name, None, auth_id, true, move |worker| async move {
let work = || async {
- if let Some(cert) = order_certificate(worker, &node_config).await? {
+ if let Some(cert) =
+ proxmox_acme_api::order_certificate(worker, &acme_config, &domains).await?
+ {
crate::config::set_proxy_certificate(&cert.certificate, &cert.private_key_pem)?;
crate::server::reload_proxy_certificate().await?;
}
@@ -562,16 +391,20 @@ pub fn revoke_acme_cert(rpcenv: &mut dyn RpcEnvironment) -> Result<String, Error
let auth_id = rpcenv.get_auth_id().unwrap();
+ let acme_config = if let Some(cfg) = node_config.acme_config().transpose()? {
+ cfg
+ } else {
+ proxmox_acme_api::parse_acme_config_string("account=default")?
+ };
+
WorkerTask::spawn(
"acme-revoke-cert",
None,
auth_id,
true,
move |_worker| async move {
- info!("Loading ACME account");
- let mut acme = node_config.acme_client().await?;
info!("Revoking old certificate");
- acme.revoke_certificate(cert_pem.as_bytes(), None).await?;
+ proxmox_acme_api::revoke_certificate(&acme_config, &cert_pem.as_bytes()).await?;
info!("Deleting certificate and regenerating a self-signed one");
delete_custom_certificate().await?;
Ok(())
diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
deleted file mode 100644
index 0ff496b6..00000000
--- a/src/api2/types/acme.rs
+++ /dev/null
@@ -1,73 +0,0 @@
-use serde::{Deserialize, Serialize};
-use serde_json::Value;
-
-use pbs_api_types::{DNS_ALIAS_FORMAT, DNS_NAME_FORMAT, PROXMOX_SAFE_ID_FORMAT};
-use proxmox_schema::{api, ApiStringFormat, ApiType, Schema, StringSchema};
-
-#[api(
- properties: {
- "domain": { format: &DNS_NAME_FORMAT },
- "alias": {
- optional: true,
- format: &DNS_ALIAS_FORMAT,
- },
- "plugin": {
- optional: true,
- format: &PROXMOX_SAFE_ID_FORMAT,
- },
- },
- default_key: "domain",
-)]
-#[derive(Deserialize, Serialize)]
-/// A domain entry for an ACME certificate.
-pub struct AcmeDomain {
- /// The domain to certify for.
- pub domain: String,
-
- /// The domain to use for challenges instead of the default acme challenge domain.
- ///
- /// This is useful if you use CNAME entries to redirect `_acme-challenge.*` domains to a
- /// different DNS server.
- #[serde(skip_serializing_if = "Option::is_none")]
- pub alias: Option<String>,
-
- /// The plugin to use to validate this domain.
- ///
- /// Empty means standalone HTTP validation is used.
- #[serde(skip_serializing_if = "Option::is_none")]
- pub plugin: Option<String>,
-}
-
-pub const ACME_DOMAIN_PROPERTY_SCHEMA: Schema =
- StringSchema::new("ACME domain configuration string")
- .format(&ApiStringFormat::PropertyString(&AcmeDomain::API_SCHEMA))
- .schema();
-
-#[api(
- properties: {
- schema: {
- type: Object,
- additional_properties: true,
- properties: {},
- },
- type: {
- type: String,
- },
- },
-)]
-#[derive(Serialize)]
-/// Schema for an ACME challenge plugin.
-pub struct AcmeChallengeSchema {
- /// Plugin ID.
- pub id: String,
-
- /// Human readable name, falls back to id.
- pub name: String,
-
- /// Plugin Type.
- #[serde(rename = "type")]
- pub ty: &'static str,
-
- /// The plugin's parameter schema.
- pub schema: Value,
-}
diff --git a/src/api2/types/mod.rs b/src/api2/types/mod.rs
index afc34b30..34193685 100644
--- a/src/api2/types/mod.rs
+++ b/src/api2/types/mod.rs
@@ -4,9 +4,6 @@ use anyhow::bail;
use proxmox_schema::*;
-mod acme;
-pub use acme::*;
-
// File names: may not contain slashes, may not start with "."
pub const FILENAME_FORMAT: ApiStringFormat = ApiStringFormat::VerifyFn(|name| {
if name.starts_with('.') {
diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
index 01ab6223..73486df9 100644
--- a/src/config/acme/mod.rs
+++ b/src/config/acme/mod.rs
@@ -5,12 +5,10 @@ use anyhow::Error;
use serde_json::Value;
use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
-use proxmox_acme_api::{AcmeAccountName, KnownAcmeDirectory, KNOWN_ACME_DIRECTORIES};
+use proxmox_acme_api::{AcmeAccountName, AcmeChallengeSchema};
use proxmox_sys::error::SysError;
use proxmox_sys::fs::{file_read_string, CreateOptions};
-use crate::api2::types::AcmeChallengeSchema;
-
pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
@@ -34,8 +32,6 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
create_acme_subdir(ACME_DIR)
}
-pub const DEFAULT_ACME_DIRECTORY_ENTRY: &KnownAcmeDirectory = &KNOWN_ACME_DIRECTORIES[0];
-
pub fn foreach_acme_account<F>(mut func: F) -> Result<(), Error>
where
F: FnMut(AcmeAccountName) -> ControlFlow<Result<(), Error>>,
@@ -79,7 +75,7 @@ pub fn load_dns_challenge_schema() -> Result<Vec<AcmeChallengeSchema>, Error> {
.and_then(Value::as_str)
.unwrap_or(id)
.to_owned(),
- ty: "dns",
+ ty: "dns".into(),
schema: schema.to_owned(),
})
.collect())
diff --git a/src/config/acme/plugin.rs b/src/config/acme/plugin.rs
index 8ce852ec..4b4a216e 100644
--- a/src/config/acme/plugin.rs
+++ b/src/config/acme/plugin.rs
@@ -1,104 +1,16 @@
use std::sync::LazyLock;
use anyhow::Error;
-use serde::{Deserialize, Serialize};
use serde_json::Value;
-use pbs_api_types::PROXMOX_SAFE_ID_FORMAT;
-use proxmox_schema::{api, ApiType, Schema, StringSchema, Updater};
+use proxmox_acme_api::{DnsPlugin, StandalonePlugin, PLUGIN_ID_SCHEMA};
+use proxmox_schema::{ApiType, Schema};
use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
use pbs_config::{open_backup_lockfile, BackupLockGuard};
-pub const PLUGIN_ID_SCHEMA: Schema = StringSchema::new("ACME Challenge Plugin ID.")
- .format(&PROXMOX_SAFE_ID_FORMAT)
- .min_length(1)
- .max_length(32)
- .schema();
-
pub static CONFIG: LazyLock<SectionConfig> = LazyLock::new(init);
-#[api(
- properties: {
- id: { schema: PLUGIN_ID_SCHEMA },
- },
-)]
-#[derive(Deserialize, Serialize)]
-/// Standalone ACME Plugin for the http-1 challenge.
-pub struct StandalonePlugin {
- /// Plugin ID.
- id: String,
-}
-
-impl Default for StandalonePlugin {
- fn default() -> Self {
- Self {
- id: "standalone".to_string(),
- }
- }
-}
-
-#[api(
- properties: {
- id: { schema: PLUGIN_ID_SCHEMA },
- disable: {
- optional: true,
- default: false,
- },
- "validation-delay": {
- default: 30,
- optional: true,
- minimum: 0,
- maximum: 2 * 24 * 60 * 60,
- },
- },
-)]
-/// DNS ACME Challenge Plugin core data.
-#[derive(Deserialize, Serialize, Updater)]
-#[serde(rename_all = "kebab-case")]
-pub struct DnsPluginCore {
- /// Plugin ID.
- #[updater(skip)]
- pub id: String,
-
- /// DNS API Plugin Id.
- pub api: String,
-
- /// Extra delay in seconds to wait before requesting validation.
- ///
- /// Allows to cope with long TTL of DNS records.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- pub validation_delay: Option<u32>,
-
- /// Flag to disable the config.
- #[serde(skip_serializing_if = "Option::is_none", default)]
- pub disable: Option<bool>,
-}
-
-#[api(
- properties: {
- core: { type: DnsPluginCore },
- },
-)]
-/// DNS ACME Challenge Plugin.
-#[derive(Deserialize, Serialize)]
-#[serde(rename_all = "kebab-case")]
-pub struct DnsPlugin {
- #[serde(flatten)]
- pub core: DnsPluginCore,
-
- // We handle this property separately in the API calls.
- /// DNS plugin data (base64url encoded without padding).
- #[serde(with = "proxmox_serde::string_as_base64url_nopad")]
- pub data: String,
-}
-
-impl DnsPlugin {
- pub fn decode_data(&self, output: &mut Vec<u8>) -> Result<(), Error> {
- Ok(proxmox_base64::url::decode_to_vec(&self.data, output)?)
- }
-}
-
fn init() -> SectionConfig {
let mut config = SectionConfig::new(&PLUGIN_ID_SCHEMA);
diff --git a/src/config/node.rs b/src/config/node.rs
index e4b66a20..6865b815 100644
--- a/src/config/node.rs
+++ b/src/config/node.rs
@@ -9,14 +9,14 @@ use pbs_api_types::{
OPENSSL_CIPHERS_TLS_1_3_SCHEMA,
};
use proxmox_acme::async_client::AcmeClient;
-use proxmox_acme_api::AcmeAccountName;
+use proxmox_acme_api::{AcmeAccountName, AcmeConfig, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA};
use proxmox_http::ProxyConfig;
use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
use pbs_buildcfg::configdir;
use pbs_config::{open_backup_lockfile, BackupLockGuard};
-use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
+use crate::api2::types::HTTP_PROXY_SCHEMA;
const CONF_FILE: &str = configdir!("/node.cfg");
const LOCK_FILE: &str = configdir!("/.node.lck");
@@ -43,20 +43,6 @@ pub fn save_config(config: &NodeConfig) -> Result<(), Error> {
pbs_config::replace_backup_config(CONF_FILE, &raw)
}
-#[api(
- properties: {
- account: { type: AcmeAccountName },
- }
-)]
-#[derive(Deserialize, Serialize)]
-/// The ACME configuration.
-///
-/// Currently only contains the name of the account use.
-pub struct AcmeConfig {
- /// Account to use to acquire ACME certificates.
- account: AcmeAccountName,
-}
-
/// All available languages in Proxmox. Taken from proxmox-i18n repository.
/// pt_BR, zh_CN, and zh_TW use the same case in the translation files.
// TODO: auto-generate from available translations
@@ -242,7 +228,7 @@ impl NodeConfig {
pub async fn acme_client(&self) -> Result<AcmeClient, Error> {
let account = if let Some(cfg) = self.acme_config().transpose()? {
- cfg.account
+ AcmeAccountName::from_string(cfg.account)?
} else {
AcmeAccountName::from_string("default".to_string())? // should really not happen
};
diff --git a/src/lib.rs b/src/lib.rs
index 8633378c..828f5842 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -27,8 +27,6 @@ pub(crate) mod auth;
pub mod tape;
-pub mod acme;
-
pub mod client_helpers;
pub mod traffic_control_cache;
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 7%]
* [pbs-devel] [PATCH proxmox v5 2/4] acme: introduce http_status module
2026-01-08 11:26 11% [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2026-01-08 11:26 10% ` [pbs-devel] [PATCH proxmox v5 1/4] acme: reduce visibility of Request type Samuel Rufinatscha
@ 2026-01-08 11:26 15% ` Samuel Rufinatscha
2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-08 11:26 14% ` [pbs-devel] [PATCH proxmox v5 3/4] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (7 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-08 11:26 UTC (permalink / raw)
To: pbs-devel
Introduce an internal http_status module with the common ACME HTTP
response codes, and replace use of crate::request::CREATED as well as
direct numeric status code usages.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme/src/account.rs | 8 ++++----
proxmox-acme/src/async_client.rs | 4 ++--
proxmox-acme/src/lib.rs | 2 ++
proxmox-acme/src/request.rs | 11 ++++++++++-
4 files changed, 18 insertions(+), 7 deletions(-)
diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
index d8eb3e73..ea1a3c60 100644
--- a/proxmox-acme/src/account.rs
+++ b/proxmox-acme/src/account.rs
@@ -84,7 +84,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::request::CREATED,
+ expected: crate::http_status::CREATED,
};
Ok(NewOrder::new(request))
@@ -106,7 +106,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: crate::http_status::OK,
})
}
@@ -131,7 +131,7 @@ impl Account {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: 200,
+ expected: crate::http_status::OK,
})
}
@@ -321,7 +321,7 @@ impl AccountCreator {
method: "POST",
content_type: crate::request::JSON_CONTENT_TYPE,
body,
- expected: crate::request::CREATED,
+ expected: crate::http_status::CREATED,
})
}
diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
index 2ff3ba22..043648bb 100644
--- a/proxmox-acme/src/async_client.rs
+++ b/proxmox-acme/src/async_client.rs
@@ -498,7 +498,7 @@ impl AcmeClient {
method: "GET",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: crate::http_status::OK,
},
nonce,
)
@@ -550,7 +550,7 @@ impl AcmeClient {
method: "HEAD",
content_type: "",
body: String::new(),
- expected: 200,
+ expected: crate::http_status::OK,
},
nonce,
)
diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
index 6722030c..6051a025 100644
--- a/proxmox-acme/src/lib.rs
+++ b/proxmox-acme/src/lib.rs
@@ -70,6 +70,8 @@ pub use order::Order;
#[cfg(feature = "impl")]
pub use order::NewOrder;
#[cfg(feature = "impl")]
+pub(crate) use request::http_status;
+#[cfg(feature = "impl")]
pub use request::ErrorResponse;
/// Header name for nonces.
diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
index dadfc5af..341ce53e 100644
--- a/proxmox-acme/src/request.rs
+++ b/proxmox-acme/src/request.rs
@@ -1,7 +1,6 @@
use serde::Deserialize;
pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
-pub(crate) const CREATED: u16 = 201;
/// A request which should be performed on the ACME provider.
pub(crate) struct Request {
@@ -21,6 +20,16 @@ pub(crate) struct Request {
pub(crate) expected: u16,
}
+/// Common HTTP status codes used in ACME responses.
+pub(crate) mod http_status {
+ /// 200 OK
+ pub(crate) const OK: u16 = 200;
+ /// 201 Created
+ pub(crate) const CREATED: u16 = 201;
+ /// 204 No Content
+ pub(crate) const NO_CONTENT: u16 = 204;
+}
+
/// An ACME error response contains a specially formatted type string, and can optionally
/// contain textual details and a set of sub problems.
#[derive(Clone, Debug, Deserialize)]
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox-backup v5 2/5] acme: include proxmox-acme-api dependency
2026-01-08 11:26 11% [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (4 preceding siblings ...)
2026-01-08 11:26 13% ` [pbs-devel] [PATCH proxmox-backup v5 1/5] acme: clean up ACME-related imports Samuel Rufinatscha
@ 2026-01-08 11:26 15% ` Samuel Rufinatscha
2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-08 11:26 6% ` [pbs-devel] [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient Samuel Rufinatscha
` (3 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-08 11:26 UTC (permalink / raw)
To: pbs-devel
PBS currently uses its own ACME client and API logic, while PDM uses the
factored out proxmox-acme and proxmox-acme-api crates. This duplication
risks differences in behaviour and requires ACME maintenance in two
places. This patch is part of a series to move PBS over to the shared
ACME stack.
Changes:
- Add proxmox-acme-api with the "impl" feature as a dependency.
- Initialize proxmox_acme_api in proxmox-backup- api, manager and proxy.
* Inits PBS config dir /acme as proxmox ACME directory
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
Cargo.toml | 3 +++
src/bin/proxmox-backup-api.rs | 2 ++
src/bin/proxmox-backup-manager.rs | 2 ++
src/bin/proxmox-backup-proxy.rs | 1 +
4 files changed, 8 insertions(+)
diff --git a/Cargo.toml b/Cargo.toml
index 1aa57ae5..feae351d 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -101,6 +101,7 @@ pbs-api-types = "1.0.8"
# other proxmox crates
pathpatterns = "1"
proxmox-acme = "1"
+proxmox-acme-api = { version = "1", features = [ "impl" ] }
pxar = "1"
# PBS workspace
@@ -251,6 +252,7 @@ pbs-api-types.workspace = true
# in their respective repo
proxmox-acme.workspace = true
+proxmox-acme-api.workspace = true
pxar.workspace = true
# proxmox-backup workspace/internal crates
@@ -269,6 +271,7 @@ proxmox-rrd-api-types.workspace = true
[patch.crates-io]
#pbs-api-types = { path = "../proxmox/pbs-api-types" }
#proxmox-acme = { path = "../proxmox/proxmox-acme" }
+#proxmox-acme-api = { path = "../proxmox/proxmox-acme-api" }
#proxmox-api-macro = { path = "../proxmox/proxmox-api-macro" }
#proxmox-apt = { path = "../proxmox/proxmox-apt" }
#proxmox-apt-api-types = { path = "../proxmox/proxmox-apt-api-types" }
diff --git a/src/bin/proxmox-backup-api.rs b/src/bin/proxmox-backup-api.rs
index 417e9e97..d0091dca 100644
--- a/src/bin/proxmox-backup-api.rs
+++ b/src/bin/proxmox-backup-api.rs
@@ -14,6 +14,7 @@ use proxmox_rest_server::{ApiConfig, RestServer};
use proxmox_router::RpcEnvironmentType;
use proxmox_sys::fs::CreateOptions;
+use pbs_buildcfg::configdir;
use proxmox_backup::auth_helpers::*;
use proxmox_backup::config;
use proxmox_backup::server::auth::check_pbs_auth;
@@ -78,6 +79,7 @@ async fn run() -> Result<(), Error> {
let mut command_sock = proxmox_daemon::command_socket::CommandSocket::new(backup_user.gid);
proxmox_product_config::init(backup_user.clone(), pbs_config::priv_user()?);
+ proxmox_acme_api::init(configdir!("/acme"), true)?;
let dir_opts = CreateOptions::new()
.owner(backup_user.uid)
diff --git a/src/bin/proxmox-backup-manager.rs b/src/bin/proxmox-backup-manager.rs
index f8365070..30bc8da9 100644
--- a/src/bin/proxmox-backup-manager.rs
+++ b/src/bin/proxmox-backup-manager.rs
@@ -19,6 +19,7 @@ use proxmox_router::{cli::*, RpcEnvironment};
use proxmox_schema::api;
use proxmox_sys::fs::CreateOptions;
+use pbs_buildcfg::configdir;
use pbs_client::{display_task_log, view_task_result};
use pbs_config::sync;
use pbs_tools::json::required_string_param;
@@ -667,6 +668,7 @@ async fn run() -> Result<(), Error> {
.init()?;
proxmox_backup::server::notifications::init()?;
proxmox_product_config::init(pbs_config::backup_user()?, pbs_config::priv_user()?);
+ proxmox_acme_api::init(configdir!("/acme"), false)?;
let cmd_def = CliCommandMap::new()
.insert("acl", acl_commands())
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 870208fe..eea44a7d 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -188,6 +188,7 @@ async fn run() -> Result<(), Error> {
proxmox_backup::server::notifications::init()?;
metric_collection::init()?;
proxmox_product_config::init(pbs_config::backup_user()?, pbs_config::priv_user()?);
+ proxmox_acme_api::init(configdir!("/acme"), false)?;
let mut indexpath = PathBuf::from(pbs_buildcfg::JS_DIR);
indexpath.push("index.hbs");
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 15%]
* [pbs-devel] [PATCH proxmox v5 1/4] acme: reduce visibility of Request type
2026-01-08 11:26 11% [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
@ 2026-01-08 11:26 10% ` Samuel Rufinatscha
2026-01-13 13:46 5% ` Fabian Grünbichler
2026-01-08 11:26 15% ` [pbs-devel] [PATCH proxmox v5 2/4] acme: introduce http_status module Samuel Rufinatscha
` (8 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-08 11:26 UTC (permalink / raw)
To: pbs-devel
Currently, the low-level ACME Request type is publicly exposed, even
though users are expected to go through AcmeClient and
proxmox-acme-api handlers. This patch reduces visibility so that
the Request type and related fields/methods are crate-internal only.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
proxmox-acme/src/account.rs | 94 ++-----------------------------
proxmox-acme/src/async_client.rs | 2 +-
proxmox-acme/src/authorization.rs | 30 ----------
proxmox-acme/src/client.rs | 6 +-
proxmox-acme/src/lib.rs | 4 --
proxmox-acme/src/order.rs | 2 +-
proxmox-acme/src/request.rs | 12 ++--
7 files changed, 16 insertions(+), 134 deletions(-)
diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
index f763c1e9..d8eb3e73 100644
--- a/proxmox-acme/src/account.rs
+++ b/proxmox-acme/src/account.rs
@@ -8,12 +8,11 @@ use openssl::pkey::{PKey, Private};
use serde::{Deserialize, Serialize};
use serde_json::Value;
-use crate::authorization::{Authorization, GetAuthorization};
use crate::b64u;
use crate::directory::Directory;
use crate::jws::Jws;
use crate::key::{Jwk, PublicKey};
-use crate::order::{NewOrder, Order, OrderData};
+use crate::order::{NewOrder, OrderData};
use crate::request::Request;
use crate::types::{AccountData, AccountStatus, ExternalAccountBinding};
use crate::Error;
@@ -92,7 +91,7 @@ impl Account {
}
/// Prepare a "POST-as-GET" request to fetch data. Low level helper.
- pub fn get_request(&self, url: &str, nonce: &str) -> Result<Request, Error> {
+ pub(crate) fn get_request(&self, url: &str, nonce: &str) -> Result<Request, Error> {
let key = PKey::private_key_from_pem(self.private_key.as_bytes())?;
let body = serde_json::to_string(&Jws::new_full(
&key,
@@ -112,7 +111,7 @@ impl Account {
}
/// Prepare a JSON POST request. Low level helper.
- pub fn post_request<T: Serialize>(
+ pub(crate) fn post_request<T: Serialize>(
&self,
url: &str,
nonce: &str,
@@ -136,31 +135,6 @@ impl Account {
})
}
- /// Prepare a JSON POST request.
- fn post_request_raw_payload(
- &self,
- url: &str,
- nonce: &str,
- payload: String,
- ) -> Result<Request, Error> {
- let key = PKey::private_key_from_pem(self.private_key.as_bytes())?;
- let body = serde_json::to_string(&Jws::new_full(
- &key,
- Some(self.location.clone()),
- url.to_owned(),
- nonce.to_owned(),
- payload,
- )?)?;
-
- Ok(Request {
- url: url.to_owned(),
- method: "POST",
- content_type: crate::request::JSON_CONTENT_TYPE,
- body,
- expected: 200,
- })
- }
-
/// Get the "key authorization" for a token.
pub fn key_authorization(&self, token: &str) -> Result<String, Error> {
let key = PKey::private_key_from_pem(self.private_key.as_bytes())?;
@@ -176,64 +150,6 @@ impl Account {
Ok(b64u::encode(digest))
}
- /// Prepare a request to update account data.
- ///
- /// This is a rather low level interface. You should know what you're doing.
- pub fn update_account_request<T: Serialize>(
- &self,
- nonce: &str,
- data: &T,
- ) -> Result<Request, Error> {
- self.post_request(&self.location, nonce, data)
- }
-
- /// Prepare a request to deactivate this account.
- pub fn deactivate_account_request<T: Serialize>(&self, nonce: &str) -> Result<Request, Error> {
- self.post_request_raw_payload(
- &self.location,
- nonce,
- r#"{"status":"deactivated"}"#.to_string(),
- )
- }
-
- /// Prepare a request to query an Authorization for an Order.
- ///
- /// Returns `Ok(None)` if `auth_index` is out of out of range. You can query the number of
- /// authorizations from via [`Order::authorization_len`] or by manually inspecting its
- /// `.data.authorization` vector.
- pub fn get_authorization(
- &self,
- order: &Order,
- auth_index: usize,
- nonce: &str,
- ) -> Result<Option<GetAuthorization>, Error> {
- match order.authorization(auth_index) {
- None => Ok(None),
- Some(url) => Ok(Some(GetAuthorization::new(self.get_request(url, nonce)?))),
- }
- }
-
- /// Prepare a request to validate a Challenge from an Authorization.
- ///
- /// Returns `Ok(None)` if `challenge_index` is out of out of range. The challenge count is
- /// available by inspecting the [`Authorization::challenges`] vector.
- ///
- /// This returns a raw `Request` since validation takes some time and the `Authorization`
- /// object has to be re-queried and its `status` inspected.
- pub fn validate_challenge(
- &self,
- authorization: &Authorization,
- challenge_index: usize,
- nonce: &str,
- ) -> Result<Option<Request>, Error> {
- match authorization.challenges.get(challenge_index) {
- None => Ok(None),
- Some(challenge) => self
- .post_request_raw_payload(&challenge.url, nonce, "{}".to_string())
- .map(Some),
- }
- }
-
/// Prepare a request to revoke a certificate.
///
/// The certificate can be either PEM or DER formatted.
@@ -274,7 +190,7 @@ pub struct CertificateRevocation<'a> {
impl CertificateRevocation<'_> {
/// Create the revocation request using the specified nonce for the given directory.
- pub fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
+ pub(crate) fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
let revoke_cert = directory.data.revoke_cert.as_ref().ok_or_else(|| {
Error::Custom("no 'revokeCert' URL specified by provider".to_string())
})?;
@@ -364,7 +280,7 @@ impl AccountCreator {
/// the resulting request.
/// Changing the private key between using the request and passing the response to
/// [`response`](AccountCreator::response()) will render the account unusable!
- pub fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
+ pub(crate) fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
let key = self.key.as_deref().ok_or(Error::MissingKey)?;
let url = directory.new_account_url().ok_or_else(|| {
Error::Custom("no 'newAccount' URL specified by provider".to_string())
diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
index dc755fb9..2ff3ba22 100644
--- a/proxmox-acme/src/async_client.rs
+++ b/proxmox-acme/src/async_client.rs
@@ -10,7 +10,7 @@ use proxmox_http::{client::Client, Body};
use crate::account::AccountCreator;
use crate::order::{Order, OrderData};
-use crate::Request as AcmeRequest;
+use crate::request::Request as AcmeRequest;
use crate::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
/// A non-blocking Acme client using tokio/hyper.
diff --git a/proxmox-acme/src/authorization.rs b/proxmox-acme/src/authorization.rs
index 28bc1b4b..7027381a 100644
--- a/proxmox-acme/src/authorization.rs
+++ b/proxmox-acme/src/authorization.rs
@@ -6,8 +6,6 @@ use serde::{Deserialize, Serialize};
use serde_json::Value;
use crate::order::Identifier;
-use crate::request::Request;
-use crate::Error;
/// Status of an [`Authorization`].
#[derive(Clone, Copy, Debug, Eq, PartialEq, Deserialize, Serialize)]
@@ -132,31 +130,3 @@ impl Challenge {
fn is_false(b: &bool) -> bool {
!*b
}
-
-/// Represents an in-flight query for an authorization.
-///
-/// This is created via [`Account::get_authorization`](crate::Account::get_authorization()).
-pub struct GetAuthorization {
- //order: OrderData,
- /// The request to send to the ACME provider. This is wrapped in an option in order to allow
- /// moving it out instead of copying the contents.
- ///
- /// When generated via [`Account::get_authorization`](crate::Account::get_authorization()),
- /// this is guaranteed to be `Some`.
- ///
- /// The response should be passed to the the [`response`](GetAuthorization::response()) method.
- pub request: Option<Request>,
-}
-
-impl GetAuthorization {
- pub(crate) fn new(request: Request) -> Self {
- Self {
- request: Some(request),
- }
- }
-
- /// Deal with the response we got from the server.
- pub fn response(self, response_body: &[u8]) -> Result<Authorization, Error> {
- Ok(serde_json::from_slice(response_body)?)
- }
-}
diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
index 931f7245..5c812567 100644
--- a/proxmox-acme/src/client.rs
+++ b/proxmox-acme/src/client.rs
@@ -7,8 +7,8 @@ use serde::{Deserialize, Serialize};
use crate::b64u;
use crate::error;
use crate::order::OrderData;
-use crate::request::ErrorResponse;
-use crate::{Account, Authorization, Challenge, Directory, Error, Order, Request};
+use crate::request::{ErrorResponse, Request};
+use crate::{Account, Authorization, Challenge, Directory, Error, Order};
macro_rules! format_err {
($($fmt:tt)*) => { Error::Client(format!($($fmt)*)) };
@@ -564,7 +564,7 @@ impl Client {
}
/// Low-level API to run an n API request. This automatically updates the current nonce!
- pub fn run_request(&mut self, request: Request) -> Result<HttpResponse, Error> {
+ pub(crate) fn run_request(&mut self, request: Request) -> Result<HttpResponse, Error> {
self.inner.run_request(request)
}
diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
index df722629..6722030c 100644
--- a/proxmox-acme/src/lib.rs
+++ b/proxmox-acme/src/lib.rs
@@ -66,10 +66,6 @@ pub use error::Error;
#[doc(inline)]
pub use order::Order;
-#[cfg(feature = "impl")]
-#[doc(inline)]
-pub use request::Request;
-
// we don't inline these:
#[cfg(feature = "impl")]
pub use order::NewOrder;
diff --git a/proxmox-acme/src/order.rs b/proxmox-acme/src/order.rs
index b6551004..432a81a4 100644
--- a/proxmox-acme/src/order.rs
+++ b/proxmox-acme/src/order.rs
@@ -153,7 +153,7 @@ pub struct NewOrder {
//order: OrderData,
/// The request to execute to place the order. When creating a [`NewOrder`] via
/// [`Account::new_order`](crate::Account::new_order) this is guaranteed to be `Some`.
- pub request: Option<Request>,
+ pub(crate) request: Option<Request>,
}
impl NewOrder {
diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
index 78a90913..dadfc5af 100644
--- a/proxmox-acme/src/request.rs
+++ b/proxmox-acme/src/request.rs
@@ -4,21 +4,21 @@ pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
pub(crate) const CREATED: u16 = 201;
/// A request which should be performed on the ACME provider.
-pub struct Request {
+pub(crate) struct Request {
/// The complete URL to send the request to.
- pub url: String,
+ pub(crate) url: String,
/// The HTTP method name to use.
- pub method: &'static str,
+ pub(crate) method: &'static str,
/// The `Content-Type` header to pass along.
- pub content_type: &'static str,
+ pub(crate) content_type: &'static str,
/// The body to pass along with request, or an empty string.
- pub body: String,
+ pub(crate) body: String,
/// The expected status code a compliant ACME provider will return on success.
- pub expected: u16,
+ pub(crate) expected: u16,
}
/// An ACME error response contains a specially formatted type string, and can optionally
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 10%]
* [pbs-devel] [PATCH proxmox-backup v5 1/5] acme: clean up ACME-related imports
2026-01-08 11:26 11% [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (3 preceding siblings ...)
2026-01-08 11:26 17% ` [pbs-devel] [PATCH proxmox v5 4/4] acme-api: add helper to load client for an account Samuel Rufinatscha
@ 2026-01-08 11:26 13% ` Samuel Rufinatscha
2026-01-13 13:45 5% ` [pbs-devel] applied: " Fabian Grünbichler
2026-01-08 11:26 15% ` [pbs-devel] [PATCH proxmox-backup v5 2/5] acme: include proxmox-acme-api dependency Samuel Rufinatscha
` (4 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-08 11:26 UTC (permalink / raw)
To: pbs-devel
Clean up ACME-related imports to make it easier to switch to
the factored out proxmox/ ACME implementation later.
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/acme/plugin.rs | 3 +--
src/api2/config/acme.rs | 10 ++++------
src/api2/node/certificates.rs | 7 +++----
src/api2/types/acme.rs | 3 +--
src/bin/proxmox-backup-manager.rs | 12 +++++-------
src/bin/proxmox-backup-proxy.rs | 14 ++++++--------
src/config/acme/mod.rs | 3 +--
src/config/acme/plugin.rs | 2 +-
src/config/node.rs | 6 ++----
9 files changed, 24 insertions(+), 36 deletions(-)
diff --git a/src/acme/plugin.rs b/src/acme/plugin.rs
index f756e9b5..993d729b 100644
--- a/src/acme/plugin.rs
+++ b/src/acme/plugin.rs
@@ -19,11 +19,10 @@ use tokio::net::TcpListener;
use tokio::process::Command;
use proxmox_acme::{Authorization, Challenge};
+use proxmox_rest_server::WorkerTask;
use crate::acme::AcmeClient;
use crate::api2::types::AcmeDomain;
-use proxmox_rest_server::WorkerTask;
-
use crate::config::acme::plugin::{DnsPlugin, PluginData};
const PROXMOX_ACME_SH_PATH: &str = "/usr/share/proxmox-acme/proxmox-acme";
diff --git a/src/api2/config/acme.rs b/src/api2/config/acme.rs
index 35c3fb77..18671639 100644
--- a/src/api2/config/acme.rs
+++ b/src/api2/config/acme.rs
@@ -10,22 +10,20 @@ use serde::{Deserialize, Serialize};
use serde_json::{json, Value};
use tracing::{info, warn};
+use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
+use proxmox_acme::types::AccountData as AcmeAccountData;
+use proxmox_acme::Account;
+use proxmox_rest_server::WorkerTask;
use proxmox_router::{
http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
};
use proxmox_schema::{api, param_bail};
-use proxmox_acme::types::AccountData as AcmeAccountData;
-use proxmox_acme::Account;
-
-use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
-
use crate::acme::AcmeClient;
use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
use crate::config::acme::plugin::{
self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
};
-use proxmox_rest_server::WorkerTask;
pub(crate) const ROUTER: Router = Router::new()
.get(&list_subdirs_api_method!(SUBDIRS))
diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.rs
index 61ef910e..6b1d87d2 100644
--- a/src/api2/node/certificates.rs
+++ b/src/api2/node/certificates.rs
@@ -5,23 +5,22 @@ use anyhow::{bail, format_err, Error};
use openssl::pkey::PKey;
use openssl::x509::X509;
use serde::{Deserialize, Serialize};
-use tracing::info;
+use tracing::{info, warn};
+use pbs_api_types::{NODE_SCHEMA, PRIV_SYS_MODIFY};
+use proxmox_rest_server::WorkerTask;
use proxmox_router::list_subdirs_api_method;
use proxmox_router::SubdirMap;
use proxmox_router::{Permission, Router, RpcEnvironment};
use proxmox_schema::api;
-use pbs_api_types::{NODE_SCHEMA, PRIV_SYS_MODIFY};
use pbs_buildcfg::configdir;
use pbs_tools::cert;
-use tracing::warn;
use crate::acme::AcmeClient;
use crate::api2::types::AcmeDomain;
use crate::config::node::NodeConfig;
use crate::server::send_certificate_renewal_mail;
-use proxmox_rest_server::WorkerTask;
pub const ROUTER: Router = Router::new()
.get(&list_subdirs_api_method!(SUBDIRS))
diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
index 210ebdbc..8661f9e8 100644
--- a/src/api2/types/acme.rs
+++ b/src/api2/types/acme.rs
@@ -1,9 +1,8 @@
use serde::{Deserialize, Serialize};
use serde_json::Value;
-use proxmox_schema::{api, ApiStringFormat, ApiType, Schema, StringSchema};
-
use pbs_api_types::{DNS_ALIAS_FORMAT, DNS_NAME_FORMAT, PROXMOX_SAFE_ID_FORMAT};
+use proxmox_schema::{api, ApiStringFormat, ApiType, Schema, StringSchema};
#[api(
properties: {
diff --git a/src/bin/proxmox-backup-manager.rs b/src/bin/proxmox-backup-manager.rs
index d9f41353..f8365070 100644
--- a/src/bin/proxmox-backup-manager.rs
+++ b/src/bin/proxmox-backup-manager.rs
@@ -5,10 +5,6 @@ use std::str::FromStr;
use anyhow::{format_err, Error};
use serde_json::{json, Value};
-use proxmox_router::{cli::*, RpcEnvironment};
-use proxmox_schema::api;
-use proxmox_sys::fs::CreateOptions;
-
use pbs_api_types::percent_encoding::percent_encode_component;
use pbs_api_types::{
BackupNamespace, GroupFilter, RateLimitConfig, SyncDirection, SyncJobConfig, DATASTORE_SCHEMA,
@@ -18,12 +14,14 @@ use pbs_api_types::{
VERIFICATION_OUTDATED_AFTER_SCHEMA, VERIFY_JOB_READ_THREADS_SCHEMA,
VERIFY_JOB_VERIFY_THREADS_SCHEMA,
};
+use proxmox_rest_server::wait_for_local_worker;
+use proxmox_router::{cli::*, RpcEnvironment};
+use proxmox_schema::api;
+use proxmox_sys::fs::CreateOptions;
+
use pbs_client::{display_task_log, view_task_result};
use pbs_config::sync;
use pbs_tools::json::required_string_param;
-
-use proxmox_rest_server::wait_for_local_worker;
-
use proxmox_backup::api2;
use proxmox_backup::client_helpers::connect_to_localhost;
use proxmox_backup::config;
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 92a8cb3c..870208fe 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -9,27 +9,25 @@ use hyper::http::request::Parts;
use hyper::http::Response;
use hyper::StatusCode;
use hyper_util::server::graceful::GracefulShutdown;
+use openssl::ssl::SslAcceptor;
+use serde_json::{json, Value};
use tracing::level_filters::LevelFilter;
use tracing::{info, warn};
use url::form_urlencoded;
-use openssl::ssl::SslAcceptor;
-use serde_json::{json, Value};
-
use proxmox_http::Body;
use proxmox_http::RateLimiterTag;
use proxmox_lang::try_block;
+use proxmox_rest_server::{
+ cleanup_old_tasks, cookie_from_header, rotate_task_log_archive, ApiConfig, Redirector,
+ RestEnvironment, RestServer, WorkerTask,
+};
use proxmox_router::{RpcEnvironment, RpcEnvironmentType};
use proxmox_sys::fs::CreateOptions;
use proxmox_sys::logrotate::LogRotate;
use pbs_datastore::DataStore;
-use proxmox_rest_server::{
- cleanup_old_tasks, cookie_from_header, rotate_task_log_archive, ApiConfig, Redirector,
- RestEnvironment, RestServer, WorkerTask,
-};
-
use proxmox_backup::{
server::{
auth::check_pbs_auth,
diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
index 274a23fd..ac89ae5e 100644
--- a/src/config/acme/mod.rs
+++ b/src/config/acme/mod.rs
@@ -5,11 +5,10 @@ use std::path::Path;
use anyhow::{bail, format_err, Error};
use serde_json::Value;
+use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
use proxmox_sys::error::SysError;
use proxmox_sys::fs::{file_read_string, CreateOptions};
-use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
-
use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
diff --git a/src/config/acme/plugin.rs b/src/config/acme/plugin.rs
index 18e71199..8ce852ec 100644
--- a/src/config/acme/plugin.rs
+++ b/src/config/acme/plugin.rs
@@ -4,10 +4,10 @@ use anyhow::Error;
use serde::{Deserialize, Serialize};
use serde_json::Value;
+use pbs_api_types::PROXMOX_SAFE_ID_FORMAT;
use proxmox_schema::{api, ApiType, Schema, StringSchema, Updater};
use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
-use pbs_api_types::PROXMOX_SAFE_ID_FORMAT;
use pbs_config::{open_backup_lockfile, BackupLockGuard};
pub const PLUGIN_ID_SCHEMA: Schema = StringSchema::new("ACME Challenge Plugin ID.")
diff --git a/src/config/node.rs b/src/config/node.rs
index d2d6e383..253b2e36 100644
--- a/src/config/node.rs
+++ b/src/config/node.rs
@@ -4,14 +4,12 @@ use anyhow::{bail, Error};
use openssl::ssl::{SslAcceptor, SslMethod};
use serde::{Deserialize, Serialize};
-use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
-
-use proxmox_http::ProxyConfig;
-
use pbs_api_types::{
EMAIL_SCHEMA, MULTI_LINE_COMMENT_SCHEMA, OPENSSL_CIPHERS_TLS_1_2_SCHEMA,
OPENSSL_CIPHERS_TLS_1_3_SCHEMA,
};
+use proxmox_http::ProxyConfig;
+use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
use pbs_buildcfg::configdir;
use pbs_config::{open_backup_lockfile, BackupLockGuard};
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests
@ 2026-01-08 11:26 11% Samuel Rufinatscha
2026-01-08 11:26 10% ` [pbs-devel] [PATCH proxmox v5 1/4] acme: reduce visibility of Request type Samuel Rufinatscha
` (9 more replies)
0 siblings, 10 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-08 11:26 UTC (permalink / raw)
To: pbs-devel
Hi,
this series fixes account registration for ACME providers that return
HTTP 204 No Content to the newNonce request. Currently, both the PBS
ACME client and the shared ACME client in proxmox-acme only accept
HTTP 200 OK for this request. The issue was observed in PBS against a
custom ACME deployment and reported as bug #6939 [1].
## Problem
During ACME account registration, PBS first fetches an anti-replay
nonce by sending a HEAD request to the CA’s newNonce URL.
RFC 8555 §7.2 [2] states that:
* the server MUST include a Replay-Nonce header with a fresh nonce,
* the server SHOULD use status 200 OK for the HEAD request,
* the server MUST also handle GET on the same resource and may return
204 No Content with an empty body.
The reporter observed the following error message:
*ACME server responded with unexpected status code: 204*
and mentioned that the issue did not appear with PVE 9 [1]. Looking at
PVE’s Perl ACME client [3], it uses a GET request instead of HEAD and
accepts any 2xx success code when retrieving the nonce. This difference
in behavior does not affect functionality but is worth noting for
consistency across implementations.
## Approach
To support ACME providers which return 204 No Content, the Rust ACME
clients in proxmox-backup and proxmox need to treat both 200 OK and 204
No Content as valid responses for the nonce request, as long as a
Replay-Nonce header is present.
This series changes the expected field of the internal Request type
from a single u16 to a list of allowed status codes
(e.g. &'static [u16]), so one request can explicitly accept multiple
success codes.
To avoid fixing the issue twice (once in PBS’ own ACME client and once
in the shared Rust client), this series first refactors PBS to use the
shared AcmeClient from proxmox-acme / proxmox-acme-api, similar to PDM,
and then applies the bug fix in that shared implementation so that all
consumers benefit from the more tolerant behavior.
## Testing
*Testing the refactor*
To test the refactor, I
(1) installed latest stable PBS on a VM
(2) created .deb package from latest PBS (master), containing the
refactor
(3) installed created .deb package
(4) installed Pebble from Let's Encrypt [5] on the same VM
(5) created an ACME account and ordered the new certificate for the
host domain.
Steps to reproduce:
(1) install latest stable PBS on a VM, create .deb package from latest
PBS (master) containing the refactor, install created .deb package
(2) install Pebble from Let's Encrypt [5] on the same VM:
cd
apt update
apt install -y golang git
git clone https://github.com/letsencrypt/pebble
cd pebble
go build ./cmd/pebble
then, download and trust the Pebble cert:
wget https://raw.githubusercontent.com/letsencrypt/pebble/main/test/certs/pebble.minica.pem
cp pebble.minica.pem /usr/local/share/ca-certificates/pebble.minica.crt
update-ca-certificates
We want Pebble to perform HTTP-01 validation against port 80, because
PBS’s standalone plugin will bind port 80. Set httpPort to 80.
nano ./test/config/pebble-config.json
Start the Pebble server in the background:
./pebble -config ./test/config/pebble-config.json &
Create a Pebble ACME account:
proxmox-backup-manager acme account register default admin@example.com --directory 'https://127.0.0.1:14000/dir'
To verify persistence of the account I checked
ls /etc/proxmox-backup/acme/accounts
Verified if update-account works
proxmox-backup-manager acme account update default --contact "a@example.com,b@example.com"
proxmox-backup-manager acme account info default
In the PBS GUI, you can create a new domain. You can use your host
domain name (see /etc/hosts). Select the created account and order the
certificate.
After a page reload, you might need to accept the new certificate in the browser.
In the PBS dashboard, you should see the new Pebble certificate.
*Note: on reboot, the created Pebble ACME account will be gone and you
will need to create a new one. Pebble does not persist account info.
In that case remove the previously created account in
/etc/proxmox-backup/acme/accounts.
*Testing the newNonce fix*
To prove the ACME newNonce fix, I put nginx in front of Pebble, to
intercept the newNonce request in order to return 204 No Content
instead of 200 OK, all other requests are unchanged and forwarded to
Pebble. Requires trusting the nginx CAs via
/usr/local/share/ca-certificates + update-ca-certificates on the VM.
Then I ran following command against nginx:
proxmox-backup-manager acme account register proxytest root@backup.local --directory 'https://nginx-address/dir
The account could be created successfully. When adjusting the nginx
configuration to return any other non-expected success status code,
PBS rejects as expected.
## Patch summary
0001 – [PATCH proxmox v5 1/4] acme: reduce visibility of Request type
Restricts the visibility of the low-level Request type. Consumers
should rely on proxmox-acme-api or AcmeClient handlers.
0002– [PATCH proxmox v5 2/4] acme: introduce http_status module
0003 – [PATCH proxmox v5 3/4] fix #6939: acme: support servers
returning 204 for nonce requests
Adjusts nonce handling to support ACME servers that return HTTP 204
(No Content) for new-nonce requests.
0004 – [PATCH proxmox v5 4/4] acme-api: add helper to load client for
an account
Introduces a helper function to load an ACME client instance for a
given account. Required for the following PBS ACME refactor.
0005 – [PATCH proxmox-backup v5 1/5] acme: clean up ACME-related imports
0006 – [PATCH proxmox-backup v5 2/5] acme: include proxmox-acme-api
dependency
Prepares the codebase to use the factored out ACME API impl.
0007 – [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient
Removes the local AcmeClient implementation. Represents the minimal
set of changes to replace it with the factored out AcmeClient.
0008 – [PATCH proxmox-backup v5 4/5] acme: change API impls to use
proxmox-acme-api handlers
0009 – [PATCH proxmox-backup v5 5/5] acme: certificate ordering through
proxmox-acme-api
Thanks for considering this patch series, I look forward to your
feedback.
Best,
Samuel Rufinatscha
## Changelog
Changes from v4 to v5:
* rebased series
* re-ordered series (proxmox-acme fix first)
* proxmox-backup: cleaned up imports based on an initial clean-up patch
* proxmox-acme: removed now unused post_request_raw_payload(),
update_account_request(), deactivate_account_request()
* proxmox-acme: removed now obsolete/unused get_authorization() and
GetAuthorization impl
Verified removal by compiling PBS, PDM, and proxmox-perl-rs
with all features.
Changes from v3 to v4:
* add proxmox-acme-api as a dependency and initialize it in
PBS so PBS can use the shared ACME API instead.
* remove the PBS-local AcmeClient implementation and switch PBS
over to the shared proxmox-acme async client.
* rework PBS’ ACME API endpoints to delegate to
proxmox-acme-api handlers instead of duplicating logic locally.
* move PBS’ ACME certificate ordering logic over to
proxmox-acme-api, keeping only certificate installation/reload in PBS.
* add a load_client_with_account helper in proxmox-acme-api so PBS
(and others) can construct an AcmeClient for a configured account
without duplicating boilerplate.
* hide the low-level Request type and its fields behind constructors
/ reduced visibility so changes to “expected” no longer affect the
public API as they did in v3.
* split out the HTTP status constants into an internal http_status
module as a separate preparatory cleanup before the bug fix, instead
of doing this inline like in v3.
* Rebased on top of the refactor: keep the same behavioural fix as in
v3 accept 204 for newNonce with Replay-Nonce present), but implement
it on top of the http_status module that is part of the refactor.
Changes from v2 to v3:
* rename `http_success` module to `http_status`
* replace `http_success` usage
* introduced `http_success` module to contain the http success codes
* replaced `Vec<u16>` with `&[u16]` for expected codes to avoid allocations.
* clarified the PVEs Perl ACME client behaviour in the commit message.
* integrated the `http_success` module, replacing `Vec<u16>` with `&[u16]`
* clarified the PVEs Perl ACME client behaviour in the commit message.
[1] Bugzilla report #6939:
[https://bugzilla.proxmox.com/show_bug.cgi?id=6939](https://bugzilla.proxmox.com/show_bug.cgi?id=6939)
[2] RFC 8555 (ACME):
[https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2](https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2)
[3] PVE’s Perl ACME client (allow 2xx codes for nonce requests):
[https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597)
[4] Pebble ACME server:
[https://github.com/letsencrypt/pebble](https://github.com/letsencrypt/pebble)
[5] Pebble ACME server (perform GET request:
[https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219)
proxmox:
Samuel Rufinatscha (4):
acme: reduce visibility of Request type
acme: introduce http_status module
fix #6939: acme: support servers returning 204 for nonce requests
acme-api: add helper to load client for an account
proxmox-acme-api/src/account_api_impl.rs | 5 ++
proxmox-acme-api/src/lib.rs | 3 +-
proxmox-acme/src/account.rs | 102 ++---------------------
proxmox-acme/src/async_client.rs | 8 +-
proxmox-acme/src/authorization.rs | 30 -------
proxmox-acme/src/client.rs | 8 +-
proxmox-acme/src/lib.rs | 6 +-
proxmox-acme/src/order.rs | 2 +-
proxmox-acme/src/request.rs | 25 ++++--
9 files changed, 44 insertions(+), 145 deletions(-)
proxmox-backup:
Samuel Rufinatscha (5):
acme: clean up ACME-related imports
acme: include proxmox-acme-api dependency
acme: drop local AcmeClient
acme: change API impls to use proxmox-acme-api handlers
acme: certificate ordering through proxmox-acme-api
Cargo.toml | 3 +
src/acme/client.rs | 691 -------------------------
src/acme/mod.rs | 5 -
src/acme/plugin.rs | 336 ------------
src/api2/config/acme.rs | 406 ++-------------
src/api2/node/certificates.rs | 232 ++-------
src/api2/types/acme.rs | 98 ----
src/api2/types/mod.rs | 3 -
src/bin/proxmox-backup-api.rs | 2 +
src/bin/proxmox-backup-manager.rs | 14 +-
src/bin/proxmox-backup-proxy.rs | 15 +-
src/bin/proxmox_backup_manager/acme.rs | 21 +-
src/config/acme/mod.rs | 55 +-
src/config/acme/plugin.rs | 92 +---
src/config/node.rs | 31 +-
src/lib.rs | 2 -
16 files changed, 109 insertions(+), 1897 deletions(-)
delete mode 100644 src/acme/client.rs
delete mode 100644 src/acme/mod.rs
delete mode 100644 src/acme/plugin.rs
delete mode 100644 src/api2/types/acme.rs
Summary over all repositories:
25 files changed, 153 insertions(+), 2042 deletions(-)
--
Generated by git-murpp 0.8.1
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 11%]
* [pbs-devel] [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient
2026-01-08 11:26 11% [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (5 preceding siblings ...)
2026-01-08 11:26 15% ` [pbs-devel] [PATCH proxmox-backup v5 2/5] acme: include proxmox-acme-api dependency Samuel Rufinatscha
@ 2026-01-08 11:26 6% ` Samuel Rufinatscha
2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-08 11:26 8% ` [pbs-devel] [PATCH proxmox-backup v5 4/5] acme: change API impls to use proxmox-acme-api handlers Samuel Rufinatscha
` (2 subsequent siblings)
9 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-08 11:26 UTC (permalink / raw)
To: pbs-devel
PBS currently uses its own ACME client and API logic, while PDM uses the
factored out proxmox-acme and proxmox-acme-api crates. This duplication
risks differences in behaviour and requires ACME maintenance in two
places. This patch is part of a series to move PBS over to the shared
ACME stack.
Changes:
- Remove the local src/acme/client.rs and switch to
proxmox_acme::async_client::AcmeClient where needed.
- Use proxmox_acme_api::load_client_with_account to the custom
AcmeClient::load() function
- Replace the local do_register() logic with
proxmox_acme_api::register_account, to further ensure accounts are persisted
- Replace the local AcmeAccountName type, required for
proxmox_acme_api::register_account
Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
---
src/acme/client.rs | 691 -------------------------
src/acme/mod.rs | 3 -
src/acme/plugin.rs | 2 +-
src/api2/config/acme.rs | 50 +-
src/api2/node/certificates.rs | 2 +-
src/api2/types/acme.rs | 8 -
src/bin/proxmox_backup_manager/acme.rs | 17 +-
src/config/acme/mod.rs | 8 +-
src/config/node.rs | 9 +-
9 files changed, 36 insertions(+), 754 deletions(-)
delete mode 100644 src/acme/client.rs
diff --git a/src/acme/client.rs b/src/acme/client.rs
deleted file mode 100644
index 9fb6ad55..00000000
--- a/src/acme/client.rs
+++ /dev/null
@@ -1,691 +0,0 @@
-//! HTTP Client for the ACME protocol.
-
-use std::fs::OpenOptions;
-use std::io;
-use std::os::unix::fs::OpenOptionsExt;
-
-use anyhow::{bail, format_err};
-use bytes::Bytes;
-use http_body_util::BodyExt;
-use hyper::Request;
-use nix::sys::stat::Mode;
-use proxmox_http::Body;
-use serde::{Deserialize, Serialize};
-
-use proxmox_acme::account::AccountCreator;
-use proxmox_acme::order::{Order, OrderData};
-use proxmox_acme::types::AccountData as AcmeAccountData;
-use proxmox_acme::Request as AcmeRequest;
-use proxmox_acme::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
-use proxmox_http::client::Client;
-use proxmox_sys::fs::{replace_file, CreateOptions};
-
-use crate::api2::types::AcmeAccountName;
-use crate::config::acme::account_path;
-use crate::tools::pbs_simple_http;
-
-/// Our on-disk format inherited from PVE's proxmox-acme code.
-#[derive(Deserialize, Serialize)]
-#[serde(rename_all = "camelCase")]
-pub struct AccountData {
- /// The account's location URL.
- location: String,
-
- /// The account data.
- account: AcmeAccountData,
-
- /// The private key as PEM formatted string.
- key: String,
-
- /// ToS URL the user agreed to.
- #[serde(skip_serializing_if = "Option::is_none")]
- tos: Option<String>,
-
- #[serde(skip_serializing_if = "is_false", default)]
- debug: bool,
-
- /// The directory's URL.
- directory_url: String,
-}
-
-#[inline]
-fn is_false(b: &bool) -> bool {
- !*b
-}
-
-pub struct AcmeClient {
- directory_url: String,
- debug: bool,
- account_path: Option<String>,
- tos: Option<String>,
- account: Option<Account>,
- directory: Option<Directory>,
- nonce: Option<String>,
- http_client: Client,
-}
-
-impl AcmeClient {
- /// Create a new ACME client for a given ACME directory URL.
- pub fn new(directory_url: String) -> Self {
- Self {
- directory_url,
- debug: false,
- account_path: None,
- tos: None,
- account: None,
- directory: None,
- nonce: None,
- http_client: pbs_simple_http(None),
- }
- }
-
- /// Load an existing ACME account by name.
- pub async fn load(account_name: &AcmeAccountName) -> Result<Self, anyhow::Error> {
- let account_path = account_path(account_name.as_ref());
- let data = match tokio::fs::read(&account_path).await {
- Ok(data) => data,
- Err(err) if err.kind() == io::ErrorKind::NotFound => {
- bail!("acme account '{}' does not exist", account_name)
- }
- Err(err) => bail!(
- "failed to load acme account from '{}' - {}",
- account_path,
- err
- ),
- };
- let data: AccountData = serde_json::from_slice(&data).map_err(|err| {
- format_err!(
- "failed to parse acme account from '{}' - {}",
- account_path,
- err
- )
- })?;
-
- let account = Account::from_parts(data.location, data.key, data.account);
-
- let mut me = Self::new(data.directory_url);
- me.debug = data.debug;
- me.account_path = Some(account_path);
- me.tos = data.tos;
- me.account = Some(account);
-
- Ok(me)
- }
-
- pub async fn new_account<'a>(
- &'a mut self,
- account_name: &AcmeAccountName,
- tos_agreed: bool,
- contact: Vec<String>,
- rsa_bits: Option<u32>,
- eab_creds: Option<(String, String)>,
- ) -> Result<&'a Account, anyhow::Error> {
- self.tos = if tos_agreed {
- self.terms_of_service_url().await?.map(str::to_owned)
- } else {
- None
- };
-
- let mut account = Account::creator()
- .set_contacts(contact)
- .agree_to_tos(tos_agreed);
-
- if let Some((eab_kid, eab_hmac_key)) = eab_creds {
- account = account.set_eab_credentials(eab_kid, eab_hmac_key)?;
- }
-
- let account = if let Some(bits) = rsa_bits {
- account.generate_rsa_key(bits)?
- } else {
- account.generate_ec_key()?
- };
-
- let _ = self.register_account(account).await?;
-
- crate::config::acme::make_acme_account_dir()?;
- let account_path = account_path(account_name.as_ref());
- let file = OpenOptions::new()
- .write(true)
- .create_new(true)
- .mode(0o600)
- .open(&account_path)
- .map_err(|err| format_err!("failed to open {:?} for writing: {}", account_path, err))?;
- self.write_to(file).map_err(|err| {
- format_err!(
- "failed to write acme account to {:?}: {}",
- account_path,
- err
- )
- })?;
- self.account_path = Some(account_path);
-
- // unwrap: Setting `self.account` is literally this function's job, we just can't keep
- // the borrow from from `self.register_account()` active due to clashes.
- Ok(self.account.as_ref().unwrap())
- }
-
- fn save(&self) -> Result<(), anyhow::Error> {
- let mut data = Vec::<u8>::new();
- self.write_to(&mut data)?;
- let account_path = self.account_path.as_ref().ok_or_else(|| {
- format_err!("no account path set, cannot save updated account information")
- })?;
- crate::config::acme::make_acme_account_dir()?;
- replace_file(
- account_path,
- &data,
- CreateOptions::new()
- .perm(Mode::from_bits_truncate(0o600))
- .owner(nix::unistd::ROOT)
- .group(nix::unistd::Gid::from_raw(0)),
- true,
- )
- }
-
- /// Shortcut to `account().ok_or_else(...).key_authorization()`.
- pub fn key_authorization(&self, token: &str) -> Result<String, anyhow::Error> {
- Ok(Self::need_account(&self.account)?.key_authorization(token)?)
- }
-
- /// Shortcut to `account().ok_or_else(...).dns_01_txt_value()`.
- /// the key authorization value.
- pub fn dns_01_txt_value(&self, token: &str) -> Result<String, anyhow::Error> {
- Ok(Self::need_account(&self.account)?.dns_01_txt_value(token)?)
- }
-
- async fn register_account(
- &mut self,
- account: AccountCreator,
- ) -> Result<&Account, anyhow::Error> {
- let mut retry = retry();
- let mut response = loop {
- retry.tick()?;
-
- let (directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
- let request = account.request(directory, nonce)?;
- match self.run_request(request).await {
- Ok(response) => break response,
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- };
-
- let account = account.response(response.location_required()?, &response.body)?;
-
- self.account = Some(account);
- Ok(self.account.as_ref().unwrap())
- }
-
- pub async fn update_account<T: Serialize>(
- &mut self,
- data: &T,
- ) -> Result<&Account, anyhow::Error> {
- let account = Self::need_account(&self.account)?;
-
- let mut retry = retry();
- let response = loop {
- retry.tick()?;
-
- let (_directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let request = account.post_request(&account.location, nonce, data)?;
- match Self::execute(&mut self.http_client, request, &mut self.nonce).await {
- Ok(response) => break response,
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- };
-
- // unwrap: we've been keeping an immutable reference to it from the top of the method
- let _ = account;
- self.account.as_mut().unwrap().data = response.json()?;
- self.save()?;
- Ok(self.account.as_ref().unwrap())
- }
-
- pub async fn new_order<I>(&mut self, domains: I) -> Result<Order, anyhow::Error>
- where
- I: IntoIterator<Item = String>,
- {
- let account = Self::need_account(&self.account)?;
-
- let order = domains
- .into_iter()
- .fold(OrderData::new(), |order, domain| order.domain(domain));
-
- let mut retry = retry();
- loop {
- retry.tick()?;
-
- let (directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let mut new_order = account.new_order(&order, directory, nonce)?;
- let mut response = match Self::execute(
- &mut self.http_client,
- new_order.request.take().unwrap(),
- &mut self.nonce,
- )
- .await
- {
- Ok(response) => response,
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- };
-
- return Ok(
- new_order.response(response.location_required()?, response.bytes().as_ref())?
- );
- }
- }
-
- /// Low level "POST-as-GET" request.
- async fn post_as_get(&mut self, url: &str) -> Result<AcmeResponse, anyhow::Error> {
- let account = Self::need_account(&self.account)?;
-
- let mut retry = retry();
- loop {
- retry.tick()?;
-
- let (_directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let request = account.get_request(url, nonce)?;
- match Self::execute(&mut self.http_client, request, &mut self.nonce).await {
- Ok(response) => return Ok(response),
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- }
- }
-
- /// Low level POST request.
- async fn post<T: Serialize>(
- &mut self,
- url: &str,
- data: &T,
- ) -> Result<AcmeResponse, anyhow::Error> {
- let account = Self::need_account(&self.account)?;
-
- let mut retry = retry();
- loop {
- retry.tick()?;
-
- let (_directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let request = account.post_request(url, nonce, data)?;
- match Self::execute(&mut self.http_client, request, &mut self.nonce).await {
- Ok(response) => return Ok(response),
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- }
- }
-
- /// Request challenge validation. Afterwards, the challenge should be polled.
- pub async fn request_challenge_validation(
- &mut self,
- url: &str,
- ) -> Result<Challenge, anyhow::Error> {
- Ok(self
- .post(url, &serde_json::Value::Object(Default::default()))
- .await?
- .json()?)
- }
-
- /// Assuming the provided URL is an 'Authorization' URL, get and deserialize it.
- pub async fn get_authorization(&mut self, url: &str) -> Result<Authorization, anyhow::Error> {
- Ok(self.post_as_get(url).await?.json()?)
- }
-
- /// Assuming the provided URL is an 'Order' URL, get and deserialize it.
- pub async fn get_order(&mut self, url: &str) -> Result<OrderData, anyhow::Error> {
- Ok(self.post_as_get(url).await?.json()?)
- }
-
- /// Finalize an Order via its `finalize` URL property and the DER encoded CSR.
- pub async fn finalize(&mut self, url: &str, csr: &[u8]) -> Result<(), anyhow::Error> {
- let csr = proxmox_base64::url::encode_no_pad(csr);
- let data = serde_json::json!({ "csr": csr });
- self.post(url, &data).await?;
- Ok(())
- }
-
- /// Download a certificate via its 'certificate' URL property.
- ///
- /// The certificate will be a PEM certificate chain.
- pub async fn get_certificate(&mut self, url: &str) -> Result<Bytes, anyhow::Error> {
- Ok(self.post_as_get(url).await?.body)
- }
-
- /// Revoke an existing certificate (PEM or DER formatted).
- pub async fn revoke_certificate(
- &mut self,
- certificate: &[u8],
- reason: Option<u32>,
- ) -> Result<(), anyhow::Error> {
- // TODO: This can also work without an account.
- let account = Self::need_account(&self.account)?;
-
- let revocation = account.revoke_certificate(certificate, reason)?;
-
- let mut retry = retry();
- loop {
- retry.tick()?;
-
- let (directory, nonce) = Self::get_dir_nonce(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?;
-
- let request = revocation.request(directory, nonce)?;
- match Self::execute(&mut self.http_client, request, &mut self.nonce).await {
- Ok(_response) => return Ok(()),
- Err(err) if err.is_bad_nonce() => continue,
- Err(err) => return Err(err.into()),
- }
- }
- }
-
- fn need_account(account: &Option<Account>) -> Result<&Account, anyhow::Error> {
- account
- .as_ref()
- .ok_or_else(|| format_err!("cannot use client without an account"))
- }
-
- pub(crate) fn account(&self) -> Result<&Account, anyhow::Error> {
- Self::need_account(&self.account)
- }
-
- pub fn tos(&self) -> Option<&str> {
- self.tos.as_deref()
- }
-
- pub fn directory_url(&self) -> &str {
- &self.directory_url
- }
-
- fn to_account_data(&self) -> Result<AccountData, anyhow::Error> {
- let account = self.account()?;
-
- Ok(AccountData {
- location: account.location.clone(),
- key: account.private_key.clone(),
- account: AcmeAccountData {
- only_return_existing: false, // don't actually write this out in case it's set
- ..account.data.clone()
- },
- tos: self.tos.clone(),
- debug: self.debug,
- directory_url: self.directory_url.clone(),
- })
- }
-
- fn write_to<T: io::Write>(&self, out: T) -> Result<(), anyhow::Error> {
- let data = self.to_account_data()?;
-
- Ok(serde_json::to_writer_pretty(out, &data)?)
- }
-}
-
-struct AcmeResponse {
- body: Bytes,
- location: Option<String>,
- got_nonce: bool,
-}
-
-impl AcmeResponse {
- /// Convenience helper to assert that a location header was part of the response.
- fn location_required(&mut self) -> Result<String, anyhow::Error> {
- self.location
- .take()
- .ok_or_else(|| format_err!("missing Location header"))
- }
-
- /// Convenience shortcut to perform json deserialization of the returned body.
- fn json<T: for<'a> Deserialize<'a>>(&self) -> Result<T, Error> {
- Ok(serde_json::from_slice(&self.body)?)
- }
-
- /// Convenience shortcut to get the body as bytes.
- fn bytes(&self) -> &[u8] {
- &self.body
- }
-}
-
-impl AcmeClient {
- /// Non-self-borrowing run_request version for borrow workarounds.
- async fn execute(
- http_client: &mut Client,
- request: AcmeRequest,
- nonce: &mut Option<String>,
- ) -> Result<AcmeResponse, Error> {
- let req_builder = Request::builder().method(request.method).uri(&request.url);
-
- let http_request = if !request.content_type.is_empty() {
- req_builder
- .header("Content-Type", request.content_type)
- .header("Content-Length", request.body.len())
- .body(request.body.into())
- } else {
- req_builder.body(Body::empty())
- }
- .map_err(|err| Error::Custom(format!("failed to create http request: {err}")))?;
-
- let response = http_client
- .request(http_request)
- .await
- .map_err(|err| Error::Custom(err.to_string()))?;
- let (parts, body) = response.into_parts();
-
- let status = parts.status.as_u16();
- let body = body
- .collect()
- .await
- .map_err(|err| Error::Custom(format!("failed to retrieve response body: {err}")))?
- .to_bytes();
-
- let got_nonce = if let Some(new_nonce) = parts.headers.get(proxmox_acme::REPLAY_NONCE) {
- let new_nonce = new_nonce.to_str().map_err(|err| {
- Error::Client(format!(
- "received invalid replay-nonce header from ACME server: {err}"
- ))
- })?;
- *nonce = Some(new_nonce.to_owned());
- true
- } else {
- false
- };
-
- if parts.status.is_success() {
- if status != request.expected {
- return Err(Error::InvalidApi(format!(
- "ACME server responded with unexpected status code: {:?}",
- parts.status
- )));
- }
-
- let location = parts
- .headers
- .get("Location")
- .map(|header| {
- header.to_str().map(str::to_owned).map_err(|err| {
- Error::Client(format!(
- "received invalid location header from ACME server: {err}"
- ))
- })
- })
- .transpose()?;
-
- return Ok(AcmeResponse {
- body,
- location,
- got_nonce,
- });
- }
-
- let error: ErrorResponse = serde_json::from_slice(&body).map_err(|err| {
- Error::Client(format!(
- "error status with improper error ACME response: {err}"
- ))
- })?;
-
- if error.ty == proxmox_acme::error::BAD_NONCE {
- if !got_nonce {
- return Err(Error::InvalidApi(
- "badNonce without a new Replay-Nonce header".to_string(),
- ));
- }
- return Err(Error::BadNonce);
- }
-
- Err(Error::Api(error))
- }
-
- /// Low-level API to run an n API request. This automatically updates the current nonce!
- async fn run_request(&mut self, request: AcmeRequest) -> Result<AcmeResponse, Error> {
- Self::execute(&mut self.http_client, request, &mut self.nonce).await
- }
-
- pub async fn directory(&mut self) -> Result<&Directory, Error> {
- Ok(Self::get_directory(
- &mut self.http_client,
- &self.directory_url,
- &mut self.directory,
- &mut self.nonce,
- )
- .await?
- .0)
- }
-
- async fn get_directory<'a, 'b>(
- http_client: &mut Client,
- directory_url: &str,
- directory: &'a mut Option<Directory>,
- nonce: &'b mut Option<String>,
- ) -> Result<(&'a Directory, Option<&'b str>), Error> {
- if let Some(d) = directory {
- return Ok((d, nonce.as_deref()));
- }
-
- let response = Self::execute(
- http_client,
- AcmeRequest {
- url: directory_url.to_string(),
- method: "GET",
- content_type: "",
- body: String::new(),
- expected: 200,
- },
- nonce,
- )
- .await?;
-
- *directory = Some(Directory::from_parts(
- directory_url.to_string(),
- response.json()?,
- ));
-
- Ok((directory.as_mut().unwrap(), nonce.as_deref()))
- }
-
- /// Like `get_directory`, but if the directory provides no nonce, also performs a `HEAD`
- /// request on the new nonce URL.
- async fn get_dir_nonce<'a, 'b>(
- http_client: &mut Client,
- directory_url: &str,
- directory: &'a mut Option<Directory>,
- nonce: &'b mut Option<String>,
- ) -> Result<(&'a Directory, &'b str), Error> {
- // this let construct is a lifetime workaround:
- let _ = Self::get_directory(http_client, directory_url, directory, nonce).await?;
- let dir = directory.as_ref().unwrap(); // the above fails if it couldn't fill this option
- if nonce.is_none() {
- // this is also a lifetime issue...
- let _ = Self::get_nonce(http_client, nonce, dir.new_nonce_url()).await?;
- };
- Ok((dir, nonce.as_deref().unwrap()))
- }
-
- pub async fn terms_of_service_url(&mut self) -> Result<Option<&str>, Error> {
- Ok(self.directory().await?.terms_of_service_url())
- }
-
- async fn get_nonce<'a>(
- http_client: &mut Client,
- nonce: &'a mut Option<String>,
- new_nonce_url: &str,
- ) -> Result<&'a str, Error> {
- let response = Self::execute(
- http_client,
- AcmeRequest {
- url: new_nonce_url.to_owned(),
- method: "HEAD",
- content_type: "",
- body: String::new(),
- expected: 200,
- },
- nonce,
- )
- .await?;
-
- if !response.got_nonce {
- return Err(Error::InvalidApi(
- "no new nonce received from new nonce URL".to_string(),
- ));
- }
-
- nonce
- .as_deref()
- .ok_or_else(|| Error::Client("failed to update nonce".to_string()))
- }
-}
-
-/// bad nonce retry count helper
-struct Retry(usize);
-
-const fn retry() -> Retry {
- Retry(0)
-}
-
-impl Retry {
- fn tick(&mut self) -> Result<(), Error> {
- if self.0 >= 3 {
- Err(Error::Client("kept getting a badNonce error!".to_string()))
- } else {
- self.0 += 1;
- Ok(())
- }
- }
-}
diff --git a/src/acme/mod.rs b/src/acme/mod.rs
index bf61811c..cc561f9a 100644
--- a/src/acme/mod.rs
+++ b/src/acme/mod.rs
@@ -1,5 +1,2 @@
-mod client;
-pub use client::AcmeClient;
-
pub(crate) mod plugin;
pub(crate) use plugin::get_acme_plugin;
diff --git a/src/acme/plugin.rs b/src/acme/plugin.rs
index 993d729b..6804243c 100644
--- a/src/acme/plugin.rs
+++ b/src/acme/plugin.rs
@@ -18,10 +18,10 @@ use tokio::io::{AsyncBufReadExt, AsyncRead, AsyncWriteExt, BufReader};
use tokio::net::TcpListener;
use tokio::process::Command;
+use proxmox_acme::async_client::AcmeClient;
use proxmox_acme::{Authorization, Challenge};
use proxmox_rest_server::WorkerTask;
-use crate::acme::AcmeClient;
use crate::api2::types::AcmeDomain;
use crate::config::acme::plugin::{DnsPlugin, PluginData};
diff --git a/src/api2/config/acme.rs b/src/api2/config/acme.rs
index 18671639..898f06dd 100644
--- a/src/api2/config/acme.rs
+++ b/src/api2/config/acme.rs
@@ -11,16 +11,16 @@ use serde_json::{json, Value};
use tracing::{info, warn};
use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
+use proxmox_acme::async_client::AcmeClient;
use proxmox_acme::types::AccountData as AcmeAccountData;
-use proxmox_acme::Account;
+use proxmox_acme_api::AcmeAccountName;
use proxmox_rest_server::WorkerTask;
use proxmox_router::{
http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
};
use proxmox_schema::{api, param_bail};
-use crate::acme::AcmeClient;
-use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
+use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
use crate::config::acme::plugin::{
self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
};
@@ -141,15 +141,15 @@ pub struct AccountInfo {
)]
/// Return existing ACME account information.
pub async fn get_account(name: AcmeAccountName) -> Result<AccountInfo, Error> {
- let client = AcmeClient::load(&name).await?;
- let account = client.account()?;
+ let account_info = proxmox_acme_api::get_account(name).await?;
+
Ok(AccountInfo {
- location: account.location.clone(),
- tos: client.tos().map(str::to_owned),
- directory: client.directory_url().to_owned(),
+ location: account_info.location,
+ tos: account_info.tos,
+ directory: account_info.directory,
account: AcmeAccountData {
only_return_existing: false, // don't actually write this out in case it's set
- ..account.data.clone()
+ ..account_info.account
},
})
}
@@ -238,41 +238,24 @@ fn register_account(
auth_id.to_string(),
true,
move |_worker| async move {
- let mut client = AcmeClient::new(directory);
-
info!("Registering ACME account '{}'...", &name);
- let account = do_register_account(
- &mut client,
+ let location = proxmox_acme_api::register_account(
&name,
- tos_url.is_some(),
contact,
- None,
+ tos_url,
+ Some(directory),
eab_kid.zip(eab_hmac_key),
)
.await?;
- info!("Registration successful, account URL: {}", account.location);
+ info!("Registration successful, account URL: {}", location);
Ok(())
},
)
}
-pub async fn do_register_account<'a>(
- client: &'a mut AcmeClient,
- name: &AcmeAccountName,
- agree_to_tos: bool,
- contact: String,
- rsa_bits: Option<u32>,
- eab_creds: Option<(String, String)>,
-) -> Result<&'a Account, Error> {
- let contact = account_contact_from_string(&contact);
- client
- .new_account(name, agree_to_tos, contact, rsa_bits, eab_creds)
- .await
-}
-
#[api(
input: {
properties: {
@@ -310,7 +293,10 @@ pub fn update_account(
None => json!({}),
};
- AcmeClient::load(&name).await?.update_account(&data).await?;
+ proxmox_acme_api::load_client_with_account(&name)
+ .await?
+ .update_account(&data)
+ .await?;
Ok(())
},
@@ -348,7 +334,7 @@ pub fn deactivate_account(
auth_id.to_string(),
true,
move |_worker| async move {
- match AcmeClient::load(&name)
+ match proxmox_acme_api::load_client_with_account(&name)
.await?
.update_account(&json!({"status": "deactivated"}))
.await
diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.rs
index 6b1d87d2..47ff8de5 100644
--- a/src/api2/node/certificates.rs
+++ b/src/api2/node/certificates.rs
@@ -8,6 +8,7 @@ use serde::{Deserialize, Serialize};
use tracing::{info, warn};
use pbs_api_types::{NODE_SCHEMA, PRIV_SYS_MODIFY};
+use proxmox_acme::async_client::AcmeClient;
use proxmox_rest_server::WorkerTask;
use proxmox_router::list_subdirs_api_method;
use proxmox_router::SubdirMap;
@@ -17,7 +18,6 @@ use proxmox_schema::api;
use pbs_buildcfg::configdir;
use pbs_tools::cert;
-use crate::acme::AcmeClient;
use crate::api2::types::AcmeDomain;
use crate::config::node::NodeConfig;
use crate::server::send_certificate_renewal_mail;
diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
index 8661f9e8..64175aff 100644
--- a/src/api2/types/acme.rs
+++ b/src/api2/types/acme.rs
@@ -59,14 +59,6 @@ pub struct KnownAcmeDirectory {
pub url: &'static str,
}
-proxmox_schema::api_string_type! {
- #[api(format: &PROXMOX_SAFE_ID_FORMAT)]
- /// ACME account name.
- #[derive(Clone, Eq, PartialEq, Hash, Deserialize, Serialize)]
- #[serde(transparent)]
- pub struct AcmeAccountName(String);
-}
-
#[api(
properties: {
schema: {
diff --git a/src/bin/proxmox_backup_manager/acme.rs b/src/bin/proxmox_backup_manager/acme.rs
index 0f0eafea..6ed61560 100644
--- a/src/bin/proxmox_backup_manager/acme.rs
+++ b/src/bin/proxmox_backup_manager/acme.rs
@@ -3,13 +3,13 @@ use std::io::Write;
use anyhow::{bail, Error};
use serde_json::Value;
+use proxmox_acme::async_client::AcmeClient;
+use proxmox_acme_api::AcmeAccountName;
use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
use proxmox_schema::api;
use proxmox_sys::fs::file_get_contents;
-use proxmox_backup::acme::AcmeClient;
use proxmox_backup::api2;
-use proxmox_backup::api2::types::AcmeAccountName;
use proxmox_backup::config::acme::plugin::DnsPluginCore;
use proxmox_backup::config::acme::KNOWN_ACME_DIRECTORIES;
@@ -188,17 +188,20 @@ async fn register_account(
println!("Attempting to register account with {directory_url:?}...");
- let account = api2::config::acme::do_register_account(
- &mut client,
+ let tos_agreed = tos_agreed
+ .then(|| directory.terms_of_service_url().map(str::to_owned))
+ .flatten();
+
+ let location = proxmox_acme_api::register_account(
&name,
- tos_agreed,
contact,
- None,
+ tos_agreed,
+ Some(directory_url),
eab_creds,
)
.await?;
- println!("Registration successful, account URL: {}", account.location);
+ println!("Registration successful, account URL: {}", location);
Ok(())
}
diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
index ac89ae5e..e4639c53 100644
--- a/src/config/acme/mod.rs
+++ b/src/config/acme/mod.rs
@@ -6,10 +6,11 @@ use anyhow::{bail, format_err, Error};
use serde_json::Value;
use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
+use proxmox_acme_api::AcmeAccountName;
use proxmox_sys::error::SysError;
use proxmox_sys::fs::{file_read_string, CreateOptions};
-use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
+use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
@@ -34,11 +35,6 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
create_acme_subdir(ACME_DIR)
}
-pub(crate) fn make_acme_account_dir() -> Result<(), Error> {
- make_acme_dir()?;
- create_acme_subdir(ACME_ACCOUNT_DIR)
-}
-
pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
KnownAcmeDirectory {
name: "Let's Encrypt V2",
diff --git a/src/config/node.rs b/src/config/node.rs
index 253b2e36..e4b66a20 100644
--- a/src/config/node.rs
+++ b/src/config/node.rs
@@ -8,16 +8,15 @@ use pbs_api_types::{
EMAIL_SCHEMA, MULTI_LINE_COMMENT_SCHEMA, OPENSSL_CIPHERS_TLS_1_2_SCHEMA,
OPENSSL_CIPHERS_TLS_1_3_SCHEMA,
};
+use proxmox_acme::async_client::AcmeClient;
+use proxmox_acme_api::AcmeAccountName;
use proxmox_http::ProxyConfig;
use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
use pbs_buildcfg::configdir;
use pbs_config::{open_backup_lockfile, BackupLockGuard};
-use crate::acme::AcmeClient;
-use crate::api2::types::{
- AcmeAccountName, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA,
-};
+use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
const CONF_FILE: &str = configdir!("/node.cfg");
const LOCK_FILE: &str = configdir!("/.node.lck");
@@ -247,7 +246,7 @@ impl NodeConfig {
} else {
AcmeAccountName::from_string("default".to_string())? // should really not happen
};
- AcmeClient::load(&account).await
+ proxmox_acme_api::load_client_with_account(&account).await
}
pub fn acme_domains(&'_ self) -> AcmeDomainIter<'_> {
--
2.47.3
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* [pbs-devel] superseded: [PATCH proxmox{-backup, } v4 0/8] fix #6939: acme: support servers returning 204 for nonce requests
2025-12-03 10:22 11% [pbs-devel] [PATCH proxmox{-backup, } v4 " Samuel Rufinatscha
` (8 preceding siblings ...)
2025-12-09 16:50 5% ` [pbs-devel] [PATCH proxmox{-backup, } v4 0/8] " Max R. Carrara
@ 2026-01-08 11:48 13% ` Samuel Rufinatscha
9 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-08 11:48 UTC (permalink / raw)
To: pbs-devel
https://lore.proxmox.com/pbs-devel/20260108112629.189670-1-s.rufinatscha@proxmox.com/T/#t
On 12/3/25 11:21 AM, Samuel Rufinatscha wrote:
> Hi,
>
> this series fixes account registration for ACME providers that return
> HTTP 204 No Content to the newNonce request. Currently, both the PBS
> ACME client and the shared ACME client in proxmox-acme only accept
> HTTP 200 OK for this request. The issue was observed in PBS against a
> custom ACME deployment and reported as bug #6939 [1].
>
> ## Problem
>
> During ACME account registration, PBS first fetches an anti-replay
> nonce by sending a HEAD request to the CA’s newNonce URL.
> RFC 8555 §7.2 [2] states that:
>
> * the server MUST include a Replay-Nonce header with a fresh nonce,
> * the server SHOULD use status 200 OK for the HEAD request,
> * the server MUST also handle GET on the same resource and may return
> 204 No Content with an empty body.
>
> The reporter observed the following error message:
>
> *ACME server responded with unexpected status code: 204*
>
> and mentioned that the issue did not appear with PVE 9 [1]. Looking at
> PVE’s Perl ACME client [3], it uses a GET request instead of HEAD and
> accepts any 2xx success code when retrieving the nonce. This difference
> in behavior does not affect functionality but is worth noting for
> consistency across implementations.
>
> ## Approach
>
> To support ACME providers which return 204 No Content, the Rust ACME
> clients in proxmox-backup and proxmox need to treat both 200 OK and 204
> No Content as valid responses for the nonce request, as long as a
> Replay-Nonce header is present.
>
> This series changes the expected field of the internal Request type
> from a single u16 to a list of allowed status codes
> (e.g. &'static [u16]), so one request can explicitly accept multiple
> success codes.
>
> To avoid fixing the issue twice (once in PBS’ own ACME client and once
> in the shared Rust client), this series first refactors PBS to use the
> shared AcmeClient from proxmox-acme / proxmox-acme-api, similar to PDM,
> and then applies the bug fix in that shared implementation so that all
> consumers benefit from the more tolerant behavior.
>
> ## Testing
>
> *Testing the refactor*
>
> To test the refactor, I
> (1) installed latest stable PBS on a VM
> (2) created .deb package from latest PBS (master), containing the
> refactor
> (3) installed created .deb package
> (4) installed Pebble from Let's Encrypt [5] on the same VM
> (5) created an ACME account and ordered the new certificate for the
> host domain.
>
> Steps to reproduce:
>
> (1) install latest stable PBS on a VM, create .deb package from latest
> PBS (master) containing the refactor, install created .deb package
> (2) install Pebble from Let's Encrypt [5] on the same VM:
>
> cd
> apt update
> apt install -y golang git
> git clone https://github.com/letsencrypt/pebble
> cd pebble
> go build ./cmd/pebble
>
> then, download and trust the Pebble cert:
>
> wget https://raw.githubusercontent.com/letsencrypt/pebble/main/test/certs/pebble.minica.pem
> cp pebble.minica.pem /usr/local/share/ca-certificates/pebble.minica.crt
> update-ca-certificates
>
> We want Pebble to perform HTTP-01 validation against port 80, because
> PBS’s standalone plugin will bind port 80. Set httpPort to 80.
>
> nano ./test/config/pebble-config.json
>
> Start the Pebble server in the background:
>
> ./pebble -config ./test/config/pebble-config.json &
>
> Create a Pebble ACME account:
>
> proxmox-backup-manager acme account register default admin@example.com --directory 'https://127.0.0.1:14000/dir'
>
> To verify persistence of the account I checked
>
> ls /etc/proxmox-backup/acme/accounts
>
> Verified if update-account works
>
> proxmox-backup-manager acme account update default --contact "a@example.com,b@example.com"
> proxmox-backup-manager acme account info default
>
> In the PBS GUI, you can create a new domain. You can use your host
> domain name (see /etc/hosts). Select the created account and order the
> certificate.
>
> After a page reload, you might need to accept the new certificate in the browser.
> In the PBS dashboard, you should see the new Pebble certificate.
>
> *Note: on reboot, the created Pebble ACME account will be gone and you
> will need to create a new one. Pebble does not persist account info.
> In that case remove the previously created account in
> /etc/proxmox-backup/acme/accounts.
>
> *Testing the newNonce fix*
>
> To prove the ACME newNonce fix, I put nginx in front of Pebble, to
> intercept the newNonce request in order to return 204 No Content
> instead of 200 OK, all other requests are unchanged and forwarded to
> Pebble. Requires trusting the nginx CAs via
> /usr/local/share/ca-certificates + update-ca-certificates on the VM.
>
> Then I ran following command against nginx:
>
> proxmox-backup-manager acme account register proxytest root@backup.local --directory 'https://nginx-address/dir
>
> The account could be created successfully. When adjusting the nginx
> configuration to return any other non-expected success status code,
> PBS rejects as expected.
>
> ## Patch summary
>
> 0001 – acme: include proxmox-acme-api dependency
> Adds proxmox-acme-api as a new dependency for the ACME code. This
> prepares the codebase to use the shared ACME API instead of local
> implementations.
>
> 0002 – acme: drop local AcmeClient
> Removes the local AcmeClient implementation. Minimal changes
> required to support the removal.
>
> 0003 – acme: change API impls to use proxmox-acme-api handler
> Updates existing ACME API implementations to use the handlers provided
> by proxmox-acme-api.
>
> 0004 – acme: certificate ordering through proxmox-acme-api
> Perform certificate ordering through proxmox-acme-api instead of local
> logic.
>
> 0005 – acme api: add helper to load client for an account
> Introduces a helper function to load an ACME client instance for a
> given account. Required for the PBS refactor.
>
> 0006 – acme: reduce visibility of Request type
> Restricts the visibility of the internal Request type.
>
> 0007 – acme: introduce http_status module
> Adds a dedicated http_status module for handling common HTTP status
> codes.
>
> 0008 – fix #6939: acme: support servers returning 204 for nonce
> Adjusts nonce handling to support ACME servers that return HTTP 204
> (No Content) for new-nonce requests.
>
> Thanks for considering this patch series, I look forward to your
> feedback.
>
> Best,
> Samuel Rufinatscha
>
> ## Changelog
>
> Changes from v3 to v4:
>
> Removed: [PATCH proxmox-backup v3 1/1].
>
> Added:
>
> [PATCH proxmox-backup v4 1/4] acme: include proxmox-acme-api dependency
> * New: add proxmox-acme-api as a dependency and initialize it in
> PBS so PBS can use the shared ACME API instead.
>
> [PATCH proxmox-backup v4 2/4] acme: drop local AcmeClient
> * New: remove the PBS-local AcmeClient implementation and switch PBS
> over to the shared proxmox-acme async client.
>
> [PATCH proxmox-backup v4 3/4] acme: change API impls to use proxmox-acme-api
> handlers
> * New: rework PBS’ ACME API endpoints to delegate to
> proxmox-acme-api handlers instead of duplicating logic locally.
>
> [PATCH proxmox-backup v4 4/4] acme: certificate ordering through
> proxmox-acme-api
> * New: move PBS’ ACME certificate ordering logic over to
> proxmox-acme-api, keeping only certificate installation/reload in
> PBS.
>
> [PATCH proxmox v4 1/4] acme-api: add helper to load client for an account
> * New: add a load_client_with_account helper in proxmox-acme-api so
> PBS (and others) can construct an AcmeClient for a configured account
> without duplicating boilerplate.
>
> [PATCH proxmox v4 2/4] acme: reduce visibility of Request type
> * New: hide the low-level Request type and its fields behind
> constructors / reduced visibility so changes to “expected” no longer
> affect the public API as they did in v3.
>
> [PATCH proxmox v4 3/4] acme: introduce http_status module
> * New: split out the HTTP status constants into an internal
> http_status module as a separate preparatory cleanup before the bug
> fix, instead of doing this inline like in v3.
>
> Changed:
>
> [PATCH proxmox v3 1/1] -> [PATCH proxmox v4 4/4]
> fix #6939: acme: support server returning 204 for nonce requests
> * Rebased on top of the refactor: keep the same behavioural fix as in v3
> (accept 204 for newNonce with Replay-Nonce present), but implement it
> on top of the http_status module that is part of the refactor.
>
> Changes from v2 to v3:
>
> [PATCH proxmox v3 1/1] fix #6939: support providers returning 204 for nonce
> requests
> * Rename `http_success` module to `http_status`
>
> [PATCH proxmox-backup v3 1/1] acme: accept HTTP 204 from newNonce endpoint
> * Replace `http_success` usage
>
> Changes from v1 to v2:
>
> [PATCH proxmox v2 1/1] fix #6939: support providers returning 204 for nonce
> requests
> * Introduced `http_success` module to contain the http success codes
> * Replaced `Vec<u16>` with `&[u16]` for expected codes to avoid
> allocations.
> * Clarified the PVEs Perl ACME client behaviour in the commit message.
>
> [PATCH proxmox-backup v2 1/1] acme: accept HTTP 204 from newNonce endpoint
> * Integrated the `http_success` module, replacing `Vec<u16>` with `&[u16]`
> * Clarified the PVEs Perl ACME client behaviour in the commit message.
>
> [1] Bugzilla report #6939:
> [https://bugzilla.proxmox.com/show_bug.cgi?id=6939](https://bugzilla.proxmox.com/show_bug.cgi?id=6939)
> [2] RFC 8555 (ACME):
> [https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2](https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2)
> [3] PVE’s Perl ACME client (allow 2xx codes for nonce requests):
> [https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597)
> [4] Pebble ACME server:
> [https://github.com/letsencrypt/pebble](https://github.com/letsencrypt/pebble)
> [5] Pebble ACME server (perform GET request:
> [https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219)
>
> proxmox-backup:
>
> Samuel Rufinatscha (4):
> acme: include proxmox-acme-api dependency
> acme: drop local AcmeClient
> acme: change API impls to use proxmox-acme-api handlers
> acme: certificate ordering through proxmox-acme-api
>
> Cargo.toml | 3 +
> src/acme/client.rs | 691 -------------------------
> src/acme/mod.rs | 5 -
> src/acme/plugin.rs | 336 ------------
> src/api2/config/acme.rs | 407 ++-------------
> src/api2/node/certificates.rs | 240 ++-------
> src/api2/types/acme.rs | 98 ----
> src/api2/types/mod.rs | 3 -
> src/bin/proxmox-backup-api.rs | 2 +
> src/bin/proxmox-backup-manager.rs | 2 +
> src/bin/proxmox-backup-proxy.rs | 1 +
> src/bin/proxmox_backup_manager/acme.rs | 21 +-
> src/config/acme/mod.rs | 51 +-
> src/config/acme/plugin.rs | 99 +---
> src/config/node.rs | 29 +-
> src/lib.rs | 2 -
> 16 files changed, 103 insertions(+), 1887 deletions(-)
> delete mode 100644 src/acme/client.rs
> delete mode 100644 src/acme/mod.rs
> delete mode 100644 src/acme/plugin.rs
> delete mode 100644 src/api2/types/acme.rs
>
>
> proxmox:
>
> Samuel Rufinatscha (4):
> acme-api: add helper to load client for an account
> acme: reduce visibility of Request type
> acme: introduce http_status module
> fix #6939: acme: support servers returning 204 for nonce requests
>
> proxmox-acme-api/src/account_api_impl.rs | 5 +++++
> proxmox-acme-api/src/lib.rs | 3 ++-
> proxmox-acme/src/account.rs | 27 +++++++++++++-----------
> proxmox-acme/src/async_client.rs | 8 +++----
> proxmox-acme/src/authorization.rs | 2 +-
> proxmox-acme/src/client.rs | 8 +++----
> proxmox-acme/src/lib.rs | 6 ++----
> proxmox-acme/src/order.rs | 2 +-
> proxmox-acme/src/request.rs | 25 +++++++++++++++-------
> 9 files changed, 51 insertions(+), 35 deletions(-)
>
>
> Summary over all repositories:
> 25 files changed, 154 insertions(+), 1922 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 13%]
* Re: [pdm-devel] [PATCH datacenter-manager] fix #7120: remote updates: drop vanished nodes/remotes from cache file
@ 2026-01-08 14:38 15% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-08 14:38 UTC (permalink / raw)
To: Proxmox Datacenter Manager development discussion, Lukas Wagner
On 1/8/26 2:06 PM, Lukas Wagner wrote:
> This commits makes sure that vanished remotes and remote cluster nodes
> are dropped from the remote updates cache file. This happens whenever
> the cache file is fully refreshed, either by the periodic update task,
> or by pressing "Refresh All" in the UI.
>
> Signed-off-by: Lukas Wagner <l.wagner@proxmox.com>
> ---
> server/src/remote_updates.rs | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
> diff --git a/server/src/remote_updates.rs b/server/src/remote_updates.rs
> index e772eef5..0490d28e 100644
> --- a/server/src/remote_updates.rs
> +++ b/server/src/remote_updates.rs
> @@ -214,6 +214,11 @@ pub async fn refresh_update_summary_cache(remotes: Vec<Remote>) -> Result<(), Er
>
> let mut content = get_cached_summary_or_default()?;
>
> + // Clean out any remotes that might have been removed from the remote config in the meanwhile.
> + content
> + .remotes
> + .retain(|remote, _| fetch_results.remote_results.contains_key(remote));
> +
> for (remote_name, result) in fetch_results.remote_results {
> let entry = content
> .remotes
> @@ -234,6 +239,11 @@ pub async fn refresh_update_summary_cache(remotes: Vec<Remote>) -> Result<(), Er
> Ok(remote_result) => {
> entry.status = RemoteUpdateStatus::Success;
>
> + // Clean out any nodes that might have been removed from the cluster in the meanwhile.
> + entry
> + .nodes
> + .retain(|name, _| remote_result.node_results.contains_key(name));
> +
> for (node_name, node_result) in remote_result.node_results {
> match node_result {
> Ok(NodeResults { data, .. }) => {
Patch looks good to me! I could reproduce the issue and can confirm
the patch works. After the patch,
/var/cache/proxmox-datacenter-manager/remote-updates.json no
longer shows the removed remote when running "Refresh All".
Reviewed-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
Tested-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel
^ permalink raw reply [relevance 15%]
* Re: [pbs-devel] [PATCH proxmox v5 4/4] acme-api: add helper to load client for an account
2026-01-08 11:26 17% ` [pbs-devel] [PATCH proxmox v5 4/4] acme-api: add helper to load client for an account Samuel Rufinatscha
@ 2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-13 16:57 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2026-01-13 13:45 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
> The PBS ACME refactoring needs a simple way to obtain an AcmeClient for
> a given configured account without duplicating config wiring. This patch
> adds a load_client_with_account helper in proxmox-acme-api that loads
> the account and constructs a matching client, similarly as PBS previous
> own AcmeClient::load() function.
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> proxmox-acme-api/src/account_api_impl.rs | 5 +++++
> proxmox-acme-api/src/lib.rs | 3 ++-
> 2 files changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/proxmox-acme-api/src/account_api_impl.rs b/proxmox-acme-api/src/account_api_impl.rs
> index ef195908..ca8c8655 100644
> --- a/proxmox-acme-api/src/account_api_impl.rs
> +++ b/proxmox-acme-api/src/account_api_impl.rs
> @@ -116,3 +116,8 @@ pub async fn update_account(name: &AcmeAccountName, contact: Option<String>) ->
>
> Ok(())
> }
> +
> +pub async fn load_client_with_account(account_name: &AcmeAccountName) -> Result<AcmeClient, Error> {
> + let account_data = super::account_config::load_account_config(&account_name).await?;
> + Ok(account_data.client())
> +}
I don't think this is needed - there is only a single callsite in PBS
and that is itself dead code that can be removed..
> diff --git a/proxmox-acme-api/src/lib.rs b/proxmox-acme-api/src/lib.rs
> index 623e9e23..96f88ae2 100644
> --- a/proxmox-acme-api/src/lib.rs
> +++ b/proxmox-acme-api/src/lib.rs
> @@ -31,7 +31,8 @@ mod plugin_config;
> mod account_api_impl;
> #[cfg(feature = "impl")]
> pub use account_api_impl::{
> - deactivate_account, get_account, get_tos, list_accounts, register_account, update_account,
> + deactivate_account, get_account, get_tos, list_accounts, load_client_with_account,
> + register_account, update_account,
> };
>
> #[cfg(feature = "impl")]
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient
2026-01-08 11:26 6% ` [pbs-devel] [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient Samuel Rufinatscha
@ 2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-14 8:56 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2026-01-13 13:45 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
> PBS currently uses its own ACME client and API logic, while PDM uses the
> factored out proxmox-acme and proxmox-acme-api crates. This duplication
> risks differences in behaviour and requires ACME maintenance in two
> places. This patch is part of a series to move PBS over to the shared
> ACME stack.
>
> Changes:
> - Remove the local src/acme/client.rs and switch to
> proxmox_acme::async_client::AcmeClient where needed.
> - Use proxmox_acme_api::load_client_with_account to the custom
> AcmeClient::load() function
> - Replace the local do_register() logic with
> proxmox_acme_api::register_account, to further ensure accounts are persisted
> - Replace the local AcmeAccountName type, required for
> proxmox_acme_api::register_account
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> src/acme/client.rs | 691 -------------------------
> src/acme/mod.rs | 3 -
> src/acme/plugin.rs | 2 +-
> src/api2/config/acme.rs | 50 +-
> src/api2/node/certificates.rs | 2 +-
> src/api2/types/acme.rs | 8 -
> src/bin/proxmox_backup_manager/acme.rs | 17 +-
> src/config/acme/mod.rs | 8 +-
> src/config/node.rs | 9 +-
> 9 files changed, 36 insertions(+), 754 deletions(-)
> delete mode 100644 src/acme/client.rs
>
[..]
> diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
> index ac89ae5e..e4639c53 100644
> --- a/src/config/acme/mod.rs
> +++ b/src/config/acme/mod.rs
I think this whole file should probably be replaced entirely by
proxmox-acme-api , which - AFAICT - would just require adding the
completion helpers there?
> @@ -6,10 +6,11 @@ use anyhow::{bail, format_err, Error};
> use serde_json::Value;
>
> use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
> +use proxmox_acme_api::AcmeAccountName;
> use proxmox_sys::error::SysError;
> use proxmox_sys::fs::{file_read_string, CreateOptions};
>
> -use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
> +use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
>
> pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
> pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
> @@ -34,11 +35,6 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
> create_acme_subdir(ACME_DIR)
> }
>
> -pub(crate) fn make_acme_account_dir() -> Result<(), Error> {
> - make_acme_dir()?;
> - create_acme_subdir(ACME_ACCOUNT_DIR)
> -}
> -
> pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
> KnownAcmeDirectory {
> name: "Let's Encrypt V2",
> diff --git a/src/config/node.rs b/src/config/node.rs
> index 253b2e36..e4b66a20 100644
> --- a/src/config/node.rs
> +++ b/src/config/node.rs
> @@ -8,16 +8,15 @@ use pbs_api_types::{
> EMAIL_SCHEMA, MULTI_LINE_COMMENT_SCHEMA, OPENSSL_CIPHERS_TLS_1_2_SCHEMA,
> OPENSSL_CIPHERS_TLS_1_3_SCHEMA,
> };
> +use proxmox_acme::async_client::AcmeClient;
> +use proxmox_acme_api::AcmeAccountName;
> use proxmox_http::ProxyConfig;
> use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
>
> use pbs_buildcfg::configdir;
> use pbs_config::{open_backup_lockfile, BackupLockGuard};
>
> -use crate::acme::AcmeClient;
> -use crate::api2::types::{
> - AcmeAccountName, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA,
> -};
> +use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
>
> const CONF_FILE: &str = configdir!("/node.cfg");
> const LOCK_FILE: &str = configdir!("/.node.lck");
> @@ -247,7 +246,7 @@ impl NodeConfig {
> } else {
> AcmeAccountName::from_string("default".to_string())? // should really not happen
> };
> - AcmeClient::load(&account).await
> + proxmox_acme_api::load_client_with_account(&account).await
> }
>
> pub fn acme_domains(&'_ self) -> AcmeDomainIter<'_> {
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 5/5] acme: certificate ordering through proxmox-acme-api
2026-01-08 11:26 7% ` [pbs-devel] [PATCH proxmox-backup v5 5/5] acme: certificate ordering through proxmox-acme-api Samuel Rufinatscha
@ 2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-13 16:51 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2026-01-13 13:45 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
> PBS currently uses its own ACME client and API logic, while PDM uses the
> factored out proxmox-acme and proxmox-acme-api crates. This duplication
> risks differences in behaviour and requires ACME maintenance in two
> places. This patch is part of a series to move PBS over to the shared
> ACME stack.
>
> Changes:
> - Replace the custom ACME order/authorization loop in node certificates
> with a call to proxmox_acme_api::order_certificate.
> - Build domain + config data as proxmox-acme-api types
> - Remove obsolete local ACME ordering and plugin glue code.
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> src/acme/mod.rs | 2 -
> src/acme/plugin.rs | 335 ----------------------------------
> src/api2/node/certificates.rs | 229 ++++-------------------
> src/api2/types/acme.rs | 73 --------
> src/api2/types/mod.rs | 3 -
> src/config/acme/mod.rs | 8 +-
> src/config/acme/plugin.rs | 92 +---------
> src/config/node.rs | 20 +-
> src/lib.rs | 2 -
> 9 files changed, 38 insertions(+), 726 deletions(-)
> delete mode 100644 src/acme/mod.rs
> delete mode 100644 src/acme/plugin.rs
> delete mode 100644 src/api2/types/acme.rs
>
[..]
> diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.rs
> index 47ff8de5..73401c41 100644
> --- a/src/api2/node/certificates.rs
> +++ b/src/api2/node/certificates.rs
> @@ -1,14 +1,11 @@
> -use std::sync::Arc;
> -use std::time::Duration;
> -
> use anyhow::{bail, format_err, Error};
> use openssl::pkey::PKey;
> use openssl::x509::X509;
> use serde::{Deserialize, Serialize};
> -use tracing::{info, warn};
> +use tracing::info;
>
> use pbs_api_types::{NODE_SCHEMA, PRIV_SYS_MODIFY};
> -use proxmox_acme::async_client::AcmeClient;
> +use proxmox_acme_api::AcmeDomain;
> use proxmox_rest_server::WorkerTask;
> use proxmox_router::list_subdirs_api_method;
> use proxmox_router::SubdirMap;
> @@ -18,8 +15,6 @@ use proxmox_schema::api;
> use pbs_buildcfg::configdir;
> use pbs_tools::cert;
>
> -use crate::api2::types::AcmeDomain;
> -use crate::config::node::NodeConfig;
> use crate::server::send_certificate_renewal_mail;
>
> pub const ROUTER: Router = Router::new()
> @@ -268,193 +263,6 @@ pub async fn delete_custom_certificate() -> Result<(), Error> {
> Ok(())
> }
>
> -struct OrderedCertificate {
> - certificate: hyper::body::Bytes,
> - private_key_pem: Vec<u8>,
> -}
> -
> -async fn order_certificate(
> - worker: Arc<WorkerTask>,
> - node_config: &NodeConfig,
> -) -> Result<Option<OrderedCertificate>, Error> {
> - use proxmox_acme::authorization::Status;
> - use proxmox_acme::order::Identifier;
> -
> - let domains = node_config.acme_domains().try_fold(
> - Vec::<AcmeDomain>::new(),
> - |mut acc, domain| -> Result<_, Error> {
> - let mut domain = domain?;
> - domain.domain.make_ascii_lowercase();
> - if let Some(alias) = &mut domain.alias {
> - alias.make_ascii_lowercase();
> - }
> - acc.push(domain);
> - Ok(acc)
> - },
> - )?;
> -
> - let get_domain_config = |domain: &str| {
> - domains
> - .iter()
> - .find(|d| d.domain == domain)
> - .ok_or_else(|| format_err!("no config for domain '{}'", domain))
> - };
> -
> - if domains.is_empty() {
> - info!("No domains configured to be ordered from an ACME server.");
> - return Ok(None);
> - }
> -
> - let (plugins, _) = crate::config::acme::plugin::config()?;
> -
> - let mut acme = node_config.acme_client().await?;
> -
> - info!("Placing ACME order");
> - let order = acme
> - .new_order(domains.iter().map(|d| d.domain.to_ascii_lowercase()))
> - .await?;
> - info!("Order URL: {}", order.location);
> -
> - let identifiers: Vec<String> = order
> - .data
> - .identifiers
> - .iter()
> - .map(|identifier| match identifier {
> - Identifier::Dns(domain) => domain.clone(),
> - })
> - .collect();
> -
> - for auth_url in &order.data.authorizations {
> - info!("Getting authorization details from '{auth_url}'");
> - let mut auth = acme.get_authorization(auth_url).await?;
> -
> - let domain = match &mut auth.identifier {
> - Identifier::Dns(domain) => domain.to_ascii_lowercase(),
> - };
> -
> - if auth.status == Status::Valid {
> - info!("{domain} is already validated!");
> - continue;
> - }
> -
> - info!("The validation for {domain} is pending");
> - let domain_config: &AcmeDomain = get_domain_config(&domain)?;
> - let plugin_id = domain_config.plugin.as_deref().unwrap_or("standalone");
> - let mut plugin_cfg = crate::acme::get_acme_plugin(&plugins, plugin_id)?
> - .ok_or_else(|| format_err!("plugin '{plugin_id}' for domain '{domain}' not found!"))?;
> -
> - info!("Setting up validation plugin");
> - let validation_url = plugin_cfg
> - .setup(&mut acme, &auth, domain_config, Arc::clone(&worker))
> - .await?;
> -
> - let result = request_validation(&mut acme, auth_url, validation_url).await;
> -
> - if let Err(err) = plugin_cfg
> - .teardown(&mut acme, &auth, domain_config, Arc::clone(&worker))
> - .await
> - {
> - warn!("Failed to teardown plugin '{plugin_id}' for domain '{domain}' - {err}");
> - }
> -
> - result?;
> - }
> -
> - info!("All domains validated");
> - info!("Creating CSR");
> -
> - let csr = proxmox_acme::util::Csr::generate(&identifiers, &Default::default())?;
> - let mut finalize_error_cnt = 0u8;
> - let order_url = &order.location;
> - let mut order;
> - loop {
> - use proxmox_acme::order::Status;
> -
> - order = acme.get_order(order_url).await?;
> -
> - match order.status {
> - Status::Pending => {
> - info!("still pending, trying to finalize anyway");
> - let finalize = order
> - .finalize
> - .as_deref()
> - .ok_or_else(|| format_err!("missing 'finalize' URL in order"))?;
> - if let Err(err) = acme.finalize(finalize, &csr.data).await {
> - if finalize_error_cnt >= 5 {
> - return Err(err);
> - }
> -
> - finalize_error_cnt += 1;
> - }
> - tokio::time::sleep(Duration::from_secs(5)).await;
> - }
> - Status::Ready => {
> - info!("order is ready, finalizing");
> - let finalize = order
> - .finalize
> - .as_deref()
> - .ok_or_else(|| format_err!("missing 'finalize' URL in order"))?;
> - acme.finalize(finalize, &csr.data).await?;
> - tokio::time::sleep(Duration::from_secs(5)).await;
> - }
> - Status::Processing => {
> - info!("still processing, trying again in 30 seconds");
> - tokio::time::sleep(Duration::from_secs(30)).await;
> - }
> - Status::Valid => {
> - info!("valid");
> - break;
> - }
> - other => bail!("order status: {:?}", other),
> - }
> - }
> -
> - info!("Downloading certificate");
> - let certificate = acme
> - .get_certificate(
> - order
> - .certificate
> - .as_deref()
> - .ok_or_else(|| format_err!("missing certificate url in finalized order"))?,
> - )
> - .await?;
> -
> - Ok(Some(OrderedCertificate {
> - certificate,
> - private_key_pem: csr.private_key_pem,
> - }))
> -}
> -
> -async fn request_validation(
> - acme: &mut AcmeClient,
> - auth_url: &str,
> - validation_url: &str,
> -) -> Result<(), Error> {
> - info!("Triggering validation");
> - acme.request_challenge_validation(validation_url).await?;
> -
> - info!("Sleeping for 5 seconds");
> - tokio::time::sleep(Duration::from_secs(5)).await;
> -
> - loop {
> - use proxmox_acme::authorization::Status;
> -
> - let auth = acme.get_authorization(auth_url).await?;
> - match auth.status {
> - Status::Pending => {
> - info!("Status is still 'pending', trying again in 10 seconds");
> - tokio::time::sleep(Duration::from_secs(10)).await;
> - }
> - Status::Valid => return Ok(()),
> - other => bail!(
> - "validating challenge '{}' failed - status: {:?}",
> - validation_url,
> - other
> - ),
> - }
> - }
> -}
> -
> #[api(
> input: {
> properties: {
> @@ -524,9 +332,30 @@ fn spawn_certificate_worker(
>
> let auth_id = rpcenv.get_auth_id().unwrap();
>
> + let acme_config = if let Some(cfg) = node_config.acme_config().transpose()? {
> + cfg
> + } else {
> + proxmox_acme_api::parse_acme_config_string("account=default")?
> + };
wouldn't it make sense to inline this into acme_config() ? the same
fallback is already there for acme_client()
> +
> + let domains = node_config.acme_domains().try_fold(
> + Vec::<AcmeDomain>::new(),
> + |mut acc, domain| -> Result<_, Error> {
> + let mut domain = domain?;
> + domain.domain.make_ascii_lowercase();
> + if let Some(alias) = &mut domain.alias {
> + alias.make_ascii_lowercase();
> + }
> + acc.push(domain);
> + Ok(acc)
> + },
> + )?;
> +
> WorkerTask::spawn(name, None, auth_id, true, move |worker| async move {
> let work = || async {
> - if let Some(cert) = order_certificate(worker, &node_config).await? {
> + if let Some(cert) =
> + proxmox_acme_api::order_certificate(worker, &acme_config, &domains).await?
> + {
> crate::config::set_proxy_certificate(&cert.certificate, &cert.private_key_pem)?;
> crate::server::reload_proxy_certificate().await?;
> }
> @@ -562,16 +391,20 @@ pub fn revoke_acme_cert(rpcenv: &mut dyn RpcEnvironment) -> Result<String, Error
>
> let auth_id = rpcenv.get_auth_id().unwrap();
>
> + let acme_config = if let Some(cfg) = node_config.acme_config().transpose()? {
> + cfg
> + } else {
> + proxmox_acme_api::parse_acme_config_string("account=default")?
> + };
here as well
> +
> WorkerTask::spawn(
> "acme-revoke-cert",
> None,
> auth_id,
> true,
> move |_worker| async move {
> - info!("Loading ACME account");
> - let mut acme = node_config.acme_client().await?;
> info!("Revoking old certificate");
> - acme.revoke_certificate(cert_pem.as_bytes(), None).await?;
> + proxmox_acme_api::revoke_certificate(&acme_config, &cert_pem.as_bytes()).await?;
> info!("Deleting certificate and regenerating a self-signed one");
> delete_custom_certificate().await?;
> Ok(())
[..]
> diff --git a/src/config/acme/plugin.rs b/src/config/acme/plugin.rs
> index 8ce852ec..4b4a216e 100644
> --- a/src/config/acme/plugin.rs
> +++ b/src/config/acme/plugin.rs
> @@ -1,104 +1,16 @@
> use std::sync::LazyLock;
>
> use anyhow::Error;
> -use serde::{Deserialize, Serialize};
> use serde_json::Value;
>
> -use pbs_api_types::PROXMOX_SAFE_ID_FORMAT;
> -use proxmox_schema::{api, ApiType, Schema, StringSchema, Updater};
> +use proxmox_acme_api::{DnsPlugin, StandalonePlugin, PLUGIN_ID_SCHEMA};
> +use proxmox_schema::{ApiType, Schema};
> use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
>
> use pbs_config::{open_backup_lockfile, BackupLockGuard};
>
> -pub const PLUGIN_ID_SCHEMA: Schema = StringSchema::new("ACME Challenge Plugin ID.")
> - .format(&PROXMOX_SAFE_ID_FORMAT)
> - .min_length(1)
> - .max_length(32)
> - .schema();
> -
> pub static CONFIG: LazyLock<SectionConfig> = LazyLock::new(init);
>
> -#[api(
> - properties: {
> - id: { schema: PLUGIN_ID_SCHEMA },
> - },
> -)]
> -#[derive(Deserialize, Serialize)]
> -/// Standalone ACME Plugin for the http-1 challenge.
> -pub struct StandalonePlugin {
> - /// Plugin ID.
> - id: String,
> -}
> -
> -impl Default for StandalonePlugin {
> - fn default() -> Self {
> - Self {
> - id: "standalone".to_string(),
> - }
> - }
> -}
> -
> -#[api(
> - properties: {
> - id: { schema: PLUGIN_ID_SCHEMA },
> - disable: {
> - optional: true,
> - default: false,
> - },
> - "validation-delay": {
> - default: 30,
> - optional: true,
> - minimum: 0,
> - maximum: 2 * 24 * 60 * 60,
> - },
> - },
> -)]
> -/// DNS ACME Challenge Plugin core data.
> -#[derive(Deserialize, Serialize, Updater)]
> -#[serde(rename_all = "kebab-case")]
> -pub struct DnsPluginCore {
> - /// Plugin ID.
> - #[updater(skip)]
> - pub id: String,
> -
> - /// DNS API Plugin Id.
> - pub api: String,
> -
> - /// Extra delay in seconds to wait before requesting validation.
> - ///
> - /// Allows to cope with long TTL of DNS records.
> - #[serde(skip_serializing_if = "Option::is_none", default)]
> - pub validation_delay: Option<u32>,
> -
> - /// Flag to disable the config.
> - #[serde(skip_serializing_if = "Option::is_none", default)]
> - pub disable: Option<bool>,
> -}
> -
> -#[api(
> - properties: {
> - core: { type: DnsPluginCore },
> - },
> -)]
> -/// DNS ACME Challenge Plugin.
> -#[derive(Deserialize, Serialize)]
> -#[serde(rename_all = "kebab-case")]
> -pub struct DnsPlugin {
> - #[serde(flatten)]
> - pub core: DnsPluginCore,
> -
> - // We handle this property separately in the API calls.
> - /// DNS plugin data (base64url encoded without padding).
> - #[serde(with = "proxmox_serde::string_as_base64url_nopad")]
> - pub data: String,
> -}
> -
> -impl DnsPlugin {
> - pub fn decode_data(&self, output: &mut Vec<u8>) -> Result<(), Error> {
> - Ok(proxmox_base64::url::decode_to_vec(&self.data, output)?)
> - }
> -}
> -
> fn init() -> SectionConfig {
> let mut config = SectionConfig::new(&PLUGIN_ID_SCHEMA);
>
> diff --git a/src/config/node.rs b/src/config/node.rs
> index e4b66a20..6865b815 100644
> --- a/src/config/node.rs
> +++ b/src/config/node.rs
> @@ -9,14 +9,14 @@ use pbs_api_types::{
> OPENSSL_CIPHERS_TLS_1_3_SCHEMA,
> };
> use proxmox_acme::async_client::AcmeClient;
> -use proxmox_acme_api::AcmeAccountName;
> +use proxmox_acme_api::{AcmeAccountName, AcmeConfig, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA};
> use proxmox_http::ProxyConfig;
> use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
>
> use pbs_buildcfg::configdir;
> use pbs_config::{open_backup_lockfile, BackupLockGuard};
>
> -use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
> +use crate::api2::types::HTTP_PROXY_SCHEMA;
>
> const CONF_FILE: &str = configdir!("/node.cfg");
> const LOCK_FILE: &str = configdir!("/.node.lck");
> @@ -43,20 +43,6 @@ pub fn save_config(config: &NodeConfig) -> Result<(), Error> {
> pbs_config::replace_backup_config(CONF_FILE, &raw)
> }
>
> -#[api(
> - properties: {
> - account: { type: AcmeAccountName },
> - }
> -)]
> -#[derive(Deserialize, Serialize)]
> -/// The ACME configuration.
> -///
> -/// Currently only contains the name of the account use.
> -pub struct AcmeConfig {
> - /// Account to use to acquire ACME certificates.
> - account: AcmeAccountName,
> -}
> -
> /// All available languages in Proxmox. Taken from proxmox-i18n repository.
> /// pt_BR, zh_CN, and zh_TW use the same case in the translation files.
> // TODO: auto-generate from available translations
> @@ -242,7 +228,7 @@ impl NodeConfig {
>
> pub async fn acme_client(&self) -> Result<AcmeClient, Error> {
> let account = if let Some(cfg) = self.acme_config().transpose()? {
> - cfg.account
> + AcmeAccountName::from_string(cfg.account)?
> } else {
> AcmeAccountName::from_string("default".to_string())? // should really not happen
> };
> diff --git a/src/lib.rs b/src/lib.rs
> index 8633378c..828f5842 100644
> --- a/src/lib.rs
> +++ b/src/lib.rs
> @@ -27,8 +27,6 @@ pub(crate) mod auth;
>
> pub mod tape;
>
> -pub mod acme;
> -
> pub mod client_helpers;
>
> pub mod traffic_control_cache;
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 4/5] acme: change API impls to use proxmox-acme-api handlers
2026-01-08 11:26 8% ` [pbs-devel] [PATCH proxmox-backup v5 4/5] acme: change API impls to use proxmox-acme-api handlers Samuel Rufinatscha
@ 2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-13 16:53 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2026-01-13 13:45 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
> PBS currently uses its own ACME client and API logic, while PDM uses the
> factored out proxmox-acme and proxmox-acme-api crates. This duplication
> risks differences in behaviour and requires ACME maintenance in two
> places. This patch is part of a series to move PBS over to the shared
> ACME stack.
>
> Changes:
> - Replace api2/config/acme.rs API logic with proxmox-acme-api handlers.
> - Drop local caching and helper types that duplicate proxmox-acme-api.
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> src/api2/config/acme.rs | 378 ++-----------------------
> src/api2/types/acme.rs | 16 --
> src/bin/proxmox_backup_manager/acme.rs | 6 +-
> src/config/acme/mod.rs | 44 +--
> 4 files changed, 33 insertions(+), 411 deletions(-)
>
> diff --git a/src/api2/config/acme.rs b/src/api2/config/acme.rs
> index 898f06dd..3314430c 100644
> --- a/src/api2/config/acme.rs
> +++ b/src/api2/config/acme.rs
> @@ -1,29 +1,18 @@
> -use std::fs;
> -use std::ops::ControlFlow;
> -use std::path::Path;
nit: this one is actually still used below
> -use std::sync::{Arc, LazyLock, Mutex};
> -use std::time::SystemTime;
> -
> -use anyhow::{bail, format_err, Error};
> -use hex::FromHex;
> -use serde::{Deserialize, Serialize};
> -use serde_json::{json, Value};
> -use tracing::{info, warn};
> +use anyhow::Error;
> +use tracing::info;
>
> use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
> -use proxmox_acme::async_client::AcmeClient;
> -use proxmox_acme::types::AccountData as AcmeAccountData;
> -use proxmox_acme_api::AcmeAccountName;
> +use proxmox_acme_api::{
> + AccountEntry, AccountInfo, AcmeAccountName, AcmeChallengeSchema, ChallengeSchemaWrapper,
> + DeletablePluginProperty, DnsPluginCore, DnsPluginCoreUpdater, KnownAcmeDirectory, PluginConfig,
> + DEFAULT_ACME_DIRECTORY_ENTRY, PLUGIN_ID_SCHEMA,
> +};
> +use proxmox_config_digest::ConfigDigest;
> use proxmox_rest_server::WorkerTask;
> use proxmox_router::{
> http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
> };
> -use proxmox_schema::{api, param_bail};
> -
> -use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
> -use crate::config::acme::plugin::{
> - self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
> -};
> +use proxmox_schema::api;
>
> pub(crate) const ROUTER: Router = Router::new()
> .get(&list_subdirs_api_method!(SUBDIRS))
> @@ -65,19 +54,6 @@ const PLUGIN_ITEM_ROUTER: Router = Router::new()
> .put(&API_METHOD_UPDATE_PLUGIN)
> .delete(&API_METHOD_DELETE_PLUGIN);
>
> -#[api(
> - properties: {
> - name: { type: AcmeAccountName },
> - },
> -)]
> -/// An ACME Account entry.
> -///
> -/// Currently only contains a 'name' property.
> -#[derive(Serialize)]
> -pub struct AccountEntry {
> - name: AcmeAccountName,
> -}
> -
> #[api(
> access: {
> permission: &Permission::Privilege(&["system", "certificates"], PRIV_SYS_MODIFY, false),
> @@ -91,40 +67,7 @@ pub struct AccountEntry {
> )]
> /// List ACME accounts.
> pub fn list_accounts() -> Result<Vec<AccountEntry>, Error> {
> - let mut entries = Vec::new();
> - crate::config::acme::foreach_acme_account(|name| {
> - entries.push(AccountEntry { name });
> - ControlFlow::Continue(())
> - })?;
> - Ok(entries)
> -}
> -
> -#[api(
> - properties: {
> - account: { type: Object, properties: {}, additional_properties: true },
> - tos: {
> - type: String,
> - optional: true,
> - },
> - },
> -)]
> -/// ACME Account information.
> -///
> -/// This is what we return via the API.
> -#[derive(Serialize)]
> -pub struct AccountInfo {
> - /// Raw account data.
> - account: AcmeAccountData,
> -
> - /// The ACME directory URL the account was created at.
> - directory: String,
> -
> - /// The account's own URL within the ACME directory.
> - location: String,
> -
> - /// The ToS URL, if the user agreed to one.
> - #[serde(skip_serializing_if = "Option::is_none")]
> - tos: Option<String>,
> + proxmox_acme_api::list_accounts()
> }
>
> #[api(
> @@ -141,23 +84,7 @@ pub struct AccountInfo {
> )]
> /// Return existing ACME account information.
> pub async fn get_account(name: AcmeAccountName) -> Result<AccountInfo, Error> {
> - let account_info = proxmox_acme_api::get_account(name).await?;
> -
> - Ok(AccountInfo {
> - location: account_info.location,
> - tos: account_info.tos,
> - directory: account_info.directory,
> - account: AcmeAccountData {
> - only_return_existing: false, // don't actually write this out in case it's set
> - ..account_info.account
> - },
> - })
> -}
> -
> -fn account_contact_from_string(s: &str) -> Vec<String> {
> - s.split(&[' ', ';', ',', '\0'][..])
> - .map(|s| format!("mailto:{s}"))
> - .collect()
> + proxmox_acme_api::get_account(name).await
> }
>
> #[api(
> @@ -222,15 +149,11 @@ fn register_account(
> );
> }
>
> - if Path::new(&crate::config::acme::account_path(&name)).exists() {
> + if std::path::Path::new(&proxmox_acme_api::account_config_filename(&name)).exists() {
here ^
> http_bail!(BAD_REQUEST, "account {} already exists", name);
> }
>
> - let directory = directory.unwrap_or_else(|| {
> - crate::config::acme::DEFAULT_ACME_DIRECTORY_ENTRY
> - .url
> - .to_owned()
> - });
> + let directory = directory.unwrap_or_else(|| DEFAULT_ACME_DIRECTORY_ENTRY.url.to_string());
>
> WorkerTask::spawn(
> "acme-register",
> @@ -286,17 +209,7 @@ pub fn update_account(
> auth_id.to_string(),
> true,
> move |_worker| async move {
> - let data = match contact {
> - Some(data) => json!({
> - "contact": account_contact_from_string(&data),
> - }),
> - None => json!({}),
> - };
> -
> - proxmox_acme_api::load_client_with_account(&name)
> - .await?
> - .update_account(&data)
> - .await?;
> + proxmox_acme_api::update_account(&name, contact).await?;
>
> Ok(())
> },
> @@ -334,18 +247,8 @@ pub fn deactivate_account(
> auth_id.to_string(),
> true,
> move |_worker| async move {
> - match proxmox_acme_api::load_client_with_account(&name)
> - .await?
> - .update_account(&json!({"status": "deactivated"}))
> - .await
> - {
> - Ok(_account) => (),
> - Err(err) if !force => return Err(err),
> - Err(err) => {
> - warn!("error deactivating account {name}, proceeding anyway - {err}");
> - }
> - }
> - crate::config::acme::mark_account_deactivated(&name)?;
> + proxmox_acme_api::deactivate_account(&name, force).await?;
> +
> Ok(())
> },
> )
> @@ -372,15 +275,7 @@ pub fn deactivate_account(
> )]
> /// Get the Terms of Service URL for an ACME directory.
> async fn get_tos(directory: Option<String>) -> Result<Option<String>, Error> {
> - let directory = directory.unwrap_or_else(|| {
> - crate::config::acme::DEFAULT_ACME_DIRECTORY_ENTRY
> - .url
> - .to_owned()
> - });
> - Ok(AcmeClient::new(directory)
> - .terms_of_service_url()
> - .await?
> - .map(str::to_owned))
> + proxmox_acme_api::get_tos(directory).await
> }
>
> #[api(
> @@ -395,52 +290,7 @@ async fn get_tos(directory: Option<String>) -> Result<Option<String>, Error> {
> )]
> /// Get named known ACME directory endpoints.
> fn get_directories() -> Result<&'static [KnownAcmeDirectory], Error> {
> - Ok(crate::config::acme::KNOWN_ACME_DIRECTORIES)
> -}
> -
> -/// Wrapper for efficient Arc use when returning the ACME challenge-plugin schema for serializing
> -struct ChallengeSchemaWrapper {
> - inner: Arc<Vec<AcmeChallengeSchema>>,
> -}
> -
> -impl Serialize for ChallengeSchemaWrapper {
> - fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
> - where
> - S: serde::Serializer,
> - {
> - self.inner.serialize(serializer)
> - }
> -}
> -
> -struct CachedSchema {
> - schema: Arc<Vec<AcmeChallengeSchema>>,
> - cached_mtime: SystemTime,
> -}
> -
> -fn get_cached_challenge_schemas() -> Result<ChallengeSchemaWrapper, Error> {
> - static CACHE: LazyLock<Mutex<Option<CachedSchema>>> = LazyLock::new(|| Mutex::new(None));
> -
> - // the actual loading code
> - let mut last = CACHE.lock().unwrap();
> -
> - let actual_mtime = fs::metadata(crate::config::acme::ACME_DNS_SCHEMA_FN)?.modified()?;
> -
> - let schema = match &*last {
> - Some(CachedSchema {
> - schema,
> - cached_mtime,
> - }) if *cached_mtime >= actual_mtime => schema.clone(),
> - _ => {
> - let new_schema = Arc::new(crate::config::acme::load_dns_challenge_schema()?);
> - *last = Some(CachedSchema {
> - schema: Arc::clone(&new_schema),
> - cached_mtime: actual_mtime,
> - });
> - new_schema
> - }
> - };
> -
> - Ok(ChallengeSchemaWrapper { inner: schema })
> + Ok(proxmox_acme_api::KNOWN_ACME_DIRECTORIES)
> }
>
> #[api(
> @@ -455,69 +305,7 @@ fn get_cached_challenge_schemas() -> Result<ChallengeSchemaWrapper, Error> {
> )]
> /// Get named known ACME directory endpoints.
> fn get_challenge_schema() -> Result<ChallengeSchemaWrapper, Error> {
> - get_cached_challenge_schemas()
> -}
> -
> -#[api]
> -#[derive(Default, Deserialize, Serialize)]
> -#[serde(rename_all = "kebab-case")]
> -/// The API's format is inherited from PVE/PMG:
> -pub struct PluginConfig {
> - /// Plugin ID.
> - plugin: String,
> -
> - /// Plugin type.
> - #[serde(rename = "type")]
> - ty: String,
> -
> - /// DNS Api name.
> - #[serde(skip_serializing_if = "Option::is_none", default)]
> - api: Option<String>,
> -
> - /// Plugin configuration data.
> - #[serde(skip_serializing_if = "Option::is_none", default)]
> - data: Option<String>,
> -
> - /// Extra delay in seconds to wait before requesting validation.
> - ///
> - /// Allows to cope with long TTL of DNS records.
> - #[serde(skip_serializing_if = "Option::is_none", default)]
> - validation_delay: Option<u32>,
> -
> - /// Flag to disable the config.
> - #[serde(skip_serializing_if = "Option::is_none", default)]
> - disable: Option<bool>,
> -}
> -
> -// See PMG/PVE's $modify_cfg_for_api sub
> -fn modify_cfg_for_api(id: &str, ty: &str, data: &Value) -> PluginConfig {
> - let mut entry = data.clone();
> -
> - let obj = entry.as_object_mut().unwrap();
> - obj.remove("id");
> - obj.insert("plugin".to_string(), Value::String(id.to_owned()));
> - obj.insert("type".to_string(), Value::String(ty.to_owned()));
> -
> - // FIXME: This needs to go once the `Updater` is fixed.
> - // None of these should be able to fail unless the user changed the files by hand, in which
> - // case we leave the unmodified string in the Value for now. This will be handled with an error
> - // later.
> - if let Some(Value::String(ref mut data)) = obj.get_mut("data") {
> - if let Ok(new) = proxmox_base64::url::decode_no_pad(&data) {
> - if let Ok(utf8) = String::from_utf8(new) {
> - *data = utf8;
> - }
> - }
> - }
> -
> - // PVE/PMG do this explicitly for ACME plugins...
> - // obj.insert("digest".to_string(), Value::String(digest.clone()));
> -
> - serde_json::from_value(entry).unwrap_or_else(|_| PluginConfig {
> - plugin: "*Error*".to_string(),
> - ty: "*Error*".to_string(),
> - ..Default::default()
> - })
> + proxmox_acme_api::get_cached_challenge_schemas()
> }
>
> #[api(
> @@ -533,12 +321,7 @@ fn modify_cfg_for_api(id: &str, ty: &str, data: &Value) -> PluginConfig {
> )]
> /// List ACME challenge plugins.
> pub fn list_plugins(rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<PluginConfig>, Error> {
> - let (plugins, digest) = plugin::config()?;
> - rpcenv["digest"] = hex::encode(digest).into();
> - Ok(plugins
> - .iter()
> - .map(|(id, (ty, data))| modify_cfg_for_api(id, ty, data))
> - .collect())
> + proxmox_acme_api::list_plugins(rpcenv)
> }
>
> #[api(
> @@ -555,13 +338,7 @@ pub fn list_plugins(rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<PluginConfig>
> )]
> /// List ACME challenge plugins.
> pub fn get_plugin(id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<PluginConfig, Error> {
> - let (plugins, digest) = plugin::config()?;
> - rpcenv["digest"] = hex::encode(digest).into();
> -
> - match plugins.get(&id) {
> - Some((ty, data)) => Ok(modify_cfg_for_api(&id, ty, data)),
> - None => http_bail!(NOT_FOUND, "no such plugin"),
> - }
> + proxmox_acme_api::get_plugin(id, rpcenv)
> }
>
> // Currently we only have "the" standalone plugin and DNS plugins so we can just flatten a
> @@ -593,30 +370,7 @@ pub fn get_plugin(id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<PluginC
> )]
> /// Add ACME plugin configuration.
> pub fn add_plugin(r#type: String, core: DnsPluginCore, data: String) -> Result<(), Error> {
> - // Currently we only support DNS plugins and the standalone plugin is "fixed":
> - if r#type != "dns" {
> - param_bail!("type", "invalid ACME plugin type: {:?}", r#type);
> - }
> -
> - let data = String::from_utf8(proxmox_base64::decode(data)?)
> - .map_err(|_| format_err!("data must be valid UTF-8"))?;
> -
> - let id = core.id.clone();
> -
> - let _lock = plugin::lock()?;
> -
> - let (mut plugins, _digest) = plugin::config()?;
> - if plugins.contains_key(&id) {
> - param_bail!("id", "ACME plugin ID {:?} already exists", id);
> - }
> -
> - let plugin = serde_json::to_value(DnsPlugin { core, data })?;
> -
> - plugins.insert(id, r#type, plugin);
> -
> - plugin::save_config(&plugins)?;
> -
> - Ok(())
> + proxmox_acme_api::add_plugin(r#type, core, data)
> }
>
> #[api(
> @@ -632,26 +386,7 @@ pub fn add_plugin(r#type: String, core: DnsPluginCore, data: String) -> Result<(
> )]
> /// Delete an ACME plugin configuration.
> pub fn delete_plugin(id: String) -> Result<(), Error> {
> - let _lock = plugin::lock()?;
> -
> - let (mut plugins, _digest) = plugin::config()?;
> - if plugins.remove(&id).is_none() {
> - http_bail!(NOT_FOUND, "no such plugin");
> - }
> - plugin::save_config(&plugins)?;
> -
> - Ok(())
> -}
> -
> -#[api()]
> -#[derive(Serialize, Deserialize)]
> -#[serde(rename_all = "kebab-case")]
> -/// Deletable property name
> -pub enum DeletableProperty {
> - /// Delete the disable property
> - Disable,
> - /// Delete the validation-delay property
> - ValidationDelay,
> + proxmox_acme_api::delete_plugin(id)
> }
>
> #[api(
> @@ -673,12 +408,12 @@ pub enum DeletableProperty {
> type: Array,
> optional: true,
> items: {
> - type: DeletableProperty,
> + type: DeletablePluginProperty,
> }
> },
> digest: {
> - description: "Digest to protect against concurrent updates",
> optional: true,
> + type: ConfigDigest,
> },
> },
> },
> @@ -692,65 +427,8 @@ pub fn update_plugin(
> id: String,
> update: DnsPluginCoreUpdater,
> data: Option<String>,
> - delete: Option<Vec<DeletableProperty>>,
> - digest: Option<String>,
> + delete: Option<Vec<DeletablePluginProperty>>,
> + digest: Option<ConfigDigest>,
> ) -> Result<(), Error> {
> - let data = data
> - .as_deref()
> - .map(proxmox_base64::decode)
> - .transpose()?
> - .map(String::from_utf8)
> - .transpose()
> - .map_err(|_| format_err!("data must be valid UTF-8"))?;
> -
> - let _lock = plugin::lock()?;
> -
> - let (mut plugins, expected_digest) = plugin::config()?;
> -
> - if let Some(digest) = digest {
> - let digest = <[u8; 32]>::from_hex(digest)?;
> - crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
> - }
> -
> - match plugins.get_mut(&id) {
> - Some((ty, ref mut entry)) => {
> - if ty != "dns" {
> - bail!("cannot update plugin of type {:?}", ty);
> - }
> -
> - let mut plugin = DnsPlugin::deserialize(&*entry)?;
> -
> - if let Some(delete) = delete {
> - for delete_prop in delete {
> - match delete_prop {
> - DeletableProperty::ValidationDelay => {
> - plugin.core.validation_delay = None;
> - }
> - DeletableProperty::Disable => {
> - plugin.core.disable = None;
> - }
> - }
> - }
> - }
> - if let Some(data) = data {
> - plugin.data = data;
> - }
> - if let Some(api) = update.api {
> - plugin.core.api = api;
> - }
> - if update.validation_delay.is_some() {
> - plugin.core.validation_delay = update.validation_delay;
> - }
> - if update.disable.is_some() {
> - plugin.core.disable = update.disable;
> - }
> -
> - *entry = serde_json::to_value(plugin)?;
> - }
> - None => http_bail!(NOT_FOUND, "no such plugin"),
> - }
> -
> - plugin::save_config(&plugins)?;
> -
> - Ok(())
> + proxmox_acme_api::update_plugin(id, update, data, delete, digest)
> }
> diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
> index 64175aff..0ff496b6 100644
> --- a/src/api2/types/acme.rs
> +++ b/src/api2/types/acme.rs
> @@ -43,22 +43,6 @@ pub const ACME_DOMAIN_PROPERTY_SCHEMA: Schema =
> .format(&ApiStringFormat::PropertyString(&AcmeDomain::API_SCHEMA))
> .schema();
>
> -#[api(
> - properties: {
> - name: { type: String },
> - url: { type: String },
> - },
> -)]
> -/// An ACME directory endpoint with a name and URL.
> -#[derive(Serialize)]
> -pub struct KnownAcmeDirectory {
> - /// The ACME directory's name.
> - pub name: &'static str,
> -
> - /// The ACME directory's endpoint URL.
> - pub url: &'static str,
> -}
> -
> #[api(
> properties: {
> schema: {
> diff --git a/src/bin/proxmox_backup_manager/acme.rs b/src/bin/proxmox_backup_manager/acme.rs
> index 6ed61560..d11d7498 100644
> --- a/src/bin/proxmox_backup_manager/acme.rs
> +++ b/src/bin/proxmox_backup_manager/acme.rs
> @@ -4,14 +4,12 @@ use anyhow::{bail, Error};
> use serde_json::Value;
>
> use proxmox_acme::async_client::AcmeClient;
> -use proxmox_acme_api::AcmeAccountName;
> +use proxmox_acme_api::{AcmeAccountName, DnsPluginCore, KNOWN_ACME_DIRECTORIES};
> use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
> use proxmox_schema::api;
> use proxmox_sys::fs::file_get_contents;
>
> use proxmox_backup::api2;
> -use proxmox_backup::config::acme::plugin::DnsPluginCore;
> -use proxmox_backup::config::acme::KNOWN_ACME_DIRECTORIES;
>
> pub fn acme_mgmt_cli() -> CommandLineInterface {
> let cmd_def = CliCommandMap::new()
> @@ -122,7 +120,7 @@ async fn register_account(
>
> match input.trim().parse::<usize>() {
> Ok(n) if n < KNOWN_ACME_DIRECTORIES.len() => {
> - break (KNOWN_ACME_DIRECTORIES[n].url.to_owned(), false);
> + break (KNOWN_ACME_DIRECTORIES[n].url.to_string(), false);
> }
> Ok(n) if n == KNOWN_ACME_DIRECTORIES.len() => {
> input.clear();
> diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
> index e4639c53..01ab6223 100644
> --- a/src/config/acme/mod.rs
> +++ b/src/config/acme/mod.rs
> @@ -1,16 +1,15 @@
> use std::collections::HashMap;
> use std::ops::ControlFlow;
> -use std::path::Path;
>
> -use anyhow::{bail, format_err, Error};
> +use anyhow::Error;
> use serde_json::Value;
>
> use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
> -use proxmox_acme_api::AcmeAccountName;
> +use proxmox_acme_api::{AcmeAccountName, KnownAcmeDirectory, KNOWN_ACME_DIRECTORIES};
> use proxmox_sys::error::SysError;
> use proxmox_sys::fs::{file_read_string, CreateOptions};
>
> -use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
> +use crate::api2::types::AcmeChallengeSchema;
>
> pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
> pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
> @@ -35,23 +34,8 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
> create_acme_subdir(ACME_DIR)
> }
>
> -pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
> - KnownAcmeDirectory {
> - name: "Let's Encrypt V2",
> - url: "https://acme-v02.api.letsencrypt.org/directory",
> - },
> - KnownAcmeDirectory {
> - name: "Let's Encrypt V2 Staging",
> - url: "https://acme-staging-v02.api.letsencrypt.org/directory",
> - },
> -];
> -
> pub const DEFAULT_ACME_DIRECTORY_ENTRY: &KnownAcmeDirectory = &KNOWN_ACME_DIRECTORIES[0];
>
> -pub fn account_path(name: &str) -> String {
> - format!("{ACME_ACCOUNT_DIR}/{name}")
> -}
> -
> pub fn foreach_acme_account<F>(mut func: F) -> Result<(), Error>
> where
> F: FnMut(AcmeAccountName) -> ControlFlow<Result<(), Error>>,
> @@ -82,28 +66,6 @@ where
> }
> }
>
> -pub fn mark_account_deactivated(name: &str) -> Result<(), Error> {
> - let from = account_path(name);
> - for i in 0..100 {
> - let to = account_path(&format!("_deactivated_{name}_{i}"));
> - if !Path::new(&to).exists() {
> - return std::fs::rename(&from, &to).map_err(|err| {
> - format_err!(
> - "failed to move account path {:?} to {:?} - {}",
> - from,
> - to,
> - err
> - )
> - });
> - }
> - }
> - bail!(
> - "No free slot to rename deactivated account {:?}, please cleanup {:?}",
> - from,
> - ACME_ACCOUNT_DIR
> - );
> -}
> -
> pub fn load_dns_challenge_schema() -> Result<Vec<AcmeChallengeSchema>, Error> {
> let raw = file_read_string(ACME_DNS_SCHEMA_FN)?;
> let schemas: serde_json::Map<String, Value> = serde_json::from_str(&raw)?;
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* [pbs-devel] applied: [PATCH proxmox-backup v5 1/5] acme: clean up ACME-related imports
2026-01-08 11:26 13% ` [pbs-devel] [PATCH proxmox-backup v5 1/5] acme: clean up ACME-related imports Samuel Rufinatscha
@ 2026-01-13 13:45 5% ` Fabian Grünbichler
0 siblings, 0 replies; 200+ results
From: Fabian Grünbichler @ 2026-01-13 13:45 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
applied this one, since it is independent.
On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
> Clean up ACME-related imports to make it easier to switch to
> the factored out proxmox/ ACME implementation later.
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> src/acme/plugin.rs | 3 +--
> src/api2/config/acme.rs | 10 ++++------
> src/api2/node/certificates.rs | 7 +++----
> src/api2/types/acme.rs | 3 +--
> src/bin/proxmox-backup-manager.rs | 12 +++++-------
> src/bin/proxmox-backup-proxy.rs | 14 ++++++--------
> src/config/acme/mod.rs | 3 +--
> src/config/acme/plugin.rs | 2 +-
> src/config/node.rs | 6 ++----
> 9 files changed, 24 insertions(+), 36 deletions(-)
>
> diff --git a/src/acme/plugin.rs b/src/acme/plugin.rs
> index f756e9b5..993d729b 100644
> --- a/src/acme/plugin.rs
> +++ b/src/acme/plugin.rs
> @@ -19,11 +19,10 @@ use tokio::net::TcpListener;
> use tokio::process::Command;
>
> use proxmox_acme::{Authorization, Challenge};
> +use proxmox_rest_server::WorkerTask;
>
> use crate::acme::AcmeClient;
> use crate::api2::types::AcmeDomain;
> -use proxmox_rest_server::WorkerTask;
> -
> use crate::config::acme::plugin::{DnsPlugin, PluginData};
>
> const PROXMOX_ACME_SH_PATH: &str = "/usr/share/proxmox-acme/proxmox-acme";
> diff --git a/src/api2/config/acme.rs b/src/api2/config/acme.rs
> index 35c3fb77..18671639 100644
> --- a/src/api2/config/acme.rs
> +++ b/src/api2/config/acme.rs
> @@ -10,22 +10,20 @@ use serde::{Deserialize, Serialize};
> use serde_json::{json, Value};
> use tracing::{info, warn};
>
> +use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
> +use proxmox_acme::types::AccountData as AcmeAccountData;
> +use proxmox_acme::Account;
> +use proxmox_rest_server::WorkerTask;
> use proxmox_router::{
> http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
> };
> use proxmox_schema::{api, param_bail};
>
> -use proxmox_acme::types::AccountData as AcmeAccountData;
> -use proxmox_acme::Account;
> -
> -use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
> -
> use crate::acme::AcmeClient;
> use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
> use crate::config::acme::plugin::{
> self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
> };
> -use proxmox_rest_server::WorkerTask;
>
> pub(crate) const ROUTER: Router = Router::new()
> .get(&list_subdirs_api_method!(SUBDIRS))
> diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.rs
> index 61ef910e..6b1d87d2 100644
> --- a/src/api2/node/certificates.rs
> +++ b/src/api2/node/certificates.rs
> @@ -5,23 +5,22 @@ use anyhow::{bail, format_err, Error};
> use openssl::pkey::PKey;
> use openssl::x509::X509;
> use serde::{Deserialize, Serialize};
> -use tracing::info;
> +use tracing::{info, warn};
>
> +use pbs_api_types::{NODE_SCHEMA, PRIV_SYS_MODIFY};
> +use proxmox_rest_server::WorkerTask;
> use proxmox_router::list_subdirs_api_method;
> use proxmox_router::SubdirMap;
> use proxmox_router::{Permission, Router, RpcEnvironment};
> use proxmox_schema::api;
>
> -use pbs_api_types::{NODE_SCHEMA, PRIV_SYS_MODIFY};
> use pbs_buildcfg::configdir;
> use pbs_tools::cert;
> -use tracing::warn;
>
> use crate::acme::AcmeClient;
> use crate::api2::types::AcmeDomain;
> use crate::config::node::NodeConfig;
> use crate::server::send_certificate_renewal_mail;
> -use proxmox_rest_server::WorkerTask;
>
> pub const ROUTER: Router = Router::new()
> .get(&list_subdirs_api_method!(SUBDIRS))
> diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
> index 210ebdbc..8661f9e8 100644
> --- a/src/api2/types/acme.rs
> +++ b/src/api2/types/acme.rs
> @@ -1,9 +1,8 @@
> use serde::{Deserialize, Serialize};
> use serde_json::Value;
>
> -use proxmox_schema::{api, ApiStringFormat, ApiType, Schema, StringSchema};
> -
> use pbs_api_types::{DNS_ALIAS_FORMAT, DNS_NAME_FORMAT, PROXMOX_SAFE_ID_FORMAT};
> +use proxmox_schema::{api, ApiStringFormat, ApiType, Schema, StringSchema};
>
> #[api(
> properties: {
> diff --git a/src/bin/proxmox-backup-manager.rs b/src/bin/proxmox-backup-manager.rs
> index d9f41353..f8365070 100644
> --- a/src/bin/proxmox-backup-manager.rs
> +++ b/src/bin/proxmox-backup-manager.rs
> @@ -5,10 +5,6 @@ use std::str::FromStr;
> use anyhow::{format_err, Error};
> use serde_json::{json, Value};
>
> -use proxmox_router::{cli::*, RpcEnvironment};
> -use proxmox_schema::api;
> -use proxmox_sys::fs::CreateOptions;
> -
> use pbs_api_types::percent_encoding::percent_encode_component;
> use pbs_api_types::{
> BackupNamespace, GroupFilter, RateLimitConfig, SyncDirection, SyncJobConfig, DATASTORE_SCHEMA,
> @@ -18,12 +14,14 @@ use pbs_api_types::{
> VERIFICATION_OUTDATED_AFTER_SCHEMA, VERIFY_JOB_READ_THREADS_SCHEMA,
> VERIFY_JOB_VERIFY_THREADS_SCHEMA,
> };
> +use proxmox_rest_server::wait_for_local_worker;
> +use proxmox_router::{cli::*, RpcEnvironment};
> +use proxmox_schema::api;
> +use proxmox_sys::fs::CreateOptions;
> +
> use pbs_client::{display_task_log, view_task_result};
> use pbs_config::sync;
> use pbs_tools::json::required_string_param;
> -
> -use proxmox_rest_server::wait_for_local_worker;
> -
> use proxmox_backup::api2;
> use proxmox_backup::client_helpers::connect_to_localhost;
> use proxmox_backup::config;
> diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
> index 92a8cb3c..870208fe 100644
> --- a/src/bin/proxmox-backup-proxy.rs
> +++ b/src/bin/proxmox-backup-proxy.rs
> @@ -9,27 +9,25 @@ use hyper::http::request::Parts;
> use hyper::http::Response;
> use hyper::StatusCode;
> use hyper_util::server::graceful::GracefulShutdown;
> +use openssl::ssl::SslAcceptor;
> +use serde_json::{json, Value};
> use tracing::level_filters::LevelFilter;
> use tracing::{info, warn};
> use url::form_urlencoded;
>
> -use openssl::ssl::SslAcceptor;
> -use serde_json::{json, Value};
> -
> use proxmox_http::Body;
> use proxmox_http::RateLimiterTag;
> use proxmox_lang::try_block;
> +use proxmox_rest_server::{
> + cleanup_old_tasks, cookie_from_header, rotate_task_log_archive, ApiConfig, Redirector,
> + RestEnvironment, RestServer, WorkerTask,
> +};
> use proxmox_router::{RpcEnvironment, RpcEnvironmentType};
> use proxmox_sys::fs::CreateOptions;
> use proxmox_sys::logrotate::LogRotate;
>
> use pbs_datastore::DataStore;
>
> -use proxmox_rest_server::{
> - cleanup_old_tasks, cookie_from_header, rotate_task_log_archive, ApiConfig, Redirector,
> - RestEnvironment, RestServer, WorkerTask,
> -};
> -
> use proxmox_backup::{
> server::{
> auth::check_pbs_auth,
> diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
> index 274a23fd..ac89ae5e 100644
> --- a/src/config/acme/mod.rs
> +++ b/src/config/acme/mod.rs
> @@ -5,11 +5,10 @@ use std::path::Path;
> use anyhow::{bail, format_err, Error};
> use serde_json::Value;
>
> +use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
> use proxmox_sys::error::SysError;
> use proxmox_sys::fs::{file_read_string, CreateOptions};
>
> -use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
> -
> use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
>
> pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
> diff --git a/src/config/acme/plugin.rs b/src/config/acme/plugin.rs
> index 18e71199..8ce852ec 100644
> --- a/src/config/acme/plugin.rs
> +++ b/src/config/acme/plugin.rs
> @@ -4,10 +4,10 @@ use anyhow::Error;
> use serde::{Deserialize, Serialize};
> use serde_json::Value;
>
> +use pbs_api_types::PROXMOX_SAFE_ID_FORMAT;
> use proxmox_schema::{api, ApiType, Schema, StringSchema, Updater};
> use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
>
> -use pbs_api_types::PROXMOX_SAFE_ID_FORMAT;
> use pbs_config::{open_backup_lockfile, BackupLockGuard};
>
> pub const PLUGIN_ID_SCHEMA: Schema = StringSchema::new("ACME Challenge Plugin ID.")
> diff --git a/src/config/node.rs b/src/config/node.rs
> index d2d6e383..253b2e36 100644
> --- a/src/config/node.rs
> +++ b/src/config/node.rs
> @@ -4,14 +4,12 @@ use anyhow::{bail, Error};
> use openssl::ssl::{SslAcceptor, SslMethod};
> use serde::{Deserialize, Serialize};
>
> -use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
> -
> -use proxmox_http::ProxyConfig;
> -
> use pbs_api_types::{
> EMAIL_SCHEMA, MULTI_LINE_COMMENT_SCHEMA, OPENSSL_CIPHERS_TLS_1_2_SCHEMA,
> OPENSSL_CIPHERS_TLS_1_3_SCHEMA,
> };
> +use proxmox_http::ProxyConfig;
> +use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
>
> use pbs_buildcfg::configdir;
> use pbs_config::{open_backup_lockfile, BackupLockGuard};
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox v5 2/4] acme: introduce http_status module
2026-01-08 11:26 15% ` [pbs-devel] [PATCH proxmox v5 2/4] acme: introduce http_status module Samuel Rufinatscha
@ 2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-14 10:29 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2026-01-13 13:45 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
> Introduce an internal http_status module with the common ACME HTTP
> response codes, and replace use of crate::request::CREATED as well as
> direct numeric status code usages.
why not use http::status ? we already have this as dependency pretty
much everywhere we do anything HTTP related.. would also for nicer error
messages in case the status is not as expected..
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> proxmox-acme/src/account.rs | 8 ++++----
> proxmox-acme/src/async_client.rs | 4 ++--
> proxmox-acme/src/lib.rs | 2 ++
> proxmox-acme/src/request.rs | 11 ++++++++++-
> 4 files changed, 18 insertions(+), 7 deletions(-)
>
> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
> index d8eb3e73..ea1a3c60 100644
> --- a/proxmox-acme/src/account.rs
> +++ b/proxmox-acme/src/account.rs
> @@ -84,7 +84,7 @@ impl Account {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: crate::request::CREATED,
> + expected: crate::http_status::CREATED,
> };
>
> Ok(NewOrder::new(request))
> @@ -106,7 +106,7 @@ impl Account {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: 200,
> + expected: crate::http_status::OK,
> })
> }
>
> @@ -131,7 +131,7 @@ impl Account {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: 200,
> + expected: crate::http_status::OK,
> })
> }
>
> @@ -321,7 +321,7 @@ impl AccountCreator {
> method: "POST",
> content_type: crate::request::JSON_CONTENT_TYPE,
> body,
> - expected: crate::request::CREATED,
> + expected: crate::http_status::CREATED,
> })
> }
>
> diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
> index 2ff3ba22..043648bb 100644
> --- a/proxmox-acme/src/async_client.rs
> +++ b/proxmox-acme/src/async_client.rs
> @@ -498,7 +498,7 @@ impl AcmeClient {
> method: "GET",
> content_type: "",
> body: String::new(),
> - expected: 200,
> + expected: crate::http_status::OK,
> },
> nonce,
> )
> @@ -550,7 +550,7 @@ impl AcmeClient {
> method: "HEAD",
> content_type: "",
> body: String::new(),
> - expected: 200,
> + expected: crate::http_status::OK,
> },
> nonce,
> )
> diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
> index 6722030c..6051a025 100644
> --- a/proxmox-acme/src/lib.rs
> +++ b/proxmox-acme/src/lib.rs
> @@ -70,6 +70,8 @@ pub use order::Order;
> #[cfg(feature = "impl")]
> pub use order::NewOrder;
> #[cfg(feature = "impl")]
> +pub(crate) use request::http_status;
> +#[cfg(feature = "impl")]
> pub use request::ErrorResponse;
>
> /// Header name for nonces.
> diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
> index dadfc5af..341ce53e 100644
> --- a/proxmox-acme/src/request.rs
> +++ b/proxmox-acme/src/request.rs
> @@ -1,7 +1,6 @@
> use serde::Deserialize;
>
> pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
> -pub(crate) const CREATED: u16 = 201;
>
> /// A request which should be performed on the ACME provider.
> pub(crate) struct Request {
> @@ -21,6 +20,16 @@ pub(crate) struct Request {
> pub(crate) expected: u16,
> }
>
> +/// Common HTTP status codes used in ACME responses.
> +pub(crate) mod http_status {
> + /// 200 OK
> + pub(crate) const OK: u16 = 200;
> + /// 201 Created
> + pub(crate) const CREATED: u16 = 201;
> + /// 204 No Content
> + pub(crate) const NO_CONTENT: u16 = 204;
> +}
> +
> /// An ACME error response contains a specially formatted type string, and can optionally
> /// contain textual details and a set of sub problems.
> #[derive(Clone, Debug, Deserialize)]
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 2/5] acme: include proxmox-acme-api dependency
2026-01-08 11:26 15% ` [pbs-devel] [PATCH proxmox-backup v5 2/5] acme: include proxmox-acme-api dependency Samuel Rufinatscha
@ 2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-13 16:41 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2026-01-13 13:45 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
> PBS currently uses its own ACME client and API logic, while PDM uses the
> factored out proxmox-acme and proxmox-acme-api crates. This duplication
> risks differences in behaviour and requires ACME maintenance in two
> places. This patch is part of a series to move PBS over to the shared
> ACME stack.
this doesn't need to be in nearly every commit here.
adding the dependency and initializing things without using them also
has no stand-alone value, so this doesn't need to be its own commit.
we could have two commits:
- add proxmox-acme-api and use it for client and API
- remove src/acme since it is now unused
or three or more if you want to split out some of the API replacement where
there isn't a 1:1 relation between old and new code..
>
> Changes:
> - Add proxmox-acme-api with the "impl" feature as a dependency.
> - Initialize proxmox_acme_api in proxmox-backup- api, manager and proxy.
> * Inits PBS config dir /acme as proxmox ACME directory
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> Cargo.toml | 3 +++
> src/bin/proxmox-backup-api.rs | 2 ++
> src/bin/proxmox-backup-manager.rs | 2 ++
> src/bin/proxmox-backup-proxy.rs | 1 +
> 4 files changed, 8 insertions(+)
>
> diff --git a/Cargo.toml b/Cargo.toml
> index 1aa57ae5..feae351d 100644
> --- a/Cargo.toml
> +++ b/Cargo.toml
> @@ -101,6 +101,7 @@ pbs-api-types = "1.0.8"
> # other proxmox crates
> pathpatterns = "1"
> proxmox-acme = "1"
> +proxmox-acme-api = { version = "1", features = [ "impl" ] }
> pxar = "1"
>
> # PBS workspace
> @@ -251,6 +252,7 @@ pbs-api-types.workspace = true
>
> # in their respective repo
> proxmox-acme.workspace = true
> +proxmox-acme-api.workspace = true
> pxar.workspace = true
>
> # proxmox-backup workspace/internal crates
> @@ -269,6 +271,7 @@ proxmox-rrd-api-types.workspace = true
> [patch.crates-io]
> #pbs-api-types = { path = "../proxmox/pbs-api-types" }
> #proxmox-acme = { path = "../proxmox/proxmox-acme" }
> +#proxmox-acme-api = { path = "../proxmox/proxmox-acme-api" }
> #proxmox-api-macro = { path = "../proxmox/proxmox-api-macro" }
> #proxmox-apt = { path = "../proxmox/proxmox-apt" }
> #proxmox-apt-api-types = { path = "../proxmox/proxmox-apt-api-types" }
> diff --git a/src/bin/proxmox-backup-api.rs b/src/bin/proxmox-backup-api.rs
> index 417e9e97..d0091dca 100644
> --- a/src/bin/proxmox-backup-api.rs
> +++ b/src/bin/proxmox-backup-api.rs
> @@ -14,6 +14,7 @@ use proxmox_rest_server::{ApiConfig, RestServer};
> use proxmox_router::RpcEnvironmentType;
> use proxmox_sys::fs::CreateOptions;
>
> +use pbs_buildcfg::configdir;
> use proxmox_backup::auth_helpers::*;
> use proxmox_backup::config;
> use proxmox_backup::server::auth::check_pbs_auth;
> @@ -78,6 +79,7 @@ async fn run() -> Result<(), Error> {
> let mut command_sock = proxmox_daemon::command_socket::CommandSocket::new(backup_user.gid);
>
> proxmox_product_config::init(backup_user.clone(), pbs_config::priv_user()?);
> + proxmox_acme_api::init(configdir!("/acme"), true)?;
>
> let dir_opts = CreateOptions::new()
> .owner(backup_user.uid)
> diff --git a/src/bin/proxmox-backup-manager.rs b/src/bin/proxmox-backup-manager.rs
> index f8365070..30bc8da9 100644
> --- a/src/bin/proxmox-backup-manager.rs
> +++ b/src/bin/proxmox-backup-manager.rs
> @@ -19,6 +19,7 @@ use proxmox_router::{cli::*, RpcEnvironment};
> use proxmox_schema::api;
> use proxmox_sys::fs::CreateOptions;
>
> +use pbs_buildcfg::configdir;
> use pbs_client::{display_task_log, view_task_result};
> use pbs_config::sync;
> use pbs_tools::json::required_string_param;
> @@ -667,6 +668,7 @@ async fn run() -> Result<(), Error> {
> .init()?;
> proxmox_backup::server::notifications::init()?;
> proxmox_product_config::init(pbs_config::backup_user()?, pbs_config::priv_user()?);
> + proxmox_acme_api::init(configdir!("/acme"), false)?;
>
> let cmd_def = CliCommandMap::new()
> .insert("acl", acl_commands())
> diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
> index 870208fe..eea44a7d 100644
> --- a/src/bin/proxmox-backup-proxy.rs
> +++ b/src/bin/proxmox-backup-proxy.rs
> @@ -188,6 +188,7 @@ async fn run() -> Result<(), Error> {
> proxmox_backup::server::notifications::init()?;
> metric_collection::init()?;
> proxmox_product_config::init(pbs_config::backup_user()?, pbs_config::priv_user()?);
> + proxmox_acme_api::init(configdir!("/acme"), false)?;
>
> let mut indexpath = PathBuf::from(pbs_buildcfg::JS_DIR);
> indexpath.push("index.hbs");
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox v5 1/4] acme: reduce visibility of Request type
2026-01-08 11:26 10% ` [pbs-devel] [PATCH proxmox v5 1/4] acme: reduce visibility of Request type Samuel Rufinatscha
@ 2026-01-13 13:46 5% ` Fabian Grünbichler
2026-01-14 15:07 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2026-01-13 13:46 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
> Currently, the low-level ACME Request type is publicly exposed, even
> though users are expected to go through AcmeClient and
> proxmox-acme-api handlers. This patch reduces visibility so that
> the Request type and related fields/methods are crate-internal only.
it also removes a lot of public and private code entirely, not just
changing visibility.. I think those were intentionally there to allow
usage without the need to using either of the provided client
implementations (which are guarded behind feature flags).
if we say the crate should only be used via either the `client` or the
`async-client` then that's fine, but it should be made explicit and
discussed.. right now this is sort of half-way there - e.g., the
Account::new_order method was not made private, even though it makes no
sense anymore with those other methods/helpers removed..
this patch also breaks a few reference in doc comments that would need
to be dropped.
a note that this breaks the current usage of proxmox-acme in PBS would
also be good to have here, if this is kept..
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> proxmox-acme/src/account.rs | 94 ++-----------------------------
> proxmox-acme/src/async_client.rs | 2 +-
> proxmox-acme/src/authorization.rs | 30 ----------
> proxmox-acme/src/client.rs | 6 +-
> proxmox-acme/src/lib.rs | 4 --
> proxmox-acme/src/order.rs | 2 +-
> proxmox-acme/src/request.rs | 12 ++--
> 7 files changed, 16 insertions(+), 134 deletions(-)
>
> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
> index f763c1e9..d8eb3e73 100644
> --- a/proxmox-acme/src/account.rs
> +++ b/proxmox-acme/src/account.rs
> @@ -8,12 +8,11 @@ use openssl::pkey::{PKey, Private};
> use serde::{Deserialize, Serialize};
> use serde_json::Value;
>
> -use crate::authorization::{Authorization, GetAuthorization};
> use crate::b64u;
> use crate::directory::Directory;
> use crate::jws::Jws;
> use crate::key::{Jwk, PublicKey};
> -use crate::order::{NewOrder, Order, OrderData};
> +use crate::order::{NewOrder, OrderData};
> use crate::request::Request;
> use crate::types::{AccountData, AccountStatus, ExternalAccountBinding};
> use crate::Error;
> @@ -92,7 +91,7 @@ impl Account {
> }
>
> /// Prepare a "POST-as-GET" request to fetch data. Low level helper.
> - pub fn get_request(&self, url: &str, nonce: &str) -> Result<Request, Error> {
> + pub(crate) fn get_request(&self, url: &str, nonce: &str) -> Result<Request, Error> {
> let key = PKey::private_key_from_pem(self.private_key.as_bytes())?;
> let body = serde_json::to_string(&Jws::new_full(
> &key,
> @@ -112,7 +111,7 @@ impl Account {
> }
>
> /// Prepare a JSON POST request. Low level helper.
> - pub fn post_request<T: Serialize>(
> + pub(crate) fn post_request<T: Serialize>(
> &self,
> url: &str,
> nonce: &str,
> @@ -136,31 +135,6 @@ impl Account {
> })
> }
>
> - /// Prepare a JSON POST request.
> - fn post_request_raw_payload(
> - &self,
> - url: &str,
> - nonce: &str,
> - payload: String,
> - ) -> Result<Request, Error> {
> - let key = PKey::private_key_from_pem(self.private_key.as_bytes())?;
> - let body = serde_json::to_string(&Jws::new_full(
> - &key,
> - Some(self.location.clone()),
> - url.to_owned(),
> - nonce.to_owned(),
> - payload,
> - )?)?;
> -
> - Ok(Request {
> - url: url.to_owned(),
> - method: "POST",
> - content_type: crate::request::JSON_CONTENT_TYPE,
> - body,
> - expected: 200,
> - })
> - }
> -
> /// Get the "key authorization" for a token.
> pub fn key_authorization(&self, token: &str) -> Result<String, Error> {
> let key = PKey::private_key_from_pem(self.private_key.as_bytes())?;
> @@ -176,64 +150,6 @@ impl Account {
> Ok(b64u::encode(digest))
> }
>
> - /// Prepare a request to update account data.
> - ///
> - /// This is a rather low level interface. You should know what you're doing.
> - pub fn update_account_request<T: Serialize>(
> - &self,
> - nonce: &str,
> - data: &T,
> - ) -> Result<Request, Error> {
> - self.post_request(&self.location, nonce, data)
> - }
> -
> - /// Prepare a request to deactivate this account.
> - pub fn deactivate_account_request<T: Serialize>(&self, nonce: &str) -> Result<Request, Error> {
> - self.post_request_raw_payload(
> - &self.location,
> - nonce,
> - r#"{"status":"deactivated"}"#.to_string(),
> - )
> - }
> -
> - /// Prepare a request to query an Authorization for an Order.
> - ///
> - /// Returns `Ok(None)` if `auth_index` is out of out of range. You can query the number of
> - /// authorizations from via [`Order::authorization_len`] or by manually inspecting its
> - /// `.data.authorization` vector.
> - pub fn get_authorization(
> - &self,
> - order: &Order,
> - auth_index: usize,
> - nonce: &str,
> - ) -> Result<Option<GetAuthorization>, Error> {
> - match order.authorization(auth_index) {
> - None => Ok(None),
> - Some(url) => Ok(Some(GetAuthorization::new(self.get_request(url, nonce)?))),
> - }
> - }
> -
> - /// Prepare a request to validate a Challenge from an Authorization.
> - ///
> - /// Returns `Ok(None)` if `challenge_index` is out of out of range. The challenge count is
> - /// available by inspecting the [`Authorization::challenges`] vector.
> - ///
> - /// This returns a raw `Request` since validation takes some time and the `Authorization`
> - /// object has to be re-queried and its `status` inspected.
> - pub fn validate_challenge(
> - &self,
> - authorization: &Authorization,
> - challenge_index: usize,
> - nonce: &str,
> - ) -> Result<Option<Request>, Error> {
> - match authorization.challenges.get(challenge_index) {
> - None => Ok(None),
> - Some(challenge) => self
> - .post_request_raw_payload(&challenge.url, nonce, "{}".to_string())
> - .map(Some),
> - }
> - }
> -
> /// Prepare a request to revoke a certificate.
> ///
> /// The certificate can be either PEM or DER formatted.
> @@ -274,7 +190,7 @@ pub struct CertificateRevocation<'a> {
>
> impl CertificateRevocation<'_> {
> /// Create the revocation request using the specified nonce for the given directory.
> - pub fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
> + pub(crate) fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
> let revoke_cert = directory.data.revoke_cert.as_ref().ok_or_else(|| {
> Error::Custom("no 'revokeCert' URL specified by provider".to_string())
> })?;
> @@ -364,7 +280,7 @@ impl AccountCreator {
> /// the resulting request.
> /// Changing the private key between using the request and passing the response to
> /// [`response`](AccountCreator::response()) will render the account unusable!
> - pub fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
> + pub(crate) fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
> let key = self.key.as_deref().ok_or(Error::MissingKey)?;
> let url = directory.new_account_url().ok_or_else(|| {
> Error::Custom("no 'newAccount' URL specified by provider".to_string())
> diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
> index dc755fb9..2ff3ba22 100644
> --- a/proxmox-acme/src/async_client.rs
> +++ b/proxmox-acme/src/async_client.rs
> @@ -10,7 +10,7 @@ use proxmox_http::{client::Client, Body};
>
> use crate::account::AccountCreator;
> use crate::order::{Order, OrderData};
> -use crate::Request as AcmeRequest;
> +use crate::request::Request as AcmeRequest;
> use crate::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
>
> /// A non-blocking Acme client using tokio/hyper.
> diff --git a/proxmox-acme/src/authorization.rs b/proxmox-acme/src/authorization.rs
> index 28bc1b4b..7027381a 100644
> --- a/proxmox-acme/src/authorization.rs
> +++ b/proxmox-acme/src/authorization.rs
> @@ -6,8 +6,6 @@ use serde::{Deserialize, Serialize};
> use serde_json::Value;
>
> use crate::order::Identifier;
> -use crate::request::Request;
> -use crate::Error;
>
> /// Status of an [`Authorization`].
> #[derive(Clone, Copy, Debug, Eq, PartialEq, Deserialize, Serialize)]
> @@ -132,31 +130,3 @@ impl Challenge {
> fn is_false(b: &bool) -> bool {
> !*b
> }
> -
> -/// Represents an in-flight query for an authorization.
> -///
> -/// This is created via [`Account::get_authorization`](crate::Account::get_authorization()).
> -pub struct GetAuthorization {
> - //order: OrderData,
> - /// The request to send to the ACME provider. This is wrapped in an option in order to allow
> - /// moving it out instead of copying the contents.
> - ///
> - /// When generated via [`Account::get_authorization`](crate::Account::get_authorization()),
> - /// this is guaranteed to be `Some`.
> - ///
> - /// The response should be passed to the the [`response`](GetAuthorization::response()) method.
> - pub request: Option<Request>,
> -}
> -
> -impl GetAuthorization {
> - pub(crate) fn new(request: Request) -> Self {
> - Self {
> - request: Some(request),
> - }
> - }
> -
> - /// Deal with the response we got from the server.
> - pub fn response(self, response_body: &[u8]) -> Result<Authorization, Error> {
> - Ok(serde_json::from_slice(response_body)?)
> - }
> -}
> diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
> index 931f7245..5c812567 100644
> --- a/proxmox-acme/src/client.rs
> +++ b/proxmox-acme/src/client.rs
> @@ -7,8 +7,8 @@ use serde::{Deserialize, Serialize};
> use crate::b64u;
> use crate::error;
> use crate::order::OrderData;
> -use crate::request::ErrorResponse;
> -use crate::{Account, Authorization, Challenge, Directory, Error, Order, Request};
> +use crate::request::{ErrorResponse, Request};
> +use crate::{Account, Authorization, Challenge, Directory, Error, Order};
>
> macro_rules! format_err {
> ($($fmt:tt)*) => { Error::Client(format!($($fmt)*)) };
> @@ -564,7 +564,7 @@ impl Client {
> }
>
> /// Low-level API to run an n API request. This automatically updates the current nonce!
> - pub fn run_request(&mut self, request: Request) -> Result<HttpResponse, Error> {
> + pub(crate) fn run_request(&mut self, request: Request) -> Result<HttpResponse, Error> {
> self.inner.run_request(request)
> }
>
> diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
> index df722629..6722030c 100644
> --- a/proxmox-acme/src/lib.rs
> +++ b/proxmox-acme/src/lib.rs
> @@ -66,10 +66,6 @@ pub use error::Error;
> #[doc(inline)]
> pub use order::Order;
>
> -#[cfg(feature = "impl")]
> -#[doc(inline)]
> -pub use request::Request;
> -
> // we don't inline these:
> #[cfg(feature = "impl")]
> pub use order::NewOrder;
> diff --git a/proxmox-acme/src/order.rs b/proxmox-acme/src/order.rs
> index b6551004..432a81a4 100644
> --- a/proxmox-acme/src/order.rs
> +++ b/proxmox-acme/src/order.rs
> @@ -153,7 +153,7 @@ pub struct NewOrder {
> //order: OrderData,
> /// The request to execute to place the order. When creating a [`NewOrder`] via
> /// [`Account::new_order`](crate::Account::new_order) this is guaranteed to be `Some`.
> - pub request: Option<Request>,
> + pub(crate) request: Option<Request>,
> }
>
> impl NewOrder {
> diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
> index 78a90913..dadfc5af 100644
> --- a/proxmox-acme/src/request.rs
> +++ b/proxmox-acme/src/request.rs
> @@ -4,21 +4,21 @@ pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
> pub(crate) const CREATED: u16 = 201;
>
> /// A request which should be performed on the ACME provider.
> -pub struct Request {
> +pub(crate) struct Request {
> /// The complete URL to send the request to.
> - pub url: String,
> + pub(crate) url: String,
>
> /// The HTTP method name to use.
> - pub method: &'static str,
> + pub(crate) method: &'static str,
>
> /// The `Content-Type` header to pass along.
> - pub content_type: &'static str,
> + pub(crate) content_type: &'static str,
>
> /// The body to pass along with request, or an empty string.
> - pub body: String,
> + pub(crate) body: String,
>
> /// The expected status code a compliant ACME provider will return on success.
> - pub expected: u16,
> + pub(crate) expected: u16,
> }
>
> /// An ACME error response contains a specially formatted type string, and can optionally
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests
2026-01-08 11:26 11% [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
` (8 preceding siblings ...)
2026-01-08 11:26 7% ` [pbs-devel] [PATCH proxmox-backup v5 5/5] acme: certificate ordering through proxmox-acme-api Samuel Rufinatscha
@ 2026-01-13 13:48 5% ` Fabian Grünbichler
2026-01-15 10:24 0% ` Max R. Carrara
9 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2026-01-13 13:48 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
> Hi,
>
> this series fixes account registration for ACME providers that return
> HTTP 204 No Content to the newNonce request. Currently, both the PBS
> ACME client and the shared ACME client in proxmox-acme only accept
> HTTP 200 OK for this request. The issue was observed in PBS against a
> custom ACME deployment and reported as bug #6939 [1].
sent some feedback for individual patches, one thing to explicitly test
is that existing accounts and DNS plugin configuration continue to work
after the switch over - AFAICT from the testing description below that
was not done (or not noted?).
>
> ## Problem
>
> During ACME account registration, PBS first fetches an anti-replay
> nonce by sending a HEAD request to the CA’s newNonce URL.
> RFC 8555 §7.2 [2] states that:
>
> * the server MUST include a Replay-Nonce header with a fresh nonce,
> * the server SHOULD use status 200 OK for the HEAD request,
> * the server MUST also handle GET on the same resource and may return
> 204 No Content with an empty body.
>
> The reporter observed the following error message:
>
> *ACME server responded with unexpected status code: 204*
>
> and mentioned that the issue did not appear with PVE 9 [1]. Looking at
> PVE’s Perl ACME client [3], it uses a GET request instead of HEAD and
> accepts any 2xx success code when retrieving the nonce. This difference
> in behavior does not affect functionality but is worth noting for
> consistency across implementations.
>
> ## Approach
>
> To support ACME providers which return 204 No Content, the Rust ACME
> clients in proxmox-backup and proxmox need to treat both 200 OK and 204
> No Content as valid responses for the nonce request, as long as a
> Replay-Nonce header is present.
>
> This series changes the expected field of the internal Request type
> from a single u16 to a list of allowed status codes
> (e.g. &'static [u16]), so one request can explicitly accept multiple
> success codes.
>
> To avoid fixing the issue twice (once in PBS’ own ACME client and once
> in the shared Rust client), this series first refactors PBS to use the
> shared AcmeClient from proxmox-acme / proxmox-acme-api, similar to PDM,
> and then applies the bug fix in that shared implementation so that all
> consumers benefit from the more tolerant behavior.
>
> ## Testing
>
> *Testing the refactor*
>
> To test the refactor, I
> (1) installed latest stable PBS on a VM
> (2) created .deb package from latest PBS (master), containing the
> refactor
> (3) installed created .deb package
> (4) installed Pebble from Let's Encrypt [5] on the same VM
> (5) created an ACME account and ordered the new certificate for the
> host domain.
>
> Steps to reproduce:
>
> (1) install latest stable PBS on a VM, create .deb package from latest
> PBS (master) containing the refactor, install created .deb package
> (2) install Pebble from Let's Encrypt [5] on the same VM:
>
> cd
> apt update
> apt install -y golang git
> git clone https://github.com/letsencrypt/pebble
> cd pebble
> go build ./cmd/pebble
>
> then, download and trust the Pebble cert:
>
> wget https://raw.githubusercontent.com/letsencrypt/pebble/main/test/certs/pebble.minica.pem
> cp pebble.minica.pem /usr/local/share/ca-certificates/pebble.minica.crt
> update-ca-certificates
>
> We want Pebble to perform HTTP-01 validation against port 80, because
> PBS’s standalone plugin will bind port 80. Set httpPort to 80.
>
> nano ./test/config/pebble-config.json
>
> Start the Pebble server in the background:
>
> ./pebble -config ./test/config/pebble-config.json &
>
> Create a Pebble ACME account:
>
> proxmox-backup-manager acme account register default admin@example.com --directory 'https://127.0.0.1:14000/dir'
>
> To verify persistence of the account I checked
>
> ls /etc/proxmox-backup/acme/accounts
>
> Verified if update-account works
>
> proxmox-backup-manager acme account update default --contact "a@example.com,b@example.com"
> proxmox-backup-manager acme account info default
>
> In the PBS GUI, you can create a new domain. You can use your host
> domain name (see /etc/hosts). Select the created account and order the
> certificate.
>
> After a page reload, you might need to accept the new certificate in the browser.
> In the PBS dashboard, you should see the new Pebble certificate.
>
> *Note: on reboot, the created Pebble ACME account will be gone and you
> will need to create a new one. Pebble does not persist account info.
> In that case remove the previously created account in
> /etc/proxmox-backup/acme/accounts.
>
> *Testing the newNonce fix*
>
> To prove the ACME newNonce fix, I put nginx in front of Pebble, to
> intercept the newNonce request in order to return 204 No Content
> instead of 200 OK, all other requests are unchanged and forwarded to
> Pebble. Requires trusting the nginx CAs via
> /usr/local/share/ca-certificates + update-ca-certificates on the VM.
>
> Then I ran following command against nginx:
>
> proxmox-backup-manager acme account register proxytest root@backup.local --directory 'https://nginx-address/dir
>
> The account could be created successfully. When adjusting the nginx
> configuration to return any other non-expected success status code,
> PBS rejects as expected.
>
> ## Patch summary
>
> 0001 – [PATCH proxmox v5 1/4] acme: reduce visibility of Request type
> Restricts the visibility of the low-level Request type. Consumers
> should rely on proxmox-acme-api or AcmeClient handlers.
>
> 0002– [PATCH proxmox v5 2/4] acme: introduce http_status module
>
> 0003 – [PATCH proxmox v5 3/4] fix #6939: acme: support servers
> returning 204 for nonce requests
> Adjusts nonce handling to support ACME servers that return HTTP 204
> (No Content) for new-nonce requests.
>
> 0004 – [PATCH proxmox v5 4/4] acme-api: add helper to load client for
> an account
> Introduces a helper function to load an ACME client instance for a
> given account. Required for the following PBS ACME refactor.
>
> 0005 – [PATCH proxmox-backup v5 1/5] acme: clean up ACME-related imports
>
> 0006 – [PATCH proxmox-backup v5 2/5] acme: include proxmox-acme-api
> dependency
> Prepares the codebase to use the factored out ACME API impl.
>
> 0007 – [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient
> Removes the local AcmeClient implementation. Represents the minimal
> set of changes to replace it with the factored out AcmeClient.
>
> 0008 – [PATCH proxmox-backup v5 4/5] acme: change API impls to use
> proxmox-acme-api handlers
>
> 0009 – [PATCH proxmox-backup v5 5/5] acme: certificate ordering through
> proxmox-acme-api
>
> Thanks for considering this patch series, I look forward to your
> feedback.
>
> Best,
> Samuel Rufinatscha
>
> ## Changelog
>
> Changes from v4 to v5:
>
> * rebased series
> * re-ordered series (proxmox-acme fix first)
> * proxmox-backup: cleaned up imports based on an initial clean-up patch
> * proxmox-acme: removed now unused post_request_raw_payload(),
> update_account_request(), deactivate_account_request()
> * proxmox-acme: removed now obsolete/unused get_authorization() and
> GetAuthorization impl
>
> Verified removal by compiling PBS, PDM, and proxmox-perl-rs
> with all features.
>
> Changes from v3 to v4:
>
> * add proxmox-acme-api as a dependency and initialize it in
> PBS so PBS can use the shared ACME API instead.
> * remove the PBS-local AcmeClient implementation and switch PBS
> over to the shared proxmox-acme async client.
> * rework PBS’ ACME API endpoints to delegate to
> proxmox-acme-api handlers instead of duplicating logic locally.
> * move PBS’ ACME certificate ordering logic over to
> proxmox-acme-api, keeping only certificate installation/reload in PBS.
> * add a load_client_with_account helper in proxmox-acme-api so PBS
> (and others) can construct an AcmeClient for a configured account
> without duplicating boilerplate.
> * hide the low-level Request type and its fields behind constructors
> / reduced visibility so changes to “expected” no longer affect the
> public API as they did in v3.
> * split out the HTTP status constants into an internal http_status
> module as a separate preparatory cleanup before the bug fix, instead
> of doing this inline like in v3.
> * Rebased on top of the refactor: keep the same behavioural fix as in
> v3 accept 204 for newNonce with Replay-Nonce present), but implement
> it on top of the http_status module that is part of the refactor.
>
> Changes from v2 to v3:
>
> * rename `http_success` module to `http_status`
> * replace `http_success` usage
> * introduced `http_success` module to contain the http success codes
> * replaced `Vec<u16>` with `&[u16]` for expected codes to avoid allocations.
> * clarified the PVEs Perl ACME client behaviour in the commit message.
> * integrated the `http_success` module, replacing `Vec<u16>` with `&[u16]`
> * clarified the PVEs Perl ACME client behaviour in the commit message.
>
> [1] Bugzilla report #6939:
> [https://bugzilla.proxmox.com/show_bug.cgi?id=6939](https://bugzilla.proxmox.com/show_bug.cgi?id=6939)
> [2] RFC 8555 (ACME):
> [https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2](https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2)
> [3] PVE’s Perl ACME client (allow 2xx codes for nonce requests):
> [https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597)
> [4] Pebble ACME server:
> [https://github.com/letsencrypt/pebble](https://github.com/letsencrypt/pebble)
> [5] Pebble ACME server (perform GET request:
> [https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219)
>
> proxmox:
>
> Samuel Rufinatscha (4):
> acme: reduce visibility of Request type
> acme: introduce http_status module
> fix #6939: acme: support servers returning 204 for nonce requests
> acme-api: add helper to load client for an account
>
> proxmox-acme-api/src/account_api_impl.rs | 5 ++
> proxmox-acme-api/src/lib.rs | 3 +-
> proxmox-acme/src/account.rs | 102 ++---------------------
> proxmox-acme/src/async_client.rs | 8 +-
> proxmox-acme/src/authorization.rs | 30 -------
> proxmox-acme/src/client.rs | 8 +-
> proxmox-acme/src/lib.rs | 6 +-
> proxmox-acme/src/order.rs | 2 +-
> proxmox-acme/src/request.rs | 25 ++++--
> 9 files changed, 44 insertions(+), 145 deletions(-)
>
>
> proxmox-backup:
>
> Samuel Rufinatscha (5):
> acme: clean up ACME-related imports
> acme: include proxmox-acme-api dependency
> acme: drop local AcmeClient
> acme: change API impls to use proxmox-acme-api handlers
> acme: certificate ordering through proxmox-acme-api
>
> Cargo.toml | 3 +
> src/acme/client.rs | 691 -------------------------
> src/acme/mod.rs | 5 -
> src/acme/plugin.rs | 336 ------------
> src/api2/config/acme.rs | 406 ++-------------
> src/api2/node/certificates.rs | 232 ++-------
> src/api2/types/acme.rs | 98 ----
> src/api2/types/mod.rs | 3 -
> src/bin/proxmox-backup-api.rs | 2 +
> src/bin/proxmox-backup-manager.rs | 14 +-
> src/bin/proxmox-backup-proxy.rs | 15 +-
> src/bin/proxmox_backup_manager/acme.rs | 21 +-
> src/config/acme/mod.rs | 55 +-
> src/config/acme/plugin.rs | 92 +---
> src/config/node.rs | 31 +-
> src/lib.rs | 2 -
> 16 files changed, 109 insertions(+), 1897 deletions(-)
> delete mode 100644 src/acme/client.rs
> delete mode 100644 src/acme/mod.rs
> delete mode 100644 src/acme/plugin.rs
> delete mode 100644 src/api2/types/acme.rs
>
>
> Summary over all repositories:
> 25 files changed, 153 insertions(+), 2042 deletions(-)
>
> --
> Generated by git-murpp 0.8.1
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 2/5] acme: include proxmox-acme-api dependency
2026-01-13 13:45 5% ` Fabian Grünbichler
@ 2026-01-13 16:41 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-13 16:41 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 1/13/26 2:45 PM, Fabian Grünbichler wrote:
> On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
>> PBS currently uses its own ACME client and API logic, while PDM uses the
>> factored out proxmox-acme and proxmox-acme-api crates. This duplication
>> risks differences in behaviour and requires ACME maintenance in two
>> places. This patch is part of a series to move PBS over to the shared
>> ACME stack.
>
> this doesn't need to be in nearly every commit here.
Makes sense, will remove!
>
> adding the dependency and initializing things without using them also
> has no stand-alone value, so this doesn't need to be its own commit.
>
I thought about this - I decided to add it as a dedicated commit to
improve visibility for review, to make sure call/init sites with params
are clear / to avoid it isn't going overlooked in the refactor.
But I see what you mean, it fits better with the changes from next
patch. Will adjust to this!
> we could have two commits:
> - add proxmox-acme-api and use it for client and API
> - remove src/acme since it is now unused
>
> or three or more if you want to split out some of the API replacement where
> there isn't a 1:1 relation between old and new code..
>
I think the 2 commits approach fits, and would hold the API
changes well together - so let's go with this!
I will try to extract the src/api2 changes from 5/5 which should work.
And then probably is src/config/acme/plugin.rs still left, which will
be removed together with src/acme as part of patch 2.
>>
>> Changes:
>> - Add proxmox-acme-api with the "impl" feature as a dependency.
>> - Initialize proxmox_acme_api in proxmox-backup- api, manager and proxy.
>> * Inits PBS config dir /acme as proxmox ACME directory
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> Cargo.toml | 3 +++
>> src/bin/proxmox-backup-api.rs | 2 ++
>> src/bin/proxmox-backup-manager.rs | 2 ++
>> src/bin/proxmox-backup-proxy.rs | 1 +
>> 4 files changed, 8 insertions(+)
>>
>> diff --git a/Cargo.toml b/Cargo.toml
>> index 1aa57ae5..feae351d 100644
>> --- a/Cargo.toml
>> +++ b/Cargo.toml
>> @@ -101,6 +101,7 @@ pbs-api-types = "1.0.8"
>> # other proxmox crates
>> pathpatterns = "1"
>> proxmox-acme = "1"
>> +proxmox-acme-api = { version = "1", features = [ "impl" ] }
>> pxar = "1"
>>
>> # PBS workspace
>> @@ -251,6 +252,7 @@ pbs-api-types.workspace = true
>>
>> # in their respective repo
>> proxmox-acme.workspace = true
>> +proxmox-acme-api.workspace = true
>> pxar.workspace = true
>>
>> # proxmox-backup workspace/internal crates
>> @@ -269,6 +271,7 @@ proxmox-rrd-api-types.workspace = true
>> [patch.crates-io]
>> #pbs-api-types = { path = "../proxmox/pbs-api-types" }
>> #proxmox-acme = { path = "../proxmox/proxmox-acme" }
>> +#proxmox-acme-api = { path = "../proxmox/proxmox-acme-api" }
>> #proxmox-api-macro = { path = "../proxmox/proxmox-api-macro" }
>> #proxmox-apt = { path = "../proxmox/proxmox-apt" }
>> #proxmox-apt-api-types = { path = "../proxmox/proxmox-apt-api-types" }
>> diff --git a/src/bin/proxmox-backup-api.rs b/src/bin/proxmox-backup-api.rs
>> index 417e9e97..d0091dca 100644
>> --- a/src/bin/proxmox-backup-api.rs
>> +++ b/src/bin/proxmox-backup-api.rs
>> @@ -14,6 +14,7 @@ use proxmox_rest_server::{ApiConfig, RestServer};
>> use proxmox_router::RpcEnvironmentType;
>> use proxmox_sys::fs::CreateOptions;
>>
>> +use pbs_buildcfg::configdir;
>> use proxmox_backup::auth_helpers::*;
>> use proxmox_backup::config;
>> use proxmox_backup::server::auth::check_pbs_auth;
>> @@ -78,6 +79,7 @@ async fn run() -> Result<(), Error> {
>> let mut command_sock = proxmox_daemon::command_socket::CommandSocket::new(backup_user.gid);
>>
>> proxmox_product_config::init(backup_user.clone(), pbs_config::priv_user()?);
>> + proxmox_acme_api::init(configdir!("/acme"), true)?;
>>
>> let dir_opts = CreateOptions::new()
>> .owner(backup_user.uid)
>> diff --git a/src/bin/proxmox-backup-manager.rs b/src/bin/proxmox-backup-manager.rs
>> index f8365070..30bc8da9 100644
>> --- a/src/bin/proxmox-backup-manager.rs
>> +++ b/src/bin/proxmox-backup-manager.rs
>> @@ -19,6 +19,7 @@ use proxmox_router::{cli::*, RpcEnvironment};
>> use proxmox_schema::api;
>> use proxmox_sys::fs::CreateOptions;
>>
>> +use pbs_buildcfg::configdir;
>> use pbs_client::{display_task_log, view_task_result};
>> use pbs_config::sync;
>> use pbs_tools::json::required_string_param;
>> @@ -667,6 +668,7 @@ async fn run() -> Result<(), Error> {
>> .init()?;
>> proxmox_backup::server::notifications::init()?;
>> proxmox_product_config::init(pbs_config::backup_user()?, pbs_config::priv_user()?);
>> + proxmox_acme_api::init(configdir!("/acme"), false)?;
>>
>> let cmd_def = CliCommandMap::new()
>> .insert("acl", acl_commands())
>> diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
>> index 870208fe..eea44a7d 100644
>> --- a/src/bin/proxmox-backup-proxy.rs
>> +++ b/src/bin/proxmox-backup-proxy.rs
>> @@ -188,6 +188,7 @@ async fn run() -> Result<(), Error> {
>> proxmox_backup::server::notifications::init()?;
>> metric_collection::init()?;
>> proxmox_product_config::init(pbs_config::backup_user()?, pbs_config::priv_user()?);
>> + proxmox_acme_api::init(configdir!("/acme"), false)?;
>>
>> let mut indexpath = PathBuf::from(pbs_buildcfg::JS_DIR);
>> indexpath.push("index.hbs");
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 5/5] acme: certificate ordering through proxmox-acme-api
2026-01-13 13:45 5% ` Fabian Grünbichler
@ 2026-01-13 16:51 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-13 16:51 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
comments inline
On 1/13/26 2:45 PM, Fabian Grünbichler wrote:
> On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
>> PBS currently uses its own ACME client and API logic, while PDM uses the
>> factored out proxmox-acme and proxmox-acme-api crates. This duplication
>> risks differences in behaviour and requires ACME maintenance in two
>> places. This patch is part of a series to move PBS over to the shared
>> ACME stack.
>>
>> Changes:
>> - Replace the custom ACME order/authorization loop in node certificates
>> with a call to proxmox_acme_api::order_certificate.
>> - Build domain + config data as proxmox-acme-api types
>> - Remove obsolete local ACME ordering and plugin glue code.
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> src/acme/mod.rs | 2 -
>> src/acme/plugin.rs | 335 ----------------------------------
>> src/api2/node/certificates.rs | 229 ++++-------------------
>> src/api2/types/acme.rs | 73 --------
>> src/api2/types/mod.rs | 3 -
>> src/config/acme/mod.rs | 8 +-
>> src/config/acme/plugin.rs | 92 +---------
>> src/config/node.rs | 20 +-
>> src/lib.rs | 2 -
>> 9 files changed, 38 insertions(+), 726 deletions(-)
>> delete mode 100644 src/acme/mod.rs
>> delete mode 100644 src/acme/plugin.rs
>> delete mode 100644 src/api2/types/acme.rs
>>
>
> [..]
>
>> diff --git a/src/api2/node/certificates.rs b/src/api2/node/certificates.rs
>> index 47ff8de5..73401c41 100644
>> --- a/src/api2/node/certificates.rs
>> +++ b/src/api2/node/certificates.rs
>> @@ -1,14 +1,11 @@
>> -use std::sync::Arc;
>> -use std::time::Duration;
>> -
>> use anyhow::{bail, format_err, Error};
>> use openssl::pkey::PKey;
>> use openssl::x509::X509;
>> use serde::{Deserialize, Serialize};
>> -use tracing::{info, warn};
>> +use tracing::info;
>>
>> use pbs_api_types::{NODE_SCHEMA, PRIV_SYS_MODIFY};
>> -use proxmox_acme::async_client::AcmeClient;
>> +use proxmox_acme_api::AcmeDomain;
>> use proxmox_rest_server::WorkerTask;
>> use proxmox_router::list_subdirs_api_method;
>> use proxmox_router::SubdirMap;
>> @@ -18,8 +15,6 @@ use proxmox_schema::api;
>> use pbs_buildcfg::configdir;
>> use pbs_tools::cert;
>>
>> -use crate::api2::types::AcmeDomain;
>> -use crate::config::node::NodeConfig;
>> use crate::server::send_certificate_renewal_mail;
>>
>> pub const ROUTER: Router = Router::new()
>> @@ -268,193 +263,6 @@ pub async fn delete_custom_certificate() -> Result<(), Error> {
>> Ok(())
>> }
>>
>> -struct OrderedCertificate {
>> - certificate: hyper::body::Bytes,
>> - private_key_pem: Vec<u8>,
>> -}
>> -
>> -async fn order_certificate(
>> - worker: Arc<WorkerTask>,
>> - node_config: &NodeConfig,
>> -) -> Result<Option<OrderedCertificate>, Error> {
>> - use proxmox_acme::authorization::Status;
>> - use proxmox_acme::order::Identifier;
>> -
>> - let domains = node_config.acme_domains().try_fold(
>> - Vec::<AcmeDomain>::new(),
>> - |mut acc, domain| -> Result<_, Error> {
>> - let mut domain = domain?;
>> - domain.domain.make_ascii_lowercase();
>> - if let Some(alias) = &mut domain.alias {
>> - alias.make_ascii_lowercase();
>> - }
>> - acc.push(domain);
>> - Ok(acc)
>> - },
>> - )?;
>> -
>> - let get_domain_config = |domain: &str| {
>> - domains
>> - .iter()
>> - .find(|d| d.domain == domain)
>> - .ok_or_else(|| format_err!("no config for domain '{}'", domain))
>> - };
>> -
>> - if domains.is_empty() {
>> - info!("No domains configured to be ordered from an ACME server.");
>> - return Ok(None);
>> - }
>> -
>> - let (plugins, _) = crate::config::acme::plugin::config()?;
>> -
>> - let mut acme = node_config.acme_client().await?;
>> -
>> - info!("Placing ACME order");
>> - let order = acme
>> - .new_order(domains.iter().map(|d| d.domain.to_ascii_lowercase()))
>> - .await?;
>> - info!("Order URL: {}", order.location);
>> -
>> - let identifiers: Vec<String> = order
>> - .data
>> - .identifiers
>> - .iter()
>> - .map(|identifier| match identifier {
>> - Identifier::Dns(domain) => domain.clone(),
>> - })
>> - .collect();
>> -
>> - for auth_url in &order.data.authorizations {
>> - info!("Getting authorization details from '{auth_url}'");
>> - let mut auth = acme.get_authorization(auth_url).await?;
>> -
>> - let domain = match &mut auth.identifier {
>> - Identifier::Dns(domain) => domain.to_ascii_lowercase(),
>> - };
>> -
>> - if auth.status == Status::Valid {
>> - info!("{domain} is already validated!");
>> - continue;
>> - }
>> -
>> - info!("The validation for {domain} is pending");
>> - let domain_config: &AcmeDomain = get_domain_config(&domain)?;
>> - let plugin_id = domain_config.plugin.as_deref().unwrap_or("standalone");
>> - let mut plugin_cfg = crate::acme::get_acme_plugin(&plugins, plugin_id)?
>> - .ok_or_else(|| format_err!("plugin '{plugin_id}' for domain '{domain}' not found!"))?;
>> -
>> - info!("Setting up validation plugin");
>> - let validation_url = plugin_cfg
>> - .setup(&mut acme, &auth, domain_config, Arc::clone(&worker))
>> - .await?;
>> -
>> - let result = request_validation(&mut acme, auth_url, validation_url).await;
>> -
>> - if let Err(err) = plugin_cfg
>> - .teardown(&mut acme, &auth, domain_config, Arc::clone(&worker))
>> - .await
>> - {
>> - warn!("Failed to teardown plugin '{plugin_id}' for domain '{domain}' - {err}");
>> - }
>> -
>> - result?;
>> - }
>> -
>> - info!("All domains validated");
>> - info!("Creating CSR");
>> -
>> - let csr = proxmox_acme::util::Csr::generate(&identifiers, &Default::default())?;
>> - let mut finalize_error_cnt = 0u8;
>> - let order_url = &order.location;
>> - let mut order;
>> - loop {
>> - use proxmox_acme::order::Status;
>> -
>> - order = acme.get_order(order_url).await?;
>> -
>> - match order.status {
>> - Status::Pending => {
>> - info!("still pending, trying to finalize anyway");
>> - let finalize = order
>> - .finalize
>> - .as_deref()
>> - .ok_or_else(|| format_err!("missing 'finalize' URL in order"))?;
>> - if let Err(err) = acme.finalize(finalize, &csr.data).await {
>> - if finalize_error_cnt >= 5 {
>> - return Err(err);
>> - }
>> -
>> - finalize_error_cnt += 1;
>> - }
>> - tokio::time::sleep(Duration::from_secs(5)).await;
>> - }
>> - Status::Ready => {
>> - info!("order is ready, finalizing");
>> - let finalize = order
>> - .finalize
>> - .as_deref()
>> - .ok_or_else(|| format_err!("missing 'finalize' URL in order"))?;
>> - acme.finalize(finalize, &csr.data).await?;
>> - tokio::time::sleep(Duration::from_secs(5)).await;
>> - }
>> - Status::Processing => {
>> - info!("still processing, trying again in 30 seconds");
>> - tokio::time::sleep(Duration::from_secs(30)).await;
>> - }
>> - Status::Valid => {
>> - info!("valid");
>> - break;
>> - }
>> - other => bail!("order status: {:?}", other),
>> - }
>> - }
>> -
>> - info!("Downloading certificate");
>> - let certificate = acme
>> - .get_certificate(
>> - order
>> - .certificate
>> - .as_deref()
>> - .ok_or_else(|| format_err!("missing certificate url in finalized order"))?,
>> - )
>> - .await?;
>> -
>> - Ok(Some(OrderedCertificate {
>> - certificate,
>> - private_key_pem: csr.private_key_pem,
>> - }))
>> -}
>> -
>> -async fn request_validation(
>> - acme: &mut AcmeClient,
>> - auth_url: &str,
>> - validation_url: &str,
>> -) -> Result<(), Error> {
>> - info!("Triggering validation");
>> - acme.request_challenge_validation(validation_url).await?;
>> -
>> - info!("Sleeping for 5 seconds");
>> - tokio::time::sleep(Duration::from_secs(5)).await;
>> -
>> - loop {
>> - use proxmox_acme::authorization::Status;
>> -
>> - let auth = acme.get_authorization(auth_url).await?;
>> - match auth.status {
>> - Status::Pending => {
>> - info!("Status is still 'pending', trying again in 10 seconds");
>> - tokio::time::sleep(Duration::from_secs(10)).await;
>> - }
>> - Status::Valid => return Ok(()),
>> - other => bail!(
>> - "validating challenge '{}' failed - status: {:?}",
>> - validation_url,
>> - other
>> - ),
>> - }
>> - }
>> -}
>> -
>> #[api(
>> input: {
>> properties: {
>> @@ -524,9 +332,30 @@ fn spawn_certificate_worker(
>>
>> let auth_id = rpcenv.get_auth_id().unwrap();
>>
>> + let acme_config = if let Some(cfg) = node_config.acme_config().transpose()? {
>> + cfg
>> + } else {
>> + proxmox_acme_api::parse_acme_config_string("account=default")?
>> + };
>
> wouldn't it make sense to inline this into acme_config() ? the same
> fallback is already there for acme_client()
>
Good catch, will refactor!
>> +
>> + let domains = node_config.acme_domains().try_fold(
>> + Vec::<AcmeDomain>::new(),
>> + |mut acc, domain| -> Result<_, Error> {
>> + let mut domain = domain?;
>> + domain.domain.make_ascii_lowercase();
>> + if let Some(alias) = &mut domain.alias {
>> + alias.make_ascii_lowercase();
>> + }
>> + acc.push(domain);
>> + Ok(acc)
>> + },
>> + )?;
>> +
>> WorkerTask::spawn(name, None, auth_id, true, move |worker| async move {
>> let work = || async {
>> - if let Some(cert) = order_certificate(worker, &node_config).await? {
>> + if let Some(cert) =
>> + proxmox_acme_api::order_certificate(worker, &acme_config, &domains).await?
>> + {
>> crate::config::set_proxy_certificate(&cert.certificate, &cert.private_key_pem)?;
>> crate::server::reload_proxy_certificate().await?;
>> }
>> @@ -562,16 +391,20 @@ pub fn revoke_acme_cert(rpcenv: &mut dyn RpcEnvironment) -> Result<String, Error
>>
>> let auth_id = rpcenv.get_auth_id().unwrap();
>>
>> + let acme_config = if let Some(cfg) = node_config.acme_config().transpose()? {
>> + cfg
>> + } else {
>> + proxmox_acme_api::parse_acme_config_string("account=default")?
>> + };
>
> here as well
>
Will adjust!
>> +
>> WorkerTask::spawn(
>> "acme-revoke-cert",
>> None,
>> auth_id,
>> true,
>> move |_worker| async move {
>> - info!("Loading ACME account");
>> - let mut acme = node_config.acme_client().await?;
>> info!("Revoking old certificate");
>> - acme.revoke_certificate(cert_pem.as_bytes(), None).await?;
>> + proxmox_acme_api::revoke_certificate(&acme_config, &cert_pem.as_bytes()).await?;
>> info!("Deleting certificate and regenerating a self-signed one");
>> delete_custom_certificate().await?;
>> Ok(())
>
> [..]
>
>> diff --git a/src/config/acme/plugin.rs b/src/config/acme/plugin.rs
>> index 8ce852ec..4b4a216e 100644
>> --- a/src/config/acme/plugin.rs
>> +++ b/src/config/acme/plugin.rs
>> @@ -1,104 +1,16 @@
>> use std::sync::LazyLock;
>>
>> use anyhow::Error;
>> -use serde::{Deserialize, Serialize};
>> use serde_json::Value;
>>
>> -use pbs_api_types::PROXMOX_SAFE_ID_FORMAT;
>> -use proxmox_schema::{api, ApiType, Schema, StringSchema, Updater};
>> +use proxmox_acme_api::{DnsPlugin, StandalonePlugin, PLUGIN_ID_SCHEMA};
>> +use proxmox_schema::{ApiType, Schema};
>> use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
>>
>> use pbs_config::{open_backup_lockfile, BackupLockGuard};
>>
>> -pub const PLUGIN_ID_SCHEMA: Schema = StringSchema::new("ACME Challenge Plugin ID.")
>> - .format(&PROXMOX_SAFE_ID_FORMAT)
>> - .min_length(1)
>> - .max_length(32)
>> - .schema();
>> -
>> pub static CONFIG: LazyLock<SectionConfig> = LazyLock::new(init);
>>
>> -#[api(
>> - properties: {
>> - id: { schema: PLUGIN_ID_SCHEMA },
>> - },
>> -)]
>> -#[derive(Deserialize, Serialize)]
>> -/// Standalone ACME Plugin for the http-1 challenge.
>> -pub struct StandalonePlugin {
>> - /// Plugin ID.
>> - id: String,
>> -}
>> -
>> -impl Default for StandalonePlugin {
>> - fn default() -> Self {
>> - Self {
>> - id: "standalone".to_string(),
>> - }
>> - }
>> -}
>> -
>> -#[api(
>> - properties: {
>> - id: { schema: PLUGIN_ID_SCHEMA },
>> - disable: {
>> - optional: true,
>> - default: false,
>> - },
>> - "validation-delay": {
>> - default: 30,
>> - optional: true,
>> - minimum: 0,
>> - maximum: 2 * 24 * 60 * 60,
>> - },
>> - },
>> -)]
>> -/// DNS ACME Challenge Plugin core data.
>> -#[derive(Deserialize, Serialize, Updater)]
>> -#[serde(rename_all = "kebab-case")]
>> -pub struct DnsPluginCore {
>> - /// Plugin ID.
>> - #[updater(skip)]
>> - pub id: String,
>> -
>> - /// DNS API Plugin Id.
>> - pub api: String,
>> -
>> - /// Extra delay in seconds to wait before requesting validation.
>> - ///
>> - /// Allows to cope with long TTL of DNS records.
>> - #[serde(skip_serializing_if = "Option::is_none", default)]
>> - pub validation_delay: Option<u32>,
>> -
>> - /// Flag to disable the config.
>> - #[serde(skip_serializing_if = "Option::is_none", default)]
>> - pub disable: Option<bool>,
>> -}
>> -
>> -#[api(
>> - properties: {
>> - core: { type: DnsPluginCore },
>> - },
>> -)]
>> -/// DNS ACME Challenge Plugin.
>> -#[derive(Deserialize, Serialize)]
>> -#[serde(rename_all = "kebab-case")]
>> -pub struct DnsPlugin {
>> - #[serde(flatten)]
>> - pub core: DnsPluginCore,
>> -
>> - // We handle this property separately in the API calls.
>> - /// DNS plugin data (base64url encoded without padding).
>> - #[serde(with = "proxmox_serde::string_as_base64url_nopad")]
>> - pub data: String,
>> -}
>> -
>> -impl DnsPlugin {
>> - pub fn decode_data(&self, output: &mut Vec<u8>) -> Result<(), Error> {
>> - Ok(proxmox_base64::url::decode_to_vec(&self.data, output)?)
>> - }
>> -}
>> -
>> fn init() -> SectionConfig {
>> let mut config = SectionConfig::new(&PLUGIN_ID_SCHEMA);
>>
>> diff --git a/src/config/node.rs b/src/config/node.rs
>> index e4b66a20..6865b815 100644
>> --- a/src/config/node.rs
>> +++ b/src/config/node.rs
>> @@ -9,14 +9,14 @@ use pbs_api_types::{
>> OPENSSL_CIPHERS_TLS_1_3_SCHEMA,
>> };
>> use proxmox_acme::async_client::AcmeClient;
>> -use proxmox_acme_api::AcmeAccountName;
>> +use proxmox_acme_api::{AcmeAccountName, AcmeConfig, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA};
>> use proxmox_http::ProxyConfig;
>> use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
>>
>> use pbs_buildcfg::configdir;
>> use pbs_config::{open_backup_lockfile, BackupLockGuard};
>>
>> -use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
>> +use crate::api2::types::HTTP_PROXY_SCHEMA;
>>
>> const CONF_FILE: &str = configdir!("/node.cfg");
>> const LOCK_FILE: &str = configdir!("/.node.lck");
>> @@ -43,20 +43,6 @@ pub fn save_config(config: &NodeConfig) -> Result<(), Error> {
>> pbs_config::replace_backup_config(CONF_FILE, &raw)
>> }
>>
>> -#[api(
>> - properties: {
>> - account: { type: AcmeAccountName },
>> - }
>> -)]
>> -#[derive(Deserialize, Serialize)]
>> -/// The ACME configuration.
>> -///
>> -/// Currently only contains the name of the account use.
>> -pub struct AcmeConfig {
>> - /// Account to use to acquire ACME certificates.
>> - account: AcmeAccountName,
>> -}
>> -
>> /// All available languages in Proxmox. Taken from proxmox-i18n repository.
>> /// pt_BR, zh_CN, and zh_TW use the same case in the translation files.
>> // TODO: auto-generate from available translations
>> @@ -242,7 +228,7 @@ impl NodeConfig {
>>
>> pub async fn acme_client(&self) -> Result<AcmeClient, Error> {
>> let account = if let Some(cfg) = self.acme_config().transpose()? {
>> - cfg.account
>> + AcmeAccountName::from_string(cfg.account)?
>> } else {
>> AcmeAccountName::from_string("default".to_string())? // should really not happen
>> };
>> diff --git a/src/lib.rs b/src/lib.rs
>> index 8633378c..828f5842 100644
>> --- a/src/lib.rs
>> +++ b/src/lib.rs
>> @@ -27,8 +27,6 @@ pub(crate) mod auth;
>>
>> pub mod tape;
>>
>> -pub mod acme;
>> -
>> pub mod client_helpers;
>>
>> pub mod traffic_control_cache;
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 4/5] acme: change API impls to use proxmox-acme-api handlers
2026-01-13 13:45 5% ` Fabian Grünbichler
@ 2026-01-13 16:53 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-13 16:53 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 1/13/26 2:45 PM, Fabian Grünbichler wrote:
> On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
>> PBS currently uses its own ACME client and API logic, while PDM uses the
>> factored out proxmox-acme and proxmox-acme-api crates. This duplication
>> risks differences in behaviour and requires ACME maintenance in two
>> places. This patch is part of a series to move PBS over to the shared
>> ACME stack.
>>
>> Changes:
>> - Replace api2/config/acme.rs API logic with proxmox-acme-api handlers.
>> - Drop local caching and helper types that duplicate proxmox-acme-api.
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> src/api2/config/acme.rs | 378 ++-----------------------
>> src/api2/types/acme.rs | 16 --
>> src/bin/proxmox_backup_manager/acme.rs | 6 +-
>> src/config/acme/mod.rs | 44 +--
>> 4 files changed, 33 insertions(+), 411 deletions(-)
>>
>> diff --git a/src/api2/config/acme.rs b/src/api2/config/acme.rs
>> index 898f06dd..3314430c 100644
>> --- a/src/api2/config/acme.rs
>> +++ b/src/api2/config/acme.rs
>> @@ -1,29 +1,18 @@
>> -use std::fs;
>> -use std::ops::ControlFlow;
>> -use std::path::Path;
>
> nit: this one is actually still used below
Ah, I see :) Good find!
>
>> -use std::sync::{Arc, LazyLock, Mutex};
>> -use std::time::SystemTime;
>> -
>> -use anyhow::{bail, format_err, Error};
>> -use hex::FromHex;
>> -use serde::{Deserialize, Serialize};
>> -use serde_json::{json, Value};
>> -use tracing::{info, warn};
>> +use anyhow::Error;
>> +use tracing::info;
>>
>> use pbs_api_types::{Authid, PRIV_SYS_MODIFY};
>> -use proxmox_acme::async_client::AcmeClient;
>> -use proxmox_acme::types::AccountData as AcmeAccountData;
>> -use proxmox_acme_api::AcmeAccountName;
>> +use proxmox_acme_api::{
>> + AccountEntry, AccountInfo, AcmeAccountName, AcmeChallengeSchema, ChallengeSchemaWrapper,
>> + DeletablePluginProperty, DnsPluginCore, DnsPluginCoreUpdater, KnownAcmeDirectory, PluginConfig,
>> + DEFAULT_ACME_DIRECTORY_ENTRY, PLUGIN_ID_SCHEMA,
>> +};
>> +use proxmox_config_digest::ConfigDigest;
>> use proxmox_rest_server::WorkerTask;
>> use proxmox_router::{
>> http_bail, list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap,
>> };
>> -use proxmox_schema::{api, param_bail};
>> -
>> -use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
>> -use crate::config::acme::plugin::{
>> - self, DnsPlugin, DnsPluginCore, DnsPluginCoreUpdater, PLUGIN_ID_SCHEMA,
>> -};
>> +use proxmox_schema::api;
>>
>> pub(crate) const ROUTER: Router = Router::new()
>> .get(&list_subdirs_api_method!(SUBDIRS))
>> @@ -65,19 +54,6 @@ const PLUGIN_ITEM_ROUTER: Router = Router::new()
>> .put(&API_METHOD_UPDATE_PLUGIN)
>> .delete(&API_METHOD_DELETE_PLUGIN);
>>
>> -#[api(
>> - properties: {
>> - name: { type: AcmeAccountName },
>> - },
>> -)]
>> -/// An ACME Account entry.
>> -///
>> -/// Currently only contains a 'name' property.
>> -#[derive(Serialize)]
>> -pub struct AccountEntry {
>> - name: AcmeAccountName,
>> -}
>> -
>> #[api(
>> access: {
>> permission: &Permission::Privilege(&["system", "certificates"], PRIV_SYS_MODIFY, false),
>> @@ -91,40 +67,7 @@ pub struct AccountEntry {
>> )]
>> /// List ACME accounts.
>> pub fn list_accounts() -> Result<Vec<AccountEntry>, Error> {
>> - let mut entries = Vec::new();
>> - crate::config::acme::foreach_acme_account(|name| {
>> - entries.push(AccountEntry { name });
>> - ControlFlow::Continue(())
>> - })?;
>> - Ok(entries)
>> -}
>> -
>> -#[api(
>> - properties: {
>> - account: { type: Object, properties: {}, additional_properties: true },
>> - tos: {
>> - type: String,
>> - optional: true,
>> - },
>> - },
>> -)]
>> -/// ACME Account information.
>> -///
>> -/// This is what we return via the API.
>> -#[derive(Serialize)]
>> -pub struct AccountInfo {
>> - /// Raw account data.
>> - account: AcmeAccountData,
>> -
>> - /// The ACME directory URL the account was created at.
>> - directory: String,
>> -
>> - /// The account's own URL within the ACME directory.
>> - location: String,
>> -
>> - /// The ToS URL, if the user agreed to one.
>> - #[serde(skip_serializing_if = "Option::is_none")]
>> - tos: Option<String>,
>> + proxmox_acme_api::list_accounts()
>> }
>>
>> #[api(
>> @@ -141,23 +84,7 @@ pub struct AccountInfo {
>> )]
>> /// Return existing ACME account information.
>> pub async fn get_account(name: AcmeAccountName) -> Result<AccountInfo, Error> {
>> - let account_info = proxmox_acme_api::get_account(name).await?;
>> -
>> - Ok(AccountInfo {
>> - location: account_info.location,
>> - tos: account_info.tos,
>> - directory: account_info.directory,
>> - account: AcmeAccountData {
>> - only_return_existing: false, // don't actually write this out in case it's set
>> - ..account_info.account
>> - },
>> - })
>> -}
>> -
>> -fn account_contact_from_string(s: &str) -> Vec<String> {
>> - s.split(&[' ', ';', ',', '\0'][..])
>> - .map(|s| format!("mailto:{s}"))
>> - .collect()
>> + proxmox_acme_api::get_account(name).await
>> }
>>
>> #[api(
>> @@ -222,15 +149,11 @@ fn register_account(
>> );
>> }
>>
>> - if Path::new(&crate::config::acme::account_path(&name)).exists() {
>> + if std::path::Path::new(&proxmox_acme_api::account_config_filename(&name)).exists() {
>
> here ^
>
>> http_bail!(BAD_REQUEST, "account {} already exists", name);
>> }
>>
>> - let directory = directory.unwrap_or_else(|| {
>> - crate::config::acme::DEFAULT_ACME_DIRECTORY_ENTRY
>> - .url
>> - .to_owned()
>> - });
>> + let directory = directory.unwrap_or_else(|| DEFAULT_ACME_DIRECTORY_ENTRY.url.to_string());
>>
>> WorkerTask::spawn(
>> "acme-register",
>> @@ -286,17 +209,7 @@ pub fn update_account(
>> auth_id.to_string(),
>> true,
>> move |_worker| async move {
>> - let data = match contact {
>> - Some(data) => json!({
>> - "contact": account_contact_from_string(&data),
>> - }),
>> - None => json!({}),
>> - };
>> -
>> - proxmox_acme_api::load_client_with_account(&name)
>> - .await?
>> - .update_account(&data)
>> - .await?;
>> + proxmox_acme_api::update_account(&name, contact).await?;
>>
>> Ok(())
>> },
>> @@ -334,18 +247,8 @@ pub fn deactivate_account(
>> auth_id.to_string(),
>> true,
>> move |_worker| async move {
>> - match proxmox_acme_api::load_client_with_account(&name)
>> - .await?
>> - .update_account(&json!({"status": "deactivated"}))
>> - .await
>> - {
>> - Ok(_account) => (),
>> - Err(err) if !force => return Err(err),
>> - Err(err) => {
>> - warn!("error deactivating account {name}, proceeding anyway - {err}");
>> - }
>> - }
>> - crate::config::acme::mark_account_deactivated(&name)?;
>> + proxmox_acme_api::deactivate_account(&name, force).await?;
>> +
>> Ok(())
>> },
>> )
>> @@ -372,15 +275,7 @@ pub fn deactivate_account(
>> )]
>> /// Get the Terms of Service URL for an ACME directory.
>> async fn get_tos(directory: Option<String>) -> Result<Option<String>, Error> {
>> - let directory = directory.unwrap_or_else(|| {
>> - crate::config::acme::DEFAULT_ACME_DIRECTORY_ENTRY
>> - .url
>> - .to_owned()
>> - });
>> - Ok(AcmeClient::new(directory)
>> - .terms_of_service_url()
>> - .await?
>> - .map(str::to_owned))
>> + proxmox_acme_api::get_tos(directory).await
>> }
>>
>> #[api(
>> @@ -395,52 +290,7 @@ async fn get_tos(directory: Option<String>) -> Result<Option<String>, Error> {
>> )]
>> /// Get named known ACME directory endpoints.
>> fn get_directories() -> Result<&'static [KnownAcmeDirectory], Error> {
>> - Ok(crate::config::acme::KNOWN_ACME_DIRECTORIES)
>> -}
>> -
>> -/// Wrapper for efficient Arc use when returning the ACME challenge-plugin schema for serializing
>> -struct ChallengeSchemaWrapper {
>> - inner: Arc<Vec<AcmeChallengeSchema>>,
>> -}
>> -
>> -impl Serialize for ChallengeSchemaWrapper {
>> - fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
>> - where
>> - S: serde::Serializer,
>> - {
>> - self.inner.serialize(serializer)
>> - }
>> -}
>> -
>> -struct CachedSchema {
>> - schema: Arc<Vec<AcmeChallengeSchema>>,
>> - cached_mtime: SystemTime,
>> -}
>> -
>> -fn get_cached_challenge_schemas() -> Result<ChallengeSchemaWrapper, Error> {
>> - static CACHE: LazyLock<Mutex<Option<CachedSchema>>> = LazyLock::new(|| Mutex::new(None));
>> -
>> - // the actual loading code
>> - let mut last = CACHE.lock().unwrap();
>> -
>> - let actual_mtime = fs::metadata(crate::config::acme::ACME_DNS_SCHEMA_FN)?.modified()?;
>> -
>> - let schema = match &*last {
>> - Some(CachedSchema {
>> - schema,
>> - cached_mtime,
>> - }) if *cached_mtime >= actual_mtime => schema.clone(),
>> - _ => {
>> - let new_schema = Arc::new(crate::config::acme::load_dns_challenge_schema()?);
>> - *last = Some(CachedSchema {
>> - schema: Arc::clone(&new_schema),
>> - cached_mtime: actual_mtime,
>> - });
>> - new_schema
>> - }
>> - };
>> -
>> - Ok(ChallengeSchemaWrapper { inner: schema })
>> + Ok(proxmox_acme_api::KNOWN_ACME_DIRECTORIES)
>> }
>>
>> #[api(
>> @@ -455,69 +305,7 @@ fn get_cached_challenge_schemas() -> Result<ChallengeSchemaWrapper, Error> {
>> )]
>> /// Get named known ACME directory endpoints.
>> fn get_challenge_schema() -> Result<ChallengeSchemaWrapper, Error> {
>> - get_cached_challenge_schemas()
>> -}
>> -
>> -#[api]
>> -#[derive(Default, Deserialize, Serialize)]
>> -#[serde(rename_all = "kebab-case")]
>> -/// The API's format is inherited from PVE/PMG:
>> -pub struct PluginConfig {
>> - /// Plugin ID.
>> - plugin: String,
>> -
>> - /// Plugin type.
>> - #[serde(rename = "type")]
>> - ty: String,
>> -
>> - /// DNS Api name.
>> - #[serde(skip_serializing_if = "Option::is_none", default)]
>> - api: Option<String>,
>> -
>> - /// Plugin configuration data.
>> - #[serde(skip_serializing_if = "Option::is_none", default)]
>> - data: Option<String>,
>> -
>> - /// Extra delay in seconds to wait before requesting validation.
>> - ///
>> - /// Allows to cope with long TTL of DNS records.
>> - #[serde(skip_serializing_if = "Option::is_none", default)]
>> - validation_delay: Option<u32>,
>> -
>> - /// Flag to disable the config.
>> - #[serde(skip_serializing_if = "Option::is_none", default)]
>> - disable: Option<bool>,
>> -}
>> -
>> -// See PMG/PVE's $modify_cfg_for_api sub
>> -fn modify_cfg_for_api(id: &str, ty: &str, data: &Value) -> PluginConfig {
>> - let mut entry = data.clone();
>> -
>> - let obj = entry.as_object_mut().unwrap();
>> - obj.remove("id");
>> - obj.insert("plugin".to_string(), Value::String(id.to_owned()));
>> - obj.insert("type".to_string(), Value::String(ty.to_owned()));
>> -
>> - // FIXME: This needs to go once the `Updater` is fixed.
>> - // None of these should be able to fail unless the user changed the files by hand, in which
>> - // case we leave the unmodified string in the Value for now. This will be handled with an error
>> - // later.
>> - if let Some(Value::String(ref mut data)) = obj.get_mut("data") {
>> - if let Ok(new) = proxmox_base64::url::decode_no_pad(&data) {
>> - if let Ok(utf8) = String::from_utf8(new) {
>> - *data = utf8;
>> - }
>> - }
>> - }
>> -
>> - // PVE/PMG do this explicitly for ACME plugins...
>> - // obj.insert("digest".to_string(), Value::String(digest.clone()));
>> -
>> - serde_json::from_value(entry).unwrap_or_else(|_| PluginConfig {
>> - plugin: "*Error*".to_string(),
>> - ty: "*Error*".to_string(),
>> - ..Default::default()
>> - })
>> + proxmox_acme_api::get_cached_challenge_schemas()
>> }
>>
>> #[api(
>> @@ -533,12 +321,7 @@ fn modify_cfg_for_api(id: &str, ty: &str, data: &Value) -> PluginConfig {
>> )]
>> /// List ACME challenge plugins.
>> pub fn list_plugins(rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<PluginConfig>, Error> {
>> - let (plugins, digest) = plugin::config()?;
>> - rpcenv["digest"] = hex::encode(digest).into();
>> - Ok(plugins
>> - .iter()
>> - .map(|(id, (ty, data))| modify_cfg_for_api(id, ty, data))
>> - .collect())
>> + proxmox_acme_api::list_plugins(rpcenv)
>> }
>>
>> #[api(
>> @@ -555,13 +338,7 @@ pub fn list_plugins(rpcenv: &mut dyn RpcEnvironment) -> Result<Vec<PluginConfig>
>> )]
>> /// List ACME challenge plugins.
>> pub fn get_plugin(id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<PluginConfig, Error> {
>> - let (plugins, digest) = plugin::config()?;
>> - rpcenv["digest"] = hex::encode(digest).into();
>> -
>> - match plugins.get(&id) {
>> - Some((ty, data)) => Ok(modify_cfg_for_api(&id, ty, data)),
>> - None => http_bail!(NOT_FOUND, "no such plugin"),
>> - }
>> + proxmox_acme_api::get_plugin(id, rpcenv)
>> }
>>
>> // Currently we only have "the" standalone plugin and DNS plugins so we can just flatten a
>> @@ -593,30 +370,7 @@ pub fn get_plugin(id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<PluginC
>> )]
>> /// Add ACME plugin configuration.
>> pub fn add_plugin(r#type: String, core: DnsPluginCore, data: String) -> Result<(), Error> {
>> - // Currently we only support DNS plugins and the standalone plugin is "fixed":
>> - if r#type != "dns" {
>> - param_bail!("type", "invalid ACME plugin type: {:?}", r#type);
>> - }
>> -
>> - let data = String::from_utf8(proxmox_base64::decode(data)?)
>> - .map_err(|_| format_err!("data must be valid UTF-8"))?;
>> -
>> - let id = core.id.clone();
>> -
>> - let _lock = plugin::lock()?;
>> -
>> - let (mut plugins, _digest) = plugin::config()?;
>> - if plugins.contains_key(&id) {
>> - param_bail!("id", "ACME plugin ID {:?} already exists", id);
>> - }
>> -
>> - let plugin = serde_json::to_value(DnsPlugin { core, data })?;
>> -
>> - plugins.insert(id, r#type, plugin);
>> -
>> - plugin::save_config(&plugins)?;
>> -
>> - Ok(())
>> + proxmox_acme_api::add_plugin(r#type, core, data)
>> }
>>
>> #[api(
>> @@ -632,26 +386,7 @@ pub fn add_plugin(r#type: String, core: DnsPluginCore, data: String) -> Result<(
>> )]
>> /// Delete an ACME plugin configuration.
>> pub fn delete_plugin(id: String) -> Result<(), Error> {
>> - let _lock = plugin::lock()?;
>> -
>> - let (mut plugins, _digest) = plugin::config()?;
>> - if plugins.remove(&id).is_none() {
>> - http_bail!(NOT_FOUND, "no such plugin");
>> - }
>> - plugin::save_config(&plugins)?;
>> -
>> - Ok(())
>> -}
>> -
>> -#[api()]
>> -#[derive(Serialize, Deserialize)]
>> -#[serde(rename_all = "kebab-case")]
>> -/// Deletable property name
>> -pub enum DeletableProperty {
>> - /// Delete the disable property
>> - Disable,
>> - /// Delete the validation-delay property
>> - ValidationDelay,
>> + proxmox_acme_api::delete_plugin(id)
>> }
>>
>> #[api(
>> @@ -673,12 +408,12 @@ pub enum DeletableProperty {
>> type: Array,
>> optional: true,
>> items: {
>> - type: DeletableProperty,
>> + type: DeletablePluginProperty,
>> }
>> },
>> digest: {
>> - description: "Digest to protect against concurrent updates",
>> optional: true,
>> + type: ConfigDigest,
>> },
>> },
>> },
>> @@ -692,65 +427,8 @@ pub fn update_plugin(
>> id: String,
>> update: DnsPluginCoreUpdater,
>> data: Option<String>,
>> - delete: Option<Vec<DeletableProperty>>,
>> - digest: Option<String>,
>> + delete: Option<Vec<DeletablePluginProperty>>,
>> + digest: Option<ConfigDigest>,
>> ) -> Result<(), Error> {
>> - let data = data
>> - .as_deref()
>> - .map(proxmox_base64::decode)
>> - .transpose()?
>> - .map(String::from_utf8)
>> - .transpose()
>> - .map_err(|_| format_err!("data must be valid UTF-8"))?;
>> -
>> - let _lock = plugin::lock()?;
>> -
>> - let (mut plugins, expected_digest) = plugin::config()?;
>> -
>> - if let Some(digest) = digest {
>> - let digest = <[u8; 32]>::from_hex(digest)?;
>> - crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
>> - }
>> -
>> - match plugins.get_mut(&id) {
>> - Some((ty, ref mut entry)) => {
>> - if ty != "dns" {
>> - bail!("cannot update plugin of type {:?}", ty);
>> - }
>> -
>> - let mut plugin = DnsPlugin::deserialize(&*entry)?;
>> -
>> - if let Some(delete) = delete {
>> - for delete_prop in delete {
>> - match delete_prop {
>> - DeletableProperty::ValidationDelay => {
>> - plugin.core.validation_delay = None;
>> - }
>> - DeletableProperty::Disable => {
>> - plugin.core.disable = None;
>> - }
>> - }
>> - }
>> - }
>> - if let Some(data) = data {
>> - plugin.data = data;
>> - }
>> - if let Some(api) = update.api {
>> - plugin.core.api = api;
>> - }
>> - if update.validation_delay.is_some() {
>> - plugin.core.validation_delay = update.validation_delay;
>> - }
>> - if update.disable.is_some() {
>> - plugin.core.disable = update.disable;
>> - }
>> -
>> - *entry = serde_json::to_value(plugin)?;
>> - }
>> - None => http_bail!(NOT_FOUND, "no such plugin"),
>> - }
>> -
>> - plugin::save_config(&plugins)?;
>> -
>> - Ok(())
>> + proxmox_acme_api::update_plugin(id, update, data, delete, digest)
>> }
>> diff --git a/src/api2/types/acme.rs b/src/api2/types/acme.rs
>> index 64175aff..0ff496b6 100644
>> --- a/src/api2/types/acme.rs
>> +++ b/src/api2/types/acme.rs
>> @@ -43,22 +43,6 @@ pub const ACME_DOMAIN_PROPERTY_SCHEMA: Schema =
>> .format(&ApiStringFormat::PropertyString(&AcmeDomain::API_SCHEMA))
>> .schema();
>>
>> -#[api(
>> - properties: {
>> - name: { type: String },
>> - url: { type: String },
>> - },
>> -)]
>> -/// An ACME directory endpoint with a name and URL.
>> -#[derive(Serialize)]
>> -pub struct KnownAcmeDirectory {
>> - /// The ACME directory's name.
>> - pub name: &'static str,
>> -
>> - /// The ACME directory's endpoint URL.
>> - pub url: &'static str,
>> -}
>> -
>> #[api(
>> properties: {
>> schema: {
>> diff --git a/src/bin/proxmox_backup_manager/acme.rs b/src/bin/proxmox_backup_manager/acme.rs
>> index 6ed61560..d11d7498 100644
>> --- a/src/bin/proxmox_backup_manager/acme.rs
>> +++ b/src/bin/proxmox_backup_manager/acme.rs
>> @@ -4,14 +4,12 @@ use anyhow::{bail, Error};
>> use serde_json::Value;
>>
>> use proxmox_acme::async_client::AcmeClient;
>> -use proxmox_acme_api::AcmeAccountName;
>> +use proxmox_acme_api::{AcmeAccountName, DnsPluginCore, KNOWN_ACME_DIRECTORIES};
>> use proxmox_router::{cli::*, ApiHandler, RpcEnvironment};
>> use proxmox_schema::api;
>> use proxmox_sys::fs::file_get_contents;
>>
>> use proxmox_backup::api2;
>> -use proxmox_backup::config::acme::plugin::DnsPluginCore;
>> -use proxmox_backup::config::acme::KNOWN_ACME_DIRECTORIES;
>>
>> pub fn acme_mgmt_cli() -> CommandLineInterface {
>> let cmd_def = CliCommandMap::new()
>> @@ -122,7 +120,7 @@ async fn register_account(
>>
>> match input.trim().parse::<usize>() {
>> Ok(n) if n < KNOWN_ACME_DIRECTORIES.len() => {
>> - break (KNOWN_ACME_DIRECTORIES[n].url.to_owned(), false);
>> + break (KNOWN_ACME_DIRECTORIES[n].url.to_string(), false);
>> }
>> Ok(n) if n == KNOWN_ACME_DIRECTORIES.len() => {
>> input.clear();
>> diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
>> index e4639c53..01ab6223 100644
>> --- a/src/config/acme/mod.rs
>> +++ b/src/config/acme/mod.rs
>> @@ -1,16 +1,15 @@
>> use std::collections::HashMap;
>> use std::ops::ControlFlow;
>> -use std::path::Path;
>>
>> -use anyhow::{bail, format_err, Error};
>> +use anyhow::Error;
>> use serde_json::Value;
>>
>> use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
>> -use proxmox_acme_api::AcmeAccountName;
>> +use proxmox_acme_api::{AcmeAccountName, KnownAcmeDirectory, KNOWN_ACME_DIRECTORIES};
>> use proxmox_sys::error::SysError;
>> use proxmox_sys::fs::{file_read_string, CreateOptions};
>>
>> -use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
>> +use crate::api2::types::AcmeChallengeSchema;
>>
>> pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
>> pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
>> @@ -35,23 +34,8 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
>> create_acme_subdir(ACME_DIR)
>> }
>>
>> -pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
>> - KnownAcmeDirectory {
>> - name: "Let's Encrypt V2",
>> - url: "https://acme-v02.api.letsencrypt.org/directory",
>> - },
>> - KnownAcmeDirectory {
>> - name: "Let's Encrypt V2 Staging",
>> - url: "https://acme-staging-v02.api.letsencrypt.org/directory",
>> - },
>> -];
>> -
>> pub const DEFAULT_ACME_DIRECTORY_ENTRY: &KnownAcmeDirectory = &KNOWN_ACME_DIRECTORIES[0];
>>
>> -pub fn account_path(name: &str) -> String {
>> - format!("{ACME_ACCOUNT_DIR}/{name}")
>> -}
>> -
>> pub fn foreach_acme_account<F>(mut func: F) -> Result<(), Error>
>> where
>> F: FnMut(AcmeAccountName) -> ControlFlow<Result<(), Error>>,
>> @@ -82,28 +66,6 @@ where
>> }
>> }
>>
>> -pub fn mark_account_deactivated(name: &str) -> Result<(), Error> {
>> - let from = account_path(name);
>> - for i in 0..100 {
>> - let to = account_path(&format!("_deactivated_{name}_{i}"));
>> - if !Path::new(&to).exists() {
>> - return std::fs::rename(&from, &to).map_err(|err| {
>> - format_err!(
>> - "failed to move account path {:?} to {:?} - {}",
>> - from,
>> - to,
>> - err
>> - )
>> - });
>> - }
>> - }
>> - bail!(
>> - "No free slot to rename deactivated account {:?}, please cleanup {:?}",
>> - from,
>> - ACME_ACCOUNT_DIR
>> - );
>> -}
>> -
>> pub fn load_dns_challenge_schema() -> Result<Vec<AcmeChallengeSchema>, Error> {
>> let raw = file_read_string(ACME_DNS_SCHEMA_FN)?;
>> let schemas: serde_json::Map<String, Value> = serde_json::from_str(&raw)?;
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox v5 4/4] acme-api: add helper to load client for an account
2026-01-13 13:45 5% ` Fabian Grünbichler
@ 2026-01-13 16:57 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-13 16:57 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 1/13/26 2:44 PM, Fabian Grünbichler wrote:
> On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
>> The PBS ACME refactoring needs a simple way to obtain an AcmeClient for
>> a given configured account without duplicating config wiring. This patch
>> adds a load_client_with_account helper in proxmox-acme-api that loads
>> the account and constructs a matching client, similarly as PBS previous
>> own AcmeClient::load() function.
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> proxmox-acme-api/src/account_api_impl.rs | 5 +++++
>> proxmox-acme-api/src/lib.rs | 3 ++-
>> 2 files changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/proxmox-acme-api/src/account_api_impl.rs b/proxmox-acme-api/src/account_api_impl.rs
>> index ef195908..ca8c8655 100644
>> --- a/proxmox-acme-api/src/account_api_impl.rs
>> +++ b/proxmox-acme-api/src/account_api_impl.rs
>> @@ -116,3 +116,8 @@ pub async fn update_account(name: &AcmeAccountName, contact: Option<String>) ->
>>
>> Ok(())
>> }
>> +
>> +pub async fn load_client_with_account(account_name: &AcmeAccountName) -> Result<AcmeClient, Error> {
>> + let account_data = super::account_config::load_account_config(&account_name).await?;
>> + Ok(account_data.client())
>> +}
>
> I don't think this is needed - there is only a single callsite in PBS
> and that is itself dead code that can be removed..
>
Will check, thanks!
>> diff --git a/proxmox-acme-api/src/lib.rs b/proxmox-acme-api/src/lib.rs
>> index 623e9e23..96f88ae2 100644
>> --- a/proxmox-acme-api/src/lib.rs
>> +++ b/proxmox-acme-api/src/lib.rs
>> @@ -31,7 +31,8 @@ mod plugin_config;
>> mod account_api_impl;
>> #[cfg(feature = "impl")]
>> pub use account_api_impl::{
>> - deactivate_account, get_account, get_tos, list_accounts, register_account, update_account,
>> + deactivate_account, get_account, get_tos, list_accounts, load_client_with_account,
>> + register_account, update_account,
>> };
>>
>> #[cfg(feature = "impl")]
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient
2026-01-13 13:45 5% ` Fabian Grünbichler
@ 2026-01-14 8:56 6% ` Samuel Rufinatscha
2026-01-14 9:58 5% ` Fabian Grünbichler
0 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-14 8:56 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 1/13/26 2:44 PM, Fabian Grünbichler wrote:
> On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
>> PBS currently uses its own ACME client and API logic, while PDM uses the
>> factored out proxmox-acme and proxmox-acme-api crates. This duplication
>> risks differences in behaviour and requires ACME maintenance in two
>> places. This patch is part of a series to move PBS over to the shared
>> ACME stack.
>>
>> Changes:
>> - Remove the local src/acme/client.rs and switch to
>> proxmox_acme::async_client::AcmeClient where needed.
>> - Use proxmox_acme_api::load_client_with_account to the custom
>> AcmeClient::load() function
>> - Replace the local do_register() logic with
>> proxmox_acme_api::register_account, to further ensure accounts are persisted
>> - Replace the local AcmeAccountName type, required for
>> proxmox_acme_api::register_account
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> src/acme/client.rs | 691 -------------------------
>> src/acme/mod.rs | 3 -
>> src/acme/plugin.rs | 2 +-
>> src/api2/config/acme.rs | 50 +-
>> src/api2/node/certificates.rs | 2 +-
>> src/api2/types/acme.rs | 8 -
>> src/bin/proxmox_backup_manager/acme.rs | 17 +-
>> src/config/acme/mod.rs | 8 +-
>> src/config/node.rs | 9 +-
>> 9 files changed, 36 insertions(+), 754 deletions(-)
>> delete mode 100644 src/acme/client.rs
>>
>
> [..]
>
>> diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
>> index ac89ae5e..e4639c53 100644
>> --- a/src/config/acme/mod.rs
>> +++ b/src/config/acme/mod.rs
>
> I think this whole file should probably be replaced entirely by
> proxmox-acme-api , which - AFAICT - would just require adding the
> completion helpers there?
>
Good point, yes I think moving the completion helpers would
allow us to get rid of this file. PDM does not make use of
them / there is atm no 1:1 code in proxmox/ for these helpers.
>> @@ -6,10 +6,11 @@ use anyhow::{bail, format_err, Error};
>> use serde_json::Value;
>>
>> use pbs_api_types::PROXMOX_SAFE_ID_REGEX;
>> +use proxmox_acme_api::AcmeAccountName;
>> use proxmox_sys::error::SysError;
>> use proxmox_sys::fs::{file_read_string, CreateOptions};
>>
>> -use crate::api2::types::{AcmeAccountName, AcmeChallengeSchema, KnownAcmeDirectory};
>> +use crate::api2::types::{AcmeChallengeSchema, KnownAcmeDirectory};
>>
>> pub(crate) const ACME_DIR: &str = pbs_buildcfg::configdir!("/acme");
>> pub(crate) const ACME_ACCOUNT_DIR: &str = pbs_buildcfg::configdir!("/acme/accounts");
>> @@ -34,11 +35,6 @@ pub(crate) fn make_acme_dir() -> Result<(), Error> {
>> create_acme_subdir(ACME_DIR)
>> }
>>
>> -pub(crate) fn make_acme_account_dir() -> Result<(), Error> {
>> - make_acme_dir()?;
>> - create_acme_subdir(ACME_ACCOUNT_DIR)
>> -}
>> -
>> pub const KNOWN_ACME_DIRECTORIES: &[KnownAcmeDirectory] = &[
>> KnownAcmeDirectory {
>> name: "Let's Encrypt V2",
>> diff --git a/src/config/node.rs b/src/config/node.rs
>> index 253b2e36..e4b66a20 100644
>> --- a/src/config/node.rs
>> +++ b/src/config/node.rs
>> @@ -8,16 +8,15 @@ use pbs_api_types::{
>> EMAIL_SCHEMA, MULTI_LINE_COMMENT_SCHEMA, OPENSSL_CIPHERS_TLS_1_2_SCHEMA,
>> OPENSSL_CIPHERS_TLS_1_3_SCHEMA,
>> };
>> +use proxmox_acme::async_client::AcmeClient;
>> +use proxmox_acme_api::AcmeAccountName;
>> use proxmox_http::ProxyConfig;
>> use proxmox_schema::{api, ApiStringFormat, ApiType, Updater};
>>
>> use pbs_buildcfg::configdir;
>> use pbs_config::{open_backup_lockfile, BackupLockGuard};
>>
>> -use crate::acme::AcmeClient;
>> -use crate::api2::types::{
>> - AcmeAccountName, AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA,
>> -};
>> +use crate::api2::types::{AcmeDomain, ACME_DOMAIN_PROPERTY_SCHEMA, HTTP_PROXY_SCHEMA};
>>
>> const CONF_FILE: &str = configdir!("/node.cfg");
>> const LOCK_FILE: &str = configdir!("/.node.lck");
>> @@ -247,7 +246,7 @@ impl NodeConfig {
>> } else {
>> AcmeAccountName::from_string("default".to_string())? // should really not happen
>> };
>> - AcmeClient::load(&account).await
>> + proxmox_acme_api::load_client_with_account(&account).await
>> }
>>
>> pub fn acme_domains(&'_ self) -> AcmeDomainIter<'_> {
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* [pbs-devel] applied-series: [PATCH proxmox-backup v6 0/4] datastore: remove config reload on hot path
2026-01-05 14:16 12% [pbs-devel] [PATCH proxmox-backup v6 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
` (3 preceding siblings ...)
2026-01-05 14:16 13% ` [pbs-devel] [PATCH proxmox-backup v6 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
@ 2026-01-14 9:54 5% ` Fabian Grünbichler
4 siblings, 0 replies; 200+ results
From: Fabian Grünbichler @ 2026-01-14 9:54 UTC (permalink / raw)
To: pbs-devel, Samuel Rufinatscha
On Mon, 05 Jan 2026 15:16:10 +0100, Samuel Rufinatscha wrote:
> this series reduces CPU time in datastore lookups by avoiding repeated
> datastore.cfg reads/parses in both `lookup_datastore()` and
> `DataStore::Drop`. It also adds a TTL so manual config edits are
> noticed without reintroducing hashing on every request.
>
> While investigating #6049 [1], cargo-flamegraph [2] showed hotspots
> during repeated `/status` calls in `lookup_datastore()` and in `Drop`,
> dominated by `pbs_config::datastore::config()` (config parse).
>
> [...]
Applied with some rewording of commit messages to make them less
verbose/boiler-platey, thanks!
[1/4] config: enable config version cache for datastore
commit: d14b7469a72f4265bcc1727a1274b207bc201be0
[2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups
commit: be6d251e4483474754cdc5f6d12f2674e22fa132
[3/4] partial fix #6049: datastore: use config fast-path in Drop
commit: 584fa961909c32565046c39f95485273c0a8cba5
[4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits
commit: 07ab13e5aaf1d6b790234d5238c1c3668c56c22e
Best regards,
--
Fabian Grünbichler <f.gruenbichler@proxmox.com>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient
2026-01-14 8:56 6% ` Samuel Rufinatscha
@ 2026-01-14 9:58 5% ` Fabian Grünbichler
2026-01-14 10:52 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2026-01-14 9:58 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Samuel Rufinatscha
On January 14, 2026 9:56 am, Samuel Rufinatscha wrote:
> On 1/13/26 2:44 PM, Fabian Grünbichler wrote:
>> On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
>>> PBS currently uses its own ACME client and API logic, while PDM uses the
>>> factored out proxmox-acme and proxmox-acme-api crates. This duplication
>>> risks differences in behaviour and requires ACME maintenance in two
>>> places. This patch is part of a series to move PBS over to the shared
>>> ACME stack.
>>>
>>> Changes:
>>> - Remove the local src/acme/client.rs and switch to
>>> proxmox_acme::async_client::AcmeClient where needed.
>>> - Use proxmox_acme_api::load_client_with_account to the custom
>>> AcmeClient::load() function
>>> - Replace the local do_register() logic with
>>> proxmox_acme_api::register_account, to further ensure accounts are persisted
>>> - Replace the local AcmeAccountName type, required for
>>> proxmox_acme_api::register_account
>>>
>>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>>> ---
>>> src/acme/client.rs | 691 -------------------------
>>> src/acme/mod.rs | 3 -
>>> src/acme/plugin.rs | 2 +-
>>> src/api2/config/acme.rs | 50 +-
>>> src/api2/node/certificates.rs | 2 +-
>>> src/api2/types/acme.rs | 8 -
>>> src/bin/proxmox_backup_manager/acme.rs | 17 +-
>>> src/config/acme/mod.rs | 8 +-
>>> src/config/node.rs | 9 +-
>>> 9 files changed, 36 insertions(+), 754 deletions(-)
>>> delete mode 100644 src/acme/client.rs
>>>
>>
>> [..]
>>
>>> diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
>>> index ac89ae5e..e4639c53 100644
>>> --- a/src/config/acme/mod.rs
>>> +++ b/src/config/acme/mod.rs
>>
>> I think this whole file should probably be replaced entirely by
>> proxmox-acme-api , which - AFAICT - would just require adding the
>> completion helpers there?
>>
>
> Good point, yes I think moving the completion helpers would
> allow us to get rid of this file. PDM does not make use of
> them / there is atm no 1:1 code in proxmox/ for these helpers.
only because https://bugzilla.proxmox.com/show_bug.cgi?id=7179 is not
yet implemented ;) so please coordinate with Shan to avoid doing the
work twice.
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox v5 2/4] acme: introduce http_status module
2026-01-13 13:45 5% ` Fabian Grünbichler
@ 2026-01-14 10:29 6% ` Samuel Rufinatscha
2026-01-15 9:25 5% ` Fabian Grünbichler
0 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-14 10:29 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 1/13/26 2:45 PM, Fabian Grünbichler wrote:
> On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
>> Introduce an internal http_status module with the common ACME HTTP
>> response codes, and replace use of crate::request::CREATED as well as
>> direct numeric status code usages.
>
> why not use http::status ? we already have this as dependency pretty
> much everywhere we do anything HTTP related.. would also for nicer error
> messages in case the status is not as expected..
>
http is only pulled in via the optional client / async-client features,
not the base impl feature. This code here is gated by impl, where http
might not be available. Adding http as a hard
dependency just for the few status code constants feels a bit overkill.
This matches what we discussed in a previous review round:
https://lore.proxmox.com/pbs-devel/2b7574fb-a3c5-4119-8fb6-9649881dba15@proxmox.com/
Also, since this is pub(crate) API, I think we can easily switch to
StatusCode later if http ever becomes a necessary dependency for impl.
OK with you?
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> proxmox-acme/src/account.rs | 8 ++++----
>> proxmox-acme/src/async_client.rs | 4 ++--
>> proxmox-acme/src/lib.rs | 2 ++
>> proxmox-acme/src/request.rs | 11 ++++++++++-
>> 4 files changed, 18 insertions(+), 7 deletions(-)
>>
>> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
>> index d8eb3e73..ea1a3c60 100644
>> --- a/proxmox-acme/src/account.rs
>> +++ b/proxmox-acme/src/account.rs
>> @@ -84,7 +84,7 @@ impl Account {
>> method: "POST",
>> content_type: crate::request::JSON_CONTENT_TYPE,
>> body,
>> - expected: crate::request::CREATED,
>> + expected: crate::http_status::CREATED,
>> };
>>
>> Ok(NewOrder::new(request))
>> @@ -106,7 +106,7 @@ impl Account {
>> method: "POST",
>> content_type: crate::request::JSON_CONTENT_TYPE,
>> body,
>> - expected: 200,
>> + expected: crate::http_status::OK,
>> })
>> }
>>
>> @@ -131,7 +131,7 @@ impl Account {
>> method: "POST",
>> content_type: crate::request::JSON_CONTENT_TYPE,
>> body,
>> - expected: 200,
>> + expected: crate::http_status::OK,
>> })
>> }
>>
>> @@ -321,7 +321,7 @@ impl AccountCreator {
>> method: "POST",
>> content_type: crate::request::JSON_CONTENT_TYPE,
>> body,
>> - expected: crate::request::CREATED,
>> + expected: crate::http_status::CREATED,
>> })
>> }
>>
>> diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
>> index 2ff3ba22..043648bb 100644
>> --- a/proxmox-acme/src/async_client.rs
>> +++ b/proxmox-acme/src/async_client.rs
>> @@ -498,7 +498,7 @@ impl AcmeClient {
>> method: "GET",
>> content_type: "",
>> body: String::new(),
>> - expected: 200,
>> + expected: crate::http_status::OK,
>> },
>> nonce,
>> )
>> @@ -550,7 +550,7 @@ impl AcmeClient {
>> method: "HEAD",
>> content_type: "",
>> body: String::new(),
>> - expected: 200,
>> + expected: crate::http_status::OK,
>> },
>> nonce,
>> )
>> diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
>> index 6722030c..6051a025 100644
>> --- a/proxmox-acme/src/lib.rs
>> +++ b/proxmox-acme/src/lib.rs
>> @@ -70,6 +70,8 @@ pub use order::Order;
>> #[cfg(feature = "impl")]
>> pub use order::NewOrder;
>> #[cfg(feature = "impl")]
>> +pub(crate) use request::http_status;
>> +#[cfg(feature = "impl")]
>> pub use request::ErrorResponse;
>>
>> /// Header name for nonces.
>> diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
>> index dadfc5af..341ce53e 100644
>> --- a/proxmox-acme/src/request.rs
>> +++ b/proxmox-acme/src/request.rs
>> @@ -1,7 +1,6 @@
>> use serde::Deserialize;
>>
>> pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
>> -pub(crate) const CREATED: u16 = 201;
>>
>> /// A request which should be performed on the ACME provider.
>> pub(crate) struct Request {
>> @@ -21,6 +20,16 @@ pub(crate) struct Request {
>> pub(crate) expected: u16,
>> }
>>
>> +/// Common HTTP status codes used in ACME responses.
>> +pub(crate) mod http_status {
>> + /// 200 OK
>> + pub(crate) const OK: u16 = 200;
>> + /// 201 Created
>> + pub(crate) const CREATED: u16 = 201;
>> + /// 204 No Content
>> + pub(crate) const NO_CONTENT: u16 = 204;
>> +}
>> +
>> /// An ACME error response contains a specially formatted type string, and can optionally
>> /// contain textual details and a set of sub problems.
>> #[derive(Clone, Debug, Deserialize)]
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup v3 2/4] pbs-config: cache verified API token secrets
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox-backup v3 2/4] pbs-config: cache verified API token secrets Samuel Rufinatscha
@ 2026-01-14 10:44 5% ` Fabian Grünbichler
0 siblings, 0 replies; 200+ results
From: Fabian Grünbichler @ 2026-01-14 10:44 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 2, 2026 5:07 pm, Samuel Rufinatscha wrote:
> Currently, every token-based API request reads the token.shadow file and
> runs the expensive password hash verification for the given token
> secret. This shows up as a hotspot in /status profiling (see
> bug #7017 [1]).
>
> This patch introduces an in-memory cache of successfully verified token
> secrets. Subsequent requests for the same token+secret combination only
> perform a comparison using openssl::memcmp::eq and avoid re-running the
> password hash. The cache is updated when a token secret is set and
> cleared when a token is deleted. Note, this does NOT include manual
> config changes, which will be covered in a subsequent patch.
>
> This patch is part of the series which fixes bug #7017 [1].
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> Changes from v1 to v2:
>
> * Replace OnceCell with LazyLock, and std::sync::RwLock with
> parking_lot::RwLock.
> * Add API_MUTATION_GENERATION and guard cache inserts
> to prevent “zombie inserts” across concurrent set/delete.
> * Refactor cache operations into cache_try_secret_matches,
> cache_try_insert_secret, and centralize write-side behavior in
> apply_api_mutation.
> * Switch fast-path cache access to try_read/try_write (best-effort).
>
> Changes from v2 to v3:
>
> * Replaced process-local cache invalidation (AtomicU64
> API_MUTATION_GENERATION) with a cross-process shared generation via
> ConfigVersionCache.
> * Validate shared generation before/after the constant-time secret
> compare; only insert into cache if the generation is unchanged.
> * invalidate_cache_state() on insert if shared generation changed.
>
> Cargo.toml | 1 +
> pbs-config/Cargo.toml | 1 +
> pbs-config/src/token_shadow.rs | 157 ++++++++++++++++++++++++++++++++-
> 3 files changed, 158 insertions(+), 1 deletion(-)
>
> diff --git a/Cargo.toml b/Cargo.toml
> index 1aa57ae5..821b63b7 100644
> --- a/Cargo.toml
> +++ b/Cargo.toml
> @@ -143,6 +143,7 @@ nom = "7"
> num-traits = "0.2"
> once_cell = "1.3.1"
> openssl = "0.10.40"
> +parking_lot = "0.12"
> percent-encoding = "2.1"
> pin-project-lite = "0.2"
> regex = "1.5.5"
> diff --git a/pbs-config/Cargo.toml b/pbs-config/Cargo.toml
> index 74afb3c6..eb81ce00 100644
> --- a/pbs-config/Cargo.toml
> +++ b/pbs-config/Cargo.toml
> @@ -13,6 +13,7 @@ libc.workspace = true
> nix.workspace = true
> once_cell.workspace = true
> openssl.workspace = true
> +parking_lot.workspace = true
> regex.workspace = true
> serde.workspace = true
> serde_json.workspace = true
> diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
> index 640fabbf..fa84aee5 100644
> --- a/pbs-config/src/token_shadow.rs
> +++ b/pbs-config/src/token_shadow.rs
> @@ -1,6 +1,8 @@
> use std::collections::HashMap;
> +use std::sync::LazyLock;
>
> use anyhow::{bail, format_err, Error};
> +use parking_lot::RwLock;
> use serde::{Deserialize, Serialize};
> use serde_json::{from_value, Value};
>
> @@ -13,6 +15,18 @@ use crate::{open_backup_lockfile, BackupLockGuard};
> const LOCK_FILE: &str = pbs_buildcfg::configdir!("/token.shadow.lock");
> const CONF_FILE: &str = pbs_buildcfg::configdir!("/token.shadow");
>
> +/// Global in-memory cache for successfully verified API token secrets.
> +/// The cache stores plain text secrets for token Authids that have already been
> +/// verified against the hashed values in `token.shadow`. This allows for cheap
> +/// subsequent authentications for the same token+secret combination, avoiding
> +/// recomputing the password hash on every request.
> +static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new(|| {
> + RwLock::new(ApiTokenSecretCache {
> + secrets: HashMap::new(),
> + shared_gen: 0,
> + })
> +});
> +
> #[derive(Serialize, Deserialize)]
> #[serde(rename_all = "kebab-case")]
> /// ApiToken id / secret pair
> @@ -54,9 +68,27 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
> bail!("not an API token ID");
> }
>
> + // Fast path
> + if cache_try_secret_matches(tokenid, secret) {
> + return Ok(());
> + }
> +
> + // Slow path
> + // First, capture the shared generation before doing the hash verification.
> + let gen_before = token_shadow_shared_gen();
> +
> let data = read_file()?;
> match data.get(tokenid) {
> - Some(hashed_secret) => proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret),
> + Some(hashed_secret) => {
> + proxmox_sys::crypt::verify_crypt_pw(secret, hashed_secret)?;
> +
> + // Try to cache only if nothing changed while verifying the secret.
> + if let Some(gen) = gen_before {
> + cache_try_insert_secret(tokenid.clone(), secret.to_owned(), gen);
> + }
> +
> + Ok(())
> + }
> None => bail!("invalid API token"),
> }
> }
> @@ -82,6 +114,8 @@ fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
> data.insert(tokenid.clone(), hashed_secret);
> write_file(data)?;
>
> + apply_api_mutation(tokenid, Some(secret));
> +
> Ok(())
> }
>
> @@ -97,5 +131,126 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
> data.remove(tokenid);
> write_file(data)?;
>
> + apply_api_mutation(tokenid, None);
> +
> Ok(())
> }
> +
> +struct ApiTokenSecretCache {
> + /// Keys are token Authids, values are the corresponding plain text secrets.
> + /// Entries are added after a successful on-disk verification in
> + /// `verify_secret` or when a new token secret is generated by
> + /// `generate_and_set_secret`. Used to avoid repeated
> + /// password-hash computation on subsequent authentications.
> + secrets: HashMap<Authid, CachedSecret>,
> + /// Shared generation to detect mutations of the underlying token.shadow file.
> + shared_gen: usize,
> +}
> +
> +/// Cached secret.
> +struct CachedSecret {
> + secret: String,
> +}
> +
> +fn cache_try_insert_secret(tokenid: Authid, secret: String, shared_gen_before: usize) {
> + let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
> + return;
> + };
> +
> + let Some(shared_gen_now) = token_shadow_shared_gen() else {
> + return;
> + };
> +
> + // If this process missed a generation bump, its cache is stale.
> + if cache.shared_gen != shared_gen_now {
> + invalidate_cache_state(&mut cache);
> + cache.shared_gen = shared_gen_now;
> + }
> +
> + // If a mutation happened while we were verifying the secret, do not insert.
> + if shared_gen_now == shared_gen_before {
> + cache.secrets.insert(tokenid, CachedSecret { secret });
> + }
> +}
> +
> +// Tries to match the given token secret against the cached secret.
> +// Checks the generation before and after the constant-time compare to avoid a
> +// TOCTOU window. If another process rotates/deletes a token while we're validating
> +// the cached secret, the generation will change, and we
> +// must not trust the cache for this request.
> +fn cache_try_secret_matches(tokenid: &Authid, secret: &str) -> bool {
> + let Some(cache) = TOKEN_SECRET_CACHE.try_read() else {
> + return false;
> + };
> + let Some(entry) = cache.secrets.get(tokenid) else {
> + return false;
> + };
> +
> + let cache_gen = cache.shared_gen;
> +
> + let Some(gen1) = token_shadow_shared_gen() else {
> + return false;
> + };
> + if gen1 != cache_gen {
> + return false;
> + }
> +
> + let eq = openssl::memcmp::eq(entry.secret.as_bytes(), secret.as_bytes());
should we invalidate the cache here for this particular authid in case
of a mismatch, to avoid making brute forcing too easy/cheap?
> + let Some(gen2) = token_shadow_shared_gen() else {
> + return false;
> + };
> +
> + eq && gen2 == cache_gen
> +}
> +
> +fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
> + // Signal cache invalidation to other processes (best-effort).
> + let new_shared_gen = bump_token_shadow_shared_gen();
> +
> + let mut cache = TOKEN_SECRET_CACHE.write();
> +
> + // If we cannot read/bump the shared generation, we cannot safely trust the cache.
> + let Some(gen) = new_shared_gen else {
> + invalidate_cache_state(&mut cache);
> + cache.shared_gen = 0;
> + return;
> + };
> +
> + // Update to the post-mutation generation.
> + cache.shared_gen = gen;
> +
> + // Apply the new mutation.
> + match new_secret {
> + Some(secret) => {
> + cache.secrets.insert(
> + tokenid.clone(),
> + CachedSecret {
> + secret: secret.to_owned(),
> + },
> + );
> + }
> + None => {
> + cache.secrets.remove(tokenid);
> + }
> + }
> +}
> +
> +/// Get the current shared generation.
> +fn token_shadow_shared_gen() -> Option<usize> {
> + crate::ConfigVersionCache::new()
> + .ok()
> + .map(|cvc| cvc.token_shadow_generation())
> +}
> +
> +/// Bump and return the new shared generation.
> +fn bump_token_shadow_shared_gen() -> Option<usize> {
> + crate::ConfigVersionCache::new()
> + .ok()
> + .map(|cvc| cvc.increase_token_shadow_generation() + 1)
> +}
> +
> +/// Invalidates the cache state and only keeps the shared generation.
both calls to this actually set the cached generation to some value
right after, so maybe this should take a generation directly and set it?
> +fn invalidate_cache_state(cache: &mut ApiTokenSecretCache) {
> + cache.secrets.clear();
> +}
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v3 3/4] pbs-config: invalidate token-secret cache on token.shadow changes
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox-backup v3 3/4] pbs-config: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
@ 2026-01-14 10:44 5% ` Fabian Grünbichler
0 siblings, 0 replies; 200+ results
From: Fabian Grünbichler @ 2026-01-14 10:44 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 2, 2026 5:07 pm, Samuel Rufinatscha wrote:
> Previously the in-memory token-secret cache was only updated via
> set_secret() and delete_secret(), so manual edits to token.shadow were
> not reflected.
>
> This patch adds file change detection to the cache. It tracks the mtime
> and length of token.shadow and clears the in-memory token secret cache
> whenever these values change.
>
> Note, this patch fetches file stats on every request. An TTL-based
> optimization will be covered in a subsequent patch of the series.
>
> This patch is part of the series which fixes bug #7017 [1].
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> Changes from v1 to v2:
>
> * Add file metadata tracking (file_mtime, file_len) and
> FILE_GENERATION.
> * Store file_gen in CachedSecret and verify it against the current
> FILE_GENERATION to ensure cached entries belong to the current file
> state.
> * Add shadow_mtime_len() helper and convert refresh to best-effort
> (try_write, returns bool).
> * Pass a pre-write metadata snapshot into apply_api_mutation and
> clear/bump generation if the cache metadata indicates missed external
> edits.
>
> Changes from v2 to v3:
>
> * Cache now tracks last_checked (epoch seconds).
> * Simplified refresh_cache_if_file_changed, removed
> FILE_GENERATION logic
> * On first load, initializes file metadata and keeps empty cache.
>
> pbs-config/src/token_shadow.rs | 122 +++++++++++++++++++++++++++++++--
> 1 file changed, 118 insertions(+), 4 deletions(-)
>
> diff --git a/pbs-config/src/token_shadow.rs b/pbs-config/src/token_shadow.rs
> index fa84aee5..02fb191b 100644
> --- a/pbs-config/src/token_shadow.rs
> +++ b/pbs-config/src/token_shadow.rs
> @@ -1,5 +1,8 @@
> use std::collections::HashMap;
> +use std::fs;
> +use std::io::ErrorKind;
> use std::sync::LazyLock;
> +use std::time::SystemTime;
>
> use anyhow::{bail, format_err, Error};
> use parking_lot::RwLock;
> @@ -7,6 +10,7 @@ use serde::{Deserialize, Serialize};
> use serde_json::{from_value, Value};
>
> use proxmox_sys::fs::CreateOptions;
> +use proxmox_time::epoch_i64;
>
> use pbs_api_types::Authid;
> //use crate::auth;
> @@ -24,6 +28,9 @@ static TOKEN_SECRET_CACHE: LazyLock<RwLock<ApiTokenSecretCache>> = LazyLock::new
> RwLock::new(ApiTokenSecretCache {
> secrets: HashMap::new(),
> shared_gen: 0,
> + file_mtime: None,
> + file_len: None,
> + last_checked: None,
> })
> });
>
> @@ -62,6 +69,63 @@ fn write_file(data: HashMap<Authid, String>) -> Result<(), Error> {
> proxmox_sys::fs::replace_file(CONF_FILE, &json, options, true)
> }
>
> +/// Refreshes the in-memory cache if the on-disk token.shadow file changed.
> +/// Returns true if the cache is valid to use, false if not.
> +fn refresh_cache_if_file_changed() -> bool {
> + let now = epoch_i64();
> +
> + // Best-effort refresh under write lock.
> + let Some(mut cache) = TOKEN_SECRET_CACHE.try_write() else {
> + return false;
> + };
> +
> + let Some(shared_gen_now) = token_shadow_shared_gen() else {
> + return false;
> + };
> +
> + // If another process bumped the generation, we don't know what changed -> clear cache
> + if cache.shared_gen != shared_gen_now {
> + invalidate_cache_state(&mut cache);
> + cache.shared_gen = shared_gen_now;
> + }
> +
> + // Stat the file to detect manual edits.
> + let Ok((new_mtime, new_len)) = shadow_mtime_len() else {
> + return false;
> + };
> +
> + // Initialize file stats if we have no prior state.
> + if cache.last_checked.is_none() {
> + cache.secrets.clear(); // ensure cache is empty on first load
> + cache.file_mtime = new_mtime;
> + cache.file_len = new_len;
> + cache.last_checked = Some(now);
> + return true;
this code here
> + }
> +
> + // No change detected.
> + if cache.file_mtime == new_mtime && cache.file_len == new_len {
> + cache.last_checked = Some(now);
> + return true;
> + }
> +
> + // Manual edit detected -> invalidate cache and update stat.
> + cache.secrets.clear();
> + cache.file_mtime = new_mtime;
> + cache.file_len = new_len;
> + cache.last_checked = Some(now);
and this code here are identical. if this is the first invocation, then
the change detection check above cannot be true (the cached mtime and
len will be None).
so we can drop the first if above, and replace the last line in this
hunk with
let prev_last_checked = cache.last_checked.replace(Some(now));
and then skip bumping the generation if this is_none()
OTOH, if we just cleared the cache here, does it make sense to return
true? the cache is empty, so likely querying it *now* makes no sense?
> +
> + // Best-effort propagation to other processes + update local view.
> + if let Some(shared_gen_new) = bump_token_shadow_shared_gen() {
> + cache.shared_gen = shared_gen_new;
> + } else {
> + // Do not fail: local cache is already safe as we cleared it above.
> + // Keep local shared_gen as-is to avoid repeated failed attempts.
> + }
> +
> + true
> +}
> +
> /// Verifies that an entry for given tokenid / API token secret exists
> pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
> if !tokenid.is_token() {
> @@ -69,7 +133,7 @@ pub fn verify_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
> }
>
> // Fast path
> - if cache_try_secret_matches(tokenid, secret) {
> + if refresh_cache_if_file_changed() && cache_try_secret_matches(tokenid, secret) {
> return Ok(());
> }
>
> @@ -109,12 +173,15 @@ fn set_secret(tokenid: &Authid, secret: &str) -> Result<(), Error> {
>
> let _guard = lock_config()?;
>
> + // Capture state before we write to detect external edits.
> + let pre_meta = shadow_mtime_len().unwrap_or((None, None));
> +
> let mut data = read_file()?;
> let hashed_secret = proxmox_sys::crypt::encrypt_pw(secret)?;
> data.insert(tokenid.clone(), hashed_secret);
> write_file(data)?;
>
> - apply_api_mutation(tokenid, Some(secret));
> + apply_api_mutation(tokenid, Some(secret), pre_meta);
>
> Ok(())
> }
> @@ -127,11 +194,14 @@ pub fn delete_secret(tokenid: &Authid) -> Result<(), Error> {
>
> let _guard = lock_config()?;
>
> + // Capture state before we write to detect external edits.
> + let pre_meta = shadow_mtime_len().unwrap_or((None, None));
> +
> let mut data = read_file()?;
> data.remove(tokenid);
> write_file(data)?;
>
> - apply_api_mutation(tokenid, None);
> + apply_api_mutation(tokenid, None, pre_meta);
>
> Ok(())
> }
> @@ -145,6 +215,12 @@ struct ApiTokenSecretCache {
> secrets: HashMap<Authid, CachedSecret>,
> /// Shared generation to detect mutations of the underlying token.shadow file.
> shared_gen: usize,
> + // shadow file mtime to detect changes
> + file_mtime: Option<SystemTime>,
> + // shadow file length to detect changes
> + file_len: Option<u64>,
> + // last time the file metadata was checked
> + last_checked: Option<i64>,
these three are always set together, so wouldn't it make more sense to
make them an Option<ShadowFileInfo> ?
> }
>
> /// Cached secret.
> @@ -204,7 +280,13 @@ fn cache_try_secret_matches(tokenid: &Authid, secret: &str) -> bool {
> eq && gen2 == cache_gen
> }
>
> -fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
> +fn apply_api_mutation(
> + tokenid: &Authid,
> + new_secret: Option<&str>,
> + pre_write_meta: (Option<SystemTime>, Option<u64>),
> +) {
> + let now = epoch_i64();
> +
> // Signal cache invalidation to other processes (best-effort).
> let new_shared_gen = bump_token_shadow_shared_gen();
>
> @@ -220,6 +302,13 @@ fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
> // Update to the post-mutation generation.
> cache.shared_gen = gen;
>
> + // If our cached file metadata does not match the on-disk state before our write,
> + // we likely missed an external/manual edit. We can no longer trust any cached secrets.
> + let (pre_mtime, pre_len) = pre_write_meta;
> + if cache.file_mtime != pre_mtime || cache.file_len != pre_len {
> + cache.secrets.clear();
> + }
> +
> // Apply the new mutation.
> match new_secret {
> Some(secret) => {
> @@ -234,6 +323,20 @@ fn apply_api_mutation(tokenid: &Authid, new_secret: Option<&str>) {
> cache.secrets.remove(tokenid);
> }
> }
> +
> + // Update our view of the file metadata to the post-write state (best-effort).
> + // (If this fails, drop local cache so callers fall back to slow path until refreshed.)
> + match shadow_mtime_len() {
> + Ok((mtime, len)) => {
> + cache.file_mtime = mtime;
> + cache.file_len = len;
> + cache.last_checked = Some(now);
> + }
> + Err(_) => {
> + // If we cannot validate state, do not trust cache.
> + invalidate_cache_state(&mut cache);
> + }
> + }
> }
>
> /// Get the current shared generation.
> @@ -253,4 +356,15 @@ fn bump_token_shadow_shared_gen() -> Option<usize> {
> /// Invalidates the cache state and only keeps the shared generation.
> fn invalidate_cache_state(cache: &mut ApiTokenSecretCache) {
> cache.secrets.clear();
> + cache.file_mtime = None;
> + cache.file_len = None;
> + cache.last_checked = None;
> +}
> +
> +fn shadow_mtime_len() -> Result<(Option<SystemTime>, Option<u64>), Error> {
> + match fs::metadata(CONF_FILE) {
> + Ok(meta) => Ok((meta.modified().ok(), Some(meta.len()))),
> + Err(e) if e.kind() == ErrorKind::NotFound => Ok((None, None)),
> + Err(e) => Err(e.into()),
> + }
> }
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v3 1/4] pbs-config: add token.shadow generation to ConfigVersionCache
2026-01-02 16:07 17% ` [pbs-devel] [PATCH proxmox-backup v3 1/4] pbs-config: add token.shadow generation to ConfigVersionCache Samuel Rufinatscha
@ 2026-01-14 10:44 5% ` Fabian Grünbichler
0 siblings, 0 replies; 200+ results
From: Fabian Grünbichler @ 2026-01-14 10:44 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 2, 2026 5:07 pm, Samuel Rufinatscha wrote:
> Currently, every token-based API request reads the token.shadow file and
> runs the expensive password hash verification for the given token
> secret. This shows up as a hotspot in /status profiling (see
> bug #7017 [1]).
>
> To solve the issue, this patch prepares the config version cache,
> so that token_shadow_generation config caching can be built on
> top of it.
>
> This patch specifically:
> (1) implements increment function in order to invalidate generations
this is needlessly verbose..
>
> This patch is part of the series which fixes bug #7017 [1].
this is already mentioned higher up and doesn't need to be repeated
here.
this patch needs a rebase. it would be good to call out why it is safe
to add to this struct, since it is accessed/mapped by both old and new
processes.
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> pbs-config/src/config_version_cache.rs | 18 ++++++++++++++++++
> 1 file changed, 18 insertions(+)
>
> diff --git a/pbs-config/src/config_version_cache.rs b/pbs-config/src/config_version_cache.rs
> index e8fb994f..1376b11d 100644
> --- a/pbs-config/src/config_version_cache.rs
> +++ b/pbs-config/src/config_version_cache.rs
> @@ -28,6 +28,8 @@ struct ConfigVersionCacheDataInner {
> // datastore (datastore.cfg) generation/version
> // FIXME: remove with PBS 3.0
> datastore_generation: AtomicUsize,
> + // Token shadow (token.shadow) generation/version.
> + token_shadow_generation: AtomicUsize,
> // Add further atomics here
> }
>
> @@ -153,4 +155,20 @@ impl ConfigVersionCache {
> .datastore_generation
> .fetch_add(1, Ordering::AcqRel)
> }
> +
> + /// Returns the token shadow generation number.
> + pub fn token_shadow_generation(&self) -> usize {
> + self.shmem
> + .data()
> + .token_shadow_generation
> + .load(Ordering::Acquire)
> + }
> +
> + /// Increase the token shadow generation number.
> + pub fn increase_token_shadow_generation(&self) -> usize {
> + self.shmem
> + .data()
> + .token_shadow_generation
> + .fetch_add(1, Ordering::AcqRel)
> + }
> }
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-datacenter-manager v3 1/2] pdm-config: implement token.shadow generation
2026-01-02 16:07 13% ` [pbs-devel] [PATCH proxmox-datacenter-manager v3 1/2] pdm-config: implement token.shadow generation Samuel Rufinatscha
@ 2026-01-14 10:45 5% ` Fabian Grünbichler
0 siblings, 0 replies; 200+ results
From: Fabian Grünbichler @ 2026-01-14 10:45 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 2, 2026 5:07 pm, Samuel Rufinatscha wrote:
> PDM depends on the shared proxmox/proxmox-access-control crate for
> token.shadow handling, which expects the product to provide a
> cross-process invalidation signal so it can safely cache verified API
> token secrets and invalidate them when token.shadow is changed.
>
> This patch
>
> * adds a token_shadow_generation to PDM’s shared-memory
> ConfigVersionCache
> * implements proxmox_access_control::init::AccessControlConfig
> for pdm_config::AccessControlConfig, which
> - delegates roles/privs/path checks to the existing
> pdm_api_types::AccessControlConfig implementation
> - implements the shadow cache generation trait functions
> * switches the AccessControlConfig init paths (server + CLI) to use
> pdm_config::AccessControlConfig instead of
> pdm_api_types::AccessControlConfig
>
> This patch is part of the series which fixes bug #7017 [1].
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> cli/admin/src/main.rs | 2 +-
> lib/pdm-config/Cargo.toml | 1 +
> lib/pdm-config/src/access_control_config.rs | 73 +++++++++++++++++++++
> lib/pdm-config/src/config_version_cache.rs | 18 +++++
> lib/pdm-config/src/lib.rs | 2 +
> server/src/acl.rs | 3 +-
> 6 files changed, 96 insertions(+), 3 deletions(-)
> create mode 100644 lib/pdm-config/src/access_control_config.rs
>
> diff --git a/cli/admin/src/main.rs b/cli/admin/src/main.rs
> index f698fa2..916c633 100644
> --- a/cli/admin/src/main.rs
> +++ b/cli/admin/src/main.rs
> @@ -19,7 +19,7 @@ fn main() {
> proxmox_product_config::init(api_user, priv_user);
>
> proxmox_access_control::init::init(
> - &pdm_api_types::AccessControlConfig,
> + &pdm_config::AccessControlConfig,
> pdm_buildcfg::configdir!("/access"),
> )
> .expect("failed to setup access control config");
> diff --git a/lib/pdm-config/Cargo.toml b/lib/pdm-config/Cargo.toml
> index d39c2ad..19781d2 100644
> --- a/lib/pdm-config/Cargo.toml
> +++ b/lib/pdm-config/Cargo.toml
> @@ -13,6 +13,7 @@ once_cell.workspace = true
> openssl.workspace = true
> serde.workspace = true
>
> +proxmox-access-control.workspace = true
> proxmox-config-digest = { workspace = true, features = [ "openssl" ] }
> proxmox-http = { workspace = true, features = [ "http-helpers" ] }
> proxmox-ldap = { workspace = true, features = [ "types" ]}
> diff --git a/lib/pdm-config/src/access_control_config.rs b/lib/pdm-config/src/access_control_config.rs
> new file mode 100644
> index 0000000..6f2e6b3
> --- /dev/null
> +++ b/lib/pdm-config/src/access_control_config.rs
> @@ -0,0 +1,73 @@
> +// e.g. in src/main.rs or server::context mod, wherever convenient
> +
> +use anyhow::Error;
> +use pdm_api_types::{Authid, Userid};
> +use proxmox_section_config::SectionConfigData;
> +use std::collections::HashMap;
> +
> +pub struct AccessControlConfig;
> +
> +impl proxmox_access_control::init::AccessControlConfig for AccessControlConfig {
should we then remove the impl from the api type?
> + fn privileges(&self) -> &HashMap<&str, u64> {
> + pdm_api_types::AccessControlConfig.privileges()
> + }
> +
> + fn roles(&self) -> &HashMap<&str, (u64, &str)> {
> + pdm_api_types::AccessControlConfig.roles()
> + }
> +
> + fn is_superuser(&self, auth_id: &Authid) -> bool {
> + pdm_api_types::AccessControlConfig.is_superuser(auth_id)
> + }
> +
> + fn is_group_member(&self, user_id: &Userid, group: &str) -> bool {
> + pdm_api_types::AccessControlConfig.is_group_member(user_id, group)
> + }
> +
> + fn role_admin(&self) -> Option<&str> {
> + pdm_api_types::AccessControlConfig.role_admin()
> + }
> +
> + fn role_no_access(&self) -> Option<&str> {
> + pdm_api_types::AccessControlConfig.role_no_access()
> + }
> +
> + fn init_user_config(&self, config: &mut SectionConfigData) -> Result<(), Error> {
> + pdm_api_types::AccessControlConfig.init_user_config(config)
> + }
> +
> + fn acl_audit_privileges(&self) -> u64 {
> + pdm_api_types::AccessControlConfig.acl_audit_privileges()
> + }
> +
> + fn acl_modify_privileges(&self) -> u64 {
> + pdm_api_types::AccessControlConfig.acl_modify_privileges()
> + }
> +
> + fn check_acl_path(&self, path: &str) -> Result<(), Error> {
> + pdm_api_types::AccessControlConfig.check_acl_path(path)
> + }
> +
> + fn allow_partial_permission_match(&self) -> bool {
> + pdm_api_types::AccessControlConfig.allow_partial_permission_match()
> + }
> +
> + fn cache_generation(&self) -> Option<usize> {
> + pdm_api_types::AccessControlConfig.cache_generation()
> + }
shouldn't this be wired up to the ConfigVersionCache?
> +
> + fn increment_cache_generation(&self) -> Result<(), Error> {
> + pdm_api_types::AccessControlConfig.increment_cache_generation()
shouldn't this be wired up to the ConfigVersionCache?
> + }
> +
> + fn token_shadow_cache_generation(&self) -> Option<usize> {
> + crate::ConfigVersionCache::new()
> + .ok()
> + .map(|c| c.token_shadow_generation())
> + }
> +
> + fn increment_token_shadow_cache_generation(&self) -> Result<usize, Error> {
> + let c = crate::ConfigVersionCache::new()?;
> + Ok(c.increase_token_shadow_generation())
> + }
> +}
> diff --git a/lib/pdm-config/src/config_version_cache.rs b/lib/pdm-config/src/config_version_cache.rs
> index 36a6a77..933140c 100644
> --- a/lib/pdm-config/src/config_version_cache.rs
> +++ b/lib/pdm-config/src/config_version_cache.rs
> @@ -27,6 +27,8 @@ struct ConfigVersionCacheDataInner {
> traffic_control_generation: AtomicUsize,
> // Tracks updates to the remote/hostname/nodename mapping cache.
> remote_mapping_cache: AtomicUsize,
> + // Token shadow (token.shadow) generation/version.
> + token_shadow_generation: AtomicUsize,
explanation why this is safe for the commit message would be nice ;)
> // Add further atomics here
> }
>
> @@ -172,4 +174,20 @@ impl ConfigVersionCache {
> .fetch_add(1, Ordering::Relaxed)
> + 1
> }
> +
> + /// Returns the token shadow generation number.
> + pub fn token_shadow_generation(&self) -> usize {
> + self.shmem
> + .data()
> + .token_shadow_generation
> + .load(Ordering::Acquire)
> + }
> +
> + /// Increase the token shadow generation number.
> + pub fn increase_token_shadow_generation(&self) -> usize {
> + self.shmem
> + .data()
> + .token_shadow_generation
> + .fetch_add(1, Ordering::AcqRel)
> + }
> }
> diff --git a/lib/pdm-config/src/lib.rs b/lib/pdm-config/src/lib.rs
> index 4c49054..a15a006 100644
> --- a/lib/pdm-config/src/lib.rs
> +++ b/lib/pdm-config/src/lib.rs
> @@ -9,6 +9,8 @@ pub mod remotes;
> pub mod setup;
> pub mod views;
>
> +mod access_control_config;
> +pub use access_control_config::AccessControlConfig;
> mod config_version_cache;
> pub use config_version_cache::ConfigVersionCache;
>
> diff --git a/server/src/acl.rs b/server/src/acl.rs
> index f421814..e6e007b 100644
> --- a/server/src/acl.rs
> +++ b/server/src/acl.rs
> @@ -1,6 +1,5 @@
> pub(crate) fn init() {
> - static ACCESS_CONTROL_CONFIG: pdm_api_types::AccessControlConfig =
> - pdm_api_types::AccessControlConfig;
> + static ACCESS_CONTROL_CONFIG: pdm_config::AccessControlConfig = pdm_config::AccessControlConfig;
>
> proxmox_access_control::init::init(&ACCESS_CONTROL_CONFIG, pdm_buildcfg::configdir!("/access"))
> .expect("failed to setup access control config");
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-datacenter-manager v3 2/2] docs: document API token-cache TTL effects
2026-01-02 16:07 17% ` [pbs-devel] [PATCH proxmox-datacenter-manager v3 2/2] docs: document API token-cache TTL effects Samuel Rufinatscha
@ 2026-01-14 10:45 5% ` Fabian Grünbichler
2026-01-14 11:24 6% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Fabian Grünbichler @ 2026-01-14 10:45 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On January 2, 2026 5:07 pm, Samuel Rufinatscha wrote:
> Documents the effects of the added API token-cache in the
> proxmox-access-control crate. This patch is part of the
> series that fixes bug #7017 [1].
>
> This patch is part of the series which fixes bug #7017 [1].
>
> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
please try to read your commit messages at least once before sending.
the bug is referenced three times here, and it is not necessary to
mention it at all..
>
> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
> ---
> Changes from v2 to v3:
>
> * Reword documentation warning for clarity.
>
> docs/access-control.rst | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/docs/access-control.rst b/docs/access-control.rst
> index adf26cd..18e57a2 100644
> --- a/docs/access-control.rst
> +++ b/docs/access-control.rst
> @@ -47,6 +47,10 @@ place of the user ID (``user@realm``) and the user password, respectively.
> The API token is passed from the client to the server by setting the ``Authorization`` HTTP header
> with method ``PDMAPIToken`` to the value ``TOKENID:TOKENSECRET``.
>
> +.. WARNING:: Direct/manual edits to ``token.shadow`` may take up to 60 seconds (or
> + longer in edge cases) to take effect due to caching. Restart services for
> + immediate effect of manual edits.
> +
> .. _access_control:
>
> Access Control
> --
> 2.47.3
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient
2026-01-14 9:58 5% ` Fabian Grünbichler
@ 2026-01-14 10:52 6% ` Samuel Rufinatscha
2026-01-14 16:41 12% ` Samuel Rufinatscha
0 siblings, 1 reply; 200+ results
From: Samuel Rufinatscha @ 2026-01-14 10:52 UTC (permalink / raw)
To: Fabian Grünbichler, Proxmox Backup Server development discussion
On 1/14/26 10:57 AM, Fabian Grünbichler wrote:
> On January 14, 2026 9:56 am, Samuel Rufinatscha wrote:
>> On 1/13/26 2:44 PM, Fabian Grünbichler wrote:
>>> On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
>>>> PBS currently uses its own ACME client and API logic, while PDM uses the
>>>> factored out proxmox-acme and proxmox-acme-api crates. This duplication
>>>> risks differences in behaviour and requires ACME maintenance in two
>>>> places. This patch is part of a series to move PBS over to the shared
>>>> ACME stack.
>>>>
>>>> Changes:
>>>> - Remove the local src/acme/client.rs and switch to
>>>> proxmox_acme::async_client::AcmeClient where needed.
>>>> - Use proxmox_acme_api::load_client_with_account to the custom
>>>> AcmeClient::load() function
>>>> - Replace the local do_register() logic with
>>>> proxmox_acme_api::register_account, to further ensure accounts are persisted
>>>> - Replace the local AcmeAccountName type, required for
>>>> proxmox_acme_api::register_account
>>>>
>>>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>>>> ---
>>>> src/acme/client.rs | 691 -------------------------
>>>> src/acme/mod.rs | 3 -
>>>> src/acme/plugin.rs | 2 +-
>>>> src/api2/config/acme.rs | 50 +-
>>>> src/api2/node/certificates.rs | 2 +-
>>>> src/api2/types/acme.rs | 8 -
>>>> src/bin/proxmox_backup_manager/acme.rs | 17 +-
>>>> src/config/acme/mod.rs | 8 +-
>>>> src/config/node.rs | 9 +-
>>>> 9 files changed, 36 insertions(+), 754 deletions(-)
>>>> delete mode 100644 src/acme/client.rs
>>>>
>>>
>>> [..]
>>>
>>>> diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
>>>> index ac89ae5e..e4639c53 100644
>>>> --- a/src/config/acme/mod.rs
>>>> +++ b/src/config/acme/mod.rs
>>>
>>> I think this whole file should probably be replaced entirely by
>>> proxmox-acme-api , which - AFAICT - would just require adding the
>>> completion helpers there?
>>>
>>
>> Good point, yes I think moving the completion helpers would
>> allow us to get rid of this file. PDM does not make use of
>> them / there is atm no 1:1 code in proxmox/ for these helpers.
>
> only because https://bugzilla.proxmox.com/show_bug.cgi?id=7179 is not
> yet implemented ;) so please coordinate with Shan to avoid doing the
> work twice.
Ah, good catch! thanks for the reference @Fabian.
@Shan: since #7179 will likely touch the same area, it probably makes
sense to factor out the required helpers as part of this series
to avoid duplicate work. If that works for you, maybe hold off on
parallel changes here until this lands. What do you think?
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-datacenter-manager v3 2/2] docs: document API token-cache TTL effects
2026-01-14 10:45 5% ` Fabian Grünbichler
@ 2026-01-14 11:24 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-14 11:24 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 1/14/26 11:44 AM, Fabian Grünbichler wrote:
> On January 2, 2026 5:07 pm, Samuel Rufinatscha wrote:
>> Documents the effects of the added API token-cache in the
>> proxmox-access-control crate. This patch is part of the
>> series that fixes bug #7017 [1].
>>
>> This patch is part of the series which fixes bug #7017 [1].
>>
>> [1] https://bugzilla.proxmox.com/show_bug.cgi?id=7017
>
> please try to read your commit messages at least once before sending.
> the bug is referenced three times here, and it is not necessary to
> mention it at all..
>
Fair point — the duplication here is a copy/paste slip. I’ll resend v4
with a cleaned-up commit message for this docs only patch and remove
the "part of the series" boilerplate from the other commits. Thanks.
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> Changes from v2 to v3:
>>
>> * Reword documentation warning for clarity.
>>
>> docs/access-control.rst | 4 ++++
>> 1 file changed, 4 insertions(+)
>>
>> diff --git a/docs/access-control.rst b/docs/access-control.rst
>> index adf26cd..18e57a2 100644
>> --- a/docs/access-control.rst
>> +++ b/docs/access-control.rst
>> @@ -47,6 +47,10 @@ place of the user ID (``user@realm``) and the user password, respectively.
>> The API token is passed from the client to the server by setting the ``Authorization`` HTTP header
>> with method ``PDMAPIToken`` to the value ``TOKENID:TOKENSECRET``.
>>
>> +.. WARNING:: Direct/manual edits to ``token.shadow`` may take up to 60 seconds (or
>> + longer in edge cases) to take effect due to caching. Restart services for
>> + immediate effect of manual edits.
>> +
>> .. _access_control:
>>
>> Access Control
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox v5 1/4] acme: reduce visibility of Request type
2026-01-13 13:46 5% ` Fabian Grünbichler
@ 2026-01-14 15:07 6% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-14 15:07 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Fabian Grünbichler
On 1/13/26 2:46 PM, Fabian Grünbichler wrote:
> On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
>> Currently, the low-level ACME Request type is publicly exposed, even
>> though users are expected to go through AcmeClient and
>> proxmox-acme-api handlers. This patch reduces visibility so that
>> the Request type and related fields/methods are crate-internal only.
>
> it also removes a lot of public and private code entirely, not just
> changing visibility.. I think those were intentionally there to allow
> usage without the need to using either of the provided client
> implementations (which are guarded behind feature flags).
>
> if we say the crate should only be used via either the `client` or the
> `async-client` then that's fine, but it should be made explicit and
> discussed.. right now this is sort of half-way there - e.g., the
> Account::new_order method was not made private, even though it makes no
> sense anymore with those other methods/helpers removed..
>
> this patch also breaks a few reference in doc comments that would need
> to be dropped.
>
> a note that this breaks the current usage of proxmox-acme in PBS would
> also be good to have here, if this is kept..
>
Makes sense.
I think the best here is to drop the visibility reductions and removals
and keep the low-level API intact (as it’s currently documented and
feature-gated).
This will keep this series focused on fixing the 204 nonce handling and
switch PBS to the factored-out client / API handlers (to be on the same
base as PDM).
>>
>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>> ---
>> proxmox-acme/src/account.rs | 94 ++-----------------------------
>> proxmox-acme/src/async_client.rs | 2 +-
>> proxmox-acme/src/authorization.rs | 30 ----------
>> proxmox-acme/src/client.rs | 6 +-
>> proxmox-acme/src/lib.rs | 4 --
>> proxmox-acme/src/order.rs | 2 +-
>> proxmox-acme/src/request.rs | 12 ++--
>> 7 files changed, 16 insertions(+), 134 deletions(-)
>>
>> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
>> index f763c1e9..d8eb3e73 100644
>> --- a/proxmox-acme/src/account.rs
>> +++ b/proxmox-acme/src/account.rs
>> @@ -8,12 +8,11 @@ use openssl::pkey::{PKey, Private};
>> use serde::{Deserialize, Serialize};
>> use serde_json::Value;
>>
>> -use crate::authorization::{Authorization, GetAuthorization};
>> use crate::b64u;
>> use crate::directory::Directory;
>> use crate::jws::Jws;
>> use crate::key::{Jwk, PublicKey};
>> -use crate::order::{NewOrder, Order, OrderData};
>> +use crate::order::{NewOrder, OrderData};
>> use crate::request::Request;
>> use crate::types::{AccountData, AccountStatus, ExternalAccountBinding};
>> use crate::Error;
>> @@ -92,7 +91,7 @@ impl Account {
>> }
>>
>> /// Prepare a "POST-as-GET" request to fetch data. Low level helper.
>> - pub fn get_request(&self, url: &str, nonce: &str) -> Result<Request, Error> {
>> + pub(crate) fn get_request(&self, url: &str, nonce: &str) -> Result<Request, Error> {
>> let key = PKey::private_key_from_pem(self.private_key.as_bytes())?;
>> let body = serde_json::to_string(&Jws::new_full(
>> &key,
>> @@ -112,7 +111,7 @@ impl Account {
>> }
>>
>> /// Prepare a JSON POST request. Low level helper.
>> - pub fn post_request<T: Serialize>(
>> + pub(crate) fn post_request<T: Serialize>(
>> &self,
>> url: &str,
>> nonce: &str,
>> @@ -136,31 +135,6 @@ impl Account {
>> })
>> }
>>
>> - /// Prepare a JSON POST request.
>> - fn post_request_raw_payload(
>> - &self,
>> - url: &str,
>> - nonce: &str,
>> - payload: String,
>> - ) -> Result<Request, Error> {
>> - let key = PKey::private_key_from_pem(self.private_key.as_bytes())?;
>> - let body = serde_json::to_string(&Jws::new_full(
>> - &key,
>> - Some(self.location.clone()),
>> - url.to_owned(),
>> - nonce.to_owned(),
>> - payload,
>> - )?)?;
>> -
>> - Ok(Request {
>> - url: url.to_owned(),
>> - method: "POST",
>> - content_type: crate::request::JSON_CONTENT_TYPE,
>> - body,
>> - expected: 200,
>> - })
>> - }
>> -
>> /// Get the "key authorization" for a token.
>> pub fn key_authorization(&self, token: &str) -> Result<String, Error> {
>> let key = PKey::private_key_from_pem(self.private_key.as_bytes())?;
>> @@ -176,64 +150,6 @@ impl Account {
>> Ok(b64u::encode(digest))
>> }
>>
>> - /// Prepare a request to update account data.
>> - ///
>> - /// This is a rather low level interface. You should know what you're doing.
>> - pub fn update_account_request<T: Serialize>(
>> - &self,
>> - nonce: &str,
>> - data: &T,
>> - ) -> Result<Request, Error> {
>> - self.post_request(&self.location, nonce, data)
>> - }
>> -
>> - /// Prepare a request to deactivate this account.
>> - pub fn deactivate_account_request<T: Serialize>(&self, nonce: &str) -> Result<Request, Error> {
>> - self.post_request_raw_payload(
>> - &self.location,
>> - nonce,
>> - r#"{"status":"deactivated"}"#.to_string(),
>> - )
>> - }
>> -
>> - /// Prepare a request to query an Authorization for an Order.
>> - ///
>> - /// Returns `Ok(None)` if `auth_index` is out of out of range. You can query the number of
>> - /// authorizations from via [`Order::authorization_len`] or by manually inspecting its
>> - /// `.data.authorization` vector.
>> - pub fn get_authorization(
>> - &self,
>> - order: &Order,
>> - auth_index: usize,
>> - nonce: &str,
>> - ) -> Result<Option<GetAuthorization>, Error> {
>> - match order.authorization(auth_index) {
>> - None => Ok(None),
>> - Some(url) => Ok(Some(GetAuthorization::new(self.get_request(url, nonce)?))),
>> - }
>> - }
>> -
>> - /// Prepare a request to validate a Challenge from an Authorization.
>> - ///
>> - /// Returns `Ok(None)` if `challenge_index` is out of out of range. The challenge count is
>> - /// available by inspecting the [`Authorization::challenges`] vector.
>> - ///
>> - /// This returns a raw `Request` since validation takes some time and the `Authorization`
>> - /// object has to be re-queried and its `status` inspected.
>> - pub fn validate_challenge(
>> - &self,
>> - authorization: &Authorization,
>> - challenge_index: usize,
>> - nonce: &str,
>> - ) -> Result<Option<Request>, Error> {
>> - match authorization.challenges.get(challenge_index) {
>> - None => Ok(None),
>> - Some(challenge) => self
>> - .post_request_raw_payload(&challenge.url, nonce, "{}".to_string())
>> - .map(Some),
>> - }
>> - }
>> -
>> /// Prepare a request to revoke a certificate.
>> ///
>> /// The certificate can be either PEM or DER formatted.
>> @@ -274,7 +190,7 @@ pub struct CertificateRevocation<'a> {
>>
>> impl CertificateRevocation<'_> {
>> /// Create the revocation request using the specified nonce for the given directory.
>> - pub fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
>> + pub(crate) fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
>> let revoke_cert = directory.data.revoke_cert.as_ref().ok_or_else(|| {
>> Error::Custom("no 'revokeCert' URL specified by provider".to_string())
>> })?;
>> @@ -364,7 +280,7 @@ impl AccountCreator {
>> /// the resulting request.
>> /// Changing the private key between using the request and passing the response to
>> /// [`response`](AccountCreator::response()) will render the account unusable!
>> - pub fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
>> + pub(crate) fn request(&self, directory: &Directory, nonce: &str) -> Result<Request, Error> {
>> let key = self.key.as_deref().ok_or(Error::MissingKey)?;
>> let url = directory.new_account_url().ok_or_else(|| {
>> Error::Custom("no 'newAccount' URL specified by provider".to_string())
>> diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
>> index dc755fb9..2ff3ba22 100644
>> --- a/proxmox-acme/src/async_client.rs
>> +++ b/proxmox-acme/src/async_client.rs
>> @@ -10,7 +10,7 @@ use proxmox_http::{client::Client, Body};
>>
>> use crate::account::AccountCreator;
>> use crate::order::{Order, OrderData};
>> -use crate::Request as AcmeRequest;
>> +use crate::request::Request as AcmeRequest;
>> use crate::{Account, Authorization, Challenge, Directory, Error, ErrorResponse};
>>
>> /// A non-blocking Acme client using tokio/hyper.
>> diff --git a/proxmox-acme/src/authorization.rs b/proxmox-acme/src/authorization.rs
>> index 28bc1b4b..7027381a 100644
>> --- a/proxmox-acme/src/authorization.rs
>> +++ b/proxmox-acme/src/authorization.rs
>> @@ -6,8 +6,6 @@ use serde::{Deserialize, Serialize};
>> use serde_json::Value;
>>
>> use crate::order::Identifier;
>> -use crate::request::Request;
>> -use crate::Error;
>>
>> /// Status of an [`Authorization`].
>> #[derive(Clone, Copy, Debug, Eq, PartialEq, Deserialize, Serialize)]
>> @@ -132,31 +130,3 @@ impl Challenge {
>> fn is_false(b: &bool) -> bool {
>> !*b
>> }
>> -
>> -/// Represents an in-flight query for an authorization.
>> -///
>> -/// This is created via [`Account::get_authorization`](crate::Account::get_authorization()).
>> -pub struct GetAuthorization {
>> - //order: OrderData,
>> - /// The request to send to the ACME provider. This is wrapped in an option in order to allow
>> - /// moving it out instead of copying the contents.
>> - ///
>> - /// When generated via [`Account::get_authorization`](crate::Account::get_authorization()),
>> - /// this is guaranteed to be `Some`.
>> - ///
>> - /// The response should be passed to the the [`response`](GetAuthorization::response()) method.
>> - pub request: Option<Request>,
>> -}
>> -
>> -impl GetAuthorization {
>> - pub(crate) fn new(request: Request) -> Self {
>> - Self {
>> - request: Some(request),
>> - }
>> - }
>> -
>> - /// Deal with the response we got from the server.
>> - pub fn response(self, response_body: &[u8]) -> Result<Authorization, Error> {
>> - Ok(serde_json::from_slice(response_body)?)
>> - }
>> -}
>> diff --git a/proxmox-acme/src/client.rs b/proxmox-acme/src/client.rs
>> index 931f7245..5c812567 100644
>> --- a/proxmox-acme/src/client.rs
>> +++ b/proxmox-acme/src/client.rs
>> @@ -7,8 +7,8 @@ use serde::{Deserialize, Serialize};
>> use crate::b64u;
>> use crate::error;
>> use crate::order::OrderData;
>> -use crate::request::ErrorResponse;
>> -use crate::{Account, Authorization, Challenge, Directory, Error, Order, Request};
>> +use crate::request::{ErrorResponse, Request};
>> +use crate::{Account, Authorization, Challenge, Directory, Error, Order};
>>
>> macro_rules! format_err {
>> ($($fmt:tt)*) => { Error::Client(format!($($fmt)*)) };
>> @@ -564,7 +564,7 @@ impl Client {
>> }
>>
>> /// Low-level API to run an n API request. This automatically updates the current nonce!
>> - pub fn run_request(&mut self, request: Request) -> Result<HttpResponse, Error> {
>> + pub(crate) fn run_request(&mut self, request: Request) -> Result<HttpResponse, Error> {
>> self.inner.run_request(request)
>> }
>>
>> diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
>> index df722629..6722030c 100644
>> --- a/proxmox-acme/src/lib.rs
>> +++ b/proxmox-acme/src/lib.rs
>> @@ -66,10 +66,6 @@ pub use error::Error;
>> #[doc(inline)]
>> pub use order::Order;
>>
>> -#[cfg(feature = "impl")]
>> -#[doc(inline)]
>> -pub use request::Request;
>> -
>> // we don't inline these:
>> #[cfg(feature = "impl")]
>> pub use order::NewOrder;
>> diff --git a/proxmox-acme/src/order.rs b/proxmox-acme/src/order.rs
>> index b6551004..432a81a4 100644
>> --- a/proxmox-acme/src/order.rs
>> +++ b/proxmox-acme/src/order.rs
>> @@ -153,7 +153,7 @@ pub struct NewOrder {
>> //order: OrderData,
>> /// The request to execute to place the order. When creating a [`NewOrder`] via
>> /// [`Account::new_order`](crate::Account::new_order) this is guaranteed to be `Some`.
>> - pub request: Option<Request>,
>> + pub(crate) request: Option<Request>,
>> }
>>
>> impl NewOrder {
>> diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
>> index 78a90913..dadfc5af 100644
>> --- a/proxmox-acme/src/request.rs
>> +++ b/proxmox-acme/src/request.rs
>> @@ -4,21 +4,21 @@ pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
>> pub(crate) const CREATED: u16 = 201;
>>
>> /// A request which should be performed on the ACME provider.
>> -pub struct Request {
>> +pub(crate) struct Request {
>> /// The complete URL to send the request to.
>> - pub url: String,
>> + pub(crate) url: String,
>>
>> /// The HTTP method name to use.
>> - pub method: &'static str,
>> + pub(crate) method: &'static str,
>>
>> /// The `Content-Type` header to pass along.
>> - pub content_type: &'static str,
>> + pub(crate) content_type: &'static str,
>>
>> /// The body to pass along with request, or an empty string.
>> - pub body: String,
>> + pub(crate) body: String,
>>
>> /// The expected status code a compliant ACME provider will return on success.
>> - pub expected: u16,
>> + pub(crate) expected: u16,
>> }
>>
>> /// An ACME error response contains a specially formatted type string, and can optionally
>> --
>> 2.47.3
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 6%]
* Re: [pbs-devel] [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient
2026-01-14 10:52 6% ` Samuel Rufinatscha
@ 2026-01-14 16:41 12% ` Samuel Rufinatscha
0 siblings, 0 replies; 200+ results
From: Samuel Rufinatscha @ 2026-01-14 16:41 UTC (permalink / raw)
To: Fabian Grünbichler, Proxmox Backup Server development discussion
On 1/14/26 11:52 AM, Samuel Rufinatscha wrote:
> On 1/14/26 10:57 AM, Fabian Grünbichler wrote:
>> On January 14, 2026 9:56 am, Samuel Rufinatscha wrote:
>>> On 1/13/26 2:44 PM, Fabian Grünbichler wrote:
>>>> On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
>>>>> PBS currently uses its own ACME client and API logic, while PDM
>>>>> uses the
>>>>> factored out proxmox-acme and proxmox-acme-api crates. This
>>>>> duplication
>>>>> risks differences in behaviour and requires ACME maintenance in two
>>>>> places. This patch is part of a series to move PBS over to the shared
>>>>> ACME stack.
>>>>>
>>>>> Changes:
>>>>> - Remove the local src/acme/client.rs and switch to
>>>>> proxmox_acme::async_client::AcmeClient where needed.
>>>>> - Use proxmox_acme_api::load_client_with_account to the custom
>>>>> AcmeClient::load() function
>>>>> - Replace the local do_register() logic with
>>>>> proxmox_acme_api::register_account, to further ensure accounts are
>>>>> persisted
>>>>> - Replace the local AcmeAccountName type, required for
>>>>> proxmox_acme_api::register_account
>>>>>
>>>>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>>>>> ---
>>>>> src/acme/client.rs | 691
>>>>> -------------------------
>>>>> src/acme/mod.rs | 3 -
>>>>> src/acme/plugin.rs | 2 +-
>>>>> src/api2/config/acme.rs | 50 +-
>>>>> src/api2/node/certificates.rs | 2 +-
>>>>> src/api2/types/acme.rs | 8 -
>>>>> src/bin/proxmox_backup_manager/acme.rs | 17 +-
>>>>> src/config/acme/mod.rs | 8 +-
>>>>> src/config/node.rs | 9 +-
>>>>> 9 files changed, 36 insertions(+), 754 deletions(-)
>>>>> delete mode 100644 src/acme/client.rs
>>>>>
>>>>
>>>> [..]
>>>>
>>>>> diff --git a/src/config/acme/mod.rs b/src/config/acme/mod.rs
>>>>> index ac89ae5e..e4639c53 100644
>>>>> --- a/src/config/acme/mod.rs
>>>>> +++ b/src/config/acme/mod.rs
>>>>
>>>> I think this whole file should probably be replaced entirely by
>>>> proxmox-acme-api , which - AFAICT - would just require adding the
>>>> completion helpers there?
>>>>
>>>
>>> Good point, yes I think moving the completion helpers would
>>> allow us to get rid of this file. PDM does not make use of
>>> them / there is atm no 1:1 code in proxmox/ for these helpers.
>>
>> only because https://bugzilla.proxmox.com/show_bug.cgi?id=7179 is not
>> yet implemented ;) so please coordinate with Shan to avoid doing the
>> work twice.
>
> Ah, good catch! thanks for the reference @Fabian.
>
> @Shan: since #7179 will likely touch the same area, it probably makes
> sense to factor out the required helpers as part of this series
> to avoid duplicate work. If that works for you, maybe hold off on
> parallel changes here until this lands. What do you think?
>
>
We discussed it, Shan has already progressed on the CLI side, but
hasn’t integrated the completion helpers yet, so there’s no duplicate
work.
The current plan is:
I’ll move/factor the completion helpers into proxmox/proxmox-acme-api
(preferably as a standalone patch so it can be applied
independently).
Shan will then consume those helpers in PDM as part of #7179, so the
CLI can make use of the completions.
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 12%]
* Re: [pbs-devel] [PATCH proxmox v5 2/4] acme: introduce http_status module
2026-01-14 10:29 6% ` Samuel Rufinatscha
@ 2026-01-15 9:25 5% ` Fabian Grünbichler
0 siblings, 0 replies; 200+ results
From: Fabian Grünbichler @ 2026-01-15 9:25 UTC (permalink / raw)
To: Proxmox Backup Server development discussion, Samuel Rufinatscha
Cc: Thomas Lamprecht
On January 14, 2026 11:29 am, Samuel Rufinatscha wrote:
> On 1/13/26 2:45 PM, Fabian Grünbichler wrote:
>> On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
>>> Introduce an internal http_status module with the common ACME HTTP
>>> response codes, and replace use of crate::request::CREATED as well as
>>> direct numeric status code usages.
>>
>> why not use http::status ? we already have this as dependency pretty
>> much everywhere we do anything HTTP related.. would also for nicer error
>> messages in case the status is not as expected..
>>
>
> http is only pulled in via the optional client / async-client features,
> not the base impl feature. This code here is gated by impl, where http
> might not be available. Adding http as a hard
> dependency just for the few status code constants feels a bit overkill.
> This matches what we discussed in a previous review round:
>
> https://lore.proxmox.com/pbs-devel/2b7574fb-a3c5-4119-8fb6-9649881dba15@proxmox.com/
your patch makes this crate unusable without enabling either client ;)
http is a small low-level crate for exactly this purpose (a common
implementation of common HTTP related types). we already pull it in
everywhere proxmox-acme is used. in fact, I think it would even make
sense to switch over more things to use http here than just the status
code, but that's a different matter/series..
anyway, I guess we can ignore this for now and discuss switching over to
http at a later point as a series on its own.
> Also, since this is pub(crate) API, I think we can easily switch to
> StatusCode later if http ever becomes a necessary dependency for impl.
> OK with you?
>
>>>
>>> Signed-off-by: Samuel Rufinatscha <s.rufinatscha@proxmox.com>
>>> ---
>>> proxmox-acme/src/account.rs | 8 ++++----
>>> proxmox-acme/src/async_client.rs | 4 ++--
>>> proxmox-acme/src/lib.rs | 2 ++
>>> proxmox-acme/src/request.rs | 11 ++++++++++-
>>> 4 files changed, 18 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/proxmox-acme/src/account.rs b/proxmox-acme/src/account.rs
>>> index d8eb3e73..ea1a3c60 100644
>>> --- a/proxmox-acme/src/account.rs
>>> +++ b/proxmox-acme/src/account.rs
>>> @@ -84,7 +84,7 @@ impl Account {
>>> method: "POST",
>>> content_type: crate::request::JSON_CONTENT_TYPE,
>>> body,
>>> - expected: crate::request::CREATED,
>>> + expected: crate::http_status::CREATED,
>>> };
>>>
>>> Ok(NewOrder::new(request))
>>> @@ -106,7 +106,7 @@ impl Account {
>>> method: "POST",
>>> content_type: crate::request::JSON_CONTENT_TYPE,
>>> body,
>>> - expected: 200,
>>> + expected: crate::http_status::OK,
>>> })
>>> }
>>>
>>> @@ -131,7 +131,7 @@ impl Account {
>>> method: "POST",
>>> content_type: crate::request::JSON_CONTENT_TYPE,
>>> body,
>>> - expected: 200,
>>> + expected: crate::http_status::OK,
>>> })
>>> }
>>>
>>> @@ -321,7 +321,7 @@ impl AccountCreator {
>>> method: "POST",
>>> content_type: crate::request::JSON_CONTENT_TYPE,
>>> body,
>>> - expected: crate::request::CREATED,
>>> + expected: crate::http_status::CREATED,
>>> })
>>> }
>>>
>>> diff --git a/proxmox-acme/src/async_client.rs b/proxmox-acme/src/async_client.rs
>>> index 2ff3ba22..043648bb 100644
>>> --- a/proxmox-acme/src/async_client.rs
>>> +++ b/proxmox-acme/src/async_client.rs
>>> @@ -498,7 +498,7 @@ impl AcmeClient {
>>> method: "GET",
>>> content_type: "",
>>> body: String::new(),
>>> - expected: 200,
>>> + expected: crate::http_status::OK,
>>> },
>>> nonce,
>>> )
>>> @@ -550,7 +550,7 @@ impl AcmeClient {
>>> method: "HEAD",
>>> content_type: "",
>>> body: String::new(),
>>> - expected: 200,
>>> + expected: crate::http_status::OK,
>>> },
>>> nonce,
>>> )
>>> diff --git a/proxmox-acme/src/lib.rs b/proxmox-acme/src/lib.rs
>>> index 6722030c..6051a025 100644
>>> --- a/proxmox-acme/src/lib.rs
>>> +++ b/proxmox-acme/src/lib.rs
>>> @@ -70,6 +70,8 @@ pub use order::Order;
>>> #[cfg(feature = "impl")]
>>> pub use order::NewOrder;
>>> #[cfg(feature = "impl")]
>>> +pub(crate) use request::http_status;
>>> +#[cfg(feature = "impl")]
>>> pub use request::ErrorResponse;
>>>
>>> /// Header name for nonces.
>>> diff --git a/proxmox-acme/src/request.rs b/proxmox-acme/src/request.rs
>>> index dadfc5af..341ce53e 100644
>>> --- a/proxmox-acme/src/request.rs
>>> +++ b/proxmox-acme/src/request.rs
>>> @@ -1,7 +1,6 @@
>>> use serde::Deserialize;
>>>
>>> pub(crate) const JSON_CONTENT_TYPE: &str = "application/jose+json";
>>> -pub(crate) const CREATED: u16 = 201;
>>>
>>> /// A request which should be performed on the ACME provider.
>>> pub(crate) struct Request {
>>> @@ -21,6 +20,16 @@ pub(crate) struct Request {
>>> pub(crate) expected: u16,
>>> }
>>>
>>> +/// Common HTTP status codes used in ACME responses.
>>> +pub(crate) mod http_status {
>>> + /// 200 OK
>>> + pub(crate) const OK: u16 = 200;
>>> + /// 201 Created
>>> + pub(crate) const CREATED: u16 = 201;
>>> + /// 204 No Content
>>> + pub(crate) const NO_CONTENT: u16 = 204;
>>> +}
>>> +
>>> /// An ACME error response contains a specially formatted type string, and can optionally
>>> /// contain textual details and a set of sub problems.
>>> #[derive(Clone, Debug, Deserialize)]
>>> --
>>> 2.47.3
>>>
>>>
>>>
>>> _______________________________________________
>>> pbs-devel mailing list
>>> pbs-devel@lists.proxmox.com
>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>>
>>>
>>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>
>
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 5%]
* Re: [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests
2026-01-13 13:48 5% ` [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Fabian Grünbichler
@ 2026-01-15 10:24 0% ` Max R. Carrara
0 siblings, 0 replies; 200+ results
From: Max R. Carrara @ 2026-01-15 10:24 UTC (permalink / raw)
To: Proxmox Backup Server development discussion
On Tue Jan 13, 2026 at 2:48 PM CET, Fabian Grünbichler wrote:
> On January 8, 2026 12:26 pm, Samuel Rufinatscha wrote:
> > Hi,
> >
> > this series fixes account registration for ACME providers that return
> > HTTP 204 No Content to the newNonce request. Currently, both the PBS
> > ACME client and the shared ACME client in proxmox-acme only accept
> > HTTP 200 OK for this request. The issue was observed in PBS against a
> > custom ACME deployment and reported as bug #6939 [1].
>
> sent some feedback for individual patches, one thing to explicitly test
> is that existing accounts and DNS plugin configuration continue to work
> after the switch over - AFAICT from the testing description below that
> was not done (or not noted?).
Ah, to chime in here: I've tested this with a custom DNS plugin config,
see: https://lore.proxmox.com/pbs-devel/DETUA6ZP3M6X.2S90QXW3EYJXU@proxmox.com/
Since a v6 is on its way, I'll re-test everything then; also the case of
configuring the DNS plugin *first* and then switching over to the new
impl.
>
> >
> > ## Problem
> >
> > During ACME account registration, PBS first fetches an anti-replay
> > nonce by sending a HEAD request to the CA’s newNonce URL.
> > RFC 8555 §7.2 [2] states that:
> >
> > * the server MUST include a Replay-Nonce header with a fresh nonce,
> > * the server SHOULD use status 200 OK for the HEAD request,
> > * the server MUST also handle GET on the same resource and may return
> > 204 No Content with an empty body.
> >
> > The reporter observed the following error message:
> >
> > *ACME server responded with unexpected status code: 204*
> >
> > and mentioned that the issue did not appear with PVE 9 [1]. Looking at
> > PVE’s Perl ACME client [3], it uses a GET request instead of HEAD and
> > accepts any 2xx success code when retrieving the nonce. This difference
> > in behavior does not affect functionality but is worth noting for
> > consistency across implementations.
> >
> > ## Approach
> >
> > To support ACME providers which return 204 No Content, the Rust ACME
> > clients in proxmox-backup and proxmox need to treat both 200 OK and 204
> > No Content as valid responses for the nonce request, as long as a
> > Replay-Nonce header is present.
> >
> > This series changes the expected field of the internal Request type
> > from a single u16 to a list of allowed status codes
> > (e.g. &'static [u16]), so one request can explicitly accept multiple
> > success codes.
> >
> > To avoid fixing the issue twice (once in PBS’ own ACME client and once
> > in the shared Rust client), this series first refactors PBS to use the
> > shared AcmeClient from proxmox-acme / proxmox-acme-api, similar to PDM,
> > and then applies the bug fix in that shared implementation so that all
> > consumers benefit from the more tolerant behavior.
> >
> > ## Testing
> >
> > *Testing the refactor*
> >
> > To test the refactor, I
> > (1) installed latest stable PBS on a VM
> > (2) created .deb package from latest PBS (master), containing the
> > refactor
> > (3) installed created .deb package
> > (4) installed Pebble from Let's Encrypt [5] on the same VM
> > (5) created an ACME account and ordered the new certificate for the
> > host domain.
> >
> > Steps to reproduce:
> >
> > (1) install latest stable PBS on a VM, create .deb package from latest
> > PBS (master) containing the refactor, install created .deb package
> > (2) install Pebble from Let's Encrypt [5] on the same VM:
> >
> > cd
> > apt update
> > apt install -y golang git
> > git clone https://github.com/letsencrypt/pebble
> > cd pebble
> > go build ./cmd/pebble
> >
> > then, download and trust the Pebble cert:
> >
> > wget https://raw.githubusercontent.com/letsencrypt/pebble/main/test/certs/pebble.minica.pem
> > cp pebble.minica.pem /usr/local/share/ca-certificates/pebble.minica.crt
> > update-ca-certificates
> >
> > We want Pebble to perform HTTP-01 validation against port 80, because
> > PBS’s standalone plugin will bind port 80. Set httpPort to 80.
> >
> > nano ./test/config/pebble-config.json
> >
> > Start the Pebble server in the background:
> >
> > ./pebble -config ./test/config/pebble-config.json &
> >
> > Create a Pebble ACME account:
> >
> > proxmox-backup-manager acme account register default admin@example.com --directory 'https://127.0.0.1:14000/dir'
> >
> > To verify persistence of the account I checked
> >
> > ls /etc/proxmox-backup/acme/accounts
> >
> > Verified if update-account works
> >
> > proxmox-backup-manager acme account update default --contact "a@example.com,b@example.com"
> > proxmox-backup-manager acme account info default
> >
> > In the PBS GUI, you can create a new domain. You can use your host
> > domain name (see /etc/hosts). Select the created account and order the
> > certificate.
> >
> > After a page reload, you might need to accept the new certificate in the browser.
> > In the PBS dashboard, you should see the new Pebble certificate.
> >
> > *Note: on reboot, the created Pebble ACME account will be gone and you
> > will need to create a new one. Pebble does not persist account info.
> > In that case remove the previously created account in
> > /etc/proxmox-backup/acme/accounts.
> >
> > *Testing the newNonce fix*
> >
> > To prove the ACME newNonce fix, I put nginx in front of Pebble, to
> > intercept the newNonce request in order to return 204 No Content
> > instead of 200 OK, all other requests are unchanged and forwarded to
> > Pebble. Requires trusting the nginx CAs via
> > /usr/local/share/ca-certificates + update-ca-certificates on the VM.
> >
> > Then I ran following command against nginx:
> >
> > proxmox-backup-manager acme account register proxytest root@backup.local --directory 'https://nginx-address/dir
> >
> > The account could be created successfully. When adjusting the nginx
> > configuration to return any other non-expected success status code,
> > PBS rejects as expected.
> >
> > ## Patch summary
> >
> > 0001 – [PATCH proxmox v5 1/4] acme: reduce visibility of Request type
> > Restricts the visibility of the low-level Request type. Consumers
> > should rely on proxmox-acme-api or AcmeClient handlers.
> >
> > 0002– [PATCH proxmox v5 2/4] acme: introduce http_status module
> >
> > 0003 – [PATCH proxmox v5 3/4] fix #6939: acme: support servers
> > returning 204 for nonce requests
> > Adjusts nonce handling to support ACME servers that return HTTP 204
> > (No Content) for new-nonce requests.
> >
> > 0004 – [PATCH proxmox v5 4/4] acme-api: add helper to load client for
> > an account
> > Introduces a helper function to load an ACME client instance for a
> > given account. Required for the following PBS ACME refactor.
> >
> > 0005 – [PATCH proxmox-backup v5 1/5] acme: clean up ACME-related imports
> >
> > 0006 – [PATCH proxmox-backup v5 2/5] acme: include proxmox-acme-api
> > dependency
> > Prepares the codebase to use the factored out ACME API impl.
> >
> > 0007 – [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient
> > Removes the local AcmeClient implementation. Represents the minimal
> > set of changes to replace it with the factored out AcmeClient.
> >
> > 0008 – [PATCH proxmox-backup v5 4/5] acme: change API impls to use
> > proxmox-acme-api handlers
> >
> > 0009 – [PATCH proxmox-backup v5 5/5] acme: certificate ordering through
> > proxmox-acme-api
> >
> > Thanks for considering this patch series, I look forward to your
> > feedback.
> >
> > Best,
> > Samuel Rufinatscha
> >
> > ## Changelog
> >
> > Changes from v4 to v5:
> >
> > * rebased series
> > * re-ordered series (proxmox-acme fix first)
> > * proxmox-backup: cleaned up imports based on an initial clean-up patch
> > * proxmox-acme: removed now unused post_request_raw_payload(),
> > update_account_request(), deactivate_account_request()
> > * proxmox-acme: removed now obsolete/unused get_authorization() and
> > GetAuthorization impl
> >
> > Verified removal by compiling PBS, PDM, and proxmox-perl-rs
> > with all features.
> >
> > Changes from v3 to v4:
> >
> > * add proxmox-acme-api as a dependency and initialize it in
> > PBS so PBS can use the shared ACME API instead.
> > * remove the PBS-local AcmeClient implementation and switch PBS
> > over to the shared proxmox-acme async client.
> > * rework PBS’ ACME API endpoints to delegate to
> > proxmox-acme-api handlers instead of duplicating logic locally.
> > * move PBS’ ACME certificate ordering logic over to
> > proxmox-acme-api, keeping only certificate installation/reload in PBS.
> > * add a load_client_with_account helper in proxmox-acme-api so PBS
> > (and others) can construct an AcmeClient for a configured account
> > without duplicating boilerplate.
> > * hide the low-level Request type and its fields behind constructors
> > / reduced visibility so changes to “expected” no longer affect the
> > public API as they did in v3.
> > * split out the HTTP status constants into an internal http_status
> > module as a separate preparatory cleanup before the bug fix, instead
> > of doing this inline like in v3.
> > * Rebased on top of the refactor: keep the same behavioural fix as in
> > v3 accept 204 for newNonce with Replay-Nonce present), but implement
> > it on top of the http_status module that is part of the refactor.
> >
> > Changes from v2 to v3:
> >
> > * rename `http_success` module to `http_status`
> > * replace `http_success` usage
> > * introduced `http_success` module to contain the http success codes
> > * replaced `Vec<u16>` with `&[u16]` for expected codes to avoid allocations.
> > * clarified the PVEs Perl ACME client behaviour in the commit message.
> > * integrated the `http_success` module, replacing `Vec<u16>` with `&[u16]`
> > * clarified the PVEs Perl ACME client behaviour in the commit message.
> >
> > [1] Bugzilla report #6939:
> > [https://bugzilla.proxmox.com/show_bug.cgi?id=6939](https://bugzilla.proxmox.com/show_bug.cgi?id=6939)
> > [2] RFC 8555 (ACME):
> > [https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2](https://datatracker.ietf.org/doc/html/rfc8555/#section-7.2)
> > [3] PVE’s Perl ACME client (allow 2xx codes for nonce requests):
> > [https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l597)
> > [4] Pebble ACME server:
> > [https://github.com/letsencrypt/pebble](https://github.com/letsencrypt/pebble)
> > [5] Pebble ACME server (perform GET request:
> > [https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219](https://git.proxmox.com/?p=proxmox-acme.git;a=blob;f=src/PVE/ACME.pm;h=f1e9bb7d316e3cea1e376c610b0479119217aecc;hb=HEAD#l219)
> >
> > proxmox:
> >
> > Samuel Rufinatscha (4):
> > acme: reduce visibility of Request type
> > acme: introduce http_status module
> > fix #6939: acme: support servers returning 204 for nonce requests
> > acme-api: add helper to load client for an account
> >
> > proxmox-acme-api/src/account_api_impl.rs | 5 ++
> > proxmox-acme-api/src/lib.rs | 3 +-
> > proxmox-acme/src/account.rs | 102 ++---------------------
> > proxmox-acme/src/async_client.rs | 8 +-
> > proxmox-acme/src/authorization.rs | 30 -------
> > proxmox-acme/src/client.rs | 8 +-
> > proxmox-acme/src/lib.rs | 6 +-
> > proxmox-acme/src/order.rs | 2 +-
> > proxmox-acme/src/request.rs | 25 ++++--
> > 9 files changed, 44 insertions(+), 145 deletions(-)
> >
> >
> > proxmox-backup:
> >
> > Samuel Rufinatscha (5):
> > acme: clean up ACME-related imports
> > acme: include proxmox-acme-api dependency
> > acme: drop local AcmeClient
> > acme: change API impls to use proxmox-acme-api handlers
> > acme: certificate ordering through proxmox-acme-api
> >
> > Cargo.toml | 3 +
> > src/acme/client.rs | 691 -------------------------
> > src/acme/mod.rs | 5 -
> > src/acme/plugin.rs | 336 ------------
> > src/api2/config/acme.rs | 406 ++-------------
> > src/api2/node/certificates.rs | 232 ++-------
> > src/api2/types/acme.rs | 98 ----
> > src/api2/types/mod.rs | 3 -
> > src/bin/proxmox-backup-api.rs | 2 +
> > src/bin/proxmox-backup-manager.rs | 14 +-
> > src/bin/proxmox-backup-proxy.rs | 15 +-
> > src/bin/proxmox_backup_manager/acme.rs | 21 +-
> > src/config/acme/mod.rs | 55 +-
> > src/config/acme/plugin.rs | 92 +---
> > src/config/node.rs | 31 +-
> > src/lib.rs | 2 -
> > 16 files changed, 109 insertions(+), 1897 deletions(-)
> > delete mode 100644 src/acme/client.rs
> > delete mode 100644 src/acme/mod.rs
> > delete mode 100644 src/acme/plugin.rs
> > delete mode 100644 src/api2/types/acme.rs
> >
> >
> > Summary over all repositories:
> > 25 files changed, 153 insertions(+), 2042 deletions(-)
> >
> > --
> > Generated by git-murpp 0.8.1
> >
> >
> > _______________________________________________
> > pbs-devel mailing list
> > pbs-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> >
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [relevance 0%]
Results 1-200 of ~300 next (newer) | reverse | sort options + mbox downloads above
-- links below jump to the message on this page --
2025-10-28 15:21 14% [pbs-devel] [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2025-10-28 15:22 15% ` [pbs-devel] [PATCH proxmox 1/1] " Samuel Rufinatscha
2025-10-29 7:23 5% ` Christian Ebner
2025-10-29 7:53 0% ` Thomas Lamprecht
2025-10-29 8:07 0% ` Christian Ebner
2025-10-29 10:36 0% ` Wolfgang Bumiller
2025-10-29 15:50 6% ` Samuel Rufinatscha
2025-10-29 10:38 5% ` Wolfgang Bumiller
2025-10-29 15:56 6% ` Samuel Rufinatscha
2025-10-28 15:22 16% ` [pbs-devel] [PATCH proxmox-backup 1/1] fix #6939: acme: accept HTTP 204 from newNonce endpoint Samuel Rufinatscha
2025-10-29 7:51 5% ` [pbs-devel] [PATCH proxmox{, -backup} 0/2] fix #6939: acme: support servers returning 204 for nonce requests Thomas Lamprecht
2025-10-29 16:02 6% ` Samuel Rufinatscha
2025-10-29 16:49 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
2025-10-29 16:45 13% [pbs-devel] [PATCH proxmox{, -backup} v2 " Samuel Rufinatscha
2025-10-29 16:45 13% ` [pbs-devel] [PATCH proxmox v2 1/1] " Samuel Rufinatscha
2025-10-31 16:21 5% ` Thomas Lamprecht
2025-11-03 8:51 9% ` Samuel Rufinatscha
2025-10-29 16:45 15% ` [pbs-devel] [PATCH proxmox-backup v2 1/1] fix #6939: acme: accept HTTP 204 from newNonce endpoint Samuel Rufinatscha
2025-11-03 10:21 13% ` [pbs-devel] superseded: [PATCH proxmox{, -backup} v2 0/2] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2025-11-03 10:13 13% [pbs-devel] [PATCH proxmox{, -backup} v3 " Samuel Rufinatscha
2025-11-03 10:13 13% ` [pbs-devel] [PATCH proxmox v3 1/1] " Samuel Rufinatscha
2025-11-04 14:11 4% ` Thomas Lamprecht
2025-11-05 10:22 9% ` Samuel Rufinatscha
2025-11-03 10:13 15% ` [pbs-devel] [PATCH proxmox-backup v3 1/1] fix #6939: acme: accept HTTP 204 from newNonce endpoint Samuel Rufinatscha
2025-12-03 10:23 13% ` [pbs-devel] superseded: [PATCH proxmox{, -backup} v3 0/2] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2025-11-03 16:26 15% [pbs-devel] [PATCH proxmox] fix #6913: auth-api: fix user ID parsing for 2-character realms Samuel Rufinatscha
2025-11-11 10:40 5% ` Fabian Grünbichler
2025-11-11 13:49 6% ` Samuel Rufinatscha
2025-11-14 10:34 5% ` [pbs-devel] applied: " Fabian Grünbichler
2025-11-11 12:29 11% [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path Samuel Rufinatscha
2025-11-11 12:29 12% ` [pbs-devel] [PATCH proxmox-backup 1/3] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
2025-11-12 13:24 4% ` Fabian Grünbichler
2025-11-13 12:59 6% ` Samuel Rufinatscha
2025-11-11 12:29 15% ` [pbs-devel] [PATCH proxmox-backup 2/3] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
2025-11-12 11:24 5% ` Fabian Grünbichler
2025-11-12 15:20 6% ` Samuel Rufinatscha
2025-11-11 12:29 15% ` [pbs-devel] [PATCH proxmox-backup 3/3] datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
2025-11-12 11:27 5% ` [pbs-devel] [PATCH proxmox-backup 0/3] datastore: remove config reload on hot path Fabian Grünbichler
2025-11-12 17:27 6% ` Samuel Rufinatscha
2025-11-14 15:08 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
2025-11-12 13:14 [pbs-devel] [PATCH proxmox-backup] task tracking: fix adding new entry if other PID is tracked Fabian Grünbichler
2025-11-17 8:41 13% ` Samuel Rufinatscha
2025-11-14 15:05 10% [pbs-devel] [PATCH proxmox-backup v2 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
2025-11-14 15:05 17% ` [pbs-devel] [PATCH proxmox-backup v2 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
2025-11-14 15:05 12% ` [pbs-devel] [PATCH proxmox-backup v2 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
2025-11-19 13:24 5% ` Fabian Grünbichler
2025-11-14 15:05 16% ` [pbs-devel] [PATCH proxmox-backup v2 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
2025-11-14 15:05 15% ` [pbs-devel] [PATCH proxmox-backup v2 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
2025-11-19 13:24 5% ` Fabian Grünbichler
2025-11-19 17:25 6% ` Samuel Rufinatscha
2025-11-20 13:07 13% ` [pbs-devel] superseded: [PATCH proxmox-backup v2 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
2025-11-20 13:03 10% [pbs-devel] [PATCH proxmox-backup v3 0/6] " Samuel Rufinatscha
2025-11-20 13:03 17% ` [pbs-devel] [PATCH proxmox-backup v3 1/6] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
2025-11-20 13:03 12% ` [pbs-devel] [PATCH proxmox-backup v3 2/6] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
2025-11-20 13:03 16% ` [pbs-devel] [PATCH proxmox-backup v3 3/6] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
2025-11-20 13:03 15% ` [pbs-devel] [PATCH proxmox-backup v3 4/6] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
2025-11-20 13:03 15% ` [pbs-devel] [PATCH proxmox-backup v3 5/6] partial fix #6049: datastore: add reload flag to config cache helper Samuel Rufinatscha
2025-11-20 14:50 5% ` Fabian Grünbichler
2025-11-20 18:15 6% ` Samuel Rufinatscha
2025-11-20 13:03 15% ` [pbs-devel] [PATCH proxmox-backup v3 6/6] datastore: only bump generation when config digest changes Samuel Rufinatscha
2025-11-20 14:50 5% ` Fabian Grünbichler
2025-11-21 8:37 6% ` Samuel Rufinatscha
2025-11-20 14:50 5% ` [pbs-devel] [PATCH proxmox-backup v3 0/6] datastore: remove config reload on hot path Fabian Grünbichler
2025-11-20 15:17 6% ` Samuel Rufinatscha
2025-11-24 15:35 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
2025-11-24 15:33 12% [pbs-devel] [PATCH proxmox-backup v4 0/4] " Samuel Rufinatscha
2025-11-24 15:33 16% ` [pbs-devel] [PATCH proxmox-backup v4 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
2025-11-24 15:33 11% ` [pbs-devel] [PATCH proxmox-backup v4 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
2025-11-24 15:33 14% ` [pbs-devel] [PATCH proxmox-backup v4 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
2025-11-24 15:33 13% ` [pbs-devel] [PATCH proxmox-backup v4 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
2025-11-24 17:06 13% ` [pbs-devel] superseded: [PATCH proxmox-backup v4 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
2025-11-24 17:04 12% [pbs-devel] [PATCH proxmox-backup v5 " Samuel Rufinatscha
2025-11-24 17:04 16% ` [pbs-devel] [PATCH proxmox-backup v5 1/4] partial fix #6049: config: enable config version cache for datastore Samuel Rufinatscha
2025-11-26 15:15 5% ` Fabian Grünbichler
2025-11-24 17:04 11% ` [pbs-devel] [PATCH proxmox-backup v5 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
2025-11-26 15:15 5% ` Fabian Grünbichler
2025-11-26 17:21 6% ` Samuel Rufinatscha
2025-11-24 17:04 14% ` [pbs-devel] [PATCH proxmox-backup v5 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
2025-11-26 15:15 5% ` Fabian Grünbichler
2025-11-28 9:03 6% ` Samuel Rufinatscha
2025-11-28 10:46 5% ` Fabian Grünbichler
2025-11-28 11:10 6% ` Samuel Rufinatscha
2025-11-24 17:04 14% ` [pbs-devel] [PATCH proxmox-backup v5 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
2025-11-26 15:15 5% ` Fabian Grünbichler
2025-11-26 15:16 5% ` [pbs-devel] [PATCH proxmox-backup v5 0/4] datastore: remove config reload on hot path Fabian Grünbichler
2025-11-26 16:10 6% ` Samuel Rufinatscha
2026-01-05 14:21 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
2025-12-02 15:56 12% [pbs-devel] [PATCH proxmox{-backup, } 0/8] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2025-12-02 15:56 15% ` [pbs-devel] [PATCH proxmox-backup 1/4] acme: include proxmox-acme-api dependency Samuel Rufinatscha
2025-12-02 15:56 6% ` [pbs-devel] [PATCH proxmox-backup 2/4] acme: drop local AcmeClient Samuel Rufinatscha
2025-12-02 15:56 8% ` [pbs-devel] [PATCH proxmox-backup 3/4] acme: change API impls to use proxmox-acme-api handlers Samuel Rufinatscha
2025-12-02 15:56 7% ` [pbs-devel] [PATCH proxmox-backup 4/4] acme: certificate ordering through proxmox-acme-api Samuel Rufinatscha
2025-12-02 15:56 12% ` [pbs-devel] [PATCH proxmox 1/4] acme: reduce visibility of Request type Samuel Rufinatscha
2025-12-02 15:56 15% ` [pbs-devel] [PATCH proxmox 2/4] acme: introduce http_status module Samuel Rufinatscha
2025-12-02 15:56 17% ` [pbs-devel] [PATCH proxmox 3/4] acme-api: add helper to load client for an account Samuel Rufinatscha
2025-12-02 15:56 14% ` [pbs-devel] [PATCH proxmox 4/4] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2025-12-02 16:02 6% ` [pbs-devel] [PATCH proxmox{-backup, } 0/8] " Samuel Rufinatscha
2025-12-03 10:22 11% [pbs-devel] [PATCH proxmox{-backup, } v4 " Samuel Rufinatscha
2025-12-03 10:22 15% ` [pbs-devel] [PATCH proxmox-backup v4 1/4] acme: include proxmox-acme-api dependency Samuel Rufinatscha
2025-12-03 10:22 6% ` [pbs-devel] [PATCH proxmox-backup v4 2/4] acme: drop local AcmeClient Samuel Rufinatscha
2025-12-09 16:50 4% ` Max R. Carrara
2025-12-03 10:22 8% ` [pbs-devel] [PATCH proxmox-backup v4 3/4] acme: change API impls to use proxmox-acme-api handlers Samuel Rufinatscha
2025-12-09 16:50 5% ` Max R. Carrara
2025-12-03 10:22 7% ` [pbs-devel] [PATCH proxmox-backup v4 4/4] acme: certificate ordering through proxmox-acme-api Samuel Rufinatscha
2025-12-09 16:50 5% ` Max R. Carrara
2025-12-03 10:22 17% ` [pbs-devel] [PATCH proxmox v4 1/4] acme-api: add helper to load client for an account Samuel Rufinatscha
2025-12-09 16:51 5% ` Max R. Carrara
2025-12-10 10:08 6% ` Samuel Rufinatscha
2025-12-03 10:22 12% ` [pbs-devel] [PATCH proxmox v4 2/4] acme: reduce visibility of Request type Samuel Rufinatscha
2025-12-09 16:51 5% ` Max R. Carrara
2025-12-03 10:22 15% ` [pbs-devel] [PATCH proxmox v4 3/4] acme: introduce http_status module Samuel Rufinatscha
2025-12-03 10:22 14% ` [pbs-devel] [PATCH proxmox v4 4/4] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2025-12-09 16:50 5% ` [pbs-devel] [PATCH proxmox{-backup, } v4 0/8] " Max R. Carrara
2025-12-10 9:44 6% ` Samuel Rufinatscha
2026-01-08 11:48 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
2025-12-05 13:25 15% [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Samuel Rufinatscha
2025-12-05 13:25 14% ` [pbs-devel] [PATCH proxmox-backup 1/3] pbs-config: cache verified API token secrets Samuel Rufinatscha
2025-12-05 14:04 5% ` Shannon Sterz
2025-12-09 13:29 6% ` Samuel Rufinatscha
2025-12-17 11:16 5% ` Christian Ebner
2025-12-17 11:25 0% ` Shannon Sterz
2025-12-10 11:47 5% ` Fabian Grünbichler
2025-12-10 15:35 6% ` Samuel Rufinatscha
2025-12-15 15:05 12% ` Samuel Rufinatscha
2025-12-15 19:00 12% ` Samuel Rufinatscha
2025-12-16 8:16 5% ` Fabian Grünbichler
2025-12-05 13:25 15% ` [pbs-devel] [PATCH proxmox-backup 2/3] pbs-config: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
2025-12-05 13:25 16% ` [pbs-devel] [PATCH proxmox-backup 3/3] pbs-config: add TTL window to token secret cache Samuel Rufinatscha
2025-12-05 13:25 14% ` [pbs-devel] [PATCH proxmox 1/3] proxmox-access-control: cache verified API token secrets Samuel Rufinatscha
2025-12-05 13:25 15% ` [pbs-devel] [PATCH proxmox 2/3] proxmox-access-control: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
2025-12-05 13:25 16% ` [pbs-devel] [PATCH proxmox 3/3] proxmox-access-control: add TTL window to token secret cache Samuel Rufinatscha
2025-12-05 14:06 5% ` [pbs-devel] [PATCH proxmox{-backup, } 0/6] Reduce token.shadow verification overhead Shannon Sterz
2025-12-09 13:58 6% ` Samuel Rufinatscha
2025-12-17 16:27 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
2025-12-17 16:25 14% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token " Samuel Rufinatscha
2025-12-17 16:25 13% ` [pbs-devel] [PATCH proxmox-backup v2 1/3] pbs-config: cache verified API token secrets Samuel Rufinatscha
2025-12-17 16:25 12% ` [pbs-devel] [PATCH proxmox-backup v2 2/3] pbs-config: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
2025-12-17 16:25 14% ` [pbs-devel] [PATCH proxmox-backup v2 3/3] pbs-config: add TTL window to token secret cache Samuel Rufinatscha
2025-12-17 16:25 13% ` [pbs-devel] [PATCH proxmox v2 1/3] proxmox-access-control: cache verified API token secrets Samuel Rufinatscha
2025-12-17 16:25 12% ` [pbs-devel] [PATCH proxmox v2 2/3] proxmox-access-control: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
2025-12-17 16:25 15% ` [pbs-devel] [PATCH proxmox v2 3/3] proxmox-access-control: add TTL window to token secret cache Samuel Rufinatscha
2025-12-17 16:25 17% ` [pbs-devel] [PATCH proxmox-datacenter-manager v2 1/1] docs: document API token-cache TTL effects Samuel Rufinatscha
2025-12-18 11:03 12% ` [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v2 0/7] token-shadow: reduce api token verification overhead Samuel Rufinatscha
2026-01-02 16:09 13% ` [pbs-devel] superseded: " Samuel Rufinatscha
2026-01-02 16:07 13% [pbs-devel] [PATCH proxmox{-backup, , -datacenter-manager} v3 00/10] " Samuel Rufinatscha
2026-01-02 16:07 17% ` [pbs-devel] [PATCH proxmox-backup v3 1/4] pbs-config: add token.shadow generation to ConfigVersionCache Samuel Rufinatscha
2026-01-14 10:44 5% ` Fabian Grünbichler
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox-backup v3 2/4] pbs-config: cache verified API token secrets Samuel Rufinatscha
2026-01-14 10:44 5% ` Fabian Grünbichler
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox-backup v3 3/4] pbs-config: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
2026-01-14 10:44 5% ` Fabian Grünbichler
2026-01-02 16:07 15% ` [pbs-devel] [PATCH proxmox-backup v3 4/4] pbs-config: add TTL window to token secret cache Samuel Rufinatscha
2026-01-02 16:07 17% ` [pbs-devel] [PATCH proxmox v3 1/4] proxmox-access-control: extend AccessControlConfig for token.shadow invalidation Samuel Rufinatscha
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox v3 2/4] proxmox-access-control: cache verified API token secrets Samuel Rufinatscha
2026-01-02 16:07 12% ` [pbs-devel] [PATCH proxmox v3 3/4] proxmox-access-control: invalidate token-secret cache on token.shadow changes Samuel Rufinatscha
2026-01-02 16:07 15% ` [pbs-devel] [PATCH proxmox v3 4/4] proxmox-access-control: add TTL window to token secret cache Samuel Rufinatscha
2026-01-02 16:07 13% ` [pbs-devel] [PATCH proxmox-datacenter-manager v3 1/2] pdm-config: implement token.shadow generation Samuel Rufinatscha
2026-01-14 10:45 5% ` Fabian Grünbichler
2026-01-02 16:07 17% ` [pbs-devel] [PATCH proxmox-datacenter-manager v3 2/2] docs: document API token-cache TTL effects Samuel Rufinatscha
2026-01-14 10:45 5% ` Fabian Grünbichler
2026-01-14 11:24 6% ` Samuel Rufinatscha
2026-01-05 10:34 [pbs-devel] [PATCH proxmox-backup 1/1] fix: s3: make s3_refresh apihandler sync Nicolas Frey
2026-01-05 15:22 13% ` Samuel Rufinatscha
2026-01-05 14:16 12% [pbs-devel] [PATCH proxmox-backup v6 0/4] datastore: remove config reload on hot path Samuel Rufinatscha
2026-01-05 14:16 16% ` [pbs-devel] [PATCH proxmox-backup v6 1/4] config: enable config version cache for datastore Samuel Rufinatscha
2026-01-05 14:16 11% ` [pbs-devel] [PATCH proxmox-backup v6 2/4] partial fix #6049: datastore: impl ConfigVersionCache fast path for lookups Samuel Rufinatscha
2026-01-05 14:16 15% ` [pbs-devel] [PATCH proxmox-backup v6 3/4] partial fix #6049: datastore: use config fast-path in Drop Samuel Rufinatscha
2026-01-05 14:16 13% ` [pbs-devel] [PATCH proxmox-backup v6 4/4] partial fix #6049: datastore: add TTL fallback to catch manual config edits Samuel Rufinatscha
2026-01-14 9:54 5% ` [pbs-devel] applied-series: [PATCH proxmox-backup v6 0/4] datastore: remove config reload on hot path Fabian Grünbichler
2026-01-07 12:46 6% [pbs-devel] [PATCH proxmox-backup v2 1/1] fix: s3: make s3_refresh apihandler sync Nicolas Frey
2026-01-08 11:26 11% [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2026-01-08 11:26 10% ` [pbs-devel] [PATCH proxmox v5 1/4] acme: reduce visibility of Request type Samuel Rufinatscha
2026-01-13 13:46 5% ` Fabian Grünbichler
2026-01-14 15:07 6% ` Samuel Rufinatscha
2026-01-08 11:26 15% ` [pbs-devel] [PATCH proxmox v5 2/4] acme: introduce http_status module Samuel Rufinatscha
2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-14 10:29 6% ` Samuel Rufinatscha
2026-01-15 9:25 5% ` Fabian Grünbichler
2026-01-08 11:26 14% ` [pbs-devel] [PATCH proxmox v5 3/4] fix #6939: acme: support servers returning 204 for nonce requests Samuel Rufinatscha
2026-01-08 11:26 17% ` [pbs-devel] [PATCH proxmox v5 4/4] acme-api: add helper to load client for an account Samuel Rufinatscha
2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-13 16:57 6% ` Samuel Rufinatscha
2026-01-08 11:26 13% ` [pbs-devel] [PATCH proxmox-backup v5 1/5] acme: clean up ACME-related imports Samuel Rufinatscha
2026-01-13 13:45 5% ` [pbs-devel] applied: " Fabian Grünbichler
2026-01-08 11:26 15% ` [pbs-devel] [PATCH proxmox-backup v5 2/5] acme: include proxmox-acme-api dependency Samuel Rufinatscha
2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-13 16:41 6% ` Samuel Rufinatscha
2026-01-08 11:26 6% ` [pbs-devel] [PATCH proxmox-backup v5 3/5] acme: drop local AcmeClient Samuel Rufinatscha
2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-14 8:56 6% ` Samuel Rufinatscha
2026-01-14 9:58 5% ` Fabian Grünbichler
2026-01-14 10:52 6% ` Samuel Rufinatscha
2026-01-14 16:41 12% ` Samuel Rufinatscha
2026-01-08 11:26 8% ` [pbs-devel] [PATCH proxmox-backup v5 4/5] acme: change API impls to use proxmox-acme-api handlers Samuel Rufinatscha
2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-13 16:53 6% ` Samuel Rufinatscha
2026-01-08 11:26 7% ` [pbs-devel] [PATCH proxmox-backup v5 5/5] acme: certificate ordering through proxmox-acme-api Samuel Rufinatscha
2026-01-13 13:45 5% ` Fabian Grünbichler
2026-01-13 16:51 6% ` Samuel Rufinatscha
2026-01-13 13:48 5% ` [pbs-devel] [PATCH proxmox{, -backup} v5 0/9] fix #6939: acme: support servers returning 204 for nonce requests Fabian Grünbichler
2026-01-15 10:24 0% ` Max R. Carrara
2026-01-08 13:06 [pdm-devel] [PATCH datacenter-manager] fix #7120: remote updates: drop vanished nodes/remotes from cache file Lukas Wagner
2026-01-08 14:38 15% ` Samuel Rufinatscha
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.