* [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications
@ 2026-04-02 10:53 Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 01/17] api: s3: add endpoint to reset s3 request counters Christian Ebner
` (16 more replies)
0 siblings, 17 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
This patch series implements request and traffic counters for the s3
client, being shared as atomic counters via shared memory and mmapping
across all s3-client instances.
The shared counters are instantiated on demand for individual
datastores with s3 backend on s3 client instantiation and stored as
part of the datastore implementation by loading the mmapped file on
datastore creation, being cached for further access. Counter threshold
values can be defined in the datastore options, allowing to send
notifications via the notification system if counters exceeded the set
threshold values. Further, a per-datastore schedule can be configured
to reset the counters with given interval.
Further, counters are being loaded during rrd metrics collection and
exposed as charts in the datastore summary, including the total
request counts per method, the total upload and download traffic as
well as the averaged upload and download rate.
Usage statistics, are not included in this series an will be tackled
as followup series.
Link to the bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=6563
Changes since version 7 (thanks a lot @Thomas for review and testing):
Adapt RRD graphs for S3 request counters to not fill area, have no
transparency and double the line width, making them more accessible
- Drop patch for notification counter threshold rendering, inline to
callside instead
- Display `None` if no notification threshold counters or reset
schedule is configured
Changes since version 6 (thanks a lot @Thomas for review and @Hannes
for review and testing):
- Return last counter values on reset and show them in the scheduled
task log (as suggested off-list by Thomas)
- Add meaningful description/title for task list and task log
- Directly use S3RequestCounterConfig instead of wrapping the options
previously stored in S3RequestCounterOptions in it
- Drop useless Arc for callback fn as it can be shared without
- Fix swapped comments in api types for upload/download traffic
- Rebased onto current master
Changes since version 5 (thanks a lot @Hannes for review and testing):
- Implemented the scheduled reset logic on top of the previous series
- Also upload/download traffic counters are now reset, not just request counters
- Fixed the possible swallowing of errors in `Content`s `Body` implementation
- request_flush() is no longer called by spawning a blocking tokio task
- Fixed the notification callback to not capture the datastore name, but provide
a `label` as callback parameter instead
- Fix the api definition for the reset_counters endpoint
- Fixed incorrect graph labels and typos discovered during review
Changes since version 4:
- Rebase on top of patch series https://lore.proxmox.com/pbs-devel/20260311095906.202410-1-c.ebner@proxmox.com/T/
as requested by Fabian in order to facilitate review flow.
Changes since version 3:
- Extend previous implementation by threshold values for counters and
mechanism to send notifications once thresholds are exceeded
Changes since version 2:
- Introduce msync flushing mechanism to periodically persist counters
state to mmap backing file.
Changes since version 1 (thanks @Robert for feedback):
- Keep tmpfs check in shmem mapping for pre-existing code, add
dedicated methods to create mmapped files in persistent locations.
- Reorder and align atomic counters to 32-byte (half default cache
line size) to reduce false sharing.
- Rework request counter init logic, avoid unsafe code and undefined
behaviour.
- Rework page size calculation based on new counter alignment.
- Avoid the need to open and mmap counter file for each rrd data
collection, keep the per-datastores filehandle cached instead.
proxmox-backup:
Christian Ebner (17):
api: s3: add endpoint to reset s3 request counters
bin: s3: expose request counter reset method as cli command
ui: datastore summary: move store to be part of summary panel
ui: expose s3 request counter statistics in the datastore summary
metrics: collect s3 datastore statistics as rrd metrics
api: admin: expose s3 statistics in datastore rrd data
partially fix #6563: ui: expose s3 rrd charts in datastore summary
datastore: refactor datastore lookup parameters into dedicated type
api: config: update notification thresholds for config and counters
ui: add notification thresholds edit window
notification: define templates and template data for thresholds
datastore: add thresholds notification callback on datastore lookup
api/ui: notifications: add 'thresholds' as notification type value
api: config: allow counter reset schedule editing
ui: expose counter reset schedule edit window
bin: proxy: periodically schedule counter reset task
ui: add task description for scheduled counter reset
debian/proxmox-backup-server.install | 2 +
pbs-datastore/src/datastore.rs | 93 ++++-
pbs-datastore/src/lib.rs | 3 +-
pbs-datastore/src/snapshot_reader.rs | 7 +-
src/api2/admin/datastore.rs | 45 ++-
src/api2/admin/namespace.rs | 9 +-
src/api2/admin/s3.rs | 71 +++-
src/api2/backup/mod.rs | 3 +-
src/api2/config/datastore.rs | 42 ++-
src/api2/config/notifications/mod.rs | 1 +
src/api2/reader/mod.rs | 3 +-
src/api2/status/mod.rs | 6 +-
src/api2/tape/backup.rs | 6 +-
src/api2/tape/restore.rs | 6 +-
src/bin/proxmox-backup-proxy.rs | 89 ++++-
src/bin/proxmox_backup_manager/s3.rs | 33 ++
src/server/metric_collection/metric_server.rs | 8 +-
src/server/metric_collection/mod.rs | 90 ++++-
src/server/metric_collection/pull_metrics.rs | 18 +-
src/server/metric_collection/rrd.rs | 34 +-
src/server/notifications/mod.rs | 39 +-
src/server/notifications/template_data.rs | 30 ++
src/server/prune_job.rs | 3 +-
src/server/pull.rs | 7 +-
src/server/push.rs | 3 +-
src/server/verify_job.rs | 3 +-
src/tools/mod.rs | 14 +-
templates/Makefile | 2 +
.../default/thresholds-exceeded-body.txt.hbs | 9 +
.../thresholds-exceeded-subject.txt.hbs | 1 +
www/Makefile | 2 +
www/Utils.js | 5 +
www/datastore/OptionView.js | 16 +
www/datastore/Summary.js | 344 ++++++++++++------
www/window/CounterResetScheduleEdit.js | 27 ++
www/window/NotificationThresholds.js | 113 ++++++
36 files changed, 996 insertions(+), 191 deletions(-)
create mode 100644 templates/default/thresholds-exceeded-body.txt.hbs
create mode 100644 templates/default/thresholds-exceeded-subject.txt.hbs
create mode 100644 www/window/CounterResetScheduleEdit.js
create mode 100644 www/window/NotificationThresholds.js
Summary over all repositories:
36 files changed, 996 insertions(+), 191 deletions(-)
--
Generated by murpp 0.11.0
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 01/17] api: s3: add endpoint to reset s3 request counters
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 02/17] bin: s3: expose request counter reset method as cli command Christian Ebner
` (15 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Allows to manually reset the current counter states via the api, in
order to provide the same functionality also via a dedicated cli
command. Regular operation will however be to reset the counters
via a scheduled task, as introduced subsequently.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>
Tested-by: Hannes Laimer <h.laimer@proxmox.com>
---
changes since version 7:
- no changes
src/api2/admin/s3.rs | 71 ++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 68 insertions(+), 3 deletions(-)
diff --git a/src/api2/admin/s3.rs b/src/api2/admin/s3.rs
index 1e098b4a1..174e8bfe7 100644
--- a/src/api2/admin/s3.rs
+++ b/src/api2/admin/s3.rs
@@ -1,13 +1,16 @@
//! S3 bucket operations
-use anyhow::{Context, Error};
+use std::path::Path;
+use std::sync::atomic::Ordering;
+
+use anyhow::{bail, Context, Error};
use serde_json::Value;
use proxmox_http::Body;
use proxmox_router::{list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap};
use proxmox_s3_client::{
S3Client, S3ClientConf, S3ClientOptions, S3ObjectKey, S3RequestCounterConfig,
- S3_BUCKET_NAME_SCHEMA, S3_CLIENT_ID_SCHEMA, S3_HTTP_REQUEST_TIMEOUT,
+ SharedRequestCounters, S3_BUCKET_NAME_SCHEMA, S3_CLIENT_ID_SCHEMA, S3_HTTP_REQUEST_TIMEOUT,
};
use proxmox_schema::*;
use proxmox_sortable_macro::sortable;
@@ -96,8 +99,70 @@ pub async fn check(
Ok(Value::Null)
}
+#[api(
+ input: {
+ properties: {
+ "s3-client-id": {
+ schema: S3_CLIENT_ID_SCHEMA,
+ },
+ bucket: {
+ schema: S3_BUCKET_NAME_SCHEMA,
+ },
+ "store-prefix": {
+ type: String,
+ description: "Store prefix within bucket for S3 object keys (commonly datastore name)",
+ optional: true,
+ },
+ },
+ },
+ access: {
+ permission: &Permission::Privilege(&[], PRIV_SYS_MODIFY, false),
+ },
+)]
+/// Reset the S3 request counters for matching endpoint, bucket or datastore (if prefix is given).
+pub async fn reset_counters(
+ s3_client_id: String,
+ bucket: String,
+ store_prefix: Option<String>,
+ _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+ let (config, _digest) = pbs_config::s3::config()?;
+ // only check if the provided endpoint id exists
+ let _config: S3ClientConf = config
+ .lookup(S3_CFG_TYPE_ID, &s3_client_id)
+ .context("config lookup failed")?;
+
+ let request_counter_id = if let Some(store) = &store_prefix {
+ format!("{s3_client_id}-{bucket}-{store}")
+ } else {
+ format!("{s3_client_id}-{bucket}")
+ };
+
+ let path = format!("{S3_CLIENT_REQUEST_COUNTER_BASE_PATH}/{request_counter_id}.shmem");
+ let path = Path::new(&path);
+ // Fail early to not create the file when opening shared memory map below. Accept that
+ // this can race, with a new counter file being created in the mean time, but that is
+ // not an issue.
+ if !path.is_file() {
+ bail!("Cannot find s3 counters file '{path:?}'");
+ }
+
+ let user = pbs_config::backup_user()?;
+ let request_counters = SharedRequestCounters::open_shared_memory_mapped(path, user)
+ .context("failed to open shared request counters")?;
+ request_counters.reset(Ordering::Release);
+
+ Ok(())
+}
+
#[sortable]
-const S3_OPERATION_SUBDIRS: SubdirMap = &[("check", &Router::new().put(&API_METHOD_CHECK))];
+const S3_OPERATION_SUBDIRS: SubdirMap = &[
+ ("check", &Router::new().put(&API_METHOD_CHECK)),
+ (
+ "reset-counters",
+ &Router::new().put(&API_METHOD_RESET_COUNTERS),
+ ),
+];
const S3_OPERATION_ROUTER: Router = Router::new()
.get(&list_subdirs_api_method!(S3_OPERATION_SUBDIRS))
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 02/17] bin: s3: expose request counter reset method as cli command
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 01/17] api: s3: add endpoint to reset s3 request counters Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 03/17] ui: datastore summary: move store to be part of summary panel Christian Ebner
` (14 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Allows to reset the s3 request counters from the cli by calling the
corresponding api method. Place it as a subcommand to `s3 endpoint`
since the endpoint as this should only be allowed for existing
endpoints.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>
Tested-by: Hannes Laimer <h.laimer@proxmox.com>
---
changes since version 7:
- no changes
src/bin/proxmox_backup_manager/s3.rs | 33 ++++++++++++++++++++++++++++
1 file changed, 33 insertions(+)
diff --git a/src/bin/proxmox_backup_manager/s3.rs b/src/bin/proxmox_backup_manager/s3.rs
index a94371e09..64154466f 100644
--- a/src/bin/proxmox_backup_manager/s3.rs
+++ b/src/bin/proxmox_backup_manager/s3.rs
@@ -86,6 +86,33 @@ fn list_s3_clients(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Valu
Ok(Value::Null)
}
+#[api(
+ input: {
+ properties: {
+ "s3-endpoint-id": {
+ schema: S3_CLIENT_ID_SCHEMA,
+ },
+ bucket: {
+ schema: S3_BUCKET_NAME_SCHEMA,
+ },
+ "store-prefix": {
+ type: String,
+ description: "Store prefix within bucket for S3 object keys (commonly datastore name)",
+ optional: true,
+ },
+ },
+ },
+)]
+/// Reset the S3 request counters for matching endpoint, bucket or datastore (if prefix is given).
+async fn reset_counters(
+ s3_endpoint_id: String,
+ bucket: String,
+ store_prefix: Option<String>,
+ rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+ api2::admin::s3::reset_counters(s3_endpoint_id, bucket, store_prefix, rpcenv).await
+}
+
pub fn s3_commands() -> CommandLineInterface {
let endpoint_cmd_def = CliCommandMap::new()
.insert("list", CliCommand::new(&API_METHOD_LIST_S3_CLIENTS))
@@ -111,6 +138,12 @@ pub fn s3_commands() -> CommandLineInterface {
CliCommand::new(&API_METHOD_LIST_BUCKETS)
.arg_param(&["s3-endpoint-id"])
.completion_cb("s3-endpoint-id", pbs_config::s3::complete_s3_client_id),
+ )
+ .insert(
+ "reset-counters",
+ CliCommand::new(&API_METHOD_RESET_COUNTERS)
+ .arg_param(&["s3-endpoint-id", "bucket"])
+ .completion_cb("s3-endpoint-id", pbs_config::s3::complete_s3_client_id),
);
let cmd_def = CliCommandMap::new()
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 03/17] ui: datastore summary: move store to be part of summary panel
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 01/17] api: s3: add endpoint to reset s3 request counters Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 02/17] bin: s3: expose request counter reset method as cli command Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 04/17] ui: expose s3 request counter statistics in the datastore summary Christian Ebner
` (13 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Move the store from the datastore info panel to the parent datastore
summary panel and refactor the store load logic. By this, the same
view model can be reused by all child items, which is required to
show the s3 statistics if present for the datastore, avoiding the
need to perform additional api requests.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>
Tested-by: Hannes Laimer <h.laimer@proxmox.com>
---
changes since version 7:
- no changes
www/datastore/Summary.js | 179 ++++++++++++++++-----------------------
1 file changed, 74 insertions(+), 105 deletions(-)
diff --git a/www/datastore/Summary.js b/www/datastore/Summary.js
index da2653c26..c2b1cedc9 100644
--- a/www/datastore/Summary.js
+++ b/www/datastore/Summary.js
@@ -48,104 +48,6 @@ Ext.define('PBS.DataStoreInfo', {
extend: 'Ext.panel.Panel',
alias: 'widget.pbsDataStoreInfo',
- viewModel: {
- data: {
- countstext: '',
- usage: {},
- stillbad: 0,
- mountpoint: '',
- },
- },
-
- controller: {
- xclass: 'Ext.app.ViewController',
-
- onLoad: function (store, data, success) {
- let me = this;
- if (!success) {
- Proxmox.Utils.API2Request({
- url: `/config/datastore/${me.view.datastore}`,
- success: function (response) {
- let maintenanceString = response.result.data['maintenance-mode'];
- let removable = !!response.result.data['backing-device'];
- if (!maintenanceString && !removable) {
- me.view.el.mask(gettext('Datastore is not available'));
- return;
- }
-
- let [_type, msg] = PBS.Utils.parseMaintenanceMode(maintenanceString);
- let isUnplugged = !maintenanceString && removable;
- let maskMessage = isUnplugged
- ? gettext('Datastore is not mounted')
- : `${gettext('Datastore is in maintenance mode')}${msg ? ': ' + msg : ''}`;
-
- let maskIcon = isUnplugged
- ? 'fa pbs-unplugged-mask'
- : 'fa pbs-maintenance-mask';
- me.view.el.mask(maskMessage, maskIcon);
- },
- });
- return;
- }
- me.view.el.unmask();
-
- let vm = me.getViewModel();
-
- let counts = store.getById('counts').data.value;
- let used = store.getById('used').data.value;
- let total = store.getById('avail').data.value + used;
- let backendType = store.getById('backend-type').data.value;
- if (backendType === 's3') {
- me.lookup('usage').title = gettext('Local Cache Usage');
- }
-
- let usage = Proxmox.Utils.render_size_usage(used, total, true);
- vm.set('usagetext', usage);
- vm.set('usage', used / total);
-
- let countstext = function (count) {
- count = count || {};
- return `${count.groups || 0} ${gettext('Groups')}, ${count.snapshots || 0} ${gettext('Snapshots')}`;
- };
- let gcstatus = store.getById('gc-status')?.data.value;
- if (gcstatus) {
- let dedup = PBS.Utils.calculate_dedup_factor(gcstatus);
- vm.set('deduplication', dedup.toFixed(2));
- vm.set('stillbad', gcstatus['still-bad']);
- }
-
- vm.set('ctcount', countstext(counts.ct));
- vm.set('vmcount', countstext(counts.vm));
- vm.set('hostcount', countstext(counts.host));
- },
-
- startStore: function () {
- this.store.startUpdate();
- },
- stopStore: function () {
- this.store.stopUpdate();
- },
- doSingleStoreLoad: function () {
- this.store.load();
- },
-
- init: function (view) {
- let me = this;
- let datastore = encodeURIComponent(view.datastore);
- me.store = Ext.create('Proxmox.data.ObjectStore', {
- interval: 5 * 1000,
- url: `/api2/json/admin/datastore/${datastore}/status/?verbose=true`,
- });
- me.store.on('load', me.onLoad, me);
- },
- },
-
- listeners: {
- activate: 'startStore',
- beforedestroy: 'stopStore',
- deactivate: 'stopStore',
- },
-
defaults: {
xtype: 'pmxInfoWidget',
},
@@ -242,6 +144,15 @@ Ext.define('PBS.DataStoreSummary', {
padding: 5,
},
+ viewModel: {
+ data: {
+ countstext: '',
+ usage: {},
+ stillbad: 0,
+ mountpoint: '',
+ },
+ },
+
tbar: [
{
xtype: 'button',
@@ -371,16 +282,19 @@ Ext.define('PBS.DataStoreSummary', {
listeners: {
activate: function () {
this.rrdstore.startUpdate();
+ this.infoStore.startUpdate();
},
afterrender: function () {
this.statusStore.startUpdate();
},
deactivate: function () {
this.rrdstore.stopUpdate();
+ this.infoStore.stopUpdate();
},
destroy: function () {
this.rrdstore.stopUpdate();
this.statusStore.stopUpdate();
+ this.infoStore.stopUpdate();
},
resize: function (panel) {
Proxmox.Utils.updateColumns(panel);
@@ -400,6 +314,11 @@ Ext.define('PBS.DataStoreSummary', {
interval: 1000,
});
+ me.infoStore = Ext.create('Proxmox.data.ObjectStore', {
+ interval: 5 * 1000,
+ url: `/api2/json/admin/datastore/${me.datastore}/status/?verbose=true`,
+ });
+
let lastRequestFailed = false;
me.mon(me.statusStore, 'load', (s, records, success) => {
let mountBtn = me.lookupReferenceHolder().lookupReference('mountButton');
@@ -409,10 +328,8 @@ Ext.define('PBS.DataStoreSummary', {
me.statusStore.stopUpdate();
me.rrdstore.stopUpdate();
-
- let infoPanelController = me.down('pbsDataStoreInfo').getController();
- infoPanelController.stopStore();
- infoPanelController.doSingleStoreLoad();
+ me.infoStore.stopUpdate();
+ me.infoStore.load();
Proxmox.Utils.API2Request({
url: `/config/datastore/${me.datastore}`,
@@ -437,7 +354,7 @@ Ext.define('PBS.DataStoreSummary', {
} else {
// only trigger on edges, else we couple our interval to the info one
if (lastRequestFailed) {
- me.down('pbsDataStoreInfo').fireEvent('activate');
+ me.infoStore.startUpdate();
me.rrdstore.startUpdate();
}
unmountBtn.setDisabled(false);
@@ -499,6 +416,60 @@ Ext.define('PBS.DataStoreSummary', {
},
});
+ me.mon(me.infoStore, 'load', (store, records, success) => {
+ if (!success) {
+ Proxmox.Utils.API2Request({
+ url: `/config/datastore/${me.datastore}`,
+ success: function (response) {
+ let maintenanceString = response.result.data['maintenance-mode'];
+ let removable = !!response.result.data['backing-device'];
+ if (!maintenanceString && !removable) {
+ me.down('pbsDataStoreInfo').mask(gettext('Datastore is not available'));
+ return;
+ }
+
+ let [_type, msg] = PBS.Utils.parseMaintenanceMode(maintenanceString);
+ let isUnplugged = !maintenanceString && removable;
+ let maskMessage = isUnplugged
+ ? gettext('Datastore is not mounted')
+ : `${gettext('Datastore is in maintenance mode')}${msg ? ': ' + msg : ''}`;
+
+ let maskIcon = isUnplugged
+ ? 'fa pbs-unplugged-mask'
+ : 'fa pbs-maintenance-mask';
+ me.down('pbsDataStoreInfo').mask(maskMessage, maskIcon);
+ },
+ });
+ return;
+ }
+ me.down('pbsDataStoreInfo').unmask();
+
+ let vm = me.getViewModel();
+
+ let counts = store.getById('counts').data.value;
+ let used = store.getById('used').data.value;
+ let total = store.getById('avail').data.value + used;
+
+ let usage = Proxmox.Utils.render_size_usage(used, total, true);
+ vm.set('usagetext', usage);
+ vm.set('usage', used / total);
+
+ let countstext = function (count) {
+ count = count || {};
+ return `${count.groups || 0} ${gettext('Groups')}, ${count.snapshots || 0} ${gettext('Snapshots')}`;
+ };
+ let gcstatus = store.getById('gc-status')?.data.value;
+ if (gcstatus) {
+ let dedup = PBS.Utils.calculate_dedup_factor(gcstatus);
+ vm.set('deduplication', dedup.toFixed(2));
+ vm.set('stillbad', gcstatus['still-bad']);
+ }
+
+ vm.set('ctcount', countstext(counts.ct));
+ vm.set('vmcount', countstext(counts.vm));
+ vm.set('hostcount', countstext(counts.host));
+ });
+
me.mon(
me.rrdstore,
'load',
@@ -513,7 +484,5 @@ Ext.define('PBS.DataStoreSummary', {
me.query('proxmoxRRDChart').forEach((chart) => {
chart.setStore(me.rrdstore);
});
-
- me.down('pbsDataStoreInfo').relayEvents(me, ['activate', 'deactivate']);
},
});
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 04/17] ui: expose s3 request counter statistics in the datastore summary
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (2 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 03/17] ui: datastore summary: move store to be part of summary panel Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 05/17] metrics: collect s3 datastore statistics as rrd metrics Christian Ebner
` (12 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Show the current s3 request counter statistics for datastore backend.
Use a dedicated info widget, only shown for s3 datastores.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>
Tested-by: Hannes Laimer <h.laimer@proxmox.com>
---
changes since version 7:
- no changes
www/datastore/Summary.js | 110 +++++++++++++++++++++++++++++++++++++++
1 file changed, 110 insertions(+)
diff --git a/www/datastore/Summary.js b/www/datastore/Summary.js
index c2b1cedc9..5dcc0e5dc 100644
--- a/www/datastore/Summary.js
+++ b/www/datastore/Summary.js
@@ -130,6 +130,95 @@ Ext.define('PBS.DataStoreInfo', {
],
});
+Ext.define('PBS.DataStoreS3Stats', {
+ extend: 'Ext.panel.Panel',
+ alias: 'widget.pbsDataStoreS3Stats',
+
+ defaults: {
+ xtype: 'pmxInfoWidget',
+ },
+
+ bodyPadding: 20,
+
+ items: [
+ {
+ xtype: 'box',
+ html: `<b>${gettext('S3 traffic:')}</b>`,
+ padding: '10 0 5 0',
+ },
+ {
+ iconCls: 'fa fa-fw fa-arrow-up',
+ title: gettext('Data uploaded'),
+ printBar: false,
+ bind: {
+ data: {
+ text: '{uploaded}',
+ },
+ },
+ },
+ {
+ iconCls: 'fa fa-fw fa-arrow-down',
+ title: gettext('Data downloaded'),
+ printBar: false,
+ bind: {
+ data: {
+ text: '{downloaded}',
+ },
+ },
+ },
+ {
+ xtype: 'box',
+ html: `<b>${gettext('S3 requests:')}</b>`,
+ padding: '10 0 5 0',
+ },
+ {
+ title: gettext('GET'),
+ printBar: false,
+ bind: {
+ data: {
+ text: '{get}',
+ },
+ },
+ },
+ {
+ title: gettext('PUT'),
+ printBar: false,
+ bind: {
+ data: {
+ text: '{put}',
+ },
+ },
+ },
+ {
+ title: gettext('POST'),
+ printBar: false,
+ bind: {
+ data: {
+ text: '{post}',
+ },
+ },
+ },
+ {
+ title: gettext('HEAD'),
+ printBar: false,
+ bind: {
+ data: {
+ text: '{head}',
+ },
+ },
+ },
+ {
+ title: gettext('DELETE'),
+ printBar: false,
+ bind: {
+ data: {
+ text: '{delete}',
+ },
+ },
+ },
+ ],
+});
+
Ext.define('PBS.DataStoreSummary', {
extend: 'Ext.panel.Panel',
alias: 'widget.pbsDataStoreSummary',
@@ -150,6 +239,7 @@ Ext.define('PBS.DataStoreSummary', {
usage: {},
stillbad: 0,
mountpoint: '',
+ showS3Stats: false,
},
},
@@ -244,10 +334,19 @@ Ext.define('PBS.DataStoreSummary', {
{
xtype: 'pbsDataStoreNotes',
flex: 1,
+ padding: '0 10 0 0',
cbind: {
datastore: '{datastore}',
},
},
+ {
+ xtype: 'pbsDataStoreS3Stats',
+ flex: 1,
+ title: gettext('S3 statistics'),
+ bind: {
+ visible: '{showS3Stats}',
+ },
+ },
],
},
{
@@ -464,6 +563,17 @@ Ext.define('PBS.DataStoreSummary', {
vm.set('deduplication', dedup.toFixed(2));
vm.set('stillbad', gcstatus['still-bad']);
}
+ let s3Stats = store.getById('s3-statistics')?.data.value;
+ if (s3Stats) {
+ vm.set('uploaded', Proxmox.Utils.format_size(s3Stats.uploaded));
+ vm.set('downloaded', Proxmox.Utils.format_size(s3Stats.downloaded));
+ vm.set('get', s3Stats.get);
+ vm.set('post', s3Stats.post);
+ vm.set('delete', s3Stats.delete);
+ vm.set('head', s3Stats.head);
+ vm.set('put', s3Stats.put);
+ vm.set('showS3Stats', true);
+ }
vm.set('ctcount', countstext(counts.ct));
vm.set('vmcount', countstext(counts.vm));
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 05/17] metrics: collect s3 datastore statistics as rrd metrics
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (3 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 04/17] ui: expose s3 request counter statistics in the datastore summary Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 06/17] api: admin: expose s3 statistics in datastore rrd data Christian Ebner
` (11 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
For datastores with s3 backend, load the shared s3 request counters
via the mmapped file (avoiding to have to do a full datastore lookup)
and include them as rrd metrics.
Combine the pre-existing DiskStat with an optional S3Statistics into
a common DatastoreStats struct as dedicated type for the internal
method interfaces.
Request counters are collected by method, total upload and download
traffic as gauge values as well as derived values to get averaged
rate statistics.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>
Tested-by: Hannes Laimer <h.laimer@proxmox.com>
---
changes since version 7:
- no changes
src/server/metric_collection/metric_server.rs | 8 +-
src/server/metric_collection/mod.rs | 90 +++++++++++++++++--
src/server/metric_collection/pull_metrics.rs | 18 +++-
src/server/metric_collection/rrd.rs | 34 ++++++-
4 files changed, 132 insertions(+), 18 deletions(-)
diff --git a/src/server/metric_collection/metric_server.rs b/src/server/metric_collection/metric_server.rs
index ba20628a0..4584fc14c 100644
--- a/src/server/metric_collection/metric_server.rs
+++ b/src/server/metric_collection/metric_server.rs
@@ -5,10 +5,10 @@ use serde_json::{json, Value};
use proxmox_metrics::MetricsData;
-use super::{DiskStat, HostStats};
+use super::{DatastoreStats, DiskStat, HostStats};
pub async fn send_data_to_metric_servers(
- stats: Arc<(HostStats, DiskStat, Vec<DiskStat>)>,
+ stats: Arc<(HostStats, DiskStat, Vec<DatastoreStats>)>,
) -> Result<(), Error> {
let (config, _digest) = pbs_config::metrics::config()?;
let channel_list = get_metric_server_connections(config)?;
@@ -66,10 +66,10 @@ pub async fn send_data_to_metric_servers(
for datastore in stats.2.iter() {
values.push(Arc::new(
- MetricsData::new("blockstat", ctime, datastore.to_value())?
+ MetricsData::new("blockstat", ctime, datastore.disk.to_value())?
.tag("object", "host")
.tag("host", nodename)
- .tag("datastore", datastore.name.clone()),
+ .tag("datastore", datastore.disk.name.clone()),
));
}
diff --git a/src/server/metric_collection/mod.rs b/src/server/metric_collection/mod.rs
index e2df7fc0a..840dc2f58 100644
--- a/src/server/metric_collection/mod.rs
+++ b/src/server/metric_collection/mod.rs
@@ -1,18 +1,27 @@
+use std::collections::hash_map::Entry;
use std::collections::HashMap;
use std::path::Path;
use std::pin::pin;
-use std::sync::{Arc, OnceLock};
+use std::sync::atomic::Ordering;
+use std::sync::{Arc, LazyLock, Mutex, OnceLock};
use std::time::{Duration, Instant};
-use anyhow::Error;
+use anyhow::{format_err, Error};
+use hyper::Method;
use tokio::join;
-use pbs_api_types::{DataStoreConfig, Operation};
+use pbs_api_types::{
+ DataStoreConfig, DatastoreBackendConfig, DatastoreBackendType, Operation, S3Statistics,
+};
+use proxmox_lang::try_block;
use proxmox_network_api::{get_network_interfaces, IpLink};
+use proxmox_s3_client::SharedRequestCounters;
+use proxmox_schema::ApiType;
use proxmox_sys::fs::FileSystemInformation;
use proxmox_sys::linux::procfs::{Loadavg, ProcFsMemInfo, ProcFsNetDev, ProcFsStat};
use crate::tools::disks::{zfs_dataset_stats, BlockDevStat, DiskManage};
+use pbs_datastore::S3_CLIENT_REQUEST_COUNTER_BASE_PATH;
mod metric_server;
pub(crate) mod pull_metrics;
@@ -109,6 +118,11 @@ struct DiskStat {
dev: Option<BlockDevStat>,
}
+struct DatastoreStats {
+ disk: DiskStat,
+ s3_stats: Option<S3Statistics>,
+}
+
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, serde::Serialize, serde::Deserialize)]
enum NetdevType {
Physical,
@@ -215,7 +229,52 @@ fn collect_host_stats_sync() -> HostStats {
}
}
-fn collect_disk_stats_sync() -> (DiskStat, Vec<DiskStat>) {
+static S3_REQUEST_COUNTERS_MAP: LazyLock<Mutex<HashMap<String, SharedRequestCounters>>> =
+ LazyLock::new(|| Mutex::new(HashMap::new()));
+
+fn collect_s3_stats(
+ store: &str,
+ backend_config: &DatastoreBackendConfig,
+) -> Result<Option<S3Statistics>, Error> {
+ let endpoint_id = backend_config
+ .client
+ .as_ref()
+ .ok_or(format_err!("missing s3 endpoint id"))?;
+ let bucket = backend_config
+ .bucket
+ .as_ref()
+ .ok_or(format_err!("missing s3 bucket name"))?;
+ let path =
+ format!("{S3_CLIENT_REQUEST_COUNTER_BASE_PATH}/{endpoint_id}-{bucket}-{store}.shmem");
+
+ let mut counters = S3_REQUEST_COUNTERS_MAP.lock().unwrap();
+ let s3_stats = match counters.entry(path.clone()) {
+ Entry::Occupied(o) => load_s3_statistics(o.get()),
+ Entry::Vacant(v) => {
+ let user = pbs_config::backup_user()?;
+ let counters = SharedRequestCounters::open_shared_memory_mapped(path, user)?;
+ let s3_stats = load_s3_statistics(&counters);
+ v.insert(counters);
+ s3_stats
+ }
+ };
+
+ Ok(Some(s3_stats))
+}
+
+fn load_s3_statistics(counters: &SharedRequestCounters) -> S3Statistics {
+ S3Statistics {
+ get: counters.load(Method::GET, Ordering::Acquire),
+ put: counters.load(Method::PUT, Ordering::Acquire),
+ post: counters.load(Method::POST, Ordering::Acquire),
+ delete: counters.load(Method::DELETE, Ordering::Acquire),
+ head: counters.load(Method::HEAD, Ordering::Acquire),
+ uploaded: counters.get_upload_traffic(Ordering::Acquire),
+ downloaded: counters.get_download_traffic(Ordering::Acquire),
+ }
+}
+
+fn collect_disk_stats_sync() -> (DiskStat, Vec<DatastoreStats>) {
let disk_manager = DiskManage::new();
let root = gather_disk_stats(disk_manager.clone(), Path::new("/"), "host");
@@ -239,11 +298,30 @@ fn collect_disk_stats_sync() -> (DiskStat, Vec<DiskStat>) {
continue;
}
- datastores.push(gather_disk_stats(
+ let s3_stats: Option<S3Statistics> = try_block!({
+ let backend_config: DatastoreBackendConfig = serde_json::from_value(
+ DatastoreBackendConfig::API_SCHEMA
+ .parse_property_string(config.backend.as_deref().unwrap_or(""))?,
+ )?;
+
+ if backend_config.ty.unwrap_or_default() == DatastoreBackendType::S3 {
+ collect_s3_stats(&config.name, &backend_config)
+ } else {
+ Ok(None)
+ }
+ })
+ .unwrap_or_else(|err: Error| {
+ eprintln!("parsing datastore backend config failed - {err}");
+ None
+ });
+
+ let disk = gather_disk_stats(
disk_manager.clone(),
Path::new(&config.absolute_path()),
&config.name,
- ));
+ );
+
+ datastores.push(DatastoreStats { disk, s3_stats });
}
}
Err(err) => {
diff --git a/src/server/metric_collection/pull_metrics.rs b/src/server/metric_collection/pull_metrics.rs
index e99662faf..4dcd336a5 100644
--- a/src/server/metric_collection/pull_metrics.rs
+++ b/src/server/metric_collection/pull_metrics.rs
@@ -6,13 +6,14 @@ use nix::sys::stat::Mode;
use pbs_api_types::{
MetricDataPoint,
MetricDataType::{self, Derive, Gauge},
+ S3Statistics,
};
use pbs_buildcfg::PROXMOX_BACKUP_RUN_DIR;
use proxmox_shared_cache::SharedCache;
use proxmox_sys::fs::CreateOptions;
use serde::{Deserialize, Serialize};
-use super::{DiskStat, HostStats, NetdevType, METRIC_COLLECTION_INTERVAL};
+use super::{DatastoreStats, DiskStat, HostStats, NetdevType, METRIC_COLLECTION_INTERVAL};
const METRIC_CACHE_TIME: Duration = Duration::from_secs(30 * 60);
const STORED_METRIC_GENERATIONS: u64 =
@@ -89,7 +90,7 @@ pub fn get_all_metrics(start_time: i64) -> Result<Vec<MetricDataPoint>, Error> {
pub(super) fn update_metrics(
host: &HostStats,
hostdisk: &DiskStat,
- datastores: &[DiskStat],
+ datastores: &[DatastoreStats],
) -> Result<(), Error> {
let mut points = MetricDataPoints::new(proxmox_time::epoch_i64());
@@ -129,8 +130,11 @@ pub(super) fn update_metrics(
update_disk_metrics(&mut points, hostdisk, "host");
for stat in datastores {
- let id = format!("datastore/{}", stat.name);
- update_disk_metrics(&mut points, stat, &id);
+ let id = format!("datastore/{}", stat.disk.name);
+ update_disk_metrics(&mut points, &stat.disk, &id);
+ if let Some(stat) = &stat.s3_stats {
+ update_s3_metrics(&mut points, stat, &id);
+ }
}
get_cache()?.set(&points, Duration::from_secs(2))?;
@@ -158,6 +162,12 @@ fn update_disk_metrics(points: &mut MetricDataPoints, disk: &DiskStat, id: &str)
}
}
+fn update_s3_metrics(points: &mut MetricDataPoints, stat: &S3Statistics, id: &str) {
+ let id = format!("{id}/s3");
+ points.add(Gauge, &id, "uploaded", stat.uploaded as f64);
+ points.add(Gauge, &id, "downloaded", stat.downloaded as f64);
+}
+
#[derive(Serialize, Deserialize)]
struct MetricDataPoints {
timestamp: i64,
diff --git a/src/server/metric_collection/rrd.rs b/src/server/metric_collection/rrd.rs
index 7b13b51ff..a0ea1a566 100644
--- a/src/server/metric_collection/rrd.rs
+++ b/src/server/metric_collection/rrd.rs
@@ -13,10 +13,11 @@ use proxmox_rrd::rrd::{AggregationFn, Archive, DataSourceType, Database};
use proxmox_rrd::Cache;
use proxmox_sys::fs::CreateOptions;
+use pbs_api_types::S3Statistics;
use pbs_buildcfg::PROXMOX_BACKUP_STATE_DIR_M;
use proxmox_rrd_api_types::{RrdMode, RrdTimeframe};
-use super::{DiskStat, HostStats, NetdevType};
+use super::{DatastoreStats, DiskStat, HostStats, NetdevType};
const RRD_CACHE_BASEDIR: &str = concat!(PROXMOX_BACKUP_STATE_DIR_M!(), "/rrdb");
@@ -148,7 +149,7 @@ fn update_derive(name: &str, value: f64) {
}
}
-pub(super) fn update_metrics(host: &HostStats, hostdisk: &DiskStat, datastores: &[DiskStat]) {
+pub(super) fn update_metrics(host: &HostStats, hostdisk: &DiskStat, datastores: &[DatastoreStats]) {
if let Some(stat) = &host.proc {
update_gauge("host/cpu", stat.cpu);
update_gauge("host/iowait", stat.iowait_percent);
@@ -182,8 +183,11 @@ pub(super) fn update_metrics(host: &HostStats, hostdisk: &DiskStat, datastores:
update_disk_metrics(hostdisk, "host");
for stat in datastores {
- let rrd_prefix = format!("datastore/{}", stat.name);
- update_disk_metrics(stat, &rrd_prefix);
+ let rrd_prefix = format!("datastore/{}", stat.disk.name);
+ update_disk_metrics(&stat.disk, &rrd_prefix);
+ if let Some(stats) = &stat.s3_stats {
+ update_s3_metrics(stats, &rrd_prefix);
+ }
}
}
@@ -212,3 +216,25 @@ fn update_disk_metrics(disk: &DiskStat, rrd_prefix: &str) {
update_derive(&rrd_key, (stat.io_ticks as f64) / 1000.0);
}
}
+
+fn update_s3_metrics(stats: &S3Statistics, rrd_prefix: &str) {
+ let rrd_key = format!("{rrd_prefix}/s3/total/uploaded");
+ update_gauge(&rrd_key, stats.uploaded as f64);
+ let rrd_key = format!("{rrd_prefix}/s3/total/downloaded");
+ update_gauge(&rrd_key, stats.downloaded as f64);
+ let rrd_key = format!("{rrd_prefix}/s3/uploaded");
+ update_derive(&rrd_key, stats.uploaded as f64);
+ let rrd_key = format!("{rrd_prefix}/s3/downloaded");
+ update_derive(&rrd_key, stats.downloaded as f64);
+
+ let rrd_key = format!("{rrd_prefix}/s3/total/get");
+ update_gauge(&rrd_key, stats.get as f64);
+ let rrd_key = format!("{rrd_prefix}/s3/total/put");
+ update_gauge(&rrd_key, stats.put as f64);
+ let rrd_key = format!("{rrd_prefix}/s3/total/post");
+ update_gauge(&rrd_key, stats.post as f64);
+ let rrd_key = format!("{rrd_prefix}/s3/total/head");
+ update_gauge(&rrd_key, stats.head as f64);
+ let rrd_key = format!("{rrd_prefix}/s3/total/delete");
+ update_gauge(&rrd_key, stats.delete as f64);
+}
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 06/17] api: admin: expose s3 statistics in datastore rrd data
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (4 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 05/17] metrics: collect s3 datastore statistics as rrd metrics Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 07/17] partially fix #6563: ui: expose s3 rrd charts in datastore summary Christian Ebner
` (10 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Includes the additional s3 related rrd data metrics in the api
response to expose them in the ui.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>
Tested-by: Hannes Laimer <h.laimer@proxmox.com>
---
changes since version 7:
- no changes
src/api2/admin/datastore.rs | 19 +++++++++++++++----
1 file changed, 15 insertions(+), 4 deletions(-)
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 0033d6cf8..44a7d2bda 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -39,10 +39,10 @@ use pbs_api_types::{
print_ns_and_snapshot, print_store_and_ns, ArchiveType, Authid, BackupArchiveName,
BackupContent, BackupGroupDeleteStats, BackupNamespace, BackupType, Counts, CryptMode,
DataStoreConfig, DataStoreListItem, DataStoreMountStatus, DataStoreStatus,
- GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
- MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SyncJobConfig,
- BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
- BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
+ DatastoreBackendType, GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus,
+ KeepOptions, MaintenanceMode, MaintenanceType, Operation, PruneJobOptions, SnapshotListItem,
+ SyncJobConfig, BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA,
+ BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
IGNORE_VERIFIED_BACKUPS_SCHEMA, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA, PRIV_DATASTORE_AUDIT,
PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE, PRIV_DATASTORE_READ,
PRIV_DATASTORE_VERIFY, PRIV_SYS_MODIFY, UPID, UPID_SCHEMA, VERIFICATION_OUTDATED_AFTER_SCHEMA,
@@ -1898,6 +1898,17 @@ pub fn get_rrd_stats(
Ok(Some((fs_type, _, _))) if fs_type.as_str() == "zfs" => {}
_ => rrd_fields.push("io_ticks"),
};
+ if datastore.backend_type() == DatastoreBackendType::S3 {
+ rrd_fields.push("s3/uploaded");
+ rrd_fields.push("s3/downloaded");
+ rrd_fields.push("s3/total/uploaded");
+ rrd_fields.push("s3/total/downloaded");
+ rrd_fields.push("s3/total/get");
+ rrd_fields.push("s3/total/put");
+ rrd_fields.push("s3/total/post");
+ rrd_fields.push("s3/total/head");
+ rrd_fields.push("s3/total/delete");
+ }
create_value_from_rrd(&format!("datastore/{store}"), &rrd_fields, timeframe, cf)
}
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 07/17] partially fix #6563: ui: expose s3 rrd charts in datastore summary
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (5 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 06/17] api: admin: expose s3 statistics in datastore rrd data Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 08/17] datastore: refactor datastore lookup parameters into dedicated type Christian Ebner
` (9 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Show the total request counts per method as well as the total and
derived upload/download s3 api statistics as rrd charts.
This partially fixes issue 6563, further information such as usage
statistics for the s3 backend will be implemented.
Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=6563
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 7:
- use seriesConfig on request counter graphs to adapt their style and
area fill configuration
www/datastore/Summary.js | 55 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 55 insertions(+)
diff --git a/www/datastore/Summary.js b/www/datastore/Summary.js
index 5dcc0e5dc..2fdb25812 100644
--- a/www/datastore/Summary.js
+++ b/www/datastore/Summary.js
@@ -22,6 +22,15 @@ Ext.define('pve-rrd-datastore', {
'write_ios',
'write_bytes',
'io_ticks',
+ 's3/uploaded',
+ 's3/downloaded',
+ 's3/total/uploaded',
+ 's3/total/downloaded',
+ 's3/total/get',
+ 's3/total/put',
+ 's3/total/post',
+ 's3/total/head',
+ 's3/total/delete',
{
name: 'io_delay',
calculate: function (data) {
@@ -349,6 +358,52 @@ Ext.define('PBS.DataStoreSummary', {
},
],
},
+ {
+ xtype: 'proxmoxRRDChart',
+ title: gettext('S3 API requests'),
+ fields: [
+ 's3/total/get',
+ 's3/total/put',
+ 's3/total/post',
+ 's3/total/head',
+ 's3/total/delete',
+ ],
+ fieldTitles: [
+ gettext('GET'),
+ gettext('PUT'),
+ gettext('POST'),
+ gettext('HEAD'),
+ gettext('DELETE'),
+ ],
+ bind: {
+ visible: '{showS3Stats}',
+ },
+ seriesConfig: {
+ fill: false,
+ style: {
+ lineWidth: 3.0,
+ opacity: 1.0,
+ },
+ },
+ },
+ {
+ xtype: 'proxmoxRRDChart',
+ title: gettext('S3 API download/upload rate (bytes/second)'),
+ fields: ['s3/downloaded', 's3/uploaded'],
+ fieldTitles: [gettext('Download'), gettext('Upload')],
+ bind: {
+ visible: '{showS3Stats}',
+ },
+ },
+ {
+ xtype: 'proxmoxRRDChart',
+ title: gettext('S3 API total download/upload (bytes)'),
+ fields: ['s3/total/downloaded', 's3/total/uploaded'],
+ fieldTitles: [gettext('Download'), gettext('Upload')],
+ bind: {
+ visible: '{showS3Stats}',
+ },
+ },
{
xtype: 'proxmoxRRDChart',
title: gettext('Storage usage (bytes)'),
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 08/17] datastore: refactor datastore lookup parameters into dedicated type
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (6 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 07/17] partially fix #6563: ui: expose s3 rrd charts in datastore summary Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 09/17] api: config: update notification thresholds for config and counters Christian Ebner
` (8 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
This will allow to easily extend the lookup by a callback method to
allow passing in additional callbacks whenever that is required for
the backend implementation. By this the provided callback
implementation details can live outside the crate boundaries, e.g.
to hook to the notification system.
By moving this to a central helper, DataStore::lookup_datastore()
calls do not need to individually set parameters.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>
Tested-by: Hannes Laimer <h.laimer@proxmox.com>
---
changes since version 7:
- no changes
pbs-datastore/src/datastore.rs | 36 ++++++++++++++++++----------
pbs-datastore/src/lib.rs | 3 ++-
pbs-datastore/src/snapshot_reader.rs | 6 +++--
src/api2/admin/datastore.rs | 26 ++++++++++----------
src/api2/admin/namespace.rs | 9 ++++---
src/api2/backup/mod.rs | 3 ++-
src/api2/reader/mod.rs | 3 ++-
src/api2/status/mod.rs | 6 +++--
src/api2/tape/backup.rs | 6 +++--
src/api2/tape/restore.rs | 6 +++--
src/bin/proxmox-backup-proxy.rs | 8 ++++---
src/server/prune_job.rs | 3 ++-
src/server/pull.rs | 7 ++++--
src/server/push.rs | 3 ++-
src/server/verify_job.rs | 3 ++-
src/tools/mod.rs | 10 +++++++-
16 files changed, 91 insertions(+), 47 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 61be2d3fd..71ff0fa3a 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -202,6 +202,17 @@ impl DataStoreImpl {
}
}
+pub struct DataStoreLookup<'a> {
+ name: &'a str,
+ operation: Operation,
+}
+
+impl<'a> DataStoreLookup<'a> {
+ pub fn with(name: &'a str, operation: Operation) -> Self {
+ Self { name, operation }
+ }
+}
+
pub struct DataStore {
inner: Arc<DataStoreImpl>,
operation: Option<Operation>,
@@ -498,18 +509,18 @@ impl DataStore {
Ok(())
}
- pub fn lookup_datastore(name: &str, operation: Operation) -> Result<Arc<DataStore>, Error> {
+ pub fn lookup_datastore(lookup: DataStoreLookup) -> Result<Arc<DataStore>, Error> {
// Avoid TOCTOU between checking maintenance mode and updating active operation counter, as
// we use it to decide whether it is okay to delete the datastore.
let _config_lock = pbs_config::datastore::lock_config()?;
// Get the current datastore.cfg generation number and cached config
let (section_config, gen_num) = datastore_section_config_cached(true)?;
- let config: DataStoreConfig = section_config.lookup("datastore", name)?;
+ let config: DataStoreConfig = section_config.lookup("datastore", lookup.name)?;
if let Some(maintenance_mode) = config.get_maintenance_mode() {
- if let Err(error) = maintenance_mode.check(operation) {
- bail!("datastore '{name}' is unavailable: {error}");
+ if let Err(error) = maintenance_mode.check(lookup.operation) {
+ bail!("datastore '{}' is unavailable: {error}", lookup.name);
}
}
@@ -520,16 +531,16 @@ impl DataStore {
bail!("datastore '{}' is not mounted", config.name);
}
- let entry = datastore_cache.get(name);
+ let entry = datastore_cache.get(lookup.name);
// reuse chunk store so that we keep using the same process locker instance!
let chunk_store = if let Some(datastore) = &entry {
// Re-use DataStoreImpl
if datastore.config_generation == gen_num && gen_num.is_some() {
- update_active_operations(name, operation, 1)?;
+ update_active_operations(lookup.name, lookup.operation, 1)?;
return Ok(Arc::new(Self {
inner: Arc::clone(datastore),
- operation: Some(operation),
+ operation: Some(lookup.operation),
}));
}
Arc::clone(&datastore.chunk_store)
@@ -539,7 +550,7 @@ impl DataStore {
.parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
)?;
Arc::new(ChunkStore::open(
- name,
+ lookup.name,
config.absolute_path(),
tuning.sync_level.unwrap_or_default(),
)?)
@@ -548,13 +559,13 @@ impl DataStore {
let datastore = DataStore::with_store_and_config(chunk_store, config, gen_num)?;
let datastore = Arc::new(datastore);
- datastore_cache.insert(name.to_string(), datastore.clone());
+ datastore_cache.insert(lookup.name.to_string(), datastore.clone());
- update_active_operations(name, operation, 1)?;
+ update_active_operations(lookup.name, lookup.operation, 1)?;
Ok(Arc::new(Self {
inner: datastore,
- operation: Some(operation),
+ operation: Some(lookup.operation),
}))
}
@@ -580,7 +591,8 @@ impl DataStore {
{
// the datastore drop handler does the checking if tasks are running and clears the
// cache entry, so we just have to trigger it here
- let _ = DataStore::lookup_datastore(name, Operation::Lookup);
+ let lookup = DataStoreLookup::with(name, Operation::Lookup);
+ let _ = DataStore::lookup_datastore(lookup);
}
Ok(())
diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
index afe340a65..9fa9dd591 100644
--- a/pbs-datastore/src/lib.rs
+++ b/pbs-datastore/src/lib.rs
@@ -217,7 +217,8 @@ pub use store_progress::StoreProgress;
mod datastore;
pub use datastore::{
check_backup_owner, ensure_datastore_is_mounted, get_datastore_mount_status, DataStore,
- DatastoreBackend, S3_CLIENT_REQUEST_COUNTER_BASE_PATH, S3_DATASTORE_IN_USE_MARKER,
+ DataStoreLookup, DatastoreBackend, S3_CLIENT_REQUEST_COUNTER_BASE_PATH,
+ S3_DATASTORE_IN_USE_MARKER,
};
mod hierarchy;
diff --git a/pbs-datastore/src/snapshot_reader.rs b/pbs-datastore/src/snapshot_reader.rs
index 231b1f493..d522a02d7 100644
--- a/pbs-datastore/src/snapshot_reader.rs
+++ b/pbs-datastore/src/snapshot_reader.rs
@@ -16,6 +16,7 @@ use pbs_api_types::{
};
use crate::backup_info::BackupDir;
+use crate::datastore::DataStoreLookup;
use crate::dynamic_index::DynamicIndexReader;
use crate::fixed_index::FixedIndexReader;
use crate::index::IndexFile;
@@ -162,10 +163,11 @@ impl<F: Fn(&[u8; 32]) -> bool> Iterator for SnapshotChunkIterator<'_, F> {
),
};
- let datastore = DataStore::lookup_datastore(
+ let lookup = DataStoreLookup::with(
self.snapshot_reader.datastore_name(),
Operation::Read,
- )?;
+ );
+ let datastore = DataStore::lookup_datastore(lookup)?;
let order =
datastore.get_chunks_in_order(&*index, &self.skip_fn, |_| Ok(()))?;
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 44a7d2bda..865c236fe 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -71,7 +71,9 @@ use crate::api2::backup::optional_ns_param;
use crate::api2::node::rrd::create_value_from_rrd;
use crate::backup::{check_ns_privs_full, ListAccessibleBackupGroups, VerifyWorker, NS_PRIVS_OK};
use crate::server::jobstate::{compute_schedule_status, Job, JobState};
-use crate::tools::{backup_info_to_snapshot_list_item, get_all_snapshot_files, read_backup_index};
+use crate::tools::{
+ backup_info_to_snapshot_list_item, get_all_snapshot_files, lookup_with, read_backup_index,
+};
// helper to unify common sequence of checks:
// 1. check privs on NS (full or limited access)
@@ -88,7 +90,7 @@ fn check_privs_and_load_store(
) -> Result<Arc<DataStore>, Error> {
let limited = check_ns_privs_full(store, ns, auth_id, full_access_privs, partial_access_privs)?;
- let datastore = DataStore::lookup_datastore(store, operation)?;
+ let datastore = DataStore::lookup_datastore(lookup_with(store, operation))?;
if limited {
let owner = datastore.get_owner(ns, backup_group)?;
@@ -134,7 +136,7 @@ pub fn list_groups(
PRIV_DATASTORE_BACKUP,
)?;
- let datastore = DataStore::lookup_datastore(&store, Operation::Read)?;
+ let datastore = DataStore::lookup_datastore(lookup_with(&store, Operation::Read))?;
datastore
.iter_backup_groups(ns.clone())? // FIXME: Namespaces and recursion parameters!
@@ -467,7 +469,7 @@ unsafe fn list_snapshots_blocking(
PRIV_DATASTORE_BACKUP,
)?;
- let datastore = DataStore::lookup_datastore(&store, Operation::Read)?;
+ let datastore = DataStore::lookup_datastore(lookup_with(&store, Operation::Read))?;
// FIXME: filter also owner before collecting, for doing that nicely the owner should move into
// backup group and provide an error free (Err -> None) accessor
@@ -601,7 +603,7 @@ pub async fn status(
}
};
- let datastore = DataStore::lookup_datastore(&store, Operation::Read)?;
+ let datastore = DataStore::lookup_datastore(lookup_with(&store, Operation::Read))?;
let (counts, gc_status) = if verbose {
let filter_owner = if store_privs & PRIV_DATASTORE_AUDIT != 0 {
@@ -731,7 +733,7 @@ pub fn verify(
PRIV_DATASTORE_BACKUP,
)?;
- let datastore = DataStore::lookup_datastore(&store, Operation::Read)?;
+ let datastore = DataStore::lookup_datastore(lookup_with(&store, Operation::Read))?;
let ignore_verified = ignore_verified.unwrap_or(true);
let worker_id;
@@ -1083,7 +1085,7 @@ pub fn prune_datastore(
true,
)?;
- let datastore = DataStore::lookup_datastore(&store, Operation::Write)?;
+ let datastore = DataStore::lookup_datastore(lookup_with(&store, Operation::Write))?;
let ns = prune_options.ns.clone().unwrap_or_default();
let worker_id = format!("{store}:{ns}");
@@ -1121,7 +1123,7 @@ pub fn start_garbage_collection(
_info: &ApiMethod,
rpcenv: &mut dyn RpcEnvironment,
) -> Result<Value, Error> {
- let datastore = DataStore::lookup_datastore(&store, Operation::Write)?;
+ let datastore = DataStore::lookup_datastore(lookup_with(&store, Operation::Write))?;
let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
let job = Job::new("garbage_collection", &store)
@@ -1168,7 +1170,7 @@ pub fn garbage_collection_status(
..Default::default()
};
- let datastore = DataStore::lookup_datastore(&store, Operation::Read)?;
+ let datastore = DataStore::lookup_datastore(lookup_with(&store, Operation::Read))?;
let status_in_memory = datastore.last_gc_status();
let state_file = JobState::load("garbage_collection", &store)
.map_err(|err| log::error!("could not open GC statefile for {store}: {err}"))
@@ -1880,7 +1882,7 @@ pub fn get_rrd_stats(
cf: RrdMode,
_param: Value,
) -> Result<Value, Error> {
- let datastore = DataStore::lookup_datastore(&store, Operation::Read)?;
+ let datastore = DataStore::lookup_datastore(lookup_with(&store, Operation::Read))?;
let disk_manager = crate::tools::disks::DiskManage::new();
let mut rrd_fields = vec![
@@ -2267,7 +2269,7 @@ pub async fn set_backup_owner(
PRIV_DATASTORE_BACKUP,
)?;
- let datastore = DataStore::lookup_datastore(&store, Operation::Write)?;
+ let datastore = DataStore::lookup_datastore(lookup_with(&store, Operation::Write))?;
let backup_group = datastore.backup_group(ns, backup_group);
let owner = backup_group.get_owner()?;
@@ -2752,7 +2754,7 @@ pub fn s3_refresh(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Valu
/// Performs an s3 refresh for given datastore. Expects the store to already be in maintenance mode
/// s3-refresh.
pub(crate) fn do_s3_refresh(store: &str, worker: &dyn WorkerTaskContext) -> Result<(), Error> {
- let datastore = DataStore::lookup_datastore(store, Operation::Lookup)?;
+ let datastore = DataStore::lookup_datastore(lookup_with(store, Operation::Lookup))?;
run_maintenance_locked(store, MaintenanceType::S3Refresh, worker, || {
proxmox_async::runtime::block_on(datastore.s3_refresh())
})
diff --git a/src/api2/admin/namespace.rs b/src/api2/admin/namespace.rs
index 30e24d8db..c885ab540 100644
--- a/src/api2/admin/namespace.rs
+++ b/src/api2/admin/namespace.rs
@@ -54,7 +54,8 @@ pub fn create_namespace(
check_ns_modification_privs(&store, &ns, &auth_id)?;
- let datastore = DataStore::lookup_datastore(&store, Operation::Write)?;
+ let lookup = crate::tools::lookup_with(&store, Operation::Write);
+ let datastore = DataStore::lookup_datastore(lookup)?;
datastore.create_namespace(&parent, name)
}
@@ -97,7 +98,8 @@ pub fn list_namespaces(
// get result up-front to avoid cloning NS, it's relatively cheap anyway (no IO normally)
let parent_access = check_ns_privs(&store, &parent, &auth_id, NS_PRIVS_OK);
- let datastore = DataStore::lookup_datastore(&store, Operation::Read)?;
+ let lookup = crate::tools::lookup_with(&store, Operation::Read);
+ let datastore = DataStore::lookup_datastore(lookup)?;
let iter = match datastore.recursive_iter_backup_ns_ok(parent, max_depth) {
Ok(iter) => iter,
@@ -162,7 +164,8 @@ pub fn delete_namespace(
check_ns_modification_privs(&store, &ns, &auth_id)?;
- let datastore = DataStore::lookup_datastore(&store, Operation::Write)?;
+ let lookup = crate::tools::lookup_with(&store, Operation::Write);
+ let datastore = DataStore::lookup_datastore(lookup)?;
let (removed_all, stats) = datastore.remove_namespace_recursive(&ns, delete_groups)?;
if !removed_all {
diff --git a/src/api2/backup/mod.rs b/src/api2/backup/mod.rs
index 6df0d34bf..86ec49487 100644
--- a/src/api2/backup/mod.rs
+++ b/src/api2/backup/mod.rs
@@ -99,7 +99,8 @@ fn upgrade_to_backup_protocol(
)
.map_err(|err| http_err!(FORBIDDEN, "{err}"))?;
- let datastore = DataStore::lookup_datastore(&store, Operation::Write)?;
+ let lookup = crate::tools::lookup_with(&store, Operation::Write);
+ let datastore = DataStore::lookup_datastore(lookup)?;
let protocols = parts
.headers
diff --git a/src/api2/reader/mod.rs b/src/api2/reader/mod.rs
index 9262eb6cb..a814ba5f7 100644
--- a/src/api2/reader/mod.rs
+++ b/src/api2/reader/mod.rs
@@ -96,7 +96,8 @@ fn upgrade_to_backup_reader_protocol(
bail!("no permissions on /{}", acl_path.join("/"));
}
- let datastore = DataStore::lookup_datastore(&store, Operation::Read)?;
+ let lookup = crate::tools::lookup_with(&store, Operation::Read);
+ let datastore = DataStore::lookup_datastore(lookup)?;
let backup_dir = pbs_api_types::BackupDir::deserialize(¶m)?;
diff --git a/src/api2/status/mod.rs b/src/api2/status/mod.rs
index a1bf44c6b..f01501336 100644
--- a/src/api2/status/mod.rs
+++ b/src/api2/status/mod.rs
@@ -74,7 +74,8 @@ pub async fn datastore_status(
};
if !allowed {
- if let Ok(datastore) = DataStore::lookup_datastore(store, Operation::Lookup) {
+ let lookup = crate::tools::lookup_with(store, Operation::Lookup);
+ if let Ok(datastore) = DataStore::lookup_datastore(lookup) {
if can_access_any_namespace(datastore, &auth_id, &user_info) {
list.push(DataStoreStatusListItem::empty(
store,
@@ -87,7 +88,8 @@ pub async fn datastore_status(
continue;
}
- let datastore = match DataStore::lookup_datastore(store, Operation::Read) {
+ let lookup = crate::tools::lookup_with(store, Operation::Read);
+ let datastore = match DataStore::lookup_datastore(lookup) {
Ok(datastore) => datastore,
Err(err) => {
list.push(DataStoreStatusListItem::empty(
diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
index 47e8d0209..c254c6d8b 100644
--- a/src/api2/tape/backup.rs
+++ b/src/api2/tape/backup.rs
@@ -152,7 +152,8 @@ pub fn do_tape_backup_job(
let worker_type = job.jobtype().to_string();
- let datastore = DataStore::lookup_datastore(&setup.store, Operation::Read)?;
+ let lookup = crate::tools::lookup_with(&setup.store, Operation::Read);
+ let datastore = DataStore::lookup_datastore(lookup)?;
let (config, _digest) = pbs_config::media_pool::config()?;
let pool_config: MediaPoolConfig = config.lookup("pool", &setup.pool)?;
@@ -310,7 +311,8 @@ pub fn backup(
check_backup_permission(&auth_id, &setup.store, &setup.pool, &setup.drive)?;
- let datastore = DataStore::lookup_datastore(&setup.store, Operation::Read)?;
+ let lookup = crate::tools::lookup_with(&setup.store, Operation::Read);
+ let datastore = DataStore::lookup_datastore(lookup)?;
let (config, _digest) = pbs_config::media_pool::config()?;
let pool_config: MediaPoolConfig = config.lookup("pool", &setup.pool)?;
diff --git a/src/api2/tape/restore.rs b/src/api2/tape/restore.rs
index 92529a76d..4356cf748 100644
--- a/src/api2/tape/restore.rs
+++ b/src/api2/tape/restore.rs
@@ -144,10 +144,12 @@ impl TryFrom<String> for DataStoreMap {
if let Some(index) = store.find('=') {
let mut target = store.split_off(index);
target.remove(0); // remove '='
- let datastore = DataStore::lookup_datastore(&target, Operation::Write)?;
+ let lookup = crate::tools::lookup_with(&target, Operation::Write);
+ let datastore = DataStore::lookup_datastore(lookup)?;
map.insert(store, datastore);
} else if default.is_none() {
- default = Some(DataStore::lookup_datastore(&store, Operation::Write)?);
+ let lookup = crate::tools::lookup_with(&store, Operation::Write);
+ default = Some(DataStore::lookup_datastore(lookup)?);
} else {
bail!("multiple default stores given");
}
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index c1fe3ac15..b0efa78ae 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -47,7 +47,7 @@ use pbs_api_types::{
use proxmox_backup::auth_helpers::*;
use proxmox_backup::config;
use proxmox_backup::server::{self, metric_collection};
-use proxmox_backup::tools::PROXMOX_BACKUP_TCP_KEEPALIVE_TIME;
+use proxmox_backup::tools::{lookup_with, PROXMOX_BACKUP_TCP_KEEPALIVE_TIME};
use proxmox_backup::api2::tape::backup::do_tape_backup_job;
use proxmox_backup::server::do_prune_job;
@@ -545,7 +545,8 @@ async fn schedule_datastore_garbage_collection() {
{
// limit datastore scope due to Op::Lookup
- let datastore = match DataStore::lookup_datastore(&store, Operation::Lookup) {
+ let lookup = lookup_with(&store, Operation::Lookup);
+ let datastore = match DataStore::lookup_datastore(lookup) {
Ok(datastore) => datastore,
Err(err) => {
eprintln!("lookup_datastore failed - {err}");
@@ -588,7 +589,8 @@ async fn schedule_datastore_garbage_collection() {
Err(_) => continue, // could not get lock
};
- let datastore = match DataStore::lookup_datastore(&store, Operation::Write) {
+ let lookup = lookup_with(&store, Operation::Write);
+ let datastore = match DataStore::lookup_datastore(lookup) {
Ok(datastore) => datastore,
Err(err) => {
log::warn!("skipping scheduled GC on {store}, could look it up - {err}");
diff --git a/src/server/prune_job.rs b/src/server/prune_job.rs
index bb86a323e..ca5c67541 100644
--- a/src/server/prune_job.rs
+++ b/src/server/prune_job.rs
@@ -133,7 +133,8 @@ pub fn do_prune_job(
auth_id: &Authid,
schedule: Option<String>,
) -> Result<String, Error> {
- let datastore = DataStore::lookup_datastore(&store, Operation::Write)?;
+ let lookup = crate::tools::lookup_with(&store, Operation::Write);
+ let datastore = DataStore::lookup_datastore(lookup)?;
let worker_type = job.jobtype().to_string();
let auth_id = auth_id.clone();
diff --git a/src/server/pull.rs b/src/server/pull.rs
index 0ac6b5b8e..bd3e8bef4 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -112,12 +112,15 @@ impl PullParameters {
client,
})
} else {
+ let lookup = crate::tools::lookup_with(remote_store, Operation::Read);
+ let store = DataStore::lookup_datastore(lookup)?;
Arc::new(LocalSource {
- store: DataStore::lookup_datastore(remote_store, Operation::Read)?,
+ store,
ns: remote_ns,
})
};
- let store = DataStore::lookup_datastore(store, Operation::Write)?;
+ let lookup = crate::tools::lookup_with(store, Operation::Write);
+ let store = DataStore::lookup_datastore(lookup)?;
let backend = store.backend()?;
let target = PullTarget { store, ns, backend };
diff --git a/src/server/push.rs b/src/server/push.rs
index 27c5b22d4..697b94f2f 100644
--- a/src/server/push.rs
+++ b/src/server/push.rs
@@ -110,7 +110,8 @@ impl PushParameters {
let remove_vanished = remove_vanished.unwrap_or(false);
let encrypted_only = encrypted_only.unwrap_or(false);
let verified_only = verified_only.unwrap_or(false);
- let store = DataStore::lookup_datastore(store, Operation::Read)?;
+ let lookup = crate::tools::lookup_with(store, Operation::Read);
+ let store = DataStore::lookup_datastore(lookup)?;
if !store.namespace_exists(&ns) {
bail!(
diff --git a/src/server/verify_job.rs b/src/server/verify_job.rs
index 2ec8c5138..ab14c7389 100644
--- a/src/server/verify_job.rs
+++ b/src/server/verify_job.rs
@@ -15,7 +15,8 @@ pub fn do_verification_job(
schedule: Option<String>,
to_stdout: bool,
) -> Result<String, Error> {
- let datastore = DataStore::lookup_datastore(&verification_job.store, Operation::Read)?;
+ let lookup = crate::tools::lookup_with(&verification_job.store, Operation::Read);
+ let datastore = DataStore::lookup_datastore(lookup)?;
let outdated_after = verification_job.outdated_after;
let ignore_verified_snapshots = verification_job.ignore_verified.unwrap_or(true);
diff --git a/src/tools/mod.rs b/src/tools/mod.rs
index 6a975bde2..a82aaf136 100644
--- a/src/tools/mod.rs
+++ b/src/tools/mod.rs
@@ -6,12 +6,14 @@ use anyhow::{bail, Error};
use std::collections::HashSet;
use pbs_api_types::{
- Authid, BackupContent, CryptMode, SnapshotListItem, SnapshotVerifyState, MANIFEST_BLOB_NAME,
+ Authid, BackupContent, CryptMode, Operation, SnapshotListItem, SnapshotVerifyState,
+ MANIFEST_BLOB_NAME,
};
use proxmox_http::{client::Client, HttpOptions, ProxyConfig};
use pbs_datastore::backup_info::{BackupDir, BackupInfo};
use pbs_datastore::manifest::BackupManifest;
+use pbs_datastore::DataStoreLookup;
pub mod config;
pub mod disks;
@@ -186,3 +188,9 @@ pub(crate) fn backup_info_to_snapshot_list_item(
}
}
}
+
+/// Lookup the datastore by name with given operation.
+#[inline(always)]
+pub fn lookup_with<'a>(name: &'a str, operation: Operation) -> DataStoreLookup<'a> {
+ DataStoreLookup::with(name, operation)
+}
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 09/17] api: config: update notification thresholds for config and counters
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (7 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 08/17] datastore: refactor datastore lookup parameters into dedicated type Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 10/17] ui: add notification thresholds edit window Christian Ebner
` (7 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Allow users to update or clear notification threshold values stored
in the datastore configuration via api or cli.
At the same time, also load and update the threshold values for the
shared request counters so they are applied immediately.
To achieve this without duplicating code, move the request counter
loading and updating of the thresholds into dedicated helpers.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>
Tested-by: Hannes Laimer <h.laimer@proxmox.com>
---
changes since version 7:
- no changes
pbs-datastore/src/datastore.rs | 27 ++++++++++++++++++++++++---
src/api2/config/datastore.rs | 33 +++++++++++++++++++++++++++++----
2 files changed, 53 insertions(+), 7 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 71ff0fa3a..2d63f7527 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -18,8 +18,8 @@ use tracing::{info, warn};
use proxmox_human_byte::HumanByte;
use proxmox_s3_client::{
- S3Client, S3ClientConf, S3ClientOptions, S3ObjectKey, S3PathPrefix, S3RateLimiterOptions,
- S3RequestCounterConfig, SharedRequestCounters,
+ RequestCounterThresholds, S3Client, S3ClientConf, S3ClientOptions, S3ObjectKey, S3PathPrefix,
+ S3RateLimiterOptions, S3RequestCounterConfig, SharedRequestCounters,
};
use proxmox_schema::ApiType;
@@ -706,7 +706,10 @@ impl DataStore {
);
let cache = LocalDatastoreLruCache::new(cache_capacity, chunk_store.clone());
- let request_counters = Self::request_counters(&config, &backend_config)?;
+
+ let mut request_counters = Self::request_counters(&config, &backend_config)?;
+
+ Self::update_notification_thresholds(&mut request_counters, &config)?;
(Some(cache), Some(Arc::new(request_counters)))
} else {
@@ -756,6 +759,24 @@ impl DataStore {
Ok(request_counters)
}
+ /// Update the notification threshold values on the counters by given config
+ pub fn update_notification_thresholds(
+ counters: &mut SharedRequestCounters,
+ config: &DataStoreConfig,
+ ) -> Result<(), Error> {
+ let thresholds: RequestCounterThresholds =
+ if let Some(thresholds) = &config.notification_thresholds {
+ serde_json::from_value(
+ RequestCounterThresholds::API_SCHEMA.parse_property_string(thresholds)?,
+ )?
+ } else {
+ RequestCounterThresholds::default()
+ };
+
+ counters.update_thresholds(&thresholds);
+ Ok(())
+ }
+
// Requires obtaining a shared chunk store lock beforehand
pub fn create_fixed_writer<P: AsRef<Path>>(
&self,
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index f845fe2d0..d51c60f43 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -14,10 +14,11 @@ use proxmox_section_config::SectionConfigData;
use proxmox_uuid::Uuid;
use pbs_api_types::{
- Authid, DataStoreConfig, DataStoreConfigUpdater, DatastoreBackendType, DatastoreNotify,
- DatastoreTuning, KeepOptions, MaintenanceMode, MaintenanceType, PruneJobConfig,
- PruneJobOptions, DATASTORE_SCHEMA, PRIV_DATASTORE_ALLOCATE, PRIV_DATASTORE_AUDIT,
- PRIV_DATASTORE_MODIFY, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA, UPID_SCHEMA,
+ Authid, DataStoreConfig, DataStoreConfigUpdater, DatastoreBackendConfig, DatastoreBackendType,
+ DatastoreNotify, DatastoreTuning, KeepOptions, MaintenanceMode, MaintenanceType,
+ PruneJobConfig, PruneJobOptions, DATASTORE_SCHEMA, PRIV_DATASTORE_ALLOCATE,
+ PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_MODIFY, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA,
+ UPID_SCHEMA,
};
use pbs_config::BackupLockGuard;
use pbs_datastore::chunk_store::ChunkStore;
@@ -436,6 +437,8 @@ pub enum DeletableProperty {
Tuning,
/// Delete the maintenance-mode property
MaintenanceMode,
+ /// Delete the notification-thresholds property
+ NotificationThresholds,
}
#[api(
@@ -534,6 +537,10 @@ pub fn update_datastore(
DeletableProperty::MaintenanceMode => {
data.set_maintenance_mode(None)?;
}
+ DeletableProperty::NotificationThresholds => {
+ data.notification_thresholds = None;
+ update_thresholds(&data)?;
+ }
}
}
}
@@ -619,6 +626,11 @@ pub fn update_datastore(
data.set_maintenance_mode(maintenance_mode)?;
}
+ if update.notification_thresholds.is_some() {
+ data.notification_thresholds = update.notification_thresholds;
+ update_thresholds(&data)?;
+ }
+
config.set_data(&name, "datastore", &data)?;
pbs_config::datastore::save_config(&config)?;
@@ -651,6 +663,19 @@ pub fn update_datastore(
Ok(())
}
+fn update_thresholds(config: &DataStoreConfig) -> Result<(), Error> {
+ let backend_config: DatastoreBackendConfig = serde_json::from_value(
+ DatastoreBackendConfig::API_SCHEMA
+ .parse_property_string(config.backend.as_deref().unwrap_or(""))?,
+ )?;
+ if backend_config.ty.unwrap_or_default() == DatastoreBackendType::S3 {
+ let mut request_counters =
+ pbs_datastore::DataStore::request_counters(config, &backend_config)?;
+ pbs_datastore::DataStore::update_notification_thresholds(&mut request_counters, config)?;
+ }
+ Ok(())
+}
+
#[api(
protected: true,
input: {
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 10/17] ui: add notification thresholds edit window
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (8 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 09/17] api: config: update notification thresholds for config and counters Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 11/17] notification: define templates and template data for thresholds Christian Ebner
` (6 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Allows setting the notification thresholds via the datastore options.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 7:
- inline and simplify notification rendering, assuring to show `None` if
not set
www/Makefile | 1 +
www/datastore/OptionView.js | 8 ++
www/window/NotificationThresholds.js | 113 +++++++++++++++++++++++++++
3 files changed, 122 insertions(+)
create mode 100644 www/window/NotificationThresholds.js
diff --git a/www/Makefile b/www/Makefile
index 9ebf0445f..5002bae90 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -81,6 +81,7 @@ JSSRC= \
window/NamespaceEdit.js \
window/MaintenanceOptions.js \
window/NotesEdit.js \
+ window/NotificationThresholds.js \
window/RemoteEdit.js \
window/TrafficControlEdit.js \
window/NotifyOptions.js \
diff --git a/www/datastore/OptionView.js b/www/datastore/OptionView.js
index 7854e64d7..238fbbf34 100644
--- a/www/datastore/OptionView.js
+++ b/www/datastore/OptionView.js
@@ -334,5 +334,13 @@ Ext.define('PBS.Datastore.Options', {
},
},
},
+ 'notification-thresholds': {
+ required: true,
+ header: gettext('Notification Thresholds'),
+ renderer: (notificationThresholds) => notificationThresholds ?? gettext('None'),
+ editor: {
+ xtype: 'pbsNotificationThresholdsEdit',
+ },
+ },
},
});
diff --git a/www/window/NotificationThresholds.js b/www/window/NotificationThresholds.js
new file mode 100644
index 000000000..19b3598e6
--- /dev/null
+++ b/www/window/NotificationThresholds.js
@@ -0,0 +1,113 @@
+Ext.define('PBS.window.NotificationThresholds', {
+ extend: 'Proxmox.window.Edit',
+ xtype: 'pbsNotificationThresholdsEdit',
+ mixins: ['Proxmox.Mixin.CBind'],
+
+ subject: gettext('Notification Thresholds'),
+
+ width: 500,
+ fieldDefaults: {
+ labelWidth: 120,
+ },
+
+ items: {
+ xtype: 'inputpanel',
+ onGetValues: function (values) {
+ if (!Ext.isArray(values.delete ?? [])) {
+ values.delete = [values.delete];
+ }
+ for (const k of values.delete ?? []) {
+ delete values[k];
+ }
+ delete values.delete;
+ let notificationThresholds = PBS.Utils.printPropertyString(values);
+ if (!notificationThresholds) {
+ return { delete: 'notification-thresholds' };
+ }
+ return { 'notification-thresholds': notificationThresholds };
+ },
+ onSetValues: function (values) {
+ if (values['notification-thresholds']) {
+ return PBS.Utils.parsePropertyString(values['notification-thresholds']);
+ }
+ },
+ items: [
+ {
+ xtype: 'box',
+ html: `<b>${gettext('S3 requests count:')}</b>`,
+ padding: '10 0 5 0',
+ },
+ {
+ xtype: 'proxmoxintegerfield',
+ name: 's3-get',
+ fieldLabel: gettext('GET'),
+ emptyText: gettext('none'),
+ fieldStyle: 'text-align: right',
+ minValue: 0,
+ deleteEmpty: true,
+ },
+ {
+ xtype: 'proxmoxintegerfield',
+ name: 's3-put',
+ fieldLabel: gettext('PUT'),
+ emptyText: gettext('none'),
+ fieldStyle: 'text-align: right',
+ minValue: 0,
+ deleteEmpty: true,
+ },
+ {
+ xtype: 'proxmoxintegerfield',
+ name: 's3-post',
+ fieldLabel: gettext('POST'),
+ emptyText: gettext('none'),
+ fieldStyle: 'text-align: right',
+ minValue: 0,
+ deleteEmpty: true,
+ },
+ {
+ xtype: 'proxmoxintegerfield',
+ name: 's3-head',
+ fieldLabel: gettext('HEAD'),
+ emptyText: gettext('none'),
+ fieldStyle: 'text-align: right',
+ minValue: 0,
+ deleteEmpty: true,
+ },
+ {
+ xtype: 'proxmoxintegerfield',
+ name: 's3-delete',
+ fieldLabel: gettext('DELETE'),
+ emptyText: gettext('none'),
+ fieldStyle: 'text-align: right',
+ minValue: 0,
+ deleteEmpty: true,
+ },
+ {
+ xtype: 'menuseparator',
+ },
+ {
+ xtype: 'box',
+ html: `<b>${gettext('S3 traffic volume:')}</b>`,
+ padding: '10 0 5 0',
+ },
+ {
+ xtype: 'pmxSizeField',
+ name: 's3-upload',
+ fieldLabel: gettext('Upload'),
+ unit: 'GiB',
+ submitAutoScaledSizeUnit: true,
+ allowZero: false,
+ emptyText: gettext('none'),
+ },
+ {
+ xtype: 'pmxSizeField',
+ name: 's3-download',
+ fieldLabel: gettext('Download'),
+ unit: 'GiB',
+ submitAutoScaledSizeUnit: true,
+ allowZero: false,
+ emptyText: gettext('none'),
+ },
+ ],
+ },
+});
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 11/17] notification: define templates and template data for thresholds
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (9 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 10/17] ui: add notification thresholds edit window Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 12/17] datastore: add thresholds notification callback on datastore lookup Christian Ebner
` (5 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Define the templates and data passed to the renderer, required for
the datastore threshold notifications to be rendered and send.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>
Tested-by: Hannes Laimer <h.laimer@proxmox.com>
---
changes since version 7:
- no changes
debian/proxmox-backup-server.install | 2 +
src/server/notifications/mod.rs | 39 +++++++++++++++++--
src/server/notifications/template_data.rs | 30 ++++++++++++++
templates/Makefile | 2 +
.../default/thresholds-exceeded-body.txt.hbs | 9 +++++
.../thresholds-exceeded-subject.txt.hbs | 1 +
6 files changed, 79 insertions(+), 4 deletions(-)
create mode 100644 templates/default/thresholds-exceeded-body.txt.hbs
create mode 100644 templates/default/thresholds-exceeded-subject.txt.hbs
diff --git a/debian/proxmox-backup-server.install b/debian/proxmox-backup-server.install
index 254a4b4b4..1f1d9b601 100644
--- a/debian/proxmox-backup-server.install
+++ b/debian/proxmox-backup-server.install
@@ -67,6 +67,8 @@ usr/share/proxmox-backup/templates/default/tape-load-body.txt.hbs
usr/share/proxmox-backup/templates/default/tape-load-subject.txt.hbs
usr/share/proxmox-backup/templates/default/test-body.txt.hbs
usr/share/proxmox-backup/templates/default/test-subject.txt.hbs
+usr/share/proxmox-backup/templates/default/thresholds-exceeded-body.txt.hbs
+usr/share/proxmox-backup/templates/default/thresholds-exceeded-subject.txt.hbs
usr/share/proxmox-backup/templates/default/verify-err-body.txt.hbs
usr/share/proxmox-backup/templates/default/verify-err-subject.txt.hbs
usr/share/proxmox-backup/templates/default/verify-ok-body.txt.hbs
diff --git a/src/server/notifications/mod.rs b/src/server/notifications/mod.rs
index 95ff9ef18..d877fc7c6 100644
--- a/src/server/notifications/mod.rs
+++ b/src/server/notifications/mod.rs
@@ -23,10 +23,10 @@ const SPOOL_DIR: &str = concatcp!(pbs_buildcfg::PROXMOX_BACKUP_STATE_DIR, "/noti
mod template_data;
use template_data::{
- AcmeErrTemplateData, CommonData, GcErrTemplateData, GcOkTemplateData,
- PackageUpdatesTemplateData, PruneErrTemplateData, PruneOkTemplateData, SyncErrTemplateData,
- SyncOkTemplateData, TapeBackupErrTemplateData, TapeBackupOkTemplateData, TapeLoadTemplateData,
- VerifyErrTemplateData, VerifyOkTemplateData,
+ AcmeErrTemplateData, CommonData, DatastoreThresholdExceededTemplateData, GcErrTemplateData,
+ GcOkTemplateData, PackageUpdatesTemplateData, PruneErrTemplateData, PruneOkTemplateData,
+ SyncErrTemplateData, SyncOkTemplateData, TapeBackupErrTemplateData, TapeBackupOkTemplateData,
+ TapeLoadTemplateData, VerifyErrTemplateData, VerifyOkTemplateData,
};
/// Initialize the notification system by setting context in proxmox_notify
@@ -575,6 +575,37 @@ pub fn send_certificate_renewal_mail(result: &Result<(), Error>) -> Result<(), E
Ok(())
}
+/// send notification if datastore values are exceeding the set threshold limit.
+pub fn send_datastore_threshold_exceeded(
+ datastore: &str,
+ threshold: &str,
+ limit: u64,
+ value: u64,
+) -> Result<(), Error> {
+ let metadata = HashMap::from([
+ ("datastore".into(), datastore.into()),
+ ("hostname".into(), proxmox_sys::nodename().into()),
+ ("type".into(), "thresholds".into()),
+ ]);
+
+ let template_data = DatastoreThresholdExceededTemplateData::new(
+ datastore.to_string(),
+ threshold.to_string(),
+ limit,
+ value,
+ );
+
+ let notification = Notification::from_template(
+ Severity::Warning,
+ "thresholds-exceeded",
+ serde_json::to_value(template_data)?,
+ metadata,
+ );
+
+ send_notification(notification)?;
+ Ok(())
+}
+
/// Lookup users email address
pub fn lookup_user_email(userid: &Userid) -> Option<String> {
if let Ok(user_config) = pbs_config::user::cached_config() {
diff --git a/src/server/notifications/template_data.rs b/src/server/notifications/template_data.rs
index f410730e1..d01744671 100644
--- a/src/server/notifications/template_data.rs
+++ b/src/server/notifications/template_data.rs
@@ -342,3 +342,33 @@ pub struct VerifyErrTemplateData {
/// The list of snapshots that failed to verify.
pub failed_snapshot_list: Vec<String>,
}
+
+/// Template data for the datastore threshold exceeded template.
+#[derive(Serialize)]
+#[serde(rename_all = "kebab-case")]
+pub struct DatastoreThresholdExceededTemplateData {
+ /// Common properties.
+ #[serde(flatten)]
+ pub common: CommonData,
+ /// The datastore.
+ pub datastore: String,
+ /// The threshold which value was exceeded.
+ pub threshold: String,
+ /// The set threshold value.
+ pub limit: u64,
+ /// The value exceeding the threshold.
+ pub value: u64,
+}
+
+impl DatastoreThresholdExceededTemplateData {
+ /// Create new a new instance.
+ pub fn new(datastore: String, threshold: String, limit: u64, value: u64) -> Self {
+ Self {
+ common: CommonData::new(),
+ datastore,
+ threshold,
+ limit,
+ value,
+ }
+ }
+}
diff --git a/templates/Makefile b/templates/Makefile
index 0539902eb..8a4586d78 100644
--- a/templates/Makefile
+++ b/templates/Makefile
@@ -25,6 +25,8 @@ NOTIFICATION_TEMPLATES= \
default/tape-load-subject.txt.hbs \
default/test-body.txt.hbs \
default/test-subject.txt.hbs \
+ default/thresholds-exceeded-body.txt.hbs \
+ default/thresholds-exceeded-subject.txt.hbs \
default/verify-err-body.txt.hbs \
default/verify-ok-body.txt.hbs \
default/verify-err-subject.txt.hbs \
diff --git a/templates/default/thresholds-exceeded-body.txt.hbs b/templates/default/thresholds-exceeded-body.txt.hbs
new file mode 100644
index 000000000..e7cab6bf3
--- /dev/null
+++ b/templates/default/thresholds-exceeded-body.txt.hbs
@@ -0,0 +1,9 @@
+Datastore: {{datastore}}
+Threshold: {{threshold}}
+Limit: {{limit}}
+
+Threshold {{threshold}}: limit {{limit}} exceeded by value {{value}}.
+
+Please visit the web interface for further details:
+
+<{{base-url}}/#DataStore-{{datastore}}>
diff --git a/templates/default/thresholds-exceeded-subject.txt.hbs b/templates/default/thresholds-exceeded-subject.txt.hbs
new file mode 100644
index 000000000..5aefb8778
--- /dev/null
+++ b/templates/default/thresholds-exceeded-subject.txt.hbs
@@ -0,0 +1 @@
+Threshold Exceeded For Datastore '{{datastore}}'
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 12/17] datastore: add thresholds notification callback on datastore lookup
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (10 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 11/17] notification: define templates and template data for thresholds Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 13/17] api/ui: notifications: add 'thresholds' as notification type value Christian Ebner
` (4 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Pass in the callback to be executed when a datastore threshold value
exceeds its set limit. This callback is then callable by the s3
client request counters implementation, currently the only component
defining thresholds.
When a threshold is exceeded, the notification is generated by the
callback and queued for being send.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>
Tested-by: Hannes Laimer <h.laimer@proxmox.com>
---
changes since version 7:
- no changes
pbs-datastore/src/datastore.rs | 36 ++++++++++++++++++++++++----
pbs-datastore/src/snapshot_reader.rs | 1 +
src/tools/mod.rs | 6 ++++-
3 files changed, 38 insertions(+), 5 deletions(-)
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 2d63f7527..66cd30cea 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -202,14 +202,25 @@ impl DataStoreImpl {
}
}
+pub type ThresholdsExceededCallback = fn(&str, &str, u64, u64) -> Result<(), Error>;
+
pub struct DataStoreLookup<'a> {
name: &'a str,
operation: Operation,
+ thresholds_exceeded_callback: Option<ThresholdsExceededCallback>,
}
impl<'a> DataStoreLookup<'a> {
- pub fn with(name: &'a str, operation: Operation) -> Self {
- Self { name, operation }
+ pub fn with(
+ name: &'a str,
+ operation: Operation,
+ thresholds_exceeded_callback: Option<ThresholdsExceededCallback>,
+ ) -> Self {
+ Self {
+ name,
+ operation,
+ thresholds_exceeded_callback,
+ }
}
}
@@ -556,7 +567,12 @@ impl DataStore {
)?)
};
- let datastore = DataStore::with_store_and_config(chunk_store, config, gen_num)?;
+ let datastore = DataStore::with_store_and_config(
+ chunk_store,
+ config,
+ gen_num,
+ lookup.thresholds_exceeded_callback,
+ )?;
let datastore = Arc::new(datastore);
datastore_cache.insert(lookup.name.to_string(), datastore.clone());
@@ -591,7 +607,7 @@ impl DataStore {
{
// the datastore drop handler does the checking if tasks are running and clears the
// cache entry, so we just have to trigger it here
- let lookup = DataStoreLookup::with(name, Operation::Lookup);
+ let lookup = DataStoreLookup::with(name, Operation::Lookup, None);
let _ = DataStore::lookup_datastore(lookup);
}
@@ -645,6 +661,7 @@ impl DataStore {
Arc::new(chunk_store),
config,
None,
+ None,
)?);
if let Some(operation) = operation {
@@ -658,6 +675,7 @@ impl DataStore {
chunk_store: Arc<ChunkStore>,
config: DataStoreConfig,
generation: Option<usize>,
+ thresholds_exceeded_callback: Option<ThresholdsExceededCallback>,
) -> Result<DataStoreImpl, Error> {
let mut gc_status_path = chunk_store.base_path();
gc_status_path.push(".gc-status");
@@ -710,6 +728,16 @@ impl DataStore {
let mut request_counters = Self::request_counters(&config, &backend_config)?;
Self::update_notification_thresholds(&mut request_counters, &config)?;
+ request_counters.set_thresholds_exceeded_callback(
+ config.name,
+ Box::new(move |label, counter_id, threshold, value| {
+ thresholds_exceeded_callback.map(|cb| {
+ if let Err(err) = cb(label, counter_id, threshold, value) {
+ log::error!("failed to send notification: {err:?}");
+ }
+ });
+ }),
+ );
(Some(cache), Some(Arc::new(request_counters)))
} else {
diff --git a/pbs-datastore/src/snapshot_reader.rs b/pbs-datastore/src/snapshot_reader.rs
index d522a02d7..7f7625f91 100644
--- a/pbs-datastore/src/snapshot_reader.rs
+++ b/pbs-datastore/src/snapshot_reader.rs
@@ -166,6 +166,7 @@ impl<F: Fn(&[u8; 32]) -> bool> Iterator for SnapshotChunkIterator<'_, F> {
let lookup = DataStoreLookup::with(
self.snapshot_reader.datastore_name(),
Operation::Read,
+ None,
);
let datastore = DataStore::lookup_datastore(lookup)?;
let order =
diff --git a/src/tools/mod.rs b/src/tools/mod.rs
index a82aaf136..5b6a4c293 100644
--- a/src/tools/mod.rs
+++ b/src/tools/mod.rs
@@ -192,5 +192,9 @@ pub(crate) fn backup_info_to_snapshot_list_item(
/// Lookup the datastore by name with given operation.
#[inline(always)]
pub fn lookup_with<'a>(name: &'a str, operation: Operation) -> DataStoreLookup<'a> {
- DataStoreLookup::with(name, operation)
+ DataStoreLookup::with(
+ name,
+ operation,
+ Some(crate::server::notifications::send_datastore_threshold_exceeded),
+ )
}
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 13/17] api/ui: notifications: add 'thresholds' as notification type value
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (11 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 12/17] datastore: add thresholds notification callback on datastore lookup Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 14/17] api: config: allow counter reset schedule editing Christian Ebner
` (3 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Allows to add notification rules regarding thresholds by matching
using this new notification type value. By adding it to the list
here, the value can be selected in the notification matcher edit
window in the UI. The user facing comment is finally defined in the
utils.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>
Tested-by: Hannes Laimer <h.laimer@proxmox.com>
---
changes since version 7:
- no changes
src/api2/config/notifications/mod.rs | 1 +
www/Utils.js | 1 +
2 files changed, 2 insertions(+)
diff --git a/src/api2/config/notifications/mod.rs b/src/api2/config/notifications/mod.rs
index 40fe4dfaa..2b94f715b 100644
--- a/src/api2/config/notifications/mod.rs
+++ b/src/api2/config/notifications/mod.rs
@@ -190,6 +190,7 @@ pub fn get_values(
"system-mail",
"tape-backup",
"tape-load",
+ "thresholds",
"verify",
] {
values.push(MatchableValue {
diff --git a/www/Utils.js b/www/Utils.js
index 25fba16f9..dc4b8b013 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -483,6 +483,7 @@ Ext.define('PBS.Utils', {
sync: gettext('Sync job'),
'tape-backup': gettext('Tape backup notifications'),
'tape-load': gettext('Tape loading request'),
+ thresholds: gettext('Datastore threshold notifications'),
verify: gettext('Verification job'),
});
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 14/17] api: config: allow counter reset schedule editing
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (12 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 13/17] api/ui: notifications: add 'thresholds' as notification type value Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 15/17] ui: expose counter reset schedule edit window Christian Ebner
` (2 subsequent siblings)
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Must allow to create/edit/delete the schedule which will be used to
periodically reset notification threshold related counters.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>
Tested-by: Hannes Laimer <h.laimer@proxmox.com>
---
changes since version 7:
- no changes
src/api2/config/datastore.rs | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index d51c60f43..a34b3e22a 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -439,6 +439,8 @@ pub enum DeletableProperty {
MaintenanceMode,
/// Delete the notification-thresholds property
NotificationThresholds,
+ /// Delete the counter reset schedule.
+ CounterResetSchedule,
}
#[api(
@@ -541,6 +543,9 @@ pub fn update_datastore(
data.notification_thresholds = None;
update_thresholds(&data)?;
}
+ DeletableProperty::CounterResetSchedule => {
+ data.counter_reset_schedule = None;
+ }
}
}
}
@@ -631,6 +636,10 @@ pub fn update_datastore(
update_thresholds(&data)?;
}
+ if update.counter_reset_schedule.is_some() {
+ data.counter_reset_schedule = update.counter_reset_schedule;
+ }
+
config.set_data(&name, "datastore", &data)?;
pbs_config::datastore::save_config(&config)?;
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 15/17] ui: expose counter reset schedule edit window
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (13 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 14/17] api: config: allow counter reset schedule editing Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 16/17] bin: proxy: periodically schedule counter reset task Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 17/17] ui: add task description for scheduled counter reset Christian Ebner
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Shows the counter reset schedule in the datastore options and allows
editing.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 7:
- render `None` if no schedule is set
www/Makefile | 1 +
www/datastore/OptionView.js | 8 ++++++++
www/window/CounterResetScheduleEdit.js | 27 ++++++++++++++++++++++++++
3 files changed, 36 insertions(+)
create mode 100644 www/window/CounterResetScheduleEdit.js
diff --git a/www/Makefile b/www/Makefile
index 5002bae90..8ac338f2d 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -77,6 +77,7 @@ JSSRC= \
window/ACLEdit.js \
window/BackupGroupChangeOwner.js \
window/CreateDirectory.js \
+ window/CounterResetScheduleEdit.js \
window/DataStoreEdit.js \
window/NamespaceEdit.js \
window/MaintenanceOptions.js \
diff --git a/www/datastore/OptionView.js b/www/datastore/OptionView.js
index 238fbbf34..d4886ecf0 100644
--- a/www/datastore/OptionView.js
+++ b/www/datastore/OptionView.js
@@ -342,5 +342,13 @@ Ext.define('PBS.Datastore.Options', {
xtype: 'pbsNotificationThresholdsEdit',
},
},
+ 'counter-reset-schedule': {
+ required: true,
+ header: gettext('Counter Reset Schedule'),
+ renderer: (schedule) => schedule ?? gettext('None'),
+ editor: {
+ xtype: 'pbsCounterResetScheduleEdit',
+ },
+ },
},
});
diff --git a/www/window/CounterResetScheduleEdit.js b/www/window/CounterResetScheduleEdit.js
new file mode 100644
index 000000000..6d0cd5d73
--- /dev/null
+++ b/www/window/CounterResetScheduleEdit.js
@@ -0,0 +1,27 @@
+Ext.define('PBS.window.CounterResetScheduleEdit', {
+ extend: 'Proxmox.window.Edit',
+ alias: 'widget.pbsCounterResetScheduleEdit',
+ mixins: ['Proxmox.Mixin.CBind'],
+
+ userid: undefined,
+ isAdd: false,
+
+ subject: gettext('Counter Reset Schedule'),
+
+ cbindData: function (initial) {
+ let me = this;
+
+ me.datastore = encodeURIComponent(me.datastore);
+ me.url = `/api2/extjs/config/datastore/${me.datastore}`;
+ me.method = 'PUT';
+ me.autoLoad = true;
+ return {};
+ },
+
+ items: {
+ xtype: 'pbsCalendarEvent',
+ name: 'counter-reset-schedule',
+ fieldLabel: gettext('Counter Reset Schedule'),
+ emptyText: gettext('none (disabled)'),
+ },
+});
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 16/17] bin: proxy: periodically schedule counter reset task
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (14 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 15/17] ui: expose counter reset schedule edit window Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 17/17] ui: add task description for scheduled counter reset Christian Ebner
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
Analogous to other recurring scheduled tasks, check the configured
counter reset schedule for each datastore and periodically execute
the reset task if set. By performing this as a dedicated job, it is
assured to keep track of the scheduled executions.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 7:
- no changes
src/bin/proxmox-backup-proxy.rs | 81 ++++++++++++++++++++++++++++++++-
1 file changed, 79 insertions(+), 2 deletions(-)
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index b0efa78ae..238637989 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -1,5 +1,6 @@
use std::path::{Path, PathBuf};
use std::pin::pin;
+use std::sync::atomic::Ordering;
use std::sync::{Arc, Mutex};
use anyhow::{bail, format_err, Context, Error};
@@ -17,6 +18,7 @@ use url::form_urlencoded;
use proxmox_http::Body;
use proxmox_http::RateLimiterTag;
+use proxmox_human_byte::HumanByte;
use proxmox_lang::try_block;
use proxmox_rest_server::{
cleanup_old_tasks, cookie_from_header, rotate_task_log_archive, ApiConfig, Redirector,
@@ -40,8 +42,8 @@ use pbs_buildcfg::configdir;
use proxmox_time::CalendarEvent;
use pbs_api_types::{
- Authid, DataStoreConfig, Operation, PruneJobConfig, SyncJobConfig, TapeBackupJobConfig,
- VerificationJobConfig,
+ Authid, DataStoreConfig, DatastoreBackendConfig, Operation, PruneJobConfig, SyncJobConfig,
+ TapeBackupJobConfig, VerificationJobConfig,
};
use proxmox_backup::auth_helpers::*;
@@ -508,6 +510,7 @@ async fn schedule_tasks() -> Result<(), Error> {
schedule_datastore_verify_jobs().await;
schedule_tape_backup_jobs().await;
schedule_task_log_rotate().await;
+ schedule_notification_threshold_counter_reset().await;
Ok(())
}
@@ -881,6 +884,80 @@ async fn schedule_task_log_rotate() {
}
}
+async fn schedule_notification_threshold_counter_reset() {
+ let config = match pbs_config::datastore::config() {
+ Err(err) => {
+ eprintln!("unable to read datastore config - {err}");
+ return;
+ }
+ Ok((config, _digest)) => config,
+ };
+
+ for (store, (_, store_config)) in config.sections {
+ let store_config: DataStoreConfig = match serde_json::from_value(store_config) {
+ Ok(c) => c,
+ Err(err) => {
+ eprintln!("datastore config from_value failed - {err}");
+ continue;
+ }
+ };
+
+ let event_str = match &store_config.counter_reset_schedule {
+ Some(event_str) => event_str,
+ None => continue,
+ };
+
+ let worker_type = "notification-threshold-counter-reset";
+ if check_schedule(worker_type, event_str, &store) {
+ let mut job = match Job::new(worker_type, &store) {
+ Ok(job) => job,
+ Err(_) => continue, // could not get lock
+ };
+
+ if let Err(err) = WorkerTask::new_thread(
+ worker_type,
+ None,
+ Authid::root_auth_id().to_string(),
+ false,
+ move |worker| {
+ job.start(&worker.upid().to_string())?;
+ info!("executing counter reset for {store}");
+
+ let result = try_block!({
+ let backend_config: DatastoreBackendConfig =
+ store_config.backend.as_deref().unwrap_or("").parse()?;
+ let request_counters =
+ DataStore::request_counters(&store_config, &backend_config)?;
+ let last_values = request_counters.reset(Ordering::Release);
+ info!("Last counter values before reset:");
+ info!("Request traffic volume:");
+ info!("Uploaded: {}", HumanByte::from(last_values.upload));
+ info!("Downloaded: {}", HumanByte::from(last_values.download));
+ info!("Request count by method:");
+ info!("GET: {}", last_values.get);
+ info!("PUT: {}", last_values.put);
+ info!("POST: {}", last_values.post);
+ info!("HEAD: {}", last_values.head);
+ info!("DELETE: {}", last_values.delete);
+
+ Ok(())
+ });
+
+ let status = worker.create_state(&result);
+
+ if let Err(err) = job.finish(status) {
+ eprintln!("could not finish job state for {worker_type}: {err}");
+ }
+
+ result
+ },
+ ) {
+ eprintln!("unable to start counter reset task: {err}");
+ }
+ }
+ }
+}
+
async fn command_reopen_access_logfiles() -> Result<(), Error> {
// only care about the most recent daemon instance for each, proxy & api, as other older ones
// should not respond to new requests anyway, but only finish their current one and then exit.
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH proxmox-backup v8 17/17] ui: add task description for scheduled counter reset
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
` (15 preceding siblings ...)
2026-04-02 10:53 ` [PATCH proxmox-backup v8 16/17] bin: proxy: periodically schedule counter reset task Christian Ebner
@ 2026-04-02 10:53 ` Christian Ebner
16 siblings, 0 replies; 18+ messages in thread
From: Christian Ebner @ 2026-04-02 10:53 UTC (permalink / raw)
To: pbs-devel
So the task list and task log window show a more user friedly
description and title for the task.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 7:
- no changes
www/Utils.js | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/www/Utils.js b/www/Utils.js
index dc4b8b013..d76be709b 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -447,6 +447,10 @@ Ext.define('PBS.Utils', {
gettext('Datastore'),
gettext('sync jobs handler triggered by mount'),
],
+ 'notification-threshold-counter-reset': [
+ null,
+ gettext('Notification Threshold Counter Reset'),
+ ],
prune: (type, id) => PBS.Utils.render_datastore_worker_id(id, gettext('Prune')),
prunejob: (type, id) => PBS.Utils.render_prune_job_worker_id(id, gettext('Prune Job')),
reader: (type, id) => PBS.Utils.render_datastore_worker_id(id, gettext('Read Objects')),
--
2.47.3
^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2026-04-02 10:54 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-04-02 10:53 [PATCH proxmox-backup v8 00/17] partially fix #6563: add s3 counter for statistics and notifications Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 01/17] api: s3: add endpoint to reset s3 request counters Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 02/17] bin: s3: expose request counter reset method as cli command Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 03/17] ui: datastore summary: move store to be part of summary panel Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 04/17] ui: expose s3 request counter statistics in the datastore summary Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 05/17] metrics: collect s3 datastore statistics as rrd metrics Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 06/17] api: admin: expose s3 statistics in datastore rrd data Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 07/17] partially fix #6563: ui: expose s3 rrd charts in datastore summary Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 08/17] datastore: refactor datastore lookup parameters into dedicated type Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 09/17] api: config: update notification thresholds for config and counters Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 10/17] ui: add notification thresholds edit window Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 11/17] notification: define templates and template data for thresholds Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 12/17] datastore: add thresholds notification callback on datastore lookup Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 13/17] api/ui: notifications: add 'thresholds' as notification type value Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 14/17] api: config: allow counter reset schedule editing Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 15/17] ui: expose counter reset schedule edit window Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 16/17] bin: proxy: periodically schedule counter reset task Christian Ebner
2026-04-02 10:53 ` [PATCH proxmox-backup v8 17/17] ui: add task description for scheduled counter reset Christian Ebner
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.