[pbs-devel] [PATCH proxmox{, -backup} v2 0/7] tuning: verify: add verify job thread options

[pbs-devel] [PATCH proxmox{, -backup} v2 0/7] tuning: verify: add verify job thread options

[Nicolas Frey]

this patch series aims to expose configuration of the number of threads
to use for reading and verifying of verify jobs through the datastore's
tuning options.

It expands on the series I sent last week [0], where @Thomas
suggested making this configurable through the tuning options too.
@Chris suggested adding a dedicated worker thread settings to the
datastore instead, which is an approach I don't have a problem with,
but I'd want some more opinions on.

If the docs should be updated to account for this change, I can either
send a follow up or do that in a v2.

Later down the line, it would also be beneficial to add the tape job
threads to these options, though that is outside of the scope of this
series.

[0] https://lore.proxmox.com/pbs-devel/5a731352-1f77-4b15-9a22-7064b70ce519@proxmox.com/T/#t

Changes since v1 (thanks Chris!):
* renamed user-facing parameters to make them less clumsy and clearer
* add the new tuning parameters to the docs
* adapt `render_tuning_options`
* cleaned up nits

proxmox:

Nicolas Frey (1):
  pbs-api-types: add default-verification-{workers,readers} to tuning
    options

 pbs-api-types/src/datastore.rs | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)


proxmox-backup:

Nicolas Frey (6):
  datastore: add new thread settings to tuning options
  verify: use tuning options' thread settings for {verify,read}-threads
  ui: tuning: make default-verification-{workers,readers} configurable
  ui: tuning: render verification threads in tuning options
  ui: verify: fetch default {read,verify} thread values from tuning
    options
  docs: tuning: document default-verification-{workers,readers}

 docs/storage.rst               |  5 +++++
 pbs-datastore/src/datastore.rs | 34 ++++++++++++++++++++++++++++++
 src/backup/verify.rs           | 14 +++++++++++--
 www/Utils.js                   | 12 +++++++++++
 www/datastore/OptionView.js    | 20 ++++++++++++++++++
 www/window/VerifyAll.js        | 38 ++++++++++++++++++++++++++++++++--
 www/window/VerifyJobEdit.js    | 37 +++++++++++++++++++++++++++++++++
 7 files changed, 156 insertions(+), 4 deletions(-)


Summary over all repositories:
  8 files changed, 170 insertions(+), 5 deletions(-)

-- 
Generated by git-murpp 0.8.1

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox v2 1/1] pbs-api-types: add default-verification-{workers, readers} to tuning options

[Nicolas Frey]

to determine the default number of threads to use for verify jobs on
the datastore level

Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Tested-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Nicolas Frey <n.frey@proxmox.com>
---
Changes since v1:
* rename the user-facing parameters to
  `default-verification-{workers,readers}`

 pbs-api-types/src/datastore.rs | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index a214ac25..c17452a8 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -19,7 +19,8 @@ use crate::{
     BACKUP_ID_RE, BACKUP_NS_RE, BACKUP_TIME_RE, BACKUP_TYPE_RE, DATASTORE_NOTIFY_STRING_SCHEMA,
     GC_SCHEDULE_SCHEMA, GROUP_OR_SNAPSHOT_PATH_REGEX_STR, PROXMOX_SAFE_ID_FORMAT,
     PROXMOX_SAFE_ID_REGEX_STR, PRUNE_SCHEDULE_SCHEMA, SHA256_HEX_REGEX, SINGLE_LINE_COMMENT_SCHEMA,
-    SNAPSHOT_PATH_REGEX_STR, UPID,
+    SNAPSHOT_PATH_REGEX_STR, UPID, VERIFY_JOB_READ_THREADS_SCHEMA,
+    VERIFY_JOB_VERIFY_THREADS_SCHEMA,
 };
 
 const_regex! {
@@ -262,6 +263,14 @@ pub const GC_CACHE_CAPACITY_SCHEMA: Schema =
             schema: GC_CACHE_CAPACITY_SCHEMA,
             optional: true,
         },
+        "default-verification-workers": {
+            schema: VERIFY_JOB_VERIFY_THREADS_SCHEMA,
+            optional: true,
+        },
+        "default-verification-readers": {
+            schema: VERIFY_JOB_READ_THREADS_SCHEMA,
+            optional: true,
+        },
     },
 )]
 #[derive(Serialize, Deserialize, Default)]
@@ -279,6 +288,10 @@ pub struct DatastoreTuning {
     pub gc_atime_cutoff: Option<usize>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub gc_cache_capacity: Option<usize>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub default_verification_workers: Option<usize>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub default_verification_readers: Option<usize>,
 }
 
 pub const DATASTORE_TUNING_STRING_SCHEMA: Schema = StringSchema::new("Datastore tuning options")
-- 
2.47.3


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v2 1/6] datastore: add new thread settings to tuning options

[Nicolas Frey]

add a new field to the `DataStoreImpl` to store the verify job thread
configuration from the tuning options

Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Tested-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Nicolas Frey <n.frey@proxmox.com>
---
Changes since v1:
* make the `new` fn not pub
* add docstring to `thread_settings` fn

 pbs-datastore/src/datastore.rs | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 65299cca..8d3dff4d 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -153,6 +153,7 @@ pub struct DataStoreImpl {
     sync_level: DatastoreFSyncLevel,
     backend_config: DatastoreBackendConfig,
     lru_store_caching: Option<LocalDatastoreLruCache>,
+    thread_settings: DatastoreThreadSettings,
 }
 
 impl DataStoreImpl {
@@ -169,6 +170,7 @@ impl DataStoreImpl {
             sync_level: Default::default(),
             backend_config: Default::default(),
             lru_store_caching: None,
+            thread_settings: Default::default(),
         })
     }
 }
@@ -263,6 +265,27 @@ impl DatastoreBackend {
     }
 }
 
+#[derive(Clone, Default)]
+/// Amount of threads to use for certain jobs in a datastore.
+pub struct DatastoreThreadSettings {
+    /// # of threads to use to verify in verify job
+    pub verify_job_verify_threads: Option<usize>,
+    /// # of threads to use to read in verify job
+    pub verify_job_read_threads: Option<usize>,
+}
+
+impl DatastoreThreadSettings {
+    fn new(
+        verify_job_verify_threads: Option<usize>,
+        verify_job_read_threads: Option<usize>,
+    ) -> Self {
+        Self {
+            verify_job_verify_threads,
+            verify_job_read_threads,
+        }
+    }
+}
+
 impl DataStore {
     // This one just panics on everything
     #[doc(hidden)]
@@ -545,6 +568,11 @@ impl DataStore {
             None
         };
 
+        let thread_settings = DatastoreThreadSettings::new(
+            tuning.default_verification_workers,
+            tuning.default_verification_readers,
+        );
+
         Ok(DataStoreImpl {
             chunk_store,
             gc_mutex: Mutex::new(None),
@@ -555,6 +583,7 @@ impl DataStore {
             sync_level: tuning.sync_level.unwrap_or_default(),
             backend_config,
             lru_store_caching,
+            thread_settings,
         })
     }
 
@@ -654,6 +683,11 @@ impl DataStore {
         self.inner.chunk_store.base_path()
     }
 
+    /// Returns the thread settings for this datastore
+    pub fn thread_settings(&self) -> &DatastoreThreadSettings {
+        &self.inner.thread_settings
+    }
+
     /// Returns the absolute path for a backup namespace on this datastore
     pub fn namespace_path(&self, ns: &BackupNamespace) -> PathBuf {
         let mut path = self.base_path();
-- 
2.47.3


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v2 2/6] verify: use tuning options' thread settings for {verify, read}-threads

[Nicolas Frey]

uses the datastores' verify job thread settings as a secondary
fallback:

read_threads   -> arg? -> datastore.read? -> 1 (default)
verify_threads -> arg? -> datastore.verify? -> 4 (default)

Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Tested-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Nicolas Frey <n.frey@proxmox.com>
---
 src/backup/verify.rs | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 734c7b30..f52d7781 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -73,6 +73,16 @@ impl VerifyWorker {
         verify_threads: Option<usize>,
     ) -> Result<Self, Error> {
         let backend = datastore.backend()?;
+        let thread_settings = datastore.thread_settings();
+
+        let read_threads = read_threads
+            .or(thread_settings.verify_job_read_threads)
+            .unwrap_or(1);
+
+        let verify_threads = verify_threads
+            .or(thread_settings.verify_job_verify_threads)
+            .unwrap_or(4);
+
         Ok(Self {
             worker,
             datastore,
@@ -81,8 +91,8 @@ impl VerifyWorker {
             // start with 64 chunks since we assume there are few corrupt ones
             corrupt_chunks: Arc::new(Mutex::new(HashSet::with_capacity(64))),
             backend,
-            read_threads: read_threads.unwrap_or(1),
-            verify_threads: verify_threads.unwrap_or(4),
+            read_threads,
+            verify_threads,
         })
     }
 
-- 
2.47.3


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v2 3/6] ui: tuning: make default-verification-{workers, readers} configurable

[Nicolas Frey]

Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Tested-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Nicolas Frey <n.frey@proxmox.com>
---
Changes since v1:
* change label to better describe the parameter

 www/datastore/OptionView.js | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/www/datastore/OptionView.js b/www/datastore/OptionView.js
index 2f9aa9eb..22bf80e6 100644
--- a/www/datastore/OptionView.js
+++ b/www/datastore/OptionView.js
@@ -310,6 +310,26 @@ Ext.define('PBS.Datastore.Options', {
                             deleteEmpty: true,
                             step: 1024,
                         },
+                        {
+                            xtype: 'proxmoxintegerfield',
+                            name: 'default-verification-workers',
+                            fieldLabel: gettext('Default # of Verification Workers'),
+                            labelWidth: 200,
+                            emptyText: '4',
+                            minValue: 1,
+                            maxValue: 32,
+                            deleteEmpty: true,
+                        },
+                        {
+                            xtype: 'proxmoxintegerfield',
+                            name: 'default-verification-readers',
+                            fieldLabel: gettext('Default # of Verification Readers'),
+                            labelWidth: 200,
+                            emptyText: '1',
+                            minValue: 1,
+                            maxValue: 32,
+                            deleteEmpty: true,
+                        },
                     ],
                 },
             },
-- 
2.47.3


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v2 4/6] ui: tuning: render verification threads in tuning options

[Nicolas Frey]

account for default-verification-{workers,readers} when rendering
tuning options

Signed-off-by: Nicolas Frey <n.frey@proxmox.com>
---
New in v2

 www/Utils.js | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/www/Utils.js b/www/Utils.js
index a80a59a5..fc9a5916 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -915,6 +915,18 @@ Ext.define('PBS.Utils', {
             `${gettext('GC cache capacity')}: ${gc_cache_capacity ?? Proxmox.Utils.defaultText}`,
         );
 
+        let verification_workers = tuning['default-verification-workers'];
+        delete tuning['default-verification-workers'];
+        options.push(
+            `${gettext('Default verification workers')}: ${verification_workers ?? 4}`,
+        );
+
+        let verification_readers = tuning['default-verification-readers'];
+        delete tuning['default-verification-readers'];
+        options.push(
+            `${gettext('Default verification readers')}: ${verification_readers ?? 1}`,
+        );
+
         for (const [k, v] of Object.entries(tuning)) {
             options.push(`${k}: ${v}`);
         }
-- 
2.47.3


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v2 5/6] ui: verify: fetch default {read, verify} thread values from tuning options

[Nicolas Frey]

making sure that the now default (e.g. emptyText) value matches the
one found at the datastore-tuning level, we fetch the tuning data
and bind it to the respective `emptyText` properties.

Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Tested-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Nicolas Frey <n.frey@proxmox.com>
---
 www/window/VerifyAll.js     | 38 +++++++++++++++++++++++++++++++++++--
 www/window/VerifyJobEdit.js | 37 ++++++++++++++++++++++++++++++++++++
 2 files changed, 73 insertions(+), 2 deletions(-)

diff --git a/www/window/VerifyAll.js b/www/window/VerifyAll.js
index 4239c215..c0fae10b 100644
--- a/www/window/VerifyAll.js
+++ b/www/window/VerifyAll.js
@@ -11,6 +11,36 @@ Ext.define('PBS.window.VerifyAll', {
         url: `/admin/datastore/{datastore}/verify`,
     },
 
+    viewModel: {
+        data: {
+            defaultReadThreads: null,
+            defaultVerifyThreads: null,
+        },
+    },
+
+    initComponent: function () {
+        let me = this;
+        me.callParent();
+
+        if (me.datastore === undefined) {
+            return;
+        }
+
+        Proxmox.Utils.API2Request({
+            url: `/api2/extjs/config/datastore/${encodeURIComponent(me.datastore)}`,
+            method: 'GET',
+            success: ({ result }) => {
+                let raw = result?.data?.tuning || {};
+                let tuning = PBS.Utils.parsePropertyString(raw);
+
+                me.getViewModel().set({
+                    defaultReadThreads: tuning['default-verification-readers'] ?? 1,
+                    defaultVerifyThreads: tuning['default-verification-workers'] ?? 4,
+                });
+            },
+        });
+    },
+
     submitText: gettext('Verify'),
     isCreate: true,
     showTaskViewer: true,
@@ -84,7 +114,9 @@ Ext.define('PBS.window.VerifyAll', {
                     xtype: 'proxmoxintegerfield',
                     name: 'read-threads',
                     fieldLabel: gettext('# of Read Threads'),
-                    emptyText: '1',
+                    bind: {
+                        emptyText: '{defaultReadThreads}',
+                    },
                     skipEmptyText: true,
                     minValue: 1,
                     maxValue: 32,
@@ -93,7 +125,9 @@ Ext.define('PBS.window.VerifyAll', {
                     xtype: 'proxmoxintegerfield',
                     name: 'verify-threads',
                     fieldLabel: gettext('# of Verify Threads'),
-                    emptyText: '4',
+                    bind: {
+                        emptyText: '{defaultVerifyThreads}',
+                    },
                     skipEmptyText: true,
                     minValue: 1,
                     maxValue: 32,
diff --git a/www/window/VerifyJobEdit.js b/www/window/VerifyJobEdit.js
index 5650ed5c..93396a4d 100644
--- a/www/window/VerifyJobEdit.js
+++ b/www/window/VerifyJobEdit.js
@@ -32,8 +32,38 @@ Ext.define('PBS.window.VerifyJobEdit', {
     viewModel: {
         data: {
             ignoreVerified: true,
+            defaultReadThreads: null,
+            defaultVerifyThreads: null,
         },
     },
+
+    loadThreadDefaults: function (datastore) {
+        Proxmox.Utils.API2Request({
+            url: `/api2/extjs/config/datastore/${encodeURIComponent(datastore)}`,
+            method: 'GET',
+            success: ({ result }) => {
+                let raw = result?.data?.tuning || {};
+                let tuning = PBS.Utils.parsePropertyString(raw);
+
+                this.getViewModel().set({
+                    defaultReadThreads: tuning['default-verification-readers'] ?? 1,
+                    defaultVerifyThreads: tuning['default-verification-workers'] ?? 4,
+                });
+            },
+        });
+    },
+
+    initComponent: function () {
+        let me = this;
+        me.callParent();
+
+        if (me.datastore === undefined) {
+            return;
+        }
+
+        me.loadThreadDefaults(me.datastore);
+    },
+
     controller: {
         xclass: 'Ext.app.ViewController',
         control: {
@@ -46,6 +76,7 @@ Ext.define('PBS.window.VerifyJobEdit', {
             let view = this.getView();
             let nsSelector = view.down('pbsNamespaceSelector');
             nsSelector.setDatastore(value);
+            view.loadThreadDefaults(value)
         },
     },
 
@@ -164,6 +195,9 @@ Ext.define('PBS.window.VerifyJobEdit', {
                 cbind: {
                     deleteEmpty: '{!isCreate}',
                 },
+                bind: {
+                    emptyText: '{defaultReadThreads}',
+                },
             },
             {
                 xtype: 'proxmoxintegerfield',
@@ -175,6 +209,9 @@ Ext.define('PBS.window.VerifyJobEdit', {
                 cbind: {
                     deleteEmpty: '{!isCreate}',
                 },
+                bind: {
+                    emptyText: '{defaultVerifyThreads}',
+                },
             },
         ],
         advancedColumn2: [
-- 
2.47.3


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] [PATCH proxmox-backup v2 6/6] docs: tuning: document default-verification-{workers, readers}

[Nicolas Frey]

Signed-off-by: Nicolas Frey <n.frey@proxmox.com>
---
New in v2

 docs/storage.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/docs/storage.rst b/docs/storage.rst
index f1218d9b..672091f8 100644
--- a/docs/storage.rst
+++ b/docs/storage.rst
@@ -618,6 +618,11 @@ There are some tuning related options for the datastore that are more advanced:
   cache slots, 1048576 (= 1024 * 1024) being the default, 8388608 (= 8192 *
   1024) the maximum value.
 
+* ``default-verification-workers`` and ``default-verification-readers``:
+  Define the default number of threads used for verification and reading of chunks,
+  respectively. By default, 4 threads are used for verification and 1 thread is used
+  for reading.
+
 If you want to set multiple tuning options simultaneously, you can separate them
 with a comma, like this:
 
-- 
2.47.3


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox{, -backup} v2 0/7] tuning: verify: add verify job thread options

[Christian Ebner]

Gave this another spin, LGTM although a second pair of eyes, especially 
for the ui patches would be appreciated.

Consider:

Reviewed-by: Christian Ebner <c.ebner@proxmox.com>
Tested-by: Christian Ebner <c.ebner@proxmox.com>


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] applied: [PATCH proxmox{, -backup} v2 0/7] tuning: verify: add verify job thread options

[Thomas Lamprecht]

On Mon, 24 Nov 2025 12:04:20 +0100, Nicolas Frey wrote:
> this patch series aims to expose configuration of the number of threads
> to use for reading and verifying of verify jobs through the datastore's
> tuning options.
> 
> It expands on the series I sent last week [0], where @Thomas
> suggested making this configurable through the tuning options too.
> @Chris suggested adding a dedicated worker thread settings to the
> datastore instead, which is an approach I don't have a problem with,
> but I'd want some more opinions on.
> 
> [...]

Applied, thanks!

[1/1] pbs-api-types: add default-verification-{workers, readers} to tuning options
      commit: 9be1e40f56224efd385d851ad1ac74070a631307
[1/6] datastore: add new thread settings to tuning options
      commit: 568e11a4913629f934e95e5b1f38159a2d506581
[2/6] verify: use tuning options' thread settings for {verify, read}-threads
      commit: 2165ea5b2023312c67728ae0d2d223ce758d6971
[3/6] ui: tuning: make default-verification-{workers, readers} configurable
      commit: ea62a3bc7572253427faec38911e02796d0acc0b
[4/6] ui: tuning: render verification threads in tuning options
      commit: 23262e2bfa182bc80437665ad20865f02a5d8d94
[5/6] ui: verify: fetch default {read, verify} thread values from tuning options
      commit: 88d419d2cee3e91d5b1a55f2d163ded1b0591db6
[6/6] docs: tuning: document default-verification-{workers, readers}
      commit: a3302af5d10a7adc3fa80e228f3715672583e80e


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel