public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH proxmox-backup v3 00/15] fix 7642: avoid expensive uid/gid lookups for lock- and config-files
@ 2026-07-01 14:03 Christian Ebner
  2026-07-01 14:03 ` [PATCH proxmox-backup v3 01/15] bin: api: early init proxmox-product-config Christian Ebner
                   ` (14 more replies)
  0 siblings, 15 replies; 16+ messages in thread
From: Christian Ebner @ 2026-07-01 14:03 UTC (permalink / raw)
  To: pbs-devel

As reported by a user in the community forum [0] and the thereof
resulting bugzilla entry [1], phase 2 of garbage collection currently
produces avoidable syscalls and parsing overhead, reading
/etc/nsswitch.conf and /etc/passwd for each per-chunk file locking
call. This is however not limited to garbage collection, but rather
affects file locking in general. The backup_user() lookup was
identified as the culprit.

This patch series fixes this by switching to proxmox-product-config
helpers for config file operations and file locks instead, which does
use the user as parsed once on process init.

[0] https://forum.proxmox.com/threads/s3-gc-phase-2-nsswitch-passwd-overhead.183709/
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7642

Change since version 2:
- Extend previous patch series by refactoring also other callers of
  pbs_config::backup_user() and use proxmox-product-config helpers
  where possible

Change since version 1 (thanks @Fabian for feedback):
- Instead of fixing this for S3 file locking only, do it for all
  file locks using the open_backup_lockfile().
- Use proxmox-product-config helpers and init user as required
Note: Above changes significantly escalated the scope of the patches,
as they do now touch any config update as well and in the calling
process the init() must have been called once before.


proxmox-backup:

Christian Ebner (15):
  bin: api: early init proxmox-product-config
  bin: daily update: refactor to use proxmox-product-config
  pbs-config: use proxmox-product-config::replace_secret_config()
  pbs-config: use proxmox-product-config::replace_config()
  fix #7642: avoid expensive user lookups on file locking
  pbs-config: use proxmox-product-config helpers
  pbs-config: drop backup_group helper, use users gid instead
  pbs-datastore: use proxmox-product-config cached backup user
  pbs-datastore: use general helpers for file lock create options
  server: auth helpers: use proxmox-product-config create options
    helpers
  api: subscription: use proxmox-product-config create options
  tape: use proxmox-product-config helper for user lookup
  tape: use proxmox-product-config lock file create options
  tape: use proxmox-product-config to generate create options
  tree-wide: use proxmox-product-config::get_api_user for user lookup

 pbs-config/Cargo.toml                         |  1 +
 pbs-config/src/acl.rs                         |  5 +-
 pbs-config/src/config_version_cache.rs        |  2 +-
 pbs-config/src/datastore.rs                   |  5 +-
 pbs-config/src/domains.rs                     |  5 +-
 pbs-config/src/drive.rs                       |  5 +-
 pbs-config/src/encryption_keys.rs             |  9 +--
 pbs-config/src/lib.rs                         | 56 ++-----------------
 pbs-config/src/media_pool.rs                  |  5 +-
 pbs-config/src/metrics.rs                     |  3 +-
 pbs-config/src/node.rs                        |  3 +-
 pbs-config/src/notifications.rs               |  8 ++-
 pbs-config/src/prune.rs                       |  5 +-
 pbs-config/src/remote.rs                      |  3 +-
 pbs-config/src/s3.rs                          |  5 +-
 pbs-config/src/sync.rs                        |  5 +-
 pbs-config/src/tape_job.rs                    |  5 +-
 pbs-config/src/token_shadow.rs                |  8 +--
 pbs-config/src/traffic_control.rs             |  5 +-
 pbs-config/src/user.rs                        |  5 +-
 pbs-config/src/verify.rs                      |  5 +-
 pbs-datastore/Cargo.toml                      |  1 +
 pbs-datastore/src/chunk_store.rs              | 27 +++++----
 pbs-datastore/src/datastore.rs                | 17 +++---
 pbs-datastore/src/move_journal.rs             | 11 +---
 pbs-datastore/src/task_tracking.rs            |  7 +--
 src/api2/admin/datastore.rs                   |  2 +-
 src/api2/admin/s3.rs                          |  4 +-
 src/api2/config/datastore.rs                  |  2 +-
 src/api2/config/s3.rs                         |  2 +-
 src/api2/node/subscription.rs                 | 11 +---
 src/auth_helpers.rs                           | 27 ++-------
 src/bin/proxmox-backup-api.rs                 |  5 +-
 src/bin/proxmox-backup-proxy.rs               |  5 +-
 src/bin/proxmox-daily-update.rs               | 12 ++--
 src/bin/proxmox-tape.rs                       |  8 +++
 .../proxmox_backup_manager/subscription.rs    |  4 +-
 src/bin/sg-tape-cmd.rs                        |  5 +-
 src/config/mod.rs                             |  9 +--
 src/server/jobstate.rs                        |  4 +-
 src/server/metric_collection/mod.rs           |  2 +-
 src/server/metric_collection/pull_metrics.rs  |  2 +-
 src/server/metric_collection/rrd.rs           |  2 +-
 src/server/mod.rs                             |  6 +-
 src/server/notifications/mod.rs               |  4 +-
 src/tape/changer/mod.rs                       |  2 +-
 src/tape/drive/mod.rs                         | 11 +---
 src/tape/encryption_keys.rs                   |  5 +-
 src/tape/inventory.rs                         |  6 +-
 src/tape/media_catalog.rs                     | 14 ++---
 src/tape/media_catalog_cache.rs               | 10 +---
 src/tape/mod.rs                               |  8 +--
 src/tape/test/alloc_writable_media.rs         | 17 ++++++
 src/traffic_control_cache.rs                  |  2 +-
 54 files changed, 171 insertions(+), 236 deletions(-)


Summary over all repositories:
  54 files changed, 171 insertions(+), 236 deletions(-)

-- 
Generated by murpp 0.11.0




^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2026-07-01 14:05 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-01 14:03 [PATCH proxmox-backup v3 00/15] fix 7642: avoid expensive uid/gid lookups for lock- and config-files Christian Ebner
2026-07-01 14:03 ` [PATCH proxmox-backup v3 01/15] bin: api: early init proxmox-product-config Christian Ebner
2026-07-01 14:03 ` [PATCH proxmox-backup v3 02/15] bin: daily update: refactor to use proxmox-product-config Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 03/15] pbs-config: use proxmox-product-config::replace_secret_config() Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 04/15] pbs-config: use proxmox-product-config::replace_config() Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 05/15] fix #7642: avoid expensive user lookups on file locking Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 06/15] pbs-config: use proxmox-product-config helpers Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 07/15] pbs-config: drop backup_group helper, use users gid instead Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 08/15] pbs-datastore: use proxmox-product-config cached backup user Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 09/15] pbs-datastore: use general helpers for file lock create options Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 10/15] server: auth helpers: use proxmox-product-config create options helpers Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 11/15] api: subscription: use proxmox-product-config create options Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 12/15] tape: use proxmox-product-config helper for user lookup Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 13/15] tape: use proxmox-product-config lock file create options Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 14/15] tape: use proxmox-product-config to generate " Christian Ebner
2026-07-01 14:04 ` [PATCH proxmox-backup v3 15/15] tree-wide: use proxmox-product-config::get_api_user for user lookup Christian Ebner

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal