[PATCH proxmox-backup v3 00/15] fix 7642: avoid expensive uid/gid lookups for lock- and config-files

[Christian Ebner <c.ebner@proxmox.com>]

As reported by a user in the community forum [0] and the thereof
resulting bugzilla entry [1], phase 2 of garbage collection currently
produces avoidable syscalls and parsing overhead, reading
/etc/nsswitch.conf and /etc/passwd for each per-chunk file locking
call. This is however not limited to garbage collection, but rather
affects file locking in general. The backup_user() lookup was
identified as the culprit.

This patch series fixes this by switching to proxmox-product-config
helpers for config file operations and file locks instead, which does
use the user as parsed once on process init.

[0] https://forum.proxmox.com/threads/s3-gc-phase-2-nsswitch-passwd-overhead.183709/
[1] https://bugzilla.proxmox.com/show_bug.cgi?id=7642

Change since version 2:
- Extend previous patch series by refactoring also other callers of
  pbs_config::backup_user() and use proxmox-product-config helpers
  where possible

Change since version 1 (thanks @Fabian for feedback):
- Instead of fixing this for S3 file locking only, do it for all
  file locks using the open_backup_lockfile().
- Use proxmox-product-config helpers and init user as required
Note: Above changes significantly escalated the scope of the patches,
as they do now touch any config update as well and in the calling
process the init() must have been called once before.


proxmox-backup:

Christian Ebner (15):
  bin: api: early init proxmox-product-config
  bin: daily update: refactor to use proxmox-product-config
  pbs-config: use proxmox-product-config::replace_secret_config()
  pbs-config: use proxmox-product-config::replace_config()
  fix #7642: avoid expensive user lookups on file locking
  pbs-config: use proxmox-product-config helpers
  pbs-config: drop backup_group helper, use users gid instead
  pbs-datastore: use proxmox-product-config cached backup user
  pbs-datastore: use general helpers for file lock create options
  server: auth helpers: use proxmox-product-config create options
    helpers
  api: subscription: use proxmox-product-config create options
  tape: use proxmox-product-config helper for user lookup
  tape: use proxmox-product-config lock file create options
  tape: use proxmox-product-config to generate create options
  tree-wide: use proxmox-product-config::get_api_user for user lookup

 pbs-config/Cargo.toml                         |  1 +
 pbs-config/src/acl.rs                         |  5 +-
 pbs-config/src/config_version_cache.rs        |  2 +-
 pbs-config/src/datastore.rs                   |  5 +-
 pbs-config/src/domains.rs                     |  5 +-
 pbs-config/src/drive.rs                       |  5 +-
 pbs-config/src/encryption_keys.rs             |  9 +--
 pbs-config/src/lib.rs                         | 56 ++-----------------
 pbs-config/src/media_pool.rs                  |  5 +-
 pbs-config/src/metrics.rs                     |  3 +-
 pbs-config/src/node.rs                        |  3 +-
 pbs-config/src/notifications.rs               |  8 ++-
 pbs-config/src/prune.rs                       |  5 +-
 pbs-config/src/remote.rs                      |  3 +-
 pbs-config/src/s3.rs                          |  5 +-
 pbs-config/src/sync.rs                        |  5 +-
 pbs-config/src/tape_job.rs                    |  5 +-
 pbs-config/src/token_shadow.rs                |  8 +--
 pbs-config/src/traffic_control.rs             |  5 +-
 pbs-config/src/user.rs                        |  5 +-
 pbs-config/src/verify.rs                      |  5 +-
 pbs-datastore/Cargo.toml                      |  1 +
 pbs-datastore/src/chunk_store.rs              | 27 +++++----
 pbs-datastore/src/datastore.rs                | 17 +++---
 pbs-datastore/src/move_journal.rs             | 11 +---
 pbs-datastore/src/task_tracking.rs            |  7 +--
 src/api2/admin/datastore.rs                   |  2 +-
 src/api2/admin/s3.rs                          |  4 +-
 src/api2/config/datastore.rs                  |  2 +-
 src/api2/config/s3.rs                         |  2 +-
 src/api2/node/subscription.rs                 | 11 +---
 src/auth_helpers.rs                           | 27 ++-------
 src/bin/proxmox-backup-api.rs                 |  5 +-
 src/bin/proxmox-backup-proxy.rs               |  5 +-
 src/bin/proxmox-daily-update.rs               | 12 ++--
 src/bin/proxmox-tape.rs                       |  8 +++
 .../proxmox_backup_manager/subscription.rs    |  4 +-
 src/bin/sg-tape-cmd.rs                        |  5 +-
 src/config/mod.rs                             |  9 +--
 src/server/jobstate.rs                        |  4 +-
 src/server/metric_collection/mod.rs           |  2 +-
 src/server/metric_collection/pull_metrics.rs  |  2 +-
 src/server/metric_collection/rrd.rs           |  2 +-
 src/server/mod.rs                             |  6 +-
 src/server/notifications/mod.rs               |  4 +-
 src/tape/changer/mod.rs                       |  2 +-
 src/tape/drive/mod.rs                         | 11 +---
 src/tape/encryption_keys.rs                   |  5 +-
 src/tape/inventory.rs                         |  6 +-
 src/tape/media_catalog.rs                     | 14 ++---
 src/tape/media_catalog_cache.rs               | 10 +---
 src/tape/mod.rs                               |  8 +--
 src/tape/test/alloc_writable_media.rs         | 17 ++++++
 src/traffic_control_cache.rs                  |  2 +-
 54 files changed, 171 insertions(+), 236 deletions(-)


Summary over all repositories:
  54 files changed, 171 insertions(+), 236 deletions(-)

-- 
Generated by murpp 0.11.0