[PATCH proxmox{,-backup} v4 00/14] keep datastore config unlocked during long running operations

[Christian Ebner <c.ebner@proxmox.com>]

Currently an s3-refresh for datastores backed by s3, mount and
unmount for removalbe datastores as well as datastore creation
(independent of datastore type) run while holding the datastore
config lock for consistency.
However, this also blocks config access by all unrelated datastores.

Instead, implement per-datastore locks for maintenance-mode changes,
and rely on block unexpected maintenance-mode changes during ongoing
operations. This allows to concurrently create datastores, run s3
refresh and perform mount/unmount without blocking config access longer
than necessary.

Based on a report in the community forum: https://forum.proxmox.com/threads/183244/

Changes since version 3 (thanks @Robert for comments):
- Refactor datastore creation validation checks and combine logic into common
  helpers, so datastore config checks and operations happen independent from the
  datastore creation.
- Add MaintenanceModeLock wrapper type to clearly distinguish from datastore
  lock.
- Extend file lock inode check to also check for device number.
- Limit DATASTORE_LOCK_DIR to pub(crate).

Changes since version 2:
- Rebased onto current master

Changes since version 1 (thanks @Fabian for review):
- Heavily reworked locking helpers for s3-refresh and mount/unmount operations
- Include new maintenance type create, used to unblock concurrent datastore creation
- Refactor and code cleanup, especially with respect to chunk store scope restriction


proxmox:

Christian Ebner (1):
  pbs-api-types: add datastore create maintenance-mode type

 pbs-api-types/src/datastore.rs   | 8 ++++++++
 pbs-api-types/src/maintenance.rs | 4 ++++
 2 files changed, 12 insertions(+)


proxmox-backup:

Christian Ebner (13):
  api: config: early perform user access checks for datastore creation
  api: config: unlocked s3 bucket access check for datastore creation
  api: config: rearrange independent code block for datastore creation
  api/datastore: refactor datastore creation helper logic
  datastore: restrict chunk store scope to pbs-datastore crate
  datastore: move lock files base path constant to central location
  datastore: move file lock helper to centralized place
  datastore: add additional check for device number on lock helper
  datastore: create lockdir with correct mode for backup user access
  api/datastore: use maintenance-mode lock to protect against changes
  pbs-config: add helper to check new datastore config sections
  api: datastore: move prune job and s3 backend logic to create helper
  datastore: protect datastore creation by maintenance-mode

 pbs-config/src/datastore.rs                   |  45 +++-
 pbs-datastore/src/backup_info.rs              |  36 +--
 pbs-datastore/src/chunk_store.rs              |  22 +-
 pbs-datastore/src/datastore.rs                | 129 +++++++++-
 pbs-datastore/src/lib.rs                      |  78 +++++-
 .../src/local_datastore_lru_cache.rs          |   2 +-
 pbs-datastore/src/move_journal.rs             |   2 +-
 proxmox-backup-client/src/main.rs             |  15 +-
 src/api2/admin/datastore.rs                   |  65 ++---
 src/api2/config/datastore.rs                  | 229 ++++--------------
 src/api2/helpers.rs                           |  78 +++++-
 src/api2/node/disks/directory.rs              |  19 +-
 src/api2/node/disks/zfs.rs                    |  21 +-
 13 files changed, 438 insertions(+), 303 deletions(-)


Summary over all repositories:
  15 files changed, 450 insertions(+), 303 deletions(-)

-- 
Generated by murpp 0.11.0