[PATCH proxmox{,-backup} v3 00/30] fix #7251: implement server side encryption support for push sync jobs

[Christian Ebner <c.ebner@proxmox.com>]

This patch series implements support for encrypting backup snapshots
when pushing from a source PBS instance to an untrusted remote target
PBS instance. Further, it adds support to decrypt snapshots being
encrypted on the remote source PBS when pulling the contents to the
local target PBS instance. This allows to perform full server side
encryption/decryption when syncing with a less trusted remote PBS.

In order to encrypt/decrypt snapshots, a new encryption key entity
is introduced, to be created as global instance on the PBS, placed and
managed by it's own dedicated config. Keys with secret are stored
in dedicated files so they only need to be loaded when accessing the
key, not for listing of configuration. Sync encryption keys can be
archived, rendering them no longer usable to encrypt new contents,
but still allowing to decrypt. In order to remove a sync encryption
key, it must be archived first and no longer associated to any
sync job config, a constrained added as safety net to avoid accidental
key removal.
The same centralized key management is also used for tape encryption
keys, so they are on-par ui wise, the configs remain however separated
for the time being.

The sync jobs in push direction are extended to receive an additional
active encryption key parameter, which will be used to encrypt
unencrypted snapshot when pushing to the remote target.
A list of associated keys is kept, adding the previous encryption key
of the push sync job if the key is rotated.
For pull sync jobs, the active encryption key parameter is not
considered, rather all associated keys will be loaded and used to
decrypt snapshots with matching fingerprint as found in the source
manifest. In order to encrypt/decrypt the contents, chunks, index
files, blobs and manifest are additionally processed, rewritten when
required.

Changes since version 2 (thanks a lot to @Thomas for review):
- Add dedicated lock file for per-key file locks when creating/deleting sync
  keys.
- Add initial documentation for server side encryption/decription during sync
  jobs.
- Adapt key archive endpoint to be able to toggle, kept as dedicated patch as
  unsure about impl details.
- Early detect unusable keys provided on key creation as upload via api.
- List all associated sync jobs when checking with encryption_key_in_use().
- Fix check for key access when setting active encryption key. It must fail for
  archived keys.
- Add flag to check for not allowing to set archived key as active encryption
  key.
- Drop associated keys also on active encryption key update, readd rotated one
  afterwards if required.
- Refactor check for un-/partially-/fully-encrypted backup snapshots.
- Include snapshot name in log message for skipped snapshots.
- Add missing return on error when requesting key archivation for tape.
- Handle errors for api calls to load tape and sync keys in ui by wrapping into
  try-catch-block.
- Also drop verify state on pull, do not rely on inherent check to better
  protect against bugs and corruptions.
- Awitch field label for associated keys based on sync direction.
- Add comment field explaining active encryption key and associated keys and
  their relation on key rotation.
- Also store key id together with key config when loading associated keys, so it
  can be logged later when key fingerprint matched.
- Squash new manifest registration into patch 26, keeping logic together
- Fix boguous check, must use change-detection-fingerprint, not key-fingerprint
  to detect changes on already existing manifest.
- Convert unprotected manifest part to json value to drop key-fingerprint.
- Log id of key used for decryption, not just fingerprint
- Switch all remaining `log` macros for sync to use `tracing`.
- Fix typos in commit message for async DataBlob reader patch.
- Double column width for `hint` field.
- Fix icons for type based menu buttons and type column
- Drop dead code `crypt-key-fp`.
- Fix error messages by s/seems/seem/ and wrap in gettext()
- Document config lock requirements for delete_key().
- Drop outdated comment on key file lock drop, it's a dedicated file now.

Changes since version 1 (thanks a lot to @all reviewers/testers!):
- Implement encryption key archiving and key rotation logic, allowing
  to specify active encryption key for push syncs, and a list of
  previously used ones. For pull multiple decryption keys can now be
  configured.
- Rework the UI to add support for key archiving, manage key association
  in sync jobs and to also manage tape encryption keys in the same
  centralized grid.
- Check for key still being in-use by sync job before removing it
- Fully encrypted snapshots are now pushed as-is if an encryption key
  is configured.
- Fixed inefficient resync of pre-existing target snapshot on pull,
  detect file changes in manifest via fingerprinting.
- Avoid overwriting pre-existing decrypted local snapshot by encrypted
  snapshot when no (or mismatching) decryption key is passed for pull
  job.
- Rename EncryptionKey to CyrptKey, as the key is also used for
  decryption.
- Remove key from config before removing keyfile
- Add locking mechansism to avoid races in key config writing
- Fix gathering of known chunks from previous snapshot in push for
  dynamic index files
- Detect config changes by checking for digest mismatch
- Guard key loading by PRIV_SYS_MODIFY
- Use tracing::info! instead of log::info!
- Fix clearing of encryption/decryption key via sync job config window
- Fix creating new sync job without crypt key configured
- Check key exists and can be accessed when set in sync job
- Fix min key id length for key edit window
- Fixed drag-and-drop for key file upload
- Fix outdated comments, typos, ecc.

Link to the bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=7251


proxmox:

Christian Ebner (2):
  pbs-api-types: define en-/decryption key type and schema
  pbs-api-types: sync job: add optional cryptographic keys to config

 pbs-api-types/src/jobs.rs           | 21 ++++++++++++++--
 pbs-api-types/src/key_derivation.rs | 38 ++++++++++++++++++++++++++---
 pbs-api-types/src/lib.rs            |  2 +-
 3 files changed, 55 insertions(+), 6 deletions(-)


proxmox-backup:

Christian Ebner (28):
  sync: push: use tracing macros instead of log
  datastore: blob: implement async reader for data blobs
  datastore: manifest: add helper for change detection fingerprint
  pbs-key-config: introduce store_with() for KeyConfig
  pbs-config: implement encryption key config handling
  pbs-config: acls: add 'encryption-keys' as valid 'system' subpath
  ui: expose 'encryption-keys' as acl subpath for 'system'
  sync: add helper to check encryption key acls and load key
  api: config: add endpoints for encryption key manipulation
  api: config: check sync owner has access to en-/decryption keys
  api: config: allow encryption key manipulation for sync job
  sync: push: rewrite manifest instead of pushing pre-existing one
  api: push sync: expose optional encryption key for push sync
  sync: push: optionally encrypt data blob on upload
  sync: push: optionally encrypt client log on upload if key is given
  sync: push: add helper for loading known chunks from previous snapshot
  fix #7251: api: push: encrypt snapshots using configured encryption
    key
  ui: define and expose encryption key management menu item and windows
  ui: expose assigning encryption key to sync jobs
  sync: pull: load encryption key if given in job config
  sync: expand source chunk reader trait by crypt config
  sync: pull: introduce and use decrypt index writer if crypt config
  sync: pull: extend encountered chunk by optional decrypted digest
  sync: pull: decrypt blob files on pull if encryption key is configured
  sync: pull: decrypt chunks and rewrite index file for matching key
  sync: pull: decrypt snapshots with matching encryption key fingerprint
  api: encryption keys: allow to toggle the archived state for keys
  docs: add section describing server side encryption for sync jobs

 docs/managing-remotes.rst          |  49 +++
 pbs-config/Cargo.toml              |   2 +
 pbs-config/src/acl.rs              |   4 +-
 pbs-config/src/encryption_keys.rs  | 217 ++++++++++++++
 pbs-config/src/lib.rs              |   1 +
 pbs-datastore/src/data_blob.rs     |  18 +-
 pbs-datastore/src/manifest.rs      |  20 ++
 pbs-key-config/src/lib.rs          |  36 ++-
 src/api2/config/encryption_keys.rs | 219 ++++++++++++++
 src/api2/config/mod.rs             |   2 +
 src/api2/config/sync.rs            |  94 +++++-
 src/api2/pull.rs                   |  15 +-
 src/api2/push.rs                   |   8 +-
 src/server/pull.rs                 | 459 ++++++++++++++++++++++++-----
 src/server/push.rs                 | 311 ++++++++++++++-----
 src/server/sync.rs                 |  58 +++-
 www/Makefile                       |   3 +
 www/NavigationTree.js              |   6 +
 www/Utils.js                       |   1 +
 www/config/EncryptionKeysView.js   | 346 ++++++++++++++++++++++
 www/form/EncryptionKeySelector.js  |  96 ++++++
 www/form/PermissionPathSelector.js |   1 +
 www/window/EncryptionKeysEdit.js   | 382 ++++++++++++++++++++++++
 www/window/SyncJobEdit.js          |  62 ++++
 24 files changed, 2248 insertions(+), 162 deletions(-)
 create mode 100644 pbs-config/src/encryption_keys.rs
 create mode 100644 src/api2/config/encryption_keys.rs
 create mode 100644 www/config/EncryptionKeysView.js
 create mode 100644 www/form/EncryptionKeySelector.js
 create mode 100644 www/window/EncryptionKeysEdit.js


Summary over all repositories:
  27 files changed, 2303 insertions(+), 168 deletions(-)

-- 
Generated by murpp 0.11.0