Re: [pbs-devel] [PATCH proxmox{, -backup} v9 00/49] fix #2943: S3 storage backend for datastores

["Hannes Laimer" <h.laimer@proxmox.com>]

This is a really cool feature, and a very well put together series!
So LGTM, other than the missed `.unwrap()`, consider this

Reviewed-by: Hannes Laimer <h.laimer@proxmox.com>

On Sat Jul 19, 2025 at 2:49 PM CEST, Christian Ebner wrote:
> Disclaimer: These patches are still in an experimental state and not
> intended for production use.
>
> This patch series aims to add S3 compatible object stores as storage
> backend for PBS datastores. A PBS local cache store using the regular
> datastore layout is used for faster operation, bypassing requests to
> the S3 api when possible. Further, the local cache store allows to
> keep frequently used chunks and is used to avoid expensive metadata
> updates on the object store, e.g. by using local marker file during
> garbage collection.
>
> Backups are created by upload chunks to the corresponding S3 bucket,
> while keeping the index files in the local cache store, on backup
> finish, the snapshot metadata are persisted to the S3 storage backend.
>
> Snapshot restores read chunks preferably from the local cache store,
> downloading and insterting them if not present from the S3 object
> store. Listing and snapsoht metadata operation currently rely soly on
> the local cache store.
>
> Currently chunks use a 1:1 mapping to S3 objects. An advanced packing
> mechanism for chunks to significantly reduce the number of api
> requests and therefore be more cost effective will be implemented as
> followup patches.
>
> Most notably changes since version 8 of the patches (thanks @Lukas for
> code review):
> - Moved s3 refresh button to the datastore content, and made it less
>   visible by placing it into a 'More' dropdown.
> - Added basic unit tests for s3 object key helpers
> - Extend missing public function and type documentation
> - Use pbs_buildcfg::configdir macro to build config paths
> - Mostly refactoring based on feedback, please refer to the per-patch
>   changelog for details.
> - Fixes typos in docs (thanks @Maximiliano)
>
> Most notably changes since version 7 of the patches (thanks @Thomas
> and @Lukas for feedback, testing and debugging):
> - Improve self-signed certificate fingerprint check, verify valid
>   expected fingerprint is passed to client on instantiation.
> - Rename previously missed host to endpoint is s3 client selector
> - Use more specific `S3 Client ID` over ambiguous `Unique Identifier`
> - Implement missing cli commands for s3 client manipulation
> - Add in-use marker to s3 object stores to avoid accitental reuse of
>   object stores which are already used as datastore by another
>   instance, adding also flags to overwrite.
> - Automatically perform an s3-refresh when recreating a datastore,
>   pre-populating the contents without further user interaction.
> - Add documentation
> - Fix formatting issue in proxmox-s3-client
>
> Most notably changes since version 6 of the patches (thanks @Thomas
> for feedback):
> - Reworked uri encoding logic, instead of doing this in the
>   S3ObjectKey, perform this in the build_uri helper used by all
>   client api requests.
> - Add cache-size optional parameter to datastore backend config,
>   allows to define the local datastore LRU cache capacity.
> - Increase s3 client timeout, as otherwise delete objects operations
>   on Cloudflare R2 would run into a timeout error.
> - Also upload client log, previously not uploaded to s3 backend.
> - Add missing documentation to some pub types in the response reader
> - Use s3 object key generation helper for index file upload, which
>   fixes the missing key prefix.
> - Add basic regression tests for uri encoder and decoder helper
>   functions.
> - Include some baseline performance tests for garbage collection as
>   well as chunk up-/download when caching.
>
> Most notably changes since version 5 of the patches (thanks @Thomas
> for feedback):
> - Move s3 client into its own, dedicated crate in the proxmox repo
> - Factor out any directly PBS related code from the client
> - Guard implementation behind feature cfg, so api types can be used
>   independently
> - Add basic example and extend on crate documentation
>
> Most notably changes since version 4 of the patches:
> - Fix race between S3 backend upload and local cache store insert,
>   avoiding possibly chunk loss for concurrent backups.
> - Use the local datastore cache also for local chunk reader instances
> - Fallback to fetching chunks from S3 backend if they should be cached
>   but the local chunk file is missing or empty, instead of failing
> - Rename chunks detected as corrupt also on the S3 object store
> - Retry chunk uploads via put objects in case of errors.
> - Add possibility to add rate limits for the s3 client put requests, as
>   otherwise object stores can be overloaded.
> - Allow for Cloudflare R2 compatible `auto` region, as otherwise AWS
>   sign v4 request authentication will fail
> - Use `Async` instead of `Sync` variant for the api handler of the
>   s3-refresh command, as otherwise this fails.
> - Take into account that some type folders might not be present when
>   performing an s3-refresh.
> - Use `Local` instead of `Regular` to refer to normal datastores in the
>   creation window.
>
> Most notably changes since version 3 of the patches:
> - Rebased onto current master, fixed incompatibilities with upgraded
>   dependencies
> - Added method to uri decode s3 object keys, as they are required in
>   order to download contents to a local store
> - Added api endpoint to allow resyncing of the datastore contents to
>   the local cache store, introducing a new maintenance mode s3-refresh
>   to guarantee consistency.
>
> Most notably changes since RFC version 2 of the patches (thanks
> @Lukas for feedback):
> - Extend S3 client implementation to also support path style bucket
>   addressing.
> - Keep bucket name as config option for the datastore, allowing more
>   flexible reuse of a configured S3 client.
> - Use the datastore name as additional object key prefix to allow for
>   multiple datastores on the same bucket.
> - Allow bucket and region templating in S3 endpoint, making this more
>   flexible with respect to possible DNS records.
> - Rework datastore create window to be less overloaded.
> - Drop dead code in the S3 client implementation, since tagging and
>   object copying is currently not required.
> - Fix missing locking when deleting chunks from s3 store during
>   garbage collection, avoiding possible chunk loss for concurrent
>   backups.
> - Remove chunks from LRU cache when deleting chunks during garbage
>   collection, avoiding possible chunk loss for concurrent backups.
> - Add dedicated types for object prefix and relative s3 key paths to
>   avoid misuse.
> - Use more fitting icon for S3 client.
>
> Link to the bugtracker issue:
> https://bugzilla.proxmox.com/show_bug.cgi?id=2943
>
> Steps to setup a local S3 object store using RADOS gateway or MinIO
> can be found at (internal only, external users might use the steps
> outlined in the cover letter and comments of RFC version 2):
> https://wiki.intra.proxmox.com/PBS_Setup_S3_Object_Store
>
> proxmox:
>
> Christian Ebner (3):
>   pbs-api-types: extend datastore config by backend config enum
>   pbs-api-types: maintenance: add new maintenance mode S3 refresh
>   s3 client: wrap upload with retry into dedicated methods
>
>  Cargo.toml                       |   1 +
>  pbs-api-types/Cargo.toml         |   1 +
>  pbs-api-types/debian/control     |   2 +
>  pbs-api-types/src/datastore.rs   | 114 ++++++++++++++++++++++++++++++-
>  pbs-api-types/src/maintenance.rs |   4 ++
>  proxmox-s3-client/src/client.rs  |  32 ++++++++-
>  6 files changed, 151 insertions(+), 3 deletions(-)
>
>
> proxmox-backup:
>
> Christian Ebner (46):
>   datastore: add helpers for path/digest to s3 object key conversion
>   config: introduce s3 object store client configuration
>   api: config: implement endpoints to manipulate and list s3 configs
>   api: datastore: check s3 backend bucket access on datastore create
>   api/cli: add endpoint and command to check s3 client connection
>   datastore: allow to get the backend for a datastore
>   api: backup: store datastore backend in runtime environment
>   api: backup: conditionally upload chunks to s3 object store backend
>   api: backup: conditionally upload blobs to s3 object store backend
>   api: backup: conditionally upload indices to s3 object store backend
>   api: backup: conditionally upload manifest to s3 object store backend
>   api: datastore: conditionally upload client log to s3 backend
>   sync: pull: conditionally upload content to s3 backend
>   api: reader: fetch chunks based on datastore backend
>   datastore: local chunk reader: read chunks based on backend
>   verify worker: add datastore backed to verify worker
>   verify: implement chunk verification for stores with s3 backend
>   datastore: create namespace marker in s3 backend
>   datastore: create/delete protected marker file on s3 storage backend
>   datastore: prune groups/snapshots from s3 object store backend
>   datastore: get and set owner for s3 store backend
>   datastore: implement garbage collection for s3 backend
>   ui: add datastore type selector and reorganize component layout
>   ui: add s3 client edit window for configuration create/edit
>   ui: add s3 client view for configuration
>   ui: expose the s3 client view in the navigation tree
>   ui: add s3 client selector and bucket field for s3 backend setup
>   tools: lru cache: add removed callback for evicted cache nodes
>   tools: async lru cache: implement insert, remove and contains methods
>   datastore: add local datastore cache for network attached storages
>   api: backup: use local datastore cache on s3 backend chunk upload
>   api: reader: use local datastore cache on s3 backend chunk fetching
>   datastore: local chunk reader: get cached chunk from local cache store
>   backup writer: refactor parameters into backup writer options struct
>   api: backup: add no-cache flag to bypass local datastore cache
>   api/datastore: implement refresh endpoint for stores with s3 backend
>   cli: add dedicated subcommand for datastore s3 refresh
>   ui: render s3 refresh as valid maintenance type and task description
>   ui: expose s3 refresh button for datastores backed by object store
>   datastore: conditionally upload atime marker chunk to s3 backend
>   bin: implement client subcommands for s3 configuration manipulation
>   bin: expose reuse-datastore flag for proxmox-backup-manager
>   datastore: mark store as in-use by setting marker on s3 backend
>   datastore: run s3-refresh when reusing a datastore with s3 backend
>   api/ui: add flag to allow overwriting in-use marker for s3 backend
>   docs: Add section describing how to setup s3 backed datastore
>
>  Cargo.toml                                    |   2 +
>  docs/storage.rst                              |  68 ++
>  examples/upload-speed.rs                      |  17 +-
>  pbs-client/src/backup_writer.rs               |  47 +-
>  pbs-config/Cargo.toml                         |   1 +
>  pbs-config/src/lib.rs                         |   1 +
>  pbs-config/src/s3.rs                          |  89 +++
>  pbs-datastore/Cargo.toml                      |   5 +
>  pbs-datastore/src/backup_info.rs              |  63 +-
>  pbs-datastore/src/cached_chunk_reader.rs      |   6 +-
>  pbs-datastore/src/chunk_store.rs              |  29 +-
>  pbs-datastore/src/datastore.rs                | 688 ++++++++++++++++--
>  pbs-datastore/src/dynamic_index.rs            |   1 +
>  pbs-datastore/src/lib.rs                      |   5 +
>  pbs-datastore/src/local_chunk_reader.rs       |  61 +-
>  .../src/local_datastore_lru_cache.rs          | 180 +++++
>  pbs-datastore/src/s3.rs                       | 114 +++
>  pbs-tools/src/async_lru_cache.rs              |  46 +-
>  pbs-tools/src/lru_cache.rs                    |  42 +-
>  proxmox-backup-client/src/benchmark.rs        |  17 +-
>  proxmox-backup-client/src/main.rs             |  26 +-
>  src/api2/admin/datastore.rs                   |  85 ++-
>  src/api2/admin/mod.rs                         |   2 +
>  src/api2/admin/s3.rs                          |  82 +++
>  src/api2/backup/environment.rs                |  82 ++-
>  src/api2/backup/mod.rs                        | 131 ++--
>  src/api2/backup/upload_chunk.rs               | 114 ++-
>  src/api2/config/datastore.rs                  | 147 +++-
>  src/api2/config/mod.rs                        |   2 +
>  src/api2/config/s3.rs                         | 310 ++++++++
>  src/api2/node/disks/directory.rs              |   2 +-
>  src/api2/node/disks/zfs.rs                    |   2 +-
>  src/api2/reader/environment.rs                |  12 +-
>  src/api2/reader/mod.rs                        |  62 +-
>  src/backup/verify.rs                          | 134 +++-
>  src/bin/proxmox-backup-manager.rs             |   1 +
>  src/bin/proxmox_backup_manager/datastore.rs   |  42 ++
>  src/bin/proxmox_backup_manager/mod.rs         |   2 +
>  src/bin/proxmox_backup_manager/s3.rs          | 102 +++
>  src/server/pull.rs                            |  68 +-
>  src/server/push.rs                            |  19 +-
>  src/server/verify_job.rs                      |   2 +-
>  www/Makefile                                  |   3 +
>  www/NavigationTree.js                         |   6 +
>  www/Utils.js                                  |   4 +
>  www/config/S3ClientView.js                    | 141 ++++
>  www/datastore/Content.js                      |  48 ++
>  www/form/S3ClientSelector.js                  |  33 +
>  www/window/DataStoreEdit.js                   | 132 +++-
>  www/window/MaintenanceOptions.js              |   6 +-
>  www/window/S3ClientEdit.js                    | 148 ++++
>  51 files changed, 3104 insertions(+), 328 deletions(-)
>  create mode 100644 pbs-config/src/s3.rs
>  create mode 100644 pbs-datastore/src/local_datastore_lru_cache.rs
>  create mode 100644 pbs-datastore/src/s3.rs
>  create mode 100644 src/api2/admin/s3.rs
>  create mode 100644 src/api2/config/s3.rs
>  create mode 100644 src/bin/proxmox_backup_manager/s3.rs
>  create mode 100644 www/config/S3ClientView.js
>  create mode 100644 www/form/S3ClientSelector.js
>  create mode 100644 www/window/S3ClientEdit.js
>
>
> Summary over all repositories:
>   57 files changed, 3255 insertions(+), 331 deletions(-)



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel