public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Hannes Duerr <h.duerr@proxmox.com>
To: Proxmox Backup Server development discussion
	<pbs-devel@lists.proxmox.com>,
	Christian Ebner <c.ebner@proxmox.com>
Subject: Re: [pbs-devel] [PATCH proxmox{, -backup} v4 00/48] fix #2943: S3 storage backend for datastores
Date: Tue, 1 Jul 2025 16:30:52 +0200	[thread overview]
Message-ID: <724f995a-0e2e-47e4-9b5f-c1d5ee71df9f@proxmox.com> (raw)
In-Reply-To: <20250623094106.299251-1-c.ebner@proxmox.com>

Hi i am currently testing this series and noticed that you've missed the 
following build dependencies in the debian/control file for the 
proxmox-backup repository:
librust-iso8601-dev,
librust-md5-dev,
librust-quick-xml-dev,
librust-serde-xml-rs-dev,

On 6/23/25 11:40, Christian Ebner wrote:
> Disclaimer: These patches are still in an experimental state and not
> intended for production use.
>
> This patch series aims to add S3 compatible object stores as storage
> backend for PBS datastores. A PBS local cache store using the regular
> datastore layout is used for faster operation, bypassing requests to
> the S3 api when possible. Further, the local cache store allows to
> keep frequently used chunks and is used to avoid expensive metadata
> updates on the object store, e.g. by using local marker file during
> garbage collection.
>
> Backups are created by upload chunks to the corresponding S3 bucket,
> while keeping the index files in the local cache store, on backup
> finish, the snapshot metadata are persisted to the S3 storage backend.
>
> Snapshot restores read chunks preferably from the local cache store,
> downloading and insterting them if not present from the S3 object
> store. Listing and snapsoht metadata operation currently rely soly on
> the local cache store.
>
> Currently chunks use a 1:1 mapping to S3 objects. An advanced packing
> mechanism for chunks to significantly reduce the number of api
> requests and therefore be more cost effective will be implemented as
> followup patches.
>
> Most notably changes since version 3 of the patches:
> - Rebased onto current master, fixed incompatibilities with upgraded
>    dependencies
> - Added method to uri decode s3 object keys, as they are required in
>    order to download contents to a local store
> - Added api endpoint to allow resyncing of the datastore contents to
>    the local cache store, introducing a new maintenance mode s3-refresh
>    to guarantee consistency.
>
> Most notably changes since RFC version 2 of the patches (thanks
> @Lukas for feedback):
> - Extend S3 client implementation to also support path style bucket
>    addressing.
> - Keep bucket name as config option for the datastore, allowing more
>    flexible reuse of a configured S3 client.
> - Use the datastore name as additional object key prefix to allow for
>    multiple datastores on the same bucket.
> - Allow bucket and region templating in S3 endpoint, making this more
>    flexible with respect to possible DNS records.
> - Rework datastore create window to be less overloaded.
> - Drop dead code in the S3 client implementation, since tagging and
>    object copying is currently not required.
> - Fix missing locking when deleting chunks from s3 store during
>    garbage collection, avoiding possible chunk loss for concurrent
>    backups.
> - Remove chunks from LRU cache when deleting chunks during garbage
>    collection, avoiding possible chunk loss for concurrent backups.
> - Add dedicated types for object prefix and relative s3 key paths to
>    avoid misuse.
> - Use more fitting icon for S3 client.
>
> Link to the bugtracker issue:
> https://bugzilla.proxmox.com/show_bug.cgi?id=2943
>
> The previous version 3 of the patch series can be found at:
> https://lore.proxmox.com/pbs-devel/20250616142156.413652-1-c.ebner@proxmox.com/T/
>
> Steps to setup a local S3 object store using RADOS gateway or MinIO
> can be found at (internal only, external users might use the steps
> outlined in the cover letter and comments of RFC version 2):
> https://wiki.intra.proxmox.com/PBS_Setup_S3_Object_Store
>
> proxmox:
>
> Christian Ebner (3):
>    pbs-api-types: add types for S3 client configs and secrets
>    pbs-api-types: extend datastore config by backend config enum
>    pbs-api-types: maintenance: add new maintenance mode S3 refresh
>
>   pbs-api-types/src/datastore.rs   | 103 ++++++++++++++++++++-
>   pbs-api-types/src/lib.rs         |   3 +
>   pbs-api-types/src/maintenance.rs |   4 +
>   pbs-api-types/src/s3.rs          | 154 +++++++++++++++++++++++++++++++
>   4 files changed, 263 insertions(+), 1 deletion(-)
>   create mode 100644 pbs-api-types/src/s3.rs
>
>
> proxmox-backup:
>
> Christian Ebner (45):
>    api: fix minor formatting issues
>    bin: sort submodules alphabetically
>    datastore: ignore missing owner file when removing group directory
>    verify: refactor verify related functions to be methods of worker
>    s3 client: add crate for AWS s3 compatible object store client
>    s3 client: implement AWS signature v4 request authentication
>    s3 client: add dedicated type for s3 object keys
>    s3 client: add type for last modified timestamp in responses
>    s3 client: add helper to parse http date headers
>    s3 client: implement methods to operate on s3 objects in bucket
>    config: introduce s3 object store client configuration
>    api: config: implement endpoints to manipulate and list s3 configs
>    api: datastore: check s3 backend bucket access on datastore create
>    api/cli: add endpoint and command to check s3 client connection
>    datastore: allow to get the backend for a datastore
>    api: backup: store datastore backend in runtime environment
>    api: backup: conditionally upload chunks to s3 object store backend
>    api: backup: conditionally upload blobs to s3 object store backend
>    api: backup: conditionally upload indices to s3 object store backend
>    api: backup: conditionally upload manifest to s3 object store backend
>    sync: pull: conditionally upload content to s3 backend
>    api: reader: fetch chunks based on datastore backend
>    datastore: local chunk reader: read chunks based on backend
>    verify worker: add datastore backed to verify worker
>    verify: implement chunk verification for stores with s3 backend
>    datastore: create namespace marker in S3 backend
>    datastore: create/delete protected marker file on s3 storage backend
>    datastore: prune groups/snapshots from s3 object store backend
>    datastore: get and set owner for S3 store backend
>    datastore: implement garbage collection for s3 backend
>    ui: add datastore type selector and reorganize component layout
>    ui: add S3 client edit window for configuration create/edit
>    ui: add S3 client view for configuration
>    ui: expose the S3 client view in the navigation tree
>    ui: add s3 client selector and bucket field for s3 backend setup
>    tools: lru cache: add removed callback for evicted cache nodes
>    tools: async lru cache: implement insert, remove and contains methods
>    datastore: add local datastore cache for network attached storages
>    api: backup: use local datastore cache on s3 backend chunk upload
>    api: reader: use local datastore cache on s3 backend chunk fetching
>    api: backup: add no-cache flag to bypass local datastore cache
>    api/datastore: implement refresh endpoint for stores with s3 backend
>    cli: add dedicated subcommand for datastore s3 refresh
>    ui: render s3 refresh as valid maintenance type and task description
>    ui: expose s3 refresh button for datastores backed by object store
>
>   Cargo.toml                                    |   8 +
>   examples/upload-speed.rs                      |   1 +
>   pbs-client/src/backup_writer.rs               |   4 +-
>   pbs-config/src/lib.rs                         |   1 +
>   pbs-config/src/s3.rs                          |  82 ++
>   pbs-datastore/Cargo.toml                      |   5 +
>   pbs-datastore/src/backup_info.rs              |  61 +-
>   pbs-datastore/src/cached_chunk_reader.rs      |   6 +-
>   pbs-datastore/src/chunk_store.rs              |   4 +
>   pbs-datastore/src/datastore.rs                | 601 +++++++++++-
>   pbs-datastore/src/dynamic_index.rs            |   1 +
>   pbs-datastore/src/lib.rs                      |   4 +
>   pbs-datastore/src/local_chunk_reader.rs       |  37 +-
>   .../src/local_datastore_lru_cache.rs          | 116 +++
>   pbs-s3-client/Cargo.toml                      |  33 +
>   pbs-s3-client/src/aws_sign_v4.rs              | 174 ++++
>   pbs-s3-client/src/client.rs                   | 549 +++++++++++
>   pbs-s3-client/src/lib.rs                      | 123 +++
>   pbs-s3-client/src/object_key.rs               |  99 ++
>   pbs-s3-client/src/response_reader.rs          | 279 ++++++
>   pbs-tools/src/async_lru_cache.rs              |  46 +-
>   pbs-tools/src/lru_cache.rs                    |  42 +-
>   proxmox-backup-client/src/benchmark.rs        |   1 +
>   proxmox-backup-client/src/main.rs             |   8 +
>   src/api2/admin/datastore.rs                   |  86 +-
>   src/api2/admin/mod.rs                         |   2 +
>   src/api2/admin/s3.rs                          |  80 ++
>   src/api2/backup/environment.rs                | 145 ++-
>   src/api2/backup/mod.rs                        | 136 +--
>   src/api2/backup/upload_chunk.rs               | 108 ++-
>   src/api2/config/datastore.rs                  |  49 +-
>   src/api2/config/mod.rs                        |   2 +
>   src/api2/config/s3.rs                         | 310 ++++++
>   src/api2/reader/environment.rs                |  12 +-
>   src/api2/reader/mod.rs                        |  61 +-
>   src/backup/verify.rs                          | 879 +++++++++---------
>   src/bin/proxmox-backup-manager.rs             |   1 +
>   src/bin/proxmox_backup_manager/datastore.rs   |  30 +
>   src/bin/proxmox_backup_manager/mod.rs         |  30 +-
>   src/bin/proxmox_backup_manager/s3.rs          |  46 +
>   src/server/pull.rs                            |  62 +-
>   src/server/push.rs                            |   1 +
>   src/server/verify_job.rs                      |  12 +-
>   www/Makefile                                  |   3 +
>   www/NavigationTree.js                         |   6 +
>   www/Utils.js                                  |   1 +
>   www/config/S3ClientView.js                    | 141 +++
>   www/datastore/Summary.js                      |  44 +
>   www/form/S3ClientSelector.js                  |  33 +
>   www/window/DataStoreEdit.js                   | 110 ++-
>   www/window/MaintenanceOptions.js              |   6 +-
>   www/window/S3ClientEdit.js                    | 148 +++
>   52 files changed, 4120 insertions(+), 709 deletions(-)
>   create mode 100644 pbs-config/src/s3.rs
>   create mode 100644 pbs-datastore/src/local_datastore_lru_cache.rs
>   create mode 100644 pbs-s3-client/Cargo.toml
>   create mode 100644 pbs-s3-client/src/aws_sign_v4.rs
>   create mode 100644 pbs-s3-client/src/client.rs
>   create mode 100644 pbs-s3-client/src/lib.rs
>   create mode 100644 pbs-s3-client/src/object_key.rs
>   create mode 100644 pbs-s3-client/src/response_reader.rs
>   create mode 100644 src/api2/admin/s3.rs
>   create mode 100644 src/api2/config/s3.rs
>   create mode 100644 src/bin/proxmox_backup_manager/s3.rs
>   create mode 100644 www/config/S3ClientView.js
>   create mode 100644 www/form/S3ClientSelector.js
>   create mode 100644 www/window/S3ClientEdit.js
>
>
> Summary over all repositories:
>    56 files changed, 4383 insertions(+), 710 deletions(-)
>


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


  parent reply	other threads:[~2025-07-01 14:30 UTC|newest]

Thread overview: 53+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-06-23  9:40 Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox v4 1/3] pbs-api-types: add types for S3 client configs and secrets Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox v4 2/3] pbs-api-types: extend datastore config by backend config enum Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox v4 3/3] pbs-api-types: maintenance: add new maintenance mode S3 refresh Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 01/45] api: fix minor formatting issues Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 02/45] bin: sort submodules alphabetically Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 03/45] datastore: ignore missing owner file when removing group directory Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 04/45] verify: refactor verify related functions to be methods of worker Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 05/45] s3 client: add crate for AWS s3 compatible object store client Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 06/45] s3 client: implement AWS signature v4 request authentication Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 07/45] s3 client: add dedicated type for s3 object keys Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 08/45] s3 client: add type for last modified timestamp in responses Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 09/45] s3 client: add helper to parse http date headers Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 10/45] s3 client: implement methods to operate on s3 objects in bucket Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 11/45] config: introduce s3 object store client configuration Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 12/45] api: config: implement endpoints to manipulate and list s3 configs Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 13/45] api: datastore: check s3 backend bucket access on datastore create Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 14/45] api/cli: add endpoint and command to check s3 client connection Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 15/45] datastore: allow to get the backend for a datastore Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 16/45] api: backup: store datastore backend in runtime environment Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 17/45] api: backup: conditionally upload chunks to s3 object store backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 18/45] api: backup: conditionally upload blobs " Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 19/45] api: backup: conditionally upload indices " Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 20/45] api: backup: conditionally upload manifest " Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 21/45] sync: pull: conditionally upload content to s3 backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 22/45] api: reader: fetch chunks based on datastore backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 23/45] datastore: local chunk reader: read chunks based on backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 24/45] verify worker: add datastore backed to verify worker Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 25/45] verify: implement chunk verification for stores with s3 backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 26/45] datastore: create namespace marker in S3 backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 27/45] datastore: create/delete protected marker file on s3 storage backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 28/45] datastore: prune groups/snapshots from s3 object store backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 29/45] datastore: get and set owner for S3 " Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 30/45] datastore: implement garbage collection for s3 backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 31/45] ui: add datastore type selector and reorganize component layout Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 32/45] ui: add S3 client edit window for configuration create/edit Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 33/45] ui: add S3 client view for configuration Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 34/45] ui: expose the S3 client view in the navigation tree Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 35/45] ui: add s3 client selector and bucket field for s3 backend setup Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 36/45] tools: lru cache: add removed callback for evicted cache nodes Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 37/45] tools: async lru cache: implement insert, remove and contains methods Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 38/45] datastore: add local datastore cache for network attached storages Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 39/45] api: backup: use local datastore cache on s3 backend chunk upload Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 40/45] api: reader: use local datastore cache on s3 backend chunk fetching Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 41/45] api: backup: add no-cache flag to bypass local datastore cache Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 42/45] api/datastore: implement refresh endpoint for stores with s3 backend Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 43/45] cli: add dedicated subcommand for datastore s3 refresh Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 44/45] ui: render s3 refresh as valid maintenance type and task description Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 45/45] ui: expose s3 refresh button for datastores backed by object store Christian Ebner
2025-06-30  9:21 ` [pbs-devel] [PATCH proxmox{, -backup} v4 00/48] fix #2943: S3 storage backend for datastores Christian Ebner
2025-07-01 14:30 ` Hannes Duerr [this message]
2025-07-01 14:53   ` Christian Ebner
2025-07-03 13:20 ` [pbs-devel] superseded: " Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=724f995a-0e2e-47e4-9b5f-c1d5ee71df9f@proxmox.com \
    --to=h.duerr@proxmox.com \
    --cc=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal