all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Hannes Duerr <h.duerr@proxmox.com>
To: Proxmox Backup Server development discussion
	<pbs-devel@lists.proxmox.com>,
	Christian Ebner <c.ebner@proxmox.com>
Subject: Re: [pbs-devel] [PATCH proxmox{, -backup} v4 00/48] fix #2943: S3 storage backend for datastores
Date: Tue, 1 Jul 2025 16:30:52 +0200	[thread overview]
Message-ID: <724f995a-0e2e-47e4-9b5f-c1d5ee71df9f@proxmox.com> (raw)
In-Reply-To: <20250623094106.299251-1-c.ebner@proxmox.com>

Hi i am currently testing this series and noticed that you've missed the 
following build dependencies in the debian/control file for the 
proxmox-backup repository:
librust-iso8601-dev,
librust-md5-dev,
librust-quick-xml-dev,
librust-serde-xml-rs-dev,

On 6/23/25 11:40, Christian Ebner wrote:
> Disclaimer: These patches are still in an experimental state and not
> intended for production use.
>
> This patch series aims to add S3 compatible object stores as storage
> backend for PBS datastores. A PBS local cache store using the regular
> datastore layout is used for faster operation, bypassing requests to
> the S3 api when possible. Further, the local cache store allows to
> keep frequently used chunks and is used to avoid expensive metadata
> updates on the object store, e.g. by using local marker file during
> garbage collection.
>
> Backups are created by upload chunks to the corresponding S3 bucket,
> while keeping the index files in the local cache store, on backup
> finish, the snapshot metadata are persisted to the S3 storage backend.
>
> Snapshot restores read chunks preferably from the local cache store,
> downloading and insterting them if not present from the S3 object
> store. Listing and snapsoht metadata operation currently rely soly on
> the local cache store.
>
> Currently chunks use a 1:1 mapping to S3 objects. An advanced packing
> mechanism for chunks to significantly reduce the number of api
> requests and therefore be more cost effective will be implemented as
> followup patches.
>
> Most notably changes since version 3 of the patches:
> - Rebased onto current master, fixed incompatibilities with upgraded
>    dependencies
> - Added method to uri decode s3 object keys, as they are required in
>    order to download contents to a local store
> - Added api endpoint to allow resyncing of the datastore contents to
>    the local cache store, introducing a new maintenance mode s3-refresh
>    to guarantee consistency.
>
> Most notably changes since RFC version 2 of the patches (thanks
> @Lukas for feedback):
> - Extend S3 client implementation to also support path style bucket
>    addressing.
> - Keep bucket name as config option for the datastore, allowing more
>    flexible reuse of a configured S3 client.
> - Use the datastore name as additional object key prefix to allow for
>    multiple datastores on the same bucket.
> - Allow bucket and region templating in S3 endpoint, making this more
>    flexible with respect to possible DNS records.
> - Rework datastore create window to be less overloaded.
> - Drop dead code in the S3 client implementation, since tagging and
>    object copying is currently not required.
> - Fix missing locking when deleting chunks from s3 store during
>    garbage collection, avoiding possible chunk loss for concurrent
>    backups.
> - Remove chunks from LRU cache when deleting chunks during garbage
>    collection, avoiding possible chunk loss for concurrent backups.
> - Add dedicated types for object prefix and relative s3 key paths to
>    avoid misuse.
> - Use more fitting icon for S3 client.
>
> Link to the bugtracker issue:
> https://bugzilla.proxmox.com/show_bug.cgi?id=2943
>
> The previous version 3 of the patch series can be found at:
> https://lore.proxmox.com/pbs-devel/20250616142156.413652-1-c.ebner@proxmox.com/T/
>
> Steps to setup a local S3 object store using RADOS gateway or MinIO
> can be found at (internal only, external users might use the steps
> outlined in the cover letter and comments of RFC version 2):
> https://wiki.intra.proxmox.com/PBS_Setup_S3_Object_Store
>
> proxmox:
>
> Christian Ebner (3):
>    pbs-api-types: add types for S3 client configs and secrets
>    pbs-api-types: extend datastore config by backend config enum
>    pbs-api-types: maintenance: add new maintenance mode S3 refresh
>
>   pbs-api-types/src/datastore.rs   | 103 ++++++++++++++++++++-
>   pbs-api-types/src/lib.rs         |   3 +
>   pbs-api-types/src/maintenance.rs |   4 +
>   pbs-api-types/src/s3.rs          | 154 +++++++++++++++++++++++++++++++
>   4 files changed, 263 insertions(+), 1 deletion(-)
>   create mode 100644 pbs-api-types/src/s3.rs
>
>
> proxmox-backup:
>
> Christian Ebner (45):
>    api: fix minor formatting issues
>    bin: sort submodules alphabetically
>    datastore: ignore missing owner file when removing group directory
>    verify: refactor verify related functions to be methods of worker
>    s3 client: add crate for AWS s3 compatible object store client
>    s3 client: implement AWS signature v4 request authentication
>    s3 client: add dedicated type for s3 object keys
>    s3 client: add type for last modified timestamp in responses
>    s3 client: add helper to parse http date headers
>    s3 client: implement methods to operate on s3 objects in bucket
>    config: introduce s3 object store client configuration
>    api: config: implement endpoints to manipulate and list s3 configs
>    api: datastore: check s3 backend bucket access on datastore create
>    api/cli: add endpoint and command to check s3 client connection
>    datastore: allow to get the backend for a datastore
>    api: backup: store datastore backend in runtime environment
>    api: backup: conditionally upload chunks to s3 object store backend
>    api: backup: conditionally upload blobs to s3 object store backend
>    api: backup: conditionally upload indices to s3 object store backend
>    api: backup: conditionally upload manifest to s3 object store backend
>    sync: pull: conditionally upload content to s3 backend
>    api: reader: fetch chunks based on datastore backend
>    datastore: local chunk reader: read chunks based on backend
>    verify worker: add datastore backed to verify worker
>    verify: implement chunk verification for stores with s3 backend
>    datastore: create namespace marker in S3 backend
>    datastore: create/delete protected marker file on s3 storage backend
>    datastore: prune groups/snapshots from s3 object store backend
>    datastore: get and set owner for S3 store backend
>    datastore: implement garbage collection for s3 backend
>    ui: add datastore type selector and reorganize component layout
>    ui: add S3 client edit window for configuration create/edit
>    ui: add S3 client view for configuration
>    ui: expose the S3 client view in the navigation tree
>    ui: add s3 client selector and bucket field for s3 backend setup
>    tools: lru cache: add removed callback for evicted cache nodes
>    tools: async lru cache: implement insert, remove and contains methods
>    datastore: add local datastore cache for network attached storages
>    api: backup: use local datastore cache on s3 backend chunk upload
>    api: reader: use local datastore cache on s3 backend chunk fetching
>    api: backup: add no-cache flag to bypass local datastore cache
>    api/datastore: implement refresh endpoint for stores with s3 backend
>    cli: add dedicated subcommand for datastore s3 refresh
>    ui: render s3 refresh as valid maintenance type and task description
>    ui: expose s3 refresh button for datastores backed by object store
>
>   Cargo.toml                                    |   8 +
>   examples/upload-speed.rs                      |   1 +
>   pbs-client/src/backup_writer.rs               |   4 +-
>   pbs-config/src/lib.rs                         |   1 +
>   pbs-config/src/s3.rs                          |  82 ++
>   pbs-datastore/Cargo.toml                      |   5 +
>   pbs-datastore/src/backup_info.rs              |  61 +-
>   pbs-datastore/src/cached_chunk_reader.rs      |   6 +-
>   pbs-datastore/src/chunk_store.rs              |   4 +
>   pbs-datastore/src/datastore.rs                | 601 +++++++++++-
>   pbs-datastore/src/dynamic_index.rs            |   1 +
>   pbs-datastore/src/lib.rs                      |   4 +
>   pbs-datastore/src/local_chunk_reader.rs       |  37 +-
>   .../src/local_datastore_lru_cache.rs          | 116 +++
>   pbs-s3-client/Cargo.toml                      |  33 +
>   pbs-s3-client/src/aws_sign_v4.rs              | 174 ++++
>   pbs-s3-client/src/client.rs                   | 549 +++++++++++
>   pbs-s3-client/src/lib.rs                      | 123 +++
>   pbs-s3-client/src/object_key.rs               |  99 ++
>   pbs-s3-client/src/response_reader.rs          | 279 ++++++
>   pbs-tools/src/async_lru_cache.rs              |  46 +-
>   pbs-tools/src/lru_cache.rs                    |  42 +-
>   proxmox-backup-client/src/benchmark.rs        |   1 +
>   proxmox-backup-client/src/main.rs             |   8 +
>   src/api2/admin/datastore.rs                   |  86 +-
>   src/api2/admin/mod.rs                         |   2 +
>   src/api2/admin/s3.rs                          |  80 ++
>   src/api2/backup/environment.rs                | 145 ++-
>   src/api2/backup/mod.rs                        | 136 +--
>   src/api2/backup/upload_chunk.rs               | 108 ++-
>   src/api2/config/datastore.rs                  |  49 +-
>   src/api2/config/mod.rs                        |   2 +
>   src/api2/config/s3.rs                         | 310 ++++++
>   src/api2/reader/environment.rs                |  12 +-
>   src/api2/reader/mod.rs                        |  61 +-
>   src/backup/verify.rs                          | 879 +++++++++---------
>   src/bin/proxmox-backup-manager.rs             |   1 +
>   src/bin/proxmox_backup_manager/datastore.rs   |  30 +
>   src/bin/proxmox_backup_manager/mod.rs         |  30 +-
>   src/bin/proxmox_backup_manager/s3.rs          |  46 +
>   src/server/pull.rs                            |  62 +-
>   src/server/push.rs                            |   1 +
>   src/server/verify_job.rs                      |  12 +-
>   www/Makefile                                  |   3 +
>   www/NavigationTree.js                         |   6 +
>   www/Utils.js                                  |   1 +
>   www/config/S3ClientView.js                    | 141 +++
>   www/datastore/Summary.js                      |  44 +
>   www/form/S3ClientSelector.js                  |  33 +
>   www/window/DataStoreEdit.js                   | 110 ++-
>   www/window/MaintenanceOptions.js              |   6 +-
>   www/window/S3ClientEdit.js                    | 148 +++
>   52 files changed, 4120 insertions(+), 709 deletions(-)
>   create mode 100644 pbs-config/src/s3.rs
>   create mode 100644 pbs-datastore/src/local_datastore_lru_cache.rs
>   create mode 100644 pbs-s3-client/Cargo.toml
>   create mode 100644 pbs-s3-client/src/aws_sign_v4.rs
>   create mode 100644 pbs-s3-client/src/client.rs
>   create mode 100644 pbs-s3-client/src/lib.rs
>   create mode 100644 pbs-s3-client/src/object_key.rs
>   create mode 100644 pbs-s3-client/src/response_reader.rs
>   create mode 100644 src/api2/admin/s3.rs
>   create mode 100644 src/api2/config/s3.rs
>   create mode 100644 src/bin/proxmox_backup_manager/s3.rs
>   create mode 100644 www/config/S3ClientView.js
>   create mode 100644 www/form/S3ClientSelector.js
>   create mode 100644 www/window/S3ClientEdit.js
>
>
> Summary over all repositories:
>    56 files changed, 4383 insertions(+), 710 deletions(-)
>


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


  parent reply	other threads:[~2025-07-01 14:30 UTC|newest]

Thread overview: 53+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-06-23  9:40 Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox v4 1/3] pbs-api-types: add types for S3 client configs and secrets Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox v4 2/3] pbs-api-types: extend datastore config by backend config enum Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox v4 3/3] pbs-api-types: maintenance: add new maintenance mode S3 refresh Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 01/45] api: fix minor formatting issues Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 02/45] bin: sort submodules alphabetically Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 03/45] datastore: ignore missing owner file when removing group directory Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 04/45] verify: refactor verify related functions to be methods of worker Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 05/45] s3 client: add crate for AWS s3 compatible object store client Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 06/45] s3 client: implement AWS signature v4 request authentication Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 07/45] s3 client: add dedicated type for s3 object keys Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 08/45] s3 client: add type for last modified timestamp in responses Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 09/45] s3 client: add helper to parse http date headers Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 10/45] s3 client: implement methods to operate on s3 objects in bucket Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 11/45] config: introduce s3 object store client configuration Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 12/45] api: config: implement endpoints to manipulate and list s3 configs Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 13/45] api: datastore: check s3 backend bucket access on datastore create Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 14/45] api/cli: add endpoint and command to check s3 client connection Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 15/45] datastore: allow to get the backend for a datastore Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 16/45] api: backup: store datastore backend in runtime environment Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 17/45] api: backup: conditionally upload chunks to s3 object store backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 18/45] api: backup: conditionally upload blobs " Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 19/45] api: backup: conditionally upload indices " Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 20/45] api: backup: conditionally upload manifest " Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 21/45] sync: pull: conditionally upload content to s3 backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 22/45] api: reader: fetch chunks based on datastore backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 23/45] datastore: local chunk reader: read chunks based on backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 24/45] verify worker: add datastore backed to verify worker Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 25/45] verify: implement chunk verification for stores with s3 backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 26/45] datastore: create namespace marker in S3 backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 27/45] datastore: create/delete protected marker file on s3 storage backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 28/45] datastore: prune groups/snapshots from s3 object store backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 29/45] datastore: get and set owner for S3 " Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 30/45] datastore: implement garbage collection for s3 backend Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 31/45] ui: add datastore type selector and reorganize component layout Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 32/45] ui: add S3 client edit window for configuration create/edit Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 33/45] ui: add S3 client view for configuration Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 34/45] ui: expose the S3 client view in the navigation tree Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 35/45] ui: add s3 client selector and bucket field for s3 backend setup Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 36/45] tools: lru cache: add removed callback for evicted cache nodes Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 37/45] tools: async lru cache: implement insert, remove and contains methods Christian Ebner
2025-06-23  9:40 ` [pbs-devel] [PATCH proxmox-backup v4 38/45] datastore: add local datastore cache for network attached storages Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 39/45] api: backup: use local datastore cache on s3 backend chunk upload Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 40/45] api: reader: use local datastore cache on s3 backend chunk fetching Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 41/45] api: backup: add no-cache flag to bypass local datastore cache Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 42/45] api/datastore: implement refresh endpoint for stores with s3 backend Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 43/45] cli: add dedicated subcommand for datastore s3 refresh Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 44/45] ui: render s3 refresh as valid maintenance type and task description Christian Ebner
2025-06-23  9:41 ` [pbs-devel] [PATCH proxmox-backup v4 45/45] ui: expose s3 refresh button for datastores backed by object store Christian Ebner
2025-06-30  9:21 ` [pbs-devel] [PATCH proxmox{, -backup} v4 00/48] fix #2943: S3 storage backend for datastores Christian Ebner
2025-07-01 14:30 ` Hannes Duerr [this message]
2025-07-01 14:53   ` Christian Ebner
2025-07-03 13:20 ` [pbs-devel] superseded: " Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=724f995a-0e2e-47e4-9b5f-c1d5ee71df9f@proxmox.com \
    --to=h.duerr@proxmox.com \
    --cc=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal