From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pbs-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
by lore.proxmox.com (Postfix) with ESMTPS id 6E49A1FF16B
for <inbox@lore.proxmox.com>; Tue, 1 Jul 2025 16:30:48 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id BD2F681C1;
Tue, 1 Jul 2025 16:31:27 +0200 (CEST)
Message-ID: <724f995a-0e2e-47e4-9b5f-c1d5ee71df9f@proxmox.com>
Date: Tue, 1 Jul 2025 16:30:52 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
To: Proxmox Backup Server development discussion
<pbs-devel@lists.proxmox.com>, Christian Ebner <c.ebner@proxmox.com>
References: <20250623094106.299251-1-c.ebner@proxmox.com>
Content-Language: en-US
From: Hannes Duerr <h.duerr@proxmox.com>
In-Reply-To: <20250623094106.299251-1-c.ebner@proxmox.com>
X-SPAM-LEVEL: Spam detection results: 0
AWL -0.040 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.237 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [maintenance.rs, benchmark.rs, verify.rs, push.rs, lib.rs,
proxmox-backup-manager.rs, client.rs, mod.rs, proxmox.com, environment.rs,
main.rs, datastore.rs, pull.rs, s3.rs, upload-speed.rs]
Subject: Re: [pbs-devel] [PATCH proxmox{,
-backup} v4 00/48] fix #2943: S3 storage backend for datastores
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
<pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox Backup Server development discussion
<pbs-devel@lists.proxmox.com>
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="us-ascii"; Format="flowed"
Errors-To: pbs-devel-bounces@lists.proxmox.com
Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com>
Hi i am currently testing this series and noticed that you've missed the
following build dependencies in the debian/control file for the
proxmox-backup repository:
librust-iso8601-dev,
librust-md5-dev,
librust-quick-xml-dev,
librust-serde-xml-rs-dev,
On 6/23/25 11:40, Christian Ebner wrote:
> Disclaimer: These patches are still in an experimental state and not
> intended for production use.
>
> This patch series aims to add S3 compatible object stores as storage
> backend for PBS datastores. A PBS local cache store using the regular
> datastore layout is used for faster operation, bypassing requests to
> the S3 api when possible. Further, the local cache store allows to
> keep frequently used chunks and is used to avoid expensive metadata
> updates on the object store, e.g. by using local marker file during
> garbage collection.
>
> Backups are created by upload chunks to the corresponding S3 bucket,
> while keeping the index files in the local cache store, on backup
> finish, the snapshot metadata are persisted to the S3 storage backend.
>
> Snapshot restores read chunks preferably from the local cache store,
> downloading and insterting them if not present from the S3 object
> store. Listing and snapsoht metadata operation currently rely soly on
> the local cache store.
>
> Currently chunks use a 1:1 mapping to S3 objects. An advanced packing
> mechanism for chunks to significantly reduce the number of api
> requests and therefore be more cost effective will be implemented as
> followup patches.
>
> Most notably changes since version 3 of the patches:
> - Rebased onto current master, fixed incompatibilities with upgraded
> dependencies
> - Added method to uri decode s3 object keys, as they are required in
> order to download contents to a local store
> - Added api endpoint to allow resyncing of the datastore contents to
> the local cache store, introducing a new maintenance mode s3-refresh
> to guarantee consistency.
>
> Most notably changes since RFC version 2 of the patches (thanks
> @Lukas for feedback):
> - Extend S3 client implementation to also support path style bucket
> addressing.
> - Keep bucket name as config option for the datastore, allowing more
> flexible reuse of a configured S3 client.
> - Use the datastore name as additional object key prefix to allow for
> multiple datastores on the same bucket.
> - Allow bucket and region templating in S3 endpoint, making this more
> flexible with respect to possible DNS records.
> - Rework datastore create window to be less overloaded.
> - Drop dead code in the S3 client implementation, since tagging and
> object copying is currently not required.
> - Fix missing locking when deleting chunks from s3 store during
> garbage collection, avoiding possible chunk loss for concurrent
> backups.
> - Remove chunks from LRU cache when deleting chunks during garbage
> collection, avoiding possible chunk loss for concurrent backups.
> - Add dedicated types for object prefix and relative s3 key paths to
> avoid misuse.
> - Use more fitting icon for S3 client.
>
> Link to the bugtracker issue:
> https://bugzilla.proxmox.com/show_bug.cgi?id=2943
>
> The previous version 3 of the patch series can be found at:
> https://lore.proxmox.com/pbs-devel/20250616142156.413652-1-c.ebner@proxmox.com/T/
>
> Steps to setup a local S3 object store using RADOS gateway or MinIO
> can be found at (internal only, external users might use the steps
> outlined in the cover letter and comments of RFC version 2):
> https://wiki.intra.proxmox.com/PBS_Setup_S3_Object_Store
>
> proxmox:
>
> Christian Ebner (3):
> pbs-api-types: add types for S3 client configs and secrets
> pbs-api-types: extend datastore config by backend config enum
> pbs-api-types: maintenance: add new maintenance mode S3 refresh
>
> pbs-api-types/src/datastore.rs | 103 ++++++++++++++++++++-
> pbs-api-types/src/lib.rs | 3 +
> pbs-api-types/src/maintenance.rs | 4 +
> pbs-api-types/src/s3.rs | 154 +++++++++++++++++++++++++++++++
> 4 files changed, 263 insertions(+), 1 deletion(-)
> create mode 100644 pbs-api-types/src/s3.rs
>
>
> proxmox-backup:
>
> Christian Ebner (45):
> api: fix minor formatting issues
> bin: sort submodules alphabetically
> datastore: ignore missing owner file when removing group directory
> verify: refactor verify related functions to be methods of worker
> s3 client: add crate for AWS s3 compatible object store client
> s3 client: implement AWS signature v4 request authentication
> s3 client: add dedicated type for s3 object keys
> s3 client: add type for last modified timestamp in responses
> s3 client: add helper to parse http date headers
> s3 client: implement methods to operate on s3 objects in bucket
> config: introduce s3 object store client configuration
> api: config: implement endpoints to manipulate and list s3 configs
> api: datastore: check s3 backend bucket access on datastore create
> api/cli: add endpoint and command to check s3 client connection
> datastore: allow to get the backend for a datastore
> api: backup: store datastore backend in runtime environment
> api: backup: conditionally upload chunks to s3 object store backend
> api: backup: conditionally upload blobs to s3 object store backend
> api: backup: conditionally upload indices to s3 object store backend
> api: backup: conditionally upload manifest to s3 object store backend
> sync: pull: conditionally upload content to s3 backend
> api: reader: fetch chunks based on datastore backend
> datastore: local chunk reader: read chunks based on backend
> verify worker: add datastore backed to verify worker
> verify: implement chunk verification for stores with s3 backend
> datastore: create namespace marker in S3 backend
> datastore: create/delete protected marker file on s3 storage backend
> datastore: prune groups/snapshots from s3 object store backend
> datastore: get and set owner for S3 store backend
> datastore: implement garbage collection for s3 backend
> ui: add datastore type selector and reorganize component layout
> ui: add S3 client edit window for configuration create/edit
> ui: add S3 client view for configuration
> ui: expose the S3 client view in the navigation tree
> ui: add s3 client selector and bucket field for s3 backend setup
> tools: lru cache: add removed callback for evicted cache nodes
> tools: async lru cache: implement insert, remove and contains methods
> datastore: add local datastore cache for network attached storages
> api: backup: use local datastore cache on s3 backend chunk upload
> api: reader: use local datastore cache on s3 backend chunk fetching
> api: backup: add no-cache flag to bypass local datastore cache
> api/datastore: implement refresh endpoint for stores with s3 backend
> cli: add dedicated subcommand for datastore s3 refresh
> ui: render s3 refresh as valid maintenance type and task description
> ui: expose s3 refresh button for datastores backed by object store
>
> Cargo.toml | 8 +
> examples/upload-speed.rs | 1 +
> pbs-client/src/backup_writer.rs | 4 +-
> pbs-config/src/lib.rs | 1 +
> pbs-config/src/s3.rs | 82 ++
> pbs-datastore/Cargo.toml | 5 +
> pbs-datastore/src/backup_info.rs | 61 +-
> pbs-datastore/src/cached_chunk_reader.rs | 6 +-
> pbs-datastore/src/chunk_store.rs | 4 +
> pbs-datastore/src/datastore.rs | 601 +++++++++++-
> pbs-datastore/src/dynamic_index.rs | 1 +
> pbs-datastore/src/lib.rs | 4 +
> pbs-datastore/src/local_chunk_reader.rs | 37 +-
> .../src/local_datastore_lru_cache.rs | 116 +++
> pbs-s3-client/Cargo.toml | 33 +
> pbs-s3-client/src/aws_sign_v4.rs | 174 ++++
> pbs-s3-client/src/client.rs | 549 +++++++++++
> pbs-s3-client/src/lib.rs | 123 +++
> pbs-s3-client/src/object_key.rs | 99 ++
> pbs-s3-client/src/response_reader.rs | 279 ++++++
> pbs-tools/src/async_lru_cache.rs | 46 +-
> pbs-tools/src/lru_cache.rs | 42 +-
> proxmox-backup-client/src/benchmark.rs | 1 +
> proxmox-backup-client/src/main.rs | 8 +
> src/api2/admin/datastore.rs | 86 +-
> src/api2/admin/mod.rs | 2 +
> src/api2/admin/s3.rs | 80 ++
> src/api2/backup/environment.rs | 145 ++-
> src/api2/backup/mod.rs | 136 +--
> src/api2/backup/upload_chunk.rs | 108 ++-
> src/api2/config/datastore.rs | 49 +-
> src/api2/config/mod.rs | 2 +
> src/api2/config/s3.rs | 310 ++++++
> src/api2/reader/environment.rs | 12 +-
> src/api2/reader/mod.rs | 61 +-
> src/backup/verify.rs | 879 +++++++++---------
> src/bin/proxmox-backup-manager.rs | 1 +
> src/bin/proxmox_backup_manager/datastore.rs | 30 +
> src/bin/proxmox_backup_manager/mod.rs | 30 +-
> src/bin/proxmox_backup_manager/s3.rs | 46 +
> src/server/pull.rs | 62 +-
> src/server/push.rs | 1 +
> src/server/verify_job.rs | 12 +-
> www/Makefile | 3 +
> www/NavigationTree.js | 6 +
> www/Utils.js | 1 +
> www/config/S3ClientView.js | 141 +++
> www/datastore/Summary.js | 44 +
> www/form/S3ClientSelector.js | 33 +
> www/window/DataStoreEdit.js | 110 ++-
> www/window/MaintenanceOptions.js | 6 +-
> www/window/S3ClientEdit.js | 148 +++
> 52 files changed, 4120 insertions(+), 709 deletions(-)
> create mode 100644 pbs-config/src/s3.rs
> create mode 100644 pbs-datastore/src/local_datastore_lru_cache.rs
> create mode 100644 pbs-s3-client/Cargo.toml
> create mode 100644 pbs-s3-client/src/aws_sign_v4.rs
> create mode 100644 pbs-s3-client/src/client.rs
> create mode 100644 pbs-s3-client/src/lib.rs
> create mode 100644 pbs-s3-client/src/object_key.rs
> create mode 100644 pbs-s3-client/src/response_reader.rs
> create mode 100644 src/api2/admin/s3.rs
> create mode 100644 src/api2/config/s3.rs
> create mode 100644 src/bin/proxmox_backup_manager/s3.rs
> create mode 100644 www/config/S3ClientView.js
> create mode 100644 www/form/S3ClientSelector.js
> create mode 100644 www/window/S3ClientEdit.js
>
>
> Summary over all repositories:
> 56 files changed, 4383 insertions(+), 710 deletions(-)
>
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel