From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pbs-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9])
by lore.proxmox.com (Postfix) with ESMTPS id 4875A1FF191
for <inbox@lore.proxmox.com>; Mon, 16 Jun 2025 16:21:56 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 748679E40;
Mon, 16 Jun 2025 16:22:14 +0200 (CEST)
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Date: Mon, 16 Jun 2025 16:21:13 +0200
Message-Id: <20250616142156.413652-1-c.ebner@proxmox.com>
X-Mailer: git-send-email 2.39.5
MIME-Version: 1.0
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.032 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DMARC_MISSING 0.1 Missing DMARC policy
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to
Validity was blocked. See
https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more
information.
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [lib.rs, push.rs, s3.rs, mod.rs, benchmark.rs, main.rs,
environment.rs, datastore.rs, upload-speed.rs, proxmox.com,
proxmox-backup-manager.rs, pull.rs, client.rs, verify.rs]
Subject: [pbs-devel] [PATCH proxmox{,
-backup} v3 00/43] fix #2943: S3 storage backend for datastores
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
<pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox Backup Server development discussion
<pbs-devel@lists.proxmox.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: pbs-devel-bounces@lists.proxmox.com
Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com>
Disclaimer: These patches are still in an experimental state and not
intended for production use.
This patch series aims to add S3 compatible object stores as storage
backend for PBS datastores. A PBS local cache store using the regular
datastore layout is used for faster operation, bypassing requests to
the S3 api when possible. Further, the local cache store allows to
keep frequently used chunks and is used to avoid expensive metadata
updates on the object store, e.g. by using local marker file during
garbage collection.
Backups are created by upload chunks to the corresponding S3 bucket,
while keeping the index files in the local cache store, on backup
finish, the snapshot metadata are persisted to the S3 storage backend.
Snapshot restores read chunks preferably from the local cache store,
downloading and insterting them if not present from the S3 object
store. Listing and snapsoht metadata operation currently rely soly on
the local cache store.
Currently chunks use a 1:1 mapping to S3 objects. An advanced packing
mechanism for chunks to significantly reduce the number of api
requests and therefore be more cost effective will be implemented as
followup patches.
Note: These patches assume the javascript files to be pre-formatted
using proxmox-biome to apply cleanly (not part of this series).
Most notably changes since RFC version 2 of the patches (thanks
@Lukas for feedback):
- Extend S3 client implementation to also support path style bucket
addressing.
- Keep bucket name as config option for the datastore, allowing more
flexible reuse of a configured S3 client.
- Use the datastore name as additional object key prefix to allow for
multiple datastores on the same bucket.
- Allow bucket and region templating in S3 endpoint, making this more
flexible with respect to possible DNS records.
- Rework datastore create window to be less overloaded.
- Drop dead code in the S3 client implementation, since tagging and
object copying is currently not required.
- Fix missing locking when deleting chunks from s3 store during
garbage collection, avoiding possible chunk loss for concurrent
backups.
- Remove chunks from LRU cache when deleting chunks during garbage
collection, avoiding possible chunk loss for concurrent backups.
- Add dedicated types for object prefix and relative s3 key paths to
avoid misuse.
- Use more fitting icon for S3 client.
Link to the bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=2943
The previous RFC version 2 of the patch series can be found at:
https://lore.proxmox.com/pbs-devel/cae15312-019a-4a95-8f4c-88c9a25f6b54@proxmox.com/T/
Steps to setup a local S3 object store using RADOS gateway or MinIO
can be found at (internal only, external users might use the steps
outlined in the cover letter and comments of RFC version 2):
https://wiki.intra.proxmox.com/PBS_Setup_S3_Object_Store
proxmox:
Christian Ebner (2):
pbs-api-types: add types for S3 client configs and secrets
pbs-api-types: extend datastore config by backend config enum
pbs-api-types/src/datastore.rs | 100 ++++++++++++++++++++-
pbs-api-types/src/lib.rs | 3 +
pbs-api-types/src/s3.rs | 154 +++++++++++++++++++++++++++++++++
3 files changed, 256 insertions(+), 1 deletion(-)
create mode 100644 pbs-api-types/src/s3.rs
proxmox-backup:
Christian Ebner (41):
api: fix minor formatting issues
bin: sort submodules alphabetically
datastore: ignore missing owner file when removing group directory
verify: refactor verify related functions to be methods of worker
s3 client: add crate for AWS S3 compatible object store client
s3 client: implement AWS signature v4 request authentication
s3 client: add dedicated type for s3 object keys
s3 client: add type for last modified timestamp in responses
s3 client: add helper to parse http date headers
s3 client: implement methods to operate on s3 objects in bucket
config: introduce s3 object store client configuration
api: config: implement endpoints to manipulate and list s3 configs
api: datastore: check S3 backend bucket access on datastore create
api/bin: add endpoint and command to check s3 client connection
datastore: allow to get the backend for a datastore
api: backup: store datastore backend in runtime environment
api: backup: conditionally upload chunks to S3 object store backend
api: backup: conditionally upload blobs to S3 object store backend
api: backup: conditionally upload indices to S3 object store backend
api: backup: conditionally upload manifest to S3 object store backend
sync: pull: conditionally upload content to S3 backend
api: reader: fetch chunks based on datastore backend
datastore: local chunk reader: read chunks based on backend
verify worker: add datastore backed to verify worker
verify: implement chunk verification for stores with s3 backend
datastore: create namespace marker in S3 backend
datastore: create/delete protected marker file on S3 storage backend
datastore: prune groups/snapshots from S3 object store backend
datastore: get and set owner for S3 store backend
datastore: implement garbage collection for s3 backend
ui: add datastore type selector and reorganize component layout
ui: add S3 client edit window for configuration create/edit
ui: add S3 client view for configuration
ui: expose the S3 client view in the navigation tree
ui: add s3 client selector and bucket field for s3 backend setup
tools: lru cache: add removed callback for evicted cache nodes
tools: async lru cache: implement insert, remove and contains methods
datastore: add local datastore cache for network attached storages
api: backup: use local datastore cache on S3 backend chunk upload
api: reader: use local datastore cache on S3 backend chunk fetching
api: backup: add no-cache flag to bypass local datastore cache
Cargo.toml | 8 +
examples/upload-speed.rs | 1 +
pbs-client/src/backup_writer.rs | 4 +-
pbs-config/src/lib.rs | 1 +
pbs-config/src/s3.rs | 82 ++
pbs-datastore/Cargo.toml | 3 +
pbs-datastore/src/backup_info.rs | 61 +-
pbs-datastore/src/cached_chunk_reader.rs | 6 +-
pbs-datastore/src/chunk_store.rs | 4 +
pbs-datastore/src/datastore.rs | 465 ++++++++-
pbs-datastore/src/dynamic_index.rs | 1 +
pbs-datastore/src/lib.rs | 4 +
pbs-datastore/src/local_chunk_reader.rs | 37 +-
.../src/local_datastore_lru_cache.rs | 116 +++
pbs-s3-client/Cargo.toml | 31 +
pbs-s3-client/src/aws_sign_v4.rs | 140 +++
pbs-s3-client/src/client.rs | 560 +++++++++++
pbs-s3-client/src/lib.rs | 122 +++
pbs-s3-client/src/object_key.rs | 102 ++
pbs-s3-client/src/response_reader.rs | 343 +++++++
pbs-tools/src/async_lru_cache.rs | 46 +-
pbs-tools/src/lru_cache.rs | 42 +-
proxmox-backup-client/src/benchmark.rs | 1 +
proxmox-backup-client/src/main.rs | 8 +
src/api2/admin/datastore.rs | 52 +-
src/api2/admin/mod.rs | 2 +
src/api2/admin/s3.rs | 80 ++
src/api2/backup/environment.rs | 145 ++-
src/api2/backup/mod.rs | 107 +--
src/api2/backup/upload_chunk.rs | 93 +-
src/api2/config/datastore.rs | 49 +-
src/api2/config/mod.rs | 2 +
src/api2/config/s3.rs | 310 ++++++
src/api2/reader/environment.rs | 12 +-
src/api2/reader/mod.rs | 59 +-
src/backup/verify.rs | 879 +++++++++---------
src/bin/proxmox-backup-manager.rs | 1 +
src/bin/proxmox_backup_manager/mod.rs | 30 +-
src/bin/proxmox_backup_manager/s3.rs | 46 +
src/server/pull.rs | 62 +-
src/server/push.rs | 1 +
src/server/verify_job.rs | 12 +-
www/Makefile | 3 +
www/NavigationTree.js | 6 +
www/config/S3ClientView.js | 137 +++
www/form/S3ClientSelector.js | 33 +
www/window/DataStoreEdit.js | 110 ++-
www/window/S3ClientEdit.js | 144 +++
48 files changed, 3883 insertions(+), 680 deletions(-)
create mode 100644 pbs-config/src/s3.rs
create mode 100644 pbs-datastore/src/local_datastore_lru_cache.rs
create mode 100644 pbs-s3-client/Cargo.toml
create mode 100644 pbs-s3-client/src/aws_sign_v4.rs
create mode 100644 pbs-s3-client/src/client.rs
create mode 100644 pbs-s3-client/src/lib.rs
create mode 100644 pbs-s3-client/src/object_key.rs
create mode 100644 pbs-s3-client/src/response_reader.rs
create mode 100644 src/api2/admin/s3.rs
create mode 100644 src/api2/config/s3.rs
create mode 100644 src/bin/proxmox_backup_manager/s3.rs
create mode 100644 www/config/S3ClientView.js
create mode 100644 www/form/S3ClientSelector.js
create mode 100644 www/window/S3ClientEdit.js
Summary over all repositories:
51 files changed, 4139 insertions(+), 681 deletions(-)
--
Generated by git-murpp 0.8.1
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel