From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [PATCH proxmox-backup v11 02/46] config: introduce s3 object store client configuration
Date: Tue, 22 Jul 2025 12:10:22 +0200 [thread overview]
Message-ID: <20250722101106.526438-7-c.ebner@proxmox.com> (raw)
In-Reply-To: <20250722101106.526438-1-c.ebner@proxmox.com>
Adds the client configuration for s3 object store as dedicated
configuration files.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
changes since version 10:
- merge secrets into client config
pbs-config/Cargo.toml | 1 +
pbs-config/src/lib.rs | 1 +
pbs-config/src/s3.rs | 67 +++++++++++++++++++++++++++++++++++++++++++
3 files changed, 69 insertions(+)
create mode 100644 pbs-config/src/s3.rs
diff --git a/pbs-config/Cargo.toml b/pbs-config/Cargo.toml
index 284149658..74afb3c64 100644
--- a/pbs-config/Cargo.toml
+++ b/pbs-config/Cargo.toml
@@ -19,6 +19,7 @@ serde_json.workspace = true
proxmox-notify.workspace = true
proxmox-router = { workspace = true, default-features = false }
+proxmox-s3-client.workspace = true
proxmox-schema.workspace = true
proxmox-section-config.workspace = true
proxmox-shared-memory.workspace = true
diff --git a/pbs-config/src/lib.rs b/pbs-config/src/lib.rs
index 9c4d77c24..d03c079ab 100644
--- a/pbs-config/src/lib.rs
+++ b/pbs-config/src/lib.rs
@@ -10,6 +10,7 @@ pub mod network;
pub mod notifications;
pub mod prune;
pub mod remote;
+pub mod s3;
pub mod sync;
pub mod tape_job;
pub mod token_shadow;
diff --git a/pbs-config/src/s3.rs b/pbs-config/src/s3.rs
new file mode 100644
index 000000000..428f097bb
--- /dev/null
+++ b/pbs-config/src/s3.rs
@@ -0,0 +1,67 @@
+use std::collections::HashMap;
+use std::sync::LazyLock;
+
+use anyhow::Error;
+
+use proxmox_s3_client::S3ClientConfig;
+use proxmox_schema::*;
+use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
+
+use pbs_api_types::JOB_ID_SCHEMA;
+
+use pbs_buildcfg::configdir;
+
+use crate::{open_backup_lockfile, replace_backup_config, BackupLockGuard};
+
+pub static CONFIG: LazyLock<SectionConfig> = LazyLock::new(init);
+
+fn init() -> SectionConfig {
+ let obj_schema = S3ClientConfig::API_SCHEMA.unwrap_object_schema();
+ let plugin =
+ SectionConfigPlugin::new("s3client".to_string(), Some(String::from("id")), obj_schema);
+ let mut config = SectionConfig::new(&JOB_ID_SCHEMA);
+ config.register_plugin(plugin);
+
+ config
+}
+
+/// Configuration file location for S3 client.
+pub const S3_CFG_FILENAME: &str = configdir!("/s3.cfg");
+/// Configuration lock file used to prevent concurrent configuration update operations.
+pub const S3_CFG_LOCKFILE: &str = configdir!("/.s3.lck");
+
+/// Config type for s3 client config entries
+pub const S3_CFG_TYPE_ID: &str = "s3client";
+
+/// Get exclusive lock for S3 client configuration update.
+pub fn lock_config() -> Result<BackupLockGuard, Error> {
+ open_backup_lockfile(S3_CFG_LOCKFILE, None, true)
+}
+
+/// Load s3 client configuration from file.
+pub fn config() -> Result<(SectionConfigData, [u8; 32]), Error> {
+ parse_config(S3_CFG_FILENAME)
+}
+
+/// Save given s3 client configuration to file.
+pub fn save_config(config: &SectionConfigData) -> Result<(), Error> {
+ let raw = CONFIG.write(S3_CFG_FILENAME, config)?;
+ replace_backup_config(S3_CFG_FILENAME, raw.as_bytes())?;
+ Ok(())
+}
+
+/// Shell completion helper to complete s3 client id's as found in the config.
+pub fn complete_s3_client_id(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
+ match config() {
+ Ok((data, _digest)) => data.sections.keys().map(|id| id.to_string()).collect(),
+ Err(_) => Vec::new(),
+ }
+}
+
+fn parse_config(path: &str) -> Result<(SectionConfigData, [u8; 32]), Error> {
+ let content = proxmox_sys::fs::file_read_optional_string(path)?;
+ let content = content.unwrap_or_default();
+ let digest = openssl::sha::sha256(content.as_bytes());
+ let data = CONFIG.parse(path, &content)?;
+ Ok((data, digest))
+}
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
next prev parent reply other threads:[~2025-07-22 10:11 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-07-22 10:10 [pbs-devel] [PATCH proxmox{, -backup} v11 00/50] fix #2943: S3 storage backend for datastores Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox v11 1/4] pbs-api-types: extend datastore config by backend config enum Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox v11 2/4] pbs-api-types: maintenance: add new maintenance mode S3 refresh Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox v11 3/4] s3 client: Add missing S3 object key max length check Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox v11 4/4] s3 client: merge secrets config with client config Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 01/46] datastore: add helpers for path/digest to s3 object key conversion Christian Ebner
2025-07-22 10:10 ` Christian Ebner [this message]
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 03/46] api: config: implement endpoints to manipulate and list s3 configs Christian Ebner
2025-07-22 12:18 ` Lukas Wagner
2025-07-22 12:32 ` Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 04/46] api: datastore: check s3 backend bucket access on datastore create Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 05/46] api/cli: add endpoint and command to check s3 client connection Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 06/46] datastore: allow to get the backend for a datastore Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 07/46] api: backup: store datastore backend in runtime environment Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 08/46] api: backup: conditionally upload chunks to s3 object store backend Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 09/46] api: backup: conditionally upload blobs " Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 10/46] api: backup: conditionally upload indices " Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 11/46] api: backup: conditionally upload manifest " Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 12/46] api: datastore: conditionally upload client log to s3 backend Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 13/46] sync: pull: conditionally upload content " Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 14/46] api: reader: fetch chunks based on datastore backend Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 15/46] datastore: local chunk reader: read chunks based on backend Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 16/46] verify worker: add datastore backed to verify worker Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 17/46] verify: implement chunk verification for stores with s3 backend Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 18/46] datastore: create namespace marker in " Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 19/46] datastore: create/delete protected marker file on s3 storage backend Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 20/46] datastore: prune groups/snapshots from s3 object store backend Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 21/46] datastore: get and set owner for s3 " Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 22/46] datastore: implement garbage collection for s3 backend Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 23/46] ui: add datastore type selector and reorganize component layout Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 24/46] ui: add s3 client edit window for configuration create/edit Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 25/46] ui: add s3 client view for configuration Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 26/46] ui: expose the s3 client view in the navigation tree Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 27/46] ui: add s3 client selector and bucket field for s3 backend setup Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 28/46] tools: lru cache: add removed callback for evicted cache nodes Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 29/46] tools: async lru cache: implement insert, remove and contains methods Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 30/46] datastore: add local datastore cache for network attached storages Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 31/46] api: backup: use local datastore cache on s3 backend chunk upload Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 32/46] api: reader: use local datastore cache on s3 backend chunk fetching Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 33/46] datastore: local chunk reader: get cached chunk from local cache store Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 34/46] backup writer: refactor parameters into backup writer options struct Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 35/46] api: backup: add no-cache flag to bypass local datastore cache Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 36/46] api/datastore: implement refresh endpoint for stores with s3 backend Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 37/46] cli: add dedicated subcommand for datastore s3 refresh Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 38/46] ui: render s3 refresh as valid maintenance type and task description Christian Ebner
2025-07-22 10:10 ` [pbs-devel] [PATCH proxmox-backup v11 39/46] ui: expose s3 refresh button for datastores backed by object store Christian Ebner
2025-07-22 10:11 ` [pbs-devel] [PATCH proxmox-backup v11 40/46] datastore: conditionally upload atime marker chunk to s3 backend Christian Ebner
2025-07-22 10:11 ` [pbs-devel] [PATCH proxmox-backup v11 41/46] bin: implement client subcommands for s3 configuration manipulation Christian Ebner
2025-07-22 10:11 ` [pbs-devel] [PATCH proxmox-backup v11 42/46] bin: expose reuse-datastore flag for proxmox-backup-manager Christian Ebner
2025-07-22 10:11 ` [pbs-devel] [PATCH proxmox-backup v11 43/46] datastore: mark store as in-use by setting marker on s3 backend Christian Ebner
2025-07-22 10:11 ` [pbs-devel] [PATCH proxmox-backup v11 44/46] datastore: run s3-refresh when reusing a datastore with " Christian Ebner
2025-07-22 10:11 ` [pbs-devel] [PATCH proxmox-backup v11 45/46] api/ui: add flag to allow overwriting in-use marker for " Christian Ebner
2025-07-22 10:11 ` [pbs-devel] [PATCH proxmox-backup v11 46/46] docs: Add section describing how to setup s3 backed datastore Christian Ebner
2025-07-22 20:25 ` [pbs-devel] applied: [PATCH proxmox{, -backup} v11 00/50] fix #2943: S3 storage backend for datastores Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250722101106.526438-7-c.ebner@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox