public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [PATCH proxmox-backup v3 15/41] datastore: allow to get the backend for a datastore
Date: Mon, 16 Jun 2025 16:21:30 +0200	[thread overview]
Message-ID: <20250616142156.413652-18-c.ebner@proxmox.com> (raw)
In-Reply-To: <20250616142156.413652-1-c.ebner@proxmox.com>

Implements an enum with variants Filesystem and S3 to distinguish
between available backends. Filesystem will be used as default, if no
backend is configured in the datastores configuration. If the
datastore has an s3 backend configured, the backend method will
instantiate and s3 client and return it with the S3 variant.

This allows to instantiate the client once, keeping and reusing the
same open connection to the api for the lifetime of task or job, e.g.
in the backup writer/readers runtime environment.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 pbs-datastore/Cargo.toml       |  1 +
 pbs-datastore/src/datastore.rs | 53 ++++++++++++++++++++++++++++++++--
 pbs-datastore/src/lib.rs       |  1 +
 3 files changed, 53 insertions(+), 2 deletions(-)

diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index d7fe71b34..182398d7e 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -46,4 +46,5 @@ pbs-api-types.workspace = true
 pbs-buildcfg.workspace = true
 pbs-config.workspace = true
 pbs-key-config.workspace = true
+pbs-s3-client.workspace = true
 pbs-tools.workspace = true
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 924d8cf9c..3e54ce31f 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -8,6 +8,7 @@ use std::time::Duration;
 
 use anyhow::{bail, format_err, Context, Error};
 use nix::unistd::{unlinkat, UnlinkatFlags};
+use pbs_s3_client::{S3Client, S3ClientOptions};
 use pbs_tools::lru_cache::LruCache;
 use tracing::{info, warn};
 
@@ -23,8 +24,9 @@ use proxmox_worker_task::WorkerTaskContext;
 
 use pbs_api_types::{
     ArchiveType, Authid, BackupGroupDeleteStats, BackupNamespace, BackupType, ChunkOrder,
-    DataStoreConfig, DatastoreFSyncLevel, DatastoreTuning, GarbageCollectionStatus,
-    MaintenanceMode, MaintenanceType, Operation, UPID,
+    DataStoreConfig, DatastoreBackendConfig, DatastoreBackendType, DatastoreFSyncLevel,
+    DatastoreTuning, GarbageCollectionStatus, MaintenanceMode, MaintenanceType, Operation,
+    S3ClientConfig, S3ClientSecretsConfig, UPID,
 };
 use pbs_config::BackupLockGuard;
 
@@ -127,6 +129,7 @@ pub struct DataStoreImpl {
     chunk_order: ChunkOrder,
     last_digest: Option<[u8; 32]>,
     sync_level: DatastoreFSyncLevel,
+    backend_config: DatastoreBackendConfig,
 }
 
 impl DataStoreImpl {
@@ -141,6 +144,7 @@ impl DataStoreImpl {
             chunk_order: Default::default(),
             last_digest: None,
             sync_level: Default::default(),
+            backend_config: Default::default(),
         })
     }
 }
@@ -196,6 +200,12 @@ impl Drop for DataStore {
     }
 }
 
+#[derive(Clone)]
+pub enum DatastoreBackend {
+    Filesystem,
+    S3(Arc<S3Client>),
+}
+
 impl DataStore {
     // This one just panics on everything
     #[doc(hidden)]
@@ -206,6 +216,39 @@ impl DataStore {
         })
     }
 
+    /// Get the backend for this datastore based on it's configuration
+    pub fn backend(&self) -> Result<DatastoreBackend, Error> {
+        let backend_type = match self.inner.backend_config.ty.unwrap_or_default() {
+            DatastoreBackendType::Filesystem => DatastoreBackend::Filesystem,
+            DatastoreBackendType::S3 => {
+                let s3_client_id = self
+                    .inner
+                    .backend_config
+                    .client
+                    .as_ref()
+                    .ok_or_else(|| format_err!("missing client for s3 backend"))?;
+                let bucket = self
+                    .inner
+                    .backend_config
+                    .bucket
+                    .clone()
+                    .ok_or_else(|| format_err!("missing bucket for s3 backend"))?;
+
+                let (config, _config_digest) = pbs_config::s3::config()?;
+                let (secrets, _secrets_digest) = pbs_config::s3::secrets_config()?;
+                let config: S3ClientConfig = config.lookup("s3client", s3_client_id)?;
+                let secrets: S3ClientSecretsConfig = secrets.lookup("s3secrets", s3_client_id)?;
+
+                let options =
+                    S3ClientOptions::from_config(config, secrets, bucket, self.name().to_owned());
+                let s3_client = S3Client::new(options)?;
+                DatastoreBackend::S3(Arc::new(s3_client))
+            }
+        };
+
+        Ok(backend_type)
+    }
+
     pub fn lookup_datastore(
         name: &str,
         operation: Option<Operation>,
@@ -383,6 +426,11 @@ impl DataStore {
                 .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
         )?;
 
+        let backend_config: DatastoreBackendConfig = serde_json::from_value(
+            DatastoreBackendConfig::API_SCHEMA
+                .parse_property_string(config.backend.as_deref().unwrap_or(""))?,
+        )?;
+
         Ok(DataStoreImpl {
             chunk_store,
             gc_mutex: Mutex::new(()),
@@ -391,6 +439,7 @@ impl DataStore {
             chunk_order: tuning.chunk_order.unwrap_or_default(),
             last_digest,
             sync_level: tuning.sync_level.unwrap_or_default(),
+            backend_config,
         })
     }
 
diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
index 5014b6c09..e6f65575b 100644
--- a/pbs-datastore/src/lib.rs
+++ b/pbs-datastore/src/lib.rs
@@ -203,6 +203,7 @@ pub use store_progress::StoreProgress;
 mod datastore;
 pub use datastore::{
     check_backup_owner, ensure_datastore_is_mounted, get_datastore_mount_status, DataStore,
+    DatastoreBackend,
 };
 
 mod hierarchy;
-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


  parent reply	other threads:[~2025-06-16 14:22 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-06-16 14:21 [pbs-devel] [PATCH proxmox{, -backup} v3 00/43] fix #2943: S3 storage backend for datastores Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox v3 1/2] pbs-api-types: add types for S3 client configs and secrets Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox v3 2/2] pbs-api-types: extend datastore config by backend config enum Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 01/41] api: fix minor formatting issues Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 02/41] bin: sort submodules alphabetically Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 03/41] datastore: ignore missing owner file when removing group directory Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 04/41] verify: refactor verify related functions to be methods of worker Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 05/41] s3 client: add crate for AWS S3 compatible object store client Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 06/41] s3 client: implement AWS signature v4 request authentication Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 07/41] s3 client: add dedicated type for s3 object keys Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 08/41] s3 client: add type for last modified timestamp in responses Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 09/41] s3 client: add helper to parse http date headers Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 10/41] s3 client: implement methods to operate on s3 objects in bucket Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 11/41] config: introduce s3 object store client configuration Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 12/41] api: config: implement endpoints to manipulate and list s3 configs Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 13/41] api: datastore: check S3 backend bucket access on datastore create Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 14/41] api/bin: add endpoint and command to check s3 client connection Christian Ebner
2025-06-16 14:21 ` Christian Ebner [this message]
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 16/41] api: backup: store datastore backend in runtime environment Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 17/41] api: backup: conditionally upload chunks to S3 object store backend Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 18/41] api: backup: conditionally upload blobs " Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 19/41] api: backup: conditionally upload indices " Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 20/41] api: backup: conditionally upload manifest " Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 21/41] sync: pull: conditionally upload content to S3 backend Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 22/41] api: reader: fetch chunks based on datastore backend Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 23/41] datastore: local chunk reader: read chunks based on backend Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 24/41] verify worker: add datastore backed to verify worker Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 25/41] verify: implement chunk verification for stores with s3 backend Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 26/41] datastore: create namespace marker in S3 backend Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 27/41] datastore: create/delete protected marker file on S3 storage backend Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 28/41] datastore: prune groups/snapshots from S3 object store backend Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 29/41] datastore: get and set owner for S3 " Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 30/41] datastore: implement garbage collection for s3 backend Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 31/41] ui: add datastore type selector and reorganize component layout Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 32/41] ui: add S3 client edit window for configuration create/edit Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 33/41] ui: add S3 client view for configuration Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 34/41] ui: expose the S3 client view in the navigation tree Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 35/41] ui: add s3 client selector and bucket field for s3 backend setup Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 36/41] tools: lru cache: add removed callback for evicted cache nodes Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 37/41] tools: async lru cache: implement insert, remove and contains methods Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 38/41] datastore: add local datastore cache for network attached storages Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 39/41] api: backup: use local datastore cache on S3 backend chunk upload Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 40/41] api: reader: use local datastore cache on S3 backend chunk fetching Christian Ebner
2025-06-16 14:21 ` [pbs-devel] [PATCH proxmox-backup v3 41/41] api: backup: add no-cache flag to bypass local datastore cache Christian Ebner
2025-06-23  9:43 ` [pbs-devel] superseded: [PATCH proxmox{, -backup} v3 00/43] fix #2943: S3 storage backend for datastores Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20250616142156.413652-18-c.ebner@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal