From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [RFC proxmox 2/2] pbs-api-types: extend datastore config by backend config enum
Date: Mon, 19 May 2025 13:46:03 +0200 [thread overview]
Message-ID: <20250519114640.303640-3-c.ebner@proxmox.com> (raw)
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
Allows to configure a backend config variant for a datastore on
creation. The current default `Filesystem` backend variant is
introduced to be compatible with existing storages. A new S3 backend
variant allows to create datastores backed by an S3 compatible object
store instead.
For S3 backends, the id of the corresponding S3 client configuration
is storered. A valid datastore backend configuration for S3 therefore
contains:
```
...
backend s3=<S3_CONFIG_ID>
...
```
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
pbs-api-types/src/datastore.rs | 58 +++++++++++++++++++++++++++++++++-
1 file changed, 57 insertions(+), 1 deletion(-)
diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 5bd953ac..2b983cb2 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -336,7 +336,11 @@ pub const DATASTORE_TUNING_STRING_SCHEMA: Schema = StringSchema::new("Datastore
optional: true,
format: &proxmox_schema::api_types::UUID_FORMAT,
type: String,
- }
+ },
+ backend: {
+ schema: DATASTORE_BACKEND_SCHEMA,
+ optional: true,
+ },
}
)]
#[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
@@ -389,8 +393,59 @@ pub struct DataStoreConfig {
#[updater(skip)]
#[serde(skip_serializing_if = "Option::is_none")]
pub backing_device: Option<String>,
+
+ /// Backend to be used by datastore
+ #[updater(skip)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ pub backend: Option<String>,
+}
+
+pub const DATASTORE_BACKEND_SCHEMA: Schema = StringSchema::new("Backend config to be used for datastore.")
+ .format(&ApiStringFormat::VerifyFn(verify_datastore_backend))
+ .type_text("<filesystem|s3=S3_CONFIG_ID>")
+ .schema();
+
+fn verify_datastore_backend(input: &str) -> Result<(), Error> {
+ DatastoreBackendConfig::from_str(input).map(|_| ())
+}
+
+#[derive(Clone, Default)]
+/// Available backend configurations for datastores.
+pub enum DatastoreBackendConfig {
+ #[default]
+ Filesystem,
+ S3(String),
}
+impl std::str::FromStr for DatastoreBackendConfig {
+ type Err = Error;
+
+ fn from_str(s: &str) -> Result<Self, Self::Err> {
+ if s == "filesystem" {
+ return Ok(Self::Filesystem);
+ }
+ match s.split_once('=') {
+ Some(("s3", value)) => {
+ let s3_config_id = value.parse()?;
+ Ok(Self::S3(s3_config_id))
+ }
+ _ => bail!("invalid datastore backend configuration"),
+ }
+ }
+}
+
+impl std::fmt::Display for DatastoreBackendConfig {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ match self {
+ Self::Filesystem => write!(f, "filesystem"),
+ Self::S3(s3_config_id) => write!(f, "s3:{s3_config_id}"),
+ }
+ }
+}
+
+proxmox_serde::forward_serialize_to_display!(DatastoreBackendConfig);
+proxmox_serde::forward_deserialize_to_from_str!(DatastoreBackendConfig);
+
#[api]
#[derive(Serialize, Deserialize, Updater, Clone, PartialEq, Default)]
#[serde(rename_all = "kebab-case")]
@@ -424,6 +479,7 @@ impl DataStoreConfig {
tuning: None,
maintenance_mode: None,
backing_device: None,
+ backend: None,
}
}
--
2.39.5
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
next prev parent reply other threads:[~2025-05-19 11:47 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-05-19 11:46 [pbs-devel] [RFC proxmox proxmox-backup 00/39] S3 storage backend for datastores Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox 1/2] pbs-api-types: add types for S3 client configs and secrets Christian Ebner
2025-05-19 11:46 ` Christian Ebner [this message]
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 03/39] fmt: fix minor formatting issues Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 04/39] verify: refactor verify related functions to be methods of worker Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 05/39] s3 client: add crate for AWS S3 compatible object store client Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 06/39] s3 client: implement AWS signature v4 request authentication Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 07/39] s3 client: add dedicated type for s3 object keys Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 08/39] s3 client: add helper for last modified timestamp parsing Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 09/39] s3 client: add helper to parse http date headers Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 10/39] s3 client: implement methods to operate on s3 objects in bucket Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 11/39] config: introduce s3 object store client configuration Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 12/39] api: config: implement endpoints to manipulate and list s3 configs Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 13/39] api: datastore: check S3 backend bucket access on datastore create Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 14/39] datastore: allow to get the backend for a datastore Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 15/39] api: backup: store datastore backend in runtime environment Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 16/39] api: backup: conditionally upload chunks to S3 object store backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 17/39] api: backup: conditionally upload blobs " Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 18/39] api: backup: conditionally upload indices " Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 19/39] api: backup: conditionally upload manifest " Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 20/39] api: reader: fetch chunks based on datastore backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 21/39] datastore: local chunk reader: read chunks based on backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 22/39] verify worker: add datastore backed to verify worker Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 23/39] verify: implement chunk verification for stores with s3 backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 24/39] api: remove snapshot from S3 backend on snapshot delete Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 25/39] datastore: prune groups/snapshots from S3 object store backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 26/39] datastore: implement garbage collection for s3 backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 27/39] ui: add S3 client edit window for configuration create/edit Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 28/39] ui: add S3 client view for configuration Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 29/39] ui: expose the S3 client view in the navigation tree Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 30/39] ui: add s3 bucket selector and allow to set s3 backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 31/39] api/bin: add endpoint and command to test s3 backend for datastore Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 32/39] tools: lru cache: add removed callback for evicted nodes Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 33/39] tools: async lru cache: implement insert, remove and contains methods Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 34/39] datastore: add local datastore cache for network attached storages Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 35/39] api: backup: use local datastore cache on S3 backend chunk upload Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 36/39] api: reader: use local datastore cache on S3 backend chunk fetching Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 37/39] api: backup: add no-cache flag to bypass local datastore cache Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 38/39] datastore: get and set owner for S3 store backend Christian Ebner
2025-05-19 11:46 ` [pbs-devel] [RFC proxmox-backup 39/39] datastore: create namespace marker in S3 backend Christian Ebner
2025-05-29 14:33 ` [pbs-devel] superseded: [RFC proxmox proxmox-backup 00/39] S3 storage backend for datastores Christian Ebner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250519114640.303640-3-c.ebner@proxmox.com \
--to=c.ebner@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal