public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [PATCH proxmox-backup v5 22/46] api: reader: fetch chunks based on datastore backend
Date: Thu,  3 Jul 2025 15:18:13 +0200	[thread overview]
Message-ID: <20250703131837.786811-26-c.ebner@proxmox.com> (raw)
In-Reply-To: <20250703131837.786811-1-c.ebner@proxmox.com>

Read the chunk based on the datastores backend, reading from local
filesystem or fetching from S3 object store.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 src/api2/reader/environment.rs | 12 ++++++----
 src/api2/reader/mod.rs         | 40 +++++++++++++++++++++++-----------
 2 files changed, 35 insertions(+), 17 deletions(-)

diff --git a/src/api2/reader/environment.rs b/src/api2/reader/environment.rs
index 3b2f06f43..8924352b0 100644
--- a/src/api2/reader/environment.rs
+++ b/src/api2/reader/environment.rs
@@ -1,13 +1,14 @@
 use std::collections::HashSet;
 use std::sync::{Arc, RwLock};
 
+use anyhow::Error;
 use serde_json::{json, Value};
 
 use proxmox_router::{RpcEnvironment, RpcEnvironmentType};
 
 use pbs_api_types::Authid;
 use pbs_datastore::backup_info::BackupDir;
-use pbs_datastore::DataStore;
+use pbs_datastore::{DataStore, DatastoreBackend};
 use proxmox_rest_server::formatter::*;
 use proxmox_rest_server::WorkerTask;
 use tracing::info;
@@ -23,6 +24,7 @@ pub struct ReaderEnvironment {
     pub worker: Arc<WorkerTask>,
     pub datastore: Arc<DataStore>,
     pub backup_dir: BackupDir,
+    pub backend: DatastoreBackend,
     allowed_chunks: Arc<RwLock<HashSet<[u8; 32]>>>,
 }
 
@@ -33,8 +35,9 @@ impl ReaderEnvironment {
         worker: Arc<WorkerTask>,
         datastore: Arc<DataStore>,
         backup_dir: BackupDir,
-    ) -> Self {
-        Self {
+    ) -> Result<Self, Error> {
+        let backend = datastore.backend()?;
+        Ok(Self {
             result_attributes: json!({}),
             env_type,
             auth_id,
@@ -43,8 +46,9 @@ impl ReaderEnvironment {
             debug: tracing::enabled!(tracing::Level::DEBUG),
             formatter: JSON_FORMATTER,
             backup_dir,
+            backend,
             allowed_chunks: Arc::new(RwLock::new(HashSet::new())),
-        }
+        })
     }
 
     pub fn log<S: AsRef<str>>(&self, msg: S) {
diff --git a/src/api2/reader/mod.rs b/src/api2/reader/mod.rs
index a77216043..36866923e 100644
--- a/src/api2/reader/mod.rs
+++ b/src/api2/reader/mod.rs
@@ -3,6 +3,7 @@
 use anyhow::{bail, format_err, Context, Error};
 use futures::*;
 use hex::FromHex;
+use http_body_util::BodyExt;
 use hyper::body::Incoming;
 use hyper::header::{self, HeaderValue, CONNECTION, UPGRADE};
 use hyper::http::request::Parts;
@@ -27,7 +28,8 @@ use pbs_api_types::{
 };
 use pbs_config::CachedUserInfo;
 use pbs_datastore::index::IndexFile;
-use pbs_datastore::{DataStore, PROXMOX_BACKUP_READER_PROTOCOL_ID_V1};
+use pbs_datastore::{DataStore, DatastoreBackend, PROXMOX_BACKUP_READER_PROTOCOL_ID_V1};
+use pbs_s3_client::S3Client;
 use pbs_tools::json::required_string_param;
 
 use crate::api2::backup::optional_ns_param;
@@ -162,7 +164,7 @@ fn upgrade_to_backup_reader_protocol(
                     worker.clone(),
                     datastore,
                     backup_dir,
-                );
+                )?;
 
                 env.debug = debug;
 
@@ -323,17 +325,10 @@ fn download_chunk(
             ));
         }
 
-        let (path, _) = env.datastore.chunk_path(&digest);
-        let path2 = path.clone();
-
-        env.debug(format!("download chunk {:?}", path));
-
-        let data =
-            proxmox_async::runtime::block_in_place(|| std::fs::read(path)).map_err(move |err| {
-                http_err!(BAD_REQUEST, "reading file {:?} failed: {}", path2, err)
-            })?;
-
-        let body = Body::from(data);
+        let body = match &env.backend {
+            DatastoreBackend::Filesystem => load_from_filesystem(env, &digest)?,
+            DatastoreBackend::S3(s3_client) => fetch_from_object_store(s3_client, &digest).await?,
+        };
 
         // fixme: set other headers ?
         Ok(Response::builder()
@@ -345,6 +340,25 @@ fn download_chunk(
     .boxed()
 }
 
+async fn fetch_from_object_store(s3_client: &S3Client, digest: &[u8; 32]) -> Result<Body, Error> {
+    if let Some(response) = s3_client.get_object(digest.into()).await? {
+        let data = response.content.collect().await?.to_bytes();
+        return Ok(Body::from(data));
+    }
+    bail!("cannot find chunk with digest {}", hex::encode(digest));
+}
+
+fn load_from_filesystem(env: &ReaderEnvironment, digest: &[u8; 32]) -> Result<Body, Error> {
+    let (path, _) = env.datastore.chunk_path(digest);
+    let path2 = path.clone();
+
+    env.debug(format!("download chunk {path:?}"));
+
+    let data = proxmox_async::runtime::block_in_place(|| std::fs::read(path))
+        .map_err(move |err| http_err!(BAD_REQUEST, "reading file {path2:?} failed: {err}"))?;
+    Ok(Body::from(data))
+}
+
 /* this is too slow
 fn download_chunk_old(
     _parts: Parts,
-- 
2.47.2



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel


  parent reply	other threads:[~2025-07-03 13:19 UTC|newest]

Thread overview: 57+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-07-03 13:17 [pbs-devel] [PATCH proxmox{, -backup} v5 00/49] fix #2943: S3 storage backend for datastores Christian Ebner
2025-07-03 13:17 ` [pbs-devel] [PATCH proxmox v5 1/3] pbs-api-types: add types for S3 client configs and secrets Christian Ebner
2025-07-04 11:37   ` Thomas Lamprecht
2025-07-04 11:56     ` Christian Ebner
2025-07-03 13:17 ` [pbs-devel] [PATCH proxmox v5 2/3] pbs-api-types: extend datastore config by backend config enum Christian Ebner
2025-07-03 13:17 ` [pbs-devel] [PATCH proxmox v5 3/3] pbs-api-types: maintenance: add new maintenance mode S3 refresh Christian Ebner
2025-07-03 13:17 ` [pbs-devel] [PATCH proxmox-backup v5 01/46] api: fix minor formatting issues Christian Ebner
2025-07-04 11:11   ` [pbs-devel] applied: " Thomas Lamprecht
2025-07-03 13:17 ` [pbs-devel] [PATCH proxmox-backup v5 02/46] bin: sort submodules alphabetically Christian Ebner
2025-07-04 11:11   ` [pbs-devel] applied: " Thomas Lamprecht
2025-07-03 13:17 ` [pbs-devel] [PATCH proxmox-backup v5 03/46] datastore: ignore missing owner file when removing group directory Christian Ebner
2025-07-04 11:11   ` [pbs-devel] applieapplied: " Thomas Lamprecht
2025-07-03 13:17 ` [pbs-devel] [PATCH proxmox-backup v5 04/46] verify: refactor verify related functions to be methods of worker Christian Ebner
2025-07-04 11:16   ` [pbs-devel] applied: " Thomas Lamprecht
2025-07-03 13:17 ` [pbs-devel] [PATCH proxmox-backup v5 05/46] s3 client: add crate for AWS s3 compatible object store client Christian Ebner
2025-07-03 13:17 ` [pbs-devel] [PATCH proxmox-backup v5 06/46] s3 client: implement AWS signature v4 request authentication Christian Ebner
2025-07-03 13:17 ` [pbs-devel] [PATCH proxmox-backup v5 07/46] s3 client: add dedicated type for s3 object keys Christian Ebner
2025-07-03 13:17 ` [pbs-devel] [PATCH proxmox-backup v5 08/46] s3 client: add type for last modified timestamp in responses Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 09/46] s3 client: add helper to parse http date headers Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 10/46] s3 client: implement methods to operate on s3 objects in bucket Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 11/46] config: introduce s3 object store client configuration Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 12/46] api: config: implement endpoints to manipulate and list s3 configs Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 13/46] api: datastore: check s3 backend bucket access on datastore create Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 14/46] api/cli: add endpoint and command to check s3 client connection Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 15/46] datastore: allow to get the backend for a datastore Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 16/46] api: backup: store datastore backend in runtime environment Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 17/46] api: backup: conditionally upload chunks to s3 object store backend Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 18/46] api: backup: conditionally upload blobs " Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 19/46] api: backup: conditionally upload indices " Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 20/46] api: backup: conditionally upload manifest " Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 21/46] sync: pull: conditionally upload content to s3 backend Christian Ebner
2025-07-03 13:18 ` Christian Ebner [this message]
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 23/46] datastore: local chunk reader: read chunks based on backend Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 24/46] verify worker: add datastore backed to verify worker Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 25/46] verify: implement chunk verification for stores with s3 backend Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 26/46] datastore: create namespace marker in " Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 27/46] datastore: create/delete protected marker file on s3 storage backend Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 28/46] datastore: prune groups/snapshots from s3 object store backend Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 29/46] datastore: get and set owner for s3 " Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 30/46] datastore: implement garbage collection for s3 backend Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 31/46] ui: add datastore type selector and reorganize component layout Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 32/46] ui: add s3 client edit window for configuration create/edit Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 33/46] ui: add s3 client view for configuration Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 34/46] ui: expose the s3 client view in the navigation tree Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 35/46] ui: add s3 client selector and bucket field for s3 backend setup Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 36/46] tools: lru cache: add removed callback for evicted cache nodes Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 37/46] tools: async lru cache: implement insert, remove and contains methods Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 38/46] datastore: add local datastore cache for network attached storages Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 39/46] api: backup: use local datastore cache on s3 backend chunk upload Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 40/46] api: reader: use local datastore cache on s3 backend chunk fetching Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 41/46] datastore: local chunk reader: get cached chunk from local cache store Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 42/46] api: backup: add no-cache flag to bypass local datastore cache Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 43/46] api/datastore: implement refresh endpoint for stores with s3 backend Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 44/46] cli: add dedicated subcommand for datastore s3 refresh Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 45/46] ui: render s3 refresh as valid maintenance type and task description Christian Ebner
2025-07-03 13:18 ` [pbs-devel] [PATCH proxmox-backup v5 46/46] ui: expose s3 refresh button for datastores backed by object store Christian Ebner
2025-07-08 17:05 ` [pbs-devel] superseded: [PATCH proxmox{, -backup} v5 00/49] fix #2943: S3 storage backend for datastores Christian Ebner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20250703131837.786811-26-c.ebner@proxmox.com \
    --to=c.ebner@proxmox.com \
    --cc=pbs-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal