From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <pbs-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
	by lore.proxmox.com (Postfix) with ESMTPS id 90B591FF17F
	for <inbox@lore.proxmox.com>; Mon, 19 May 2025 13:47:58 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
	by firstgate.proxmox.com (Proxmox) with ESMTP id C9EEF8B09;
	Mon, 19 May 2025 13:47:44 +0200 (CEST)
From: Christian Ebner <c.ebner@proxmox.com>
To: pbs-devel@lists.proxmox.com
Date: Mon, 19 May 2025 13:46:20 +0200
Message-Id: <20250519114640.303640-20-c.ebner@proxmox.com>
X-Mailer: git-send-email 2.39.5
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
MIME-Version: 1.0
X-SPAM-LEVEL: Spam detection results:  0
 AWL 0.031 Adjusted score from AWL reputation of From: address
 BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
 DMARC_MISSING             0.1 Missing DMARC policy
 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
Subject: [pbs-devel] [RFC proxmox-backup 19/39] api: backup: conditionally
 upload manifest to S3 object store backend
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
 <pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, 
 <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
Reply-To: Proxmox Backup Server development discussion
 <pbs-devel@lists.proxmox.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: pbs-devel-bounces@lists.proxmox.com
Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com>

Upload the manifest to the S3 object store backend after it has been
finished in the backup api call handler, if s3 is configured as
backend. Keep also the locally cached version for fast and efficient
listing of contents without the need to perform expensive (as in
monetary cost and IO latency) requests.

The datastore's metadata contents will be synced from the S3 backend
during datastore opening.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
 src/api2/backup/environment.rs | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 72e369bcf..685b78e89 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -12,7 +12,7 @@ use serde_json::{json, Value};
 use proxmox_router::{RpcEnvironment, RpcEnvironmentType};
 use proxmox_sys::fs::{replace_file, CreateOptions};
 
-use pbs_api_types::Authid;
+use pbs_api_types::{Authid, MANIFEST_BLOB_NAME};
 use pbs_datastore::backup_info::{BackupDir, BackupInfo};
 use pbs_datastore::dynamic_index::DynamicIndexWriter;
 use pbs_datastore::fixed_index::FixedIndexWriter;
@@ -719,6 +719,37 @@ impl BackupEnvironment {
             }
         }
 
+        if let DatastoreBackend::S3(s3_client) = &self.backend {
+            // Upload manifest to S3 object store
+            let mut object_key = self.backup_dir.relative_path();
+            object_key.push(MANIFEST_BLOB_NAME.as_ref());
+            let mut path = self.datastore.base_path();
+            path.push(&object_key);
+            let mut manifest = std::fs::File::open(&path)?;
+            let mut buffer = Vec::new();
+            manifest.read_to_end(&mut buffer)?;
+            let data = Body::from(buffer);
+            let object_key = object_key
+                .as_os_str()
+                .to_str()
+                .ok_or_else(|| format_err!("invalid path"))?;
+            match proxmox_async::runtime::block_on(s3_client.put_object(object_key.into(), data))? {
+                PutObjectResponse::PreconditionFailed => {
+                    self.log(format!(
+                        "Upload of manifest failed, object {object_key} already present."
+                    ));
+                    bail!("upload of manifest failed");
+                }
+                PutObjectResponse::NeedsRetry => {
+                    self.log("Upload of manifest failed, reupload required.");
+                    bail!("concurrent operation, reupload required");
+                }
+                PutObjectResponse::Success(_content) => {
+                    self.log(format!("Uploaded manifest to object store: {object_key}"))
+                }
+            }
+        }
+
         self.datastore.try_ensure_sync_level()?;
 
         // marks the backup as successful
-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel