From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pbs-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 90B591FF17F for <inbox@lore.proxmox.com>; Mon, 19 May 2025 13:47:58 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C9EEF8B09; Mon, 19 May 2025 13:47:44 +0200 (CEST) From: Christian Ebner <c.ebner@proxmox.com> To: pbs-devel@lists.proxmox.com Date: Mon, 19 May 2025 13:46:20 +0200 Message-Id: <20250519114640.303640-20-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com> References: <20250519114640.303640-1-c.ebner@proxmox.com> MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.031 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [RFC proxmox-backup 19/39] api: backup: conditionally upload manifest to S3 object store backend X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion <pbs-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/> List-Post: <mailto:pbs-devel@lists.proxmox.com> List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe> Reply-To: Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com> Upload the manifest to the S3 object store backend after it has been finished in the backup api call handler, if s3 is configured as backend. Keep also the locally cached version for fast and efficient listing of contents without the need to perform expensive (as in monetary cost and IO latency) requests. The datastore's metadata contents will be synced from the S3 backend during datastore opening. Signed-off-by: Christian Ebner <c.ebner@proxmox.com> --- src/api2/backup/environment.rs | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs index 72e369bcf..685b78e89 100644 --- a/src/api2/backup/environment.rs +++ b/src/api2/backup/environment.rs @@ -12,7 +12,7 @@ use serde_json::{json, Value}; use proxmox_router::{RpcEnvironment, RpcEnvironmentType}; use proxmox_sys::fs::{replace_file, CreateOptions}; -use pbs_api_types::Authid; +use pbs_api_types::{Authid, MANIFEST_BLOB_NAME}; use pbs_datastore::backup_info::{BackupDir, BackupInfo}; use pbs_datastore::dynamic_index::DynamicIndexWriter; use pbs_datastore::fixed_index::FixedIndexWriter; @@ -719,6 +719,37 @@ impl BackupEnvironment { } } + if let DatastoreBackend::S3(s3_client) = &self.backend { + // Upload manifest to S3 object store + let mut object_key = self.backup_dir.relative_path(); + object_key.push(MANIFEST_BLOB_NAME.as_ref()); + let mut path = self.datastore.base_path(); + path.push(&object_key); + let mut manifest = std::fs::File::open(&path)?; + let mut buffer = Vec::new(); + manifest.read_to_end(&mut buffer)?; + let data = Body::from(buffer); + let object_key = object_key + .as_os_str() + .to_str() + .ok_or_else(|| format_err!("invalid path"))?; + match proxmox_async::runtime::block_on(s3_client.put_object(object_key.into(), data))? { + PutObjectResponse::PreconditionFailed => { + self.log(format!( + "Upload of manifest failed, object {object_key} already present." + )); + bail!("upload of manifest failed"); + } + PutObjectResponse::NeedsRetry => { + self.log("Upload of manifest failed, reupload required."); + bail!("concurrent operation, reupload required"); + } + PutObjectResponse::Success(_content) => { + self.log(format!("Uploaded manifest to object store: {object_key}")) + } + } + } + self.datastore.try_ensure_sync_level()?; // marks the backup as successful -- 2.39.5 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel