From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id B87591FF16F for ; Tue, 22 Jul 2025 22:24:07 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id A9AEE1175E; Tue, 22 Jul 2025 22:25:22 +0200 (CEST) Message-ID: <733bccbc-0a04-425e-9c14-cfb30389be81@proxmox.com> Date: Tue, 22 Jul 2025 22:25:20 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta To: Proxmox Backup Server development discussion , Christian Ebner References: <20250722101106.526438-1-c.ebner@proxmox.com> Content-Language: en-US From: Thomas Lamprecht In-Reply-To: <20250722101106.526438-1-c.ebner@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1753215910942 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.031 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] applied: [PATCH proxmox{, -backup} v11 00/50] fix #2943: S3 storage backend for datastores X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Cc: Lukas Wagner Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" Am 22.07.25 um 12:11 schrieb Christian Ebner:> Disclaimer: These patches are still in an experimental state and not > intended for production use. > > This patch series aims to add S3 compatible object stores as storage > backend for PBS datastores. A PBS local cache store using the regular > datastore layout is used for faster operation, bypassing requests to > the S3 api when possible. Further, the local cache store allows to > keep frequently used chunks and is used to avoid expensive metadata > updates on the object store, e.g. by using local marker file during > garbage collection. > > Backups are created by upload chunks to the corresponding S3 bucket, > while keeping the index files in the local cache store, on backup > finish, the snapshot metadata are persisted to the S3 storage backend. > > Snapshot restores read chunks preferably from the local cache store, > downloading and insterting them if not present from the S3 object > store. Listing and snapsoht metadata operation currently rely soly on > the local cache store. > > Currently chunks use a 1:1 mapping to S3 objects. An advanced packing > mechanism for chunks to significantly reduce the number of api > requests and therefore be more cost effective will be implemented as > followup patches. applied, huge thanks to you for this quality implementation and the flexibility for some non-trivial requested changes towards the end, appreciated! Also thanks to Lukas and Hannes for their review and testing, that gave me much more confidence with this and definitively helped to catch some issues upfront. _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel