From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 9C1871FF390 for ; Fri, 7 Jun 2024 11:43:27 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 648911C060; Fri, 7 Jun 2024 11:44:00 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Fri, 7 Jun 2024 11:43:05 +0200 Message-Id: <20240607094313.178742-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.028 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pbs-devel] [PATCH proxmox-backup 0/8] drop catalog encoding for split pxar archives X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" The catalog stores metadata needed for fast lookups via e.g. the file browser for single file restore. Since it is now possible to create snapshots with split pxar archives, the same metadata can be accessed in a reasonable performat way also via the metadata archive, effectively making the catalog unneeded duplicate data for this case. This patch series therefore allows to lookup directory entries where needed via the metadata archive instead of the catalog and conditionally drops the catalog encoding for snapshots using split archive encoding. The patches adapt not only the `catalog` api endpoint, used by the Proxmox Backup WebUI to access the snapshots contents but also the proxomox-file-restore, which is used by the Proxmox Virtual Environments PBS storage plugin to access and restore contents. Tested that regular backups can still be inspected and restored and that the same functionality is still available for split archive backups, without now having a catalog encoded. Functionally was tested with the following path series applied to PVE: https://lists.proxmox.com/pipermail/pve-devel/2024-May/064024.html Please note: For directories with may entries a noticeable performance drop is present, which might be improved upon by optimizing decoding. Christian Ebner (8): api: datastore: factor out path decoding for catalog api: datastore: move reusable code out of thread client: tools: add helper to lookup `ArchiveEntry`s via pxar api: datastore: conditional lookup for catalog endpoint api: datastore: add optional archive-name to file-restore file-restore: fallback to mpxar if catalog not present www: content: lookup via metadata archive instead of catalog client: backup: conditionally write catalog for file level backups pbs-client/src/pxar_backup_stream.rs | 11 ++- pbs-client/src/tools/mod.rs | 84 +++++++++++++++++ proxmox-backup-client/src/main.rs | 21 +++-- proxmox-file-restore/src/main.rs | 72 +++++++++++---- src/api2/admin/datastore.rs | 129 ++++++++++++++++++--------- www/datastore/Content.js | 3 + 6 files changed, 251 insertions(+), 69 deletions(-) -- 2.39.2 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel