From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <pbs-devel-bounces@lists.proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id A95891FF3A0 for <inbox@lore.proxmox.com>; Fri, 7 Jun 2024 13:37:33 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 9E3681DF3C; Fri, 7 Jun 2024 13:38:05 +0200 (CEST) From: Christian Ebner <c.ebner@proxmox.com> To: pbs-devel@lists.proxmox.com Date: Fri, 7 Jun 2024 13:37:43 +0200 Message-Id: <20240607113752.324017-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.027 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pbs-devel] [PATCH v2 proxmox-backup 0/9] drop catalog encoding for split pxar archives X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion <pbs-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/> List-Post: <mailto:pbs-devel@lists.proxmox.com> List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe> Reply-To: Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com> Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" <pbs-devel-bounces@lists.proxmox.com> The catalog stores metadata needed for fast lookups via e.g. the file browser for single file restore. Since it is now possible to create snapshots with split pxar archives, the same metadata can be accessed in a reasonable performat way also via the metadata archive, effectively making the catalog unneeded duplicate data for this case. This patch series therefore allows to lookup directory entries where needed via the metadata archive instead of the catalog and conditionally drops the catalog encoding for snapshots using split archive encoding. The patches adapt not only the `catalog` api endpoint, used by the Proxmox Backup WebUI to access the snapshots contents but also the proxomox-file-restore, which is used by the Proxmox Virtual Environments PBS storage plugin to access and restore contents. Tested that regular backups can still be inspected and restored and that the same functionality is still available for split archive backups, without now having a catalog encoded. Functionally was tested with the following path series applied to PVE: https://lists.proxmox.com/pipermail/pve-devel/2024-May/064024.html Please note: For directories with may entries a noticeable performance drop is present, which might be improved upon by optimizing decoding. Changes since version 1: - Do not base64 encode archive name - Split off file restore archive name listing The rest of the comments will be addressed as followups as discussed off-list. Christian Ebner (9): api: datastore: factor out path decoding for catalog api: datastore: move reusable code out of thread client: tools: add helper to lookup `ArchiveEntry`s via pxar api: datastore: conditional lookup for catalog endpoint api: datastore: add optional archive-name to file-restore file-restore: never list ppxar as archive file-restore: fallback to mpxar if catalog not present www: content: lookup via metadata archive instead of catalog client: backup: conditionally write catalog for file level backups pbs-client/src/pxar_backup_stream.rs | 11 ++- pbs-client/src/tools/mod.rs | 84 +++++++++++++++++++ proxmox-backup-client/src/main.rs | 21 +++-- proxmox-file-restore/src/main.rs | 72 ++++++++++++---- src/api2/admin/datastore.rs | 121 ++++++++++++++++++--------- www/datastore/Content.js | 3 + 6 files changed, 243 insertions(+), 69 deletions(-) -- 2.39.2 >From 8dffaab3ce2c20b0c376d6c0dc8b3d561499fa24 Mon Sep 17 00:00:00 2001 From: Christian Ebner <c.ebner@proxmox.com> Date: Fri, 7 Jun 2024 11:25:09 +0200 Christian Ebner (8): api: datastore: factor out path decoding for catalog api: datastore: move reusable code out of thread client: tools: add helper to lookup `ArchiveEntry`s via pxar api: datastore: conditional lookup for catalog endpoint api: datastore: add optional archive-name to file-restore file-restore: fallback to mpxar if catalog not present www: content: lookup via metadata archive instead of catalog client: backup: conditionally write catalog for file level backups pbs-client/src/pxar_backup_stream.rs | 11 ++- pbs-client/src/tools/mod.rs | 84 +++++++++++++++++ proxmox-backup-client/src/main.rs | 21 +++-- proxmox-file-restore/src/main.rs | 72 +++++++++++---- src/api2/admin/datastore.rs | 129 ++++++++++++++++++--------- www/datastore/Content.js | 3 + 6 files changed, 251 insertions(+), 69 deletions(-) -- 2.39.2 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel