From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 0BB4B1FF0AA for ; Tue, 17 Nov 2026 21:40:18 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id D4E7A21696; Tue, 17 Nov 2026 21:40:05 +0100 (CET) From: "Max R. Carrara" To: pve-devel@lists.proxmox.com Subject: [PATCH manager/storage v3 00/51] Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types Date: Tue, 17 Nov 2026 21:38:32 +0100 Message-ID: <20261117203953.913035-1-m.carrara@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794947993513 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.123 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_ASCII_DIVIDERS 0.8 Email that uses ascii formatting dividers and possible spam tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: YD7KMH3Q4RR6MKWQ5DWHEO7P3ROF4ULQ X-Message-ID-Hash: YD7KMH3Q4RR6MKWQ5DWHEO7P3ROF4ULQ X-MailFrom: m.carrara@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Wolfgang Bumiller X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types - v3 =========================================================================== Basically what the title says. Implement subdirectory scanning for directory-based storage types, which includes directories (duh), NFS, CIFS, CephFS, and BTRFS. This fixes #2884 [2884] and the more narrowly-scoped #623 [623]. See the initial series's cover letter [init-cover] for the entire context. *Huge* thanks to @Wolfgang for the thorough reviews and the correspondence off-list! Maintainer Notes ---------------- - pve-manager will need a version bump on pve-storage Notable Changes Since v2 ------------------------ - Rebase on the latest master - Take vtype-split changes that were recently introduced into account, notably the new `vm-vol`, `vm-state`, and `ct-vol` vtypes - Do a huge API bump at the end of the changes for pve-storage instead of in between, since that is actually easier to manage (and rebase) - Ignore symlinks when scanning through nested subdirectories, as that can otherwise lead to loops during the scan -- and those loops can then cause an exponential blowup in subdirectories being scanned - Keep track of content directories while scanning subdirs to avoid traversing content dirs that may be contained in another content dir, e.g. with `iso=data,snippets=data/snippets`; use device + inode here, to *really* make sure we catch all edge cases (e.g. bind mounts) - Explicitly remove trailing slashes from content directory overrides; with 'snippets=/' and 'images=imgs/', for example, a VM disk could have resolved to a snippet (yikes) - Retain calls to `PVE::Storage::Plugin->get_subdir()` throughout the code, as that would otherwise break API -- third-party plugin authors are still encouraged to switch to the new helpers instead of overriding `get_subdir()` Testing ------- If anyone could give this series a spin, I'd be most grateful! Here are some interesting things you could check out (non-exhaustive): - Configuring the new 'max-scan-depth' property in the UI - Adding subdirectories in your ISO, LXC template and snippets dirs (and populating those dirs afterwards) - Checking whether the depth limit is honored --> 0 is the default, which retains the current behavior of not scanning through any subdirs - Checking whether imports (.ova files etc) still work as expected - ISO / CT template upload / deletion - ... Also note that I ran the tests in the repository for every single patch that I added; if you want to do this for yourself as a sanity check, try the following: git rebase -i --autostash --autosquash origin/master -x 'cd src && make test' Previous Versions ----------------- v1: https://lore.proxmox.com/pve-devel/20260422111322.257380-1-m.carrara@proxmox.com/ v2: https://lore.proxmox.com/pve-devel/20260923150606.531239-1-m.carrara@proxmox.com/ References ---------- [2884]: https://bugzilla.proxmox.com/show_bug.cgi?id=2884 [623]: https://bugzilla.proxmox.com/show_bug.cgi?id=623 [init-cover]: https://lore.proxmox.com/pve-devel/20260422111322.257380-1-m.carrara@proxmox.com/ Summary of Changes ------------------ pve-storage: Max R. Carrara (50): test: plugin tests: run tests with at most 4 jobs plugin, common: remove superfluous use of =pod command paragraph common: use Exporter module for PVE::Storage::Common plugin: make get_subdir_files a proper subroutine and update style plugin api: replace subdir helper methods with standalone functions common: prevent autovivification in plugin_get_vtype_subdir helper plugin: break up if-elsif chain into separate if-blocks plugin: adapt get_subdir_files helper of list_volumes API method plugin: update code style of list_volumes plugin API method plugin: use closure for obtaining raw volume data in list_volumes plugin: use closure for inner loop logic in list_volumes storage: update code style in function path_to_volume_id storage: break up if-elsif chain in path_to_volume_id storage: heave vtype file path parsing logic inside loop into helper storage: clean up code that was moved into helper in path_to_volume_id api: status: move content type assert for up-/downloads into helper api: status: use get_subdir helper method to get content directory api: status: move up-/download file path parsing code into helper api: status: simplify file content assertion logic for up-/download test: guest import: add tests for PVE::GuestImport tree-wide: introduce parsing module and replace usages of ISO_EXT_RE_0 common: test: set up parser testing code, add tests for 'iso' vtype tree-wide: replace usages of VZTMPL_EXT_RE_1 with parsing functions tree-wide: replace usages of BACKUP_EXT_RE_2 with parsing functions tree-wide: replace usages of inline regexes for snippets with parsers tree-wide: partially replace usages of regexes for 'import' vtype tree-wide: replace remaining usages of regexes for 'import' vtype tree-wide: simplify recently refactored parsing logic test: list volumes: reorganize and modernize test running code test: list volumes: fix broken test checking for vmlist modifications test: list volumes: introduce new format for test cases test: list volumes: remove legacy code and migrate cases to new format plugin: do not return volumes of undeclared content types anymore plugin: correct comment in get_subdir_files helper test: parse volname: modernize code test: parse volname: adapt tests regarding 'import' volume type test: parse volname: move VM disk test creation into separate block test: parse volname: move backup file test creation into sep. block test: parse volname: parameterize test case creation for some vtypes test: volume id: modernize code test: volume id: rename 'volname' test case parameter to 'file' test: filesystem path: modernize code plugin: list content directories without globbing fix #2884: implement nested subdir scanning and support 'iso' vtype fix #2884: support nested subdir scanning for 'vztmpl' volume type fix #2884: support nested subdir scanning for 'snippets' vtype test: add more tests for 'import' vtype & guard against nested subdirs test: add tests guarding against subdir scanning for vtypes storage: resolve images in content dir overrides ending with a slash bump the storage plugin API version ApiChangeLog | 77 + debian/control | 1 + src/PVE/API2/Storage/Status.pm | 199 +- src/PVE/BackupProvider/Plugin/Base.pm | 6 +- src/PVE/GuestImport.pm | 28 +- src/PVE/GuestImport/OVF.pm | 24 +- src/PVE/Makefile | 1 + src/PVE/Storage.pm | 225 +- src/PVE/Storage/BTRFSPlugin.pm | 17 +- src/PVE/Storage/CIFSPlugin.pm | 1 + src/PVE/Storage/CephFSPlugin.pm | 1 + src/PVE/Storage/Common.pm | 102 +- src/PVE/Storage/Common/Makefile | 5 + src/PVE/Storage/Common/Parse.pm | 705 ++++++ src/PVE/Storage/Common/test/Makefile | 6 + src/PVE/Storage/Common/test/parser_tests.pl | 1256 ++++++++++ src/PVE/Storage/Common/test/run_tests.pl | 25 + src/PVE/Storage/DirPlugin.pm | 1 + src/PVE/Storage/ESXiPlugin.pm | 6 - src/PVE/Storage/Makefile | 4 + src/PVE/Storage/NFSPlugin.pm | 1 + src/PVE/Storage/Plugin.pm | 462 ++-- src/test/filesystem_path_test.pm | 118 +- src/test/get_subdir_test.pm | 12 +- src/test/guest_import_test.pl | 948 ++++++++ src/test/list_volumes_test.pm | 2405 +++++++++++++++---- src/test/parse_volname_test.pm | 776 ++++-- src/test/path_to_volume_id_test.pm | 300 ++- src/test/run_plugin_tests.pl | 28 +- src/test/run_volume_access_tests.pl | 5 +- 30 files changed, 6616 insertions(+), 1129 deletions(-) create mode 100644 src/PVE/Storage/Common/Parse.pm create mode 100644 src/PVE/Storage/Common/test/Makefile create mode 100755 src/PVE/Storage/Common/test/parser_tests.pl create mode 100755 src/PVE/Storage/Common/test/run_tests.pl create mode 100755 src/test/guest_import_test.pl pve-manager: Max R. Carrara (1): fix #2884: ui: storage: add field for 'max-scan-depth' property www/manager6/storage/Base.js | 14 ++++++++++++++ 1 file changed, 14 insertions(+) Summary over all repositories: 31 files changed, 6630 insertions(+), 1129 deletions(-) -- Generated by murpp 0.12.0