From: "Max R. Carrara" <m.carrara@proxmox.com>
To: pve-devel@lists.proxmox.com
Cc: Wolfgang Bumiller <w.bumiller@proxmox.com>
Subject: [PATCH manager/storage v3 00/51] Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types
Date: Tue, 17 Nov 2026 21:38:32 +0100 [thread overview]
Message-ID: <20261117203953.913035-1-m.carrara@proxmox.com> (raw)
Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types - v3
===========================================================================
Basically what the title says. Implement subdirectory scanning for
directory-based storage types, which includes directories (duh), NFS,
CIFS, CephFS, and BTRFS. This fixes #2884 [2884] and the more
narrowly-scoped #623 [623].
See the initial series's cover letter [init-cover] for the entire
context.
*Huge* thanks to @Wolfgang for the thorough reviews and the
correspondence off-list!
Maintainer Notes
----------------
- pve-manager will need a version bump on pve-storage
Notable Changes Since v2
------------------------
- Rebase on the latest master
- Take vtype-split changes that were recently introduced into account,
notably the new `vm-vol`, `vm-state`, and `ct-vol` vtypes
- Do a huge API bump at the end of the changes for pve-storage instead
of in between, since that is actually easier to manage (and rebase)
- Ignore symlinks when scanning through nested subdirectories, as that
can otherwise lead to loops during the scan -- and those loops can
then cause an exponential blowup in subdirectories being scanned
- Keep track of content directories while scanning subdirs to avoid
traversing content dirs that may be contained in another content dir,
e.g. with `iso=data,snippets=data/snippets`; use device + inode here,
to *really* make sure we catch all edge cases (e.g. bind mounts)
- Explicitly remove trailing slashes from content directory overrides;
with 'snippets=/' and 'images=imgs/', for example, a VM disk could
have resolved to a snippet (yikes)
- Retain calls to `PVE::Storage::Plugin->get_subdir()` throughout the
code, as that would otherwise break API -- third-party plugin authors
are still encouraged to switch to the new helpers instead of
overriding `get_subdir()`
Testing
-------
If anyone could give this series a spin, I'd be most grateful!
Here are some interesting things you could check out (non-exhaustive):
- Configuring the new 'max-scan-depth' property in the UI
- Adding subdirectories in your ISO, LXC template and snippets dirs
(and populating those dirs afterwards)
- Checking whether the depth limit is honored
--> 0 is the default, which retains the current behavior of not
scanning through any subdirs
- Checking whether imports (.ova files etc) still work as expected
- ISO / CT template upload / deletion
- ...
Also note that I ran the tests in the repository for every single patch
that I added; if you want to do this for yourself as a sanity check, try
the following:
git rebase -i --autostash --autosquash origin/master -x 'cd src && make test'
Previous Versions
-----------------
v1: https://lore.proxmox.com/pve-devel/20260422111322.257380-1-m.carrara@proxmox.com/
v2: https://lore.proxmox.com/pve-devel/20260923150606.531239-1-m.carrara@proxmox.com/
References
----------
[2884]: https://bugzilla.proxmox.com/show_bug.cgi?id=2884
[623]: https://bugzilla.proxmox.com/show_bug.cgi?id=623
[init-cover]: https://lore.proxmox.com/pve-devel/20260422111322.257380-1-m.carrara@proxmox.com/
Summary of Changes
------------------
pve-storage:
Max R. Carrara (50):
test: plugin tests: run tests with at most 4 jobs
plugin, common: remove superfluous use of =pod command paragraph
common: use Exporter module for PVE::Storage::Common
plugin: make get_subdir_files a proper subroutine and update style
plugin api: replace subdir helper methods with standalone functions
common: prevent autovivification in plugin_get_vtype_subdir helper
plugin: break up if-elsif chain into separate if-blocks
plugin: adapt get_subdir_files helper of list_volumes API method
plugin: update code style of list_volumes plugin API method
plugin: use closure for obtaining raw volume data in list_volumes
plugin: use closure for inner loop logic in list_volumes
storage: update code style in function path_to_volume_id
storage: break up if-elsif chain in path_to_volume_id
storage: heave vtype file path parsing logic inside loop into helper
storage: clean up code that was moved into helper in path_to_volume_id
api: status: move content type assert for up-/downloads into helper
api: status: use get_subdir helper method to get content directory
api: status: move up-/download file path parsing code into helper
api: status: simplify file content assertion logic for up-/download
test: guest import: add tests for PVE::GuestImport
tree-wide: introduce parsing module and replace usages of ISO_EXT_RE_0
common: test: set up parser testing code, add tests for 'iso' vtype
tree-wide: replace usages of VZTMPL_EXT_RE_1 with parsing functions
tree-wide: replace usages of BACKUP_EXT_RE_2 with parsing functions
tree-wide: replace usages of inline regexes for snippets with parsers
tree-wide: partially replace usages of regexes for 'import' vtype
tree-wide: replace remaining usages of regexes for 'import' vtype
tree-wide: simplify recently refactored parsing logic
test: list volumes: reorganize and modernize test running code
test: list volumes: fix broken test checking for vmlist modifications
test: list volumes: introduce new format for test cases
test: list volumes: remove legacy code and migrate cases to new format
plugin: do not return volumes of undeclared content types anymore
plugin: correct comment in get_subdir_files helper
test: parse volname: modernize code
test: parse volname: adapt tests regarding 'import' volume type
test: parse volname: move VM disk test creation into separate block
test: parse volname: move backup file test creation into sep. block
test: parse volname: parameterize test case creation for some vtypes
test: volume id: modernize code
test: volume id: rename 'volname' test case parameter to 'file'
test: filesystem path: modernize code
plugin: list content directories without globbing
fix #2884: implement nested subdir scanning and support 'iso' vtype
fix #2884: support nested subdir scanning for 'vztmpl' volume type
fix #2884: support nested subdir scanning for 'snippets' vtype
test: add more tests for 'import' vtype & guard against nested subdirs
test: add tests guarding against subdir scanning for vtypes
storage: resolve images in content dir overrides ending with a slash
bump the storage plugin API version
ApiChangeLog | 77 +
debian/control | 1 +
src/PVE/API2/Storage/Status.pm | 199 +-
src/PVE/BackupProvider/Plugin/Base.pm | 6 +-
src/PVE/GuestImport.pm | 28 +-
src/PVE/GuestImport/OVF.pm | 24 +-
src/PVE/Makefile | 1 +
src/PVE/Storage.pm | 225 +-
src/PVE/Storage/BTRFSPlugin.pm | 17 +-
src/PVE/Storage/CIFSPlugin.pm | 1 +
src/PVE/Storage/CephFSPlugin.pm | 1 +
src/PVE/Storage/Common.pm | 102 +-
src/PVE/Storage/Common/Makefile | 5 +
src/PVE/Storage/Common/Parse.pm | 705 ++++++
src/PVE/Storage/Common/test/Makefile | 6 +
src/PVE/Storage/Common/test/parser_tests.pl | 1256 ++++++++++
src/PVE/Storage/Common/test/run_tests.pl | 25 +
src/PVE/Storage/DirPlugin.pm | 1 +
src/PVE/Storage/ESXiPlugin.pm | 6 -
src/PVE/Storage/Makefile | 4 +
src/PVE/Storage/NFSPlugin.pm | 1 +
src/PVE/Storage/Plugin.pm | 462 ++--
src/test/filesystem_path_test.pm | 118 +-
src/test/get_subdir_test.pm | 12 +-
src/test/guest_import_test.pl | 948 ++++++++
src/test/list_volumes_test.pm | 2405 +++++++++++++++----
src/test/parse_volname_test.pm | 776 ++++--
src/test/path_to_volume_id_test.pm | 300 ++-
src/test/run_plugin_tests.pl | 28 +-
src/test/run_volume_access_tests.pl | 5 +-
30 files changed, 6616 insertions(+), 1129 deletions(-)
create mode 100644 src/PVE/Storage/Common/Parse.pm
create mode 100644 src/PVE/Storage/Common/test/Makefile
create mode 100755 src/PVE/Storage/Common/test/parser_tests.pl
create mode 100755 src/PVE/Storage/Common/test/run_tests.pl
create mode 100755 src/test/guest_import_test.pl
pve-manager:
Max R. Carrara (1):
fix #2884: ui: storage: add field for 'max-scan-depth' property
www/manager6/storage/Base.js | 14 ++++++++++++++
1 file changed, 14 insertions(+)
Summary over all repositories:
31 files changed, 6630 insertions(+), 1129 deletions(-)
--
Generated by murpp 0.12.0
next reply other threads:[~2026-11-17 20:40 UTC|newest]
Thread overview: 52+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 20:38 Max R. Carrara [this message]
2026-11-17 20:38 ` [PATCH pve-storage v3 01/51] test: plugin tests: run tests with at most 4 jobs Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 02/51] plugin, common: remove superfluous use of =pod command paragraph Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 03/51] common: use Exporter module for PVE::Storage::Common Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 04/51] plugin: make get_subdir_files a proper subroutine and update style Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 05/51] plugin api: replace subdir helper methods with standalone functions Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 06/51] common: prevent autovivification in plugin_get_vtype_subdir helper Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 07/51] plugin: break up if-elsif chain into separate if-blocks Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 08/51] plugin: adapt get_subdir_files helper of list_volumes API method Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 09/51] plugin: update code style of list_volumes plugin " Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 10/51] plugin: use closure for obtaining raw volume data in list_volumes Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 11/51] plugin: use closure for inner loop logic " Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 12/51] storage: update code style in function path_to_volume_id Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 13/51] storage: break up if-elsif chain in path_to_volume_id Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 14/51] storage: heave vtype file path parsing logic inside loop into helper Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 15/51] storage: clean up code that was moved into helper in path_to_volume_id Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 16/51] api: status: move content type assert for up-/downloads into helper Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 17/51] api: status: use get_subdir helper method to get content directory Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 18/51] api: status: move up-/download file path parsing code into helper Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 19/51] api: status: simplify file content assertion logic for up-/download Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 20/51] test: guest import: add tests for PVE::GuestImport Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 21/51] tree-wide: introduce parsing module and replace usages of ISO_EXT_RE_0 Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 22/51] common: test: set up parser testing code, add tests for 'iso' vtype Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 23/51] tree-wide: replace usages of VZTMPL_EXT_RE_1 with parsing functions Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 24/51] tree-wide: replace usages of BACKUP_EXT_RE_2 " Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 25/51] tree-wide: replace usages of inline regexes for snippets with parsers Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 26/51] tree-wide: partially replace usages of regexes for 'import' vtype Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 27/51] tree-wide: replace remaining " Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 28/51] tree-wide: simplify recently refactored parsing logic Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 29/51] test: list volumes: reorganize and modernize test running code Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 30/51] test: list volumes: fix broken test checking for vmlist modifications Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 31/51] test: list volumes: introduce new format for test cases Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 32/51] test: list volumes: remove legacy code and migrate cases to new format Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 33/51] plugin: do not return volumes of undeclared content types anymore Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 34/51] plugin: correct comment in get_subdir_files helper Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 35/51] test: parse volname: modernize code Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 36/51] test: parse volname: adapt tests regarding 'import' volume type Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 37/51] test: parse volname: move VM disk test creation into separate block Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 38/51] test: parse volname: move backup file test creation into sep. block Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 39/51] test: parse volname: parameterize test case creation for some vtypes Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 40/51] test: volume id: modernize code Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 41/51] test: volume id: rename 'volname' test case parameter to 'file' Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 42/51] test: filesystem path: modernize code Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 43/51] plugin: list content directories without globbing Max R. Carrara
2026-11-18 9:12 ` [PATCH pve-storage v3 44/51] fix #2884: implement nested subdir scanning and support 'iso' vtype Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-storage v3 45/51] fix #2884: support nested subdir scanning for 'vztmpl' volume type Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-storage v3 46/51] fix #2884: support nested subdir scanning for 'snippets' vtype Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-storage v3 47/51] test: add more tests for 'import' vtype & guard against nested subdirs Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-storage v3 48/51] test: add tests guarding against subdir scanning for vtypes Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-storage v3 49/51] storage: resolve images in content dir overrides ending with a slash Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-storage v3 50/51] bump the storage plugin API version Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-manager v3 51/51] fix #2884: ui: storage: add field for 'max-scan-depth' property Max R. Carrara
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261117203953.913035-1-m.carrara@proxmox.com \
--to=m.carrara@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
--cc=w.bumiller@proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox