From: "Max R. Carrara" <m.carrara@proxmox.com>
To: pve-devel@lists.proxmox.com
Cc: Wolfgang Bumiller <w.bumiller@proxmox.com>
Subject: [PATCH manager/storage v3 00/51] Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types
Date: Tue, 17 Nov 2026 21:38:32 +0100 [thread overview]
Message-ID: <20261117203953.913035-1-m.carrara@proxmox.com> (raw)
Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types - v3
===========================================================================
Basically what the title says. Implement subdirectory scanning for
directory-based storage types, which includes directories (duh), NFS,
CIFS, CephFS, and BTRFS. This fixes #2884 [2884] and the more
narrowly-scoped #623 [623].
See the initial series's cover letter [init-cover] for the entire
context.
*Huge* thanks to @Wolfgang for the thorough reviews and the
correspondence off-list!
Maintainer Notes
----------------
- pve-manager will need a version bump on pve-storage
Notable Changes Since v2
------------------------
- Rebase on the latest master
- Take vtype-split changes that were recently introduced into account,
notably the new `vm-vol`, `vm-state`, and `ct-vol` vtypes
- Do a huge API bump at the end of the changes for pve-storage instead
of in between, since that is actually easier to manage (and rebase)
- Ignore symlinks when scanning through nested subdirectories, as that
can otherwise lead to loops during the scan -- and those loops can
then cause an exponential blowup in subdirectories being scanned
- Keep track of content directories while scanning subdirs to avoid
traversing content dirs that may be contained in another content dir,
e.g. with `iso=data,snippets=data/snippets`; use device + inode here,
to *really* make sure we catch all edge cases (e.g. bind mounts)
- Explicitly remove trailing slashes from content directory overrides;
with 'snippets=/' and 'images=imgs/', for example, a VM disk could
have resolved to a snippet (yikes)
- Retain calls to `PVE::Storage::Plugin->get_subdir()` throughout the
code, as that would otherwise break API -- third-party plugin authors
are still encouraged to switch to the new helpers instead of
overriding `get_subdir()`
Testing
-------
If anyone could give this series a spin, I'd be most grateful!
Here are some interesting things you could check out (non-exhaustive):
- Configuring the new 'max-scan-depth' property in the UI
- Adding subdirectories in your ISO, LXC template and snippets dirs
(and populating those dirs afterwards)
- Checking whether the depth limit is honored
--> 0 is the default, which retains the current behavior of not
scanning through any subdirs
- Checking whether imports (.ova files etc) still work as expected
- ISO / CT template upload / deletion
- ...
Also note that I ran the tests in the repository for every single patch
that I added; if you want to do this for yourself as a sanity check, try
the following:
git rebase -i --autostash --autosquash origin/master -x 'cd src && make test'
Previous Versions
-----------------
v1: https://lore.proxmox.com/pve-devel/20260422111322.257380-1-m.carrara@proxmox.com/
v2: https://lore.proxmox.com/pve-devel/20260923150606.531239-1-m.carrara@proxmox.com/
References
----------
[2884]: https://bugzilla.proxmox.com/show_bug.cgi?id=2884
[623]: https://bugzilla.proxmox.com/show_bug.cgi?id=623
[init-cover]: https://lore.proxmox.com/pve-devel/20260422111322.257380-1-m.carrara@proxmox.com/
Summary of Changes
------------------
pve-storage:
Max R. Carrara (50):
test: plugin tests: run tests with at most 4 jobs
plugin, common: remove superfluous use of =pod command paragraph
common: use Exporter module for PVE::Storage::Common
plugin: make get_subdir_files a proper subroutine and update style
plugin api: replace subdir helper methods with standalone functions
common: prevent autovivification in plugin_get_vtype_subdir helper
plugin: break up if-elsif chain into separate if-blocks
plugin: adapt get_subdir_files helper of list_volumes API method
plugin: update code style of list_volumes plugin API method
plugin: use closure for obtaining raw volume data in list_volumes
plugin: use closure for inner loop logic in list_volumes
storage: update code style in function path_to_volume_id
storage: break up if-elsif chain in path_to_volume_id
storage: heave vtype file path parsing logic inside loop into helper
storage: clean up code that was moved into helper in path_to_volume_id
api: status: move content type assert for up-/downloads into helper
api: status: use get_subdir helper method to get content directory
api: status: move up-/download file path parsing code into helper
api: status: simplify file content assertion logic for up-/download
test: guest import: add tests for PVE::GuestImport
tree-wide: introduce parsing module and replace usages of ISO_EXT_RE_0
common: test: set up parser testing code, add tests for 'iso' vtype
tree-wide: replace usages of VZTMPL_EXT_RE_1 with parsing functions
tree-wide: replace usages of BACKUP_EXT_RE_2 with parsing functions
tree-wide: replace usages of inline regexes for snippets with parsers
tree-wide: partially replace usages of regexes for 'import' vtype
tree-wide: replace remaining usages of regexes for 'import' vtype
tree-wide: simplify recently refactored parsing logic
test: list volumes: reorganize and modernize test running code
test: list volumes: fix broken test checking for vmlist modifications
test: list volumes: introduce new format for test cases
test: list volumes: remove legacy code and migrate cases to new format
plugin: do not return volumes of undeclared content types anymore
plugin: correct comment in get_subdir_files helper
test: parse volname: modernize code
test: parse volname: adapt tests regarding 'import' volume type
test: parse volname: move VM disk test creation into separate block
test: parse volname: move backup file test creation into sep. block
test: parse volname: parameterize test case creation for some vtypes
test: volume id: modernize code
test: volume id: rename 'volname' test case parameter to 'file'
test: filesystem path: modernize code
plugin: list content directories without globbing
fix #2884: implement nested subdir scanning and support 'iso' vtype
fix #2884: support nested subdir scanning for 'vztmpl' volume type
fix #2884: support nested subdir scanning for 'snippets' vtype
test: add more tests for 'import' vtype & guard against nested subdirs
test: add tests guarding against subdir scanning for vtypes
storage: resolve images in content dir overrides ending with a slash
bump the storage plugin API version
ApiChangeLog | 77 +
debian/control | 1 +
src/PVE/API2/Storage/Status.pm | 199 +-
src/PVE/BackupProvider/Plugin/Base.pm | 6 +-
src/PVE/GuestImport.pm | 28 +-
src/PVE/GuestImport/OVF.pm | 24 +-
src/PVE/Makefile | 1 +
src/PVE/Storage.pm | 225 +-
src/PVE/Storage/BTRFSPlugin.pm | 17 +-
src/PVE/Storage/CIFSPlugin.pm | 1 +
src/PVE/Storage/CephFSPlugin.pm | 1 +
src/PVE/Storage/Common.pm | 102 +-
src/PVE/Storage/Common/Makefile | 5 +
src/PVE/Storage/Common/Parse.pm | 705 ++++++
src/PVE/Storage/Common/test/Makefile | 6 +
src/PVE/Storage/Common/test/parser_tests.pl | 1256 ++++++++++
src/PVE/Storage/Common/test/run_tests.pl | 25 +
src/PVE/Storage/DirPlugin.pm | 1 +
src/PVE/Storage/ESXiPlugin.pm | 6 -
src/PVE/Storage/Makefile | 4 +
src/PVE/Storage/NFSPlugin.pm | 1 +
src/PVE/Storage/Plugin.pm | 462 ++--
src/test/filesystem_path_test.pm | 118 +-
src/test/get_subdir_test.pm | 12 +-
src/test/guest_import_test.pl | 948 ++++++++
src/test/list_volumes_test.pm | 2405 +++++++++++++++----
src/test/parse_volname_test.pm | 776 ++++--
src/test/path_to_volume_id_test.pm | 300 ++-
src/test/run_plugin_tests.pl | 28 +-
src/test/run_volume_access_tests.pl | 5 +-
30 files changed, 6616 insertions(+), 1129 deletions(-)
create mode 100644 src/PVE/Storage/Common/Parse.pm
create mode 100644 src/PVE/Storage/Common/test/Makefile
create mode 100755 src/PVE/Storage/Common/test/parser_tests.pl
create mode 100755 src/PVE/Storage/Common/test/run_tests.pl
create mode 100755 src/test/guest_import_test.pl
pve-manager:
Max R. Carrara (1):
fix #2884: ui: storage: add field for 'max-scan-depth' property
www/manager6/storage/Base.js | 14 ++++++++++++++
1 file changed, 14 insertions(+)
Summary over all repositories:
31 files changed, 6630 insertions(+), 1129 deletions(-)
--
Generated by murpp 0.12.0
next reply other threads:[~2026-11-17 20:40 UTC|newest]
Thread overview: 52+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 20:38 Max R. Carrara [this message]
2026-11-17 20:38 ` [PATCH pve-storage v3 01/51] test: plugin tests: run tests with at most 4 jobs Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 02/51] plugin, common: remove superfluous use of =pod command paragraph Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 03/51] common: use Exporter module for PVE::Storage::Common Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 04/51] plugin: make get_subdir_files a proper subroutine and update style Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 05/51] plugin api: replace subdir helper methods with standalone functions Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 06/51] common: prevent autovivification in plugin_get_vtype_subdir helper Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 07/51] plugin: break up if-elsif chain into separate if-blocks Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 08/51] plugin: adapt get_subdir_files helper of list_volumes API method Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 09/51] plugin: update code style of list_volumes plugin " Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 10/51] plugin: use closure for obtaining raw volume data in list_volumes Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 11/51] plugin: use closure for inner loop logic " Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 12/51] storage: update code style in function path_to_volume_id Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 13/51] storage: break up if-elsif chain in path_to_volume_id Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 14/51] storage: heave vtype file path parsing logic inside loop into helper Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 15/51] storage: clean up code that was moved into helper in path_to_volume_id Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 16/51] api: status: move content type assert for up-/downloads into helper Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 17/51] api: status: use get_subdir helper method to get content directory Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 18/51] api: status: move up-/download file path parsing code into helper Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 19/51] api: status: simplify file content assertion logic for up-/download Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 20/51] test: guest import: add tests for PVE::GuestImport Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 21/51] tree-wide: introduce parsing module and replace usages of ISO_EXT_RE_0 Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 22/51] common: test: set up parser testing code, add tests for 'iso' vtype Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 23/51] tree-wide: replace usages of VZTMPL_EXT_RE_1 with parsing functions Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 24/51] tree-wide: replace usages of BACKUP_EXT_RE_2 " Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 25/51] tree-wide: replace usages of inline regexes for snippets with parsers Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 26/51] tree-wide: partially replace usages of regexes for 'import' vtype Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 27/51] tree-wide: replace remaining " Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 28/51] tree-wide: simplify recently refactored parsing logic Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 29/51] test: list volumes: reorganize and modernize test running code Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 30/51] test: list volumes: fix broken test checking for vmlist modifications Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 31/51] test: list volumes: introduce new format for test cases Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 32/51] test: list volumes: remove legacy code and migrate cases to new format Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 33/51] plugin: do not return volumes of undeclared content types anymore Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 34/51] plugin: correct comment in get_subdir_files helper Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 35/51] test: parse volname: modernize code Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 36/51] test: parse volname: adapt tests regarding 'import' volume type Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 37/51] test: parse volname: move VM disk test creation into separate block Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 38/51] test: parse volname: move backup file test creation into sep. block Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 39/51] test: parse volname: parameterize test case creation for some vtypes Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 40/51] test: volume id: modernize code Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 41/51] test: volume id: rename 'volname' test case parameter to 'file' Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 42/51] test: filesystem path: modernize code Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 43/51] plugin: list content directories without globbing Max R. Carrara
2026-11-18 9:12 ` [PATCH pve-storage v3 44/51] fix #2884: implement nested subdir scanning and support 'iso' vtype Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-storage v3 45/51] fix #2884: support nested subdir scanning for 'vztmpl' volume type Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-storage v3 46/51] fix #2884: support nested subdir scanning for 'snippets' vtype Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-storage v3 47/51] test: add more tests for 'import' vtype & guard against nested subdirs Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-storage v3 48/51] test: add tests guarding against subdir scanning for vtypes Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-storage v3 49/51] storage: resolve images in content dir overrides ending with a slash Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-storage v3 50/51] bump the storage plugin API version Max R. Carrara
2026-11-18 9:14 ` [PATCH pve-manager v3 51/51] fix #2884: ui: storage: add field for 'max-scan-depth' property Max R. Carrara
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261117203953.913035-1-m.carrara@proxmox.com \
--to=m.carrara@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
--cc=w.bumiller@proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.