public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: "Max R. Carrara" <m.carrara@proxmox.com>
To: pve-devel@lists.proxmox.com
Cc: Wolfgang Bumiller <w.bumiller@proxmox.com>
Subject: [PATCH manager/storage v3 00/51] Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types
Date: Tue, 17 Nov 2026 21:38:32 +0100	[thread overview]
Message-ID: <20261117203953.913035-1-m.carrara@proxmox.com> (raw)

Fix #2884: Implement Subdirectory Scanning for Dir-Based Storage Types - v3
===========================================================================

Basically what the title says. Implement subdirectory scanning for
directory-based storage types, which includes directories (duh), NFS,
CIFS, CephFS, and BTRFS. This fixes #2884 [2884] and the more
narrowly-scoped #623 [623].

See the initial series's cover letter [init-cover] for the entire
context.

*Huge* thanks to @Wolfgang for the thorough reviews and the
correspondence off-list!

Maintainer Notes
----------------

- pve-manager will need a version bump on pve-storage

Notable Changes Since v2
------------------------

- Rebase on the latest master

- Take vtype-split changes that were recently introduced into account,
  notably the new `vm-vol`, `vm-state`, and `ct-vol` vtypes

- Do a huge API bump at the end of the changes for pve-storage instead
  of in between, since that is actually easier to manage (and rebase)

- Ignore symlinks when scanning through nested subdirectories, as that
  can otherwise lead to loops during the scan -- and those loops can
  then cause an exponential blowup in subdirectories being scanned

- Keep track of content directories while scanning subdirs to avoid
  traversing content dirs that may be contained in another content dir,
  e.g. with `iso=data,snippets=data/snippets`; use device + inode here,
  to *really* make sure we catch all edge cases (e.g. bind mounts)

- Explicitly remove trailing slashes from content directory overrides;
  with 'snippets=/' and 'images=imgs/', for example, a VM disk could
  have resolved to a snippet (yikes)

- Retain calls to `PVE::Storage::Plugin->get_subdir()` throughout the
  code, as that would otherwise break API -- third-party plugin authors
  are still encouraged to switch to the new helpers instead of
  overriding `get_subdir()`

Testing
-------

If anyone could give this series a spin, I'd be most grateful!

Here are some interesting things you could check out (non-exhaustive):
- Configuring the new 'max-scan-depth' property in the UI
- Adding subdirectories in your ISO, LXC template and snippets dirs
  (and populating those dirs afterwards)
- Checking whether the depth limit is honored
  --> 0 is the default, which retains the current behavior of not
      scanning through any subdirs
- Checking whether imports (.ova files etc) still work as expected
- ISO / CT template upload / deletion
- ...

Also note that I ran the tests in the repository for every single patch
that I added; if you want to do this for yourself as a sanity check, try
the following:

    git rebase -i --autostash --autosquash origin/master -x 'cd src && make test'

Previous Versions
-----------------

v1: https://lore.proxmox.com/pve-devel/20260422111322.257380-1-m.carrara@proxmox.com/
v2: https://lore.proxmox.com/pve-devel/20260923150606.531239-1-m.carrara@proxmox.com/

References
----------

[2884]: https://bugzilla.proxmox.com/show_bug.cgi?id=2884
[623]: https://bugzilla.proxmox.com/show_bug.cgi?id=623
[init-cover]: https://lore.proxmox.com/pve-devel/20260422111322.257380-1-m.carrara@proxmox.com/

Summary of Changes
------------------


pve-storage:

Max R. Carrara (50):
  test: plugin tests: run tests with at most 4 jobs
  plugin, common: remove superfluous use of =pod command paragraph
  common: use Exporter module for PVE::Storage::Common
  plugin: make get_subdir_files a proper subroutine and update style
  plugin api: replace subdir helper methods with standalone functions
  common: prevent autovivification in plugin_get_vtype_subdir helper
  plugin: break up if-elsif chain into separate if-blocks
  plugin: adapt get_subdir_files helper of list_volumes API method
  plugin: update code style of list_volumes plugin API method
  plugin: use closure for obtaining raw volume data in list_volumes
  plugin: use closure for inner loop logic in list_volumes
  storage: update code style in function path_to_volume_id
  storage: break up if-elsif chain in path_to_volume_id
  storage: heave vtype file path parsing logic inside loop into helper
  storage: clean up code that was moved into helper in path_to_volume_id
  api: status: move content type assert for up-/downloads into helper
  api: status: use get_subdir helper method to get content directory
  api: status: move up-/download file path parsing code into helper
  api: status: simplify file content assertion logic for up-/download
  test: guest import: add tests for PVE::GuestImport
  tree-wide: introduce parsing module and replace usages of ISO_EXT_RE_0
  common: test: set up parser testing code, add tests for 'iso' vtype
  tree-wide: replace usages of VZTMPL_EXT_RE_1 with parsing functions
  tree-wide: replace usages of BACKUP_EXT_RE_2 with parsing functions
  tree-wide: replace usages of inline regexes for snippets with parsers
  tree-wide: partially replace usages of regexes for 'import' vtype
  tree-wide: replace remaining usages of regexes for 'import' vtype
  tree-wide: simplify recently refactored parsing logic
  test: list volumes: reorganize and modernize test running code
  test: list volumes: fix broken test checking for vmlist modifications
  test: list volumes: introduce new format for test cases
  test: list volumes: remove legacy code and migrate cases to new format
  plugin: do not return volumes of undeclared content types anymore
  plugin: correct comment in get_subdir_files helper
  test: parse volname: modernize code
  test: parse volname: adapt tests regarding 'import' volume type
  test: parse volname: move VM disk test creation into separate block
  test: parse volname: move backup file test creation into sep. block
  test: parse volname: parameterize test case creation for some vtypes
  test: volume id: modernize code
  test: volume id: rename 'volname' test case parameter to 'file'
  test: filesystem path: modernize code
  plugin: list content directories without globbing
  fix #2884: implement nested subdir scanning and support 'iso' vtype
  fix #2884: support nested subdir scanning for 'vztmpl' volume type
  fix #2884: support nested subdir scanning for 'snippets' vtype
  test: add more tests for 'import' vtype & guard against nested subdirs
  test: add tests guarding against subdir scanning for vtypes
  storage: resolve images in content dir overrides ending with a slash
  bump the storage plugin API version

 ApiChangeLog                                |   77 +
 debian/control                              |    1 +
 src/PVE/API2/Storage/Status.pm              |  199 +-
 src/PVE/BackupProvider/Plugin/Base.pm       |    6 +-
 src/PVE/GuestImport.pm                      |   28 +-
 src/PVE/GuestImport/OVF.pm                  |   24 +-
 src/PVE/Makefile                            |    1 +
 src/PVE/Storage.pm                          |  225 +-
 src/PVE/Storage/BTRFSPlugin.pm              |   17 +-
 src/PVE/Storage/CIFSPlugin.pm               |    1 +
 src/PVE/Storage/CephFSPlugin.pm             |    1 +
 src/PVE/Storage/Common.pm                   |  102 +-
 src/PVE/Storage/Common/Makefile             |    5 +
 src/PVE/Storage/Common/Parse.pm             |  705 ++++++
 src/PVE/Storage/Common/test/Makefile        |    6 +
 src/PVE/Storage/Common/test/parser_tests.pl | 1256 ++++++++++
 src/PVE/Storage/Common/test/run_tests.pl    |   25 +
 src/PVE/Storage/DirPlugin.pm                |    1 +
 src/PVE/Storage/ESXiPlugin.pm               |    6 -
 src/PVE/Storage/Makefile                    |    4 +
 src/PVE/Storage/NFSPlugin.pm                |    1 +
 src/PVE/Storage/Plugin.pm                   |  462 ++--
 src/test/filesystem_path_test.pm            |  118 +-
 src/test/get_subdir_test.pm                 |   12 +-
 src/test/guest_import_test.pl               |  948 ++++++++
 src/test/list_volumes_test.pm               | 2405 +++++++++++++++----
 src/test/parse_volname_test.pm              |  776 ++++--
 src/test/path_to_volume_id_test.pm          |  300 ++-
 src/test/run_plugin_tests.pl                |   28 +-
 src/test/run_volume_access_tests.pl         |    5 +-
 30 files changed, 6616 insertions(+), 1129 deletions(-)
 create mode 100644 src/PVE/Storage/Common/Parse.pm
 create mode 100644 src/PVE/Storage/Common/test/Makefile
 create mode 100755 src/PVE/Storage/Common/test/parser_tests.pl
 create mode 100755 src/PVE/Storage/Common/test/run_tests.pl
 create mode 100755 src/test/guest_import_test.pl


pve-manager:

Max R. Carrara (1):
  fix #2884: ui: storage: add field for 'max-scan-depth' property

 www/manager6/storage/Base.js | 14 ++++++++++++++
 1 file changed, 14 insertions(+)


Summary over all repositories:
  31 files changed, 6630 insertions(+), 1129 deletions(-)

-- 
Generated by murpp 0.12.0




             reply	other threads:[~2026-11-17 20:40 UTC|newest]

Thread overview: 52+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-17 20:38 Max R. Carrara [this message]
2026-11-17 20:38 ` [PATCH pve-storage v3 01/51] test: plugin tests: run tests with at most 4 jobs Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 02/51] plugin, common: remove superfluous use of =pod command paragraph Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 03/51] common: use Exporter module for PVE::Storage::Common Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 04/51] plugin: make get_subdir_files a proper subroutine and update style Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 05/51] plugin api: replace subdir helper methods with standalone functions Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 06/51] common: prevent autovivification in plugin_get_vtype_subdir helper Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 07/51] plugin: break up if-elsif chain into separate if-blocks Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 08/51] plugin: adapt get_subdir_files helper of list_volumes API method Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 09/51] plugin: update code style of list_volumes plugin " Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 10/51] plugin: use closure for obtaining raw volume data in list_volumes Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 11/51] plugin: use closure for inner loop logic " Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 12/51] storage: update code style in function path_to_volume_id Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 13/51] storage: break up if-elsif chain in path_to_volume_id Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 14/51] storage: heave vtype file path parsing logic inside loop into helper Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 15/51] storage: clean up code that was moved into helper in path_to_volume_id Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 16/51] api: status: move content type assert for up-/downloads into helper Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 17/51] api: status: use get_subdir helper method to get content directory Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 18/51] api: status: move up-/download file path parsing code into helper Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 19/51] api: status: simplify file content assertion logic for up-/download Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 20/51] test: guest import: add tests for PVE::GuestImport Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 21/51] tree-wide: introduce parsing module and replace usages of ISO_EXT_RE_0 Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 22/51] common: test: set up parser testing code, add tests for 'iso' vtype Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 23/51] tree-wide: replace usages of VZTMPL_EXT_RE_1 with parsing functions Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 24/51] tree-wide: replace usages of BACKUP_EXT_RE_2 " Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 25/51] tree-wide: replace usages of inline regexes for snippets with parsers Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 26/51] tree-wide: partially replace usages of regexes for 'import' vtype Max R. Carrara
2026-11-17 20:38 ` [PATCH pve-storage v3 27/51] tree-wide: replace remaining " Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 28/51] tree-wide: simplify recently refactored parsing logic Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 29/51] test: list volumes: reorganize and modernize test running code Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 30/51] test: list volumes: fix broken test checking for vmlist modifications Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 31/51] test: list volumes: introduce new format for test cases Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 32/51] test: list volumes: remove legacy code and migrate cases to new format Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 33/51] plugin: do not return volumes of undeclared content types anymore Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 34/51] plugin: correct comment in get_subdir_files helper Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 35/51] test: parse volname: modernize code Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 36/51] test: parse volname: adapt tests regarding 'import' volume type Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 37/51] test: parse volname: move VM disk test creation into separate block Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 38/51] test: parse volname: move backup file test creation into sep. block Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 39/51] test: parse volname: parameterize test case creation for some vtypes Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 40/51] test: volume id: modernize code Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 41/51] test: volume id: rename 'volname' test case parameter to 'file' Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 42/51] test: filesystem path: modernize code Max R. Carrara
2026-11-17 20:39 ` [PATCH pve-storage v3 43/51] plugin: list content directories without globbing Max R. Carrara
2026-11-18  9:12 ` [PATCH pve-storage v3 44/51] fix #2884: implement nested subdir scanning and support 'iso' vtype Max R. Carrara
2026-11-18  9:14 ` [PATCH pve-storage v3 45/51] fix #2884: support nested subdir scanning for 'vztmpl' volume type Max R. Carrara
2026-11-18  9:14   ` [PATCH pve-storage v3 46/51] fix #2884: support nested subdir scanning for 'snippets' vtype Max R. Carrara
2026-11-18  9:14   ` [PATCH pve-storage v3 47/51] test: add more tests for 'import' vtype & guard against nested subdirs Max R. Carrara
2026-11-18  9:14   ` [PATCH pve-storage v3 48/51] test: add tests guarding against subdir scanning for vtypes Max R. Carrara
2026-11-18  9:14   ` [PATCH pve-storage v3 49/51] storage: resolve images in content dir overrides ending with a slash Max R. Carrara
2026-11-18  9:14   ` [PATCH pve-storage v3 50/51] bump the storage plugin API version Max R. Carrara
2026-11-18  9:14   ` [PATCH pve-manager v3 51/51] fix #2884: ui: storage: add field for 'max-scan-depth' property Max R. Carrara

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261117203953.913035-1-m.carrara@proxmox.com \
    --to=m.carrara@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    --cc=w.bumiller@proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal