Re: [pve-devel] [PATCH ceph 1/4 v5] cherry-pick changes to use system packages for bundled deps

["Kefu Chai" <k.chai@proxmox.com>]

Hi Fabian,

Is there anything I can do to help move this patchset forward? I'd be
happy to extract the libxsimd-dev changes into a separate commit if that
would make review easier, as it might be less controversial on its own.

On Mon Jan 26, 2026 at 5:06 PM CST, Fabian Grünbichler wrote:
> On January 24, 2026 5:13 am, Kefu Chai wrote:
>> On Fri Jan 23, 2026 at 9:03 PM CST, Fabian Grünbichler wrote:
>>> one small question inline, otherwise this one LGTM
>>>
>>> On January 23, 2026 8:56 am, Kefu Chai wrote:
>>>> Previously, cephadm's build process pulled dependencies from PyPI,
>>>> which failed in environments with restricted network access. This
>>>> change backports upstream improvements that enable building cephadm
>>>> using system packages instead, mirroring the existing RPM packaging
>>>> workflow.
>>>> 
>>>> And the vendored Arrow pulled xsimd from GitHub, this also failed in
>>>> building environment without network access. This change backports
>>>> upstream improvements enabling the build to use libxsimd-dev package
>>>> when building debian packages.
>>>> 
>>>> Backported changes:
>>>> - https://github.com/ceph/ceph/pull/66256 - Use system packages for
>>>>   cephadm bundled dependencies
>>>> - https://github.com/ceph/ceph/pull/66248 - Add libxsimd-dev build
>>>>   dependency for vendored Arrow
>>>> - https://github.com/ceph/ceph/pull/65292 - cephadm: fix building
>>>>   rpm-sourced cephadm zippapp on el10
>>>> 
>>>> Note: Test commits from these PRs are excluded from this backport.
>>>> 
>>>> Signed-off-by: Kefu Chai <k.chai@proxmox.com>
>>>> ---
>>>>  ...-issues-running-existing-cephadm-bui.patch |  66 +++++
>>>>  ...root_entries-population-in-version-c.patch |  62 +++++
>>>>  ...cephadm-rpm-based-builds-without-top.patch |  89 ++++++
>>>>  ...d-Debian-package-support-for-bundled.patch | 261 ++++++++++++++++++
>>>>  ...m-packages-for-cephadm-bundled-depen.patch |  54 ++++
>>>>  ...-Use-AUTO-mode-for-xsimd-dependency-.patch |  55 ++++
>>>>  ...dd-libxsimd-dev-build-dependency-for.patch |  41 +++
>>>>  patches/series                                |   8 +
>>>>  8 files changed, 636 insertions(+)
>>>>  create mode 100644 patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
>>>>  create mode 100644 patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
>>>>  create mode 100644 patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
>>>>  create mode 100644 patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
>>>>  create mode 100644 patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
>>>>  create mode 100644 patches/0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch
>>>>  create mode 100644 patches/0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch
>>>> 
>>>> diff --git a/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch b/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
>>>> new file mode 100644
>>>> index 00000000000..c6208b0db2a
>>>> --- /dev/null
>>>> +++ b/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
>>>> @@ -0,0 +1,66 @@
>>>> +From 2685f6c35245d033e73e16ece3e356fcead44cea Mon Sep 17 00:00:00 2001
>>>> +From: John Mulligan <jmulligan@redhat.com>
>>>> +Date: Thu, 21 Aug 2025 12:30:55 -0400
>>>> +Subject: [PATCH 42/46] cephadm: fix some issues running existing cephadm build
>>>> + tests
>>>> +
>>>> +As time has marched on and people changed things our tests no longer
>>>> +match the expected inputs.
>>>> +
>>>> +Signed-off-by: John Mulligan <jmulligan@redhat.com>
>>>> +(cherry picked from commit 31c8010faa417ca53614bd30379a9b9c0c9199de)
>>>> +---
>>>> + src/cephadm/tests/build/test_cephadm_build.py | 18 ++++++++++++------
>>>> + 1 file changed, 12 insertions(+), 6 deletions(-)
>>>> +
>>>> +diff --git a/src/cephadm/tests/build/test_cephadm_build.py b/src/cephadm/tests/build/test_cephadm_build.py
>>>> +index c2995a76d4b..9beef67ffe0 100644
>>>> +--- a/src/cephadm/tests/build/test_cephadm_build.py
>>>> ++++ b/src/cephadm/tests/build/test_cephadm_build.py
>>>> +@@ -34,12 +34,12 @@ CONTAINERS = {
>>>> +     },
>>>> +     'ubuntu-20.04': {
>>>> +         'name': 'cephadm-build-test:ubuntu-20-04-py3',
>>>> +-        'base_image': 'quay.io/library/ubuntu:20.04',
>>>> ++        'base_image': 'docker.io/library/ubuntu:20.04',
>>>> +         'script': 'apt update && apt install -y python3-venv',
>>>> +     },
>>>> +     'ubuntu-22.04': {
>>>> +         'name': 'cephadm-build-test:ubuntu-22-04-py3',
>>>> +-        'base_image': 'quay.io/library/ubuntu:22.04',
>>>> ++        'base_image': 'docker.io/library/ubuntu:22.04',
>>>> +         'script': 'apt update && apt install -y python3-venv',
>>>> +     },
>>>> + }
>>>> +@@ -128,8 +128,8 @@ def test_cephadm_build(env, source_dir, tmp_path):
>>>> +     assert all('requirements_entry' in v for v in data['bundled_packages'])
>>>> +     assert 'zip_root_entries' in data
>>>> +     zre = data['zip_root_entries']
>>>> +-    assert any(e.startswith('Jinja2') for e in zre)
>>>> +-    assert any(e.startswith('MarkupSafe') for e in zre)
>>>> ++    assert any(_dist_info(e, 'Jinja2') for e in zre)
>>>> ++    assert any(_dist_info(e, 'MarkupSafe') for e in zre)
>>>> +     assert any(e.startswith('jinja2') for e in zre)
>>>> +     assert any(e.startswith('markupsafe') for e in zre)
>>>> +     assert any(e.startswith('cephadmlib') for e in zre)
>>>> +@@ -184,9 +184,15 @@ def test_cephadm_build_from_rpms(env, source_dir, tmp_path):
>>>> +     assert all('requirements_entry' in v for v in data['bundled_packages'])
>>>> +     assert 'zip_root_entries' in data
>>>> +     zre = data['zip_root_entries']
>>>> +-    assert any(e.startswith('Jinja2') for e in zre)
>>>> +-    assert any(e.startswith('MarkupSafe') for e in zre)
>>>> ++    assert any(_dist_info(e, 'Jinja2') for e in zre)
>>>> ++    assert any(_dist_info(e, 'MarkupSafe') for e in zre)
>>>> +     assert any(e.startswith('jinja2') for e in zre)
>>>> +     assert any(e.startswith('markupsafe') for e in zre)
>>>> +     assert any(e.startswith('cephadmlib') for e in zre)
>>>> +     assert any(e.startswith('_cephadmmeta') for e in zre)
>>>> ++
>>>> ++
>>>> ++def _dist_info(entry, name):
>>>> ++    return (
>>>> ++        entry.startswith(entry) or entry.startswith(entry.lower())
>>>> ++    ) and (entry.endswith('.dist-info') or entry.endswith('.egg-info'))
>>>> +-- 
>>>> +2.47.3
>>>> +
>>>> diff --git a/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch b/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
>>>> new file mode 100644
>>>> index 00000000000..7a0b1a18d68
>>>> --- /dev/null
>>>> +++ b/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
>>>> @@ -0,0 +1,62 @@
>>>> +From 2c20ae95f32a2bdf78576252f1c9171f00a2790e Mon Sep 17 00:00:00 2001
>>>> +From: Kefu Chai <k.chai@proxmox.com>
>>>> +Date: Mon, 10 Nov 2025 12:11:08 +0800
>>>> +Subject: [PATCH 43/46] cephadm: fix zip_root_entries population in version
>>>> + command
>>>> +
>>>> +The 'cephadm version --verbose' command was returning an empty
>>>> +zip_root_entries list because it relied on the private '_files'
>>>> +attribute of zipimport.zipimporter, which is not reliably populated
>>>> +across Python versions.
>>>> +
>>>> +This commit fixes the issue by using the zipfile module to properly
>>>> +read the archive contents via the loader.archive path. This ensures
>>>> +that zip_root_entries is correctly populated with the root-level
>>>> +directories in the zipapp.
>>>> +
>>>> +This fix is necessary for the cephadm build tests to properly validate
>>>> +that all expected packages and modules are included in the built zipapp.
>>>> +
>>>> +Signed-off-by: Kefu Chai <k.chai@proxmox.com>
>>>> +(cherry picked from commit 2c68c1496dbb7cd01bf783e31510940445040a34)
>>>> +---
>>>> + src/cephadm/cephadm.py | 16 ++++++++++++----
>>>> + 1 file changed, 12 insertions(+), 4 deletions(-)
>>>> +
>>>> +diff --git a/src/cephadm/cephadm.py b/src/cephadm/cephadm.py
>>>> +index f75aaa86dac..8d08c700868 100755
>>>> +--- a/src/cephadm/cephadm.py
>>>> ++++ b/src/cephadm/cephadm.py
>>>> +@@ -1688,6 +1688,7 @@ def command_version(ctx):
>>>> +     # type: (CephadmContext) -> int
>>>> +     import importlib
>>>> +     import zipimport
>>>> ++    import zipfile
>>>> +     import types
>>>> + 
>>>> +     vmod: Optional[types.ModuleType]
>>>> +@@ -1744,10 +1745,17 @@ def command_version(ctx):
>>>> +             out['bundled_packages'] = deps_info
>>>> +         except OSError:
>>>> +             pass
>>>> +-        files = getattr(loader, '_files', {})
>>>> +-        out['zip_root_entries'] = sorted(
>>>> +-            {p.split('/')[0] for p in files.keys()}
>>>> +-        )
>>>> ++        # Use zipfile module to properly read the archive contents
>>>> ++        # loader.archive contains the path to the zip file
>>>> ++        try:
>>>> ++            with zipfile.ZipFile(loader.archive, 'r') as zf:
>>>> ++                files = zf.namelist()
>>>> ++                out['zip_root_entries'] = sorted(
>>>> ++                    {p.split('/')[0] for p in files if p}
>>>> ++                )
>>>> ++        except (OSError, zipfile.BadZipFile):
>>>> ++            # Fallback to empty list if we can't read the zip
>>>> ++            out['zip_root_entries'] = []
>>>> + 
>>>> +     json.dump(out, sys.stdout, indent=2)
>>>> +     print()
>>>> +-- 
>>>> +2.47.3
>>>> +
>>>> diff --git a/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch b/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
>>>> new file mode 100644
>>>> index 00000000000..6e1b4dd2e0e
>>>> --- /dev/null
>>>> +++ b/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
>>>> @@ -0,0 +1,89 @@
>>>> +From 5ce72178a782dfe2e80648063b025ba23d96d5d8 Mon Sep 17 00:00:00 2001
>>>> +From: John Mulligan <jmulligan@redhat.com>
>>>> +Date: Thu, 21 Aug 2025 12:44:17 -0400
>>>> +Subject: [PATCH 44/46] cephadm: support cephadm rpm based builds without
>>>> + top_level.txt
>>>> +
>>>> +Signed-off-by: John Mulligan <jmulligan@redhat.com>
>>>> +(cherry picked from commit 26a499a8da339d870af193ea964368afbc84c694)
>>>> +---
>>>> + src/cephadm/build.py | 56 +++++++++++++++++++++++++++++++++-----------
>>>> + 1 file changed, 42 insertions(+), 14 deletions(-)
>>>> +
>>>> +diff --git a/src/cephadm/build.py b/src/cephadm/build.py
>>>> +index 43bc58a4003..01c91fc1680 100755
>>>> +--- a/src/cephadm/build.py
>>>> ++++ b/src/cephadm/build.py
>>>> +@@ -434,6 +434,47 @@ def _install_rpm_deps(tempdir, config):
>>>> +     return dinfo
>>>> + 
>>>> + 
>>>> ++def _gather_rpm_package_dirs(paths):
>>>> ++    # = The easy way =
>>>> ++    # the top_level.txt file can be used to determine where the python packages
>>>> ++    # actually are. We need all of those and the meta-data dir (parent of
>>>> ++    # top_level.txt) to be included in our zipapp
>>>> ++    top_level = None
>>>> ++    for path in paths:
>>>> ++        if path.endswith('top_level.txt'):
>>>> ++            top_level = pathlib.Path(path)
>>>> ++    if top_level:
>>>> ++        meta_dir = top_level.parent
>>>> ++        pkg_dirs = [
>>>> ++            top_level.parent.parent / p
>>>> ++            for p in top_level.read_text().splitlines()
>>>> ++        ]
>>>> ++        return meta_dir, pkg_dirs
>>>> ++    # = The hard way =
>>>> ++    # loop through the directories to find the .dist-info dir (containing the
>>>> ++    # mandatory METADATA file, according to the spec) and once we know the
>>>> ++    # location of dist info we find the sibling paths from the rpm listing
>>>> ++    dist_info = None
>>>> ++    ppaths = []
>>>> ++    for path in paths:
>>>> ++        ppath = pathlib.Path(path)
>>>> ++        ppaths.append(ppath)
>>>> ++        if ppath.name == 'METADATA' and ppath.parent.name.endswith('.dist-info'):
>>>> ++            dist_info = ppath.parent
>>>> ++            break
>>>> ++    if not dist_info:
>>>> ++        raise ValueError('no .dist-info METADATA found')
>>>> ++    if not dist_info.parent.name == 'site-packages':
>>>> ++        raise ValueError(
>>>> ++            'unexpected parent directory (not site-packages):'
>>>> ++            f' {dist_info.parent.name}'
>>>> ++        )
>>>> ++    siblings = [
>>>> ++        p for p in ppaths if p.parent == dist_info.parent and p != dist_info
>>>> ++    ]
>>>> ++    return dist_info, siblings
>>>> ++
>>>> ++
>>>> + def _deps_from_rpm(tempdir, config, dinfo, pkg):
>>>> +     # first, figure out what rpm provides a particular python lib
>>>> +     dist = f'python3.{sys.version_info.minor}dist({pkg})'.lower()
>>>> +@@ -469,20 +510,7 @@ def _deps_from_rpm(tempdir, config, dinfo, pkg):
>>>> +         ['rpm', '-ql', rpmname], check=True, stdout=subprocess.PIPE
>>>> +     )
>>>> +     paths = [l.decode('utf8') for l in res.stdout.splitlines()]
>>>> +-    # the top_level.txt file can be used to determine where the python packages
>>>> +-    # actually are. We need all of those and the meta-data dir (parent of
>>>> +-    # top_level.txt) to be included in our zipapp
>>>> +-    top_level = None
>>>> +-    for path in paths:
>>>> +-        if path.endswith('top_level.txt'):
>>>> +-            top_level = pathlib.Path(path)
>>>> +-    if not top_level:
>>>> +-        raise ValueError('top_level not found')
>>>> +-    meta_dir = top_level.parent
>>>> +-    pkg_dirs = [
>>>> +-        top_level.parent.parent / p
>>>> +-        for p in top_level.read_text().splitlines()
>>>> +-    ]
>>>> ++    meta_dir, pkg_dirs = _gather_rpm_package_dirs(paths)
>>>> +     meta_dest = tempdir / meta_dir.name
>>>> +     log.info(f"Copying {meta_dir} to {meta_dest}")
>>>> +     # copy the meta data directory
>>>> +-- 
>>>> +2.47.3
>>>> +
>>>> diff --git a/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch b/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
>>>> new file mode 100644
>>>> index 00000000000..94241490b64
>>>> --- /dev/null
>>>> +++ b/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
>>>> @@ -0,0 +1,261 @@
>>>> +From a4744feb1457aa7fb3f47b42f29138682cc7d41c Mon Sep 17 00:00:00 2001
>>>> +From: Kefu Chai <k.chai@proxmox.com>
>>>> +Date: Tue, 14 Oct 2025 21:04:42 +0800
>>>> +Subject: [PATCH 45/46] cephadm/build: Add Debian package support for bundled
>>>> + dependencies
>>>> +
>>>> +Extends the cephadm build script to support bundling dependencies from
>>>> +Debian packages in addition to pip and RPM packages. This allows building
>>>> +cephadm on Debian-based distributions using system packages.
>>>> +
>>>> +Key changes:
>>>> +- Add 'deb' to DependencyMode enum to enable Debian package mode
>>>> +- Implement _setup_deb() to configure Debian dependency requirements
>>>> +- Add _install_deb_deps() to orchestrate Debian package installation
>>>> +- Add _gather_deb_package_dirs() to parse Debian package file listings
>>>> +  and locate Python package directories (handles both site-packages and
>>>> +  dist-packages directories used by Debian)
>>>> +- Add _deps_from_deb() to extract Python dependencies from installed
>>>> +  Debian packages using dpkg/apt-cache tools
>>>> +- Fix variable reference bug in _install_deps() (deps.mode -> config.deps_mode)
>>>> +
>>>> +The Debian implementation follows a similar pattern to the existing RPM
>>>> +support, using dpkg-query and dpkg -L to locate installed packages and
>>>> +their files, with special handling for Debian naming conventions
>>>> +(e.g., PyYAML -> python3-yaml).
>>>> +
>>>> +Signed-off-by: Kefu Chai <k.chai@proxmox.com>
>>>> +(cherry picked from commit 3ff9b0c24e33debe95a0a0c6b42da30be788871c)
>>>> +---
>>>> + src/cephadm/build.py | 158 +++++++++++++++++++++++++++++++++++++++----
>>>> + 1 file changed, 143 insertions(+), 15 deletions(-)
>>>> +
>>>> +diff --git a/src/cephadm/build.py b/src/cephadm/build.py
>>>> +index 01c91fc1680..53d3983de89 100755
>>>> +--- a/src/cephadm/build.py
>>>> ++++ b/src/cephadm/build.py
>>>> +@@ -154,6 +154,7 @@ class PipEnv(enum.Enum):
>>>> + class DependencyMode(enum.Enum):
>>>> +     pip = enum.auto()
>>>> +     rpm = enum.auto()
>>>> ++    deb = enum.auto()
>>>> +     none = enum.auto()
>>>> + 
>>>> + 
>>>> +@@ -169,6 +170,8 @@ class Config:
>>>> +             self._setup_pip()
>>>> +         elif self.deps_mode == DependencyMode.rpm:
>>>> +             self._setup_rpm()
>>>> ++        elif self.deps_mode == DependencyMode.deb:
>>>> ++            self._setup_deb()
>>>> + 
>>>> +     def _setup_pip(self):
>>>> +         if self._maj_min == (3, 6):
>>>> +@@ -180,6 +183,9 @@ class Config:
>>>> +     def _setup_rpm(self):
>>>> +         self.requirements = [InstallSpec(**v) for v in PY_REQUIREMENTS]
>>>> + 
>>>> ++    def _setup_deb(self):
>>>> ++        self.requirements = [InstallSpec(**v) for v in PY_REQUIREMENTS]
>>>> ++
>>>> + 
>>>> + class DependencyInfo:
>>>> +     """Type for tracking bundled dependencies."""
>>>> +@@ -333,7 +339,9 @@ def _install_deps(tempdir, config):
>>>> +         return _install_pip_deps(tempdir, config)
>>>> +     if config.deps_mode == DependencyMode.rpm:
>>>> +         return _install_rpm_deps(tempdir, config)
>>>> +-    raise ValueError(f'unexpected deps mode: {deps.mode}')
>>>> ++    if config.deps_mode == DependencyMode.deb:
>>>> ++        return _install_deb_deps(tempdir, config)
>>>> ++    raise ValueError(f'unexpected deps mode: {config.deps_mode}')
>>>> + 
>>>> + 
>>>> + def _install_pip_deps(tempdir, config):
>>>> +@@ -434,7 +442,26 @@ def _install_rpm_deps(tempdir, config):
>>>> +     return dinfo
>>>> + 
>>>> + 
>>>> +-def _gather_rpm_package_dirs(paths):
>>>> ++def _install_deb_deps(tempdir, config):
>>>> ++    log.info("Installing dependencies using Debian packages")
>>>> ++    dinfo = DependencyInfo(config)
>>>> ++    for pkg in config.requirements:
>>>> ++        log.info(f"Looking for debian package for: {pkg.name!r}")
>>>> ++        _deps_from_deb(tempdir, config, dinfo, pkg.name)
>>>> ++    return dinfo
>>>> ++
>>>> ++
>>>> ++def _gather_package_dirs(paths, expected_parent_dir):
>>>> ++    """Parse package file listing to find Python package directories.
>>>> ++
>>>> ++    Args:
>>>> ++        paths: List of file paths from package listing
>>>> ++        expected_parent_dir: Expected parent directory name (e.g., 'site-packages' for RPM,
>>>> ++                           'dist-packages' for Debian)
>>>> ++
>>>> ++    Returns:
>>>> ++        Tuple of (metadata_dir, package_dirs)
>>>> ++    """
>>>> +     # = The easy way =
>>>> +     # the top_level.txt file can be used to determine where the python packages
>>>> +     # actually are. We need all of those and the meta-data dir (parent of
>>>> +@@ -453,7 +480,7 @@ def _gather_rpm_package_dirs(paths):
>>>> +     # = The hard way =
>>>> +     # loop through the directories to find the .dist-info dir (containing the
>>>> +     # mandatory METADATA file, according to the spec) and once we know the
>>>> +-    # location of dist info we find the sibling paths from the rpm listing
>>>> ++    # location of dist info we find the sibling paths from the package listing
>>>> +     dist_info = None
>>>> +     ppaths = []
>>>> +     for path in paths:
>>>> +@@ -464,9 +491,9 @@ def _gather_rpm_package_dirs(paths):
>>>> +             break
>>>> +     if not dist_info:
>>>> +         raise ValueError('no .dist-info METADATA found')
>>>> +-    if not dist_info.parent.name == 'site-packages':
>>>> ++    if dist_info.parent.name != expected_parent_dir:
>>>> +         raise ValueError(
>>>> +-            'unexpected parent directory (not site-packages):'
>>>> ++            f'unexpected parent directory (not {expected_parent_dir}):'
>>>> +             f' {dist_info.parent.name}'
>>>> +         )
>>>> +     siblings = [
>>>> +@@ -475,6 +502,31 @@ def _gather_rpm_package_dirs(paths):
>>>> +     return dist_info, siblings
>>>> + 
>>>> + 
>>>> ++def _copy_package_files(tempdir, paths, expected_parent_dir):
>>>> ++    """Copy package files to the build directory.
>>>> ++
>>>> ++    Args:
>>>> ++        tempdir: Temporary directory to copy files to
>>>> ++        paths: List of file paths from package listing
>>>> ++        expected_parent_dir: Expected parent directory name per packaging convention:
>>>> ++            - 'site-packages' for RPM-based distributions
>>>> ++            - 'dist-packages' for Debian-based distributions
>>>> ++
>>>> ++    Returns:
>>>> ++        None
>>>> ++    """
>>>> ++    meta_dir, pkg_dirs = _gather_package_dirs(paths, expected_parent_dir)
>>>> ++    meta_dest = tempdir / meta_dir.name
>>>> ++    log.info(f"Copying {meta_dir} to {meta_dest}")
>>>> ++    # copy the meta data directory
>>>> ++    shutil.copytree(meta_dir, meta_dest, ignore=_ignore_cephadmlib)
>>>> ++    # copy all the package directories
>>>> ++    for pkg_dir in pkg_dirs:
>>>> ++        pkg_dest = tempdir / pkg_dir.name
>>>> ++        log.info(f"Copying {pkg_dir} to {pkg_dest}")
>>>> ++        shutil.copytree(pkg_dir, pkg_dest, ignore=_ignore_cephadmlib)
>>>> ++
>>>> ++
>>>> + def _deps_from_rpm(tempdir, config, dinfo, pkg):
>>>> +     # first, figure out what rpm provides a particular python lib
>>>> +     dist = f'python3.{sys.version_info.minor}dist({pkg})'.lower()
>>>> +@@ -510,16 +562,92 @@ def _deps_from_rpm(tempdir, config, dinfo, pkg):
>>>> +         ['rpm', '-ql', rpmname], check=True, stdout=subprocess.PIPE
>>>> +     )
>>>> +     paths = [l.decode('utf8') for l in res.stdout.splitlines()]
>>>> +-    meta_dir, pkg_dirs = _gather_rpm_package_dirs(paths)
>>>> +-    meta_dest = tempdir / meta_dir.name
>>>> +-    log.info(f"Copying {meta_dir} to {meta_dest}")
>>>> +-    # copy the meta data directory
>>>> +-    shutil.copytree(meta_dir, meta_dest, ignore=_ignore_cephadmlib)
>>>> +-    # copy all the package directories
>>>> +-    for pkg_dir in pkg_dirs:
>>>> +-        pkg_dest = tempdir / pkg_dir.name
>>>> +-        log.info(f"Copying {pkg_dir} to {pkg_dest}")
>>>> +-        shutil.copytree(pkg_dir, pkg_dest, ignore=_ignore_cephadmlib)
>>>> ++    # RPM-based distributions use 'site-packages' for Python packages
>>>> ++    _copy_package_files(tempdir, paths, 'site-packages')
>>>> ++
>>>> ++
>>>> ++def _deps_from_deb(tempdir, config, dinfo, pkg):
>>>> ++    """Extract Python dependencies from Debian packages.
>>>> ++
>>>> ++    Args:
>>>> ++        tempdir: Temporary directory to copy package files to
>>>> ++        config: Build configuration
>>>> ++        dinfo: DependencyInfo instance to track dependencies
>>>> ++        pkg: Python package name (e.g., 'MarkupSafe', 'Jinja2', 'PyYAML')
>>>> ++    """
>>>> ++    # Convert Python package name to Debian package name
>>>> ++    # Python packages are typically named python3-<lowercase-name>
>>>> ++    # Handle special cases: PyYAML -> python3-yaml, MarkupSafe -> python3-markupsafe
>>>> ++    pkg_lower = pkg.lower()
>>>> ++    if pkg_lower == 'pyyaml':
>>>> ++        deb_pkg_name = 'python3-yaml'
>>>> ++    else:
>>>> ++        deb_pkg_name = f'python3-{pkg_lower}'
>>>> ++
>>>> ++    # First, try to find the package using apt-cache
>>>> ++    # This helps verify the package exists before trying to list its files
>>>> ++    try:
>>>> ++        res = subprocess.run(
>>>> ++            ['apt-cache', 'show', deb_pkg_name],
>>>> ++            check=True,
>>>> ++            stdout=subprocess.PIPE,
>>>> ++            stderr=subprocess.PIPE,
>>>> ++        )
>>>> ++    except subprocess.CalledProcessError:
>>>> ++        # Package not found, try alternative naming
>>>> ++        log.warning(f"Package {deb_pkg_name} not found via apt-cache, trying dpkg -S")
>>>> ++        # Try to search for files that might belong to this package
>>>> ++        # Search for the Python module in site-packages
>>>> ++        search_pattern = f'/usr/lib/python3*/dist-packages/{pkg.lower()}*'
>>>> ++        try:
>>>> ++            res = subprocess.run(
>>>> ++                ['dpkg', '-S', search_pattern],
>>>> ++                check=True,
>>>> ++                stdout=subprocess.PIPE,
>>>> ++                stderr=subprocess.PIPE,
>>>> ++            )
>>>> ++            # dpkg -S output format: "package: /path/to/file"
>>>> ++            deb_pkg_name = res.stdout.decode('utf8').split(':')[0].strip()
>>>> ++        except subprocess.CalledProcessError as err:
>>>> ++            log.error(f"Could not find Debian package for {pkg}")
>>>> ++            log.error(f"Tried: {deb_pkg_name} and pattern search")
>>>> ++            sys.exit(1)
>>>> ++
>>>> ++    # Get version information using dpkg-query
>>>> ++    try:
>>>> ++        res = subprocess.run(
>>>> ++            ['dpkg-query', '-W', '-f=${Version}\\n', deb_pkg_name],
>>>> ++            check=True,
>>>> ++            stdout=subprocess.PIPE,
>>>> ++        )
>>>> ++        version = res.stdout.decode('utf8').strip()
>>>> ++    except subprocess.CalledProcessError as err:
>>>> ++        log.error(f"Could not query version for package {deb_pkg_name}: {err}")
>>>> ++        sys.exit(1)
>>>> ++
>>>> ++    log.info(f"Debian Package: {deb_pkg_name} (version: {version})")
>>>> ++    dinfo.add(
>>>> ++        pkg,
>>>> ++        deb_name=deb_pkg_name,
>>>> ++        version=version,
>>>> ++        package_source='deb',
>>>> ++    )
>>>> ++
>>>> ++    # Get the list of files provided by the Debian package
>>>> ++    try:
>>>> ++        res = subprocess.run(
>>>> ++            ['dpkg', '-L', deb_pkg_name],
>>>> ++            check=True,
>>>> ++            stdout=subprocess.PIPE,
>>>> ++        )
>>>> ++    except subprocess.CalledProcessError as err:
>>>> ++        log.error(f"Could not list files for package {deb_pkg_name}: {err}")
>>>> ++        sys.exit(1)
>>>> ++
>>>> ++    paths = [l.decode('utf8') for l in res.stdout.splitlines()]
>>>> ++    # Debian-based distributions use 'dist-packages' for system-managed Python packages
>>>> ++    # per Debian Python Policy: https://www.debian.org/doc/packaging-manuals/python-policy/
>>>> ++    _copy_package_files(tempdir, paths, 'dist-packages')
>>>> + 
>>>> + 
>>>> + def generate_version_file(versioning_vars, dest):
>>>> +-- 
>>>> +2.47.3
>>>> +
>>>> diff --git a/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch b/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
>>>> new file mode 100644
>>>> index 00000000000..0b4daa63f79
>>>> --- /dev/null
>>>> +++ b/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
>>>> @@ -0,0 +1,54 @@
>>>> +From 779c9251669b65b7ed77a841f281e2d957174db2 Mon Sep 17 00:00:00 2001
>>>> +From: Kefu Chai <k.chai@proxmox.com>
>>>> +Date: Tue, 21 Oct 2025 11:25:00 +0800
>>>> +Subject: [PATCH 46/46] debian: Use system packages for cephadm bundled
>>>> + dependencies
>>>> +
>>>> +Configure the Debian build to use CEPHADM_BUNDLED_DEPENDENCIES=deb,
>>>> +which instructs the cephadm build script to bundle dependencies from
>>>> +system-installed Debian packages instead of downloading from PyPI.
>>>> +
>>>> +This change addresses build failures in restricted network environments
>>>> +where Debian build tools do not permit internet access. By leveraging
>>>> +the Debian package support added in commit 9378a2988e1, the build now
>>>> +uses python3-markupsafe, python3-jinja2, and python3-yaml packages
>>>> +that are already installed as build dependencies.
>>>> +
>>>> +This approach mirrors the existing RPM packaging workflow, ensuring
>>>> +consistent behavior across different distribution package formats.
>>>> +
>>>> +Signed-off-by: Kefu Chai <k.chai@proxmox.com>
>>>> +(cherry picked from commit 25680021ee2422f8b8b4075b3ab77af39126eecc)
>>>
>>> long term, it might be even better to not bundle at all, but add those
>>> python deps as runtime dependencies?
>> 
>> I concur with your concerns, as bundling violates Debian's phylosophy of 
>> avoiding "vendored" dependencies. The distro intends to control and
>> track all packages. And bunlding also implies more disk space usage and 
>> duplications.
>> 
>> But it's also an upstream design choice: cephadm was explicitly designed
>> to be standalone. Part of cephadm'svalue is that you can just curl |
>> python3 it onto a fresh system. If it requires installing 10+ Python
>> packages first, you've lost that simplicity.
>> 
>> What do you think?
>
> I understand that reasoning, and cephadm isn't something we are using
> anyway (atm)..
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel