Re: [pve-devel] [PATCH ceph 1/4 v5] cherry-pick changes to use system packages for bundled deps

["Fabian Grünbichler" <f.gruenbichler@proxmox.com>]

On January 24, 2026 5:13 am, Kefu Chai wrote:
> On Fri Jan 23, 2026 at 9:03 PM CST, Fabian Grünbichler wrote:
>> one small question inline, otherwise this one LGTM
>>
>> On January 23, 2026 8:56 am, Kefu Chai wrote:
>>> Previously, cephadm's build process pulled dependencies from PyPI,
>>> which failed in environments with restricted network access. This
>>> change backports upstream improvements that enable building cephadm
>>> using system packages instead, mirroring the existing RPM packaging
>>> workflow.
>>> 
>>> And the vendored Arrow pulled xsimd from GitHub, this also failed in
>>> building environment without network access. This change backports
>>> upstream improvements enabling the build to use libxsimd-dev package
>>> when building debian packages.
>>> 
>>> Backported changes:
>>> - https://github.com/ceph/ceph/pull/66256 - Use system packages for
>>>   cephadm bundled dependencies
>>> - https://github.com/ceph/ceph/pull/66248 - Add libxsimd-dev build
>>>   dependency for vendored Arrow
>>> - https://github.com/ceph/ceph/pull/65292 - cephadm: fix building
>>>   rpm-sourced cephadm zippapp on el10
>>> 
>>> Note: Test commits from these PRs are excluded from this backport.
>>> 
>>> Signed-off-by: Kefu Chai <k.chai@proxmox.com>
>>> ---
>>>  ...-issues-running-existing-cephadm-bui.patch |  66 +++++
>>>  ...root_entries-population-in-version-c.patch |  62 +++++
>>>  ...cephadm-rpm-based-builds-without-top.patch |  89 ++++++
>>>  ...d-Debian-package-support-for-bundled.patch | 261 ++++++++++++++++++
>>>  ...m-packages-for-cephadm-bundled-depen.patch |  54 ++++
>>>  ...-Use-AUTO-mode-for-xsimd-dependency-.patch |  55 ++++
>>>  ...dd-libxsimd-dev-build-dependency-for.patch |  41 +++
>>>  patches/series                                |   8 +
>>>  8 files changed, 636 insertions(+)
>>>  create mode 100644 patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
>>>  create mode 100644 patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
>>>  create mode 100644 patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
>>>  create mode 100644 patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
>>>  create mode 100644 patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
>>>  create mode 100644 patches/0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch
>>>  create mode 100644 patches/0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch
>>> 
>>> diff --git a/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch b/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
>>> new file mode 100644
>>> index 00000000000..c6208b0db2a
>>> --- /dev/null
>>> +++ b/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
>>> @@ -0,0 +1,66 @@
>>> +From 2685f6c35245d033e73e16ece3e356fcead44cea Mon Sep 17 00:00:00 2001
>>> +From: John Mulligan <jmulligan@redhat.com>
>>> +Date: Thu, 21 Aug 2025 12:30:55 -0400
>>> +Subject: [PATCH 42/46] cephadm: fix some issues running existing cephadm build
>>> + tests
>>> +
>>> +As time has marched on and people changed things our tests no longer
>>> +match the expected inputs.
>>> +
>>> +Signed-off-by: John Mulligan <jmulligan@redhat.com>
>>> +(cherry picked from commit 31c8010faa417ca53614bd30379a9b9c0c9199de)
>>> +---
>>> + src/cephadm/tests/build/test_cephadm_build.py | 18 ++++++++++++------
>>> + 1 file changed, 12 insertions(+), 6 deletions(-)
>>> +
>>> +diff --git a/src/cephadm/tests/build/test_cephadm_build.py b/src/cephadm/tests/build/test_cephadm_build.py
>>> +index c2995a76d4b..9beef67ffe0 100644
>>> +--- a/src/cephadm/tests/build/test_cephadm_build.py
>>> ++++ b/src/cephadm/tests/build/test_cephadm_build.py
>>> +@@ -34,12 +34,12 @@ CONTAINERS = {
>>> +     },
>>> +     'ubuntu-20.04': {
>>> +         'name': 'cephadm-build-test:ubuntu-20-04-py3',
>>> +-        'base_image': 'quay.io/library/ubuntu:20.04',
>>> ++        'base_image': 'docker.io/library/ubuntu:20.04',
>>> +         'script': 'apt update && apt install -y python3-venv',
>>> +     },
>>> +     'ubuntu-22.04': {
>>> +         'name': 'cephadm-build-test:ubuntu-22-04-py3',
>>> +-        'base_image': 'quay.io/library/ubuntu:22.04',
>>> ++        'base_image': 'docker.io/library/ubuntu:22.04',
>>> +         'script': 'apt update && apt install -y python3-venv',
>>> +     },
>>> + }
>>> +@@ -128,8 +128,8 @@ def test_cephadm_build(env, source_dir, tmp_path):
>>> +     assert all('requirements_entry' in v for v in data['bundled_packages'])
>>> +     assert 'zip_root_entries' in data
>>> +     zre = data['zip_root_entries']
>>> +-    assert any(e.startswith('Jinja2') for e in zre)
>>> +-    assert any(e.startswith('MarkupSafe') for e in zre)
>>> ++    assert any(_dist_info(e, 'Jinja2') for e in zre)
>>> ++    assert any(_dist_info(e, 'MarkupSafe') for e in zre)
>>> +     assert any(e.startswith('jinja2') for e in zre)
>>> +     assert any(e.startswith('markupsafe') for e in zre)
>>> +     assert any(e.startswith('cephadmlib') for e in zre)
>>> +@@ -184,9 +184,15 @@ def test_cephadm_build_from_rpms(env, source_dir, tmp_path):
>>> +     assert all('requirements_entry' in v for v in data['bundled_packages'])
>>> +     assert 'zip_root_entries' in data
>>> +     zre = data['zip_root_entries']
>>> +-    assert any(e.startswith('Jinja2') for e in zre)
>>> +-    assert any(e.startswith('MarkupSafe') for e in zre)
>>> ++    assert any(_dist_info(e, 'Jinja2') for e in zre)
>>> ++    assert any(_dist_info(e, 'MarkupSafe') for e in zre)
>>> +     assert any(e.startswith('jinja2') for e in zre)
>>> +     assert any(e.startswith('markupsafe') for e in zre)
>>> +     assert any(e.startswith('cephadmlib') for e in zre)
>>> +     assert any(e.startswith('_cephadmmeta') for e in zre)
>>> ++
>>> ++
>>> ++def _dist_info(entry, name):
>>> ++    return (
>>> ++        entry.startswith(entry) or entry.startswith(entry.lower())
>>> ++    ) and (entry.endswith('.dist-info') or entry.endswith('.egg-info'))
>>> +-- 
>>> +2.47.3
>>> +
>>> diff --git a/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch b/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
>>> new file mode 100644
>>> index 00000000000..7a0b1a18d68
>>> --- /dev/null
>>> +++ b/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
>>> @@ -0,0 +1,62 @@
>>> +From 2c20ae95f32a2bdf78576252f1c9171f00a2790e Mon Sep 17 00:00:00 2001
>>> +From: Kefu Chai <k.chai@proxmox.com>
>>> +Date: Mon, 10 Nov 2025 12:11:08 +0800
>>> +Subject: [PATCH 43/46] cephadm: fix zip_root_entries population in version
>>> + command
>>> +
>>> +The 'cephadm version --verbose' command was returning an empty
>>> +zip_root_entries list because it relied on the private '_files'
>>> +attribute of zipimport.zipimporter, which is not reliably populated
>>> +across Python versions.
>>> +
>>> +This commit fixes the issue by using the zipfile module to properly
>>> +read the archive contents via the loader.archive path. This ensures
>>> +that zip_root_entries is correctly populated with the root-level
>>> +directories in the zipapp.
>>> +
>>> +This fix is necessary for the cephadm build tests to properly validate
>>> +that all expected packages and modules are included in the built zipapp.
>>> +
>>> +Signed-off-by: Kefu Chai <k.chai@proxmox.com>
>>> +(cherry picked from commit 2c68c1496dbb7cd01bf783e31510940445040a34)
>>> +---
>>> + src/cephadm/cephadm.py | 16 ++++++++++++----
>>> + 1 file changed, 12 insertions(+), 4 deletions(-)
>>> +
>>> +diff --git a/src/cephadm/cephadm.py b/src/cephadm/cephadm.py
>>> +index f75aaa86dac..8d08c700868 100755
>>> +--- a/src/cephadm/cephadm.py
>>> ++++ b/src/cephadm/cephadm.py
>>> +@@ -1688,6 +1688,7 @@ def command_version(ctx):
>>> +     # type: (CephadmContext) -> int
>>> +     import importlib
>>> +     import zipimport
>>> ++    import zipfile
>>> +     import types
>>> + 
>>> +     vmod: Optional[types.ModuleType]
>>> +@@ -1744,10 +1745,17 @@ def command_version(ctx):
>>> +             out['bundled_packages'] = deps_info
>>> +         except OSError:
>>> +             pass
>>> +-        files = getattr(loader, '_files', {})
>>> +-        out['zip_root_entries'] = sorted(
>>> +-            {p.split('/')[0] for p in files.keys()}
>>> +-        )
>>> ++        # Use zipfile module to properly read the archive contents
>>> ++        # loader.archive contains the path to the zip file
>>> ++        try:
>>> ++            with zipfile.ZipFile(loader.archive, 'r') as zf:
>>> ++                files = zf.namelist()
>>> ++                out['zip_root_entries'] = sorted(
>>> ++                    {p.split('/')[0] for p in files if p}
>>> ++                )
>>> ++        except (OSError, zipfile.BadZipFile):
>>> ++            # Fallback to empty list if we can't read the zip
>>> ++            out['zip_root_entries'] = []
>>> + 
>>> +     json.dump(out, sys.stdout, indent=2)
>>> +     print()
>>> +-- 
>>> +2.47.3
>>> +
>>> diff --git a/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch b/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
>>> new file mode 100644
>>> index 00000000000..6e1b4dd2e0e
>>> --- /dev/null
>>> +++ b/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
>>> @@ -0,0 +1,89 @@
>>> +From 5ce72178a782dfe2e80648063b025ba23d96d5d8 Mon Sep 17 00:00:00 2001
>>> +From: John Mulligan <jmulligan@redhat.com>
>>> +Date: Thu, 21 Aug 2025 12:44:17 -0400
>>> +Subject: [PATCH 44/46] cephadm: support cephadm rpm based builds without
>>> + top_level.txt
>>> +
>>> +Signed-off-by: John Mulligan <jmulligan@redhat.com>
>>> +(cherry picked from commit 26a499a8da339d870af193ea964368afbc84c694)
>>> +---
>>> + src/cephadm/build.py | 56 +++++++++++++++++++++++++++++++++-----------
>>> + 1 file changed, 42 insertions(+), 14 deletions(-)
>>> +
>>> +diff --git a/src/cephadm/build.py b/src/cephadm/build.py
>>> +index 43bc58a4003..01c91fc1680 100755
>>> +--- a/src/cephadm/build.py
>>> ++++ b/src/cephadm/build.py
>>> +@@ -434,6 +434,47 @@ def _install_rpm_deps(tempdir, config):
>>> +     return dinfo
>>> + 
>>> + 
>>> ++def _gather_rpm_package_dirs(paths):
>>> ++    # = The easy way =
>>> ++    # the top_level.txt file can be used to determine where the python packages
>>> ++    # actually are. We need all of those and the meta-data dir (parent of
>>> ++    # top_level.txt) to be included in our zipapp
>>> ++    top_level = None
>>> ++    for path in paths:
>>> ++        if path.endswith('top_level.txt'):
>>> ++            top_level = pathlib.Path(path)
>>> ++    if top_level:
>>> ++        meta_dir = top_level.parent
>>> ++        pkg_dirs = [
>>> ++            top_level.parent.parent / p
>>> ++            for p in top_level.read_text().splitlines()
>>> ++        ]
>>> ++        return meta_dir, pkg_dirs
>>> ++    # = The hard way =
>>> ++    # loop through the directories to find the .dist-info dir (containing the
>>> ++    # mandatory METADATA file, according to the spec) and once we know the
>>> ++    # location of dist info we find the sibling paths from the rpm listing
>>> ++    dist_info = None
>>> ++    ppaths = []
>>> ++    for path in paths:
>>> ++        ppath = pathlib.Path(path)
>>> ++        ppaths.append(ppath)
>>> ++        if ppath.name == 'METADATA' and ppath.parent.name.endswith('.dist-info'):
>>> ++            dist_info = ppath.parent
>>> ++            break
>>> ++    if not dist_info:
>>> ++        raise ValueError('no .dist-info METADATA found')
>>> ++    if not dist_info.parent.name == 'site-packages':
>>> ++        raise ValueError(
>>> ++            'unexpected parent directory (not site-packages):'
>>> ++            f' {dist_info.parent.name}'
>>> ++        )
>>> ++    siblings = [
>>> ++        p for p in ppaths if p.parent == dist_info.parent and p != dist_info
>>> ++    ]
>>> ++    return dist_info, siblings
>>> ++
>>> ++
>>> + def _deps_from_rpm(tempdir, config, dinfo, pkg):
>>> +     # first, figure out what rpm provides a particular python lib
>>> +     dist = f'python3.{sys.version_info.minor}dist({pkg})'.lower()
>>> +@@ -469,20 +510,7 @@ def _deps_from_rpm(tempdir, config, dinfo, pkg):
>>> +         ['rpm', '-ql', rpmname], check=True, stdout=subprocess.PIPE
>>> +     )
>>> +     paths = [l.decode('utf8') for l in res.stdout.splitlines()]
>>> +-    # the top_level.txt file can be used to determine where the python packages
>>> +-    # actually are. We need all of those and the meta-data dir (parent of
>>> +-    # top_level.txt) to be included in our zipapp
>>> +-    top_level = None
>>> +-    for path in paths:
>>> +-        if path.endswith('top_level.txt'):
>>> +-            top_level = pathlib.Path(path)
>>> +-    if not top_level:
>>> +-        raise ValueError('top_level not found')
>>> +-    meta_dir = top_level.parent
>>> +-    pkg_dirs = [
>>> +-        top_level.parent.parent / p
>>> +-        for p in top_level.read_text().splitlines()
>>> +-    ]
>>> ++    meta_dir, pkg_dirs = _gather_rpm_package_dirs(paths)
>>> +     meta_dest = tempdir / meta_dir.name
>>> +     log.info(f"Copying {meta_dir} to {meta_dest}")
>>> +     # copy the meta data directory
>>> +-- 
>>> +2.47.3
>>> +
>>> diff --git a/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch b/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
>>> new file mode 100644
>>> index 00000000000..94241490b64
>>> --- /dev/null
>>> +++ b/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
>>> @@ -0,0 +1,261 @@
>>> +From a4744feb1457aa7fb3f47b42f29138682cc7d41c Mon Sep 17 00:00:00 2001
>>> +From: Kefu Chai <k.chai@proxmox.com>
>>> +Date: Tue, 14 Oct 2025 21:04:42 +0800
>>> +Subject: [PATCH 45/46] cephadm/build: Add Debian package support for bundled
>>> + dependencies
>>> +
>>> +Extends the cephadm build script to support bundling dependencies from
>>> +Debian packages in addition to pip and RPM packages. This allows building
>>> +cephadm on Debian-based distributions using system packages.
>>> +
>>> +Key changes:
>>> +- Add 'deb' to DependencyMode enum to enable Debian package mode
>>> +- Implement _setup_deb() to configure Debian dependency requirements
>>> +- Add _install_deb_deps() to orchestrate Debian package installation
>>> +- Add _gather_deb_package_dirs() to parse Debian package file listings
>>> +  and locate Python package directories (handles both site-packages and
>>> +  dist-packages directories used by Debian)
>>> +- Add _deps_from_deb() to extract Python dependencies from installed
>>> +  Debian packages using dpkg/apt-cache tools
>>> +- Fix variable reference bug in _install_deps() (deps.mode -> config.deps_mode)
>>> +
>>> +The Debian implementation follows a similar pattern to the existing RPM
>>> +support, using dpkg-query and dpkg -L to locate installed packages and
>>> +their files, with special handling for Debian naming conventions
>>> +(e.g., PyYAML -> python3-yaml).
>>> +
>>> +Signed-off-by: Kefu Chai <k.chai@proxmox.com>
>>> +(cherry picked from commit 3ff9b0c24e33debe95a0a0c6b42da30be788871c)
>>> +---
>>> + src/cephadm/build.py | 158 +++++++++++++++++++++++++++++++++++++++----
>>> + 1 file changed, 143 insertions(+), 15 deletions(-)
>>> +
>>> +diff --git a/src/cephadm/build.py b/src/cephadm/build.py
>>> +index 01c91fc1680..53d3983de89 100755
>>> +--- a/src/cephadm/build.py
>>> ++++ b/src/cephadm/build.py
>>> +@@ -154,6 +154,7 @@ class PipEnv(enum.Enum):
>>> + class DependencyMode(enum.Enum):
>>> +     pip = enum.auto()
>>> +     rpm = enum.auto()
>>> ++    deb = enum.auto()
>>> +     none = enum.auto()
>>> + 
>>> + 
>>> +@@ -169,6 +170,8 @@ class Config:
>>> +             self._setup_pip()
>>> +         elif self.deps_mode == DependencyMode.rpm:
>>> +             self._setup_rpm()
>>> ++        elif self.deps_mode == DependencyMode.deb:
>>> ++            self._setup_deb()
>>> + 
>>> +     def _setup_pip(self):
>>> +         if self._maj_min == (3, 6):
>>> +@@ -180,6 +183,9 @@ class Config:
>>> +     def _setup_rpm(self):
>>> +         self.requirements = [InstallSpec(**v) for v in PY_REQUIREMENTS]
>>> + 
>>> ++    def _setup_deb(self):
>>> ++        self.requirements = [InstallSpec(**v) for v in PY_REQUIREMENTS]
>>> ++
>>> + 
>>> + class DependencyInfo:
>>> +     """Type for tracking bundled dependencies."""
>>> +@@ -333,7 +339,9 @@ def _install_deps(tempdir, config):
>>> +         return _install_pip_deps(tempdir, config)
>>> +     if config.deps_mode == DependencyMode.rpm:
>>> +         return _install_rpm_deps(tempdir, config)
>>> +-    raise ValueError(f'unexpected deps mode: {deps.mode}')
>>> ++    if config.deps_mode == DependencyMode.deb:
>>> ++        return _install_deb_deps(tempdir, config)
>>> ++    raise ValueError(f'unexpected deps mode: {config.deps_mode}')
>>> + 
>>> + 
>>> + def _install_pip_deps(tempdir, config):
>>> +@@ -434,7 +442,26 @@ def _install_rpm_deps(tempdir, config):
>>> +     return dinfo
>>> + 
>>> + 
>>> +-def _gather_rpm_package_dirs(paths):
>>> ++def _install_deb_deps(tempdir, config):
>>> ++    log.info("Installing dependencies using Debian packages")
>>> ++    dinfo = DependencyInfo(config)
>>> ++    for pkg in config.requirements:
>>> ++        log.info(f"Looking for debian package for: {pkg.name!r}")
>>> ++        _deps_from_deb(tempdir, config, dinfo, pkg.name)
>>> ++    return dinfo
>>> ++
>>> ++
>>> ++def _gather_package_dirs(paths, expected_parent_dir):
>>> ++    """Parse package file listing to find Python package directories.
>>> ++
>>> ++    Args:
>>> ++        paths: List of file paths from package listing
>>> ++        expected_parent_dir: Expected parent directory name (e.g., 'site-packages' for RPM,
>>> ++                           'dist-packages' for Debian)
>>> ++
>>> ++    Returns:
>>> ++        Tuple of (metadata_dir, package_dirs)
>>> ++    """
>>> +     # = The easy way =
>>> +     # the top_level.txt file can be used to determine where the python packages
>>> +     # actually are. We need all of those and the meta-data dir (parent of
>>> +@@ -453,7 +480,7 @@ def _gather_rpm_package_dirs(paths):
>>> +     # = The hard way =
>>> +     # loop through the directories to find the .dist-info dir (containing the
>>> +     # mandatory METADATA file, according to the spec) and once we know the
>>> +-    # location of dist info we find the sibling paths from the rpm listing
>>> ++    # location of dist info we find the sibling paths from the package listing
>>> +     dist_info = None
>>> +     ppaths = []
>>> +     for path in paths:
>>> +@@ -464,9 +491,9 @@ def _gather_rpm_package_dirs(paths):
>>> +             break
>>> +     if not dist_info:
>>> +         raise ValueError('no .dist-info METADATA found')
>>> +-    if not dist_info.parent.name == 'site-packages':
>>> ++    if dist_info.parent.name != expected_parent_dir:
>>> +         raise ValueError(
>>> +-            'unexpected parent directory (not site-packages):'
>>> ++            f'unexpected parent directory (not {expected_parent_dir}):'
>>> +             f' {dist_info.parent.name}'
>>> +         )
>>> +     siblings = [
>>> +@@ -475,6 +502,31 @@ def _gather_rpm_package_dirs(paths):
>>> +     return dist_info, siblings
>>> + 
>>> + 
>>> ++def _copy_package_files(tempdir, paths, expected_parent_dir):
>>> ++    """Copy package files to the build directory.
>>> ++
>>> ++    Args:
>>> ++        tempdir: Temporary directory to copy files to
>>> ++        paths: List of file paths from package listing
>>> ++        expected_parent_dir: Expected parent directory name per packaging convention:
>>> ++            - 'site-packages' for RPM-based distributions
>>> ++            - 'dist-packages' for Debian-based distributions
>>> ++
>>> ++    Returns:
>>> ++        None
>>> ++    """
>>> ++    meta_dir, pkg_dirs = _gather_package_dirs(paths, expected_parent_dir)
>>> ++    meta_dest = tempdir / meta_dir.name
>>> ++    log.info(f"Copying {meta_dir} to {meta_dest}")
>>> ++    # copy the meta data directory
>>> ++    shutil.copytree(meta_dir, meta_dest, ignore=_ignore_cephadmlib)
>>> ++    # copy all the package directories
>>> ++    for pkg_dir in pkg_dirs:
>>> ++        pkg_dest = tempdir / pkg_dir.name
>>> ++        log.info(f"Copying {pkg_dir} to {pkg_dest}")
>>> ++        shutil.copytree(pkg_dir, pkg_dest, ignore=_ignore_cephadmlib)
>>> ++
>>> ++
>>> + def _deps_from_rpm(tempdir, config, dinfo, pkg):
>>> +     # first, figure out what rpm provides a particular python lib
>>> +     dist = f'python3.{sys.version_info.minor}dist({pkg})'.lower()
>>> +@@ -510,16 +562,92 @@ def _deps_from_rpm(tempdir, config, dinfo, pkg):
>>> +         ['rpm', '-ql', rpmname], check=True, stdout=subprocess.PIPE
>>> +     )
>>> +     paths = [l.decode('utf8') for l in res.stdout.splitlines()]
>>> +-    meta_dir, pkg_dirs = _gather_rpm_package_dirs(paths)
>>> +-    meta_dest = tempdir / meta_dir.name
>>> +-    log.info(f"Copying {meta_dir} to {meta_dest}")
>>> +-    # copy the meta data directory
>>> +-    shutil.copytree(meta_dir, meta_dest, ignore=_ignore_cephadmlib)
>>> +-    # copy all the package directories
>>> +-    for pkg_dir in pkg_dirs:
>>> +-        pkg_dest = tempdir / pkg_dir.name
>>> +-        log.info(f"Copying {pkg_dir} to {pkg_dest}")
>>> +-        shutil.copytree(pkg_dir, pkg_dest, ignore=_ignore_cephadmlib)
>>> ++    # RPM-based distributions use 'site-packages' for Python packages
>>> ++    _copy_package_files(tempdir, paths, 'site-packages')
>>> ++
>>> ++
>>> ++def _deps_from_deb(tempdir, config, dinfo, pkg):
>>> ++    """Extract Python dependencies from Debian packages.
>>> ++
>>> ++    Args:
>>> ++        tempdir: Temporary directory to copy package files to
>>> ++        config: Build configuration
>>> ++        dinfo: DependencyInfo instance to track dependencies
>>> ++        pkg: Python package name (e.g., 'MarkupSafe', 'Jinja2', 'PyYAML')
>>> ++    """
>>> ++    # Convert Python package name to Debian package name
>>> ++    # Python packages are typically named python3-<lowercase-name>
>>> ++    # Handle special cases: PyYAML -> python3-yaml, MarkupSafe -> python3-markupsafe
>>> ++    pkg_lower = pkg.lower()
>>> ++    if pkg_lower == 'pyyaml':
>>> ++        deb_pkg_name = 'python3-yaml'
>>> ++    else:
>>> ++        deb_pkg_name = f'python3-{pkg_lower}'
>>> ++
>>> ++    # First, try to find the package using apt-cache
>>> ++    # This helps verify the package exists before trying to list its files
>>> ++    try:
>>> ++        res = subprocess.run(
>>> ++            ['apt-cache', 'show', deb_pkg_name],
>>> ++            check=True,
>>> ++            stdout=subprocess.PIPE,
>>> ++            stderr=subprocess.PIPE,
>>> ++        )
>>> ++    except subprocess.CalledProcessError:
>>> ++        # Package not found, try alternative naming
>>> ++        log.warning(f"Package {deb_pkg_name} not found via apt-cache, trying dpkg -S")
>>> ++        # Try to search for files that might belong to this package
>>> ++        # Search for the Python module in site-packages
>>> ++        search_pattern = f'/usr/lib/python3*/dist-packages/{pkg.lower()}*'
>>> ++        try:
>>> ++            res = subprocess.run(
>>> ++                ['dpkg', '-S', search_pattern],
>>> ++                check=True,
>>> ++                stdout=subprocess.PIPE,
>>> ++                stderr=subprocess.PIPE,
>>> ++            )
>>> ++            # dpkg -S output format: "package: /path/to/file"
>>> ++            deb_pkg_name = res.stdout.decode('utf8').split(':')[0].strip()
>>> ++        except subprocess.CalledProcessError as err:
>>> ++            log.error(f"Could not find Debian package for {pkg}")
>>> ++            log.error(f"Tried: {deb_pkg_name} and pattern search")
>>> ++            sys.exit(1)
>>> ++
>>> ++    # Get version information using dpkg-query
>>> ++    try:
>>> ++        res = subprocess.run(
>>> ++            ['dpkg-query', '-W', '-f=${Version}\\n', deb_pkg_name],
>>> ++            check=True,
>>> ++            stdout=subprocess.PIPE,
>>> ++        )
>>> ++        version = res.stdout.decode('utf8').strip()
>>> ++    except subprocess.CalledProcessError as err:
>>> ++        log.error(f"Could not query version for package {deb_pkg_name}: {err}")
>>> ++        sys.exit(1)
>>> ++
>>> ++    log.info(f"Debian Package: {deb_pkg_name} (version: {version})")
>>> ++    dinfo.add(
>>> ++        pkg,
>>> ++        deb_name=deb_pkg_name,
>>> ++        version=version,
>>> ++        package_source='deb',
>>> ++    )
>>> ++
>>> ++    # Get the list of files provided by the Debian package
>>> ++    try:
>>> ++        res = subprocess.run(
>>> ++            ['dpkg', '-L', deb_pkg_name],
>>> ++            check=True,
>>> ++            stdout=subprocess.PIPE,
>>> ++        )
>>> ++    except subprocess.CalledProcessError as err:
>>> ++        log.error(f"Could not list files for package {deb_pkg_name}: {err}")
>>> ++        sys.exit(1)
>>> ++
>>> ++    paths = [l.decode('utf8') for l in res.stdout.splitlines()]
>>> ++    # Debian-based distributions use 'dist-packages' for system-managed Python packages
>>> ++    # per Debian Python Policy: https://www.debian.org/doc/packaging-manuals/python-policy/
>>> ++    _copy_package_files(tempdir, paths, 'dist-packages')
>>> + 
>>> + 
>>> + def generate_version_file(versioning_vars, dest):
>>> +-- 
>>> +2.47.3
>>> +
>>> diff --git a/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch b/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
>>> new file mode 100644
>>> index 00000000000..0b4daa63f79
>>> --- /dev/null
>>> +++ b/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
>>> @@ -0,0 +1,54 @@
>>> +From 779c9251669b65b7ed77a841f281e2d957174db2 Mon Sep 17 00:00:00 2001
>>> +From: Kefu Chai <k.chai@proxmox.com>
>>> +Date: Tue, 21 Oct 2025 11:25:00 +0800
>>> +Subject: [PATCH 46/46] debian: Use system packages for cephadm bundled
>>> + dependencies
>>> +
>>> +Configure the Debian build to use CEPHADM_BUNDLED_DEPENDENCIES=deb,
>>> +which instructs the cephadm build script to bundle dependencies from
>>> +system-installed Debian packages instead of downloading from PyPI.
>>> +
>>> +This change addresses build failures in restricted network environments
>>> +where Debian build tools do not permit internet access. By leveraging
>>> +the Debian package support added in commit 9378a2988e1, the build now
>>> +uses python3-markupsafe, python3-jinja2, and python3-yaml packages
>>> +that are already installed as build dependencies.
>>> +
>>> +This approach mirrors the existing RPM packaging workflow, ensuring
>>> +consistent behavior across different distribution package formats.
>>> +
>>> +Signed-off-by: Kefu Chai <k.chai@proxmox.com>
>>> +(cherry picked from commit 25680021ee2422f8b8b4075b3ab77af39126eecc)
>>
>> long term, it might be even better to not bundle at all, but add those
>> python deps as runtime dependencies?
> 
> I concur with your concerns, as bundling violates Debian's phylosophy of 
> avoiding "vendored" dependencies. The distro intends to control and
> track all packages. And bunlding also implies more disk space usage and 
> duplications.
> 
> But it's also an upstream design choice: cephadm was explicitly designed
> to be standalone. Part of cephadm'svalue is that you can just curl |
> python3 it onto a fresh system. If it requires installing 10+ Python
> packages first, you've lost that simplicity.
> 
> What do you think?

I understand that reasoning, and cephadm isn't something we are using
anyway (atm)..


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel