Re: [pve-devel] [PATCH ceph 1/4 v5] cherry-pick changes to use system packages for bundled deps

["Fabian Grünbichler" <f.gruenbichler@proxmox.com>]

one small question inline, otherwise this one LGTM

On January 23, 2026 8:56 am, Kefu Chai wrote:
> Previously, cephadm's build process pulled dependencies from PyPI,
> which failed in environments with restricted network access. This
> change backports upstream improvements that enable building cephadm
> using system packages instead, mirroring the existing RPM packaging
> workflow.
> 
> And the vendored Arrow pulled xsimd from GitHub, this also failed in
> building environment without network access. This change backports
> upstream improvements enabling the build to use libxsimd-dev package
> when building debian packages.
> 
> Backported changes:
> - https://github.com/ceph/ceph/pull/66256 - Use system packages for
>   cephadm bundled dependencies
> - https://github.com/ceph/ceph/pull/66248 - Add libxsimd-dev build
>   dependency for vendored Arrow
> - https://github.com/ceph/ceph/pull/65292 - cephadm: fix building
>   rpm-sourced cephadm zippapp on el10
> 
> Note: Test commits from these PRs are excluded from this backport.
> 
> Signed-off-by: Kefu Chai <k.chai@proxmox.com>
> ---
>  ...-issues-running-existing-cephadm-bui.patch |  66 +++++
>  ...root_entries-population-in-version-c.patch |  62 +++++
>  ...cephadm-rpm-based-builds-without-top.patch |  89 ++++++
>  ...d-Debian-package-support-for-bundled.patch | 261 ++++++++++++++++++
>  ...m-packages-for-cephadm-bundled-depen.patch |  54 ++++
>  ...-Use-AUTO-mode-for-xsimd-dependency-.patch |  55 ++++
>  ...dd-libxsimd-dev-build-dependency-for.patch |  41 +++
>  patches/series                                |   8 +
>  8 files changed, 636 insertions(+)
>  create mode 100644 patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
>  create mode 100644 patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
>  create mode 100644 patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
>  create mode 100644 patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
>  create mode 100644 patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
>  create mode 100644 patches/0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch
>  create mode 100644 patches/0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch
> 
> diff --git a/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch b/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
> new file mode 100644
> index 00000000000..c6208b0db2a
> --- /dev/null
> +++ b/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
> @@ -0,0 +1,66 @@
> +From 2685f6c35245d033e73e16ece3e356fcead44cea Mon Sep 17 00:00:00 2001
> +From: John Mulligan <jmulligan@redhat.com>
> +Date: Thu, 21 Aug 2025 12:30:55 -0400
> +Subject: [PATCH 42/46] cephadm: fix some issues running existing cephadm build
> + tests
> +
> +As time has marched on and people changed things our tests no longer
> +match the expected inputs.
> +
> +Signed-off-by: John Mulligan <jmulligan@redhat.com>
> +(cherry picked from commit 31c8010faa417ca53614bd30379a9b9c0c9199de)
> +---
> + src/cephadm/tests/build/test_cephadm_build.py | 18 ++++++++++++------
> + 1 file changed, 12 insertions(+), 6 deletions(-)
> +
> +diff --git a/src/cephadm/tests/build/test_cephadm_build.py b/src/cephadm/tests/build/test_cephadm_build.py
> +index c2995a76d4b..9beef67ffe0 100644
> +--- a/src/cephadm/tests/build/test_cephadm_build.py
> ++++ b/src/cephadm/tests/build/test_cephadm_build.py
> +@@ -34,12 +34,12 @@ CONTAINERS = {
> +     },
> +     'ubuntu-20.04': {
> +         'name': 'cephadm-build-test:ubuntu-20-04-py3',
> +-        'base_image': 'quay.io/library/ubuntu:20.04',
> ++        'base_image': 'docker.io/library/ubuntu:20.04',
> +         'script': 'apt update && apt install -y python3-venv',
> +     },
> +     'ubuntu-22.04': {
> +         'name': 'cephadm-build-test:ubuntu-22-04-py3',
> +-        'base_image': 'quay.io/library/ubuntu:22.04',
> ++        'base_image': 'docker.io/library/ubuntu:22.04',
> +         'script': 'apt update && apt install -y python3-venv',
> +     },
> + }
> +@@ -128,8 +128,8 @@ def test_cephadm_build(env, source_dir, tmp_path):
> +     assert all('requirements_entry' in v for v in data['bundled_packages'])
> +     assert 'zip_root_entries' in data
> +     zre = data['zip_root_entries']
> +-    assert any(e.startswith('Jinja2') for e in zre)
> +-    assert any(e.startswith('MarkupSafe') for e in zre)
> ++    assert any(_dist_info(e, 'Jinja2') for e in zre)
> ++    assert any(_dist_info(e, 'MarkupSafe') for e in zre)
> +     assert any(e.startswith('jinja2') for e in zre)
> +     assert any(e.startswith('markupsafe') for e in zre)
> +     assert any(e.startswith('cephadmlib') for e in zre)
> +@@ -184,9 +184,15 @@ def test_cephadm_build_from_rpms(env, source_dir, tmp_path):
> +     assert all('requirements_entry' in v for v in data['bundled_packages'])
> +     assert 'zip_root_entries' in data
> +     zre = data['zip_root_entries']
> +-    assert any(e.startswith('Jinja2') for e in zre)
> +-    assert any(e.startswith('MarkupSafe') for e in zre)
> ++    assert any(_dist_info(e, 'Jinja2') for e in zre)
> ++    assert any(_dist_info(e, 'MarkupSafe') for e in zre)
> +     assert any(e.startswith('jinja2') for e in zre)
> +     assert any(e.startswith('markupsafe') for e in zre)
> +     assert any(e.startswith('cephadmlib') for e in zre)
> +     assert any(e.startswith('_cephadmmeta') for e in zre)
> ++
> ++
> ++def _dist_info(entry, name):
> ++    return (
> ++        entry.startswith(entry) or entry.startswith(entry.lower())
> ++    ) and (entry.endswith('.dist-info') or entry.endswith('.egg-info'))
> +-- 
> +2.47.3
> +
> diff --git a/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch b/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
> new file mode 100644
> index 00000000000..7a0b1a18d68
> --- /dev/null
> +++ b/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
> @@ -0,0 +1,62 @@
> +From 2c20ae95f32a2bdf78576252f1c9171f00a2790e Mon Sep 17 00:00:00 2001
> +From: Kefu Chai <k.chai@proxmox.com>
> +Date: Mon, 10 Nov 2025 12:11:08 +0800
> +Subject: [PATCH 43/46] cephadm: fix zip_root_entries population in version
> + command
> +
> +The 'cephadm version --verbose' command was returning an empty
> +zip_root_entries list because it relied on the private '_files'
> +attribute of zipimport.zipimporter, which is not reliably populated
> +across Python versions.
> +
> +This commit fixes the issue by using the zipfile module to properly
> +read the archive contents via the loader.archive path. This ensures
> +that zip_root_entries is correctly populated with the root-level
> +directories in the zipapp.
> +
> +This fix is necessary for the cephadm build tests to properly validate
> +that all expected packages and modules are included in the built zipapp.
> +
> +Signed-off-by: Kefu Chai <k.chai@proxmox.com>
> +(cherry picked from commit 2c68c1496dbb7cd01bf783e31510940445040a34)
> +---
> + src/cephadm/cephadm.py | 16 ++++++++++++----
> + 1 file changed, 12 insertions(+), 4 deletions(-)
> +
> +diff --git a/src/cephadm/cephadm.py b/src/cephadm/cephadm.py
> +index f75aaa86dac..8d08c700868 100755
> +--- a/src/cephadm/cephadm.py
> ++++ b/src/cephadm/cephadm.py
> +@@ -1688,6 +1688,7 @@ def command_version(ctx):
> +     # type: (CephadmContext) -> int
> +     import importlib
> +     import zipimport
> ++    import zipfile
> +     import types
> + 
> +     vmod: Optional[types.ModuleType]
> +@@ -1744,10 +1745,17 @@ def command_version(ctx):
> +             out['bundled_packages'] = deps_info
> +         except OSError:
> +             pass
> +-        files = getattr(loader, '_files', {})
> +-        out['zip_root_entries'] = sorted(
> +-            {p.split('/')[0] for p in files.keys()}
> +-        )
> ++        # Use zipfile module to properly read the archive contents
> ++        # loader.archive contains the path to the zip file
> ++        try:
> ++            with zipfile.ZipFile(loader.archive, 'r') as zf:
> ++                files = zf.namelist()
> ++                out['zip_root_entries'] = sorted(
> ++                    {p.split('/')[0] for p in files if p}
> ++                )
> ++        except (OSError, zipfile.BadZipFile):
> ++            # Fallback to empty list if we can't read the zip
> ++            out['zip_root_entries'] = []
> + 
> +     json.dump(out, sys.stdout, indent=2)
> +     print()
> +-- 
> +2.47.3
> +
> diff --git a/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch b/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
> new file mode 100644
> index 00000000000..6e1b4dd2e0e
> --- /dev/null
> +++ b/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
> @@ -0,0 +1,89 @@
> +From 5ce72178a782dfe2e80648063b025ba23d96d5d8 Mon Sep 17 00:00:00 2001
> +From: John Mulligan <jmulligan@redhat.com>
> +Date: Thu, 21 Aug 2025 12:44:17 -0400
> +Subject: [PATCH 44/46] cephadm: support cephadm rpm based builds without
> + top_level.txt
> +
> +Signed-off-by: John Mulligan <jmulligan@redhat.com>
> +(cherry picked from commit 26a499a8da339d870af193ea964368afbc84c694)
> +---
> + src/cephadm/build.py | 56 +++++++++++++++++++++++++++++++++-----------
> + 1 file changed, 42 insertions(+), 14 deletions(-)
> +
> +diff --git a/src/cephadm/build.py b/src/cephadm/build.py
> +index 43bc58a4003..01c91fc1680 100755
> +--- a/src/cephadm/build.py
> ++++ b/src/cephadm/build.py
> +@@ -434,6 +434,47 @@ def _install_rpm_deps(tempdir, config):
> +     return dinfo
> + 
> + 
> ++def _gather_rpm_package_dirs(paths):
> ++    # = The easy way =
> ++    # the top_level.txt file can be used to determine where the python packages
> ++    # actually are. We need all of those and the meta-data dir (parent of
> ++    # top_level.txt) to be included in our zipapp
> ++    top_level = None
> ++    for path in paths:
> ++        if path.endswith('top_level.txt'):
> ++            top_level = pathlib.Path(path)
> ++    if top_level:
> ++        meta_dir = top_level.parent
> ++        pkg_dirs = [
> ++            top_level.parent.parent / p
> ++            for p in top_level.read_text().splitlines()
> ++        ]
> ++        return meta_dir, pkg_dirs
> ++    # = The hard way =
> ++    # loop through the directories to find the .dist-info dir (containing the
> ++    # mandatory METADATA file, according to the spec) and once we know the
> ++    # location of dist info we find the sibling paths from the rpm listing
> ++    dist_info = None
> ++    ppaths = []
> ++    for path in paths:
> ++        ppath = pathlib.Path(path)
> ++        ppaths.append(ppath)
> ++        if ppath.name == 'METADATA' and ppath.parent.name.endswith('.dist-info'):
> ++            dist_info = ppath.parent
> ++            break
> ++    if not dist_info:
> ++        raise ValueError('no .dist-info METADATA found')
> ++    if not dist_info.parent.name == 'site-packages':
> ++        raise ValueError(
> ++            'unexpected parent directory (not site-packages):'
> ++            f' {dist_info.parent.name}'
> ++        )
> ++    siblings = [
> ++        p for p in ppaths if p.parent == dist_info.parent and p != dist_info
> ++    ]
> ++    return dist_info, siblings
> ++
> ++
> + def _deps_from_rpm(tempdir, config, dinfo, pkg):
> +     # first, figure out what rpm provides a particular python lib
> +     dist = f'python3.{sys.version_info.minor}dist({pkg})'.lower()
> +@@ -469,20 +510,7 @@ def _deps_from_rpm(tempdir, config, dinfo, pkg):
> +         ['rpm', '-ql', rpmname], check=True, stdout=subprocess.PIPE
> +     )
> +     paths = [l.decode('utf8') for l in res.stdout.splitlines()]
> +-    # the top_level.txt file can be used to determine where the python packages
> +-    # actually are. We need all of those and the meta-data dir (parent of
> +-    # top_level.txt) to be included in our zipapp
> +-    top_level = None
> +-    for path in paths:
> +-        if path.endswith('top_level.txt'):
> +-            top_level = pathlib.Path(path)
> +-    if not top_level:
> +-        raise ValueError('top_level not found')
> +-    meta_dir = top_level.parent
> +-    pkg_dirs = [
> +-        top_level.parent.parent / p
> +-        for p in top_level.read_text().splitlines()
> +-    ]
> ++    meta_dir, pkg_dirs = _gather_rpm_package_dirs(paths)
> +     meta_dest = tempdir / meta_dir.name
> +     log.info(f"Copying {meta_dir} to {meta_dest}")
> +     # copy the meta data directory
> +-- 
> +2.47.3
> +
> diff --git a/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch b/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
> new file mode 100644
> index 00000000000..94241490b64
> --- /dev/null
> +++ b/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
> @@ -0,0 +1,261 @@
> +From a4744feb1457aa7fb3f47b42f29138682cc7d41c Mon Sep 17 00:00:00 2001
> +From: Kefu Chai <k.chai@proxmox.com>
> +Date: Tue, 14 Oct 2025 21:04:42 +0800
> +Subject: [PATCH 45/46] cephadm/build: Add Debian package support for bundled
> + dependencies
> +
> +Extends the cephadm build script to support bundling dependencies from
> +Debian packages in addition to pip and RPM packages. This allows building
> +cephadm on Debian-based distributions using system packages.
> +
> +Key changes:
> +- Add 'deb' to DependencyMode enum to enable Debian package mode
> +- Implement _setup_deb() to configure Debian dependency requirements
> +- Add _install_deb_deps() to orchestrate Debian package installation
> +- Add _gather_deb_package_dirs() to parse Debian package file listings
> +  and locate Python package directories (handles both site-packages and
> +  dist-packages directories used by Debian)
> +- Add _deps_from_deb() to extract Python dependencies from installed
> +  Debian packages using dpkg/apt-cache tools
> +- Fix variable reference bug in _install_deps() (deps.mode -> config.deps_mode)
> +
> +The Debian implementation follows a similar pattern to the existing RPM
> +support, using dpkg-query and dpkg -L to locate installed packages and
> +their files, with special handling for Debian naming conventions
> +(e.g., PyYAML -> python3-yaml).
> +
> +Signed-off-by: Kefu Chai <k.chai@proxmox.com>
> +(cherry picked from commit 3ff9b0c24e33debe95a0a0c6b42da30be788871c)
> +---
> + src/cephadm/build.py | 158 +++++++++++++++++++++++++++++++++++++++----
> + 1 file changed, 143 insertions(+), 15 deletions(-)
> +
> +diff --git a/src/cephadm/build.py b/src/cephadm/build.py
> +index 01c91fc1680..53d3983de89 100755
> +--- a/src/cephadm/build.py
> ++++ b/src/cephadm/build.py
> +@@ -154,6 +154,7 @@ class PipEnv(enum.Enum):
> + class DependencyMode(enum.Enum):
> +     pip = enum.auto()
> +     rpm = enum.auto()
> ++    deb = enum.auto()
> +     none = enum.auto()
> + 
> + 
> +@@ -169,6 +170,8 @@ class Config:
> +             self._setup_pip()
> +         elif self.deps_mode == DependencyMode.rpm:
> +             self._setup_rpm()
> ++        elif self.deps_mode == DependencyMode.deb:
> ++            self._setup_deb()
> + 
> +     def _setup_pip(self):
> +         if self._maj_min == (3, 6):
> +@@ -180,6 +183,9 @@ class Config:
> +     def _setup_rpm(self):
> +         self.requirements = [InstallSpec(**v) for v in PY_REQUIREMENTS]
> + 
> ++    def _setup_deb(self):
> ++        self.requirements = [InstallSpec(**v) for v in PY_REQUIREMENTS]
> ++
> + 
> + class DependencyInfo:
> +     """Type for tracking bundled dependencies."""
> +@@ -333,7 +339,9 @@ def _install_deps(tempdir, config):
> +         return _install_pip_deps(tempdir, config)
> +     if config.deps_mode == DependencyMode.rpm:
> +         return _install_rpm_deps(tempdir, config)
> +-    raise ValueError(f'unexpected deps mode: {deps.mode}')
> ++    if config.deps_mode == DependencyMode.deb:
> ++        return _install_deb_deps(tempdir, config)
> ++    raise ValueError(f'unexpected deps mode: {config.deps_mode}')
> + 
> + 
> + def _install_pip_deps(tempdir, config):
> +@@ -434,7 +442,26 @@ def _install_rpm_deps(tempdir, config):
> +     return dinfo
> + 
> + 
> +-def _gather_rpm_package_dirs(paths):
> ++def _install_deb_deps(tempdir, config):
> ++    log.info("Installing dependencies using Debian packages")
> ++    dinfo = DependencyInfo(config)
> ++    for pkg in config.requirements:
> ++        log.info(f"Looking for debian package for: {pkg.name!r}")
> ++        _deps_from_deb(tempdir, config, dinfo, pkg.name)
> ++    return dinfo
> ++
> ++
> ++def _gather_package_dirs(paths, expected_parent_dir):
> ++    """Parse package file listing to find Python package directories.
> ++
> ++    Args:
> ++        paths: List of file paths from package listing
> ++        expected_parent_dir: Expected parent directory name (e.g., 'site-packages' for RPM,
> ++                           'dist-packages' for Debian)
> ++
> ++    Returns:
> ++        Tuple of (metadata_dir, package_dirs)
> ++    """
> +     # = The easy way =
> +     # the top_level.txt file can be used to determine where the python packages
> +     # actually are. We need all of those and the meta-data dir (parent of
> +@@ -453,7 +480,7 @@ def _gather_rpm_package_dirs(paths):
> +     # = The hard way =
> +     # loop through the directories to find the .dist-info dir (containing the
> +     # mandatory METADATA file, according to the spec) and once we know the
> +-    # location of dist info we find the sibling paths from the rpm listing
> ++    # location of dist info we find the sibling paths from the package listing
> +     dist_info = None
> +     ppaths = []
> +     for path in paths:
> +@@ -464,9 +491,9 @@ def _gather_rpm_package_dirs(paths):
> +             break
> +     if not dist_info:
> +         raise ValueError('no .dist-info METADATA found')
> +-    if not dist_info.parent.name == 'site-packages':
> ++    if dist_info.parent.name != expected_parent_dir:
> +         raise ValueError(
> +-            'unexpected parent directory (not site-packages):'
> ++            f'unexpected parent directory (not {expected_parent_dir}):'
> +             f' {dist_info.parent.name}'
> +         )
> +     siblings = [
> +@@ -475,6 +502,31 @@ def _gather_rpm_package_dirs(paths):
> +     return dist_info, siblings
> + 
> + 
> ++def _copy_package_files(tempdir, paths, expected_parent_dir):
> ++    """Copy package files to the build directory.
> ++
> ++    Args:
> ++        tempdir: Temporary directory to copy files to
> ++        paths: List of file paths from package listing
> ++        expected_parent_dir: Expected parent directory name per packaging convention:
> ++            - 'site-packages' for RPM-based distributions
> ++            - 'dist-packages' for Debian-based distributions
> ++
> ++    Returns:
> ++        None
> ++    """
> ++    meta_dir, pkg_dirs = _gather_package_dirs(paths, expected_parent_dir)
> ++    meta_dest = tempdir / meta_dir.name
> ++    log.info(f"Copying {meta_dir} to {meta_dest}")
> ++    # copy the meta data directory
> ++    shutil.copytree(meta_dir, meta_dest, ignore=_ignore_cephadmlib)
> ++    # copy all the package directories
> ++    for pkg_dir in pkg_dirs:
> ++        pkg_dest = tempdir / pkg_dir.name
> ++        log.info(f"Copying {pkg_dir} to {pkg_dest}")
> ++        shutil.copytree(pkg_dir, pkg_dest, ignore=_ignore_cephadmlib)
> ++
> ++
> + def _deps_from_rpm(tempdir, config, dinfo, pkg):
> +     # first, figure out what rpm provides a particular python lib
> +     dist = f'python3.{sys.version_info.minor}dist({pkg})'.lower()
> +@@ -510,16 +562,92 @@ def _deps_from_rpm(tempdir, config, dinfo, pkg):
> +         ['rpm', '-ql', rpmname], check=True, stdout=subprocess.PIPE
> +     )
> +     paths = [l.decode('utf8') for l in res.stdout.splitlines()]
> +-    meta_dir, pkg_dirs = _gather_rpm_package_dirs(paths)
> +-    meta_dest = tempdir / meta_dir.name
> +-    log.info(f"Copying {meta_dir} to {meta_dest}")
> +-    # copy the meta data directory
> +-    shutil.copytree(meta_dir, meta_dest, ignore=_ignore_cephadmlib)
> +-    # copy all the package directories
> +-    for pkg_dir in pkg_dirs:
> +-        pkg_dest = tempdir / pkg_dir.name
> +-        log.info(f"Copying {pkg_dir} to {pkg_dest}")
> +-        shutil.copytree(pkg_dir, pkg_dest, ignore=_ignore_cephadmlib)
> ++    # RPM-based distributions use 'site-packages' for Python packages
> ++    _copy_package_files(tempdir, paths, 'site-packages')
> ++
> ++
> ++def _deps_from_deb(tempdir, config, dinfo, pkg):
> ++    """Extract Python dependencies from Debian packages.
> ++
> ++    Args:
> ++        tempdir: Temporary directory to copy package files to
> ++        config: Build configuration
> ++        dinfo: DependencyInfo instance to track dependencies
> ++        pkg: Python package name (e.g., 'MarkupSafe', 'Jinja2', 'PyYAML')
> ++    """
> ++    # Convert Python package name to Debian package name
> ++    # Python packages are typically named python3-<lowercase-name>
> ++    # Handle special cases: PyYAML -> python3-yaml, MarkupSafe -> python3-markupsafe
> ++    pkg_lower = pkg.lower()
> ++    if pkg_lower == 'pyyaml':
> ++        deb_pkg_name = 'python3-yaml'
> ++    else:
> ++        deb_pkg_name = f'python3-{pkg_lower}'
> ++
> ++    # First, try to find the package using apt-cache
> ++    # This helps verify the package exists before trying to list its files
> ++    try:
> ++        res = subprocess.run(
> ++            ['apt-cache', 'show', deb_pkg_name],
> ++            check=True,
> ++            stdout=subprocess.PIPE,
> ++            stderr=subprocess.PIPE,
> ++        )
> ++    except subprocess.CalledProcessError:
> ++        # Package not found, try alternative naming
> ++        log.warning(f"Package {deb_pkg_name} not found via apt-cache, trying dpkg -S")
> ++        # Try to search for files that might belong to this package
> ++        # Search for the Python module in site-packages
> ++        search_pattern = f'/usr/lib/python3*/dist-packages/{pkg.lower()}*'
> ++        try:
> ++            res = subprocess.run(
> ++                ['dpkg', '-S', search_pattern],
> ++                check=True,
> ++                stdout=subprocess.PIPE,
> ++                stderr=subprocess.PIPE,
> ++            )
> ++            # dpkg -S output format: "package: /path/to/file"
> ++            deb_pkg_name = res.stdout.decode('utf8').split(':')[0].strip()
> ++        except subprocess.CalledProcessError as err:
> ++            log.error(f"Could not find Debian package for {pkg}")
> ++            log.error(f"Tried: {deb_pkg_name} and pattern search")
> ++            sys.exit(1)
> ++
> ++    # Get version information using dpkg-query
> ++    try:
> ++        res = subprocess.run(
> ++            ['dpkg-query', '-W', '-f=${Version}\\n', deb_pkg_name],
> ++            check=True,
> ++            stdout=subprocess.PIPE,
> ++        )
> ++        version = res.stdout.decode('utf8').strip()
> ++    except subprocess.CalledProcessError as err:
> ++        log.error(f"Could not query version for package {deb_pkg_name}: {err}")
> ++        sys.exit(1)
> ++
> ++    log.info(f"Debian Package: {deb_pkg_name} (version: {version})")
> ++    dinfo.add(
> ++        pkg,
> ++        deb_name=deb_pkg_name,
> ++        version=version,
> ++        package_source='deb',
> ++    )
> ++
> ++    # Get the list of files provided by the Debian package
> ++    try:
> ++        res = subprocess.run(
> ++            ['dpkg', '-L', deb_pkg_name],
> ++            check=True,
> ++            stdout=subprocess.PIPE,
> ++        )
> ++    except subprocess.CalledProcessError as err:
> ++        log.error(f"Could not list files for package {deb_pkg_name}: {err}")
> ++        sys.exit(1)
> ++
> ++    paths = [l.decode('utf8') for l in res.stdout.splitlines()]
> ++    # Debian-based distributions use 'dist-packages' for system-managed Python packages
> ++    # per Debian Python Policy: https://www.debian.org/doc/packaging-manuals/python-policy/
> ++    _copy_package_files(tempdir, paths, 'dist-packages')
> + 
> + 
> + def generate_version_file(versioning_vars, dest):
> +-- 
> +2.47.3
> +
> diff --git a/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch b/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
> new file mode 100644
> index 00000000000..0b4daa63f79
> --- /dev/null
> +++ b/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
> @@ -0,0 +1,54 @@
> +From 779c9251669b65b7ed77a841f281e2d957174db2 Mon Sep 17 00:00:00 2001
> +From: Kefu Chai <k.chai@proxmox.com>
> +Date: Tue, 21 Oct 2025 11:25:00 +0800
> +Subject: [PATCH 46/46] debian: Use system packages for cephadm bundled
> + dependencies
> +
> +Configure the Debian build to use CEPHADM_BUNDLED_DEPENDENCIES=deb,
> +which instructs the cephadm build script to bundle dependencies from
> +system-installed Debian packages instead of downloading from PyPI.
> +
> +This change addresses build failures in restricted network environments
> +where Debian build tools do not permit internet access. By leveraging
> +the Debian package support added in commit 9378a2988e1, the build now
> +uses python3-markupsafe, python3-jinja2, and python3-yaml packages
> +that are already installed as build dependencies.
> +
> +This approach mirrors the existing RPM packaging workflow, ensuring
> +consistent behavior across different distribution package formats.
> +
> +Signed-off-by: Kefu Chai <k.chai@proxmox.com>
> +(cherry picked from commit 25680021ee2422f8b8b4075b3ab77af39126eecc)

long term, it might be even better to not bundle at all, but add those
python deps as runtime dependencies?

> +---
> + debian/control | 2 ++
> + debian/rules   | 1 +
> + 2 files changed, 3 insertions(+)
> +
> +diff --git a/debian/control b/debian/control
> +index 35c1dbe5bd2..94997827bd2 100644
> +--- a/debian/control
> ++++ b/debian/control
> +@@ -106,6 +106,8 @@ Build-Depends: automake,
> +                python3-requests <pkg.ceph.check>,
> +                python3-scipy <pkg.ceph.check>,
> +                python3-onelogin-saml2 <pkg.ceph.check>,
> ++               python3-jinja2,
> ++               python3-markupsafe,
> +                python3-setuptools,
> +                python3-sphinx,
> +                python3-venv,
> +diff --git a/debian/rules b/debian/rules
> +index b935efaa8cd..ee4eff2bdc5 100755
> +--- a/debian/rules
> ++++ b/debian/rules
> +@@ -30,6 +30,7 @@ extraopts += -DWITH_CEPHFS_JAVA=ON
> + extraopts += -DWITH_CEPHFS_SHELL=ON
> + extraopts += -DWITH_SYSTEMD=ON -DCEPH_SYSTEMD_ENV_DIR=/etc/default
> + extraopts += -DWITH_GRAFANA=ON
> ++extraopts += -DCEPHADM_BUNDLED_DEPENDENCIES=deb
> + ifeq ($(DEB_HOST_ARCH), amd64)
> +   extraopts += -DWITH_RBD_RWL=ON
> + else
> +-- 
> +2.47.3
> +
> diff --git a/patches/0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch b/patches/0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch
> new file mode 100644
> index 00000000000..ab5681bece0
> --- /dev/null
> +++ b/patches/0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch
> @@ -0,0 +1,55 @@
> +From 46803fcaa3b2b4f14318b1320e8e3beb9cf3eec4 Mon Sep 17 00:00:00 2001
> +From: Kefu Chai <k.chai@proxmox.com>
> +Date: Wed, 15 Oct 2025 15:46:22 +0800
> +Subject: [PATCH 47/48] cmake/BuildArrow: Use AUTO mode for xsimd dependency
> + detection
> +MIME-Version: 1.0
> +Content-Type: text/plain; charset=UTF-8
> +Content-Transfer-Encoding: 8bit
> +
> +Arrow requires xsimd >= 9.0.1 according to arrow/cpp/thirdparty/versions.txt.
> +Previously, we unconditionally set -Dxsimd_SOURCE=BUNDLED, forcing the use
> +of Arrow's vendored xsimd regardless of system package availability.
> +
> +This commit changes to -Dxsimd_SOURCE=AUTO, which allows Arrow's
> +resolve_dependency mechanism to automatically:
> +1. Try to find system xsimd package
> +2. Check if version >= 9.0.1
> +3. Use system version if found and sufficient
> +4. Fall back to bundled version otherwise
> +
> +This reduces build time and dependencies on systems with sufficient xsimd,
> +while maintaining compatibility with older distributions.
> +
> +Distribution availability:
> +- Ubuntu Noble (24.04): libxsimd-dev 12.1.1 (✓ will use system)
> +- Ubuntu Jammy (22.04): libxsimd-dev 7.6.0 (✗ will use bundled)
> +- Debian Trixie (13): libxsimd-dev 13.2.0 (✓ will use system)
> +- CentOS Stream 9: xsimd-devel 7.4.9 (✗ will use bundled)
> +
> +Signed-off-by: Kefu Chai <k.chai@proxmox.com>
> +(cherry picked from commit 2defde30e4c61cd9f33196b91ac16505928ce774)
> +---
> + cmake/modules/BuildArrow.cmake | 6 +++---
> + 1 file changed, 3 insertions(+), 3 deletions(-)
> +
> +diff --git a/cmake/modules/BuildArrow.cmake b/cmake/modules/BuildArrow.cmake
> +index 0ee1d85b49f..cadc2909fa4 100644
> +--- a/cmake/modules/BuildArrow.cmake
> ++++ b/cmake/modules/BuildArrow.cmake
> +@@ -69,9 +69,9 @@ function(build_arrow)
> +     list(APPEND arrow_DEPENDS Boost)
> +   endif()
> + 
> +-  # since Arrow 15.0.0 needs xsimd>=8.1.0 and since Ubuntu Jammy
> +-  # Jellyfish only provides 7.6.0, we'll have arrow build it as source
> +-  list(APPEND arrow_CMAKE_ARGS -Dxsimd_SOURCE=BUNDLED)
> ++  # Arrow requires xsimd >= 9.0.1 (see arrow/cpp/thirdparty/versions.txt).
> ++  # Use AUTO to let Arrow detect system xsimd and fall back to bundled if needed.
> ++  list(APPEND arrow_CMAKE_ARGS -Dxsimd_SOURCE=AUTO)
> + 
> +   # cmake doesn't properly handle arguments containing ";", such as
> +   # CMAKE_PREFIX_PATH, for which reason we'll have to use some other separator.
> +-- 
> +2.47.3
> +
> diff --git a/patches/0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch b/patches/0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch
> new file mode 100644
> index 00000000000..c431fb6edc1
> --- /dev/null
> +++ b/patches/0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch
> @@ -0,0 +1,41 @@
> +From 20fb1b8eff9d9b4e683c33e556752085b408c24b Mon Sep 17 00:00:00 2001
> +From: Kefu Chai <k.chai@proxmox.com>
> +Date: Wed, 15 Oct 2025 15:46:26 +0800
> +Subject: [PATCH 48/48] debian/control: Add libxsimd-dev build dependency for
> + vendored Arrow
> +
> +In commit e8460cbd, we introduced the "pkg.ceph.arrow" build profile to
> +support building with system Arrow packages. However, neither Debian nor
> +Ubuntu currently ships Arrow packages.
> +
> +Since WITH_RADOSGW_SELECT_PARQUET is always enabled in debian/rules,
> +Arrow support is required for all builds. When the pkg.ceph.arrow profile
> +is not selected, the build uses vendored Arrow. With the recent change to
> +use AUTO mode for xsimd detection, Arrow will attempt to find system xsimd
> +>= 9.0.1. Adding libxsimd-dev as a build dependency ensures it's available
> +for Arrow to detect and use, reducing build time on supported distributions.
> +
> +On distributions with insufficient xsimd versions (< 9.0.1), Arrow will
> +automatically fall back to its bundled version.
> +
> +Signed-off-by: Kefu Chai <k.chai@proxmox.com>
> +(cherry picked from commit 929b47f83880683b376b9dcc0060938447f11c88)
> +---
> + debian/control | 1 +
> + 1 file changed, 1 insertion(+)
> +
> +diff --git a/debian/control b/debian/control
> +index 94997827bd2..7d30e1faa70 100644
> +--- a/debian/control
> ++++ b/debian/control
> +@@ -85,6 +85,7 @@ Build-Depends: automake,
> +                libpmem-dev <pkg.ceph.pmdk>,
> +                libpmemobj-dev (>= 1.8) <pkg.ceph.pmdk>,
> +                libprotobuf-dev <pkg.ceph.crimson>,
> ++               libxsimd-dev <!pkg.ceph.arrow>,
> +                ninja-build,
> +                nlohmann-json3-dev,
> +                patch,
> +-- 
> +2.47.3
> +
> diff --git a/patches/series b/patches/series
> index d775a8a068d..f2bfed1b939 100644
> --- a/patches/series
> +++ b/patches/series
> @@ -39,3 +39,11 @@
>  0039-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch
>  0040-pybind-rbd-disable-on_progress-callbacks-to-prevent-.patch
>  0041-systemd-services-fix-installing-ceph-volume.patch
> +0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
> +0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
> +0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
> +0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
> +0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
> +0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch
> +0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch
> +  
> -- 
> 2.47.3
> 
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel