[pve-devel] [PATCH ceph] cherry-pick changes to use system packages for bundled deps

* [pve-devel] [PATCH ceph] cherry-pick changes to use system packages for bundled deps
@ 2026-01-21 15:12 Kefu Chai
  2026-01-22  8:29 ` [pve-devel] [PATCH ceph] cherry-pick changes to buiild crimson package Kefu Chai
  0 siblings, 1 reply; 15+ messages in thread
From: Kefu Chai @ 2026-01-21 15:12 UTC (permalink / raw)
  To: pve-devel

Previously, cephadm's build process pulled dependencies from PyPI,
which failed in environments with restricted network access. This
change backports upstream improvements that enable building cephadm
using system packages instead, mirroring the existing RPM packaging
workflow.

And the vendored Arrow pulled xsimd from GitHub, this also failed in
building environment without network access. This change backports
upstream improvements enabling the build to use libxsimd-dev package
when building debian packages.

Backported changes:
- https://github.com/ceph/ceph/pull/66256 - Use system packages for
  cephadm bundled dependencies
- https://github.com/ceph/ceph/pull/66248 - Add libxsimd-dev build
  dependency for vendored Arrow
- https://github.com/ceph/ceph/pull/65292 - cephadm: fix building
  rpm-sourced cephadm zippapp on el10

Note: Test commits from these PRs are excluded from this backport.

Signed-off-by: Kefu Chai <k.chai@proxmox.com>
---
 ...-issues-running-existing-cephadm-bui.patch |  66 +++++
 ...root_entries-population-in-version-c.patch |  62 +++++
 ...cephadm-rpm-based-builds-without-top.patch |  89 ++++++
 ...d-Debian-package-support-for-bundled.patch | 261 ++++++++++++++++++
 ...m-packages-for-cephadm-bundled-depen.patch |  54 ++++
 ...-Use-AUTO-mode-for-xsimd-dependency-.patch |  55 ++++
 ...dd-libxsimd-dev-build-dependency-for.patch |  41 +++
 patches/series                                |   8 +
 8 files changed, 636 insertions(+)
 create mode 100644 patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
 create mode 100644 patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
 create mode 100644 patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
 create mode 100644 patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
 create mode 100644 patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
 create mode 100644 patches/0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch
 create mode 100644 patches/0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch

diff --git a/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch b/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
new file mode 100644
index 00000000000..c6208b0db2a
--- /dev/null
+++ b/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
@@ -0,0 +1,66 @@
+From 2685f6c35245d033e73e16ece3e356fcead44cea Mon Sep 17 00:00:00 2001
+From: John Mulligan <jmulligan@redhat.com>
+Date: Thu, 21 Aug 2025 12:30:55 -0400
+Subject: [PATCH 42/46] cephadm: fix some issues running existing cephadm build
+ tests
+
+As time has marched on and people changed things our tests no longer
+match the expected inputs.
+
+Signed-off-by: John Mulligan <jmulligan@redhat.com>
+(cherry picked from commit 31c8010faa417ca53614bd30379a9b9c0c9199de)
+---
+ src/cephadm/tests/build/test_cephadm_build.py | 18 ++++++++++++------
+ 1 file changed, 12 insertions(+), 6 deletions(-)
+
+diff --git a/src/cephadm/tests/build/test_cephadm_build.py b/src/cephadm/tests/build/test_cephadm_build.py
+index c2995a76d4b..9beef67ffe0 100644
+--- a/src/cephadm/tests/build/test_cephadm_build.py
++++ b/src/cephadm/tests/build/test_cephadm_build.py
+@@ -34,12 +34,12 @@ CONTAINERS = {
+     },
+     'ubuntu-20.04': {
+         'name': 'cephadm-build-test:ubuntu-20-04-py3',
+-        'base_image': 'quay.io/library/ubuntu:20.04',
++        'base_image': 'docker.io/library/ubuntu:20.04',
+         'script': 'apt update && apt install -y python3-venv',
+     },
+     'ubuntu-22.04': {
+         'name': 'cephadm-build-test:ubuntu-22-04-py3',
+-        'base_image': 'quay.io/library/ubuntu:22.04',
++        'base_image': 'docker.io/library/ubuntu:22.04',
+         'script': 'apt update && apt install -y python3-venv',
+     },
+ }
+@@ -128,8 +128,8 @@ def test_cephadm_build(env, source_dir, tmp_path):
+     assert all('requirements_entry' in v for v in data['bundled_packages'])
+     assert 'zip_root_entries' in data
+     zre = data['zip_root_entries']
+-    assert any(e.startswith('Jinja2') for e in zre)
+-    assert any(e.startswith('MarkupSafe') for e in zre)
++    assert any(_dist_info(e, 'Jinja2') for e in zre)
++    assert any(_dist_info(e, 'MarkupSafe') for e in zre)
+     assert any(e.startswith('jinja2') for e in zre)
+     assert any(e.startswith('markupsafe') for e in zre)
+     assert any(e.startswith('cephadmlib') for e in zre)
+@@ -184,9 +184,15 @@ def test_cephadm_build_from_rpms(env, source_dir, tmp_path):
+     assert all('requirements_entry' in v for v in data['bundled_packages'])
+     assert 'zip_root_entries' in data
+     zre = data['zip_root_entries']
+-    assert any(e.startswith('Jinja2') for e in zre)
+-    assert any(e.startswith('MarkupSafe') for e in zre)
++    assert any(_dist_info(e, 'Jinja2') for e in zre)
++    assert any(_dist_info(e, 'MarkupSafe') for e in zre)
+     assert any(e.startswith('jinja2') for e in zre)
+     assert any(e.startswith('markupsafe') for e in zre)
+     assert any(e.startswith('cephadmlib') for e in zre)
+     assert any(e.startswith('_cephadmmeta') for e in zre)
++
++
++def _dist_info(entry, name):
++    return (
++        entry.startswith(entry) or entry.startswith(entry.lower())
++    ) and (entry.endswith('.dist-info') or entry.endswith('.egg-info'))
+-- 
+2.47.3
+
diff --git a/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch b/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
new file mode 100644
index 00000000000..7a0b1a18d68
--- /dev/null
+++ b/patches/0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
@@ -0,0 +1,62 @@
+From 2c20ae95f32a2bdf78576252f1c9171f00a2790e Mon Sep 17 00:00:00 2001
+From: Kefu Chai <k.chai@proxmox.com>
+Date: Mon, 10 Nov 2025 12:11:08 +0800
+Subject: [PATCH 43/46] cephadm: fix zip_root_entries population in version
+ command
+
+The 'cephadm version --verbose' command was returning an empty
+zip_root_entries list because it relied on the private '_files'
+attribute of zipimport.zipimporter, which is not reliably populated
+across Python versions.
+
+This commit fixes the issue by using the zipfile module to properly
+read the archive contents via the loader.archive path. This ensures
+that zip_root_entries is correctly populated with the root-level
+directories in the zipapp.
+
+This fix is necessary for the cephadm build tests to properly validate
+that all expected packages and modules are included in the built zipapp.
+
+Signed-off-by: Kefu Chai <k.chai@proxmox.com>
+(cherry picked from commit 2c68c1496dbb7cd01bf783e31510940445040a34)
+---
+ src/cephadm/cephadm.py | 16 ++++++++++++----
+ 1 file changed, 12 insertions(+), 4 deletions(-)
+
+diff --git a/src/cephadm/cephadm.py b/src/cephadm/cephadm.py
+index f75aaa86dac..8d08c700868 100755
+--- a/src/cephadm/cephadm.py
++++ b/src/cephadm/cephadm.py
+@@ -1688,6 +1688,7 @@ def command_version(ctx):
+     # type: (CephadmContext) -> int
+     import importlib
+     import zipimport
++    import zipfile
+     import types
+ 
+     vmod: Optional[types.ModuleType]
+@@ -1744,10 +1745,17 @@ def command_version(ctx):
+             out['bundled_packages'] = deps_info
+         except OSError:
+             pass
+-        files = getattr(loader, '_files', {})
+-        out['zip_root_entries'] = sorted(
+-            {p.split('/')[0] for p in files.keys()}
+-        )
++        # Use zipfile module to properly read the archive contents
++        # loader.archive contains the path to the zip file
++        try:
++            with zipfile.ZipFile(loader.archive, 'r') as zf:
++                files = zf.namelist()
++                out['zip_root_entries'] = sorted(
++                    {p.split('/')[0] for p in files if p}
++                )
++        except (OSError, zipfile.BadZipFile):
++            # Fallback to empty list if we can't read the zip
++            out['zip_root_entries'] = []
+ 
+     json.dump(out, sys.stdout, indent=2)
+     print()
+-- 
+2.47.3
+
diff --git a/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch b/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
new file mode 100644
index 00000000000..6e1b4dd2e0e
--- /dev/null
+++ b/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
@@ -0,0 +1,89 @@
+From 5ce72178a782dfe2e80648063b025ba23d96d5d8 Mon Sep 17 00:00:00 2001
+From: John Mulligan <jmulligan@redhat.com>
+Date: Thu, 21 Aug 2025 12:44:17 -0400
+Subject: [PATCH 44/46] cephadm: support cephadm rpm based builds without
+ top_level.txt
+
+Signed-off-by: John Mulligan <jmulligan@redhat.com>
+(cherry picked from commit 26a499a8da339d870af193ea964368afbc84c694)
+---
+ src/cephadm/build.py | 56 +++++++++++++++++++++++++++++++++-----------
+ 1 file changed, 42 insertions(+), 14 deletions(-)
+
+diff --git a/src/cephadm/build.py b/src/cephadm/build.py
+index 43bc58a4003..01c91fc1680 100755
+--- a/src/cephadm/build.py
++++ b/src/cephadm/build.py
+@@ -434,6 +434,47 @@ def _install_rpm_deps(tempdir, config):
+     return dinfo
+ 
+ 
++def _gather_rpm_package_dirs(paths):
++    # = The easy way =
++    # the top_level.txt file can be used to determine where the python packages
++    # actually are. We need all of those and the meta-data dir (parent of
++    # top_level.txt) to be included in our zipapp
++    top_level = None
++    for path in paths:
++        if path.endswith('top_level.txt'):
++            top_level = pathlib.Path(path)
++    if top_level:
++        meta_dir = top_level.parent
++        pkg_dirs = [
++            top_level.parent.parent / p
++            for p in top_level.read_text().splitlines()
++        ]
++        return meta_dir, pkg_dirs
++    # = The hard way =
++    # loop through the directories to find the .dist-info dir (containing the
++    # mandatory METADATA file, according to the spec) and once we know the
++    # location of dist info we find the sibling paths from the rpm listing
++    dist_info = None
++    ppaths = []
++    for path in paths:
++        ppath = pathlib.Path(path)
++        ppaths.append(ppath)
++        if ppath.name == 'METADATA' and ppath.parent.name.endswith('.dist-info'):
++            dist_info = ppath.parent
++            break
++    if not dist_info:
++        raise ValueError('no .dist-info METADATA found')
++    if not dist_info.parent.name == 'site-packages':
++        raise ValueError(
++            'unexpected parent directory (not site-packages):'
++            f' {dist_info.parent.name}'
++        )
++    siblings = [
++        p for p in ppaths if p.parent == dist_info.parent and p != dist_info
++    ]
++    return dist_info, siblings
++
++
+ def _deps_from_rpm(tempdir, config, dinfo, pkg):
+     # first, figure out what rpm provides a particular python lib
+     dist = f'python3.{sys.version_info.minor}dist({pkg})'.lower()
+@@ -469,20 +510,7 @@ def _deps_from_rpm(tempdir, config, dinfo, pkg):
+         ['rpm', '-ql', rpmname], check=True, stdout=subprocess.PIPE
+     )
+     paths = [l.decode('utf8') for l in res.stdout.splitlines()]
+-    # the top_level.txt file can be used to determine where the python packages
+-    # actually are. We need all of those and the meta-data dir (parent of
+-    # top_level.txt) to be included in our zipapp
+-    top_level = None
+-    for path in paths:
+-        if path.endswith('top_level.txt'):
+-            top_level = pathlib.Path(path)
+-    if not top_level:
+-        raise ValueError('top_level not found')
+-    meta_dir = top_level.parent
+-    pkg_dirs = [
+-        top_level.parent.parent / p
+-        for p in top_level.read_text().splitlines()
+-    ]
++    meta_dir, pkg_dirs = _gather_rpm_package_dirs(paths)
+     meta_dest = tempdir / meta_dir.name
+     log.info(f"Copying {meta_dir} to {meta_dest}")
+     # copy the meta data directory
+-- 
+2.47.3
+
diff --git a/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch b/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
new file mode 100644
index 00000000000..94241490b64
--- /dev/null
+++ b/patches/0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
@@ -0,0 +1,261 @@
+From a4744feb1457aa7fb3f47b42f29138682cc7d41c Mon Sep 17 00:00:00 2001
+From: Kefu Chai <k.chai@proxmox.com>
+Date: Tue, 14 Oct 2025 21:04:42 +0800
+Subject: [PATCH 45/46] cephadm/build: Add Debian package support for bundled
+ dependencies
+
+Extends the cephadm build script to support bundling dependencies from
+Debian packages in addition to pip and RPM packages. This allows building
+cephadm on Debian-based distributions using system packages.
+
+Key changes:
+- Add 'deb' to DependencyMode enum to enable Debian package mode
+- Implement _setup_deb() to configure Debian dependency requirements
+- Add _install_deb_deps() to orchestrate Debian package installation
+- Add _gather_deb_package_dirs() to parse Debian package file listings
+  and locate Python package directories (handles both site-packages and
+  dist-packages directories used by Debian)
+- Add _deps_from_deb() to extract Python dependencies from installed
+  Debian packages using dpkg/apt-cache tools
+- Fix variable reference bug in _install_deps() (deps.mode -> config.deps_mode)
+
+The Debian implementation follows a similar pattern to the existing RPM
+support, using dpkg-query and dpkg -L to locate installed packages and
+their files, with special handling for Debian naming conventions
+(e.g., PyYAML -> python3-yaml).
+
+Signed-off-by: Kefu Chai <k.chai@proxmox.com>
+(cherry picked from commit 3ff9b0c24e33debe95a0a0c6b42da30be788871c)
+---
+ src/cephadm/build.py | 158 +++++++++++++++++++++++++++++++++++++++----
+ 1 file changed, 143 insertions(+), 15 deletions(-)
+
+diff --git a/src/cephadm/build.py b/src/cephadm/build.py
+index 01c91fc1680..53d3983de89 100755
+--- a/src/cephadm/build.py
++++ b/src/cephadm/build.py
+@@ -154,6 +154,7 @@ class PipEnv(enum.Enum):
+ class DependencyMode(enum.Enum):
+     pip = enum.auto()
+     rpm = enum.auto()
++    deb = enum.auto()
+     none = enum.auto()
+ 
+ 
+@@ -169,6 +170,8 @@ class Config:
+             self._setup_pip()
+         elif self.deps_mode == DependencyMode.rpm:
+             self._setup_rpm()
++        elif self.deps_mode == DependencyMode.deb:
++            self._setup_deb()
+ 
+     def _setup_pip(self):
+         if self._maj_min == (3, 6):
+@@ -180,6 +183,9 @@ class Config:
+     def _setup_rpm(self):
+         self.requirements = [InstallSpec(**v) for v in PY_REQUIREMENTS]
+ 
++    def _setup_deb(self):
++        self.requirements = [InstallSpec(**v) for v in PY_REQUIREMENTS]
++
+ 
+ class DependencyInfo:
+     """Type for tracking bundled dependencies."""
+@@ -333,7 +339,9 @@ def _install_deps(tempdir, config):
+         return _install_pip_deps(tempdir, config)
+     if config.deps_mode == DependencyMode.rpm:
+         return _install_rpm_deps(tempdir, config)
+-    raise ValueError(f'unexpected deps mode: {deps.mode}')
++    if config.deps_mode == DependencyMode.deb:
++        return _install_deb_deps(tempdir, config)
++    raise ValueError(f'unexpected deps mode: {config.deps_mode}')
+ 
+ 
+ def _install_pip_deps(tempdir, config):
+@@ -434,7 +442,26 @@ def _install_rpm_deps(tempdir, config):
+     return dinfo
+ 
+ 
+-def _gather_rpm_package_dirs(paths):
++def _install_deb_deps(tempdir, config):
++    log.info("Installing dependencies using Debian packages")
++    dinfo = DependencyInfo(config)
++    for pkg in config.requirements:
++        log.info(f"Looking for debian package for: {pkg.name!r}")
++        _deps_from_deb(tempdir, config, dinfo, pkg.name)
++    return dinfo
++
++
++def _gather_package_dirs(paths, expected_parent_dir):
++    """Parse package file listing to find Python package directories.
++
++    Args:
++        paths: List of file paths from package listing
++        expected_parent_dir: Expected parent directory name (e.g., 'site-packages' for RPM,
++                           'dist-packages' for Debian)
++
++    Returns:
++        Tuple of (metadata_dir, package_dirs)
++    """
+     # = The easy way =
+     # the top_level.txt file can be used to determine where the python packages
+     # actually are. We need all of those and the meta-data dir (parent of
+@@ -453,7 +480,7 @@ def _gather_rpm_package_dirs(paths):
+     # = The hard way =
+     # loop through the directories to find the .dist-info dir (containing the
+     # mandatory METADATA file, according to the spec) and once we know the
+-    # location of dist info we find the sibling paths from the rpm listing
++    # location of dist info we find the sibling paths from the package listing
+     dist_info = None
+     ppaths = []
+     for path in paths:
+@@ -464,9 +491,9 @@ def _gather_rpm_package_dirs(paths):
+             break
+     if not dist_info:
+         raise ValueError('no .dist-info METADATA found')
+-    if not dist_info.parent.name == 'site-packages':
++    if dist_info.parent.name != expected_parent_dir:
+         raise ValueError(
+-            'unexpected parent directory (not site-packages):'
++            f'unexpected parent directory (not {expected_parent_dir}):'
+             f' {dist_info.parent.name}'
+         )
+     siblings = [
+@@ -475,6 +502,31 @@ def _gather_rpm_package_dirs(paths):
+     return dist_info, siblings
+ 
+ 
++def _copy_package_files(tempdir, paths, expected_parent_dir):
++    """Copy package files to the build directory.
++
++    Args:
++        tempdir: Temporary directory to copy files to
++        paths: List of file paths from package listing
++        expected_parent_dir: Expected parent directory name per packaging convention:
++            - 'site-packages' for RPM-based distributions
++            - 'dist-packages' for Debian-based distributions
++
++    Returns:
++        None
++    """
++    meta_dir, pkg_dirs = _gather_package_dirs(paths, expected_parent_dir)
++    meta_dest = tempdir / meta_dir.name
++    log.info(f"Copying {meta_dir} to {meta_dest}")
++    # copy the meta data directory
++    shutil.copytree(meta_dir, meta_dest, ignore=_ignore_cephadmlib)
++    # copy all the package directories
++    for pkg_dir in pkg_dirs:
++        pkg_dest = tempdir / pkg_dir.name
++        log.info(f"Copying {pkg_dir} to {pkg_dest}")
++        shutil.copytree(pkg_dir, pkg_dest, ignore=_ignore_cephadmlib)
++
++
+ def _deps_from_rpm(tempdir, config, dinfo, pkg):
+     # first, figure out what rpm provides a particular python lib
+     dist = f'python3.{sys.version_info.minor}dist({pkg})'.lower()
+@@ -510,16 +562,92 @@ def _deps_from_rpm(tempdir, config, dinfo, pkg):
+         ['rpm', '-ql', rpmname], check=True, stdout=subprocess.PIPE
+     )
+     paths = [l.decode('utf8') for l in res.stdout.splitlines()]
+-    meta_dir, pkg_dirs = _gather_rpm_package_dirs(paths)
+-    meta_dest = tempdir / meta_dir.name
+-    log.info(f"Copying {meta_dir} to {meta_dest}")
+-    # copy the meta data directory
+-    shutil.copytree(meta_dir, meta_dest, ignore=_ignore_cephadmlib)
+-    # copy all the package directories
+-    for pkg_dir in pkg_dirs:
+-        pkg_dest = tempdir / pkg_dir.name
+-        log.info(f"Copying {pkg_dir} to {pkg_dest}")
+-        shutil.copytree(pkg_dir, pkg_dest, ignore=_ignore_cephadmlib)
++    # RPM-based distributions use 'site-packages' for Python packages
++    _copy_package_files(tempdir, paths, 'site-packages')
++
++
++def _deps_from_deb(tempdir, config, dinfo, pkg):
++    """Extract Python dependencies from Debian packages.
++
++    Args:
++        tempdir: Temporary directory to copy package files to
++        config: Build configuration
++        dinfo: DependencyInfo instance to track dependencies
++        pkg: Python package name (e.g., 'MarkupSafe', 'Jinja2', 'PyYAML')
++    """
++    # Convert Python package name to Debian package name
++    # Python packages are typically named python3-<lowercase-name>
++    # Handle special cases: PyYAML -> python3-yaml, MarkupSafe -> python3-markupsafe
++    pkg_lower = pkg.lower()
++    if pkg_lower == 'pyyaml':
++        deb_pkg_name = 'python3-yaml'
++    else:
++        deb_pkg_name = f'python3-{pkg_lower}'
++
++    # First, try to find the package using apt-cache
++    # This helps verify the package exists before trying to list its files
++    try:
++        res = subprocess.run(
++            ['apt-cache', 'show', deb_pkg_name],
++            check=True,
++            stdout=subprocess.PIPE,
++            stderr=subprocess.PIPE,
++        )
++    except subprocess.CalledProcessError:
++        # Package not found, try alternative naming
++        log.warning(f"Package {deb_pkg_name} not found via apt-cache, trying dpkg -S")
++        # Try to search for files that might belong to this package
++        # Search for the Python module in site-packages
++        search_pattern = f'/usr/lib/python3*/dist-packages/{pkg.lower()}*'
++        try:
++            res = subprocess.run(
++                ['dpkg', '-S', search_pattern],
++                check=True,
++                stdout=subprocess.PIPE,
++                stderr=subprocess.PIPE,
++            )
++            # dpkg -S output format: "package: /path/to/file"
++            deb_pkg_name = res.stdout.decode('utf8').split(':')[0].strip()
++        except subprocess.CalledProcessError as err:
++            log.error(f"Could not find Debian package for {pkg}")
++            log.error(f"Tried: {deb_pkg_name} and pattern search")
++            sys.exit(1)
++
++    # Get version information using dpkg-query
++    try:
++        res = subprocess.run(
++            ['dpkg-query', '-W', '-f=${Version}\\n', deb_pkg_name],
++            check=True,
++            stdout=subprocess.PIPE,
++        )
++        version = res.stdout.decode('utf8').strip()
++    except subprocess.CalledProcessError as err:
++        log.error(f"Could not query version for package {deb_pkg_name}: {err}")
++        sys.exit(1)
++
++    log.info(f"Debian Package: {deb_pkg_name} (version: {version})")
++    dinfo.add(
++        pkg,
++        deb_name=deb_pkg_name,
++        version=version,
++        package_source='deb',
++    )
++
++    # Get the list of files provided by the Debian package
++    try:
++        res = subprocess.run(
++            ['dpkg', '-L', deb_pkg_name],
++            check=True,
++            stdout=subprocess.PIPE,
++        )
++    except subprocess.CalledProcessError as err:
++        log.error(f"Could not list files for package {deb_pkg_name}: {err}")
++        sys.exit(1)
++
++    paths = [l.decode('utf8') for l in res.stdout.splitlines()]
++    # Debian-based distributions use 'dist-packages' for system-managed Python packages
++    # per Debian Python Policy: https://www.debian.org/doc/packaging-manuals/python-policy/
++    _copy_package_files(tempdir, paths, 'dist-packages')
+ 
+ 
+ def generate_version_file(versioning_vars, dest):
+-- 
+2.47.3
+
diff --git a/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch b/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
new file mode 100644
index 00000000000..0b4daa63f79
--- /dev/null
+++ b/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
@@ -0,0 +1,54 @@
+From 779c9251669b65b7ed77a841f281e2d957174db2 Mon Sep 17 00:00:00 2001
+From: Kefu Chai <k.chai@proxmox.com>
+Date: Tue, 21 Oct 2025 11:25:00 +0800
+Subject: [PATCH 46/46] debian: Use system packages for cephadm bundled
+ dependencies
+
+Configure the Debian build to use CEPHADM_BUNDLED_DEPENDENCIES=deb,
+which instructs the cephadm build script to bundle dependencies from
+system-installed Debian packages instead of downloading from PyPI.
+
+This change addresses build failures in restricted network environments
+where Debian build tools do not permit internet access. By leveraging
+the Debian package support added in commit 9378a2988e1, the build now
+uses python3-markupsafe, python3-jinja2, and python3-yaml packages
+that are already installed as build dependencies.
+
+This approach mirrors the existing RPM packaging workflow, ensuring
+consistent behavior across different distribution package formats.
+
+Signed-off-by: Kefu Chai <k.chai@proxmox.com>
+(cherry picked from commit 25680021ee2422f8b8b4075b3ab77af39126eecc)
+---
+ debian/control | 2 ++
+ debian/rules   | 1 +
+ 2 files changed, 3 insertions(+)
+
+diff --git a/debian/control b/debian/control
+index 35c1dbe5bd2..94997827bd2 100644
+--- a/debian/control
++++ b/debian/control
+@@ -106,6 +106,8 @@ Build-Depends: automake,
+                python3-requests <pkg.ceph.check>,
+                python3-scipy <pkg.ceph.check>,
+                python3-onelogin-saml2 <pkg.ceph.check>,
++               python3-jinja2,
++               python3-markupsafe,
+                python3-setuptools,
+                python3-sphinx,
+                python3-venv,
+diff --git a/debian/rules b/debian/rules
+index b935efaa8cd..ee4eff2bdc5 100755
+--- a/debian/rules
++++ b/debian/rules
+@@ -30,6 +30,7 @@ extraopts += -DWITH_CEPHFS_JAVA=ON
+ extraopts += -DWITH_CEPHFS_SHELL=ON
+ extraopts += -DWITH_SYSTEMD=ON -DCEPH_SYSTEMD_ENV_DIR=/etc/default
+ extraopts += -DWITH_GRAFANA=ON
++extraopts += -DCEPHADM_BUNDLED_DEPENDENCIES=deb
+ ifeq ($(DEB_HOST_ARCH), amd64)
+   extraopts += -DWITH_RBD_RWL=ON
+ else
+-- 
+2.47.3
+
diff --git a/patches/0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch b/patches/0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch
new file mode 100644
index 00000000000..ab5681bece0
--- /dev/null
+++ b/patches/0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch
@@ -0,0 +1,55 @@
+From 46803fcaa3b2b4f14318b1320e8e3beb9cf3eec4 Mon Sep 17 00:00:00 2001
+From: Kefu Chai <k.chai@proxmox.com>
+Date: Wed, 15 Oct 2025 15:46:22 +0800
+Subject: [PATCH 47/48] cmake/BuildArrow: Use AUTO mode for xsimd dependency
+ detection
+MIME-Version: 1.0
+Content-Type: text/plain; charset=UTF-8
+Content-Transfer-Encoding: 8bit
+
+Arrow requires xsimd >= 9.0.1 according to arrow/cpp/thirdparty/versions.txt.
+Previously, we unconditionally set -Dxsimd_SOURCE=BUNDLED, forcing the use
+of Arrow's vendored xsimd regardless of system package availability.
+
+This commit changes to -Dxsimd_SOURCE=AUTO, which allows Arrow's
+resolve_dependency mechanism to automatically:
+1. Try to find system xsimd package
+2. Check if version >= 9.0.1
+3. Use system version if found and sufficient
+4. Fall back to bundled version otherwise
+
+This reduces build time and dependencies on systems with sufficient xsimd,
+while maintaining compatibility with older distributions.
+
+Distribution availability:
+- Ubuntu Noble (24.04): libxsimd-dev 12.1.1 (✓ will use system)
+- Ubuntu Jammy (22.04): libxsimd-dev 7.6.0 (✗ will use bundled)
+- Debian Trixie (13): libxsimd-dev 13.2.0 (✓ will use system)
+- CentOS Stream 9: xsimd-devel 7.4.9 (✗ will use bundled)
+
+Signed-off-by: Kefu Chai <k.chai@proxmox.com>
+(cherry picked from commit 2defde30e4c61cd9f33196b91ac16505928ce774)
+---
+ cmake/modules/BuildArrow.cmake | 6 +++---
+ 1 file changed, 3 insertions(+), 3 deletions(-)
+
+diff --git a/cmake/modules/BuildArrow.cmake b/cmake/modules/BuildArrow.cmake
+index 0ee1d85b49f..cadc2909fa4 100644
+--- a/cmake/modules/BuildArrow.cmake
++++ b/cmake/modules/BuildArrow.cmake
+@@ -69,9 +69,9 @@ function(build_arrow)
+     list(APPEND arrow_DEPENDS Boost)
+   endif()
+ 
+-  # since Arrow 15.0.0 needs xsimd>=8.1.0 and since Ubuntu Jammy
+-  # Jellyfish only provides 7.6.0, we'll have arrow build it as source
+-  list(APPEND arrow_CMAKE_ARGS -Dxsimd_SOURCE=BUNDLED)
++  # Arrow requires xsimd >= 9.0.1 (see arrow/cpp/thirdparty/versions.txt).
++  # Use AUTO to let Arrow detect system xsimd and fall back to bundled if needed.
++  list(APPEND arrow_CMAKE_ARGS -Dxsimd_SOURCE=AUTO)
+ 
+   # cmake doesn't properly handle arguments containing ";", such as
+   # CMAKE_PREFIX_PATH, for which reason we'll have to use some other separator.
+-- 
+2.47.3
+
diff --git a/patches/0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch b/patches/0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch
new file mode 100644
index 00000000000..c431fb6edc1
--- /dev/null
+++ b/patches/0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch
@@ -0,0 +1,41 @@
+From 20fb1b8eff9d9b4e683c33e556752085b408c24b Mon Sep 17 00:00:00 2001
+From: Kefu Chai <k.chai@proxmox.com>
+Date: Wed, 15 Oct 2025 15:46:26 +0800
+Subject: [PATCH 48/48] debian/control: Add libxsimd-dev build dependency for
+ vendored Arrow
+
+In commit e8460cbd, we introduced the "pkg.ceph.arrow" build profile to
+support building with system Arrow packages. However, neither Debian nor
+Ubuntu currently ships Arrow packages.
+
+Since WITH_RADOSGW_SELECT_PARQUET is always enabled in debian/rules,
+Arrow support is required for all builds. When the pkg.ceph.arrow profile
+is not selected, the build uses vendored Arrow. With the recent change to
+use AUTO mode for xsimd detection, Arrow will attempt to find system xsimd
+>= 9.0.1. Adding libxsimd-dev as a build dependency ensures it's available
+for Arrow to detect and use, reducing build time on supported distributions.
+
+On distributions with insufficient xsimd versions (< 9.0.1), Arrow will
+automatically fall back to its bundled version.
+
+Signed-off-by: Kefu Chai <k.chai@proxmox.com>
+(cherry picked from commit 929b47f83880683b376b9dcc0060938447f11c88)
+---
+ debian/control | 1 +
+ 1 file changed, 1 insertion(+)
+
+diff --git a/debian/control b/debian/control
+index 94997827bd2..7d30e1faa70 100644
+--- a/debian/control
++++ b/debian/control
+@@ -85,6 +85,7 @@ Build-Depends: automake,
+                libpmem-dev <pkg.ceph.pmdk>,
+                libpmemobj-dev (>= 1.8) <pkg.ceph.pmdk>,
+                libprotobuf-dev <pkg.ceph.crimson>,
++               libxsimd-dev <!pkg.ceph.arrow>,
+                ninja-build,
+                nlohmann-json3-dev,
+                patch,
+-- 
+2.47.3
+
diff --git a/patches/series b/patches/series
index d775a8a068d..f2bfed1b939 100644
--- a/patches/series
+++ b/patches/series
@@ -39,3 +39,11 @@
 0039-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch
 0040-pybind-rbd-disable-on_progress-callbacks-to-prevent-.patch
 0041-systemd-services-fix-installing-ceph-volume.patch
+0042-cephadm-fix-some-issues-running-existing-cephadm-bui.patch
+0043-cephadm-fix-zip_root_entries-population-in-version-c.patch
+0044-cephadm-support-cephadm-rpm-based-builds-without-top.patch
+0045-cephadm-build-Add-Debian-package-support-for-bundled.patch
+0046-debian-Use-system-packages-for-cephadm-bundled-depen.patch
+0047-cmake-BuildArrow-Use-AUTO-mode-for-xsimd-dependency-.patch
+0048-debian-control-Add-libxsimd-dev-build-dependency-for.patch
+  
-- 
2.47.3



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

^ permalink raw reply	[flat|nested] 15+ messages in thread