From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id A184F1FF141 for ; Fri, 30 Jan 2026 03:18:37 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 434BC1188D; Fri, 30 Jan 2026 03:19:02 +0100 (CET) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Fri, 30 Jan 2026 10:18:06 +0800 Message-Id: Cc: "pve-devel" Subject: Re: [pve-devel] [PATCH ceph 1/4 v5] cherry-pick changes to use system packages for bundled deps From: "Kefu Chai" To: "Proxmox VE development discussion" X-Mailer: aerc 0.20.0 References: <20260123075619.2731724-3-k.chai@proxmox.com> <1769170653.ul6jn3m5d0.astroid@yuna.none> <1769418342.xjabsrxuw3.astroid@yuna.none> In-Reply-To: <1769418342.xjabsrxuw3.astroid@yuna.none> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1769739433360 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.036 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_LOTSOFHASH 0.25 Emails with lots of hash-like gibberish KAM_SHORT 0.001 Use of a URL Shortener for very short URL RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [parent.name,proxmox.com,cephadm.py,docker.io,quay.io,ppath.name] Message-ID-Hash: RGHLD3EM7X3WNLGU25WAFBUXOFAWPX4A X-Message-ID-Hash: RGHLD3EM7X3WNLGU25WAFBUXOFAWPX4A X-MailFrom: k.chai@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi Fabian, Is there anything I can do to help move this patchset forward? I'd be happy to extract the libxsimd-dev changes into a separate commit if that would make review easier, as it might be less controversial on its own. On Mon Jan 26, 2026 at 5:06 PM CST, Fabian Gr=C3=BCnbichler wrote: > On January 24, 2026 5:13 am, Kefu Chai wrote: >> On Fri Jan 23, 2026 at 9:03 PM CST, Fabian Gr=C3=BCnbichler wrote: >>> one small question inline, otherwise this one LGTM >>> >>> On January 23, 2026 8:56 am, Kefu Chai wrote: >>>> Previously, cephadm's build process pulled dependencies from PyPI, >>>> which failed in environments with restricted network access. This >>>> change backports upstream improvements that enable building cephadm >>>> using system packages instead, mirroring the existing RPM packaging >>>> workflow. >>>>=20 >>>> And the vendored Arrow pulled xsimd from GitHub, this also failed in >>>> building environment without network access. This change backports >>>> upstream improvements enabling the build to use libxsimd-dev package >>>> when building debian packages. >>>>=20 >>>> Backported changes: >>>> - https://github.com/ceph/ceph/pull/66256 - Use system packages for >>>> cephadm bundled dependencies >>>> - https://github.com/ceph/ceph/pull/66248 - Add libxsimd-dev build >>>> dependency for vendored Arrow >>>> - https://github.com/ceph/ceph/pull/65292 - cephadm: fix building >>>> rpm-sourced cephadm zippapp on el10 >>>>=20 >>>> Note: Test commits from these PRs are excluded from this backport. >>>>=20 >>>> Signed-off-by: Kefu Chai >>>> --- >>>> ...-issues-running-existing-cephadm-bui.patch | 66 +++++ >>>> ...root_entries-population-in-version-c.patch | 62 +++++ >>>> ...cephadm-rpm-based-builds-without-top.patch | 89 ++++++ >>>> ...d-Debian-package-support-for-bundled.patch | 261 +++++++++++++++++= + >>>> ...m-packages-for-cephadm-bundled-depen.patch | 54 ++++ >>>> ...-Use-AUTO-mode-for-xsimd-dependency-.patch | 55 ++++ >>>> ...dd-libxsimd-dev-build-dependency-for.patch | 41 +++ >>>> patches/series | 8 + >>>> 8 files changed, 636 insertions(+) >>>> create mode 100644 patches/0042-cephadm-fix-some-issues-running-exist= ing-cephadm-bui.patch >>>> create mode 100644 patches/0043-cephadm-fix-zip_root_entries-populati= on-in-version-c.patch >>>> create mode 100644 patches/0044-cephadm-support-cephadm-rpm-based-bui= lds-without-top.patch >>>> create mode 100644 patches/0045-cephadm-build-Add-Debian-package-supp= ort-for-bundled.patch >>>> create mode 100644 patches/0046-debian-Use-system-packages-for-cephad= m-bundled-depen.patch >>>> create mode 100644 patches/0047-cmake-BuildArrow-Use-AUTO-mode-for-xs= imd-dependency-.patch >>>> create mode 100644 patches/0048-debian-control-Add-libxsimd-dev-build= -dependency-for.patch >>>>=20 >>>> diff --git a/patches/0042-cephadm-fix-some-issues-running-existing-cep= hadm-bui.patch b/patches/0042-cephadm-fix-some-issues-running-existing-ceph= adm-bui.patch >>>> new file mode 100644 >>>> index 00000000000..c6208b0db2a >>>> --- /dev/null >>>> +++ b/patches/0042-cephadm-fix-some-issues-running-existing-cephadm-bu= i.patch >>>> @@ -0,0 +1,66 @@ >>>> +From 2685f6c35245d033e73e16ece3e356fcead44cea Mon Sep 17 00:00:00 200= 1 >>>> +From: John Mulligan >>>> +Date: Thu, 21 Aug 2025 12:30:55 -0400 >>>> +Subject: [PATCH 42/46] cephadm: fix some issues running existing ceph= adm build >>>> + tests >>>> + >>>> +As time has marched on and people changed things our tests no longer >>>> +match the expected inputs. >>>> + >>>> +Signed-off-by: John Mulligan >>>> +(cherry picked from commit 31c8010faa417ca53614bd30379a9b9c0c9199de) >>>> +--- >>>> + src/cephadm/tests/build/test_cephadm_build.py | 18 ++++++++++++-----= - >>>> + 1 file changed, 12 insertions(+), 6 deletions(-) >>>> + >>>> +diff --git a/src/cephadm/tests/build/test_cephadm_build.py b/src/ceph= adm/tests/build/test_cephadm_build.py >>>> +index c2995a76d4b..9beef67ffe0 100644 >>>> +--- a/src/cephadm/tests/build/test_cephadm_build.py >>>> ++++ b/src/cephadm/tests/build/test_cephadm_build.py >>>> +@@ -34,12 +34,12 @@ CONTAINERS =3D { >>>> + }, >>>> + 'ubuntu-20.04': { >>>> + 'name': 'cephadm-build-test:ubuntu-20-04-py3', >>>> +- 'base_image': 'quay.io/library/ubuntu:20.04', >>>> ++ 'base_image': 'docker.io/library/ubuntu:20.04', >>>> + 'script': 'apt update && apt install -y python3-venv', >>>> + }, >>>> + 'ubuntu-22.04': { >>>> + 'name': 'cephadm-build-test:ubuntu-22-04-py3', >>>> +- 'base_image': 'quay.io/library/ubuntu:22.04', >>>> ++ 'base_image': 'docker.io/library/ubuntu:22.04', >>>> + 'script': 'apt update && apt install -y python3-venv', >>>> + }, >>>> + } >>>> +@@ -128,8 +128,8 @@ def test_cephadm_build(env, source_dir, tmp_path)= : >>>> + assert all('requirements_entry' in v for v in data['bundled_pack= ages']) >>>> + assert 'zip_root_entries' in data >>>> + zre =3D data['zip_root_entries'] >>>> +- assert any(e.startswith('Jinja2') for e in zre) >>>> +- assert any(e.startswith('MarkupSafe') for e in zre) >>>> ++ assert any(_dist_info(e, 'Jinja2') for e in zre) >>>> ++ assert any(_dist_info(e, 'MarkupSafe') for e in zre) >>>> + assert any(e.startswith('jinja2') for e in zre) >>>> + assert any(e.startswith('markupsafe') for e in zre) >>>> + assert any(e.startswith('cephadmlib') for e in zre) >>>> +@@ -184,9 +184,15 @@ def test_cephadm_build_from_rpms(env, source_dir= , tmp_path): >>>> + assert all('requirements_entry' in v for v in data['bundled_pack= ages']) >>>> + assert 'zip_root_entries' in data >>>> + zre =3D data['zip_root_entries'] >>>> +- assert any(e.startswith('Jinja2') for e in zre) >>>> +- assert any(e.startswith('MarkupSafe') for e in zre) >>>> ++ assert any(_dist_info(e, 'Jinja2') for e in zre) >>>> ++ assert any(_dist_info(e, 'MarkupSafe') for e in zre) >>>> + assert any(e.startswith('jinja2') for e in zre) >>>> + assert any(e.startswith('markupsafe') for e in zre) >>>> + assert any(e.startswith('cephadmlib') for e in zre) >>>> + assert any(e.startswith('_cephadmmeta') for e in zre) >>>> ++ >>>> ++ >>>> ++def _dist_info(entry, name): >>>> ++ return ( >>>> ++ entry.startswith(entry) or entry.startswith(entry.lower()) >>>> ++ ) and (entry.endswith('.dist-info') or entry.endswith('.egg-info= ')) >>>> +--=20 >>>> +2.47.3 >>>> + >>>> diff --git a/patches/0043-cephadm-fix-zip_root_entries-population-in-v= ersion-c.patch b/patches/0043-cephadm-fix-zip_root_entries-population-in-ve= rsion-c.patch >>>> new file mode 100644 >>>> index 00000000000..7a0b1a18d68 >>>> --- /dev/null >>>> +++ b/patches/0043-cephadm-fix-zip_root_entries-population-in-version-= c.patch >>>> @@ -0,0 +1,62 @@ >>>> +From 2c20ae95f32a2bdf78576252f1c9171f00a2790e Mon Sep 17 00:00:00 200= 1 >>>> +From: Kefu Chai >>>> +Date: Mon, 10 Nov 2025 12:11:08 +0800 >>>> +Subject: [PATCH 43/46] cephadm: fix zip_root_entries population in ve= rsion >>>> + command >>>> + >>>> +The 'cephadm version --verbose' command was returning an empty >>>> +zip_root_entries list because it relied on the private '_files' >>>> +attribute of zipimport.zipimporter, which is not reliably populated >>>> +across Python versions. >>>> + >>>> +This commit fixes the issue by using the zipfile module to properly >>>> +read the archive contents via the loader.archive path. This ensures >>>> +that zip_root_entries is correctly populated with the root-level >>>> +directories in the zipapp. >>>> + >>>> +This fix is necessary for the cephadm build tests to properly validat= e >>>> +that all expected packages and modules are included in the built zipa= pp. >>>> + >>>> +Signed-off-by: Kefu Chai >>>> +(cherry picked from commit 2c68c1496dbb7cd01bf783e31510940445040a34) >>>> +--- >>>> + src/cephadm/cephadm.py | 16 ++++++++++++---- >>>> + 1 file changed, 12 insertions(+), 4 deletions(-) >>>> + >>>> +diff --git a/src/cephadm/cephadm.py b/src/cephadm/cephadm.py >>>> +index f75aaa86dac..8d08c700868 100755 >>>> +--- a/src/cephadm/cephadm.py >>>> ++++ b/src/cephadm/cephadm.py >>>> +@@ -1688,6 +1688,7 @@ def command_version(ctx): >>>> + # type: (CephadmContext) -> int >>>> + import importlib >>>> + import zipimport >>>> ++ import zipfile >>>> + import types >>>> +=20 >>>> + vmod: Optional[types.ModuleType] >>>> +@@ -1744,10 +1745,17 @@ def command_version(ctx): >>>> + out['bundled_packages'] =3D deps_info >>>> + except OSError: >>>> + pass >>>> +- files =3D getattr(loader, '_files', {}) >>>> +- out['zip_root_entries'] =3D sorted( >>>> +- {p.split('/')[0] for p in files.keys()} >>>> +- ) >>>> ++ # Use zipfile module to properly read the archive contents >>>> ++ # loader.archive contains the path to the zip file >>>> ++ try: >>>> ++ with zipfile.ZipFile(loader.archive, 'r') as zf: >>>> ++ files =3D zf.namelist() >>>> ++ out['zip_root_entries'] =3D sorted( >>>> ++ {p.split('/')[0] for p in files if p} >>>> ++ ) >>>> ++ except (OSError, zipfile.BadZipFile): >>>> ++ # Fallback to empty list if we can't read the zip >>>> ++ out['zip_root_entries'] =3D [] >>>> +=20 >>>> + json.dump(out, sys.stdout, indent=3D2) >>>> + print() >>>> +--=20 >>>> +2.47.3 >>>> + >>>> diff --git a/patches/0044-cephadm-support-cephadm-rpm-based-builds-wit= hout-top.patch b/patches/0044-cephadm-support-cephadm-rpm-based-builds-with= out-top.patch >>>> new file mode 100644 >>>> index 00000000000..6e1b4dd2e0e >>>> --- /dev/null >>>> +++ b/patches/0044-cephadm-support-cephadm-rpm-based-builds-without-to= p.patch >>>> @@ -0,0 +1,89 @@ >>>> +From 5ce72178a782dfe2e80648063b025ba23d96d5d8 Mon Sep 17 00:00:00 200= 1 >>>> +From: John Mulligan >>>> +Date: Thu, 21 Aug 2025 12:44:17 -0400 >>>> +Subject: [PATCH 44/46] cephadm: support cephadm rpm based builds with= out >>>> + top_level.txt >>>> + >>>> +Signed-off-by: John Mulligan >>>> +(cherry picked from commit 26a499a8da339d870af193ea964368afbc84c694) >>>> +--- >>>> + src/cephadm/build.py | 56 +++++++++++++++++++++++++++++++++---------= -- >>>> + 1 file changed, 42 insertions(+), 14 deletions(-) >>>> + >>>> +diff --git a/src/cephadm/build.py b/src/cephadm/build.py >>>> +index 43bc58a4003..01c91fc1680 100755 >>>> +--- a/src/cephadm/build.py >>>> ++++ b/src/cephadm/build.py >>>> +@@ -434,6 +434,47 @@ def _install_rpm_deps(tempdir, config): >>>> + return dinfo >>>> +=20 >>>> +=20 >>>> ++def _gather_rpm_package_dirs(paths): >>>> ++ # =3D The easy way =3D >>>> ++ # the top_level.txt file can be used to determine where the pyth= on packages >>>> ++ # actually are. We need all of those and the meta-data dir (pare= nt of >>>> ++ # top_level.txt) to be included in our zipapp >>>> ++ top_level =3D None >>>> ++ for path in paths: >>>> ++ if path.endswith('top_level.txt'): >>>> ++ top_level =3D pathlib.Path(path) >>>> ++ if top_level: >>>> ++ meta_dir =3D top_level.parent >>>> ++ pkg_dirs =3D [ >>>> ++ top_level.parent.parent / p >>>> ++ for p in top_level.read_text().splitlines() >>>> ++ ] >>>> ++ return meta_dir, pkg_dirs >>>> ++ # =3D The hard way =3D >>>> ++ # loop through the directories to find the .dist-info dir (conta= ining the >>>> ++ # mandatory METADATA file, according to the spec) and once we kn= ow the >>>> ++ # location of dist info we find the sibling paths from the rpm l= isting >>>> ++ dist_info =3D None >>>> ++ ppaths =3D [] >>>> ++ for path in paths: >>>> ++ ppath =3D pathlib.Path(path) >>>> ++ ppaths.append(ppath) >>>> ++ if ppath.name =3D=3D 'METADATA' and ppath.parent.name.endswi= th('.dist-info'): >>>> ++ dist_info =3D ppath.parent >>>> ++ break >>>> ++ if not dist_info: >>>> ++ raise ValueError('no .dist-info METADATA found') >>>> ++ if not dist_info.parent.name =3D=3D 'site-packages': >>>> ++ raise ValueError( >>>> ++ 'unexpected parent directory (not site-packages):' >>>> ++ f' {dist_info.parent.name}' >>>> ++ ) >>>> ++ siblings =3D [ >>>> ++ p for p in ppaths if p.parent =3D=3D dist_info.parent and p = !=3D dist_info >>>> ++ ] >>>> ++ return dist_info, siblings >>>> ++ >>>> ++ >>>> + def _deps_from_rpm(tempdir, config, dinfo, pkg): >>>> + # first, figure out what rpm provides a particular python lib >>>> + dist =3D f'python3.{sys.version_info.minor}dist({pkg})'.lower() >>>> +@@ -469,20 +510,7 @@ def _deps_from_rpm(tempdir, config, dinfo, pkg): >>>> + ['rpm', '-ql', rpmname], check=3DTrue, stdout=3Dsubprocess.P= IPE >>>> + ) >>>> + paths =3D [l.decode('utf8') for l in res.stdout.splitlines()] >>>> +- # the top_level.txt file can be used to determine where the pyth= on packages >>>> +- # actually are. We need all of those and the meta-data dir (pare= nt of >>>> +- # top_level.txt) to be included in our zipapp >>>> +- top_level =3D None >>>> +- for path in paths: >>>> +- if path.endswith('top_level.txt'): >>>> +- top_level =3D pathlib.Path(path) >>>> +- if not top_level: >>>> +- raise ValueError('top_level not found') >>>> +- meta_dir =3D top_level.parent >>>> +- pkg_dirs =3D [ >>>> +- top_level.parent.parent / p >>>> +- for p in top_level.read_text().splitlines() >>>> +- ] >>>> ++ meta_dir, pkg_dirs =3D _gather_rpm_package_dirs(paths) >>>> + meta_dest =3D tempdir / meta_dir.name >>>> + log.info(f"Copying {meta_dir} to {meta_dest}") >>>> + # copy the meta data directory >>>> +--=20 >>>> +2.47.3 >>>> + >>>> diff --git a/patches/0045-cephadm-build-Add-Debian-package-support-for= -bundled.patch b/patches/0045-cephadm-build-Add-Debian-package-support-for-= bundled.patch >>>> new file mode 100644 >>>> index 00000000000..94241490b64 >>>> --- /dev/null >>>> +++ b/patches/0045-cephadm-build-Add-Debian-package-support-for-bundle= d.patch >>>> @@ -0,0 +1,261 @@ >>>> +From a4744feb1457aa7fb3f47b42f29138682cc7d41c Mon Sep 17 00:00:00 200= 1 >>>> +From: Kefu Chai >>>> +Date: Tue, 14 Oct 2025 21:04:42 +0800 >>>> +Subject: [PATCH 45/46] cephadm/build: Add Debian package support for = bundled >>>> + dependencies >>>> + >>>> +Extends the cephadm build script to support bundling dependencies fro= m >>>> +Debian packages in addition to pip and RPM packages. This allows buil= ding >>>> +cephadm on Debian-based distributions using system packages. >>>> + >>>> +Key changes: >>>> +- Add 'deb' to DependencyMode enum to enable Debian package mode >>>> +- Implement _setup_deb() to configure Debian dependency requirements >>>> +- Add _install_deb_deps() to orchestrate Debian package installation >>>> +- Add _gather_deb_package_dirs() to parse Debian package file listing= s >>>> + and locate Python package directories (handles both site-packages a= nd >>>> + dist-packages directories used by Debian) >>>> +- Add _deps_from_deb() to extract Python dependencies from installed >>>> + Debian packages using dpkg/apt-cache tools >>>> +- Fix variable reference bug in _install_deps() (deps.mode -> config.= deps_mode) >>>> + >>>> +The Debian implementation follows a similar pattern to the existing R= PM >>>> +support, using dpkg-query and dpkg -L to locate installed packages an= d >>>> +their files, with special handling for Debian naming conventions >>>> +(e.g., PyYAML -> python3-yaml). >>>> + >>>> +Signed-off-by: Kefu Chai >>>> +(cherry picked from commit 3ff9b0c24e33debe95a0a0c6b42da30be788871c) >>>> +--- >>>> + src/cephadm/build.py | 158 +++++++++++++++++++++++++++++++++++++++--= -- >>>> + 1 file changed, 143 insertions(+), 15 deletions(-) >>>> + >>>> +diff --git a/src/cephadm/build.py b/src/cephadm/build.py >>>> +index 01c91fc1680..53d3983de89 100755 >>>> +--- a/src/cephadm/build.py >>>> ++++ b/src/cephadm/build.py >>>> +@@ -154,6 +154,7 @@ class PipEnv(enum.Enum): >>>> + class DependencyMode(enum.Enum): >>>> + pip =3D enum.auto() >>>> + rpm =3D enum.auto() >>>> ++ deb =3D enum.auto() >>>> + none =3D enum.auto() >>>> +=20 >>>> +=20 >>>> +@@ -169,6 +170,8 @@ class Config: >>>> + self._setup_pip() >>>> + elif self.deps_mode =3D=3D DependencyMode.rpm: >>>> + self._setup_rpm() >>>> ++ elif self.deps_mode =3D=3D DependencyMode.deb: >>>> ++ self._setup_deb() >>>> +=20 >>>> + def _setup_pip(self): >>>> + if self._maj_min =3D=3D (3, 6): >>>> +@@ -180,6 +183,9 @@ class Config: >>>> + def _setup_rpm(self): >>>> + self.requirements =3D [InstallSpec(**v) for v in PY_REQUIREM= ENTS] >>>> +=20 >>>> ++ def _setup_deb(self): >>>> ++ self.requirements =3D [InstallSpec(**v) for v in PY_REQUIREM= ENTS] >>>> ++ >>>> +=20 >>>> + class DependencyInfo: >>>> + """Type for tracking bundled dependencies.""" >>>> +@@ -333,7 +339,9 @@ def _install_deps(tempdir, config): >>>> + return _install_pip_deps(tempdir, config) >>>> + if config.deps_mode =3D=3D DependencyMode.rpm: >>>> + return _install_rpm_deps(tempdir, config) >>>> +- raise ValueError(f'unexpected deps mode: {deps.mode}') >>>> ++ if config.deps_mode =3D=3D DependencyMode.deb: >>>> ++ return _install_deb_deps(tempdir, config) >>>> ++ raise ValueError(f'unexpected deps mode: {config.deps_mode}') >>>> +=20 >>>> +=20 >>>> + def _install_pip_deps(tempdir, config): >>>> +@@ -434,7 +442,26 @@ def _install_rpm_deps(tempdir, config): >>>> + return dinfo >>>> +=20 >>>> +=20 >>>> +-def _gather_rpm_package_dirs(paths): >>>> ++def _install_deb_deps(tempdir, config): >>>> ++ log.info("Installing dependencies using Debian packages") >>>> ++ dinfo =3D DependencyInfo(config) >>>> ++ for pkg in config.requirements: >>>> ++ log.info(f"Looking for debian package for: {pkg.name!r}") >>>> ++ _deps_from_deb(tempdir, config, dinfo, pkg.name) >>>> ++ return dinfo >>>> ++ >>>> ++ >>>> ++def _gather_package_dirs(paths, expected_parent_dir): >>>> ++ """Parse package file listing to find Python package directories= . >>>> ++ >>>> ++ Args: >>>> ++ paths: List of file paths from package listing >>>> ++ expected_parent_dir: Expected parent directory name (e.g., '= site-packages' for RPM, >>>> ++ 'dist-packages' for Debian) >>>> ++ >>>> ++ Returns: >>>> ++ Tuple of (metadata_dir, package_dirs) >>>> ++ """ >>>> + # =3D The easy way =3D >>>> + # the top_level.txt file can be used to determine where the pyth= on packages >>>> + # actually are. We need all of those and the meta-data dir (pare= nt of >>>> +@@ -453,7 +480,7 @@ def _gather_rpm_package_dirs(paths): >>>> + # =3D The hard way =3D >>>> + # loop through the directories to find the .dist-info dir (conta= ining the >>>> + # mandatory METADATA file, according to the spec) and once we kn= ow the >>>> +- # location of dist info we find the sibling paths from the rpm l= isting >>>> ++ # location of dist info we find the sibling paths from the packa= ge listing >>>> + dist_info =3D None >>>> + ppaths =3D [] >>>> + for path in paths: >>>> +@@ -464,9 +491,9 @@ def _gather_rpm_package_dirs(paths): >>>> + break >>>> + if not dist_info: >>>> + raise ValueError('no .dist-info METADATA found') >>>> +- if not dist_info.parent.name =3D=3D 'site-packages': >>>> ++ if dist_info.parent.name !=3D expected_parent_dir: >>>> + raise ValueError( >>>> +- 'unexpected parent directory (not site-packages):' >>>> ++ f'unexpected parent directory (not {expected_parent_dir}= ):' >>>> + f' {dist_info.parent.name}' >>>> + ) >>>> + siblings =3D [ >>>> +@@ -475,6 +502,31 @@ def _gather_rpm_package_dirs(paths): >>>> + return dist_info, siblings >>>> +=20 >>>> +=20 >>>> ++def _copy_package_files(tempdir, paths, expected_parent_dir): >>>> ++ """Copy package files to the build directory. >>>> ++ >>>> ++ Args: >>>> ++ tempdir: Temporary directory to copy files to >>>> ++ paths: List of file paths from package listing >>>> ++ expected_parent_dir: Expected parent directory name per pack= aging convention: >>>> ++ - 'site-packages' for RPM-based distributions >>>> ++ - 'dist-packages' for Debian-based distributions >>>> ++ >>>> ++ Returns: >>>> ++ None >>>> ++ """ >>>> ++ meta_dir, pkg_dirs =3D _gather_package_dirs(paths, expected_pare= nt_dir) >>>> ++ meta_dest =3D tempdir / meta_dir.name >>>> ++ log.info(f"Copying {meta_dir} to {meta_dest}") >>>> ++ # copy the meta data directory >>>> ++ shutil.copytree(meta_dir, meta_dest, ignore=3D_ignore_cephadmlib= ) >>>> ++ # copy all the package directories >>>> ++ for pkg_dir in pkg_dirs: >>>> ++ pkg_dest =3D tempdir / pkg_dir.name >>>> ++ log.info(f"Copying {pkg_dir} to {pkg_dest}") >>>> ++ shutil.copytree(pkg_dir, pkg_dest, ignore=3D_ignore_cephadml= ib) >>>> ++ >>>> ++ >>>> + def _deps_from_rpm(tempdir, config, dinfo, pkg): >>>> + # first, figure out what rpm provides a particular python lib >>>> + dist =3D f'python3.{sys.version_info.minor}dist({pkg})'.lower() >>>> +@@ -510,16 +562,92 @@ def _deps_from_rpm(tempdir, config, dinfo, pkg)= : >>>> + ['rpm', '-ql', rpmname], check=3DTrue, stdout=3Dsubprocess.P= IPE >>>> + ) >>>> + paths =3D [l.decode('utf8') for l in res.stdout.splitlines()] >>>> +- meta_dir, pkg_dirs =3D _gather_rpm_package_dirs(paths) >>>> +- meta_dest =3D tempdir / meta_dir.name >>>> +- log.info(f"Copying {meta_dir} to {meta_dest}") >>>> +- # copy the meta data directory >>>> +- shutil.copytree(meta_dir, meta_dest, ignore=3D_ignore_cephadmlib= ) >>>> +- # copy all the package directories >>>> +- for pkg_dir in pkg_dirs: >>>> +- pkg_dest =3D tempdir / pkg_dir.name >>>> +- log.info(f"Copying {pkg_dir} to {pkg_dest}") >>>> +- shutil.copytree(pkg_dir, pkg_dest, ignore=3D_ignore_cephadml= ib) >>>> ++ # RPM-based distributions use 'site-packages' for Python package= s >>>> ++ _copy_package_files(tempdir, paths, 'site-packages') >>>> ++ >>>> ++ >>>> ++def _deps_from_deb(tempdir, config, dinfo, pkg): >>>> ++ """Extract Python dependencies from Debian packages. >>>> ++ >>>> ++ Args: >>>> ++ tempdir: Temporary directory to copy package files to >>>> ++ config: Build configuration >>>> ++ dinfo: DependencyInfo instance to track dependencies >>>> ++ pkg: Python package name (e.g., 'MarkupSafe', 'Jinja2', 'PyY= AML') >>>> ++ """ >>>> ++ # Convert Python package name to Debian package name >>>> ++ # Python packages are typically named python3- >>>> ++ # Handle special cases: PyYAML -> python3-yaml, MarkupSafe -> py= thon3-markupsafe >>>> ++ pkg_lower =3D pkg.lower() >>>> ++ if pkg_lower =3D=3D 'pyyaml': >>>> ++ deb_pkg_name =3D 'python3-yaml' >>>> ++ else: >>>> ++ deb_pkg_name =3D f'python3-{pkg_lower}' >>>> ++ >>>> ++ # First, try to find the package using apt-cache >>>> ++ # This helps verify the package exists before trying to list its= files >>>> ++ try: >>>> ++ res =3D subprocess.run( >>>> ++ ['apt-cache', 'show', deb_pkg_name], >>>> ++ check=3DTrue, >>>> ++ stdout=3Dsubprocess.PIPE, >>>> ++ stderr=3Dsubprocess.PIPE, >>>> ++ ) >>>> ++ except subprocess.CalledProcessError: >>>> ++ # Package not found, try alternative naming >>>> ++ log.warning(f"Package {deb_pkg_name} not found via apt-cache= , trying dpkg -S") >>>> ++ # Try to search for files that might belong to this package >>>> ++ # Search for the Python module in site-packages >>>> ++ search_pattern =3D f'/usr/lib/python3*/dist-packages/{pkg.lo= wer()}*' >>>> ++ try: >>>> ++ res =3D subprocess.run( >>>> ++ ['dpkg', '-S', search_pattern], >>>> ++ check=3DTrue, >>>> ++ stdout=3Dsubprocess.PIPE, >>>> ++ stderr=3Dsubprocess.PIPE, >>>> ++ ) >>>> ++ # dpkg -S output format: "package: /path/to/file" >>>> ++ deb_pkg_name =3D res.stdout.decode('utf8').split(':')[0]= .strip() >>>> ++ except subprocess.CalledProcessError as err: >>>> ++ log.error(f"Could not find Debian package for {pkg}") >>>> ++ log.error(f"Tried: {deb_pkg_name} and pattern search") >>>> ++ sys.exit(1) >>>> ++ >>>> ++ # Get version information using dpkg-query >>>> ++ try: >>>> ++ res =3D subprocess.run( >>>> ++ ['dpkg-query', '-W', '-f=3D${Version}\\n', deb_pkg_name]= , >>>> ++ check=3DTrue, >>>> ++ stdout=3Dsubprocess.PIPE, >>>> ++ ) >>>> ++ version =3D res.stdout.decode('utf8').strip() >>>> ++ except subprocess.CalledProcessError as err: >>>> ++ log.error(f"Could not query version for package {deb_pkg_nam= e}: {err}") >>>> ++ sys.exit(1) >>>> ++ >>>> ++ log.info(f"Debian Package: {deb_pkg_name} (version: {version})") >>>> ++ dinfo.add( >>>> ++ pkg, >>>> ++ deb_name=3Ddeb_pkg_name, >>>> ++ version=3Dversion, >>>> ++ package_source=3D'deb', >>>> ++ ) >>>> ++ >>>> ++ # Get the list of files provided by the Debian package >>>> ++ try: >>>> ++ res =3D subprocess.run( >>>> ++ ['dpkg', '-L', deb_pkg_name], >>>> ++ check=3DTrue, >>>> ++ stdout=3Dsubprocess.PIPE, >>>> ++ ) >>>> ++ except subprocess.CalledProcessError as err: >>>> ++ log.error(f"Could not list files for package {deb_pkg_name}:= {err}") >>>> ++ sys.exit(1) >>>> ++ >>>> ++ paths =3D [l.decode('utf8') for l in res.stdout.splitlines()] >>>> ++ # Debian-based distributions use 'dist-packages' for system-mana= ged Python packages >>>> ++ # per Debian Python Policy: https://www.debian.org/doc/packaging= -manuals/python-policy/ >>>> ++ _copy_package_files(tempdir, paths, 'dist-packages') >>>> +=20 >>>> +=20 >>>> + def generate_version_file(versioning_vars, dest): >>>> +--=20 >>>> +2.47.3 >>>> + >>>> diff --git a/patches/0046-debian-Use-system-packages-for-cephadm-bundl= ed-depen.patch b/patches/0046-debian-Use-system-packages-for-cephadm-bundle= d-depen.patch >>>> new file mode 100644 >>>> index 00000000000..0b4daa63f79 >>>> --- /dev/null >>>> +++ b/patches/0046-debian-Use-system-packages-for-cephadm-bundled-depe= n.patch >>>> @@ -0,0 +1,54 @@ >>>> +From 779c9251669b65b7ed77a841f281e2d957174db2 Mon Sep 17 00:00:00 200= 1 >>>> +From: Kefu Chai >>>> +Date: Tue, 21 Oct 2025 11:25:00 +0800 >>>> +Subject: [PATCH 46/46] debian: Use system packages for cephadm bundle= d >>>> + dependencies >>>> + >>>> +Configure the Debian build to use CEPHADM_BUNDLED_DEPENDENCIES=3Ddeb, >>>> +which instructs the cephadm build script to bundle dependencies from >>>> +system-installed Debian packages instead of downloading from PyPI. >>>> + >>>> +This change addresses build failures in restricted network environmen= ts >>>> +where Debian build tools do not permit internet access. By leveraging >>>> +the Debian package support added in commit 9378a2988e1, the build now >>>> +uses python3-markupsafe, python3-jinja2, and python3-yaml packages >>>> +that are already installed as build dependencies. >>>> + >>>> +This approach mirrors the existing RPM packaging workflow, ensuring >>>> +consistent behavior across different distribution package formats. >>>> + >>>> +Signed-off-by: Kefu Chai >>>> +(cherry picked from commit 25680021ee2422f8b8b4075b3ab77af39126eecc) >>> >>> long term, it might be even better to not bundle at all, but add those >>> python deps as runtime dependencies? >>=20 >> I concur with your concerns, as bundling violates Debian's phylosophy of= =20 >> avoiding "vendored" dependencies. The distro intends to control and >> track all packages. And bunlding also implies more disk space usage and= =20 >> duplications. >>=20 >> But it's also an upstream design choice: cephadm was explicitly designed >> to be standalone. Part of cephadm'svalue is that you can just curl | >> python3 it onto a fresh system. If it requires installing 10+ Python >> packages first, you've lost that simplicity. >>=20 >> What do you think? > > I understand that reasoning, and cephadm isn't something we are using > anyway (atm).. > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel