[pve-devel] [PATCH v1 master ceph 0/2] Fix Ceph Squid Module Loading

[pve-devel] [PATCH v1 master ceph 0/2] Fix Ceph Squid Module Loading

[Max R. Carrara]

Fix Ceph Squid Module Loading - v1
==================================

The first patch provides a workaround for the PyO3 ImportError that's
being thrown by the `restful` Ceph mgr Python module upon import.
This fixes the following error:

    2025-07-16T17:40:40.932+0200 7c315bc77080  1 mgr[py] Loading python module 'restful'
    2025-07-16T17:40:41.220+0200 7c315bc77080 -1 mgr[py] Module not found: 'restful'
    2025-07-16T17:40:41.220+0200 7c315bc77080 -1 mgr[py] Traceback (most recent call last):
      File "/usr/share/ceph/mgr/restful/__init__.py", line 1, in <module>
        from .module import Module
      File "/usr/share/ceph/mgr/restful/module.py", line 22, in <module>
        from OpenSSL import crypto
      File "/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
        from OpenSSL import SSL, crypto
      File "/lib/python3/dist-packages/OpenSSL/SSL.py", line 15, in <module>
        from cryptography import x509
      File "/lib/python3/dist-packages/cryptography/x509/__init__.py", line 7, in <module>
        from cryptography.x509 import certificate_transparency, verification
      File "/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py", line 11, in <module>
        from cryptography.hazmat.bindings._rust import x509 as rust_x509
    ImportError: PyO3 modules compiled for CPython 3.8 or older may only be initialized once per interpreter process

    2025-07-16T17:40:41.222+0200 7c315bc77080 -1 mgr[py] Class not found in module 'restful'
    2025-07-16T17:40:41.222+0200 7c315bc77080 -1 mgr[py] Error loading module 'restful': (2) No such file or directory


The second patch fixes how the Ceph mgr looks up the NOTIFY_TYPES
attribute of the mgr Python modules' classes. This also happens to fix
the `mgr_module` Python module import failure. In total, this fixes
errors like the following:

    2025-07-16T17:40:41.222+0200 7c315bc77080  1 mgr[py] Loading python module 'selftest'
    2025-07-16T17:40:41.377+0200 7c315bc77080 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member
    2025-07-16T17:40:41.377+0200 7c315bc77080 -1 mgr[py] Module not found: 'mgr_module'
    2025-07-16T17:40:41.377+0200 7c315bc77080 -1 mgr[py] AttributeError: type object 'Module' has no attribute 'NOTIFY_TYPES'
    
    2025-07-16T17:40:41.377+0200 7c315bc77080  1 mgr[py] Loading python module 'snap_schedule'
    2025-07-16T17:40:41.497+0200 7c315bc77080 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member
    2025-07-16T17:40:41.497+0200 7c315bc77080 -1 mgr[py] Module not found: 'mgr_module'
    2025-07-16T17:40:41.497+0200 7c315bc77080 -1 mgr[py] AttributeError: type object 'Module' has no attribute 'NOTIFY_TYPES'
    
    2025-07-16T17:40:41.497+0200 7c315bc77080  1 mgr[py] Loading python module 'stats'
    2025-07-16T17:40:41.636+0200 7c315bc77080  1 mgr[py] Loading python module 'status'
    2025-07-16T17:40:41.785+0200 7c315bc77080 -1 mgr[py] Module status has missing NOTIFY_TYPES member
    2025-07-16T17:40:41.785+0200 7c315bc77080 -1 mgr[py] Module not found: 'mgr_module'
    2025-07-16T17:40:41.785+0200 7c315bc77080 -1 mgr[py] AttributeError: type object 'Module' has no attribute 'NOTIFY_TYPES'

Note: These logs can be found inside the ceph-mgr logs in the
/var/log/ceph directory.

The commit messages of the supplied patches elaborate on further
details.

Summary of Changes
------------------

Max R. Carrara (2):
  provide workaround for PyO3 ImportError
  mgr: fix errors regarding module imports and NOTIFY_TYPES attributes

 ...ul-provide-workaround-for-PyO3-Impor.patch | 152 ++++++++++++++++++
 ...mport-by-making-NOTIFY_TYPES-in-py-m.patch |  56 +++++++
 patches/series                                |   2 +
 3 files changed, 210 insertions(+)
 create mode 100644 patches/0058-pybind-mgr-restful-provide-workaround-for-PyO3-Impor.patch
 create mode 100644 patches/0059-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch

-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v1 master ceph 1/2] provide workaround for PyO3 ImportError

[Max R. Carrara]

By moving the self-signed cert generation into a separate module and
consequently running it via `subprocess.run`--so, as a new,
independent Python process--the PyO3 ImportError is successfully
avoided.

Inspired by an upstream PR [0].

[0]: https://github.com/ceph/ceph/pull/62951

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 ...ul-provide-workaround-for-PyO3-Impor.patch | 152 ++++++++++++++++++
 patches/series                                |   1 +
 2 files changed, 153 insertions(+)
 create mode 100644 patches/0058-pybind-mgr-restful-provide-workaround-for-PyO3-Impor.patch

diff --git a/patches/0058-pybind-mgr-restful-provide-workaround-for-PyO3-Impor.patch b/patches/0058-pybind-mgr-restful-provide-workaround-for-PyO3-Impor.patch
new file mode 100644
index 0000000000..25fb720410
--- /dev/null
+++ b/patches/0058-pybind-mgr-restful-provide-workaround-for-PyO3-Impor.patch
@@ -0,0 +1,152 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: "Max R. Carrara" <m.carrara@proxmox.com>
+Date: Wed, 16 Jul 2025 13:14:39 +0200
+Subject: [PATCH 58/59] pybind/mgr/restful: provide workaround for PyO3
+ ImportError
+
+Move the self-signed cert generation into a separate module
+inside python-common/ceph and run the module in a separate Python
+process.
+
+This provides a workaround for the ImportError thrown by PyO3 when
+the `restful` module is loaded in the context of multiple Python
+sub-interpreters being present. In particular, the ImportError is
+thrown by the `crypto` module of the `OpenSSL` package.
+
+Inspired by an upstream PR [0].
+
+[0]: https://github.com/ceph/ceph/pull/62951
+
+Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
+---
+ src/pybind/mgr/restful/module.py       | 24 +++------
+ src/python-common/ceph/_crypto_wrap.py | 69 ++++++++++++++++++++++++++
+ 2 files changed, 76 insertions(+), 17 deletions(-)
+ create mode 100644 src/python-common/ceph/_crypto_wrap.py
+
+diff --git a/src/pybind/mgr/restful/module.py b/src/pybind/mgr/restful/module.py
+index 0f8c78e0bd8..7f93c41f1e6 100644
+--- a/src/pybind/mgr/restful/module.py
++++ b/src/pybind/mgr/restful/module.py
+@@ -7,6 +7,7 @@ import json
+ import time
+ import errno
+ import inspect
++import subprocess
+ import tempfile
+ import threading
+ import traceback
+@@ -19,7 +20,6 @@ from . import context
+ 
+ from uuid import uuid4
+ from pecan import jsonify, make_app
+-from OpenSSL import crypto
+ from pecan.rest import RestController
+ from werkzeug.serving import make_server, make_ssl_devcert
+ 
+@@ -401,24 +401,14 @@ class Module(MgrModule):
+ 
+ 
+     def create_self_signed_cert(self):
+-        # create a key pair
+-        pkey = crypto.PKey()
+-        pkey.generate_key(crypto.TYPE_RSA, 2048)
+-
+-        # create a self-signed cert
+-        cert = crypto.X509()
+-        cert.get_subject().O = "IT"
+-        cert.get_subject().CN = "ceph-restful"
+-        cert.set_serial_number(int(uuid4()))
+-        cert.gmtime_adj_notBefore(0)
+-        cert.gmtime_adj_notAfter(10*365*24*60*60)
+-        cert.set_issuer(cert.get_subject())
+-        cert.set_pubkey(pkey)
+-        cert.sign(pkey, 'sha512')
++        cmd = ["python3", "-m", "ceph._crypto_wrap", "create_self_signed_cert"]
++
++        response = subprocess.run(cmd, capture_output=True, check=True)
++        response_obj = json.loads(response.stdout)
+ 
+         return (
+-            crypto.dump_certificate(crypto.FILETYPE_PEM, cert),
+-            crypto.dump_privatekey(crypto.FILETYPE_PEM, pkey)
++            response_obj["cert"].encode("utf-8"),
++            response_obj["key"].encode("utf-8"),
+         )
+ 
+ 
+diff --git a/src/python-common/ceph/_crypto_wrap.py b/src/python-common/ceph/_crypto_wrap.py
+new file mode 100644
+index 00000000000..16a19a5345e
+--- /dev/null
++++ b/src/python-common/ceph/_crypto_wrap.py
+@@ -0,0 +1,69 @@
++"""CLI wrapper for cryptographic functions of the :mod:`restful` module.
++
++To be called via :func:`subprocess.run()` as a workaround for
++:class:`ImportError`s related to PyO3's current lack of sub-interpreter
++support.
++
++Note:
++    Since this module is installed as part of the ``ceph`` package,
++    it should be called like so::
++
++        python3 -m ceph._crypto_wrap create_self_signed_cert
++"""
++
++import argparse
++import sys
++import json
++
++from argparse import Namespace
++from typing import Any
++from uuid import uuid4
++
++from OpenSSL import crypto
++
++
++def _respond(data: dict[str, Any]) -> None:
++    json.dump(data, sys.stdout)
++    sys.stdout.flush()
++
++
++def create_self_signed_cert(args: Namespace) -> None:
++    cert_key_pair = _create_self_signed_cert()
++    _respond(cert_key_pair)
++
++
++def _create_self_signed_cert() -> dict[str, str]:
++    # create a key pair
++    pubkey = crypto.PKey()
++    pubkey.generate_key(crypto.TYPE_RSA, 2048)
++
++    # create a self-signed cert
++    cert = crypto.X509()
++    cert.get_subject().O = "IT"
++    cert.get_subject().CN = "ceph-restful"
++    cert.set_serial_number(int(uuid4()))
++    cert.gmtime_adj_notBefore(0)
++    cert.gmtime_adj_notAfter(10 * 365 * 24 * 60 * 60)
++    cert.set_issuer(cert.get_subject())
++    cert.set_pubkey(pubkey)
++    cert.sign(pubkey, "sha512")
++
++    return {
++        "cert": crypto.dump_certificate(crypto.FILETYPE_PEM, cert).decode(),
++        "key": crypto.dump_privatekey(crypto.FILETYPE_PEM, pubkey).decode(),
++    }
++
++
++def main() -> None:
++    parser = argparse.ArgumentParser(prog="_crypto_wrap.py")
++    subparsers = parser.add_subparsers(required=True)
++
++    parser_cssc = subparsers.add_parser("create_self_signed_cert")
++    parser_cssc.set_defaults(func=create_self_signed_cert)
++
++    args = parser.parse_args()
++    args.func(args)
++
++
++if __name__ == "__main__":
++    main()
diff --git a/patches/series b/patches/series
index 728a9f935e..ff23f8b640 100644
--- a/patches/series
+++ b/patches/series
@@ -51,3 +51,4 @@
 0055-python-common-cryptotools-catch-all-failures-to-read.patch
 0056-mgr-cephadm-always-use-the-internal-cryptocaller.patch
 0057-mgr-dashboard-add-an-option-to-control-the-dashboard.patch
+0058-pybind-mgr-restful-provide-workaround-for-PyO3-Impor.patch
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH v1 master ceph 2/2] mgr: fix errors regarding module imports and NOTIFY_TYPES attributes

[Max R. Carrara]

By handling the AttributeError that is thrown internally, the import
of several mgr modules also works again.

See the patch's message for all details.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 ...mport-by-making-NOTIFY_TYPES-in-py-m.patch | 56 +++++++++++++++++++
 patches/series                                |  1 +
 2 files changed, 57 insertions(+)
 create mode 100644 patches/0059-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch

diff --git a/patches/0059-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch b/patches/0059-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch
new file mode 100644
index 0000000000..5595c96c9e
--- /dev/null
+++ b/patches/0059-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch
@@ -0,0 +1,56 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: "Max R. Carrara" <m.carrara@proxmox.com>
+Date: Wed, 16 Jul 2025 16:31:43 +0200
+Subject: [PATCH 59/59] mgr: fix module import by making NOTIFY_TYPES in py
+ modules optional
+
+If NOTIFY_TYPES isn't an attribute of the passed class, the Python
+(sub-)interpreter raises an AttributeError that must be handled or cleared
+explicitly via the Python C-API. Unfortunately, this isn't done here,
+which means that the exception sticks around until handled.
+
+This caused a call to PyModule::load_subclass_of() to fail and
+incorrectly report the AttributeError as cause.
+
+Checking whether the class has NOTIFY_TYPES as attribute in the first
+place fixes this.
+
+Note that there's an upstream PR [0] that wasn't backported that aimed
+to fix this, but does so incorrectly, as the exception is still not
+cleared there. The warnings regarding NOTIFY_TYPES missing also occurs
+on Reef but doesn't cause any module imports to fail there. As the
+affected Ceph code has stayed mostly the same between bookworm and
+trixie releases, this suggests that some behavior between Python 3.11
+and 3.13 likely changed.
+
+Either way, avoiding the AttributeError altogether fixes this.
+
+[0]: https://github.com/ceph/ceph/pull/57106
+
+Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
+---
+ src/mgr/PyModule.cc | 10 ++++++----
+ 1 file changed, 6 insertions(+), 4 deletions(-)
+
+diff --git a/src/mgr/PyModule.cc b/src/mgr/PyModule.cc
+index 084cf3ffc1e..e6fd269dca5 100644
+--- a/src/mgr/PyModule.cc
++++ b/src/mgr/PyModule.cc
+@@ -513,11 +513,13 @@ int PyModule::register_options(PyObject *cls)
+ 
+ int PyModule::load_notify_types()
+ {
+-  PyObject *ls = PyObject_GetAttrString(pClass, "NOTIFY_TYPES");
+-  if (ls == nullptr) {
+-    derr << "Module " << get_name() << " has missing NOTIFY_TYPES member" << dendl;
+-    return -EINVAL;
++  if (!PyObject_HasAttrString(pClass, "NOTIFY_TYPES")) {
++    dout(10) << "Module " << get_name() << " has no NOTIFY_TYPES member" << dendl;
++    return 0;
+   }
++
++  PyObject *ls = PyObject_GetAttrString(pClass, "NOTIFY_TYPES");
++
+   if (!PyObject_TypeCheck(ls, &PyList_Type)) {
+     // Relatively easy mistake for human to make, e.g. defining COMMANDS
+     // as a {} instead of a []
diff --git a/patches/series b/patches/series
index ff23f8b640..b820614566 100644
--- a/patches/series
+++ b/patches/series
@@ -52,3 +52,4 @@
 0056-mgr-cephadm-always-use-the-internal-cryptocaller.patch
 0057-mgr-dashboard-add-an-option-to-control-the-dashboard.patch
 0058-pybind-mgr-restful-provide-workaround-for-PyO3-Impor.patch
+0059-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied-series: [PATCH v1 master ceph 0/2] Fix Ceph Squid Module Loading

[Thomas Lamprecht]

On Wed, 16 Jul 2025 19:39:54 +0200, Max R. Carrara wrote:
> Fix Ceph Squid Module Loading - v1
> ==================================
> 
> The first patch provides a workaround for the PyO3 ImportError that's
> being thrown by the `restful` Ceph mgr Python module upon import.
> This fixes the following error:
> 
> [...]

Patches seem relatively straight forward and I saw no new crash reports about
mgr modules in my test cluster, so:

Applied, thanks!

[1/2] provide workaround for PyO3 ImportError
      commit: 51c4e16ffa4d46944b03b376e14aa92c7ca758ab
[2/2] mgr: fix errors regarding module imports and NOTIFY_TYPES attributes
      commit: 099772ef89be90332860fc9606b70ee5ad6b6d38


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel