From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id F1BF51FF0AE for ; Tue, 13 Oct 2026 11:25:28 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id C602E21376; Tue, 13 Oct 2026 11:25:26 +0200 (CEST) From: Kefu Chai To: pve-devel@lists.proxmox.com Subject: [RFC] letting Ceph read its own config as the ceph user Date: Tue, 13 Oct 2026 17:25:16 +0800 Message-ID: <20261013092516.1596661-1-k.chai@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1791883520732 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.386 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: NQOV5YKINI74JFQPSIZLTDLFJMX742XX X-Message-ID-Hash: NQOV5YKINI74JFQPSIZLTDLFJMX742XX X-MailFrom: k.chai@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi all, Maximiliano pointed out to me that the way we store Ceph's config and keys has been a pain in PVE for a long time. I'm still fairly new to this part of the code, so before writing any patches I'd like to ask for your thoughts. After reading through the bugs he mentioned, they seem to me to come down to the same problem, but I may well be missing some history here. How things are today -------------------- Ceph daemons start as root and switch to ceph:ceph right away. We keep ceph.conf and the keyrings on pmxcfs, which only knows about root and www-data: /etc/pve/ceph.conf is root:www-data 0640, /etc/pve/priv is root only, and /etc/ceph/ceph.conf is just a symlink to /etc/pve/ceph.conf. That's fine as long as Ceph only reads these files while it's still root. But Ceph does more and more as ceph, and from what I can see, every time it does, something breaks and we add another workaround: - #4759: once ceph-crash started dropping privileges, it couldn't authenticate anymore. We fixed it by running it with group www-data and giving it its own keyring in /etc/pve/ceph [1][2]. - #7011: "ceph-mon --mkfs" ran as root and created the monitor's log file as root, so the monitor couldn't write to it once it started as ceph. Now we chown the log after creating the monitor. The comment there says we can't create it as ceph "without losing access to the mon keyring inside pmxcfs" [3][4]. - #5705: when ceph-mgr loses the active role, it re-execs itself. By then it's already running as ceph, so it can't read ceph.conf and just exits ("can't open ceph.conf"), and systemd has to restart it. I could reproduce this on my test cluster with a single "ceph mgr fail". Friedrich found that a few failovers in a row trip systemd's start limit, and then you end up with no mgr at all [5]. The original report is about the dashboard's RGW pages, which fail for the same reason, since radosgw-admin runs as ceph too. The bug has been open since 2024, and at least one user runs their own path unit that copies ceph.conf somewhere ceph can read it. - #6816: ceph-exporter can't read the admin keyring and goes into a crash loop. Max Carrara's v4 fixes it the same way we fixed ceph-crash: its own keyring, and group www-data [6][7]. - ceph-crash logs this on every boot: Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)') I looked into it on a 9.2 test cluster and found two causes. The unit isn't ordered after pve-cluster, so it starts before /etc/pve is mounted. And its startup "ceph -s" uses client.admin, whose keyring in priv it can't read even once /etc/pve is up. Nothing actually breaks, since the crash posting itself uses client.crash, but it's the same problem again. - When we create a MON, MDS or MGR, we run things as root and chown the result afterwards. If a new Ceph release creates one more file there, we get a daemon that can't read it. A rough idea ------------ What I have in mind is that whatever Ceph reads as ceph should be readable by ceph from the start. Then we could drop the www-data group tricks and stop fixing owners afterwards. Here is how I think we could get there, but please take it as a starting point for discussion rather than a finished plan. 1. A local ceph.conf that ceph can read This is Fabian's idea from #5705 [8]: treat ceph.conf like corosync.conf. The master copy stays in /etc/pve, and every node keeps a local copy that gets updated whenever the master changes. pmxcfs already does this for corosync.conf (dcdb_sync_corosync_conf() in dcdb.c), so roughly: - /etc/pve/ceph.conf stays the master copy, so the API, the GUI and everything else that writes it keep working as they do now. - pmxcfs writes /etc/ceph/ceph.conf as root:ceph 0640 when it starts and whenever the file changes. It looks up the ceph group by name and skips the write if Ceph isn't installed. - A postinst step replaces the /etc/ceph/ceph.conf symlink with that file. pmxcfs ownership would stay root and www-data only. If I'm not wrong, this would fix #5705 for all the daemons, the dashboard and ceph-crash's config read. A daemon that starts before pmxcfs would also find the last known config instead of an empty /etc/pve. You know pmxcfs much better than I do, so if this doesn't belong there, pve-manager could instead ship a systemd path unit that does the copy, like the one a user posted in #5705. That's simpler, but the copy can be stale for a moment after each change. 2. Create monitors as ceph Copy the mon keyring to a temporary file owned by ceph, run "ceph-mon --mkfs --setuser ceph --setgroup ceph", then remove the copy. That would let us drop the chown calls after mkfs in MON.pm, including the one from #7011. I only tried it on my small test cluster, but there nothing was left owned by root. Upstream seems to do the same: the manual deployment guide runs mkfs as ceph, and cephadm runs it with --setuser ceph. 3. Per-service keys that ceph can read ceph-crash and ceph-exporter would each get their own key with minimal caps, stored somewhere ceph can read it. Then ceph-crash no longer needs the www-data group. The admin and mon keys stay in priv, root only. 4. While we're at it, we could write the MDS and MGR keyrings with the right owner in the first place, instead of running chown -R afterwards. Steps 2 and 4 only touch pve-manager, so they could go in first if you think they make sense. Step 1 needs pve-cluster. For step 3 I'd build on Max Carrara's #6816 series rather than send something competing with it. A few questions for you ----------------------- 1. Would you be OK with pmxcfs syncing ceph.conf the way it syncs corosync.conf, or would you prefer this lived in pve-manager? 2. For per-service keys like client.crash: should we sync them to a local file owned by ceph too, or keep the www-data group approach? 3. Any concerns about /etc/ceph/ceph.conf turning from a symlink into a regular file? Scripts that follow the link would see the copy instead. I'm sure there are things I haven't thought of, so any corrections or better ideas are very welcome. Thanks, Kefu [1] https://bugzilla.proxmox.com/show_bug.cgi?id=4759 [2] https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=11edd5d88dda057252ed0da3530d192580342874 [3] https://bugzilla.proxmox.com/show_bug.cgi?id=7011 [4] https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=3c231167128366458688c347d992f1e403d35e8e [5] https://bugzilla.proxmox.com/show_bug.cgi?id=5705#c9 [6] https://bugzilla.proxmox.com/show_bug.cgi?id=6816 [7] https://lore.proxmox.com/pve-devel/20260623134454.710441-1-m.carrara@proxmox.com/T/#t [8] https://bugzilla.proxmox.com/show_bug.cgi?id=5705#c2