[RFC] letting Ceph read its own config as the ceph user

[RFC] letting Ceph read its own config as the ceph user

[Kefu Chai]

Hi all,

Maximiliano pointed out to me that the way we store Ceph's config and
keys has been a pain in PVE for a long time. I'm still fairly new to
this part of the code, so before writing any patches I'd like to ask
for your thoughts. After reading through the bugs he mentioned, they
seem to me to come down to the same problem, but I may well be missing
some history here.

How things are today
--------------------

Ceph daemons start as root and switch to ceph:ceph right away. We keep
ceph.conf and the keyrings on pmxcfs, which only knows about root and
www-data: /etc/pve/ceph.conf is root:www-data 0640, /etc/pve/priv is
root only, and /etc/ceph/ceph.conf is just a symlink to
/etc/pve/ceph.conf.

That's fine as long as Ceph only reads these files while it's still
root. But Ceph does more and more as ceph, and from what I can see,
every time it does, something breaks and we add another workaround:

- #4759: once ceph-crash started dropping privileges, it couldn't
  authenticate anymore. We fixed it by running it with group www-data
  and giving it its own keyring in /etc/pve/ceph [1][2].

- #7011: "ceph-mon --mkfs" ran as root and created the monitor's log
  file as root, so the monitor couldn't write to it once it started as
  ceph. Now we chown the log after creating the monitor. The comment
  there says we can't create it as ceph "without losing access to
  the mon keyring inside pmxcfs" [3][4].

- #5705: when ceph-mgr loses the active role, it re-execs itself. By
  then it's already running as ceph, so it can't read ceph.conf and
  just exits ("can't open ceph.conf"), and systemd has to restart it.
  I could reproduce this on my test cluster with a single "ceph mgr
  fail". Friedrich found that a few failovers in a row trip systemd's
  start limit, and then you end up with no mgr at all [5]. The original
  report is about the dashboard's RGW pages, which fail for the same
  reason, since radosgw-admin runs as ceph too. The bug has been open
  since 2024, and at least one user runs their own path unit that
  copies ceph.conf somewhere ceph can read it.

- #6816: ceph-exporter can't read the admin keyring and goes into a
  crash loop. Max Carrara's v4 fixes it the same way we fixed
  ceph-crash: its own keyring, and group www-data [6][7].

- ceph-crash logs this on every boot:

    Error initializing cluster client: ObjectNotFound('RADOS object
    not found (error calling conf_read_file)')

  I looked into it on a 9.2 test cluster and found two causes. The unit
  isn't ordered after pve-cluster, so it starts before /etc/pve is
  mounted. And its startup "ceph -s" uses client.admin, whose keyring in
  priv it can't read even once /etc/pve is up. Nothing actually breaks,
  since the crash posting itself uses client.crash, but it's the same
  problem again.

- When we create a MON, MDS or MGR, we run things as root and chown the
  result afterwards. If a new Ceph release creates one more file there,
  we get a daemon that can't read it.

A rough idea
------------

What I have in mind is that whatever Ceph reads as ceph should be
readable by ceph from the start. Then we could drop the www-data group
tricks and stop fixing owners afterwards. Here is how I think we could
get there, but please take it as a starting point for discussion
rather than a finished plan.

1. A local ceph.conf that ceph can read

   This is Fabian's idea from #5705 [8]: treat ceph.conf like
   corosync.conf. The master copy stays in /etc/pve, and every node
   keeps a local copy that gets updated whenever the master changes.
   pmxcfs already does this for corosync.conf
   (dcdb_sync_corosync_conf() in dcdb.c), so roughly:

   - /etc/pve/ceph.conf stays the master copy, so the API, the GUI and
     everything else that writes it keep working as they do now.
   - pmxcfs writes /etc/ceph/ceph.conf as root:ceph 0640 when it starts
     and whenever the file changes. It looks up the ceph group by name
     and skips the write if Ceph isn't installed.
   - A postinst step replaces the /etc/ceph/ceph.conf symlink with that
     file.

   pmxcfs ownership would stay root and www-data only. If I'm not wrong,
   this would fix #5705 for all the daemons, the dashboard and
   ceph-crash's config read. A daemon that starts before pmxcfs would
   also find the last known config instead of an empty /etc/pve.

   You know pmxcfs much better than I do, so if this doesn't belong
   there, pve-manager could instead ship a systemd path unit that does
   the copy, like the one a user posted in #5705. That's simpler, but
   the copy can be stale for a moment after each change.

2. Create monitors as ceph

   Copy the mon keyring to a temporary file owned by ceph, run "ceph-mon
   --mkfs --setuser ceph --setgroup ceph", then remove the copy. That
   would let us drop the chown calls after mkfs in MON.pm, including the
   one from #7011. I only tried it on my small test cluster, but there
   nothing was left owned by root. Upstream seems to do the same: the
   manual deployment guide runs mkfs as ceph, and cephadm runs it with
   --setuser ceph.

3. Per-service keys that ceph can read

   ceph-crash and ceph-exporter would each get their own key with
   minimal caps, stored somewhere ceph can read it. Then ceph-crash no
   longer needs the www-data group. The admin and mon keys stay in
   priv, root only.

4. While we're at it, we could write the MDS and MGR keyrings with the
   right owner in the first place, instead of running chown -R
   afterwards.

Steps 2 and 4 only touch pve-manager, so they could go in first if you
think they make sense. Step 1 needs pve-cluster. For step 3 I'd build
on Max Carrara's #6816 series rather than send something competing
with it.

A few questions for you
-----------------------

1. Would you be OK with pmxcfs syncing ceph.conf the way it syncs
   corosync.conf, or would you prefer this lived in pve-manager?

2. For per-service keys like client.crash: should we sync them to a
   local file owned by ceph too, or keep the www-data group approach?

3. Any concerns about /etc/ceph/ceph.conf turning from a symlink into a
   regular file? Scripts that follow the link would see the copy
   instead.

I'm sure there are things I haven't thought of, so any corrections or
better ideas are very welcome.

Thanks,
Kefu

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=4759
[2] https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=11edd5d88dda057252ed0da3530d192580342874
[3] https://bugzilla.proxmox.com/show_bug.cgi?id=7011
[4] https://git.proxmox.com/?p=pve-manager.git;a=commitdiff;h=3c231167128366458688c347d992f1e403d35e8e
[5] https://bugzilla.proxmox.com/show_bug.cgi?id=5705#c9
[6] https://bugzilla.proxmox.com/show_bug.cgi?id=6816
[7] https://lore.proxmox.com/pve-devel/20260623134454.710441-1-m.carrara@proxmox.com/T/#t
[8] https://bugzilla.proxmox.com/show_bug.cgi?id=5705#c2

Re: [RFC] letting Ceph read its own config as the ceph user

[Kefu Chai]

Hi all,

Maximiliano kindly went through the ceph.conf part with me off-list,
and I'd like to change a few things from the RFC:

- I'd keep the /etc/ceph/ceph.conf symlink after all. Otherwise edits
  made through it would stay on one node and get overwritten, and PVE
  itself relies on the symlink in a few places (setup_pve_symlinks(),
  check_ceph_enabled()).

- The copy would go to a separate path instead, as root:ceph 0440. I
  think its directory should be owned by root, since the mon and OSDs
  read the config before dropping privileges. That rules out
  /var/lib/ceph and /etc/ceph, which are both owned by ceph. pmxcfs's
  own /var/lib/pve-cluster would work, so I'll use
  /var/lib/pve-cluster/ceph.conf below, but better ideas are welcome.

- The Ceph services could find the copy through a single drop-in in
  ceph-.service.d/:

    Environment=CEPH_CONF=/var/lib/pve-cluster/ceph.conf,/etc/ceph/ceph.conf

  If the copy is missing, Ceph falls back to the symlink. I tried this
  with one mgr: after "ceph mgr fail" it came back as ceph and found
  the copy, without systemd having to restart it.

- Sorry, the systemd path unit I suggested as a fallback doesn't work.
  It only notices changes made on the local node, not the ones coming
  from other nodes.

I haven't figured out yet how to make a failed copy visible. Maybe a
log message, or a health check.

The ceph-crash error at boot had two causes in the end, and I sent a
fix for each:

- ceph-crash wasn't ordered after pve-cluster. The pve-manager patch
  [1] replaces the per-unit drop-ins with one for all ceph-* units, so
  it now covers ceph-crash and a few other units we had missed.

- Its startup ping ran "ceph -s" as client.admin, whose key ceph-crash
  can't read. With the Ceph PR [2], it uses the same keys it posts
  crashes with.

The rest of the RFC stays the same. Thanks again, Maximiliano, for
your help!

Cheers,
Kefu

[1] https://lore.proxmox.com/pve-devel/20261014104321.2652781-1-k.chai@proxmox.com/T/#u
[2] https://github.com/ceph/ceph/pull/72674