Re: [RFC] letting Ceph read its own config as the ceph user

[Kefu Chai <k.chai@proxmox.com>]

Hi all,

Maximiliano kindly went through the ceph.conf part with me off-list,
and I'd like to change a few things from the RFC:

- I'd keep the /etc/ceph/ceph.conf symlink after all. Otherwise edits
  made through it would stay on one node and get overwritten, and PVE
  itself relies on the symlink in a few places (setup_pve_symlinks(),
  check_ceph_enabled()).

- The copy would go to a separate path instead, as root:ceph 0440. I
  think its directory should be owned by root, since the mon and OSDs
  read the config before dropping privileges. That rules out
  /var/lib/ceph and /etc/ceph, which are both owned by ceph. pmxcfs's
  own /var/lib/pve-cluster would work, so I'll use
  /var/lib/pve-cluster/ceph.conf below, but better ideas are welcome.

- The Ceph services could find the copy through a single drop-in in
  ceph-.service.d/:

    Environment=CEPH_CONF=/var/lib/pve-cluster/ceph.conf,/etc/ceph/ceph.conf

  If the copy is missing, Ceph falls back to the symlink. I tried this
  with one mgr: after "ceph mgr fail" it came back as ceph and found
  the copy, without systemd having to restart it.

- Sorry, the systemd path unit I suggested as a fallback doesn't work.
  It only notices changes made on the local node, not the ones coming
  from other nodes.

I haven't figured out yet how to make a failed copy visible. Maybe a
log message, or a health check.

The ceph-crash error at boot had two causes in the end, and I sent a
fix for each:

- ceph-crash wasn't ordered after pve-cluster. The pve-manager patch
  [1] replaces the per-unit drop-ins with one for all ceph-* units, so
  it now covers ceph-crash and a few other units we had missed.

- Its startup ping ran "ceph -s" as client.admin, whose key ceph-crash
  can't read. With the Ceph PR [2], it uses the same keys it posts
  crashes with.

The rest of the RFC stays the same. Thanks again, Maximiliano, for
your help!

Cheers,
Kefu

[1] https://lore.proxmox.com/pve-devel/20261014104321.2652781-1-k.chai@proxmox.com/T/#u
[2] https://github.com/ceph/ceph/pull/72674