Re: [PVE-User] Caution: ceph-mon service does not start after today's updates

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

Some news.

There are a few things at play, it boils down to two things:
* a update of various service orderings in ceph with 14.2.12 (released a bit
  ago), they introduced pretty much everywhere a `Before=remote-fs-pre.target`
  order enforcement.

* rrdcached, a service used by pve-cluster.service (pmxcfs), this has no native
  systemd service file, so systemd auto generates one, with an `Before=remote-pre.target`
  order enforcement which then has ordering for the aforementioned
  `Before=remote-fs-pre.target`


Thus you get the cycle (-> means an after odering, all befores where transformed
to after by reversing them (systemd does that too)):


.> pve-cluster -> rrdcached -> remote-pre -> remote-fs-pre -> ceph-mgr@ -.
|                                                                        |
`------------------------------------------------------------------------'

We're building a new ceph version with the Before=remote-fs-pre.target removed,
it is bogus for the ceph mgr, mds, mon, .. services as is.

As you probably guessed, one can also fix this by adapting rrdcached, and as
a work around you can do so:

1. copy over the generated ephemeral service file from /run to /etc, which
   has higher priority.

# cp /run/systemd/generator.late/rrdcached.service /etc/systemd/system/

2. Drop the after ordering for remote-fs.target
# sed -i '/^After=remote-fs.target/d' /etc/systemd/system/rrdcached.service

3. reboot 

A ceph 14.2.15-pve2 package will soon be available, we'll also see if we can
improve the rrdcached situation in the future, it has no fault on its own
naturally, the systemd auto generators heuristic is to blame, but maybe we
can see if upstream or Debian has interest in adding an hand crafted systemd
unit file, avoiding auto-generation. Otionally we could maintain it for PVE,
or do like in Proxmox Backup Server - use our own rust based RRD implementation

regards,
Thomas