Re: [RFC cluster/manager 00/10] pmxcfs: add a change notification socket

[Hannes Laimer <h.laimer@proxmox.com>]

On 2026-09-25 14:43, Wolfgang Bumiller wrote:
> On Fri, Sep 25, 2026 at 02:23:53PM +0200, Hannes Laimer wrote:
>> On 2026-09-25 14:01, Wolfgang Bumiller wrote:
>>> On Fri, Sep 18, 2026 at 04:41:42PM +0200, Hannes Laimer wrote:
>>>> Every daemon that cares about /etc/pve learns about changes by calling
>>>> cfs_update on each loop iteration and comparing the version vector it
>>>> gets over the libqb IPC. inotify cannot replace that, since remote
>>>
>>> Not strictly true - pmxcfs could add inotify support vis fuse, but it is
>>> limited to regular files and would not support whole directory watches.
>>>
>>>> changes arrive through corosync and never touch the local VFS, so
>>>> pvestatd, pve-firewall, the HA daemons and pvescheduler all wake up on
>>>> timers and re-read what usually has not changed.
>>>>
>>>> This series adds a push path. pmxcfs gets a Unix stream socket at
>>>> /run/pve-cluster/pmxcfs.sock, served by a Rust thread linked into the C
>>>> daemon as a static library with a small C ABI. A client subscribes with
>>>> named path templates such as nodes/{node}/qemu-server/{vmid}.conf and
>>>> receives one JSON line per matching mutation, carrying the memdb version
>>>> as sequence number, the event type, the path and the values the
>>>> placeholders captured. Connections authorized by group membership never
>>>> see private paths, as on the IPC and FUSE side.
>>>>
>>>> The daemon keeps the last mutations in a fixed size ring and each
>>>> connection a cursor into it, so the mutating thread only appends and
>>>> never waits for a client. A slow client is caught up from the ring, one
>>>> that fell off it gets a single resync event, and a reconnecting client
>>>> resumes at the last sequence number it saw, also across a restart of
>>>> pmxcfs when nothing changed meanwhile. A node that takes the whole
>>>> state from the cluster after a membership change hands its clients a
>>>> resync event, since no sequence of mutations describes that. A panic in
>>>> the Rust code only disables the notifier for the rest of the daemon's
>>>> life. A sequence number identifies a state within one line of history
>>>> only. A client that missed a resync while disconnected and returns at
>>>> the same version after a pmxcfs restart resumes as if nothing changed.
>>>> Carrying the root entry's mtime next to the version would close that.
>>>>
>>>> On top of the socket, pve-cluster ships a Perl client with reconnect
>>>> and resync handling and a hook registry, where a hook is registered
>>>> like an API method with a path template whose placeholders become the
>>>> parameters of a run. pve-manager gets the runner that executes those
>>>> runs in children of a listener hosted by pvescheduler, one run per hook
>>>> and parameter set in flight and later events for the same set collapsed
>>>> into a single rerun, so a config update through the API costs one extra
>>>> run rather than one per save. The listener records the position it has
>>>> processed up to under /run and resumes there after a reload. No hook
>>>> ships in this series. A consumer registers one as the example below
>>>> shows.
>>>>
>>>> pve-manager depends on the pve-cluster packages of this series for the
>>>> new modules and the socket, at build time as well, since its tests run
>>>> through the check target.
>>>>
>>>> Example usage:
>>>>
>>>>     package PVE::Network::Hooks;
>>>>
>>>>     use base qw(PVE::Cluster::Hooks);
>>>>
>>>>     __PACKAGE__->register_hook({
>>>>         name => 'guest-firewall',
>>>
>>> ^ This particular example would have to run in the pve-firewall daemon,
>>> not pvescheduler, though.
>>> What about the remaining use cases?
>>
>> oops, brain just skipped this one :P
>>
>> things like sdn apply, or load-balancing that can avoid having to parse
>> all guest configs periodically. i think there is probably a bunch more,
>> things we do periodically are probably good candidates
> 
> What I meant is, do those all make sense in one big does-it-all
> pvescheduler daemon?
> It seems to be meant for "jobs", in the sense of backup jobs etc., so
> I'm not convinced moving every *event* based thing into it - or any one
> giant does-it-all service - is the right way here.
> I'd expect the SDN code to also move further into rust in a not too
> distant future.

agree, no hard feelings either way. just chose this cause there are
domains without their own daemon, and this one seemed general enough