Re: [RFC cluster/manager 00/10] pmxcfs: add a change notification socket

[Hannes Laimer <h.laimer@proxmox.com>]

On 2026-10-01 09:31, Wolfgang Bumiller wrote:
> On Fri, Sep 18, 2026 at 04:41:42PM +0200, Hannes Laimer wrote:
>> Every daemon that cares about /etc/pve learns about changes by calling
>> cfs_update on each loop iteration and comparing the version vector it
>> gets over the libqb IPC. inotify cannot replace that, since remote
>> changes arrive through corosync and never touch the local VFS, so
>> pvestatd, pve-firewall, the HA daemons and pvescheduler all wake up on
>> timers and re-read what usually has not changed.
>>
>> This series adds a push path. pmxcfs gets a Unix stream socket at
>> /run/pve-cluster/pmxcfs.sock, served by a Rust thread linked into the C
>> daemon as a static library with a small C ABI. A client subscribes with
>> named path templates such as nodes/{node}/qemu-server/{vmid}.conf and
>> receives one JSON line per matching mutation, carrying the memdb version
>> as sequence number, the event type, the path and the values the
>> placeholders captured. Connections authorized by group membership never
>> see private paths, as on the IPC and FUSE side.
>>
>> The daemon keeps the last mutations in a fixed size ring and each
>> connection a cursor into it, so the mutating thread only appends and
>> never waits for a client. A slow client is caught up from the ring, one
>> that fell off it gets a single resync event, and a reconnecting client
>> resumes at the last sequence number it saw, also across a restart of
>> pmxcfs when nothing changed meanwhile. A node that takes the whole
>> state from the cluster after a membership change hands its clients a
>> resync event, since no sequence of mutations describes that. A panic in
> 
> So a high level question:
> 
> Do we actually have any use case where this particular way processing
> changes on the *client* side would give us any *real* benefit?
> 
> Given the complexity and custom non-blocking handling if socket states,
> I'd like to see a real need for this.
> 
> Why isn't a simple async-based server with no sequence numbers where a
> time-out just disconnects the client, and the disconnect itself being
> the equivalent of a resync event, sufficient?

tbf, that would very likely be more than fine. especially because that
realistically shouldn't really happen a lot. the main use-case I had in
mind was having to expensively parse all guest configs periodically for
load-balancing, with something like this that could be avoided. and with
that in mind i wanted to minimize how often that has to happen. but yes,
given the complexity that adds, and that disconnects or alike
realistically happen only very rarely, the added complexity is not a
really good tradeoff.

> 
> The server would be just the accept loop which `spawn()`s a
> client-handler with its socket and event channel, the send-side of the
> channel pushed to the list of client channels. The client handling task
> just loops through a channel.recv() followed by a socket.send() with a
> timeout. On timeout it just returns, the channel gets closed, the next
> event-emit sees the channel.send() fail and removes the channel.

this sounds like a good alternative approach. i'll prepare a v1 with
this.

thanks a lot for taking a look, and the feedback on this! :)