Re: [RFC cluster/manager 00/10] pmxcfs: add a change notification socket

[Hannes Laimer <h.laimer@proxmox.com>]

On 2026-09-25 14:01, Wolfgang Bumiller wrote:
> On Fri, Sep 18, 2026 at 04:41:42PM +0200, Hannes Laimer wrote:
>> Every daemon that cares about /etc/pve learns about changes by calling
>> cfs_update on each loop iteration and comparing the version vector it
>> gets over the libqb IPC. inotify cannot replace that, since remote
> 
> Not strictly true - pmxcfs could add inotify support vis fuse, but it is
> limited to regular files and would not support whole directory watches.
> 
>> changes arrive through corosync and never touch the local VFS, so
>> pvestatd, pve-firewall, the HA daemons and pvescheduler all wake up on
>> timers and re-read what usually has not changed.
>>
>> This series adds a push path. pmxcfs gets a Unix stream socket at
>> /run/pve-cluster/pmxcfs.sock, served by a Rust thread linked into the C
>> daemon as a static library with a small C ABI. A client subscribes with
>> named path templates such as nodes/{node}/qemu-server/{vmid}.conf and
>> receives one JSON line per matching mutation, carrying the memdb version
>> as sequence number, the event type, the path and the values the
>> placeholders captured. Connections authorized by group membership never
>> see private paths, as on the IPC and FUSE side.
>>
>> The daemon keeps the last mutations in a fixed size ring and each
>> connection a cursor into it, so the mutating thread only appends and
>> never waits for a client. A slow client is caught up from the ring, one
>> that fell off it gets a single resync event, and a reconnecting client
>> resumes at the last sequence number it saw, also across a restart of
>> pmxcfs when nothing changed meanwhile. A node that takes the whole
>> state from the cluster after a membership change hands its clients a
>> resync event, since no sequence of mutations describes that. A panic in
>> the Rust code only disables the notifier for the rest of the daemon's
>> life. A sequence number identifies a state within one line of history
>> only. A client that missed a resync while disconnected and returns at
>> the same version after a pmxcfs restart resumes as if nothing changed.
>> Carrying the root entry's mtime next to the version would close that.
>>
>> On top of the socket, pve-cluster ships a Perl client with reconnect
>> and resync handling and a hook registry, where a hook is registered
>> like an API method with a path template whose placeholders become the
>> parameters of a run. pve-manager gets the runner that executes those
>> runs in children of a listener hosted by pvescheduler, one run per hook
>> and parameter set in flight and later events for the same set collapsed
>> into a single rerun, so a config update through the API costs one extra
>> run rather than one per save. The listener records the position it has
>> processed up to under /run and resumes there after a reload. No hook
>> ships in this series. A consumer registers one as the example below
>> shows.
>>
>> pve-manager depends on the pve-cluster packages of this series for the
>> new modules and the socket, at build time as well, since its tests run
>> through the check target.
>>
>> Example usage:
>>
>>     package PVE::Network::Hooks;
>>
>>     use base qw(PVE::Cluster::Hooks);
>>
>>     __PACKAGE__->register_hook({
>>         name => 'guest-firewall',
> 
> ^ This particular example would have to run in the pve-firewall daemon,
> not pvescheduler, though.

well, maybe, what the daemon does currently is basically translate
config into nft kernel state periodically. the idea was that it could
stop being periodic, and only one daemon would then work on config
updates.. but could also just run in the fw daemon, not sure if there
are good arguments against having only one daemon that handles such
events

> What about the remaining use cases?
> 
>>         path => 'firewall/{vmid}.fw',
>>         code => sub {
>>             my ($param, $event) = @_;
>>
>>             my $vmids = $event->{type} eq 'resync'
>>                 ? [] # every guest with a firewall config
>>                 : [ $param->{vmid} ];
>>
>>             for my $vmid ($vmids->@*) {
>>                 # reload and apply the firewall for guest $vmid
>>             }
>>         },
>>     });
>>
>>     1;