Re: [RFC cluster/manager 00/10] pmxcfs: add a change notification socket

[Hannes Laimer <h.laimer@proxmox.com>]

On 2026-09-25 14:01, Wolfgang Bumiller wrote:
> On Fri, Sep 18, 2026 at 04:41:42PM +0200, Hannes Laimer wrote:
>> Every daemon that cares about /etc/pve learns about changes by calling
>> cfs_update on each loop iteration and comparing the version vector it
>> gets over the libqb IPC. inotify cannot replace that, since remote
> 
> Not strictly true - pmxcfs could add inotify support vis fuse, but it is
> limited to regular files and would not support whole directory watches.
> 
>> changes arrive through corosync and never touch the local VFS, so
>> pvestatd, pve-firewall, the HA daemons and pvescheduler all wake up on
>> timers and re-read what usually has not changed.
>>
>> This series adds a push path. pmxcfs gets a Unix stream socket at
>> /run/pve-cluster/pmxcfs.sock, served by a Rust thread linked into the C
>> daemon as a static library with a small C ABI. A client subscribes with
>> named path templates such as nodes/{node}/qemu-server/{vmid}.conf and
>> receives one JSON line per matching mutation, carrying the memdb version
>> as sequence number, the event type, the path and the values the
>> placeholders captured. Connections authorized by group membership never
>> see private paths, as on the IPC and FUSE side.
>>
>> The daemon keeps the last mutations in a fixed size ring and each
>> connection a cursor into it, so the mutating thread only appends and
>> never waits for a client. A slow client is caught up from the ring, one
>> that fell off it gets a single resync event, and a reconnecting client
>> resumes at the last sequence number it saw, also across a restart of
>> pmxcfs when nothing changed meanwhile. A node that takes the whole
>> state from the cluster after a membership change hands its clients a
>> resync event, since no sequence of mutations describes that. A panic in
>> the Rust code only disables the notifier for the rest of the daemon's
>> life. A sequence number identifies a state within one line of history
>> only. A client that missed a resync while disconnected and returns at
>> the same version after a pmxcfs restart resumes as if nothing changed.
>> Carrying the root entry's mtime next to the version would close that.
>>
>> On top of the socket, pve-cluster ships a Perl client with reconnect
>> and resync handling and a hook registry, where a hook is registered
>> like an API method with a path template whose placeholders become the
>> parameters of a run. pve-manager gets the runner that executes those
>> runs in children of a listener hosted by pvescheduler, one run per hook
>> and parameter set in flight and later events for the same set collapsed
>> into a single rerun, so a config update through the API costs one extra
>> run rather than one per save. The listener records the position it has
>> processed up to under /run and resumes there after a reload. No hook
>> ships in this series. A consumer registers one as the example below
>> shows.
>>
>> pve-manager depends on the pve-cluster packages of this series for the
>> new modules and the socket, at build time as well, since its tests run
>> through the check target.
>>
>> Example usage:
>>
>>     package PVE::Network::Hooks;
>>
>>     use base qw(PVE::Cluster::Hooks);
>>
>>     __PACKAGE__->register_hook({
>>         name => 'guest-firewall',
> 
> ^ This particular example would have to run in the pve-firewall daemon,
> not pvescheduler, though.
> What about the remaining use cases?

oops, brain just skipped this one :P

things like sdn apply, or load-balancing that can avoid having to parse
all guest configs periodically. i think there is probably a bunch more,
things we do periodically are probably good candidates

thanks for taking a look! :)

> 
>>         path => 'firewall/{vmid}.fw',
>>         code => sub {
>>             my ($param, $event) = @_;
>>
>>             my $vmids = $event->{type} eq 'resync'
>>                 ? [] # every guest with a firewall config
>>                 : [ $param->{vmid} ];
>>
>>             for my $vmid ($vmids->@*) {
>>                 # reload and apply the firewall for guest $vmid
>>             }
>>         },
>>     });
>>
>>     1;