Re: [RFC cluster/manager 00/10] pmxcfs: add a change notification socket

[Wolfgang Bumiller <w.bumiller@proxmox.com>]

On Fri, Sep 18, 2026 at 04:41:42PM +0200, Hannes Laimer wrote:
> Every daemon that cares about /etc/pve learns about changes by calling
> cfs_update on each loop iteration and comparing the version vector it
> gets over the libqb IPC. inotify cannot replace that, since remote

Not strictly true - pmxcfs could add inotify support vis fuse, but it is
limited to regular files and would not support whole directory watches.

> changes arrive through corosync and never touch the local VFS, so
> pvestatd, pve-firewall, the HA daemons and pvescheduler all wake up on
> timers and re-read what usually has not changed.
> 
> This series adds a push path. pmxcfs gets a Unix stream socket at
> /run/pve-cluster/pmxcfs.sock, served by a Rust thread linked into the C
> daemon as a static library with a small C ABI. A client subscribes with
> named path templates such as nodes/{node}/qemu-server/{vmid}.conf and
> receives one JSON line per matching mutation, carrying the memdb version
> as sequence number, the event type, the path and the values the
> placeholders captured. Connections authorized by group membership never
> see private paths, as on the IPC and FUSE side.
> 
> The daemon keeps the last mutations in a fixed size ring and each
> connection a cursor into it, so the mutating thread only appends and
> never waits for a client. A slow client is caught up from the ring, one
> that fell off it gets a single resync event, and a reconnecting client
> resumes at the last sequence number it saw, also across a restart of
> pmxcfs when nothing changed meanwhile. A node that takes the whole
> state from the cluster after a membership change hands its clients a
> resync event, since no sequence of mutations describes that. A panic in
> the Rust code only disables the notifier for the rest of the daemon's
> life. A sequence number identifies a state within one line of history
> only. A client that missed a resync while disconnected and returns at
> the same version after a pmxcfs restart resumes as if nothing changed.
> Carrying the root entry's mtime next to the version would close that.
> 
> On top of the socket, pve-cluster ships a Perl client with reconnect
> and resync handling and a hook registry, where a hook is registered
> like an API method with a path template whose placeholders become the
> parameters of a run. pve-manager gets the runner that executes those
> runs in children of a listener hosted by pvescheduler, one run per hook
> and parameter set in flight and later events for the same set collapsed
> into a single rerun, so a config update through the API costs one extra
> run rather than one per save. The listener records the position it has
> processed up to under /run and resumes there after a reload. No hook
> ships in this series. A consumer registers one as the example below
> shows.
> 
> pve-manager depends on the pve-cluster packages of this series for the
> new modules and the socket, at build time as well, since its tests run
> through the check target.
> 
> Example usage:
> 
>     package PVE::Network::Hooks;
> 
>     use base qw(PVE::Cluster::Hooks);
> 
>     __PACKAGE__->register_hook({
>         name => 'guest-firewall',

^ This particular example would have to run in the pve-firewall daemon,
not pvescheduler, though.
What about the remaining use cases?

>         path => 'firewall/{vmid}.fw',
>         code => sub {
>             my ($param, $event) = @_;
> 
>             my $vmids = $event->{type} eq 'resync'
>                 ? [] # every guest with a firewall config
>                 : [ $param->{vmid} ];
> 
>             for my $vmid ($vmids->@*) {
>                 # reload and apply the firewall for guest $vmid
>             }
>         },
>     });
> 
>     1;