superseded: [RFC cluster/manager 00/10] pmxcfs: add a change notification socket

[Hannes Laimer <h.laimer@proxmox.com>]

superseded-by:
https://lore.proxmox.com/pve-devel/20261022142205.930536-1-h.laimer@proxmox.com/

On 2026-09-18 16:42, Hannes Laimer wrote:
> Every daemon that cares about /etc/pve learns about changes by calling
> cfs_update on each loop iteration and comparing the version vector it
> gets over the libqb IPC. inotify cannot replace that, since remote
> changes arrive through corosync and never touch the local VFS, so
> pvestatd, pve-firewall, the HA daemons and pvescheduler all wake up on
> timers and re-read what usually has not changed.
> 
> This series adds a push path. pmxcfs gets a Unix stream socket at
> /run/pve-cluster/pmxcfs.sock, served by a Rust thread linked into the C
> daemon as a static library with a small C ABI. A client subscribes with
> named path templates such as nodes/{node}/qemu-server/{vmid}.conf and
> receives one JSON line per matching mutation, carrying the memdb version
> as sequence number, the event type, the path and the values the
> placeholders captured. Connections authorized by group membership never
> see private paths, as on the IPC and FUSE side.
> 
> The daemon keeps the last mutations in a fixed size ring and each
> connection a cursor into it, so the mutating thread only appends and
> never waits for a client. A slow client is caught up from the ring, one
> that fell off it gets a single resync event, and a reconnecting client
> resumes at the last sequence number it saw, also across a restart of
> pmxcfs when nothing changed meanwhile. A node that takes the whole
> state from the cluster after a membership change hands its clients a
> resync event, since no sequence of mutations describes that. A panic in
> the Rust code only disables the notifier for the rest of the daemon's
> life. A sequence number identifies a state within one line of history
> only. A client that missed a resync while disconnected and returns at
> the same version after a pmxcfs restart resumes as if nothing changed.
> Carrying the root entry's mtime next to the version would close that.
> 
> On top of the socket, pve-cluster ships a Perl client with reconnect
> and resync handling and a hook registry, where a hook is registered
> like an API method with a path template whose placeholders become the
> parameters of a run. pve-manager gets the runner that executes those
> runs in children of a listener hosted by pvescheduler, one run per hook
> and parameter set in flight and later events for the same set collapsed
> into a single rerun, so a config update through the API costs one extra
> run rather than one per save. The listener records the position it has
> processed up to under /run and resumes there after a reload. No hook
> ships in this series. A consumer registers one as the example below
> shows.
> 
> pve-manager depends on the pve-cluster packages of this series for the
> new modules and the socket, at build time as well, since its tests run
> through the check target.
> 
> Example usage:
> 
>     package PVE::Network::Hooks;
> 
>     use base qw(PVE::Cluster::Hooks);
> 
>     __PACKAGE__->register_hook({
>         name => 'guest-firewall',
>         path => 'firewall/{vmid}.fw',
>         code => sub {
>             my ($param, $event) = @_;
> 
>             my $vmids = $event->{type} eq 'resync'
>                 ? [] # every guest with a firewall config
>                 : [ $param->{vmid} ];
> 
>             for my $vmid ($vmids->@*) {
>                 # reload and apply the firewall for guest $vmid
>             }
>         },
>     });
> 
>     1;
> 
> 
> pve-cluster:
> 
> Hannes Laimer (8):
>   buildsys: add rust workspace under src/rust
>   rust: notify: add change notification socket server
>   rust: ffi: add C ABI staticlib for pmxcfs
>   pmxcfs: memdb: add change notification hook
>   buildsys: link pmxcfs against the rust notify staticlib
>   pmxcfs: notify: emit change events over the notification socket
>   cfs: add perl client for the change notification socket
>   cfs: add hook registry for change notification consumers
> 
>  .gitignore                                  |   2 +
>  Makefile                                    |   1 +
>  debian/control                              |  16 +-
>  debian/pve-cluster.install                  |   2 +
>  debian/rules                                |  16 +
>  src/Makefile                                |   2 +-
>  src/PVE/Cluster/Hooks.pm                    | 134 ++++
>  src/PVE/Cluster/Makefile                    |   2 +-
>  src/PVE/Cluster/Watch.pm                    | 371 ++++++++++
>  src/pmxcfs/Makefile                         |  15 +-
>  src/pmxcfs/cfs-utils.h                      |   4 +
>  src/pmxcfs/database.c                       |   2 +
>  src/pmxcfs/memdb.c                          |  24 +
>  src/pmxcfs/memdb.h                          |  11 +
>  src/pmxcfs/pmxcfs.c                         |  25 +
>  src/rust/.cargo/config.toml                 |   8 +
>  src/rust/Cargo.toml                         |  23 +
>  src/rust/Makefile                           |  18 +
>  src/rust/pmxcfs-ffi/Cargo.toml              |  17 +
>  src/rust/pmxcfs-ffi/include/pmxcfs-notify.h |  29 +
>  src/rust/pmxcfs-ffi/src/lib.rs              | 309 ++++++++
>  src/rust/pmxcfs-notify/Cargo.toml           |  16 +
>  src/rust/pmxcfs-notify/src/conn.rs          | 317 ++++++++
>  src/rust/pmxcfs-notify/src/lib.rs           |  43 ++
>  src/rust/pmxcfs-notify/src/protocol.rs      | 237 ++++++
>  src/rust/pmxcfs-notify/src/registry.rs      | 405 ++++++++++
>  src/rust/pmxcfs-notify/src/ring.rs          | 171 +++++
>  src/rust/pmxcfs-notify/src/server.rs        | 780 ++++++++++++++++++++
>  src/rust/pmxcfs-notify/src/template.rs      | 254 +++++++
>  src/rust/rustfmt.toml                       |   2 +
>  src/test/Makefile                           |  10 +-
>  src/test/hooks_test.pl                      | 219 ++++++
>  src/test/watch_client_test.pl               | 294 ++++++++
>  33 files changed, 3773 insertions(+), 6 deletions(-)
>  create mode 100644 src/PVE/Cluster/Hooks.pm
>  create mode 100644 src/PVE/Cluster/Watch.pm
>  create mode 100644 src/rust/.cargo/config.toml
>  create mode 100644 src/rust/Cargo.toml
>  create mode 100644 src/rust/Makefile
>  create mode 100644 src/rust/pmxcfs-ffi/Cargo.toml
>  create mode 100644 src/rust/pmxcfs-ffi/include/pmxcfs-notify.h
>  create mode 100644 src/rust/pmxcfs-ffi/src/lib.rs
>  create mode 100644 src/rust/pmxcfs-notify/Cargo.toml
>  create mode 100644 src/rust/pmxcfs-notify/src/conn.rs
>  create mode 100644 src/rust/pmxcfs-notify/src/lib.rs
>  create mode 100644 src/rust/pmxcfs-notify/src/protocol.rs
>  create mode 100644 src/rust/pmxcfs-notify/src/registry.rs
>  create mode 100644 src/rust/pmxcfs-notify/src/ring.rs
>  create mode 100644 src/rust/pmxcfs-notify/src/server.rs
>  create mode 100644 src/rust/pmxcfs-notify/src/template.rs
>  create mode 100644 src/rust/rustfmt.toml
>  create mode 100644 src/test/hooks_test.pl
>  create mode 100644 src/test/watch_client_test.pl
> 
> 
> pve-manager:
> 
> Hannes Laimer (2):
>   hooks: add runner executing cluster change hooks in children
>   pvescheduler: run cluster change hooks from a listener child
> 
>  PVE/HookRunner.pm           | 368 ++++++++++++++++++++++++++++++++
>  PVE/Makefile                |   1 +
>  PVE/Service/pvescheduler.pm |  26 ++-
>  test/Makefile               |   6 +-
>  test/hook_runner_test.pl    | 405 ++++++++++++++++++++++++++++++++++++
>  5 files changed, 801 insertions(+), 5 deletions(-)
>  create mode 100644 PVE/HookRunner.pm
>  create mode 100755 test/hook_runner_test.pl
> 
> 
> Summary over all repositories:
>   38 files changed, 4574 insertions(+), 11 deletions(-)
>