From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 3B32B1FF0C2 for ; Thu, 22 Oct 2026 16:22:14 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id BE7BD215F8; Thu, 22 Oct 2026 16:22:13 +0200 (CEST) From: Hannes Laimer To: pve-devel@lists.proxmox.com Subject: [PATCH cluster 00/12] pmxcfs: add a change notification socket Date: Thu, 22 Oct 2026 16:21:53 +0200 Message-ID: <20261022142205.930536-1-h.laimer@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1792678927528 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.349 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: SAEOVOOD4X5MBCIXFL2CKZ7NLZXWGVGX X-Message-ID-Hash: SAEOVOOD4X5MBCIXFL2CKZ7NLZXWGVGX X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: pmxcfs gets a Unix stream socket at /run/pve-cluster/pmxcfs.sock. It pushes one JSON line per change to /etc/pve to every client whose path template matches the changed path. A client subscribes with named templates such as nodes/{node}/qemu-server/{vmid}.conf. An event carries the type of the change, the path, the rename target and the values the placeholders captured. So a client reads the one file that changed. Today a daemon that cares about /etc/pve compares the version vector on every loop pass and re-reads what usually has not changed. A connection admitted by its group id never sees private paths, as on the IPC and FUSE side. The socket is served by a Rust thread linked into the C daemon as a static library with a small C ABI. The daemon never waits on a client. A client that falls too far behind gets a resync event in place of what it missed, and one that stopped reading is cut off. A node that takes its whole state from the cluster sends a resync as well. Nothing is kept across connections, so every subscription starts with a resync and the client reads everything once. A panic in the Rust code never takes pmxcfs down, at worst it switches the notifier off. On top of the socket, pve-cluster ships three Perl modules. A client handles the connection and its reconnects. A hook registry takes hooks registered like API methods, with a path template whose placeholders become the parameters of a run. And a runner executes those runs in children. It keeps one run per hook and parameter set in flight and collapses later events for the same set into a single rerun. The series ships no hook and nothing that runs the runner. Example, hook registration: ``` package PVE::Example::Hooks; use base qw(PVE::Cluster::Hooks); __PACKAGE__->register_hook({ name => 'guest-config', path => 'nodes/{node}/qemu-server/{vmid}.conf', code => sub { my ($param, $event) = @_; if ($event->{type} eq 'resync') { # reread every guest config return; } # reread guest $param->{vmid} on $param->{node} }, }); 1; ``` Example, in a child of the daemon: ``` use PVE::Example::Hooks; use PVE::Cluster::HookRunner; my $pid = fork() // die "fork failed: $!\n"; if (!$pid) { my $ok = eval { PVE::Cluster::HookRunner->new()->run(); 1 }; warn "hook runner failed: $@" if !$ok; POSIX::_exit($ok ? 0 : 1); } ``` Example, in the daemon's own loop: ``` use PVE::Example::Hooks; use PVE::Cluster; use PVE::Cluster::Hooks; use PVE::Cluster::Watch; my $watch = PVE::Cluster::Watch->new(); $watch->subscribe(PVE::Cluster::Hooks->subscription()->%*); # handle events for ten seconds. Events that arrive while the daemon # does other work wait in the socket until the next pass. If the # socket is full for 30s the server drops the connection, the next # wait reconnects and starts with a resync. my $deadline = time() + 10; while ((my $left = $deadline - time()) > 0) { my @events = $watch->wait($left); # so hooks reading through the cluster cache see the change PVE::Cluster::cfs_update() if @events; for my $event (@events) { for my $run (PVE::Cluster::Hooks->dispatch($event)->@*) { $run->{hook}->{code}->($run->{param}, $run->{event}); } } } ``` Changes since the RFC, thanks @Wolfgang! * resuming is gone, every subscribe is answered with a resync, so there are no sequence numbers, no ring and no saved positions * the server runs on tokio with one task per connection and one bounded channel * the protocol is parsed by serde alone * the ffi takes the event enum directly and lost the version argument, the memdb hook lost it as well * the build goes through cargo's dependency file with the build mode from d/rules * the runner moved to pve-cluster and the pve-manager part is gone. The series adds the machinery only, the examples above show how to use it pve-cluster: Hannes Laimer (12): buildsys: add rust workspace under src/rust rust: notify: add path templates with placeholder capture rust: notify: add the wire protocol rust: notify: add subscription matching rust: notify: add the socket server rust: ffi: add C ABI staticlib for pmxcfs pmxcfs: memdb: add change notification hook buildsys: link pmxcfs against the rust notify staticlib pmxcfs: notify: emit change events over the notification socket cfs: add perl client for the change notification socket cfs: add hook registry for change notification consumers cfs: add runner executing cluster change hooks in children .gitignore | 2 + Makefile | 1 + debian/control | 22 +- debian/pve-cluster.install | 3 + debian/rules | 17 + src/Makefile | 2 +- src/PVE/Cluster/HookRunner.pm | 268 ++++++++++ src/PVE/Cluster/Hooks.pm | 135 +++++ src/PVE/Cluster/Makefile | 2 +- src/PVE/Cluster/Watch.pm | 325 ++++++++++++ src/defines.mk | 11 + src/pmxcfs/Makefile | 20 +- src/pmxcfs/cfs-utils.h | 4 + src/pmxcfs/database.c | 2 + src/pmxcfs/memdb.c | 22 + src/pmxcfs/memdb.h | 8 + src/pmxcfs/pmxcfs.c | 24 + src/rust/.cargo/config.toml | 8 + src/rust/Cargo.toml | 24 + src/rust/Makefile | 20 + src/rust/pmxcfs-ffi/Cargo.toml | 17 + src/rust/pmxcfs-ffi/include/pmxcfs-notify.h | 24 + src/rust/pmxcfs-ffi/src/lib.rs | 306 +++++++++++ src/rust/pmxcfs-notify/Cargo.toml | 17 + src/rust/pmxcfs-notify/src/conn.rs | 334 ++++++++++++ src/rust/pmxcfs-notify/src/lib.rs | 32 ++ src/rust/pmxcfs-notify/src/protocol.rs | 189 +++++++ src/rust/pmxcfs-notify/src/server.rs | 542 ++++++++++++++++++++ src/rust/pmxcfs-notify/src/template.rs | 285 ++++++++++ src/rust/rustfmt.toml | 2 + src/test/Makefile | 14 +- src/test/hook_runner_test.pl | 302 +++++++++++ src/test/hooks_test.pl | 219 ++++++++ src/test/watch_client_test.pl | 229 +++++++++ 34 files changed, 3426 insertions(+), 6 deletions(-) create mode 100644 src/PVE/Cluster/HookRunner.pm create mode 100644 src/PVE/Cluster/Hooks.pm create mode 100644 src/PVE/Cluster/Watch.pm create mode 100644 src/defines.mk create mode 100644 src/rust/.cargo/config.toml create mode 100644 src/rust/Cargo.toml create mode 100644 src/rust/Makefile create mode 100644 src/rust/pmxcfs-ffi/Cargo.toml create mode 100644 src/rust/pmxcfs-ffi/include/pmxcfs-notify.h create mode 100644 src/rust/pmxcfs-ffi/src/lib.rs create mode 100644 src/rust/pmxcfs-notify/Cargo.toml create mode 100644 src/rust/pmxcfs-notify/src/conn.rs create mode 100644 src/rust/pmxcfs-notify/src/lib.rs create mode 100644 src/rust/pmxcfs-notify/src/protocol.rs create mode 100644 src/rust/pmxcfs-notify/src/server.rs create mode 100644 src/rust/pmxcfs-notify/src/template.rs create mode 100644 src/rust/rustfmt.toml create mode 100644 src/test/hook_runner_test.pl create mode 100644 src/test/hooks_test.pl create mode 100644 src/test/watch_client_test.pl Summary over all repositories: 34 files changed, 3426 insertions(+), 6 deletions(-) -- Generated by murpp 0.12.0