From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id DC19F1FF0C1 for ; Fri, 18 Sep 2026 16:42:55 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id D6A402164F; Fri, 18 Sep 2026 16:42:12 +0200 (CEST) From: Hannes Laimer To: pve-devel@lists.proxmox.com Subject: [RFC cluster/manager 00/10] pmxcfs: add a change notification socket Date: Fri, 18 Sep 2026 16:41:42 +0200 Message-ID: <20260918144152.575163-1-h.laimer@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1789742522780 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.502 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust RCVD_IN_MSPIKE_H2 0.001 Average reputation (+2) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: A7T566MIZ7OLJYDF72WEK7KK4SH3XC2J X-Message-ID-Hash: A7T566MIZ7OLJYDF72WEK7KK4SH3XC2J X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Every daemon that cares about /etc/pve learns about changes by calling cfs_update on each loop iteration and comparing the version vector it gets over the libqb IPC. inotify cannot replace that, since remote changes arrive through corosync and never touch the local VFS, so pvestatd, pve-firewall, the HA daemons and pvescheduler all wake up on timers and re-read what usually has not changed. This series adds a push path. pmxcfs gets a Unix stream socket at /run/pve-cluster/pmxcfs.sock, served by a Rust thread linked into the C daemon as a static library with a small C ABI. A client subscribes with named path templates such as nodes/{node}/qemu-server/{vmid}.conf and receives one JSON line per matching mutation, carrying the memdb version as sequence number, the event type, the path and the values the placeholders captured. Connections authorized by group membership never see private paths, as on the IPC and FUSE side. The daemon keeps the last mutations in a fixed size ring and each connection a cursor into it, so the mutating thread only appends and never waits for a client. A slow client is caught up from the ring, one that fell off it gets a single resync event, and a reconnecting client resumes at the last sequence number it saw, also across a restart of pmxcfs when nothing changed meanwhile. A node that takes the whole state from the cluster after a membership change hands its clients a resync event, since no sequence of mutations describes that. A panic in the Rust code only disables the notifier for the rest of the daemon's life. A sequence number identifies a state within one line of history only. A client that missed a resync while disconnected and returns at the same version after a pmxcfs restart resumes as if nothing changed. Carrying the root entry's mtime next to the version would close that. On top of the socket, pve-cluster ships a Perl client with reconnect and resync handling and a hook registry, where a hook is registered like an API method with a path template whose placeholders become the parameters of a run. pve-manager gets the runner that executes those runs in children of a listener hosted by pvescheduler, one run per hook and parameter set in flight and later events for the same set collapsed into a single rerun, so a config update through the API costs one extra run rather than one per save. The listener records the position it has processed up to under /run and resumes there after a reload. No hook ships in this series. A consumer registers one as the example below shows. pve-manager depends on the pve-cluster packages of this series for the new modules and the socket, at build time as well, since its tests run through the check target. Example usage: package PVE::Network::Hooks; use base qw(PVE::Cluster::Hooks); __PACKAGE__->register_hook({ name => 'guest-firewall', path => 'firewall/{vmid}.fw', code => sub { my ($param, $event) = @_; my $vmids = $event->{type} eq 'resync' ? [] # every guest with a firewall config : [ $param->{vmid} ]; for my $vmid ($vmids->@*) { # reload and apply the firewall for guest $vmid } }, }); 1; pve-cluster: Hannes Laimer (8): buildsys: add rust workspace under src/rust rust: notify: add change notification socket server rust: ffi: add C ABI staticlib for pmxcfs pmxcfs: memdb: add change notification hook buildsys: link pmxcfs against the rust notify staticlib pmxcfs: notify: emit change events over the notification socket cfs: add perl client for the change notification socket cfs: add hook registry for change notification consumers .gitignore | 2 + Makefile | 1 + debian/control | 16 +- debian/pve-cluster.install | 2 + debian/rules | 16 + src/Makefile | 2 +- src/PVE/Cluster/Hooks.pm | 134 ++++ src/PVE/Cluster/Makefile | 2 +- src/PVE/Cluster/Watch.pm | 371 ++++++++++ src/pmxcfs/Makefile | 15 +- src/pmxcfs/cfs-utils.h | 4 + src/pmxcfs/database.c | 2 + src/pmxcfs/memdb.c | 24 + src/pmxcfs/memdb.h | 11 + src/pmxcfs/pmxcfs.c | 25 + src/rust/.cargo/config.toml | 8 + src/rust/Cargo.toml | 23 + src/rust/Makefile | 18 + src/rust/pmxcfs-ffi/Cargo.toml | 17 + src/rust/pmxcfs-ffi/include/pmxcfs-notify.h | 29 + src/rust/pmxcfs-ffi/src/lib.rs | 309 ++++++++ src/rust/pmxcfs-notify/Cargo.toml | 16 + src/rust/pmxcfs-notify/src/conn.rs | 317 ++++++++ src/rust/pmxcfs-notify/src/lib.rs | 43 ++ src/rust/pmxcfs-notify/src/protocol.rs | 237 ++++++ src/rust/pmxcfs-notify/src/registry.rs | 405 ++++++++++ src/rust/pmxcfs-notify/src/ring.rs | 171 +++++ src/rust/pmxcfs-notify/src/server.rs | 780 ++++++++++++++++++++ src/rust/pmxcfs-notify/src/template.rs | 254 +++++++ src/rust/rustfmt.toml | 2 + src/test/Makefile | 10 +- src/test/hooks_test.pl | 219 ++++++ src/test/watch_client_test.pl | 294 ++++++++ 33 files changed, 3773 insertions(+), 6 deletions(-) create mode 100644 src/PVE/Cluster/Hooks.pm create mode 100644 src/PVE/Cluster/Watch.pm create mode 100644 src/rust/.cargo/config.toml create mode 100644 src/rust/Cargo.toml create mode 100644 src/rust/Makefile create mode 100644 src/rust/pmxcfs-ffi/Cargo.toml create mode 100644 src/rust/pmxcfs-ffi/include/pmxcfs-notify.h create mode 100644 src/rust/pmxcfs-ffi/src/lib.rs create mode 100644 src/rust/pmxcfs-notify/Cargo.toml create mode 100644 src/rust/pmxcfs-notify/src/conn.rs create mode 100644 src/rust/pmxcfs-notify/src/lib.rs create mode 100644 src/rust/pmxcfs-notify/src/protocol.rs create mode 100644 src/rust/pmxcfs-notify/src/registry.rs create mode 100644 src/rust/pmxcfs-notify/src/ring.rs create mode 100644 src/rust/pmxcfs-notify/src/server.rs create mode 100644 src/rust/pmxcfs-notify/src/template.rs create mode 100644 src/rust/rustfmt.toml create mode 100644 src/test/hooks_test.pl create mode 100644 src/test/watch_client_test.pl pve-manager: Hannes Laimer (2): hooks: add runner executing cluster change hooks in children pvescheduler: run cluster change hooks from a listener child PVE/HookRunner.pm | 368 ++++++++++++++++++++++++++++++++ PVE/Makefile | 1 + PVE/Service/pvescheduler.pm | 26 ++- test/Makefile | 6 +- test/hook_runner_test.pl | 405 ++++++++++++++++++++++++++++++++++++ 5 files changed, 801 insertions(+), 5 deletions(-) create mode 100644 PVE/HookRunner.pm create mode 100755 test/hook_runner_test.pl Summary over all repositories: 38 files changed, 4574 insertions(+), 11 deletions(-) -- Generated by murpp 0.12.0