From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id A7B661FF0C2 for ; Thu, 22 Oct 2026 16:24:22 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 02A3A21690; Thu, 22 Oct 2026 16:24:22 +0200 (CEST) Message-ID: Date: Thu, 22 Oct 2026 16:22:59 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: superseded: [RFC cluster/manager 00/10] pmxcfs: add a change notification socket To: pve-devel@lists.proxmox.com References: <20260918144152.575163-1-h.laimer@proxmox.com> From: Hannes Laimer Content-Language: en-US In-Reply-To: <20260918144152.575163-1-h.laimer@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1792678979865 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.340 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: I3VPL5VLYVMI4QJLXN23I6Z6WT3ESAQN X-Message-ID-Hash: I3VPL5VLYVMI4QJLXN23I6Z6WT3ESAQN X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: superseded-by: https://lore.proxmox.com/pve-devel/20261022142205.930536-1-h.laimer@proxmox.com/ On 2026-09-18 16:42, Hannes Laimer wrote: > Every daemon that cares about /etc/pve learns about changes by calling > cfs_update on each loop iteration and comparing the version vector it > gets over the libqb IPC. inotify cannot replace that, since remote > changes arrive through corosync and never touch the local VFS, so > pvestatd, pve-firewall, the HA daemons and pvescheduler all wake up on > timers and re-read what usually has not changed. > > This series adds a push path. pmxcfs gets a Unix stream socket at > /run/pve-cluster/pmxcfs.sock, served by a Rust thread linked into the C > daemon as a static library with a small C ABI. A client subscribes with > named path templates such as nodes/{node}/qemu-server/{vmid}.conf and > receives one JSON line per matching mutation, carrying the memdb version > as sequence number, the event type, the path and the values the > placeholders captured. Connections authorized by group membership never > see private paths, as on the IPC and FUSE side. > > The daemon keeps the last mutations in a fixed size ring and each > connection a cursor into it, so the mutating thread only appends and > never waits for a client. A slow client is caught up from the ring, one > that fell off it gets a single resync event, and a reconnecting client > resumes at the last sequence number it saw, also across a restart of > pmxcfs when nothing changed meanwhile. A node that takes the whole > state from the cluster after a membership change hands its clients a > resync event, since no sequence of mutations describes that. A panic in > the Rust code only disables the notifier for the rest of the daemon's > life. A sequence number identifies a state within one line of history > only. A client that missed a resync while disconnected and returns at > the same version after a pmxcfs restart resumes as if nothing changed. > Carrying the root entry's mtime next to the version would close that. > > On top of the socket, pve-cluster ships a Perl client with reconnect > and resync handling and a hook registry, where a hook is registered > like an API method with a path template whose placeholders become the > parameters of a run. pve-manager gets the runner that executes those > runs in children of a listener hosted by pvescheduler, one run per hook > and parameter set in flight and later events for the same set collapsed > into a single rerun, so a config update through the API costs one extra > run rather than one per save. The listener records the position it has > processed up to under /run and resumes there after a reload. No hook > ships in this series. A consumer registers one as the example below > shows. > > pve-manager depends on the pve-cluster packages of this series for the > new modules and the socket, at build time as well, since its tests run > through the check target. > > Example usage: > > package PVE::Network::Hooks; > > use base qw(PVE::Cluster::Hooks); > > __PACKAGE__->register_hook({ > name => 'guest-firewall', > path => 'firewall/{vmid}.fw', > code => sub { > my ($param, $event) = @_; > > my $vmids = $event->{type} eq 'resync' > ? [] # every guest with a firewall config > : [ $param->{vmid} ]; > > for my $vmid ($vmids->@*) { > # reload and apply the firewall for guest $vmid > } > }, > }); > > 1; > > > pve-cluster: > > Hannes Laimer (8): > buildsys: add rust workspace under src/rust > rust: notify: add change notification socket server > rust: ffi: add C ABI staticlib for pmxcfs > pmxcfs: memdb: add change notification hook > buildsys: link pmxcfs against the rust notify staticlib > pmxcfs: notify: emit change events over the notification socket > cfs: add perl client for the change notification socket > cfs: add hook registry for change notification consumers > > .gitignore | 2 + > Makefile | 1 + > debian/control | 16 +- > debian/pve-cluster.install | 2 + > debian/rules | 16 + > src/Makefile | 2 +- > src/PVE/Cluster/Hooks.pm | 134 ++++ > src/PVE/Cluster/Makefile | 2 +- > src/PVE/Cluster/Watch.pm | 371 ++++++++++ > src/pmxcfs/Makefile | 15 +- > src/pmxcfs/cfs-utils.h | 4 + > src/pmxcfs/database.c | 2 + > src/pmxcfs/memdb.c | 24 + > src/pmxcfs/memdb.h | 11 + > src/pmxcfs/pmxcfs.c | 25 + > src/rust/.cargo/config.toml | 8 + > src/rust/Cargo.toml | 23 + > src/rust/Makefile | 18 + > src/rust/pmxcfs-ffi/Cargo.toml | 17 + > src/rust/pmxcfs-ffi/include/pmxcfs-notify.h | 29 + > src/rust/pmxcfs-ffi/src/lib.rs | 309 ++++++++ > src/rust/pmxcfs-notify/Cargo.toml | 16 + > src/rust/pmxcfs-notify/src/conn.rs | 317 ++++++++ > src/rust/pmxcfs-notify/src/lib.rs | 43 ++ > src/rust/pmxcfs-notify/src/protocol.rs | 237 ++++++ > src/rust/pmxcfs-notify/src/registry.rs | 405 ++++++++++ > src/rust/pmxcfs-notify/src/ring.rs | 171 +++++ > src/rust/pmxcfs-notify/src/server.rs | 780 ++++++++++++++++++++ > src/rust/pmxcfs-notify/src/template.rs | 254 +++++++ > src/rust/rustfmt.toml | 2 + > src/test/Makefile | 10 +- > src/test/hooks_test.pl | 219 ++++++ > src/test/watch_client_test.pl | 294 ++++++++ > 33 files changed, 3773 insertions(+), 6 deletions(-) > create mode 100644 src/PVE/Cluster/Hooks.pm > create mode 100644 src/PVE/Cluster/Watch.pm > create mode 100644 src/rust/.cargo/config.toml > create mode 100644 src/rust/Cargo.toml > create mode 100644 src/rust/Makefile > create mode 100644 src/rust/pmxcfs-ffi/Cargo.toml > create mode 100644 src/rust/pmxcfs-ffi/include/pmxcfs-notify.h > create mode 100644 src/rust/pmxcfs-ffi/src/lib.rs > create mode 100644 src/rust/pmxcfs-notify/Cargo.toml > create mode 100644 src/rust/pmxcfs-notify/src/conn.rs > create mode 100644 src/rust/pmxcfs-notify/src/lib.rs > create mode 100644 src/rust/pmxcfs-notify/src/protocol.rs > create mode 100644 src/rust/pmxcfs-notify/src/registry.rs > create mode 100644 src/rust/pmxcfs-notify/src/ring.rs > create mode 100644 src/rust/pmxcfs-notify/src/server.rs > create mode 100644 src/rust/pmxcfs-notify/src/template.rs > create mode 100644 src/rust/rustfmt.toml > create mode 100644 src/test/hooks_test.pl > create mode 100644 src/test/watch_client_test.pl > > > pve-manager: > > Hannes Laimer (2): > hooks: add runner executing cluster change hooks in children > pvescheduler: run cluster change hooks from a listener child > > PVE/HookRunner.pm | 368 ++++++++++++++++++++++++++++++++ > PVE/Makefile | 1 + > PVE/Service/pvescheduler.pm | 26 ++- > test/Makefile | 6 +- > test/hook_runner_test.pl | 405 ++++++++++++++++++++++++++++++++++++ > 5 files changed, 801 insertions(+), 5 deletions(-) > create mode 100644 PVE/HookRunner.pm > create mode 100755 test/hook_runner_test.pl > > > Summary over all repositories: > 38 files changed, 4574 insertions(+), 11 deletions(-) >