From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 9865B1FF0B2 for ; Fri, 25 Sep 2026 14:24:00 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id E398B216CB; Fri, 25 Sep 2026 14:23:57 +0200 (CEST) Message-ID: Date: Fri, 25 Sep 2026 14:23:53 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC cluster/manager 00/10] pmxcfs: add a change notification socket To: Wolfgang Bumiller References: <20260918144152.575163-1-h.laimer@proxmox.com> <4kp2pm6gasdbailmstjwxrvsc3ozmpiln4tz67pjh4ii3od2eg@ec5poxqld7gh> From: Hannes Laimer Content-Language: en-US In-Reply-To: <4kp2pm6gasdbailmstjwxrvsc3ozmpiln4tz67pjh4ii3od2eg@ec5poxqld7gh> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1790339034113 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.475 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 4OFQOCAGDO5LQR3XS2OUVML4L2DTM634 X-Message-ID-Hash: 4OFQOCAGDO5LQR3XS2OUVML4L2DTM634 X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pve-devel@lists.proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 2026-09-25 14:01, Wolfgang Bumiller wrote: > On Fri, Sep 18, 2026 at 04:41:42PM +0200, Hannes Laimer wrote: >> Every daemon that cares about /etc/pve learns about changes by calling >> cfs_update on each loop iteration and comparing the version vector it >> gets over the libqb IPC. inotify cannot replace that, since remote > > Not strictly true - pmxcfs could add inotify support vis fuse, but it is > limited to regular files and would not support whole directory watches. > >> changes arrive through corosync and never touch the local VFS, so >> pvestatd, pve-firewall, the HA daemons and pvescheduler all wake up on >> timers and re-read what usually has not changed. >> >> This series adds a push path. pmxcfs gets a Unix stream socket at >> /run/pve-cluster/pmxcfs.sock, served by a Rust thread linked into the C >> daemon as a static library with a small C ABI. A client subscribes with >> named path templates such as nodes/{node}/qemu-server/{vmid}.conf and >> receives one JSON line per matching mutation, carrying the memdb version >> as sequence number, the event type, the path and the values the >> placeholders captured. Connections authorized by group membership never >> see private paths, as on the IPC and FUSE side. >> >> The daemon keeps the last mutations in a fixed size ring and each >> connection a cursor into it, so the mutating thread only appends and >> never waits for a client. A slow client is caught up from the ring, one >> that fell off it gets a single resync event, and a reconnecting client >> resumes at the last sequence number it saw, also across a restart of >> pmxcfs when nothing changed meanwhile. A node that takes the whole >> state from the cluster after a membership change hands its clients a >> resync event, since no sequence of mutations describes that. A panic in >> the Rust code only disables the notifier for the rest of the daemon's >> life. A sequence number identifies a state within one line of history >> only. A client that missed a resync while disconnected and returns at >> the same version after a pmxcfs restart resumes as if nothing changed. >> Carrying the root entry's mtime next to the version would close that. >> >> On top of the socket, pve-cluster ships a Perl client with reconnect >> and resync handling and a hook registry, where a hook is registered >> like an API method with a path template whose placeholders become the >> parameters of a run. pve-manager gets the runner that executes those >> runs in children of a listener hosted by pvescheduler, one run per hook >> and parameter set in flight and later events for the same set collapsed >> into a single rerun, so a config update through the API costs one extra >> run rather than one per save. The listener records the position it has >> processed up to under /run and resumes there after a reload. No hook >> ships in this series. A consumer registers one as the example below >> shows. >> >> pve-manager depends on the pve-cluster packages of this series for the >> new modules and the socket, at build time as well, since its tests run >> through the check target. >> >> Example usage: >> >> package PVE::Network::Hooks; >> >> use base qw(PVE::Cluster::Hooks); >> >> __PACKAGE__->register_hook({ >> name => 'guest-firewall', > > ^ This particular example would have to run in the pve-firewall daemon, > not pvescheduler, though. > What about the remaining use cases? oops, brain just skipped this one :P things like sdn apply, or load-balancing that can avoid having to parse all guest configs periodically. i think there is probably a bunch more, things we do periodically are probably good candidates thanks for taking a look! :) > >> path => 'firewall/{vmid}.fw', >> code => sub { >> my ($param, $event) = @_; >> >> my $vmids = $event->{type} eq 'resync' >> ? [] # every guest with a firewall config >> : [ $param->{vmid} ]; >> >> for my $vmid ($vmids->@*) { >> # reload and apply the firewall for guest $vmid >> } >> }, >> }); >> >> 1;