From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id B998A1FF0B2 for ; Fri, 25 Sep 2026 14:48:11 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 4B50B21707; Fri, 25 Sep 2026 14:48:10 +0200 (CEST) Message-ID: <337d9d70-a6c1-459f-80e9-ed0c3fcec886@proxmox.com> Date: Fri, 25 Sep 2026 14:48:06 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC cluster/manager 00/10] pmxcfs: add a change notification socket To: Wolfgang Bumiller References: <20260918144152.575163-1-h.laimer@proxmox.com> <4kp2pm6gasdbailmstjwxrvsc3ozmpiln4tz67pjh4ii3od2eg@ec5poxqld7gh> From: Hannes Laimer Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1790340486700 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.472 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: DXGD2BZN5HQWWC7UW4RSMXJ6OXNUK23I X-Message-ID-Hash: DXGD2BZN5HQWWC7UW4RSMXJ6OXNUK23I X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pve-devel@lists.proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 2026-09-25 14:43, Wolfgang Bumiller wrote: > On Fri, Sep 25, 2026 at 02:23:53PM +0200, Hannes Laimer wrote: >> On 2026-09-25 14:01, Wolfgang Bumiller wrote: >>> On Fri, Sep 18, 2026 at 04:41:42PM +0200, Hannes Laimer wrote: >>>> Every daemon that cares about /etc/pve learns about changes by calling >>>> cfs_update on each loop iteration and comparing the version vector it >>>> gets over the libqb IPC. inotify cannot replace that, since remote >>> >>> Not strictly true - pmxcfs could add inotify support vis fuse, but it is >>> limited to regular files and would not support whole directory watches. >>> >>>> changes arrive through corosync and never touch the local VFS, so >>>> pvestatd, pve-firewall, the HA daemons and pvescheduler all wake up on >>>> timers and re-read what usually has not changed. >>>> >>>> This series adds a push path. pmxcfs gets a Unix stream socket at >>>> /run/pve-cluster/pmxcfs.sock, served by a Rust thread linked into the C >>>> daemon as a static library with a small C ABI. A client subscribes with >>>> named path templates such as nodes/{node}/qemu-server/{vmid}.conf and >>>> receives one JSON line per matching mutation, carrying the memdb version >>>> as sequence number, the event type, the path and the values the >>>> placeholders captured. Connections authorized by group membership never >>>> see private paths, as on the IPC and FUSE side. >>>> >>>> The daemon keeps the last mutations in a fixed size ring and each >>>> connection a cursor into it, so the mutating thread only appends and >>>> never waits for a client. A slow client is caught up from the ring, one >>>> that fell off it gets a single resync event, and a reconnecting client >>>> resumes at the last sequence number it saw, also across a restart of >>>> pmxcfs when nothing changed meanwhile. A node that takes the whole >>>> state from the cluster after a membership change hands its clients a >>>> resync event, since no sequence of mutations describes that. A panic in >>>> the Rust code only disables the notifier for the rest of the daemon's >>>> life. A sequence number identifies a state within one line of history >>>> only. A client that missed a resync while disconnected and returns at >>>> the same version after a pmxcfs restart resumes as if nothing changed. >>>> Carrying the root entry's mtime next to the version would close that. >>>> >>>> On top of the socket, pve-cluster ships a Perl client with reconnect >>>> and resync handling and a hook registry, where a hook is registered >>>> like an API method with a path template whose placeholders become the >>>> parameters of a run. pve-manager gets the runner that executes those >>>> runs in children of a listener hosted by pvescheduler, one run per hook >>>> and parameter set in flight and later events for the same set collapsed >>>> into a single rerun, so a config update through the API costs one extra >>>> run rather than one per save. The listener records the position it has >>>> processed up to under /run and resumes there after a reload. No hook >>>> ships in this series. A consumer registers one as the example below >>>> shows. >>>> >>>> pve-manager depends on the pve-cluster packages of this series for the >>>> new modules and the socket, at build time as well, since its tests run >>>> through the check target. >>>> >>>> Example usage: >>>> >>>> package PVE::Network::Hooks; >>>> >>>> use base qw(PVE::Cluster::Hooks); >>>> >>>> __PACKAGE__->register_hook({ >>>> name => 'guest-firewall', >>> >>> ^ This particular example would have to run in the pve-firewall daemon, >>> not pvescheduler, though. >>> What about the remaining use cases? >> >> oops, brain just skipped this one :P >> >> things like sdn apply, or load-balancing that can avoid having to parse >> all guest configs periodically. i think there is probably a bunch more, >> things we do periodically are probably good candidates > > What I meant is, do those all make sense in one big does-it-all > pvescheduler daemon? > It seems to be meant for "jobs", in the sense of backup jobs etc., so > I'm not convinced moving every *event* based thing into it - or any one > giant does-it-all service - is the right way here. > I'd expect the SDN code to also move further into rust in a not too > distant future. agree, no hard feelings either way. just chose this cause there are domains without their own daemon, and this one seemed general enough