From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id ADA0E1FF0A3 for ; Thu, 01 Oct 2026 09:43:43 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 8D3E821640; Thu, 01 Oct 2026 09:43:41 +0200 (CEST) Message-ID: <6de5f1fc-f769-4fbc-abdf-bbd96b74abaa@proxmox.com> Date: Thu, 1 Oct 2026 09:43:36 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC cluster/manager 00/10] pmxcfs: add a change notification socket To: Wolfgang Bumiller References: <20260918144152.575163-1-h.laimer@proxmox.com> From: Hannes Laimer Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1790840616919 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.479 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: M34TGUYNOZ7DN4XUQH32KWAMLY3KEJYT X-Message-ID-Hash: M34TGUYNOZ7DN4XUQH32KWAMLY3KEJYT X-MailFrom: h.laimer@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: pve-devel@lists.proxmox.com X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 2026-10-01 09:31, Wolfgang Bumiller wrote: > On Fri, Sep 18, 2026 at 04:41:42PM +0200, Hannes Laimer wrote: >> Every daemon that cares about /etc/pve learns about changes by calling >> cfs_update on each loop iteration and comparing the version vector it >> gets over the libqb IPC. inotify cannot replace that, since remote >> changes arrive through corosync and never touch the local VFS, so >> pvestatd, pve-firewall, the HA daemons and pvescheduler all wake up on >> timers and re-read what usually has not changed. >> >> This series adds a push path. pmxcfs gets a Unix stream socket at >> /run/pve-cluster/pmxcfs.sock, served by a Rust thread linked into the C >> daemon as a static library with a small C ABI. A client subscribes with >> named path templates such as nodes/{node}/qemu-server/{vmid}.conf and >> receives one JSON line per matching mutation, carrying the memdb version >> as sequence number, the event type, the path and the values the >> placeholders captured. Connections authorized by group membership never >> see private paths, as on the IPC and FUSE side. >> >> The daemon keeps the last mutations in a fixed size ring and each >> connection a cursor into it, so the mutating thread only appends and >> never waits for a client. A slow client is caught up from the ring, one >> that fell off it gets a single resync event, and a reconnecting client >> resumes at the last sequence number it saw, also across a restart of >> pmxcfs when nothing changed meanwhile. A node that takes the whole >> state from the cluster after a membership change hands its clients a >> resync event, since no sequence of mutations describes that. A panic in > > So a high level question: > > Do we actually have any use case where this particular way processing > changes on the *client* side would give us any *real* benefit? > > Given the complexity and custom non-blocking handling if socket states, > I'd like to see a real need for this. > > Why isn't a simple async-based server with no sequence numbers where a > time-out just disconnects the client, and the disconnect itself being > the equivalent of a resync event, sufficient? tbf, that would very likely be more than fine. especially because that realistically shouldn't really happen a lot. the main use-case I had in mind was having to expensively parse all guest configs periodically for load-balancing, with something like this that could be avoided. and with that in mind i wanted to minimize how often that has to happen. but yes, given the complexity that adds, and that disconnects or alike realistically happen only very rarely, the added complexity is not a really good tradeoff. > > The server would be just the accept loop which `spawn()`s a > client-handler with its socket and event channel, the send-side of the > channel pushed to the list of client channels. The client handling task > just loops through a channel.recv() followed by a socket.send() with a > timeout. On timeout it just returns, the channel gets closed, the next > event-emit sees the channel.send() fail and removes the channel. this sounds like a good alternative approach. i'll prepare a v1 with this. thanks a lot for taking a look, and the feedback on this! :)