From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 05ED61FF0AF for ; Fri, 23 Oct 2026 18:12:19 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 38D482161A; Fri, 23 Oct 2026 18:12:18 +0200 (CEST) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com, Daniel Kral Subject: Re: [PATCH-SERIES cluster 00/12] addressing large ipc message responses Date: Fri, 23 Oct 2026 17:45:17 +0200 Message-ID: <179277014467.3697247.5808391556729339767.b4-ty@b4> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260824085610.111211-2-d.kral@proxmox.com> References: <20260824085610.111211-2-d.kral@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1792771933189 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.376 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_LOTSOFHASH 0.25 Emails with lots of hash-like gibberish RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: MDXO5GS4TIUK7ULZV7IWLVFA7VGSQTGV X-Message-ID-Hash: MDXO5GS4TIUK7ULZV7IWLVFA7VGSQTGV X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Mon, 24 Aug 2026 10:55:59 +0200, Daniel Kral wrote: > Currently, the IPC server can only send MAX_MSG_SIZE (currently, 1 MiB) > to IPC clients, i.e. users of PVE::IPCC::ipcc_send_recv(). Otherwise, > the IPC clients will fail with: > > ipcc_send_rec[13] failed: Transport endpoint is not connected > > and the IPC server will log: > > [...] Applied, thanks! I had to fix a few issues on top though that I then fixed up and folded in (and replaced on commit) as I had it locally already anyway (and missed you twice in person), the main ones being: - 11/12 made pmxcfs crash on every API token verification, as the new struct with a `char *token` member reads the start of the token as a pointer. The handler now uses the request body directly. - 12/12 was not upgrade safe in either direction. A pmxcfs that is not yet restarted read the new request header as part of the body, so requests without a body failed and status updates got stored under a bogus key. And the new client parsed the old, shorter responses with the extended header, which made Perl die with "panic: memory wrap". - Detecting new clients through a magic value at the start of the request body is ambiguous, as a request from an old client can start with the same bytes, e.g. a cluster log request with max=48164. - Pushing all chunks at once made the single IPC worker retry with a sleep while the client's ring buffer was full, so a client that stops reading blocked all IPC on the node. - The client gave up after 2 seconds, whereas it previously waited as long as pmxcfs is connected. It also retried only after send failures, reported negative errno values, and its size limit for old clients was 16 bytes too strict. So I basically rewrote 12/12: clients now announce chunk support with a flag bit in the request id, which an old pmxcfs answers with -ECHRNG without handling the request, so the client can fall back to plain requests. And the client fetches each further chunk with its own request, so the server never waits on a client. As most of the code is new now, I took over authorship and added you as Co-authored-by. Separately, testing on a 15 node cluster showed that pmxcfs crashed whenever a response could not be sent, because it disconnected the client from within the libqb message callback. That bug predates this series, and 05/12 already avoids its most common trigger, oversized responses. I fixed it in a follow-up commit. [01/12] buildsys: include pmxcfs tidy target in top-level tidy target https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=0d6998a5cfa1d6165039c86c31fbfa891bbc8229 [02/12] run make tidy superfluous due to recent other cahnges [03/12] pmxcfs: server: simplify non-negative ipc message result responses https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=af840e5f86bbd92b48a425d7637bc2369d8d48a9 [04/12] pmxcfs: server: simplify ipc message response logic https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=0e4c64f53efbce58e2b9058cec9a427d9ef51ace [05/12] pmxcfs: server: report responses exceeding the maximum message size https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=8aa1d4be0285984bfd65af16ffca3ef50c2b789f [06/12] pmxcfs: server: remove unnecessary local variable alignment attributes https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=2982458dda26363b7f4f2a3d0c736869673e769c [07/12] pmxcfs: server: move message handling to separate function https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=391e395274fb219ccea37b5fd8f94dce07755ca8 [08/12] pmxcfs: server: move message response sending to separate function https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=c4daf3cf26f04d1001b490306ce2e3fb54861aa2 [09/12] pmxcfs: server: simplify message response size calculation https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=4d19d09e811588b7f20c2d67c00b8aa7e1827f1c [10/12] ipcc: move the data pointer initialization up https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=87f576fe2bb603169a913531de5ee1cb105dad26 [11/12] pmxcfs: server: do not forward qb_ipc_request_header to s1_msg_handle_request https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=f44af776b8067d7db4be1a508017137ff88eb1a8 [12/12] allow large ipc responses by sending the response message in chunks https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=8f0871672323457beb5acfebfd10c367e15658c8