From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 2F2EB1FF0B2 for ; Mon, 24 Aug 2026 10:58:52 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 02F302173C; Mon, 24 Aug 2026 10:57:25 +0200 (CEST) From: Daniel Kral To: pve-devel@lists.proxmox.com Subject: [PATCH-SERIES cluster 00/12] addressing large ipc message responses Date: Mon, 24 Aug 2026 10:55:59 +0200 Message-ID: <20260824085610.111211-2-d.kral@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1787561805954 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.973 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: NE6ONTGF5AFIGFQ6FGM5D6X5GVYQMF25 X-Message-ID-Hash: NE6ONTGF5AFIGFQ6FGM5D6X5GVYQMF25 X-MailFrom: d.kral@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Currently, the IPC server can only send MAX_MSG_SIZE (currently, 1 MiB) to IPC clients, i.e. users of PVE::IPCC::ipcc_send_recv(). Otherwise, the IPC clients will fail with: ipcc_send_rec[13] failed: Transport endpoint is not connected and the IPC server will log: [ipcs] crit: qb_ipcs_response_send: Resource temporarily unavailable [libqb] error: error receiving from setup sock (/dev/shm/qb-[...]/qb): Bad file descriptor (9) I've reached the MAX_MSG_SIZE using CFS_IPC_GET_GUEST_CONFIG_PROPERTIES while working on a patch series, which intends the pve-ha-crm to be able to infer certain node affinity constraints from the guest configs, e.g. guests being limited to certain nodes by their storages and devices. While testing it by only gathering the 'efidisk0' property from 10,000 VMs with identical efidisk property strings (except the 5-digit vmid): efidisk0: local:10000/vm-10000-disk-0.qcow2,efitype=4m,format=qcow2,ms-cert=2023k,pre-enrolled-keys=1,size=1M the MAX_MSG_SIZE was already exceeded (1250253 bytes > 1048576 bytes). As noted in the patch message in patch #5, some of the current responses are already quite large and could be exceeded in large clusters: For example, in a test cluster with 10,000 stopped guests, the response of CFS_IPC_GET_RRD_DUMP does already have 919 KiB, where many of the runtime values are undefined (U) and therefore the response size would be much longer if they were all running. Similar response sizes can be reached with CFS_IPC_GET_GUEST_LIST and CFS_IPC_GET_GUEST_CONFIG_PROPERTIES. The first few non-RFC patches of this series do some cleanup and report such occurrences of large IPC response messages: PATCH 1-2 make root `make tidy` run pmxcfs' tidy target PATCH 3-4 some prepatory cleanup PATCH 5 report IPC responses exceeding MAX_MSG_SIZE PATCH 6-10 some after cleanup (prepatory for #11-#12) PATCH 11 preparation for RFC patch #12 The RFC patch #12 implements a simple mechanism to split these large IPC messages in chunks, send them to the client and make the client reconstruct these message chunks to the original IPC response. This solution uses an additional request header and an additional response header to communicate whether the client can handle multiple message chunk responses and when to stop the client to receive messages. See more information about the design in patch #12. Daniel Kral (12): buildsys: include pmxcfs tidy target in top-level tidy target run make tidy pmxcfs: server: simplify non-negative ipc message result responses pmxcfs: server: simplify ipc message response logic pmxcfs: server: report responses exceeding the maximum message size pmxcfs: server: remove unnecessary local variable alignment attributes pmxcfs: server: move message handling to separate function pmxcfs: server: move message response sending to separate function pmxcfs: server: simplify message response size calculation ipcc: move the data pointer initialization up pmxcfs: server: do not forward qb_ipc_request_header to s1_msg_handle_request allow large ipc responses by sending the response message in chunks Makefile | 1 + src/Makefile | 3 + src/PVE/IPCC.xs | 89 +++++++++++++----- src/PVE/Makefile | 1 + src/pmxcfs/cfs-ipc-common.h | 56 ++++++++++++ src/pmxcfs/server.c | 175 +++++++++++++++++++++++++----------- src/pmxcfs/status.c | 2 +- 7 files changed, 254 insertions(+), 73 deletions(-) create mode 100644 src/pmxcfs/cfs-ipc-common.h -- 2.47.3