Re: [PATCH-SERIES cluster 00/12] addressing large ipc message responses

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

On Mon, 24 Aug 2026 10:55:59 +0200, Daniel Kral wrote:
> Currently, the IPC server can only send MAX_MSG_SIZE (currently, 1 MiB)
> to IPC clients, i.e. users of PVE::IPCC::ipcc_send_recv(). Otherwise,
> the IPC clients will fail with:
> 
>     ipcc_send_rec[13] failed: Transport endpoint is not connected
> 
> and the IPC server will log:
> 
> [...]

Applied, thanks!

I had to fix a few issues on top though that I then fixed up and folded in
(and replaced on commit) as I had it locally already anyway (and missed you
twice in person), the main ones being:

- 11/12 made pmxcfs crash on every API token verification, as the new
  struct with a `char *token` member reads the start of the token as a
  pointer. The handler now uses the request body directly.

- 12/12 was not upgrade safe in either direction. A pmxcfs that is not
  yet restarted read the new request header as part of the body, so
  requests without a body failed and status updates got stored under a
  bogus key. And the new client parsed the old, shorter responses with
  the extended header, which made Perl die with "panic: memory wrap".

- Detecting new clients through a magic value at the start of the
  request body is ambiguous, as a request from an old client can start
  with the same bytes, e.g. a cluster log request with max=48164.

- Pushing all chunks at once made the single IPC worker retry with a
  sleep while the client's ring buffer was full, so a client that stops
  reading blocked all IPC on the node.

- The client gave up after 2 seconds, whereas it previously waited as
  long as pmxcfs is connected. It also retried only after send failures,
  reported negative errno values, and its size limit for old clients was
  16 bytes too strict.

So I basically rewrote 12/12: clients now announce chunk support with a
flag bit in the request id, which an old pmxcfs answers with -ECHRNG
without handling the request, so the client can fall back to plain
requests. And the client fetches each further chunk with its own
request, so the server never waits on a client. As most of the code is
new now, I took over authorship and added you as Co-authored-by.

Separately, testing on a 15 node cluster showed that pmxcfs crashed
whenever a response could not be sent, because it disconnected the
client from within the libqb message callback. That bug predates this
series, and 05/12 already avoids its most common trigger, oversized
responses. I fixed it in a follow-up commit.

[01/12] buildsys: include pmxcfs tidy target in top-level tidy target
        https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=0d6998a5cfa1d6165039c86c31fbfa891bbc8229
[02/12] run make tidy
        superfluous due to recent other cahnges
[03/12] pmxcfs: server: simplify non-negative ipc message result responses
        https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=af840e5f86bbd92b48a425d7637bc2369d8d48a9
[04/12] pmxcfs: server: simplify ipc message response logic
        https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=0e4c64f53efbce58e2b9058cec9a427d9ef51ace
[05/12] pmxcfs: server: report responses exceeding the maximum message size
        https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=8aa1d4be0285984bfd65af16ffca3ef50c2b789f
[06/12] pmxcfs: server: remove unnecessary local variable alignment attributes
        https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=2982458dda26363b7f4f2a3d0c736869673e769c
[07/12] pmxcfs: server: move message handling to separate function
        https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=391e395274fb219ccea37b5fd8f94dce07755ca8
[08/12] pmxcfs: server: move message response sending to separate function
        https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=c4daf3cf26f04d1001b490306ce2e3fb54861aa2
[09/12] pmxcfs: server: simplify message response size calculation
        https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=4d19d09e811588b7f20c2d67c00b8aa7e1827f1c
[10/12] ipcc: move the data pointer initialization up
        https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=87f576fe2bb603169a913531de5ee1cb105dad26
[11/12] pmxcfs: server: do not forward qb_ipc_request_header to s1_msg_handle_request
        https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=f44af776b8067d7db4be1a508017137ff88eb1a8
[12/12] allow large ipc responses by sending the response message in chunks
        https://git.proxmox.com/?p=pve-cluster.git;a=commitdiff;h=8f0871672323457beb5acfebfd10c367e15658c8