[PATCH-SERIES cluster 00/12] addressing large ipc message responses

[Daniel Kral <d.kral@proxmox.com>]

Currently, the IPC server can only send MAX_MSG_SIZE (currently, 1 MiB)
to IPC clients, i.e. users of PVE::IPCC::ipcc_send_recv(). Otherwise,
the IPC clients will fail with:

    ipcc_send_rec[13] failed: Transport endpoint is not connected

and the IPC server will log:

    [ipcs] crit: qb_ipcs_response_send: Resource temporarily unavailable
    [libqb] error: error receiving from setup sock (/dev/shm/qb-[...]/qb): Bad file descriptor (9)


I've reached the MAX_MSG_SIZE using CFS_IPC_GET_GUEST_CONFIG_PROPERTIES
while working on a patch series, which intends the pve-ha-crm to be able
to infer certain node affinity constraints from the guest configs, e.g.
guests being limited to certain nodes by their storages and devices.

While testing it by only gathering the 'efidisk0' property from 10,000
VMs with identical efidisk property strings (except the 5-digit vmid):

    efidisk0: local:10000/vm-10000-disk-0.qcow2,efitype=4m,format=qcow2,ms-cert=2023k,pre-enrolled-keys=1,size=1M

the MAX_MSG_SIZE was already exceeded (1250253 bytes > 1048576 bytes).

As noted in the patch message in patch #5, some of the current responses
are already quite large and could be exceeded in large clusters:

    For example, in a test cluster with 10,000 stopped guests, the
    response of CFS_IPC_GET_RRD_DUMP does already have 919 KiB, where
    many of the runtime values are undefined (U) and therefore the
    response size would be much longer if they were all running. Similar
    response sizes can be reached with CFS_IPC_GET_GUEST_LIST and
    CFS_IPC_GET_GUEST_CONFIG_PROPERTIES.


The first few non-RFC patches of this series do some cleanup and report
such occurrences of large IPC response messages:

PATCH 1-2   make root `make tidy` run pmxcfs' tidy target
PATCH 3-4   some prepatory cleanup
PATCH 5     report IPC responses exceeding MAX_MSG_SIZE
PATCH 6-10  some after cleanup (prepatory for #11-#12)
PATCH 11    preparation for RFC patch #12


The RFC patch #12 implements a simple mechanism to split these large IPC
messages in chunks, send them to the client and make the client
reconstruct these message chunks to the original IPC response.

This solution uses an additional request header and an additional
response header to communicate whether the client can handle multiple
message chunk responses and when to stop the client to receive messages.
See more information about the design in patch #12.



Daniel Kral (12):
  buildsys: include pmxcfs tidy target in top-level tidy target
  run make tidy
  pmxcfs: server: simplify non-negative ipc message result responses
  pmxcfs: server: simplify ipc message response logic
  pmxcfs: server: report responses exceeding the maximum message size
  pmxcfs: server: remove unnecessary local variable alignment attributes
  pmxcfs: server: move message handling to separate function
  pmxcfs: server: move message response sending to separate function
  pmxcfs: server: simplify message response size calculation
  ipcc: move the data pointer initialization up
  pmxcfs: server: do not forward qb_ipc_request_header to
    s1_msg_handle_request
  allow large ipc responses by sending the response message in chunks

 Makefile                    |   1 +
 src/Makefile                |   3 +
 src/PVE/IPCC.xs             |  89 +++++++++++++-----
 src/PVE/Makefile            |   1 +
 src/pmxcfs/cfs-ipc-common.h |  56 ++++++++++++
 src/pmxcfs/server.c         | 175 +++++++++++++++++++++++++-----------
 src/pmxcfs/status.c         |   2 +-
 7 files changed, 254 insertions(+), 73 deletions(-)
 create mode 100644 src/pmxcfs/cfs-ipc-common.h

-- 
2.47.3