From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id CCF081FF0AA for ; Tue, 17 Nov 2026 19:58:10 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 3BEFE21653; Tue, 17 Nov 2026 19:58:07 +0100 (CET) Content-Type: text/plain; charset=UTF-8 Date: Tue, 17 Nov 2026 19:58:02 +0100 Message-Id: From: =?utf-8?q?Michael_K=C3=B6ppl?= To: "Daniel Kral" , Subject: Re: [PATCH-SERIES ha-manager v2 00/16] some sid parsing and LRM speedup improvements Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Mailer: aerc 0.22.0 References: <20261112163113.294302-3-d.kral@proxmox.com> In-Reply-To: <20261112163113.294302-3-d.kral@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794941882465 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.185 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: IZVH52R4SKE5RUQMDQ7DW2PAZZDKCRBR X-Message-ID-Hash: IZVH52R4SKE5RUQMDQ7DW2PAZZDKCRBR X-MailFrom: m.koeppl@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Had a bit of a closer look at this and did some manual testing and ran the E2E tests on this to check that nothing broke. - Ran through the following scenarios manually, generally just to be sure that everything behaves as I'd expect it to: - Added, read, updated, removed HA resources by plain VM/CT ID. Checked that invalid IDs are rejected. - Initiated migrations through resource affinity rules as well as node affinity rules. - Killed a guest manually to check that it's restarted. - The E2E tests all passed in multiple runs. I attempted to build a small benchmark suite based on your ha-test-scripts. I benchmarked the LRM with 300 started HA resources on one node of a local 3-node test cluster, with no state changes during the measurement and the default max_workers (4). The script does the following 3x for the baseline, 3x for the patched version, in alternating order: - install the package (baseline or patched), restart pve-ha-lrm - wait until all resources are started and let it settle for 5 minutes - measure for 10 minutes: CPU time of the LRM main process and of its reaped workers from /proc//stat, and the number of forked workers with bpftrace - then do the same with the patched pve-ha-manager package Mean (sd) of the 3 runs: baseline patched change main CPU (in secs) 144.2 (8.8) 96.0 (1.5) -33.4% worker CPU (in secs) 296.5 (11.4) 300.9 (2.5) +1.5% total CPU (in secs) 440.7 (20.1) 396.9 (4.0) -9.9% workers forked 16793 (1207) 18141 (32) +8.0% main CPU/worker (in ms) 8.6 (0.2) 5.3 (0.1) -38.4% The main process went from 24% to 16% of a core and its CPU time per worker dropped consistently in every run, which to me seems to be in line with your profiling numbers. Happy to share the script if you want to re-run the measurements. Apart from the issue I mentioned on 12/16, the implementation looks solid to me. IMO the issue on 12/16 is worth fixing, although that could also be a separate commit on top of this series. You mentioned having a fix in separate series, but might make sense to send the fix for this individually if this is applied. Assuming the fix is either included in a v3 or sent as a follow-up, consider this: Reviewed-by: Michael K=C3=B6ppl Tested-by: Michael K=C3=B6ppl On Thu Nov 12, 2026 at 5:30 PM CET, Daniel Kral wrote: > Changes since v1: > - rebase on master > - cleanup the patch messages a bit > - cache the read_service_config() instead of putting it in the 'new' > update_lrm_status() to not change the current behavior as well > unnecessarily reading the HA resource config in idle rounds > - do not prune LRM results in flush_lrm_status() but properly prune > these in update_lrm_status() and resource_command_finished(), because > flush_lrm_status() could be called by a signal handler calling > shutdown_request() > > v1: https://lore.proxmox.com/pve-devel/20260720123705.216089-1-d.kral@pro= xmox.com/ > > It was reported in the enterprise support, that nodes with high counts > of HA resources (250+ HA resources) cause the pve-ha-lrm daemon to > consumes quite a lot of CPU time while being idle. Furthermore, > migration commands for HA resources might take quite a long time to be > actually committed when the max_workers value is not high enough. > > This smaller patch series tries to mitigate the former issue first, > which consists of making some performance improvements for the hot LRM [snip]