From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 69A5B1FF0C0 for ; Wed, 18 Nov 2026 15:45:53 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id C8CFD217F9; Wed, 18 Nov 2026 15:45:24 +0100 (CET) From: Daniel Kral To: pve-devel@lists.proxmox.com Subject: [PATCH-SERIES ha-manager v4 00/17] some sid parsing and LRM speedup improvements Date: Wed, 18 Nov 2026 15:44:45 +0100 Message-ID: <20261118144502.231134-1-d.kral@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1795013118631 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.484 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: QHSDGW6IPCN3FO2EUJVR4GFQQ2QJKKGS X-Message-ID-Hash: QHSDGW6IPCN3FO2EUJVR4GFQQ2QJKKGS X-MailFrom: d.kral@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Changes since v3: - rebase on master - drop R-b and T-b from @Michael from patch #8 (now also includes the new parse_sid() uses Fiona's JSON migrate schema introduced) Changes since v2 (thanks a lot @Michael!): - add R-b and T-b from @Michael to patches except #2 and #9 - drop s/$name/$vmid/ from PVE::HA::LRM and PVE::HA::Resources in #2 - new patch #9: use an eval around the maintenance work branch, similar to what is catched in the active work branch Before it gets unnoticed: A user reported some results for v1 of this patch series, which shows a test on another larger setup [0]. Changes since v1: - rebase on master - cleanup the patch messages a bit - cache the read_service_config() instead of putting it in the 'new' update_lrm_status() to not change the current behavior as well unnecessarily reading the HA resource config in idle rounds - do not prune LRM results in flush_lrm_status() but properly prune these in update_lrm_status() and resource_command_finished(), because flush_lrm_status() could be called by a signal handler calling shutdown_request() v1: https://lore.proxmox.com/pve-devel/20260720123705.216089-1-d.kral@proxmox.com/ v2: https://lore.proxmox.com/pve-devel/20261112163113.294302-3-d.kral@proxmox.com/ v3: https://lore.proxmox.com/pve-devel/20261118142556.218589-1-d.kral@proxmox.com/ It was reported in the enterprise support, that nodes with high counts of HA resources (250+ HA resources) cause the pve-ha-lrm daemon to consumes quite a lot of CPU time while being idle. Furthermore, migration commands for HA resources might take quite a long time to be actually committed when the max_workers value is not high enough. This smaller patch series tries to mitigate the former issue first, which consists of making some performance improvements for the hot LRM code paths, which consumed the most resources: - the service config is read and validated very often - because of that, parse_sid() is called many times as well - other smaller loops are redundant and are moved out of the hot paths I started out with making parse_sid() a little lighter, though because of the latter changes the performance bottleneck of parse_sid() became rather meaningless. I left the refactoring and changes in though as having a stricter separation where both VMIDs and SIDs can be used and where only SIDs can be used is a little nicer now. The patch series consists of: PATCH 1-2 small cleanups PATCH 3-8 refactoring and splitting up parse_sid() and its users PATCH 9 eval block around maintenance work branch PATCH 10-12 prep work to add documentation and rename some methods PATCH 13 reading the manager status only once per work iteration PATCH 14 reading the service config only once per work iteration PATCH 15-17 factoring out loops which are invariant per work iteration A test startup of 768 HA resources on a single physical host at the same time, profiling with Devel::NYTProf 6.14 [0] showed the following: +---------------------------+---------------+--------------+ | Subroutine | Before Change | After Change | +---------------------------+---------------+--------------+ | check_active_workers | 327ms | 998µs | | resource_command_finished | 22.7ms | 50µs | | handle_service_exitcode | 21.1ms | 12µs | +---------------------------+---------------+--------------+ Considering that these are called quite often per LRM cycle, these drops in execution time per call are significant and reduce how much CPU% the pve-ha-lrm main thread uses. [0] https://bugzilla.proxmox.com/show_bug.cgi?id=8233 Daniel Kral (17): resources: remove commented ipaddr resource type tree-wide: use the term vmid instead of name when referring to VMs/CTs tree-wide: remove unused return values at parse_sid callsites config: make update_single_resource_config_inplace only allow sids make parse_sid always return an array config: use early returns in parse_sid config: introduce separate parse_vmid_or_sid helper subroutine drop the unmodified sid return array entry value from parse_sid lrm: catch and report dies while the LRM is in maintenance state lrm: rename update_lrm_status to flush_lrm_status lrm: rename update_service_status to update_lrm_status lrm: document and initialize LRM instance properties lrm: update the service status once per work iteration lrm: read service config once per work iteration lrm: compute valid service uids hash set once per work iteration lrm: resource_command_finished: avoid writing stale LRM results lrm: prune stale LRM results once per work iteration src/PVE/API2/HA/Resources.pm | 19 ++-- src/PVE/CLI/ha_manager.pm | 6 +- src/PVE/HA/Config.pm | 42 +++---- src/PVE/HA/Env/PVE2.pm | 6 +- src/PVE/HA/LRM.pm | 128 ++++++++++++++-------- src/PVE/HA/Manager.pm | 6 +- src/PVE/HA/Resources.pm | 26 ----- src/PVE/HA/Resources/PVECT.pm | 4 +- src/PVE/HA/Resources/PVEVM.pm | 4 +- src/PVE/HA/Sim/Env.pm | 4 +- src/test/test-cfs-unavailable1/log.expect | 5 - src/test/test-group-migrate4/log.expect | 10 -- 12 files changed, 132 insertions(+), 128 deletions(-) -- 2.47.3