public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH-SERIES ha-manager v4 00/17] some sid parsing and LRM speedup improvements
@ 2026-11-18 14:44 Daniel Kral
  2026-11-18 14:44 ` [PATCH ha-manager v4 01/17] resources: remove commented ipaddr resource type Daniel Kral
                   ` (17 more replies)
  0 siblings, 18 replies; 22+ messages in thread
From: Daniel Kral @ 2026-11-18 14:44 UTC (permalink / raw)
  To: pve-devel

Changes since v3:
- rebase on master
- drop R-b and T-b from @Michael from patch #8 (now also includes the
  new parse_sid() uses Fiona's JSON migrate schema introduced)

Changes since v2 (thanks a lot @Michael!):
- add R-b and T-b from @Michael to patches except #2 and #9
- drop s/$name/$vmid/ from PVE::HA::LRM and PVE::HA::Resources in #2
- new patch #9: use an eval around the maintenance work branch, similar
  to what is catched in the active work branch

Before it gets unnoticed: A user reported some results for v1 of this
patch series, which shows a test on another larger setup [0].

Changes since v1:
- rebase on master
- cleanup the patch messages a bit
- cache the read_service_config() instead of putting it in the 'new'
  update_lrm_status() to not change the current behavior as well
  unnecessarily reading the HA resource config in idle rounds
- do not prune LRM results in flush_lrm_status() but properly prune
  these in update_lrm_status() and resource_command_finished(), because
  flush_lrm_status() could be called by a signal handler calling
  shutdown_request()

v1: https://lore.proxmox.com/pve-devel/20260720123705.216089-1-d.kral@proxmox.com/
v2: https://lore.proxmox.com/pve-devel/20261112163113.294302-3-d.kral@proxmox.com/
v3: https://lore.proxmox.com/pve-devel/20261118142556.218589-1-d.kral@proxmox.com/

It was reported in the enterprise support, that nodes with high counts
of HA resources (250+ HA resources) cause the pve-ha-lrm daemon to
consumes quite a lot of CPU time while being idle. Furthermore,
migration commands for HA resources might take quite a long time to be
actually committed when the max_workers value is not high enough.

This smaller patch series tries to mitigate the former issue first,
which consists of making some performance improvements for the hot LRM
code paths, which consumed the most resources:

- the service config is read and validated very often
- because of that, parse_sid() is called many times as well
- other smaller loops are redundant and are moved out of the hot paths

I started out with making parse_sid() a little lighter, though because
of the latter changes the performance bottleneck of parse_sid() became
rather meaningless. I left the refactoring and changes in though as
having a stricter separation where both VMIDs and SIDs can be used and
where only SIDs can be used is a little nicer now.


The patch series consists of:

PATCH 1-2   small cleanups
PATCH 3-8   refactoring and splitting up parse_sid() and its users
PATCH 9     eval block around maintenance work branch
PATCH 10-12 prep work to add documentation and rename some methods
PATCH 13    reading the manager status only once per work iteration
PATCH 14    reading the service config only once per work iteration
PATCH 15-17 factoring out loops which are invariant per work iteration


A test startup of 768 HA resources on a single physical host at the same
time, profiling with Devel::NYTProf 6.14 [0] showed the following:

+---------------------------+---------------+--------------+
|        Subroutine         | Before Change | After Change |
+---------------------------+---------------+--------------+
| check_active_workers      | 327ms         | 998µs        |
| resource_command_finished | 22.7ms        | 50µs         |
| handle_service_exitcode   | 21.1ms        | 12µs         |
+---------------------------+---------------+--------------+

Considering that these are called quite often per LRM cycle, these drops
in execution time per call are significant and reduce how much CPU% the
pve-ha-lrm main thread uses.


[0] https://bugzilla.proxmox.com/show_bug.cgi?id=8233

Daniel Kral (17):
  resources: remove commented ipaddr resource type
  tree-wide: use the term vmid instead of name when referring to VMs/CTs
  tree-wide: remove unused return values at parse_sid callsites
  config: make update_single_resource_config_inplace only allow sids
  make parse_sid always return an array
  config: use early returns in parse_sid
  config: introduce separate parse_vmid_or_sid helper subroutine
  drop the unmodified sid return array entry value from parse_sid
  lrm: catch and report dies while the LRM is in maintenance state
  lrm: rename update_lrm_status to flush_lrm_status
  lrm: rename update_service_status to update_lrm_status
  lrm: document and initialize LRM instance properties
  lrm: update the service status once per work iteration
  lrm: read service config once per work iteration
  lrm: compute valid service uids hash set once per work iteration
  lrm: resource_command_finished: avoid writing stale LRM results
  lrm: prune stale LRM results once per work iteration

 src/PVE/API2/HA/Resources.pm              |  19 ++--
 src/PVE/CLI/ha_manager.pm                 |   6 +-
 src/PVE/HA/Config.pm                      |  42 +++----
 src/PVE/HA/Env/PVE2.pm                    |   6 +-
 src/PVE/HA/LRM.pm                         | 128 ++++++++++++++--------
 src/PVE/HA/Manager.pm                     |   6 +-
 src/PVE/HA/Resources.pm                   |  26 -----
 src/PVE/HA/Resources/PVECT.pm             |   4 +-
 src/PVE/HA/Resources/PVEVM.pm             |   4 +-
 src/PVE/HA/Sim/Env.pm                     |   4 +-
 src/test/test-cfs-unavailable1/log.expect |   5 -
 src/test/test-group-migrate4/log.expect   |  10 --
 12 files changed, 132 insertions(+), 128 deletions(-)

-- 
2.47.3





^ permalink raw reply	[flat|nested] 22+ messages in thread

end of thread, other threads:[~2026-11-18 20:40 UTC | newest]

Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 14:44 [PATCH-SERIES ha-manager v4 00/17] some sid parsing and LRM speedup improvements Daniel Kral
2026-11-18 14:44 ` [PATCH ha-manager v4 01/17] resources: remove commented ipaddr resource type Daniel Kral
2026-11-18 14:44 ` [PATCH ha-manager v4 02/17] tree-wide: use the term vmid instead of name when referring to VMs/CTs Daniel Kral
2026-11-18 14:44 ` [PATCH ha-manager v4 03/17] tree-wide: remove unused return values at parse_sid callsites Daniel Kral
2026-11-18 15:20   ` Michael Köppl
2026-11-18 14:44 ` [PATCH ha-manager v4 04/17] config: make update_single_resource_config_inplace only allow sids Daniel Kral
2026-11-18 14:44 ` [PATCH ha-manager v4 05/17] make parse_sid always return an array Daniel Kral
2026-11-18 14:44 ` [PATCH ha-manager v4 06/17] config: use early returns in parse_sid Daniel Kral
2026-11-18 14:44 ` [PATCH ha-manager v4 07/17] config: introduce separate parse_vmid_or_sid helper subroutine Daniel Kral
2026-11-18 16:02   ` Michael Köppl
2026-11-18 14:44 ` [PATCH ha-manager v4 08/17] drop the unmodified sid return array entry value from parse_sid Daniel Kral
2026-11-18 14:44 ` [PATCH ha-manager v4 09/17] lrm: catch and report dies while the LRM is in maintenance state Daniel Kral
2026-11-18 16:12   ` Michael Köppl
2026-11-18 14:44 ` [PATCH ha-manager v4 10/17] lrm: rename update_lrm_status to flush_lrm_status Daniel Kral
2026-11-18 14:44 ` [PATCH ha-manager v4 11/17] lrm: rename update_service_status to update_lrm_status Daniel Kral
2026-11-18 14:44 ` [PATCH ha-manager v4 12/17] lrm: document and initialize LRM instance properties Daniel Kral
2026-11-18 14:44 ` [PATCH ha-manager v4 13/17] lrm: update the service status once per work iteration Daniel Kral
2026-11-18 14:44 ` [PATCH ha-manager v4 14/17] lrm: read service config " Daniel Kral
2026-11-18 14:45 ` [PATCH ha-manager v4 15/17] lrm: compute valid service uids hash set " Daniel Kral
2026-11-18 14:45 ` [PATCH ha-manager v4 16/17] lrm: resource_command_finished: avoid writing stale LRM results Daniel Kral
2026-11-18 14:45 ` [PATCH ha-manager v4 17/17] lrm: prune stale LRM results once per work iteration Daniel Kral
2026-11-18 20:39 ` applied: [PATCH-SERIES ha-manager v4 00/17] some sid parsing and LRM speedup improvements Thomas Lamprecht

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal