all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: "Michael Köppl" <m.koeppl@proxmox.com>
To: "Daniel Kral" <d.kral@proxmox.com>, <pve-devel@lists.proxmox.com>
Subject: Re: [PATCH-SERIES ha-manager v2 00/16] some sid parsing and LRM speedup improvements
Date: Tue, 17 Nov 2026 19:58:02 +0100	[thread overview]
Message-ID: <DMXPSL7YY5CD.2Z008K4DT4FDF@proxmox.com> (raw)
In-Reply-To: <20261112163113.294302-3-d.kral@proxmox.com>

Had a bit of a closer look at this and did some manual testing and ran
the E2E tests on this to check that nothing broke.

- Ran through the following scenarios manually, generally just to be
  sure that everything behaves as I'd expect it to:
  - Added, read, updated, removed HA resources by plain VM/CT ID.
    Checked that invalid IDs are rejected.
  - Initiated migrations through resource affinity rules as well as
    node affinity rules.
  - Killed a guest manually to check that it's restarted.
- The E2E tests all passed in multiple runs.

I attempted to build a small benchmark suite based on your
ha-test-scripts. I benchmarked the LRM with 300 started HA resources on
one node of a local 3-node test cluster, with no state changes during
the measurement and the default max_workers (4). The script does the
following 3x for the baseline, 3x for the patched version, in
alternating order:

- install the package (baseline or patched), restart pve-ha-lrm
- wait until all resources are started and let it settle for 5 minutes
- measure for 10 minutes: CPU time of the LRM main process and of its
  reaped workers from /proc/<pid>/stat, and the number of forked
  workers with bpftrace
- then do the same with the patched pve-ha-manager package

Mean (sd) of the 3 runs:
                            baseline       patched        change
  main CPU (in secs)        144.2 (8.8)    96.0 (1.5)     -33.4%
  worker CPU (in secs)      296.5 (11.4)   300.9 (2.5)     +1.5%
  total CPU (in secs)       440.7 (20.1)   396.9 (4.0)     -9.9%
  workers forked            16793 (1207)   18141 (32)      +8.0%
  main CPU/worker (in ms)   8.6 (0.2)      5.3 (0.1)      -38.4%

The main process went from 24% to 16% of a core and its CPU time per
worker dropped consistently in every run, which to me seems to be in
line with your profiling numbers. Happy to share the script if you want
to re-run the measurements.

Apart from the issue I mentioned on 12/16, the implementation looks
solid to me. IMO the issue on 12/16 is worth fixing, although that
could also be a separate commit on top of this series. You mentioned
having a fix in separate series, but might make sense to send the fix
for this individually if this is applied. Assuming the fix is either
included in a v3 or sent as a follow-up, consider this:
Reviewed-by: Michael Köppl <m.koeppl@proxmox.com>

Tested-by: Michael Köppl <m.koeppl@proxmox.com>

On Thu Nov 12, 2026 at 5:30 PM CET, Daniel Kral wrote:
> Changes since v1:
> - rebase on master
> - cleanup the patch messages a bit
> - cache the read_service_config() instead of putting it in the 'new'
>   update_lrm_status() to not change the current behavior as well
>   unnecessarily reading the HA resource config in idle rounds
> - do not prune LRM results in flush_lrm_status() but properly prune
>   these in update_lrm_status() and resource_command_finished(), because
>   flush_lrm_status() could be called by a signal handler calling
>   shutdown_request()
>
> v1: https://lore.proxmox.com/pve-devel/20260720123705.216089-1-d.kral@proxmox.com/
>
> It was reported in the enterprise support, that nodes with high counts
> of HA resources (250+ HA resources) cause the pve-ha-lrm daemon to
> consumes quite a lot of CPU time while being idle. Furthermore,
> migration commands for HA resources might take quite a long time to be
> actually committed when the max_workers value is not high enough.
>
> This smaller patch series tries to mitigate the former issue first,
> which consists of making some performance improvements for the hot LRM

[snip]




  parent reply	other threads:[~2026-11-17 18:58 UTC|newest]

Thread overview: 23+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12 16:30 [PATCH-SERIES ha-manager v2 00/16] some sid parsing and LRM speedup improvements Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 01/16] resources: remove commented ipaddr resource type Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 02/16] tree-wide: use the term vmid instead of name when referring to VMs/CTs Daniel Kral
2026-11-12 22:33   ` Thomas Lamprecht
2026-11-13 15:04     ` Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 03/16] tree-wide: remove unused return values at parse_sid callsites Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 04/16] config: make update_single_resource_config_inplace only allow sids Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 05/16] make parse_sid always return an array Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 06/16] config: use early returns in parse_sid Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 07/16] config: introduce separate parse_vmid_or_sid helper subroutine Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 08/16] drop the unmodified sid return array entry value from parse_sid Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 09/16] lrm: rename update_lrm_status to flush_lrm_status Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 10/16] lrm: rename update_service_status to update_lrm_status Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 11/16] lrm: document and initialize LRM instance properties Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 12/16] lrm: update the service status once per work iteration Daniel Kral
2026-11-16 10:32   ` Michael Köppl
2026-11-16 12:40     ` Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 13/16] lrm: read service config " Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 14/16] lrm: compute valid service uids hash set " Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 15/16] lrm: resource_command_finished: avoid writing stale LRM results Daniel Kral
2026-11-12 16:31 ` [PATCH ha-manager v2 16/16] lrm: prune stale LRM results once per work iteration Daniel Kral
2026-11-17 18:58 ` Michael Köppl [this message]
2026-11-18 14:41 ` superseded: [PATCH-SERIES ha-manager v2 00/16] some sid parsing and LRM speedup improvements Daniel Kral

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=DMXPSL7YY5CD.2Z008K4DT4FDF@proxmox.com \
    --to=m.koeppl@proxmox.com \
    --cc=d.kral@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal