[pve-devel] [PATCH ha-manager/perl-rs/proxmox/qemu-server v2 00/12] Granular online_node_usage accounting

[Daniel Kral <d.kral@proxmox.com>]

v1: https://lore.proxmox.com/pve-devel/20250930142021.366529-1-d.kral@proxmox.com/

Changes since v1:
 - rebased all patches on master
 - improve interface for static cache (thanks @Fiona!)
 - improve get_used_service_nodes() signature (thanks @Fiona!)
 - move get_used_service_nodes() to PVE::HA::Usage (thanks @Fiona!)
 - make service_nodes HashMap Value a HashSet (thanks @Fiona!)
 - move static cache patch to the end of series (as it shows its
   improvement there best)
 - make add_service_usage() helper part of $online_node_usage
 - various other style nits (thanks @Fiona!)
 - dropped ha-manager patch #9

Follow-up on making online_node_usage accounting more granular.


= Patches =

Build-dependency bump and versioned breaks for pve-rs needed!
Build-dependency and dependency bump for pve-ha-manager needed!

See pve-rs #1 and ha-manager #6 for more information.

qemu-server patch #1   fetches default values only when needed
proxmox patch #1       necessary for pve-rs patch
pve-rs patch #1        allow removing service usage
pve-rs patch #2        small refactor
ha-manager patch #1-#3 remove redundant $online_node_usage updates
ha-manager patch #4-#5 some decoupling and refactoring
ha-manager patch #6-#7 setup $online_node_usage only once per round and
                       make changes granular inbetween
ha-manager patch #8    implement static cache and use
                       PVE::Cluster::get_guest_config_properties(...)


= Benchmarks =

Here are some benchmarks with a 3 nodes cluster, static load scheduler,
and rebalance_on_request_start set in a virtualized environment, where
all HA resources are added and started at once in the first manage(...)
call. The columns are for HA resource count and the rows are for
different patches applied (qm #1 = qemu-server patch #1).


Run-times for the first manage(...) call to rebalance HA resources:

                300             3,000           10,002

master          19.9 s          -               -
#7              909 ms          10.0 s          33.5 s
#8              390 ms          3.83 s          13.4 s
#8 + qm #1      219 ms          1.92 s          7.11 s


The following small breakdown of the #8 + qm #1 benchmark with 10,002 HA
resources shows the following top 10 most (exclusively) time-consuming
functions. This shows that:

- with the patches from the HA rules follow-up [0] should improve the
  time for get_node_affinity (and therefore select_service_node), and
- there's definitely room to improve the call to get_current_memory(...)
  in get_derived_property(...) in qemu-server, which cascades to
  $change_service_state (add_service_usage / add_service_usage_to_node)
  and calls parse_property_string and check_prop.

This should still give a nice foundation for the upcoming dynamic load
information + load balancing series'. Also it should be pretty rare for
the HA Manager to handle 10,002 HA resource state changes + it's still
well enough under the time limit.

+-------------------------------------------+------------+------------+
|                 Function                  | Excl. time | Incl. time |
+-------------------------------------------+------------+------------+
| Sys::Syslog::syslog                       | 1.12 s     | 2.45 s     |
| Sys::Syslog::xlate                        | 404 ms     | 533 ms     |
| PVE::(...)NodeAffinity::get_node_affinity | 307 ms     | 667 ms     |
| PVE::HA::Manager::select_service_node     | 276 ms     | 2.35 s     |
| POSIX::strftime (xsub)                    | 242 ms     | 242 ms     |
| PVE::JSONSchema::parse_property_string    | 241 ms     | 782 ms     |
| PVE::JSONSchema::check_prop               | 214 ms     | 488 ms     |
| Sys::Syslog::CORE:syswrite (opcode)       | 201 ms     | 201 ms     |
| PVE::HA::Manager::$change_service_state   | 198 ms     | 2.43 s     |
| PVE::HA::Manager::manage                  | 189 ms     | 7.11 s     |
+-------------------------------------------+------------+------------+

[0] https://lore.proxmox.com/pve-devel/20250909083539.39675-1-d.kral@proxmox.com/

qemu-server:

Daniel Kral (1):
  config: only fetch necessary default values in get_derived_property
    helper

 src/PVE/QemuConfig.pm | 8 +++-----
 src/PVE/QemuServer.pm | 6 ++++++
 2 files changed, 9 insertions(+), 5 deletions(-)


proxmox:

Daniel Kral (1):
  resource-scheduling: change score_nodes_to_start_service signature

 proxmox-resource-scheduling/src/pve_static.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


perl-rs:

Daniel Kral (2):
  pve-rs: resource_scheduling: allow granular usage changes
  test: resource_scheduling: use score_nodes helper to imitate HA
    Manager

 .../bindings/resource_scheduling_static.rs    | 108 +++++++++++++++---
 pve-rs/test/resource_scheduling.pl            | 106 ++++++++++++-----
 2 files changed, 170 insertions(+), 44 deletions(-)


ha-manager:

Daniel Kral (8):
  manager: remove redundant recompute_online_node_usage from
    next_state_recovery
  manager: remove redundant add_service_usage_to_node from
    next_state_recovery
  manager: remove redundant add_service_usage_to_node from
    next_state_started
  rules: resource affinity: decouple get_resource_affinity helper from
    Usage class
  manager: make recompute_online_node_usage use add_service_usage helper
  usage: allow granular changes to Usage implementations
  manager: make online node usage computation granular
  implement static service stats cache

 src/PVE/HA/Env.pm                    | 12 ++++
 src/PVE/HA/Env/PVE2.pm               | 35 ++++++++++++
 src/PVE/HA/Manager.pm                | 82 +++++++---------------------
 src/PVE/HA/Resources/PVECT.pm        |  3 +-
 src/PVE/HA/Resources/PVEVM.pm        |  3 +-
 src/PVE/HA/Rules/ResourceAffinity.pm | 23 ++++----
 src/PVE/HA/Sim/Env.pm                | 12 ++++
 src/PVE/HA/Sim/Hardware.pm           | 31 +++++++----
 src/PVE/HA/Sim/Resources.pm          |  3 +-
 src/PVE/HA/Usage.pm                  | 69 +++++++++++++++++------
 src/PVE/HA/Usage/Basic.pm            | 35 +++++-------
 src/PVE/HA/Usage/Static.pm           | 43 ++++++---------
 src/test/test_failover1.pl           | 17 +++---
 13 files changed, 209 insertions(+), 159 deletions(-)


Summary over all repositories:
  18 files changed, 389 insertions(+), 209 deletions(-)

-- 
Generated by git-murpp 0.8.0


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel