[PATCH-SERIES ha-manager v2 00/10] include non-HA resources in usage accounting

[PATCH-SERIES ha-manager v2 00/10] include non-HA resources in usage accounting

[Daniel Kral]

This series is split out from [0] to make the static/dynamic load
balancer series a little more straight forward to reason through.

This series adds any running non-HA resource into the existing basic and
static usage scheduler accounting to base decisions on the usage of the
whole cluster instead of just the HA resources.

For clusters where every guest is configured as a HA resource nothing
should change.

I tested a few simple cases in a 3 node cluster for both the basic and
static usage which are variations of:

- 1 running HA resource on each node, starting them on any node (with
  homogeneous nodes and guests) will rebalance them on the
  alphabetically first node as before
- 1 running HA resource on node1, running 3 non-HA resources on node2,
  starting another HA resource on node1 will rebalance it to node3
- 3 non-HA resources on node1, starting a HA resource on node1 will
  rebalance it to node2
- non-running HA and non-HA resources are excluded from the usage
  accounting
- starting non-HA resources doesn't have any effect obviously

[0] https://lore.proxmox.com/pve-devel/20260217141437.584852-1-d.kral@proxmox.com/

Daniel Kral (10):
  rename static node stats to be consistent with similar interfaces
  resources: remove redundant load_config fallback for static config
  remove redundant service_node and migration_target parameter
  factor out common pve to ha resource type mapping
  derive static service stats while filling the service stats repository
  test: make static service usage explicit for all resources
  make static service stats indexable by sid
  move static service stats repository to PVE::HA::Usage::Static
  usage: augment service stats with node and state information
  include running non-HA resources in the scheduler's accounting

 src/PVE/HA/Config.pm                          |  8 +-
 src/PVE/HA/Env.pm                             | 16 ++--
 src/PVE/HA/Env/PVE2.pm                        | 81 +++++++++++++------
 src/PVE/HA/Manager.pm                         | 16 +++-
 src/PVE/HA/Resources.pm                       |  4 +-
 src/PVE/HA/Resources/PVECT.pm                 |  7 +-
 src/PVE/HA/Resources/PVEVM.pm                 |  7 +-
 src/PVE/HA/Sim/Env.pm                         | 16 ++--
 src/PVE/HA/Sim/Hardware.pm                    | 56 ++++++++-----
 src/PVE/HA/Sim/Resources.pm                   | 17 ----
 src/PVE/HA/Tools.pm                           | 23 ++++--
 src/PVE/HA/Usage.pm                           | 12 ++-
 src/PVE/HA/Usage/Basic.pm                     |  6 +-
 src/PVE/HA/Usage/Static.pm                    | 43 +++-------
 .../hardware_status                           |  6 +-
 .../hardware_status                           |  6 +-
 .../hardware_status                           | 10 +--
 .../hardware_status                           |  6 +-
 .../static_service_stats                      | 52 +++++++++++-
 .../hardware_status                           |  6 +-
 .../static_service_stats                      |  9 ++-
 src/test/test-crs-static1/hardware_status     |  6 +-
 src/test/test-crs-static2/hardware_status     | 10 +--
 src/test/test-crs-static3/hardware_status     |  6 +-
 src/test/test-crs-static4/hardware_status     |  6 +-
 src/test/test-crs-static5/hardware_status     |  6 +-
 26 files changed, 260 insertions(+), 181 deletions(-)

-- 
2.47.3

[PATCH ha-manager v2 01/10] rename static node stats to be consistent with similar interfaces

[Daniel Kral]

The names `maxcpu` and `maxmem` are used in the static load scheduler
itself and is more telling that these properties provide the maximum
configured amount of CPU cores and memory.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 src/PVE/HA/Env/PVE2.pm                                 |  9 ++++++++-
 src/PVE/HA/Sim/Hardware.pm                             |  8 ++++----
 src/PVE/HA/Usage/Static.pm                             |  6 +++---
 .../hardware_status                                    |  6 +++---
 .../hardware_status                                    |  6 +++---
 .../hardware_status                                    | 10 +++++-----
 src/test/test-crs-static-rebalance1/hardware_status    |  6 +++---
 src/test/test-crs-static-rebalance2/hardware_status    |  6 +++---
 src/test/test-crs-static1/hardware_status              |  6 +++---
 src/test/test-crs-static2/hardware_status              | 10 +++++-----
 src/test/test-crs-static3/hardware_status              |  6 +++---
 src/test/test-crs-static4/hardware_status              |  6 +++---
 src/test/test-crs-static5/hardware_status              |  6 +++---
 13 files changed, 49 insertions(+), 42 deletions(-)

diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm
index 4a38cc3b..5b8aaae6 100644
--- a/src/PVE/HA/Env/PVE2.pm
+++ b/src/PVE/HA/Env/PVE2.pm
@@ -548,7 +548,14 @@ sub get_static_node_stats {
 
     my $stats = PVE::Cluster::get_node_kv('static-info');
     for my $node (keys $stats->%*) {
-        $stats->{$node} = eval { decode_json($stats->{$node}) };
+        $stats->{$node} = eval {
+            my $node_stats = decode_json($stats->{$node});
+
+            return {
+                maxcpu => $node_stats->{cpus},
+                maxmem => $node_stats->{memory},
+            };
+        };
         $self->log('err', "unable to decode static node info for '$node' - $@") if $@;
     }
 
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index 8cbf48df..60eb9867 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -488,9 +488,9 @@ sub new {
             || die "Copy failed: $!\n";
     } else {
         my $cstatus = {
-            node1 => { power => 'off', network => 'off', cpus => 24, memory => 131072 },
-            node2 => { power => 'off', network => 'off', cpus => 24, memory => 131072 },
-            node3 => { power => 'off', network => 'off', cpus => 24, memory => 131072 },
+            node1 => { power => 'off', network => 'off', maxcpu => 24, maxmem => 131072 },
+            node2 => { power => 'off', network => 'off', maxcpu => 24, maxmem => 131072 },
+            node3 => { power => 'off', network => 'off', maxcpu => 24, maxmem => 131072 },
         };
         $self->write_hardware_status_nolock($cstatus);
     }
@@ -1088,7 +1088,7 @@ sub get_static_node_stats {
 
     my $stats = {};
     for my $node (keys $cstatus->%*) {
-        $stats->{$node} = { $cstatus->{$node}->%{qw(cpus memory)} };
+        $stats->{$node} = { $cstatus->{$node}->%{qw(maxcpu maxmem)} };
     }
 
     return $stats;
diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm
index d586b603..395be871 100644
--- a/src/PVE/HA/Usage/Static.pm
+++ b/src/PVE/HA/Usage/Static.pm
@@ -33,10 +33,10 @@ sub add_node {
 
     my $stats = $self->{'node-stats'}->{$nodename}
         or die "did not get static node usage information for '$nodename'\n";
-    die "static node usage information for '$nodename' missing cpu count\n" if !$stats->{cpus};
-    die "static node usage information for '$nodename' missing memory\n" if !$stats->{memory};
+    die "static node usage information for '$nodename' missing cpu count\n" if !$stats->{maxcpu};
+    die "static node usage information for '$nodename' missing memory\n" if !$stats->{maxmem};
 
-    eval { $self->{scheduler}->add_node($nodename, int($stats->{cpus}), int($stats->{memory})); };
+    eval { $self->{scheduler}->add_node($nodename, int($stats->{maxcpu}), int($stats->{maxmem})); };
     die "initializing static node usage for '$nodename' failed - $@" if $@;
 }
 
diff --git a/src/test/test-crs-static-rebalance-resource-affinity1/hardware_status b/src/test/test-crs-static-rebalance-resource-affinity1/hardware_status
index 84484af1..3d4cf91f 100644
--- a/src/test/test-crs-static-rebalance-resource-affinity1/hardware_status
+++ b/src/test/test-crs-static-rebalance-resource-affinity1/hardware_status
@@ -1,5 +1,5 @@
 {
-  "node1": { "power": "off", "network": "off", "cpus": 8, "memory": 112000000000 },
-  "node2": { "power": "off", "network": "off", "cpus": 8, "memory": 112000000000 },
-  "node3": { "power": "off", "network": "off", "cpus": 8, "memory": 112000000000 }
+  "node1": { "power": "off", "network": "off", "maxcpu": 8, "maxmem": 112000000000 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 8, "maxmem": 112000000000 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 8, "maxmem": 112000000000 }
 }
diff --git a/src/test/test-crs-static-rebalance-resource-affinity2/hardware_status b/src/test/test-crs-static-rebalance-resource-affinity2/hardware_status
index 84484af1..3d4cf91f 100644
--- a/src/test/test-crs-static-rebalance-resource-affinity2/hardware_status
+++ b/src/test/test-crs-static-rebalance-resource-affinity2/hardware_status
@@ -1,5 +1,5 @@
 {
-  "node1": { "power": "off", "network": "off", "cpus": 8, "memory": 112000000000 },
-  "node2": { "power": "off", "network": "off", "cpus": 8, "memory": 112000000000 },
-  "node3": { "power": "off", "network": "off", "cpus": 8, "memory": 112000000000 }
+  "node1": { "power": "off", "network": "off", "maxcpu": 8, "maxmem": 112000000000 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 8, "maxmem": 112000000000 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 8, "maxmem": 112000000000 }
 }
diff --git a/src/test/test-crs-static-rebalance-resource-affinity3/hardware_status b/src/test/test-crs-static-rebalance-resource-affinity3/hardware_status
index b6dcb1a5..7bc741f1 100644
--- a/src/test/test-crs-static-rebalance-resource-affinity3/hardware_status
+++ b/src/test/test-crs-static-rebalance-resource-affinity3/hardware_status
@@ -1,7 +1,7 @@
 {
-  "node1": { "power": "off", "network": "off", "cpus": 8, "memory": 48000000000 },
-  "node2": { "power": "off", "network": "off", "cpus": 32, "memory": 36000000000 },
-  "node3": { "power": "off", "network": "off", "cpus": 16, "memory": 24000000000 },
-  "node4": { "power": "off", "network": "off", "cpus": 32, "memory": 36000000000 },
-  "node5": { "power": "off", "network": "off", "cpus": 8, "memory": 48000000000 }
+  "node1": { "power": "off", "network": "off", "maxcpu": 8, "maxmem": 48000000000 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 36000000000 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 16, "maxmem": 24000000000 },
+  "node4": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 36000000000 },
+  "node5": { "power": "off", "network": "off", "maxcpu": 8, "maxmem": 48000000000 }
 }
diff --git a/src/test/test-crs-static-rebalance1/hardware_status b/src/test/test-crs-static-rebalance1/hardware_status
index 651ad792..bfdbbf7b 100644
--- a/src/test/test-crs-static-rebalance1/hardware_status
+++ b/src/test/test-crs-static-rebalance1/hardware_status
@@ -1,5 +1,5 @@
 {
-  "node1": { "power": "off", "network": "off", "cpus": 32, "memory": 256000000000 },
-  "node2": { "power": "off", "network": "off", "cpus": 32, "memory": 256000000000 },
-  "node3": { "power": "off", "network": "off", "cpus": 32, "memory": 256000000000 }
+  "node1": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 256000000000 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 256000000000 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 256000000000 }
 }
diff --git a/src/test/test-crs-static-rebalance2/hardware_status b/src/test/test-crs-static-rebalance2/hardware_status
index 9be70a40..c5cbde3d 100644
--- a/src/test/test-crs-static-rebalance2/hardware_status
+++ b/src/test/test-crs-static-rebalance2/hardware_status
@@ -1,5 +1,5 @@
 {
-  "node1": { "power": "off", "network": "off", "cpus": 40, "memory": 384000000000 },
-  "node2": { "power": "off", "network": "off", "cpus": 32, "memory": 256000000000 },
-  "node3": { "power": "off", "network": "off", "cpus": 32, "memory": 256000000000 }
+  "node1": { "power": "off", "network": "off", "maxcpu": 40, "maxmem": 384000000000 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 256000000000 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 256000000000 }
 }
diff --git a/src/test/test-crs-static1/hardware_status b/src/test/test-crs-static1/hardware_status
index 0fa8c265..bbe44a96 100644
--- a/src/test/test-crs-static1/hardware_status
+++ b/src/test/test-crs-static1/hardware_status
@@ -1,5 +1,5 @@
 {
-  "node1": { "power": "off", "network": "off", "cpus": 32, "memory": 100000000000 },
-  "node2": { "power": "off", "network": "off", "cpus": 32, "memory": 200000000000 },
-  "node3": { "power": "off", "network": "off", "cpus": 32, "memory": 300000000000 }
+  "node1": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 100000000000 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 200000000000 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 300000000000 }
 }
diff --git a/src/test/test-crs-static2/hardware_status b/src/test/test-crs-static2/hardware_status
index d426023a..815436ef 100644
--- a/src/test/test-crs-static2/hardware_status
+++ b/src/test/test-crs-static2/hardware_status
@@ -1,7 +1,7 @@
 {
-  "node1": { "power": "off", "network": "off", "cpus": 32, "memory": 100000000000 },
-  "node2": { "power": "off", "network": "off", "cpus": 32, "memory": 200000000000 },
-  "node3": { "power": "off", "network": "off", "cpus": 32, "memory": 300000000000 },
-  "node4": { "power": "off", "network": "off", "cpus": 64, "memory": 300000000000 },
-  "node5": { "power": "off", "network": "off", "cpus": 32, "memory": 100000000000 }
+  "node1": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 100000000000 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 200000000000 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 300000000000 },
+  "node4": { "power": "off", "network": "off", "maxcpu": 64, "maxmem": 300000000000 },
+  "node5": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 100000000000 }
 }
diff --git a/src/test/test-crs-static3/hardware_status b/src/test/test-crs-static3/hardware_status
index dfbf496e..ed84b8bd 100644
--- a/src/test/test-crs-static3/hardware_status
+++ b/src/test/test-crs-static3/hardware_status
@@ -1,5 +1,5 @@
 {
-  "node1": { "power": "off", "network": "off", "cpus": 32, "memory": 100000000000 },
-  "node2": { "power": "off", "network": "off", "cpus": 64, "memory": 200000000000 },
-  "node3": { "power": "off", "network": "off", "cpus": 32, "memory": 100000000000 }
+  "node1": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 100000000000 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 64, "maxmem": 200000000000 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 100000000000 }
 }
diff --git a/src/test/test-crs-static4/hardware_status b/src/test/test-crs-static4/hardware_status
index a83a2dcc..b08ba7f9 100644
--- a/src/test/test-crs-static4/hardware_status
+++ b/src/test/test-crs-static4/hardware_status
@@ -1,5 +1,5 @@
 {
-  "node1": { "power": "off", "network": "off", "cpus": 32, "memory": 100000000000 },
-  "node2": { "power": "off", "network": "off", "cpus": 32, "memory": 100000000000 },
-  "node3": { "power": "off", "network": "off", "cpus": 32, "memory": 100000000000 }
+  "node1": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 100000000000 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 100000000000 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 100000000000 }
 }
diff --git a/src/test/test-crs-static5/hardware_status b/src/test/test-crs-static5/hardware_status
index 3eb9e735..edfd6db2 100644
--- a/src/test/test-crs-static5/hardware_status
+++ b/src/test/test-crs-static5/hardware_status
@@ -1,5 +1,5 @@
 {
-  "node1": { "power": "off", "network": "off", "cpus": 32, "memory": 100000000000 },
-  "node2": { "power": "off", "network": "off", "cpus": 32, "memory": 100000000000 },
-  "node3": { "power": "off", "network": "off", "cpus": 128, "memory": 100000000000 }
+  "node1": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 100000000000 },
+  "node2": { "power": "off", "network": "off", "maxcpu": 32, "maxmem": 100000000000 },
+  "node3": { "power": "off", "network": "off", "maxcpu": 128, "maxmem": 100000000000 }
 }
-- 
2.47.3

[PATCH ha-manager v2 02/10] resources: remove redundant load_config fallback for static config

[Daniel Kral]

The return value of get_static_service_stats(...) is fetched through
PVE::Cluster::get_guest_config_properties(...), which in turn reads all
guest configuration files with a memdb_read_nolock(...) in the pmxcfs.

As PVE::AbstractConfig::load_config(...) internally gets the content of
the guest configuration file through cfs_read_file(...), which in turn
receives the return value of the equivalent memdb_read(...) from a
CFS_IPC_GET_CONFIG message, the fallback is likely to fail as well.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 src/PVE/HA/Resources/PVECT.pm | 1 -
 src/PVE/HA/Resources/PVEVM.pm | 1 -
 2 files changed, 2 deletions(-)

diff --git a/src/PVE/HA/Resources/PVECT.pm b/src/PVE/HA/Resources/PVECT.pm
index 4cbf6db3..b9ce2ac3 100644
--- a/src/PVE/HA/Resources/PVECT.pm
+++ b/src/PVE/HA/Resources/PVECT.pm
@@ -163,7 +163,6 @@ sub get_static_stats {
     my ($class, $haenv, $id, $service_node) = @_;
 
     my $conf = $haenv->get_static_service_stats($id);
-    $conf = PVE::LXC::Config->load_config($id, $service_node) if !defined($conf);
 
     return {
         maxcpu => PVE::LXC::Config->get_derived_property($conf, 'max-cpu'),
diff --git a/src/PVE/HA/Resources/PVEVM.pm b/src/PVE/HA/Resources/PVEVM.pm
index 7586da84..303334ba 100644
--- a/src/PVE/HA/Resources/PVEVM.pm
+++ b/src/PVE/HA/Resources/PVEVM.pm
@@ -184,7 +184,6 @@ sub get_static_stats {
     my ($class, $haenv, $id, $service_node) = @_;
 
     my $conf = $haenv->get_static_service_stats($id);
-    $conf = PVE::QemuConfig->load_config($id, $service_node) if !defined($conf);
 
     return {
         maxcpu => PVE::QemuConfig->get_derived_property($conf, 'max-cpu'),
-- 
2.47.3

[PATCH ha-manager v2 03/10] remove redundant service_node and migration_target parameter

[Daniel Kral]

As the retrieval of the static service stats are not dependent on the
location of the guest's config file, there is no need for providing the
$service_node and $migration_target arguments anymore.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 src/PVE/HA/Resources.pm       |  2 +-
 src/PVE/HA/Resources/PVECT.pm |  2 +-
 src/PVE/HA/Resources/PVEVM.pm |  2 +-
 src/PVE/HA/Sim/Resources.pm   |  2 +-
 src/PVE/HA/Usage.pm           | 10 ++++------
 src/PVE/HA/Usage/Basic.pm     |  4 ++--
 src/PVE/HA/Usage/Static.pm    | 19 +++++++------------
 7 files changed, 17 insertions(+), 24 deletions(-)

diff --git a/src/PVE/HA/Resources.pm b/src/PVE/HA/Resources.pm
index b6ccd44a..3b20d44c 100644
--- a/src/PVE/HA/Resources.pm
+++ b/src/PVE/HA/Resources.pm
@@ -178,7 +178,7 @@ sub remove_locks {
 }
 
 sub get_static_stats {
-    my ($class, $haenv, $id, $service_node) = @_;
+    my ($class, $haenv, $id) = @_;
 
     die "implement in subclass";
 }
diff --git a/src/PVE/HA/Resources/PVECT.pm b/src/PVE/HA/Resources/PVECT.pm
index b9ce2ac3..1f4eb2e9 100644
--- a/src/PVE/HA/Resources/PVECT.pm
+++ b/src/PVE/HA/Resources/PVECT.pm
@@ -160,7 +160,7 @@ sub remove_locks {
 }
 
 sub get_static_stats {
-    my ($class, $haenv, $id, $service_node) = @_;
+    my ($class, $haenv, $id) = @_;
 
     my $conf = $haenv->get_static_service_stats($id);
 
diff --git a/src/PVE/HA/Resources/PVEVM.pm b/src/PVE/HA/Resources/PVEVM.pm
index 303334ba..760259e4 100644
--- a/src/PVE/HA/Resources/PVEVM.pm
+++ b/src/PVE/HA/Resources/PVEVM.pm
@@ -181,7 +181,7 @@ sub remove_locks {
 }
 
 sub get_static_stats {
-    my ($class, $haenv, $id, $service_node) = @_;
+    my ($class, $haenv, $id) = @_;
 
     my $conf = $haenv->get_static_service_stats($id);
 
diff --git a/src/PVE/HA/Sim/Resources.pm b/src/PVE/HA/Sim/Resources.pm
index 34462522..48c15dfe 100644
--- a/src/PVE/HA/Sim/Resources.pm
+++ b/src/PVE/HA/Sim/Resources.pm
@@ -138,7 +138,7 @@ sub remove_locks {
 }
 
 sub get_static_stats {
-    my ($class, $haenv, $id, $service_node) = @_;
+    my ($class, $haenv, $id) = @_;
 
     my $sid = $class->type() . ":$id";
     my $hardware = $haenv->hardware();
diff --git a/src/PVE/HA/Usage.pm b/src/PVE/HA/Usage.pm
index 92e575cb..1be5fa09 100644
--- a/src/PVE/HA/Usage.pm
+++ b/src/PVE/HA/Usage.pm
@@ -35,7 +35,7 @@ sub contains_node {
 
 # Logs a warning to $haenv upon failure, but does not die.
 sub add_service_usage_to_node {
-    my ($self, $nodename, $sid, $service_node, $migration_target) = @_;
+    my ($self, $nodename, $sid) = @_;
 
     die "implement in subclass";
 }
@@ -49,10 +49,8 @@ sub add_service_usage {
     my ($current_node, $target_node) =
         get_used_service_nodes($online_nodes, $service_state, $service_node, $migration_target);
 
-    $self->add_service_usage_to_node($current_node, $sid, $service_node, $migration_target)
-        if $current_node;
-    $self->add_service_usage_to_node($target_node, $sid, $service_node, $migration_target)
-        if $target_node;
+    $self->add_service_usage_to_node($current_node, $sid) if $current_node;
+    $self->add_service_usage_to_node($target_node, $sid) if $target_node;
 }
 
 sub remove_service_usage {
@@ -63,7 +61,7 @@ sub remove_service_usage {
 
 # Returns a hash with $nodename => $score pairs. A lower $score is better.
 sub score_nodes_to_start_service {
-    my ($self, $sid, $service_node) = @_;
+    my ($self, $sid) = @_;
 
     die "implement in subclass";
 }
diff --git a/src/PVE/HA/Usage/Basic.pm b/src/PVE/HA/Usage/Basic.pm
index 43817bf6..ef9ae3d6 100644
--- a/src/PVE/HA/Usage/Basic.pm
+++ b/src/PVE/HA/Usage/Basic.pm
@@ -39,7 +39,7 @@ sub contains_node {
 }
 
 sub add_service_usage_to_node {
-    my ($self, $nodename, $sid, $service_node, $migration_target) = @_;
+    my ($self, $nodename, $sid) = @_;
 
     if ($self->contains_node($nodename)) {
         $self->{nodes}->{$nodename}->{$sid} = 1;
@@ -60,7 +60,7 @@ sub remove_service_usage {
 }
 
 sub score_nodes_to_start_service {
-    my ($self, $sid, $service_node) = @_;
+    my ($self, $sid) = @_;
 
     my $nodes = $self->{nodes};
 
diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm
index 395be871..2304139c 100644
--- a/src/PVE/HA/Usage/Static.pm
+++ b/src/PVE/HA/Usage/Static.pm
@@ -61,20 +61,15 @@ sub contains_node {
 }
 
 my sub get_service_usage {
-    my ($self, $sid, $service_node, $migration_target) = @_;
+    my ($self, $sid) = @_;
 
     return $self->{'service-stats'}->{$sid} if $self->{'service-stats'}->{$sid};
 
     my (undef, $type, $id) = $self->{haenv}->parse_sid($sid);
     my $plugin = PVE::HA::Resources->lookup($type);
 
-    my $stats = eval { $plugin->get_static_stats($self->{haenv}, $id, $service_node) };
-    if (my $err = $@) {
-        # config might've already moved during a migration
-        $stats = eval { $plugin->get_static_stats($self->{haenv}, $id, $migration_target); }
-            if $migration_target;
-        die "did not get static service usage information for '$sid' - $err\n" if !$stats;
-    }
+    my $stats = eval { $plugin->get_static_stats($self->{haenv}, $id) };
+    die "did not get static service usage information for '$sid'\n" if !$stats;
 
     my $service_stats = {
         maxcpu => $stats->{maxcpu} + 0.0, # containers allow non-integer cpulimit
@@ -87,12 +82,12 @@ my sub get_service_usage {
 }
 
 sub add_service_usage_to_node {
-    my ($self, $nodename, $sid, $service_node, $migration_target) = @_;
+    my ($self, $nodename, $sid) = @_;
 
     $self->{'node-services'}->{$nodename}->{$sid} = 1;
 
     eval {
-        my $service_usage = get_service_usage($self, $sid, $service_node, $migration_target);
+        my $service_usage = get_service_usage($self, $sid);
         $self->{scheduler}->add_service_usage_to_node($nodename, $sid, $service_usage);
     };
     $self->{haenv}->log('warning', "unable to add service '$sid' usage to node '$nodename' - $@")
@@ -111,10 +106,10 @@ sub remove_service_usage {
 }
 
 sub score_nodes_to_start_service {
-    my ($self, $sid, $service_node) = @_;
+    my ($self, $sid) = @_;
 
     my $score_list = eval {
-        my $service_usage = get_service_usage($self, $sid, $service_node);
+        my $service_usage = get_service_usage($self, $sid);
         $self->{scheduler}->score_nodes_to_start_service($service_usage);
     };
     if (my $err = $@) {
-- 
2.47.3

[PATCH ha-manager v2 04/10] factor out common pve to ha resource type mapping

[Daniel Kral]

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 src/PVE/HA/Config.pm |  8 +-------
 src/PVE/HA/Tools.pm  | 23 +++++++++++++++--------
 2 files changed, 16 insertions(+), 15 deletions(-)

diff --git a/src/PVE/HA/Config.pm b/src/PVE/HA/Config.pm
index 19eec2a4..cafbae8c 100644
--- a/src/PVE/HA/Config.pm
+++ b/src/PVE/HA/Config.pm
@@ -197,13 +197,7 @@ sub parse_sid {
         my $vmlist = PVE::Cluster::get_vmlist();
         if (defined($vmlist->{ids}->{$name})) {
             my $vm_type = $vmlist->{ids}->{$name}->{type};
-            if ($vm_type eq 'lxc') {
-                $type = 'ct';
-            } elsif ($vm_type eq 'qemu') {
-                $type = 'vm';
-            } else {
-                die "internal error";
-            }
+            $type = PVE::HA::Tools::get_ha_resource_type($vm_type);
             $sid = "$type:$name";
         } else {
             die "unable to detect SID from VMID - VM/CT $1 does not exist\n";
diff --git a/src/PVE/HA/Tools.pm b/src/PVE/HA/Tools.pm
index 1fa53df8..26629fb5 100644
--- a/src/PVE/HA/Tools.pm
+++ b/src/PVE/HA/Tools.pm
@@ -289,6 +289,18 @@ sub has_min_version {
     return 1;
 }
 
+sub get_ha_resource_type {
+    my ($pve_resource_type) = @_;
+
+    if ($pve_resource_type eq 'lxc') {
+        return 'ct';
+    } elsif ($pve_resource_type eq 'qemu') {
+        return 'vm';
+    } else {
+        die "unknown PVE resource type '$pve_resource_type'";
+    }
+}
+
 # bash auto completion helper
 
 # NOTE: we use PVE::HA::Config here without declaring an 'use' clause above as
@@ -309,15 +321,10 @@ sub complete_sid {
 
         while (my ($vmid, $info) = each %{ $vmlist->{ids} }) {
 
-            my $sid;
+            my $type = eval { get_ha_resource_type($info->{type}) };
+            next if $@; # silently ignore unknown pve types
 
-            if ($info->{type} eq 'lxc') {
-                $sid = "ct:$vmid";
-            } elsif ($info->{type} eq 'qemu') {
-                $sid = "vm:$vmid";
-            } else {
-                next; # should not happen
-            }
+            my $sid = "$type:$vmid";
 
             next if $cfg->{ids}->{$sid};
 
-- 
2.47.3

[PATCH ha-manager v2 05/10] derive static service stats while filling the service stats repository

[Daniel Kral]

This is needed to be able to derive the proper static service stats for
non-HA resources in a following patch as well.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 src/PVE/HA/Env/PVE2.pm        | 14 ++++++++------
 src/PVE/HA/Resources.pm       |  6 ++++++
 src/PVE/HA/Resources/PVECT.pm | 12 ++++++++----
 src/PVE/HA/Resources/PVEVM.pm | 12 ++++++++----
 4 files changed, 30 insertions(+), 14 deletions(-)

diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm
index 5b8aaae6..94f30d73 100644
--- a/src/PVE/HA/Env/PVE2.pm
+++ b/src/PVE/HA/Env/PVE2.pm
@@ -523,17 +523,19 @@ sub update_static_service_stats {
 
     my $properties = ['cores', 'cpulimit', 'memory', 'sockets', 'vcpus'];
     my $service_stats = eval {
-        my $stats = PVE::Cluster::get_guest_config_properties($properties);
+        my $stats = {};
+        my $confs = PVE::Cluster::get_guest_config_properties($properties);
 
-        # get_guest_config_properties(...) doesn't add guests which do not
-        # specify any of the given properties, but we need to make a distinction
-        # between "not cached" and "not specified" here
         my $vmlist = PVE::Cluster::get_vmlist();
         my $idlist = $vmlist->{ids} // {};
         for my $id (keys %$idlist) {
-            next if defined($stats->{$id});
+            my $type = eval { PVE::HA::Tools::get_ha_resource_type($idlist->{$id}->{type}) };
+            next if $@; # silently ignore unknown pve types
 
-            $stats->{$id} = {};
+            my $conf = $confs->{$id} // {};
+            my $plugin = PVE::HA::Resources->lookup($type);
+
+            $stats->{$id} = $plugin->get_static_stats_from_config($conf);
         }
 
         return $stats;
diff --git a/src/PVE/HA/Resources.pm b/src/PVE/HA/Resources.pm
index 3b20d44c..cba2d4c2 100644
--- a/src/PVE/HA/Resources.pm
+++ b/src/PVE/HA/Resources.pm
@@ -177,6 +177,12 @@ sub remove_locks {
     die "implement in subclass";
 }
 
+sub get_static_stats_from_config {
+    my ($class, $conf) = @_;
+
+    die "implement in subclass";
+}
+
 sub get_static_stats {
     my ($class, $haenv, $id) = @_;
 
diff --git a/src/PVE/HA/Resources/PVECT.pm b/src/PVE/HA/Resources/PVECT.pm
index 1f4eb2e9..0dc500d1 100644
--- a/src/PVE/HA/Resources/PVECT.pm
+++ b/src/PVE/HA/Resources/PVECT.pm
@@ -159,10 +159,8 @@ sub remove_locks {
     return undef;
 }
 
-sub get_static_stats {
-    my ($class, $haenv, $id) = @_;
-
-    my $conf = $haenv->get_static_service_stats($id);
+sub get_static_stats_from_config {
+    my ($class, $conf) = @_;
 
     return {
         maxcpu => PVE::LXC::Config->get_derived_property($conf, 'max-cpu'),
@@ -170,4 +168,10 @@ sub get_static_stats {
     };
 }
 
+sub get_static_stats {
+    my ($class, $haenv, $id) = @_;
+
+    return $haenv->get_static_service_stats($id);
+}
+
 1;
diff --git a/src/PVE/HA/Resources/PVEVM.pm b/src/PVE/HA/Resources/PVEVM.pm
index 760259e4..579a7fca 100644
--- a/src/PVE/HA/Resources/PVEVM.pm
+++ b/src/PVE/HA/Resources/PVEVM.pm
@@ -180,10 +180,8 @@ sub remove_locks {
     return undef;
 }
 
-sub get_static_stats {
-    my ($class, $haenv, $id) = @_;
-
-    my $conf = $haenv->get_static_service_stats($id);
+sub get_static_stats_from_config {
+    my ($class, $conf) = @_;
 
     return {
         maxcpu => PVE::QemuConfig->get_derived_property($conf, 'max-cpu'),
@@ -191,4 +189,10 @@ sub get_static_stats {
     };
 }
 
+sub get_static_stats {
+    my ($class, $haenv, $id) = @_;
+
+    return $haenv->get_static_service_stats($id);
+}
+
 1;
-- 
2.47.3

[PATCH ha-manager v2 06/10] test: make static service usage explicit for all resources

[Daniel Kral]

Even though deriving these from the HA resource sid is convenient, this
is needed to get the proper static service stats for all HA resources at
once.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 src/PVE/HA/Sim/Resources.pm                   | 10 +---
 .../static_service_stats                      | 52 ++++++++++++++++++-
 .../static_service_stats                      |  9 +++-
 3 files changed, 60 insertions(+), 11 deletions(-)

diff --git a/src/PVE/HA/Sim/Resources.pm b/src/PVE/HA/Sim/Resources.pm
index 48c15dfe..ad52437d 100644
--- a/src/PVE/HA/Sim/Resources.pm
+++ b/src/PVE/HA/Sim/Resources.pm
@@ -143,15 +143,7 @@ sub get_static_stats {
     my $sid = $class->type() . ":$id";
     my $hardware = $haenv->hardware();
 
-    if (my $service_stats = $hardware->get_static_service_stats($sid)) {
-        return $service_stats;
-    } elsif ($id =~ /^(\d)(\d\d)/) {
-        # auto assign usage calculated from ID for convenience
-        my ($maxcpu, $maxmeory) = (int($1) + 1, (int($2) + 1) * 1 << 29);
-        return { maxcpu => $maxcpu, maxmem => $maxmeory };
-    } else {
-        return {};
-    }
+    return $hardware->get_static_service_stats($sid);
 }
 
 1;
diff --git a/src/test/test-crs-static-rebalance1/static_service_stats b/src/test/test-crs-static-rebalance1/static_service_stats
index 7fb992dd..666af861 100644
--- a/src/test/test-crs-static-rebalance1/static_service_stats
+++ b/src/test/test-crs-static-rebalance1/static_service_stats
@@ -1,3 +1,53 @@
 {
-    "vm:102": { "maxcpu": 2, "maxmem": 4000000000 }
+    "vm:101": { "maxcpu": 2, "maxmem": 1073741824 },
+    "vm:102": { "maxcpu": 2, "maxmem": 4000000000 },
+    "vm:103": { "maxcpu": 2, "maxmem": 2147483648 },
+    "vm:104": { "maxcpu": 2, "maxmem": 2684354560 },
+    "vm:105": { "maxcpu": 2, "maxmem": 3221225472 },
+    "vm:106": { "maxcpu": 2, "maxmem": 3758096384 },
+    "vm:107": { "maxcpu": 2, "maxmem": 4294967296 },
+    "vm:108": { "maxcpu": 2, "maxmem": 4831838208 },
+    "vm:109": { "maxcpu": 2, "maxmem": 5368709120 },
+    "vm:110": { "maxcpu": 2, "maxmem": 5905580032 },
+    "vm:111": { "maxcpu": 2, "maxmem": 6442450944 },
+    "vm:112": { "maxcpu": 2, "maxmem": 6979321856 },
+    "vm:113": { "maxcpu": 2, "maxmem": 7516192768 },
+    "vm:114": { "maxcpu": 2, "maxmem": 8053063680 },
+    "vm:115": { "maxcpu": 2, "maxmem": 8589934592 },
+    "vm:116": { "maxcpu": 2, "maxmem": 9126805504 },
+    "vm:117": { "maxcpu": 2, "maxmem": 9663676416 },
+    "vm:118": { "maxcpu": 2, "maxmem": 10200547328 },
+    "vm:119": { "maxcpu": 2, "maxmem": 10737418240 },
+    "vm:120": { "maxcpu": 2, "maxmem": 11274289152 },
+    "vm:121": { "maxcpu": 2, "maxmem": 11811160064 },
+    "vm:122": { "maxcpu": 2, "maxmem": 12348030976 },
+    "vm:123": { "maxcpu": 2, "maxmem": 12884901888 },
+    "vm:124": { "maxcpu": 2, "maxmem": 13421772800 },
+    "vm:125": { "maxcpu": 2, "maxmem": 13958643712 },
+    "vm:126": { "maxcpu": 2, "maxmem": 14495514624 },
+    "vm:127": { "maxcpu": 2, "maxmem": 15032385536 },
+    "vm:128": { "maxcpu": 2, "maxmem": 15569256448 },
+    "vm:129": { "maxcpu": 2, "maxmem": 16106127360 },
+    "vm:130": { "maxcpu": 2, "maxmem": 16642998272 },
+    "vm:131": { "maxcpu": 2, "maxmem": 17179869184 },
+    "vm:132": { "maxcpu": 2, "maxmem": 17716740096 },
+    "vm:133": { "maxcpu": 2, "maxmem": 18253611008 },
+    "vm:134": { "maxcpu": 2, "maxmem": 18790481920 },
+    "vm:135": { "maxcpu": 2, "maxmem": 19327352832 },
+    "vm:136": { "maxcpu": 2, "maxmem": 19864223744 },
+    "vm:137": { "maxcpu": 2, "maxmem": 20401094656 },
+    "vm:138": { "maxcpu": 2, "maxmem": 20937965568 },
+    "vm:139": { "maxcpu": 2, "maxmem": 21474836480 },
+    "vm:140": { "maxcpu": 2, "maxmem": 22011707392 },
+    "vm:141": { "maxcpu": 2, "maxmem": 22548578304 },
+    "vm:142": { "maxcpu": 2, "maxmem": 23085449216 },
+    "vm:143": { "maxcpu": 2, "maxmem": 23622320128 },
+    "vm:144": { "maxcpu": 2, "maxmem": 24159191040 },
+    "vm:145": { "maxcpu": 2, "maxmem": 24696061952 },
+    "vm:146": { "maxcpu": 2, "maxmem": 25232932864 },
+    "vm:147": { "maxcpu": 2, "maxmem": 25769803776 },
+    "vm:148": { "maxcpu": 2, "maxmem": 26306674688 },
+    "vm:149": { "maxcpu": 2, "maxmem": 26843545600 },
+    "vm:150": { "maxcpu": 2, "maxmem": 27380416512 },
+    "vm:151": { "maxcpu": 2, "maxmem": 27917287424 }
 }
diff --git a/src/test/test-crs-static-rebalance2/static_service_stats b/src/test/test-crs-static-rebalance2/static_service_stats
index 0967ef42..a4049f80 100644
--- a/src/test/test-crs-static-rebalance2/static_service_stats
+++ b/src/test/test-crs-static-rebalance2/static_service_stats
@@ -1 +1,8 @@
-{}
+{
+    "vm:100": { "maxcpu": 2, "maxmem": 536870912 },
+    "vm:101": { "maxcpu": 2, "maxmem": 1073741824 },
+    "vm:102": { "maxcpu": 2, "maxmem": 1610612736 },
+    "vm:103": { "maxcpu": 2, "maxmem": 2147483648 },
+    "vm:104": { "maxcpu": 2, "maxmem": 2684354560 },
+    "vm:105": { "maxcpu": 2, "maxmem": 3221225472 }
+}
-- 
2.47.3

[PATCH ha-manager v2 07/10] make static service stats indexable by sid

[Daniel Kral]

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 src/PVE/HA/Env/PVE2.pm        | 3 ++-
 src/PVE/HA/Resources.pm       | 2 +-
 src/PVE/HA/Resources/PVECT.pm | 4 ++--
 src/PVE/HA/Resources/PVEVM.pm | 4 ++--
 src/PVE/HA/Sim/Resources.pm   | 3 +--
 src/PVE/HA/Usage/Static.pm    | 4 ++--
 6 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm
index 94f30d73..54270a40 100644
--- a/src/PVE/HA/Env/PVE2.pm
+++ b/src/PVE/HA/Env/PVE2.pm
@@ -532,10 +532,11 @@ sub update_static_service_stats {
             my $type = eval { PVE::HA::Tools::get_ha_resource_type($idlist->{$id}->{type}) };
             next if $@; # silently ignore unknown pve types
 
+            my $sid = "$type:$id";
             my $conf = $confs->{$id} // {};
             my $plugin = PVE::HA::Resources->lookup($type);
 
-            $stats->{$id} = $plugin->get_static_stats_from_config($conf);
+            $stats->{$sid} = $plugin->get_static_stats_from_config($conf);
         }
 
         return $stats;
diff --git a/src/PVE/HA/Resources.pm b/src/PVE/HA/Resources.pm
index cba2d4c2..01facac8 100644
--- a/src/PVE/HA/Resources.pm
+++ b/src/PVE/HA/Resources.pm
@@ -184,7 +184,7 @@ sub get_static_stats_from_config {
 }
 
 sub get_static_stats {
-    my ($class, $haenv, $id) = @_;
+    my ($class, $haenv, $sid) = @_;
 
     die "implement in subclass";
 }
diff --git a/src/PVE/HA/Resources/PVECT.pm b/src/PVE/HA/Resources/PVECT.pm
index 0dc500d1..bda33717 100644
--- a/src/PVE/HA/Resources/PVECT.pm
+++ b/src/PVE/HA/Resources/PVECT.pm
@@ -169,9 +169,9 @@ sub get_static_stats_from_config {
 }
 
 sub get_static_stats {
-    my ($class, $haenv, $id) = @_;
+    my ($class, $haenv, $sid) = @_;
 
-    return $haenv->get_static_service_stats($id);
+    return $haenv->get_static_service_stats($sid);
 }
 
 1;
diff --git a/src/PVE/HA/Resources/PVEVM.pm b/src/PVE/HA/Resources/PVEVM.pm
index 579a7fca..786c5130 100644
--- a/src/PVE/HA/Resources/PVEVM.pm
+++ b/src/PVE/HA/Resources/PVEVM.pm
@@ -190,9 +190,9 @@ sub get_static_stats_from_config {
 }
 
 sub get_static_stats {
-    my ($class, $haenv, $id) = @_;
+    my ($class, $haenv, $sid) = @_;
 
-    return $haenv->get_static_service_stats($id);
+    return $haenv->get_static_service_stats($sid);
 }
 
 1;
diff --git a/src/PVE/HA/Sim/Resources.pm b/src/PVE/HA/Sim/Resources.pm
index ad52437d..ed7d9c5a 100644
--- a/src/PVE/HA/Sim/Resources.pm
+++ b/src/PVE/HA/Sim/Resources.pm
@@ -138,9 +138,8 @@ sub remove_locks {
 }
 
 sub get_static_stats {
-    my ($class, $haenv, $id) = @_;
+    my ($class, $haenv, $sid) = @_;
 
-    my $sid = $class->type() . ":$id";
     my $hardware = $haenv->hardware();
 
     return $hardware->get_static_service_stats($sid);
diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm
index 2304139c..acc3533c 100644
--- a/src/PVE/HA/Usage/Static.pm
+++ b/src/PVE/HA/Usage/Static.pm
@@ -65,10 +65,10 @@ my sub get_service_usage {
 
     return $self->{'service-stats'}->{$sid} if $self->{'service-stats'}->{$sid};
 
-    my (undef, $type, $id) = $self->{haenv}->parse_sid($sid);
+    my (undef, $type, undef) = $self->{haenv}->parse_sid($sid);
     my $plugin = PVE::HA::Resources->lookup($type);
 
-    my $stats = eval { $plugin->get_static_stats($self->{haenv}, $id) };
+    my $stats = eval { $plugin->get_static_stats($self->{haenv}, $sid) };
     die "did not get static service usage information for '$sid'\n" if !$stats;
 
     my $service_stats = {
-- 
2.47.3

[PATCH ha-manager v2 08/10] move static service stats repository to PVE::HA::Usage::Static

[Daniel Kral]

Since the static service stats are already processed by their individual
plugin's implementation of get_static_stats_from_config(...), the static
service stats repository can be used by the static load scheduler
directly.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 src/PVE/HA/Env.pm             |  8 +------
 src/PVE/HA/Env/PVE2.pm        | 42 +++++++++++------------------------
 src/PVE/HA/Manager.pm         |  1 -
 src/PVE/HA/Resources.pm       |  6 -----
 src/PVE/HA/Resources/PVECT.pm |  6 -----
 src/PVE/HA/Resources/PVEVM.pm |  6 -----
 src/PVE/HA/Sim/Env.pm         |  8 +------
 src/PVE/HA/Sim/Hardware.pm    | 13 +----------
 src/PVE/HA/Sim/Resources.pm   |  8 -------
 src/PVE/HA/Usage/Static.pm    | 23 +++++--------------
 10 files changed, 22 insertions(+), 99 deletions(-)

diff --git a/src/PVE/HA/Env.pm b/src/PVE/HA/Env.pm
index 39ed68cb..76216893 100644
--- a/src/PVE/HA/Env.pm
+++ b/src/PVE/HA/Env.pm
@@ -301,15 +301,9 @@ sub get_datacenter_settings {
 }
 
 sub get_static_service_stats {
-    my ($self, $id) = @_;
-
-    return $self->{plug}->get_static_service_stats($id);
-}
-
-sub update_static_service_stats {
     my ($self) = @_;
 
-    return $self->{plug}->update_static_service_stats();
+    return $self->{plug}->get_static_service_stats();
 }
 
 sub get_static_node_stats {
diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm
index 54270a40..9c64d955 100644
--- a/src/PVE/HA/Env/PVE2.pm
+++ b/src/PVE/HA/Env/PVE2.pm
@@ -49,8 +49,6 @@ sub new {
 
     $self->{nodename} = $nodename;
 
-    $self->{static_service_stats} = undef;
-
     return $self;
 }
 
@@ -510,40 +508,26 @@ sub get_datacenter_settings {
 }
 
 sub get_static_service_stats {
-    my ($self, $id) = @_;
-
-    # undef if update_static_service_stats(...) failed before
-    return undef if !defined($self->{static_service_stats});
-
-    return $self->{static_service_stats}->{$id};
-}
-
-sub update_static_service_stats {
     my ($self) = @_;
 
     my $properties = ['cores', 'cpulimit', 'memory', 'sockets', 'vcpus'];
-    my $service_stats = eval {
-        my $stats = {};
-        my $confs = PVE::Cluster::get_guest_config_properties($properties);
+    my $stats = {};
+    my $confs = PVE::Cluster::get_guest_config_properties($properties);
 
-        my $vmlist = PVE::Cluster::get_vmlist();
-        my $idlist = $vmlist->{ids} // {};
-        for my $id (keys %$idlist) {
-            my $type = eval { PVE::HA::Tools::get_ha_resource_type($idlist->{$id}->{type}) };
-            next if $@; # silently ignore unknown pve types
+    my $vmlist = PVE::Cluster::get_vmlist();
+    my $idlist = $vmlist->{ids} // {};
+    for my $id (keys %$idlist) {
+        my $type = eval { PVE::HA::Tools::get_ha_resource_type($idlist->{$id}->{type}) };
+        next if $@; # silently ignore unknown pve types
 
-            my $sid = "$type:$id";
-            my $conf = $confs->{$id} // {};
-            my $plugin = PVE::HA::Resources->lookup($type);
+        my $sid = "$type:$id";
+        my $conf = $confs->{$id} // {};
+        my $plugin = PVE::HA::Resources->lookup($type);
 
-            $stats->{$sid} = $plugin->get_static_stats_from_config($conf);
-        }
+        $stats->{$sid} = $plugin->get_static_stats_from_config($conf);
+    }
 
-        return $stats;
-    };
-    $self->log('warning', "unable to update static service stats cache - $@") if $@;
-
-    $self->{static_service_stats} = $service_stats;
+    return $stats;
 }
 
 sub get_static_node_stats {
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index b1dbe6a8..90c5be50 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -253,7 +253,6 @@ sub recompute_online_node_usage {
                 $online_node_usage = eval {
                     my $scheduler = PVE::HA::Usage::Static->new($haenv);
                     $scheduler->add_node($_) for $online_nodes->@*;
-                    $haenv->update_static_service_stats();
                     return $scheduler;
                 };
             } else {
diff --git a/src/PVE/HA/Resources.pm b/src/PVE/HA/Resources.pm
index 01facac8..4238d9be 100644
--- a/src/PVE/HA/Resources.pm
+++ b/src/PVE/HA/Resources.pm
@@ -183,12 +183,6 @@ sub get_static_stats_from_config {
     die "implement in subclass";
 }
 
-sub get_static_stats {
-    my ($class, $haenv, $sid) = @_;
-
-    die "implement in subclass";
-}
-
 # package PVE::HA::Resources::IPAddr;
 
 # use strict;
diff --git a/src/PVE/HA/Resources/PVECT.pm b/src/PVE/HA/Resources/PVECT.pm
index bda33717..79bb7c83 100644
--- a/src/PVE/HA/Resources/PVECT.pm
+++ b/src/PVE/HA/Resources/PVECT.pm
@@ -168,10 +168,4 @@ sub get_static_stats_from_config {
     };
 }
 
-sub get_static_stats {
-    my ($class, $haenv, $sid) = @_;
-
-    return $haenv->get_static_service_stats($sid);
-}
-
 1;
diff --git a/src/PVE/HA/Resources/PVEVM.pm b/src/PVE/HA/Resources/PVEVM.pm
index 786c5130..5a0ac348 100644
--- a/src/PVE/HA/Resources/PVEVM.pm
+++ b/src/PVE/HA/Resources/PVEVM.pm
@@ -189,10 +189,4 @@ sub get_static_stats_from_config {
     };
 }
 
-sub get_static_stats {
-    my ($class, $haenv, $sid) = @_;
-
-    return $haenv->get_static_service_stats($sid);
-}
-
 1;
diff --git a/src/PVE/HA/Sim/Env.pm b/src/PVE/HA/Sim/Env.pm
index 32b5224c..0e3b02e5 100644
--- a/src/PVE/HA/Sim/Env.pm
+++ b/src/PVE/HA/Sim/Env.pm
@@ -489,15 +489,9 @@ sub get_datacenter_settings {
 }
 
 sub get_static_service_stats {
-    my ($self, $id) = @_;
-
-    return $self->{hardware}->get_static_service_stats($id);
-}
-
-sub update_static_service_stats {
     my ($self) = @_;
 
-    return $self->{hardware}->update_static_service_stats();
+    return $self->{hardware}->get_static_service_stats();
 }
 
 sub get_static_node_stats {
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index 60eb9867..062d9846 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -525,8 +525,6 @@ sub new {
 
     $self->{service_config} = $self->read_service_config();
 
-    $self->{static_service_stats} = undef;
-
     return $self;
 }
 
@@ -1063,22 +1061,13 @@ sub watchdog_update {
 }
 
 sub get_static_service_stats {
-    my ($self, $id) = @_;
-
-    # undef if update_static_service_stats(...) failed before
-    return undef if !defined($self->{static_service_stats});
-
-    return $self->{static_service_stats}->{$id};
-}
-
-sub update_static_service_stats {
     my ($self) = @_;
 
     my $filename = "$self->{statusdir}/static_service_stats";
     my $stats = eval { PVE::HA::Tools::read_json_from_file($filename) };
     $self->log('warning', "unable to update static service stats cache - $@") if $@;
 
-    $self->{static_service_stats} = $stats;
+    return $stats;
 }
 
 sub get_static_node_stats {
diff --git a/src/PVE/HA/Sim/Resources.pm b/src/PVE/HA/Sim/Resources.pm
index ed7d9c5a..9b2f3b60 100644
--- a/src/PVE/HA/Sim/Resources.pm
+++ b/src/PVE/HA/Sim/Resources.pm
@@ -137,12 +137,4 @@ sub remove_locks {
     return undef;
 }
 
-sub get_static_stats {
-    my ($class, $haenv, $sid) = @_;
-
-    my $hardware = $haenv->hardware();
-
-    return $hardware->get_static_service_stats($sid);
-}
-
 1;
diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm
index acc3533c..48622d62 100644
--- a/src/PVE/HA/Usage/Static.pm
+++ b/src/PVE/HA/Usage/Static.pm
@@ -14,12 +14,15 @@ sub new {
     my $node_stats = eval { $haenv->get_static_node_stats() };
     die "did not get static node usage information - $@" if $@;
 
+    my $service_stats = eval { $haenv->get_static_service_stats() };
+    die "did not get static service usage information - $@" if $@;
+
     my $scheduler = eval { PVE::RS::ResourceScheduling::Static->new(); };
     die "unable to initialize static scheduling - $@" if $@;
 
     return bless {
         'node-stats' => $node_stats,
-        'service-stats' => {},
+        'service-stats' => $service_stats,
         haenv => $haenv,
         scheduler => $scheduler,
         'node-services' => {}, # Services on each node. Fallback if scoring calculation fails.
@@ -63,20 +66,8 @@ sub contains_node {
 my sub get_service_usage {
     my ($self, $sid) = @_;
 
-    return $self->{'service-stats'}->{$sid} if $self->{'service-stats'}->{$sid};
-
-    my (undef, $type, undef) = $self->{haenv}->parse_sid($sid);
-    my $plugin = PVE::HA::Resources->lookup($type);
-
-    my $stats = eval { $plugin->get_static_stats($self->{haenv}, $sid) };
-    die "did not get static service usage information for '$sid'\n" if !$stats;
-
-    my $service_stats = {
-        maxcpu => $stats->{maxcpu} + 0.0, # containers allow non-integer cpulimit
-        maxmem => int($stats->{maxmem}),
-    };
-
-    $self->{'service-stats'}->{$sid} = $service_stats;
+    my $service_stats = $self->{'service-stats'}->{$sid}
+        or die "did not get static service usage information for '$sid'\n";
 
     return $service_stats;
 }
@@ -101,8 +92,6 @@ sub remove_service_usage {
 
     eval { $self->{scheduler}->remove_service_usage($sid) };
     $self->{haenv}->log('warning', "unable to remove service '$sid' usage - $@") if $@;
-
-    delete $self->{'service-stats'}->{$sid}; # Invalidate old service stats
 }
 
 sub score_nodes_to_start_service {
-- 
2.47.3

[PATCH ha-manager v2 09/10] usage: augment service stats with node and state information

[Daniel Kral]

Augment the service stats with the node and state information, which is
necessary for non-HA resources to be added to the accounting done by the
implementations of PVE::HA::Usage in an upcoming patch.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 src/PVE/HA/Env/PVE2.pm     | 49 +++++++++++++++++++++++++++++++-------
 src/PVE/HA/Sim/Hardware.pm | 27 ++++++++++++++++++++-
 src/PVE/HA/Usage/Static.pm |  2 +-
 3 files changed, 68 insertions(+), 10 deletions(-)

diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm
index 9c64d955..b544de4c 100644
--- a/src/PVE/HA/Env/PVE2.pm
+++ b/src/PVE/HA/Env/PVE2.pm
@@ -38,6 +38,12 @@ PVE::HA::Rules->init(property_isolation => 1);
 
 my $lockdir = "/etc/pve/priv/lock";
 
+# rrd entry indices for VM and CT guests
+# taken from PVE::Service::pvestatd::update_{lxc,qemu}_status()
+use constant {
+    RRD_VM_INDEX_STATUS => 2,
+};
+
 sub new {
     my ($this, $nodename) = @_;
 
@@ -507,24 +513,51 @@ sub get_datacenter_settings {
     };
 }
 
-sub get_static_service_stats {
-    my ($self) = @_;
-
-    my $properties = ['cores', 'cpulimit', 'memory', 'sockets', 'vcpus'];
-    my $stats = {};
-    my $confs = PVE::Cluster::get_guest_config_properties($properties);
-
+my sub get_cluster_service_stats {
     my $vmlist = PVE::Cluster::get_vmlist();
     my $idlist = $vmlist->{ids} // {};
+
+    my $rrd = PVE::Cluster::rrd_dump();
+
+    my $stats = {};
     for my $id (keys %$idlist) {
         my $type = eval { PVE::HA::Tools::get_ha_resource_type($idlist->{$id}->{type}) };
         next if $@; # silently ignore unknown pve types
 
         my $sid = "$type:$id";
+        my $nodename = $idlist->{$id}->{node};
+
+        my $rrdentry = $rrd->{"pve-vm-9.0/$id"} // [];
+        # can be any QMP RunState, but 'running' is the only active VM state
+        my $status = $rrdentry->[RRD_VM_INDEX_STATUS] // "stopped";
+        my $state = $status eq "running" ? "started" : "stopped";
+
+        $stats->{$sid} = {
+            id => $id,
+            node => $nodename,
+            state => $state,
+            type => $type,
+            usage => {},
+        };
+    }
+
+    return $stats;
+}
+
+sub get_static_service_stats {
+    my ($self) = @_;
+
+    my $properties = ['cores', 'cpulimit', 'memory', 'sockets', 'vcpus'];
+    my $stats = get_cluster_service_stats();
+    my $confs = PVE::Cluster::get_guest_config_properties($properties);
+
+    for my $sid (keys %$stats) {
+        my ($id, $type) = $stats->{$sid}->@{qw(id type)};
+
         my $conf = $confs->{$id} // {};
         my $plugin = PVE::HA::Resources->lookup($type);
 
-        $stats->{$sid} = $plugin->get_static_stats_from_config($conf);
+        $stats->{$sid}->{usage} = $plugin->get_static_stats_from_config($conf);
     }
 
     return $stats;
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index 062d9846..dd7d277c 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -1060,13 +1060,38 @@ sub watchdog_update {
     return &$modify_watchog($self, $code);
 }
 
+my sub get_cluster_service_stats {
+    my ($self) = @_;
+
+    my $stats = {};
+    for my $sid (keys $self->{service_config}->%*) {
+        my $cfg = $self->{service_config}->{$sid};
+
+        $stats->{$sid} = {
+            node => $cfg->{node},
+            state => $cfg->{state},
+            usage => {},
+        };
+    }
+
+    return $stats;
+}
+
 sub get_static_service_stats {
     my ($self) = @_;
 
+    my $stats = get_cluster_service_stats($self);
+
     my $filename = "$self->{statusdir}/static_service_stats";
-    my $stats = eval { PVE::HA::Tools::read_json_from_file($filename) };
+    my $usage_stats = eval { PVE::HA::Tools::read_json_from_file($filename) };
     $self->log('warning', "unable to update static service stats cache - $@") if $@;
 
+    for my $sid (keys %$stats) {
+        next if !defined($usage_stats->{$sid});
+
+        $stats->{$sid}->{usage} = $usage_stats->{$sid};
+    }
+
     return $stats;
 }
 
diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm
index 48622d62..98752691 100644
--- a/src/PVE/HA/Usage/Static.pm
+++ b/src/PVE/HA/Usage/Static.pm
@@ -66,7 +66,7 @@ sub contains_node {
 my sub get_service_usage {
     my ($self, $sid) = @_;
 
-    my $service_stats = $self->{'service-stats'}->{$sid}
+    my $service_stats = $self->{'service-stats'}->{$sid}->{usage}
         or die "did not get static service usage information for '$sid'\n";
 
     return $service_stats;
-- 
2.47.3

[PATCH ha-manager v2 10/10] include running non-HA resources in the scheduler's accounting

[Daniel Kral]

As the service stats repository includes non-HA resources as well, use
it to add the static usage stats of these running, non-HA resources to
the scheduler.

Signed-off-by: Daniel Kral <d.kral@proxmox.com>
---
 src/PVE/HA/Env.pm          |  6 ++++++
 src/PVE/HA/Env/PVE2.pm     |  6 ++++++
 src/PVE/HA/Manager.pm      | 15 ++++++++++++++-
 src/PVE/HA/Sim/Env.pm      |  6 ++++++
 src/PVE/HA/Sim/Hardware.pm |  6 ++++++
 src/PVE/HA/Usage.pm        |  2 +-
 src/PVE/HA/Usage/Basic.pm  |  2 +-
 src/PVE/HA/Usage/Static.pm |  5 +----
 8 files changed, 41 insertions(+), 7 deletions(-)

diff --git a/src/PVE/HA/Env.pm b/src/PVE/HA/Env.pm
index 76216893..3643292e 100644
--- a/src/PVE/HA/Env.pm
+++ b/src/PVE/HA/Env.pm
@@ -300,6 +300,12 @@ sub get_datacenter_settings {
     return $self->{plug}->get_datacenter_settings();
 }
 
+sub get_basic_service_stats {
+    my ($self) = @_;
+
+    return $self->{plug}->get_basic_service_stats();
+}
+
 sub get_static_service_stats {
     my ($self) = @_;
 
diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm
index b544de4c..04cd1bfe 100644
--- a/src/PVE/HA/Env/PVE2.pm
+++ b/src/PVE/HA/Env/PVE2.pm
@@ -544,6 +544,12 @@ my sub get_cluster_service_stats {
     return $stats;
 }
 
+sub get_basic_service_stats {
+    my ($self) = @_;
+
+    return get_cluster_service_stats();
+}
+
 sub get_static_service_stats {
     my ($self) = @_;
 
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 90c5be50..421c17da 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -245,13 +245,15 @@ sub recompute_online_node_usage {
 
     my $online_nodes = $self->{ns}->list_online_nodes();
 
+    my $service_stats;
     my $online_node_usage;
 
     if (my $mode = $self->{crs}->{scheduler}) {
         if ($mode eq 'static') {
             if ($have_static_scheduling) {
                 $online_node_usage = eval {
-                    my $scheduler = PVE::HA::Usage::Static->new($haenv);
+                    $service_stats = $haenv->get_static_service_stats();
+                    my $scheduler = PVE::HA::Usage::Static->new($haenv, $service_stats);
                     $scheduler->add_node($_) for $online_nodes->@*;
                     return $scheduler;
                 };
@@ -271,6 +273,7 @@ sub recompute_online_node_usage {
 
     # fallback to the basic algorithm in any case
     if (!$online_node_usage) {
+        $service_stats = $haenv->get_basic_service_stats();
         $online_node_usage = PVE::HA::Usage::Basic->new($haenv);
         $online_node_usage->add_node($_) for $online_nodes->@*;
     }
@@ -281,6 +284,16 @@ sub recompute_online_node_usage {
         $online_node_usage->add_service_usage($sid, $sd->{state}, $sd->{node}, $sd->{target});
     }
 
+    # add remaining non-HA resources to online node usage
+    for my $sid (sort keys %$service_stats) {
+        next if $self->{ss}->{$sid};
+
+        my ($node, $state) = $service_stats->{$sid}->@{qw(node state)};
+
+        # the migration target is not known for non-HA resources
+        $online_node_usage->add_service_usage($sid, $state, $node, undef);
+    }
+
     $self->{online_node_usage} = $online_node_usage;
 }
 
diff --git a/src/PVE/HA/Sim/Env.pm b/src/PVE/HA/Sim/Env.pm
index 0e3b02e5..ad51245c 100644
--- a/src/PVE/HA/Sim/Env.pm
+++ b/src/PVE/HA/Sim/Env.pm
@@ -488,6 +488,12 @@ sub get_datacenter_settings {
     };
 }
 
+sub get_basic_service_stats {
+    my ($self) = @_;
+
+    return $self->{hardware}->get_basic_service_stats();
+}
+
 sub get_static_service_stats {
     my ($self) = @_;
 
diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm
index dd7d277c..59afb44a 100644
--- a/src/PVE/HA/Sim/Hardware.pm
+++ b/src/PVE/HA/Sim/Hardware.pm
@@ -1077,6 +1077,12 @@ my sub get_cluster_service_stats {
     return $stats;
 }
 
+sub get_basic_service_stats {
+    my ($self) = @_;
+
+    return get_cluster_service_stats($self);
+}
+
 sub get_static_service_stats {
     my ($self) = @_;
 
diff --git a/src/PVE/HA/Usage.pm b/src/PVE/HA/Usage.pm
index 1be5fa09..9f19a82b 100644
--- a/src/PVE/HA/Usage.pm
+++ b/src/PVE/HA/Usage.pm
@@ -4,7 +4,7 @@ use strict;
 use warnings;
 
 sub new {
-    my ($class, $haenv) = @_;
+    my ($class, $haenv, $service_stats) = @_;
 
     die "implement in subclass";
 }
diff --git a/src/PVE/HA/Usage/Basic.pm b/src/PVE/HA/Usage/Basic.pm
index ef9ae3d6..2584727b 100644
--- a/src/PVE/HA/Usage/Basic.pm
+++ b/src/PVE/HA/Usage/Basic.pm
@@ -6,7 +6,7 @@ use warnings;
 use base qw(PVE::HA::Usage);
 
 sub new {
-    my ($class, $haenv) = @_;
+    my ($class, $haenv, $service_stats) = @_;
 
     return bless {
         nodes => {},
diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm
index 98752691..c8460fd7 100644
--- a/src/PVE/HA/Usage/Static.pm
+++ b/src/PVE/HA/Usage/Static.pm
@@ -9,14 +9,11 @@ use PVE::RS::ResourceScheduling::Static;
 use base qw(PVE::HA::Usage);
 
 sub new {
-    my ($class, $haenv) = @_;
+    my ($class, $haenv, $service_stats) = @_;
 
     my $node_stats = eval { $haenv->get_static_node_stats() };
     die "did not get static node usage information - $@" if $@;
 
-    my $service_stats = eval { $haenv->get_static_service_stats() };
-    die "did not get static service usage information - $@" if $@;
-
     my $scheduler = eval { PVE::RS::ResourceScheduling::Static->new(); };
     die "unable to initialize static scheduling - $@" if $@;
 
-- 
2.47.3

Re: [PATCH-SERIES ha-manager v2 00/10] include non-HA resources in usage accounting

[Daniel Kral]

On Thu Mar 19, 2026 at 5:33 PM CET, Daniel Kral wrote:
> This series is split out from [0] to make the static/dynamic load
> balancer series a little more straight forward to reason through.
>
> This series adds any running non-HA resource into the existing basic and
> static usage scheduler accounting to base decisions on the usage of the
> whole cluster instead of just the HA resources.
>
> For clusters where every guest is configured as a HA resource nothing
> should change.
>
> I tested a few simple cases in a 3 node cluster for both the basic and
> static usage which are variations of:
>
> - 1 running HA resource on each node, starting them on any node (with
>   homogeneous nodes and guests) will rebalance them on the
>   alphabetically first node as before
> - 1 running HA resource on node1, running 3 non-HA resources on node2,
>   starting another HA resource on node1 will rebalance it to node3
> - 3 non-HA resources on node1, starting a HA resource on node1 will
>   rebalance it to node2
> - non-running HA and non-HA resources are excluded from the usage
>   accounting
> - starting non-HA resources doesn't have any effect obviously
>

Sorry for the noise, there are only two changes in the series in patch
#9:
- removing the duplicate $sid as suggested by @Thomas, and
- adding constants for indexing the rrd status indirectly suggested by
  @Thomas in a later patch in the series.

Otherwise the series is identical to the patches from [0].

> [0] https://lore.proxmox.com/pve-devel/20260217141437.584852-1-d.kral@proxmox.com/

Re: [PATCH ha-manager v2 01/10] rename static node stats to be consistent with similar interfaces

[Daniel Kral]

On Thu Mar 19, 2026 at 5:33 PM CET, Daniel Kral wrote:
> diff --git a/src/PVE/HA/Usage/Static.pm b/src/PVE/HA/Usage/Static.pm
> index d586b603..395be871 100644
> --- a/src/PVE/HA/Usage/Static.pm
> +++ b/src/PVE/HA/Usage/Static.pm
> @@ -33,10 +33,10 @@ sub add_node {
>  
>      my $stats = $self->{'node-stats'}->{$nodename}
>          or die "did not get static node usage information for '$nodename'\n";
> -    die "static node usage information for '$nodename' missing cpu count\n" if !$stats->{cpus};
> -    die "static node usage information for '$nodename' missing memory\n" if !$stats->{memory};
> +    die "static node usage information for '$nodename' missing cpu count\n" if !$stats->{maxcpu};
> +    die "static node usage information for '$nodename' missing memory\n" if !$stats->{maxmem};

@Dominik pointed this out to me off-list: even though it's unlikely that
the node max stats will ever be exactly 0, this should be a
!defined($stats->{...}) rather than a !$stats->{...}.

This was existent, so I'd send a follow-up instead of re-sending the
series to minimize noise on the list.

>  
> -    eval { $self->{scheduler}->add_node($nodename, int($stats->{cpus}), int($stats->{memory})); };
> +    eval { $self->{scheduler}->add_node($nodename, int($stats->{maxcpu}), int($stats->{maxmem})); };
>      die "initializing static node usage for '$nodename' failed - $@" if $@;
>  }
>