From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 8EBDA1FF0AD for ; Sun, 20 Sep 2026 18:23:36 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 1C097215BB; Sun, 20 Sep 2026 18:23:25 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=SZ0UfK6p0q/e8IcvL2Rzmzu7yGaOQikrQnYgggb00znLFYId934dKa1B4iKv7+tRD8ldjzE3ECQeb7nhpJyWF8Kaf9Pj1ifQqtErkcG1QrE6+E8C0kn20ycxLBF72oJxEgQuO2VD0U5Wn5VAfDhe16LltoSHvLy/JdVFAj8NvEhzYdVyrj0ZH386dLOxXou+p5KyVv2Bepd+sWqtnajvAYLqgVX31IAB7xqcTTZo/Ugtk7wDg0D5tBRvhUEqkIVfrcw4qoDpgdWu79ZMyAO6E5kt7h8Sf02vxoZiI2vkhWdpxxo/eOmMG7+jnU3nCHvQWSOdRLryOjesb5MfngC/aQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=t+drnIuyZTybwc/UwGLjERR6RxDIY8LQAunCz3nqrmI=; b=Fg1NbjDpl3flX9CnXKDQzWOY01ncrIzh9/z3eVLTHVfu/55OvU8IpKgN2+q8p/MJp7vdJWl5w3N5PylUuG77zFN9tX48mxlf2B3pwyGNZ+Sxzr7lpj8epyCglKWgvFBFGRw5nwKhKmPcSHIk/UvHpO0jo9vLtr7VapLKY9WAkSVQBXDtUMLZemmvAEfWZ2QLbOna0Yx1hO37k3iu5Q614QX00sdxIAiBB85Vt3ry0usEP8O2yTi+cXdhPQzamzV7MbMrDvHWuWQ3ZOm0V0DWUSnV/Kd6braHIGdz4k3W/wByx6OjRRt+JNZV/ogK4NEIVAcLDS1P884NIh32QGvqJw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ryomherold.dk; dmarc=pass action=none header.from=ryomherold.dk; dkim=pass header.d=ryomherold.dk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RyomHerold.dk; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=t+drnIuyZTybwc/UwGLjERR6RxDIY8LQAunCz3nqrmI=; b=o6jVHirpaobX+FAqoUWN0ET2s7YpTzy7FkPLm61kBTfXPB6MbzJjcmO/RzrfywRmkiVYhgRbkgtwC4PVM3CFgnnnoZg3hrS09Q9hTqFiBHpfIBfacwPf8KBcT9mr1QuRCO5rgzog8cXQRCb3shN3wI/ndm0uieX1yWSwRucx6CAqcoj8u3DzIk3UjZnJ/MFQ9GE08bn+Rz8g2Q5GSBqrO6f+6pJaK+IYi6FdROISTTGQZU6vZ19K4kDWiF2Tn2tJuKiEx0tNqctAmG0vXwk5CN9b//kKEcgwWj6e5dH9fskULkibHhpOxJ5ZDjIXDor/ftSfQ+0a2uwr+ViF0iOZFg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=RyomHerold.dk; From: Michael Ryom To: pve-devel@lists.proxmox.com Subject: [PATCH ha-manager 7/7] manager: auto rebalance: only balance under actual node resource pressure Date: Sun, 20 Sep 2026 18:22:14 +0200 Message-ID: <20260920162220.574802-8-Michael@RyomHerold.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260920162220.574802-1-Michael@RyomHerold.dk> References: <20260920162220.574802-1-Michael@RyomHerold.dk> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: TL2P290CA0008.ISRP290.PROD.OUTLOOK.COM (2603:1096:950:2::11) To AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:619::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR10MB7231:EE_|AM0PR10MB883391:EE_ X-MS-Office365-Filtering-Correlation-Id: 56fc5857-3d4e-49e8-3173-08df173377e7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|376014|1800799024|366016|10067099003|3023799007|6133799003|5023799004|22082099003|18002099003|56012099006; X-Microsoft-Antispam-Message-Info: 2I7OkhNlHKzifADsF+IOAyfDOhhpbZrSs3DDs40pKa43G+A8J7b8/y+Cn3csw/QFZKe/oKyC52Bg4gtuw8TwaXuecnlauADT5o7b0xiPLOGVOno0JL0DtFs+9OG5ZFRMVYTFr9K6n4aIwo4sUFTMu9I53a0lnO+Z3mJjGdHgXxxUBrzYNhMo21NylZ113k9E6YAnKsWKVWjk9O7qZ8XnjVW0iFuGI7jHTuNyW38EqNhOQ8qw15LKy/XHxR901YeQjQX4HNc+ylsuwWQN6cX5E0GHX+iUe+XOOJmJBpkcSHaS3l18hXe3CdCmvYV+rmgFHtK58lo0spUAPpwEBlAPK2N3VaTpNCsWB9ncuKALsAxvy7wSV/gLOIaDX5SARwXv+eCC6ZhnabxDHbAxfj6AX8BkSiF3WygIsU5yTN6qojhYYqra05LwUhJ4NWF8VQ2YUK1hwfi7jJE2oTa70wNBr3XfB+XdwV7jnwEn5a9zL/Ll2diCDepem1JKn18jIFd5u8iE0RkjKUexdkN5PQ5pwL9dtV0WIrk4UaopsmPqoWq/n1oeHPvM52Ivgco9XFqsBJdl6h4Lne3OhGJi3gK0tumOpeuZaQipNlrRQWq/5a8iZSIzXCarVvSjbiP520AJuQAQTmnVtlRwsYsiEyzxYrmcgdY0fGNPTRGIoI/9N7Q= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(376014)(1800799024)(366016)(10067099003)(3023799007)(6133799003)(5023799004)(22082099003)(18002099003)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?CGbaGVQyAd2OJZ7yy55Hjg4SFEEJp4vVagLAmp0UU+GtupiR7298jJsDEbF2?= =?us-ascii?Q?NGcUVY0yOTILMPDnoeoEzLkaNgfALzlSw/2qHyPqsdH34eAvq1T8tbfKJEu6?= =?us-ascii?Q?yvzosZSZM+Dwr49FE8LOKxRDDgWeW1X0gDSKz2LBiST2Rgzli9y/81/uncC8?= =?us-ascii?Q?KU4899eZvpTNlIXDYkW6AfT/QFmqny9ks8aLKiGFOymH+7EQBrBAc6jztxza?= =?us-ascii?Q?PC5731VCN/fIFWNwNS3M0uU0rnAv/z09Zkb4tSRm5Pv0VPXN+woH1nxiVvCJ?= =?us-ascii?Q?bh2gvBBGfSngUzeSELQq6r0XKQFaM9s8CDmbvAYWC/ebfx3H9E777bPQF7DD?= =?us-ascii?Q?CBCCooiX8JFlqI3jEqEVoUcGi2OGVSRweyAjxPE5rE0TrKgM9MCOZqTVn6ve?= =?us-ascii?Q?MskImKyVoDLV3CxjJaLNvdO5DsWowqFomUxmthgXvYCvHLqKpHhWTcgiGiPC?= =?us-ascii?Q?kWOl7BkteHvDx5CaOwUuJItbz2GftTvJdGLTNHk7nB79IuqFxI8jggHlWnjp?= =?us-ascii?Q?zSwMxy2nEeX269oqqrTzo3R4mo6Y2019tz2+EkXnnnjkngiRAFyJahvjsCva?= =?us-ascii?Q?Vgy6kiSTUH22rHS7m4iu9tas5CCO5hr0GiMEKufm9HWrrxm/Ixi4hLt7yfOc?= =?us-ascii?Q?3pzC7XmABBMbAyCv9dBTgi2RNa4hPpuQlw1rL59A79pvXiTcaW8FgeUj9e3Y?= =?us-ascii?Q?UDmja/q56kKzuJuGDRTtLNxA2YvfrlGKY2hfLPf5sniZRHNpUx2i5E92OVsj?= =?us-ascii?Q?S0zT8gxtEt1owmPf7HgSf6ouyKi8+LnE+Ba8M3BKDyfMKZbbU7KQOXPDiNYO?= =?us-ascii?Q?UILvAIRdGaKXkfN3dgmeqGNpYKk17MFJLtxWOoIoMzLWHXJf+Pkc4sxjBxRJ?= =?us-ascii?Q?xGFpD0RG0Yd0l8CqZJIbejE+ua8XhVL0aEqBbOUJdA4ylboi0sr0Cz0i2I9D?= =?us-ascii?Q?IsqML122Be4xEzW2ljwdoTnlG0B/AOGW82IoQW/jLFTWUkXHYuAYyIEBetrb?= =?us-ascii?Q?wHanKywQtLMPP9yyMHYfLBjIOj9ItCbIIG77SnmzOvOgBxdgDo8Ut7jKL8S1?= =?us-ascii?Q?nR0wDn34T20JBXv8om3vmEjd7d6FkncOA1lveQLBusugNKsMSHZIaivIImuR?= =?us-ascii?Q?uMmhUu7G2aMGtEjrN0MyFEdgzOpbjIR6HnPqUAnGtZrp5JJOvOxXfQu7JZAe?= =?us-ascii?Q?3Vxy9l2OFoLFmtYQwtVux0ebIjBzpF3uTEfGQuKkmtAGBwwLYyvVYgT8X64s?= =?us-ascii?Q?x7iaqAdDtj3qfOqRgGH3zWzAZuK95MxlaV1xux7b8vZU2Ii0A1OvC3VOon2b?= =?us-ascii?Q?oQmsGq0l2r7L0Kyr4IqWIE43jRGdy0X5TSMHRh18TY2NjahOvutfvvWDXEHz?= =?us-ascii?Q?jJW/zzGmHzHiCISV/ISPnPqgHL2XKz3su3If6jHtazF8jFp8VlWqeeCIHI75?= =?us-ascii?Q?BYgrKylX2G1a6Z/6npAW8zchSMtws6e79Dbb9Up1GVsaw965Aips/3bteJ8V?= =?us-ascii?Q?kHLyfMKpeNAyQcwxOI+rCYfOGTJuNCT9vTn5G2nwKOEkar5bxwr9gyk0Xo4P?= =?us-ascii?Q?RVxuKAsJ4UHQHwiW5OGNlsH18gIjvq6AzsKLii1Vc7Uqz/WKlREf4fcZKGc0?= =?us-ascii?Q?rLoweospEdO+wDAG2Q35AOtOkSXJHW2GX+9Q9u0jku+ulfXGiz75NWat96BM?= =?us-ascii?Q?Zk33w1bglj3jHZgHNpwJ3vrSuRRxktQtAjspOSLK9Hdehc0v/hZD7vKUUrHZ?= =?us-ascii?Q?mnuUPMYfuA=3D=3D?= X-OriginatorOrg: RyomHerold.dk X-MS-Exchange-CrossTenant-Network-Message-Id: 56fc5857-3d4e-49e8-3173-08df173377e7 X-MS-Exchange-CrossTenant-AuthSource: AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2026 16:23:13.4398 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 26a2499e-4646-4888-a695-c64736a56807 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: daQStweOTCrNXIHLXxMHKaBVyOrm+kGmAz8ENszJ8HhurdDVymw1G9jlZLAYX6MisM30Fy/NHz4GFKXfdUvPNQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR10MB883391 X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: WEKZF47D7CO53LLXTQON65UYZETTUKXC X-Message-ID-Hash: WEKZF47D7CO53LLXTQON65UYZETTUKXC X-MailFrom: Michael@RyomHerold.dk X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Michael Ryom X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The load balancer acted on load asymmetry alone. But asymmetric node loads are not, by themselves, a performance problem: on an almost idle cluster, the imbalance metric (a coefficient of variation, std/mean) becomes hypersensitive, and observed live, a barely-qualifying migration (60.4% -> 54.0% expected imbalance) was issued at 2-7% node load, with nothing to gain for any workload. The purpose of the load balancer is better performance for the guests, not symmetric load figures. Gate the balancer on the pressure stall information (PSI) that the nodes already broadcast in their RRD stats: only balance while some node reports a cpu or memory some-pressure (avg10) of at least 10%, i.e. work on that node actually loses a meaningful share of its time waiting on resources and could run better with more headroom. Unlike a load threshold, this criterion is independent of the cluster's size and load level, and it directly expresses the question "would anything perform better elsewhere?" - akin to the contention-driven (rather than load-driven) approach of other schedulers like VMware DRS. Deliberately only the node pressure is considered, although the guests' own pressure stats are collected as well: a guest saturating its own vCPUs or its own memory reports pressure too, but gets neither more cores nor more memory by being migrated, while host-caused starvation (contention with other guests or host processes) shows up in the node pressure - and only that kind can be improved by a migration. The kernel's avg10 pressure values are already exponentially averaged over ten seconds, so no additional smoothing is applied. IO pressure is deliberately not considered for now, since with shared storage it usually cannot be improved by a migration. If no pressure information is available at all (e.g. nodes still broadcasting an older stats schema), balancing proceeds as before. The minimum pressure is a constant for now; like the other balancer constants, it could be exposed as a ha-auto-rebalance-* option later. Signed-off-by: Michael Ryom --- src/PVE/HA/Env/PVE2.pm | 25 +++++++++++++ src/PVE/HA/Manager.pm | 16 +++++++++ src/PVE/HA/Sim/Hardware.pm | 16 +++++++-- src/PVE/HA/Usage.pm | 9 +++++ src/PVE/HA/Usage/Dynamic.pm | 23 ++++++++++++ .../test-crs-dynamic-auto-rebalance9/README | 15 ++++++++ .../test-crs-dynamic-auto-rebalance9/cmdlist | 4 +++ .../datacenter.cfg | 6 ++++ .../dynamic_service_stats | 4 +++ .../hardware_status | 4 +++ .../log.expect | 36 +++++++++++++++++++ .../manager_status | 1 + .../service_config | 4 +++ .../static_service_stats | 4 +++ 14 files changed, 165 insertions(+), 2 deletions(-) create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/README create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/cmdlist create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/hardware_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/log.expect create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/manager_status create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/service_config create mode 100644 src/test/test-crs-dynamic-auto-rebalance9/static_service_stats diff --git a/src/PVE/HA/Env/PVE2.pm b/src/PVE/HA/Env/PVE2.pm index 8c2b03d..7d7fceb 100644 --- a/src/PVE/HA/Env/PVE2.pm +++ b/src/PVE/HA/Env/PVE2.pm @@ -47,6 +47,8 @@ use constant { RRD_VM_INDEX_MAXMEM => 7, RRD_VM_INDEX_MEM => 8, RRD_VM_INDEX_MEMHOST => 15, + RRD_VM_INDEX_PRESSURE_CPU_SOME => 16, + RRD_VM_INDEX_PRESSURE_MEM_SOME => 20, }; # rrd entry indices for PVE nodes @@ -56,8 +58,21 @@ use constant { RRD_NODE_INDEX_CPU => 5, RRD_NODE_INDEX_MAXMEM => 7, RRD_NODE_INDEX_MEM => 8, + RRD_NODE_INDEX_PRESSURE_CPU_SOME => 17, + RRD_NODE_INDEX_PRESSURE_MEM_SOME => 20, }; +# returns the numeric value of the given RRD column, or undef if the column +# is not present (e.g. nodes broadcasting an older schema) +my sub rrd_column_or_undef { + my ($rrdentry, $index) = @_; + + my $value = $rrdentry->[$index]; + return undef if !defined($value) || $value eq '' || $value eq 'U'; + + return $value + 0.0; +} + my $HOSTNAME_RE = qr/(?:[a-zA-Z0-9](?:[a-zA-Z0-9\-]{,61}?[a-zA-Z0-9])?)/; sub new { @@ -615,6 +630,11 @@ sub get_dynamic_service_stats { cpu => (($rrdentry->[RRD_VM_INDEX_CPU] || 0.0) + 0.0) * $maxcpu, maxmem => int($rrdentry->[RRD_VM_INDEX_MAXMEM] || 0), mem => $mem, + # PSI avg10 in percent; undef if the guest broadcasts an older + # schema without pressure information + pressurecpusome => rrd_column_or_undef($rrdentry, RRD_VM_INDEX_PRESSURE_CPU_SOME), + pressurememorysome => + rrd_column_or_undef($rrdentry, RRD_VM_INDEX_PRESSURE_MEM_SOME), }; } @@ -660,6 +680,11 @@ sub get_dynamic_node_stats { cpu => (($rrdentry->[RRD_NODE_INDEX_CPU] || 0.0) + 0.0) * $maxcpu, maxmem => int($rrdentry->[RRD_NODE_INDEX_MAXMEM] || 0), mem => int($rrdentry->[RRD_NODE_INDEX_MEM] || 0), + # PSI avg10 in percent; undef if the node broadcasts an older + # schema without pressure information + pressurecpusome => rrd_column_or_undef($rrdentry, RRD_NODE_INDEX_PRESSURE_CPU_SOME), + pressurememorysome => + rrd_column_or_undef($rrdentry, RRD_NODE_INDEX_PRESSURE_MEM_SOME), }; } diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm index 6e1827f..c34da89 100644 --- a/src/PVE/HA/Manager.pm +++ b/src/PVE/HA/Manager.pm @@ -155,6 +155,15 @@ my $auto_rebalance_failure_backoff_max = 3600; # minimum time before a resource is considered for rebalancing again after it # was successfully moved by the load balancer my $auto_rebalance_resource_cooldown = 600; +# minimum pressure stall value (PSI avg10, in percent) among the nodes for +# the load balancer to act at all: load asymmetry by itself is not a +# performance problem, and the node pressure stall information directly +# measures which share of its time work on the node spends waiting on +# resources - i.e. whether anything could actually run better with more +# headroom - independent of the cluster's size or load level; if no pressure +# information is available at all (older stats schema), balancing proceeds +# as before +my $auto_rebalance_min_pressure = 10.0; # minimum absolute improvement of the node imbalance for a rebalance motion; # the relative margin alone provides almost no hysteresis when a single # resource dominates the cluster load: with two nodes, a resource load v and @@ -368,6 +377,13 @@ sub load_balance { return; } + # nothing to gain from balancing while no workload is short on resources + my $max_pressure = $online_node_usage->max_pressure(); + if (defined($max_pressure) && $max_pressure < $auto_rebalance_min_pressure) { + $self->{sustained_imbalance_round} = 0; + return; + } + my $imbalance = $online_node_usage->calculate_node_imbalance(); # do not load balance unless imbalance threshold has been exceeded diff --git a/src/PVE/HA/Sim/Hardware.pm b/src/PVE/HA/Sim/Hardware.pm index aa1a886..8766a6f 100644 --- a/src/PVE/HA/Sim/Hardware.pm +++ b/src/PVE/HA/Sim/Hardware.pm @@ -791,7 +791,8 @@ sub get_cfs_state { # crm enable-node-maintenance # crm disable-node-maintenance # pve-manager-version set # note: this is NOT the *ha*-manager version -# node set-dynamic-stats [cpu ] [mem ] # load outside HA services +# node set-dynamic-stats [cpu ] [mem ] [pressurecpusome ] [pressurememorysome ] +# (load and pressure outside of the HA-managed services) # reboot # shutdown # restart-lrm @@ -888,7 +889,12 @@ sub sim_hardware_cmd { die "sim_hardware_cmd: missing target stat for '$action' command" if !@params; - my $conversions = { cpu => sub { 0.0 + $_[0] }, mem => sub { $_[0] * 1024**2 } }; + my $conversions = { + cpu => sub { 0.0 + $_[0] }, + mem => sub { $_[0] * 1024**2 }, + pressurecpusome => sub { 0.0 + $_[0] }, + pressurememorysome => sub { 0.0 + $_[0] }, + }; for my ($target, $val) (@params) { die "sim_hardware_cmd: missing value for '$action $target' command" @@ -1312,6 +1318,12 @@ sub get_dynamic_node_stats { $stats->{$node}->{cpu} = $cstatus->{$node}->{cpu} // 0.0; $stats->{$node}->{maxmem} = $stats->{$node}->{maxmem} // $default_node_maxmem; $stats->{$node}->{mem} = $cstatus->{$node}->{mem} // 0; + # PSI avg10 in percent; keep undef (pressure information not + # available) distinct from an explicit zero (no pressure) + for my $pressure (qw(pressurecpusome pressurememorysome)) { + $stats->{$node}->{$pressure} = $cstatus->{$node}->{$pressure} + if defined($cstatus->{$node}->{$pressure}); + } } my $service_conf = $self->read_service_config(); diff --git a/src/PVE/HA/Usage.pm b/src/PVE/HA/Usage.pm index 659ab30..e0890fb 100644 --- a/src/PVE/HA/Usage.pm +++ b/src/PVE/HA/Usage.pm @@ -66,6 +66,15 @@ sub calculate_node_imbalance { die "implement in subclass"; } +# Returns the highest pressure stall value (PSI avg10, in percent) among the +# nodes, or undef if the usage implementation has no notion of pressure +# (e.g. basic or static) or no node reported pressure information. +sub max_pressure { + my ($self) = @_; + + return undef; +} + sub score_best_balancing_migrations { my ($self, $migration_candidates, $limit) = @_; diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm index fc88ddf..b2ed79a 100644 --- a/src/PVE/HA/Usage/Dynamic.pm +++ b/src/PVE/HA/Usage/Dynamic.pm @@ -184,6 +184,29 @@ sub calculate_node_imbalance { return $node_imbalance // 0.0; } +# NOTE only the *node* pressure is considered: a guest saturating its own +# vCPUs or its own memory reports pressure too, but gets neither more cores +# nor more memory by being migrated, while starvation caused by the host +# (contention with other guests or host processes) shows up in the node's +# pressure - and only that kind can be improved by a migration +sub max_pressure { + my ($self) = @_; + + my $max_pressure; + + for my $node ($self->list_nodes()) { + my $stats = $self->{'node-stats'}->{$node} or next; + + for my $pressure ($stats->@{qw(pressurecpusome pressurememorysome)}) { + next if !defined($pressure); + $max_pressure = $pressure + if !defined($max_pressure) || $pressure > $max_pressure; + } + } + + return $max_pressure; +} + sub score_best_balancing_migrations { my ($self, $migration_candidates, $limit) = @_; diff --git a/src/test/test-crs-dynamic-auto-rebalance9/README b/src/test/test-crs-dynamic-auto-rebalance9/README new file mode 100644 index 0000000..d19bd97 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance9/README @@ -0,0 +1,15 @@ +Test that the auto rebalance system does not act while no node or HA-managed +service reports meaningful resource pressure, even though the relative +imbalance is high and a strongly qualifying rebalance motion exists. + +The cluster has two nodes with all load on node1: a HA resource excluded +from auto rebalancing (vm:101) and a movable HA resource vm:100. The +imbalance is 100% and migrating vm:100 to the empty node2 would improve it +far beyond the margin and the minimum absolute improvement. But both nodes +report zero pressure stall values, i.e. no workload spends any time waiting +on resources, so there is no performance problem a migration could solve and +no motion may be issued - regardless of node1's 30% load. + +Then node1 reports a CPU pressure of 25%, i.e. some workload now loses a +quarter of its time waiting for CPU, and the pending migration of vm:100 to +node2 must be carried out. diff --git a/src/test/test-crs-dynamic-auto-rebalance9/cmdlist b/src/test/test-crs-dynamic-auto-rebalance9/cmdlist new file mode 100644 index 0000000..3cb6c2a --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance9/cmdlist @@ -0,0 +1,4 @@ +[ + [ "power node1 on", "power node2 on" ], + [ "node node1 set-dynamic-stats pressurecpusome 25" ] +] diff --git a/src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg b/src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg new file mode 100644 index 0000000..01c8114 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance9/datacenter.cfg @@ -0,0 +1,6 @@ +{ + "crs": { + "ha": "dynamic", + "ha-auto-rebalance": 1 + } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats b/src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats new file mode 100644 index 0000000..7705333 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance9/dynamic_service_stats @@ -0,0 +1,4 @@ +{ + "vm:100": { "cpu": 9.6, "mem": 0 }, + "vm:101": { "cpu": 4.8, "mem": 0 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance9/hardware_status b/src/test/test-crs-dynamic-auto-rebalance9/hardware_status new file mode 100644 index 0000000..eeef22a --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance9/hardware_status @@ -0,0 +1,4 @@ +{ + "node1": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368, "pressurecpusome": 0.0, "pressurememorysome": 0.0 }, + "node2": { "power": "off", "network": "off", "maxcpu": 24, "maxmem": 34359738368, "pressurecpusome": 0.0, "pressurememorysome": 0.0 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance9/log.expect b/src/test/test-crs-dynamic-auto-rebalance9/log.expect new file mode 100644 index 0000000..aed4165 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance9/log.expect @@ -0,0 +1,36 @@ +info 0 hardware: starting simulation +info 20 cmdlist: execute power node1 on +info 20 node1/crm: status change startup => wait_for_quorum +info 20 node1/lrm: status change startup => wait_for_agent_lock +info 20 cmdlist: execute power node2 on +info 20 node2/crm: status change startup => wait_for_quorum +info 20 node2/lrm: status change startup => wait_for_agent_lock +info 20 node1/crm: got lock 'ha_manager_lock' +info 20 node1/crm: status change wait_for_quorum => master +info 20 node1/crm: using scheduler mode 'dynamic' +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online' +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online' +info 20 node1/crm: adding new service 'vm:100' on node 'node1' +info 20 node1/crm: adding new service 'vm:101' on node 'node1' +info 20 node1/crm: service 'vm:100': state changed from 'request_start' to 'started' (node = node1) +info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1) +info 21 node1/lrm: got lock 'ha_agent_node1_lock' +info 21 node1/lrm: status change wait_for_agent_lock => active +info 21 node1/lrm: starting service vm:100 +info 21 node1/lrm: service status vm:100 started +info 21 node1/lrm: starting service vm:101 +info 21 node1/lrm: service status vm:101 started +info 22 node2/crm: status change wait_for_quorum => slave +info 120 cmdlist: execute node node1 set-dynamic-stats pressurecpusome 25 +info 160 node1/crm: auto rebalance - migrate vm:100 to node2 (expected change for imbalance from 100.0% to 33.3%) +info 160 node1/crm: got crm command: migrate vm:100 node2 +info 160 node1/crm: migrate service 'vm:100' to node 'node2' +info 160 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node1, target = node2) +info 161 node1/lrm: service vm:100 - start migrate to node 'node2' +info 161 node1/lrm: service vm:100 - end migrate to node 'node2' +info 163 node2/lrm: got lock 'ha_agent_node2_lock' +info 163 node2/lrm: status change wait_for_agent_lock => active +info 180 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node2) +info 183 node2/lrm: starting service vm:100 +info 183 node2/lrm: service status vm:100 started +info 720 hardware: exit simulation - done diff --git a/src/test/test-crs-dynamic-auto-rebalance9/manager_status b/src/test/test-crs-dynamic-auto-rebalance9/manager_status new file mode 100644 index 0000000..0967ef4 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance9/manager_status @@ -0,0 +1 @@ +{} diff --git a/src/test/test-crs-dynamic-auto-rebalance9/service_config b/src/test/test-crs-dynamic-auto-rebalance9/service_config new file mode 100644 index 0000000..3f12542 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance9/service_config @@ -0,0 +1,4 @@ +{ + "vm:100": { "node": "node1", "state": "started" }, + "vm:101": { "node": "node1", "state": "started", "auto-rebalance": 0 } +} diff --git a/src/test/test-crs-dynamic-auto-rebalance9/static_service_stats b/src/test/test-crs-dynamic-auto-rebalance9/static_service_stats new file mode 100644 index 0000000..0dfad75 --- /dev/null +++ b/src/test/test-crs-dynamic-auto-rebalance9/static_service_stats @@ -0,0 +1,4 @@ +{ + "vm:100": { "maxcpu": 12.0, "maxmem": 8589934592 }, + "vm:101": { "maxcpu": 6.0, "maxmem": 8589934592 } +} -- 2.47.3