From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id ACFB81FF0AD for ; Sun, 18 Oct 2026 14:48:40 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id E16032178D; Sun, 18 Oct 2026 14:47:16 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=bpdmUcr6+R6oLEnK7uR44gmWtQiVp/mcBYcq8t4caCrlHcXqCmJm7PPJiZNFnE45f0l0T+jDMZImuT37bZiunWPqFZhJlrpb4qWODX2iYRntyYjZsFJO9yTPjgaC/OAFrOVk9odXTljhQoxmiq8YvQ+jyqMVyU2Pldb+wlyF1FWUclvxyBIwXMLNipYAb4nxjJKPDd4DHdkGRHlynCQOV9weAZ832s/gEGZaop19X1ulW6AdZo8K5FJA2LJdIqW/0gXT5gv5og2U4Jv04b147Ai3UFFbshLTh35mHO6mP4urEl77oV68uNeU59cWUJlVB/XyOe2sz+f+vBUo6PfHuQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=eZi43PihjTh6QWpBltdmqUrQigQGm8pLtQXawCb61G8=; b=B3EJBIc0s7btYE8rSEfkKvetsDvhbq2poDCsZsDV6IFcF6ejuijGUymgLM9TrF81vqU+HqLoOhQRXehQdwmG+IaT5XPISRPP7LBumP70oKftx+FgTtIezgsmp/xVrARXSKjoD2O0DkMNu4kfZ8ItkZ7eQdFp3KOLVnrGQMEpL7zkKGg89AX15L16GCGWyknX1xmbGYw5tZpixEFmBcdrmcl/I+bRIB9t6Peo3+PLy+0FWxVJ5xe9GIhXGSuACat5JyjYQnKpFW7hMWm31/NqEe+H+5IN22EVn8BW28ZS2e4t9f/tClgqfyE9xYRvjzc8ce8ahFc+2M6FlGV09ivGrA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=ryomherold.dk; dmarc=pass action=none header.from=ryomherold.dk; dkim=pass header.d=ryomherold.dk; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=RyomHerold.dk; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eZi43PihjTh6QWpBltdmqUrQigQGm8pLtQXawCb61G8=; b=3wF/BevbFA9xUB75ZcVmhg0zXiYjT6AX/J9sH7+qNm8j8zcZawv66d8fOyHEgOXgNGkFyd9qy5RVojHSySHU8sblCU6mBJT5YHPz/+yp71YWt31lm0kW7qtTP8hoEAM6io+LF6c/lrfH1ivWs5Za7vRZL29BfNolktJeweY6CYEYtWk3X7J8tWsag3ad0Drdq0FO9PptRGJzX+Mb0ImAORX69C02HgH+iciS+EXaod4J17kDNxQ3iw/t8kpQTgBf6mAbAYd/D88RIba1N+6/fSucpg8x24jCurwnAAE+Z06VeQkGc72W1+FrGteeCTXyZ2XpnNX72CoYfbjgKJjkyA== Authentication-Results: mx.microsoft.com 1; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=RyomHerold.dk; From: Michael Ryom To: pve-devel@lists.proxmox.com Subject: [PATCH ha-manager v2 12/14] usage: dynamic: smooth the unaccounted node load Date: Sun, 18 Oct 2026 14:45:56 +0200 Message-ID: <20261018124559.24305-13-Michael@RyomHerold.dk> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20261018124559.24305-1-Michael@RyomHerold.dk> References: <20261018124559.24305-1-Michael@RyomHerold.dk> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: MM0P280CA0046.SWEP280.PROD.OUTLOOK.COM (2603:10a6:190:b::20) To AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:619::17) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AS8PR10MB7231:EE_|PA4PR10MB5706:EE_ X-MS-Office365-Filtering-Correlation-Id: 9d3a9902-62f1-401c-58a0-08df2d15cdb4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|23010399003|6133799003|22082099003|10067099003|18002099003|261009223027099003|56012099006; X-Microsoft-Antispam-Message-Info: x4K/yiTh5mM90P7ntQv0uZ/5q/hhikPxNZRftALax4rIqri3gv2jl12M1NJd+oJSr2Yegnt9eQXGPOdkY4CHeK1e2+7akA71R7yyZ/wwx0a5JH/cfir5ChXcqlhQKqXSKozkwWAeFtnqiWNy39JRdz97LrDVS8J9V0JJu5/f1A/xhXySfltMUkJX2Y0+WkoqJoVnqQvL2JUBR4xmCnPvrEeMVWYPhADoElAiZ4D4ul5Kqrsk2rjB1+lyYC82QQC2+RhKQGsX/wglIURRSUpg8HCb+KV6cknIr+ck1GSWadDFTvJ9ZFEAg0r8vvh2zYOb7fMAG5K3ldKItqRRp+2tfGHzcYsy2D6BQmFAXcvX7Hp940h38xxjCXC6qNvVmMwdbhtX9BJMePXZ2zRcMdx4os9FvcZ3eK5L1WNgBP8j9N7APY/XlFnKT8Kq7W+Baeh/tAMTVG63lmffyc66Ijh4w0fGVzN9JfQ+MzzCpqR7tgS+i5Gk0/OQO+MbVCP2onR/+tX1ECDHqOwmjFQzf/AzeVmLvLIpoc+doh0wsqyB0IWQsPV9HX8e/m5JB/NlQawgfqHPyPANsb4WVyaOGoUoVbutiCrw5tvGmawe+/wOacYNeP1LBpJ9G/yFe7daAutXtIhPQKc5DJl5e1IdgC5UbiGf49UdNGC1exC1G0djeVQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(23010399003)(6133799003)(22082099003)(10067099003)(18002099003)(261009223027099003)(56012099006);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?85AZ2pCTDxcHglTxi+lX7t0vKhonyJoud6gS6scivDporNmFUWyztsO+7x7/?= =?us-ascii?Q?yxFNYVQb5yUiaOxxRIkNti8mxnm8UaX/7mKK4UOwqT9rsC1nBboBoZ4Med0K?= =?us-ascii?Q?wgO/B5wW0xt8F8C7+jBQ4lO9SGdwvml0FxCu4VzS8sL6VAtE2X9xmjbcqbZt?= =?us-ascii?Q?dfxSHZ388z0LFqcXu6pmXYWzhLEmeUu3lzm20vTv/F86/YX2z13IMuAASRtP?= =?us-ascii?Q?ImnQm9NdQr1HHcYbOJossQV/tdj/mGqMyb89oEsAzH6FU/YUY5ciKhTUTEwz?= =?us-ascii?Q?8KO97LGR+9UsMv9GmNjsvDRGGfv8yA4p7VbSMWsj2HMhxw9CWTRLgqQps0h+?= =?us-ascii?Q?kR07jTQ6+3tN6Nl18UrOdL/F3zL1Was4D/ymWzN6W/2IrglpBMW2VdflmMgR?= =?us-ascii?Q?NC4KETHgIzgZQfO4JHOFdGwQb0yRN5Lb865ieaGymJq2o9tBDP2E+7unewTM?= =?us-ascii?Q?BYtdncoo33CXY/sI/dKI7ZO5igBc94j6Ootpt1+J2nRJUuwPAszf+RtYc96n?= =?us-ascii?Q?s6N5eQtIy0z/RRQM1gZ3jpXeGuni7y9qfEA+UVfjN63hOBp0yyKP9dnvHO+N?= =?us-ascii?Q?n6ttycmJAdD9UkXnyZdeGTt3qY1QgGz7Mryze+EHKFPumeWdyrK36yhtcczU?= =?us-ascii?Q?MS/IDa5Jp4ofnAQJBVIrb0P5Y1ihYqeauYx7m+Doe2Vf+LmvrIiCgOx2UREt?= =?us-ascii?Q?Ed1es7s7ETGissVs62Uwc5P4Wo22hVLZYxv4ajtVSK77iUvXH42q4PGZoBW8?= =?us-ascii?Q?reHUA8r7dvvV/KVI4gd4Xrh62cWjqeNT3WZXeNERPEs0IP01uvj9PT7m3OZ/?= =?us-ascii?Q?hR8m8a5YZBcIwa/LYiusiQuoRVNPiu2NzZQn/2xmDCVn3T8RxFm8AGMUnpSS?= =?us-ascii?Q?H4WWdQImukHoNlLd2E8Bdy3et67pIDvgg8Rba7s5Qc+onuMQU9aYfHpHb1go?= =?us-ascii?Q?IBc5fVyErrXXx46RIIyWN9s4Mx47wYK2TreHaF1nIyaIoN0N/DTM8/axCwUH?= =?us-ascii?Q?EnN9pT9DYake2NwRHPJ2uogVqGRiDbd21J43lG6TL3LZPcWGBbnI9pbORJPU?= =?us-ascii?Q?lfC+2q829hs/PqAo+04E9WyVVmpOn3miQbubcSga4xW98nXOJPeDt02UPH4m?= =?us-ascii?Q?9rfhzXx1W9F0A/JgmMMqlJBWa0iF6XE1RWjgBsnCHjpb2JGukiqiFzQgUiaT?= =?us-ascii?Q?i7fsDmQIwitez0KQaT+Z4Y6c1oHrw16GYtqLt3uuggrojTgfLptcem/m67by?= =?us-ascii?Q?47qQ+/dSyNideK5yoIJgo86FdgJ0/UYMNYG6iVSJcPA/Ok2lxEGEwfhgYsmR?= =?us-ascii?Q?VfcVXNGCtrAsQm5WbChXNC18tiYyloHRlCwE3eqrkLy0DWHEsu3kjZGwWmUJ?= =?us-ascii?Q?UyfqyiD4xJgjgWbqC0c0SPIanAUwz2MrBo2MD+70y12GdnpEVhc76+xA3/PG?= =?us-ascii?Q?W16pwcTMtOauUT9gbgCziuHtmStDS+9Ynh7yiPoHD4apxzYoOVASDmvVFyFF?= =?us-ascii?Q?nwLJkaDuDpI8/6IzfLP2letGbpMbTPESmuFG34uMd0XqJ2IpaSOoNuhZr9j0?= =?us-ascii?Q?4+CcqUUU3YknFWpvepTW27C4dX7ZtVo51MOKSH5HG80ng7Jx9MrQlFckud5q?= =?us-ascii?Q?/dVuZSPx8FsqOm9fvdpY7jw/fv4npB7Y5yHuyRCQ05QRQectinjFcm7ZL4ht?= =?us-ascii?Q?ESiFnMmFodQF2QMyHJAmEvtaxGs1Oq2/l0yzu0M7vt0bDs1VufBZytHsT1CU?= =?us-ascii?Q?FraRfSvNAw=3D=3D?= X-OriginatorOrg: RyomHerold.dk X-MS-Exchange-CrossTenant-Network-Message-Id: 9d3a9902-62f1-401c-58a0-08df2d15cdb4 X-MS-Exchange-CrossTenant-AuthSource: AS8PR10MB7231.EURPRD10.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Oct 2026 12:46:17.9897 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 26a2499e-4646-4888-a695-c64736a56807 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hYM9y0mLpn9EmRc+jPDq+YWriBdU6Sa+QnWr3Jsoy9LP0go4lzuxsfxH3bGuqFm6e1B49eZFhUsuWTY8NZt4yA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PA4PR10MB5706 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.002 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy GB_MSAUTHSOURCE_EUROPE 0.01 Mail authenticated via Microsoft Europe (EURP) datacenter RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: AUFUSRLPK53TIL3SEIEBSTRANHUDVBBL X-Message-ID-Hash: AUFUSRLPK53TIL3SEIEBSTRANHUDVBBL X-MailFrom: Michael@RyomHerold.dk X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Michael Ryom X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The dynamic scheduler acts on point-in-time samples of the node and guest usage stats. The part of the node load that is not caused by any HA-managed service (non-HA guests, the ZFS ARC, Ceph daemons, the IO of a rebalance migration itself, sampling skew between node and guest stats) is noisy, and acting on a single sample makes the automatic load balancer prone to oscillation: a single dominant HA resource on a small cluster qualifies for a migration (and afterwards, for its reverse) as soon as the node base loads fluctuate around each other by a fraction of the resource's own load. Decompose each node's load into the usage of the HA-managed services actually running on it plus a residual, and smooth only the residual with an exponentially moving average that the manager keeps across scheduling rounds (not persisted for a CRM failover, like the other balancer state). The service usages themselves are deliberately not smoothed: they are fully modeled by the scheduler, so known structural changes (a service was migrated, started or stopped, or its measured load changed) take effect immediately, transient service load spikes remain handled by the hold duration, and the expected log of no existing regression test changes. The residual, on the other hand, cannot be attributed to anything the balancer could act on, and its expected reaction to a rebalance motion is zero, so a lagging average is safe to act on. The expected log of test-crs-dynamic-auto-rebalance9 changes, as the effective base load difference now stays well below the qualification point and vm:100 is not migrated back and forth anymore. Signed-off-by: Michael Ryom --- src/PVE/HA/Manager.pm | 10 ++- src/PVE/HA/Usage/Dynamic.pm | 73 ++++++++++++++++++- .../test-crs-dynamic-auto-rebalance9/README | 4 + .../log.expect | 38 ---------- 4 files changed, 85 insertions(+), 40 deletions(-) diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm index b804602..1721fae 100644 --- a/src/PVE/HA/Manager.pm +++ b/src/PVE/HA/Manager.pm @@ -95,6 +95,10 @@ sub new { # load_balance() is not considered for rebalancing cooldown => {}, }, + # exponentially moving average of the unaccounted node load, kept + # across rounds by PVE::HA::Usage::Dynamic; also not persisted for a + # CRM failover + dynamic_stats_smoothing_state => {}, group_migration_round => 3, # wait a little bit }, $class; @@ -623,7 +627,11 @@ sub recompute_online_node_usage { if ($have_dynamic_scheduling) { $online_node_usage = eval { $service_stats = $haenv->get_dynamic_service_stats(); - my $scheduler = PVE::HA::Usage::Dynamic->new($haenv, $service_stats); + my $scheduler = PVE::HA::Usage::Dynamic->new( + $haenv, + $service_stats, + $self->{dynamic_stats_smoothing_state}, + ); $scheduler->add_node($_) for $online_nodes->@*; return $scheduler; }; diff --git a/src/PVE/HA/Usage/Dynamic.pm b/src/PVE/HA/Usage/Dynamic.pm index 76d0fea..fc88ddf 100644 --- a/src/PVE/HA/Usage/Dynamic.pm +++ b/src/PVE/HA/Usage/Dynamic.pm @@ -8,12 +8,83 @@ use PVE::RS::ResourceScheduling::Dynamic; use base qw(PVE::HA::Usage); +# weight of the newest sample in the exponentially moving average of the +# unaccounted node load; with one sample per CRM scheduling round (~10s), +# 0.25 averages over roughly the last minute +my $usage_smoothing_alpha = 0.25; + +# Smooths the *unaccounted* part of each node's load with an exponentially +# moving average kept in $state across invocations. +# +# The node load is decomposed into the usage of the HA-managed services +# running on it plus a residual, and only the residual is smoothed. The +# residual captures the load that cannot be attributed to any HA-managed +# service (non-HA guests, ZFS ARC, Ceph daemons, the IO of a rebalance +# migration itself, sampling skew between the node and guest stats). Those +# point-in-time samples are noisy, and acting on a single sample makes the +# load balancer prone to oscillation, while their expected reaction to a +# rebalance motion is zero, so a lagging average is safe to act on. +# +# The service usages themselves are deliberately not smoothed: they are +# fully modeled by the scheduler, i.e. known structural changes (a service +# was migrated, started or stopped, or its load changed) take effect +# immediately, and transient service load spikes are already handled by the +# hold duration of the load balancer. +my $smooth_unaccounted_node_load = sub { + my ($state, $node_stats, $service_stats) = @_; + + # usage per node of the HA-managed services actually running on it + my $accounted = {}; + for my $sid (keys %$service_stats) { + my ($node, $running, $usage) = $service_stats->{$sid}->@{qw(node running usage)}; + next if !$running || !defined($node) || !$usage; + next if !defined($usage->{cpu}) || !defined($usage->{mem}); + + $accounted->{$node}->{cpu} += $usage->{cpu}; + $accounted->{$node}->{mem} += $usage->{mem}; + } + + for my $node (sort keys %$node_stats) { + my $stats = $node_stats->{$node}; + next if !defined($stats->{cpu}) || !defined($stats->{mem}); + + # the residual may become negative if the samples disagree, e.g. + # right after a service stopped; only the reconstructed node load is + # clamped, so that the residual average is not biased upwards + my $residual_sample = { + cpu => $stats->{cpu} - ($accounted->{$node}->{cpu} // 0.0), + mem => $stats->{mem} - ($accounted->{$node}->{mem} // 0), + }; + + my $residual = $state->{$node}; + if (!defined($residual)) { + $residual = $state->{$node} = $residual_sample; + } else { + for my $field (qw(cpu mem)) { + $residual->{$field} = $usage_smoothing_alpha * $residual_sample->{$field} + + (1 - $usage_smoothing_alpha) * $residual->{$field}; + } + } + + my $cpu = $residual->{cpu} + ($accounted->{$node}->{cpu} // 0.0); + my $mem = $residual->{mem} + ($accounted->{$node}->{mem} // 0); + $stats->{cpu} = $cpu > 0.0 ? $cpu : 0.0; + $stats->{mem} = $mem > 0 ? int($mem) : 0; + } + + # drop state of meanwhile removed nodes + delete $state->{$_} for grep { !$node_stats->{$_} } keys %$state; +}; + sub new { - my ($class, $haenv, $service_stats) = @_; + my ($class, $haenv, $service_stats, $smoothing_state) = @_; my $node_stats = eval { $haenv->get_dynamic_node_stats() }; die "did not get dynamic node usage information - $@" if $@; + $smooth_unaccounted_node_load->($smoothing_state, $node_stats, $service_stats) + if defined($smoothing_state); + my $scheduler = eval { PVE::RS::ResourceScheduling::Dynamic->new() }; die "unable to initialize dynamic scheduling - $@" if $@; diff --git a/src/test/test-crs-dynamic-auto-rebalance9/README b/src/test/test-crs-dynamic-auto-rebalance9/README index 93b27c4..56bb608 100644 --- a/src/test/test-crs-dynamic-auto-rebalance9/README +++ b/src/test/test-crs-dynamic-auto-rebalance9/README @@ -14,3 +14,7 @@ imbalance improvement of about 10.4% exceeds the default margin of 10%. Acting on the raw samples moves the dominant resource back and forth with every flip, even though the fluctuation is well within the noise of a real node's base load. + +With the exponentially moving average over the unaccounted node load, the +effective base load difference stays well below the point where a motion +qualifies, so no motion may be issued at any point. diff --git a/src/test/test-crs-dynamic-auto-rebalance9/log.expect b/src/test/test-crs-dynamic-auto-rebalance9/log.expect index efd4d2c..73e381a 100644 --- a/src/test/test-crs-dynamic-auto-rebalance9/log.expect +++ b/src/test/test-crs-dynamic-auto-rebalance9/log.expect @@ -19,48 +19,10 @@ info 21 node1/lrm: service status vm:100 started info 22 node2/crm: status change wait_for_quorum => slave info 120 cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0 info 120 cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0 -info 140 node1/crm: auto rebalance - migrate vm:100 to node2 (expected change for imbalance from 52.8% to 47.2%) -info 140 node1/crm: got crm command: migrate vm:100 node2 -info 140 node1/crm: migrate service 'vm:100' to node 'node2' -info 140 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node1, target = node2) -info 141 node1/lrm: service vm:100 - start migrate to node 'node2' -info 141 node1/lrm: service vm:100 - end migrate to node 'node2' -info 143 node2/lrm: got lock 'ha_agent_node2_lock' -info 143 node2/lrm: status change wait_for_agent_lock => active -info 160 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node2) -info 163 node2/lrm: starting service vm:100 -info 163 node2/lrm: service status vm:100 started info 220 cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0 info 220 cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0 -info 220 node1/crm: auto rebalance - migrate vm:100 to node1 (expected change for imbalance from 52.8% to 47.2%) -info 220 node1/crm: got crm command: migrate vm:100 node1 -info 220 node1/crm: migrate service 'vm:100' to node 'node1' -info 220 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node2, target = node1) -info 223 node2/lrm: service vm:100 - start migrate to node 'node1' -info 223 node2/lrm: service vm:100 - end migrate to node 'node1' -info 240 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node1) -info 241 node1/lrm: starting service vm:100 -info 241 node1/lrm: service status vm:100 started info 320 cmdlist: execute node node1 set-dynamic-stats cpu 5.064 mem 0 info 320 cmdlist: execute node node2 set-dynamic-stats cpu 4.536 mem 0 -info 360 node1/crm: auto rebalance - migrate vm:100 to node2 (expected change for imbalance from 52.8% to 47.2%) -info 360 node1/crm: got crm command: migrate vm:100 node2 -info 360 node1/crm: migrate service 'vm:100' to node 'node2' -info 360 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node1, target = node2) -info 361 node1/lrm: service vm:100 - start migrate to node 'node2' -info 361 node1/lrm: service vm:100 - end migrate to node 'node2' -info 380 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node2) -info 383 node2/lrm: starting service vm:100 -info 383 node2/lrm: service status vm:100 started info 420 cmdlist: execute node node1 set-dynamic-stats cpu 4.536 mem 0 info 420 cmdlist: execute node node2 set-dynamic-stats cpu 5.064 mem 0 -info 440 node1/crm: auto rebalance - migrate vm:100 to node1 (expected change for imbalance from 52.8% to 47.2%) -info 440 node1/crm: got crm command: migrate vm:100 node1 -info 440 node1/crm: migrate service 'vm:100' to node 'node1' -info 440 node1/crm: service 'vm:100': state changed from 'started' to 'migrate' (node = node2, target = node1) -info 443 node2/lrm: service vm:100 - start migrate to node 'node1' -info 443 node2/lrm: service vm:100 - end migrate to node 'node1' -info 460 node1/crm: service 'vm:100': state changed from 'migrate' to 'started' (node = node1) -info 461 node1/lrm: starting service vm:100 -info 461 node1/lrm: service status vm:100 started info 1020 hardware: exit simulation - done -- 2.47.3