From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH] lrm: refuse targets dropping the guest identity before stopping guests
Date: Fri, 20 Nov 2026 03:54:19 +0100 [thread overview]
Message-ID: <20261120025517.3031086-1-t.lamprecht@proxmox.com> (raw)
The guest migration APIs reject nodes that would drop guest identities,
but HA resource plugins can stop a container or shut down a VM for
relocation before calling those APIs. A refused requested relocation
therefore causes an unnecessary stop and restart.
For automatic moves, including maintenance, shutdown with the 'migrate'
policy, failback, and affinity rules, the CRM can select the same target
again before restarting the guest. The guest then stays stopped while
relocation is retried. Automatic rebalancing can repeatedly restart a
container instead.
Have the resource plugins check identity support before the LRM calls
migrate, and treat a refusal as a migration error. A running guest then
stays running on its source node.
Guest API requests already fail before reaching HA. Direct ha-manager
requests report the refusal in the LRM log. The CRM still retries
automatic moves to unsupported targets, so maintenance or shutdown
with the 'migrate' policy remains blocked until the selected targets
are upgraded, even if other nodes already support identities.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
I'm a bit torn here about the specific check employed, but the overal
mechanism is simple enough and could be reused for other migration
blockers in HA.
src/PVE/HA/LRM.pm | 7 ++
src/PVE/HA/Resources.pm | 8 ++
src/PVE/HA/Resources/PVECT.pm | 9 ++
src/PVE/HA/Resources/PVEVM.pm | 9 ++
src/PVE/HA/Sim/Resources.pm | 21 ++++
src/test/test-migrate-old-target1/README | 8 ++
src/test/test-migrate-old-target1/cmdlist | 6 +
.../test-migrate-old-target1/hardware_status | 5 +
src/test/test-migrate-old-target1/log.expect | 73 ++++++++++++
.../test-migrate-old-target1/manager_status | 1 +
.../test-migrate-old-target1/service_config | 4 +
src/test/test-migrate-old-target2/README | 10 ++
src/test/test-migrate-old-target2/cmdlist | 5 +
.../test-migrate-old-target2/hardware_status | 5 +
src/test/test-migrate-old-target2/log.expect | 109 ++++++++++++++++++
.../test-migrate-old-target2/manager_status | 1 +
.../test-migrate-old-target2/service_config | 5 +
src/test/test-migrate-old-target3/README | 11 ++
src/test/test-migrate-old-target3/cmdlist | 6 +
.../test-migrate-old-target3/hardware_status | 5 +
src/test/test-migrate-old-target3/log.expect | 86 ++++++++++++++
.../test-migrate-old-target3/manager_status | 1 +
.../test-migrate-old-target3/service_config | 4 +
src/test/test-migrate-old-target4/README | 7 ++
src/test/test-migrate-old-target4/cmdlist | 4 +
.../test-migrate-old-target4/datacenter.cfg | 6 +
.../test-migrate-old-target4/hardware_status | 5 +
src/test/test-migrate-old-target4/log.expect | 52 +++++++++
.../test-migrate-old-target4/manager_status | 1 +
.../test-migrate-old-target4/service_config | 5 +
.../static_service_stats | 5 +
31 files changed, 484 insertions(+)
create mode 100644 src/test/test-migrate-old-target1/README
create mode 100644 src/test/test-migrate-old-target1/cmdlist
create mode 100644 src/test/test-migrate-old-target1/hardware_status
create mode 100644 src/test/test-migrate-old-target1/log.expect
create mode 100644 src/test/test-migrate-old-target1/manager_status
create mode 100644 src/test/test-migrate-old-target1/service_config
create mode 100644 src/test/test-migrate-old-target2/README
create mode 100644 src/test/test-migrate-old-target2/cmdlist
create mode 100644 src/test/test-migrate-old-target2/hardware_status
create mode 100644 src/test/test-migrate-old-target2/log.expect
create mode 100644 src/test/test-migrate-old-target2/manager_status
create mode 100644 src/test/test-migrate-old-target2/service_config
create mode 100644 src/test/test-migrate-old-target3/README
create mode 100644 src/test/test-migrate-old-target3/cmdlist
create mode 100644 src/test/test-migrate-old-target3/hardware_status
create mode 100644 src/test/test-migrate-old-target3/log.expect
create mode 100644 src/test/test-migrate-old-target3/manager_status
create mode 100644 src/test/test-migrate-old-target3/service_config
create mode 100644 src/test/test-migrate-old-target4/README
create mode 100644 src/test/test-migrate-old-target4/cmdlist
create mode 100644 src/test/test-migrate-old-target4/datacenter.cfg
create mode 100644 src/test/test-migrate-old-target4/hardware_status
create mode 100644 src/test/test-migrate-old-target4/log.expect
create mode 100644 src/test/test-migrate-old-target4/manager_status
create mode 100644 src/test/test-migrate-old-target4/service_config
create mode 100644 src/test/test-migrate-old-target4/static_service_stats
diff --git a/src/PVE/HA/LRM.pm b/src/PVE/HA/LRM.pm
index 5a5d77a..92194cd 100644
--- a/src/PVE/HA/LRM.pm
+++ b/src/PVE/HA/LRM.pm
@@ -1083,6 +1083,13 @@ sub exec_resource_agent {
return IGNORED;
}
+ eval { $plugin->assert_migration_target($haenv, $id, $target) };
+ if (my $err = $@) {
+ chomp($err);
+ $haenv->log("err", "service $sid not moved - $err");
+ return ERROR;
+ }
+
my $online = ($cmd eq 'migrate') ? 1 : 0;
my $res = $plugin->migrate($haenv, $id, $target, $online, $params->{'migrate-options'});
diff --git a/src/PVE/HA/Resources.pm b/src/PVE/HA/Resources.pm
index 7a7e7c9..6a119d9 100644
--- a/src/PVE/HA/Resources.pm
+++ b/src/PVE/HA/Resources.pm
@@ -180,6 +180,14 @@ sub migrate_json_properties {
die "implement in subclass";
}
+# Dies if the service $id must not be moved to the node $target. The LRM checks this before calling
+# migrate(), as that already shuts the service down for a relocation.
+sub assert_migration_target {
+ my ($class, $haenv, $id, $target) = @_;
+
+ return;
+}
+
sub config_file {
my ($class, $id, $nodename) = @_;
diff --git a/src/PVE/HA/Resources/PVECT.pm b/src/PVE/HA/Resources/PVECT.pm
index f00a92c..3ec8d92 100644
--- a/src/PVE/HA/Resources/PVECT.pm
+++ b/src/PVE/HA/Resources/PVECT.pm
@@ -9,6 +9,8 @@ use PVE::HA::Tools;
BEGIN {
if (!$ENV{PVE_GENERATING_DOCS}) {
+ require PVE::GuestIdentity;
+ import PVE::GuestIdentity;
require PVE::LXC;
import PVE::LXC;
require PVE::LXC::Config;
@@ -145,6 +147,13 @@ sub migrate {
return !(-f $oldconfig);
}
+sub assert_migration_target {
+ my ($class, $haenv, $id, $target) = @_;
+
+ # the migrate API refuses such a node as well, but only after the container got stopped
+ PVE::GuestIdentity::assert_node_supports_identity($target);
+}
+
sub check_running {
my ($class, $haenv, $vmid) = @_;
diff --git a/src/PVE/HA/Resources/PVEVM.pm b/src/PVE/HA/Resources/PVEVM.pm
index 2ae2b55..25cf28e 100644
--- a/src/PVE/HA/Resources/PVEVM.pm
+++ b/src/PVE/HA/Resources/PVEVM.pm
@@ -9,6 +9,8 @@ use PVE::HA::Tools;
BEGIN {
if (!$ENV{PVE_GENERATING_DOCS}) {
+ require PVE::GuestIdentity;
+ import PVE::GuestIdentity;
require PVE::QemuConfig;
import PVE::QemuConfig;
require PVE::QemuServer;
@@ -151,6 +153,13 @@ sub migrate {
return !(-f $oldconfig);
}
+sub assert_migration_target {
+ my ($class, $haenv, $id, $target) = @_;
+
+ # the migrate API refuses such a node as well, but only after a relocation stopped the VM
+ PVE::GuestIdentity::assert_node_supports_identity($target);
+}
+
sub check_running {
my ($class, $haenv, $vmid) = @_;
diff --git a/src/PVE/HA/Sim/Resources.pm b/src/PVE/HA/Sim/Resources.pm
index 16963f0..2008593 100644
--- a/src/PVE/HA/Sim/Resources.pm
+++ b/src/PVE/HA/Sim/Resources.pm
@@ -3,12 +3,19 @@ package PVE::HA::Sim::Resources;
use strict;
use warnings;
+use PVE::HA::Tools;
+
use base qw(PVE::HA::Resources);
# provides some base methods for virtual resources (used in simulation/testing).
# reduces code duplication and it's targeted for the main PVE service types, namely
# virtual machines (VM) and container (CT)
+# Plays the role of PVE::GuestIdentity::MIN_PVE_MANAGER_VERSION for the simulated versions of the
+# tests, with a value of its own: nodes with at least this version keep the guest identity and
+# refuse to move guests to nodes with an older version, which would drop it.
+my $GUEST_IDENTITY_MIN_VERSION = '9.2.29';
+
sub verify_name {
my ($class, $id) = @_;
@@ -139,6 +146,20 @@ sub migrate_json_properties {
return $props;
}
+# models the target check of the migration API of guests
+sub assert_migration_target {
+ my ($class, $haenv, $id, $target) = @_;
+
+ my $version = $haenv->get_node_version($haenv->nodename());
+ return if !PVE::HA::Tools::has_min_version($version, $GUEST_IDENTITY_MIN_VERSION);
+
+ my $target_version = $haenv->get_node_version($target);
+ if (!PVE::HA::Tools::has_min_version($target_version, $GUEST_IDENTITY_MIN_VERSION)) {
+ die "node '$target' runs Proxmox VE $target_version, keeping the guest identity requires"
+ . " $GUEST_IDENTITY_MIN_VERSION or newer - upgrade the node first\n";
+ }
+}
+
sub remove_locks {
my ($self, $haenv, $id, $locks, $service_node) = @_;
diff --git a/src/test/test-migrate-old-target1/README b/src/test/test-migrate-old-target1/README
new file mode 100644
index 0000000..d852f3e
--- /dev/null
+++ b/src/test/test-migrate-old-target1/README
@@ -0,0 +1,8 @@
+Test requested migrations and relocations to a node whose Proxmox VE version
+would drop the guest identity, while the source node keeps it.
+
+The expected outcome is:
+- The moves of vm:101 and ct:102 to node2 are refused before the services are
+ stopped, both keep running on node1
+- The moves of vm:101 and ct:102 to node3, which keeps the guest identity, are
+ done as usual
diff --git a/src/test/test-migrate-old-target1/cmdlist b/src/test/test-migrate-old-target1/cmdlist
new file mode 100644
index 0000000..043688c
--- /dev/null
+++ b/src/test/test-migrate-old-target1/cmdlist
@@ -0,0 +1,6 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "service vm:101 migrate node2", "service ct:102 relocate node2" ],
+ [ "service vm:101 relocate node2" ],
+ [ "service vm:101 migrate node3", "service ct:102 relocate node3" ]
+]
diff --git a/src/test/test-migrate-old-target1/hardware_status b/src/test/test-migrate-old-target1/hardware_status
new file mode 100644
index 0000000..a22a500
--- /dev/null
+++ b/src/test/test-migrate-old-target1/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "pve-manager-version": "9.2.29" },
+ "node2": { "power": "off", "network": "off", "pve-manager-version": "9.2.28" },
+ "node3": { "power": "off", "network": "off", "pve-manager-version": "9.2.29" }
+}
diff --git a/src/test/test-migrate-old-target1/log.expect b/src/test/test-migrate-old-target1/log.expect
new file mode 100644
index 0000000..6d8594b
--- /dev/null
+++ b/src/test/test-migrate-old-target1/log.expect
@@ -0,0 +1,73 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'ct:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: service 'ct:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service ct:102
+info 21 node1/lrm: service status ct:102 started
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 120 cmdlist: execute service vm:101 migrate node2
+info 120 cmdlist: execute service ct:102 relocate node2
+info 120 node1/crm: got crm command: migrate vm:101 node2
+info 120 node1/crm: got crm command: relocate ct:102 node2
+info 120 node1/crm: relocate service 'ct:102' to node 'node2'
+info 120 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 120 node1/crm: migrate service 'vm:101' to node 'node2'
+info 120 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node2)
+err 121 node1/lrm: service ct:102 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err 121 node1/lrm: service vm:101 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info 123 node2/lrm: got lock 'ha_agent_node2_lock'
+info 123 node2/lrm: status change wait_for_agent_lock => active
+err 140 node1/crm: service 'ct:102' - migration failed (exit code 1)
+info 140 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node1)
+err 140 node1/crm: service 'vm:101' - migration failed (exit code 1)
+info 140 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node1)
+info 220 cmdlist: execute service vm:101 relocate node2
+info 220 node1/crm: got crm command: relocate vm:101 node2
+info 220 node1/crm: relocate service 'vm:101' to node 'node2'
+info 220 node1/crm: service 'vm:101': state changed from 'started' to 'relocate' (node = node1, target = node2)
+err 221 node1/lrm: service vm:101 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err 240 node1/crm: service 'vm:101' - migration failed (exit code 1)
+info 240 node1/crm: service 'vm:101': state changed from 'relocate' to 'started' (node = node1)
+info 320 cmdlist: execute service vm:101 migrate node3
+info 320 cmdlist: execute service ct:102 relocate node3
+info 320 node1/crm: got crm command: migrate vm:101 node3
+info 320 node1/crm: got crm command: relocate ct:102 node3
+info 320 node1/crm: relocate service 'ct:102' to node 'node3'
+info 320 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node1, target = node3)
+info 320 node1/crm: migrate service 'vm:101' to node 'node3'
+info 320 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 321 node1/lrm: service ct:102 - start relocate to node 'node3'
+info 321 node1/lrm: stopping service ct:102 (relocate)
+info 321 node1/lrm: service status ct:102 stopped
+info 321 node1/lrm: service ct:102 - end relocate to node 'node3'
+info 321 node1/lrm: service vm:101 - start migrate to node 'node3'
+info 321 node1/lrm: service vm:101 - end migrate to node 'node3'
+info 325 node3/lrm: got lock 'ha_agent_node3_lock'
+info 325 node3/lrm: status change wait_for_agent_lock => active
+info 340 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node3)
+info 340 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3)
+info 345 node3/lrm: starting service ct:102
+info 345 node3/lrm: service status ct:102 started
+info 345 node3/lrm: starting service vm:101
+info 345 node3/lrm: service status vm:101 started
+info 920 hardware: exit simulation - done
diff --git a/src/test/test-migrate-old-target1/manager_status b/src/test/test-migrate-old-target1/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-migrate-old-target1/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-migrate-old-target1/service_config b/src/test/test-migrate-old-target1/service_config
new file mode 100644
index 0000000..61bafec
--- /dev/null
+++ b/src/test/test-migrate-old-target1/service_config
@@ -0,0 +1,4 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "ct:102": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-migrate-old-target2/README b/src/test/test-migrate-old-target2/README
new file mode 100644
index 0000000..c1bdd20
--- /dev/null
+++ b/src/test/test-migrate-old-target2/README
@@ -0,0 +1,10 @@
+Test enabling the maintenance mode on a node that keeps the guest identity,
+where one of the other nodes runs a Proxmox VE version that would drop it.
+
+The expected outcome is:
+- vm:101 is migrated to node3
+- The migration of vm:103 and the relocation of ct:102 to node2 are refused
+ before the services are stopped, both keep running on node1
+- As for any move that keeps failing, the CRM retries vm:103 and ct:102 with
+ node2 until the maintenance mode is disabled again
+- vm:101 is migrated back to node1 after the maintenance mode is disabled
diff --git a/src/test/test-migrate-old-target2/cmdlist b/src/test/test-migrate-old-target2/cmdlist
new file mode 100644
index 0000000..5aa42a6
--- /dev/null
+++ b/src/test/test-migrate-old-target2/cmdlist
@@ -0,0 +1,5 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "crm node1 enable-node-maintenance" ],
+ [ "crm node1 disable-node-maintenance" ]
+]
diff --git a/src/test/test-migrate-old-target2/hardware_status b/src/test/test-migrate-old-target2/hardware_status
new file mode 100644
index 0000000..a22a500
--- /dev/null
+++ b/src/test/test-migrate-old-target2/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "pve-manager-version": "9.2.29" },
+ "node2": { "power": "off", "network": "off", "pve-manager-version": "9.2.28" },
+ "node3": { "power": "off", "network": "off", "pve-manager-version": "9.2.29" }
+}
diff --git a/src/test/test-migrate-old-target2/log.expect b/src/test/test-migrate-old-target2/log.expect
new file mode 100644
index 0000000..73d29f6
--- /dev/null
+++ b/src/test/test-migrate-old-target2/log.expect
@@ -0,0 +1,109 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'ct:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node1'
+info 20 node1/crm: service 'ct:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:103': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service ct:102
+info 21 node1/lrm: service status ct:102 started
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 21 node1/lrm: starting service vm:103
+info 21 node1/lrm: service status vm:103 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 120 cmdlist: execute crm node1 enable-node-maintenance
+info 121 node1/lrm: status change active => maintenance
+info 140 node1/crm: node 'node1': state changed from 'online' => 'maintenance'
+info 140 node1/crm: relocate service 'ct:102' to node 'node2'
+info 140 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 140 node1/crm: migrate service 'vm:101' to node 'node3' (running)
+info 140 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node3)
+info 140 node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info 140 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node1, target = node2)
+err 141 node1/lrm: service ct:102 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info 141 node1/lrm: service vm:101 - start migrate to node 'node3'
+info 141 node1/lrm: service vm:101 - end migrate to node 'node3'
+err 141 node1/lrm: service vm:103 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info 143 node2/lrm: got lock 'ha_agent_node2_lock'
+info 143 node2/lrm: status change wait_for_agent_lock => active
+info 145 node3/lrm: got lock 'ha_agent_node3_lock'
+info 145 node3/lrm: status change wait_for_agent_lock => active
+err 160 node1/crm: service 'ct:102' - migration failed (exit code 1)
+info 160 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node1)
+info 160 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3)
+err 160 node1/crm: service 'vm:103' - migration failed (exit code 1)
+info 160 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node1)
+info 160 node1/crm: relocate service 'ct:102' to node 'node2'
+info 160 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 160 node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info 160 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node1, target = node2)
+err 161 node1/lrm: service ct:102 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err 161 node1/lrm: service vm:103 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info 165 node3/lrm: starting service vm:101
+info 165 node3/lrm: service status vm:101 started
+err 180 node1/crm: service 'ct:102' - migration failed (exit code 1)
+info 180 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node1)
+err 180 node1/crm: service 'vm:103' - migration failed (exit code 1)
+info 180 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node1)
+info 180 node1/crm: relocate service 'ct:102' to node 'node2'
+info 180 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 180 node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info 180 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node1, target = node2)
+err 181 node1/lrm: service ct:102 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err 181 node1/lrm: service vm:103 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err 200 node1/crm: service 'ct:102' - migration failed (exit code 1)
+info 200 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node1)
+err 200 node1/crm: service 'vm:103' - migration failed (exit code 1)
+info 200 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node1)
+info 200 node1/crm: relocate service 'ct:102' to node 'node2'
+info 200 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 200 node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info 200 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node1, target = node2)
+err 201 node1/lrm: service ct:102 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err 201 node1/lrm: service vm:103 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info 220 cmdlist: execute crm node1 disable-node-maintenance
+err 220 node1/crm: service 'ct:102' - migration failed (exit code 1)
+info 220 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node1)
+err 220 node1/crm: service 'vm:103' - migration failed (exit code 1)
+info 220 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node1)
+info 220 node1/crm: relocate service 'ct:102' to node 'node2'
+info 220 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 220 node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info 220 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 221 node1/lrm: status change maintenance => active
+err 221 node1/lrm: service ct:102 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err 221 node1/lrm: service vm:103 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info 240 node1/crm: node 'node1': state changed from 'maintenance' => 'online'
+err 240 node1/crm: service 'ct:102' - migration failed (exit code 1)
+info 240 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node1)
+info 240 node1/crm: moving service 'vm:101' back to 'node1', node came back from maintenance.
+info 240 node1/crm: migrate service 'vm:101' to node 'node1' (running)
+info 240 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node3, target = node1)
+err 240 node1/crm: service 'vm:103' - migration failed (exit code 1)
+info 240 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node1)
+info 240 node1/crm: service 'ct:102': clearing stale maintenance node 'node1' setting (is current node)
+info 240 node1/crm: service 'vm:103': clearing stale maintenance node 'node1' setting (is current node)
+info 245 node3/lrm: service vm:101 - start migrate to node 'node1'
+info 245 node3/lrm: service vm:101 - end migrate to node 'node1'
+info 260 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node1)
+info 261 node1/lrm: starting service vm:101
+info 261 node1/lrm: service status vm:101 started
+info 820 hardware: exit simulation - done
diff --git a/src/test/test-migrate-old-target2/manager_status b/src/test/test-migrate-old-target2/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-migrate-old-target2/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-migrate-old-target2/service_config b/src/test/test-migrate-old-target2/service_config
new file mode 100644
index 0000000..671ad7c
--- /dev/null
+++ b/src/test/test-migrate-old-target2/service_config
@@ -0,0 +1,5 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "ct:102": { "node": "node1", "state": "started" },
+ "vm:103": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-migrate-old-target3/README b/src/test/test-migrate-old-target3/README
new file mode 100644
index 0000000..512bec2
--- /dev/null
+++ b/src/test/test-migrate-old-target3/README
@@ -0,0 +1,11 @@
+Test requested moves between nodes whose Proxmox VE versions differ in whether
+they keep the guest identity. Only a source node that keeps the identity
+refuses a target that would drop it.
+
+The expected outcome is:
+- The moves of vm:101 and ct:102 from node1 to node2 are done, as node1 does
+ not keep the guest identity either
+- The moves of vm:101 and ct:102 from node2 to node3 are done, as node3 keeps
+ the guest identity
+- The moves of vm:101 and ct:102 from node3 back to node1 are refused before
+ the services are stopped, both keep running on node3
diff --git a/src/test/test-migrate-old-target3/cmdlist b/src/test/test-migrate-old-target3/cmdlist
new file mode 100644
index 0000000..be88920
--- /dev/null
+++ b/src/test/test-migrate-old-target3/cmdlist
@@ -0,0 +1,6 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "service vm:101 migrate node2", "service ct:102 relocate node2" ],
+ [ "service vm:101 migrate node3", "service ct:102 relocate node3" ],
+ [ "service vm:101 migrate node1", "service ct:102 relocate node1" ]
+]
diff --git a/src/test/test-migrate-old-target3/hardware_status b/src/test/test-migrate-old-target3/hardware_status
new file mode 100644
index 0000000..b5d1ded
--- /dev/null
+++ b/src/test/test-migrate-old-target3/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "pve-manager-version": "9.2.28" },
+ "node2": { "power": "off", "network": "off", "pve-manager-version": "9.2.28" },
+ "node3": { "power": "off", "network": "off", "pve-manager-version": "9.2.29" }
+}
diff --git a/src/test/test-migrate-old-target3/log.expect b/src/test/test-migrate-old-target3/log.expect
new file mode 100644
index 0000000..8cf50a7
--- /dev/null
+++ b/src/test/test-migrate-old-target3/log.expect
@@ -0,0 +1,86 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'ct:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: service 'ct:102': state changed from 'request_start' to 'started' (node = node1)
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'started' (node = node1)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service ct:102
+info 21 node1/lrm: service status ct:102 started
+info 21 node1/lrm: starting service vm:101
+info 21 node1/lrm: service status vm:101 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 120 cmdlist: execute service vm:101 migrate node2
+info 120 cmdlist: execute service ct:102 relocate node2
+info 120 node1/crm: got crm command: migrate vm:101 node2
+info 120 node1/crm: got crm command: relocate ct:102 node2
+info 120 node1/crm: relocate service 'ct:102' to node 'node2'
+info 120 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 120 node1/crm: migrate service 'vm:101' to node 'node2'
+info 120 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 121 node1/lrm: service ct:102 - start relocate to node 'node2'
+info 121 node1/lrm: stopping service ct:102 (relocate)
+info 121 node1/lrm: service status ct:102 stopped
+info 121 node1/lrm: service ct:102 - end relocate to node 'node2'
+info 121 node1/lrm: service vm:101 - start migrate to node 'node2'
+info 121 node1/lrm: service vm:101 - end migrate to node 'node2'
+info 123 node2/lrm: got lock 'ha_agent_node2_lock'
+info 123 node2/lrm: status change wait_for_agent_lock => active
+info 140 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node2)
+info 140 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node2)
+info 143 node2/lrm: starting service ct:102
+info 143 node2/lrm: service status ct:102 started
+info 143 node2/lrm: starting service vm:101
+info 143 node2/lrm: service status vm:101 started
+info 220 cmdlist: execute service vm:101 migrate node3
+info 220 cmdlist: execute service ct:102 relocate node3
+info 220 node1/crm: got crm command: migrate vm:101 node3
+info 220 node1/crm: got crm command: relocate ct:102 node3
+info 220 node1/crm: relocate service 'ct:102' to node 'node3'
+info 220 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node2, target = node3)
+info 220 node1/crm: migrate service 'vm:101' to node 'node3'
+info 220 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node2, target = node3)
+info 223 node2/lrm: service ct:102 - start relocate to node 'node3'
+info 223 node2/lrm: stopping service ct:102 (relocate)
+info 223 node2/lrm: service status ct:102 stopped
+info 223 node2/lrm: service ct:102 - end relocate to node 'node3'
+info 223 node2/lrm: service vm:101 - start migrate to node 'node3'
+info 223 node2/lrm: service vm:101 - end migrate to node 'node3'
+info 225 node3/lrm: got lock 'ha_agent_node3_lock'
+info 225 node3/lrm: status change wait_for_agent_lock => active
+info 240 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node3)
+info 240 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3)
+info 245 node3/lrm: starting service ct:102
+info 245 node3/lrm: service status ct:102 started
+info 245 node3/lrm: starting service vm:101
+info 245 node3/lrm: service status vm:101 started
+info 320 cmdlist: execute service vm:101 migrate node1
+info 320 cmdlist: execute service ct:102 relocate node1
+info 320 node1/crm: got crm command: migrate vm:101 node1
+info 320 node1/crm: got crm command: relocate ct:102 node1
+info 320 node1/crm: relocate service 'ct:102' to node 'node1'
+info 320 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node3, target = node1)
+info 320 node1/crm: migrate service 'vm:101' to node 'node1'
+info 320 node1/crm: service 'vm:101': state changed from 'started' to 'migrate' (node = node3, target = node1)
+err 325 node3/lrm: service ct:102 not moved - node 'node1' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err 325 node3/lrm: service vm:101 not moved - node 'node1' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err 340 node1/crm: service 'ct:102' - migration failed (exit code 1)
+info 340 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node3)
+err 340 node1/crm: service 'vm:101' - migration failed (exit code 1)
+info 340 node1/crm: service 'vm:101': state changed from 'migrate' to 'started' (node = node3)
+info 920 hardware: exit simulation - done
diff --git a/src/test/test-migrate-old-target3/manager_status b/src/test/test-migrate-old-target3/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-migrate-old-target3/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-migrate-old-target3/service_config b/src/test/test-migrate-old-target3/service_config
new file mode 100644
index 0000000..61bafec
--- /dev/null
+++ b/src/test/test-migrate-old-target3/service_config
@@ -0,0 +1,4 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "ct:102": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-migrate-old-target4/README b/src/test/test-migrate-old-target4/README
new file mode 100644
index 0000000..6f38bda
--- /dev/null
+++ b/src/test/test-migrate-old-target4/README
@@ -0,0 +1,7 @@
+Test the rebalancing on start of a service, where the selected node runs a
+Proxmox VE version that would drop the guest identity, while the current node
+keeps it.
+
+The expected outcome is:
+- The move of ct:103 to node2 for its start is refused
+- ct:103 is started on its current node node1 instead
diff --git a/src/test/test-migrate-old-target4/cmdlist b/src/test/test-migrate-old-target4/cmdlist
new file mode 100644
index 0000000..1172dcc
--- /dev/null
+++ b/src/test/test-migrate-old-target4/cmdlist
@@ -0,0 +1,4 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on" ],
+ [ "service ct:103 started" ]
+]
diff --git a/src/test/test-migrate-old-target4/datacenter.cfg b/src/test/test-migrate-old-target4/datacenter.cfg
new file mode 100644
index 0000000..f2671a5
--- /dev/null
+++ b/src/test/test-migrate-old-target4/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+ "crs": {
+ "ha": "static",
+ "ha-rebalance-on-start": 1
+ }
+}
diff --git a/src/test/test-migrate-old-target4/hardware_status b/src/test/test-migrate-old-target4/hardware_status
new file mode 100644
index 0000000..eaede03
--- /dev/null
+++ b/src/test/test-migrate-old-target4/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off", "pve-manager-version": "9.2.29", "maxcpu": 24, "maxmem": 34359738368 },
+ "node2": { "power": "off", "network": "off", "pve-manager-version": "9.2.28", "maxcpu": 24, "maxmem": 34359738368 },
+ "node3": { "power": "off", "network": "off", "pve-manager-version": "9.2.28", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-migrate-old-target4/log.expect b/src/test/test-migrate-old-target4/log.expect
new file mode 100644
index 0000000..6b77377
--- /dev/null
+++ b/src/test/test-migrate-old-target4/log.expect
@@ -0,0 +1,52 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: using scheduler mode 'static'
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'ct:103' on node 'node1'
+info 20 node1/crm: adding new service 'vm:101' on node 'node1'
+info 20 node1/crm: adding new service 'vm:102' on node 'node1'
+info 20 node1/crm: service vm:101: re-balance selected new node node2 for startup
+info 20 node1/crm: service 'vm:101': state changed from 'request_start' to 'request_start_balance' (node = node1, target = node2)
+info 20 node1/crm: service vm:102: re-balance selected new node node3 for startup
+info 20 node1/crm: service 'vm:102': state changed from 'request_start' to 'request_start_balance' (node = node1, target = node3)
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+err 21 node1/lrm: service vm:101 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err 21 node1/lrm: service vm:102 not moved - node 'node3' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info 22 node2/crm: status change wait_for_quorum => slave
+info 23 node2/lrm: got lock 'ha_agent_node2_lock'
+info 23 node2/lrm: status change wait_for_agent_lock => active
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 40 node1/crm: service 'ct:103': state changed from 'request_stop' to 'stopped'
+err 40 node1/crm: service 'vm:101' - migration failed (exit code 1)
+info 40 node1/crm: service 'vm:101': state changed from 'request_start_balance' to 'started' (node = node1)
+err 40 node1/crm: service 'vm:102' - migration failed (exit code 1)
+info 40 node1/crm: service 'vm:102': state changed from 'request_start_balance' to 'started' (node = node1)
+info 41 node1/lrm: starting service vm:101
+info 41 node1/lrm: service status vm:101 started
+info 41 node1/lrm: starting service vm:102
+info 41 node1/lrm: service status vm:102 started
+info 120 cmdlist: execute service ct:103 started
+info 120 node1/crm: service 'ct:103': state changed from 'stopped' to 'request_start' (node = node1)
+info 120 node1/crm: service ct:103: re-balance selected new node node2 for startup
+info 120 node1/crm: service 'ct:103': state changed from 'request_start' to 'request_start_balance' (node = node1, target = node2)
+err 121 node1/lrm: service ct:103 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err 140 node1/crm: service 'ct:103' - migration failed (exit code 1)
+info 140 node1/crm: service 'ct:103': state changed from 'request_start_balance' to 'started' (node = node1)
+info 141 node1/lrm: starting service ct:103
+info 141 node1/lrm: service status ct:103 started
+info 720 hardware: exit simulation - done
diff --git a/src/test/test-migrate-old-target4/manager_status b/src/test/test-migrate-old-target4/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-migrate-old-target4/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-migrate-old-target4/service_config b/src/test/test-migrate-old-target4/service_config
new file mode 100644
index 0000000..98eb368
--- /dev/null
+++ b/src/test/test-migrate-old-target4/service_config
@@ -0,0 +1,5 @@
+{
+ "vm:101": { "node": "node1", "state": "started" },
+ "vm:102": { "node": "node1", "state": "started" },
+ "ct:103": { "node": "node1", "state": "stopped" }
+}
diff --git a/src/test/test-migrate-old-target4/static_service_stats b/src/test/test-migrate-old-target4/static_service_stats
new file mode 100644
index 0000000..b3be93e
--- /dev/null
+++ b/src/test/test-migrate-old-target4/static_service_stats
@@ -0,0 +1,5 @@
+{
+ "vm:101": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "vm:102": { "maxcpu": 2.0, "maxmem": 8589934592 },
+ "ct:103": { "maxcpu": 2.0, "maxmem": 8589934592 }
+}
--
2.47.3
reply other threads:[~2026-11-20 2:55 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261120025517.3031086-1-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox