public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH] lrm: refuse targets dropping the guest identity before stopping guests
Date: Fri, 20 Nov 2026 03:54:19 +0100	[thread overview]
Message-ID: <20261120025517.3031086-1-t.lamprecht@proxmox.com> (raw)

The guest migration APIs reject nodes that would drop guest identities,
but HA resource plugins can stop a container or shut down a VM for
relocation before calling those APIs. A refused requested relocation
therefore causes an unnecessary stop and restart.

For automatic moves, including maintenance, shutdown with the 'migrate'
policy, failback, and affinity rules, the CRM can select the same target
again before restarting the guest. The guest then stays stopped while
relocation is retried. Automatic rebalancing can repeatedly restart a
container instead.

Have the resource plugins check identity support before the LRM calls
migrate, and treat a refusal as a migration error. A running guest then
stays running on its source node.

Guest API requests already fail before reaching HA. Direct ha-manager
requests report the refusal in the LRM log. The CRM still retries
automatic moves to unsupported targets, so maintenance or shutdown
with the 'migrate' policy remains blocked until the selected targets
are upgraded, even if other nodes already support identities.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

I'm a bit torn here about the specific check employed, but the overal
mechanism is simple enough and could be reused for other migration
blockers in HA.

 src/PVE/HA/LRM.pm                             |   7 ++
 src/PVE/HA/Resources.pm                       |   8 ++
 src/PVE/HA/Resources/PVECT.pm                 |   9 ++
 src/PVE/HA/Resources/PVEVM.pm                 |   9 ++
 src/PVE/HA/Sim/Resources.pm                   |  21 ++++
 src/test/test-migrate-old-target1/README      |   8 ++
 src/test/test-migrate-old-target1/cmdlist     |   6 +
 .../test-migrate-old-target1/hardware_status  |   5 +
 src/test/test-migrate-old-target1/log.expect  |  73 ++++++++++++
 .../test-migrate-old-target1/manager_status   |   1 +
 .../test-migrate-old-target1/service_config   |   4 +
 src/test/test-migrate-old-target2/README      |  10 ++
 src/test/test-migrate-old-target2/cmdlist     |   5 +
 .../test-migrate-old-target2/hardware_status  |   5 +
 src/test/test-migrate-old-target2/log.expect  | 109 ++++++++++++++++++
 .../test-migrate-old-target2/manager_status   |   1 +
 .../test-migrate-old-target2/service_config   |   5 +
 src/test/test-migrate-old-target3/README      |  11 ++
 src/test/test-migrate-old-target3/cmdlist     |   6 +
 .../test-migrate-old-target3/hardware_status  |   5 +
 src/test/test-migrate-old-target3/log.expect  |  86 ++++++++++++++
 .../test-migrate-old-target3/manager_status   |   1 +
 .../test-migrate-old-target3/service_config   |   4 +
 src/test/test-migrate-old-target4/README      |   7 ++
 src/test/test-migrate-old-target4/cmdlist     |   4 +
 .../test-migrate-old-target4/datacenter.cfg   |   6 +
 .../test-migrate-old-target4/hardware_status  |   5 +
 src/test/test-migrate-old-target4/log.expect  |  52 +++++++++
 .../test-migrate-old-target4/manager_status   |   1 +
 .../test-migrate-old-target4/service_config   |   5 +
 .../static_service_stats                      |   5 +
 31 files changed, 484 insertions(+)
 create mode 100644 src/test/test-migrate-old-target1/README
 create mode 100644 src/test/test-migrate-old-target1/cmdlist
 create mode 100644 src/test/test-migrate-old-target1/hardware_status
 create mode 100644 src/test/test-migrate-old-target1/log.expect
 create mode 100644 src/test/test-migrate-old-target1/manager_status
 create mode 100644 src/test/test-migrate-old-target1/service_config
 create mode 100644 src/test/test-migrate-old-target2/README
 create mode 100644 src/test/test-migrate-old-target2/cmdlist
 create mode 100644 src/test/test-migrate-old-target2/hardware_status
 create mode 100644 src/test/test-migrate-old-target2/log.expect
 create mode 100644 src/test/test-migrate-old-target2/manager_status
 create mode 100644 src/test/test-migrate-old-target2/service_config
 create mode 100644 src/test/test-migrate-old-target3/README
 create mode 100644 src/test/test-migrate-old-target3/cmdlist
 create mode 100644 src/test/test-migrate-old-target3/hardware_status
 create mode 100644 src/test/test-migrate-old-target3/log.expect
 create mode 100644 src/test/test-migrate-old-target3/manager_status
 create mode 100644 src/test/test-migrate-old-target3/service_config
 create mode 100644 src/test/test-migrate-old-target4/README
 create mode 100644 src/test/test-migrate-old-target4/cmdlist
 create mode 100644 src/test/test-migrate-old-target4/datacenter.cfg
 create mode 100644 src/test/test-migrate-old-target4/hardware_status
 create mode 100644 src/test/test-migrate-old-target4/log.expect
 create mode 100644 src/test/test-migrate-old-target4/manager_status
 create mode 100644 src/test/test-migrate-old-target4/service_config
 create mode 100644 src/test/test-migrate-old-target4/static_service_stats

diff --git a/src/PVE/HA/LRM.pm b/src/PVE/HA/LRM.pm
index 5a5d77a..92194cd 100644
--- a/src/PVE/HA/LRM.pm
+++ b/src/PVE/HA/LRM.pm
@@ -1083,6 +1083,13 @@ sub exec_resource_agent {
             return IGNORED;
         }
 
+        eval { $plugin->assert_migration_target($haenv, $id, $target) };
+        if (my $err = $@) {
+            chomp($err);
+            $haenv->log("err", "service $sid not moved - $err");
+            return ERROR;
+        }
+
         my $online = ($cmd eq 'migrate') ? 1 : 0;
 
         my $res = $plugin->migrate($haenv, $id, $target, $online, $params->{'migrate-options'});
diff --git a/src/PVE/HA/Resources.pm b/src/PVE/HA/Resources.pm
index 7a7e7c9..6a119d9 100644
--- a/src/PVE/HA/Resources.pm
+++ b/src/PVE/HA/Resources.pm
@@ -180,6 +180,14 @@ sub migrate_json_properties {
     die "implement in subclass";
 }
 
+# Dies if the service $id must not be moved to the node $target. The LRM checks this before calling
+# migrate(), as that already shuts the service down for a relocation.
+sub assert_migration_target {
+    my ($class, $haenv, $id, $target) = @_;
+
+    return;
+}
+
 sub config_file {
     my ($class, $id, $nodename) = @_;
 
diff --git a/src/PVE/HA/Resources/PVECT.pm b/src/PVE/HA/Resources/PVECT.pm
index f00a92c..3ec8d92 100644
--- a/src/PVE/HA/Resources/PVECT.pm
+++ b/src/PVE/HA/Resources/PVECT.pm
@@ -9,6 +9,8 @@ use PVE::HA::Tools;
 
 BEGIN {
     if (!$ENV{PVE_GENERATING_DOCS}) {
+        require PVE::GuestIdentity;
+        import PVE::GuestIdentity;
         require PVE::LXC;
         import PVE::LXC;
         require PVE::LXC::Config;
@@ -145,6 +147,13 @@ sub migrate {
     return !(-f $oldconfig);
 }
 
+sub assert_migration_target {
+    my ($class, $haenv, $id, $target) = @_;
+
+    # the migrate API refuses such a node as well, but only after the container got stopped
+    PVE::GuestIdentity::assert_node_supports_identity($target);
+}
+
 sub check_running {
     my ($class, $haenv, $vmid) = @_;
 
diff --git a/src/PVE/HA/Resources/PVEVM.pm b/src/PVE/HA/Resources/PVEVM.pm
index 2ae2b55..25cf28e 100644
--- a/src/PVE/HA/Resources/PVEVM.pm
+++ b/src/PVE/HA/Resources/PVEVM.pm
@@ -9,6 +9,8 @@ use PVE::HA::Tools;
 
 BEGIN {
     if (!$ENV{PVE_GENERATING_DOCS}) {
+        require PVE::GuestIdentity;
+        import PVE::GuestIdentity;
         require PVE::QemuConfig;
         import PVE::QemuConfig;
         require PVE::QemuServer;
@@ -151,6 +153,13 @@ sub migrate {
     return !(-f $oldconfig);
 }
 
+sub assert_migration_target {
+    my ($class, $haenv, $id, $target) = @_;
+
+    # the migrate API refuses such a node as well, but only after a relocation stopped the VM
+    PVE::GuestIdentity::assert_node_supports_identity($target);
+}
+
 sub check_running {
     my ($class, $haenv, $vmid) = @_;
 
diff --git a/src/PVE/HA/Sim/Resources.pm b/src/PVE/HA/Sim/Resources.pm
index 16963f0..2008593 100644
--- a/src/PVE/HA/Sim/Resources.pm
+++ b/src/PVE/HA/Sim/Resources.pm
@@ -3,12 +3,19 @@ package PVE::HA::Sim::Resources;
 use strict;
 use warnings;
 
+use PVE::HA::Tools;
+
 use base qw(PVE::HA::Resources);
 
 # provides some base methods for virtual resources (used in simulation/testing).
 # reduces code duplication and it's targeted for the main PVE service types, namely
 # virtual machines (VM) and container (CT)
 
+# Plays the role of PVE::GuestIdentity::MIN_PVE_MANAGER_VERSION for the simulated versions of the
+# tests, with a value of its own: nodes with at least this version keep the guest identity and
+# refuse to move guests to nodes with an older version, which would drop it.
+my $GUEST_IDENTITY_MIN_VERSION = '9.2.29';
+
 sub verify_name {
     my ($class, $id) = @_;
 
@@ -139,6 +146,20 @@ sub migrate_json_properties {
     return $props;
 }
 
+# models the target check of the migration API of guests
+sub assert_migration_target {
+    my ($class, $haenv, $id, $target) = @_;
+
+    my $version = $haenv->get_node_version($haenv->nodename());
+    return if !PVE::HA::Tools::has_min_version($version, $GUEST_IDENTITY_MIN_VERSION);
+
+    my $target_version = $haenv->get_node_version($target);
+    if (!PVE::HA::Tools::has_min_version($target_version, $GUEST_IDENTITY_MIN_VERSION)) {
+        die "node '$target' runs Proxmox VE $target_version, keeping the guest identity requires"
+            . " $GUEST_IDENTITY_MIN_VERSION or newer - upgrade the node first\n";
+    }
+}
+
 sub remove_locks {
     my ($self, $haenv, $id, $locks, $service_node) = @_;
 
diff --git a/src/test/test-migrate-old-target1/README b/src/test/test-migrate-old-target1/README
new file mode 100644
index 0000000..d852f3e
--- /dev/null
+++ b/src/test/test-migrate-old-target1/README
@@ -0,0 +1,8 @@
+Test requested migrations and relocations to a node whose Proxmox VE version
+would drop the guest identity, while the source node keeps it.
+
+The expected outcome is:
+- The moves of vm:101 and ct:102 to node2 are refused before the services are
+  stopped, both keep running on node1
+- The moves of vm:101 and ct:102 to node3, which keeps the guest identity, are
+  done as usual
diff --git a/src/test/test-migrate-old-target1/cmdlist b/src/test/test-migrate-old-target1/cmdlist
new file mode 100644
index 0000000..043688c
--- /dev/null
+++ b/src/test/test-migrate-old-target1/cmdlist
@@ -0,0 +1,6 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on" ],
+    [ "service vm:101 migrate node2", "service ct:102 relocate node2" ],
+    [ "service vm:101 relocate node2" ],
+    [ "service vm:101 migrate node3", "service ct:102 relocate node3" ]
+]
diff --git a/src/test/test-migrate-old-target1/hardware_status b/src/test/test-migrate-old-target1/hardware_status
new file mode 100644
index 0000000..a22a500
--- /dev/null
+++ b/src/test/test-migrate-old-target1/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off", "pve-manager-version": "9.2.29" },
+  "node2": { "power": "off", "network": "off", "pve-manager-version": "9.2.28" },
+  "node3": { "power": "off", "network": "off", "pve-manager-version": "9.2.29" }
+}
diff --git a/src/test/test-migrate-old-target1/log.expect b/src/test/test-migrate-old-target1/log.expect
new file mode 100644
index 0000000..6d8594b
--- /dev/null
+++ b/src/test/test-migrate-old-target1/log.expect
@@ -0,0 +1,73 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'ct:102' on node 'node1'
+info     20    node1/crm: adding new service 'vm:101' on node 'node1'
+info     20    node1/crm: service 'ct:102': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:101': state changed from 'request_start' to 'started'  (node = node1)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service ct:102
+info     21    node1/lrm: service status ct:102 started
+info     21    node1/lrm: starting service vm:101
+info     21    node1/lrm: service status vm:101 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     24    node3/crm: status change wait_for_quorum => slave
+info    120      cmdlist: execute service vm:101 migrate node2
+info    120      cmdlist: execute service ct:102 relocate node2
+info    120    node1/crm: got crm command: migrate vm:101 node2
+info    120    node1/crm: got crm command: relocate ct:102 node2
+info    120    node1/crm: relocate service 'ct:102' to node 'node2'
+info    120    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node1, target = node2)
+info    120    node1/crm: migrate service 'vm:101' to node 'node2'
+info    120    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+err     121    node1/lrm: service ct:102 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err     121    node1/lrm: service vm:101 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info    123    node2/lrm: got lock 'ha_agent_node2_lock'
+info    123    node2/lrm: status change wait_for_agent_lock => active
+err     140    node1/crm: service 'ct:102' - migration failed (exit code 1)
+info    140    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node1)
+err     140    node1/crm: service 'vm:101' - migration failed (exit code 1)
+info    140    node1/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node1)
+info    220      cmdlist: execute service vm:101 relocate node2
+info    220    node1/crm: got crm command: relocate vm:101 node2
+info    220    node1/crm: relocate service 'vm:101' to node 'node2'
+info    220    node1/crm: service 'vm:101': state changed from 'started' to 'relocate'  (node = node1, target = node2)
+err     221    node1/lrm: service vm:101 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err     240    node1/crm: service 'vm:101' - migration failed (exit code 1)
+info    240    node1/crm: service 'vm:101': state changed from 'relocate' to 'started'  (node = node1)
+info    320      cmdlist: execute service vm:101 migrate node3
+info    320      cmdlist: execute service ct:102 relocate node3
+info    320    node1/crm: got crm command: migrate vm:101 node3
+info    320    node1/crm: got crm command: relocate ct:102 node3
+info    320    node1/crm: relocate service 'ct:102' to node 'node3'
+info    320    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node1, target = node3)
+info    320    node1/crm: migrate service 'vm:101' to node 'node3'
+info    320    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node1, target = node3)
+info    321    node1/lrm: service ct:102 - start relocate to node 'node3'
+info    321    node1/lrm: stopping service ct:102 (relocate)
+info    321    node1/lrm: service status ct:102 stopped
+info    321    node1/lrm: service ct:102 - end relocate to node 'node3'
+info    321    node1/lrm: service vm:101 - start migrate to node 'node3'
+info    321    node1/lrm: service vm:101 - end migrate to node 'node3'
+info    325    node3/lrm: got lock 'ha_agent_node3_lock'
+info    325    node3/lrm: status change wait_for_agent_lock => active
+info    340    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node3)
+info    340    node1/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node3)
+info    345    node3/lrm: starting service ct:102
+info    345    node3/lrm: service status ct:102 started
+info    345    node3/lrm: starting service vm:101
+info    345    node3/lrm: service status vm:101 started
+info    920     hardware: exit simulation - done
diff --git a/src/test/test-migrate-old-target1/manager_status b/src/test/test-migrate-old-target1/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-migrate-old-target1/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-migrate-old-target1/service_config b/src/test/test-migrate-old-target1/service_config
new file mode 100644
index 0000000..61bafec
--- /dev/null
+++ b/src/test/test-migrate-old-target1/service_config
@@ -0,0 +1,4 @@
+{
+    "vm:101": { "node": "node1", "state": "started" },
+    "ct:102": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-migrate-old-target2/README b/src/test/test-migrate-old-target2/README
new file mode 100644
index 0000000..c1bdd20
--- /dev/null
+++ b/src/test/test-migrate-old-target2/README
@@ -0,0 +1,10 @@
+Test enabling the maintenance mode on a node that keeps the guest identity,
+where one of the other nodes runs a Proxmox VE version that would drop it.
+
+The expected outcome is:
+- vm:101 is migrated to node3
+- The migration of vm:103 and the relocation of ct:102 to node2 are refused
+  before the services are stopped, both keep running on node1
+- As for any move that keeps failing, the CRM retries vm:103 and ct:102 with
+  node2 until the maintenance mode is disabled again
+- vm:101 is migrated back to node1 after the maintenance mode is disabled
diff --git a/src/test/test-migrate-old-target2/cmdlist b/src/test/test-migrate-old-target2/cmdlist
new file mode 100644
index 0000000..5aa42a6
--- /dev/null
+++ b/src/test/test-migrate-old-target2/cmdlist
@@ -0,0 +1,5 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on" ],
+    [ "crm node1 enable-node-maintenance" ],
+    [ "crm node1 disable-node-maintenance" ]
+]
diff --git a/src/test/test-migrate-old-target2/hardware_status b/src/test/test-migrate-old-target2/hardware_status
new file mode 100644
index 0000000..a22a500
--- /dev/null
+++ b/src/test/test-migrate-old-target2/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off", "pve-manager-version": "9.2.29" },
+  "node2": { "power": "off", "network": "off", "pve-manager-version": "9.2.28" },
+  "node3": { "power": "off", "network": "off", "pve-manager-version": "9.2.29" }
+}
diff --git a/src/test/test-migrate-old-target2/log.expect b/src/test/test-migrate-old-target2/log.expect
new file mode 100644
index 0000000..73d29f6
--- /dev/null
+++ b/src/test/test-migrate-old-target2/log.expect
@@ -0,0 +1,109 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'ct:102' on node 'node1'
+info     20    node1/crm: adding new service 'vm:101' on node 'node1'
+info     20    node1/crm: adding new service 'vm:103' on node 'node1'
+info     20    node1/crm: service 'ct:102': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:101': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:103': state changed from 'request_start' to 'started'  (node = node1)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service ct:102
+info     21    node1/lrm: service status ct:102 started
+info     21    node1/lrm: starting service vm:101
+info     21    node1/lrm: service status vm:101 started
+info     21    node1/lrm: starting service vm:103
+info     21    node1/lrm: service status vm:103 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     24    node3/crm: status change wait_for_quorum => slave
+info    120      cmdlist: execute crm node1 enable-node-maintenance
+info    121    node1/lrm: status change active => maintenance
+info    140    node1/crm: node 'node1': state changed from 'online' => 'maintenance'
+info    140    node1/crm: relocate service 'ct:102' to node 'node2'
+info    140    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node1, target = node2)
+info    140    node1/crm: migrate service 'vm:101' to node 'node3' (running)
+info    140    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node1, target = node3)
+info    140    node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info    140    node1/crm: service 'vm:103': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+err     141    node1/lrm: service ct:102 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info    141    node1/lrm: service vm:101 - start migrate to node 'node3'
+info    141    node1/lrm: service vm:101 - end migrate to node 'node3'
+err     141    node1/lrm: service vm:103 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info    143    node2/lrm: got lock 'ha_agent_node2_lock'
+info    143    node2/lrm: status change wait_for_agent_lock => active
+info    145    node3/lrm: got lock 'ha_agent_node3_lock'
+info    145    node3/lrm: status change wait_for_agent_lock => active
+err     160    node1/crm: service 'ct:102' - migration failed (exit code 1)
+info    160    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node1)
+info    160    node1/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node3)
+err     160    node1/crm: service 'vm:103' - migration failed (exit code 1)
+info    160    node1/crm: service 'vm:103': state changed from 'migrate' to 'started'  (node = node1)
+info    160    node1/crm: relocate service 'ct:102' to node 'node2'
+info    160    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node1, target = node2)
+info    160    node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info    160    node1/crm: service 'vm:103': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+err     161    node1/lrm: service ct:102 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err     161    node1/lrm: service vm:103 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info    165    node3/lrm: starting service vm:101
+info    165    node3/lrm: service status vm:101 started
+err     180    node1/crm: service 'ct:102' - migration failed (exit code 1)
+info    180    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node1)
+err     180    node1/crm: service 'vm:103' - migration failed (exit code 1)
+info    180    node1/crm: service 'vm:103': state changed from 'migrate' to 'started'  (node = node1)
+info    180    node1/crm: relocate service 'ct:102' to node 'node2'
+info    180    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node1, target = node2)
+info    180    node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info    180    node1/crm: service 'vm:103': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+err     181    node1/lrm: service ct:102 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err     181    node1/lrm: service vm:103 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err     200    node1/crm: service 'ct:102' - migration failed (exit code 1)
+info    200    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node1)
+err     200    node1/crm: service 'vm:103' - migration failed (exit code 1)
+info    200    node1/crm: service 'vm:103': state changed from 'migrate' to 'started'  (node = node1)
+info    200    node1/crm: relocate service 'ct:102' to node 'node2'
+info    200    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node1, target = node2)
+info    200    node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info    200    node1/crm: service 'vm:103': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+err     201    node1/lrm: service ct:102 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err     201    node1/lrm: service vm:103 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info    220      cmdlist: execute crm node1 disable-node-maintenance
+err     220    node1/crm: service 'ct:102' - migration failed (exit code 1)
+info    220    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node1)
+err     220    node1/crm: service 'vm:103' - migration failed (exit code 1)
+info    220    node1/crm: service 'vm:103': state changed from 'migrate' to 'started'  (node = node1)
+info    220    node1/crm: relocate service 'ct:102' to node 'node2'
+info    220    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node1, target = node2)
+info    220    node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info    220    node1/crm: service 'vm:103': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+info    221    node1/lrm: status change maintenance => active
+err     221    node1/lrm: service ct:102 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err     221    node1/lrm: service vm:103 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info    240    node1/crm: node 'node1': state changed from 'maintenance' => 'online'
+err     240    node1/crm: service 'ct:102' - migration failed (exit code 1)
+info    240    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node1)
+info    240    node1/crm: moving service 'vm:101' back to 'node1', node came back from maintenance.
+info    240    node1/crm: migrate service 'vm:101' to node 'node1' (running)
+info    240    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node3, target = node1)
+err     240    node1/crm: service 'vm:103' - migration failed (exit code 1)
+info    240    node1/crm: service 'vm:103': state changed from 'migrate' to 'started'  (node = node1)
+info    240    node1/crm: service 'ct:102': clearing stale maintenance node 'node1' setting (is current node)
+info    240    node1/crm: service 'vm:103': clearing stale maintenance node 'node1' setting (is current node)
+info    245    node3/lrm: service vm:101 - start migrate to node 'node1'
+info    245    node3/lrm: service vm:101 - end migrate to node 'node1'
+info    260    node1/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node1)
+info    261    node1/lrm: starting service vm:101
+info    261    node1/lrm: service status vm:101 started
+info    820     hardware: exit simulation - done
diff --git a/src/test/test-migrate-old-target2/manager_status b/src/test/test-migrate-old-target2/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-migrate-old-target2/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-migrate-old-target2/service_config b/src/test/test-migrate-old-target2/service_config
new file mode 100644
index 0000000..671ad7c
--- /dev/null
+++ b/src/test/test-migrate-old-target2/service_config
@@ -0,0 +1,5 @@
+{
+    "vm:101": { "node": "node1", "state": "started" },
+    "ct:102": { "node": "node1", "state": "started" },
+    "vm:103": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-migrate-old-target3/README b/src/test/test-migrate-old-target3/README
new file mode 100644
index 0000000..512bec2
--- /dev/null
+++ b/src/test/test-migrate-old-target3/README
@@ -0,0 +1,11 @@
+Test requested moves between nodes whose Proxmox VE versions differ in whether
+they keep the guest identity. Only a source node that keeps the identity
+refuses a target that would drop it.
+
+The expected outcome is:
+- The moves of vm:101 and ct:102 from node1 to node2 are done, as node1 does
+  not keep the guest identity either
+- The moves of vm:101 and ct:102 from node2 to node3 are done, as node3 keeps
+  the guest identity
+- The moves of vm:101 and ct:102 from node3 back to node1 are refused before
+  the services are stopped, both keep running on node3
diff --git a/src/test/test-migrate-old-target3/cmdlist b/src/test/test-migrate-old-target3/cmdlist
new file mode 100644
index 0000000..be88920
--- /dev/null
+++ b/src/test/test-migrate-old-target3/cmdlist
@@ -0,0 +1,6 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on" ],
+    [ "service vm:101 migrate node2", "service ct:102 relocate node2" ],
+    [ "service vm:101 migrate node3", "service ct:102 relocate node3" ],
+    [ "service vm:101 migrate node1", "service ct:102 relocate node1" ]
+]
diff --git a/src/test/test-migrate-old-target3/hardware_status b/src/test/test-migrate-old-target3/hardware_status
new file mode 100644
index 0000000..b5d1ded
--- /dev/null
+++ b/src/test/test-migrate-old-target3/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off", "pve-manager-version": "9.2.28" },
+  "node2": { "power": "off", "network": "off", "pve-manager-version": "9.2.28" },
+  "node3": { "power": "off", "network": "off", "pve-manager-version": "9.2.29" }
+}
diff --git a/src/test/test-migrate-old-target3/log.expect b/src/test/test-migrate-old-target3/log.expect
new file mode 100644
index 0000000..8cf50a7
--- /dev/null
+++ b/src/test/test-migrate-old-target3/log.expect
@@ -0,0 +1,86 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'ct:102' on node 'node1'
+info     20    node1/crm: adding new service 'vm:101' on node 'node1'
+info     20    node1/crm: service 'ct:102': state changed from 'request_start' to 'started'  (node = node1)
+info     20    node1/crm: service 'vm:101': state changed from 'request_start' to 'started'  (node = node1)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+info     21    node1/lrm: starting service ct:102
+info     21    node1/lrm: service status ct:102 started
+info     21    node1/lrm: starting service vm:101
+info     21    node1/lrm: service status vm:101 started
+info     22    node2/crm: status change wait_for_quorum => slave
+info     24    node3/crm: status change wait_for_quorum => slave
+info    120      cmdlist: execute service vm:101 migrate node2
+info    120      cmdlist: execute service ct:102 relocate node2
+info    120    node1/crm: got crm command: migrate vm:101 node2
+info    120    node1/crm: got crm command: relocate ct:102 node2
+info    120    node1/crm: relocate service 'ct:102' to node 'node2'
+info    120    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node1, target = node2)
+info    120    node1/crm: migrate service 'vm:101' to node 'node2'
+info    120    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node1, target = node2)
+info    121    node1/lrm: service ct:102 - start relocate to node 'node2'
+info    121    node1/lrm: stopping service ct:102 (relocate)
+info    121    node1/lrm: service status ct:102 stopped
+info    121    node1/lrm: service ct:102 - end relocate to node 'node2'
+info    121    node1/lrm: service vm:101 - start migrate to node 'node2'
+info    121    node1/lrm: service vm:101 - end migrate to node 'node2'
+info    123    node2/lrm: got lock 'ha_agent_node2_lock'
+info    123    node2/lrm: status change wait_for_agent_lock => active
+info    140    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node2)
+info    140    node1/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node2)
+info    143    node2/lrm: starting service ct:102
+info    143    node2/lrm: service status ct:102 started
+info    143    node2/lrm: starting service vm:101
+info    143    node2/lrm: service status vm:101 started
+info    220      cmdlist: execute service vm:101 migrate node3
+info    220      cmdlist: execute service ct:102 relocate node3
+info    220    node1/crm: got crm command: migrate vm:101 node3
+info    220    node1/crm: got crm command: relocate ct:102 node3
+info    220    node1/crm: relocate service 'ct:102' to node 'node3'
+info    220    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node2, target = node3)
+info    220    node1/crm: migrate service 'vm:101' to node 'node3'
+info    220    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node2, target = node3)
+info    223    node2/lrm: service ct:102 - start relocate to node 'node3'
+info    223    node2/lrm: stopping service ct:102 (relocate)
+info    223    node2/lrm: service status ct:102 stopped
+info    223    node2/lrm: service ct:102 - end relocate to node 'node3'
+info    223    node2/lrm: service vm:101 - start migrate to node 'node3'
+info    223    node2/lrm: service vm:101 - end migrate to node 'node3'
+info    225    node3/lrm: got lock 'ha_agent_node3_lock'
+info    225    node3/lrm: status change wait_for_agent_lock => active
+info    240    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node3)
+info    240    node1/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node3)
+info    245    node3/lrm: starting service ct:102
+info    245    node3/lrm: service status ct:102 started
+info    245    node3/lrm: starting service vm:101
+info    245    node3/lrm: service status vm:101 started
+info    320      cmdlist: execute service vm:101 migrate node1
+info    320      cmdlist: execute service ct:102 relocate node1
+info    320    node1/crm: got crm command: migrate vm:101 node1
+info    320    node1/crm: got crm command: relocate ct:102 node1
+info    320    node1/crm: relocate service 'ct:102' to node 'node1'
+info    320    node1/crm: service 'ct:102': state changed from 'started' to 'relocate'  (node = node3, target = node1)
+info    320    node1/crm: migrate service 'vm:101' to node 'node1'
+info    320    node1/crm: service 'vm:101': state changed from 'started' to 'migrate'  (node = node3, target = node1)
+err     325    node3/lrm: service ct:102 not moved - node 'node1' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err     325    node3/lrm: service vm:101 not moved - node 'node1' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err     340    node1/crm: service 'ct:102' - migration failed (exit code 1)
+info    340    node1/crm: service 'ct:102': state changed from 'relocate' to 'started'  (node = node3)
+err     340    node1/crm: service 'vm:101' - migration failed (exit code 1)
+info    340    node1/crm: service 'vm:101': state changed from 'migrate' to 'started'  (node = node3)
+info    920     hardware: exit simulation - done
diff --git a/src/test/test-migrate-old-target3/manager_status b/src/test/test-migrate-old-target3/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-migrate-old-target3/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-migrate-old-target3/service_config b/src/test/test-migrate-old-target3/service_config
new file mode 100644
index 0000000..61bafec
--- /dev/null
+++ b/src/test/test-migrate-old-target3/service_config
@@ -0,0 +1,4 @@
+{
+    "vm:101": { "node": "node1", "state": "started" },
+    "ct:102": { "node": "node1", "state": "started" }
+}
diff --git a/src/test/test-migrate-old-target4/README b/src/test/test-migrate-old-target4/README
new file mode 100644
index 0000000..6f38bda
--- /dev/null
+++ b/src/test/test-migrate-old-target4/README
@@ -0,0 +1,7 @@
+Test the rebalancing on start of a service, where the selected node runs a
+Proxmox VE version that would drop the guest identity, while the current node
+keeps it.
+
+The expected outcome is:
+- The move of ct:103 to node2 for its start is refused
+- ct:103 is started on its current node node1 instead
diff --git a/src/test/test-migrate-old-target4/cmdlist b/src/test/test-migrate-old-target4/cmdlist
new file mode 100644
index 0000000..1172dcc
--- /dev/null
+++ b/src/test/test-migrate-old-target4/cmdlist
@@ -0,0 +1,4 @@
+[
+    [ "power node1 on", "power node2 on", "power node3 on" ],
+    [ "service ct:103 started" ]
+]
diff --git a/src/test/test-migrate-old-target4/datacenter.cfg b/src/test/test-migrate-old-target4/datacenter.cfg
new file mode 100644
index 0000000..f2671a5
--- /dev/null
+++ b/src/test/test-migrate-old-target4/datacenter.cfg
@@ -0,0 +1,6 @@
+{
+    "crs": {
+        "ha": "static",
+        "ha-rebalance-on-start": 1
+    }
+}
diff --git a/src/test/test-migrate-old-target4/hardware_status b/src/test/test-migrate-old-target4/hardware_status
new file mode 100644
index 0000000..eaede03
--- /dev/null
+++ b/src/test/test-migrate-old-target4/hardware_status
@@ -0,0 +1,5 @@
+{
+  "node1": { "power": "off", "network": "off", "pve-manager-version": "9.2.29", "maxcpu": 24, "maxmem": 34359738368 },
+  "node2": { "power": "off", "network": "off", "pve-manager-version": "9.2.28", "maxcpu": 24, "maxmem": 34359738368 },
+  "node3": { "power": "off", "network": "off", "pve-manager-version": "9.2.28", "maxcpu": 24, "maxmem": 34359738368 }
+}
diff --git a/src/test/test-migrate-old-target4/log.expect b/src/test/test-migrate-old-target4/log.expect
new file mode 100644
index 0000000..6b77377
--- /dev/null
+++ b/src/test/test-migrate-old-target4/log.expect
@@ -0,0 +1,52 @@
+info      0     hardware: starting simulation
+info     20      cmdlist: execute power node1 on
+info     20    node1/crm: status change startup => wait_for_quorum
+info     20    node1/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node2 on
+info     20    node2/crm: status change startup => wait_for_quorum
+info     20    node2/lrm: status change startup => wait_for_agent_lock
+info     20      cmdlist: execute power node3 on
+info     20    node3/crm: status change startup => wait_for_quorum
+info     20    node3/lrm: status change startup => wait_for_agent_lock
+info     20    node1/crm: got lock 'ha_manager_lock'
+info     20    node1/crm: status change wait_for_quorum => master
+info     20    node1/crm: using scheduler mode 'static'
+info     20    node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info     20    node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info     20    node1/crm: adding new service 'ct:103' on node 'node1'
+info     20    node1/crm: adding new service 'vm:101' on node 'node1'
+info     20    node1/crm: adding new service 'vm:102' on node 'node1'
+info     20    node1/crm: service vm:101: re-balance selected new node node2 for startup
+info     20    node1/crm: service 'vm:101': state changed from 'request_start' to 'request_start_balance'  (node = node1, target = node2)
+info     20    node1/crm: service vm:102: re-balance selected new node node3 for startup
+info     20    node1/crm: service 'vm:102': state changed from 'request_start' to 'request_start_balance'  (node = node1, target = node3)
+info     21    node1/lrm: got lock 'ha_agent_node1_lock'
+info     21    node1/lrm: status change wait_for_agent_lock => active
+err      21    node1/lrm: service vm:101 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err      21    node1/lrm: service vm:102 not moved - node 'node3' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+info     22    node2/crm: status change wait_for_quorum => slave
+info     23    node2/lrm: got lock 'ha_agent_node2_lock'
+info     23    node2/lrm: status change wait_for_agent_lock => active
+info     24    node3/crm: status change wait_for_quorum => slave
+info     25    node3/lrm: got lock 'ha_agent_node3_lock'
+info     25    node3/lrm: status change wait_for_agent_lock => active
+info     40    node1/crm: service 'ct:103': state changed from 'request_stop' to 'stopped'
+err      40    node1/crm: service 'vm:101' - migration failed (exit code 1)
+info     40    node1/crm: service 'vm:101': state changed from 'request_start_balance' to 'started'  (node = node1)
+err      40    node1/crm: service 'vm:102' - migration failed (exit code 1)
+info     40    node1/crm: service 'vm:102': state changed from 'request_start_balance' to 'started'  (node = node1)
+info     41    node1/lrm: starting service vm:101
+info     41    node1/lrm: service status vm:101 started
+info     41    node1/lrm: starting service vm:102
+info     41    node1/lrm: service status vm:102 started
+info    120      cmdlist: execute service ct:103 started
+info    120    node1/crm: service 'ct:103': state changed from 'stopped' to 'request_start'  (node = node1)
+info    120    node1/crm: service ct:103: re-balance selected new node node2 for startup
+info    120    node1/crm: service 'ct:103': state changed from 'request_start' to 'request_start_balance'  (node = node1, target = node2)
+err     121    node1/lrm: service ct:103 not moved - node 'node2' runs Proxmox VE 9.2.28, keeping the guest identity requires 9.2.29 or newer - upgrade the node first
+err     140    node1/crm: service 'ct:103' - migration failed (exit code 1)
+info    140    node1/crm: service 'ct:103': state changed from 'request_start_balance' to 'started'  (node = node1)
+info    141    node1/lrm: starting service ct:103
+info    141    node1/lrm: service status ct:103 started
+info    720     hardware: exit simulation - done
diff --git a/src/test/test-migrate-old-target4/manager_status b/src/test/test-migrate-old-target4/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-migrate-old-target4/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-migrate-old-target4/service_config b/src/test/test-migrate-old-target4/service_config
new file mode 100644
index 0000000..98eb368
--- /dev/null
+++ b/src/test/test-migrate-old-target4/service_config
@@ -0,0 +1,5 @@
+{
+    "vm:101": { "node": "node1", "state": "started" },
+    "vm:102": { "node": "node1", "state": "started" },
+    "ct:103": { "node": "node1", "state": "stopped" }
+}
diff --git a/src/test/test-migrate-old-target4/static_service_stats b/src/test/test-migrate-old-target4/static_service_stats
new file mode 100644
index 0000000..b3be93e
--- /dev/null
+++ b/src/test/test-migrate-old-target4/static_service_stats
@@ -0,0 +1,5 @@
+{
+    "vm:101": { "maxcpu": 2.0, "maxmem": 8589934592 },
+    "vm:102": { "maxcpu": 2.0, "maxmem": 8589934592 },
+    "ct:103": { "maxcpu": 2.0, "maxmem": 8589934592 }
+}
-- 
2.47.3





                 reply	other threads:[~2026-11-20  2:55 UTC|newest]

Thread overview: [no followups] expand[flat|nested]  mbox.gz  Atom feed

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120025517.3031086-1-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal