[PATCH qemu-server] migrate: escalate cpu throttling and abort when RAM migration cannot converge

[PATCH qemu-server] migrate: escalate cpu throttling and abort when RAM migration cannot converge

[Michael Ryom]

When a guest dirties its memory faster than the RAM migration can
transfer it, the migration runs for an unbounded time: QEMU's
auto-converge only throttles the guest's vCPUs in small steps per
migration iteration, and the status loop merely appends "VM dirties
lots of memory" to the progress log while auto-increasing the downtime
limit. Observed live: a 24 GiB guest with ~430 MiB/s transfer speed and
a 600-760 MiB/s dirty rate had transferred 62 GiB (2.6 full passes over
the guest memory) after 2.5 minutes, with the downtime limit already
escalated to 800 ms, and only stopped because it was cancelled
manually. Meanwhile the migration saturates the migration network and,
for a rebalance migration issued by the HA manager, occupies the
serialized motion pipeline indefinitely.

Every full additional pass over the guest memory means the guest
dirtied all of it again before the transfer could finish. Use that as a
robust convergence measure (the reported dirty-pages-rate only pulses
during sync phases) and act in two stages:

- after 1.5 passes, escalate the auto-converge parameters
  (cpu-throttle-increment 20, max-cpu-throttle 99), so the guest is
  throttled harder and faster to force convergence at the cost of guest
  cpu time;
- after 4 passes, i.e. if even the escalated throttling does not lead
  to convergence, abort the migration with a clear error instead of
  transferring indefinitely.

The factors are constants for now and could become migration options
later. For rebalance migrations, the HA manager backs off failed
motions, so an aborted non-converging migration is not simply retried
immediately.

Add a regression test simulating a non-converging RAM migration, which
checks that the throttle escalation is attempted and the migration is
aborted afterwards.

Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
 src/PVE/QemuMigrate.pm                    | 38 +++++++++++++++++++++++
 src/test/MigrationTest/QemuMigrateMock.pm | 21 +++++++++++++
 src/test/run_qemu_migrate_tests.pl        | 29 +++++++++++++++++
 3 files changed, 88 insertions(+)

diff --git a/src/PVE/QemuMigrate.pm b/src/PVE/QemuMigrate.pm
index 8da6f15d..c6a6799c 100644
--- a/src/PVE/QemuMigrate.pm
+++ b/src/PVE/QemuMigrate.pm
@@ -1176,6 +1176,14 @@ sub phase2_start_remote_cluster {
 
 my $migrate_downtime_max = 2000 * 1000; # as defined in QEMU's migration/options.c
 
+# when the RAM migration has transferred more than this factor of the guest
+# memory without completing, the guest dirties memory faster than it can be
+# transferred; escalate the auto-converge cpu throttling to force convergence
+my $migrate_throttle_boost_factor = 1.5;
+# ... and when even that does not lead to convergence, abort the migration
+# instead of transferring indefinitely
+my $migrate_abort_factor = 4;
+
 my sub cap_migrate_downtime {
     my ($self, $migrate_downtime) = @_;
 
@@ -1442,6 +1450,8 @@ sub phase2 {
     my $err_count = 0;
     my $lastrem = undef;
     my $downtimecounter = 0;
+    my $cpu_throttle_boosted = 0;
+
     while (1) {
         $i++;
         my $avglstat = $last_mem_transferred ? $last_mem_transferred / $i : 0;
@@ -1560,6 +1570,34 @@ sub phase2 {
                 $self->log('info', "xbzrle: $msg") if $should_log;
             }
 
+            # every full additional pass over the guest memory means the
+            # guest dirtied all of it again before the transfer could finish
+            if ($total > 0 && $mem_transferred > $migrate_abort_factor * $total) {
+                my $passes = sprintf("%.1f", $mem_transferred / $total);
+                die "aborting migration - no convergence: transferred ${passes}x the guest"
+                    . " memory without completing, the guest dirties its memory faster than"
+                    . " it can be transferred\n";
+            } elsif (
+                $total > 0
+                && !$cpu_throttle_boosted
+                && $mem_transferred > $migrate_throttle_boost_factor * $total
+            ) {
+                $cpu_throttle_boosted = 1;
+                $self->log(
+                    'info',
+                    "migration does not converge - escalating the auto-converge cpu"
+                        . " throttling of the guest to force convergence",
+                );
+                eval {
+                    mon_cmd(
+                        $vmid, "migrate-set-parameters",
+                        'cpu-throttle-increment' => 20,
+                        'max-cpu-throttle' => 99,
+                    );
+                };
+                $self->log('info', "migrate-set-parameters error: $@") if $@;
+            }
+
             if (($lastrem && $rem > $lastrem) || ($rem == 0)) {
                 $downtimecounter++;
             }
diff --git a/src/test/MigrationTest/QemuMigrateMock.pm b/src/test/MigrationTest/QemuMigrateMock.pm
index 3a483817..9ecdd502 100644
--- a/src/test/MigrationTest/QemuMigrateMock.pm
+++ b/src/test/MigrationTest/QemuMigrateMock.pm
@@ -31,6 +31,7 @@ my $test_vmid = $migrate_params->{vmid};
 my $test_target = $migrate_params->{target};
 my $test_opts = $migrate_params->{opts};
 my $current_log = '';
+my $query_migrate_calls = 0;
 
 my $vm_stop_executed = 0;
 
@@ -102,10 +103,30 @@ $qemu_migrate_module->mock(
             return;
         } elsif ($command eq 'query-migrate') {
             return { status => 'failed' } if $fail_config->{'query-migrate'};
+            if ($fail_config->{'query-migrate-nonconverging'}) {
+                $query_migrate_calls++;
+                my $gib = 1024 * 1024 * 1024;
+                # the guest dirties its memory faster than the transfer speed,
+                # so the transferred amount keeps growing by more than a full
+                # pass over the guest memory without the migration finishing
+                return {
+                    status => 'active',
+                    ram => {
+                        total => 4 * $gib,
+                        transferred => $query_migrate_calls * 7 * $gib,
+                        remaining => 2 * $gib,
+                        'page-size' => 4096,
+                        'pages-per-second' => 102400, # 400 MiB/s
+                        'dirty-pages-rate' => 179200, # 700 MiB/s
+                    },
+                };
+            }
             return { status => 'completed' };
         } elsif ($command eq 'migrate') {
             return;
         } elsif ($command eq 'migrate-set-parameters') {
+            delete $expected_calls->{'migrate-set-parameters-throttle'}
+                if defined($params{'cpu-throttle-increment'});
             return;
         } elsif ($command eq 'migrate_cancel') {
             return;
diff --git a/src/test/run_qemu_migrate_tests.pl b/src/test/run_qemu_migrate_tests.pl
index 05eed1d9..861011ea 100755
--- a/src/test/run_qemu_migrate_tests.pl
+++ b/src/test/run_qemu_migrate_tests.pl
@@ -717,6 +717,35 @@ my $tests = [
             },
         },
     },
+    {
+        name => '1033_running_nonconverging',
+        target => 'pve2',
+        vmid => 1033,
+        vm_status => {
+            running => 1,
+            runningmachine => 'pc-q35-5.0+pve0',
+        },
+        opts => {
+            online => 1,
+        },
+        fail_config => {
+            'query-migrate-nonconverging' => 1,
+        },
+        # the cpu throttle escalation must have been attempted before aborting
+        expected_calls => {
+            'migrate-set-parameters-throttle' => 1,
+        },
+        expect_die => 'online migrate failure - aborting migration - no convergence',
+        expected => {
+            source_volids => {},
+            target_volids => {},
+            vm_config => $vm_configs->{1033},
+            vm_status => {
+                running => 1,
+                runningmachine => 'pc-q35-5.0+pve0',
+            },
+        },
+    },
     {
         name => '4567_targetstorage_dirotherdir',
         target => 'pve1',
-- 
2.47.3