From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id CAA851FF09B for ; Mon, 14 Sep 2026 10:58:21 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 3695F21481; Mon, 14 Sep 2026 10:58:14 +0200 (CEST) From: Dominik Csapak To: pve-devel@lists.proxmox.com Subject: [PATCH qemu-server v2 1/6] tests: hotplug: add initial hotplug test harness Date: Mon, 14 Sep 2026 10:54:30 +0200 Message-ID: <20260914085725.1299009-2-d.csapak@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20260914085725.1299009-1-d.csapak@proxmox.com> References: <20260914085725.1299009-1-d.csapak@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.530 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_MAILER 2 Automated Mailer Tag Left in Email RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 3JYT7HE64RICDT7TKTHWDQW3HDHFMRJE X-Message-ID-Hash: 3JYT7HE64RICDT7TKTHWDQW3HDHFMRJE X-MailFrom: d.csapak@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Add a new test harness for testing the hotplug mechanism. A test case consists of failure modes and a vm config with a [PENDING] section. After setting up the initial device list via config_to_command, the pending section is applied and all qmp/hmp/etc. calls are recorded and compared with the expected file for each test. In addition, there are certain constraints tested, like hotplugging into a hotplugged bridge fails, a bus is not hotpluggable, or the device already exists. Note that errors in tests don't let the tests fail, but are recorded in the output so that we can test behavior that we might need to keep to preserve backwards compatibility. After hotplugging, the resulting config is checked again with config_to_command and the device lists are compared (any difference here could lead to a resource leak and/or live migration failure). The harness is modeled after cfg2cmd. It supports cases for x86 and aarch64 host architectures. Since most mocks and setup steps overlap with the cfg2cmd tests, factor out the relevant code into separate modules which can used by multiple test. The exact setup and diff code change slightly, but the comparison and output should behave the same. No test cases are added yet, these come in separate patches. Signed-off-by: Dominik Csapak --- src/test/Makefile | 8 +- src/test/TestsCommon/CommandLineMocks.pm | 537 +++++++++++++++ src/test/TestsCommon/Utils.pm | 31 + src/test/run_config2command_tests.pl | 553 +-------------- src/test/run_hotplug_tests.pl | 833 +++++++++++++++++++++++ src/test/run_parse_config_tests.pl | 25 +- 6 files changed, 1432 insertions(+), 555 deletions(-) create mode 100644 src/test/TestsCommon/CommandLineMocks.pm create mode 100644 src/test/TestsCommon/Utils.pm create mode 100644 src/test/run_hotplug_tests.pl diff --git a/src/test/Makefile b/src/test/Makefile index cf589f41..aab0789e 100644 --- a/src/test/Makefile +++ b/src/test/Makefile @@ -6,10 +6,10 @@ test_snapshot: run_snapshot_tests.pl ./run_snapshot_tests.pl ./test_get_replicatable_volumes.pl -test_cfg_to_cmd: run_config2command_tests.pl cfg2cmd/*.conf +test_cfg_to_cmd: run_config2command_tests.pl TestsCommon/*.pm cfg2cmd/*.conf perl -I../ ./run_config2command_tests.pl -test_cfg_to_cmd_aarch64: run_config2command_tests.pl cfg2cmd/aarch64/*.conf +test_cfg_to_cmd_aarch64: run_config2command_tests.pl TestsCommon/*.pm cfg2cmd/aarch64/*.conf perl -I../ ./run_config2command_tests.pl cfg2cmd/aarch64 test_qemu_img_convert: run_qemu_img_convert_tests.pl @@ -31,9 +31,9 @@ $(MIGRATION_TEST_TARGETS): test_restore_config: run_qemu_restore_config_tests.pl ./run_qemu_restore_config_tests.pl -test_parse_config: run_parse_config_tests.pl +test_parse_config: run_parse_config_tests.pl TestsCommon/Utils.pm ./run_parse_config_tests.pl .PHONY: clean clean: - rm -rf MigrationTest/run parse-config-output + rm -rf MigrationTest/run diff --git a/src/test/TestsCommon/CommandLineMocks.pm b/src/test/TestsCommon/CommandLineMocks.pm new file mode 100644 index 00000000..50540e8f --- /dev/null +++ b/src/test/TestsCommon/CommandLineMocks.pm @@ -0,0 +1,537 @@ +package TestsCommon::CommandLineMocks; + +# Mocks for generating the QEMU command line of a VM config in tests, shared +# between the test scripts for the command line itself and for hotplugging. The +# environment (storage configuration, PCI devices, mappings, ...) is fixed, +# while a few properties of the host can be set per test via set_test_env(). +# Also contains helpers that are useful and needed by multiple test harnesses. + +use v5.36; + +use JSON qw(decode_json); +use Socket qw(AF_INET AF_INET6); +use Test::MockModule; + +use PVE::INotify; +use PVE::Mapping::PCI; +use PVE::Mapping::USB; +use PVE::ProcFSTools; +use PVE::QemuServer::CPUConfig; +use PVE::QemuServer::Drive; +use PVE::QemuServer::Helpers; +use PVE::QemuServer::Memory; +use PVE::QemuServer::OVMF; +use PVE::QemuServer::PCI; +use PVE::QemuServer; +use PVE::Storage::RBDPlugin; +use PVE::Storage::ZFSPlugin; +use PVE::Storage; +use PVE::SysFSTools; +use PVE::Tools qw(run_command); + +use base 'Exporter'; + +our @EXPORT_OK = qw( + get_storage_config + get_test_qemu_version + set_test_env +); + +my $real_qemu_version = PVE::QemuServer::Helpers::kvm_user_version(); # can be overwritten by individual tests + +# the properties of the host for the current test, see set_test_env() +my $test_env = {}; + +my $storage_config = { + ids => { + local => { + content => { + images => 1, + iso => 1, + }, + path => '/var/lib/vz', + type => 'dir', + shared => 0, + }, + localsnapext => { + content => { + images => 1, + }, + path => '/var/lib/vzsnapext', + type => 'dir', + shared => 0, + 'snapshot-as-volume-chain' => 1, + }, + noimages => { + content => { + iso => 1, + }, + path => '/var/lib/vz', + type => 'dir', + }, + 'btrfs-store' => { + content => { + images => 1, + }, + path => '/butter/bread', + type => 'btrfs', + }, + 'cifs-store' => { + shared => 1, + path => '/mnt/pve/cifs-store', + username => 'guest', + server => '127.0.0.42', + type => 'cifs', + share => 'CIFShare', + content => { + images => 1, + iso => 1, + }, + }, + 'rbd-store' => { + monhost => '127.0.0.42,127.0.0.21,::1', + fsid => 'fc4181a6-56eb-4f68-b452-8ba1f381ca2a', + content => { + images => 1, + }, + type => 'rbd', + pool => 'cpool', + username => 'admin', + shared => 1, + }, + 'krbd-store' => { + monhost => '127.0.0.42,127.0.0.21,::1', + fsid => 'fc4181a6-56eb-4f68-b452-8ba1f381ca2a', + content => { + images => 1, + }, + type => 'rbd', + pool => 'cpool', + username => 'admin', + shared => 1, + krbd => 1, + }, + 'zfs-over-iscsi-store' => { + type => 'zfs', + iscsiprovider => "comstar", + lio_tpg => "tpg1", + portal => "127.0.0.1", + target => "iqn.2019-10.org.test:foobar", + pool => "tank", + content => { + images => 1, + }, + }, + 'lvm-store' => { + vgname => 'veegee', + type => 'lvm', + content => { + images => 1, + }, + }, + 'local-lvm' => { + vgname => 'pve', + bwlimit => 'restore=1024', + type => 'lvmthin', + thinpool => 'data', + content => { + images => 1, + }, + }, + }, +}; + +my $pci_devs = [ + "0000:00:02.0", + "0000:00:43.1", + "0000:00:f4.0", + "0000:00:ff.1", + "0000:0f:f2.0", + "0000:d0:13.0", + "0000:d0:15.1", + "0000:d0:15.2", + "0000:d0:17.0", + "0000:f0:42.0", + "0000:f0:43.0", + "0000:f0:43.1", + "1234:f0:43.1", + "0000:01:00.4", + "0000:01:00.5", + "0000:01:00.6", + "0000:07:10.0", + "0000:07:10.1", + "0000:07:10.4", +]; + +my $pci_map_config = { + ids => { + someGpu => { + type => 'pci', + mdev => 1, + map => [ + 'node=localhost,path=0000:01:00.4,id=10de:2231,iommugroup=1', + 'node=localhost,path=0000:01:00.5,id=10de:2231,iommugroup=1', + 'node=localhost,path=0000:01:00.6,id=10de:2231,iommugroup=1', + ], + }, + someNic => { + type => 'pci', + map => [ + 'node=localhost,path=0000:07:10.0,id=8086:1520,iommugroup=2', + 'node=localhost,path=0000:07:10.1,id=8086:1520,iommugroup=2', + 'node=localhost,path=0000:07:10.4,id=8086:1520,iommugroup=2', + ], + }, + }, +}; + +my $usb_map_config = {}; + +my $cpu_hw_capabilities = { + # Gathered from an AMD EPYC 9475F running kernel 6.11.11-2-pve + 'amd-turin-9005' => + '{ "amd-sev": { "cbitpos": 51, "reduced-phys-bits": 6, "sev-support": true,' + . ' "sev-support-es": true, "sev-support-snp": true } }', + # TODO: others? +}; + +# Set the properties of the (mocked) host for the following test, fields are: +# qemu_version: version of the installed QEMU binary (defaults to the real version) +# host_arch: x86_64 | aarch64 (defaults to x86_64) +# host-cpu-vendor: AuthenticAMD | GenuineIntel | etc. (defaults to GenuineIntel) +# hw_capabilities: a CPU from $cpu_hw_capabilities above or a raw JSON string +sub set_test_env($env) { + $test_env = $env; +} + +sub get_test_qemu_version() { + return $test_env->{qemu_version} // $real_qemu_version // '2.12'; +} + +sub get_storage_config() { + return $storage_config; +} + +# Use 'our' for the mocker objects, since they need to stay alive after loading. Using 'my' here +# would free them after loading the module and negate the mocking. + +our $procfs_tools_module = Test::MockModule->new('PVE::ProcFSTools'); +$procfs_tools_module->mock( + read_cpuinfo => sub { + my $res = $procfs_tools_module->original('read_cpuinfo')->(); + my $vendor = $test_env->{'host-cpu-vendor'} // 'GenuineIntel'; + $res->{vendor} = $vendor; + return $res; + }, +); + +our $qemu_server_module; +$qemu_server_module = Test::MockModule->new('PVE::QemuServer'); +$qemu_server_module->mock( + kvm_user_version => sub { + return get_test_qemu_version(); + }, + kvm_version => sub { + return get_test_qemu_version(); + }, + kernel_has_vhost_net => sub { + return 1; # TODO: make this per-test configurable? + }, + get_iscsi_initiator_name => sub { + return 'iqn.1993-08.org.debian:01:aabbccddeeff'; + }, + cleanup_pci_devices => { + # do nothing + }, +); + +our $qemu_server_ovmf_module = Test::MockModule->new("PVE::QemuServer::OVMF"); +$qemu_server_ovmf_module->mock( + file_exists => sub { + my ($path) = @_; + return 1; + }, + file_get_size => sub { + my ($path) = @_; + if ($path =~ m/OVMF(32)?_(SEV_)?VARS_4M/) { + return 528 * 1024; + } elsif ($path =~ m/OVMF_VARS/) { + return 128 * 1024; + } elsif ($path =~ m/AAVMF_VARS/) { + return 64 * 1024 * 1024; + } elsif ($path =~ m/RISCV_VIRT_VARS/) { + return 32 * 1024 * 1024; + } else { + die "unknown ovmf vars image '$path' - implement me"; + } + }, +); + +our $storage_module = Test::MockModule->new("PVE::Storage"); +$storage_module->mock( + activate_volumes => sub { + return; + }, + deactivate_volumes => sub { + return; + }, + volume_snapshot_info => sub { + my ($cfg, $volid) = @_; + + my ($storeid, $volname) = PVE::Storage::parse_volume_id($volid); + + my $snapshots = {}; + if ($storeid eq 'localsnapext') { + $snapshots = { + current => { + file => 'var/lib/vzsnapext/images/8006/vm-8006-disk-0.qcow2', + parent => 'snap2', + }, + snap2 => { + file => '/var/lib/vzsnapext/images/8006/snap2-vm-8006-disk-0.qcow2', + parent => 'snap1', + }, + snap1 => { + file => '/var/lib/vzsnapext/images/8006/snap1-vm-8006-disk-0.qcow2', + }, + }; + } elsif ($storeid eq 'lvm-store') { + $snapshots = { + current => { + file => '/dev/veegee/vm-8006-disk-0.qcow2', + parent => 'snap2', + }, + snap2 => { + file => '/dev/veegee/snap2-vm-8006-disk-0.qcow2', + parent => 'snap1', + }, + snap1 => { + file => '/dev/veegee/snap1-vm-8006-disk-0.qcow2', + }, + }; + } + return $snapshots; + }, +); + +our $file_stat_module = Test::MockModule->new("File::stat"); +$file_stat_module->mock( + stat => sub { + my ($path) = @_; + my $st = $file_stat_module->original('stat')->($0); + $st->[2] = 25008 if $path =~ m!/dev/!; # block device + return $st; + }, +); + +our $zfsplugin_module = Test::MockModule->new("PVE::Storage::ZFSPlugin"); +$zfsplugin_module->mock( + zfs_get_lu_name => sub { + return "foobar"; + }, + zfs_get_lun_number => sub { + return "0"; + }, +); + +our $rbdplugin_module = Test::MockModule->new("PVE::Storage::RBDPlugin"); +$rbdplugin_module->mock( + rbd_volume_config_set => sub { + return; + }, +); + +our $qemu_server_helpers; +$qemu_server_helpers = Test::MockModule->new('PVE::QemuServer::Helpers'); +$qemu_server_helpers->mock( + get_host_phys_address_bits => sub { + return 46; + }, +); + +our $qemu_server_memory; +$qemu_server_memory = Test::MockModule->new('PVE::QemuServer::Memory'); +$qemu_server_memory->mock( + hugepages_chunk_size_supported => sub { + return 1; + }, + host_numanode_exists => sub { + my ($id) = @_; + return 1; + }, +); + +our $pve_common_tools; +$pve_common_tools = Test::MockModule->new('PVE::Tools'); +$pve_common_tools->mock( + next_vnc_port => sub { + my ($family, $address) = @_; + + return '5900'; + }, + next_spice_port => sub { + my ($family, $address) = @_; + + return '61000'; + }, + getaddrinfo_all => sub { + my ($hostname, @opts) = @_; + die "need stable hostname" if $hostname ne 'localhost'; + return ( + { + addr => Socket::pack_sockaddr_in(0, Socket::INADDR_LOOPBACK), + family => AF_INET, # IPv4 + protocol => 6, + socktype => 1, + }, + ); + }, + get_host_arch => sub { + return $test_env->{host_arch} // 'x86_64'; + }, +); + +our $pve_common_file; +$pve_common_file = Test::MockModule->new('PVE::File'); +$pve_common_file->mock( + file_copy => sub { + my ($filename, $dst, $max, $perm) = @_; + if ($dst =~ m|^/run/qemu-server/efidisk|) { + return; + } + return $pve_common_file->original('file_copy')->($filename, $dst, $max, $perm); + }, +); + +our $pve_cpuconfig; +$pve_cpuconfig = Test::MockModule->new('PVE::QemuServer::CPUConfig'); +$pve_cpuconfig->mock( + load_custom_cpu_model_config => sub { + # mock custom CPU model config + return PVE::QemuServer::CPUConfig->parse_config( + "cpu-models.conf", + < sub { + my $hw_capabilities_raw; + if (!defined($test_env->{hw_capabilities})) { + # default to barebone uncapable HW + $hw_capabilities_raw = + '{"amd-sev":{"cbitpos":0,"reduced-phys-bits":0,"sev-support":false,' + . '"sev-support-es":false,"sev-support-snp":false}}'; + } elsif (my $cpu_hw_caps = $cpu_hw_capabilities->{ lc($test_env->{hw_capabilities}) }) { + $hw_capabilities_raw = $cpu_hw_caps; + } else { + $hw_capabilities_raw = $test_env->{hw_capabilities}; + } + + my $hw_capabilities = decode_json($hw_capabilities_raw); + return $hw_capabilities; + }, +); + +our $pve_common_network; +$pve_common_network = Test::MockModule->new('PVE::Network'); +$pve_common_network->mock( + read_bridge_mtu => sub { + my ($bridge_name) = @_; + + if ($bridge_name eq 'vxlan_bridge') { + return 1450; + } + + return 1500; + }, +); + +our $pve_common_inotify; +$pve_common_inotify = Test::MockModule->new('PVE::INotify'); +$pve_common_inotify->mock( + nodename => sub { + return 'localhost'; + }, +); + +our $pve_common_sysfstools; +$pve_common_sysfstools = Test::MockModule->new('PVE::SysFSTools'); +$pve_common_sysfstools->mock( + lspci => sub { + my ($filter, $verbose) = @_; + + return [ + map { { id => $_ } } + grep { + !defined($filter) + || (!ref($filter) && $_ =~ m/^(0000:)?\Q$filter\E/) + || (ref($filter) eq 'CODE' && $filter->({ id => $_ })) + } sort @$pci_devs + ]; + }, + pci_device_info => sub { + my ($path, $noerr) = @_; + + if ($path =~ m/^0000:01:00/) { + return { + mdev => 1, + iommugroup => 1, + mdev => 1, + vendor => "0x10de", + device => "0x2231", + }; + } elsif ($path =~ m/^0000:07:10/) { + return { + iommugroup => 2, + vendor => "0x8086", + device => "0x1520", + }; + } else { + return {}; + } + }, +); + +our $qemu_drive_module; +$qemu_drive_module = Test::MockModule->new('PVE::QemuServer::Drive'); +$qemu_drive_module->mock( + get_cdrom_path => sub { + return "/dev/cdrom"; + }, +); + +our $mapping_usb_module = Test::MockModule->new("PVE::Mapping::USB"); +$mapping_usb_module->mock( + config => sub { + return $usb_map_config; + }, +); + +our $mapping_pci_module = Test::MockModule->new("PVE::Mapping::PCI"); +$mapping_pci_module->mock( + config => sub { + return $pci_map_config; + }, +); + +our $pci_module = Test::MockModule->new("PVE::QemuServer::PCI"); +$pci_module->mock( + reserve_pci_usage => sub { + die "reserve_pci_usage should not be called for 'qm showcmd'\n"; + }, + create_nvidia_device => sub { + die "create_nvidia_device should not be called for 'qm showcmd'\n"; + }, +); + +1; diff --git a/src/test/TestsCommon/Utils.pm b/src/test/TestsCommon/Utils.pm new file mode 100644 index 00000000..732ac52f --- /dev/null +++ b/src/test/TestsCommon/Utils.pm @@ -0,0 +1,31 @@ +package TestsCommon::Utils; + +use v5.36; + +use File::Temp; + +use PVE::Tools qw(run_command); + +use base 'Exporter'; + +our @EXPORT_OK = qw( + diff +); + +sub diff($expected, $got) { + return if $expected eq $got; + + my $tmp = File::Temp->new(); + print $tmp $expected; + $tmp->flush(); + + my $diff = ''; + run_command( + ['diff', '-up', '--label', 'expected', $tmp->filename(), '--label', 'got', '-'], + input => $got, + outfunc => sub { $diff .= "$_[0]\n"; }, + noerr => 1, + ); + + return $diff; +} diff --git a/src/test/run_config2command_tests.pl b/src/test/run_config2command_tests.pl index 28f538d3..ec8595f9 100755 --- a/src/test/run_config2command_tests.pl +++ b/src/test/run_config2command_tests.pl @@ -2,184 +2,23 @@ use v5.36; -use lib qw(..); +use lib qw(.. .); -use JSON qw(decode_json); use Test::More; use Test::MockModule; -use Socket qw(AF_INET AF_INET6); use PVE::File qw(file_get_contents file_set_contents); -use PVE::Tools qw(run_command); -use PVE::INotify; -use PVE::SysFSTools; - use PVE::QemuConfig; use PVE::QemuServer; -use PVE::QemuServer::Drive; -use PVE::QemuServer::Helpers; use PVE::QemuServer::Monitor; -use PVE::QemuServer::OVMF; use PVE::QemuServer::QMPHelpers; use PVE::QemuServer::CPUConfig; use PVE::Storage; -my $base_env = { - storage_config => { - ids => { - local => { - content => { - images => 1, - iso => 1, - }, - path => '/var/lib/vz', - type => 'dir', - shared => 0, - }, - localsnapext => { - content => { - images => 1, - }, - path => '/var/lib/vzsnapext', - type => 'dir', - shared => 0, - 'snapshot-as-volume-chain' => 1, - }, - noimages => { - content => { - iso => 1, - }, - path => '/var/lib/vz', - type => 'dir', - }, - 'btrfs-store' => { - content => { - images => 1, - }, - path => '/butter/bread', - type => 'btrfs', - }, - 'cifs-store' => { - shared => 1, - path => '/mnt/pve/cifs-store', - username => 'guest', - server => '127.0.0.42', - type => 'cifs', - share => 'CIFShare', - content => { - images => 1, - iso => 1, - }, - }, - 'rbd-store' => { - monhost => '127.0.0.42,127.0.0.21,::1', - fsid => 'fc4181a6-56eb-4f68-b452-8ba1f381ca2a', - content => { - images => 1, - }, - type => 'rbd', - pool => 'cpool', - username => 'admin', - shared => 1, - }, - 'krbd-store' => { - monhost => '127.0.0.42,127.0.0.21,::1', - fsid => 'fc4181a6-56eb-4f68-b452-8ba1f381ca2a', - content => { - images => 1, - }, - type => 'rbd', - pool => 'cpool', - username => 'admin', - shared => 1, - krbd => 1, - }, - 'zfs-over-iscsi-store' => { - type => 'zfs', - iscsiprovider => "comstar", - lio_tpg => "tpg1", - portal => "127.0.0.1", - target => "iqn.2019-10.org.test:foobar", - pool => "tank", - content => { - images => 1, - }, - }, - 'lvm-store' => { - vgname => 'veegee', - type => 'lvm', - content => { - images => 1, - }, - }, - 'local-lvm' => { - vgname => 'pve', - bwlimit => 'restore=1024', - type => 'lvmthin', - thinpool => 'data', - content => { - images => 1, - }, - }, - }, - }, - vmid => 8006, - real_qemu_version => PVE::QemuServer::Helpers::kvm_user_version(), # not yet mocked -}; +use TestsCommon::CommandLineMocks qw(get_storage_config get_test_qemu_version set_test_env); +use TestsCommon::Utils qw(diff); -my $pci_devs = [ - "0000:00:02.0", - "0000:00:43.1", - "0000:00:f4.0", - "0000:00:ff.1", - "0000:0f:f2.0", - "0000:d0:13.0", - "0000:d0:15.1", - "0000:d0:15.2", - "0000:d0:17.0", - "0000:f0:42.0", - "0000:f0:43.0", - "0000:f0:43.1", - "1234:f0:43.1", - "0000:01:00.4", - "0000:01:00.5", - "0000:01:00.6", - "0000:07:10.0", - "0000:07:10.1", - "0000:07:10.4", -]; - -my $pci_map_config = { - ids => { - someGpu => { - type => 'pci', - mdev => 1, - map => [ - 'node=localhost,path=0000:01:00.4,id=10de:2231,iommugroup=1', - 'node=localhost,path=0000:01:00.5,id=10de:2231,iommugroup=1', - 'node=localhost,path=0000:01:00.6,id=10de:2231,iommugroup=1', - ], - }, - someNic => { - type => 'pci', - map => [ - 'node=localhost,path=0000:07:10.0,id=8086:1520,iommugroup=2', - 'node=localhost,path=0000:07:10.1,id=8086:1520,iommugroup=2', - 'node=localhost,path=0000:07:10.4,id=8086:1520,iommugroup=2', - ], - }, - }, -}; - -my $usb_map_config = {}, - - my $cpu_hw_capabilities = { - # Gathered from an AMD EPYC 9475F running kernel 6.11.11-2-pve - 'amd-turin-9005' => - '{ "amd-sev": { "cbitpos": 51, "reduced-phys-bits": 6, "sev-support": true,' - . ' "sev-support-es": true, "sev-support-snp": true } }', - # TODO: others? - }; +my $vmid = 8006; my $current_test; # = { # description => 'Test description', # if available @@ -212,8 +51,6 @@ sub parse_test($config_fn) { my $description = $config->{description} // ''; - $current_test->{'host-cpu-vendor'} = 'GenuineIntel'; - while ($description =~ /^\h*(.*?)\h*$/gm) { my $line = $1; next if !$line || $line =~ /^#/; @@ -244,140 +81,11 @@ sub parse_test($config_fn) { } $current_test->{testname} = $testname; - PVE::QemuServer::CPUConfig::initialize_cpu_models(); -} + set_test_env($current_test); -sub get_test_qemu_version { - $current_test->{qemu_version} // $base_env->{real_qemu_version} // '2.12'; + PVE::QemuServer::CPUConfig::initialize_cpu_models(); } -my $procfs_tools_module = Test::MockModule->new('PVE::ProcFSTools'); -$procfs_tools_module->mock( - read_cpuinfo => sub { - my $res = $procfs_tools_module->original('read_cpuinfo')->(); - my $vendor = $current_test->{'host-cpu-vendor'} - or die "internal error - host cpu vendor not set"; - $res->{vendor} = $vendor; - return $res; - }, -); - -my $qemu_server_module; -$qemu_server_module = Test::MockModule->new('PVE::QemuServer'); -$qemu_server_module->mock( - kvm_user_version => sub { - return get_test_qemu_version(); - }, - kvm_version => sub { - return get_test_qemu_version(); - }, - kernel_has_vhost_net => sub { - return 1; # TODO: make this per-test configurable? - }, - get_iscsi_initiator_name => sub { - return 'iqn.1993-08.org.debian:01:aabbccddeeff'; - }, - cleanup_pci_devices => { - # do nothing - }, -); - -my $qemu_server_ovmf_module = Test::MockModule->new("PVE::QemuServer::OVMF"); -$qemu_server_ovmf_module->mock( - file_exists => sub { - my ($path) = @_; - return 1; - }, - file_get_size => sub { - my ($path) = @_; - if ($path =~ m/OVMF(32)?_(SEV_)?VARS_4M/) { - return 528 * 1024; - } elsif ($path =~ m/OVMF_VARS/) { - return 128 * 1024; - } elsif ($path =~ m/AAVMF_VARS/) { - return 64 * 1024 * 1024; - } elsif ($path =~ m/RISCV_VIRT_VARS/) { - return 32 * 1024 * 1024; - } else { - die "unknown ovmf vars image '$path' - implement me"; - } - }, -); - -my $storage_module = Test::MockModule->new("PVE::Storage"); -$storage_module->mock( - activate_volumes => sub { - return; - }, - deactivate_volumes => sub { - return; - }, - volume_snapshot_info => sub { - my ($cfg, $volid) = @_; - - my ($storeid, $volname) = PVE::Storage::parse_volume_id($volid); - - my $snapshots = {}; - if ($storeid eq 'localsnapext') { - $snapshots = { - current => { - file => 'var/lib/vzsnapext/images/8006/vm-8006-disk-0.qcow2', - parent => 'snap2', - }, - snap2 => { - file => '/var/lib/vzsnapext/images/8006/snap2-vm-8006-disk-0.qcow2', - parent => 'snap1', - }, - snap1 => { - file => '/var/lib/vzsnapext/images/8006/snap1-vm-8006-disk-0.qcow2', - }, - }; - } elsif ($storeid eq 'lvm-store') { - $snapshots = { - current => { - file => '/dev/veegee/vm-8006-disk-0.qcow2', - parent => 'snap2', - }, - snap2 => { - file => '/dev/veegee/snap2-vm-8006-disk-0.qcow2', - parent => 'snap1', - }, - snap1 => { - file => '/dev/veegee/snap1-vm-8006-disk-0.qcow2', - }, - }; - } - return $snapshots; - }, -); - -my $file_stat_module = Test::MockModule->new("File::stat"); -$file_stat_module->mock( - stat => sub { - my ($path) = @_; - my $st = $file_stat_module->original('stat')->('./run_config2command_tests.pl'); - $st->[2] = 25008 if $path =~ m!/dev/!; # block device - return $st; - }, -); - -my $zfsplugin_module = Test::MockModule->new("PVE::Storage::ZFSPlugin"); -$zfsplugin_module->mock( - zfs_get_lu_name => sub { - return "foobar"; - }, - zfs_get_lun_number => sub { - return "0"; - }, -); - -my $rbdplugin_module = Test::MockModule->new("PVE::Storage::RBDPlugin"); -$rbdplugin_module->mock( - rbd_volume_config_set => sub { - return; - }, -); - my $qemu_server_config; $qemu_server_config = Test::MockModule->new('PVE::QemuConfig'); $qemu_server_config->mock( @@ -388,185 +96,14 @@ $qemu_server_config->mock( }, ); -my $qemu_server_helpers; -$qemu_server_helpers = Test::MockModule->new('PVE::QemuServer::Helpers'); -$qemu_server_helpers->mock( - get_host_phys_address_bits => sub { - return 46; - }, -); - -my $qemu_server_memory; -$qemu_server_memory = Test::MockModule->new('PVE::QemuServer::Memory'); -$qemu_server_memory->mock( - hugepages_chunk_size_supported => sub { - return 1; - }, - host_numanode_exists => sub { - my ($id) = @_; - return 1; - }, -); - -my $pve_common_tools; -$pve_common_tools = Test::MockModule->new('PVE::Tools'); -$pve_common_tools->mock( - next_vnc_port => sub { - my ($family, $address) = @_; - - return '5900'; - }, - next_spice_port => sub { - my ($family, $address) = @_; - - return '61000'; - }, - getaddrinfo_all => sub { - my ($hostname, @opts) = @_; - die "need stable hostname" if $hostname ne 'localhost'; - return ( - { - addr => Socket::pack_sockaddr_in(0, Socket::INADDR_LOOPBACK), - family => AF_INET, # IPv4 - protocol => 6, - socktype => 1, - }, - ); - }, - get_host_arch => sub { - return $current_test->{host_arch} // 'x86_64'; - }, -); - -my $pve_common_file; -$pve_common_file = Test::MockModule->new('PVE::File'); -$pve_common_file->mock( - file_copy => sub { - my ($filename, $dst, $max, $perm) = @_; - if ($dst =~ m|^/run/qemu-server/efidisk|) { - return; - } - return $pve_common_file->original('file_copy')->($filename, $dst, $max, $perm); - }, -); - -my $pve_cpuconfig; -$pve_cpuconfig = Test::MockModule->new('PVE::QemuServer::CPUConfig'); -$pve_cpuconfig->mock( - load_custom_cpu_model_config => sub { - # mock custom CPU model config - return PVE::QemuServer::CPUConfig->parse_config( - "cpu-models.conf", - < sub { - my $hw_capabilities_raw; - if (!defined($current_test->{hw_capabilities})) { - # default to barebone uncapable HW - $hw_capabilities_raw = - '{"amd-sev":{"cbitpos":0,"reduced-phys-bits":0,"sev-support":false,' - . '"sev-support-es":false,"sev-support-snp":false}}'; - } elsif ( - my $cpu_hw_caps = $cpu_hw_capabilities->{ lc($current_test->{hw_capabilities}) } - ) { - $hw_capabilities_raw = $cpu_hw_caps; - } else { - $hw_capabilities_raw = $current_test->{hw_capabilities}; - } - - my $hw_capabilities = decode_json($hw_capabilities_raw); - return $hw_capabilities; - }, -); - -my $pve_common_network; -$pve_common_network = Test::MockModule->new('PVE::Network'); -$pve_common_network->mock( - read_bridge_mtu => sub { - my ($bridge_name) = @_; - - if ($bridge_name eq 'vxlan_bridge') { - return 1450; - } - - return 1500; - }, -); - -my $pve_common_inotify; -$pve_common_inotify = Test::MockModule->new('PVE::INotify'); -$pve_common_inotify->mock( - nodename => sub { - return 'localhost'; - }, -); - -my $pve_common_sysfstools; -$pve_common_sysfstools = Test::MockModule->new('PVE::SysFSTools'); -$pve_common_sysfstools->mock( - lspci => sub { - my ($filter, $verbose) = @_; - - return [ - map { { id => $_ } } - grep { - !defined($filter) - || (!ref($filter) && $_ =~ m/^(0000:)?\Q$filter\E/) - || (ref($filter) eq 'CODE' && $filter->({ id => $_ })) - } sort @$pci_devs - ]; - }, - pci_device_info => sub { - my ($path, $noerr) = @_; - - if ($path =~ m/^0000:01:00/) { - return { - mdev => 1, - iommugroup => 1, - mdev => 1, - vendor => "0x10de", - device => "0x2231", - }; - } elsif ($path =~ m/^0000:07:10/) { - return { - iommugroup => 2, - vendor => "0x8086", - device => "0x1520", - }; - } else { - return {}; - } - }, -); - -my $qemu_drive_module; -$qemu_drive_module = Test::MockModule->new('PVE::QemuServer::Drive'); -$qemu_drive_module->mock( - get_cdrom_path => sub { - return "/dev/cdrom"; - }, -); - my $qemu_monitor_module; $qemu_monitor_module = Test::MockModule->new('PVE::QemuServer::Monitor'); $qemu_monitor_module->mock( mon_cmd => sub { - my ($vmid, $cmd) = @_; + my ($id, $cmd) = @_; - die "invalid vmid: $vmid (expected: $base_env->{vmid})" - if $vmid != $base_env->{vmid}; + die "invalid vmid: $id (expected: $vmid)" + if $id != $vmid; if ($cmd eq 'query-version') { my $ver = get_test_qemu_version(); @@ -584,61 +121,6 @@ $qemu_monitor_module->mock( }, ); -my $mapping_usb_module = Test::MockModule->new("PVE::Mapping::USB"); -$mapping_usb_module->mock( - config => sub { - return $usb_map_config; - }, -); - -my $mapping_pci_module = Test::MockModule->new("PVE::Mapping::PCI"); -$mapping_pci_module->mock( - config => sub { - return $pci_map_config; - }, -); - -my $pci_module = Test::MockModule->new("PVE::QemuServer::PCI"); -$pci_module->mock( - reserve_pci_usage => sub { - die "reserve_pci_usage should not be called for 'qm showcmd'\n"; - }, - create_nvidia_device => sub { - die "create_nvidia_device should not be called for 'qm showcmd'\n"; - }, -); - -sub diff($a, $b) { - return if $a eq $b; - - my ($ra, $wa) = POSIX::pipe(); - my ($rb, $wb) = POSIX::pipe(); - my $ha = IO::Handle->new_from_fd($wa, 'w'); - my $hb = IO::Handle->new_from_fd($wb, 'w'); - - open my $diffproc, '-|', 'diff', '-up', "/proc/self/fd/$ra", "/proc/self/fd/$rb" ## no critic - or die "failed to run program 'diff': $!"; - POSIX::close($ra); - POSIX::close($rb); - - open my $f1, '<', \$a; - open my $f2, '<', \$b; - my ($line1, $line2); - do { - $ha->print($line1) if defined($line1 = <$f1>); - $hb->print($line2) if defined($line2 = <$f2>); - } while (defined($line1 // $line2)); - close $f1; - close $f2; - close $ha; - close $hb; - - local $/ = undef; - my $diff = <$diffproc>; - close $diffproc; - die "files differ:\n$diff"; -} - $SIG{__WARN__} = sub { my $warning = shift; chomp $warning; @@ -663,7 +145,7 @@ sub do_test($config_fn) { my $testname = $current_test->{testname}; - my ($vmid, $storecfg) = $base_env->@{qw(vmid storage_config)}; + my $storecfg = get_storage_config(); my $cmdline = eval { PVE::QemuServer::vm_commandline($storecfg, $vmid) }; my $err = $@; @@ -699,26 +181,23 @@ sub do_test($config_fn) { my $cmd_fn = "$config_fn.cmd"; - if (-f $cmd_fn) { + if (!-f $cmd_fn) { + file_set_contents($cmd_fn, $cmdline); + } else { my $cmdline_expected = file_get_contents($cmd_fn); my $cmd_expected = [split /\s*\\?\n\s*/, $cmdline_expected]; my $cmd = [split /\s*\\?\n\s*/, $cmdline]; - # uncomment for easier debugging - #file_set_contents("$cmd_fn.tmp", $cmdline); - my $exp = join("\n", @$cmd_expected); my $got = join("\n", @$cmd); - eval { diff($exp, $got) }; - if (my $err = $@) { + + if (my $diff = diff($exp, $got)) { fail("$testname"); - note($err); + note("files differ:\n$diff"); } else { pass("$testname"); } - } else { - file_set_contents($cmd_fn, $cmdline); } } diff --git a/src/test/run_hotplug_tests.pl b/src/test/run_hotplug_tests.pl new file mode 100644 index 00000000..df93cb9f --- /dev/null +++ b/src/test/run_hotplug_tests.pl @@ -0,0 +1,833 @@ +#!/usr/bin/perl + +# Regression tests for applying pending changes to a running VM via hotplug. +# +# Each test is a VM config in 'hotplug/*.conf' with the changes to apply in a [PENDING] section. The +# monitor of the VM is mocked with a minimal model of a running QEMU instance, that tracks the +# present devices, buses, block nodes, backends and QOM objects and enforces the constraints between +# them. Every monitor command and every host-side action (network, SDN, storage, cgroup) that is +# issued while applying the pending changes is recorded. Afterwards, the model is compared to the +# model of a VM freshly started with the resulting config, which is what the target of a live +# migration is, to detect deviations like differing PCI addresses or left-over devices. The +# recording are compared to the corresponding 'hotplug/*.conf.expected' file. If that file does not +# exist yet, it is generated and needs to be manually verified. Tests for an aarch64 host are in +# 'hotplug/aarch64/'. + +use v5.36; + +use lib qw(.. .); + +# hmp does not return an error for us, just strings so we use verify afterwards to check if +# it worked. The helpers then wait between retries when verifying (un)plugged devices. Overwrite +# `sleep` behavior so tests do not wait around. +BEGIN { + *CORE::GLOBAL::sleep = sub { return 0; }; +} + +use JSON; +use Storable qw(dclone); +use Test::More; +use Test::MockModule; + +use PVE::File qw(file_get_contents file_set_contents); + +use PVE::QemuConfig; +use PVE::QemuServer; +use PVE::QemuServer::Blockdev; +use PVE::QemuServer::CGroup; +use PVE::QemuServer::CPUConfig; +use PVE::QemuServer::Drive; +use PVE::QemuServer::DriveDevice; +use PVE::QemuServer::Helpers; +use PVE::QemuServer::Machine; +use PVE::QemuServer::Monitor; +use PVE::QemuServer::Network; +use PVE::Storage; + +# the mocks for generating the command line of a VM, shared with the cfg2cmd tests +use TestsCommon::CommandLineMocks qw(get_storage_config get_test_qemu_version set_test_env); +use TestsCommon::Utils qw(diff); + +my $vmid = 8006; +my $fake_config_fn = "hotplug/qemu-server/$vmid.conf"; + +my $storecfg = get_storage_config(); + +my $current_test; # = { +# testname => 'file name and description', +# description => 'Test description', # if available +# qemu_version => '10.1', +# host_arch => 'x86_64', +# fail_device_add => { deviceid => 1 }, +# fail_device_del => { deviceid => 1 }, +# fail_command => { command => 1 }, +# config => { config hash }, +# }; + +# The model of the running QEMU instance, set up from the command line that config_to_command() +# generates for the config. The number of online vCPUs and the present memory DIMMs are derived +# from the 'cpuN' and 'dimmN' devices. Buses are derived from the machine type and the present +# bridges and controllers. +my $vm_state; # = { +# machine => 'pc-i440fx-10.1+pve0', # the machine type of the running VM +# buses => { bus => 1 }, # buses provided by the machine itself +# devices => { qdev_id => { device options referencing buses and backends } }, +# hotplugged => { qdev_id => 1 }, # devices that were added during the test +# netdevs => { netdev_id => 1 }, +# chardevs => { chardev_id => 1 }, +# drives => { drive_id => 1 }, # legacy '-drive' backends (machine version < 10.0) +# blocknodes => { node_name => 1 }, # explicitly added block nodes (machine version >= 10.0) +# objects => { object_id => { arguments of object-add } }, # QOM objects +# }; + +my $recorded_actions = []; + +# use the config description to describe the test and the state of the running VM, fields are: +# TEST: A single line describing the test, gets outputted +# QEMU_VERSION: \d+\.\d+(\.\d+)? version of the running QEMU binary (defaults to current version) +# HOST_ARCH: x86_64 | aarch64 (default to x86_64, to make tests stable) +# FAIL_DEVICE_ADD: QEMU accepts 'device_add' for this device, but it never shows up +# FAIL_DEVICE_DEL: QEMU accepts 'device_del' for this device, but it never goes away +# FAIL_COMMAND: the QMP or HMP command with this name fails +# EXPECTED_ERROR: the expected error message(s) +# STAYS_PENDING: bool +# all fields are optional, FAIL_* and EXPECTED_ERROR comments can be specified multiple times +sub parse_test($config_fn) { + $current_test = { + fail_device_add => {}, + fail_device_del => {}, + fail_command => {}, + expected_errors => [], + }; + + my $config_raw = file_get_contents($config_fn); + my $config = PVE::QemuServer::parse_vm_config($fake_config_fn, $config_raw); + + $current_test->{config} = $config; + + my $description = $config->{description} // ''; + + while ($description =~ /^\h*(.*?)\h*$/gm) { + my $line = $1; + next if !$line || $line =~ /^#/; + + if ($line =~ /^TEST:\s*(.*)\s*$/) { + $current_test->{description} = "$1"; + } elsif ($line =~ /^QEMU_VERSION:\s*(.*)\s*$/) { + $current_test->{qemu_version} = "$1"; + } elsif ($line =~ /^HOST_ARCH:\s*(.*)\s*$/) { + $current_test->{host_arch} = "$1"; + } elsif ($line =~ /^FAIL_DEVICE_ADD:\s*(\S+)\s*$/) { + $current_test->{fail_device_add}->{$1} = 1; + } elsif ($line =~ /^FAIL_DEVICE_DEL:\s*(\S+)\s*$/) { + $current_test->{fail_device_del}->{$1} = 1; + } elsif ($line =~ /^FAIL_COMMAND:\s*(\S+)\s*$/) { + $current_test->{fail_command}->{$1} = 1; + } elsif ($line =~ /^EXPECTED_ERROR:\s*(.*)$/) { + push $current_test->{expected_errors}->@*, $1; + } elsif ($line =~ /^STAYS_PENDING:\s*(.*)$/) { + $current_test->{stays_pending} = $1; + } else { + die "invalid test configuration\n"; + } + } + + # the description only describes the test and is not part of the resulting config + delete $config->{description}; + + $config_fn =~ /([^\/]+)$/; + my $testname = "$1"; + if (my $desc = $current_test->{description}) { + $testname = "'$testname' - $desc"; + } + $current_test->{testname} = $testname; + + set_test_env($current_test); + + # the 'host' CPU model is registered for the host architecture of the current test + PVE::QemuServer::CPUConfig::initialize_cpu_models(); +} + +# The buses a machine provides on its own. Everything else (PCI bridges, SCSI and USB controllers) +# comes from devices on the command line or from the config files read via -readconfig. +sub default_buses_for_machine($conf, $machine) { + return { map { $_ => 1 } qw(pcie.0) } if $machine =~ m/^virt/; + return { map { $_ => 1 } ('pcie.0', map { "ide.$_" } 0 .. 5) } + if PVE::QemuServer::Machine::machine_type_is_q35($conf); + return { map { $_ => 1 } qw(pci.0 ide.0 ide.1) }; +} + +# Parse a comma separated option string like 'driver,id=x,bus=y' into a hash. A leading value +# without key is stored under $first_key. +sub parse_options($string, $first_key = undef) { + my $options = {}; + for my $part (split(/,/, $string)) { + if ($part =~ m/^([^=]+)=(.*)$/) { + $options->{$1} = $2; + } elsif (defined($first_key)) { + $options->{$first_key} = $part; + } + } + return $options; +} + +# Parse the devices from a QEMU config file (-readconfig), which is shipped in 'usr/' of the repo. +sub parse_readconfig($path) { + my ($name) = $path =~ m|([^/]+)$|; + my $devices = {}; + my $current; + for my $line (split(/\n/, file_get_contents("../usr/$name"))) { + if ($line =~ m/^\[device "([^"]+)"\]/) { + $current = $devices->{$1} = { id => $1 }; + } elsif ($line =~ m/^\[/) { + $current = undef; + } elsif ($current && $line =~ m/^\s*(\S+)\s*=\s*"([^"]*)"/) { + $current->{$1} = $2; + } + } + return $devices; +} + +# Resolve the full final machine version, even for values like 'pc' or 'q35'. +sub resolve_machine_alias($machine) { + my ($type, $pve_version) = split(/\+/, $machine, 2); + if ($type && $type !~ m/^(?:pc|q35|virt)$/) { + return $machine; + } + + my $kvm_version = get_test_qemu_version(); + my $new_machine = + PVE::QemuServer::Machine::windows_get_pinned_machine_version($type, undef, $kvm_version); + + if ($new_machine !~ m/\+pve\d+$/) { + $new_machine = $new_machine . ($pve_version ? "+$pve_version" : "+pve0"); + } + return $new_machine; +} + +# Returns the model of a VM freshly started with the given config, derived from the command line +# that config_to_command() generates for it. +sub vm_state_from_config($conf) { + my $cmd = PVE::QemuServer::config_to_command( + $storecfg, $vmid, dclone($conf), PVE::QemuServer::load_defaults(), {}, + ); + + my $machine = PVE::QemuServer::Machine::get_vm_machine($conf); + + my $state = { + machine => resolve_machine_alias($machine), + devices => {}, + hotplugged => {}, + netdevs => {}, + chardevs => {}, + drives => {}, + blocknodes => {}, + objects => {}, + }; + + for (my $i = 0; $i < scalar($cmd->@*) - 1; $i++) { + my ($opt, $arg) = $cmd->@[$i, $i + 1]; + if ($opt eq '-machine') { + $state->{machine} = resolve_machine_alias(parse_options($arg)->{type}); + } elsif ($opt eq '-device') { + my $device = parse_options($arg, 'driver'); + $state->{devices}->{ $device->{id} } = $device if $device->{id}; + } elsif ($opt eq '-readconfig') { + my $devices = parse_readconfig($arg); + $state->{devices}->{$_} = $devices->{$_} for keys $devices->%*; + } elsif ($opt eq '-netdev') { + $state->{netdevs}->{ parse_options($arg)->{id} } = 1; + } elsif ($opt eq '-chardev') { + $state->{chardevs}->{ parse_options($arg, 'backend')->{id} } = 1; + } elsif ($opt eq '-drive') { + $state->{drives}->{ parse_options($arg)->{id} } = 1; + } elsif ($opt eq '-blockdev') { + $state->{blocknodes}->{ decode_json($arg)->{'node-name'} } = 1; + } elsif ($opt eq '-object') { + my $object = $arg =~ m/^\{/ ? decode_json($arg) : parse_options($arg, 'qom-type'); + # sizes on the command line have a unit, QMP uses bytes + if (defined($object->{size}) && $object->{size} =~ m/^(\d+)([MG])$/) { + $object->{size} = $1 * 1024 * 1024 * ($2 eq 'G' ? 1024 : 1); + } + $state->{objects}->{ $object->{id} } = $object; + } + } + + $state->{buses} = default_buses_for_machine($conf, $state->{machine}); + + return $state; +} + +sub setup_vm_state($conf) { + $vm_state = vm_state_from_config($conf); +} + +# Throttle limits set via QMP contain all properties, while the throttle group generated for the +# command line only contains the configured ones. Drop the defaults to make them comparable. +sub normalized_object($object) { + $object = dclone($object); + if (($object->{'qom-type'} // '') eq 'throttle-group' && $object->{limits}) { + my $limits = $object->{limits}; + my $is_default = sub { $limits->{ $_[0] } == ($_[0] =~ m/-max-length$/ ? 1 : 0) }; + $object->{limits} = + { map { $_ => $limits->{$_} } grep { !$is_default->($_) } keys %$limits }; + } + return to_json($object, { canonical => 1 }); +} + +# Compare the model of the running VM with the model of a VM freshly started with the resulting +# config, like the target of a live migration is. Returns a list of the differences. The boot index +# is not compared, as hotplugged devices are added without one. +sub compare_with_fresh_vm($conf) { + my $running = $vm_state; + my $fresh = vm_state_from_config($conf); + my $differences = []; + + my $compare_ids = sub { + my ($kind, $running_ids, $fresh_ids) = @_; + my $all_keys = [keys $running_ids->%*, keys $fresh_ids->%*]; + for my $key ($all_keys->@*) { + push $differences->@*, "$kind: $key: only present in running VM" + if !$running_ids->{$key}; + push $differences->@*, "$kind: $key: only present in freshly started VM" + if !$fresh_ids->{$key}; + } + }; + + push $differences->@*, "machine: running '$running->{machine}' vs fresh '$fresh->{machine}'" + if $running->{machine} ne $fresh->{machine}; + + $compare_ids->('device', $running->{devices}, $fresh->{devices}); + for my $id (sort grep { $fresh->{devices}->{$_} } keys $running->{devices}->%*) { + my $running_device = $running->{devices}->{$id}; + my $fresh_device = $fresh->{devices}->{$id}; + my %options = map { $_ => 1 } keys $running_device->%*, keys $fresh_device->%*; + delete $options{bootindex}; + for my $option (sort keys %options) { + my $running_value = $running_device->{$option} // ''; + my $fresh_value = $fresh_device->{$option} // ''; + push $differences->@*, + "device $id option $option: running '$running_value' vs fresh '$fresh_value'" + if "$running_value" ne "$fresh_value"; + } + } + + $compare_ids->('object', $running->{objects}, $fresh->{objects}); + for my $id (sort grep { $fresh->{objects}->{$_} } keys $running->{objects}->%*) { + my $running_object = normalized_object($running->{objects}->{$id}); + my $fresh_object = normalized_object($fresh->{objects}->{$id}); + push $differences->@*, "object $id: running $running_object vs fresh $fresh_object" + if $running_object ne $fresh_object; + } + + for my $kind (qw(netdev chardev drive blocknode)) { + $compare_ids->($kind, $running->{"${kind}s"}, $fresh->{"${kind}s"}); + } + + return $differences; +} + +# Record an action with its arguments. HMP commands are recorded as the full command line. Arguments +# of QMP commands are recorded as they are. +sub record_action($layer, $action, @args) { + my $line = "$layer $action"; + $line .= ' ' . to_json([@args], { canonical => 1 }) if scalar(@args); + push $recorded_actions->@*, $line; +} + +sub record_qmp_action($action, $args) { + my $line = "qmp $action"; + $line .= ' ' . to_json($args, { canonical => 1 }); + push $recorded_actions->@*, $line; +} + +sub sorted_devices($regex = qr/./) { + my @list = sort grep { $_ =~ $regex } keys $vm_state->{devices}->%*; + return @list; +} + +sub bus_exists_in_vm_state($bus) { + return 1 if $vm_state->{buses}->{$bus}; + return 1 if $bus =~ m/^pci\.\d+$/ && $vm_state->{devices}->{$bus}; # PCI bridge + return 1 if $bus =~ m/^(.+)\.0$/ && $vm_state->{devices}->{$1}; # SCSI and USB controllers + return 0; +} + +# Check the constraints for adding the device with the given options. Returns an error message if +# the device cannot be added, otherwise adds it and returns nothing. +sub add_device($options) { + my $id = $options->{id} or die "device_add without ID\n"; + my $driver = $options->{driver}; + + return "Duplicate device ID '$id'" if $vm_state->{devices}->{$id}; + return "simulated failure adding device '$id'" if $current_test->{fail_device_add}->{$id}; + + if (my $bus = $options->{bus}) { + return "Bus '$bus' does not support hotplugging" if $bus =~ m/^(ide\.\d+|ahci\d+\.\d+)$/; + # devices cannot be hotplugged into a PCI bridge that was itself hotplugged, as the guest + # firmware did not assign any resources to it + return "Bus '$bus' does not support hotplugging" if $vm_state->{hotplugged}->{$bus}; + return "Bus '$bus' not found" if !bus_exists_in_vm_state($bus); + } + + my $backends = { + netdev => $vm_state->{netdevs}, + chardev => $vm_state->{chardevs}, + drive => { $vm_state->{drives}->%*, $vm_state->{blocknodes}->%* }, + iothread => $vm_state->{objects}, + memdev => $vm_state->{objects}, + }; + for my $option (sort keys $backends->%*) { + my $value = $options->{$option} or next; + return "Property '$driver.$option' can't find value '$value'" + if !$backends->{$option}->{$value}; + } + + $vm_state->{devices}->{$id} = $options; + $vm_state->{hotplugged}->{$id} = 1; + return; +} + +# Handle QMP commands querying the state of the VM. Returns nothing for other commands. +sub mocked_qmp_query($execute, $arguments) { + if ($execute eq 'query-version') { + my ($major, $minor, $micro) = get_test_qemu_version() =~ m/^(\d+)\.(\d+)(?:\.(\d+))?/; + return { qemu => { major => $major, minor => $minor, micro => $micro // 0 } }; + } elsif ($execute eq 'query-machines') { + my ($name, $pve_version) = split(/\+/, $vm_state->{machine}, 2); + my $machine = { name => $name, 'is-current' => JSON::true }; + $machine->{'pve-version'} = $pve_version if $pve_version; + return [$machine]; + } elsif ($execute eq 'query-pci') { + # everything that is not a disk or memory DIMM is reported as a PCI device with the qdev ID + my @devices = + map { { qdev_id => $_ } } + grep { !PVE::QemuServer::Drive::is_valid_drivename($_) && $_ !~ m/^dimm\d+$/ } + sorted_devices(); + return [{ bus => 0, devices => \@devices }]; + } elsif ($execute eq 'query-block') { + my @blocks = (); + for my $id (grep { PVE::QemuServer::Drive::is_valid_drivename($_) } sorted_devices()) { + my $qdev = $id =~ m/^virtio\d+$/ ? "/machine/peripheral/$id/virtio-backend" : $id; + push @blocks, { device => '', qdev => $qdev }; + } + return \@blocks; + } elsif ($execute eq 'query-mice') { + return [{ name => 'QEMU HID Tablet' }] if $vm_state->{devices}->{tablet}; + return [{ name => 'QEMU PS/2 Mouse' }]; + } elsif ($execute eq 'qom-list') { + die "unexpected qom-list path '$arguments->{path}'\n" + if $arguments->{path} ne '/machine/peripheral'; + return [map { { name => $_ } } sorted_devices()]; + } elsif ($execute eq 'query-iothreads') { + my $objects = $vm_state->{objects}; + return [ + map { { id => $_, 'thread-id' => 1 } } + sort grep { $objects->{$_}->{'qom-type'} eq 'iothread' } keys $objects->%* + ]; + } elsif ($execute eq 'query-cpus-fast') { + # the first vCPU is always present, the others are 'cpuN' devices + my @hotplugged_cpus = sorted_devices(qr/^cpu\d+$/); + return [map { { 'cpu-index' => $_ } } (0 .. scalar(@hotplugged_cpus))]; + } elsif ($execute eq 'query-memory-devices') { + my $objects = $vm_state->{objects}; + return [ + map { { + type => 'dimm', + data => { id => $_, size => $objects->{"mem-$_"}->{size} }, + } } sorted_devices(qr/^dimm\d+$/) + ]; + } elsif ($execute eq 'query-named-block-nodes') { + return [ + map { { 'node-name' => $_, children => [] } } + sort keys $vm_state->{blocknodes}->%* + ]; + } + + return; +} + +# Handle commands issued via the human monitor. Errors are reported as text output like QEMU does. +sub mocked_hmp_cmd($cmdline) { + record_action('hmp', $cmdline); + + my ($command, $args) = $cmdline =~ m/^(\S+)\s+(.*)$/; + return "Error: simulated failure of HMP command '$command'" + if $current_test->{fail_command}->{$command}; + + if ($command eq 'device_add') { + my $options = { map { split(/=/, $_, 2) } split(/,/, $args) }; + return "Error: Parameter 'driver' is missing" if !$options->{driver}; + my $err = add_device($options); + return $err ? "Error: $err" : ''; + } elsif ($command eq 'device_del') { + my $id = $args; + return "Error: Device '$id' not found" if !$vm_state->{devices}->{$id}; + # the guest never releases the device + return '' if $current_test->{fail_device_del}->{$id}; + my $device = delete $vm_state->{devices}->{$id}; + # usb-redir deletes its chardev when it is unrealized, see usbredir_unrealize() in QEMU + delete $vm_state->{chardevs}->{ $device->{chardev} } + if ($device->{driver} // '') eq 'usb-redir' && $device->{chardev}; + return ''; + } elsif ($command eq 'drive_add') { + my ($id) = $args =~ m/,id=([^,"]+)/ or die "drive_add without ID: $cmdline\n"; + return "Error: Duplicate ID '$id' for drive" if $vm_state->{drives}->{$id}; + $vm_state->{drives}->{$id} = 1; + return 'OK'; + } elsif ($command eq 'drive_del') { + my $id = $args; + return "Error: Device '$id' not found" if !delete $vm_state->{drives}->{$id}; + return ''; + } + + die "unexpected HMP command: '$cmdline'\n"; +} + +my $qmp_command_does_not_change_state = { + balloon => 1, + block_set_io_throttle => 1, + 'blockdev-change-medium' => 1, + 'blockdev-close-tray' => 1, + 'blockdev-open-tray' => 1, + eject => 1, +}; + +sub assert_backend_not_in_use($type, $id) { + my $devices = $vm_state->{devices}; + for my $key (keys $devices->%*) { + if (($devices->{$key}->{$type} // '') eq $id) { + die "$type $id is in use by device $key"; + } + } +} + +# Replacement for PVE::QemuServer::Monitor::qmp_cmd, handling all monitor communication with the VM. +my sub qmp_command($execute, %arguments) { + die "simulated failure of QMP command '$execute'" + if $current_test->{fail_command}->{$execute}; + + return {} if $qmp_command_does_not_change_state->{$execute}; + + my $id = $arguments{id}; + my $node_name = $arguments{'node-name'}; + my $devices = $vm_state->{devices}; + + if ($execute eq 'device_add') { + my $err = add_device(\%arguments); + die $err if $err; + } elsif ($execute eq 'netdev_add') { + die "Duplicate ID '$id' for netdev" if $vm_state->{netdevs}->{$id}; + $vm_state->{netdevs}->{$id} = 1; + } elsif ($execute eq 'netdev_del') { + die "Device '$id' not found" if !$vm_state->{netdevs}->{$id}; + assert_backend_not_in_use('netdev', $id); + delete $vm_state->{netdevs}->{$id}; + } elsif ($execute eq 'set_link') { + my $name = $arguments{name}; + die "Device '$name' not found" + if !$devices->{$name} && !$vm_state->{netdevs}->{$name}; + } elsif ($execute eq 'chardev-add') { + die "Duplicate ID '$id' for chardev" if $vm_state->{chardevs}->{$id}; + $vm_state->{chardevs}->{$id} = 1; + } elsif ($execute eq 'object-add') { + die "Duplicate object ID '$id'" if $vm_state->{objects}->{$id}; + $vm_state->{objects}->{$id} = \%arguments; + } elsif ($execute eq 'object-del') { + die "Object '$id' not found" if !$vm_state->{objects}->{$id}; + assert_backend_not_in_use('iothread', $id); + assert_backend_not_in_use('memory', $id); + # the top block node of a drive uses the throttle group with the same name + if (my ($node) = $id =~ m/^throttle-(drive-.+)$/) { + die "Object '$id' is in use by node '$node'" + if $vm_state->{blocknodes}->{$node}; + } + delete $vm_state->{objects}->{$id}; + } elsif ($execute eq 'qom-set') { + my $object = $vm_state->{objects}->{ $arguments{path} }; + die "Object '$arguments{path}' not found" if !$object; + $object->{ $arguments{property} } = $arguments{value}; + } elsif ($execute eq 'blockdev-add') { + die "Duplicate nodes with node-name='$node_name'" + if $vm_state->{blocknodes}->{$node_name}; + $vm_state->{blocknodes}->{$node_name} = 1; + } elsif ($execute eq 'blockdev-del') { + die "Failed to find node with node-name='$node_name'" + if !$vm_state->{blocknodes}->{$node_name}; + assert_backend_not_in_use('drive', $node_name); + delete $vm_state->{blocknodes}->{$node_name}; + } elsif ($execute eq 'blockdev-remove-medium') { + die "Device '$id' not found" if !$devices->{$id}; + delete $devices->{$id}->{drive}; + } elsif ($execute eq 'blockdev-insert-medium') { + die "Device '$id' not found" if !$devices->{$id}; + die "Node '$node_name' not found" if !$vm_state->{blocknodes}->{$node_name}; + $devices->{$id}->{drive} = $node_name; + } else { + die "unexpected QMP command: '$execute'\n"; + } + + return {}; +} + +# wraps qmp_command and decides if errors should be bubbled or returned as string +sub mocked_qmp_cmd($peer, $execute, %arguments) { + + die "unexpected QMP peer '$peer->{name}' of type '$peer->{type}'\n" + if $peer->{type} ne 'qmp' || $peer->{id} != $vmid; + + delete $arguments{timeout}; + my $noerr = delete $arguments{noerr}; + + return mocked_hmp_cmd($arguments{'command-line'}) if $execute eq 'human-monitor-command'; + + # queries do not change the state of the VM and are not recorded + if (my $result = mocked_qmp_query($execute, \%arguments)) { + return $result; + } + + record_qmp_action($execute, \%arguments); + + my $res = eval { qmp_command($execute, %arguments) }; + if (my $err = $@) { + return { error => $err } if $noerr; + die $err; + } + + return $res; +} + +my $monitor_module = Test::MockModule->new('PVE::QemuServer::Monitor'); +$monitor_module->mock(qmp_cmd => \&mocked_qmp_cmd); + +# qmp_cmd is imported by these modules, so the imported copies need to be replaced too +my $qemu_server_module = Test::MockModule->new('PVE::QemuServer'); +$qemu_server_module->mock(qmp_cmd => \&mocked_qmp_cmd); + +my $blockdev_module = Test::MockModule->new('PVE::QemuServer::Blockdev'); +$blockdev_module->mock(qmp_cmd => \&mocked_qmp_cmd); + +# the machine type of the running VM is derived from the QEMU version, so it has to be the mocked +# one everywhere, not only for the imported copy in PVE::QemuServer +my $qemu_server_helpers = Test::MockModule->new('PVE::QemuServer::Helpers'); +$qemu_server_helpers->mock( + kvm_user_version => \&get_test_qemu_version, + vm_running_locally => sub { + return 1; + }, +); + +my $drive_device_module = Test::MockModule->new('PVE::QemuServer::DriveDevice'); +$drive_device_module->mock(kvm_user_version => \&get_test_qemu_version); + +my $qemu_server_config = Test::MockModule->new('PVE::QemuConfig'); +$qemu_server_config->mock( + write_config => sub { + my ($class, $vmid, $conf) = @_; + return; + }, + load_config => sub { + die "config was reloaded unexpectedly"; + }, +); + +my $pve_cluster_module = Test::MockModule->new('PVE::Cluster'); +$pve_cluster_module->mock( + cfs_read_file => sub { + my ($file) = @_; + + if ($file eq 'datacenter.cfg') { + return {}; + } else { + die "'cfs_read_file' called for '$file' - missing mock?\n"; + } + }, +); + +# returns a mock that records the call with all its arguments as an action of the given layer +my sub recorder($layer, $action) { + return sub { + record_action($layer, $action, @_); + return; + }; +} + +my $pve_common_network = Test::MockModule->new('PVE::Network'); +$pve_common_network->mock( + tap_unplug => recorder('net', 'tap_unplug'), + tap_rate_limit => recorder('net', 'tap_rate_limit'), +); + +my $qemu_network_module = Test::MockModule->new('PVE::QemuServer::Network'); +$qemu_network_module->mock(tap_plug => recorder('net', 'tap_plug')); + +my $sdn_vnets_module = Test::MockModule->new('PVE::Network::SDN::Vnets'); +$sdn_vnets_module->mock( + add_next_free_cidr => recorder('sdn', 'add_next_free_cidr'), + add_dhcp_mapping => recorder('sdn', 'add_dhcp_mapping'), + del_ips_from_mac => recorder('sdn', 'del_ips_from_mac'), +); + +# the shared mocks turn these into no-ops, but here they are actions worth recording +my $storage_module = Test::MockModule->new('PVE::Storage'); +$storage_module->mock( + activate_volumes => sub { + my ($cfg, $vollist) = @_; + record_action('storage', 'activate_volumes', $vollist->@*); + return; + }, + vdisk_free => sub { + my ($cfg, $volid) = @_; + record_action('storage', 'vdisk_free', $volid); + return; + }, +); + +# detaching a disk checks permissions via the RPC environment, which is not initialized in tests +my $rpcenv_module = Test::MockModule->new('PVE::RPCEnvironment'); +$rpcenv_module->mock( + get => sub { + return bless({}, 'PVE::RPCEnvironment'); + }, + get_user => sub { + return 'root@pam'; + }, + check => sub { + return 1; + }, + check_vm_perm => sub { + return 1; + }, +); + +my $cgroup_module = Test::MockModule->new('PVE::QemuServer::CGroup'); +$cgroup_module->mock( + change_cpu_shares => sub { + my ($self, $shares) = @_; + record_action('cgroup', 'change_cpu_shares', $shares); + return; + }, + change_cpu_quota => sub { + my ($self, $quota, $period) = @_; + record_action('cgroup', 'change_cpu_quota', $quota, $period); + return; + }, +); + +# warnings are part of the expected outcome, e.g. when cleaning up after a failed hotplug +$SIG{__WARN__} = sub { + my $warning = shift; + chomp $warning; + $warning =~ s/ at \S+ line \d+\.?$//; + record_action('warn', $warning); +}; + +sub do_test($config_fn) { + die "no such input test config: $config_fn\n" if !-f $config_fn; + + $recorded_actions = []; # also record warnings while parsing the config + parse_test($config_fn); + + my $testname = $current_test->{testname}; + my $conf = $current_test->{config}; + + my $errors = {}; + eval { + setup_vm_state($conf); + PVE::QemuServer::vmconfig_hotplug_pending($vmid, $conf, $storecfg, undef, $errors); + }; + if (my $err = $@) { + $errors->{died} = $err; + } + + # error messages without trailing newline contain the source location, which changes too often + my $clean_error = sub { + my ($key, $msg) = @_; + chomp $msg; + $msg =~ s/ at \S+ line \d+\.?$//; + return "$key: $msg"; + }; + + my $result = ""; + $result .= "$_\n" for $recorded_actions->@*; + + my $expected_errors = $current_test->{expected_errors}; + if (scalar($expected_errors->@*)) { + my $expected = join("\n", $expected_errors->@*); + my $got = join("\n", map { $clean_error->($_, $errors->{$_}) } sort keys $errors->%*); + + if ($expected ne $got) { + note("unexpected error output:"); + note("got: $got"); + note("expected: $expected"); + fail($testname); + return; + } + } + + # the target of a live migration is started with the resulting config, so it must match + my $differences = eval { compare_with_fresh_vm($conf) }; + if (my $err = $@) { + push $differences->@*, + "failed to start a VM with the resulting config: " . $clean_error->('start', $err); + } + + if (scalar($differences->@*) > 0) { + note("there are differences to freshly started vm:"); + for my $diff (sort $differences->@*) { + note($diff); + } + fail("$testname"); + return; + } + + if (scalar($conf->{pending}->%*) > 0 && !$current_test->{stays_pending}) { + note("could not hotplug (unexpectedly): " . join(',', keys $conf->{pending}->%*)); + fail("$testname"); + return; + } + + my $expected_fn = "$config_fn.expected"; + + # only write expected file out if there is any output + if (!-f $expected_fn) { + if ($result ne "") { + file_set_contents($expected_fn, $result); + pass("$testname (generated expected output)"); + } else { + pass("$testname"); + } + return; + } + + my $expected = file_get_contents($expected_fn); + if (my $diff = diff($expected, $result)) { + note("files differ:\n$diff"); + fail("$testname"); + } else { + pass("$testname"); + } +} + +print "testing hotplug of pending changes\n"; + +# exec tests +my $test_target = shift // 'hotplug'; + +if (-f $test_target) { + do_test($test_target); +} elsif (-d $test_target) { + PVE::File::dir_glob_foreach( + $test_target, + qr/.+\.conf/, + sub { + my ($file) = @_; + + do_test("${test_target}/${file}"); + }, + ); +} else { + die "test target '$test_target' is neither file nor directory, cannot proceed\n"; +} + +done_testing(); diff --git a/src/test/run_parse_config_tests.pl b/src/test/run_parse_config_tests.pl index 62e36ee0..f1ee70c2 100755 --- a/src/test/run_parse_config_tests.pl +++ b/src/test/run_parse_config_tests.pl @@ -11,9 +11,7 @@ use strict; use warnings; -use lib qw(..); - -use File::Path qw(make_path remove_tree); +use lib qw(.. .); use Test::MockModule; use Test::More; @@ -21,8 +19,9 @@ use Test::More; use PVE::QemuServer; use PVE::Tools; +use TestsCommon::Utils qw(diff); + my $INPUT_DIR = './parse-config-input'; -my $OUTPUT_DIR = './parse-config-output'; my $EXPECTED_DIR = './parse-config-expected'; # NOTE update when you add/remove tests @@ -72,25 +71,23 @@ sub run_tests { return; } - my $output_file = "${OUTPUT_DIR}/${file}"; - PVE::Tools::file_set_contents($output_file, $output); - my $expected_file = "${EXPECTED_DIR}/${file}"; - $expected_file = $input_file if !-f $expected_file; + my $expected = $input; + if (-f $expected_file) { + $expected = PVE::Tools::file_get_contents($expected_file); + } - my $cmd = ['diff', '-u', $expected_file, $output_file]; - if (system(@$cmd) == 0) { - pass($file); - } else { + if (my $diff = diff($expected, $output)) { + note($diff); fail($file); + } else { + pass($file); } }, ); } -make_path(${OUTPUT_DIR}); run_tests(0); run_tests(1); -remove_tree(${OUTPUT_DIR}) or die "failed to remove output directory\n"; done_testing(); -- 2.47.3