* [PATCH test-tools 01/38] ansible: rename snd_test bridge to tt_test as multi-purpose bridge
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 02/38] plans: rename sdn_test bridge to tt_test Michael Köppl
` (37 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Additional tests might also require a separate bridge. Since tests are
not run in parallel, the bridge can easily be shared, which also allows
tests that want to use a separate NIC to uniformly refer to ens19
without any additional changes.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
ansible/README.md | 6 +++---
ansible/group_vars/test_hosts.yml | 12 ++++++------
2 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/ansible/README.md b/ansible/README.md
index fc82d8d7..5b5e9f93 100644
--- a/ansible/README.md
+++ b/ansible/README.md
@@ -72,9 +72,9 @@ lying around there, the play stops and displays a warning message. Remove the st
builds or point `tt_deb_source_dir` somewhere else.
**Test hosts** get their own SDN simple zone (`172.17.0.0/24` with DHCP and
-SNAT, sized from `sdn_subnet_cidr`), an isolated `sdn_test` bridge that the
-plans attach the SDN instances' second NIC to, a PVE API token for `proxmox-test-instance`,
-an SSH keypair for reaching the test instances,
+SNAT, sized from `sdn_subnet_cidr`), an isolated `tt_test` bridge that the
+plans attach the cluster instances' second NIC to, a PVE API token for
+`proxmox-test-instance`, an SSH keypair for reaching the test instances,
`~/.config/proxmox-test-tools/{proxmox-test-instance,secrets}.ron`, and the
`proxmox-e2e-tests` package with `tests`, `scripts` and `files` linked from
`/usr/share/proxmox-e2e-tests` into `/root`. Those links matter because
diff --git a/ansible/group_vars/test_hosts.yml b/ansible/group_vars/test_hosts.yml
index 005cf0e3..0669d8df 100644
--- a/ansible/group_vars/test_hosts.yml
+++ b/ansible/group_vars/test_hosts.yml
@@ -16,12 +16,12 @@ sdn_gateway: "172.17.0.254"
sdn_dhcp_start: "172.17.0.100"
sdn_dhcp_end: "172.17.0.200"
-# Separate bridge for the second NIC of the SDN-related instances and
-# the router and internet containers, over which the SDN tests build
-# their topology. Note that the plans hardcode this name
-# (plans/complete.ron, plans/sdn.ron), so only change it together with
-# them.
-tt_test_bridge: sdn_test
+# Separate bridge for the second NIC of the cluster instances (for SDN
+# and Ceph-related tests) and the router and internet containers, over
+# which the tests build their topology. Note that the plans hardcode
+# this name (plans/complete.ron, plans/sdn.ron), so only change it
+# together with them.
+tt_test_bridge: tt_test
# PVE API access used by proxmox-test-instance
#
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 02/38] plans: rename sdn_test bridge to tt_test
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
2026-11-04 19:03 ` [PATCH test-tools 01/38] ansible: rename snd_test bridge to tt_test as multi-purpose bridge Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 03/38] debian: add missing jq dependency Michael Köppl
` (36 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Besides the SDN tests, the Ceph tests will use the bridge for their
dedicated networks, so give it a name that is not tied to SDN.
The test hosts have to provide the bridge unter its new name.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
plans/complete.ron | 12 ++++++------
plans/sdn.ron | 12 ++++++------
2 files changed, 12 insertions(+), 12 deletions(-)
diff --git a/plans/complete.ron b/plans/complete.ron
index 2d723ad..75cd5b1 100644
--- a/plans/complete.ron
+++ b/plans/complete.ron
@@ -89,7 +89,7 @@ Plan(
],
additional_nics: [
NIC(
- bridge: "sdn_test",
+ bridge: "tt_test",
),
],
),
@@ -126,7 +126,7 @@ Plan(
],
additional_nics: [
NIC(
- bridge: "sdn_test",
+ bridge: "tt_test",
),
],
),
@@ -163,7 +163,7 @@ Plan(
],
additional_nics: [
NIC(
- bridge: "sdn_test",
+ bridge: "tt_test",
),
],
),
@@ -179,7 +179,7 @@ Plan(
template: "local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst",
net: [
"name=eth0,bridge=ttvnet,ip=dhcp",
- "name=eth1,bridge=sdn_test",
+ "name=eth1,bridge=tt_test",
],
root_password: Ref("default"),
),
@@ -193,7 +193,7 @@ Plan(
template: "local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst",
net: [
"name=eth0,bridge=ttvnet,ip=dhcp",
- "name=eth1,bridge=sdn_test",
+ "name=eth1,bridge=tt_test",
],
root_password: Ref("default"),
),
@@ -207,7 +207,7 @@ Plan(
template: "local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst",
net: [
"name=eth0,bridge=ttvnet,ip=dhcp",
- "name=eth1,bridge=sdn_test",
+ "name=eth1,bridge=tt_test",
],
root_password: Ref("default"),
),
diff --git a/plans/sdn.ron b/plans/sdn.ron
index 8426f9a..887b6a0 100644
--- a/plans/sdn.ron
+++ b/plans/sdn.ron
@@ -13,7 +13,7 @@ Plan(
root_password: Ref("default"),
additional_nics: [
NIC(
- bridge: "sdn_test",
+ bridge: "tt_test",
),
],
),
@@ -33,7 +33,7 @@ Plan(
root_password: Ref("default"),
additional_nics: [
NIC(
- bridge: "sdn_test",
+ bridge: "tt_test",
),
],
),
@@ -53,7 +53,7 @@ Plan(
root_password: Ref("default"),
additional_nics: [
NIC(
- bridge: "sdn_test",
+ bridge: "tt_test",
),
],
),
@@ -68,7 +68,7 @@ Plan(
template: "local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst",
net: [
"name=eth0,bridge=ttvnet,ip=dhcp",
- "name=eth1,bridge=sdn_test",
+ "name=eth1,bridge=tt_test",
],
root_password: Ref("default"),
),
@@ -82,7 +82,7 @@ Plan(
template: "local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst",
net: [
"name=eth0,bridge=ttvnet,ip=dhcp",
- "name=eth1,bridge=sdn_test",
+ "name=eth1,bridge=tt_test",
],
root_password: Ref("default"),
),
@@ -96,7 +96,7 @@ Plan(
template: "local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst",
net: [
"name=eth0,bridge=ttvnet,ip=dhcp",
- "name=eth1,bridge=sdn_test",
+ "name=eth1,bridge=tt_test",
],
root_password: Ref("default"),
),
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 03/38] debian: add missing jq dependency
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
2026-11-04 19:03 ` [PATCH test-tools 01/38] ansible: rename snd_test bridge to tt_test as multi-purpose bridge Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 02/38] plans: rename sdn_test bridge to tt_test Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 04/38] test framework: move the default container template to the helper Michael Köppl
` (35 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
The scripts that the plans run as post-setup steps read the SSH key,
host, and user of the test instance from the JSON file in $CONFIG_PATH
with jq. Without jq, they run ssh with empty arguments, which fails
with an error that does not point at the missing jq.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
debian/control | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/debian/control b/debian/control
index 12ff2cc..b7abfdf 100644
--- a/debian/control
+++ b/debian/control
@@ -10,7 +10,8 @@ Architecture: all
Depends: ${misc:Depends}, ${perl:Depends},
libjson-perl,
libio-socket-ssl-perl,
- libnet-openssh-perl
+ libnet-openssh-perl,
+ jq
Description: Proxmox End-to-End Tests and Helpers
This package contains the end-to-end test suite and Perl helper
modules for Proxmox projects. It requires proxmox-test-runner
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 04/38] test framework: move the default container template to the helper
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (2 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 03/38] debian: add missing jq dependency Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 05/38] lib: instance: add helper to check if an address answers ping Michael Köppl
` (34 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
The default container template is not specific to the SDN tests. The
firewall test already uses it and future tests will too. Move it to
Proxmox::Test::Helper, so that tests do not have to import from the SDN
helpers.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
Proxmox/Test/Helper.pm | 8 +++++++-
Proxmox/Test/SDN.pm | 5 -----
tests/pve_firewall.pl | 3 +--
tests/sdn/test_vlan_zone.pl | 4 ++--
4 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/Proxmox/Test/Helper.pm b/Proxmox/Test/Helper.pm
index 17b31db..87a0a6d 100644
--- a/Proxmox/Test/Helper.pm
+++ b/Proxmox/Test/Helper.pm
@@ -3,7 +3,8 @@ use strict;
use warnings FATAL => 'all';
use parent 'Exporter';
-our @EXPORT_OK = qw(get_pve_instance get_pbs_instance get_instance get_pve_instances);
+our @EXPORT_OK =
+ qw(CONTAINER_TEMPLATE get_pve_instance get_pbs_instance get_instance get_pve_instances);
use JSON;
@@ -12,6 +13,11 @@ use Proxmox::Test::ProxmoxInstance;
use Proxmox::Test::PVEInstance;
use Proxmox::Test::PBSInstance;
+# Default container OS template used by various test containers. Single source of truth so a
+# template version bump only needs to be done in one place. Tests that need specific templates
+# import them separately.
+use constant CONTAINER_TEMPLATE => 'local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst';
+
my $config_cache;
sub read_config {
diff --git a/Proxmox/Test/SDN.pm b/Proxmox/Test/SDN.pm
index 4f0b63c..8c27bb8 100644
--- a/Proxmox/Test/SDN.pm
+++ b/Proxmox/Test/SDN.pm
@@ -4,7 +4,6 @@ use warnings FATAL => 'all';
use parent 'Exporter';
our @EXPORT_OK = qw(
- CONTAINER_TEMPLATE
apply_sdn_config
wait_for_vnet
create_test_containers
@@ -21,10 +20,6 @@ use Test::More;
use Proxmox::Test::Helper qw(get_pve_instance);
use Proxmox::Test::Util qw(poll_until);
-# Container OS template used by all SDN test containers. Single source of truth so a template
-# version bump only needs to be done in one place.
-use constant CONTAINER_TEMPLATE => 'local:vztmpl/debian-12-standard_12.12-1_amd64.tar.zst';
-
# Wait until the interface backing $vnet exists on every instance in $instances.
#
# Until a node's reload has run, guests attached to the VNet fail to start with "bridge '<vnet>'
diff --git a/tests/pve_firewall.pl b/tests/pve_firewall.pl
index e1f6ec2..fab35b5 100755
--- a/tests/pve_firewall.pl
+++ b/tests/pve_firewall.pl
@@ -5,9 +5,8 @@ use warnings;
use Test::More;
use lib '.';
-use Proxmox::Test::Helper qw(get_pve_instance);
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pve_instance);
use Proxmox::Test::Util qw(poll_until);
-use Proxmox::Test::SDN qw(CONTAINER_TEMPLATE);
# End-to-end test of the PVE firewall: verifies a rule actually affects the dataplane, rather than
# only exercising the firewall config API (which is covered by unit and API-schema tests).
diff --git a/tests/sdn/test_vlan_zone.pl b/tests/sdn/test_vlan_zone.pl
index d0a412a..0c41dfa 100755
--- a/tests/sdn/test_vlan_zone.pl
+++ b/tests/sdn/test_vlan_zone.pl
@@ -19,8 +19,8 @@ use warnings;
use Test::More;
use lib '.';
-use Proxmox::Test::Helper qw(get_pve_instance get_pve_instances get_instance);
-use Proxmox::Test::SDN qw(CONTAINER_TEMPLATE apply_sdn_config);
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pve_instance get_pve_instances get_instance);
+use Proxmox::Test::SDN qw(apply_sdn_config);
use Proxmox::Test::Testcase;
use PVE::JSONSchema;
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 05/38] lib: instance: add helper to check if an address answers ping
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (3 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 04/38] test framework: move the default container template to the helper Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 06/38] lib: pveinstance: look up disks by their SCSI serial Michael Köppl
` (33 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Move the helper function from pve_firewall to Proxmox::Test::Instance
because it's about to be used for additional tests.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
Proxmox/Test/Instance.pm | 7 +++++++
tests/pve_firewall.pl | 23 ++++++++++++-----------
2 files changed, 19 insertions(+), 11 deletions(-)
diff --git a/Proxmox/Test/Instance.pm b/Proxmox/Test/Instance.pm
index 38f1607..8c0f710 100644
--- a/Proxmox/Test/Instance.pm
+++ b/Proxmox/Test/Instance.pm
@@ -74,4 +74,11 @@ sub run_command {
return ($stdout, $stderr);
}
+# Returns true if $ip answers a ping sent from this instance.
+sub node_can_ping {
+ my ($self, $ip) = @_;
+
+ return eval { $self->run_command("ping -c2 -W2 $ip"); 1 } ? 1 : 0;
+}
+
1;
diff --git a/tests/pve_firewall.pl b/tests/pve_firewall.pl
index fab35b5..d3991a9 100755
--- a/tests/pve_firewall.pl
+++ b/tests/pve_firewall.pl
@@ -34,11 +34,6 @@ my $vmid = $client->get('/cluster/nextid');
my $ct_ip = '10.250.250.10';
my $node_ip = '10.250.250.1';
-# Returns true if the node can ping the container.
-my $can_ping = sub {
- return eval { $pve->run_command("ping -c2 -W2 $ct_ip"); 1 } ? 1 : 0;
-};
-
# Create and start the container
my $upid = $client->post(
@@ -65,12 +60,15 @@ $pve->run_command("ip addr add $node_ip/24 dev vmbr0");
eval {
# Baseline: ping works before any firewall
poll_until(
- $can_ping,
+ sub { $pve->node_can_ping($ct_ip) },
timeout => 30,
interval => 3,
description => "container $vmid to be pingable",
);
- ok($can_ping->(), 'node can ping the container before the firewall is enabled');
+ ok(
+ $pve->node_can_ping($ct_ip),
+ 'node can ping the container before the firewall is enabled',
+ );
# Enable the datacenter and container firewall
$client->put('/cluster/firewall/options', { enable => 1 });
@@ -79,12 +77,15 @@ eval {
# With the firewall enabled the default inbound policy is DROP, so ping is blocked. pve-firewall
# applies config changes asynchronously, so poll for the effect.
poll_until(
- sub { !$can_ping->() },
+ sub { !$pve->node_can_ping($ct_ip) },
timeout => 30,
interval => 3,
description => "the firewall to block ICMP to the container",
);
- ok(!$can_ping->(), 'ping is blocked once the container firewall defaults to DROP');
+ ok(
+ !$pve->node_can_ping($ct_ip),
+ 'ping is blocked once the container firewall defaults to DROP',
+ );
# Add an ACCEPT rule for ICMP: ping works again
$client->post(
@@ -99,12 +100,12 @@ eval {
);
poll_until(
- $can_ping,
+ sub { $pve->node_can_ping($ct_ip) },
timeout => 30,
interval => 3,
description => "the ACCEPT rule to allow ICMP to the container",
);
- ok($can_ping->(), 'ping works again after adding an ACCEPT rule for ICMP');
+ ok($pve->node_can_ping($ct_ip), 'ping works again after adding an ACCEPT rule for ICMP');
1;
} or do {
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 06/38] lib: pveinstance: look up disks by their SCSI serial
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (4 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 05/38] lib: instance: add helper to check if an address answers ping Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 07/38] lib: pveinstance: add helper to find a task started by another task Michael Köppl
` (32 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
The disk list of the API reports the serial from udev's
ID_SERIAL_SHORT. For QEMU SCSI disks, udev derives that from the
device identifier, which is the drive name such as 'drive-scsi1',
not the serial the plan set. get_disk_by_serial() could therefore
never find a disk by the serial a plan assigned to it.
Similarly to setup-zfs-storage.sh, look the disk up by ID_SCSI_SERIAL
instead.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
Could be fixed by [0], but that might be better to postpone for a major
version.
[0] https://lore.proxmox.com/pve-devel/20261028190713.2788477-1-m.koeppl@proxmox.com
Proxmox/Test/PVEInstance.pm | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/Proxmox/Test/PVEInstance.pm b/Proxmox/Test/PVEInstance.pm
index 7f82171..dc40489 100644
--- a/Proxmox/Test/PVEInstance.pm
+++ b/Proxmox/Test/PVEInstance.pm
@@ -28,16 +28,19 @@ sub new_local {
return $class->SUPER::new($nodename, $config, 8006, 'PVEAuthCookie');
}
+# Return the device node of the SCSI disk the plan gave the serial $serial, or undef if there is
+# none.
sub get_disk_by_serial {
my ($self, $serial) = @_;
- my $node = $self->get_nodename();
- my $res = $self->client()->get("nodes/$node/disks/list", {});
+ my ($out) =
+ $self->run_command('for disk in /dev/sd?; do'
+ . ' echo "$disk $(udevadm info --query=property --name="$disk"'
+ . ' | sed -n "s/^ID_SCSI_SERIAL=//p")"; done');
- for my $item ($res->@*) {
- if ($item->{serial} eq $serial) {
- return $item->{devpath};
- }
+ for my $line (split(/\n/, $out // '')) {
+ my ($dev, $dev_serial) = split(/\s+/, $line, 2);
+ return $dev if defined($dev_serial) && $dev_serial eq $serial;
}
return undef;
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 07/38] lib: pveinstance: add helper to find a task started by another task
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (5 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 06/38] lib: pveinstance: look up disks by their SCSI serial Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 08/38] lib: pveinstance: add helpers to start and stop guests Michael Köppl
` (31 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Some tasks start another task for the same guest and finish before
it does (e.g. a container rollback with 'start' set runs the start in a
separate 'vzstart' task). A test that only waits for the first task then
races against the second one.
Add a helper that finds such a task through the node's task list, so
tests can wait for it and check its result.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
Proxmox/Test/PVEInstance.pm | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/Proxmox/Test/PVEInstance.pm b/Proxmox/Test/PVEInstance.pm
index dc40489..646ff48 100644
--- a/Proxmox/Test/PVEInstance.pm
+++ b/Proxmox/Test/PVEInstance.pm
@@ -109,6 +109,36 @@ sub destroy_volume {
return $ok;
}
+# Return the UPID of the task of $type that the task $upid started for the same guest, such as the
+# 'vzstart' task of a rollback with 'start' set. Such a task runs separately and can still be
+# running, or not have started yet, when $upid has finished. Dies if no such task shows up within
+# $timeout seconds (default 60).
+sub find_spawned_task {
+ my ($self, $upid, $type, $timeout) = @_;
+
+ my (undef, $node, undef, undef, $starttime, undef, $vmid) = split(/:/, $upid);
+ die "cannot parse UPID '$upid'\n" if !defined($vmid) || $vmid eq '';
+
+ return poll_until(
+ sub {
+ my $tasks = $self->client()->get(
+ "/nodes/$node/tasks",
+ {
+ vmid => $vmid,
+ typefilter => $type,
+ since => hex($starttime),
+ source => 'all',
+ },
+ );
+ my ($task) = grep { $_->{upid} ne $upid } $tasks->@*;
+ return $task ? $task->{upid} : undef;
+ },
+ timeout => $timeout // 60,
+ interval => 2,
+ description => "a '$type' task for guest $vmid started by $upid",
+ );
+}
+
sub wait_for_qga {
my ($self, $vmid, $timeout) = @_;
return $self->wait_for_qga_on($self->get_nodename(), $vmid, $timeout);
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 08/38] lib: pveinstance: add helpers to start and stop guests
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (6 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 07/38] lib: pveinstance: add helper to find a task started by another task Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 09/38] lib: pveinstance: add helper to run commands in containers Michael Köppl
` (30 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Most tests start or stop a guest at some point and need to check that
the task succeeded that the guest reports status 'running' after a
start. Without a shared helper, each test has to repeat this.
Add start_guest() and stop_guest() for both VMs and containers, which
record the task result as a test point and die if it failed.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
Proxmox/Test/PVEInstance.pm | 36 ++++++++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/Proxmox/Test/PVEInstance.pm b/Proxmox/Test/PVEInstance.pm
index 646ff48..957e3e7 100644
--- a/Proxmox/Test/PVEInstance.pm
+++ b/Proxmox/Test/PVEInstance.pm
@@ -51,6 +51,42 @@ sub get_additional_disks {
return $self->{config}->{additional_disks};
}
+my $guest_label = sub {
+ my ($type) = @_;
+ return $type eq 'qemu' ? 'VM' : 'container';
+};
+
+# Start a guest and assert that the start task succeeded. Dies if it failed or if the guest does not
+# report status 'running' afterwards. $type can either be 'qemu' or 'lxc'.
+sub start_guest {
+ my ($self, $type, $vmid) = @_;
+
+ my $client = $self->client();
+ my $path = "/nodes/" . $self->get_nodename() . "/$type/$vmid";
+ my $label = $guest_label->($type);
+
+ my $upid = $client->post("$path/status/start", {});
+ $self->task_ok($upid, "started $label $vmid") or die "starting $label $vmid failed\n";
+
+ poll_until(
+ sub { $client->get("$path/status/current", {})->{status} eq 'running' },
+ timeout => 120,
+ description => "$label $vmid to report status 'running'",
+ );
+}
+
+# Stop a guest and assert that the stop task succeeded. Dies if it failed. $type can either be
+# 'qemu' or 'lxc'.
+sub stop_guest {
+ my ($self, $type, $vmid) = @_;
+
+ my $path = "/nodes/" . $self->get_nodename() . "/$type/$vmid";
+ my $label = $guest_label->($type);
+
+ my $upid = $self->client()->post("$path/status/stop", {});
+ $self->task_ok($upid, "stopped $label $vmid") or die "stopping $label $vmid failed\n";
+}
+
sub destroy_guest {
my ($self, $type, $vmid) = @_;
return $self->destroy_guest_on($self->get_nodename(), $type, $vmid);
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 09/38] lib: pveinstance: add helper to run commands in containers
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (7 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 08/38] lib: pveinstance: add helpers to start and stop guests Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 10/38] lib: pveinstance: add helper to wait for qmeventd's cleanup Michael Köppl
` (29 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Most container tests run commands inside their container with pct
exec on the node, and each of them defines its own closure for it.
Add ct_exec() so they can share one.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
Proxmox/Test/PVEInstance.pm | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/Proxmox/Test/PVEInstance.pm b/Proxmox/Test/PVEInstance.pm
index 957e3e7..132a275 100644
--- a/Proxmox/Test/PVEInstance.pm
+++ b/Proxmox/Test/PVEInstance.pm
@@ -87,6 +87,15 @@ sub stop_guest {
$self->task_ok($upid, "stopped $label $vmid") or die "stopping $label $vmid failed\n";
}
+# Run $cmd inside the running container $ctid on this node and return its standard output. Dies if
+# the command fails.
+sub ct_exec {
+ my ($self, $ctid, $cmd) = @_;
+
+ my ($out) = $self->run_command("pct exec $ctid -- $cmd");
+ return $out // '';
+}
+
sub destroy_guest {
my ($self, $type, $vmid) = @_;
return $self->destroy_guest_on($self->get_nodename(), $type, $vmid);
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 10/38] lib: pveinstance: add helper to wait for qmeventd's cleanup
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (8 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 09/38] lib: pveinstance: add helper to run commands in containers Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 11/38] lib: pveinstance: add helper to check if guest exists Michael Köppl
` (28 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
When a QEMU process exits, qmeventd runs 'qm cleanup' for the VM on
its own. If the VM runs again by the time the cleanup gets the config
lock, for example after a hibernate and start or a rollback to a
snapshot with RAM, the cleanup holds the lock for up to 30 seconds.
Operations on the VM within that timeframe fail with a lock timeout.
Add a helper that waits for the cleanup to finish, for tests that
need to change the VM right after such an operation.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
Proxmox/Test/PVEInstance.pm | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/Proxmox/Test/PVEInstance.pm b/Proxmox/Test/PVEInstance.pm
index 132a275..7e0c611 100644
--- a/Proxmox/Test/PVEInstance.pm
+++ b/Proxmox/Test/PVEInstance.pm
@@ -96,6 +96,24 @@ sub ct_exec {
return $out // '';
}
+# Wait until qmeventd has cleaned up after a QEMU process of VM $vmid that exited after $since, a
+# timestamp of the node's clock. Helps prevent a race for the config lock between the cleanup
+# process and the VM.
+sub wait_for_qemu_cleanup {
+ my ($self, $vmid, $since, $timeout) = @_;
+
+ poll_until(
+ sub {
+ my ($log) = $self->run_command("journalctl -u qmeventd --since \@$since -o cat");
+ return 0 if ($log // '') !~ m/^Starting cleanup for \Q$vmid\E$/m;
+ return !eval { $self->run_command("pgrep -f 'qm cleanup $vmid( |\$)'"); 1 };
+ },
+ timeout => $timeout // 60,
+ interval => 1,
+ description => "qmeventd to clean up after VM $vmid",
+ );
+}
+
sub destroy_guest {
my ($self, $type, $vmid) = @_;
return $self->destroy_guest_on($self->get_nodename(), $type, $vmid);
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 11/38] lib: pveinstance: add helper to check if guest exists
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (9 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 10/38] lib: pveinstance: add helper to wait for qmeventd's cleanup Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 12/38] lib: add helpers to use a PBS datastore as PVE storage Michael Köppl
` (27 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
In preparation for the upcoming tests, of which multiple check for the
existence of a guest that they're using to test.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
Proxmox/Test/PVEInstance.pm | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/Proxmox/Test/PVEInstance.pm b/Proxmox/Test/PVEInstance.pm
index 7e0c611..d55aa4a 100644
--- a/Proxmox/Test/PVEInstance.pm
+++ b/Proxmox/Test/PVEInstance.pm
@@ -87,6 +87,15 @@ sub stop_guest {
$self->task_ok($upid, "stopped $label $vmid") or die "stopping $label $vmid failed\n";
}
+# Return whether a guest of $type ('qemu', 'lxc') with $vmid exists on this node.
+sub guest_exists {
+ my ($self, $type, $vmid) = @_;
+
+ my $node = $self->get_nodename();
+ my $guests = $self->client()->get("/nodes/$node/$type", {});
+ return scalar(grep { $_->{vmid} == $vmid } $guests->@*);
+}
+
# Run $cmd inside the running container $ctid on this node and return its standard output. Dies if
# the command fails.
sub ct_exec {
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 12/38] lib: add helpers to use a PBS datastore as PVE storage
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (10 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 11/38] lib: pveinstance: add helper to check if guest exists Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 13/38] scripts: retry container template download, fail if it fails Michael Köppl
` (26 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Tests that back up guests to PBS first create a datastore on PBS and
add it to PVE as a storage, and remove both again afterwards. Add
helpers for creating and removing the datastore and for adding it as
a storage, so these tests do not have to repeat the same API calls.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
Proxmox/Test/PBSInstance.pm | 19 +++++++++++++++++++
Proxmox/Test/PVEInstance.pm | 19 +++++++++++++++++++
2 files changed, 38 insertions(+)
diff --git a/Proxmox/Test/PBSInstance.pm b/Proxmox/Test/PBSInstance.pm
index 80138e3..165e4bc 100644
--- a/Proxmox/Test/PBSInstance.pm
+++ b/Proxmox/Test/PBSInstance.pm
@@ -10,4 +10,23 @@ sub new {
return $class->SUPER::new($name, $config, 8007, 'PBSAuthCookie');
}
+# Create a datastore named $name in /mnt/$name and assert that the creation task succeeded. Dies if
+# it failed.
+sub create_datastore {
+ my ($self, $name) = @_;
+
+ my $upid = $self->client()->post('/config/datastore', { name => $name, path => "/mnt/$name" });
+ $self->task_ok($upid, "created datastore '$name' on PBS")
+ or die "creating datastore '$name' failed\n";
+}
+
+# Remove a datastore created by create_datastore() together with its contents.
+sub remove_datastore {
+ my ($self, $name) = @_;
+
+ my $upid = $self->client()->delete("/config/datastore/$name", { 'destroy-data' => 1 });
+ $self->wait_for_task($upid);
+ $self->run_command("rm -rf /mnt/$name");
+}
+
1;
diff --git a/Proxmox/Test/PVEInstance.pm b/Proxmox/Test/PVEInstance.pm
index d55aa4a..e95480d 100644
--- a/Proxmox/Test/PVEInstance.pm
+++ b/Proxmox/Test/PVEInstance.pm
@@ -123,6 +123,25 @@ sub wait_for_qemu_cleanup {
);
}
+# Add the datastore $datastore of the PBS instance $pbs as storage $storage_id for backups.
+sub add_pbs_storage {
+ my ($self, $storage_id, $pbs, $datastore) = @_;
+
+ $self->client()->post(
+ '/storage',
+ {
+ storage => $storage_id,
+ type => 'pbs',
+ server => $pbs->get_ip(),
+ datastore => $datastore,
+ username => 'root@pam',
+ password => $pbs->get_root_password(),
+ fingerprint => $pbs->get_fingerprint(),
+ content => 'backup',
+ },
+ );
+}
+
sub destroy_guest {
my ($self, $type, $vmid) = @_;
return $self->destroy_guest_on($self->get_nodename(), $type, $vmid);
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 13/38] scripts: retry container template download, fail if it fails
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (11 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 12/38] lib: add helpers to use a PBS datastore as PVE storage Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 14/38] tests: container: verify lifecycle and guest setup Michael Köppl
` (25 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Since the download can sometimes be rejected with a 503, retry it after
a while instead of failing silently. Also, do not fail silently if the
download is unsuccessful even with multiple attempts. The setup then
would not work anyway.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
scripts/download-container-image.sh | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/scripts/download-container-image.sh b/scripts/download-container-image.sh
index b14a636..5db8af8 100755
--- a/scripts/download-container-image.sh
+++ b/scripts/download-container-image.sh
@@ -9,5 +9,14 @@ IMAGE=$2
ssh -i $SSH_KEY $USER@$HOST -o StrictHostKeyChecking=no <<EOF
set -e
-pveam download $STORAGE $IMAGE
+path=\$(pvesm path $STORAGE:vztmpl/$IMAGE)
+# Sometimes requests are rejected with 503, to try again after a while.
+for attempt in 1 2 3 4 5; do
+ pveam download $STORAGE $IMAGE || true
+ # Explicitly check for the path, since download will still return 0 if it fails.
+ [ -e "\$path" ] && exit 0
+ sleep \$((attempt * 10))
+done
+echo "downloading $IMAGE to $STORAGE failed" >&2
+exit 1
EOF
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 14/38] tests: container: verify lifecycle and guest setup
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (12 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 13/38] scripts: retry container template download, fail if it fails Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 15/38] tests: container: verify volume and bind mountpoints Michael Köppl
` (24 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Add test that covers creating, starting, shutting down, and destroying
an unprivileged container. Also check that hostname, hosts, DNS,
timezone, static network configuration are actually set up inside the
guest.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_ct_lifecycle.pl | 182 ++++++++++++++++++++++++++++++++++++++
tests/testcases.ron | 9 ++
2 files changed, 191 insertions(+)
create mode 100755 tests/pve_ct_lifecycle.pl
diff --git a/tests/pve_ct_lifecycle.pl b/tests/pve_ct_lifecycle.pl
new file mode 100755
index 0000000..2a2920e
--- /dev/null
+++ b/tests/pve_ct_lifecycle.pl
@@ -0,0 +1,182 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_ct_lifecycle> - Test the lifecycle and guest setup of a container.
+
+=head1 DESCRIPTION
+
+This testcase creates an unprivileged container, starts it, shuts it down and
+destroys it again. While it is running, it verifies that the hostname, hosts
+file, DNS, timezone and static network configuration were set up inside the
+guest, and that the container and the node can reach each other over vmbr0.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pve_instance);
+use Proxmox::Test::SDN qw(ping_from_container);
+use Proxmox::Test::Testcase;
+use Proxmox::Test::Util qw(poll_until);
+
+my $pve = get_pve_instance('pve1');
+my $node = $pve->get_nodename();
+my $client = $pve->client();
+
+my $ct_ip = '10.250.251.10';
+my $node_ip = '10.250.251.1';
+
+my $test_container = {
+ vmid => $client->get('/cluster/nextid'),
+ node => $node,
+ ostemplate => CONTAINER_TEMPLATE,
+ rootfs => 'local-lvm:2',
+ storage => 'local-lvm',
+ memory => 256,
+ unprivileged => 1,
+ features => 'nesting=1',
+ hostname => 'ct-lifecycle',
+ searchdomain => 'e2e.test',
+ nameserver => '10.250.251.53',
+ timezone => 'Europe/Vienna',
+ net0 => "name=eth0,bridge=vmbr0,ip=$ct_ip/24",
+};
+
+my $vmid = $test_container->{vmid};
+my $rootfs_volid;
+
+Proxmox::Test::Testcase->new('pve_ct_lifecycle')->with([{}])->step(
+ 'create container',
+ sub {
+ my $upid = $client->post("/nodes/$node/lxc", $test_container);
+ $pve->task_ok($upid, "created container $vmid")
+ or die "creating container $vmid failed\n";
+ },
+ sub {
+ $pve->destroy_guest('lxc', $vmid);
+ },
+)->step(
+ 'verify container config',
+ sub {
+ my $conf = $client->get("/nodes/$node/lxc/$vmid/config", {});
+
+ is($conf->{hostname}, $test_container->{hostname}, 'config has hostname');
+ is($conf->{unprivileged}, 1, 'config marks container as unprivileged');
+ is($conf->{memory}, $test_container->{memory}, 'config has memory');
+ is($conf->{features}, $test_container->{features}, 'config has features');
+ is(
+ $conf->{searchdomain}, $test_container->{searchdomain}, 'config has search domain',
+ );
+ is($conf->{nameserver}, $test_container->{nameserver}, 'config has nameserver');
+ is($conf->{timezone}, $test_container->{timezone}, 'config has timezone');
+
+ like($conf->{rootfs} // '', qr/^local-lvm:/, 'rootfs is allocated on local-lvm');
+ my %net0 = map { split(/=/, $_, 2) } split(/,/, $conf->{net0} // '');
+ is($net0{name}, 'eth0', 'net0 has interface name');
+ is($net0{bridge}, 'vmbr0', 'net0 has bridge');
+ is($net0{ip}, "$ct_ip/24", 'net0 has static address');
+
+ ($rootfs_volid) = ($conf->{rootfs} // '') =~ m/^([^,]+)/;
+ },
+)->step(
+ 'start container',
+ sub {
+ $pve->start_guest('lxc', $vmid);
+ pass("container $vmid reports status 'running'");
+ },
+)->step(
+ 'add node address on vmbr0',
+ sub {
+ $pve->run_command("ip addr add $node_ip/24 dev vmbr0");
+ },
+ sub {
+ $pve->run_command("ip addr del $node_ip/24 dev vmbr0 2>/dev/null; true");
+ },
+)->step(
+ 'verify guest setup',
+ sub {
+ my $hostname = $test_container->{hostname};
+ my $searchdomain = $test_container->{searchdomain};
+ my $nameserver = $test_container->{nameserver};
+ my $timezone = $test_container->{timezone};
+
+ is($pve->ct_exec($vmid, 'cat /etc/hostname'), "$hostname\n", '/etc/hostname is set up');
+ like(
+ $pve->ct_exec($vmid, 'cat /etc/hosts'),
+ qr/^\Q$ct_ip\E\s+\Q$hostname.$searchdomain\E\s+\Q$hostname\E\s*$/m,
+ '/etc/hosts maps the container address to its hostname',
+ );
+
+ my $resolv = $pve->ct_exec($vmid, 'cat /etc/resolv.conf');
+ like($resolv, qr/^search \Q$searchdomain\E$/m,
+ '/etc/resolv.conf has the search domain');
+ like($resolv, qr/^nameserver \Q$nameserver\E$/m, '/etc/resolv.conf has the nameserver');
+
+ is(
+ $pve->ct_exec($vmid, 'readlink /etc/localtime'),
+ "/usr/share/zoneinfo/$timezone\n",
+ '/etc/localtime points to the configured timezone',
+ );
+ },
+)->step(
+ 'verify network connectivity',
+ sub {
+ # The static address can take a bit of time after the start task has finished, as the
+ # static address is applied by the container during boot.
+ poll_until(
+ sub {
+ $pve->ct_exec($vmid, 'ip -4 -o addr show dev eth0') =~
+ m/\binet \Q$ct_ip\E\/24\b/;
+ },
+ timeout => 60,
+ interval => 2,
+ description => "eth0 in container $vmid to get $ct_ip",
+ );
+ pass("eth0 in container $vmid has $ct_ip");
+
+ ping_from_container($test_container, $node_ip, 5);
+
+ poll_until(
+ sub { $pve->node_can_ping($ct_ip) },
+ timeout => 30,
+ interval => 3,
+ description => "container $vmid to be pingable from the node",
+ );
+ ok($pve->node_can_ping($ct_ip), "node can ping container $vmid");
+ },
+)->step(
+ 'shutdown container',
+ sub {
+ my $upid = $client->post("/nodes/$node/lxc/$vmid/status/shutdown", { timeout => 60 });
+ $pve->task_ok($upid, "shut down container $vmid")
+ or die "shutting down container $vmid failed\n";
+
+ is(
+ $client->get("/nodes/$node/lxc/$vmid/status/current", {})->{status},
+ 'stopped',
+ "container $vmid reports status 'stopped' after shutdown",
+ );
+ },
+)->step(
+ 'destroy container',
+ sub {
+ my $upid = $client->delete("/nodes/$node/lxc/$vmid", {});
+ $pve->task_ok($upid, "destroyed container $vmid")
+ or die "destroying container $vmid failed\n";
+
+ ok(!$pve->guest_exists('lxc', $vmid), "container $vmid is gone");
+
+ my $content = $client->get("/nodes/$node/storage/local-lvm/content", { vmid => $vmid });
+ ok(
+ defined($rootfs_volid) && !grep({ $_->{volid} eq $rootfs_volid } $content->@*),
+ "rootfs volume of container $vmid is gone",
+ );
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index eac2926..391eb09 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -20,6 +20,15 @@ TestcaseConfig(
],
tag: "basic",
),
+ Testcase(
+ name: "pve_ct_lifecycle",
+ run: "./tests/pve_ct_lifecycle.pl",
+ timeout: 300,
+ instances: [
+ "pve1",
+ ],
+ tag: "basic",
+ ),
Testcase(
name: "pve_ldap_realm",
run: "./tests/ldap_realm.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 15/38] tests: container: verify volume and bind mountpoints
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (13 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 14/38] tests: container: verify lifecycle and guest setup Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 16/38] tests: container: verify snapshots, clones, templates Michael Köppl
` (23 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Cover volume mountpoints on LVM-thin and on a directory storage as well
as a bind mountpoint from the host. All of them must be mounted in the
running container, an online resize must grow the file system, and
deleting a mountpoint must keep its volume as unused until that is
deleted as well.
Also destroy the container after its mountpoint storage was removed
from the storage configuration, which must succeed and only warn
about the volume it could not delete.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_ct_mountpoints.pl | 198 ++++++++++++++++++++++++++++++++++++
tests/testcases.ron | 9 ++
2 files changed, 207 insertions(+)
create mode 100755 tests/pve_ct_mountpoints.pl
diff --git a/tests/pve_ct_mountpoints.pl b/tests/pve_ct_mountpoints.pl
new file mode 100755
index 0000000..d6cbe70
--- /dev/null
+++ b/tests/pve_ct_mountpoints.pl
@@ -0,0 +1,198 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_ct_mountpoints> - Test volume and bind mountpoints of a container.
+
+=head1 DESCRIPTION
+
+This testcase creates an unprivileged container with a volume mountpoint on
+local-lvm, a bind mountpoint from the host, and a volume mountpoint on a
+temporary directory storage. It verifies that all of them are mounted inside
+the running container, resizes the local-lvm mountpoint online, and detaches
+and deletes it again.
+
+Finally, the directory storage is removed from the storage configuration while
+the container still has a mountpoint on it. Destroying the container must
+still succeed and only warn about the volume it could not delete.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pve_instance);
+use Proxmox::Test::Testcase;
+
+my $pve = get_pve_instance('pve1');
+my $node = $pve->get_nodename();
+my $client = $pve->client();
+
+my $dir_storage = 'ct-mp';
+my $dir_storage_path = '/var/lib/ct-mp-store';
+my $bind_source = '/var/lib/ct-bind';
+my $bind_marker = 'test';
+
+my $test_container = {
+ vmid => $client->get('/cluster/nextid'),
+ node => $node,
+ ostemplate => CONTAINER_TEMPLATE,
+ rootfs => 'local-lvm:2',
+ storage => 'local-lvm',
+ memory => 256,
+ unprivileged => 1,
+ features => 'nesting=1',
+ hostname => 'ct-mountpoints',
+ mp0 => 'local-lvm:1,mp=/data',
+ mp1 => "$bind_source,mp=/bind",
+ mp2 => "$dir_storage:1,mp=/dirstore",
+};
+
+my $vmid = $test_container->{vmid};
+my $mp0_volid;
+
+my $volume_exists = sub {
+ my ($storage, $volid) = @_;
+ my $content = $client->get("/nodes/$node/storage/$storage/content", { vmid => $vmid });
+ return grep { $_->{volid} eq $volid } $content->@*;
+};
+
+Proxmox::Test::Testcase->new('pve_ct_mountpoints')->with([{}])->step(
+ 'create directory storage',
+ sub {
+ $pve->run_command("mkdir -p $dir_storage_path");
+ $client->post(
+ '/storage',
+ {
+ storage => $dir_storage,
+ type => 'dir',
+ path => $dir_storage_path,
+ content => 'rootdir',
+ },
+ );
+ },
+ sub {
+ eval { $client->delete("/storage/$dir_storage") };
+ $pve->run_command("rm -rf $dir_storage_path");
+ },
+)->step(
+ 'create bind mount source',
+ sub {
+ $pve->run_command("mkdir -p $bind_source && echo $bind_marker > $bind_source/marker");
+ },
+ sub {
+ $pve->run_command("rm -rf $bind_source");
+ },
+)->step(
+ 'create container',
+ sub {
+ my $upid = $client->post("/nodes/$node/lxc", $test_container);
+ $pve->task_ok($upid, "created container $vmid")
+ or die "creating container $vmid failed\n";
+ },
+ sub {
+ $pve->destroy_guest('lxc', $vmid);
+ },
+)->step(
+ 'verify mountpoint config',
+ sub {
+ my $conf = $client->get("/nodes/$node/lxc/$vmid/config", {});
+
+ like(
+ $conf->{mp0} // '',
+ qr/^local-lvm:[^,]+,mp=\/data\b/,
+ 'mp0 is a volume on local-lvm',
+ );
+ like($conf->{mp1} // '', qr/^\Q$bind_source\E,mp=\/bind\b/, 'mp1 is a bind mount');
+ like(
+ $conf->{mp2} // '',
+ qr/^\Q$dir_storage\E:[^,]+,mp=\/dirstore\b/,
+ "mp2 is a volume on $dir_storage",
+ );
+
+ ($mp0_volid) = ($conf->{mp0} // '') =~ m/^([^,]+)/;
+ },
+)->step(
+ 'start container',
+ sub {
+ $pve->start_guest('lxc', $vmid);
+ pass("container $vmid reports status 'running'");
+ },
+)->step(
+ 'verify mountpoints inside container',
+ sub {
+ for my $path ('/data', '/bind', '/dirstore') {
+ is(
+ $pve->ct_exec($vmid, "findmnt -n -o TARGET $path"),
+ "$path\n",
+ "$path is mounted",
+ );
+ }
+
+ is(
+ $pve->ct_exec($vmid, 'cat /bind/marker'),
+ "$bind_marker\n",
+ 'bind mount shows the host file',
+ );
+
+ $pve->ct_exec($vmid, "sh -c 'echo e2e > /data/marker'");
+ is($pve->ct_exec($vmid, 'cat /data/marker'), "e2e\n", 'volume mountpoint is writable');
+ },
+)->step(
+ 'resize volume mountpoint online',
+ sub {
+ my $upid =
+ $client->put("/nodes/$node/lxc/$vmid/resize", { disk => 'mp0', size => '+1G' });
+ $pve->task_ok($upid, 'resized mp0 by 1 GiB') or die "resizing mp0 failed\n";
+
+ my $conf = $client->get("/nodes/$node/lxc/$vmid/config", {});
+ like($conf->{mp0} // '', qr/\bsize=2G\b/, 'config reports the new mp0 size');
+
+ # File system on the mountpoint grows as part of the resize, check that the container sees
+ # the change right away.
+ my ($size_mib) = $pve->ct_exec($vmid, 'df -B1M --output=size /data') =~ m/(\d+)\s*$/;
+ cmp_ok($size_mib // 0, '>', 1024, 'file system on /data has grown beyond 1 GiB');
+ },
+)->step(
+ 'stop container',
+ sub {
+ $pve->stop_guest('lxc', $vmid);
+ },
+)->step(
+ 'detach and delete volume mountpoint',
+ sub {
+ $client->put("/nodes/$node/lxc/$vmid/config", { delete => 'mp0' });
+
+ my $conf = $client->get("/nodes/$node/lxc/$vmid/config", {});
+ ok(!defined($conf->{mp0}), 'mp0 is gone from the config');
+ my ($unused) = grep { m/^unused\d+$/ && $conf->{$_} eq $mp0_volid } keys $conf->%*;
+ ok(defined($unused), 'the mp0 volume is kept as an unused volume');
+
+ $client->put("/nodes/$node/lxc/$vmid/config", { delete => $unused })
+ if defined($unused);
+
+ $conf = $client->get("/nodes/$node/lxc/$vmid/config", {});
+ ok(!(grep { m/^unused\d+$/ } keys $conf->%*), 'no unused volume is left in the config');
+ ok(!$volume_exists->('local-lvm', $mp0_volid), 'the mp0 volume is deleted');
+ },
+)->step(
+ 'destroy container with unavailable mountpoint storage',
+ sub {
+ $client->delete("/storage/$dir_storage");
+
+ my $upid = $client->delete("/nodes/$node/lxc/$vmid", {});
+ my $status = $pve->wait_for_task($upid, 1);
+ like(
+ $status,
+ qr/^WARNINGS: \d+$/,
+ 'destroying the container succeeds with warnings about the unavailable storage',
+ );
+
+ ok(!$pve->guest_exists('lxc', $vmid), "container $vmid is gone");
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index 391eb09..79841ca 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -151,6 +151,15 @@ TestcaseConfig(
],
tag: "extended",
),
+ Testcase(
+ name: "pve_ct_mountpoints",
+ run: "./tests/pve_ct_mountpoints.pl",
+ timeout: 600,
+ instances: [
+ "pve1",
+ ],
+ tag: "extended",
+ ),
Testcase(
name: "pve_ha_resource_config",
run: "./tests/pve_ha_resource_config.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 16/38] tests: container: verify snapshots, clones, templates
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (14 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 15/38] tests: container: verify volume and bind mountpoints Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 17/38] tests: container: verify device passthrough Michael Köppl
` (22 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Snapshot a running container and roll it back with 'start' set and
create a full clone, which must keep the data but get its own hostname
and machine ID. Also create a linked clone of a template.
On LVM-thin, a linked clone does not depend on its origin, so the
template can be destroyed while the clone exists. Check that the clone
keeps working afterwards.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_ct_snapshot_clone.pl | 267 +++++++++++++++++++++++++++++++++
tests/testcases.ron | 9 ++
2 files changed, 276 insertions(+)
create mode 100755 tests/pve_ct_snapshot_clone.pl
diff --git a/tests/pve_ct_snapshot_clone.pl b/tests/pve_ct_snapshot_clone.pl
new file mode 100755
index 0000000..e9119e8
--- /dev/null
+++ b/tests/pve_ct_snapshot_clone.pl
@@ -0,0 +1,267 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_ct_snapshot_clone> - Test snapshots, clones and templates of a container.
+
+=head1 DESCRIPTION
+
+This testcase creates an unprivileged container on local-lvm, writes a marker
+file and takes a snapshot of the running container. After changing the marker,
+it rolls the snapshot back and verifies that the original content is restored,
+then deletes the snapshot.
+
+Afterwards, it creates a full clone and checks that the clone carries the data
+but gets its own hostname and machine ID. Finally, it converts the container to
+a template and creates a linked clone backed by the template's base volume. The
+template can then be destroyed and the linked clone must keep its data.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pve_instance);
+use Proxmox::Test::Testcase;
+use Proxmox::Test::Util qw(poll_until);
+
+my $pve = get_pve_instance('pve1');
+my $node = $pve->get_nodename();
+my $client = $pve->client();
+
+my $snapname = 'e2esnap';
+
+my $test_container = {
+ vmid => $client->get('/cluster/nextid'),
+ node => $node,
+ ostemplate => CONTAINER_TEMPLATE,
+ rootfs => 'local-lvm:2',
+ storage => 'local-lvm',
+ memory => 256,
+ unprivileged => 1,
+ features => 'nesting=1',
+ hostname => 'ct-snapshot',
+};
+
+my $vmid = $test_container->{vmid};
+my $full_clone = { vmid => undef, hostname => 'ct-clone-full' };
+my $linked_clone = { vmid => undef, hostname => 'ct-clone-linked' };
+my $machine_id;
+
+my $snapshot_names = sub {
+ my $snapshots = $client->get("/nodes/$node/lxc/$vmid/snapshot", {});
+ return map { $_->{name} } grep { $_->{name} ne 'current' } $snapshots->@*;
+};
+
+my $create_clone = sub {
+ my ($clone, $full) = @_;
+
+ $clone->{vmid} = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/lxc/$vmid/clone",
+ {
+ newid => $clone->{vmid},
+ hostname => $clone->{hostname},
+ full => $full,
+ ($full ? (storage => 'local-lvm') : ()),
+ },
+ );
+ $pve->task_ok($upid, ($full ? 'full' : 'linked') . " clone $clone->{vmid} created")
+ or die "cloning container $vmid failed\n";
+};
+
+Proxmox::Test::Testcase->new('pve_ct_snapshot_clone')->with([{}])->step(
+ 'create container',
+ sub {
+ my $upid = $client->post("/nodes/$node/lxc", $test_container);
+ $pve->task_ok($upid, "created container $vmid")
+ or die "creating container $vmid failed\n";
+ },
+ sub {
+ $pve->destroy_guest('lxc', $vmid);
+ },
+)->step(
+ 'start container and write marker',
+ sub {
+ $pve->start_guest('lxc', $vmid);
+ $pve->ct_exec($vmid, "sh -c 'echo before-snapshot > /root/marker'");
+
+ # Required later to check if the full clone has a different machine ID
+ $machine_id = $pve->ct_exec($vmid, 'cat /etc/machine-id');
+ },
+)->step(
+ 'create snapshot of running container',
+ sub {
+ my $upid = $client->post("/nodes/$node/lxc/$vmid/snapshot", { snapname => $snapname });
+ $pve->task_ok($upid, "created snapshot '$snapname'")
+ or die "creating snapshot '$snapname' failed\n";
+
+ ok((grep { $_ eq $snapname } $snapshot_names->()), "snapshot '$snapname' is listed");
+ },
+)->step(
+ 'rollback snapshot',
+ sub {
+ $pve->ct_exec($vmid, "sh -c 'echo after-snapshot > /root/marker'");
+
+ # Rolling back stops the running container, 'start' brings it back afterwards in a separate
+ # task.
+ my $upid =
+ $client->post("/nodes/$node/lxc/$vmid/snapshot/$snapname/rollback", { start => 1 });
+ $pve->task_ok($upid, "rolled back to snapshot '$snapname'")
+ or die "rolling back snapshot '$snapname' failed\n";
+
+ my $start_upid = $pve->find_spawned_task($upid, 'vzstart');
+ $pve->task_ok($start_upid, "started container $vmid after the rollback")
+ or die "starting container $vmid after the rollback failed\n";
+
+ poll_until(
+ sub {
+ $client->get("/nodes/$node/lxc/$vmid/status/current", {})->{status} eq
+ 'running';
+ },
+ # A status query can take some time after teh start, so allow more than usual.
+ timeout => 120,
+ description => "container $vmid to run again after the rollback",
+ );
+ is(
+ $pve->ct_exec($vmid, 'cat /root/marker'),
+ "before-snapshot\n",
+ 'rollback restored the content from the time of the snapshot',
+ );
+ },
+)->step(
+ 'delete snapshot',
+ sub {
+ my $upid = $client->delete("/nodes/$node/lxc/$vmid/snapshot/$snapname", {});
+ $pve->task_ok($upid, "deleted snapshot '$snapname'")
+ or die "deleting snapshot '$snapname' failed\n";
+
+ ok(!(grep { $_ eq $snapname } $snapshot_names->()), "snapshot '$snapname' is gone");
+ },
+)->step(
+ 'stop container',
+ sub {
+ $pve->stop_guest('lxc', $vmid);
+ },
+)->step(
+ 'create full clone',
+ sub {
+ $create_clone->($full_clone, 1);
+ },
+ sub {
+ $pve->destroy_guest('lxc', $full_clone->{vmid}) if defined($full_clone->{vmid});
+ },
+)->step(
+ 'verify full clone',
+ sub {
+ my $id = $full_clone->{vmid};
+
+ my $conf = $client->get("/nodes/$node/lxc/$id/config", {});
+ is($conf->{hostname}, $full_clone->{hostname}, 'full clone has its own hostname');
+ like(
+ $conf->{rootfs} // '',
+ qr/^local-lvm:vm-$id-disk-\d+\b/,
+ 'full clone has own rootfs',
+ );
+
+ $pve->start_guest('lxc', $id);
+ is(
+ $pve->ct_exec($id, 'cat /root/marker'),
+ "before-snapshot\n",
+ 'full clone has the data',
+ );
+ is(
+ $pve->ct_exec($id, 'cat /etc/hostname'),
+ "$full_clone->{hostname}\n",
+ 'hostname is set up inside the full clone',
+ );
+
+ # Cloning clears the machine ID, so the guest generates a new one on its first boot.
+ my $clone_machine_id = $pve->ct_exec($id, 'cat /etc/machine-id');
+ like($clone_machine_id, qr/^[0-9a-f]{32}$/, 'full clone has a machine ID');
+ isnt($clone_machine_id, $machine_id, 'full clone has a different machine ID');
+
+ $pve->stop_guest('lxc', $id);
+ },
+)->step(
+ 'convert container to template',
+ sub {
+ # Conversion runs in a worker task, but the API call does not return its task ID.
+ $client->post("/nodes/$node/lxc/$vmid/template", {});
+
+ my $conf;
+ eval {
+ poll_until(
+ sub {
+ $conf = $client->get("/nodes/$node/lxc/$vmid/config", {});
+ return $conf->{template};
+ },
+ timeout => 60,
+ interval => 2,
+ description => "container $vmid to be converted to a template",
+ );
+ };
+ is($conf->{template}, 1, 'container is marked as template');
+ like(
+ $conf->{rootfs} // '',
+ qr/^local-lvm:base-$vmid-disk-\d+\b/,
+ 'template rootfs was converted to a base volume',
+ );
+ },
+)->step(
+ 'create linked clone',
+ sub {
+ $create_clone->($linked_clone, 0);
+ },
+ sub {
+ $pve->destroy_guest('lxc', $linked_clone->{vmid}) if defined($linked_clone->{vmid});
+ },
+)->step(
+ 'verify linked clone',
+ sub {
+ my $id = $linked_clone->{vmid};
+
+ my $conf = $client->get("/nodes/$node/lxc/$id/config", {});
+ my ($volname) = ($conf->{rootfs} // '') =~ m/^local-lvm:([^,]+)/
+ or die "linked clone $id has no rootfs on local-lvm\n";
+
+ my $vgname = $client->get('/storage/local-lvm', {})->{vgname};
+ my ($origin) = $pve->run_command("lvs --noheadings -o origin $vgname/$volname");
+ like(
+ $origin // '',
+ qr/^\s*base-$vmid-disk-\d+\s*$/,
+ 'linked clone rootfs is a thin snapshot of the template base volume',
+ );
+
+ $pve->start_guest('lxc', $id);
+ is(
+ $pve->ct_exec($id, 'cat /root/marker'),
+ "before-snapshot\n",
+ 'linked clone has the data',
+ );
+ $pve->stop_guest('lxc', $id);
+ },
+)->step(
+ 'linked clone survives destroying the template',
+ sub {
+ # A snapshot on LVM-thin does not depend on its origin volume, so the template can be
+ # destroyed while linked clones of it exist and they must keep working.
+ my $upid = $client->delete("/nodes/$node/lxc/$vmid", {});
+ $pve->task_ok($upid, "destroyed template $vmid")
+ or die "destroying the template failed\n";
+
+ my $id = $linked_clone->{vmid};
+ $pve->start_guest('lxc', $id);
+ is(
+ $pve->ct_exec($id, 'cat /root/marker'),
+ "before-snapshot\n",
+ 'linked clone still has the data after the template is gone',
+ );
+ $pve->stop_guest('lxc', $id);
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index 79841ca..c0369e4 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -160,6 +160,15 @@ TestcaseConfig(
],
tag: "extended",
),
+ Testcase(
+ name: "pve_ct_snapshot_clone",
+ run: "./tests/pve_ct_snapshot_clone.pl",
+ timeout: 600,
+ instances: [
+ "pve1",
+ ],
+ tag: "extended",
+ ),
Testcase(
name: "pve_ha_resource_config",
run: "./tests/pve_ha_resource_config.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 17/38] tests: container: verify device passthrough
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (15 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 16/38] tests: container: verify snapshots, clones, templates Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 18/38] plans,scripts: build a minimal OCI image onto pve1 Michael Köppl
` (21 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Pass two loop devices through to a container, one of them with
'deny-write' set, and check mode, read and write access inside the
container. Run it for an unprivileged container and for a privileged
one with the 'unmanaged' OS type.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_ct_device_passthrough.pl | 162 +++++++++++++++++++++++++++++
tests/testcases.ron | 9 ++
2 files changed, 171 insertions(+)
create mode 100755 tests/pve_ct_device_passthrough.pl
diff --git a/tests/pve_ct_device_passthrough.pl b/tests/pve_ct_device_passthrough.pl
new file mode 100755
index 0000000..15ca4ab
--- /dev/null
+++ b/tests/pve_ct_device_passthrough.pl
@@ -0,0 +1,162 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_ct_device_passthrough> - Test passing host devices through to a container.
+
+=head1 DESCRIPTION
+
+This testcase attaches two loop devices on the host, each backed by a small
+scratch file, and passes them through to a container, the second one with write
+access denied. It verifies that both devices show up inside the container with
+the configured mode, that data written to the writable one can be read back,
+and that writes to the other one are refused while reads still work.
+
+Loop devices are used because a container cannot access them unless they are
+passed through and they are easy to set up.
+
+It runs once for an unprivileged container and once for a privileged container
+with the 'unmanaged' OS type, for which LXC ships no OS-specific config.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pve_instance);
+use Proxmox::Test::Testcase;
+
+my $pve = get_pve_instance('pve1');
+my $node = $pve->get_nodename();
+my $client = $pve->client();
+
+my $backing_dir = '/var/lib/ct-passthrough';
+
+# The device type and number of $dev as reported by stat on the host.
+my $host_device_id = sub {
+ my ($dev) = @_;
+
+ my ($out) = $pve->run_command("stat -c '%F %t:%T' $dev");
+ chomp($out);
+ return $out;
+};
+
+Proxmox::Test::Testcase->new('pve_ct_device_passthrough')->with([
+ {
+ name => 'unprivileged',
+ unprivileged => 1,
+ },
+ {
+ name => 'privileged-unmanaged',
+ unprivileged => 0,
+ ostype => 'unmanaged',
+ },
+])->step(
+ 'attach loop devices on the host',
+ sub {
+ my ($testcase) = @_;
+
+ $pve->run_command("mkdir -p $backing_dir");
+ for my $kind ('writable', 'readonly') {
+ my $file = "$backing_dir/$kind.img";
+ $pve->run_command("truncate -s 1M $file");
+ my ($dev) = $pve->run_command("losetup --find --show $file");
+ chomp($dev);
+ $testcase->{"${kind}_dev"} = $dev;
+ }
+ },
+ sub {
+ my ($testcase) = @_;
+
+ for my $kind ('writable', 'readonly') {
+ my $dev = $testcase->{"${kind}_dev"} // next;
+ $pve->run_command("losetup --detach $dev");
+ }
+ $pve->run_command("rm -rf $backing_dir");
+ },
+)->step(
+ 'create container with passthrough devices',
+ sub {
+ my ($testcase) = @_;
+
+ $testcase->{vmid} = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/lxc",
+ {
+ vmid => $testcase->{vmid},
+ ostemplate => CONTAINER_TEMPLATE,
+ rootfs => 'local-lvm:2',
+ storage => 'local-lvm',
+ memory => 256,
+ unprivileged => $testcase->{unprivileged},
+ features => 'nesting=1',
+ hostname => "ct-dev-$testcase->{name}",
+ ($testcase->{ostype} ? (ostype => $testcase->{ostype}) : ()),
+ dev0 => "$testcase->{writable_dev},mode=0660",
+ dev1 => "$testcase->{readonly_dev},mode=0640,deny-write=1",
+ },
+ );
+ $pve->task_ok($upid, "created $testcase->{name} container $testcase->{vmid}")
+ or die "creating container failed\n";
+ },
+ sub {
+ my ($testcase) = @_;
+ $pve->destroy_guest('lxc', $testcase->{vmid}) if defined($testcase->{vmid});
+ },
+)->step(
+ 'start container',
+ sub {
+ my ($testcase) = @_;
+ my $vmid = $testcase->{vmid};
+
+ $pve->start_guest('lxc', $vmid);
+ },
+)->step(
+ 'verify devices inside container',
+ sub {
+ my ($testcase) = @_;
+ my $vmid = $testcase->{vmid};
+ my $writable_dev = $testcase->{writable_dev};
+ my $readonly_dev = $testcase->{readonly_dev};
+
+ is(
+ $pve->ct_exec($vmid, "stat -c '%F %t:%T %a' $writable_dev"),
+ $host_device_id->($writable_dev) . " 660\n",
+ "$writable_dev is passed through with the configured mode",
+ );
+ is(
+ $pve->ct_exec($vmid, "stat -c '%F %t:%T %a' $readonly_dev"),
+ $host_device_id->($readonly_dev) . " 640\n",
+ "$readonly_dev is passed through with the configured mode",
+ );
+
+ my $data = 'test';
+
+ ok(
+ eval { $pve->ct_exec($vmid, "sh -c 'printf $data > $writable_dev'"); 1 },
+ "$writable_dev is writable",
+ );
+ is(
+ $pve->ct_exec($vmid, "head -c " . length($data) . " $writable_dev"),
+ $data,
+ "data written to $writable_dev can be read back",
+ );
+
+ # Opening for writing should already fail because the device cgroup checks access when the
+ # device is opened.
+ ok(
+ !eval { $pve->ct_exec($vmid, "sh -c 'printf $data > $readonly_dev'"); 1 },
+ "writing to $readonly_dev is denied",
+ );
+ is(
+ $pve->ct_exec($vmid, "sh -c 'head -c 4 $readonly_dev | wc -c'"),
+ "4\n",
+ "$readonly_dev is still readable",
+ );
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index c0369e4..1d5b7ad 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -169,6 +169,15 @@ TestcaseConfig(
],
tag: "extended",
),
+ Testcase(
+ name: "pve_ct_device_passthrough",
+ run: "./tests/pve_ct_device_passthrough.pl",
+ timeout: 600,
+ instances: [
+ "pve1",
+ ],
+ tag: "extended",
+ ),
Testcase(
name: "pve_ha_resource_config",
run: "./tests/pve_ha_resource_config.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 18/38] plans,scripts: build a minimal OCI image onto pve1
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (16 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 17/38] tests: container: verify device passthrough Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 19/38] tests: container: verify application containers from OCI images Michael Köppl
` (20 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Build the image for the OCI container test during setup instead of
pulling one from a registry. The test then depends neither on the
registry nor on how a third-party image behaves.
The image consists of a static BusyBox and an entrypoint that only
exits on the image's stop signal. Its config sets the fields that
pve-container takes over into the container config, so the entrypoint
and command, environment, user, working directory, stop signal and a
volume.
The image is built on the test runner, so umoci and busybox-static
become dependencies of the test package.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
debian/control | 4 ++-
plans/complete.ron | 1 +
scripts/build-oci-image.sh | 69 ++++++++++++++++++++++++++++++++++++++
3 files changed, 73 insertions(+), 1 deletion(-)
create mode 100755 scripts/build-oci-image.sh
diff --git a/debian/control b/debian/control
index b7abfdf..56a1185 100644
--- a/debian/control
+++ b/debian/control
@@ -11,7 +11,9 @@ Depends: ${misc:Depends}, ${perl:Depends},
libjson-perl,
libio-socket-ssl-perl,
libnet-openssh-perl,
- jq
+ busybox-static,
+ jq,
+ umoci
Description: Proxmox End-to-End Tests and Helpers
This package contains the end-to-end test suite and Perl helper
modules for Proxmox projects. It requires proxmox-test-runner
diff --git a/plans/complete.ron b/plans/complete.ron
index 75cd5b1..dd299fe 100644
--- a/plans/complete.ron
+++ b/plans/complete.ron
@@ -15,6 +15,7 @@ Plan(
post_setup: [
"./scripts/update-packages.sh pve staging",
"./scripts/download-container-image.sh local debian-12-standard_12.12-1_amd64.tar.zst",
+ "./scripts/build-oci-image.sh local e2e-oci",
"./scripts/setup-pebble.sh",
],
),
diff --git a/scripts/build-oci-image.sh b/scripts/build-oci-image.sh
new file mode 100755
index 0000000..81ddabc
--- /dev/null
+++ b/scripts/build-oci-image.sh
@@ -0,0 +1,69 @@
+#!/usr/bin/bash
+# Builds a minimal OCI image for the application container tests and stores it as a container
+# template. The image sets various image config fields that pve-container takes over into the
+# container config. It also only exits on the configured stop signal.
+set -e
+
+SSH_KEY=$(jq -r '.ssh_private_keyfile' "$CONFIG_PATH")
+HOST=$(jq -r '.host' "$CONFIG_PATH")
+USER=$(jq -r '.ssh_user' "$CONFIG_PATH")
+
+STORAGE=$1
+FILENAME=$2
+
+workdir=$(mktemp -d)
+trap 'rm -rf "$workdir"' EXIT
+umask 022
+
+# Root filesystem
+rootfs="$workdir/rootfs"
+mkdir -p "$rootfs"/{bin,dev,etc,home/e2e,proc,run,sys,tmp}
+chmod 1777 "$rootfs/tmp"
+cp "$(command -v busybox)" "$rootfs/bin/"
+for applet in $("$rootfs/bin/busybox" --list); do
+ [ "$applet" = busybox ] || ln -s busybox "$rootfs/bin/$applet"
+done
+
+# User database for the image's User field
+cat > "$rootfs/etc/passwd" <<'PASSWD'
+root:x:0:0:root:/root:/bin/sh
+e2e:x:1000:1000:e2e:/home/e2e:/bin/sh
+PASSWD
+cat > "$rootfs/etc/group" <<'GROUP'
+root:x:0:
+e2e:x:1000:
+GROUP
+
+# Entrypoint that exits on the image's stop signal only, so that stopping the container with any
+# other signal times out. The shell ignores all signals it has no trap for
+cat > "$rootfs/entrypoint.sh" <<'ENTRYPOINT'
+#!/bin/sh
+trap 'exit 0' USR1
+while :; do
+ sleep 1
+done
+ENTRYPOINT
+chmod 755 "$rootfs/entrypoint.sh"
+
+umoci init --layout "$workdir/layout"
+umoci new --image "$workdir/layout:e2e"
+umoci unpack --rootless --image "$workdir/layout:e2e" "$workdir/bundle"
+cp -a "$rootfs/." "$workdir/bundle/rootfs/"
+umoci repack --image "$workdir/layout:e2e" "$workdir/bundle"
+umoci config --image "$workdir/layout:e2e" \
+ --config.entrypoint /entrypoint.sh \
+ --config.cmd e2e-cmd \
+ --config.env PATH=/bin \
+ --config.env E2E_IMAGE_ENV=from-image \
+ --config.stopsignal SIGUSR1 \
+ --config.user e2e:e2e \
+ --config.workingdir /home/e2e \
+ --config.volume /data
+umoci gc --layout "$workdir/layout"
+
+# Required for pve-container to detect the archive as OCI format
+tar -C "$workdir/layout" -cf "$workdir/$FILENAME.tar" oci-layout index.json blobs
+
+ssh -i "$SSH_KEY" "$USER"@"$HOST" -o StrictHostKeyChecking=no \
+ "cat > \"\$(pvesm path '$STORAGE:vztmpl/$FILENAME.tar')\"" \
+ < "$workdir/$FILENAME.tar"
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 19/38] tests: container: verify application containers from OCI images
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (17 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 18/38] plans,scripts: build a minimal OCI image onto pve1 Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 20/38] tests: vm: verify snapshots, clones, templates Michael Köppl
` (19 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Create a container from the OCI image built during setup and check
that the image's entrypoint, environment, user, working directory
and stop signal end up in the config, and that the host manages the
network.
Then check that init runs the entrypoint with the image's user,
working directory and environment, including a variable added
through the 'env' property, and that the image's volume path
exists. Since 'pct exec' does not get variables that are meant for
init only, the test reads them from /proc/1/environ. Finally, shut
the container down. The image's init only exits on the image's stop
signal, so the shutdown fails if that signal is not taken over from
the OCI image.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_ct_oci.pl | 190 ++++++++++++++++++++++++++++++++++++++++++++
tests/testcases.ron | 9 +++
2 files changed, 199 insertions(+)
create mode 100755 tests/pve_ct_oci.pl
diff --git a/tests/pve_ct_oci.pl b/tests/pve_ct_oci.pl
new file mode 100755
index 0000000..b987a8a
--- /dev/null
+++ b/tests/pve_ct_oci.pl
@@ -0,0 +1,190 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_ct_oci> - Test an application container created from an OCI image.
+
+=head1 DESCRIPTION
+
+This testcase creates an unprivileged container from the minimal OCI image that
+the setup plan builds and puts into pve1's local storage. It verifies that the
+image's entrypoint and command, environment, user, working directory and stop
+signal are taken over into the container config and that the network is managed
+by the host, since the image has no network management of its own.
+
+It then adds an environment variable through the API and starts the container.
+It verifies that the image's entrypoint runs as init with the image's user,
+working directory and environment, including the added variable, that the
+image's volume path exists, and that the container is reachable from the node
+at its host-configured address. Finally, it shuts the container down. The
+image's init only exits on the image's stop signal.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(get_pve_instance);
+use Proxmox::Test::Testcase;
+use Proxmox::Test::Util qw(poll_until);
+
+my $pve = get_pve_instance('pve1');
+my $node = $pve->get_nodename();
+my $client = $pve->client();
+
+my $ct_ip = '10.250.252.10';
+my $node_ip = '10.250.252.1';
+
+my $test_container = {
+ vmid => $client->get('/cluster/nextid'),
+ node => $node,
+ ostemplate => 'local:vztmpl/e2e-oci.tar',
+ rootfs => 'local-lvm:1',
+ storage => 'local-lvm',
+ memory => 256,
+ unprivileged => 1,
+ hostname => 'ct-oci',
+ net0 => "name=eth0,bridge=vmbr0,ip=$ct_ip/24",
+};
+
+my $vmid = $test_container->{vmid};
+my $env_marker_key = 'MARKER_ENV';
+my $env_marker_value = 'test';
+
+Proxmox::Test::Testcase->new('pve_ct_oci')->with([{}])->step(
+ 'create container from OCI image',
+ sub {
+ my $upid = $client->post("/nodes/$node/lxc", $test_container);
+ $pve->task_ok($upid, "created container $vmid from OCI image")
+ or die "creating container $vmid failed\n";
+ },
+ sub {
+ $pve->destroy_guest('lxc', $vmid);
+ },
+)->step(
+ 'verify config taken from the image',
+ sub {
+ my $conf = $client->get("/nodes/$node/lxc/$vmid/config", {});
+
+ is(
+ $conf->{entrypoint},
+ '/entrypoint.sh e2e-cmd',
+ 'entrypoint and command are taken from the image',
+ );
+ is($conf->{cmode}, 'console', 'console mode is used for the custom entrypoint');
+ # This is set by the OCI image.
+ is($conf->{env}, "PATH=/bin\0E2E_IMAGE_ENV=from-image", 'image environment is set');
+ like($conf->{net0} // '', qr/\bhost-managed=1\b/, 'network is managed by the host');
+
+ my %raw = map { $_->[0] => $_->[1] } ($conf->{lxc} // [])->@*;
+ is($raw{'lxc.init.uid'}, 1000, 'init user is resolved from the image user database');
+ is($raw{'lxc.init.gid'}, 1000, 'init group is resolved from the image group database');
+ is($raw{'lxc.init.cwd'}, '/home/e2e', 'working directory is taken from the image');
+ is($raw{'lxc.signal.halt'}, 'SIGUSR1', 'stop signal is taken from the image');
+ },
+)->step(
+ 'add environment variable',
+ sub {
+ my $env = $client->get("/nodes/$node/lxc/$vmid/config", {})->{env};
+ $client->put(
+ "/nodes/$node/lxc/$vmid/config",
+ { env => "$env\0$env_marker_key=$env_marker_value" },
+ );
+
+ my $conf = $client->get("/nodes/$node/lxc/$vmid/config", {});
+ like(
+ $conf->{env} // '',
+ qr/\0$env_marker_key=$env_marker_value$/,
+ 'config has the added variable',
+ );
+ },
+)->step(
+ 'add node address on vmbr0',
+ sub {
+ $pve->run_command("ip addr add $node_ip/24 dev vmbr0");
+ },
+ sub {
+ $pve->run_command("ip addr del $node_ip/24 dev vmbr0 2>/dev/null; true");
+ },
+)->step(
+ 'start container',
+ sub {
+ $pve->start_guest('lxc', $vmid);
+ },
+)->step(
+ 'verify the image entrypoint runs as init',
+ sub {
+ is(
+ $pve->ct_exec($vmid, 'cat /proc/1/cmdline'),
+ "/bin/sh\0/entrypoint.sh\0e2e-cmd\0",
+ 'init runs the entrypoint with the command as argument',
+ );
+
+ my $status = $pve->ct_exec($vmid, 'cat /proc/1/status');
+ like($status, qr/^Uid:\s+1000\s+1000\s+1000\s+1000$/m, 'init runs as the image user');
+ like($status, qr/^Gid:\s+1000\s+1000\s+1000\s+1000$/m,
+ 'init runs with the image group');
+
+ is(
+ $pve->ct_exec($vmid, 'readlink /proc/1/cwd'),
+ "/home/e2e\n",
+ 'init runs in the working directory',
+ );
+ },
+)->step(
+ 'verify the environment of init',
+ sub {
+ my %env =
+ map { split(/=/, $_, 2) } split(/\0/, $pve->ct_exec($vmid, 'cat /proc/1/environ'));
+
+ is($env{E2E_IMAGE_ENV}, 'from-image', 'init has the image environment');
+ is($env{$env_marker_key}, $env_marker_value, 'init has the added variable');
+ },
+)->step(
+ 'verify the image volume path exists',
+ sub {
+ ok(
+ eval { $pve->ct_exec($vmid, 'test -d /data'); 1 },
+ 'directory for the image volume exists',
+ );
+ },
+)->step(
+ 'verify host-managed network',
+ sub {
+ poll_until(
+ sub {
+ $pve->ct_exec($vmid, 'ip -4 addr show dev eth0') =~ m/\binet \Q$ct_ip\E\/24\b/;
+ },
+ timeout => 30,
+ interval => 2,
+ description => "eth0 in container $vmid to get $ct_ip",
+ );
+ pass("host configured $ct_ip on eth0 in container $vmid");
+
+ poll_until(
+ sub { $pve->node_can_ping($ct_ip) },
+ timeout => 30,
+ interval => 3,
+ description => "container $vmid to be pingable from the node",
+ );
+ ok($pve->node_can_ping($ct_ip), "node can ping container $vmid");
+ },
+)->step(
+ 'shutdown container',
+ sub {
+ my $upid = $client->post("/nodes/$node/lxc/$vmid/status/shutdown", { timeout => 30 });
+ $pve->task_ok($upid, "shut down container $vmid")
+ or die "shutting down container $vmid failed\n";
+
+ is(
+ $client->get("/nodes/$node/lxc/$vmid/status/current", {})->{status},
+ 'stopped',
+ "container $vmid reports status 'stopped' after shutdown",
+ );
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index 1d5b7ad..afbaa5b 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -178,6 +178,15 @@ TestcaseConfig(
],
tag: "extended",
),
+ Testcase(
+ name: "pve_ct_oci",
+ run: "./tests/pve_ct_oci.pl",
+ timeout: 600,
+ instances: [
+ "pve1",
+ ],
+ tag: "extended",
+ ),
Testcase(
name: "pve_ha_resource_config",
run: "./tests/pve_ha_resource_config.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 20/38] tests: vm: verify snapshots, clones, templates
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (18 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 19/38] tests: container: verify application containers from OCI images Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 21/38] tests: vm: verify adding, resizing, and removing disks Michael Köppl
` (18 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Take a snapshot of a running VM including its RAM and check that a
rollback restores the config from the time of the snapshot and
resumes the VM right away, and that deleting the snapshot removes
the saved RAM.
Clone the running VM and a template made from it, and check that
each clone gets its own disk and a new MAC address, SMBIOS UUID and
VM generation ID, so that the clones can be told apart. Check that the
template itself can no longer be started.
The VM has no OS, so the test does not check disk contents.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_vm_snapshot_clone.pl | 282 +++++++++++++++++++++++++++++++++
tests/testcases.ron | 9 ++
2 files changed, 291 insertions(+)
create mode 100755 tests/pve_vm_snapshot_clone.pl
diff --git a/tests/pve_vm_snapshot_clone.pl b/tests/pve_vm_snapshot_clone.pl
new file mode 100755
index 0000000..69eee60
--- /dev/null
+++ b/tests/pve_vm_snapshot_clone.pl
@@ -0,0 +1,282 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_vm_snapshot_clone> - Test snapshots, clones and templates of a VM.
+
+=head1 DESCRIPTION
+
+This testcase creates a VM with a disk on local-lvm and a network device, starts
+it and takes a snapshot with RAM. After changing, adding and removing options and
+leaving a change pending, it rolls the snapshot back and verifies that the VM
+runs again right away (snapshot with RAM), that the config from the time of the
+snapshot is restored with a new VM generation ID, and that the pending change is
+discarded. Deleting the snapshot must also remove the saved VM state.
+
+Afterwards, it creates a full clone of the running VM and checks that the clone
+has its own disk and its own identity (i.e. MAC address, SMBIOS UUID, VM
+generation ID). Finally, it converts the VM to a template and creates a linked
+clone of the template, which must get its own identity as well and start.
+
+The testcase does not check disk contents, as the VM has no operating system.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(get_pve_instance);
+use Proxmox::Test::Testcase;
+
+my $pve = get_pve_instance('pve1');
+my $node = $pve->get_nodename();
+my $client = $pve->client();
+
+my $snapname = 'e2esnap';
+
+my $test_vm = {
+ vmid => $client->get('/cluster/nextid'),
+ name => 'vm-snapshot',
+ memory => 256,
+ scsihw => 'virtio-scsi-pci',
+ scsi0 => 'local-lvm:1',
+ net0 => 'virtio,bridge=vmbr0',
+ tags => 'e2e',
+};
+
+my $vmid = $test_vm->{vmid};
+my $full_clone = { vmid => undef, name => 'vm-clone-full' };
+my $linked_clone = { vmid => undef, name => 'vm-clone-linked' };
+my $vmstate;
+my $snapshot_conf;
+
+my $vm_config = sub {
+ my ($id) = @_;
+ return $client->get("/nodes/$node/qemu/$id/config", {});
+};
+
+# The config in effect, without pending changes applied.
+my $current_config = sub {
+ return $client->get("/nodes/$node/qemu/$vmid/config", { current => 1 });
+};
+
+my $pending_changes = sub {
+ my $pending = $client->get("/nodes/$node/qemu/$vmid/pending", {});
+ return {
+ map { $_->{key} => 1 }
+ grep { defined($_->{pending}) || $_->{delete} } $pending->@*
+ };
+};
+
+my $vm_identity = sub {
+ my ($conf) = @_;
+
+ my ($mac) = ($conf->{net0} // '') =~ m/^[^=,]+=([0-9A-Fa-f:]{17})\b/;
+ my ($uuid) = ($conf->{smbios1} // '') =~ m/\buuid=([0-9a-fA-F-]+)/;
+
+ return { mac => $mac, uuid => $uuid, vmgenid => $conf->{vmgenid} };
+};
+
+my $volume_exists = sub {
+ my ($volid, $owner) = @_;
+
+ my ($storage) = $volid =~ m/^([^:]+):/;
+ my $content = $client->get("/nodes/$node/storage/$storage/content", { vmid => $owner });
+ return grep { $_->{volid} eq $volid } $content->@*;
+};
+
+my $verify_clone = sub {
+ my ($clone, $source_conf) = @_;
+ my $id = $clone->{vmid};
+
+ my $conf = $vm_config->($id);
+ is($conf->{name}, $clone->{name}, "clone $id has its own name");
+ is($conf->{memory}, $source_conf->{memory}, "clone $id has the memory of the source");
+
+ my ($disk) = ($conf->{scsi0} // '') =~ m/^([^,]+)/;
+ my ($source_disk) = ($source_conf->{scsi0} // '') =~ m/^([^,]+)/;
+ ok(
+ defined($disk) && $disk ne $source_disk && $volume_exists->($disk, $id),
+ "clone $id has its own disk",
+ );
+
+ my $clone_identity = $vm_identity->($conf);
+ my $source_identity = $vm_identity->($source_conf);
+ for my $key (sort keys $clone_identity->%*) {
+ ok(defined($clone_identity->{$key}), "clone $id has a $key");
+ isnt($clone_identity->{$key}, $source_identity->{$key}, "clone $id has its own $key");
+ }
+
+ $pve->start_guest('qemu', $id);
+ $pve->stop_guest('qemu', $id);
+};
+
+Proxmox::Test::Testcase->new('pve_vm_snapshot_clone')->with([{}])->step(
+ 'create VM',
+ sub {
+ my $upid = $client->post("/nodes/$node/qemu", $test_vm);
+ $pve->task_ok($upid, "created VM $vmid") or die "creating VM $vmid failed\n";
+ },
+ sub {
+ $pve->destroy_guest('qemu', $vmid);
+ },
+)->step(
+ 'verify generated identity',
+ sub {
+ my $identity = $vm_identity->($vm_config->($vmid));
+
+ ok($identity->{mac}, 'a MAC address is generated for net0');
+ ok($identity->{uuid}, 'an SMBIOS UUID is generated');
+ ok($identity->{vmgenid}, 'a vmgenid is generated');
+ },
+)->step(
+ 'start VM',
+ sub {
+ $pve->start_guest('qemu', $vmid);
+ },
+)->step(
+ 'create snapshot including RAM',
+ sub {
+ my $upid = $client->post(
+ "/nodes/$node/qemu/$vmid/snapshot",
+ { snapname => $snapname, vmstate => 1 },
+ );
+ $pve->task_ok($upid, "created snapshot '$snapname' of the running VM")
+ or die "creating snapshot '$snapname' failed\n";
+
+ my ($snapshot) = grep { $_->{name} eq $snapname }
+ $client->get("/nodes/$node/qemu/$vmid/snapshot", {})->@*;
+ ok($snapshot, "snapshot '$snapname' is listed");
+ is($snapshot->{vmstate}, 1, "snapshot '$snapname' includes the RAM");
+
+ $vmstate =
+ $client->get("/nodes/$node/qemu/$vmid/snapshot/$snapname/config", {})->{vmstate};
+ ok(defined($vmstate) && $volume_exists->($vmstate, $vmid), 'RAM is saved to a volume');
+
+ $snapshot_conf = $current_config->();
+ },
+)->step(
+ 'change config after the snapshot',
+ sub {
+ # Changing the memory of a running VM without memory hotplug stays pending, the other
+ # changes apply right away.
+ $client->put(
+ "/nodes/$node/qemu/$vmid/config",
+ { name => 'vm-snapshot-changed', onboot => 1, delete => 'tags', memory => 512 },
+ );
+
+ my $conf = $current_config->();
+ is($conf->{name}, 'vm-snapshot-changed', 'name is changed');
+ is($conf->{onboot}, 1, 'onboot is added');
+ ok(!defined($conf->{tags}), 'tags are removed');
+ is_deeply($pending_changes->(), { memory => 1 }, 'memory change is pending');
+ },
+)->step(
+ 'rollback snapshot',
+ sub {
+ my ($since) = $pve->run_command('date +%s');
+ chomp($since);
+
+ my $upid = $client->post("/nodes/$node/qemu/$vmid/snapshot/$snapname/rollback", {});
+ $pve->task_ok($upid, "rolled back to snapshot '$snapname'")
+ or die "rolling back snapshot '$snapname' failed\n";
+
+ is(
+ $client->get("/nodes/$node/qemu/$vmid/status/current", {})->{status},
+ 'running',
+ 'VM runs again right after the rollback, as the snapshot includes its RAM',
+ );
+
+ my $conf = $current_config->();
+ isnt($conf->{vmgenid}, $snapshot_conf->{vmgenid}, 'rollback generates a new vmgenid');
+ my %expected = $snapshot_conf->%*;
+ # Rollback gives the VM a new generation ID, so this can never match.
+ delete $expected{$_} for qw(digest vmgenid);
+ delete $conf->{$_} for qw(digest vmgenid);
+ is_deeply(
+ $conf,
+ \%expected,
+ 'rollback restored the config from the time of the snapshot',
+ );
+ is_deeply($pending_changes->(), {}, 'rollback discarded the pending change');
+
+ $pve->wait_for_qemu_cleanup($vmid, $since);
+ },
+)->step(
+ 'delete snapshot',
+ sub {
+ my $upid = $client->delete("/nodes/$node/qemu/$vmid/snapshot/$snapname", {});
+ $pve->task_ok($upid, "deleted snapshot '$snapname'")
+ or die "deleting snapshot '$snapname' failed\n";
+
+ ok(
+ !(
+ grep { $_->{name} eq $snapname }
+ $client->get("/nodes/$node/qemu/$vmid/snapshot", {})->@*
+ ),
+ "snapshot '$snapname' is gone",
+ );
+ ok(!$volume_exists->($vmstate, $vmid), 'saved VM state is removed with the snapshot');
+ },
+)->step(
+ 'create full clone of the running VM',
+ sub {
+ $full_clone->{vmid} = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/qemu/$vmid/clone",
+ { newid => $full_clone->{vmid}, name => $full_clone->{name}, full => 1 },
+ );
+ $pve->task_ok($upid, "created full clone $full_clone->{vmid}")
+ or die "cloning VM $vmid failed\n";
+ },
+ sub {
+ $pve->destroy_guest('qemu', $full_clone->{vmid}) if defined($full_clone->{vmid});
+ },
+)->step(
+ 'verify full clone',
+ sub {
+ $verify_clone->($full_clone, $vm_config->($vmid));
+ },
+)->step(
+ 'stop VM',
+ sub {
+ $pve->stop_guest('qemu', $vmid);
+ },
+)->step(
+ 'convert VM to template',
+ sub {
+ my $upid = $client->post("/nodes/$node/qemu/$vmid/template", {});
+ $pve->task_ok($upid, "converted VM $vmid to a template")
+ or die "converting VM $vmid to a template failed\n";
+
+ is($vm_config->($vmid)->{template}, 1, "VM $vmid is marked as template");
+
+ $upid = eval { $client->post("/nodes/$node/qemu/$vmid/status/start", {}) };
+ my $status = defined($upid) ? $pve->wait_for_task($upid, 1) : "API error: $@";
+ ok(!$pve->task_successful($status), 'starting the template is refused');
+ },
+)->step(
+ 'create linked clone',
+ sub {
+ $linked_clone->{vmid} = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/qemu/$vmid/clone",
+ { newid => $linked_clone->{vmid}, name => $linked_clone->{name}, full => 0 },
+ );
+ $pve->task_ok($upid, "created linked clone $linked_clone->{vmid}")
+ or die "creating the linked clone failed\n";
+ },
+ sub {
+ $pve->destroy_guest('qemu', $linked_clone->{vmid}) if defined($linked_clone->{vmid});
+ },
+)->step(
+ 'verify linked clone',
+ sub {
+ $verify_clone->($linked_clone, $vm_config->($vmid));
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index afbaa5b..e7f8917 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -187,6 +187,15 @@ TestcaseConfig(
],
tag: "extended",
),
+ Testcase(
+ name: "pve_vm_snapshot_clone",
+ run: "./tests/pve_vm_snapshot_clone.pl",
+ timeout: 600,
+ instances: [
+ "pve1",
+ ],
+ tag: "extended",
+ ),
Testcase(
name: "pve_ha_resource_config",
run: "./tests/pve_ha_resource_config.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 21/38] tests: vm: verify adding, resizing, and removing disks
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (19 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 20/38] tests: vm: verify snapshots, clones, templates Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 22/38] plans,scripts: download a Debian cloud image onto pve1 Michael Köppl
` (17 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Check that disks on local-lvm and on a directory storage are
attached to the running VM, and that a disk added while the VM runs
is hotplugged instead of left pending. Resize it online, then remove
it again, which must unplug it and keep its volume as unused until
that is deleted explicitly.
As for containers, destroying a VM whose disk storage was removed
from the storage configuration must succeed with warnings.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_vm_disks.pl | 195 ++++++++++++++++++++++++++++++++++++++++++
tests/testcases.ron | 9 ++
2 files changed, 204 insertions(+)
create mode 100755 tests/pve_vm_disks.pl
diff --git a/tests/pve_vm_disks.pl b/tests/pve_vm_disks.pl
new file mode 100755
index 0000000..1ff7704
--- /dev/null
+++ b/tests/pve_vm_disks.pl
@@ -0,0 +1,195 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_vm_disks> - Test adding, resizing, and removing disks of a VM.
+
+=head1 DESCRIPTION
+
+This testcase creates a VM with a disk on local-lvm and a disk on a temporary
+directory storage and verifies that both are attached to the running VM. While
+the VM is running, it adds another disk on local-lvm, which must be hotplugged,
+resizes it, and removes it again, which must unplug it and keep its volume as an
+unused volume until it is deleted explicitly.
+
+Finally, the directory storage is removed from the storage configuration while
+the VM still has a disk on it. Destroying the VM must still succeed and only
+warn about the volume it could not delete.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(get_pve_instance);
+use Proxmox::Test::Testcase;
+
+my $pve = get_pve_instance('pve1');
+my $node = $pve->get_nodename();
+my $client = $pve->client();
+
+my $dir_storage = 'vm-disks';
+my $dir_storage_path = '/var/lib/vm-disks-store';
+
+my $test_vm = {
+ vmid => $client->get('/cluster/nextid'),
+ name => 'vm-disks',
+ memory => 256,
+ scsihw => 'virtio-scsi-pci',
+ scsi0 => 'local-lvm:1',
+ scsi1 => "$dir_storage:1",
+};
+
+my $vmid = $test_vm->{vmid};
+my $hotplug_volid;
+
+my $vm_config = sub {
+ return $client->get("/nodes/$node/qemu/$vmid/config", {});
+};
+
+my $attached_disks = sub {
+ return $client->get("/nodes/$node/qemu/$vmid/status/current", {})->{blockstat} // {};
+};
+
+my $pending = sub {
+ my ($key) = @_;
+ my ($entry) =
+ grep { $_->{key} eq $key } $client->get("/nodes/$node/qemu/$vmid/pending", {})->@*;
+ return $entry && (defined($entry->{pending}) || $entry->{delete});
+};
+
+my $volume_size = sub {
+ my ($storage, $volid) = @_;
+ my $content = $client->get("/nodes/$node/storage/$storage/content", { vmid => $vmid });
+ my ($volume) = grep { $_->{volid} eq $volid } $content->@*;
+ return $volume ? $volume->{size} : undef;
+};
+
+my $update_config = sub {
+ my ($params, $name) = @_;
+ my $upid = $client->post("/nodes/$node/qemu/$vmid/config", $params);
+ $pve->task_ok($upid, $name) or die "updating the config of VM $vmid failed\n";
+};
+
+Proxmox::Test::Testcase->new('pve_vm_disks')->with([{}])->step(
+ 'create directory storage',
+ sub {
+ $pve->run_command("mkdir -p $dir_storage_path");
+ $client->post(
+ '/storage',
+ {
+ storage => $dir_storage,
+ type => 'dir',
+ path => $dir_storage_path,
+ content => 'images',
+ },
+ );
+ },
+ sub {
+ eval { $client->delete("/storage/$dir_storage") };
+ $pve->run_command("rm -rf $dir_storage_path");
+ },
+)->step(
+ 'create VM',
+ sub {
+ my $upid = $client->post("/nodes/$node/qemu", $test_vm);
+ $pve->task_ok($upid, "created VM $vmid") or die "creating VM $vmid failed\n";
+ },
+ sub {
+ $pve->destroy_guest('qemu', $vmid);
+ },
+)->step(
+ 'verify disk config',
+ sub {
+ my $conf = $vm_config->();
+
+ like($conf->{scsi0} // '', qr/^local-lvm:[^,]+,.*\bsize=1G\b/, 'scsi0 is on local-lvm');
+ like(
+ $conf->{scsi1} // '',
+ qr/^\Q$dir_storage\E:[^,]+,.*\bsize=1G\b/,
+ "scsi1 is on $dir_storage",
+ );
+ },
+)->step(
+ 'start VM',
+ sub {
+ $pve->start_guest('qemu', $vmid);
+
+ my $attached = $attached_disks->();
+ ok($attached->{scsi0}, 'scsi0 is attached to the running VM');
+ ok($attached->{scsi1}, 'scsi1 is attached to the running VM');
+ },
+)->step(
+ 'hotplug disk',
+ sub {
+ $update_config->({ scsi2 => 'local-lvm:1' }, 'added scsi2 to the running VM');
+
+ ($hotplug_volid) = ($vm_config->()->{scsi2} // '') =~ m/^([^,]+)/;
+ ok(defined($hotplug_volid), 'config has scsi2');
+ ok(!$pending->('scsi2'), 'adding scsi2 is not pending');
+ ok($attached_disks->()->{scsi2}, 'scsi2 is attached to the running VM');
+ },
+)->step(
+ 'resize hotplugged disk',
+ sub {
+ my $upid =
+ $client->put("/nodes/$node/qemu/$vmid/resize", { disk => 'scsi2', size => '+1G' });
+ $pve->task_ok($upid, 'resized scsi2 by 1 GiB') or die "resizing scsi2 failed\n";
+
+ like(
+ $vm_config->()->{scsi2} // '',
+ qr/\bsize=2G\b/,
+ 'config reports the new scsi2 size',
+ );
+ is(
+ $volume_size->('local-lvm', $hotplug_volid),
+ 2 * 1024 * 1024 * 1024,
+ 'scsi2 volume has the new size',
+ );
+ },
+)->step(
+ 'remove hotplugged disk',
+ sub {
+ $update_config->({ delete => 'scsi2' }, 'removed scsi2 from the running VM');
+
+ my $conf = $vm_config->();
+ ok(!defined($conf->{scsi2}), 'scsi2 is gone from the config');
+ ok(!$pending->('scsi2'), 'removing scsi2 is not pending');
+ ok(!$attached_disks->()->{scsi2}, 'scsi2 is no longer attached to the running VM');
+
+ my ($unused) = grep { m/^unused\d+$/ && $conf->{$_} eq $hotplug_volid } keys $conf->%*;
+ ok(defined($unused), 'the scsi2 volume is kept as an unused volume');
+
+ $update_config->({ delete => $unused }, "deleted $unused") if defined($unused);
+
+ $conf = $vm_config->();
+ ok(!(grep { m/^unused\d+$/ } keys $conf->%*), 'no unused volume is left in the config');
+ ok(!defined($volume_size->('local-lvm', $hotplug_volid)),
+ 'the scsi2 volume is deleted');
+ },
+)->step(
+ 'stop VM',
+ sub {
+ $pve->stop_guest('qemu', $vmid);
+ },
+)->step(
+ 'destroy VM with unavailable disk storage',
+ sub {
+ $client->delete("/storage/$dir_storage");
+
+ my $upid = $client->delete("/nodes/$node/qemu/$vmid", {});
+ my $status = $pve->wait_for_task($upid, 1);
+ like(
+ $status,
+ qr/^WARNINGS: \d+$/,
+ 'destroying the VM succeeds with warnings about the unavailable storage',
+ );
+
+ ok(!$pve->guest_exists('qemu', $vmid), "VM $vmid is gone");
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index e7f8917..b1a35b6 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -187,6 +187,15 @@ TestcaseConfig(
],
tag: "extended",
),
+ Testcase(
+ name: "pve_vm_disks",
+ run: "./tests/pve_vm_disks.pl",
+ timeout: 600,
+ instances: [
+ "pve1",
+ ],
+ tag: "extended",
+ ),
Testcase(
name: "pve_vm_snapshot_clone",
run: "./tests/pve_vm_snapshot_clone.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 22/38] plans,scripts: download a Debian cloud image onto pve1
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (20 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 21/38] tests: vm: verify adding, resizing, and removing disks Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 23/38] tests: vm: verify lifecycle operations and cloud-init guest setup Michael Köppl
` (16 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Download the image through the storage's download-url API during
setup, so that a VM test can import its disk from the image with
'import-from' and boot a real guest.
The image is pinned to a dated build and only kept if it matches its
checksum, so all test runs use the same guest.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
plans/complete.ron | 1 +
scripts/download-cloud-image.sh | 21 +++++++++++++++++++++
2 files changed, 22 insertions(+)
create mode 100755 scripts/download-cloud-image.sh
diff --git a/plans/complete.ron b/plans/complete.ron
index dd299fe..5d91647 100644
--- a/plans/complete.ron
+++ b/plans/complete.ron
@@ -16,6 +16,7 @@ Plan(
"./scripts/update-packages.sh pve staging",
"./scripts/download-container-image.sh local debian-12-standard_12.12-1_amd64.tar.zst",
"./scripts/build-oci-image.sh local e2e-oci",
+ "./scripts/download-cloud-image.sh local https://cloud.debian.org/images/cloud/trixie/20261017-2634/debian-13-genericcloud-amd64-20261017-2634.qcow2 debian-13-genericcloud-amd64.qcow2 ec584ccabfef7120b4c4f7482adf0e615894d9292cedec446b64194b951ebe5dd97c336b5dcbae458333764b0edd516334506017f33c4f2020ef8d38b569ffe3",
"./scripts/setup-pebble.sh",
],
),
diff --git a/scripts/download-cloud-image.sh b/scripts/download-cloud-image.sh
new file mode 100755
index 0000000..8d410d3
--- /dev/null
+++ b/scripts/download-cloud-image.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/bash
+# Downloads a VM disk image to the import content of a storage, so that VMs can be created from it
+# with 'import-from'. The download is only kept if it matches the given SHA-512 checksum.
+SSH_KEY=$(jq -r '.ssh_private_keyfile' "$CONFIG_PATH")
+HOST=$(jq -r '.host' "$CONFIG_PATH")
+USER=$(jq -r '.ssh_user' "$CONFIG_PATH")
+
+STORAGE=$1
+URL=$2
+FILENAME=$3
+SHA512=$4
+
+ssh -i "$SSH_KEY" "$USER"@"$HOST" -o StrictHostKeyChecking=no <<EOF
+set -e
+
+pvesh create /nodes/localhost/storage/$STORAGE/download-url --content import \
+ --url '$URL' --filename '$FILENAME' --checksum $SHA512 --checksum-algorithm sha512
+# Check for existence of the file, so this fails if it cannot get the file. pvesh exits with 0 even
+# if the download failed.
+test -e "\$(pvesm path $STORAGE:import/$FILENAME)"
+EOF
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 23/38] tests: vm: verify lifecycle operations and cloud-init guest setup
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (21 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 22/38] plans,scripts: download a Debian cloud image onto pve1 Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 24/38] plans,scripts: install Ceph Tentacle on the cluster nodes Michael Köppl
` (15 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Create a VM from the Debian cloud image with a user, an SSH key, a
static address and DNS settings for cloud-init. Check the user,
network and meta data that Proxmox VE generates from them, then boot
the VM and check that the node can reach it at the address, log in
as the user with the key, and that the guest uses the VM name as
hostname. Package upgrades on first boot are disabled, as the guest
cannot reach any mirror.
While the VM runs, change a cloud-init option and check that it is
pending until the cloud-init drive is regenerated, and that it
changes the instance ID.
Then pause and resume the VM, reset it, reboot it to apply a pending
memory change. Then hibernate and start again. The guest's boot ID shows
whether the guest kept running or booted again. Finally, shut the VM
down through ACPI and check that destroying it removes all of its
volumes, including the cloud-init drive. A second VM without an OS
ignores ACPI requests, so the shutdown succeeds when forced.
Checks inside the guest are limited to what follows directly from the
config.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_vm_lifecycle.pl | 442 ++++++++++++++++++++++++++++++++++++++
tests/testcases.ron | 9 +
2 files changed, 451 insertions(+)
create mode 100755 tests/pve_vm_lifecycle.pl
diff --git a/tests/pve_vm_lifecycle.pl b/tests/pve_vm_lifecycle.pl
new file mode 100755
index 0000000..2168258
--- /dev/null
+++ b/tests/pve_vm_lifecycle.pl
@@ -0,0 +1,442 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_vm_lifecycle> - Test the lifecycle operations and cloud-init guest setup of a VM.
+
+=head1 DESCRIPTION
+
+This testcase creates a VM with a disk imported from the Debian cloud image that
+the setup plan downloads onto pve1's local storage, and a cloud-init drive. It
+configures a user with an SSH key, a static address and DNS settings through
+cloud-init and verifies the cloud-init configuration that Proxmox VE generates
+from them.
+
+It then starts the VM and verifies that the node can reach it at the configured
+address and log in as the configured user with the configured key, and that the
+guest uses the VM's name as hostname. While the VM is running, it changes a
+cloud-init option, which must be reported as pending until the cloud-init drive
+is regenerated.
+
+Afterwards, it pauses and resumes the VM, resets it, reboots it to apply a
+pending memory change and hibernates it to disk and starts it again. The guest's
+boot ID shows whether the guest kept running or was restarted by each of these.
+Finally, it shuts the VM down through ACPI and destroys it, which must remove all
+of its volumes. A second VM without an OS, which ignores ACPI, must only stop
+when the shutdown is forced.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use URI::Escape qw(uri_escape);
+
+use lib '.';
+use Proxmox::Test::Helper qw(get_pve_instance);
+use Proxmox::Test::Testcase;
+use Proxmox::Test::Util qw(poll_until);
+
+my $pve = get_pve_instance('pve1');
+my $node = $pve->get_nodename();
+my $client = $pve->client();
+
+my $vm_ip = '10.250.253.10';
+my $node_ip = '10.250.253.1';
+my $nameserver = '10.250.253.53';
+my $searchdomain = 'e2e.test';
+my $changed_searchdomain = 'changed.e2e.test';
+my $user = 'e2e';
+
+my $vmid = $client->get('/cluster/nextid');
+my $name = 'vm-lifecycle';
+my $no_os_vmid;
+my $key_dir;
+my $public_key;
+my $instance_id;
+
+my $vm_config = sub {
+ my ($id) = @_;
+ return $client->get("/nodes/$node/qemu/$id/config", { current => 1 });
+};
+
+my $vm_status = sub {
+ my ($id) = @_;
+ return $client->get("/nodes/$node/qemu/$id/status/current", {});
+};
+
+my $volumes = sub {
+ my $content = $client->get("/nodes/$node/storage/local-lvm/content", { vmid => $vmid });
+ return { map { $_->{volid} => 1 } $content->@* };
+};
+
+my $volume_exists = sub {
+ my ($volid) = @_;
+
+ my ($storage) = $volid =~ m/^([^:]+):/;
+ my $content = $client->get("/nodes/$node/storage/$storage/content", { vmid => $vmid });
+ return grep { $_->{volid} eq $volid } $content->@*;
+};
+
+my $pending_value = sub {
+ my ($key) = @_;
+ my ($entry) =
+ grep { $_->{key} eq $key } $client->get("/nodes/$node/qemu/$vmid/pending", {})->@*;
+ return $entry ? $entry->{pending} : undef;
+};
+
+my $cloudinit_dump = sub {
+ my ($type) = @_;
+ return $client->get("/nodes/$node/qemu/$vmid/cloudinit/dump", { type => $type });
+};
+
+my $cloudinit_option = sub {
+ my ($key) = @_;
+ my ($entry) =
+ grep { $_->{key} eq $key } $client->get("/nodes/$node/qemu/$vmid/cloudinit", {})->@*;
+ return $entry // {};
+};
+
+my $guest_exec = sub {
+ my ($cmd) = @_;
+ my ($out) =
+ $pve->run_command("ssh -i $key_dir/id_ed25519 -o BatchMode=yes -o ConnectTimeout=5"
+ . " -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR"
+ . " $user\@$vm_ip $cmd");
+ return $out // '';
+};
+
+my $wait_for_ssh = sub {
+ poll_until(
+ sub {
+ eval { $guest_exec->('true'); 1 }
+ },
+ timeout => 300,
+ interval => 5,
+ description => "SSH login to VM $vmid as '$user'",
+ );
+};
+
+# Changes whenever the guest boots, but not while paused or hibernated
+my $boot_id = sub {
+ my $id = $guest_exec->('cat /proc/sys/kernel/random/boot_id');
+ chomp($id);
+ return $id;
+};
+
+Proxmox::Test::Testcase->new('pve_vm_lifecycle')->with([{}])->step(
+ 'create SSH key pair on the node',
+ sub {
+ ($key_dir) = $pve->run_command('mktemp -d');
+ chomp($key_dir);
+ $pve->run_command("ssh-keygen -q -t ed25519 -N '' -f $key_dir/id_ed25519");
+
+ ($public_key) = $pve->run_command("cat $key_dir/id_ed25519.pub");
+ chomp($public_key);
+ },
+ sub {
+ $pve->run_command("rm -rf $key_dir") if defined($key_dir);
+ },
+)->step(
+ 'create VM from cloud image',
+ sub {
+ my $upid = $client->post(
+ "/nodes/$node/qemu",
+ {
+ vmid => $vmid,
+ name => $name,
+ ostype => 'l26',
+ memory => 1024,
+ scsihw => 'virtio-scsi-pci',
+ scsi0 =>
+ 'local-lvm:0,import-from=local:import/debian-13-genericcloud-amd64.qcow2',
+ ide2 => 'local-lvm:cloudinit',
+ boot => 'order=scsi0',
+ net0 => 'virtio,bridge=vmbr0',
+ ciuser => $user,
+ sshkeys => uri_escape($public_key),
+ ipconfig0 => "ip=$vm_ip/24,gw=$node_ip",
+ nameserver => $nameserver,
+ searchdomain => $searchdomain,
+ # The guest cannot reach any package mirror.
+ ciupgrade => 0,
+ },
+ );
+ $pve->task_ok($upid, "created VM $vmid from the cloud image")
+ or die "creating VM $vmid failed\n";
+ },
+ sub {
+ $pve->destroy_guest('qemu', $vmid);
+ },
+)->step(
+ 'verify VM volumes',
+ sub {
+ my $conf = $vm_config->($vmid);
+
+ my ($disk) = ($conf->{scsi0} // '') =~ m/^(local-lvm:[^,]+)/;
+ ok(defined($disk) && $volumes->()->{$disk},
+ 'disk is imported to a volume on local-lvm');
+
+ my ($cidrive) = ($conf->{ide2} // '') =~ m/^(local-lvm:[^,]+),media=cdrom\b/;
+ ok(
+ defined($cidrive) && $volumes->()->{$cidrive},
+ 'cloud-init drive is a volume on local-lvm',
+ );
+ },
+)->step(
+ 'verify generated cloud-init config',
+ sub {
+ my $user_data = $cloudinit_dump->('user');
+ like($user_data, qr/^hostname: \Q$name\E$/m, 'user data sets the VM name as hostname');
+ like($user_data, qr/^fqdn: \Q$name.$searchdomain\E$/m, 'user data sets the FQDN');
+ like($user_data, qr/^user: \Q$user\E$/m, 'user data sets the user');
+ like($user_data, qr/^\s*- \Q$public_key\E$/m, 'user data authorizes the SSH key');
+
+ my ($mac) = ($vm_config->($vmid)->{net0} // '') =~ m/^[^=,]+=([0-9A-Fa-f:]{17})\b/
+ or die "net0 has no MAC address\n";
+ my $network_data = $cloudinit_dump->('network');
+ like(
+ $network_data,
+ qr/\bmac_address: '\Q$mac\E'/i,
+ 'network data configures the interface with the MAC address of net0',
+ );
+ like($network_data, qr/\baddress: '\Q$vm_ip\E'/, 'network data sets the address');
+ like($network_data, qr/\bnetmask: '255\.255\.255\.0'/, 'network data sets the netmask');
+ like($network_data, qr/\bgateway: '\Q$node_ip\E'/, 'network data sets the gateway');
+ like($network_data, qr/- '\Q$nameserver\E'/, 'network data sets the nameserver');
+ like($network_data, qr/- '\Q$searchdomain\E'/, 'network data sets the search domain');
+
+ ($instance_id) = $cloudinit_dump->('meta') =~ m/^instance-id: (\S+)$/m;
+ ok(defined($instance_id), 'meta data sets an instance ID');
+ },
+)->step(
+ 'add node address on vmbr0',
+ sub {
+ $pve->run_command("ip addr add $node_ip/24 dev vmbr0");
+ },
+ sub {
+ $pve->run_command("ip addr del $node_ip/24 dev vmbr0 2>/dev/null; true");
+ },
+)->step(
+ 'start VM',
+ sub {
+ $pve->start_guest('qemu', $vmid);
+ },
+)->step(
+ 'verify guest setup',
+ sub {
+ poll_until(
+ sub { $pve->node_can_ping($vm_ip) },
+ timeout => 300,
+ interval => 5,
+ description => "VM $vmid to answer on $vm_ip",
+ );
+ pass("node can ping VM $vmid at the configured address");
+
+ $wait_for_ssh->();
+ is($guest_exec->('id -un'), "$user\n", 'node logs in as the user with the SSH key');
+ is($guest_exec->('hostname'), "$name\n", 'guest uses the VM name as hostname');
+ like(
+ $guest_exec->('ip -4 -o addr show'),
+ qr/\binet \Q$vm_ip\E\/24\b/,
+ 'guest has the configured address and prefix length',
+ );
+ },
+)->step(
+ 'change cloud-init option of the running VM',
+ sub {
+ my $upid = $client->post(
+ "/nodes/$node/qemu/$vmid/config",
+ { searchdomain => $changed_searchdomain },
+ );
+ $pve->task_ok($upid, 'changed the search domain') or die "changing the config failed\n";
+
+ my $option = $cloudinit_option->('searchdomain');
+ is($option->{value}, $searchdomain, 'cloud-init drive still has the old search domain');
+ is($option->{pending}, $changed_searchdomain, 'new search domain is pending');
+
+ my ($new_instance_id) = $cloudinit_dump->('meta') =~ m/^instance-id: (\S+)$/m;
+ isnt($new_instance_id, $instance_id, 'changed config gets a new instance ID');
+ },
+)->step(
+ 'regenerate cloud-init drive',
+ sub {
+ $client->put("/nodes/$node/qemu/$vmid/cloudinit", {});
+
+ my $option = $cloudinit_option->('searchdomain');
+ is(
+ $option->{value},
+ $changed_searchdomain,
+ 'cloud-init drive has the new search domain',
+ );
+ ok(!defined($option->{pending}), 'no change of the search domain is pending');
+ },
+)->step(
+ 'pause and resume VM',
+ sub {
+ my $old_boot_id = $boot_id->();
+
+ my $upid = $client->post("/nodes/$node/qemu/$vmid/status/suspend", {});
+ $pve->task_ok($upid, "paused VM $vmid") or die "pausing VM $vmid failed\n";
+
+ my $status = $vm_status->($vmid);
+ is($status->{status}, 'running', 'paused VM still reports status running');
+ is($status->{qmpstatus}, 'paused', 'QEMU reports the VM as paused');
+ ok(!$pve->node_can_ping($vm_ip), 'paused VM does not answer ping');
+
+ eval { $client->post("/nodes/$node/qemu/$vmid/status/reboot", {}) };
+ ok($@, 'rebooting the paused VM fails');
+ is($vm_status->($vmid)->{qmpstatus}, 'paused', 'VM is still paused');
+
+ $upid = $client->post("/nodes/$node/qemu/$vmid/status/resume", {});
+ $pve->task_ok($upid, "resumed VM $vmid") or die "resuming VM $vmid failed\n";
+
+ is($vm_status->($vmid)->{qmpstatus}, 'running', 'QEMU reports the VM as running');
+ $wait_for_ssh->();
+ is($boot_id->(), $old_boot_id, 'guest continues without booting after the resume');
+ },
+)->step(
+ 'reset VM',
+ sub {
+ my $old_pid = $vm_status->($vmid)->{pid};
+ my $old_boot_id = $boot_id->();
+
+ my $upid = $client->post("/nodes/$node/qemu/$vmid/status/reset", {});
+ $pve->task_ok($upid, "reset VM $vmid") or die "resetting VM $vmid failed\n";
+
+ my $status = $vm_status->($vmid);
+ is($status->{status}, 'running', 'VM keeps running during the reset');
+ is($status->{pid}, $old_pid, 'reset keeps the QEMU process');
+
+ $wait_for_ssh->();
+ isnt($boot_id->(), $old_boot_id, 'guest boots again after the reset');
+ },
+)->step(
+ 'reboot VM to apply pending change',
+ sub {
+ my $upid = $client->post("/nodes/$node/qemu/$vmid/config", { memory => 1536 });
+ $pve->task_ok($upid, 'changed the memory') or die "changing the memory failed\n";
+
+ is($pending_value->('memory'), 1536, 'memory change of the running VM is pending');
+
+ my $old_pid = $vm_status->($vmid)->{pid};
+ my $old_boot_id = $boot_id->();
+
+ $upid = $client->post("/nodes/$node/qemu/$vmid/status/reboot", { timeout => 120 });
+ $pve->task_ok($upid, "rebooted VM $vmid") or die "rebooting VM $vmid failed\n";
+
+ # The reboot task only shuts the VM down, a separate start brings it back up.
+ poll_until(
+ sub {
+ my $status = $vm_status->($vmid);
+ return $status->{status} eq 'running' && $status->{pid} != $old_pid;
+ },
+ timeout => 120,
+ interval => 2,
+ description => "VM $vmid to run in a new QEMU process",
+ );
+ pass('reboot starts the VM in a new QEMU process');
+
+ is($vm_config->($vmid)->{memory}, 1536, 'reboot applied the pending memory change');
+ ok(!defined($pending_value->('memory')), 'memory change is no longer pending');
+
+ $wait_for_ssh->();
+ isnt($boot_id->(), $old_boot_id, 'guest boots again after the reboot');
+
+ # The guest only sees part of the memory, but more than before the change.
+ my ($mem_kib) = $guest_exec->('cat /proc/meminfo') =~ m/^MemTotal:\s+(\d+) kB$/m;
+ cmp_ok($mem_kib // 0, '>', 1024 * 1024, 'guest has more than 1 GiB of memory');
+ },
+)->step(
+ 'hibernate VM',
+ sub {
+ my $old_boot_id = $boot_id->();
+ my ($since) = $pve->run_command('date +%s');
+ chomp($since);
+
+ my $upid = $client->post("/nodes/$node/qemu/$vmid/status/suspend", { todisk => 1 });
+ $pve->task_ok($upid, "hibernated VM $vmid") or die "hibernating VM $vmid failed\n";
+
+ is($vm_status->($vmid)->{status}, 'stopped', 'hibernated VM reports status stopped');
+
+ my $conf = $vm_config->($vmid);
+ is($conf->{lock}, 'suspended', 'hibernated VM is locked');
+ my $vmstate = $conf->{vmstate};
+ ok(defined($vmstate) && $volume_exists->($vmstate), 'RAM is saved to a volume');
+
+ eval { $client->put("/nodes/$node/qemu/$vmid/config", { name => "$name-changed" }) };
+ ok($@, 'changing the config of the hibernated VM fails');
+ is($vm_config->($vmid)->{name}, $name, 'name of the hibernated VM is unchanged');
+
+ # Nothing waits for the cleanup operation of qmeventd after hibernation. If the VM is
+ # started before the cleanup takes the config lock, operations after that would fail.
+ $pve->wait_for_qemu_cleanup($vmid, $since);
+
+ $pve->start_guest('qemu', $vmid);
+
+ $wait_for_ssh->();
+ is($boot_id->(), $old_boot_id, 'guest continues without booting after the start');
+
+ $conf = $vm_config->($vmid);
+ ok(!defined($conf->{lock}), 'VM is no longer locked');
+ ok(!defined($conf->{vmstate}), 'config no longer references the saved RAM');
+ ok(!defined($vmstate) || !$volume_exists->($vmstate), 'saved RAM is removed');
+ },
+)->step(
+ 'shutdown VM',
+ sub {
+ my $upid = $client->post("/nodes/$node/qemu/$vmid/status/shutdown", { timeout => 120 });
+ $pve->task_ok($upid, "shut down VM $vmid") or die "shutting down VM $vmid failed\n";
+
+ is($vm_status->($vmid)->{status}, 'stopped', "VM $vmid reports status 'stopped'");
+ },
+)->step(
+ 'destroy VM',
+ sub {
+ my $upid = $client->delete("/nodes/$node/qemu/$vmid", {});
+ $pve->task_ok($upid, "destroyed VM $vmid") or die "destroying VM $vmid failed\n";
+
+ ok(!$pve->guest_exists('qemu', $vmid), "VM $vmid is gone");
+ is_deeply($volumes->(), {}, "no volume of VM $vmid is left on local-lvm");
+ },
+)->step(
+ 'create and start VM without OS',
+ sub {
+ $no_os_vmid = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/qemu",
+ { vmid => $no_os_vmid, name => 'vm-lifecycle-no-os', memory => 128 },
+ );
+ $pve->task_ok($upid, "created VM $no_os_vmid")
+ or die "creating VM $no_os_vmid failed\n";
+
+ $pve->start_guest('qemu', $no_os_vmid);
+ },
+ sub {
+ $pve->destroy_guest('qemu', $no_os_vmid) if defined($no_os_vmid);
+ },
+)->step(
+ 'shutdown VM without OS',
+ sub {
+ my $upid = $client->post(
+ "/nodes/$node/qemu/$no_os_vmid/status/shutdown", { timeout => 10 },
+ );
+ ok(
+ !$pve->task_successful($pve->wait_for_task($upid, 1)),
+ 'shutdown fails as nothing handles the ACPI request',
+ );
+ is($vm_status->($no_os_vmid)->{status}, 'running', 'VM without OS keeps running');
+
+ $upid = $client->post(
+ "/nodes/$node/qemu/$no_os_vmid/status/shutdown",
+ { timeout => 10, forceStop => 1 },
+ );
+ $pve->task_ok($upid, "forced shutdown of VM $no_os_vmid")
+ or die "forcing the shutdown of VM $no_os_vmid failed\n";
+ is($vm_status->($no_os_vmid)->{status}, 'stopped', 'forced shutdown stops the VM');
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index b1a35b6..fba3e06 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -29,6 +29,15 @@ TestcaseConfig(
],
tag: "basic",
),
+ Testcase(
+ name: "pve_vm_lifecycle",
+ run: "./tests/pve_vm_lifecycle.pl",
+ timeout: 600,
+ instances: [
+ "pve1",
+ ],
+ tag: "basic",
+ ),
Testcase(
name: "pve_ldap_realm",
run: "./tests/ldap_realm.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 24/38] plans,scripts: install Ceph Tentacle on the cluster nodes
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (22 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 23/38] tests: vm: verify lifecycle operations and cloud-init guest setup Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 25/38] test framework: add Ceph test helpers Michael Köppl
` (14 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Installing the packages during setup puts them into the snapshot the
tests start from, so Ceph tests do not have to download them.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
plans/complete.ron | 3 +++
scripts/install-ceph.sh | 13 +++++++++++++
2 files changed, 16 insertions(+)
create mode 100755 scripts/install-ceph.sh
diff --git a/plans/complete.ron b/plans/complete.ron
index 5d91647..3210ac4 100644
--- a/plans/complete.ron
+++ b/plans/complete.ron
@@ -99,6 +99,7 @@ Plan(
"./scripts/update-packages.sh pve staging",
"./scripts/download-container-image.sh local debian-12-standard_12.12-1_amd64.tar.zst",
"./scripts/setup-zfs-storage.sh zfs0 zfstest",
+ "./scripts/install-ceph.sh tentacle test",
],
),
SetupInstance(
@@ -136,6 +137,7 @@ Plan(
"./scripts/update-packages.sh pve staging",
"./scripts/download-container-image.sh local debian-12-standard_12.12-1_amd64.tar.zst",
"./scripts/setup-zfs-storage.sh zfs0 zfstest",
+ "./scripts/install-ceph.sh tentacle test",
],
),
SetupInstance(
@@ -173,6 +175,7 @@ Plan(
"./scripts/update-packages.sh pve staging",
"./scripts/download-container-image.sh local debian-12-standard_12.12-1_amd64.tar.zst",
"./scripts/setup-zfs-storage.sh zfs0 zfstest",
+ "./scripts/install-ceph.sh tentacle test",
],
),
SetupInstance(
diff --git a/scripts/install-ceph.sh b/scripts/install-ceph.sh
new file mode 100755
index 0000000..2ca2d91
--- /dev/null
+++ b/scripts/install-ceph.sh
@@ -0,0 +1,13 @@
+#!/usr/bin/bash
+SSH_KEY=$(jq -r '.ssh_private_keyfile' $CONFIG_PATH)
+HOST=$(jq -r '.host' $CONFIG_PATH)
+USER=$(jq -r '.ssh_user' $CONFIG_PATH)
+
+VERSION=$1
+REPOSITORY=$2
+
+ssh -i $SSH_KEY $USER@$HOST -o StrictHostKeyChecking=no <<EOT
+set -e
+
+yes | pveceph install --version $VERSION --repository $REPOSITORY
+EOT
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 25/38] test framework: add Ceph test helpers
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (23 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 24/38] plans,scripts: install Ceph Tentacle on the cluster nodes Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-06 12:24 ` Kefu Chai
2026-11-04 19:03 ` [PATCH e2e-tests 26/38] tests: ceph: verify setting up a hyper-converged cluster Michael Köppl
` (13 subsequent siblings)
38 siblings, 1 reply; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Add helpers to set up a hyper-converged Ceph cluster on the cluster
nodes and to wait until it is ready. Ceph needs the PVE cluster to
exist, but post-setup scripts run before it is created, and the
tests roll back to the state after setup. So every Ceph test sets up
its own Ceph cluster with these helpers. The setup is based mostly on
the documentation [0].
The cluster uses dedicated public and cluster networks on the second
NIC of the nodes, as the documentation recommends keeping Ceph traffic
off the network of the Proxmox VE cluster. Without a network, the
monitors fall back to the node's cluster address.
The OSD memory target is lowered to 1 GiB, as the test nodes only have
4 GiB of memory for two OSDs, a monitor, and a manager.
[0] https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
Proxmox/Test/Ceph.pm | 338 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 338 insertions(+)
create mode 100644 Proxmox/Test/Ceph.pm
diff --git a/Proxmox/Test/Ceph.pm b/Proxmox/Test/Ceph.pm
new file mode 100644
index 0000000..357e0ed
--- /dev/null
+++ b/Proxmox/Test/Ceph.pm
@@ -0,0 +1,338 @@
+package Proxmox::Test::Ceph;
+use strict;
+use warnings FATAL => 'all';
+
+use parent 'Exporter';
+our @EXPORT_OK = qw(
+ CEPH_PUBLIC_NETWORK
+ CEPH_CLUSTER_NETWORK
+ ceph_disks
+ ceph_osds
+ ceph_status
+ wait_for_ceph
+ check_storage_active
+ setup_ceph_network
+ init_ceph
+ create_monitors
+ create_managers
+ create_osds
+ create_ceph_cluster
+ create_pool
+);
+
+use JSON;
+use Test::More;
+
+use Proxmox::Test::Util qw(poll_until);
+
+# Lower the OSD memory target from the default of 4 GiB, which would otherwise leave the test nodes
+# without memory for anything else.
+use constant OSD_MEMORY_TARGET => 1024 * 1024 * 1024;
+
+# Ceph traffic is kept off the network of the Proxmox VE cluster, as recommended for hyper-converged
+# setups. The Ceph networks use the NIC that the plan attaches to the 'tt_test' bridge.
+use constant CEPH_NIC => 'ens19';
+use constant CEPH_PUBLIC_PREFIX => '10.250.240';
+use constant CEPH_CLUSTER_PREFIX => '10.250.241';
+use constant CEPH_PUBLIC_NETWORK => CEPH_PUBLIC_PREFIX . '.0/24';
+use constant CEPH_CLUSTER_NETWORK => CEPH_CLUSTER_PREFIX . '.0/24';
+
+# Disks with serials ceph0, ceph1, ... are supplied by the plan.
+sub ceph_disks {
+ my ($instance) = @_;
+
+ my @disks;
+ for my $disk (($instance->get_additional_disks() // [])->@*) {
+ next if ($disk->{serial} // '') !~ m/^ceph\d+$/;
+
+ my $path = $instance->get_disk_by_serial($disk->{serial})
+ // die "no disk with serial '$disk->{serial}' on " . $instance->get_nodename() . "\n";
+ push @disks, $path;
+ }
+
+ return \@disks;
+}
+
+# The OSDs on the node of $instance, as listed in the OSD tree of the API.
+sub ceph_osds {
+ my ($instance) = @_;
+
+ my $node = $instance->get_nodename();
+ my $tree = $instance->client()->get("/nodes/$node/ceph/osd", {});
+
+ my @buckets = ($tree->{root});
+ while (my $bucket = shift @buckets) {
+ my $children = $bucket->{children} // [];
+ if (($bucket->{type} // '') eq 'host' && $bucket->{name} eq $node) {
+ return [grep { $_->{type} eq 'osd' } $children->@*];
+ }
+ push @buckets, $children->@*;
+ }
+
+ return [];
+}
+
+# The output of 'ceph status' on $instance.
+sub ceph_status {
+ my ($instance) = @_;
+
+ my ($out) = $instance->run_command('ceph status --format json');
+ return decode_json($out);
+}
+
+# Wait until the Ceph cluster seen from $instance has $opts{monitors} monitors in quorum,
+# $opts{osds} OSDs that are all up and in, and all placement groups active+clean. If
+# $opts{health_ok} is set as well, also wait for HEALTH OK.
+# Returns the final status.
+sub wait_for_ceph {
+ my ($instance, %opts) = @_;
+
+ my $monitors = $opts{monitors} // 0;
+ my $osds = $opts{osds} // 0;
+ my $last;
+
+ my $ready = eval {
+ poll_until(
+ sub {
+ $last = ceph_status($instance);
+
+ my $health = $last->{health}->{status} // '';
+ return 0 if $health eq 'HEALTH_ERR';
+ return 0 if $opts{health_ok} && $health ne 'HEALTH_OK';
+
+ return 0 if scalar(($last->{quorum_names} // [])->@*) < $monitors;
+
+ my $osdmap = $last->{osdmap} // {};
+ return 0 if ($osdmap->{num_osds} // 0) < $osds;
+ return 0 if ($osdmap->{num_up_osds} // 0) != ($osdmap->{num_osds} // 0);
+ return 0 if ($osdmap->{num_in_osds} // 0) != ($osdmap->{num_osds} // 0);
+
+ my $pgmap = $last->{pgmap} // {};
+ my $clean = 0;
+ for my $state (($pgmap->{pgs_by_state} // [])->@*) {
+ $clean += $state->{count} if $state->{state_name} eq 'active+clean';
+ }
+ return $clean == ($pgmap->{num_pgs} // 0) ? $last : 0;
+ },
+ timeout => $opts{timeout} // 600,
+ interval => 5,
+ description => $opts{description} // 'the Ceph cluster to become ready',
+ );
+ };
+
+ if (!$ready) {
+ my $err = $@;
+ diag('last Ceph health: ' . encode_json($last->{health} // {})) if $last;
+ die $err;
+ }
+
+ return $ready;
+}
+
+# Check that $storage is active on $instance. If it is not, print the result of `pvesm status`.
+sub check_storage_active {
+ my ($instance, $storage) = @_;
+
+ my $node = $instance->get_nodename();
+ my $status = $instance->client()->get("/nodes/$node/storage/$storage/status", {});
+ return 1 if ok($status->{active}, "storage '$storage' is active on node '$node'");
+
+ my ($out, $err) = eval { $instance->run_command("pvesm status --storage $storage") };
+ diag("'pvesm status --storage $storage' on '$node':\n"
+ . join('', grep { defined } $out, $err, $@));
+ return 0;
+}
+
+# Give every instance in $instances an address in the Ceph public and the Ceph cluster network,
+# numbered in the order of $instances, and wait until the instances reach each other on both.
+# Returns the addresses as { <nodename> => { public => ..., cluster => ... } }.
+sub setup_ceph_network {
+ my ($instances) = @_;
+
+ my $nic = CEPH_NIC;
+ my $addresses = {};
+ for my $i (0 .. $instances->$#*) {
+ my $instance = $instances->[$i];
+ my $public = CEPH_PUBLIC_PREFIX . '.' . ($i + 1);
+ my $cluster = CEPH_CLUSTER_PREFIX . '.' . ($i + 1);
+
+ $instance->run_command("ip link set $nic up && ip addr add $public/24 dev $nic"
+ . " && ip addr add $cluster/24 dev $nic");
+ $addresses->{ $instance->get_nodename() } = { public => $public, cluster => $cluster };
+ }
+
+ for my $instance ($instances->@*) {
+ my $node = $instance->get_nodename();
+ for my $other (grep { $_ ne $node } sort keys $addresses->%*) {
+ for my $address (sort values $addresses->{$other}->%*) {
+ poll_until(
+ sub { $instance->node_can_ping($address) },
+ timeout => 30,
+ interval => 2,
+ description => "'$node' to reach '$other' at $address",
+ );
+ }
+ }
+ }
+
+ return $addresses;
+}
+
+# Write the initial Ceph configuration through the first instance in $instances for the whole
+# cluster. %opts is passed on as additional parameters of the init call.
+sub init_ceph {
+ my ($instances, %opts) = @_;
+
+ my $first = $instances->[0];
+ my $node = $first->get_nodename();
+
+ $first->client()->post("/nodes/$node/ceph/init", \%opts);
+}
+
+# Prints the end of the monitor logs to show why creating a monitor or getting quorum failed.
+my $diag_monitor_logs = sub {
+ my ($instances) = @_;
+
+ for my $instance ($instances->@*) {
+ my $node = $instance->get_nodename();
+ my ($log) =
+ eval { $instance->run_command("tail -n 30 /var/log/ceph/ceph-mon.$node.log") };
+ diag("end of the monitor log on '$node':\n" . ($log // "not available: $@"));
+ }
+};
+
+# Create a monitor on every instance in $instances, one at a time, and wait until each new monitor
+# has joined the quorum before creating the next one. The first monitor also gets a manager.
+sub create_monitors {
+ my ($instances) = @_;
+
+ my $first = $instances->[0];
+ my @with_monitor;
+
+ for my $instance ($instances->@*) {
+ my $node = $instance->get_nodename();
+
+ eval {
+ # Before creating another monitor, ensure connectivity to the existing ones.
+ if (@with_monitor) {
+ poll_until(
+ sub {
+ eval {
+ $instance->run_command(
+ 'ceph --conf /etc/pve/ceph.conf --connect-timeout 5 status'
+ . ' >/dev/null');
+ 1;
+ };
+ },
+ timeout => 60,
+ interval => 2,
+ description => "'$node' to connect to the existing monitors",
+ );
+ }
+
+ my $upid = $instance->client()->post("/nodes/$node/ceph/mon/$node", {});
+ $instance->task_ok($upid, "created monitor on '$node'")
+ or die "creating the monitor on '$node' failed\n";
+
+ poll_until(
+ sub {
+ grep { $_ eq $node } (ceph_status($first)->{quorum_names} // [])->@*;
+ },
+ timeout => 120,
+ interval => 2,
+ description => "monitor '$node' to join the quorum",
+ );
+ };
+ if (my $err = $@) {
+ $diag_monitor_logs->([@with_monitor, $instance]);
+ die $err;
+ }
+
+ push @with_monitor, $instance;
+ }
+}
+
+# Create a manager on every instance in $instances that does not have one yet.
+sub create_managers {
+ my ($instances) = @_;
+
+ for my $instance ($instances->@*) {
+ my $node = $instance->get_nodename();
+
+ my $managers = $instance->client()->get("/nodes/$node/ceph/mgr", {});
+ next if grep { ($_->{name} // '') eq $node } $managers->@*;
+
+ my $upid = $instance->client()->post("/nodes/$node/ceph/mgr/$node", {});
+ $instance->task_ok($upid, "created manager on '$node'")
+ or die "creating the manager on '$node' failed\n";
+ }
+}
+
+# Create an OSD on every Ceph disk of every instance in $instances. %opts is passed on as
+# additional parameters of the OSD create call. Returns the number of OSDs created.
+sub create_osds {
+ my ($instances, %opts) = @_;
+
+ $instances->[0]->run_command('ceph config set osd osd_memory_target ' . OSD_MEMORY_TARGET);
+
+ my $count = 0;
+ for my $instance ($instances->@*) {
+ my $node = $instance->get_nodename();
+
+ for my $dev (ceph_disks($instance)->@*) {
+ my $upid =
+ $instance->client()->post("/nodes/$node/ceph/osd", { dev => $dev, %opts });
+ $instance->task_ok($upid, "created OSD on '$node' for '$dev'")
+ or die "creating the OSD on '$node' for '$dev' failed\n";
+ $count++;
+ }
+ }
+
+ return $count;
+}
+
+# Set up a Ceph cluster on the Ceph public and cluster network with a monitor, a manager and OSDs on
+# every instance in $instances, and wait until it is ready. The Ceph packages must already be
+# installed. Returns the number of OSDs.
+sub create_ceph_cluster {
+ my ($instances) = @_;
+
+ setup_ceph_network($instances);
+ init_ceph(
+ $instances,
+ network => CEPH_PUBLIC_NETWORK,
+ 'cluster-network' => CEPH_CLUSTER_NETWORK,
+ );
+ create_monitors($instances);
+ create_managers($instances);
+ my $osds = create_osds($instances);
+
+ wait_for_ceph($instances->[0], monitors => scalar($instances->@*), osds => $osds);
+
+ return $osds;
+}
+
+# Create the replicated pool $name through $instance and wait for its placement groups. With
+# $opts{add_storages}, an RBD storage of the same name is added for the pool.
+sub create_pool {
+ my ($instance, $name, %opts) = @_;
+
+ my $node = $instance->get_nodename();
+ my $upid = $instance->client()
+ ->post("/nodes/$node/ceph/pool", { name => $name, pg_autoscale_mode => "on", %opts });
+ $instance->task_ok($upid, "created pool '$name'") or die "creating pool '$name' failed\n";
+
+ poll_until(
+ sub {
+ my ($out) = $instance->run_command("ceph df --format json");
+ return grep { $_->{name} eq $name } (decode_json($out)->{pools} // [])->@*;
+ },
+ timeout => 60,
+ interval => 2,
+ description => "pool '$name' to show up in the usage stats",
+ );
+
+ wait_for_ceph($instance);
+}
+
+1;
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* Re: [PATCH e2e-tests 25/38] test framework: add Ceph test helpers
2026-11-04 19:03 ` [PATCH e2e-tests 25/38] test framework: add Ceph test helpers Michael Köppl
@ 2026-11-06 12:24 ` Kefu Chai
2026-11-06 13:04 ` Michael Köppl
0 siblings, 1 reply; 47+ messages in thread
From: Kefu Chai @ 2026-11-06 12:24 UTC (permalink / raw)
To: m.koeppl; +Cc: pve-devel
Hi Michael,
> [snip]
> + my $pgmap = $last->{pgmap} // {};
> + my $clean = 0;
> + for my $state (($pgmap->{pgs_by_state} // [])->@*) {
> + $clean += $state->{count} if $state->{state_name} eq 'active+clean';
> + }
> + return $clean == ($pgmap->{num_pgs} // 0) ? $last : 0;
> [snip]
Waiting for quorum and all expected OSDs to be up and in makes sense.
For the PGs, I'd allow scrubbing alongside active+clean.
A healthy, fully replicated PG can be active+clean+scrubbing or
active+clean+scrubbing+deep. The exact match here means the test has
to wait for scrubbing to finish, which could cause an unnecessary
delay or timeout.
Could we accept those two states as well? I'd allow the scrub flags
explicitly: checking only that active and clean are present could
also accept an inconsistent PG.
The OSD drain check in the replacement test later in the series
has the same issue.
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH e2e-tests 25/38] test framework: add Ceph test helpers
2026-11-06 12:24 ` Kefu Chai
@ 2026-11-06 13:04 ` Michael Köppl
0 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-06 13:04 UTC (permalink / raw)
To: Kefu Chai, m.koeppl; +Cc: pve-devel
On Fri Nov 6, 2026 at 1:24 PM CET, Kefu Chai wrote:
> Hi Michael,
>
>> [snip]
>> + my $pgmap = $last->{pgmap} // {};
>> + my $clean = 0;
>> + for my $state (($pgmap->{pgs_by_state} // [])->@*) {
>> + $clean += $state->{count} if $state->{state_name} eq 'active+clean';
>> + }
>> + return $clean == ($pgmap->{num_pgs} // 0) ? $last : 0;
>> [snip]
>
> Waiting for quorum and all expected OSDs to be up and in makes sense.
> For the PGs, I'd allow scrubbing alongside active+clean.
>
> A healthy, fully replicated PG can be active+clean+scrubbing or
> active+clean+scrubbing+deep. The exact match here means the test has
> to wait for scrubbing to finish, which could cause an unnecessary
> delay or timeout.
>
> Could we accept those two states as well? I'd allow the scrub flags
> explicitly: checking only that active and clean are present could
> also accept an inconsistent PG.
Thanks for taking the time to look at these tests and your feedback!
Can definitely add these for a v2. I'll add a list of explicitly
accepted states as in the following example and accept either state
while checking:
```
my $CLEAN_PG_STATES = {
'active+clean' => 1,
'active+clean+scrubbing' => 1,
'active+clean+scrubbing+deep' => 1,
};
```
>
> The OSD drain check in the replacement test later in the series
> has the same issue.
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH e2e-tests 26/38] tests: ceph: verify setting up a hyper-converged cluster
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (24 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 25/38] test framework: add Ceph test helpers Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-06 12:50 ` Kefu Chai
2026-11-04 19:03 ` [PATCH e2e-tests 27/38] tests: ceph: verify guests on RBD storage Michael Köppl
` (12 subsequent siblings)
38 siblings, 1 reply; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Set up Ceph on the main cluster through the API with dedicated
public and cluster networks, as the setup wizard requires a public
network and the documentation recommends separating Ceph traffic.
Check that the configuration on pmxcfs has the networks and is linked
to from /etc/ceph on every node, that the monitors form a quorum on
the public network, and that the OSDs use the public network for
clients and the cluster network for replication. Also check the
managers and creating a pool with an RBD storage.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/ceph/test_ceph_cluster_setup.pl | 206 ++++++++++++++++++++++++++
tests/testcases.ron | 11 ++
2 files changed, 217 insertions(+)
create mode 100755 tests/ceph/test_ceph_cluster_setup.pl
diff --git a/tests/ceph/test_ceph_cluster_setup.pl b/tests/ceph/test_ceph_cluster_setup.pl
new file mode 100755
index 0000000..5bd47d3
--- /dev/null
+++ b/tests/ceph/test_ceph_cluster_setup.pl
@@ -0,0 +1,206 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<test_ceph_cluster_setup> - Set up a hyper-converged Ceph cluster on 3 nodes.
+
+=head1 DESCRIPTION
+
+This testcase sets up Ceph on the cluster nodes. It gives every node an address
+in a dedicated Ceph public and cluster network and writes the initial
+configuration with these networks. It creates a monitor and a manager on every
+node and an OSD on every Ceph disk.
+
+It verifies that the configuration on pmxcfs has the networks and is linked to
+from /etc/ceph on every node, that the monitors form a quorum and use the public
+network, that one manager is active with the others on standby, and that all
+OSDs are up and in and use the public network for clients and the cluster
+network for replication. Finally, it creates a pool together with an RBD
+storage and verifies that the storage is active on every node and the cluster
+reports HEALTH_OK.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use JSON;
+
+use lib '.';
+use Proxmox::Test::Ceph qw(
+ CEPH_CLUSTER_NETWORK
+ CEPH_PUBLIC_NETWORK
+ ceph_disks
+ ceph_osds
+ ceph_status
+ check_storage_active
+ create_managers
+ create_monitors
+ create_osds
+ create_pool
+ init_ceph
+ setup_ceph_network
+ wait_for_ceph
+);
+use Proxmox::Test::Helper qw(get_pve_instances);
+use Proxmox::Test::Testcase;
+use Proxmox::Test::Util qw(poll_until);
+
+my $pve_instances = get_pve_instances();
+die "not enough PVE instances" if scalar($pve_instances->@*) < 3;
+
+my $first = $pve_instances->[0];
+my $monitors = scalar($pve_instances->@*);
+my $pool = 'e2e-setup';
+my $osds = 0;
+$osds += scalar(ceph_disks($_)->@*) for $pve_instances->@*;
+
+my $public_network = CEPH_PUBLIC_NETWORK;
+my $cluster_network = CEPH_CLUSTER_NETWORK;
+my $addresses;
+
+# Commands like 'ceph' find the configuration on pmxcfs through this link.
+my $check_config_link = sub {
+ my ($instance) = @_;
+
+ my ($target) = eval { $instance->run_command('readlink /etc/ceph/ceph.conf') };
+ is(
+ $target,
+ "/etc/pve/ceph.conf\n",
+ "/etc/ceph/ceph.conf links to the config on pmxcfs on '"
+ . $instance->get_nodename() . "'",
+ );
+};
+
+Proxmox::Test::Testcase->new('ceph_cluster_setup')->with([{}])->step(
+ 'set up Ceph networks',
+ sub {
+ $addresses = setup_ceph_network($pve_instances);
+ pass('nodes reach each other in the Ceph public and cluster network');
+ },
+)->step(
+ 'initialize Ceph',
+ sub {
+ init_ceph(
+ $pve_instances,
+ network => $public_network,
+ 'cluster-network' => $cluster_network,
+ );
+
+ my $node = $first->get_nodename();
+ my $conf = $first->client()->get("/nodes/$node/ceph/cfg/raw", {});
+ like($conf, qr/^\s*fsid\s*=\s*[0-9a-f-]{36}\s*$/m, 'ceph.conf has a cluster fsid');
+ like(
+ $conf,
+ qr/^\s*public_network\s*=\s*\Q$public_network\E\s*$/m,
+ 'ceph.conf has the public network',
+ );
+ like(
+ $conf,
+ qr/^\s*cluster_network\s*=\s*\Q$cluster_network\E\s*$/m,
+ 'ceph.conf has the cluster network',
+ );
+
+ $check_config_link->($first);
+ },
+)->step(
+ 'create monitors',
+ sub {
+ create_monitors($pve_instances);
+
+ my @quorum = sort(ceph_status($first)->{quorum_names}->@*);
+ my @nodes = sort map { $_->get_nodename() } $pve_instances->@*;
+ is_deeply(\@quorum, \@nodes, 'every node has a monitor in the quorum');
+
+ my $node = $first->get_nodename();
+ my $monitor_list = $first->client()->get("/nodes/$node/ceph/mon", {});
+ for my $instance ($pve_instances->@*) {
+ my $name = $instance->get_nodename();
+ my ($monitor) = grep { $_->{name} eq $name } $monitor_list->@*;
+ like(
+ ($monitor // {})->{addr} // '',
+ qr/^\Q$addresses->{$name}->{public}\E:/,
+ "monitor on '$name' uses its address in the public network",
+ );
+
+ # The nodes other than the one that initialized Ceph should by now also have the link.
+ $check_config_link->($instance);
+ }
+ },
+)->step(
+ 'create managers',
+ sub {
+ create_managers($pve_instances);
+
+ # A new manager is not immediately registered as standby
+ my $mgrmap;
+ eval {
+ poll_until(
+ sub {
+ $mgrmap = ceph_status($first)->{mgrmap};
+ return $mgrmap->{available} && $mgrmap->{num_standbys} == $monitors - 1;
+ },
+ timeout => 60,
+ interval => 2,
+ description => 'one active and ' . ($monitors - 1) . ' standby managers',
+ );
+ };
+ diag("manager map: " . JSON::encode_json($mgrmap // {})) if $@;
+ ok($mgrmap->{available}, 'a manager is active');
+ is($mgrmap->{num_standbys}, $monitors - 1, 'the other managers are on standby');
+ },
+)->step(
+ 'create OSDs',
+ sub {
+ is(create_osds($pve_instances), $osds, "created $osds OSDs");
+ wait_for_ceph($first, monitors => $monitors, osds => $osds);
+ pass("all $osds OSDs are up and in");
+
+ for my $instance ($pve_instances->@*) {
+ my $name = $instance->get_nodename();
+ my $node_osds = ceph_osds($instance);
+ is(
+ scalar($node_osds->@*),
+ scalar(ceph_disks($instance)->@*),
+ "OSD tree lists an OSD for every Ceph disk under '$name'",
+ );
+
+ for my $osd ($node_osds->@*) {
+ my $metadata =
+ $instance->client()->get("/nodes/$name/ceph/osd/$osd->{id}/metadata", {})
+ ->{osd};
+ like(
+ $metadata->{front_addr} // '',
+ qr/\b\Q$addresses->{$name}->{public}\E:/,
+ "osd.$osd->{id} serves clients in the public network",
+ );
+ like(
+ $metadata->{back_addr} // '',
+ qr/\b\Q$addresses->{$name}->{cluster}\E:/,
+ "osd.$osd->{id} replicates in the cluster network",
+ );
+ }
+ }
+ },
+)->step(
+ 'create pool with RBD storage',
+ sub {
+ create_pool($first, $pool, add_storages => 1);
+
+ my $storage = $first->client()->get("/storage/$pool", {});
+ is($storage->{type}, 'rbd', "RBD storage '$pool' was added");
+ is($storage->{pool}, $pool, "storage '$pool' uses the new pool");
+
+ check_storage_active($_, $pool) for $pve_instances->@*;
+ },
+)->step(
+ 'verify cluster health',
+ sub {
+ wait_for_ceph($first, monitors => $monitors, osds => $osds, health_ok => 1);
+ pass('the Ceph cluster reports HEALTH_OK');
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index fba3e06..47c06b0 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -392,6 +392,17 @@ TestcaseConfig(
],
tag: "sdn",
),
+ Testcase(
+ name: "ceph_cluster_setup",
+ run: "./tests/ceph/test_ceph_cluster_setup.pl",
+ timeout: 1800,
+ instances: [
+ "pve1-cluster",
+ "pve2-cluster",
+ "pve3-cluster",
+ ],
+ tag: "ceph",
+ ),
],
rollback_between_tests: Some("pre-test"),
)
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* Re: [PATCH e2e-tests 26/38] tests: ceph: verify setting up a hyper-converged cluster
2026-11-04 19:03 ` [PATCH e2e-tests 26/38] tests: ceph: verify setting up a hyper-converged cluster Michael Köppl
@ 2026-11-06 12:50 ` Kefu Chai
2026-11-06 13:05 ` Michael Köppl
0 siblings, 1 reply; 47+ messages in thread
From: Kefu Chai @ 2026-11-06 12:50 UTC (permalink / raw)
To: m.koeppl; +Cc: pve-devel
Hi Michael,
> [snip]
> + create_pool($first, $pool, add_storages => 1);
> +
> + my $storage = $first->client()->get("/storage/$pool", {});
> + is($storage->{type}, 'rbd', "RBD storage '$pool' was added");
> + is($storage->{pool}, $pool, "storage '$pool' uses the new pool");
> [snip]
Could we also check that the pool has size=3 and min_size=2?
These are defaults set by the PVE pool creation API, so this would
be a small sanity check that they reach Ceph correctly.
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH e2e-tests 26/38] tests: ceph: verify setting up a hyper-converged cluster
2026-11-06 12:50 ` Kefu Chai
@ 2026-11-06 13:05 ` Michael Köppl
0 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-06 13:05 UTC (permalink / raw)
To: Kefu Chai, m.koeppl; +Cc: pve-devel
On Fri Nov 6, 2026 at 1:50 PM CET, Kefu Chai wrote:
> Hi Michael,
>
>> [snip]
>> + create_pool($first, $pool, add_storages => 1);
>> +
>> + my $storage = $first->client()->get("/storage/$pool", {});
>> + is($storage->{type}, 'rbd', "RBD storage '$pool' was added");
>> + is($storage->{pool}, $pool, "storage '$pool' uses the new pool");
>> [snip]
>
> Could we also check that the pool has size=3 and min_size=2?
> These are defaults set by the PVE pool creation API, so this would
> be a small sanity check that they reach Ceph correctly.
Thanks for pointing this out, will add it for v2!
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH e2e-tests 27/38] tests: ceph: verify guests on RBD storage
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (25 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 26/38] tests: ceph: verify setting up a hyper-converged cluster Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-06 13:01 ` Kefu Chai
2026-11-04 19:03 ` [PATCH e2e-tests 28/38] tests: ceph: verify setting up, using and destroying a CephFS Michael Köppl
` (11 subsequent siblings)
38 siblings, 1 reply; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Containers always access their volumes through krbd, so the container
part runs once, on a storage without krbd, and checks that the volume
is mapped. Other tests already cover operations on local-lvm, so this
test checks what the RBD storage does for them. A backup in snapshot
mode takes an RBD snapshot and removes it and its mapping afterwards.
Migrating the container in restart mode leaves its volume on the storage
and moves the mapping to the target node, as the file system on it must
only ever be mounted on one node. Converting the container to a template
gives it a base image with a protected snapshot, from which the image of
a linked clone is cloned. The template cannot be destroyed while the
linked clone exists.
For VMs, the krbd option of the storage decides whether QEMU accesses
the disk through librbd or through a mapped device. With librbd, QEMU
takes snapshots and resizes the disk of a running VM itself. So with
and without krbd, check that the disk is only mapped with krbd,
snapshot a running VM and roll the snapshot back, delete it while the
VM runs, and resize the disk of the running VM. Also live-migrate the
VM to another node, which must not copy its disk and, with krbd, must
move the mapping of its image to the target node.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/ceph/test_ceph_rbd_storage.pl | 597 ++++++++++++++++++++++++++++
tests/testcases.ron | 11 +
2 files changed, 608 insertions(+)
create mode 100755 tests/ceph/test_ceph_rbd_storage.pl
diff --git a/tests/ceph/test_ceph_rbd_storage.pl b/tests/ceph/test_ceph_rbd_storage.pl
new file mode 100755
index 0000000..9bdde6a
--- /dev/null
+++ b/tests/ceph/test_ceph_rbd_storage.pl
@@ -0,0 +1,597 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<test_ceph_rbd_storage> - Test guests on an RBD storage of a hyper-converged Ceph cluster.
+
+=head1 DESCRIPTION
+
+This testcase sets up Ceph on the cluster nodes and creates a pool. It adds RBD
+storages for the pool and verifies that they are active on every node.
+
+Containers always access their volumes through krbd, independent of the krbd
+option of the storage, so the container part runs once, on a storage without
+krbd. It verifies that the volume of a running container is mapped through krbd
+anyway. Other tests cover the generic container operations on local-lvm, so on
+that storage, it verifies what the RBD storage does for them:
+
+ - Backs the running container up in snapshot mode, which must take an RBD
+ snapshot of the volume and remove it and its mapping afterwards.
+ - Migrates the running container to another node and back in restart mode.
+ The volume must stay on the shared storage and only be mapped on the node
+ the container runs on.
+ - Converts the container to a template, which must rename its image to a base
+ image with a protected snapshot. Creates a linked clone, whose image must be
+ cloned from that snapshot, and verifies that the template cannot be
+ destroyed while the linked clone exists.
+
+For VMs, the krbd option decides whether QEMU accesses the disk through librbd
+or through a device mapped with krbd, which also changes how snapshots and
+resizing of a running VM are done. So the VM part runs once without and once
+with krbd. It writes data to the disk of a VM, starts it and verifies that the
+disk is only mapped with krbd. It then performs the following:
+
+ - Takes a snapshot of the running VM, which QEMU takes itself with librbd,
+ and verifies that rolling it back restores the data. Deletes the snapshot
+ while the VM is running again.
+ - Resizes the disk of the running VM.
+ - Live-migrates the VM to another node, which must not copy the disk since the
+ storage is shared. With krbd, the image must also move from the source
+ node's mapped devices to the target node's.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use JSON;
+
+use lib '.';
+use Proxmox::Test::Ceph qw(create_ceph_cluster create_pool check_storage_active);
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pve_instances);
+use Proxmox::Test::Testcase;
+use Proxmox::Test::Util qw(poll_until);
+
+my $pve_instances = get_pve_instances();
+die "not enough PVE instances" if scalar($pve_instances->@*) < 3;
+
+my $first = $pve_instances->[0];
+my $second = $pve_instances->[1];
+my $node = $first->get_nodename();
+my $target = $second->get_nodename();
+my $client = $first->client();
+my $pool = 'e2e-rbd';
+my $snapname = 'e2esnap';
+my $marker = 'e2e-rbd-container';
+
+my $wait_for_status = sub {
+ my ($instance, $type, $vmid, $status) = @_;
+ my $nodename = $instance->get_nodename();
+
+ poll_until(
+ sub {
+ $instance->client()->get("/nodes/$nodename/$type/$vmid/status/current", {})
+ ->{status} eq $status;
+ },
+ timeout => 120,
+ description => "$type $vmid on '$nodename' to report status '$status'",
+ );
+};
+
+# The RBD images mapped through krbd on $instance, by image name, and the mapped snapshots, by
+# '<image>@<snapshot>'.
+my $mapped_images = sub {
+ my ($instance) = @_;
+ my ($out) = $instance->run_command('rbd device list --format json');
+
+ my $mapped = {};
+ for my $device (decode_json($out)->@*) {
+ my $snap = $device->{snap} // '-';
+ $mapped->{ $snap eq '-' ? $device->{name} : "$device->{name}\@$snap" } = 1;
+ }
+ return $mapped;
+};
+
+# The snapshots of the RBD image $image, by name.
+my $rbd_snapshots = sub {
+ my ($image) = @_;
+ my ($out) = $first->run_command("rbd snap ls --format json $pool/$image");
+ return { map { $_->{name} => $_ } decode_json($out)->@* };
+};
+
+# The information 'rbd info' shows about the RBD image $image, or about a snapshot with
+# '$image@<snapshot>'.
+my $rbd_info = sub {
+ my ($image) = @_;
+ my ($out) = $first->run_command("rbd info --format json $pool/$image");
+ return decode_json($out);
+};
+
+# A checksum of the data of the RBD image $image.
+my $image_checksum = sub {
+ my ($image) = @_;
+ my ($out) =
+ $first->run_command("set -o pipefail; rbd export --no-progress $pool/$image - | sha256sum");
+ my ($sum) = split(/\s+/, $out // '');
+ return $sum;
+};
+
+# Overwrite part of the RBD image $image, which must not be in use, with random data.
+my $write_random_data = sub {
+ my ($image) = @_;
+
+ my ($dev) = $first->run_command("rbd device map $pool/$image");
+ chomp($dev);
+ eval {
+ $first->run_command(
+ "dd if=/dev/urandom of=$dev bs=1M count=8 seek=16 oflag=direct status=none");
+ };
+ my $err = $@;
+ $first->run_command("rbd device unmap $dev");
+ die $err if $err;
+};
+
+my $add_storage = sub {
+ my ($testcase, $content) = @_;
+
+ $client->post(
+ '/storage',
+ {
+ storage => $testcase->{storage},
+ type => 'rbd',
+ pool => $pool,
+ krbd => $testcase->{krbd},
+ content => $content,
+ },
+ );
+
+ check_storage_active($_, $testcase->{storage}) for $pve_instances->@*;
+};
+
+create_ceph_cluster($pve_instances);
+create_pool($first, $pool);
+
+Proxmox::Test::Testcase->new('ceph_rbd_containers')->with([
+ {
+ storage => 'e2e-rbd-ct',
+ krbd => 0,
+ },
+])->step(
+ 'add RBD storage without krbd',
+ sub {
+ my ($testcase) = @_;
+ $add_storage->($testcase, 'rootdir');
+ },
+ sub {
+ my ($testcase) = @_;
+ $client->delete("/storage/$testcase->{storage}");
+ },
+)->step(
+ 'create and start container',
+ sub {
+ my ($testcase) = @_;
+
+ $testcase->{ctid} = $client->get('/cluster/nextid');
+ $testcase->{ct_node} = $node;
+ my $upid = $client->post(
+ "/nodes/$node/lxc",
+ {
+ vmid => $testcase->{ctid},
+ ostemplate => CONTAINER_TEMPLATE,
+ rootfs => "$testcase->{storage}:2",
+ memory => 256,
+ unprivileged => 1,
+ features => 'nesting=1',
+ hostname => "e2e-ct-$testcase->{storage}",
+ },
+ );
+ $first->task_ok($upid, "created container $testcase->{ctid} on '$testcase->{storage}'")
+ or die "creating the container failed\n";
+
+ $first->start_guest('lxc', $testcase->{ctid});
+
+ my $conf = $client->get("/nodes/$node/lxc/$testcase->{ctid}/config", {});
+ my ($image) = ($conf->{rootfs} // '') =~ m/^[^:]+:([^,]+)/;
+ $testcase->{image} = $image;
+ ok(
+ $image && $mapped_images->($first)->{$image},
+ 'container volume is mapped through krbd, although the storage does not enable it',
+ );
+
+ $first->ct_exec($testcase->{ctid}, "sh -c 'echo $marker > /root/marker'");
+ },
+ sub {
+ my ($testcase) = @_;
+ return if !defined($testcase->{ctid});
+ $first->destroy_guest_on($testcase->{ct_node}, 'lxc', $testcase->{ctid});
+ },
+)->step(
+ 'back up running container in snapshot mode',
+ sub {
+ my ($testcase) = @_;
+ my $ctid = $testcase->{ctid};
+
+ my $upid = $client->post(
+ "/nodes/$node/vzdump",
+ { vmid => $ctid, storage => 'local', mode => 'snapshot' },
+ );
+ $first->task_ok($upid, 'backed up the running container') or die "backup failed\n";
+
+ my $content =
+ $client->get("/nodes/$node/storage/local/content", { content => 'backup' });
+ ($testcase->{backup}) = map { $_->{volid} } grep { $_->{vmid} == $ctid } $content->@*;
+ ok($testcase->{backup}, 'backup archive exists on local storage');
+
+ # The backup reads the container's data from an RBD snapshot 'vzdump' of its volume, which
+ # it maps through krbd.
+ my $log = $client->get("/nodes/$node/tasks/$upid/log", { limit => 1000 });
+ ok(
+ (grep { ($_->{t} // '') =~ m/create storage snapshot 'vzdump'/ } $log->@*),
+ 'backup used a storage snapshot of the container volume',
+ );
+
+ my $image = $testcase->{image};
+ ok(
+ !$rbd_snapshots->($image)->{vzdump},
+ 'backup snapshot is removed from the RBD image',
+ );
+ ok(
+ !$mapped_images->($first)->{"$image\@vzdump"},
+ 'backup snapshot is no longer mapped',
+ );
+ },
+ sub {
+ my ($testcase) = @_;
+ $first->destroy_volume('local', $testcase->{backup}) if $testcase->{backup};
+ },
+)->step(
+ 'migrate running container in restart mode',
+ sub {
+ my ($testcase) = @_;
+ my $ctid = $testcase->{ctid};
+ my $image = $testcase->{image};
+ my $volid = "$testcase->{storage}:$image";
+
+ # Migrate the container to another node (and back, so that the remaining steps find it on
+ # the first node again).
+ for my $instances ([$first, $second], [$second, $first]) {
+ my ($source, $dest) = $instances->@*;
+ my $from = $source->get_nodename();
+ my $to = $dest->get_nodename();
+
+ my $upid = $client->post(
+ "/nodes/$from/lxc/$ctid/migrate",
+ { target => $to, restart => 1, timeout => 60 },
+ );
+ $source->task_ok($upid, "migrated container $ctid to '$to' in restart mode")
+ or die "migrating the container failed\n";
+ $testcase->{ct_node} = $to;
+
+ $wait_for_status->($dest, 'lxc', $ctid, 'running');
+
+ my $log = $client->get("/nodes/$from/tasks/$upid/log", { limit => 1000 });
+ my $kept =
+ grep { ($_->{t} // '') =~ m/'\Q$volid\E' is on shared storage/ } $log->@*;
+ ok($kept, "migration to '$to' leaves the volume on the shared storage");
+ like(
+ $client->get("/nodes/$to/lxc/$ctid/config", {})->{rootfs} // '',
+ qr/^\Q$volid\E,/,
+ "container uses the same volume on '$to'",
+ );
+
+ # The file system on the volume must only ever be mounted on one node.
+ ok($mapped_images->($dest)->{$image}, "container volume is mapped on '$to'");
+ ok(
+ !$mapped_images->($source)->{$image},
+ "container volume is no longer mapped on '$from'",
+ );
+
+ is(
+ $dest->ct_exec($ctid, 'cat /root/marker'),
+ "$marker\n",
+ "container data is intact on '$to'",
+ );
+ }
+ },
+)->step(
+ 'convert container to template',
+ sub {
+ my ($testcase) = @_;
+ my $ctid = $testcase->{ctid};
+
+ # Only a stopped container can become a template.
+ $first->stop_guest('lxc', $ctid);
+
+ # The conversion runs in a worker task, but the API call does not return its task ID.
+ $client->post("/nodes/$node/lxc/$ctid/template", {});
+
+ my $conf;
+ eval {
+ poll_until(
+ sub {
+ $conf = $client->get("/nodes/$node/lxc/$ctid/config", {});
+ return $conf->{template};
+ },
+ timeout => 60,
+ interval => 2,
+ description => "container $ctid to be converted to a template",
+ );
+ };
+ is($conf->{template}, 1, "container $ctid is marked as template")
+ or die "converting the container to a template failed\n";
+
+ # Linked clones of the template are cloned from the protected snapshot of the base image.
+ my $base = $testcase->{image} =~ s/^vm-/base-/r;
+ my ($volume) = ($conf->{rootfs} // '') =~ m/^[^:]+:([^,]+)/;
+ is($volume, $base, "container volume is renamed to the base image '$base'");
+ $testcase->{base} = $base;
+
+ my $info = eval { $rbd_info->("$base\@__base__") } // {};
+ is($info->{protected}, 'true', "base image has the protected snapshot '__base__'");
+ },
+)->step(
+ 'create linked clone',
+ sub {
+ my ($testcase) = @_;
+
+ $testcase->{clone} = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/lxc/$testcase->{ctid}/clone",
+ {
+ newid => $testcase->{clone},
+ hostname => "e2e-clone-$testcase->{storage}",
+ full => 0,
+ },
+ );
+ $first->task_ok($upid, "created linked clone $testcase->{clone}")
+ or die "creating the linked clone failed\n";
+
+ my $conf = $client->get("/nodes/$node/lxc/$testcase->{clone}/config", {});
+ my ($base, $image) = ($conf->{rootfs} // '') =~ m!^[^:]+:([^/,]+)/([^,]+)!;
+ is(
+ $base,
+ $testcase->{base},
+ 'linked clone volume refers to the base image of the template',
+ );
+
+ my $parent = eval { $rbd_info->($image // '')->{parent} } // {};
+ is_deeply(
+ [$parent->@{qw(pool image snapshot)}],
+ [$pool, $testcase->{base}, '__base__'],
+ "RBD image of the linked clone is cloned from '$testcase->{base}\@__base__'",
+ );
+ },
+ sub {
+ my ($testcase) = @_;
+ $first->destroy_guest('lxc', $testcase->{clone}) if defined($testcase->{clone});
+ },
+)->step(
+ 'verify linked clone',
+ sub {
+ my ($testcase) = @_;
+ my $clone = $testcase->{clone};
+
+ $first->start_guest('lxc', $clone);
+
+ is(
+ $first->ct_exec($clone, 'cat /root/marker'),
+ "$marker\n",
+ 'linked clone has the data of the template',
+ );
+
+ $first->stop_guest('lxc', $clone);
+ },
+)->step(
+ 'template is protected while a linked clone exists',
+ sub {
+ my ($testcase) = @_;
+ my $ctid = $testcase->{ctid};
+
+ my $upid = eval { $client->delete("/nodes/$node/lxc/$ctid", {}) };
+ my $status = defined($upid) ? $first->wait_for_task($upid, 1) : "API error: $@";
+ ok(!$first->task_successful($status), 'destroying the template is refused');
+ like(
+ $status,
+ qr/still in use by linked clone/,
+ 'destroying the template is refused because of the linked clone',
+ );
+
+ my $conf = eval { $client->get("/nodes/$node/lxc/$ctid/config", {}) };
+ ok($conf && $conf->{template}, 'template still exists');
+ },
+)->run();
+
+Proxmox::Test::Testcase->new('ceph_rbd_vms')->with([
+ {
+ storage => 'e2e-rbd-librbd',
+ krbd => 0,
+ },
+ {
+ storage => 'e2e-rbd-krbd',
+ krbd => 1,
+ },
+])->step(
+ 'add RBD storage',
+ sub {
+ my ($testcase) = @_;
+ $add_storage->($testcase, 'images');
+ },
+ sub {
+ my ($testcase) = @_;
+ $client->delete("/storage/$testcase->{storage}");
+ },
+)->step(
+ 'create VM and write data to its disk',
+ sub {
+ my ($testcase) = @_;
+
+ $testcase->{vmid} = $client->get('/cluster/nextid');
+ $testcase->{vm_node} = $node;
+ my $upid = $client->post(
+ "/nodes/$node/qemu",
+ {
+ vmid => $testcase->{vmid},
+ memory => 512,
+ scsihw => 'virtio-scsi-pci',
+ scsi0 => "$testcase->{storage}:1",
+ },
+ );
+ $first->task_ok($upid, "created VM $testcase->{vmid} on '$testcase->{storage}'")
+ or die "creating the VM failed\n";
+
+ my $conf = $client->get("/nodes/$node/qemu/$testcase->{vmid}/config", {});
+ ($testcase->{disk}) = ($conf->{scsi0} // '') =~ m/^([^,]+)/;
+ ($testcase->{image}) = ($testcase->{disk} // '') =~ m/^[^:]+:(.+)$/;
+ ok($testcase->{image}, "VM disk is '$testcase->{disk}'") or die "VM has no disk\n";
+
+ $write_random_data->($testcase->{image});
+ $testcase->{checksum} = $image_checksum->($testcase->{image});
+ },
+ sub {
+ my ($testcase) = @_;
+ return if !defined($testcase->{vmid});
+ $first->destroy_guest_on($testcase->{vm_node}, 'qemu', $testcase->{vmid});
+ },
+)->step(
+ 'start VM',
+ sub {
+ my ($testcase) = @_;
+
+ $first->start_guest('qemu', $testcase->{vmid});
+
+ my $mapped = $mapped_images->($first)->{ $testcase->{image} };
+ if ($testcase->{krbd}) {
+ ok($mapped, "VM disk is mapped through krbd on '$node'");
+ } else {
+ ok(!$mapped, 'VM disk is not mapped, as QEMU accesses it through librbd');
+ }
+ },
+)->step(
+ 'snapshot running VM',
+ sub {
+ my ($testcase) = @_;
+ my $vmid = $testcase->{vmid};
+
+ my $upid = $client->post("/nodes/$node/qemu/$vmid/snapshot", { snapname => $snapname });
+ $first->task_ok($upid, 'created snapshot of the running VM')
+ or die "creating the snapshot failed\n";
+
+ my $log = $client->get("/nodes/$node/tasks/$upid/log", { limit => 1000 });
+ my $by_qemu = grep { ($_->{t} // '') eq 'internal qemu snapshot' } $log->@*;
+ if ($testcase->{krbd}) {
+ ok(!$by_qemu, 'the storage takes the snapshot of the mapped disk');
+ } else {
+ ok($by_qemu, 'QEMU takes the snapshot itself through librbd');
+ }
+
+ ok(
+ $rbd_snapshots->($testcase->{image})->{$snapname},
+ "RBD image of the disk has the snapshot '$snapname'",
+ );
+ },
+)->step(
+ 'roll back snapshot',
+ sub {
+ my ($testcase) = @_;
+ my $vmid = $testcase->{vmid};
+
+ $first->stop_guest('qemu', $vmid);
+
+ $write_random_data->($testcase->{image});
+ isnt(
+ $image_checksum->($testcase->{image}),
+ $testcase->{checksum},
+ 'data on the disk changed after the snapshot',
+ );
+
+ my $upid = $client->post("/nodes/$node/qemu/$vmid/snapshot/$snapname/rollback", {});
+ $first->task_ok($upid, 'rolled back the snapshot') or die "rolling back failed\n";
+
+ is(
+ $image_checksum->($testcase->{image}),
+ $testcase->{checksum},
+ 'rollback restored the data from the time of the snapshot',
+ );
+ },
+)->step(
+ 'delete snapshot of running VM',
+ sub {
+ my ($testcase) = @_;
+ my $vmid = $testcase->{vmid};
+
+ $first->start_guest('qemu', $vmid);
+
+ my $upid = $client->delete("/nodes/$node/qemu/$vmid/snapshot/$snapname", {});
+ $first->task_ok($upid, 'deleted the snapshot of the running VM')
+ or die "deleting the snapshot failed\n";
+
+ is_deeply(
+ [keys $rbd_snapshots->($testcase->{image})->%*],
+ [],
+ 'snapshot is removed from the RBD image',
+ );
+ },
+)->step(
+ 'resize disk of running VM',
+ sub {
+ my ($testcase) = @_;
+ my $vmid = $testcase->{vmid};
+
+ my $upid =
+ $client->put("/nodes/$node/qemu/$vmid/resize", { disk => 'scsi0', size => '+1G' });
+ $first->task_ok($upid, 'resized the disk of the running VM by 1 GiB')
+ or die "resizing the disk failed\n";
+
+ like(
+ $client->get("/nodes/$node/qemu/$vmid/config", {})->{scsi0} // '',
+ qr/\bsize=2G\b/,
+ 'config reports the new disk size',
+ );
+
+ is(
+ $rbd_info->($testcase->{image})->{size},
+ 2 * 1024 * 1024 * 1024,
+ 'RBD image has the new size',
+ );
+ },
+)->step(
+ 'live-migrate VM',
+ sub {
+ my ($testcase) = @_;
+ my $vmid = $testcase->{vmid};
+
+ my $upid =
+ $client->post("/nodes/$node/qemu/$vmid/migrate", { target => $target, online => 1 });
+ $first->task_ok($upid, "live-migrated VM $vmid to '$target'")
+ or die "migrating the VM failed\n";
+ $testcase->{vm_node} = $target;
+
+ $wait_for_status->($second, 'qemu', $vmid, 'running');
+
+ my $conf = $second->client()->get("/nodes/$target/qemu/$vmid/config", {});
+ like(
+ $conf->{scsi0} // '',
+ qr/^\Q$testcase->{disk}\E\b/,
+ 'VM still uses the same disk on the shared storage',
+ );
+
+ my $log = $second->client()->get("/nodes/$node/tasks/$upid/log", { limit => 10000 });
+ ok(
+ !(grep { ($_->{t} // '') =~ m/drive mirror|starting storage migration/ } $log->@*),
+ 'migration did not copy the disk',
+ );
+
+ my $mapped_on_target = $mapped_images->($second)->{ $testcase->{image} };
+ my $mapped_on_source = $mapped_images->($first)->{ $testcase->{image} };
+ if ($testcase->{krbd}) {
+ ok($mapped_on_target, "VM disk is mapped on '$target'");
+ ok(!$mapped_on_source, "VM disk is no longer mapped on '$node'");
+ } else {
+ ok(!$mapped_on_target && !$mapped_on_source, 'VM disk is mapped on neither node');
+ }
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index 47c06b0..fec8667 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -403,6 +403,17 @@ TestcaseConfig(
],
tag: "ceph",
),
+ Testcase(
+ name: "ceph_rbd_storage",
+ run: "./tests/ceph/test_ceph_rbd_storage.pl",
+ timeout: 2400,
+ instances: [
+ "pve1-cluster",
+ "pve2-cluster",
+ "pve3-cluster",
+ ],
+ tag: "ceph",
+ ),
],
rollback_between_tests: Some("pre-test"),
)
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* Re: [PATCH e2e-tests 27/38] tests: ceph: verify guests on RBD storage
2026-11-04 19:03 ` [PATCH e2e-tests 27/38] tests: ceph: verify guests on RBD storage Michael Köppl
@ 2026-11-06 13:01 ` Kefu Chai
2026-11-06 13:07 ` Michael Köppl
0 siblings, 1 reply; 47+ messages in thread
From: Kefu Chai @ 2026-11-06 13:01 UTC (permalink / raw)
To: m.koeppl; +Cc: pve-devel
Hi Michael,
> [snip]
> + $first->start_guest('qemu', $testcase->{vmid});
> +
> + my $mapped = $mapped_images->($first)->{ $testcase->{image} };
> + if ($testcase->{krbd}) {
> + ok($mapped, "VM disk is mapped through krbd on '$node'");
> + } else {
> + ok(!$mapped, 'VM disk is not mapped, as QEMU accesses it through librbd');
> + }
> [snip]
Could we mirror the container test here: boot a small guest, write
a marker inside it, and check the marker after migration, for both
krbd settings?
That would give us the same basic data check for VMs and exercise
guest I/O through both QEMU storage paths.
^ permalink raw reply [flat|nested] 47+ messages in thread* Re: [PATCH e2e-tests 27/38] tests: ceph: verify guests on RBD storage
2026-11-06 13:01 ` Kefu Chai
@ 2026-11-06 13:07 ` Michael Köppl
0 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-06 13:07 UTC (permalink / raw)
To: Kefu Chai, m.koeppl; +Cc: pve-devel
On Fri Nov 6, 2026 at 2:01 PM CET, Kefu Chai wrote:
> Hi Michael,
>
>> [snip]
>> + $first->start_guest('qemu', $testcase->{vmid});
>> +
>> + my $mapped = $mapped_images->($first)->{ $testcase->{image} };
>> + if ($testcase->{krbd}) {
>> + ok($mapped, "VM disk is mapped through krbd on '$node'");
>> + } else {
>> + ok(!$mapped, 'VM disk is not mapped, as QEMU accesses it through librbd');
>> + }
>> [snip]
>
> Could we mirror the container test here: boot a small guest, write
> a marker inside it, and check the marker after migration, for both
> krbd settings?
>
> That would give us the same basic data check for VMs and exercise
> guest I/O through both QEMU storage paths.
I'm not even sure why I left this out here. I'll add it for v2 since
it's definitely worthwhile to have this additional check.
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH e2e-tests 28/38] tests: ceph: verify setting up, using and destroying a CephFS
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (26 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 27/38] tests: ceph: verify guests on RBD storage Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 29/38] tests: ceph: verify replacing an OSD Michael Köppl
` (10 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Create a metadata server on every node and a CephFS with a storage
for it. Check that the CephFS gets its own pools, that one metadata
server becomes active with the others on standby, and that the
storage is mounted on every node. Backup a container to the storage
and restore it on another node, which lists the backup, as the
storage is shared.
Destroy the CephFS as the documentation describes. Destroying it
together with its storage is refused while the storage is enabled
(based on `man pveceph(1)`). Once the storage is disabled and unmounted
and the metadata servers are stopped, it succeeds and removes the
storage and the pools too.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/ceph/test_ceph_cephfs.pl | 311 +++++++++++++++++++++++++++++++++
tests/testcases.ron | 11 ++
2 files changed, 322 insertions(+)
create mode 100755 tests/ceph/test_ceph_cephfs.pl
diff --git a/tests/ceph/test_ceph_cephfs.pl b/tests/ceph/test_ceph_cephfs.pl
new file mode 100755
index 0000000..40fdb3c
--- /dev/null
+++ b/tests/ceph/test_ceph_cephfs.pl
@@ -0,0 +1,311 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<test_ceph_cephfs> - Test setting up, using and destroying a CephFS.
+
+=head1 DESCRIPTION
+
+This testcase sets up Ceph on the cluster nodes and creates an MDS on every
+node, which are all on standby while there is no CephFS. It then creates a
+CephFS together with a storage for it and verifies that the CephFS has its
+own data and metadata pool, that one MDS became active with the others
+still on standby (only one can be active at a time), and that the storage is
+active and mounted on every node.
+
+It backs up a container on the first node to the CephFS storage and restores
+the backup on the second node, which lists it as the storage is shared.
+
+Finally, it destroys the CephFS as the documentation describes. Destroying it
+together with its storage must be refused while the storage is enabled. Once
+the storage is disabled and unmounted on every node and all MDS are stopped,
+destroying the CephFS must remove it together with its storage and its pools.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use JSON;
+
+use lib '.';
+use Proxmox::Test::Ceph qw(check_storage_active create_ceph_cluster wait_for_ceph);
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pve_instances);
+use Proxmox::Test::Testcase;
+use Proxmox::Test::Util qw(poll_until);
+
+my $pve_instances = get_pve_instances();
+die "not enough PVE instances" if scalar($pve_instances->@*) < 3;
+
+my $first = $pve_instances->[0];
+my $second = $pve_instances->[1];
+my $node = $first->get_nodename();
+my $target = $second->get_nodename();
+my $client = $first->client();
+
+my $fs = 'e2e-cephfs';
+my $mountpoint = "/mnt/pve/$fs";
+my $marker = 'e2e-cephfs-backup';
+
+my $ctid;
+my $restored_ctid;
+my $backup_volid;
+
+# The state of every MDS by name, as reported by the API.
+my $mds_states = sub {
+ my $list = $client->get("/nodes/$node/ceph/mds", {});
+ return { map { $_->{name} => $_->{state} // '' } $list->@* };
+};
+
+my $wait_for_mds_states = sub {
+ my ($active, $standby) = @_;
+
+ my $states;
+ eval {
+ poll_until(
+ sub {
+ $states = $mds_states->();
+ my @values = values $states->%*;
+ return scalar(grep { $_ eq 'up:active' } @values) == $active
+ && scalar(grep { $_ eq 'up:standby' } @values) == $standby;
+ },
+ timeout => 120,
+ interval => 2,
+ description => "$active active and $standby standby MDS",
+ );
+ };
+ diag('MDS states: ' . encode_json($states // {})) if $@;
+ return $states;
+};
+
+my $is_mounted = sub {
+ my ($instance) = @_;
+ my ($fstype) = eval { $instance->run_command("findmnt -n -o FSTYPE $mountpoint") };
+ return ($fstype // '') eq "ceph\n";
+};
+
+my $cephfs_entry = sub {
+ my ($entry) = grep { $_->{name} eq $fs } $client->get("/nodes/$node/ceph/fs", {})->@*;
+ return $entry;
+};
+
+Proxmox::Test::Testcase->new('ceph_cephfs')->before_all(sub {
+ create_ceph_cluster($pve_instances);
+})->with([{}])->step(
+ 'create metadata servers',
+ sub {
+ for my $instance ($pve_instances->@*) {
+ my $name = $instance->get_nodename();
+ my $upid = $instance->client()->post("/nodes/$name/ceph/mds/$name", {});
+ $instance->task_ok($upid, "created MDS on '$name'")
+ or die "creating the MDS on '$name' failed\n";
+ }
+
+ my $states = $wait_for_mds_states->(0, scalar($pve_instances->@*));
+ is_deeply(
+ [sort keys $states->%*],
+ [sort map { $_->get_nodename() } $pve_instances->@*],
+ 'every node has an MDS',
+ );
+ ok(
+ !(grep { $_ ne 'up:standby' } values $states->%*),
+ 'all MDS are on standby without a CephFS',
+ );
+ },
+)->step(
+ 'create CephFS with storage',
+ sub {
+ my $upid =
+ $client->post("/nodes/$node/ceph/fs/$fs", { pg_num => 32, 'add-storage' => 1 });
+ $first->task_ok($upid, "created CephFS '$fs'") or die "creating the CephFS failed\n";
+
+ my $entry = $cephfs_entry->();
+ ok($entry, "CephFS '$fs' is listed") or die "CephFS '$fs' not found\n";
+ is($entry->{metadata_pool}, "${fs}_metadata", 'CephFS has its own metadata pool');
+ is_deeply($entry->{data_pools}, ["${fs}_data"], 'CephFS has its own data pool');
+
+ my $states = $wait_for_mds_states->(1, scalar($pve_instances->@*) - 1);
+ my ($active) = grep { $states->{$_} eq 'up:active' } keys $states->%*;
+ ok($active, "MDS on '" . ($active // '') . "' became active for the CephFS");
+
+ my $storage = $client->get("/storage/$fs", {});
+ is($storage->{type}, 'cephfs', "CephFS storage '$fs' was added");
+ is($storage->{'fs-name'}, $fs, "storage '$fs' uses the new CephFS");
+ like($storage->{content} // '', qr/\bbackup\b/, "storage '$fs' can hold backups");
+
+ # The fs is not immediately available on all nodes. It is mounted on the first and mounted
+ # shortly after on the other nodes by pvestatd. When querying this too early, the storage
+ # is reported ad inactive.
+ for my $instance ($pve_instances->@*) {
+ my $name = $instance->get_nodename();
+ eval {
+ poll_until(
+ sub { $is_mounted->($instance) },
+ timeout => 60,
+ interval => 2,
+ description => "CephFS to be mounted on '$name'",
+ );
+ };
+ ok($is_mounted->($instance), "CephFS is mounted at $mountpoint on '$name'");
+ check_storage_active($instance, $fs);
+ }
+
+ wait_for_ceph($first);
+ },
+)->step(
+ 'back up container to CephFS',
+ sub {
+ $ctid = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/lxc",
+ {
+ vmid => $ctid,
+ ostemplate => CONTAINER_TEMPLATE,
+ rootfs => 'local-lvm:2',
+ memory => 256,
+ unprivileged => 1,
+ features => 'nesting=1',
+ hostname => 'ct-cephfs-backup',
+ },
+ );
+ $first->task_ok($upid, "created container $ctid") or die "creating container failed\n";
+
+ $first->start_guest('lxc', $ctid);
+ $first->ct_exec($ctid, "sh -c 'echo $marker > /root/marker'");
+
+ $upid = $client->post(
+ "/nodes/$node/vzdump",
+ { vmid => $ctid, storage => $fs, mode => 'snapshot' },
+ );
+ $first->task_ok($upid, 'backed up the container to CephFS') or die "backup failed\n";
+
+ my $content = $client->get("/nodes/$node/storage/$fs/content", { content => 'backup' });
+ ($backup_volid) = map { $_->{volid} } grep { ($_->{vmid} // 0) == $ctid } $content->@*;
+ ok($backup_volid, 'backup is listed on the CephFS storage')
+ or die "backup not found\n";
+ },
+ sub {
+ $first->destroy_guest('lxc', $ctid) if defined($ctid);
+ },
+)->step(
+ 'restore backup on another node',
+ sub {
+ my $target_client = $second->client();
+
+ my $content =
+ $target_client->get("/nodes/$target/storage/$fs/content", { content => 'backup' });
+ ok(
+ (grep { $_->{volid} eq $backup_volid } $content->@*),
+ "node '$target' lists the backup on the shared storage",
+ );
+
+ $restored_ctid = $client->get('/cluster/nextid');
+ my $upid = $target_client->post(
+ "/nodes/$target/lxc",
+ {
+ vmid => $restored_ctid,
+ ostemplate => $backup_volid,
+ restore => 1,
+ storage => 'local-lvm',
+ },
+ );
+ $second->task_ok($upid, "restored the backup as container $restored_ctid on '$target'")
+ or die "restoring the backup failed\n";
+
+ $second->start_guest('lxc', $restored_ctid);
+ is(
+ $second->ct_exec($restored_ctid, 'cat /root/marker'),
+ "$marker\n",
+ 'restored container has the data from the backup',
+ );
+ $second->stop_guest('lxc', $restored_ctid);
+ },
+ sub {
+ $second->destroy_guest('lxc', $restored_ctid) if defined($restored_ctid);
+ },
+)->step(
+ 'destroying CephFS with an enabled storage is refused',
+ sub {
+ my $upid = eval {
+ $client->delete(
+ "/nodes/$node/ceph/fs/$fs",
+ { 'remove-storages' => 1, 'remove-pools' => 1 },
+ );
+ };
+ my $status = defined($upid) ? $first->wait_for_task($upid, 1) : "API error: $@";
+ ok(!$first->task_successful($status), 'destroying the CephFS is refused');
+ like($status, qr/not disabled/, 'destroying is refused because the storage is enabled');
+ ok($cephfs_entry->(), 'CephFS still exists');
+ },
+)->step(
+ 'destroy CephFS',
+ sub {
+ $client->put("/storage/$fs", { disable => 1 });
+
+ # Done manually according to the admin guide.
+ for my $instance ($pve_instances->@*) {
+ $instance->run_command("umount $mountpoint") if $is_mounted->($instance);
+ ok(
+ !$is_mounted->($instance),
+ "CephFS is unmounted on '" . $instance->get_nodename() . "'",
+ );
+ }
+
+ # Stop the standby MDS first, so that none of them takes over from the active one.
+ my $states = $mds_states->();
+ my @standby = grep { $states->{$_} ne 'up:active' } sort keys $states->%*;
+ my @active = grep { $states->{$_} eq 'up:active' } sort keys $states->%*;
+ for my $name (@standby, @active) {
+ my ($instance) = grep { $_->get_nodename() eq $name } $pve_instances->@*;
+ my $upid =
+ $instance->client()->post("/nodes/$name/ceph/stop", { service => "mds.$name" });
+ $instance->task_ok($upid, "stopped MDS on '$name'")
+ or die "stopping the MDS on '$name' failed\n";
+ }
+
+ poll_until(
+ sub {
+ my ($out) = $first->run_command("ceph fs get $fs --format json");
+ return !(keys(decode_json($out)->{mdsmap}->{up}->%*));
+ },
+ timeout => 120,
+ interval => 2,
+ description => "CephFS '$fs' to have no MDS up",
+ );
+
+ my $upid = $client->delete(
+ "/nodes/$node/ceph/fs/$fs",
+ { 'remove-storages' => 1, 'remove-pools' => 1 },
+ );
+ $first->task_ok($upid, "destroyed CephFS '$fs'")
+ or die "destroying the CephFS failed\n";
+
+ ok(!$cephfs_entry->(), 'CephFS is gone');
+
+ my $storages = $client->get('/storage', {});
+ ok(!(grep { $_->{storage} eq $fs } $storages->@*), "storage '$fs' is removed");
+
+ # Pool deletion runs in the background.
+ my %pools;
+ eval {
+ poll_until(
+ sub {
+ %pools =
+ map { $_->{pool_name} => 1 } $client->get("/nodes/$node/ceph/pool", {})->@*;
+ return !$pools{"${fs}_data"} && !$pools{"${fs}_metadata"};
+ },
+ timeout => 60,
+ interval => 2,
+ description => 'the pools of the CephFS to be removed',
+ );
+ };
+ ok(
+ !$pools{"${fs}_data"} && !$pools{"${fs}_metadata"},
+ 'pools of the CephFS are removed',
+ );
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index fec8667..9a94d11 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -414,6 +414,17 @@ TestcaseConfig(
],
tag: "ceph",
),
+ Testcase(
+ name: "ceph_cephfs",
+ run: "./tests/ceph/test_ceph_cephfs.pl",
+ timeout: 1800,
+ instances: [
+ "pve1-cluster",
+ "pve2-cluster",
+ "pve3-cluster",
+ ],
+ tag: "ceph",
+ ),
],
rollback_between_tests: Some("pre-test"),
)
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 29/38] tests: ceph: verify replacing an OSD
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (27 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 28/38] tests: ceph: verify setting up, using and destroying a CephFS Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-06 13:14 ` Kefu Chai
2026-11-04 19:03 ` [PATCH e2e-tests 30/38] tests: ceph: drop the outdated Ceph setup Michael Köppl
` (9 subsequent siblings)
38 siblings, 1 reply; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Replace an OSD as the documentation describes, while a container
runs on an RBD storage. Mark the OSD out, wait until its placement
groups have moved, check that it is safe to stop and stop it, then
destroy it with cleanup and create a new OSD on the same disk.
Destroying with cleanup must remove the OSD from the OSD tree, leave
neither an enabled service nor a data directory behind and leave the
disk unused, so that it can be reused right away. Afterwards, the
cluster must be healthy with all OSDs up and in, and the container
must have kept running with its data intact.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/ceph/test_ceph_osd_replace.pl | 237 ++++++++++++++++++++++++++++
tests/testcases.ron | 11 ++
2 files changed, 248 insertions(+)
create mode 100755 tests/ceph/test_ceph_osd_replace.pl
diff --git a/tests/ceph/test_ceph_osd_replace.pl b/tests/ceph/test_ceph_osd_replace.pl
new file mode 100755
index 0000000..51857ec
--- /dev/null
+++ b/tests/ceph/test_ceph_osd_replace.pl
@@ -0,0 +1,237 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<test_ceph_osd_replace> - Test replacing an OSD of a hyper-converged Ceph cluster.
+
+=head1 DESCRIPTION
+
+This testcase sets up Ceph on the cluster nodes with a pool and an RBD storage
+and starts a container on that storage. It then replaces an OSD on the second
+node as the documentation describes. It marks the OSD out, waits until Ceph has
+moved all placement groups off it, checks that it is safe to stop and stops it.
+It destroys the OSD with cleanup, which must remove the OSD from the OSD tree,
+leave neither an enabled service nor a data directory of it behind and leave
+its disk unused, so that a new OSD can be created on the same disk right away.
+
+Afterwards, the cluster must report HEALTH_OK with all OSDs up and in again.
+The container must have kept running with its data intact.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Ceph qw(
+ ceph_disks
+ ceph_osds
+ ceph_status
+ create_ceph_cluster
+ create_pool
+ wait_for_ceph
+);
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pve_instances);
+use Proxmox::Test::Testcase;
+use Proxmox::Test::Util qw(poll_until);
+
+my $pve_instances = get_pve_instances();
+die "not enough PVE instances" if scalar($pve_instances->@*) < 3;
+
+my $first = $pve_instances->[0];
+my $second = $pve_instances->[1];
+my $node = $first->get_nodename();
+my $osd_node = $second->get_nodename();
+my $client = $first->client();
+my $osd_client = $second->client();
+
+my $pool = 'e2e-replace';
+my $marker = 'e2e-osd-replace';
+
+my $osds;
+my $ctid;
+my $osd = {};
+
+# The entry of osd.$id in the OSD tree of the second (!) node, or undef if it is not listed.
+my $osd_entry = sub {
+ my ($id) = @_;
+ my ($entry) = grep { $_->{id} == $id } ceph_osds($second)->@*;
+ return $entry;
+};
+
+# The entry of the disk $devpath in the disk list of the second (!) node.
+my $disk_entry = sub {
+ my ($devpath) = @_;
+ my ($entry) =
+ grep { $_->{devpath} eq $devpath } $osd_client->get("/nodes/$osd_node/disks/list", {})->@*;
+ return $entry // {};
+};
+
+Proxmox::Test::Testcase->new('ceph_osd_replace')->before_all(sub {
+ $osds = create_ceph_cluster($pve_instances);
+ create_pool($first, $pool, add_storages => 1);
+})->with([{}])->step(
+ 'start container on RBD storage',
+ sub {
+ $ctid = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/lxc",
+ {
+ vmid => $ctid,
+ ostemplate => CONTAINER_TEMPLATE,
+ rootfs => "$pool:2",
+ memory => 256,
+ unprivileged => 1,
+ features => 'nesting=1',
+ hostname => 'ct-osd-replace',
+ },
+ );
+ $first->task_ok($upid, "created container $ctid on '$pool'")
+ or die "creating the container failed\n";
+
+ $first->start_guest('lxc', $ctid);
+ $first->ct_exec($ctid, "sh -c 'echo $marker > /root/marker'");
+ },
+ sub {
+ $first->destroy_guest('lxc', $ctid) if defined($ctid);
+ },
+)->step(
+ 'mark OSD out',
+ sub {
+ my ($entry) = ceph_osds($second)->@*;
+ die "no OSD on '$osd_node'\n" if !$entry;
+ my $id = $osd->{id} = $entry->{id};
+
+ my ($disk) = grep { ($_->{osdid} // -1) == $id }
+ $osd_client->get("/nodes/$osd_node/disks/list", {})->@*;
+ $osd->{disk} = ($disk // {})->{devpath};
+ ok($osd->{disk}, "disk list shows the disk of osd.$id") or die "disk not found\n";
+
+ $osd_client->post("/nodes/$osd_node/ceph/osd/$id/out", {});
+
+ poll_until(
+ sub { !($osd_entry->($id) // {})->{in} },
+ timeout => 30,
+ interval => 2,
+ description => "osd.$id to be out",
+ );
+ pass("osd.$id is out");
+ },
+)->step(
+ 'wait for the data to move off the OSD',
+ sub {
+ my $id = $osd->{id};
+
+ poll_until(
+ sub {
+ return 0 if (($osd_entry->($id) // {})->{pgs} // -1) != 0;
+
+ my $pgmap = ceph_status($first)->{pgmap} // {};
+ my ($clean) = map { $_->{count} }
+ grep { $_->{state_name} eq 'active+clean' } ($pgmap->{pgs_by_state} // [])->@*;
+ return ($clean // 0) == ($pgmap->{num_pgs} // -1);
+ },
+ timeout => 600,
+ interval => 5,
+ description => "osd.$id to hold no placement groups",
+ );
+ pass("Ceph moved all placement groups off osd.$id");
+ },
+)->step(
+ 'stop OSD',
+ sub {
+ my $id = $osd->{id};
+
+ my $safety = $osd_client->get(
+ "/nodes/$osd_node/ceph/cmd-safety",
+ { service => 'osd', id => $id, action => 'stop' },
+ );
+ ok($safety->{safe}, "Ceph reports that stopping osd.$id is safe")
+ or diag($safety->{status} // 'no status');
+
+ my $upid = $osd_client->post("/nodes/$osd_node/ceph/stop", { service => "osd.$id" });
+ $second->task_ok($upid, "stopped osd.$id") or die "stopping osd.$id failed\n";
+
+ poll_until(
+ sub { (($osd_entry->($id) // {})->{status} // '') eq 'down' },
+ timeout => 60,
+ interval => 2,
+ description => "osd.$id to be down",
+ );
+ pass("osd.$id is down");
+ },
+)->step(
+ 'destroy OSD with cleanup',
+ sub {
+ my $id = $osd->{id};
+
+ my $upid = $osd_client->delete("/nodes/$osd_node/ceph/osd/$id", { cleanup => 1 });
+ $second->task_ok($upid, "destroyed osd.$id") or die "destroying osd.$id failed\n";
+
+ ok(!$osd_entry->($id), "osd.$id is gone from the OSD tree");
+
+ my ($enabled) =
+ $second->run_command("systemctl is-enabled ceph-osd\@$id.service || true");
+ unlike($enabled // '', qr/^enabled/, "service of osd.$id is no longer enabled");
+ ok(
+ eval { $second->run_command("test ! -e /var/lib/ceph/osd/ceph-$id"); 1 },
+ "data directory of osd.$id is removed",
+ );
+
+ # The change in the disk list is not immediately reflected.
+ my $disk;
+ eval {
+ poll_until(
+ sub {
+ $disk = $disk_entry->($osd->{disk});
+ return !$disk->{used} && ($disk->{osdid} // -1) == -1;
+ },
+ timeout => 60,
+ interval => 2,
+ description => "disk '$osd->{disk}' to be unused",
+ );
+ };
+ ok(!$disk->{used}, "disk '$osd->{disk}' is unused")
+ or diag("disk is used by: $disk->{used}");
+ is($disk->{osdid} // -1, -1, "disk '$osd->{disk}' no longer belongs to an OSD");
+ },
+)->step(
+ 'create OSD on the same disk',
+ sub {
+ my $upid = $osd_client->post("/nodes/$osd_node/ceph/osd", { dev => $osd->{disk} });
+ $second->task_ok($upid, "created OSD on '$osd->{disk}' again")
+ or die "creating the OSD failed\n";
+
+ wait_for_ceph(
+ $first,
+ monitors => scalar($pve_instances->@*),
+ osds => $osds,
+ health_ok => 1,
+ );
+ pass("the Ceph cluster reports HEALTH_OK with all $osds OSDs up and in");
+
+ is(
+ scalar(ceph_osds($second)->@*),
+ scalar(ceph_disks($second)->@*),
+ "'$osd_node' has an OSD on every Ceph disk again",
+ );
+ },
+)->step(
+ 'container kept running',
+ sub {
+ is(
+ $client->get("/nodes/$node/lxc/$ctid/status/current", {})->{status},
+ 'running',
+ "container $ctid kept running",
+ );
+ is($first->ct_exec($ctid, 'cat /root/marker'), "$marker\n", 'container data is intact');
+ ok(
+ eval { $first->ct_exec($ctid, "sh -c 'echo written > /root/after-replace'"); 1 },
+ 'container can still write to its volume',
+ );
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index 9a94d11..340ac81 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -425,6 +425,17 @@ TestcaseConfig(
],
tag: "ceph",
),
+ Testcase(
+ name: "ceph_osd_replace",
+ run: "./tests/ceph/test_ceph_osd_replace.pl",
+ timeout: 1800,
+ instances: [
+ "pve1-cluster",
+ "pve2-cluster",
+ "pve3-cluster",
+ ],
+ tag: "ceph",
+ ),
],
rollback_between_tests: Some("pre-test"),
)
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* Re: [PATCH e2e-tests 29/38] tests: ceph: verify replacing an OSD
2026-11-04 19:03 ` [PATCH e2e-tests 29/38] tests: ceph: verify replacing an OSD Michael Köppl
@ 2026-11-06 13:14 ` Kefu Chai
0 siblings, 0 replies; 47+ messages in thread
From: Kefu Chai @ 2026-11-06 13:14 UTC (permalink / raw)
To: m.koeppl; +Cc: pve-devel
Hi Michael,
> [snip]
> + $first->start_guest('lxc', $ctid);
> + $first->ct_exec($ctid, "sh -c 'echo $marker > /root/marker'");
> [snip]
Could we sync the marker before replacing the OSD, then stop/start
the container and read it again after the "kept running" check?
PVE's stop cleanup unmounts the root filesystem and unmaps its RBD
device, so the final read would no longer use the old filesystem's
page cache.
> [snip]
> + my $upid = $osd_client->delete("/nodes/$osd_node/ceph/osd/$id", { cleanup => 1 });
> + $second->task_ok($upid, "destroyed osd.$id") or die "destroying osd.$id failed\n";
> [snip]
Could we also check cmd-safety with action=destroy before this?
The earlier action=stop checks ok-to-stop; action=destroy would
cover the safe-to-destroy check as well.
^ permalink raw reply [flat|nested] 47+ messages in thread
* [PATCH e2e-tests 30/38] tests: ceph: drop the outdated Ceph setup
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (28 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 29/38] tests: ceph: verify replacing an OSD Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 31/38] tests: pbs: verify container backup and restore Michael Köppl
` (8 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
The Ceph plan uses an old plan format, its instance names do not
match the ones the test cases expect, and the runner script refers
to a setup file that does not exist, so these test cases no longer
run. The Ceph tests on the main cluster replace them.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
debian/proxmox-e2e-tests-scripts.install | 1 -
plans/ceph.ron | 87 ------------------------
run_ceph_tests.sh | 10 ---
tests/ceph_testcases.ron | 29 --------
tests/create_ceph_cluster.pl | 69 -------------------
tests/test_create_cluster.pl | 46 -------------
6 files changed, 242 deletions(-)
delete mode 100644 plans/ceph.ron
delete mode 100755 run_ceph_tests.sh
delete mode 100644 tests/ceph_testcases.ron
delete mode 100755 tests/create_ceph_cluster.pl
delete mode 100755 tests/test_create_cluster.pl
diff --git a/debian/proxmox-e2e-tests-scripts.install b/debian/proxmox-e2e-tests-scripts.install
index 63f3521..42d1e98 100644
--- a/debian/proxmox-e2e-tests-scripts.install
+++ b/debian/proxmox-e2e-tests-scripts.install
@@ -1,4 +1,3 @@
run_tests.sh /usr/bin/proxmox-e2e-tests/
run_tests_local.sh /usr/bin/proxmox-e2e-tests/
-run_ceph_tests.sh /usr/bin/proxmox-e2e-tests/
run_ldap.sh /usr/bin/proxmox-e2e-tests/
diff --git a/plans/ceph.ron b/plans/ceph.ron
deleted file mode 100644
index e57d7c6..0000000
--- a/plans/ceph.ron
+++ /dev/null
@@ -1,87 +0,0 @@
-#![enable(implicit_some)]
-
-[
- SetupInstance(
- name: "pve1",
- install: (
- fqdn: "pve1.proxmox.com",
- iso: "local:iso/proxmox-ve_8.2-2-auto-from-http-url.iso",
- cores: 2,
- memory: 3000,
- disk_setup: Xfs(
- size: 10,
- ),
- root_password: Ref("default"),
- additional_disks: [
- Disk(
- serial: "ceph0",
- size: 10,
- ),
- Disk(
- serial: "ceph1",
- size: 10,
- ),
- ]
- ),
- post_setup: [
- "scripts/update-packages.sh pve staging",
- ],
-
- ),
- SetupInstance(
- name: "pve2",
- install: (
- fqdn: "pve2.proxmox.com",
- iso: "local:iso/proxmox-ve_8.2-2-auto-from-http-url.iso",
- cores: 2,
- memory: 3000,
- disk_setup: Ext4(
- size: 10,
- ),
- root_password: Ref("default"),
- additional_disks: [
- Disk(
- serial: "ceph0",
- size: 10,
- ),
- Disk(
- serial: "ceph1",
- size: 10,
- ),
- ]
- ),
- post_setup: [
- "scripts/update-packages.sh pve staging",
- ],
-
- ),
- SetupInstance(
- name: "pve3",
- install: (
- fqdn: "pve3.proxmox.com",
- iso: "local:iso/proxmox-ve_8.2-2-auto-from-http-url.iso",
- cores: 2,
- memory: 3000,
- disk_setup: Zfs(
- size: 10,
- raid: Raid1,
- ),
- root_password: Ref("default"),
- additional_disks: [
- Disk(
- serial: "ceph0",
- size: 10,
- ),
- Disk(
- serial: "ceph1",
- size: 10,
- ),
- ]
- ),
- post_setup: [
- "scripts/update-packages.sh pve staging",
- ],
-
- ),
-]
-
diff --git a/run_ceph_tests.sh b/run_ceph_tests.sh
deleted file mode 100755
index f80d768..0000000
--- a/run_ceph_tests.sh
+++ /dev/null
@@ -1,10 +0,0 @@
-#!/usr/bin/bash
-set -e
-
-inventory="inventory_ceph.ron"
-
-proxmox-test-instance setup setups/ceph.ron --output $inventory
-#proxmox-test-instance rollback $inventory post-setup
-proxmox-test-runner run $inventory tests/ceph_testcases.ron
-proxmox-test-instance cleanup $inventory
-
diff --git a/tests/ceph_testcases.ron b/tests/ceph_testcases.ron
deleted file mode 100644
index b8894bc..0000000
--- a/tests/ceph_testcases.ron
+++ /dev/null
@@ -1,29 +0,0 @@
-#![enable(implicit_some)]
-
-TestcaseConfig(
- testcases: [
- Testcase(
- name: "test_create_cluster",
- run: "./tests/test_create_cluster.pl",
- // There are some sleeps in the test case
- timeout: 60,
- instances: [
- "pve-ceph-1",
- "pve-ceph-2",
- "pve-ceph-3",
- ],
- ),
- Testcase(
- name: "test_create_ceph_cluster",
- run: "./tests/create_ceph_cluster.pl",
- // There are some sleeps in the test case
- timeout: 600,
- instances: [
- "pve-ceph-1",
- "pve-ceph-2",
- "pve-ceph-3",
- ],
- ),
- ],
- rollback_between_tests: Some("pre-test"),
-)
diff --git a/tests/create_ceph_cluster.pl b/tests/create_ceph_cluster.pl
deleted file mode 100755
index 434afe2..0000000
--- a/tests/create_ceph_cluster.pl
+++ /dev/null
@@ -1,69 +0,0 @@
-#!/usr/bin/perl
-use strict;
-use warnings;
-use JSON;
-use File::Slurp;
-
-use lib '.';
-use Proxmox::Test::Helper qw(get_pve_instances);
-
-use Data::Dumper;
-
-use Test::More;
-
-my $instances = get_pve_instances();
-
-for my $instance ($instances->@*) {
- # Make sure that we are in sync
- $instance->run_command("systemctl restart chrony");
- $instance->run_command("yes | pveceph install -version squid -repository no-subscription");
-}
-
-## Init ceph
-for my $instance ($instances->@*) {
- my $nodename = $instance->get_nodename();
- $instance->client()->post("/nodes/$nodename/ceph/init", {});
-}
-
-# Create monitors
-for my $instance ($instances->@*) {
- my $nodename = $instance->get_nodename();
- my $upid = $instance->client()->post("/nodes/$nodename/ceph/mon/$nodename", {});
-
- is($instance->wait_for_task($upid), "OK", "monitor created on '$nodename'");
-
- # TODO: Can we avoid this? Something seems to be a bit racy here if we don't wait a bit
- sleep(3);
-}
-
-# Create OSDs
-for my $instance ($instances->@*) {
- my $nodename = $instance->get_nodename();
-
- for my $disk ($instance->get_additional_disks()->@*) {
- my $serial = $disk->{serial};
- my $path = $instance->get_disk_by_serial($serial);
-
- my $upid = $instance->client()->post("/nodes/$nodename/ceph/osd", {
- dev => $path,
- });
-
- is(
- $instance->wait_for_task($upid),
- "OK",
- "OSD created on '$nodename' for disk with serial '$serial'",
- );
- }
-}
-
-# Create pool
-my $instance = $instances->[0];
-my $nodename = $instance->get_nodename();
-
-my $upid = $instance->client()->post("/nodes/$nodename/ceph/pool", {
- name => "testpool",
-});
-
-is($instance->wait_for_task($upid), "OK", "'testpool' created");
-
-done_testing();
diff --git a/tests/test_create_cluster.pl b/tests/test_create_cluster.pl
deleted file mode 100755
index 64a80e1..0000000
--- a/tests/test_create_cluster.pl
+++ /dev/null
@@ -1,46 +0,0 @@
-#!/usr/bin/perl
-use strict;
-use warnings;
-
-use Test::More;
-
-use lib '.';
-use Proxmox::Test::Helper qw(get_pve_instances);
-use Data::Dumper;
-
-my $pve_instances = get_pve_instances();
-
-my $first = splice(@$pve_instances, 0, 1);
-
-if (scalar($pve_instances->@*) < 1) {
- die "not enough instances";
-}
-
-my $upid;
-$upid = $first->client()->post('/cluster/config', {
- clustername => 'test',
-});
-
-is($first->wait_for_task($upid), "OK", "created the cluster");
-
-for my $instance (@$pve_instances) {
- $upid = $instance->client->post(
- '/cluster/config/join',
- {
- hostname => $first->get_ip(),
- password => $first->get_root_password(),
- fingerprint => $first->get_fingerprint(),
- },
- );
-
- is($instance->wait_for_task($upid), "OK", "joined the cluster");
-}
-
-my $cluster_nodes = $first->client()->get("/cluster/config/nodes");
-
-my @members = sort map $_->{name}, $cluster_nodes->@*;
-my @nodenames = sort map $_->get_nodename(), get_pve_instances()->@*;
-is_deeply(\@members, \@nodenames, "all instances are member of the cluster");
-
-done_testing();
-
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 31/38] tests: pbs: verify container backup and restore
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (29 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 30/38] tests: ceph: drop the outdated Ceph setup Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 32/38] tests: backup: verify VM backup and restore, locally and on PBS Michael Köppl
` (7 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Back up a running container in snapshot mode to a local datastore on
PBS, restore it as a new container and check its config and data.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pbs/test_pbs_ct_backup_restore.pl | 150 ++++++++++++++++++++++++
tests/testcases.ron | 10 ++
2 files changed, 160 insertions(+)
create mode 100755 tests/pbs/test_pbs_ct_backup_restore.pl
diff --git a/tests/pbs/test_pbs_ct_backup_restore.pl b/tests/pbs/test_pbs_ct_backup_restore.pl
new file mode 100755
index 0000000..43fb66f
--- /dev/null
+++ b/tests/pbs/test_pbs_ct_backup_restore.pl
@@ -0,0 +1,150 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<test_pbs_ct_backup_restore> - Back up a running container to PBS and restore it.
+
+=head1 DESCRIPTION
+
+This testcase creates a local datastore on PBS and adds it to PVE as a storage.
+It then creates an unprivileged container, starts it, and writes a marker file.
+It backs up the running container the PBS datastore in snapshot mode, restores
+the backup as a new container, and verifies that the restored container keeps
+its configuration and data.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pbs_instance get_pve_instance);
+use Proxmox::Test::Testcase;
+
+my $pve = get_pve_instance('pve1');
+my $pbs = get_pbs_instance('pbs1');
+
+my $datastore = 'e2e-ct';
+my $storage_id = 'e2e-pbs-ct';
+
+my $node = $pve->get_nodename();
+my $client = $pve->client();
+
+my $test_container = {
+ vmid => $client->get('/cluster/nextid'),
+ node => $node,
+ ostemplate => CONTAINER_TEMPLATE,
+ rootfs => 'local-lvm:2',
+ storage => 'local-lvm',
+ memory => 256,
+ unprivileged => 1,
+ features => 'nesting=1',
+ hostname => 'e2e-ct-pbs-backup',
+};
+
+my $vmid = $test_container->{vmid};
+my $marker = "e2e-backup-$vmid";
+my $backup_volid;
+my $restore_vmid;
+
+Proxmox::Test::Testcase->new('pbs_ct_backup_restore')->with([{}])->step(
+ 'create datastore on PBS',
+ sub {
+ $pbs->create_datastore($datastore);
+ },
+ sub {
+ $pbs->remove_datastore($datastore);
+ },
+)->step(
+ 'add PBS as storage on PVE',
+ sub {
+ $pve->add_pbs_storage($storage_id, $pbs, $datastore);
+ },
+ sub {
+ $client->delete("/storage/$storage_id");
+ },
+)->step(
+ 'create container',
+ sub {
+ my $upid = $client->post("/nodes/$node/lxc", $test_container);
+ $pve->task_ok($upid, "created container $vmid")
+ or die "creating container $vmid failed\n";
+ },
+ sub {
+ $pve->destroy_guest('lxc', $vmid);
+ },
+)->step(
+ 'start container and write marker',
+ sub {
+ $pve->start_guest('lxc', $vmid);
+ $pve->ct_exec($vmid, "sh -c 'echo $marker > /root/marker'");
+ },
+)->step(
+ 'back up running container to PBS',
+ sub {
+ my $upid = $client->post(
+ "/nodes/$node/vzdump",
+ {
+ vmid => $vmid,
+ storage => $storage_id,
+ mode => 'snapshot',
+ },
+ );
+ $pve->task_ok($upid, 'backed up container to PBS') or die "backup failed\n";
+
+ my $content =
+ $client->get("/nodes/$node/storage/$storage_id/content", { content => 'backup' });
+ my ($backup) = grep { $_->{vmid} == $vmid } $content->@*;
+ $backup_volid = $backup->{volid};
+ ok($backup_volid, 'found container backup on PBS storage') or die "backup not found\n";
+ is($backup->{format}, 'pbs-ct', 'backup has the PBS container format');
+
+ is(
+ $client->get("/nodes/$node/lxc/$vmid/status/current", {})->{status},
+ 'running',
+ 'container keeps running during a snapshot mode backup',
+ );
+ },
+)->step(
+ 'restore backup as new container',
+ sub {
+ $restore_vmid = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/lxc",
+ {
+ vmid => $restore_vmid,
+ ostemplate => $backup_volid,
+ restore => 1,
+ storage => 'local-lvm',
+ },
+ );
+ $pve->task_ok($upid, "restored container $restore_vmid from PBS")
+ or die "restoring the backup failed\n";
+ },
+ sub {
+ $pve->destroy_guest('lxc', $restore_vmid) if defined($restore_vmid);
+ },
+)->step(
+ 'verify restored container',
+ sub {
+ my $conf = $client->get("/nodes/$node/lxc/$restore_vmid/config", {});
+ is($conf->{hostname}, $test_container->{hostname}, 'restored config has the hostname');
+ is($conf->{unprivileged}, 1, 'restored container is still unprivileged');
+ like(
+ $conf->{rootfs} // '',
+ qr/^local-lvm:vm-$restore_vmid-disk-\d+\b/,
+ 'restored rootfs is allocated for the new container',
+ );
+
+ $pve->start_guest('lxc', $restore_vmid);
+ is(
+ $pve->ct_exec($restore_vmid, 'cat /root/marker'),
+ "$marker\n",
+ 'restored data is intact',
+ );
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index 340ac81..ace5792 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -273,6 +273,16 @@ TestcaseConfig(
],
tag: "pbs",
),
+ Testcase(
+ name: "pbs_ct_backup_restore",
+ run: "./tests/pbs/test_pbs_ct_backup_restore.pl",
+ timeout: 600,
+ instances: [
+ "pve1",
+ "pbs1",
+ ],
+ tag: "pbs",
+ ),
Testcase(
name: "sdn_create_vlan_zone",
run: "./tests/sdn/test_vlan_zone.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 32/38] tests: backup: verify VM backup and restore, locally and on PBS
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (30 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 31/38] tests: pbs: verify container backup and restore Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 33/38] tests: pbs: verify single file restore from a container backup Michael Köppl
` (6 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Back up a running VM in snapshot mode to the local storage and to a
PBS datastore, and check that the archives are listed with the
expected format and with the notes from the notes template. Restore
the local backup as a new VM, which keeps the MAC address, and the
PBS backup as a new VM with a unique MAC address. Finally, restore
the PBS backup over the original VM, which needs to be forced.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_vm_backup_restore.pl | 269 +++++++++++++++++++++++++++++++++
tests/testcases.ron | 10 ++
2 files changed, 279 insertions(+)
create mode 100755 tests/pve_vm_backup_restore.pl
diff --git a/tests/pve_vm_backup_restore.pl b/tests/pve_vm_backup_restore.pl
new file mode 100755
index 0000000..3aa5add
--- /dev/null
+++ b/tests/pve_vm_backup_restore.pl
@@ -0,0 +1,269 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_vm_backup_restore> - Test backing up and restoring a VM, locally and on PBS.
+
+=head1 DESCRIPTION
+
+This testcase creates a VM with a disk on a temporary directory storage, writes
+random data to the disk and keeps a copy of it. It backs up the running VM in
+snapshot mode to the local storage and to a datastore on PBS (including a notes
+template) and verifies that the archives are listed with the expected format and
+notes while the VM keeps running.
+
+It restores the local backup as a new VM, which must keep the MAC address of the
+original, and the PBS backup as a new VM with a unique MAC address. Both restored
+VMs are expected to have the original disk content.
+
+Finally, it changes the data on the disk of the stopped original VM and restores
+the PBS backup over it. Restoring over an existing VM must be refused unless it
+is forced. The forced restore must bring back the original disk content.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(get_pbs_instance get_pve_instance);
+use Proxmox::Test::Testcase;
+
+my $pve = get_pve_instance('pve1');
+my $pbs = get_pbs_instance('pbs1');
+my $node = $pve->get_nodename();
+my $client = $pve->client();
+
+my $disk_storage = 'e2e-backup-disks';
+my $disk_storage_path = '/var/lib/e2e-backup-disks';
+my $datastore = 'e2e-vm';
+my $pbs_storage = 'e2e-pbs-vm';
+my $name = 'vm-backup';
+
+my $vmid;
+my $reference_dir;
+my $backups = {};
+my $restored_vmids = {};
+
+my $config = sub {
+ my ($id) = @_;
+ return $client->get("/nodes/$node/qemu/$id/config", {});
+};
+
+my $mac_of = sub {
+ my ($conf) = @_;
+ my ($mac) = ($conf->{net0} // '') =~ m/^[^=,]+=([0-9A-Fa-f:]{17})\b/;
+ return $mac;
+};
+
+my $disk_path = sub {
+ my ($id) = @_;
+ my ($volid) = ($config->($id)->{scsi0} // '') =~ m/^([^,]+)/;
+ die "VM $id has no scsi0\n" if !defined($volid);
+ my ($path) = $pve->run_command("pvesm path $volid");
+ chomp($path);
+ return $path;
+};
+
+my $has_reference_content = sub {
+ my ($id) = @_;
+ my $path = $disk_path->($id);
+ return eval {
+ $pve->run_command("qemu-img compare -f raw $reference_dir/disk.raw $path");
+ 1;
+ };
+};
+
+my $restore = sub {
+ my ($id, $archive, %opts) = @_;
+ return $client->post(
+ "/nodes/$node/qemu",
+ { vmid => $id, archive => $archive, storage => $disk_storage, %opts },
+ );
+};
+
+Proxmox::Test::Testcase->new('pve_vm_backup_restore')->with([{}])->step(
+ 'create directory storage for VM disks',
+ sub {
+ $pve->run_command("mkdir -p $disk_storage_path");
+ $client->post(
+ '/storage',
+ {
+ storage => $disk_storage,
+ type => 'dir',
+ path => $disk_storage_path,
+ content => 'images',
+ },
+ );
+ },
+ sub {
+ eval { $client->delete("/storage/$disk_storage") };
+ $pve->run_command("rm -rf $disk_storage_path");
+ },
+)->step(
+ 'create datastore on PBS',
+ sub {
+ $pbs->create_datastore($datastore);
+ },
+ sub {
+ $pbs->remove_datastore($datastore);
+ },
+)->step(
+ 'add PBS as storage on PVE',
+ sub {
+ $pve->add_pbs_storage($pbs_storage, $pbs, $datastore);
+ },
+ sub {
+ $client->delete("/storage/$pbs_storage");
+ },
+)->step(
+ 'create VM',
+ sub {
+ $vmid = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/qemu",
+ {
+ vmid => $vmid,
+ name => $name,
+ memory => 256,
+ scsihw => 'virtio-scsi-pci',
+ scsi0 => "$disk_storage:1,format=raw",
+ net0 => 'virtio,bridge=vmbr0',
+ },
+ );
+ $pve->task_ok($upid, "created VM $vmid") or die "creating VM $vmid failed\n";
+ },
+ sub {
+ $pve->destroy_guest('qemu', $vmid) if defined($vmid);
+ },
+)->step(
+ 'write data to the disk and keep a copy',
+ sub {
+ my $path = $disk_path->($vmid);
+ $pve->run_command("dd if=/dev/urandom of=$path bs=1M count=8 seek=16 conv=notrunc");
+
+ ($reference_dir) = $pve->run_command('mktemp -d');
+ chomp($reference_dir);
+ $pve->run_command("cp --sparse=always $path $reference_dir/disk.raw");
+ },
+ sub {
+ $pve->run_command("rm -rf $reference_dir") if defined($reference_dir);
+ },
+)->step(
+ 'start VM',
+ sub {
+ $pve->start_guest('qemu', $vmid);
+ },
+)->step(
+ 'back up running VM',
+ sub {
+ for my $storage ('local', $pbs_storage) {
+ my $upid = $client->post(
+ "/nodes/$node/vzdump",
+ {
+ vmid => $vmid,
+ storage => $storage,
+ mode => 'snapshot',
+ 'notes-template' => 'e2e {{guestname}}',
+ },
+ );
+ $pve->task_ok($upid, "backed up VM $vmid to $storage")
+ or die "backing up VM $vmid to $storage failed\n";
+
+ my $content =
+ $client->get("/nodes/$node/storage/$storage/content", { content => 'backup' });
+ my ($backup) = grep { ($_->{vmid} // 0) == $vmid } $content->@*;
+ ok($backup, "backup is listed on $storage") or die "backup not found\n";
+ $backups->{$storage} = $backup->{volid};
+
+ like(
+ $backup->{format},
+ $storage eq 'local' ? qr/^vma\b/ : qr/^pbs-vm$/,
+ "backup on $storage has the expected format",
+ );
+ is($backup->{notes}, "e2e $name", "backup on $storage has notes from the template");
+ }
+
+ is(
+ $client->get("/nodes/$node/qemu/$vmid/status/current", {})->{status},
+ 'running',
+ 'VM keeps running during the backups',
+ );
+ },
+ sub {
+ $pve->destroy_volume('local', $backups->{local}) if $backups->{local};
+ },
+)->step(
+ 'restore local backup as new VM',
+ sub {
+ my $id = $restored_vmids->{local} = $client->get('/cluster/nextid');
+ my $upid = $restore->($id, $backups->{local});
+ $pve->task_ok($upid, "restored local backup as VM $id")
+ or die "restoring the local backup failed\n";
+
+ my $conf = $config->($id);
+ is($conf->{name}, $name, 'restored VM has the name of the original');
+ is(
+ $mac_of->($conf),
+ $mac_of->($config->($vmid)),
+ 'restored VM keeps the MAC address of the original',
+ );
+ like(
+ $conf->{scsi0} // '',
+ qr/^\Q$disk_storage\E:/,
+ "disk is restored to $disk_storage",
+ );
+ ok($has_reference_content->($id), 'restored disk has the original content');
+ },
+ sub {
+ $pve->destroy_guest('qemu', $restored_vmids->{local})
+ if defined($restored_vmids->{local});
+ },
+)->step(
+ 'restore PBS backup as new VM with unique MAC address',
+ sub {
+ my $id = $restored_vmids->{pbs} = $client->get('/cluster/nextid');
+ my $upid = $restore->($id, $backups->{$pbs_storage}, unique => 1);
+ $pve->task_ok($upid, "restored PBS backup as VM $id")
+ or die "restoring the PBS backup failed\n";
+
+ my $conf = $config->($id);
+ is($conf->{name}, $name, 'restored VM has the name of the original');
+ my $mac = $mac_of->($conf);
+ ok(defined($mac), 'restored VM has a MAC address');
+ isnt($mac, $mac_of->($config->($vmid)), 'restored VM has a new MAC address');
+ ok($has_reference_content->($id), 'restored disk has the original content');
+ },
+ sub {
+ $pve->destroy_guest('qemu', $restored_vmids->{pbs})
+ if defined($restored_vmids->{pbs});
+ },
+)->step(
+ 'restore PBS backup over the changed original VM',
+ sub {
+ $pve->stop_guest('qemu', $vmid);
+
+ my $path = $disk_path->($vmid);
+ $pve->run_command("dd if=/dev/urandom of=$path bs=1M count=8 seek=16 conv=notrunc");
+ ok(!$has_reference_content->($vmid), 'disk content differs from the backup');
+
+ my $mac = $mac_of->($config->($vmid));
+
+ eval { $restore->($vmid, $backups->{$pbs_storage}) };
+ ok($@, 'restoring over the existing VM without force fails');
+ ok(!$has_reference_content->($vmid), 'disk content is unchanged');
+
+ my $upid = $restore->($vmid, $backups->{$pbs_storage}, force => 1);
+ $pve->task_ok($upid, "restored PBS backup over VM $vmid")
+ or die "restoring over VM $vmid failed\n";
+
+ my $conf = $config->($vmid);
+ is($conf->{name}, $name, 'VM keeps its name');
+ is($mac_of->($conf), $mac, 'VM keeps its MAC address');
+ ok($has_reference_content->($vmid), 'disk has the content from the time of the backup');
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index ace5792..b107835 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -283,6 +283,16 @@ TestcaseConfig(
],
tag: "pbs",
),
+ Testcase(
+ name: "pve_vm_backup_restore",
+ run: "./tests/pve_vm_backup_restore.pl",
+ timeout: 900,
+ instances: [
+ "pve1",
+ "pbs1",
+ ],
+ tag: "pbs",
+ ),
Testcase(
name: "sdn_create_vlan_zone",
run: "./tests/sdn/test_vlan_zone.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 33/38] tests: pbs: verify single file restore from a container backup
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (31 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 32/38] tests: backup: verify VM backup and restore, locally and on PBS Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 34/38] tests: storage: verify moving guest volumes between storages Michael Köppl
` (5 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Back up a container with a marker file to PBS, browse the backup
through the file restore API down to the file and download it.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
Notes:
The test accepts curl's exit code 56 for the download, which curl
with OpenSSL 3 returns because pveproxy ends the streamed response
without a TLS close_notify. This workaround is only meant to stay
until http-server patch [0] is applied, or the issue is resolved in
some other way.
[0] https://lore.proxmox.com/pve-devel/20261103155223.1960754-1-m.koeppl@proxmox.com/
tests/pbs/test_pbs_file_restore.pl | 151 +++++++++++++++++++++++++++++
tests/testcases.ron | 10 ++
2 files changed, 161 insertions(+)
create mode 100755 tests/pbs/test_pbs_file_restore.pl
diff --git a/tests/pbs/test_pbs_file_restore.pl b/tests/pbs/test_pbs_file_restore.pl
new file mode 100755
index 0000000..1d9a20e
--- /dev/null
+++ b/tests/pbs/test_pbs_file_restore.pl
@@ -0,0 +1,151 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<test_pbs_file_restore> - Restore single files from a container backup on PBS.
+
+=head1 DESCRIPTION
+
+This testcase creates a local datastore on PBS and adds it to PVE as a storage.
+It creates a container, writes a file to it and backs it up to the PBS datastore.
+It then browses the backup through the single file restore API down to the file,
+which must be listed with its size. It then downloads the file, which must have
+the content written before the backup.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use URI::Escape qw(uri_escape);
+
+use lib '.';
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pbs_instance get_pve_instance);
+use Proxmox::Test::Testcase;
+
+my $pve = get_pve_instance('pve1');
+my $pbs = get_pbs_instance('pbs1');
+my $node = $pve->get_nodename();
+my $client = $pve->client();
+
+my $datastore = 'e2e-file-restore';
+my $storage_id = 'e2e-pbs-file-restore';
+
+my $ctid;
+my $backup_volid;
+my $marker_file = 'e2e-marker';
+my $marker = "e2e-file-restore-$$";
+my $marker_entry;
+
+# Entries below $filepath in the backup
+my $list = sub {
+ my ($filepath) = @_;
+ return $client->get(
+ "/nodes/$node/storage/$storage_id/file-restore/list",
+ { volume => $backup_volid, filepath => $filepath },
+ );
+};
+
+my $find_entry = sub {
+ my ($entries, $match) = @_;
+ my ($entry) = grep { $_->{text} =~ $match } $entries->@*;
+ return $entry;
+};
+
+Proxmox::Test::Testcase->new('pbs_file_restore')->with([{}])->step(
+ 'create datastore on PBS',
+ sub {
+ $pbs->create_datastore($datastore);
+ },
+ sub {
+ $pbs->remove_datastore($datastore);
+ },
+)->step(
+ 'add PBS as storage on PVE',
+ sub {
+ $pve->add_pbs_storage($storage_id, $pbs, $datastore);
+ },
+ sub {
+ $client->delete("/storage/$storage_id");
+ },
+)->step(
+ 'create container and write file',
+ sub {
+ $ctid = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/lxc",
+ {
+ vmid => $ctid,
+ ostemplate => CONTAINER_TEMPLATE,
+ rootfs => 'local-lvm:2',
+ memory => 256,
+ unprivileged => 1,
+ features => 'nesting=1',
+ hostname => 'ct-file-restore',
+ },
+ );
+ $pve->task_ok($upid, "created container $ctid")
+ or die "creating container $ctid failed\n";
+
+ $pve->start_guest('lxc', $ctid);
+ $pve->ct_exec($ctid, "sh -c 'echo $marker > /root/$marker_file'");
+ },
+ sub {
+ $pve->destroy_guest('lxc', $ctid) if defined($ctid);
+ },
+)->step(
+ 'back up container to PBS',
+ sub {
+ my $upid = $client->post(
+ "/nodes/$node/vzdump",
+ { vmid => $ctid, storage => $storage_id, mode => 'snapshot' },
+ );
+ $pve->task_ok($upid, 'backed up container to PBS') or die "backup failed\n";
+
+ my $content =
+ $client->get("/nodes/$node/storage/$storage_id/content", { content => 'backup' });
+ my ($backup) = grep { ($_->{vmid} // 0) == $ctid } $content->@*;
+ $backup_volid = $backup->{volid};
+ ok($backup_volid, 'found container backup on PBS storage') or die "backup not found\n";
+ },
+)->step(
+ 'browse backup down to the file',
+ sub {
+ my $archive = $find_entry->($list->('/'), qr/^root\.pxar/);
+ ok($archive, 'backup contains the root file system archive')
+ or die "archive not found\n";
+
+ my $root_dir = $find_entry->($list->($archive->{filepath}), qr/^root$/);
+ ok($root_dir, "archive contains the directory '/root'") or die "'/root' not found\n";
+
+ $marker_entry = $find_entry->($list->($root_dir->{filepath}), qr/^\Q$marker_file\E$/);
+ ok($marker_entry, 'directory contains the file') or die "file not found\n";
+ is($marker_entry->{size}, length("$marker\n"), 'file is listed with its size');
+ },
+)->step(
+ 'download file',
+ sub {
+ # This doesn't work with the API client (expects JSON), to use ticket from the node for
+ # manual request.
+ my $ticket = $client->post(
+ '/access/ticket',
+ { username => 'root@pam', password => $pve->get_root_password() },
+ )->{ticket};
+
+ my $query = join(
+ '&',
+ 'volume=' . uri_escape($backup_volid),
+ 'filepath=' . uri_escape($marker_entry->{filepath}),
+ );
+ # FIXME: remove workaround once api server sends TLS close notify
+ my ($content) =
+ $pve->run_command("curl -sSfk -H 'Cookie: PVEAuthCookie=$ticket'"
+ . " 'https://localhost:8006/api2/json/nodes/$node/storage/$storage_id"
+ . "/file-restore/download?$query'; rc=\$?; [ \$rc -eq 0 ] || [ \$rc -eq 56 ]");
+ is($content, "$marker\n", 'downloaded file has the content from the backup');
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index b107835..51ceb81 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -293,6 +293,16 @@ TestcaseConfig(
],
tag: "pbs",
),
+ Testcase(
+ name: "pbs_file_restore",
+ run: "./tests/pbs/test_pbs_file_restore.pl",
+ timeout: 600,
+ instances: [
+ "pve1",
+ "pbs1",
+ ],
+ tag: "pbs",
+ ),
Testcase(
name: "sdn_create_vlan_zone",
run: "./tests/sdn/test_vlan_zone.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 34/38] tests: storage: verify moving guest volumes between storages
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (32 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 33/38] tests: pbs: verify single file restore from a container backup Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 35/38] tests: migration: verify container migration between nodes Michael Köppl
` (4 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Move the disk of a running VM from a directory storage to local-lvm
and delete the source, then move it back as qcow2 while keeping the
source, which must be kept as an unused volume. The disk content is
compared against a copy taken before the moves.
Also move the root file system of a stopped container, which must
start from the moved volume with its data intact, and check that
moving a volume of a running container is refused.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_move_disks.pl | 262 ++++++++++++++++++++++++++++++++++++++++
tests/testcases.ron | 9 ++
2 files changed, 271 insertions(+)
create mode 100755 tests/pve_move_disks.pl
diff --git a/tests/pve_move_disks.pl b/tests/pve_move_disks.pl
new file mode 100755
index 0000000..59dd780
--- /dev/null
+++ b/tests/pve_move_disks.pl
@@ -0,0 +1,262 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_move_disks> - Test moving guest volumes between storages.
+
+=head1 DESCRIPTION
+
+This testcase creates a VM with a disk on a temporary directory storage, writes
+random data to the disk and keeps a copy of it. While the VM is running, it moves
+the disk to local-lvm and deletes the source, then moves it back to the directory
+storage in qcow2 format and keeps the source, which must then be kept as an
+unused volume. Afterwards, the moved disk must still have the original content.
+
+It also creates a container on local-lvm and writes a file to it. Moving the
+volume of the running container must be refused. Once the container is stopped,
+its root file system is moved to the directory storage, and the container must
+start from the moved volume with the file intact.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pve_instance);
+use Proxmox::Test::Testcase;
+
+my $pve = get_pve_instance('pve1');
+my $node = $pve->get_nodename();
+my $client = $pve->client();
+
+my $dir_storage = 'e2e-move';
+my $dir_storage_path = '/var/lib/e2e-move-store';
+
+my $vmid;
+my $ctid;
+my $reference_dir;
+my $marker = 'e2e-move-volume';
+
+my $config = sub {
+ my ($type, $id) = @_;
+ return $client->get("/nodes/$node/$type/$id/config", {});
+};
+
+my $volid_of = sub {
+ my ($value) = @_;
+ my ($volid) = ($value // '') =~ m/^([^,]+)/;
+ return $volid;
+};
+
+my $volume = sub {
+ my ($storage, $id, $volid) = @_;
+ my $content = $client->get("/nodes/$node/storage/$storage/content", { vmid => $id });
+ my ($entry) = grep { $_->{volid} eq $volid } $content->@*;
+ return $entry;
+};
+
+my $move = sub {
+ my ($path, $params, $name) = @_;
+ my $upid = $client->post($path, $params);
+ $pve->task_ok($upid, $name) or die "$name failed\n";
+};
+
+Proxmox::Test::Testcase->new('pve_move_disks')->with([{}])->step(
+ 'create directory storage',
+ sub {
+ $pve->run_command("mkdir -p $dir_storage_path");
+ $client->post(
+ '/storage',
+ {
+ storage => $dir_storage,
+ type => 'dir',
+ path => $dir_storage_path,
+ content => 'images,rootdir',
+ },
+ );
+ },
+ sub {
+ eval { $client->delete("/storage/$dir_storage") };
+ $pve->run_command("rm -rf $dir_storage_path");
+ },
+)->step(
+ 'create VM',
+ sub {
+ $vmid = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/qemu",
+ {
+ vmid => $vmid,
+ name => 'vm-move-disk',
+ memory => 256,
+ scsihw => 'virtio-scsi-pci',
+ scsi0 => "$dir_storage:1,format=raw",
+ },
+ );
+ $pve->task_ok($upid, "created VM $vmid") or die "creating VM $vmid failed\n";
+ },
+ sub {
+ $pve->destroy_guest('qemu', $vmid) if defined($vmid);
+ },
+)->step(
+ 'write data to the disk and keep a copy',
+ sub {
+ my $volid = $volid_of->($config->('qemu', $vmid)->{scsi0});
+ my ($path) = $pve->run_command("pvesm path $volid");
+ chomp($path);
+
+ $pve->run_command("dd if=/dev/urandom of=$path bs=1M count=8 seek=16 conv=notrunc");
+
+ ($reference_dir) = $pve->run_command('mktemp -d');
+ chomp($reference_dir);
+ $pve->run_command("cp --sparse=always $path $reference_dir/disk.raw");
+ },
+ sub {
+ $pve->run_command("rm -rf $reference_dir") if defined($reference_dir);
+ },
+)->step(
+ 'start VM',
+ sub {
+ $pve->start_guest('qemu', $vmid);
+ },
+)->step(
+ 'move disk of the running VM and delete the source',
+ sub {
+ my $source = $volid_of->($config->('qemu', $vmid)->{scsi0});
+
+ $move->(
+ "/nodes/$node/qemu/$vmid/move_disk",
+ { disk => 'scsi0', storage => 'local-lvm', delete => 1 },
+ 'moved scsi0 to local-lvm',
+ );
+
+ my $conf = $config->('qemu', $vmid);
+ like($conf->{scsi0} // '', qr/^local-lvm:/, 'scsi0 is on local-lvm');
+ ok(!$volume->($dir_storage, $vmid, $source), 'source volume is deleted');
+ ok(!(grep { m/^unused\d+$/ } keys $conf->%*), 'no unused volume is left');
+ is(
+ $client->get("/nodes/$node/qemu/$vmid/status/current", {})->{status},
+ 'running',
+ 'VM keeps running during the move',
+ );
+ },
+)->step(
+ 'move disk of the running VM to qcow2 and keep the source',
+ sub {
+ my $source = $volid_of->($config->('qemu', $vmid)->{scsi0});
+
+ $move->(
+ "/nodes/$node/qemu/$vmid/move_disk",
+ { disk => 'scsi0', storage => $dir_storage, format => 'qcow2' },
+ "moved scsi0 to $dir_storage as qcow2",
+ );
+
+ my $conf = $config->('qemu', $vmid);
+ my $target = $volid_of->($conf->{scsi0});
+ like($target // '', qr/^\Q$dir_storage\E:/, "scsi0 is on $dir_storage");
+ is(
+ ($volume->($dir_storage, $vmid, $target) // {})->{format},
+ 'qcow2',
+ 'moved volume has the qcow2 format',
+ );
+
+ my ($unused) = grep { m/^unused\d+$/ && $conf->{$_} eq $source } keys $conf->%*;
+ ok(defined($unused), 'source volume is kept as an unused volume');
+ ok($volume->('local-lvm', $vmid, $source), 'source volume still exists');
+ },
+)->step(
+ 'stop VM',
+ sub {
+ $pve->stop_guest('qemu', $vmid);
+ },
+)->step(
+ 'verify disk content',
+ sub {
+ my $volid = $volid_of->($config->('qemu', $vmid)->{scsi0});
+ my ($path) = $pve->run_command("pvesm path $volid");
+ chomp($path);
+
+ ok(
+ eval {
+ $pve->run_command(
+ "qemu-img compare -f raw -F qcow2 $reference_dir/disk.raw $path");
+ 1;
+ },
+ 'moved disk has the original content',
+ ) or diag($@);
+ },
+)->step(
+ 'create container',
+ sub {
+ $ctid = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node/lxc",
+ {
+ vmid => $ctid,
+ ostemplate => CONTAINER_TEMPLATE,
+ rootfs => 'local-lvm:2',
+ memory => 256,
+ unprivileged => 1,
+ features => 'nesting=1',
+ hostname => 'ct-move-volume',
+ },
+ );
+ $pve->task_ok($upid, "created container $ctid")
+ or die "creating container $ctid failed\n";
+ },
+ sub {
+ $pve->destroy_guest('lxc', $ctid) if defined($ctid);
+ },
+)->step(
+ 'write file in the running container',
+ sub {
+ $pve->start_guest('lxc', $ctid);
+ $pve->ct_exec($ctid, "sh -c 'echo $marker > /root/marker'");
+
+ eval {
+ $client->post(
+ "/nodes/$node/lxc/$ctid/move_volume",
+ { volume => 'rootfs', storage => $dir_storage },
+ );
+ };
+ ok($@, 'moving the root file system of the running container fails');
+ like(
+ $config->('lxc', $ctid)->{rootfs} // '',
+ qr/^local-lvm:/,
+ 'root file system is still on local-lvm',
+ );
+
+ $pve->stop_guest('lxc', $ctid);
+ },
+)->step(
+ 'move root file system of the stopped container',
+ sub {
+ my $source = $volid_of->($config->('lxc', $ctid)->{rootfs});
+
+ $move->(
+ "/nodes/$node/lxc/$ctid/move_volume",
+ { volume => 'rootfs', storage => $dir_storage, delete => 1 },
+ "moved rootfs to $dir_storage",
+ );
+
+ like(
+ $config->('lxc', $ctid)->{rootfs} // '',
+ qr/^\Q$dir_storage\E:/,
+ "rootfs is on $dir_storage",
+ );
+ ok(!$volume->('local-lvm', $ctid, $source), 'source volume is deleted');
+
+ $pve->start_guest('lxc', $ctid);
+ is(
+ $pve->ct_exec($ctid, 'cat /root/marker'),
+ "$marker\n",
+ 'file is intact after the move',
+ );
+ $pve->stop_guest('lxc', $ctid);
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index 51ceb81..f664c62 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -214,6 +214,15 @@ TestcaseConfig(
],
tag: "extended",
),
+ Testcase(
+ name: "pve_move_disks",
+ run: "./tests/pve_move_disks.pl",
+ timeout: 600,
+ instances: [
+ "pve1",
+ ],
+ tag: "extended",
+ ),
Testcase(
name: "pve_ha_resource_config",
run: "./tests/pve_ha_resource_config.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 35/38] tests: migration: verify container migration between nodes
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (33 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 34/38] tests: storage: verify moving guest volumes between storages Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 36/38] tests: migration: verify VM migration with local disks Michael Köppl
` (3 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Migrate a stopped container with its root file system on local-lvm
to another node, then migrate it back while it runs, in restart
mode. Check that the container and its volume end up on the target
node, that nothing is left behind on the source node and that its
data is intact. Migrating a running container without restart mode
must be refused.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_ct_migration.pl | 159 ++++++++++++++++++++++++++++++++++++++
tests/testcases.ron | 11 +++
2 files changed, 170 insertions(+)
create mode 100755 tests/pve_ct_migration.pl
diff --git a/tests/pve_ct_migration.pl b/tests/pve_ct_migration.pl
new file mode 100755
index 0000000..b29faf7
--- /dev/null
+++ b/tests/pve_ct_migration.pl
@@ -0,0 +1,159 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_ct_migration> - Test migrating a container between cluster nodes.
+
+=head1 DESCRIPTION
+
+This testcase creates a container with its root file system on the local-lvm
+storage of the first cluster node and writes a file to it. Migrating the running
+container must be refused unless restart mode is used.
+
+It migrates the stopped container to the second node, which must move both the
+container and its volume there. It then starts the container on the second node
+and migrates it back in restart mode, after which the container must run on the
+first node again. The file must be intact after each migration.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pve_instance);
+use Proxmox::Test::Testcase;
+use Proxmox::Test::Util qw(poll_until);
+
+my $pve_a = get_pve_instance('pve1-cluster');
+my $pve_b = get_pve_instance('pve2-cluster');
+my $node_a = $pve_a->get_nodename();
+my $node_b = $pve_b->get_nodename();
+my $client = $pve_a->client();
+
+my $ctid;
+my $marker = 'e2e-ct-migration';
+
+# The node the container lives on according to the cluster resources.
+my $resource_node = sub {
+ my $resources = $client->get('/cluster/resources', { type => 'vm' });
+ my ($entry) = grep { ($_->{vmid} // 0) == $ctid } $resources->@*;
+ return $entry ? $entry->{node} : undef;
+};
+
+my $wait_for_node = sub {
+ my ($node) = @_;
+ poll_until(
+ sub { ($resource_node->() // '') eq $node },
+ timeout => 60,
+ description => "container $ctid to show up on node $node",
+ );
+};
+
+my $volumes_on = sub {
+ my ($node) = @_;
+ my $content = $client->get("/nodes/$node/storage/local-lvm/content", { vmid => $ctid });
+ return scalar($content->@*);
+};
+
+# Verifies that both the container and its volume are on the expected node.
+my $verify_location = sub {
+ my ($instance, $other) = @_;
+ my $node = $instance->get_nodename();
+ my $other_node = $other->get_nodename();
+
+ $wait_for_node->($node);
+ is($resource_node->(), $node, "container $ctid is on node $node");
+ ok(
+ (grep { $_->{vmid} == $ctid } $client->get("/nodes/$node/lxc", {})->@*),
+ "node $node lists container $ctid",
+ );
+ ok($volumes_on->($node), "volume of container $ctid is on node $node");
+ ok(!$volumes_on->($other_node), "no volume of container $ctid is left on node $other_node");
+};
+
+Proxmox::Test::Testcase->new('pve_ct_migration')->with([{}])->step(
+ 'create container and write file',
+ sub {
+ $ctid = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node_a/lxc",
+ {
+ vmid => $ctid,
+ ostemplate => CONTAINER_TEMPLATE,
+ rootfs => 'local-lvm:2',
+ memory => 256,
+ unprivileged => 1,
+ features => 'nesting=1',
+ hostname => 'ct-migration',
+ },
+ );
+ $pve_a->task_ok($upid, "created container $ctid on node $node_a")
+ or die "creating container $ctid failed\n";
+
+ $pve_a->start_guest('lxc', $ctid);
+ $pve_a->ct_exec($ctid, "sh -c 'echo $marker > /root/marker'");
+ },
+ sub {
+ my $node = $resource_node->();
+ $pve_a->destroy_guest_on($node, 'lxc', $ctid) if defined($node);
+ },
+)->step(
+ 'migrating the running container needs restart mode',
+ sub {
+ eval { $client->post("/nodes/$node_a/lxc/$ctid/migrate", { target => $node_b }) };
+ # The next step sends the same request for the stopped container, which only succeeds if
+ # this one failed because hte container is running.
+ ok($@, 'migrating the running container without restart mode fails');
+ is($resource_node->(), $node_a, "container $ctid stays on node $node_a");
+ },
+)->step(
+ 'migrate stopped container',
+ sub {
+ $pve_a->stop_guest('lxc', $ctid);
+
+ my $upid = $client->post("/nodes/$node_a/lxc/$ctid/migrate", { target => $node_b });
+ $pve_a->task_ok($upid, "migrated container $ctid to node $node_b")
+ or die "migrating container $ctid failed\n";
+
+ $verify_location->($pve_b, $pve_a);
+
+ $pve_b->start_guest('lxc', $ctid);
+ is(
+ $pve_b->ct_exec($ctid, 'cat /root/marker'),
+ "$marker\n",
+ 'file is intact after the migration',
+ );
+ },
+)->step(
+ 'migrate running container in restart mode',
+ sub {
+ my $upid = $pve_b->client()->post(
+ "/nodes/$node_b/lxc/$ctid/migrate",
+ { target => $node_a, restart => 1, timeout => 60 },
+ );
+ $pve_b->task_ok($upid, "migrated container $ctid back to node $node_a in restart mode")
+ or die "migrating container $ctid back failed\n";
+
+ $verify_location->($pve_a, $pve_b);
+
+ poll_until(
+ sub {
+ $client->get("/nodes/$node_a/lxc/$ctid/status/current", {})->{status} eq
+ 'running';
+ },
+ timeout => 60,
+ description => "container $ctid to run on node $node_a",
+ );
+ pass("container $ctid runs again after the restart migration");
+ is(
+ $pve_a->ct_exec($ctid, 'cat /root/marker'),
+ "$marker\n",
+ 'file is intact after the migration',
+ );
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index f664c62..9903d79 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -140,6 +140,17 @@ TestcaseConfig(
],
tag: "extended",
),
+ Testcase(
+ name: "pve_ct_migration",
+ run: "./tests/pve_ct_migration.pl",
+ timeout: 600,
+ instances: [
+ "pve1-cluster",
+ "pve2-cluster",
+ "pve3-cluster",
+ ],
+ tag: "extended",
+ ),
Testcase(
name: "pve_replication",
run: "./tests/pve_replication.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 36/38] tests: migration: verify VM migration with local disks
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (34 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 35/38] tests: migration: verify container migration between nodes Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 37/38] tests: ha: verify moving guests off a node for maintenance Michael Köppl
` (2 subsequent siblings)
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Live migrate a VM with a disk on node-local ZFS storage to another node
onto local-lvm and back onto ZFS, then migrate the stopped VM to the
other node again. Check that the disk is moved to the target storage
each time, that no volume is left behind and that its content stays the
same. A live migration without allowing local disks must fail and leave
the VM running on its node.
The offline migration keeps the disk on ZFS if the disk cannot be
converted in a format both storages support. The disk content is
compared by checksum on the node.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_vm_migration_local_disks.pl | 213 ++++++++++++++++++++++++++
tests/testcases.ron | 11 ++
2 files changed, 224 insertions(+)
create mode 100755 tests/pve_vm_migration_local_disks.pl
diff --git a/tests/pve_vm_migration_local_disks.pl b/tests/pve_vm_migration_local_disks.pl
new file mode 100755
index 0000000..55d2e14
--- /dev/null
+++ b/tests/pve_vm_migration_local_disks.pl
@@ -0,0 +1,213 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_vm_migration_local_disks> - Test migrating a VM with local disks between cluster nodes.
+
+=head1 DESCRIPTION
+
+This testcase creates a VM with a disk on the node-local ZFS storage of the first
+cluster node, writes random data to the disk and records its checksum. The
+migration preconditions must report the disk as local.
+
+It starts the VM, and a live migration without allowing the migration of local
+disks must fail, leaving the VM running on the first node. A live migration with
+local disks to the second node, onto its local-lvm storage, must move the VM and
+its disk, and so must a live migration back onto the ZFS storage. After stopping
+the VM, an offline migration to the second node must move it to the storage with
+the same storage ID. The disk must have the original content after each
+migration, and no volume may be left behind on the source node.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(get_pve_instance);
+use Proxmox::Test::Testcase;
+use Proxmox::Test::Util qw(poll_until);
+
+my $pve_a = get_pve_instance('pve1-cluster');
+my $pve_b = get_pve_instance('pve2-cluster');
+my $node_a = $pve_a->get_nodename();
+my $node_b = $pve_b->get_nodename();
+my $client = $pve_a->client();
+
+my $zfs_storage = 'zfstest';
+
+my $vmid;
+my $checksum;
+
+# The node the VM lives on according to the cluster resources. undef if the node could not be
+# determined.
+my $resource_node = sub {
+ my $resources = $client->get('/cluster/resources', { type => 'vm' });
+ my ($entry) = grep { ($_->{vmid} // 0) == $vmid } $resources->@*;
+ return $entry ? $entry->{node} : undef;
+};
+
+my $disk_volid = sub {
+ my ($node) = @_;
+ my $conf = $client->get("/nodes/$node/qemu/$vmid/config", {});
+ my ($volid) = ($conf->{scsi0} // '') =~ m/^([^,]+)/;
+ return $volid;
+};
+
+my $volumes_on = sub {
+ my ($node, $storage) = @_;
+ my $content = $client->get("/nodes/$node/storage/$storage/content", { vmid => $vmid });
+ return { map { $_->{volid} => 1 } $content->@* };
+};
+
+my $disk_checksum = sub {
+ my ($instance) = @_;
+ my $volid = $disk_volid->($instance->get_nodename());
+ my ($path) = $instance->run_command("pvesm path $volid");
+ chomp($path);
+ my ($out) = $instance->run_command("sha256sum $path");
+ my ($sum) = split(/\s+/, $out // '');
+ return $sum;
+};
+
+my $migrate = sub {
+ my ($instance, $params) = @_;
+ my $node = $instance->get_nodename();
+ return $instance->client()->post("/nodes/$node/qemu/$vmid/migrate", $params);
+};
+
+my $verify_location = sub {
+ my ($instance, $storage, $source, $source_storage) = @_;
+ my $node = $instance->get_nodename();
+
+ poll_until(
+ sub { ($resource_node->() // '') eq $node },
+ timeout => 60,
+ description => "VM $vmid to show up on node $node",
+ );
+ pass("VM $vmid is on node $node");
+
+ my $volid = $disk_volid->($node);
+ like($volid // '', qr/^\Q$storage\E:/, "disk is on $storage");
+ ok($volid && $volumes_on->($node, $storage)->{$volid}, "disk volume exists on node $node");
+ is_deeply(
+ $volumes_on->($source->get_nodename(), $source_storage),
+ {},
+ 'no volume is left on the source node',
+ );
+ is($disk_checksum->($instance), $checksum, 'disk has the original content');
+};
+
+Proxmox::Test::Testcase->new('pve_vm_migration_local_disks')->with([{}])->step(
+ 'create VM with a local disk',
+ sub {
+ $vmid = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node_a/qemu",
+ {
+ vmid => $vmid,
+ name => 'vm-migration-local-disks',
+ memory => 256,
+ scsihw => 'virtio-scsi-pci',
+ scsi0 => "$zfs_storage:1",
+ },
+ );
+ $pve_a->task_ok($upid, "created VM $vmid on node $node_a")
+ or die "creating VM $vmid failed\n";
+ },
+ sub {
+ my $node = $resource_node->();
+ $pve_a->destroy_guest_on($node, 'qemu', $vmid) if defined($node);
+ },
+)->step(
+ 'write data to the disk',
+ sub {
+ my ($path) = $pve_a->run_command('pvesm path ' . $disk_volid->($node_a));
+ chomp($path);
+ $pve_a->run_command("dd if=/dev/urandom of=$path bs=1M count=8 seek=16 conv=notrunc");
+
+ $checksum = $disk_checksum->($pve_a);
+ ok($checksum, 'recorded the checksum of the disk');
+ },
+)->step(
+ 'check migration preconditions',
+ sub {
+ my $precondition = $client->get("/nodes/$node_a/qemu/$vmid/migrate", {});
+ my $volid = $disk_volid->($node_a);
+ ok(
+ (grep { $_->{volid} eq $volid } ($precondition->{local_disks} // [])->@*),
+ 'disk reported as local',
+ );
+ },
+)->step(
+ 'live migration needs to allow local disks',
+ sub {
+ $pve_a->start_guest('qemu', $vmid);
+
+ my $upid = $migrate->($pve_a, { target => $node_b, online => 1 });
+ ok(
+ !$pve_a->task_successful($pve_a->wait_for_task($upid, 1)),
+ 'live migration without local disks allowed fails',
+ );
+ is($resource_node->(), $node_a, "VM $vmid stays on node $node_a");
+ is(
+ $client->get("/nodes/$node_a/qemu/$vmid/status/current", {})->{status},
+ 'running',
+ "VM $vmid keeps running",
+ );
+ },
+)->step(
+ 'live migrate VM with local disk to another storage',
+ sub {
+ my $upid = $migrate->(
+ $pve_a,
+ {
+ target => $node_b,
+ online => 1,
+ 'with-local-disks' => 1,
+ targetstorage => 'local-lvm',
+ },
+ );
+ $pve_a->task_ok($upid, "live migrated VM $vmid to node $node_b")
+ or die "live migrating VM $vmid failed\n";
+
+ is(
+ $client->get("/nodes/$node_b/qemu/$vmid/status/current", {})->{status},
+ 'running',
+ "VM $vmid keeps running during the live migration",
+ );
+ $verify_location->($pve_b, 'local-lvm', $pve_a, $zfs_storage);
+ },
+)->step(
+ 'live migrate VM with local disk back to the ZFS storage',
+ sub {
+ my $upid = $migrate->(
+ $pve_b,
+ {
+ target => $node_a,
+ online => 1,
+ 'with-local-disks' => 1,
+ targetstorage => $zfs_storage,
+ },
+ );
+ $pve_b->task_ok($upid, "live migrated VM $vmid back to node $node_a")
+ or die "live migrating VM $vmid back failed\n";
+
+ $verify_location->($pve_a, $zfs_storage, $pve_b, 'local-lvm');
+ },
+)->step(
+ 'migrate stopped VM with local disk',
+ sub {
+ $pve_a->stop_guest('qemu', $vmid);
+
+ my $upid = $migrate->($pve_a, { target => $node_b });
+ $pve_a->task_ok($upid, "migrated VM $vmid to node $node_b")
+ or die "migrating VM $vmid failed\n";
+
+ $verify_location->($pve_b, $zfs_storage, $pve_a, $zfs_storage);
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index 9903d79..e0d622c 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -151,6 +151,17 @@ TestcaseConfig(
],
tag: "extended",
),
+ Testcase(
+ name: "pve_vm_migration_local_disks",
+ run: "./tests/pve_vm_migration_local_disks.pl",
+ timeout: 600,
+ instances: [
+ "pve1-cluster",
+ "pve2-cluster",
+ "pve3-cluster",
+ ],
+ tag: "extended",
+ ),
Testcase(
name: "pve_replication",
run: "./tests/pve_replication.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 37/38] tests: ha: verify moving guests off a node for maintenance
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (35 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 36/38] tests: migration: verify VM migration with local disks Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-04 19:03 ` [PATCH e2e-tests 38/38] tests: migration: verify migrating guests in bulk Michael Köppl
2026-11-06 13:25 ` [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Kefu Chai
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Put a node with an HA-managed VM into HA maintenance mode and check
that the HA stack moves the VM to another node, and back once the
maintenance mode is disabled. HA maintenance mode is only available
through ha-manager on the node.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_node_maintenance.pl | 136 ++++++++++++++++++++++++++++++++++
tests/testcases.ron | 11 +++
2 files changed, 147 insertions(+)
create mode 100755 tests/pve_node_maintenance.pl
diff --git a/tests/pve_node_maintenance.pl b/tests/pve_node_maintenance.pl
new file mode 100755
index 0000000..6673761
--- /dev/null
+++ b/tests/pve_node_maintenance.pl
@@ -0,0 +1,136 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_node_maintenance> - Test moving HA-managed guests off a node in maintenance mode.
+
+=head1 DESCRIPTION
+
+An HA-managed VM is started on the first node. Putting the first node into HA
+maintenance mode must make the HA stack move the VM to another node, where it
+keeps running. Taking the node out of maintenance mode must move the VM back.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(get_pve_instance);
+use Proxmox::Test::Testcase;
+use Proxmox::Test::Util qw(poll_until);
+
+my $pve_a = get_pve_instance('pve1-cluster');
+my $node_a = $pve_a->get_nodename();
+my $client = $pve_a->client();
+
+my $ha_vmid;
+
+my $resource = sub {
+ my ($id) = @_;
+ my $resources = $client->get('/cluster/resources', { type => 'vm' });
+ my ($entry) = grep { ($_->{vmid} // 0) == $id } $resources->@*;
+ return $entry // {};
+};
+
+my $wait_for_guest = sub {
+ my ($id, $node, $timeout) = @_;
+ poll_until(
+ sub {
+ my $entry = $resource->($id);
+ return ($entry->{node} // '') eq $node && ($entry->{status} // '') eq 'running';
+ },
+ timeout => $timeout,
+ interval => 5,
+ description => "guest $id to run on node $node",
+ );
+};
+
+my $lrm_status = sub {
+ my ($node) = @_;
+ my ($entry) = grep { $_->{type} eq 'lrm' && $_->{node} eq $node }
+ $client->get('/cluster/ha/status/current', {})->@*;
+ return $entry ? $entry->{status} : '';
+};
+
+my $set_maintenance = sub {
+ my ($mode) = @_;
+ $pve_a->run_command("ha-manager crm-command node-maintenance $mode $node_a");
+};
+
+my $destroy = sub {
+ my ($type, $id) = @_;
+ my $node = $resource->($id)->{node};
+ $pve_a->destroy_guest_on($node, $type, $id) if defined($node);
+};
+
+Proxmox::Test::Testcase->new('pve_node_maintenance')->with([{}])->step(
+ 'start HA-managed VM',
+ sub {
+ $ha_vmid = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node_a/qemu",
+ { vmid => $ha_vmid, name => 'vm-ha-maintenance', memory => 256 },
+ );
+ $pve_a->task_ok($upid, "created VM $ha_vmid on node $node_a")
+ or die "creating VM $ha_vmid failed\n";
+
+ $client->post('/cluster/ha/resources', { sid => "vm:$ha_vmid", state => 'started' });
+ $pve_a->wait_for_ha_manager();
+
+ $wait_for_guest->($ha_vmid, $node_a, 120);
+ pass("HA started VM $ha_vmid on node $node_a");
+ },
+ sub {
+ eval { $client->delete("/cluster/ha/resources/vm:$ha_vmid", {}) };
+ $destroy->('qemu', $ha_vmid) if defined($ha_vmid);
+ },
+)->step(
+ 'enable maintenance mode',
+ sub {
+ $set_maintenance->('enable');
+
+ poll_until(
+ sub { $lrm_status->($node_a) =~ m/\bmaintenance mode\b/ },
+ timeout => 120,
+ interval => 5,
+ description => "node $node_a to enter maintenance mode",
+ );
+ pass("node $node_a is in maintenance mode");
+
+ poll_until(
+ sub {
+ my $entry = $resource->($ha_vmid);
+ return ($entry->{node} // $node_a) ne $node_a
+ && ($entry->{status} // '') eq 'running';
+ },
+ timeout => 240,
+ interval => 5,
+ description => "HA to move VM $ha_vmid off node $node_a",
+ );
+ pass("HA moved VM $ha_vmid to node " . $resource->($ha_vmid)->{node});
+ },
+ sub {
+ eval { $set_maintenance->('disable') };
+ },
+)->step(
+ 'disable maintenance mode',
+ sub {
+ $set_maintenance->('disable');
+
+ poll_until(
+ sub { $lrm_status->($node_a) !~ m/\bmaintenance mode\b/ },
+ timeout => 120,
+ interval => 5,
+ description => "node $node_a to leave maintenance mode",
+ );
+ pass("node $node_a is no longer in maintenance mode");
+
+ $wait_for_guest->($ha_vmid, $node_a, 240);
+ pass("HA moved VM $ha_vmid back to node $node_a");
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index e0d622c..0e4cea0 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -293,6 +293,17 @@ TestcaseConfig(
],
tag: "ha",
),
+ Testcase(
+ name: "pve_node_maintenance",
+ run: "./tests/pve_node_maintenance.pl",
+ timeout: 900,
+ instances: [
+ "pve1-cluster",
+ "pve2-cluster",
+ "pve3-cluster",
+ ],
+ tag: "ha",
+ ),
Testcase(
name: "pbs_s3_datastore",
run: "./tests/pbs/test_pbs_s3_datastore.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* [PATCH e2e-tests 38/38] tests: migration: verify migrating guests in bulk
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (36 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 37/38] tests: ha: verify moving guests off a node for maintenance Michael Köppl
@ 2026-11-04 19:03 ` Michael Köppl
2026-11-06 13:25 ` [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Kefu Chai
38 siblings, 0 replies; 47+ messages in thread
From: Michael Köppl @ 2026-11-04 19:03 UTC (permalink / raw)
To: pve-devel
Start a VM and a container that are not Ha-managed on one node and
migrate both to another node with a single bulk migration. Check that
both run on the target node afterwards.
Signed-off-by: Michael Köppl <m.koeppl@proxmox.com>
---
tests/pve_bulk_migration.pl | 111 ++++++++++++++++++++++++++++++++++++
tests/testcases.ron | 11 ++++
2 files changed, 122 insertions(+)
create mode 100755 tests/pve_bulk_migration.pl
diff --git a/tests/pve_bulk_migration.pl b/tests/pve_bulk_migration.pl
new file mode 100755
index 0000000..3b5c493
--- /dev/null
+++ b/tests/pve_bulk_migration.pl
@@ -0,0 +1,111 @@
+#!/usr/bin/perl
+
+=head1 NAME
+
+C<pve_bulk_migration> - Test migrating several guests between cluster nodes at once.
+
+=head1 DESCRIPTION
+
+This testcase starts a VM and a container that are not HA-managed on the first
+cluster node and migrates both to the second node in bulk. Afterwards, both must
+run on the second node.
+
+=cut
+
+use strict;
+use warnings;
+
+use Test::More;
+
+use lib '.';
+use Proxmox::Test::Helper qw(CONTAINER_TEMPLATE get_pve_instance);
+use Proxmox::Test::Testcase;
+use Proxmox::Test::Util qw(poll_until);
+
+my $pve_a = get_pve_instance('pve1-cluster');
+my $pve_b = get_pve_instance('pve2-cluster');
+my $node_a = $pve_a->get_nodename();
+my $node_b = $pve_b->get_nodename();
+my $client = $pve_a->client();
+
+my $vmid;
+my $ctid;
+
+# The node and status of a guest according to the cluster resources.
+my $resource = sub {
+ my ($id) = @_;
+ my $resources = $client->get('/cluster/resources', { type => 'vm' });
+ my ($entry) = grep { ($_->{vmid} // 0) == $id } $resources->@*;
+ return $entry // {};
+};
+
+my $wait_for_guest = sub {
+ my ($id, $node, $timeout) = @_;
+ poll_until(
+ sub {
+ my $entry = $resource->($id);
+ return ($entry->{node} // '') eq $node && ($entry->{status} // '') eq 'running';
+ },
+ timeout => $timeout,
+ interval => 5,
+ description => "guest $id to run on node $node",
+ );
+};
+
+my $destroy = sub {
+ my ($type, $id) = @_;
+ my $node = $resource->($id)->{node};
+ $pve_a->destroy_guest_on($node, $type, $id) if defined($node);
+};
+
+Proxmox::Test::Testcase->new('pve_bulk_migration')->with([{}])->step(
+ 'start VM and container',
+ sub {
+ $vmid = $client->get('/cluster/nextid');
+ my $upid = $client->post(
+ "/nodes/$node_a/qemu",
+ { vmid => $vmid, name => 'vm-bulk-migrate', memory => 256 },
+ );
+ $pve_a->task_ok($upid, "created VM $vmid on node $node_a")
+ or die "creating VM $vmid failed\n";
+ $pve_a->start_guest('qemu', $vmid);
+
+ $ctid = $client->get('/cluster/nextid');
+ $upid = $client->post(
+ "/nodes/$node_a/lxc",
+ {
+ vmid => $ctid,
+ ostemplate => CONTAINER_TEMPLATE,
+ rootfs => 'local-lvm:2',
+ memory => 256,
+ unprivileged => 1,
+ features => 'nesting=1',
+ hostname => 'ct-bulk-migrate',
+ },
+ );
+ $pve_a->task_ok($upid, "created container $ctid on node $node_a")
+ or die "creating container $ctid failed\n";
+ $pve_a->start_guest('lxc', $ctid);
+ },
+ sub {
+ $destroy->('lxc', $ctid) if defined($ctid);
+ $destroy->('qemu', $vmid) if defined($vmid);
+ },
+)->step(
+ 'migrate guests in bulk',
+ sub {
+ my $upid = $client->post(
+ "/nodes/$node_a/migrateall",
+ { target => $node_b, vms => "$vmid,$ctid", maxworkers => 2 },
+ );
+ $pve_a->task_ok($upid, "migrated guests from node $node_a to node $node_b")
+ or die "bulk migration failed\n";
+
+ $wait_for_guest->($vmid, $node_b, 60);
+ pass("VM $vmid runs on node $node_b");
+ $wait_for_guest->($ctid, $node_b, 60);
+ pass("container $ctid runs on node $node_b");
+ },
+)->run();
+
+done_testing();
diff --git a/tests/testcases.ron b/tests/testcases.ron
index 0e4cea0..7522ef2 100644
--- a/tests/testcases.ron
+++ b/tests/testcases.ron
@@ -304,6 +304,17 @@ TestcaseConfig(
],
tag: "ha",
),
+ Testcase(
+ name: "pve_bulk_migration",
+ run: "./tests/pve_bulk_migration.pl",
+ timeout: 600,
+ instances: [
+ "pve1-cluster",
+ "pve2-cluster",
+ "pve3-cluster",
+ ],
+ tag: "extended",
+ ),
Testcase(
name: "pbs_s3_datastore",
run: "./tests/pbs/test_pbs_s3_datastore.pl",
--
2.47.3
^ permalink raw reply related [flat|nested] 47+ messages in thread* Re: [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows
2026-11-04 19:03 [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Michael Köppl
` (37 preceding siblings ...)
2026-11-04 19:03 ` [PATCH e2e-tests 38/38] tests: migration: verify migrating guests in bulk Michael Köppl
@ 2026-11-06 13:25 ` Kefu Chai
38 siblings, 0 replies; 47+ messages in thread
From: Kefu Chai @ 2026-11-06 13:25 UTC (permalink / raw)
To: m.koeppl; +Cc: pve-devel
Hi Michael,
> [snip]
> - Ceph: setting up a hyper-converged cluster with dedicated public and
> cluster networks, containers and VMs on RBD storage (VMs with and
> without krbd), a CephFS with a storage for backups, and replacing an
> OSD. The old Ceph setup that was outdated anyway is replaced.
> [snip]
The recent aes256k migration got me thinking about upgrade coverage.
What do you think about a small upgrade test as a follow-up?
For Ceph, we could create a cluster on an older supported version,
put some known data in a guest on RBD, then upgrade and check that
the cluster is healthy and the guest can still read its data.
Since these tests create the cluster after updating packages,
that would give us a basic check of upgrading an existing setup.
Would it be straightforward to do with the current framework?
^ permalink raw reply [flat|nested] 47+ messages in thread