From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 901551FF0C0 for ; Wed, 04 Nov 2026 20:05:53 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id E36BA2164B; Wed, 04 Nov 2026 20:05:12 +0100 (CET) From: =?UTF-8?q?Michael=20K=C3=B6ppl?= To: pve-devel@lists.proxmox.com Subject: [PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows Date: Wed, 4 Nov 2026 20:03:07 +0100 Message-ID: <20261104190345.2573464-1-m.koeppl@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1793819033225 X-SPAM-LEVEL: Spam detection results: 0 AWL -2.284 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_DBL_SPAM 5 Contains a spam URL listed in the Spamhaus DBL blocklist [sdn.pm] Message-ID-Hash: 6VIIPKK5X64Z7RBQP6NKQPTSIE3PW4XS X-Message-ID-Hash: 6VIIPKK5X64Z7RBQP6NKQPTSIE3PW4XS X-MailFrom: m.koeppl@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This series adds E2E tests for tasks that are regularly performed on PVE and PBS, together with the helpers and setup steps they need. Generally, the tests also check the effect of an operation, not only that the API accepts it (e.g. checking manually inside the guest, reading disk content, etc.). New tests: - CTs: lifecycle and guest setup, volume and bind mountpoints, snapshots, clones and templates, device passthrough, application containers created from an OCI image - VMs: snapshots with RAM, clones and templates, hotplugging, resizing and removing disks, and lifecycle operations of a VM set up through cloud-init. - Ceph: setting up a hyper-converged cluster with dedicated public and cluster networks, containers and VMs on RBD storage (VMs with and without krbd), a CephFS with a storage for backups, and replacing an OSD. The old Ceph setup that was outdated anyway is replaced. - Backups: container and VM backup and restore, locally and on PBS, and single file restore from a container backup on PBS. - Storage: moving the volumes of VMs and containers between storages. - Cluster: container migration, VM migration with local disks, and moving guests off a node for maintenance, through HA maintenance mode and bulk migration. The setup plan now builds a minimal OCI image (to avoid dependency on a specific image or a single source for this image) and downloads a Debian cloud image onto pve1. It also installs Ceph on the cluster nodes. Added dependencies are jq (which was missing before even though it was needed), busybox-static and umoci (for the OCI image). The Ceph networks use the second NIC of the cluster nodes, which so far only the SDN tests used. As its bridge is no longer specific to SDN, the test-tools patch renames it from sdn_test to tt_test, and the e2e-tests patch updates the plans accordingly. 3 issues in PVE came up while writing the tests: - Starting a hibernated VM right after hibernating it can leave its config locked a number of seconds, as the qmeventd cleanup of the old QEMU process then waits for the new one to exit. The VM lifecycle test for now works around this by waiting for the cleanup before starting the VM again. The fix for #8286 might be pointer how to fix this. - pveproxy ends streamed downloads (e.g. single file restore) without a TLS close_notify, which curl reports as an error. Until patch [0] or another fix is applied, the file restore test accepts that error and compares the downloaded content instead. It it marked with FIXME. - Querying the status of a CephFS storage on a node where it is not mounted yet mounts it, but reports it as inactive. The CephFS test waits until pvestatd has mounted the storage before checking its status. [0] https://lore.proxmox.com/pve-devel/20261103155223.1960754-1-m.koeppl@proxmox.com/ test-tools: Michael Köppl (1): ansible: rename snd_test bridge to tt_test as multi-purpose bridge ansible/README.md | 6 +++--- ansible/group_vars/test_hosts.yml | 12 ++++++------ 2 files changed, 9 insertions(+), 9 deletions(-) e2e-tests: Michael Köppl (37): plans: rename sdn_test bridge to tt_test debian: add missing jq dependency test framework: move the default container template to the helper lib: instance: add helper to check if an address answers ping lib: pveinstance: look up disks by their SCSI serial lib: pveinstance: add helper to find a task started by another task lib: pveinstance: add helpers to start and stop guests lib: pveinstance: add helper to run commands in containers lib: pveinstance: add helper to wait for qmeventd's cleanup lib: pveinstance: add helper to check if guest exists lib: add helpers to use a PBS datastore as PVE storage scripts: retry container template download, fail if it fails tests: container: verify lifecycle and guest setup tests: container: verify volume and bind mountpoints tests: container: verify snapshots, clones, templates tests: container: verify device passthrough plans,scripts: build a minimal OCI image onto pve1 tests: container: verify application containers from OCI images tests: vm: verify snapshots, clones, templates tests: vm: verify adding, resizing, and removing disks plans,scripts: download a Debian cloud image onto pve1 tests: vm: verify lifecycle operations and cloud-init guest setup plans,scripts: install Ceph Tentacle on the cluster nodes test framework: add Ceph test helpers tests: ceph: verify setting up a hyper-converged cluster tests: ceph: verify guests on RBD storage tests: ceph: verify setting up, using and destroying a CephFS tests: ceph: verify replacing an OSD tests: ceph: drop the outdated Ceph setup tests: pbs: verify container backup and restore tests: backup: verify VM backup and restore, locally and on PBS tests: pbs: verify single file restore from a container backup tests: storage: verify moving guest volumes between storages tests: migration: verify container migration between nodes tests: migration: verify VM migration with local disks tests: ha: verify moving guests off a node for maintenance tests: migration: verify migrating guests in bulk Proxmox/Test/Ceph.pm | 338 +++++++++++++ Proxmox/Test/Helper.pm | 8 +- Proxmox/Test/Instance.pm | 7 + Proxmox/Test/PBSInstance.pm | 19 + Proxmox/Test/PVEInstance.pm | 136 +++++- Proxmox/Test/SDN.pm | 5 - debian/control | 5 +- debian/proxmox-e2e-tests-scripts.install | 1 - plans/ceph.ron | 87 ---- plans/complete.ron | 17 +- plans/sdn.ron | 12 +- run_ceph_tests.sh | 10 - scripts/build-oci-image.sh | 69 +++ scripts/download-cloud-image.sh | 21 + scripts/download-container-image.sh | 11 +- scripts/install-ceph.sh | 13 + tests/ceph/test_ceph_cephfs.pl | 311 ++++++++++++ tests/ceph/test_ceph_cluster_setup.pl | 206 ++++++++ tests/ceph/test_ceph_osd_replace.pl | 237 +++++++++ tests/ceph/test_ceph_rbd_storage.pl | 597 +++++++++++++++++++++++ tests/ceph_testcases.ron | 29 -- tests/create_ceph_cluster.pl | 69 --- tests/pbs/test_pbs_ct_backup_restore.pl | 150 ++++++ tests/pbs/test_pbs_file_restore.pl | 151 ++++++ tests/pve_bulk_migration.pl | 111 +++++ tests/pve_ct_device_passthrough.pl | 162 ++++++ tests/pve_ct_lifecycle.pl | 182 +++++++ tests/pve_ct_migration.pl | 159 ++++++ tests/pve_ct_mountpoints.pl | 198 ++++++++ tests/pve_ct_oci.pl | 190 ++++++++ tests/pve_ct_snapshot_clone.pl | 267 ++++++++++ tests/pve_firewall.pl | 26 +- tests/pve_move_disks.pl | 262 ++++++++++ tests/pve_node_maintenance.pl | 136 ++++++ tests/pve_vm_backup_restore.pl | 269 ++++++++++ tests/pve_vm_disks.pl | 195 ++++++++ tests/pve_vm_lifecycle.pl | 442 +++++++++++++++++ tests/pve_vm_migration_local_disks.pl | 213 ++++++++ tests/pve_vm_snapshot_clone.pl | 282 +++++++++++ tests/sdn/test_vlan_zone.pl | 4 +- tests/test_create_cluster.pl | 46 -- tests/testcases.ron | 199 ++++++++ 42 files changed, 5569 insertions(+), 283 deletions(-) create mode 100644 Proxmox/Test/Ceph.pm delete mode 100644 plans/ceph.ron delete mode 100755 run_ceph_tests.sh create mode 100755 scripts/build-oci-image.sh create mode 100755 scripts/download-cloud-image.sh create mode 100755 scripts/install-ceph.sh create mode 100755 tests/ceph/test_ceph_cephfs.pl create mode 100755 tests/ceph/test_ceph_cluster_setup.pl create mode 100755 tests/ceph/test_ceph_osd_replace.pl create mode 100755 tests/ceph/test_ceph_rbd_storage.pl delete mode 100644 tests/ceph_testcases.ron delete mode 100755 tests/create_ceph_cluster.pl create mode 100755 tests/pbs/test_pbs_ct_backup_restore.pl create mode 100755 tests/pbs/test_pbs_file_restore.pl create mode 100755 tests/pve_bulk_migration.pl create mode 100755 tests/pve_ct_device_passthrough.pl create mode 100755 tests/pve_ct_lifecycle.pl create mode 100755 tests/pve_ct_migration.pl create mode 100755 tests/pve_ct_mountpoints.pl create mode 100755 tests/pve_ct_oci.pl create mode 100755 tests/pve_ct_snapshot_clone.pl create mode 100755 tests/pve_move_disks.pl create mode 100755 tests/pve_node_maintenance.pl create mode 100755 tests/pve_vm_backup_restore.pl create mode 100755 tests/pve_vm_disks.pl create mode 100755 tests/pve_vm_lifecycle.pl create mode 100755 tests/pve_vm_migration_local_disks.pl create mode 100755 tests/pve_vm_snapshot_clone.pl delete mode 100755 tests/test_create_cluster.pl Summary over all repositories: 44 files changed, 5578 insertions(+), 292 deletions(-) -- Generated by murpp 0.12.1