[PATCH e2e-tests/test-tools 00/38] tests: add tests for guests, Ceph, PBS and cluster workflows

["Michael Köppl" <m.koeppl@proxmox.com>]

This series adds E2E tests for tasks that are regularly performed on
PVE and PBS, together with the helpers and setup steps they need.
Generally, the tests also check the effect of an operation, not only
that the API accepts it (e.g. checking manually inside the guest,
reading disk content, etc.).

New tests:
- CTs: lifecycle and guest setup, volume and bind mountpoints,
  snapshots, clones and templates, device passthrough, application
  containers created from an OCI image
- VMs: snapshots with RAM, clones and templates, hotplugging, resizing
  and removing disks, and lifecycle operations of a VM set up through
  cloud-init.
- Ceph: setting up a hyper-converged cluster with dedicated public and
  cluster networks, containers and VMs on RBD storage (VMs with and
  without krbd), a CephFS with a storage for backups, and replacing an
  OSD. The old Ceph setup that was outdated anyway is replaced.
- Backups: container and VM backup and restore, locally and on PBS,
  and single file restore from a container backup on PBS.
- Storage: moving the volumes of VMs and containers between storages.
- Cluster: container migration, VM migration with local disks, and
  moving guests off a node for maintenance, through HA maintenance
  mode and bulk migration.

The setup plan now builds a minimal OCI image (to avoid dependency on
a specific image or a single source for this image) and downloads a
Debian cloud image onto pve1. It also installs Ceph on the cluster
nodes. Added dependencies are jq (which was missing before even though
it was needed), busybox-static and umoci (for the OCI image).

The Ceph networks use the second NIC of the cluster nodes, which so
far only the SDN tests used. As its bridge is no longer specific to
SDN, the test-tools patch renames it from sdn_test to tt_test, and the
e2e-tests patch updates the plans accordingly.

3 issues in PVE came up while writing the tests:
- Starting a hibernated VM right after hibernating it can leave its
  config locked a number of seconds, as the qmeventd cleanup of the old
  QEMU process then waits for the new one to exit. The VM lifecycle
  test for now works around this by waiting for the cleanup before
  starting the VM again. The fix for #8286 might be pointer how to fix
  this.
- pveproxy ends streamed downloads (e.g. single file restore) without
  a TLS close_notify, which curl reports as an error. Until patch [0]
  or another fix is applied, the file restore test accepts that error
  and compares the downloaded content instead. It it marked with FIXME.
- Querying the status of a CephFS storage on a node where it is not
  mounted yet mounts it, but reports it as inactive. The CephFS test
  waits until pvestatd has mounted the storage before checking its
  status.

[0] https://lore.proxmox.com/pve-devel/20261103155223.1960754-1-m.koeppl@proxmox.com/


test-tools:

Michael Köppl (1):
  ansible: rename snd_test bridge to tt_test as multi-purpose bridge

 ansible/README.md                 |  6 +++---
 ansible/group_vars/test_hosts.yml | 12 ++++++------
 2 files changed, 9 insertions(+), 9 deletions(-)


e2e-tests:

Michael Köppl (37):
  plans: rename sdn_test bridge to tt_test
  debian: add missing jq dependency
  test framework: move the default container template to the helper
  lib: instance: add helper to check if an address answers ping
  lib: pveinstance: look up disks by their SCSI serial
  lib: pveinstance: add helper to find a task started by another task
  lib: pveinstance: add helpers to start and stop guests
  lib: pveinstance: add helper to run commands in containers
  lib: pveinstance: add helper to wait for qmeventd's cleanup
  lib: pveinstance: add helper to check if guest exists
  lib: add helpers to use a PBS datastore as PVE storage
  scripts: retry container template download, fail if it fails
  tests: container: verify lifecycle and guest setup
  tests: container: verify volume and bind mountpoints
  tests: container: verify snapshots, clones, templates
  tests: container: verify device passthrough
  plans,scripts: build a minimal OCI image onto pve1
  tests: container: verify application containers from OCI images
  tests: vm: verify snapshots, clones, templates
  tests: vm: verify adding, resizing, and removing disks
  plans,scripts: download a Debian cloud image onto pve1
  tests: vm: verify lifecycle operations and cloud-init guest setup
  plans,scripts: install Ceph Tentacle on the cluster nodes
  test framework: add Ceph test helpers
  tests: ceph: verify setting up a hyper-converged cluster
  tests: ceph: verify guests on RBD storage
  tests: ceph: verify setting up, using and destroying a CephFS
  tests: ceph: verify replacing an OSD
  tests: ceph: drop the outdated Ceph setup
  tests: pbs: verify container backup and restore
  tests: backup: verify VM backup and restore, locally and on PBS
  tests: pbs: verify single file restore from a container backup
  tests: storage: verify moving guest volumes between storages
  tests: migration: verify container migration between nodes
  tests: migration: verify VM migration with local disks
  tests: ha: verify moving guests off a node for maintenance
  tests: migration: verify migrating guests in bulk

 Proxmox/Test/Ceph.pm                     | 338 +++++++++++++
 Proxmox/Test/Helper.pm                   |   8 +-
 Proxmox/Test/Instance.pm                 |   7 +
 Proxmox/Test/PBSInstance.pm              |  19 +
 Proxmox/Test/PVEInstance.pm              | 136 +++++-
 Proxmox/Test/SDN.pm                      |   5 -
 debian/control                           |   5 +-
 debian/proxmox-e2e-tests-scripts.install |   1 -
 plans/ceph.ron                           |  87 ----
 plans/complete.ron                       |  17 +-
 plans/sdn.ron                            |  12 +-
 run_ceph_tests.sh                        |  10 -
 scripts/build-oci-image.sh               |  69 +++
 scripts/download-cloud-image.sh          |  21 +
 scripts/download-container-image.sh      |  11 +-
 scripts/install-ceph.sh                  |  13 +
 tests/ceph/test_ceph_cephfs.pl           | 311 ++++++++++++
 tests/ceph/test_ceph_cluster_setup.pl    | 206 ++++++++
 tests/ceph/test_ceph_osd_replace.pl      | 237 +++++++++
 tests/ceph/test_ceph_rbd_storage.pl      | 597 +++++++++++++++++++++++
 tests/ceph_testcases.ron                 |  29 --
 tests/create_ceph_cluster.pl             |  69 ---
 tests/pbs/test_pbs_ct_backup_restore.pl  | 150 ++++++
 tests/pbs/test_pbs_file_restore.pl       | 151 ++++++
 tests/pve_bulk_migration.pl              | 111 +++++
 tests/pve_ct_device_passthrough.pl       | 162 ++++++
 tests/pve_ct_lifecycle.pl                | 182 +++++++
 tests/pve_ct_migration.pl                | 159 ++++++
 tests/pve_ct_mountpoints.pl              | 198 ++++++++
 tests/pve_ct_oci.pl                      | 190 ++++++++
 tests/pve_ct_snapshot_clone.pl           | 267 ++++++++++
 tests/pve_firewall.pl                    |  26 +-
 tests/pve_move_disks.pl                  | 262 ++++++++++
 tests/pve_node_maintenance.pl            | 136 ++++++
 tests/pve_vm_backup_restore.pl           | 269 ++++++++++
 tests/pve_vm_disks.pl                    | 195 ++++++++
 tests/pve_vm_lifecycle.pl                | 442 +++++++++++++++++
 tests/pve_vm_migration_local_disks.pl    | 213 ++++++++
 tests/pve_vm_snapshot_clone.pl           | 282 +++++++++++
 tests/sdn/test_vlan_zone.pl              |   4 +-
 tests/test_create_cluster.pl             |  46 --
 tests/testcases.ron                      | 199 ++++++++
 42 files changed, 5569 insertions(+), 283 deletions(-)
 create mode 100644 Proxmox/Test/Ceph.pm
 delete mode 100644 plans/ceph.ron
 delete mode 100755 run_ceph_tests.sh
 create mode 100755 scripts/build-oci-image.sh
 create mode 100755 scripts/download-cloud-image.sh
 create mode 100755 scripts/install-ceph.sh
 create mode 100755 tests/ceph/test_ceph_cephfs.pl
 create mode 100755 tests/ceph/test_ceph_cluster_setup.pl
 create mode 100755 tests/ceph/test_ceph_osd_replace.pl
 create mode 100755 tests/ceph/test_ceph_rbd_storage.pl
 delete mode 100644 tests/ceph_testcases.ron
 delete mode 100755 tests/create_ceph_cluster.pl
 create mode 100755 tests/pbs/test_pbs_ct_backup_restore.pl
 create mode 100755 tests/pbs/test_pbs_file_restore.pl
 create mode 100755 tests/pve_bulk_migration.pl
 create mode 100755 tests/pve_ct_device_passthrough.pl
 create mode 100755 tests/pve_ct_lifecycle.pl
 create mode 100755 tests/pve_ct_migration.pl
 create mode 100755 tests/pve_ct_mountpoints.pl
 create mode 100755 tests/pve_ct_oci.pl
 create mode 100755 tests/pve_ct_snapshot_clone.pl
 create mode 100755 tests/pve_move_disks.pl
 create mode 100755 tests/pve_node_maintenance.pl
 create mode 100755 tests/pve_vm_backup_restore.pl
 create mode 100755 tests/pve_vm_disks.pl
 create mode 100755 tests/pve_vm_lifecycle.pl
 create mode 100755 tests/pve_vm_migration_local_disks.pl
 create mode 100755 tests/pve_vm_snapshot_clone.pl
 delete mode 100755 tests/test_create_cluster.pl


Summary over all repositories:
  44 files changed, 5578 insertions(+), 292 deletions(-)

-- 
Generated by murpp 0.12.1