Re: [pve-devel] [RFC] towards automated integration testing

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

Am 16/10/2023 um 17:33 schrieb Lukas Wagner:
>> Or is this some part that takes place in the test, i.e., a
>> generalization of product to test and supplementary tool/app that helps
>> on that test?
> 
> It was intended to be a 'general VM/CT base thingy' that can be
> instantiated and managed by the test runner, so either a PVE/PBS/PMG
> base installation, or some auxiliary resource, e.g. a Debian VM with
> an already-set-up LDAP server.
> 
> I'll see if I can find good terms with the newly added focus on
> bare-metal testing / the decoupling between environment setup and test
> execution.

Hmm, yeah OK, having some additional info on top of "template" like e.g.,
"system-template", or "app-template", could be already slightly better
then.

While slightly details, IMO still important for overall future
direction, I'd possibly split "restore" into "source-type" and "source",
where the "source-type" can be e.g., "disk-image" for a qcow2 or the
like to work directly on, or "backup-image" for your backup restore
process, or some type for bootstrap tools like debootstrap or the VM
specific vmdb2.

Also having re-use configurable, i.e., if the app-template-instance
is destroyed after some test run is done. For that, writing a simple
info about mapping instantiated templates to other identifiers (VMID,
IP, ...) in e.g. /var/cache/ (or some XDG_ directory to cater also to
any users running this as non-root).

Again, can be classified as details, but IMO important for the
direction this is going, and not too much work, so should be at least
on the radar.

>> Is the order of test-cases guaranteed by toml parsing, or how are intra-
>> fixture dependencies ensured?
>>
> 
> Good point. With rollbacks in between test cases it probably does not
> matter much, but on 'real hardware' with no rollback this could
> definitely be a concern.
> A super simple thing that could just work fine is ordering test
> execution by testcase-names, sorted alphabetically. Ideally you'd write
> test cases that do not depend on each other any way, and *if* you ever
> find yourself in the situation where you *need* some ordering, you
> could> just encode the order in the test-case name by adding an integer
> prefix> - similar how you would name config files in /etc/sysctl.d/*,
> for instance.


While it can be OK to leave that for later, encoding such things
in names is IMO brittle and hard to manage if more than a handful
of tests, and we hopefully got lots more ;-)


From top of my head I'd rather do some attribute based dependency
annotation, so that one can depend on single tests, or whole fixture
on others single tests or whole fixture.