From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id F16739AF79 for ; Mon, 16 Oct 2023 17:33:38 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id CC20F1A949 for ; Mon, 16 Oct 2023 17:33:08 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 16 Oct 2023 17:33:07 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 71BEA40688 for ; Mon, 16 Oct 2023 17:33:07 +0200 (CEST) Message-ID: <832bc039-57d0-4d27-ac48-721c5c82af83@proxmox.com> Date: Mon, 16 Oct 2023 17:33:06 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: de-AT, en-US To: Thomas Lamprecht , Proxmox VE development discussion References: <1bb25817-66e7-406a-bd4b-0699de6cba31@proxmox.com> From: Lukas Wagner In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.027 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [RFC] towards automated integration testing X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Oct 2023 15:33:39 -0000 Thanks for the summary from our discussion and the additional feedback! On 10/16/23 15:57, Thomas Lamprecht wrote: >> - create some sort of test report > > As Stefan mentioned, test-output can be good to have. Our buildbot > instance provides that, and while I don't look at them in 99% of the > builds, when I need to its worth *a lot*. > Agreed, test output is always valuable and will definitely captured. >> >> ## Introduction >> >> The goal is to establish a framework that allows us to write >> automated integration tests for our products. >> These tests are intended to run in the following situations: >> - When new packages are uploaded to the staging repos (by triggering >> a test run from repoman, or similar) > > *debian repos, as we could also trigger some when git commits are > pushed, just like we do now through Buildbot. Doing so is IMO nice as it > will catch issues before a package was bumped, but is still quite a bit > simpler to implement than an "apply patch from list to git repos" thing > from the next point, but could still act as a preparation for that. > >> - Later, this tests could also be run when patch series are posted to >> our mailing lists. This requires a mechanism to automatically >> discover, fetch and build patches, which will be a separate, >> follow-up project. > >> >> As a main mode of operation, the Systems under Test (SUTs) >> will be virtualized on top of a Proxmox VE node. > > For the fully-automated test system this can be OK as primary mode, as > it indeed makes things like going back to an older software state much > easier. > > But, if we decouple the test harness and running them from that more > automated system, we can also run the harness periodically on our > bare-metal test servers. > >> ## Terminology >> - Template: A backup/VM template that can be instantiated by the test >> runner > > I.e., the base of the test host? I'd call this test-host, template is a > bit to overloaded/generic and might focus too much on the virtual test > environment. True, 'template' is a bit overloaded. > > Or is this some part that takes place in the test, i.e., a > generalization of product to test and supplementary tool/app that helps > on that test? It was intended to be a 'general VM/CT base thingy' that can be instantiated and managed by the test runner, so either a PVE/PBS/PMG base installation, or some auxiliary resource, e.g. a Debian VM with an already-set-up LDAP server. I'll see if I can find good terms with the newly added focus on bare-metal testing / the decoupling between environment setup and test execution. > Is the order of test-cases guaranteed by toml parsing, or how are intra- > fixture dependencies ensured? > Good point. With rollbacks in between test cases it probably does not matter much, but on 'real hardware' with no rollback this could definitely be a concern. A super simple thing that could just work fine is ordering test execution by testcase-names, sorted alphabetically. Ideally you'd write test cases that do not depend on each other any way, and *if* you ever find yourself in the situation where you *need* some ordering, you could just encode the order in the test-case name by adding an integer prefix - similar how you would name config files in /etc/sysctl.d/*, for instance. -- - Lukas