From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 8CCAC9AF92 for ; Mon, 16 Oct 2023 15:57:58 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 6351E18C18 for ; Mon, 16 Oct 2023 15:57:28 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 16 Oct 2023 15:57:27 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 2802A417C2 for ; Mon, 16 Oct 2023 15:57:27 +0200 (CEST) Message-ID: Date: Mon, 16 Oct 2023 15:57:25 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Thomas Lamprecht To: Proxmox VE development discussion , Lukas Wagner References: <1bb25817-66e7-406a-bd4b-0699de6cba31@proxmox.com> Content-Language: de-AT, en-US Autocrypt: addr=t.lamprecht@proxmox.com; keydata= xsFNBFsLjcYBEACsaQP6uTtw/xHTUCKF4VD4/Wfg7gGn47+OfCKJQAD+Oyb3HSBkjclopC5J uXsB1vVOfqVYE6PO8FlD2L5nxgT3SWkc6Ka634G/yGDU3ZC3C/7NcDVKhSBI5E0ww4Qj8s9w OQRloemb5LOBkJNEUshkWRTHHOmk6QqFB/qBPW2COpAx6oyxVUvBCgm/1S0dAZ9gfkvpqFSD 90B5j3bL6i9FIv3YGUCgz6Ue3f7u+HsEAew6TMtlt90XV3vT4M2IOuECG/pXwTy7NtmHaBQ7 UJBcwSOpDEweNob50+9B4KbnVn1ydx+K6UnEcGDvUWBkREccvuExvupYYYQ5dIhRFf3fkS4+ wMlyAFh8PQUgauod+vqs45FJaSgTqIALSBsEHKEs6IoTXtnnpbhu3p6XBin4hunwoBFiyYt6 YHLAM1yLfCyX510DFzX/Ze2hLqatqzY5Wa7NIXqYYelz7tXiuCLHP84+sV6JtEkeSUCuOiUY virj6nT/nJK8m0BzdR6FgGtNxp7RVXFRz/+mwijJVLpFsyG1i0Hmv2zTn3h2nyGK/I6yhFNt dX69y5hbo6LAsRjLUvZeHXpTU4TrpN/WiCjJblbj5um5eEr4yhcwhVmG102puTtuCECsDucZ jpKpUqzXlpLbzG/dp9dXFH3MivvfuaHrg3MtjXY1i+/Oxyp5iwARAQABzTNUaG9tYXMgTGFt cHJlY2h0IChBdXRoLTQpIDx0LmxhbXByZWNodEBwcm94bW94LmNvbT7CwY4EEwEIADgWIQQO R4qbEl/pah9K6VrTZCM6gDZWBgUCWwuNxgIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAK CRDTZCM6gDZWBm/jD/4+6JB2s67eaqoP6x9VGaXNGJPCscwzLuxDTCG90G9FYu29VcXtubH/ bPwsyBbNUQpqTm/s4XboU2qpS5ykCuTjqavrcP33tdkYfGcItj2xMipJ1i3TWvpikQVsX42R G64wovLs/dvpTYphRZkg5DwhgTmy3mRkmofFCTa+//MOcNOORltemp984tWjpR3bUJETNWpF sKGZHa3N4kCNxb7A+VMsJZ/1gN3jbQbQG7GkJtnHlWkw9rKCYqBtWrnrHa4UAvSa9M/XCIAB FThFGqZI1ojdVlv5gd6b/nWxfOPrLlSxbUo5FZ1i/ycj7/24nznW1V4ykG9iUld4uYUY86bB UGSjew1KYp9FmvKiwEoB+zxNnuEQfS7/Bj1X9nxizgweiHIyFsRqgogTvLh403QMSGNSoArk tqkorf1U+VhEncIn4H3KksJF0njZKfilrieOO7Vuot1xKr9QnYrZzJ7m7ZxJ/JfKGaRHXkE1 feMmrvZD1AtdUATZkoeQtTOpMu4r6IQRfSdwm/CkppZXfDe50DJxAMDWwfK2rr2bVkNg/yZI tKLBS0YgRTIynkvv0h8d9dIjiicw3RMeYXyqOnSWVva2r+tl+JBaenr8YTQw0zARrhC0mttu cIZGnVEvQuDwib57QLqMjQaC1gazKHvhA15H5MNxUhwm229UmdH3KM7BTQRbC43GARAAyTkR D6KRJ9Xa2fVMh+6f186q0M3ni+5tsaVhUiykxjsPgkuWXWW9MbLpYXkzX6h/RIEKlo2BGA95 QwG5+Ya2Bo3g7FGJHAkXY6loq7DgMp5/TVQ8phsSv3WxPTJLCBq6vNBamp5hda4cfXFUymsy HsJy4dtgkrPQ/bnsdFDCRUuhJHopnAzKHN8APXpKU6xV5e3GE4LwFsDhNHfH/m9+2yO/trcD txSFpyftbK2gaMERHgA8SKkzRhiwRTt9w5idOfpJVkYRsgvuSGZ0pcD4kLCOIFrer5xXudk6 NgJc36XkFRMnwqrL/bB4k6Pi2u5leyqcXSLyBgeHsZJxg6Lcr2LZ35+8RQGPOw9C0ItmRjtY ZpGKPlSxjxA1WHT2YlF9CEt3nx7c4C3thHHtqBra6BGPyW8rvtq4zRqZRLPmZ0kt/kiMPhTM 8wZAlObbATVrUMcZ/uNjRv2vU9O5aTAD9E5r1B0dlqKgxyoImUWB0JgpILADaT3VybDd3C8X s6Jt8MytUP+1cEWt9VKo4vY4Jh5vwrJUDLJvzpN+TsYCZPNVj18+jf9uGRaoK6W++DdMAr5l gQiwsNgf9372dbMI7pt2gnT5/YdG+ZHnIIlXC6OUonA1Ro/Itg90Q7iQySnKKkqqnWVc+qO9 GJbzcGykxD6EQtCSlurt3/5IXTA7t6sAEQEAAcLBdgQYAQgAIBYhBA5HipsSX+lqH0rpWtNk IzqANlYGBQJbC43GAhsMAAoJENNkIzqANlYGD1sP/ikKgHgcspEKqDED9gQrTBvipH85si0j /Jwu/tBtnYjLgKLh2cjv1JkgYYjb3DyZa1pLsIv6rGnPX9bH9IN03nqirC/Q1Y1lnbNTynPk IflgvsJjoTNZjgu1wUdQlBgL/JhUp1sIYID11jZphgzfDgp/E6ve/8xE2HMAnf4zAfJaKgD0 F+fL1DlcdYUditAiYEuN40Ns/abKs8I1MYx7Yglu3RzJfBzV4t86DAR+OvuF9v188WrFwXCS RSf4DmJ8tntyNej+DVGUnmKHupLQJO7uqCKB/1HLlMKc5G3GLoGqJliHjUHUAXNzinlpE2Vj C78pxpwxRNg2ilE3AhPoAXrY5qED5PLE9sLnmQ9AzRcMMJUXjTNEDxEYbF55SdGBHHOAcZtA kEQKub86e+GHA+Z8oXQSGeSGOkqHi7zfgW1UexddTvaRwE6AyZ6FxTApm8wq8NT2cryWPWTF BDSGB3ujWHMM8ERRYJPcBSjTvt0GcEqnd+OSGgxTkGOdufn51oz82zfpVo1t+J/FNz6MRMcg 8nEC+uKvgzH1nujxJ5pRCBOquFZaGn/p71Yr0oVitkttLKblFsqwa+10Lt6HBxm+2+VLp4Ja 0WZNncZciz3V3cuArpan/ZhhyiWYV5FD0pOXPCJIx7WS9PTtxiv0AOS4ScWEUmBxyhFeOpYa DrEx In-Reply-To: <1bb25817-66e7-406a-bd4b-0699de6cba31@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.076 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [RFC] towards automated integration testing X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Oct 2023 13:57:58 -0000 A few things, most of which we talked off-list already anyway. We should eye if we can integrate existing regression testing in there too, i.e.: - The qemu autotest that Stefan Reiter started and Fiona still uses, here we should drop the in-git tracked backup that the test VM is restored from (replace with something like vmdb2 [0] managed Debian image that gets generated on demand), replace some hard coded configs with a simple config and make it public. [0]: https://vmdb2.liw.fi/ - The selenium based end-to-end tests which we also use to generate most screenshots with (they can run headless too). Here we also need a few clean-ups, but not that many, and make the repo public. Am 13/10/2023 um 15:33 schrieb Lukas Wagner:> I am currently doing the groundwork that should eventually enable us > to write automated integration tests for our products. > > Part of that endeavor will be to write a custom test runner, which will > - setup a specified test environment > - execute test cases in that environment This should be decoupled from all else, so that I can run it on any existing installation, bare-metal or not. This allows devs using it in their existing setups with almost no change required. We can then also add it easily in our existing buildbot instance relatively easily, so it would be worth doing so even if we might deprecate Buildbot in the future (for what little it can do, it could be simpler). > - create some sort of test report As Stefan mentioned, test-output can be good to have. Our buildbot instance provides that, and while I don't look at them in 99% of the builds, when I need to its worth *a lot*. > > ## Introduction > > The goal is to establish a framework that allows us to write > automated integration tests for our products. > These tests are intended to run in the following situations: > - When new packages are uploaded to the staging repos (by triggering > a test run from repoman, or similar) *debian repos, as we could also trigger some when git commits are pushed, just like we do now through Buildbot. Doing so is IMO nice as it will catch issues before a package was bumped, but is still quite a bit simpler to implement than an "apply patch from list to git repos" thing from the next point, but could still act as a preparation for that. > - Later, this tests could also be run when patch series are posted to > our mailing lists. This requires a mechanism to automatically > discover, fetch and build patches, which will be a separate, > follow-up project. > > As a main mode of operation, the Systems under Test (SUTs) > will be virtualized on top of a Proxmox VE node. For the fully-automated test system this can be OK as primary mode, as it indeed makes things like going back to an older software state much easier. But, if we decouple the test harness and running them from that more automated system, we can also run the harness periodically on our bare-metal test servers. > ## Terminology > - Template: A backup/VM template that can be instantiated by the test > runner I.e., the base of the test host? I'd call this test-host, template is a bit to overloaded/generic and might focus too much on the virtual test environment. Or is this some part that takes place in the test, i.e., a generalization of product to test and supplementary tool/app that helps on that test? Hmm, could work out ok, and we should be able to specialize stuff relatively easier later too, if wanted. > - Test Case: Some script/executable executed by the test runner, success > is determined via exit code. > - Fixture: Description of a test setup (e.g. which templates are needed, > additional setup steps to run, etc.) > > ## Approach > Test writers write template, fixture, test case definition in > declarative configuration files (most likely TOML). The test case > references a test executable/script, which performs the actual test. > > The test script is executed by the test runner; the test outcome is > determined by the exit code of the script. Test scripts could be written > in any language, e.g. they could be Perl scripts that use the official > `libpve-apiclient-perl` to test-drive the SUTs. > If we notice any emerging patterns, we could write additional helper > libs that reduce the amount of boilerplate in test scripts. > > In essence, the test runner would do the following: > - Group testcases by fixture > - For every fixture: > - Instantiate needed templates from their backup snapshot Should be optional, possible a default-on boolean option that conveys > - Start VMs Same. > - Run any specified `setup-hooks` (update system, deploy packages, > etc.) Should be as idempotent as possible. > - Take a snapshot, including RAM Should be optional (as in, don't care if it cannot be done, e.g., on bare metal). > - For every testcase using that fixture: > - Run testcase (execute test executable, check exit code) > - Rollback to snapshot (iff `rollback = true` for that template) > - destroy test instances (or at least those which are not needed by > other fixtures) Might be optional for l1 hosts, l2 test VMs might be a separate switch. > In the beginning, the test scripts would primarily drive the Systems > under Test (SUTs) via their API. However, the system would also offer > the flexibility for us to venture into the realm of automated GUI > testing at some point (e.g. using selenium) - without having to > change the overall test architecture. Our existing selenium based UI test simple use the API to create stuff that it needs, if it's not existing, and sometimes remove also some. It uses some special ranges or values to avoid most conflicts with real systems, allowing one to point it at existing (production) systems without problems. IMO this has a big value, and I actually added a bit of resiliency, as I find that having to set up clean states a bit annoying and for one of the main use cases of that tooling, creating screenshots, too sterile. But always starting out from a very clean state is IMO not only "ugly" for screenshots, but can also sometimes mas issues that test can run into on systems with a longer uptime and the "organic mess" that comes from long-term maintenance. In practice one naturally wants both, starting from a clean state and from existing one, both have their advantages and disadvantages. Like messy systems also might have more false-positives on regression tracking. > > ## Mock Test Runner Config > > Beside the actual test scripts, test writers would write test > configuration. Based on the current requirements and approach that > I have chose, a example config *could* look like the one following. > These would likely be split into multiple files/folders > (e.g. to group test case definition and the test script logically). > > ```toml > [template.pve-default] > # Backup image to restore from, in this case this would be a previously > # set up PVE installation > restore = '...' > # To check if node is booted successfully, also made available to hook > # scripts, in case they need to SSH in to setup things. > ip = "10.0.0.1" > # Define credentials in separate file - most template could use a > # default password/SSH key/API token etc. > credentials = "default" > # Update to latest packages, install test .debs > # credentials are passed via env var > # Maybe this could also be ansible playbooks, if the need arises. fwiw, one could also define a config-deployment-system, like - none (already is setup) - cloudinit - QGA but that can be added later on too. > setup-hooks = [ > "update.sh", > ] > # Take snapshot after setup-hook, roll back after each test case > rollback = true > > > [template.ldap-server] > # Backup image to restore from > restore = '...' > credentials = "default" > ip = "10.0.0.3" > # No need to roll back in between test cases, there won't be any changes > rollback = false > > > > # Example fixture. They can be used by multiple testcases. > [fixture.pve-with-ldap-server] > # Maybe one could specify additional setup-hooks here as well, in case > # one wants a 'per-fixture' setup? So that we can reduce the number of > # base images? > templates = [ > 'pve-default', > 'ldap-server', > ] > > > # testcases.toml (might be split to multiple files/folders?) maybe some sort of predicates could be also nice (even if not there from the start), like to place condition where a test is skipped if that's not met, like the existence of a ZFS-storage or something like that. While those seem like details, having a general (simple) dependency and, so to say, anti-dependency system might influence overall design more. > [testcase.test-ldap-realms] > fixture = 'pve-with-ldap-server' > > # - return code is check to determine test case success > # - stderr/stdout is captured for the final test report > # - some data is passed via env var: > # - name of the test case > # - template configuration (IPs, credentials, etc.) > # - ... > test-exec = './test-ldap-realms.pl' > # Consider test as failed if test script does not finish fast enough > test-timeout = 60 > # Additional params for the test script, allowing for parameterized > # tests. > # Could also turn this into an array and loop over the values, in > # order to create multiple test cases from the same definition. > test-params = { foo = "bar" } > > # Second test case, using the same fixture > [testcase.test-ldap-something-else] > fixture = 'pve-with-ldap-server' > test-exec = './test-ldap-something-else.pl' > > ``` > Is the order of test-cases guaranteed by toml parsing, or how are intra- fixture dependencies ensured? Anyway, the most important thing is to start out here, so I don't want to block anything on base on minorish stuff. The most important thing for me is that the following parts are decoupled and ideally shippable by a separate debian package each: - parts that manage automated testing, including how the test host base system is set up (the latter could be even its own thing) - running test itself inclusive some helper modules/scripts - the test definitions As then we can run them anywhere easily and extend, or possible even rework some parts independently, if ever needed. - Thomas