From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 1B7899B1AE for ; Tue, 17 Oct 2023 09:35:29 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id ED68630708 for ; Tue, 17 Oct 2023 09:34:58 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 17 Oct 2023 09:34:58 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id C310842034 for ; Tue, 17 Oct 2023 09:34:57 +0200 (CEST) Message-ID: <347d64bd-486b-4078-b396-ff75dbd38996@proxmox.com> Date: Tue, 17 Oct 2023 09:34:56 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-GB, de-AT To: Proxmox VE development discussion , Lukas Wagner , Stefan Hanreich References: <1bb25817-66e7-406a-bd4b-0699de6cba31@proxmox.com> <44f89ce6-043f-7b05-75ad-ac66550eb3e8@proxmox.com> <9780afc5-6bf9-40da-8a2f-0c5e02ded605@proxmox.com> From: Thomas Lamprecht Autocrypt: addr=t.lamprecht@proxmox.com; keydata= xsFNBFsLjcYBEACsaQP6uTtw/xHTUCKF4VD4/Wfg7gGn47+OfCKJQAD+Oyb3HSBkjclopC5J uXsB1vVOfqVYE6PO8FlD2L5nxgT3SWkc6Ka634G/yGDU3ZC3C/7NcDVKhSBI5E0ww4Qj8s9w OQRloemb5LOBkJNEUshkWRTHHOmk6QqFB/qBPW2COpAx6oyxVUvBCgm/1S0dAZ9gfkvpqFSD 90B5j3bL6i9FIv3YGUCgz6Ue3f7u+HsEAew6TMtlt90XV3vT4M2IOuECG/pXwTy7NtmHaBQ7 UJBcwSOpDEweNob50+9B4KbnVn1ydx+K6UnEcGDvUWBkREccvuExvupYYYQ5dIhRFf3fkS4+ wMlyAFh8PQUgauod+vqs45FJaSgTqIALSBsEHKEs6IoTXtnnpbhu3p6XBin4hunwoBFiyYt6 YHLAM1yLfCyX510DFzX/Ze2hLqatqzY5Wa7NIXqYYelz7tXiuCLHP84+sV6JtEkeSUCuOiUY virj6nT/nJK8m0BzdR6FgGtNxp7RVXFRz/+mwijJVLpFsyG1i0Hmv2zTn3h2nyGK/I6yhFNt dX69y5hbo6LAsRjLUvZeHXpTU4TrpN/WiCjJblbj5um5eEr4yhcwhVmG102puTtuCECsDucZ jpKpUqzXlpLbzG/dp9dXFH3MivvfuaHrg3MtjXY1i+/Oxyp5iwARAQABzTNUaG9tYXMgTGFt cHJlY2h0IChBdXRoLTQpIDx0LmxhbXByZWNodEBwcm94bW94LmNvbT7CwY4EEwEIADgWIQQO R4qbEl/pah9K6VrTZCM6gDZWBgUCWwuNxgIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAK CRDTZCM6gDZWBm/jD/4+6JB2s67eaqoP6x9VGaXNGJPCscwzLuxDTCG90G9FYu29VcXtubH/ bPwsyBbNUQpqTm/s4XboU2qpS5ykCuTjqavrcP33tdkYfGcItj2xMipJ1i3TWvpikQVsX42R G64wovLs/dvpTYphRZkg5DwhgTmy3mRkmofFCTa+//MOcNOORltemp984tWjpR3bUJETNWpF sKGZHa3N4kCNxb7A+VMsJZ/1gN3jbQbQG7GkJtnHlWkw9rKCYqBtWrnrHa4UAvSa9M/XCIAB FThFGqZI1ojdVlv5gd6b/nWxfOPrLlSxbUo5FZ1i/ycj7/24nznW1V4ykG9iUld4uYUY86bB UGSjew1KYp9FmvKiwEoB+zxNnuEQfS7/Bj1X9nxizgweiHIyFsRqgogTvLh403QMSGNSoArk tqkorf1U+VhEncIn4H3KksJF0njZKfilrieOO7Vuot1xKr9QnYrZzJ7m7ZxJ/JfKGaRHXkE1 feMmrvZD1AtdUATZkoeQtTOpMu4r6IQRfSdwm/CkppZXfDe50DJxAMDWwfK2rr2bVkNg/yZI tKLBS0YgRTIynkvv0h8d9dIjiicw3RMeYXyqOnSWVva2r+tl+JBaenr8YTQw0zARrhC0mttu cIZGnVEvQuDwib57QLqMjQaC1gazKHvhA15H5MNxUhwm229UmdH3KM7BTQRbC43GARAAyTkR D6KRJ9Xa2fVMh+6f186q0M3ni+5tsaVhUiykxjsPgkuWXWW9MbLpYXkzX6h/RIEKlo2BGA95 QwG5+Ya2Bo3g7FGJHAkXY6loq7DgMp5/TVQ8phsSv3WxPTJLCBq6vNBamp5hda4cfXFUymsy HsJy4dtgkrPQ/bnsdFDCRUuhJHopnAzKHN8APXpKU6xV5e3GE4LwFsDhNHfH/m9+2yO/trcD txSFpyftbK2gaMERHgA8SKkzRhiwRTt9w5idOfpJVkYRsgvuSGZ0pcD4kLCOIFrer5xXudk6 NgJc36XkFRMnwqrL/bB4k6Pi2u5leyqcXSLyBgeHsZJxg6Lcr2LZ35+8RQGPOw9C0ItmRjtY ZpGKPlSxjxA1WHT2YlF9CEt3nx7c4C3thHHtqBra6BGPyW8rvtq4zRqZRLPmZ0kt/kiMPhTM 8wZAlObbATVrUMcZ/uNjRv2vU9O5aTAD9E5r1B0dlqKgxyoImUWB0JgpILADaT3VybDd3C8X s6Jt8MytUP+1cEWt9VKo4vY4Jh5vwrJUDLJvzpN+TsYCZPNVj18+jf9uGRaoK6W++DdMAr5l gQiwsNgf9372dbMI7pt2gnT5/YdG+ZHnIIlXC6OUonA1Ro/Itg90Q7iQySnKKkqqnWVc+qO9 GJbzcGykxD6EQtCSlurt3/5IXTA7t6sAEQEAAcLBdgQYAQgAIBYhBA5HipsSX+lqH0rpWtNk IzqANlYGBQJbC43GAhsMAAoJENNkIzqANlYGD1sP/ikKgHgcspEKqDED9gQrTBvipH85si0j /Jwu/tBtnYjLgKLh2cjv1JkgYYjb3DyZa1pLsIv6rGnPX9bH9IN03nqirC/Q1Y1lnbNTynPk IflgvsJjoTNZjgu1wUdQlBgL/JhUp1sIYID11jZphgzfDgp/E6ve/8xE2HMAnf4zAfJaKgD0 F+fL1DlcdYUditAiYEuN40Ns/abKs8I1MYx7Yglu3RzJfBzV4t86DAR+OvuF9v188WrFwXCS RSf4DmJ8tntyNej+DVGUnmKHupLQJO7uqCKB/1HLlMKc5G3GLoGqJliHjUHUAXNzinlpE2Vj C78pxpwxRNg2ilE3AhPoAXrY5qED5PLE9sLnmQ9AzRcMMJUXjTNEDxEYbF55SdGBHHOAcZtA kEQKub86e+GHA+Z8oXQSGeSGOkqHi7zfgW1UexddTvaRwE6AyZ6FxTApm8wq8NT2cryWPWTF BDSGB3ujWHMM8ERRYJPcBSjTvt0GcEqnd+OSGgxTkGOdufn51oz82zfpVo1t+J/FNz6MRMcg 8nEC+uKvgzH1nujxJ5pRCBOquFZaGn/p71Yr0oVitkttLKblFsqwa+10Lt6HBxm+2+VLp4Ja 0WZNncZciz3V3cuArpan/ZhhyiWYV5FD0pOXPCJIx7WS9PTtxiv0AOS4ScWEUmBxyhFeOpYa DrEx In-Reply-To: <9780afc5-6bf9-40da-8a2f-0c5e02ded605@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL -0.076 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [RFC] towards automated integration testing X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Oct 2023 07:35:29 -0000 Am 16/10/2023 um 17:18 schrieb Lukas Wagner: > On 10/16/23 13:20, Stefan Hanreich wrote: >> I can imagine having to setup VMs inside the Test Setup as well for >> doing various tests. Doing this manually every time could be quite >> cumbersome / hard to automate. Do you have a mechanism in mind to >> deploy VMs inside the test system as well? Again, PBS could be an >> interesting option for this imo. >> > Several options come to mind. We could use a virtualized PBS instance > with a datastore containing the VM backup as part of the fixture. We > could use some external backup store (so the same 'source' as for the > templates themselves) - however that means that the systems under test > must have network access to that. We could also think about using > iPXE to boot test VMs, with the boot image either be provided by some > template from the fixture, or by some external server. For both > approaches, the 'as part of the fixture' approaches seem a bit nicer, > as they are more self-contained. What about the following approach: The test state that they need one or more VMs with certain properties, i.e., something like "none" (don't care), "ostype=3Dwin*", "memory>=3D10G= " or the like (can start out easy w.r.t to supported comparison features, as long the base system is there it can be extended relatively easily later on). Then, on a run of a test first all those asset-dependencies are collected. Then they can be, depending on further config, get newly created or selected from existing candidates on the target test-host system. In general the test-system can add a specific tag (like "test-asset") to such virtual guests by default, and also add that as implicit property condition (if no explicit tag-condition is already present) for when searching for existing assets, this way one can either re-use guests, be it because they exist due to running on a bare-metal system, that won't get rolled back, or even in some virtual system that gets rolled back to a state that already has to virtual-guest test-assets configured and thus can also reduce the time required to set up a clean environment by a lot, benefiting both use cases. Extra config, and/or command line, knobs can then force re-creation of all, or some asses of, a test, or the base search path for images, here it's probably enough to have some simpler definitively wanted ones to provide the core-infra for how to add others, maybe more complex knobs in the future more easily (creating new things is IMO always harder than extending existing ones, at least if non-trivial). > Also, the vmbd2 thingy that thomas mentioned might be interesting for Because I stumbled upon it today, systemd's mkosi tool could be also interesting here: https://github.com/systemd/mkosi https://github.com/systemd/mkosi/blob/main/mkosi/resources/mkosi.md > this - i've only glanced at it so far though. >=20 > As of now it seems that this question will not influence the design of > the test runner much, so it can probably be postponed to a later > stage. Not of the runner itself, but all set up stuff for it, so I'd at least try to keep it in mind =E2=80=93 above features might not be that much wo= rk, but would create lots of flexibility to allow devs using it more easily for declarative reproduction tries of bugs too. At least I see it a big mental roadblock if I have to set up specific environments for using such tools, and cannot just re-use my existing ones 1:1. >=20 >>> In theory, the test runner would also be able to drive tests on real >>> hardware, but of course with some limitations (harder to have a >>> predictable, reproducible environment, etc.) >> >> Maybe utilizing Aaron's installer for setting up those test systems >> could at least produce somewhat identical setups? Although it is >> really hard managing systems with different storage types, network >> cards, ... . >=20 > In general my biggest concern with 'bare-metal' tests - and to > precise, that does not really have anything to do with being > 'bare-metal', more about testing on something that is harder roll back > into a clean state that can be used for the next test execution, is > that I'm afraid that a setup like this could become quite brittle and > a maintenance burden I don't see that as issue, just as two separate thing, one is regression testing in clean states where we can turn up reporting of test-failures to the max and the other is integration testing where we don't report widely but only allow some way to see list of issues for admins to decide. Bugs in the test system or configuration issue breaking idempotency assumptions can then be fixed, other issues that are not visible in those clean-room tests can become visible, I see no reason why both cannot co-exist and have equivalent priority/focus. New tests can be checked for basic idempotency by running them twice, with the second run not doing any rollback. >> I've seen GitLab using tags for runners that specify certain >> capabilities of systems. Maybe we could also introduce something like >> that here for different bare-metal systems? E.g. a test case >> specifies it needs a system with tag `ZFS` and then you can run / >> skip the respective test case on that system. Managing those tags can >> introduce quite a lot of churn though, so I'm not sure if this would >> be a good idea. >=20 > I have thought about a tag system as well - not necessarily for test > runners, but for test cases. E.g. you could tag tests for the > authentication system with 'auth' - because at least for the local > development cycle it might not make much sense to run tests for > clusters, ceph, etc. while working on the authentication system. Yes, I thought about something like that too, a known set of tags (i.e., centrally managed set and bail, or at least warn if test uses unknown one) =E2=80=93 having test runs be filtered by their use classes, like "migration" or "windows" or your "auth" example would be definitively nice. >>> The test script is executed by the test runner; the test outcome is >>> determined by the exit code of the script. Test scripts could be >>> written >> Are you considering capturing output as well? That would make sense >> when using assertions at least, so in case of failures developers >> have a starting point for debugging. > Yup, I'd capture stdout/stderr from all test executables/scripts and > include it in the final test report. I guess there would be a (optional) notification to a set of addresses, passed to the test system via CLI/Config by the tester (human on manual tests or derived from changes and maintainers for automated tests), and that would only have a summary and link/point to the full report that provides the longer outputs of test harness and possibly system logs. > Test output is indeed very useful when determining *why* something > went wrong. Journalctl of all nodes that took part of a test might be useful too. >> Would it make sense to allow specifying a expected exit code for >> tests that actually should fail - or do you consider this something >> that should be handled by the test script? >=20 > I guess that's a matter of taste. Personally I'd keep the contract > between test runner and test script simple and say 0 =3D=3D success, > everything else is a failure. If there are any test cases that expect > a failure of some API call, then the script should 'translate' the > exit code. W.r.t. exit code I find that fine, but maybe we want to allow passing a more formal result text back, but we always can extend this by just using some special files that the test script writes to, or something like that, in the future, here starting out with simply checking exit code seems fine enough to me.