From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 6E82C9B2E2 for ; Tue, 17 Oct 2023 14:33:22 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 449F132FD4 for ; Tue, 17 Oct 2023 14:33:22 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 17 Oct 2023 14:33:21 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 67DEA417F4 for ; Tue, 17 Oct 2023 14:33:21 +0200 (CEST) Message-ID: <9406389e-da89-475e-b8ad-e37efb72df10@proxmox.com> Date: Tue, 17 Oct 2023 14:33:20 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: de-AT, en-US To: Thomas Lamprecht , Proxmox VE development discussion References: <1bb25817-66e7-406a-bd4b-0699de6cba31@proxmox.com> <832bc039-57d0-4d27-ac48-721c5c82af83@proxmox.com> <125370ed-9b55-42d2-b544-28683a17be79@proxmox.com> From: Lukas Wagner In-Reply-To: <125370ed-9b55-42d2-b544-28683a17be79@proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.026 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [RFC] towards automated integration testing X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Oct 2023 12:33:22 -0000 On 10/17/23 08:35, Thomas Lamprecht wrote: >>> Is the order of test-cases guaranteed by toml parsing, or how are intra- >>> fixture dependencies ensured? >>> >> >> Good point. With rollbacks in between test cases it probably does not >> matter much, but on 'real hardware' with no rollback this could >> definitely be a concern. >> A super simple thing that could just work fine is ordering test >> execution by testcase-names, sorted alphabetically. Ideally you'd write >> test cases that do not depend on each other any way, and *if* you ever >> find yourself in the situation where you *need* some ordering, you >> could> just encode the order in the test-case name by adding an integer >> prefix> - similar how you would name config files in /etc/sysctl.d/*, >> for instance. > > > While it can be OK to leave that for later, encoding such things > in names is IMO brittle and hard to manage if more than a handful > of tests, and we hopefully got lots more ;-) > > > From top of my head I'd rather do some attribute based dependency > annotation, so that one can depend on single tests, or whole fixture > on others single tests or whole fixture. > The more thought I spend on it, the more I believe that inter-testcase deps should be avoided as much as possible. In unit testing, (hidden) dependencies between tests are in my experience the no. 1 cause of flaky tests, and I see no reason why this would not also apply for end-to-end integration testing. I'd suggest to only allow test cases to depend on fixtures. The fixtures themselves could have setup/teardown hooks that allow setting up and cleaning up a test scenario. If needed, we could also have something like 'fixture inheritance', where a fixture can 'extend' another, supplying additional setup/teardown. Example: the 'outermost' or 'parent' fixture might define that we want a 'basic PVE installation' with the latest .debs deployed, while another fixture that inherits from that one might set up a storage of a certain type, useful for all tests that require specific that type of storage. On the other hand, instead of inheritance, a 'role/trait'-based system might also work (composition >>> inheritance, after all) - and maybe that also aligns better with the 'properties' mentioned in your other mail (I mean this here: "ostype=win*", "memory>=10G"). This is essentially a very similar pattern as in numerous other testing frameworks (xUnit, pytest, etc.); I think it makes sense to build upon this battle-proven approach. Regarding execution order, I'd now even suggest the polar opposite of my prior idea. Instead of enforcing some execution order, we could also actively shuffle execution order from run to run, at least for tests using the same fixture. The seed used for the RNG should be put into the test report and could also be provided via a flag to the test runner, in case we need to repeat a specific test sequence . In that way, the runner would actively help us to hunt down hidden inter-TC deps, making our test suite hopefully less brittle and more robust in the long term. Any way, lots of details to figure out. Thanks again for your input. -- - Lukas