From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id DACB49B479 for ; Tue, 17 Oct 2023 18:28:24 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id BB3A036946 for ; Tue, 17 Oct 2023 18:28:24 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 17 Oct 2023 18:28:23 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 8134B42375 for ; Tue, 17 Oct 2023 18:28:23 +0200 (CEST) Message-ID: <57929886-9ef9-45b9-94c4-f0f66dc2532a@proxmox.com> Date: Tue, 17 Oct 2023 18:28:22 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Lukas Wagner , Proxmox VE development discussion References: <1bb25817-66e7-406a-bd4b-0699de6cba31@proxmox.com> <832bc039-57d0-4d27-ac48-721c5c82af83@proxmox.com> <125370ed-9b55-42d2-b544-28683a17be79@proxmox.com> <9406389e-da89-475e-b8ad-e37efb72df10@proxmox.com> Content-Language: en-GB, de-AT From: Thomas Lamprecht Autocrypt: addr=t.lamprecht@proxmox.com; keydata= xsFNBFsLjcYBEACsaQP6uTtw/xHTUCKF4VD4/Wfg7gGn47+OfCKJQAD+Oyb3HSBkjclopC5J uXsB1vVOfqVYE6PO8FlD2L5nxgT3SWkc6Ka634G/yGDU3ZC3C/7NcDVKhSBI5E0ww4Qj8s9w OQRloemb5LOBkJNEUshkWRTHHOmk6QqFB/qBPW2COpAx6oyxVUvBCgm/1S0dAZ9gfkvpqFSD 90B5j3bL6i9FIv3YGUCgz6Ue3f7u+HsEAew6TMtlt90XV3vT4M2IOuECG/pXwTy7NtmHaBQ7 UJBcwSOpDEweNob50+9B4KbnVn1ydx+K6UnEcGDvUWBkREccvuExvupYYYQ5dIhRFf3fkS4+ wMlyAFh8PQUgauod+vqs45FJaSgTqIALSBsEHKEs6IoTXtnnpbhu3p6XBin4hunwoBFiyYt6 YHLAM1yLfCyX510DFzX/Ze2hLqatqzY5Wa7NIXqYYelz7tXiuCLHP84+sV6JtEkeSUCuOiUY virj6nT/nJK8m0BzdR6FgGtNxp7RVXFRz/+mwijJVLpFsyG1i0Hmv2zTn3h2nyGK/I6yhFNt dX69y5hbo6LAsRjLUvZeHXpTU4TrpN/WiCjJblbj5um5eEr4yhcwhVmG102puTtuCECsDucZ jpKpUqzXlpLbzG/dp9dXFH3MivvfuaHrg3MtjXY1i+/Oxyp5iwARAQABzTNUaG9tYXMgTGFt cHJlY2h0IChBdXRoLTQpIDx0LmxhbXByZWNodEBwcm94bW94LmNvbT7CwY4EEwEIADgWIQQO R4qbEl/pah9K6VrTZCM6gDZWBgUCWwuNxgIbAwULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAK CRDTZCM6gDZWBm/jD/4+6JB2s67eaqoP6x9VGaXNGJPCscwzLuxDTCG90G9FYu29VcXtubH/ bPwsyBbNUQpqTm/s4XboU2qpS5ykCuTjqavrcP33tdkYfGcItj2xMipJ1i3TWvpikQVsX42R G64wovLs/dvpTYphRZkg5DwhgTmy3mRkmofFCTa+//MOcNOORltemp984tWjpR3bUJETNWpF sKGZHa3N4kCNxb7A+VMsJZ/1gN3jbQbQG7GkJtnHlWkw9rKCYqBtWrnrHa4UAvSa9M/XCIAB FThFGqZI1ojdVlv5gd6b/nWxfOPrLlSxbUo5FZ1i/ycj7/24nznW1V4ykG9iUld4uYUY86bB UGSjew1KYp9FmvKiwEoB+zxNnuEQfS7/Bj1X9nxizgweiHIyFsRqgogTvLh403QMSGNSoArk tqkorf1U+VhEncIn4H3KksJF0njZKfilrieOO7Vuot1xKr9QnYrZzJ7m7ZxJ/JfKGaRHXkE1 feMmrvZD1AtdUATZkoeQtTOpMu4r6IQRfSdwm/CkppZXfDe50DJxAMDWwfK2rr2bVkNg/yZI tKLBS0YgRTIynkvv0h8d9dIjiicw3RMeYXyqOnSWVva2r+tl+JBaenr8YTQw0zARrhC0mttu cIZGnVEvQuDwib57QLqMjQaC1gazKHvhA15H5MNxUhwm229UmdH3KM7BTQRbC43GARAAyTkR D6KRJ9Xa2fVMh+6f186q0M3ni+5tsaVhUiykxjsPgkuWXWW9MbLpYXkzX6h/RIEKlo2BGA95 QwG5+Ya2Bo3g7FGJHAkXY6loq7DgMp5/TVQ8phsSv3WxPTJLCBq6vNBamp5hda4cfXFUymsy HsJy4dtgkrPQ/bnsdFDCRUuhJHopnAzKHN8APXpKU6xV5e3GE4LwFsDhNHfH/m9+2yO/trcD txSFpyftbK2gaMERHgA8SKkzRhiwRTt9w5idOfpJVkYRsgvuSGZ0pcD4kLCOIFrer5xXudk6 NgJc36XkFRMnwqrL/bB4k6Pi2u5leyqcXSLyBgeHsZJxg6Lcr2LZ35+8RQGPOw9C0ItmRjtY ZpGKPlSxjxA1WHT2YlF9CEt3nx7c4C3thHHtqBra6BGPyW8rvtq4zRqZRLPmZ0kt/kiMPhTM 8wZAlObbATVrUMcZ/uNjRv2vU9O5aTAD9E5r1B0dlqKgxyoImUWB0JgpILADaT3VybDd3C8X s6Jt8MytUP+1cEWt9VKo4vY4Jh5vwrJUDLJvzpN+TsYCZPNVj18+jf9uGRaoK6W++DdMAr5l gQiwsNgf9372dbMI7pt2gnT5/YdG+ZHnIIlXC6OUonA1Ro/Itg90Q7iQySnKKkqqnWVc+qO9 GJbzcGykxD6EQtCSlurt3/5IXTA7t6sAEQEAAcLBdgQYAQgAIBYhBA5HipsSX+lqH0rpWtNk IzqANlYGBQJbC43GAhsMAAoJENNkIzqANlYGD1sP/ikKgHgcspEKqDED9gQrTBvipH85si0j /Jwu/tBtnYjLgKLh2cjv1JkgYYjb3DyZa1pLsIv6rGnPX9bH9IN03nqirC/Q1Y1lnbNTynPk IflgvsJjoTNZjgu1wUdQlBgL/JhUp1sIYID11jZphgzfDgp/E6ve/8xE2HMAnf4zAfJaKgD0 F+fL1DlcdYUditAiYEuN40Ns/abKs8I1MYx7Yglu3RzJfBzV4t86DAR+OvuF9v188WrFwXCS RSf4DmJ8tntyNej+DVGUnmKHupLQJO7uqCKB/1HLlMKc5G3GLoGqJliHjUHUAXNzinlpE2Vj C78pxpwxRNg2ilE3AhPoAXrY5qED5PLE9sLnmQ9AzRcMMJUXjTNEDxEYbF55SdGBHHOAcZtA kEQKub86e+GHA+Z8oXQSGeSGOkqHi7zfgW1UexddTvaRwE6AyZ6FxTApm8wq8NT2cryWPWTF BDSGB3ujWHMM8ERRYJPcBSjTvt0GcEqnd+OSGgxTkGOdufn51oz82zfpVo1t+J/FNz6MRMcg 8nEC+uKvgzH1nujxJ5pRCBOquFZaGn/p71Yr0oVitkttLKblFsqwa+10Lt6HBxm+2+VLp4Ja 0WZNncZciz3V3cuArpan/ZhhyiWYV5FD0pOXPCJIx7WS9PTtxiv0AOS4ScWEUmBxyhFeOpYa DrEx In-Reply-To: <9406389e-da89-475e-b8ad-e37efb72df10@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL -0.076 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [RFC] towards automated integration testing X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Oct 2023 16:28:24 -0000 Am 17/10/2023 um 14:33 schrieb Lukas Wagner: > On 10/17/23 08:35, Thomas Lamprecht wrote: >> =C2=A0From top of my head I'd rather do some attribute based dependenc= y >> annotation, so that one can depend on single tests, or whole fixture >> on others single tests or whole fixture. >> >=20 > The more thought I spend on it, the more I believe that inter-testcase > deps should be avoided as much as possible. In unit testing, (hidden) We don't plan unit testing here though and the dependencies I proposed are the contrary from hidden, rather explicit annotated ones. > dependencies between tests are in my experience the no. 1 cause of > flaky tests, and I see no reason why this would not also apply for > end-to-end integration testing. Any source on that being the no 1 source of flaky tests? IMO that should not make any difference, in the end you just allow better reuse through composition of other tests (e.g., migration builds upon clustering *set up*, not tests, if I just want to run migration I can do clustering setup without executing its tests). Not providing that could also mean that one has to move all logic in the test-script, resulting in a single test per "fixture", reducing granularity and parallelity of some running tests. I also think that=20 > I'd suggest to only allow test cases to depend on fixtures. The fixture= s > themselves could have setup/teardown hooks that allow setting up and > cleaning up a test scenario. If needed, we could also have something > like 'fixture inheritance', where a fixture can 'extend' another, > supplying additional setup/teardown. > Example: the 'outermost' or 'parent' fixture might define that we > want a 'basic PVE installation' with the latest .debs deployed, > while another fixture that inherits from that one might set up a > storage of a certain type, useful for all tests that require specific > that type of storage. Maybe our disagreement stems mostly from different design pictures in our head, I probably am a bit less fixed (heh) on the fixtures, or at least the naming of that term and might use test system, or intra test system when for your design plan fixture would be the better word. > On the other hand, instead of inheritance, a 'role/trait'-based system > might also work (composition >>> inheritance, after all) - and > maybe that also aligns better with the 'properties' mentioned in > your other mail (I mean this here:=C2=A0 "ostype=3Dwin*", "memory>=3D10= G"). >=20 > This is essentially a very similar pattern as in numerous other testing= > frameworks (xUnit, pytest, etc.); I think it makes sense to > build upon this battle-proven approach. Those are all unit testing tools though that we do already in the sources and IIRC those do not really provide what we need here. While starting out simple(r) and avoiding too much complexity has certainly it's merits, I don't think we should try to draw/align too many parallels with those tools here for us. >=20 > Regarding execution order, I'd now even suggest the polar opposite of m= y > prior idea. Instead of enforcing some execution order, we could also > actively shuffle execution order from run to run, at least for tests > using the same fixture. > The seed used for the RNG should be put into the test > report and could also be provided via a flag to the test runner, in cas= e > we need to repeat a specific test sequence . Hmm, this also has a chance to make tests flaky and get a bit annoying, like perl's hash scrambling, but not a bad idea, I'd just not do that by default on the "armed" test system that builds on package/git/patch updat= es, but possibly in addition with reporting turned off like the double tests for idempotency-checking I wrote in my previous message. > In that way, the runner would actively help us to hunt down > hidden inter-TC deps, making our test suite hopefully less brittle and > more robust in the long term. Agree, but as mentioned above I'd not enable it by default on the dev facing automated systems, but possibly for manual runs from devs and a separate "test-test-system" ^^ In summary, the most important points for me is a decoupled test-system from the automation system that can manage it, ideally such that I can decide relatively flexible on manual runs, IMO that should not be to much= work and it guarantees for clean cut APIs from which future development, or integration surely will benefit too. The rest is possibly hard to determine clearly on this stage, as it's eas= y (at least for me) to get lost in different understandings of terms and design perception, but hard to convey those very clearly about "pipe drea= ms", so at this stage I'll cede to add discussion churn until there's somethin= g more concrete that I can grasp on my terms (through reading/writing code)= , but that should not deter others from giving input still while at this st= age. Thanks for your work on this. - Thomas