From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 643E5C2689 for ; Tue, 23 Jan 2024 18:00:58 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3C8A2AB5B for ; Tue, 23 Jan 2024 18:00:58 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 23 Jan 2024 18:00:56 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id BC1F54926B for ; Tue, 23 Jan 2024 18:00:56 +0100 (CET) From: Aaron Lauterer To: pve-devel@lists.proxmox.com Date: Tue, 23 Jan 2024 18:00:35 +0100 Message-Id: <20240123170053.490250-1-a.lauterer@proxmox.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.068 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pve-devel] [PATCH v1 installer/docs 00/18] add automated/unattended installation X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jan 2024 17:00:58 -0000 This patch series adds the possibility to do an automated / unattended installation of Proxmox VE. It assumes that the patch series to use JSON output on the low-level-installer is already applied [1]. The overall idea is that we will have a dedicated ISO for the unattended installation. It should be configured in such a way that it will start the installation without any user interaction. Though the integration in the installation environmend isn't part of this patch series. The information for the installer that is usually gathered interactively from the user is provided via an `answer.toml` file. The answer file allows to select disks and the network card via filters. The installer also allows to run custom commands pre and post installation. This should give users plenty of possibilities to either further customize/prepare the installation or integrate it into a larger automated installation setup. For example, one could issue HTTP requests to signal the status and progress of the installation. The install environment needs to call the 'proxmox-fetch-answer' binary. It tries to find the answer file and once found, will start the 'proxmox-auto-installer' binary and pass the contents to it via stdin. The auto-installer then parses the answer file and determines what parameters need to be passed to the low-level installer. For example, which disks and NIC to use, network IP settings and so forth. The current status reporting of the actual installation is kept rather simple. Both binaries log into the /tmp/ directory. There is a third binary, the 'proxmox-installer-filter'. It is meant as a pure utility for users to make it easier to see what properties they can write filters against and to test the filters. The fetch-answer binary is currently searching for a partition/file-system labeled 'proxmoxinst' in lower or uppercase. It can be located on an additioan USB flash drive, or maybe on the install medium itself if it is possible to write to it. We do have some ideas for additional steps to fetch an answer file. The main one is that we could download the answer file from a URL. Ideally we would send unique properties along with the request (MAC addresses, serial numbers, ...) so that it is possible to have a script on the receiving side that can then generate the answer file dynamically. The big question is, where the URL comes from, for which we have also some ideas: * custom DHCP options * kernel cmdline (might be an option with PXE boot) * TXT DNS record in a predefined subdomain of the search domain received via DHCP, basically a 'dig TXT proxmoxinst.{search domain}'. * We should also make it possible to provide an SSL fingerprint in a similar manner in case the listening server is not trusted out of the box. Other plans / ideas for the future: * add option to define remote SSH access (password and,or public key). This could make remote debugging in case of problems easier Regarding the patch series itself: The first patches are needed to move some code into the common crate and make structs/functions already in the common crate accessible. I did split up the individual parts of the auto installer into their own patches as much as possible, and (hopefully) in the order they depend on each other. Areas that can be improved/extended: * Testing possibility integrated in the Makefile * Documentation: explain process, additional examples for answer.toml [0] https://lists.proxmox.com/pipermail/pve-devel/2023-September/059020.html [1] https://lists.proxmox.com/pipermail/pve-devel/2023-December/060961.html installer: Aaron Lauterer (17): tui: common: move InstallConfig struct to common crate common: make InstallZfsOption members public common: tui: use BTreeMap for predictable ordering Makefile: fix handling of multiple usr_bin files low-level: add dump-udev command add auto-installer crate auto-installer: add dependencies auto-installer: add answer file definition auto-installer: add struct to hold udev info auto-installer: add utils auto-installer: add simple logging auto-installer: add tests for answer file parsing auto-installer: add auto-installer binary auto-installer: add fetch answer binary auto-installer: use glob crate for pattern matching auto-installer: utils: make get_udev_index functions public auto-installer: add proxmox-installer-filter helper tool docs: Aaron Lauterer (1): installation: add unattended documentation pve-installation.adoc | 267 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 267 insertions(+) Cargo.toml | 1 + Makefile | 9 +- Proxmox/Makefile | 1 + Proxmox/Sys/Udev.pm | 54 ++ proxmox-auto-installer/Cargo.toml | 19 + proxmox-auto-installer/src/answer.rs | 148 ++++++ .../src/bin/proxmox-auto-installer.rs | 192 ++++++++ .../src/bin/proxmox-fetch-answer.rs | 73 +++ .../src/bin/proxmox-installer-filter.rs | 298 +++++++++++ .../src/fetch_plugins/mod.rs | 1 + .../src/fetch_plugins/partition.rs | 102 ++++ proxmox-auto-installer/src/lib.rs | 5 + proxmox-auto-installer/src/log.rs | 38 ++ proxmox-auto-installer/src/udevinfo.rs | 9 + proxmox-auto-installer/src/utils.rs | 461 ++++++++++++++++++ proxmox-auto-installer/tests/parse-answer.rs | 102 ++++ .../tests/resources/iso-info.json | 1 + .../tests/resources/locales.json | 1 + .../resources/parse_answer/disk_match.json | 29 ++ .../resources/parse_answer/disk_match.toml | 14 + .../parse_answer/disk_match_all.json | 26 + .../parse_answer/disk_match_all.toml | 16 + .../parse_answer/disk_match_any.json | 33 ++ .../parse_answer/disk_match_any.toml | 16 + .../tests/resources/parse_answer/minimal.json | 17 + .../tests/resources/parse_answer/minimal.toml | 14 + .../resources/parse_answer/nic_matching.json | 17 + .../resources/parse_answer/nic_matching.toml | 19 + .../tests/resources/parse_answer/readme | 4 + .../resources/parse_answer/specific_nic.json | 17 + .../resources/parse_answer/specific_nic.toml | 19 + .../tests/resources/parse_answer/zfs.json | 27 + .../tests/resources/parse_answer/zfs.toml | 19 + .../tests/resources/run-env-info.json | 1 + .../tests/resources/run-env-udev.json | 1 + proxmox-installer-common/src/setup.rs | 100 +++- proxmox-low-level-installer | 13 + proxmox-tui-installer/src/options.rs | 4 +- proxmox-tui-installer/src/setup.rs | 100 +--- .../src/views/install_progress.rs | 4 +- 40 files changed, 1915 insertions(+), 110 deletions(-) create mode 100644 Proxmox/Sys/Udev.pm create mode 100644 proxmox-auto-installer/Cargo.toml create mode 100644 proxmox-auto-installer/src/answer.rs create mode 100644 proxmox-auto-installer/src/bin/proxmox-auto-installer.rs create mode 100644 proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs create mode 100644 proxmox-auto-installer/src/bin/proxmox-installer-filter.rs create mode 100644 proxmox-auto-installer/src/fetch_plugins/mod.rs create mode 100644 proxmox-auto-installer/src/fetch_plugins/partition.rs create mode 100644 proxmox-auto-installer/src/lib.rs create mode 100644 proxmox-auto-installer/src/log.rs create mode 100644 proxmox-auto-installer/src/udevinfo.rs create mode 100644 proxmox-auto-installer/src/utils.rs create mode 100644 proxmox-auto-installer/tests/parse-answer.rs create mode 100644 proxmox-auto-installer/tests/resources/iso-info.json create mode 100644 proxmox-auto-installer/tests/resources/locales.json create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match.json create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match.toml create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.json create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.toml create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.json create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.toml create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/minimal.json create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/minimal.toml create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/nic_matching.json create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/nic_matching.toml create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/readme create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/specific_nic.json create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/specific_nic.toml create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.json create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.toml create mode 100644 proxmox-auto-installer/tests/resources/run-env-info.json create mode 100644 proxmox-auto-installer/tests/resources/run-env-udev.json -- 2.39.2