From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 31C0A93F3F for ; Thu, 8 Feb 2024 11:26:05 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 169611B975 for ; Thu, 8 Feb 2024 11:26:05 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 8 Feb 2024 11:26:03 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 4009F45193 for ; Thu, 8 Feb 2024 11:26:03 +0100 (CET) Date: Thu, 8 Feb 2024 11:26:01 +0100 From: Christoph Heiss To: Aaron Lauterer Cc: Proxmox VE development discussion Message-ID: References: <20240123170053.490250-1-a.lauterer@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240123170053.490250-1-a.lauterer@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.003 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pve-devel] [PATCH v1 installer/docs 00/18] add automated/unattended installation X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Feb 2024 10:26:05 -0000 Did some testing, doing various installs using * different filesystems * pre/post hooks * simple udev matching for disks and network interfaces I did not exercise the udev-matching exhaustively, but as it is pretty generic, that should cover it. The `proxmox-installer-filter` is a nice tooling addition as well! Maybe we could also provide some tool in the future which would do some (basic) sanity checking/validating on the `answer.toml`, to avoid people having to do test cycles on a real machine. As for the global `{pre,post}_commands` hooks - , So please consider this: Tested-by: Christoph Heiss Also, already left some review comments on the individual patches, but nothing too major. On Tue, Jan 23, 2024 at 06:00:35PM +0100, Aaron Lauterer wrote: > This patch series adds the possibility to do an automated / unattended > installation of Proxmox VE. > > It assumes that the patch series to use JSON output on the > low-level-installer is already applied [1]. > > The overall idea is that we will have a dedicated ISO for the unattended > installation. It should be configured in such a way that it will start > the installation without any user interaction. > Though the integration in the installation environmend isn't part of > this patch series. > > The information for the installer that is usually gathered interactively > from the user is provided via an `answer.toml` file. > > The answer file allows to select disks and the network card via filters. > > The installer also allows to run custom commands pre and post > installation. This should give users plenty of possibilities to either > further customize/prepare the installation or integrate it into a larger > automated installation setup. > For example, one could issue HTTP requests to signal the status and > progress of the installation. > > > The install environment needs to call the 'proxmox-fetch-answer' binary. > It tries to find the answer file and once found, will start the > 'proxmox-auto-installer' binary and pass the contents to it via stdin. > > The auto-installer then parses the answer file and determines what > parameters need to be passed to the low-level installer. For example, > which disks and NIC to use, network IP settings and so forth. > > The current status reporting of the actual installation is kept rather > simple. > > Both binaries log into the /tmp/ directory. > > There is a third binary, the 'proxmox-installer-filter'. It is meant as > a pure utility for users to make it easier to see what properties they > can write filters against and to test the filters. > > > The fetch-answer binary is currently searching for a > partition/file-system labeled 'proxmoxinst' in lower or uppercase. It > can be located on an additioan USB flash drive, or maybe on the install > medium itself if it is possible to write to it. > > > We do have some ideas for additional steps to fetch an answer file. The > main one is that we could download the answer file from a URL. Ideally > we would send unique properties along with the request (MAC addresses, > serial numbers, ...) so that it is possible to have a script on the > receiving side that can then generate the answer file dynamically. > > The big question is, where the URL comes from, for which we have also > some ideas: > * custom DHCP options > * kernel cmdline (might be an option with PXE boot) > * TXT DNS record in a predefined subdomain of the search domain received > via DHCP, basically a 'dig TXT proxmoxinst.{search domain}'. > * We should also make it possible to provide an SSL fingerprint in a > similar manner in case the listening server is not trusted out of the > box. > > Other plans / ideas for the future: > > * add option to define remote SSH access (password and,or public key). > This could make remote debugging in case of problems easier > > > Regarding the patch series itself: > The first patches are needed to move some code into the common crate and > make structs/functions already in the common crate accessible. > > I did split up the individual parts of the auto installer into their own > patches as much as possible, and (hopefully) in the order they depend on > each other. > > Areas that can be improved/extended: > * Testing possibility integrated in the Makefile > * Documentation: explain process, additional examples for answer.toml > > [0] https://lists.proxmox.com/pipermail/pve-devel/2023-September/059020.html > [1] https://lists.proxmox.com/pipermail/pve-devel/2023-December/060961.html > > installer: Aaron Lauterer (17): > tui: common: move InstallConfig struct to common crate > common: make InstallZfsOption members public > common: tui: use BTreeMap for predictable ordering > Makefile: fix handling of multiple usr_bin files > low-level: add dump-udev command > add auto-installer crate > auto-installer: add dependencies > auto-installer: add answer file definition > auto-installer: add struct to hold udev info > auto-installer: add utils > auto-installer: add simple logging > auto-installer: add tests for answer file parsing > auto-installer: add auto-installer binary > auto-installer: add fetch answer binary > auto-installer: use glob crate for pattern matching > auto-installer: utils: make get_udev_index functions public > auto-installer: add proxmox-installer-filter helper tool > > > docs: Aaron Lauterer (1): > installation: add unattended documentation > > pve-installation.adoc | 267 ++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 267 insertions(+) > > Cargo.toml | 1 + > Makefile | 9 +- > Proxmox/Makefile | 1 + > Proxmox/Sys/Udev.pm | 54 ++ > proxmox-auto-installer/Cargo.toml | 19 + > proxmox-auto-installer/src/answer.rs | 148 ++++++ > .../src/bin/proxmox-auto-installer.rs | 192 ++++++++ > .../src/bin/proxmox-fetch-answer.rs | 73 +++ > .../src/bin/proxmox-installer-filter.rs | 298 +++++++++++ > .../src/fetch_plugins/mod.rs | 1 + > .../src/fetch_plugins/partition.rs | 102 ++++ > proxmox-auto-installer/src/lib.rs | 5 + > proxmox-auto-installer/src/log.rs | 38 ++ > proxmox-auto-installer/src/udevinfo.rs | 9 + > proxmox-auto-installer/src/utils.rs | 461 ++++++++++++++++++ > proxmox-auto-installer/tests/parse-answer.rs | 102 ++++ > .../tests/resources/iso-info.json | 1 + > .../tests/resources/locales.json | 1 + > .../resources/parse_answer/disk_match.json | 29 ++ > .../resources/parse_answer/disk_match.toml | 14 + > .../parse_answer/disk_match_all.json | 26 + > .../parse_answer/disk_match_all.toml | 16 + > .../parse_answer/disk_match_any.json | 33 ++ > .../parse_answer/disk_match_any.toml | 16 + > .../tests/resources/parse_answer/minimal.json | 17 + > .../tests/resources/parse_answer/minimal.toml | 14 + > .../resources/parse_answer/nic_matching.json | 17 + > .../resources/parse_answer/nic_matching.toml | 19 + > .../tests/resources/parse_answer/readme | 4 + > .../resources/parse_answer/specific_nic.json | 17 + > .../resources/parse_answer/specific_nic.toml | 19 + > .../tests/resources/parse_answer/zfs.json | 27 + > .../tests/resources/parse_answer/zfs.toml | 19 + > .../tests/resources/run-env-info.json | 1 + > .../tests/resources/run-env-udev.json | 1 + > proxmox-installer-common/src/setup.rs | 100 +++- > proxmox-low-level-installer | 13 + > proxmox-tui-installer/src/options.rs | 4 +- > proxmox-tui-installer/src/setup.rs | 100 +--- > .../src/views/install_progress.rs | 4 +- > 40 files changed, 1915 insertions(+), 110 deletions(-) > create mode 100644 Proxmox/Sys/Udev.pm > create mode 100644 proxmox-auto-installer/Cargo.toml > create mode 100644 proxmox-auto-installer/src/answer.rs > create mode 100644 proxmox-auto-installer/src/bin/proxmox-auto-installer.rs > create mode 100644 proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs > create mode 100644 proxmox-auto-installer/src/bin/proxmox-installer-filter.rs > create mode 100644 proxmox-auto-installer/src/fetch_plugins/mod.rs > create mode 100644 proxmox-auto-installer/src/fetch_plugins/partition.rs > create mode 100644 proxmox-auto-installer/src/lib.rs > create mode 100644 proxmox-auto-installer/src/log.rs > create mode 100644 proxmox-auto-installer/src/udevinfo.rs > create mode 100644 proxmox-auto-installer/src/utils.rs > create mode 100644 proxmox-auto-installer/tests/parse-answer.rs > create mode 100644 proxmox-auto-installer/tests/resources/iso-info.json > create mode 100644 proxmox-auto-installer/tests/resources/locales.json > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match.json > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match.toml > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.json > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.toml > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.json > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.toml > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/minimal.json > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/minimal.toml > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/nic_matching.json > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/nic_matching.toml > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/readme > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/specific_nic.json > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/specific_nic.toml > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.json > create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.toml > create mode 100644 proxmox-auto-installer/tests/resources/run-env-info.json > create mode 100644 proxmox-auto-installer/tests/resources/run-env-udev.json > > -- > 2.39.2 > > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel >