Re: [pve-devel] [PATCH installer v5 00/36] add automated/unattended installation

[Aaron Lauterer <a.lauterer@proxmox.com>]

a new v6 has been posted that includes the t-b and r-b tags as well as 
some smaller style fixes in the most recent patches

https://lists.proxmox.com/pipermail/pve-devel/2024-April/063139.html

On  2024-04-16  17:32, Aaron Lauterer wrote:
> patches until 31 got a [0,1]
> 
> Tested-by: Christoph Heiss <c.heiss@proxmox.com>
> Reviewed-by: Christoph Heiss <c.heiss@proxmox.com>
> 
> changes since v4:
> 
> Patches 32-36 finalize how to prepare an ISO for automated installation
> and introduce a slight change in behavior, as it is now also possible to
> include the needed parameters into the ISO directly:
> 
> * answer file itself
> * URL to fetch it from
> * SSL cert fingerprint
> 
> The 'proxmox-autoinst-helper' tool got a new subcommand to prepare an
> ISO.
> 
> The cover letter iself:
> 
> The overall idea is to prepare an ISO for automated installation. A
> prepare ISO will automatically boot into the installation.
> 
> The information for the installer that is usually gathered interactively
> from the user is provided via an `answer.toml` file.
> 
> The answer file allows to select disks and the network card via filters.
> 
> The installer also allows to run custom commands pre and post
> installation. This should give users plenty of possibilities to either
> further customize/prepare the installation or integrate it into a larger
> automated installation setup.
> For example, one could issue HTTP requests to signal the status and
> progress of the installation.
> 
> When the installer is called with 'proxauto' in the kernel cmdline, the
> 'proxmox-fetch-answer' binary is called. It tries to find the answer
> file and once found, will start the 'proxmox-auto-installer' binary and
> pass the contents to it via stdin.
> 
> The auto-installer then parses the answer file and determines what
> parameters need to be passed to the low-level installer. For example,
> which disks and NIC to use, network IP settings and so forth.
> 
> The current status reporting of the actual installation is kept rather
> simple.
> 
> Both binaries log into the tmp directory.
> 
> There is a third binary, the 'proxmox-autoinst-helper'. It provides a
> few subcommands, from the help:
>    prepare-iso      Prepare an ISO for automated installation.
>    validate-answer  Validate if an answer file is formatted correctly
>    device-match     Test which devices the given filter matches against
>    device-info      Show device information that can be used for filters
>    identifiers      Show identifiers for the current machine. This information is part of the POST request to fetch an answer file
>    help             Print this message or the help of the given subcommand(s)
> 
> The fetch-answer binary is trying to get an answer file. If included in
> the ISO, it will use that one. If no answer file is included, it first
> searches for a partition/FS labeled `proxmoxinst`, or all upper case,
> and an `answer.toml` in there. This could be provided by another USB
> flash drive.
> If that is not successful, the next step is to send an HTTP POST request
> to a URL to get the TOML contents in return. A POST request was chosen
> because we also send information to identify the host in JSON format.
> 
> The question then is, where to get that URL from. Right now, there are
> three options implemented. The first is to hardcode it in the prepared
> iso. The second is to look for a custom DHCP option
> and the third is to query for a TXT record in the `proxmoxinst`
> subdomain of the search domain.
> 
> It is possible to provide a SHA256 fingerprint of the SSL cert used by
> the answer server. The safest option is to include it in the ISO itself.
> If that is not found, then it can be provided by a second custom DHCP
> option or placed as TXT record in the subdomain `proxmoxinst-fp`.
> 
> This patch series now also separates the 3 binaries into their own
> crate. The 'proxmox-fetch-answer' to keep the OpenSSL dependency as
> localized as possible, and the 'proxmox-autoinst-helper' to make it easy
> to compile just that binary.
> 
> The new `proxmox-chroot` utility helps to prepare everything to chroot
> into a fresh installation and clean it up once done.
> This will be useful in the post commands when further customizing the
> installation.
> 
> 
> Other plans / ideas for the future:
> 
> * add option to define remote SSH access (password and,or public key).
>    This could make remote debugging in case of problems easier
> 
> 
> Regarding the patch series itself:
> 01-03 are needed to move some code into the common crate and
> make structs/functions already in the common crate accessible.
> 
> I did split up the individual parts of the auto installer into their own
> patches as much as possible, and (hopefully) in the order they depend on
> each other.
> 
> Patches after the `unconfigured` one (16), switch the pattern matching
> to the glob crate, add the helper tool and the fetching via HTTP.
> 
> Patch 26 factors our the binaries into their own crates.
> 
> Patches 27-30 are for the 'proxmox-chroot' utility and preparations for
> it to work.
> 
> Patch 31 makes sure that the answer file can only contain known keys.
> 
> Patches 32 - 36 finalize the ISO preparation, add the subcommand for it
> to the autoinst-helper tool and adapt the fetch-answer binary to also
> handle the new options to include the infos in the ISO itself.
> 
> Areas that can be improved/extended:
> * Testing possibility integrated in the Makefile
> * make build target for statically compiled proxmox-autoinst-helper
> 
> I did test it with all 3 installers, PVE, PMG and PBS and it worked.
> 
> WIP: Documentation. A first draft is available in the inernal wiki, as
> we will most likely keep it in wiki format since it applies for all 3
> products, if we provide ISOs for it.
> 
> [0] https://lists.proxmox.com/pipermail/pve-devel/2024-April/062634.html
> [1] https://lists.proxmox.com/pipermail/pve-devel/2024-April/062690.html
> 
> Aaron Lauterer (36):
>    tui: common: move InstallConfig struct to common crate
>    common: make InstallZfsOption members public
>    common: tui: use BTreeMap for predictable ordering
>    common: utils: add deserializer for CidrAddress
>    common: options: add Deserialize trait
>    low-level: add dump-udev command
>    add auto-installer crate
>    auto-installer: add dependencies
>    auto-installer: add answer file definition
>    auto-installer: add struct to hold udev info
>    auto-installer: add utils
>    auto-installer: add simple logging
>    auto-installer: add tests for answer file parsing
>    auto-installer: add auto-installer binary
>    auto-installer: add fetch answer binary
>    unconfigured: add proxauto as option to start auto installer
>    auto-installer: use glob crate for pattern matching
>    auto-installer: utils: make get_udev_index functions public
>    auto-installer: add proxmox-autoinst-helper tool
>    common: add Display trait to ProxmoxProduct
>    auto-installer: fetch: add gathering of system identifiers and
>      restructure code
>    auto-installer: helper: add subcommand to view indentifiers
>    auto-installer: fetch: add http post utility module
>    auto-installer: fetch: add http plugin to fetch answer
>    control: update build depends for auto installer
>    auto installer: factor out fetch-answer and autoinst-helper
>    low-level: write low level config to /tmp
>    common: add deserializer for FsType
>    common: skip target_hd when deserializing InstallConfig
>    add proxmox-chroot utility
>    auto-installer: answer: deny unknown fields
>    fetch-answer: move get_answer_file to utils
>    auto-installer: utils: define ISO specified settings
>    fetch-answer: use ISO specified configurations
>    fetch-answer: dpcp: improve logging of steps taken
>    autoinst-helper: add prepare-iso subcommand
> 
>   Cargo.toml                                    |   4 +
>   Makefile                                      |  19 +-
>   Proxmox/Makefile                              |   1 +
>   Proxmox/Sys/Udev.pm                           |  54 ++
>   debian/control                                |  10 +
>   proxmox-auto-installer/Cargo.toml             |  20 +
>   proxmox-auto-installer/src/answer.rs          | 255 ++++++++
>   .../src/bin/proxmox-auto-installer.rs         | 195 ++++++
>   proxmox-auto-installer/src/lib.rs             |   5 +
>   proxmox-auto-installer/src/log.rs             |  38 ++
>   proxmox-auto-installer/src/sysinfo.rs         |  81 +++
>   proxmox-auto-installer/src/udevinfo.rs        |   9 +
>   proxmox-auto-installer/src/utils.rs           | 455 ++++++++++++++
>   proxmox-auto-installer/tests/parse-answer.rs  | 106 ++++
>   .../tests/resources/iso-info.json             |   1 +
>   .../tests/resources/locales.json              |   1 +
>   .../resources/parse_answer/disk_match.json    |  29 +
>   .../resources/parse_answer/disk_match.toml    |  17 +
>   .../parse_answer/disk_match_all.json          |  26 +
>   .../parse_answer/disk_match_all.toml          |  17 +
>   .../parse_answer/disk_match_any.json          |  33 ++
>   .../parse_answer/disk_match_any.toml          |  17 +
>   .../tests/resources/parse_answer/minimal.json |  17 +
>   .../tests/resources/parse_answer/minimal.toml |  14 +
>   .../resources/parse_answer/nic_matching.json  |  17 +
>   .../resources/parse_answer/nic_matching.toml  |  19 +
>   .../tests/resources/parse_answer/readme       |   4 +
>   .../resources/parse_answer/specific_nic.json  |  17 +
>   .../resources/parse_answer/specific_nic.toml  |  19 +
>   .../tests/resources/parse_answer/zfs.json     |  27 +
>   .../tests/resources/parse_answer/zfs.toml     |  20 +
>   .../tests/resources/run-env-info.json         |   1 +
>   .../tests/resources/run-env-udev.json         |   1 +
>   proxmox-autoinst-helper/Cargo.toml            |  22 +
>   proxmox-autoinst-helper/src/main.rs           | 561 ++++++++++++++++++
>   proxmox-chroot/Cargo.toml                     |  16 +
>   proxmox-chroot/src/main.rs                    | 356 +++++++++++
>   proxmox-fetch-answer/Cargo.toml               |  23 +
>   .../src/fetch_plugins/http.rs                 | 182 ++++++
>   proxmox-fetch-answer/src/fetch_plugins/mod.rs |   3 +
>   .../src/fetch_plugins/partition.rs            |  21 +
>   .../src/fetch_plugins/utils/mod.rs            |  96 +++
>   .../src/fetch_plugins/utils/post.rs           |  94 +++
>   proxmox-fetch-answer/src/main.rs              | 116 ++++
>   proxmox-installer-common/Cargo.toml           |   1 +
>   proxmox-installer-common/src/options.rs       |  21 +-
>   proxmox-installer-common/src/setup.rs         | 141 ++++-
>   proxmox-installer-common/src/utils.rs         |  11 +
>   proxmox-low-level-installer                   |  14 +
>   proxmox-tui-installer/src/options.rs          |   4 +-
>   proxmox-tui-installer/src/setup.rs            | 100 +---
>   .../src/views/install_progress.rs             |   4 +-
>   unconfigured.sh                               |  17 +
>   53 files changed, 3234 insertions(+), 118 deletions(-)
>   create mode 100644 Proxmox/Sys/Udev.pm
>   create mode 100644 proxmox-auto-installer/Cargo.toml
>   create mode 100644 proxmox-auto-installer/src/answer.rs
>   create mode 100644 proxmox-auto-installer/src/bin/proxmox-auto-installer.rs
>   create mode 100644 proxmox-auto-installer/src/lib.rs
>   create mode 100644 proxmox-auto-installer/src/log.rs
>   create mode 100644 proxmox-auto-installer/src/sysinfo.rs
>   create mode 100644 proxmox-auto-installer/src/udevinfo.rs
>   create mode 100644 proxmox-auto-installer/src/utils.rs
>   create mode 100644 proxmox-auto-installer/tests/parse-answer.rs
>   create mode 100644 proxmox-auto-installer/tests/resources/iso-info.json
>   create mode 100644 proxmox-auto-installer/tests/resources/locales.json
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match.json
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match.toml
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.json
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.toml
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.json
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.toml
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/minimal.json
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/minimal.toml
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/nic_matching.json
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/nic_matching.toml
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/readme
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/specific_nic.json
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/specific_nic.toml
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.json
>   create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.toml
>   create mode 100644 proxmox-auto-installer/tests/resources/run-env-info.json
>   create mode 100644 proxmox-auto-installer/tests/resources/run-env-udev.json
>   create mode 100644 proxmox-autoinst-helper/Cargo.toml
>   create mode 100644 proxmox-autoinst-helper/src/main.rs
>   create mode 100644 proxmox-chroot/Cargo.toml
>   create mode 100644 proxmox-chroot/src/main.rs
>   create mode 100644 proxmox-fetch-answer/Cargo.toml
>   create mode 100644 proxmox-fetch-answer/src/fetch_plugins/http.rs
>   create mode 100644 proxmox-fetch-answer/src/fetch_plugins/mod.rs
>   create mode 100644 proxmox-fetch-answer/src/fetch_plugins/partition.rs
>   create mode 100644 proxmox-fetch-answer/src/fetch_plugins/utils/mod.rs
>   create mode 100644 proxmox-fetch-answer/src/fetch_plugins/utils/post.rs
>   create mode 100644 proxmox-fetch-answer/src/main.rs
> 


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel