[pve-devel] [PATCH v1 installer/docs 00/18] add automated/unattended installation

[Aaron Lauterer <a.lauterer@proxmox.com>]

This patch series adds the possibility to do an automated / unattended
installation of Proxmox VE.

It assumes that the patch series to use JSON output on the
low-level-installer is already applied [1].

The overall idea is that we will have a dedicated ISO for the unattended
installation. It should be configured in such a way that it will start
the installation without any user interaction.
Though the integration in the installation environmend isn't part of
this patch series.

The information for the installer that is usually gathered interactively
from the user is provided via an `answer.toml` file.

The answer file allows to select disks and the network card via filters.

The installer also allows to run custom commands pre and post
installation. This should give users plenty of possibilities to either
further customize/prepare the installation or integrate it into a larger
automated installation setup.
For example, one could issue HTTP requests to signal the status and
progress of the installation.


The install environment needs to call the 'proxmox-fetch-answer' binary.
It tries to find the answer file and once found, will start the
'proxmox-auto-installer' binary and pass the contents to it via stdin.

The auto-installer then parses the answer file and determines what
parameters need to be passed to the low-level installer. For example,
which disks and NIC to use, network IP settings and so forth.

The current status reporting of the actual installation is kept rather
simple.

Both binaries log into the /tmp/ directory.

There is a third binary, the 'proxmox-installer-filter'. It is meant as
a pure utility for users to make it easier to see what properties they
can write filters against and to test the filters.


The fetch-answer binary is currently searching for a
partition/file-system labeled 'proxmoxinst' in lower or uppercase. It
can be located on an additioan USB flash drive, or maybe on the install
medium itself if it is possible to write to it.


We do have some ideas for additional steps to fetch an answer file. The
main one is that we could download the answer file from a URL. Ideally
we would send unique properties along with the request (MAC addresses,
serial numbers, ...) so that it is possible to have a script on the
receiving side that can then generate the answer file dynamically.

The big question is, where the URL comes from, for which we have also
some ideas:
* custom DHCP options
* kernel cmdline (might be an option with PXE boot)
* TXT DNS record in a predefined subdomain of the search domain received
  via DHCP, basically a 'dig TXT proxmoxinst.{search domain}'.
* We should also make it possible to provide an SSL fingerprint in a
  similar manner in case the listening server is not trusted out of the
  box.

Other plans / ideas for the future:

* add option to define remote SSH access (password and,or public key).
  This could make remote debugging in case of problems easier


Regarding the patch series itself:
The first patches are needed to move some code into the common crate and
make structs/functions already in the common crate accessible.

I did split up the individual parts of the auto installer into their own
patches as much as possible, and (hopefully) in the order they depend on
each other.

Areas that can be improved/extended:
* Testing possibility integrated in the Makefile
* Documentation: explain process, additional examples for answer.toml

[0] https://lists.proxmox.com/pipermail/pve-devel/2023-September/059020.html
[1] https://lists.proxmox.com/pipermail/pve-devel/2023-December/060961.html

installer: Aaron Lauterer (17):
  tui: common: move InstallConfig struct to common crate
  common: make InstallZfsOption members public
  common: tui: use BTreeMap for predictable ordering
  Makefile: fix handling of multiple usr_bin files
  low-level: add dump-udev command
  add auto-installer crate
  auto-installer: add dependencies
  auto-installer: add answer file definition
  auto-installer: add struct to hold udev info
  auto-installer: add utils
  auto-installer: add simple logging
  auto-installer: add tests for answer file parsing
  auto-installer: add auto-installer binary
  auto-installer: add fetch answer binary
  auto-installer: use glob crate for pattern matching
  auto-installer: utils: make get_udev_index functions public
  auto-installer: add proxmox-installer-filter helper tool


docs: Aaron Lauterer (1):
  installation: add unattended documentation

 pve-installation.adoc | 267 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 267 insertions(+)

 Cargo.toml                                    |   1 +
 Makefile                                      |   9 +-
 Proxmox/Makefile                              |   1 +
 Proxmox/Sys/Udev.pm                           |  54 ++
 proxmox-auto-installer/Cargo.toml             |  19 +
 proxmox-auto-installer/src/answer.rs          | 148 ++++++
 .../src/bin/proxmox-auto-installer.rs         | 192 ++++++++
 .../src/bin/proxmox-fetch-answer.rs           |  73 +++
 .../src/bin/proxmox-installer-filter.rs       | 298 +++++++++++
 .../src/fetch_plugins/mod.rs                  |   1 +
 .../src/fetch_plugins/partition.rs            | 102 ++++
 proxmox-auto-installer/src/lib.rs             |   5 +
 proxmox-auto-installer/src/log.rs             |  38 ++
 proxmox-auto-installer/src/udevinfo.rs        |   9 +
 proxmox-auto-installer/src/utils.rs           | 461 ++++++++++++++++++
 proxmox-auto-installer/tests/parse-answer.rs  | 102 ++++
 .../tests/resources/iso-info.json             |   1 +
 .../tests/resources/locales.json              |   1 +
 .../resources/parse_answer/disk_match.json    |  29 ++
 .../resources/parse_answer/disk_match.toml    |  14 +
 .../parse_answer/disk_match_all.json          |  26 +
 .../parse_answer/disk_match_all.toml          |  16 +
 .../parse_answer/disk_match_any.json          |  33 ++
 .../parse_answer/disk_match_any.toml          |  16 +
 .../tests/resources/parse_answer/minimal.json |  17 +
 .../tests/resources/parse_answer/minimal.toml |  14 +
 .../resources/parse_answer/nic_matching.json  |  17 +
 .../resources/parse_answer/nic_matching.toml  |  19 +
 .../tests/resources/parse_answer/readme       |   4 +
 .../resources/parse_answer/specific_nic.json  |  17 +
 .../resources/parse_answer/specific_nic.toml  |  19 +
 .../tests/resources/parse_answer/zfs.json     |  27 +
 .../tests/resources/parse_answer/zfs.toml     |  19 +
 .../tests/resources/run-env-info.json         |   1 +
 .../tests/resources/run-env-udev.json         |   1 +
 proxmox-installer-common/src/setup.rs         | 100 +++-
 proxmox-low-level-installer                   |  13 +
 proxmox-tui-installer/src/options.rs          |   4 +-
 proxmox-tui-installer/src/setup.rs            | 100 +---
 .../src/views/install_progress.rs             |   4 +-
 40 files changed, 1915 insertions(+), 110 deletions(-)
 create mode 100644 Proxmox/Sys/Udev.pm
 create mode 100644 proxmox-auto-installer/Cargo.toml
 create mode 100644 proxmox-auto-installer/src/answer.rs
 create mode 100644 proxmox-auto-installer/src/bin/proxmox-auto-installer.rs
 create mode 100644 proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
 create mode 100644 proxmox-auto-installer/src/bin/proxmox-installer-filter.rs
 create mode 100644 proxmox-auto-installer/src/fetch_plugins/mod.rs
 create mode 100644 proxmox-auto-installer/src/fetch_plugins/partition.rs
 create mode 100644 proxmox-auto-installer/src/lib.rs
 create mode 100644 proxmox-auto-installer/src/log.rs
 create mode 100644 proxmox-auto-installer/src/udevinfo.rs
 create mode 100644 proxmox-auto-installer/src/utils.rs
 create mode 100644 proxmox-auto-installer/tests/parse-answer.rs
 create mode 100644 proxmox-auto-installer/tests/resources/iso-info.json
 create mode 100644 proxmox-auto-installer/tests/resources/locales.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/minimal.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/minimal.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/nic_matching.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/nic_matching.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/readme
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/specific_nic.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/specific_nic.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.toml
 create mode 100644 proxmox-auto-installer/tests/resources/run-env-info.json
 create mode 100644 proxmox-auto-installer/tests/resources/run-env-udev.json

-- 
2.39.2