public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pve-devel] [PATCH installer v6 00/36] add automated/unattended installation
@ 2024-04-17 12:30 Aaron Lauterer
  2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 01/36] tui: common: move InstallConfig struct to common crate Aaron Lauterer
                   ` (35 more replies)
  0 siblings, 36 replies; 41+ messages in thread
From: Aaron Lauterer @ 2024-04-17 12:30 UTC (permalink / raw)
  To: pve-devel

patches until 31 got a [0,1]

Tested-by: Christoph Heiss <c.heiss@proxmox.com>
Reviewed-by: Christoph Heiss <c.heiss@proxmox.com>

changes since v5:

some small style fixups and using cert_fingerprint or TLS instead of
ssl_fingerprint and SSL in patches 32-36.

changes since v4:

Patches 32-36 finalize how to prepare an ISO for automated installation
and introduce a slight change in behavior, as it is now also possible to
include the needed parameters into the ISO directly:

* answer file itself
* URL to fetch it from
* SSL cert fingerprint

The 'proxmox-autoinst-helper' tool got a new subcommand to prepare an
ISO.

The cover letter iself:

The overall idea is to prepare an ISO for automated installation. A
prepare ISO will automatically boot into the installation.

The information for the installer that is usually gathered interactively
from the user is provided via an `answer.toml` file.

The answer file allows to select disks and the network card via filters.

The installer also allows to run custom commands pre and post
installation. This should give users plenty of possibilities to either
further customize/prepare the installation or integrate it into a larger
automated installation setup.
For example, one could issue HTTP requests to signal the status and
progress of the installation.

When the installer is called with 'proxauto' in the kernel cmdline, the
'proxmox-fetch-answer' binary is called. It tries to find the answer
file and once found, will start the 'proxmox-auto-installer' binary and
pass the contents to it via stdin.

The auto-installer then parses the answer file and determines what
parameters need to be passed to the low-level installer. For example,
which disks and NIC to use, network IP settings and so forth.

The current status reporting of the actual installation is kept rather
simple.

Both binaries log into the tmp directory.

There is a third binary, the 'proxmox-autoinst-helper'. It provides a
few subcommands, from the help:
  prepare-iso      Prepare an ISO for automated installation.
  validate-answer  Validate if an answer file is formatted correctly
  device-match     Test which devices the given filter matches against
  device-info      Show device information that can be used for filters
  identifiers      Show identifiers for the current machine. This information is part of the POST request to fetch an answer file
  help             Print this message or the help of the given subcommand(s)

The fetch-answer binary is trying to get an answer file. If included in
the ISO, it will use that one. If no answer file is included, it first
searches for a partition/FS labeled `proxmoxinst`, or all upper case,
and an `answer.toml` in there. This could be provided by another USB
flash drive.
If that is not successful, the next step is to send an HTTP POST request
to a URL to get the TOML contents in return. A POST request was chosen
because we also send information to identify the host in JSON format.

The question then is, where to get that URL from. Right now, there are
three options implemented. The first is to hardcode it in the prepared
iso. The second is to look for a custom DHCP option
and the third is to query for a TXT record in the `proxmoxinst`
subdomain of the search domain.

It is possible to provide a SHA256 fingerprint of the SSL cert used by
the answer server. The safest option is to include it in the ISO itself.
If that is not found, then it can be provided by a second custom DHCP
option or placed as TXT record in the subdomain `proxmoxinst-fp`.

This patch series now also separates the 3 binaries into their own
crate. The 'proxmox-fetch-answer' to keep the OpenSSL dependency as
localized as possible, and the 'proxmox-autoinst-helper' to make it easy
to compile just that binary.

The new `proxmox-chroot` utility helps to prepare everything to chroot
into a fresh installation and clean it up once done.
This will be useful in the post commands when further customizing the
installation.


Other plans / ideas for the future:

* add option to define remote SSH access (password and,or public key).
  This could make remote debugging in case of problems easier
* make it possible to flash prepared ISO directly to USB flash drives


Regarding the patch series itself:
01-03 are needed to move some code into the common crate and
make structs/functions already in the common crate accessible.

I did split up the individual parts of the auto installer into their own
patches as much as possible, and (hopefully) in the order they depend on
each other.

Patches after the `unconfigured` one (16), switch the pattern matching
to the glob crate, add the helper tool and the fetching via HTTP.

Patch 26 factors our the binaries into their own crates.

Patches 27-30 are for the 'proxmox-chroot' utility and preparations for
it to work.

Patch 31 makes sure that the answer file can only contain known keys.

Patches 32 - 36 finalize the ISO preparation, add the subcommand for it
to the autoinst-helper tool and adapt the fetch-answer binary to also
handle the new options to include the infos in the ISO itself.

Areas that can be improved/extended:
* Testing possibility integrated in the Makefile
* make build target for statically compiled proxmox-autoinst-helper

I did test it with all 3 installers, PVE, PMG and PBS and it worked.
With PVE I tested all variants on how to inject infos into the prepared
ISO.

WIP: Documentation. A first draft is available in the inernal wiki, as
we will most likely keep it in wiki format since it applies for all 3
products, if we provide ISOs for it.

[0] https://lists.proxmox.com/pipermail/pve-devel/2024-April/062634.html
[1] https://lists.proxmox.com/pipermail/pve-devel/2024-April/062690.html

Aaron Lauterer (36):
  tui: common: move InstallConfig struct to common crate
  common: make InstallZfsOption members public
  common: tui: use BTreeMap for predictable ordering
  common: utils: add deserializer for CidrAddress
  common: options: add Deserialize trait
  low-level: add dump-udev command
  add auto-installer crate
  auto-installer: add dependencies
  auto-installer: add answer file definition
  auto-installer: add struct to hold udev info
  auto-installer: add utils
  auto-installer: add simple logging
  auto-installer: add tests for answer file parsing
  auto-installer: add auto-installer binary
  auto-installer: add fetch answer binary
  unconfigured: add proxauto as option to start auto installer
  auto-installer: use glob crate for pattern matching
  auto-installer: utils: make get_udev_index functions public
  auto-installer: add proxmox-autoinst-helper tool
  common: add Display trait to ProxmoxProduct
  auto-installer: fetch: add gathering of system identifiers and
    restructure code
  auto-installer: helper: add subcommand to view indentifiers
  auto-installer: fetch: add http post utility module
  auto-installer: fetch: add http plugin to fetch answer
  control: update build depends for auto installer
  auto installer: factor out fetch-answer and autoinst-helper
  low-level: write low level config to /tmp
  common: add deserializer for FsType
  common: skip target_hd when deserializing InstallConfig
  add proxmox-chroot utility
  auto-installer: answer: deny unknown fields
  fetch-answer: move get_answer_file to utils
  auto-installer: utils: define ISO specified settings
  fetch-answer: use ISO specified configurations
  fetch-answer: dpcp: improve logging of steps taken
  autoinst-helper: add prepare-iso subcommand

 Cargo.toml                                    |   4 +
 Makefile                                      |  19 +-
 Proxmox/Makefile                              |   1 +
 Proxmox/Sys/Udev.pm                           |  54 ++
 debian/control                                |  10 +
 proxmox-auto-installer/Cargo.toml             |  20 +
 proxmox-auto-installer/src/answer.rs          | 255 ++++++++
 .../src/bin/proxmox-auto-installer.rs         | 195 ++++++
 proxmox-auto-installer/src/lib.rs             |   5 +
 proxmox-auto-installer/src/log.rs             |  38 ++
 proxmox-auto-installer/src/sysinfo.rs         |  81 +++
 proxmox-auto-installer/src/udevinfo.rs        |   9 +
 proxmox-auto-installer/src/utils.rs           | 455 ++++++++++++++
 proxmox-auto-installer/tests/parse-answer.rs  | 106 ++++
 .../tests/resources/iso-info.json             |   1 +
 .../tests/resources/locales.json              |   1 +
 .../resources/parse_answer/disk_match.json    |  29 +
 .../resources/parse_answer/disk_match.toml    |  17 +
 .../parse_answer/disk_match_all.json          |  26 +
 .../parse_answer/disk_match_all.toml          |  17 +
 .../parse_answer/disk_match_any.json          |  33 +
 .../parse_answer/disk_match_any.toml          |  17 +
 .../tests/resources/parse_answer/minimal.json |  17 +
 .../tests/resources/parse_answer/minimal.toml |  14 +
 .../resources/parse_answer/nic_matching.json  |  17 +
 .../resources/parse_answer/nic_matching.toml  |  19 +
 .../tests/resources/parse_answer/readme       |   4 +
 .../resources/parse_answer/specific_nic.json  |  17 +
 .../resources/parse_answer/specific_nic.toml  |  19 +
 .../tests/resources/parse_answer/zfs.json     |  27 +
 .../tests/resources/parse_answer/zfs.toml     |  20 +
 .../tests/resources/run-env-info.json         |   1 +
 .../tests/resources/run-env-udev.json         |   1 +
 proxmox-autoinst-helper/Cargo.toml            |  22 +
 proxmox-autoinst-helper/src/main.rs           | 562 ++++++++++++++++++
 proxmox-chroot/Cargo.toml                     |  16 +
 proxmox-chroot/src/main.rs                    | 356 +++++++++++
 proxmox-fetch-answer/Cargo.toml               |  23 +
 .../src/fetch_plugins/http.rs                 | 182 ++++++
 proxmox-fetch-answer/src/fetch_plugins/mod.rs |   3 +
 .../src/fetch_plugins/partition.rs            |  21 +
 .../src/fetch_plugins/utils/mod.rs            |  96 +++
 .../src/fetch_plugins/utils/post.rs           |  94 +++
 proxmox-fetch-answer/src/main.rs              | 116 ++++
 proxmox-installer-common/Cargo.toml           |   1 +
 proxmox-installer-common/src/options.rs       |  21 +-
 proxmox-installer-common/src/setup.rs         | 141 ++++-
 proxmox-installer-common/src/utils.rs         |  11 +
 proxmox-low-level-installer                   |  14 +
 proxmox-tui-installer/src/options.rs          |   4 +-
 proxmox-tui-installer/src/setup.rs            | 100 +---
 .../src/views/install_progress.rs             |   4 +-
 unconfigured.sh                               |  17 +
 53 files changed, 3235 insertions(+), 118 deletions(-)
 create mode 100644 Proxmox/Sys/Udev.pm
 create mode 100644 proxmox-auto-installer/Cargo.toml
 create mode 100644 proxmox-auto-installer/src/answer.rs
 create mode 100644 proxmox-auto-installer/src/bin/proxmox-auto-installer.rs
 create mode 100644 proxmox-auto-installer/src/lib.rs
 create mode 100644 proxmox-auto-installer/src/log.rs
 create mode 100644 proxmox-auto-installer/src/sysinfo.rs
 create mode 100644 proxmox-auto-installer/src/udevinfo.rs
 create mode 100644 proxmox-auto-installer/src/utils.rs
 create mode 100644 proxmox-auto-installer/tests/parse-answer.rs
 create mode 100644 proxmox-auto-installer/tests/resources/iso-info.json
 create mode 100644 proxmox-auto-installer/tests/resources/locales.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/minimal.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/minimal.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/nic_matching.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/nic_matching.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/readme
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/specific_nic.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/specific_nic.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.toml
 create mode 100644 proxmox-auto-installer/tests/resources/run-env-info.json
 create mode 100644 proxmox-auto-installer/tests/resources/run-env-udev.json
 create mode 100644 proxmox-autoinst-helper/Cargo.toml
 create mode 100644 proxmox-autoinst-helper/src/main.rs
 create mode 100644 proxmox-chroot/Cargo.toml
 create mode 100644 proxmox-chroot/src/main.rs
 create mode 100644 proxmox-fetch-answer/Cargo.toml
 create mode 100644 proxmox-fetch-answer/src/fetch_plugins/http.rs
 create mode 100644 proxmox-fetch-answer/src/fetch_plugins/mod.rs
 create mode 100644 proxmox-fetch-answer/src/fetch_plugins/partition.rs
 create mode 100644 proxmox-fetch-answer/src/fetch_plugins/utils/mod.rs
 create mode 100644 proxmox-fetch-answer/src/fetch_plugins/utils/post.rs
 create mode 100644 proxmox-fetch-answer/src/main.rs

-- 
2.39.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel


^ permalink raw reply	[flat|nested] 41+ messages in thread

end of thread, other threads:[~2024-04-18 13:39 UTC | newest]

Thread overview: 41+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-04-17 12:30 [pve-devel] [PATCH installer v6 00/36] add automated/unattended installation Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 01/36] tui: common: move InstallConfig struct to common crate Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 02/36] common: make InstallZfsOption members public Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 03/36] common: tui: use BTreeMap for predictable ordering Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 04/36] common: utils: add deserializer for CidrAddress Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 05/36] common: options: add Deserialize trait Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 06/36] low-level: add dump-udev command Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 07/36] add auto-installer crate Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 08/36] auto-installer: add dependencies Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 09/36] auto-installer: add answer file definition Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 10/36] auto-installer: add struct to hold udev info Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 11/36] auto-installer: add utils Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 12/36] auto-installer: add simple logging Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 13/36] auto-installer: add tests for answer file parsing Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 14/36] auto-installer: add auto-installer binary Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 15/36] auto-installer: add fetch answer binary Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 16/36] unconfigured: add proxauto as option to start auto installer Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 17/36] auto-installer: use glob crate for pattern matching Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 18/36] auto-installer: utils: make get_udev_index functions public Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 19/36] auto-installer: add proxmox-autoinst-helper tool Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 20/36] common: add Display trait to ProxmoxProduct Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 21/36] auto-installer: fetch: add gathering of system identifiers and restructure code Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 22/36] auto-installer: helper: add subcommand to view indentifiers Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 23/36] auto-installer: fetch: add http post utility module Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 24/36] auto-installer: fetch: add http plugin to fetch answer Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 25/36] control: update build depends for auto installer Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 26/36] auto installer: factor out fetch-answer and autoinst-helper Aaron Lauterer
2024-04-17 12:30 ` [pve-devel] [PATCH installer v6 27/36] low-level: write low level config to /tmp Aaron Lauterer
2024-04-17 12:31 ` [pve-devel] [PATCH installer v6 28/36] common: add deserializer for FsType Aaron Lauterer
2024-04-17 12:31 ` [pve-devel] [PATCH installer v6 29/36] common: skip target_hd when deserializing InstallConfig Aaron Lauterer
2024-04-17 12:31 ` [pve-devel] [PATCH installer v6 30/36] add proxmox-chroot utility Aaron Lauterer
2024-04-17 12:31 ` [pve-devel] [PATCH installer v6 31/36] auto-installer: answer: deny unknown fields Aaron Lauterer
2024-04-17 12:31 ` [pve-devel] [PATCH installer v6 32/36] fetch-answer: move get_answer_file to utils Aaron Lauterer
2024-04-17 12:31 ` [pve-devel] [PATCH installer v6 33/36] auto-installer: utils: define ISO specified settings Aaron Lauterer
2024-04-17 12:31 ` [pve-devel] [PATCH installer v6 34/36] fetch-answer: use ISO specified configurations Aaron Lauterer
2024-04-17 12:31 ` [pve-devel] [PATCH installer v6 35/36] fetch-answer: dpcp: improve logging of steps taken Aaron Lauterer
2024-04-17 12:31 ` [pve-devel] [PATCH installer v6 36/36] autoinst-helper: add prepare-iso subcommand Aaron Lauterer
2024-04-18  8:48   ` Christoph Heiss
2024-04-18  9:13     ` Aaron Lauterer
2024-04-18 13:39     ` Thomas Lamprecht
2024-04-18 11:38   ` [pve-devel] [PATCH installer v6 36/36 follow-up] " Aaron Lauterer

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal