public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pve-devel] [PATCH v2 00/22] add automated/unattended installation
@ 2024-02-21 11:07 Aaron Lauterer
  2024-02-21 11:07 ` [pve-devel] [PATCH v2 01/22] tui: common: move InstallConfig struct to common crate Aaron Lauterer
                   ` (23 more replies)
  0 siblings, 24 replies; 29+ messages in thread
From: Aaron Lauterer @ 2024-02-21 11:07 UTC (permalink / raw)
  To: pve-devel

This patch series adds the possibility to do an automated / unattended
installation of Proxmox VE.

It assumes that the patch series to use JSON output on the
low-level-installer is already applied [1].

The overall idea is that we will have a dedicated ISO for the unattended
installation. It should be configured in such a way that it will start
the installation without any user interaction.
Though the integration in the installation environmend isn't part of
this patch series.

The information for the installer that is usually gathered interactively
from the user is provided via an `answer.toml` file.

The answer file allows to select disks and the network card via filters.

The installer also allows to run custom commands pre and post
installation. This should give users plenty of possibilities to either
further customize/prepare the installation or integrate it into a larger
automated installation setup.
For example, one could issue HTTP requests to signal the status and
progress of the installation.


The install environment needs to call the 'proxmox-fetch-answer' binary.
It tries to find the answer file and once found, will start the
'proxmox-auto-installer' binary and pass the contents to it via stdin.

The auto-installer then parses the answer file and determines what
parameters need to be passed to the low-level installer. For example,
which disks and NIC to use, network IP settings and so forth.

The current status reporting of the actual installation is kept rather
simple.

Both binaries log into the tmp directory.

There is a third binary, the 'proxmox-autoinst-helper'. It provides a
few subcommands, from the help:
  answer        Validate if an answer file is formatted correctly
  device-match  Test which devices the given filter matches against
  device-info   Show device information that can be used for filters
  identifiers   Show identifiers for the current machine. This information is part of the POST request to fetch an answer file

The fetch-answer binary is trying to get an answer file. It does so by
first searching for a partition/FS labeled `proxmoxinst`, or all upper
case, and an `answer.toml` in there. This could be provided by another
USB flash drive.
If that is not successful, the next step is to send an HTTP POST request
to a URL to get the TOML contents in return. A POST request was chosen
because we also send information to identify the host in JSON format.

The question then is, where to get that URL from. Right now, there are
two options implemented. The first is looking for a custom DHCP option
and the second is querying for a TXT record in the `proxmoxinst`
subdomain of the search domain.

It is possible to provide a SHA256 fingerprint of the SSL cert used by
the answer server. The safest option is to place a
`cert_fingerprint.txt` file in the same `proxmoxinst` partition as where
you alternatively would place the `answer.toml`.
If that is not found, then it can be provided by a second custom DHCP
option or placed as TXT record in the subdomain `proxmoxinst-fp`.

Other plans / ideas for the future:

* add a chroot helper tool to make it easier to modify the installation
  in a post-command (I might send a follow up patch in the next days)
* add option to define remote SSH access (password and,or public key).
  This could make remote debugging in case of problems easier


Regarding the patch series itself:
01-03 are needed to move some code into the common crate and
make structs/functions already in the common crate accessible.

I did split up the individual parts of the auto installer into their own
patches as much as possible, and (hopefully) in the order they depend on
each other.

Patches after the `unconfigured` one (14), switch the pattern matching
to the glob crate, add the helper tool and the fetching via HTTP.

The last patch updates the build dependencies. I am not sure though if
I did it correctly so take it with a grain of salt.

I did skip on the doc patch this time as it needs more time and will be
either sent as a dedicated patch or follow up to this series.

Areas that can be improved/extended:
* Testing possibility integrated in the Makefile
* Documentation: explain process, additional examples for answer.toml


Changes since v1:
* incorporated feedback (thx @cheiss)
* the proxmox-installer-filter tool got renamed and extended to the
  proxmox-autoinst-helper
* added HTTP fetch functionality
* moved the fetch_plugins/utils.rs into its own subdirectory as it grew
  quite a bit
* added builddeps

[0] https://lists.proxmox.com/pipermail/pve-devel/2023-September/059020.html
[1] https://lists.proxmox.com/pipermail/pve-devel/2023-December/060961.html


Aaron Lauterer (22):
  tui: common: move InstallConfig struct to common crate
  common: make InstallZfsOption members public
  common: tui: use BTreeMap for predictable ordering
  low-level: add dump-udev command
  add auto-installer crate
  auto-installer: add dependencies
  auto-installer: add answer file definition
  auto-installer: add struct to hold udev info
  auto-installer: add utils
  auto-installer: add simple logging
  auto-installer: add tests for answer file parsing
  auto-installer: add auto-installer binary
  auto-installer: add fetch answer binary
  unconfigured: add proxauto as option to start auto installer
  auto-installer: use glob crate for pattern matching
  auto-installer: utils: make get_udev_index functions public
  auto-installer: add proxmox-autoinst-helper tool
  auto-installer: fetch: add gathering of system identifiers and
    restructure code
  auto-installer: helper: add subcommand to view indentifiers
  auto-installer: fetch: add http post utility module
  auto-installer: fetch: add http plugin to fetch answer
  control: update build depends for auto installer

 Cargo.toml                                    |   1 +
 Makefile                                      |   7 +-
 Proxmox/Makefile                              |   1 +
 Proxmox/Sys/Udev.pm                           |  54 +++
 debian/control                                |  10 +
 proxmox-auto-installer/Cargo.toml             |  25 +
 proxmox-auto-installer/src/answer.rs          | 133 +++++
 .../src/bin/proxmox-auto-installer.rs         | 193 ++++++++
 .../src/bin/proxmox-autoinst-helper.rs        | 333 +++++++++++++
 .../src/bin/proxmox-fetch-answer.rs           |  76 +++
 .../src/fetch_plugins/http.rs                 | 185 +++++++
 .../src/fetch_plugins/mod.rs                  |   3 +
 .../src/fetch_plugins/partition.rs            |  32 ++
 .../src/fetch_plugins/utils/mod.rs            | 114 +++++
 .../src/fetch_plugins/utils/post.rs           |  93 ++++
 .../src/fetch_plugins/utils/sysinfo.rs        | 200 ++++++++
 proxmox-auto-installer/src/lib.rs             |   5 +
 proxmox-auto-installer/src/log.rs             |  38 ++
 proxmox-auto-installer/src/udevinfo.rs        |   9 +
 proxmox-auto-installer/src/utils.rs           | 459 ++++++++++++++++++
 proxmox-auto-installer/tests/parse-answer.rs  | 102 ++++
 .../tests/resources/iso-info.json             |   1 +
 .../tests/resources/locales.json              |   1 +
 .../resources/parse_answer/disk_match.json    |  29 ++
 .../resources/parse_answer/disk_match.toml    |  14 +
 .../parse_answer/disk_match_all.json          |  26 +
 .../parse_answer/disk_match_all.toml          |  16 +
 .../parse_answer/disk_match_any.json          |  33 ++
 .../parse_answer/disk_match_any.toml          |  16 +
 .../tests/resources/parse_answer/minimal.json |  17 +
 .../tests/resources/parse_answer/minimal.toml |  14 +
 .../resources/parse_answer/nic_matching.json  |  17 +
 .../resources/parse_answer/nic_matching.toml  |  19 +
 .../tests/resources/parse_answer/readme       |   4 +
 .../resources/parse_answer/specific_nic.json  |  17 +
 .../resources/parse_answer/specific_nic.toml  |  19 +
 .../tests/resources/parse_answer/zfs.json     |  27 ++
 .../tests/resources/parse_answer/zfs.toml     |  19 +
 .../tests/resources/run-env-info.json         |   1 +
 .../tests/resources/run-env-udev.json         |   1 +
 proxmox-installer-common/src/setup.rs         | 100 +++-
 proxmox-low-level-installer                   |  13 +
 proxmox-tui-installer/src/options.rs          |   4 +-
 proxmox-tui-installer/src/setup.rs            | 100 +---
 .../src/views/install_progress.rs             |   4 +-
 unconfigured.sh                               |  17 +
 46 files changed, 2493 insertions(+), 109 deletions(-)
 create mode 100644 Proxmox/Sys/Udev.pm
 create mode 100644 proxmox-auto-installer/Cargo.toml
 create mode 100644 proxmox-auto-installer/src/answer.rs
 create mode 100644 proxmox-auto-installer/src/bin/proxmox-auto-installer.rs
 create mode 100644 proxmox-auto-installer/src/bin/proxmox-autoinst-helper.rs
 create mode 100644 proxmox-auto-installer/src/bin/proxmox-fetch-answer.rs
 create mode 100644 proxmox-auto-installer/src/fetch_plugins/http.rs
 create mode 100644 proxmox-auto-installer/src/fetch_plugins/mod.rs
 create mode 100644 proxmox-auto-installer/src/fetch_plugins/partition.rs
 create mode 100644 proxmox-auto-installer/src/fetch_plugins/utils/mod.rs
 create mode 100644 proxmox-auto-installer/src/fetch_plugins/utils/post.rs
 create mode 100644 proxmox-auto-installer/src/fetch_plugins/utils/sysinfo.rs
 create mode 100644 proxmox-auto-installer/src/lib.rs
 create mode 100644 proxmox-auto-installer/src/log.rs
 create mode 100644 proxmox-auto-installer/src/udevinfo.rs
 create mode 100644 proxmox-auto-installer/src/utils.rs
 create mode 100644 proxmox-auto-installer/tests/parse-answer.rs
 create mode 100644 proxmox-auto-installer/tests/resources/iso-info.json
 create mode 100644 proxmox-auto-installer/tests/resources/locales.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_all.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/disk_match_any.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/minimal.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/minimal.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/nic_matching.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/nic_matching.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/readme
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/specific_nic.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/specific_nic.toml
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.json
 create mode 100644 proxmox-auto-installer/tests/resources/parse_answer/zfs.toml
 create mode 100644 proxmox-auto-installer/tests/resources/run-env-info.json
 create mode 100644 proxmox-auto-installer/tests/resources/run-env-udev.json

-- 
2.39.2





^ permalink raw reply	[flat|nested] 29+ messages in thread

end of thread, other threads:[~2024-02-23 11:37 UTC | newest]

Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-21 11:07 [pve-devel] [PATCH v2 00/22] add automated/unattended installation Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 01/22] tui: common: move InstallConfig struct to common crate Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 02/22] common: make InstallZfsOption members public Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 03/22] common: tui: use BTreeMap for predictable ordering Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 04/22] low-level: add dump-udev command Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 05/22] add auto-installer crate Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 06/22] auto-installer: add dependencies Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 07/22] auto-installer: add answer file definition Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 08/22] auto-installer: add struct to hold udev info Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 09/22] auto-installer: add utils Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 10/22] auto-installer: add simple logging Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 11/22] auto-installer: add tests for answer file parsing Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 12/22] auto-installer: add auto-installer binary Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 13/22] auto-installer: add fetch answer binary Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 14/22] unconfigured: add proxauto as option to start auto installer Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 15/22] auto-installer: use glob crate for pattern matching Aaron Lauterer
2024-02-21 11:07 ` [pve-devel] [PATCH v2 16/22] auto-installer: utils: make get_udev_index functions public Aaron Lauterer
2024-02-21 11:08 ` [pve-devel] [PATCH v2 17/22] auto-installer: add proxmox-autoinst-helper tool Aaron Lauterer
2024-02-21 11:08 ` [pve-devel] [PATCH v2 18/22] auto-installer: fetch: add gathering of system identifiers and restructure code Aaron Lauterer
2024-02-21 14:09   ` Christoph Heiss
2024-02-21 16:07     ` Aaron Lauterer
2024-02-21 11:08 ` [pve-devel] [PATCH v2 19/22] auto-installer: helper: add subcommand to view indentifiers Aaron Lauterer
2024-02-21 11:08 ` [pve-devel] [PATCH v2 20/22] auto-installer: fetch: add http post utility module Aaron Lauterer
2024-02-21 12:21   ` Christoph Heiss
2024-02-21 11:08 ` [pve-devel] [PATCH v2 21/22] auto-installer: fetch: add http plugin to fetch answer Aaron Lauterer
2024-02-21 11:08 ` [pve-devel] [PATCH v2 22/22] control: update build depends for auto installer Aaron Lauterer
2024-02-21 13:39 ` [pve-devel] [PATCH v2 00/22] add automated/unattended installation Christoph Heiss
2024-02-23 10:19 ` Friedrich Weber
2024-02-23 11:37   ` Aaron Lauterer

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal