[PATCH datacenter-manager/installer/proxmox/yew-comp v4 00/40] add auto-installer integration

[Christoph Heiss <c.heiss@proxmox.com>]

This series adds integration with our automated installer [0] for all
our products. With this, Proxmox Datacenter Manager can be used for
serving answer files via HTTPS in an automated fashion.

It provides three panels under the "Automated Installations" tab in the
"Remotes" menu:

- Automated Installations: List all past and currently ongoing
  installations.
- Prepared Answers: Enables users to do the whole CRUD cycle for
  prepared answers, i.e. create new answers (from scratch or based on
  existing ones), edit and delete.
- Authentication Tokens: Simple token for the /answer endpoint. What
  tokens can be used to obtain which answer can be set when editing a
  prepared answer under the "Authentication" tab.

When a new token was created, either through the dedicated panel or
directly while creating a new prepared answer, the secret itself as well
as the command line for preparing an installation ISO is shown in the
UI.

Permission-wise, everything is currently scoped under
/system/auto-installation.

I can provide pre-built packages for testing, if needed - just holler at
me.

User interface
==============

By the nature of it, the auto-installer has a lot more options than the
standard GUI/TUI installer, and it's kind of hard to not fully cramp up
the UI while still providing all options.

The viewing of the system information and post-installation hook
notification data could be maybe considered a bit bare-bones. For the
first one, displaying it as a tree could work, if we don't want to
simply display raw JSON documents to the user.

Answer endpoint
===============

The endpoint for serving answer files to the automated installer lives
under `/api2/json/auto-install/answer`. This makes it (unfortunately)
incompatible with older ISOs, as currently they only support answer
files formatted as TOML. See also the discussion in [1].

Installer changes
=================

Most of the changes to pve-installer are churn from switching to the
moved types in proxmox-installer-types. They were heavily intertwined
between the different crates in pve-installer, so
a) untwisting that, b) doing some long-overdue quality-of-life
improvements and c) making them #[api] ready created a lot of churn,
although nothing in the actual TOML/JSON interface changed for any of
these types.

Other possible future work
==========================

- Target filter selection by dropdown/tree. Based on the system
  information schema, a list of all possible JSON pointers, or even a
  tree, could be generated and displayed to the user, in a form
  consisting of the key, a text input for the value/glob to match and a
  set of add/remove buttons for each filter.

- Display/saving the installation progress/log. The progress log
  generated by the auto-installer can be sent to PDM, showing it there
  similar to tasks.

- Implement a "wait for answer" system. Core idea here is that the
  installer waits for PDM to provide in answer by implemented e.g. an
  retry system. The use case is for to be able to effectively install
  systems on demand, without the need for preparations.

- Providing the option for automatically adding the installed node to
  PDM (for PVE/PBS)

- Maybe mTLS authentication for the answer endpoint, as suggested by
  Lukas/Thomas?

[0] https://pve.proxmox.com/wiki/Automated_Installation
[1] https://lore.proxmox.com/pdm-devel/11b27b4a-7e1f-4af9-8384-12d54d72ef17@proxmox.com/
[2] https://lore.proxmox.com/pdm-devel/DETMUXY1Q877.32G593TWC52WW@proxmox.com/#:~:text=%20I%20think%20this%20is%20dangerous

History
=======

v3: https://lore.proxmox.com/pdm-devel/20260403165437.2166551-1-c.heiss@proxmox.com/
v2: https://lore.proxmox.com/pdm-devel/20251205112528.373387-1-c.heiss@proxmox.com/
v1: https://lore.proxmox.com/pdm-devel/20251204125122.945961-1-c.heiss@proxmox.com/

Notable Changes v3 -> v4:
  * answer tokens are now automatically created when preparing a new
    answer, if no existing one is selected
  * use minijinja instead of handlebars as templating engine
  * add some more validation to some ui form fields
  * .. and udev filter/template counters in the backend
  * fix root password on prepared answer creation
  * add "Authorized tokens" column to prepared answers panel
  * use `Bearer` instead of `ProxmoxInstallerToken` in `Authorization`
    HTTP header

Notable Changes v2 -> v3:
  * moved "Automated Installations" panels to a tab under "Remotes"
  * added authentication to answer endpoint
  * answers are now served as JSON unconditionally
  * moved installations state file to
    /var/lib/proxmox-datacenter-manager/
  * added required pve-installer changes
  * ui: use `DataTable`s instead of text areas for filter entries
  * ui: use `PdmClient` for interacting with API instead of sending
    requests directly

Notable Changes v1 -> v2:
  * add documentation patch
  * fixed compilation due to leftover type

Diffstat
========

proxmox:

Christoph Heiss (12):
  api-macro: allow $ in identifier name
  schema: oneOf: allow single string variant
  schema: implement UpdaterType for HashMap and BTreeMap
  network-types: move `Fqdn` type from proxmox-installer-common
  network-types: implement api type for Fqdn
  network-types: add api wrapper type for std::net::IpAddr
  network-types: cidr: implement generic `IpAddr::new` constructor
  network-types: fqdn: implement standard library Error for Fqdn
  node-status: make KernelVersionInformation Clone + PartialEq
  installer-types: add common types used by the installer
  installer-types: add types used by the auto-installer
  installer-types: implement api type for all externally-used types

 Cargo.toml                                   |    2 +
 proxmox-api-macro/src/util.rs                |    2 +-
 proxmox-installer-types/Cargo.toml           |   28 +
 proxmox-installer-types/debian/changelog     |    5 +
 proxmox-installer-types/debian/control       |   67 +
 proxmox-installer-types/debian/debcargo.toml |    7 +
 proxmox-installer-types/src/answer.rs        | 1199 ++++++++++++++++++
 proxmox-installer-types/src/lib.rs           |  182 +++
 proxmox-installer-types/src/post_hook.rs     |  215 ++++
 proxmox-network-types/Cargo.toml             |    3 +-
 proxmox-network-types/debian/control         |    8 +-
 proxmox-network-types/src/fqdn.rs            |  257 ++++
 proxmox-network-types/src/ip_address.rs      |   73 +-
 proxmox-network-types/src/lib.rs             |    1 +
 proxmox-node-status/src/types.rs             |    2 +-
 proxmox-schema/src/schema.rs                 |   78 +-
 proxmox-schema/tests/schema.rs               |   91 +-
 17 files changed, 2207 insertions(+), 13 deletions(-)
 create mode 100644 proxmox-installer-types/Cargo.toml
 create mode 100644 proxmox-installer-types/debian/changelog
 create mode 100644 proxmox-installer-types/debian/control
 create mode 100644 proxmox-installer-types/debian/debcargo.toml
 create mode 100644 proxmox-installer-types/src/answer.rs
 create mode 100644 proxmox-installer-types/src/lib.rs
 create mode 100644 proxmox-installer-types/src/post_hook.rs
 create mode 100644 proxmox-network-types/src/fqdn.rs

proxmox-yew-comp:

Christoph Heiss (1):
  widget: kvlist: add widget for user-modifiable data tables

 src/key_value_list.rs | 348 ++++++++++++++++++++++++++++++++++++++++++
 src/lib.rs            |   3 +
 2 files changed, 351 insertions(+)
 create mode 100644 src/key_value_list.rs

proxmox-datacenter-manager:

Christoph Heiss (11):
  api-types, cli: use ReturnType::new() instead of constructing it
    manually
  api-types: add api types for auto-installer integration
  config: add auto-installer configuration module
  acl: wire up new /system/auto-installation acl path
  server: api: add auto-installer integration module
  server: api: auto-installer: add access token management endpoints
  client: add bindings for auto-installer endpoints
  ui: auto-installer: add installations overview panel
  ui: auto-installer: add prepared answer configuration panel
  ui: auto-installer: add access token configuration panel
  docs: add documentation for auto-installer integration

 Cargo.toml                                    |    6 +
 cli/client/src/pbs.rs                         |   10 +-
 cli/client/src/pve.rs                         |   15 +-
 cli/client/src/remotes.rs                     |    5 +-
 debian/control                                |    5 +
 docs/automated-installations.rst              |  146 ++
 docs/index.rst                                |    1 +
 lib/pdm-api-types/Cargo.toml                  |    3 +
 lib/pdm-api-types/src/acl.rs                  |    4 +-
 lib/pdm-api-types/src/auto_installer.rs       |  458 ++++++
 lib/pdm-api-types/src/lib.rs                  |   10 +-
 lib/pdm-buildcfg/src/lib.rs                   |   10 +
 lib/pdm-client/src/lib.rs                     |  245 ++++
 lib/pdm-config/Cargo.toml                     |    3 +
 lib/pdm-config/src/auto_install.rs            |  575 ++++++++
 lib/pdm-config/src/lib.rs                     |    1 +
 lib/pdm-config/src/setup.rs                   |    7 +
 server/Cargo.toml                             |    5 +
 server/src/api/auto_installer/mod.rs          | 1259 +++++++++++++++++
 server/src/api/mod.rs                         |    2 +
 ui/Cargo.toml                                 |    2 +
 .../configuration/permission_path_selector.rs |    1 +
 .../auto_installer/installations_panel.rs     |  303 ++++
 ui/src/remotes/auto_installer/mod.rs          |   86 ++
 .../prepared_answer_add_wizard.rs             |  219 +++
 .../prepared_answer_edit_window.rs            |  217 +++
 .../auto_installer/prepared_answer_form.rs    | 1089 ++++++++++++++
 .../auto_installer/prepared_answers_panel.rs  |  311 ++++
 ui/src/remotes/auto_installer/token_panel.rs  |  420 ++++++
 .../remotes/auto_installer/token_selector.rs  |  159 +++
 ui/src/remotes/mod.rs                         |   10 +
 31 files changed, 5557 insertions(+), 30 deletions(-)
 create mode 100644 docs/automated-installations.rst
 create mode 100644 lib/pdm-api-types/src/auto_installer.rs
 create mode 100644 lib/pdm-config/src/auto_install.rs
 create mode 100644 server/src/api/auto_installer/mod.rs
 create mode 100644 ui/src/remotes/auto_installer/installations_panel.rs
 create mode 100644 ui/src/remotes/auto_installer/mod.rs
 create mode 100644 ui/src/remotes/auto_installer/prepared_answer_add_wizard.rs
 create mode 100644 ui/src/remotes/auto_installer/prepared_answer_edit_window.rs
 create mode 100644 ui/src/remotes/auto_installer/prepared_answer_form.rs
 create mode 100644 ui/src/remotes/auto_installer/prepared_answers_panel.rs
 create mode 100644 ui/src/remotes/auto_installer/token_panel.rs
 create mode 100644 ui/src/remotes/auto_installer/token_selector.rs

pve-installer:

Christoph Heiss (16):
  install: iso env: use JSON boolean literals for product config
  common: http: allow passing custom headers to post()
  common: http: retrieve error message from body on post()
  common: options: move regex construction out of loop
  assistant: support adding an authorization token for HTTP-based
    answers
  post-hook: run cargo fmt
  tree-wide: used moved `Fqdn` type to proxmox-network-types
  tree-wide: use `Cidr` type from proxmox-network-types
  tree-wide: switch to filesystem types from proxmox-installer-types
  auto: sysinfo: switch to types from proxmox-installer-types
  fetch-answer: switch to types from proxmox-installer-types
  fetch-answer: http: prefer json over toml for answer format
  fetch-answer: send auto-installer HTTP authorization token if set
  fetch-answer: print full error messages when fetching failed
  tree-wide: switch out `Answer` -> `AutoInstallerConfig` types
  auto: drop now-dead answer file definitions

 Cargo.toml                                    |   8 +
 Proxmox/Install/ISOEnv.pm                     |  16 +-
 proxmox-auto-install-assistant/Cargo.toml     |   1 +
 proxmox-auto-install-assistant/src/main.rs    |  35 +-
 proxmox-auto-installer/Cargo.toml             |   2 +
 proxmox-auto-installer/src/answer.rs          | 499 -------------
 .../src/bin/proxmox-auto-installer.rs         |  20 +-
 proxmox-auto-installer/src/lib.rs             |   2 -
 proxmox-auto-installer/src/sysinfo.rs         |  91 +--
 proxmox-auto-installer/src/udevinfo.rs        |  11 -
 proxmox-auto-installer/src/utils.rs           | 154 ++--
 proxmox-auto-installer/tests/parse-answer.rs  |   6 +-
 .../tests/resources/iso-info.json             |   4 +-
 .../ipv4_and_subnet_mask_33.json              |   2 +-
 ...rface_pinning_overlong_interface_name.json |   2 +-
 proxmox-chroot/Cargo.toml                     |   1 +
 proxmox-chroot/src/main.rs                    |  60 +-
 proxmox-fetch-answer/Cargo.toml               |   2 +-
 .../src/fetch_plugins/http.rs                 | 116 ++-
 proxmox-fetch-answer/src/main.rs              |  21 +-
 proxmox-installer-common/Cargo.toml           |   2 +
 proxmox-installer-common/src/disk_checks.rs   |   5 +-
 proxmox-installer-common/src/dmi.rs           |  43 ++
 proxmox-installer-common/src/http.rs          |  85 ++-
 proxmox-installer-common/src/lib.rs           |   6 +-
 proxmox-installer-common/src/options.rs       | 365 +++------
 proxmox-installer-common/src/setup.rs         | 103 +--
 proxmox-installer-common/src/sysinfo.rs       |  52 --
 proxmox-installer-common/src/utils.rs         | 382 ----------
 proxmox-post-hook/Cargo.toml                  |   4 +-
 proxmox-post-hook/src/main.rs                 | 691 +++++++-----------
 proxmox-tui-installer/Cargo.toml              |   2 +
 proxmox-tui-installer/src/main.rs             |  12 +-
 proxmox-tui-installer/src/options.rs          |  23 +-
 proxmox-tui-installer/src/setup.rs            |   5 +-
 proxmox-tui-installer/src/views/bootdisk.rs   |  44 +-
 proxmox-tui-installer/src/views/mod.rs        |  21 +-
 proxmox-tui-installer/src/views/network.rs    |  15 +-
 38 files changed, 824 insertions(+), 2089 deletions(-)
 delete mode 100644 proxmox-auto-installer/src/answer.rs
 delete mode 100644 proxmox-auto-installer/src/udevinfo.rs
 create mode 100644 proxmox-installer-common/src/dmi.rs
 delete mode 100644 proxmox-installer-common/src/sysinfo.rs
 delete mode 100644 proxmox-installer-common/src/utils.rs

Summary over all repositories:
  88 files changed, 8939 insertions(+), 2132 deletions(-)

--