[pdm-devel] [RFC network/proxmox{, -backup, -api-types, -yew-comp, -datacenter-manager} v2 00/32] Add initial SDN / EVPN integration

[Stefan Hanreich <s.hanreich@proxmox.com>]

## Introduction

This patch series adds a new panel to the PDM that shows an overview of the
current state of all EVPN zones across all remotes. It includes two different
tree views:

* IP-VRFs: that shows the contents of all IP-VRFs (identified by their Route
  Target = ASN:VNI) across all remotes.
* Zones: that shows the contents of a specific zone on a specific remote.

For more information on the two tree views, consult the respective commits that
introduce the components.

The panel also allows users to create new Zones / VNets on multiple remotes
simultaneously by utilizing the new SDN locking functionality.

I have provided prebuilt packages on the share in the folder pdm-evpn


## API

It introduces the following API endpoints on PDM:

/sdn
    GET /controllers - list the controllers of all remotes
    GET /zones - list the zones of all remotes
    POST /zones - create a zone on multiple remotes
    GET /vnets - list the vnets of all remotes
    POST /vnets - create a vnet on multiple remotes


## Additional remarks

This patch series contains some preparatory patches that are not directly
related to the implemented functionality:

* One fix for proxmox-schema so values that are larger than i32 can be used in
  the integer schema definition (required for e.g. 64-bit ASNs)
* Add JSONSchema to a lot of SDN API endpoints that were previously undocumented

I have sent them initially as separate patch series, but since they are a hard
requirement for this patch series I have merged all of them into one patch
series now. This way it is easier to keep track of the requirements.


## Open questions for reviewers

* The LockedSdnClient(s) are abstractions for locked SDN remotes. I'm still a
bit unsure about the design / implementation but for future features I will be
making more complex changes across multiple remotes so I figured an abstraction
for this will come in handy in the future.

I'd love some inputs / opinions on the API design as well as the general concept
of locking config -> making changes -> rolling back / applying.

I will work on a more sophisticated implementation utilizing tokio-specific
functions in the following days, but I wanted to get the patch series out now
and validate the API / general idea.

* We might wanna move the EvpnRouteTarget type out of the UI, even though it is
currently only used there.

* Should we introduce a caching mechanism for the SDN API calls?

I have shortly talked about this with @Lukas, but we decided against
implementing such a mechanism for now after some deliberation.

Showing outdated information is particularly problematic with configuration,
especially because the create dialogues rely on that information.

After creating a new zone / vnet we would have to hit the remotes anyway, in
order to be able to show the updated data immediately.

The downside is of course a long load time for the EVPN panel, as well as a long
load if even one of the remotes is not available.

For an initial release I think it is fine to go forward without caching and see
how it works out in practice based on reports from our users. Any input on this
matter would be greatly appreciated!


## Future Work
* show the output of the new status API calls created by Gabriel in the views.
* add a functionality for grouping remotes together, instead of implicitly
  grouping them based on ASN:VNI
* introduce a caching mechanism for the SDN API calls (?)
* integration tests with mocked SDN clients
* add some QoL to the UI (e.g expand/collapse all)


## Changelog

Changes since the RFC v1:
* overhauled the structure of the trees completely
  * split the initial tree view into two distinct tree views
  * changed the grouping of elements
  * improved and unified the terms used across all UI elements
* improved toolbar design
* removed the controller data table, since the tree views should now include
  that information
* improved locked SDN client and added a collection type for locked SDN clients
* improved error handling and logging considerably for the worker tasks


## Dependencies:
pbs-api-types depends on proxmox-schema
proxmox-backup depends on proxmox-schema
proxmox-datacenter-manager depends on proxmox-schema

proxmox-api-types depends on pve-network
proxmox-datacenter-manager depends on proxmox-api-types
proxmox-datacenter-manager depends on proxmox-yew-comp

proxmox:

Stefan Hanreich (2):
  schema: use i64 for minimum / maximum / default integer values
  pbs-api-types: fix values for integer schemas

 pbs-api-types/src/datastore.rs  |  6 +++---
 proxmox-schema/src/de/mod.rs    |  3 +--
 proxmox-schema/src/de/verify.rs | 13 ++++++++-----
 proxmox-schema/src/schema.rs    | 18 +++++++++---------
 4 files changed, 21 insertions(+), 19 deletions(-)


proxmox-backup:

Stefan Hanreich (1):
  api: change integer schema parameters to i64

 pbs-tape/src/bin/pmt.rs           |  6 +++---
 proxmox-backup-client/src/main.rs |  2 +-
 pxar-bin/src/main.rs              |  6 +++---
 src/api2/backup/upload_chunk.rs   | 15 ++++++---------
 4 files changed, 13 insertions(+), 16 deletions(-)


pve-network:

Stefan Hanreich (6):
  sdn: api: return null for rollback / lock endpoints
  controllers: fix maximum value for ASN
  api: add state standard option
  api: controllers: update schema of endpoints
  api: vnets: update schema of endpoints
  api: zones: update schema of endpoints

 src/PVE/API2/Network/SDN.pm                   |   4 +
 src/PVE/API2/Network/SDN/Controllers.pm       | 116 +++++++++-
 src/PVE/API2/Network/SDN/Vnets.pm             |  92 +++++++-
 src/PVE/API2/Network/SDN/Zones.pm             | 203 ++++++++++++++++--
 src/PVE/Network/SDN.pm                        |  10 +
 src/PVE/Network/SDN/Controllers/BgpPlugin.pm  |   7 +-
 src/PVE/Network/SDN/Controllers/EvpnPlugin.pm |   2 +-
 src/PVE/Network/SDN/Controllers/IsisPlugin.pm |   6 +-
 src/PVE/Network/SDN/VnetPlugin.pm             |  21 +-
 src/PVE/Network/SDN/Zones/EvpnPlugin.pm       |  22 +-
 src/PVE/Network/SDN/Zones/QinQPlugin.pm       |   6 +-
 src/PVE/Network/SDN/Zones/VlanPlugin.pm       |   1 +
 src/PVE/Network/SDN/Zones/VxlanPlugin.pm      |  15 +-
 13 files changed, 457 insertions(+), 48 deletions(-)


proxmox-api-types:

Stefan Hanreich (7):
  add QemuMigratePreconditionsNotAllowedNodesBlockingHaResources struct
  sdn: add list/create zone endpoints
  sdn: add list/create vnet endpoints
  sdn: add list/create controller endpoints
  sdn: add sdn configuration locking endpoints
  tasks: add helper for querying successfully finished tasks
  sdn: add helpers for pending values

 pve-api-types/generate.pl      | 40 ++++++++++++++++++++++++++++++++++
 pve-api-types/src/lib.rs       |  1 +
 pve-api-types/src/sdn.rs       | 33 ++++++++++++++++++++++++++++
 pve-api-types/src/types/mod.rs |  4 ++++
 4 files changed, 78 insertions(+)
 create mode 100644 pve-api-types/src/sdn.rs


proxmox-yew-comp:

Stefan Hanreich (1):
  sdn: add descriptions for sdn tasks

 src/utils.rs | 3 +++
 1 file changed, 3 insertions(+)


proxmox-datacenter-manager:

Stefan Hanreich (15):
  server: add locked sdn client helpers
  api: sdn: add list_zones endpoint
  api: sdn: add create_zone endpoint
  api: sdn: add list_vnets endpoint
  api: sdn: add create_vnet endpoint
  api: sdn: add list_controllers endpoint
  ui: sdn: add EvpnRouteTarget type
  ui: sdn: add vnet icon
  ui: sdn: add remote tree component
  ui: add view for showing ip vrfs
  ui: sdn: add AddVnetWindow component
  ui: sdn: add AddZoneWindow component
  ui: sdn: add EvpnPanel
  ui: sdn: add EvpnPanel to main menu
  pve: sdn: add descriptions for sdn tasks

 lib/pdm-api-types/Cargo.toml      |   2 +
 lib/pdm-api-types/src/lib.rs      |   2 +
 lib/pdm-api-types/src/sdn.rs      | 168 +++++++++++++
 lib/pdm-client/src/lib.rs         |  61 +++++
 server/src/api/mod.rs             |   2 +
 server/src/api/sdn/controllers.rs |  78 ++++++
 server/src/api/sdn/mod.rs         |  17 ++
 server/src/api/sdn/vnets.rs       | 149 +++++++++++
 server/src/api/sdn/zones.rs       | 174 +++++++++++++
 server/src/lib.rs                 |   1 +
 server/src/sdn_client.rs          | 387 ++++++++++++++++++++++++++++
 ui/css/pdm.scss                   |  14 +-
 ui/images/icon-sdn-vnet.svg       |   6 +
 ui/src/lib.rs                     |   2 +
 ui/src/main_menu.rs               |  10 +
 ui/src/sdn/evpn/add_vnet.rs       | 273 ++++++++++++++++++++
 ui/src/sdn/evpn/add_zone.rs       | 281 +++++++++++++++++++++
 ui/src/sdn/evpn/evpn_panel.rs     | 224 +++++++++++++++++
 ui/src/sdn/evpn/mod.rs            |  41 +++
 ui/src/sdn/evpn/remote_tree.rs    | 403 ++++++++++++++++++++++++++++++
 ui/src/sdn/evpn/vrf_tree.rs       | 345 +++++++++++++++++++++++++
 ui/src/sdn/mod.rs                 |   1 +
 ui/src/tasks.rs                   |   2 +
 23 files changed, 2642 insertions(+), 1 deletion(-)
 create mode 100644 lib/pdm-api-types/src/sdn.rs
 create mode 100644 server/src/api/sdn/controllers.rs
 create mode 100644 server/src/api/sdn/mod.rs
 create mode 100644 server/src/api/sdn/vnets.rs
 create mode 100644 server/src/api/sdn/zones.rs
 create mode 100644 server/src/sdn_client.rs
 create mode 100644 ui/images/icon-sdn-vnet.svg
 create mode 100644 ui/src/sdn/evpn/add_vnet.rs
 create mode 100644 ui/src/sdn/evpn/add_zone.rs
 create mode 100644 ui/src/sdn/evpn/evpn_panel.rs
 create mode 100644 ui/src/sdn/evpn/mod.rs
 create mode 100644 ui/src/sdn/evpn/remote_tree.rs
 create mode 100644 ui/src/sdn/evpn/vrf_tree.rs
 create mode 100644 ui/src/sdn/mod.rs


Summary over all repositories:
  49 files changed, 3214 insertions(+), 84 deletions(-)

-- 
Generated by git-murpp 0.8.0

_______________________________________________
pdm-devel mailing list
pdm-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pdm-devel