[PATCH pve-docs v2 11/11] sdn: add example for IPv6 in an EVPN zone

[Hannes Laimer <h.laimer@proxmox.com>]

End-to-end walkthrough of an EVPN zone serving IPv6, with a /64 SLAAC
subnet and a /96 announce-only subnet, to show how the per-vnet RA
toggle and per-prefix overrides interact in practice.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
---
 pvesdn.adoc | 216 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 216 insertions(+)

diff --git a/pvesdn.adoc b/pvesdn.adoc
index 74ce953..2618506 100644
--- a/pvesdn.adoc
+++ b/pvesdn.adoc
@@ -1551,6 +1551,222 @@ can reply back.
 If you have configured an external BGP router, the BGP-EVPN routes (10.0.1.0/24
 and 10.0.2.0/24 in this example), will be announced dynamically.
 
+[[pvesdn_setup_example_evpn_ipv6]]
+EVPN with IPv6 Setup Example
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The example assumes a cluster with three nodes (node1, node2, node3) with P2P
+connections between `node1<->node2` and `node2<->node3`. Additionally we'll
+also need an `/48` IPv6 prefix routed to the exit node.
+
+We will start by creating an OpenFabric which will be the underlay for our EVPN
+zone. Then we'll set up a VNet with two subnets, one with SLAAC and a `/96` one
+with RA enabled and statically assigned IPs.
+
+.Prefixes
+. to `node2` routed `/48`: `2001:db8:abcd::/48`
+. `/64` subnet with SLAAC: `2001:db8:abcd:1::/64`
+. `/96` subnet without SLAAC: `2001:db8:abcd:2::/96`
+
+.IPv6 forwarding
+This has to be enabled on all nodes.
+----
+sysctl -w net.ipv6.conf.all.forwarding=1
+----
+
+NOTE: The underlay will be IPv4 based, the overlay IPv6.
+
+Setup `OpenFabric`
+^^^^^^^^^^^^^^^^^^
+
+. Create the fabric with `10.0.0.0/24` as its IPv4 prefix and `underlay` as its
+  name, we can leave the defaults for the rest
+. Add the three nodes to the fabric,
+.. `node1` with `10.0.0.1`, select P2P NIC between `node1<->node2`
+.. `node2` with `10.0.0.2`, select both P2P NICs between `node1<->node2` and
+   `node2<->node3`
+.. `node3` with `10.0.0.3`, select P2P NIC between `node2<->node3`
+. Apply the changes
+
+The resulting config in `/etc/pve/sdn/fabrics.cfg` should look something like
+this:
+
+----
+openfabric_fabric: underlay
+	ip_prefix 10.0.0.0/24
+
+openfabric_node: underlay_node1
+	interfaces name=ens19
+	ip 10.0.0.1
+
+openfabric_node: underlay_node2
+	interfaces name=ens20
+	interfaces name=ens19
+	ip 10.0.0.2
+
+openfabric_node: underlay_node3
+	interfaces name=ens19
+	ip 10.0.0.3
+----
+
+Each node should now also be able to reach all of the others, this can quickly
+be checked with a
+
+----
+ping 10.0.0.X
+----
+
+With `ip -4 r` you should also be able to see the respective `onlink` routes.
+
+----
+10.0.0.2 nhid 28 via 10.0.0.2 dev ens19 proto openfabric src 10.0.0.1 metric 20 onlink
+10.0.0.3 nhid 28 via 10.0.0.2 dev ens19 proto openfabric src 10.0.0.1 metric 20 onlink
+----
+
+If this works we can continue.
+
+Create controller
+^^^^^^^^^^^^^^^^^
+
+Create an EVPN controller, keep `65000` as `ASN#` and select the `underlay`
+fabric we have just created. For this we'll name the controller `v6ctl`. Then
+apply the changes.
+
+The relevant config here is `/etc/pve/sdn/controllers.cfg`, and should look
+like this:
+
+----
+evpn: v6ctl
+	asn 65000
+	fabric underlay
+----
+
+Setup the zone
+^^^^^^^^^^^^^^
+
+Create a new EVPN zone, select the `v6ctl` as its controller, set the
+`VRF_VXLAN Tag` to `6` and name it `v6zone`. As exit node select the node that
+the `/48` prefix is routed to. Then apply the changes.
+
+The relevant config here is `/etc/pve/sdn/zones.cfg`, and after applying should
+look something like this:
+
+----
+evpn: v6zone
+	controller v6ctl
+	vrf-vxlan 6
+	exitnodes node2
+	ipam pve
+	mac BC:24:11:F2:72:B8
+----
+
+
+Create the VNets
+^^^^^^^^^^^^^^^^
+
+.The bigger `/64` with SLAAC
+Create a new VNet, name it `net64`, select `v6zone` as zone, set `64` as tag,
+and under "IPv6 Router Advertisement" check `Send Router Advertisements`.
+
+.The smaller `/96` without SLAAC and static IPs
+Create a new VNet, name it `net96`, select `v6zone` as zone, set `96` as tag,
+and under "IPv6 Router Advertisement" check `Send Router Advertisements`.
+
+After applying `/etc/pve/sdn/vnets.cfg` should contain:
+
+----
+vnet: net64
+	zone v6zone
+	tag 64
+	ipv6-ra 1
+
+vnet: net96
+	zone v6zone
+	tag 96
+	ipv6-ra 1
+----
+
+
+Setup the subnets
+^^^^^^^^^^^^^^^^^
+
+First we'll configure the `/64` subnet in `net64`. The defaults already enable
+SLAAC, so no overrides are needed:
+
+. Select the `net64` vnet
+. Create subnet with `2001:db8:abcd:1::/64` and `2001:db8:abcd:1::1/64` as
+  gateway
+
+Then configure the `/96` subnet in `net96`. SLAAC is only valid on `/64`, so
+under "IPv6 Prefix Options" uncheck `SLAAC (A)`:
+
+. Select the `net96` vnet
+. Create subnet with `2001:db8:abcd:2::/96` and `2001:db8:abcd:2::1/96` as
+  gateway
+. Under "IPv6 Prefix Options" uncheck `SLAAC (A)`
+
+NOTE: With RAs enabled a default gateway is advertised so we don't have to
+specify a gateway explicitly for guests with a static IP. The vnet's RDNSS list
+is also advertised, which can be handy because usually we'd need a whole DHCP
+server for that. Whether the advertised DNS is used is somewhat client
+dependent, as sometimes this option is ignored.
+
+The relevant config for these is `/etc/pve/sdn/subnets.cfg`, and after applying
+these changes should contain
+
+----
+subnet: v6zone-2001:db8:abcd:1::-64
+	vnet net64
+	gateway 2001:db8:abcd:1::1
+
+subnet: v6zone-2001:db8:abcd:2::-96
+	vnet net96
+	gateway 2001:db8:abcd:2::1
+	nd-prefix-autonomous 0
+----
+
+Result
+^^^^^^
+
+Now we can create two guests to test this, we'll use Debian CTs here, but any
+guest will do.
+
+. configure first guest
+.. select `net64` as bridge
+.. leave the IPv4 section `static` and the IP field empty
+.. for IPv6 select `SLAAC`
+. configure second guest
+.. select `net96` as bridge
+.. leave the IPv4 section `static` and the IP field empty
+.. for IPv6 select `static` and set `2001:db8:abcd:2::200/96` as `IPv6`
+
+NOTE: We can leave `gateway` empty because we have RAs enabled on `net96`,
+setting it is also not a problem, just not needed.
+
+Both guests should have a default route via the gateway's link-local address,
+`ip -6 r`
+
+----
+default via fe80::be24:11ff:fef2:72b8 dev eth0 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium
+----
+
+This address will match the link-local address on the host's `net64` (or `net96`) interfaces, `ip a show net64`.
+
+----
+7: net64: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master vrf_v6zone state UP group default qlen 1000
+    link/ether bc:24:11:f2:72:b8 brd ff:ff:ff:ff:ff:ff
+    inet6 2001:db8:abcd:1::1/64 scope global
+       valid_lft forever preferred_lft forever
+    inet6 fe80::be24:11ff:fef2:72b8/64 scope link proto kernel_ll
+       valid_lft forever preferred_lft forever
+----
+
+Before the guest's first traffic, `ip -6 neigh show vrf vrf_v6zone` will only
+contain an entry for the guest's link-local address. After the first `ping
+2001:db8::1`, also the guest's GUA should have an entry.
+
+
+
 [[pvesdn_notes]]
 Notes
 -----
-- 
2.47.3