Re: SPAM: [RFC cluster/docs/ifupdown2/manager/network/proxmox{-ve-rs,-ebpf,-perl-rs} v2 00/27] sdn: add microsegmentation support

["Lukas Sichert" <l.sichert@proxmox.com>]

On 2026-07-09 11:18, Hannes Laimer <h.laimer@proxmox.com> wrote:

> This adds support for microsegmentation using eBPF programs attached to
> interfaces. Mostly the tap/veth interfaces on the guests directly.
>

Overall, the microsegmentation setup worked for both VMs and containers
across the three-node EVPN cluster.

What worked:
- Group assignments were applied to QEMU VM TAP interfaces and LXC container veth interfaces.
- Allowed traffic paths worked across nodes:
│   - web <-> app
│   - app <-> db
│   - managed guests <-> gateway/untagged

- Default-denied paths, such as web <->  db, were blocked.
- VM-to-CT and CT-to-VM policy enforcement worked.
- Migration between cluster nodes preserved the expected policy.
- VXLAN-GBP carried the source identity across hosts after the VXLAN
  devices were recreated with gbp.
- A normal SDN reload preserved the VXLAN gbp flag.
- A second SDN apply correctly realized a NIC that had been added after
  the previous render.

A bigger design change I have also already talked with Hannes about already:

In my opinion the any/all/exact architecture is superior to the
hierachical structure from v1. Still, I think hierarchy has some useful
properties. In particular, it allows the group structure to resemble the
internal structure of an organization.

In an example:
Lets say we have
testing
└── database

production
└── database

A testing database usually requires very different rules from a
production database. It may need to communicate with untagged developer
machines, test services, and debugging tools. A production database
should typically communicate only with production application servers
and explicitly approved external services.
A hierarchy makes both systems’ database role visible while clearly
separating their inherited testing and production policies.

The question remains how this can be best integrated into the
any/all/exact architecture.

The most promising way we are currently exploring is deplicating the
policies for the subgroups.

For ALL and EXACT policies, a policy is duplicated with the parent group
replaced by the subgroup. For ANY policies, only occurrences of the
parent group in the ANY part are expanded.
For any duplicated rule the priority gets increased by 0.01.
To not directly creating conflicts it is also important that longer
matching rules get proritized when compared against other policies with
the conflicting rules.
For every group in the all set the priority gets increased by 0.1.
ANY sets get increase by 0.1.

This means if one has
ALL(a,u,v) -> ALL(s,t) allow priority=5
ANY(a,x,y) -> All(r) allow priority=4

and the subgroup a/b
He would get the rules:
ALL(a,u,v) -> ALL(s,t) allow priority=5.5
ALL(a/b,u,v) -> ALL(s,t) allow priority=5.51
ANY(a,x,y) -> All(r) allow priority=4.1
ANY(a/b) -> All(r) allow priority=4.11

This would allow hierarchical groups while retaining the ANY/ALL/EXACT
matching model. It also seems more intuitive than computing
intersections and denying everything outside the intersection.

The problem is, that this can also create confusing behaviour:

Consider
ALL(v,u) -> ALL(s) deny priority=5
All(a,o) -> ALL(x) allow prioriy=5
and subgroup a/b, this would result in:

ALL(v,u) -> ALL(s) deny priority=5.3
All(a,o) -> ALL(x) allow prioriy=5.3
All(a/b,o) -> ALL(x) allow prioriy=5.31

This means a vm with set {a,v,u,o} can't send to a vm with set {s,x},
because allow and deny rules both match equally and we conservatively
then do not allow sending.
But for a vm with set {a/b,v,u,o} it is possible, because the rule with
priority 5.31 overrules the deny rule. This means that the policies for a
and a/b differ even though no additional policies are set.

I don't know if this added complexity is a valid tradeoff for the
additional configurability.

Some other things that could be adjusted in the next version:

1. stale bpf entries after interface

Stopping and restarting a container changed its veth ifindex, but BPF
links and tap_to_group entries for the old ifindex remained pinned.

This is visible with:

root@pve1-cluster:~# cat /sys/class/net/veth300i0/ifindex
 52
root@pve1-cluster:~# ls -1 /sys/fs/bpf/proxmox-ebpf/policy/links
 52-egress
 52-ingress
root@pve1-cluster:~# pct stop 300
│ pct start 300
│ sleep 2
│ cat /sys/class/net/veth300i0/ifindex
 53
root@pve1-cluster:~#  ls -1 /sys/fs/bpf/proxmox-ebpf/policy/links
 52-egress
 52-ingress
 53-egress
 53-ingress

The commands here are exact, the output is shortend for better
readability.

2. Missing assignments fail open as untagged

In a normal workflow, adding a NIC after the SDN render left it absent
from realized. The agent TREATED IT AS UNASSIGNED AND ATTACHED NO BPF
PROGRAMS. THE INTERFACE DOES NOT STAMP A MANAGED- Source identity. Its
packets retain mark 0 and are treated as untagged. This means pings were
possible until a SDN reload.

I think this is because in proxmox-ebpf/src/state.rs the wire id is
called with
let wire_id = registry
          .id_of(&set, &policies)
          .unwrap_or(microseg::identity::UNTAGGED_ID);

and the documentation of id_of states:
Returns `None` only for a set no current class carries..., but as the
nic is created after the ids are set, the id of the set, the nic belongs
to is UNTAGGED_ID.

3. CTs are also named VMx

When adding an assignment for an container, one can only selects the
CT/VM via --vmid 300. This then creates vm300 in the GUI and entries
with "id" : "vm300i1" even though in the cluster/resources it is listed
as "vmid": 300, "type": "lxc". I think it would be nice to rename
containers into ct300.

4. Gui resets when reloading

When adding an assignment in the Gui, the Policy Window resets and jumps back to
the All Policies page. This is even more annoying, because the menu selection
stays at the selected group, so one has to click on a different group
and the on the group that was selected before.