From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id BBAB41FF0A7 for ; Mon, 17 Aug 2026 13:17:16 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 2CEBC23C2D; Mon, 17 Aug 2026 13:17:16 +0200 (CEST) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Mon, 17 Aug 2026 13:17:01 +0200 Message-Id: From: "Lukas Sichert" To: "Hannes Laimer" , Subject: Re: SPAM: [RFC cluster/docs/ifupdown2/manager/network/proxmox{-ve-rs,-ebpf,-perl-rs} v2 00/27] sdn: add microsegmentation support References: <20260709091852.538885-1-h.laimer@proxmox.com> In-Reply-To: <20260709091852.538885-1-h.laimer@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1786965410575 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.108 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust RDNS_NONE 1.274 Delivered to internal network by a host with no rDNS SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: SRXXSWWX4LO7QD2JDAU2HXWYUPX6LTZM X-Message-ID-Hash: SRXXSWWX4LO7QD2JDAU2HXWYUPX6LTZM X-MailFrom: l.sichert@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 2026-07-09 11:18, Hannes Laimer wrote: > This adds support for microsegmentation using eBPF programs attached to > interfaces. Mostly the tap/veth interfaces on the guests directly. > Overall, the microsegmentation setup worked for both VMs and containers across the three-node EVPN cluster. What worked: - Group assignments were applied to QEMU VM TAP interfaces and LXC containe= r veth interfaces. - Allowed traffic paths worked across nodes: =E2=94=82 - web <-> app =E2=94=82 - app <-> db =E2=94=82 - managed guests <-> gateway/untagged - Default-denied paths, such as web <-> db, were blocked. - VM-to-CT and CT-to-VM policy enforcement worked. - Migration between cluster nodes preserved the expected policy. - VXLAN-GBP carried the source identity across hosts after the VXLAN devices were recreated with gbp. - A normal SDN reload preserved the VXLAN gbp flag. - A second SDN apply correctly realized a NIC that had been added after the previous render. A bigger design change I have also already talked with Hannes about already= : In my opinion the any/all/exact architecture is superior to the hierachical structure from v1. Still, I think hierarchy has some useful properties. In particular, it allows the group structure to resemble the internal structure of an organization. In an example: Lets say we have testing =E2=94=94=E2=94=80=E2=94=80 database production =E2=94=94=E2=94=80=E2=94=80 database A testing database usually requires very different rules from a production database. It may need to communicate with untagged developer machines, test services, and debugging tools. A production database should typically communicate only with production application servers and explicitly approved external services. A hierarchy makes both systems=E2=80=99 database role visible while clearly separating their inherited testing and production policies. The question remains how this can be best integrated into the any/all/exact architecture. The most promising way we are currently exploring is deplicating the policies for the subgroups. For ALL and EXACT policies, a policy is duplicated with the parent group replaced by the subgroup. For ANY policies, only occurrences of the parent group in the ANY part are expanded. For any duplicated rule the priority gets increased by 0.01. To not directly creating conflicts it is also important that longer matching rules get proritized when compared against other policies with the conflicting rules. For every group in the all set the priority gets increased by 0.1. ANY sets get increase by 0.1. This means if one has ALL(a,u,v) -> ALL(s,t) allow priority=3D5 ANY(a,x,y) -> All(r) allow priority=3D4 and the subgroup a/b He would get the rules: ALL(a,u,v) -> ALL(s,t) allow priority=3D5.5 ALL(a/b,u,v) -> ALL(s,t) allow priority=3D5.51 ANY(a,x,y) -> All(r) allow priority=3D4.1 ANY(a/b) -> All(r) allow priority=3D4.11 This would allow hierarchical groups while retaining the ANY/ALL/EXACT matching model. It also seems more intuitive than computing intersections and denying everything outside the intersection. The problem is, that this can also create confusing behaviour: Consider ALL(v,u) -> ALL(s) deny priority=3D5 All(a,o) -> ALL(x) allow prioriy=3D5 and subgroup a/b, this would result in: ALL(v,u) -> ALL(s) deny priority=3D5.3 All(a,o) -> ALL(x) allow prioriy=3D5.3 All(a/b,o) -> ALL(x) allow prioriy=3D5.31 This means a vm with set {a,v,u,o} can't send to a vm with set {s,x}, because allow and deny rules both match equally and we conservatively then do not allow sending. But for a vm with set {a/b,v,u,o} it is possible, because the rule with priority 5.31 overrules the deny rule. This means that the policies for a and a/b differ even though no additional policies are set. I don't know if this added complexity is a valid tradeoff for the additional configurability. Some other things that could be adjusted in the next version: 1. stale bpf entries after interface Stopping and restarting a container changed its veth ifindex, but BPF links and tap_to_group entries for the old ifindex remained pinned. This is visible with: root@pve1-cluster:~# cat /sys/class/net/veth300i0/ifindex 52 root@pve1-cluster:~# ls -1 /sys/fs/bpf/proxmox-ebpf/policy/links 52-egress 52-ingress root@pve1-cluster:~# pct stop 300 =E2=94=82 pct start 300 =E2=94=82 sleep 2 =E2=94=82 cat /sys/class/net/veth300i0/ifindex 53 root@pve1-cluster:~# ls -1 /sys/fs/bpf/proxmox-ebpf/policy/links 52-egress 52-ingress 53-egress 53-ingress The commands here are exact, the output is shortend for better readability. 2. Missing assignments fail open as untagged In a normal workflow, adding a NIC after the SDN render left it absent from realized. The agent TREATED IT AS UNASSIGNED AND ATTACHED NO BPF PROGRAMS. THE INTERFACE DOES NOT STAMP A MANAGED- Source identity. Its packets retain mark 0 and are treated as untagged. This means pings were possible until a SDN reload. I think this is because in proxmox-ebpf/src/state.rs the wire id is called with let wire_id =3D registry .id_of(&set, &policies) .unwrap_or(microseg::identity::UNTAGGED_ID); and the documentation of id_of states: Returns `None` only for a set no current class carries..., but as the nic is created after the ids are set, the id of the set, the nic belongs to is UNTAGGED_ID. 3. CTs are also named VMx When adding an assignment for an container, one can only selects the CT/VM via --vmid 300. This then creates vm300 in the GUI and entries with "id" : "vm300i1" even though in the cluster/resources it is listed as "vmid": 300, "type": "lxc". I think it would be nice to rename containers into ct300. 4. Gui resets when reloading When adding an assignment in the Gui, the Policy Window resets and jumps ba= ck to the All Policies page. This is even more annoying, because the menu selecti= on stays at the selected group, so one has to click on a different group and the on the group that was selected before.