[PVE-User] PVE-firewall and multicast with linux bridging

[PVE-User] PVE-firewall and multicast with linux bridging

[Bryan Fields]

I've run into this multicast issue.  Even with the interface as unfirewalled, 
proxmox is blocking multicast/invalid traffic at the server and datacenter level.

I have the following VM's and LXC's all attached to an interface vmbr46.

100.120.255.128/28
100.120.255.129 - vrrp gateway
100.120.255.130 - Core 1 (router VM)
100.120.255.131 - Core 2 (router VM)
100.120.255.132 - NMS LXC
100.120.255.133 - vm0 - debian testing VM
100.120.255.134 - vm1 - debian testing VM
100.120.255.135 - Hypervisor vmbr46

This is an isolated bridge on linux, and is only used for testing of these 
servers/multicast network.  None of the ports have the firewall enabled.

During testing, I've had PIM between the routers come up and several weird 
groups back and forth.  I first assumed it was the fact I was testing from an 
LXC, and made the VMs.  This was not the case, as the VM's would have the same 
issues of only some ICMP pings to the multicast addresses working and testing 
with socat showed one way multicast between the Hypervisor and one VM.

After much mocking this up on another host and locally with real servers, I 
was able to isolate it to the bridge device.  There was not firewall logs for 
any of this, and pings to 224.0.0.1 wouldn't even work.  This is the all 
multicast address, everything that is participating in multicast should reply.

I configured the vmbr46 as 100.120.255.135/28 on the hypervisor to test this. 
I had the management firewall on the hypervisor disabled and confirmed I 
wasn't seeing any drops in the logs.  eventually after troubleshooting this, I 
discovered there is a built in rule that blocks BROADCAST, MULTICAST, and 
ANYCAST across all interfaces, even though it's not on the forward chain. 
Owing to how multicast is handled on the bridge, it appears the INPUT chain is 
filtering this.

This is in the rules, and appears to be harcoded in
/usr/share/perl5/PVE/Firewall.pm
-A PVEFW-DropBroadcast -m addrtype --dst-type BROADCAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type MULTICAST -j DROP
-A PVEFW-DropBroadcast -m addrtype --dst-type ANYCAST -j DROP
-A PVEFW-DropBroadcast -d 224.0.0.0/4 -j DROP

I'd be ok with the hypervisor not being able to talk directly to the VMs on 
the vmbr46 interface, but I need the VMs and CT's to pass multicast with each 
other.  Is there some way to exempt an interface totally from filtering?

-- 
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] PVE-firewall and multicast with linux bridging

[Bryan Fields]

On 6/22/25 02:22, Bryan Fields wrote:
> I've run into this multicast issue.  Even with the interface as unfirewalled,
> proxmox is blocking multicast/invalid traffic at the server and datacenter level.

did more digging on this using the TRACE tool in iptables.  I believe it's 
100% a bug at this point, as there is no way to disable these rules other than 
to disable the firewall at the cluster level.  Disabling the entire firewall 
is unacceptable.

I've updated this and filed a bug here:

https://bugzilla.proxmox.com/show_bug.cgi?id=6478


-- 
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] PVE-firewall and multicast with linux bridging

[Bryan Fields]

I've got somewhat of a work around, as it needs to be applied manually each
time the firewall is reset.

Example here is the devices I want to have this enabled on, and then the first
command replaces the first rule and then the next insert the following rules
at 2 in the chain.

iptables -R PVEFW-FORWARD 1 -m conntrack --ctstate INVALID --in-interface vmbr8 -j DROP
iptables -I PVEFW-FORWARD 2 -m conntrack --ctstate INVALID --in-interface vmbr44 -j DROP
iptables -I PVEFW-FORWARD 2 -m conntrack --ctstate INVALID --in-interface vmbr45 -j DROP
iptables -I PVEFW-FORWARD 2 -m conntrack --ctstate INVALID --in-interface vmbr192 -j DROP
iptables -I PVEFW-FORWARD 2 -m conntrack --ctstate INVALID --in-interface vmbr199 -j DROP

As there's no way to exclude multiple interfaces on the iptables command, the
only way to do this is white list interfaces.  This should really be how
proxmox does it, asking about connection tracking at the per bridge
level.  I do want it on some of the bridges, but on others, it needs to be
optional.

I'm frankly surprised that there's no one else who's run into this as it
appears many issues are caused by this.
-- 
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] PVE-firewall and multicast with linux bridging

[g.husson_proxmox-pve-user--- via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 5471 bytes --]

From: g.husson_proxmox-pve-user@liberasys.com
To: pve-user@lists.proxmox.com
Subject: Re: [PVE-User] PVE-firewall and multicast with linux bridging
Date: Mon, 30 Jun 2025 08:16:48 +0200
Message-ID: <e9552fb7-d058-4f60-8709-48a2a07dcac3@liberasys.com>

Hello Bryan,

"It is not a bug, it is a feature" :-)
Look at the documentation :
===
The following traffic is dropped, but not logged even with logging enabled:
- Broadcast, multicast and anycast traffic not related to corosync, 
i.e., not coming through ports 5405-5412
===

Again, from the documentation :
===
proxmox-firewall will create two tables that are managed by the 
proxmox-firewall service: proxmox-firewall and proxmox-firewall-guests. 
If you want to create custom rules that live outside the Proxmox VE 
firewall configuration you can create your own tables to manage your 
custom firewall rules. proxmox-firewall will only touch the tables it 
generates, so you can easily extend and modify the behavior of the 
proxmox-firewall by adding your own tables.
===

Now you can use rc.local, or crontab @reboot or better a systemd file 
that chains after proxmox VE firewall start in order to apply the manual 
rules you found.

Best regards,
Gautier Husson.


On 29/06/2025 10:14, Bryan Fields wrote:
> I've got somewhat of a work around, as it needs to be applied manually 
> each
> time the firewall is reset.
>
> Example here is the devices I want to have this enabled on, and then 
> the first
> command replaces the first rule and then the next insert the following 
> rules
> at 2 in the chain.
>
> iptables -R PVEFW-FORWARD 1 -m conntrack --ctstate INVALID 
> --in-interface vmbr8 -j DROP
> iptables -I PVEFW-FORWARD 2 -m conntrack --ctstate INVALID 
> --in-interface vmbr44 -j DROP
> iptables -I PVEFW-FORWARD 2 -m conntrack --ctstate INVALID 
> --in-interface vmbr45 -j DROP
> iptables -I PVEFW-FORWARD 2 -m conntrack --ctstate INVALID 
> --in-interface vmbr192 -j DROP
> iptables -I PVEFW-FORWARD 2 -m conntrack --ctstate INVALID 
> --in-interface vmbr199 -j DROP
>
> As there's no way to exclude multiple interfaces on the iptables 
> command, the
> only way to do this is white list interfaces.  This should really be how
> proxmox does it, asking about connection tracking at the per bridge
> level.  I do want it on some of the bridges, but on others, it needs 
> to be
> optional.
>
> I'm frankly surprised that there's no one else who's run into this as it
> appears many issues are caused by this.




[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] PVE-firewall and multicast with linux bridging

[Bryan Fields]

On 6/30/25 2:16 AM, g.husson_proxmox-pve-user--- via pve-user wrote:
> "It is not a bug, it is a feature" :-)
> Look at the documentation :
> ===
> The following traffic is dropped, but not logged even with logging enabled:
> - Broadcast, multicast and anycast traffic not related to corosync, 
> i.e., not coming through ports 5405-5412
> ===
> 
> Again, from the documentation :
> ===
> proxmox-firewall will create two tables that are managed by the proxmox- 
> firewall service: proxmox-firewall and proxmox-firewall-guests. If you 
> want to create custom rules that live outside the Proxmox VE firewall 
> configuration you can create your own tables to manage your custom 
> firewall rules. proxmox-firewall will only touch the tables it 
> generates, so you can easily extend and modify the behavior of the 
> proxmox-firewall by adding your own tables.
> ===

None of this mentions that connection tracking is enabled globally, even 
on interfaces that are not firewalled.  It's an undocumented "feature".

This kills multicast traffic globally, and owing to connection tracking 
being unable to match traffic.

> Chain PVEFW-FORWARD (1 references)
> num  target     prot opt source               destination
> 1    DROP       0    --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID

> Now you can use rc.local, or crontab @reboot or better a systemd file 
> that chains after proxmox VE firewall start in order to apply the manual 
> rules you found.

Given how iptables works, I can make a new table and insert it before 
the PVEFW-FORWARD chain.  However there is no way to negate a later rule 
other than by allowing/forwarding it, which I may not want to do 'permit 
any any' on an interface globally.   The only option is to edit the 
PVEFW-FORWARD chain directly, but this will get overwritten on reboots 
and when the firewall settings are changed in pve.

If there's a way to kick off a script when pve-firewall updates, this 
would be an option, but the better option would be to fix this so it can 
be enabled on a per interface basis, rather than all or none.

-- 
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user