[pve-devel] [PATCH pve-cluster] sysctl: disable net.ipv4.igmp_link_local_mcast_reports

[pve-devel] [PATCH pve-cluster] sysctl: disable net.ipv4.igmp_link_local_mcast_reports

[Alexandre Derumier]

currently, when veth or tap interfaces are plugged to bridge,
an igmp v3 report is broadcasted to the network, with the
bridge mac adddress.

Users have reported problems with hetzner for example, blocking the server
because of the unknown mac flooding the network.
https://forum.proxmox.com/threads/proxmox-claiming-mac-address.52601/page-6#post-421676

some traces:

ip addr:

190: fwbr109i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 22:5f:0b:cb:ac:42 brd ff:ff:ff:ff:ff:ff

ebtable log:
Oct  6 09:46:24 kvmformation3 kernel: [437256.753355] MAC-FLOOD-F IN=fwpr109p0 OUT=eno1 MAC source = 22:5f:0b:cb:ac:42 MAC dest = 01:00:5e:00:00:16 proto = 0x0800 IP SRC=0.0.0.0 IP DST=224.0.0.22, IP tos=0xC0, IP proto=2

tcpdump -e -i eno1 igmp
09:53:23.914825 22:5f:0b:cb:ac:42 (oui Unknown) > 01:00:5e:00:00:16 (oui Unknown), ethertype IPv4 (0x0800), length 54: 0.0.0.0 > igmp.mcast.net: igmp v3 report, 1 group record(s)

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 debian/sysctl.d/pve.conf | 1 +
 1 file changed, 1 insertion(+)

diff --git a/debian/sysctl.d/pve.conf b/debian/sysctl.d/pve.conf
index 929698f..85b59b9 100644
--- a/debian/sysctl.d/pve.conf
+++ b/debian/sysctl.d/pve.conf
@@ -2,4 +2,5 @@ net.bridge.bridge-nf-call-ip6tables = 0
 net.bridge.bridge-nf-call-iptables = 0
 net.bridge.bridge-nf-call-arptables = 0
 net.bridge.bridge-nf-filter-vlan-tagged = 0
+net.ipv4.igmp_link_local_mcast_reports = 0
 fs.aio-max-nr = 1048576
-- 
2.30.2

Re: [pve-devel] [PATCH pve-cluster] sysctl: disable net.ipv4.igmp_link_local_mcast_reports

[Thomas Lamprecht]

On 06.10.21 10:32, Alexandre Derumier wrote:
> currently, when veth or tap interfaces are plugged to bridge,
> an igmp v3 report is broadcasted to the network, with the
> bridge mac adddress.

but this disables it for all, couldn't there be repercussions for people relying
on multicast?

Should it be an FW option?

> 
> Users have reported problems with hetzner for example, blocking the server
> because of the unknown mac flooding the network.
> https://forum.proxmox.com/threads/proxmox-claiming-mac-address.52601/page-6#post-421676
> 
> some traces:
> 
> ip addr:
> 
> 190: fwbr109i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
>     link/ether 22:5f:0b:cb:ac:42 brd ff:ff:ff:ff:ff:ff
> 
> ebtable log:
> Oct  6 09:46:24 kvmformation3 kernel: [437256.753355] MAC-FLOOD-F IN=fwpr109p0 OUT=eno1 MAC source = 22:5f:0b:cb:ac:42 MAC dest = 01:00:5e:00:00:16 proto = 0x0800 IP SRC=0.0.0.0 IP DST=224.0.0.22, IP tos=0xC0, IP proto=2
> 
> tcpdump -e -i eno1 igmp
> 09:53:23.914825 22:5f:0b:cb:ac:42 (oui Unknown) > 01:00:5e:00:00:16 (oui Unknown), ethertype IPv4 (0x0800), length 54: 0.0.0.0 > igmp.mcast.net: igmp v3 report, 1 group record(s)
> 
> Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
> ---
>  debian/sysctl.d/pve.conf | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/debian/sysctl.d/pve.conf b/debian/sysctl.d/pve.conf
> index 929698f..85b59b9 100644
> --- a/debian/sysctl.d/pve.conf
> +++ b/debian/sysctl.d/pve.conf
> @@ -2,4 +2,5 @@ net.bridge.bridge-nf-call-ip6tables = 0
>  net.bridge.bridge-nf-call-iptables = 0
>  net.bridge.bridge-nf-call-arptables = 0
>  net.bridge.bridge-nf-filter-vlan-tagged = 0
> +net.ipv4.igmp_link_local_mcast_reports = 0
>  fs.aio-max-nr = 1048576
> 

Re: [pve-devel] [PATCH pve-cluster] sysctl: disable net.ipv4.igmp_link_local_mcast_reports

[DERUMIER, Alexandre]

Le vendredi 05 novembre 2021 à 14:20 +0100, Thomas Lamprecht a écrit :
On 06.10.21 10:32, Alexandre Derumier wrote:
currently, when veth or tap interfaces are plugged to bridge,
an igmp v3 report is broadcasted to the network, with the
bridge mac adddress.

but this disables it for all, couldn't there be repercussions for
people relying
on multicast?

This is really specific to local-link multicast, and it's should only
be use for some specific routing protocol

https://yhbt.net/lore/all/1439396033-6264-1-git-send-email-pdowney@brocade.com/T/
https://www.omnisecu.com/tcpip/ipv4-link-local-multicast-addresses.php

So, I'll not break multicast services inside the vm.

Maybe if hypervisor use ospf routing protocol, but anyway, we don't
have any infos about true vm ip/mac on fwbr bridges.


another workaround possible:

the igmp report is send when the fwbr bridge is going up.

actually corretly activate the fwbr bridge before plugging to vmbr,

my $create_firewall_bridge_linux = sub {
  ...
  &$cond_create_bridge($fwbr);
  &$activate_interface($fwbr);
  copy_bridge_config($bridge, $fwbr);
  veth_create($vethfw, $vethfwpeer, $bridge);

  &$bridge_add_interface($fwbr, $vethfw);
  &$bridge_add_interface($bridge, $vethfwpeer, $tag, $trunks);
  &$bridge_add_interface($fwbr, $iface);
};

but it seem that igmp is sent some millisecond later

A simple sleep like,

  &$cond_create_bridge($fwbr);
  &$activate_interface($fwbr);
  sleep(1);
  &$bridge_add_interface($fwbr, $vethfw);

and the igmp report from fwbr is not going to vmbr.
(but, maybe this is more ugly than a sysctl knob)





Should it be an FW option?

It could be.

but it need to be persistant at firewall service stop, as when we
shutdown the server, igmp report could be emit on vm/ct shutdown.
and at boot, it should be enabled before the vm auto-start

Personnaly, I think it should be disabled by default, with an knob to
enable it.
, as a majority of basic users don't known what it is. (And advanced
users using routing protocol, should be aware of this option).






Users have reported problems with hetzner for example, blocking the
server
because of the unknown mac flooding the network.
https://antiphishing.cetsi.fr/proxy/v3?i=MlZSTzBhZFZ6Nzl4c3EyN7fbSKDe
PLMxi5u5_onpAoI&r=cm1qVmRYUWk2WXhYZVFHWA0PXtTaYxz7-FIOTkZBm34_dHdSch-
gXn7ST9eGhQLN&f=S1Zkd042VWdrZG5qQUxxWkoxusdz-
0duEYVP4tn9qrY6ihzNtzMZon4NP5plKzc3&u=https%3A//forum.proxmox.com/thr
eads/proxmox-claiming-mac-address.52601/page-6%23post-421676&k=F1is

some traces:

ip addr:

190: fwbr109i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP group default qlen 1000
    link/ether 22:5f:0b:cb:ac:42 brd ff:ff:ff:ff:ff:ff

ebtable log:
Oct  6 09:46:24 kvmformation3 kernel: [437256.753355] MAC-FLOOD-F
IN=fwpr109p0 OUT=eno1 MAC source = 22:5f:0b:cb:ac:42 MAC dest =
01:00:5e:00:00:16 proto = 0x0800 IP SRC=0.0.0.0 IP DST=224.0.0.22, IP
tos=0xC0, IP proto=2

tcpdump -e -i eno1 igmp
09:53:23.914825 22:5f:0b:cb:ac:42 (oui Unknown) > 01:00:5e:00:00:16
(oui Unknown), ethertype IPv4 (0x0800), length 54: 0.0.0.0 >
igmp.mcast.net: igmp v3 report, 1 group record(s)

Signed-off-by: Alexandre Derumier <aderumier@odiso.com<mailto:aderumier@odiso.com>>
---
 debian/sysctl.d/pve.conf | 1 +
 1 file changed, 1 insertion(+)

diff --git a/debian/sysctl.d/pve.conf b/debian/sysctl.d/pve.conf
index 929698f..85b59b9 100644
--- a/debian/sysctl.d/pve.conf
+++ b/debian/sysctl.d/pve.conf
@@ -2,4 +2,5 @@ net.bridge.bridge-nf-call-ip6tables = 0
 net.bridge.bridge-nf-call-iptables = 0
 net.bridge.bridge-nf-call-arptables = 0
 net.bridge.bridge-nf-filter-vlan-tagged = 0
+net.ipv4.igmp_link_local_mcast_reports = 0
 fs.aio-max-nr = 1048576

Re: [pve-devel] [PATCH pve-cluster] sysctl: disable net.ipv4.igmp_link_local_mcast_reports

[Thomas Lamprecht]

On 09.11.21 16:52, DERUMIER, Alexandre wrote:
> This is really specific to local-link multicast, and it's should only
> be use for some specific routing protocol
> 
> https://yhbt.net/lore/all/1439396033-6264-1-git-send-email-pdowney@brocade.com/T/
> https://www.omnisecu.com/tcpip/ipv4-link-local-multicast-addresses.php
> 
> So, I'll not break multicast services inside the vm.
> 
> Maybe if hypervisor use ospf routing protocol, but anyway, we don't
> have any infos about true vm ip/mac on fwbr bridges.

hmm, ack thx for the info, so the sysctl can be fine.

> actually corretly activate the fwbr bridge before plugging to vmbr,
> 
> my $create_firewall_bridge_linux = sub {
>   ...
>   &$cond_create_bridge($fwbr);
>   &$activate_interface($fwbr);
>   copy_bridge_config($bridge, $fwbr);
>   veth_create($vethfw, $vethfwpeer, $bridge);
> 
>   &$bridge_add_interface($fwbr, $vethfw);
>   &$bridge_add_interface($bridge, $vethfwpeer, $tag, $trunks);
>   &$bridge_add_interface($fwbr, $iface);
> };
> 
> but it seem that igmp is sent some millisecond later
> 
> A simple sleep like,
> 
>   &$cond_create_bridge($fwbr);
>   &$activate_interface($fwbr);
>   sleep(1);
>   &$bridge_add_interface($fwbr, $vethfw);
> 
> and the igmp report from fwbr is not going to vmbr.
> (but, maybe this is more ugly than a sysctl knob)
> 

yeah such sleeps are ugly and still racy, we could poll for the fwbr to be up?

E.g., something like:

my $round = 0;
while (1) {
   last if PVE::Tools::file_read_firstline("/sys/class/net/$fwbr/operstate") =~ "up";
   die "timeout on waiting for $fwbr to become ready\n" if ++$round > 100;
   usleep(10 * 1000); # 100 * 10ms = 1s
}

(did not test it)

>> Should it be an FW option?
> 
> It could be.
> 
> but it need to be persistant at firewall service stop, as when we
> shutdown the server, igmp report could be emit on vm/ct shutdown.
> and at boot, it should be enabled before the vm auto-start
> 
> Personnaly, I think it should be disabled by default, with an knob to
> enable it.
> , as a majority of basic users don't known what it is. (And advanced
> users using routing protocol, should be aware of this option).


ah yea, with default on I meant that the sysctl would be default on (so the local
mcast reporting default off), sorry for the confusion.
We wouldn't need a actual config knob either as an admin that depends on this can
just configure a higher-priority sysctl.d file..

[pve-devel] applied: [PATCH pve-cluster] sysctl: disable net.ipv4.igmp_link_local_mcast_reports

[Thomas Lamprecht]

On 06.10.21 10:32, Alexandre Derumier wrote:
> currently, when veth or tap interfaces are plugged to bridge,
> an igmp v3 report is broadcasted to the network, with the
> bridge mac adddress.
> 
> Users have reported problems with hetzner for example, blocking the server
> because of the unknown mac flooding the network.
> https://forum.proxmox.com/threads/proxmox-claiming-mac-address.52601/page-6#post-421676
> 
> some traces:
> 
> ip addr:
> 
> 190: fwbr109i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
>     link/ether 22:5f:0b:cb:ac:42 brd ff:ff:ff:ff:ff:ff
> 
> ebtable log:
> Oct  6 09:46:24 kvmformation3 kernel: [437256.753355] MAC-FLOOD-F IN=fwpr109p0 OUT=eno1 MAC source = 22:5f:0b:cb:ac:42 MAC dest = 01:00:5e:00:00:16 proto = 0x0800 IP SRC=0.0.0.0 IP DST=224.0.0.22, IP tos=0xC0, IP proto=2
> 
> tcpdump -e -i eno1 igmp
> 09:53:23.914825 22:5f:0b:cb:ac:42 (oui Unknown) > 01:00:5e:00:00:16 (oui Unknown), ethertype IPv4 (0x0800), length 54: 0.0.0.0 > igmp.mcast.net: igmp v3 report, 1 group record(s)
> 
> Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
> ---
>  debian/sysctl.d/pve.conf | 1 +
>  1 file changed, 1 insertion(+)
> 
>

applied, thanks!