From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 0BBF31FF0AB for ; Wed, 07 Oct 2026 11:46:43 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 8B3A4213F7; Wed, 07 Oct 2026 11:46:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=internet.wien; s=drsa2048_mx01_01; t=1791366393; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=r62QHEb8J35KVl0ydjNTUw7w4tvfH8ADkO33OWFMkXE=; b=ZQrqOJXLLSBc49CGkKSNoEcuqLAP0sguBM/WEM1NFYQMFR2S27AnujKrivqlkzpwkGjxLa NSlf1YQ1sPltk2xOModQB92sRGJ1sah0JepHE0CTDhYXvP5W0/uHAhF2EzMZ12lzkt3YxW PaSCELMXsQx6JARDRIYH9vVZlEGU2SSUQvdTncKm5CPpBA1qztwOHmeWpTYgv1RwyNLBtp S8EHJbxuckknqBb+YhtQIL2RC5v50wA1NXg7WiMgv3o1jwcQcZkp9mLyOGr6tZ1l0F+sNu hh6Pqw3nYTVPn3uj4+iDtwe3e32rtsflDrHvm/2kShGg3RbzcHbW+6s7M/2VtQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=internet.wien; s=drsa1024_mx01_01; t=1791366393; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=r62QHEb8J35KVl0ydjNTUw7w4tvfH8ADkO33OWFMkXE=; b=pZZQMZ7wHfQTIrrQPiHIbNi5ZqwXX9EwN1PfQ1pmzlvorSkmCnwx3q+HcO4Mtz6/13ITOP UnczvllD4GEtZRVtrJJfar6tML1pue+PuVBwdnCH4nn9fiBRoscJPLvrScnwfveAuKEAAQ 73f+x2WlMcrd3LhyXCHon5Ny3wNJCyU= Message-ID: <8de6cd729969eb4e318d4e76aa1137ddf47c7954.camel@internet.wien> Subject: Re: SDN with BGP Fabric and IPv6 Only Underlay for VXLAN From: Tobias Fiebig To: Hannes Laimer , pve-devel@lists.proxmox.com Date: Wed, 07 Oct 2026 11:46:32 +0200 In-Reply-To: References: <6a01449fcaee735648dbd8424d64c737a0428c9f.camel@internet.wien> <870c6dac7eb6683febc6bedaf679c500315216bd.camel@internet.wien> Autocrypt: addr=tobias@internet.wien; prefer-encrypt=mutual; keydata=mDMEZxprVxYJKwYBBAHaRw8BAQdArrtud5RQu9EE0uVs+iR5P8srdhFF2MWG0ll6VnhOt 1S0IFRvYmlhcyBGaWViaWcgPHRvYmlhc0BmaWViaWcubmw+iJEEExYIADkCGwMFCQWjmoACF4ACHg cWIQTZ+3sTGjvUXcRACVeZMd9l/7ol/AUCZ44PdgQLCQgHAiICBBUICwIACgkQmTHfZf+6JfyN0AE AunVWmj6nUDJCtwIlDqhzLrw0bf20C0hcrI6lITEsVW4A/0H56Tz96R9IliIPlqPX5Ww6kH1w6Nlg 51a2x+gUoNMOtDxUb2JpYXMgRmllYmlnIChSZXNlYXJjaCBVbml0IEFkZHJlc3MpIDx0b2JpYXNAa W50ZXJuZXQud2llbj6IsAQTFggAWBYhBNn7exMaO9RdxEAJV5kx32X/uiX8BQJprpreGxSAAAAAAA QADm1hbnUyLDIuNSsxLjExLDIsMgIbAwUJBaOagAULCQgHAwMVCAsFFgIDAQACHgUCF4AACgkQmTH fZf+6JfzPpAEA7WxbtpIJsyblG1kEO7wqXtWEZHZ+PPuYG5x2XojLT2EBAIpQZv5PpYcOGc+FCXB3 K7M8n3aoHdY+LXCQUB27IXQAtEFUb2JpYXMgRmllYmlnIChUVSBXaWVuIE1haWwgQWRkcmVzcykgP HRvYmlhcy5maWViaWdAdHV3aWVuLmFjLmF0PoiwBBMWCABYFiEE2ft7Exo71F3EQAlXmTHfZf+6Jf wFAmmumv0bFIAAAAAABAAObWFudTIsMi41KzEuMTEsMiwyAhsDBQkFo5qABQsJCAcDAxUICwUWAgM BAAIeBQIXgAAKCRCZMd9l/7ol/Dk/AQDME9RFGDoUln1nzXjPwFSJiEzgF/by/vpd+5I7WfU1XAD+ LAemdvz+gw41+SyOIgUl25YaVcofaD0hW5PrIij5fw20N1RvYmlhcyBGaWViaWcgKFByaXZhdGUgb WFpbCBBdXN0cmlhKSA8dG9iaWFzQGZpZWJpZy5hdD6IsAQTFggAWBYhBNn7exMaO9RdxEAJV5kx32 X/uiX8BQJprpsNGxSAAAAAAAQADm1hbnUyLDIuNSsxLjExLDIsMgIbAwUJBaOagAULCQgHAwMVCAs FFgIDAQACHgUCF4AACgkQmTHfZf+6Jfx6AwD/Wv+jfW0PCfWyb9P1/y+88gudOj6xtJMDP0719XAd gHwBAKPzSDowHwQ743tArxpatFqqzFQwfxSzEp63HsZGBdwP Organization: TU Wien - Research Unit 'Internet Infrastructures' Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.60.2 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: WSY2EYCACBJSV2RQ4WPDEOM4PXCPNSBS X-Message-ID-Hash: WSY2EYCACBJSV2RQ4WPDEOM4PXCPNSBS X-MailFrom: tobias@internet.wien X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hello Hannes, > yes, a patch for this specifically already exists [1], and with [2] > also v6 fabrics should work. Thanks, looking forward to it being merged. In the context of the L3 fabric, i stumbled over two additional things. 1. BGP Pseudo interface MTU If the L3 fabric is via links running on jumbos, and there is no additional L2 network between the proxmox nodes, VM migration breaks as soon as the BGP dummy interface for the loopback address in the fabric is added on nodes, as that interface is created with an MTU of 1500 and used as the default source address for contacting other nodes, i.e., while the other node originates packets for an MTU of, e.g., 9000, those reach the initiating node, but cannot be forwarded to the (dummy) interface where the selected source address resides. I am also not sure why there are not PTB ICMPv6 packets originated that should fix this. But on migration, copying over the memory state just hangs. This can be fixed by either adjusting the MTU of the dummy interface to match that of the actual links, or by creating a dedicated shared L2, e.g., via another VXLAN, that is than addressed and explicitly configured as the migration network. 2. Pure L3 Fabric + RFC8950 / IPv4 with IPv6 Nexthops Beyond that, there is also the overall network concept. Not sure whether this would be viable feature; But dumping it in here for reference. What I am essentially trying to build is a network (not only the PVE components) that runs mostly IPv6 only, and has IPv4 as /32 routed to GUA, handled via (i)BGP. A description of the concept can be found here: https://ripe88.ripe.net/presentations/15-ripe_88_v6_wg.pdf The general advantages would be: - No need for bridge devices - No need for any L2 within the whole network; This also means that, e.g., LACP is no longer needed on switches; Instead the L3 fabric is BGP based (numbered or unnumbered); Also, with ECMP, traffic actually balances etc. - Most importantly: No ARP ;-) Things I noticed that stand in the way there: - Technically, nodes would have to be able to create a VM interface that is not added to a bridge; Instead, it would just get DHCP/SLAAC with a prefix selected from a configured covering prefix; The more specific (automatically) selected for the VM would then just be announced in the BGP fabric, and upon VM migration, the prefix would be moved. The same could be done with v4, considering the VM OS supports v6 NH for v4 default routes. - There is no way to add communities via configurable routemaps in proxmox atm; This makes it a bit more difficult to properly configure filters when interacting with hosts speaking BGP outside of the proxmox cluster; Same holds for import filters based on matching communities. - The imported routes webinterface in the fabric view is a bit overwhelmed when the nodes get a (v6) fulltable (questionable concept, but... it can make sense ;-)) Pagination there would probably be helpful (also, in general, for larger networks). Happy to hear your thoughts; If this is the wrong place for discussing this, please also let me know. With best regards, Tobias --=20 My working day may not be your working day. Please do not feel obliged=20 to reply to my email outside of your normal working hours. ----------------------------------------------------------------- Univ.Prof. Dr.-Ing. Tobias Fiebig, E191-06 - Internet Infrastructures TU Wien, Fakult=C3=A4t f=C3=BCr Informatik, Treitl Str. 3, 1040 Wien Room DE01 15 mobile phone: +31 616 80 98 99 mail: tobias@internet.wien