From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id E0D276993E for ; Tue, 23 Mar 2021 13:12:16 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id DC40B2E8B6 for ; Tue, 23 Mar 2021 13:12:16 +0100 (CET) Received: from smtp.jostedal.no (smtp.jostedal.no [217.17.211.26]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 40B122E8AC for ; Tue, 23 Mar 2021 13:12:15 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by smtp.jostedal.no (Postfix) with ESMTP id AF7735430B for ; Tue, 23 Mar 2021 13:02:55 +0100 (CET) Received: from smtp.jostedal.no ([127.0.0.1]) by localhost (dpmx.servers.jostedal.no [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4W3rWZOUu4EK for ; Tue, 23 Mar 2021 13:02:55 +0100 (CET) Received: from [10.234.0.29] (95.62-50-191.enivest.net [62.50.191.95]) (Authenticated sender: ronny@aasen.cx) by smtp.jostedal.no (Postfix) with ESMTPSA id 6DFCD54291 for ; Tue, 23 Mar 2021 13:02:55 +0100 (CET) Reply-To: ronny+pve-user@aasen.cx To: pve-user@lists.proxmox.com References: <45b53a59-fd1d-03f0-eaea-4bdce366354f@merit.unu.edu> From: Ronny Aasen Message-ID: <56cb4d98-84e4-3183-daaa-48b34d79544c@aasen.cx> Date: Tue, 23 Mar 2021 13:02:52 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.0 MIME-Version: 1.0 In-Reply-To: <45b53a59-fd1d-03f0-eaea-4bdce366354f@merit.unu.edu> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.293 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_NUMSUBJECT 0.5 Subject ends in numbers excluding current years NICE_REPLY_A -0.001 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [PVE-User] ip address on both bond0 and vmbr0 X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Mar 2021 12:12:16 -0000 On 23.03.2021 11:42, mj wrote: > Hi all, > > First some info: > 10.0.0.0/24 is ceph storage > 192.168.143.0/24 is our LAN > > I am trying to make this /etc/networking/interfaces work in in pve: > >> auto enp2s0f0 >> iface enp2s0f0 inet manual >> #mlag1 >> >> auto enp2s0f1 >> iface enp2s0f1 inet manual >> #mlag2 >> >> iface enp0s25 inet manual >> #management >> >> auto bond0 >> iface bond0 inet static >>         address 10.0.0.10/24 >>         bond-slaves enp2s0f0 enp2s0f1 >>         bond-miimon 100 >>         bond-mode active-backup >>         bond-primary enp2s0f0 >> >> auto vmbr0 >> iface vmbr0 inet static >>         address 192.168.143.10/24 >>         gateway 192.168.143.1 >>         bridge-ports bond0 >>         bridge-stp off >>         bridge-fd 0 > > We will connect pve servers to two mlagged arista 40G switches. The > 10.0.0.0/24 ceph network will remain local on the two aristas, and > 192.168.143.0/24 will be routed to our core switch. > > The VM IPs are in the LAN 192.168.143.0/24 range, and obviously don't > require access to 10.0.0.0/24 > > We connect the VMs to vmbr0 and assign VLANs to them by configuring a > VLAN tag in the proxmox VM config. This works. :-) > > However, assigning the IP address to bond0 does NOT work. The IP address > is ignored. bond0 works, but is IP-less. Adding the IP address manually > after boot works, using: >> ip addr add 10.0.0.10/24 dev bond0 > > Why is this ip address not assigned to bond0 at boot time? > > Is it not possible to have an IP on both bond0 and vmbr0, when bond0 is > also used as a bridge port? > No you can not use the ip on the bond and the bridge; while you can run 2 ip's on bridge, that is a bit ugly. the way we do it is running vlan's on the bond, into a vlan aware bridge auto ens6f0 iface ens6f0 inet manual mtu 9700 auto ens6f1 iface ens6f1 inet manual mtu 9700 auto bond0 iface bond0 inet manual slaves ens6f0 ens6f1 bond_miimon 100 bond_mode 1 bond_xmit_hash_policy layer3+4 mtu 9700 auto vmbr0 iface vmbr0 inet manual bridge_ports bond0 bridge_stp off bridge_maxage 0 bridge_ageing 0 bridge_maxwait 0 bridge_fd 0 bridge_vlan_aware yes mtu 9700 up echo 1 > /sys/devices/virtual/net/vmbr0/bridge/multicast_querier up echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping then define an vlan interface per subnet auto vmbr0.10 iface vmbr0.10 inet6 static address 2001:db8:2323::11 netmask 64 gateway 2001:bd8:2323::1 mtu 1500 vm's attach to vmbr0 + the tag for the vlan they should be in. good luck