From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <lists@merit.unu.edu> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 3BDAB69BAB for <pve-user@lists.proxmox.com>; Tue, 23 Mar 2021 15:28:20 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 329562FF05 for <pve-user@lists.proxmox.com>; Tue, 23 Mar 2021 15:28:20 +0100 (CET) Received: from mail.merit.unu.edu (webmail.merit.unu.edu [192.87.143.6]) by firstgate.proxmox.com (Proxmox) with ESMTP id 463F52FEF9 for <pve-user@lists.proxmox.com>; Tue, 23 Mar 2021 15:28:18 +0100 (CET) Received: from [192.87.143.100] (ws100.merit.unu.edu [192.87.143.100]) by mail.merit.unu.edu (Postfix) with ESMTPSA id EFB55809F5DE9 for <pve-user@lists.proxmox.com>; Tue, 23 Mar 2021 15:28:17 +0100 (CET) To: pve-user@lists.proxmox.com References: <45b53a59-fd1d-03f0-eaea-4bdce366354f@merit.unu.edu> <56cb4d98-84e4-3183-daaa-48b34d79544c@aasen.cx> From: mj <lists@merit.unu.edu> Message-ID: <73e84c78-51a7-edf2-a898-1f983996d1b0@merit.unu.edu> Date: Tue, 23 Mar 2021 15:28:17 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: <56cb4d98-84e4-3183-daaa-48b34d79544c@aasen.cx> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.225 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_NUMSUBJECT 0.5 Subject ends in numbers excluding current years NICE_REPLY_A -0.001 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [PVE-User] ip address on both bond0 and vmbr0 X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list <pve-user.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-user/> List-Post: <mailto:pve-user@lists.proxmox.com> List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Tue, 23 Mar 2021 14:28:20 -0000 Hi all, Thanks for all suggestions! I will try with Bastian's: > bond0 (slaves enp2...) > vmbr0 (slave bond0) 192.168.143.10/24 > bond0.10 10.0.0.10/24 as that will also give proper separation of ceph traffic, as indicated by Dorsy. Also thank you Ronny, for showing your elaborate config! MJ On 23/03/2021 13:02, Ronny Aasen wrote: > On 23.03.2021 11:42, mj wrote: >> Hi all, >> >> First some info: >> 10.0.0.0/24 is ceph storage >> 192.168.143.0/24 is our LAN >> >> I am trying to make this /etc/networking/interfaces work in in pve: >> >>> auto enp2s0f0 >>> iface enp2s0f0 inet manual >>> #mlag1 >>> >>> auto enp2s0f1 >>> iface enp2s0f1 inet manual >>> #mlag2 >>> >>> iface enp0s25 inet manual >>> #management >>> >>> auto bond0 >>> iface bond0 inet static >>> address 10.0.0.10/24 >>> bond-slaves enp2s0f0 enp2s0f1 >>> bond-miimon 100 >>> bond-mode active-backup >>> bond-primary enp2s0f0 >>> >>> auto vmbr0 >>> iface vmbr0 inet static >>> address 192.168.143.10/24 >>> gateway 192.168.143.1 >>> bridge-ports bond0 >>> bridge-stp off >>> bridge-fd 0 >> >> We will connect pve servers to two mlagged arista 40G switches. The >> 10.0.0.0/24 ceph network will remain local on the two aristas, and >> 192.168.143.0/24 will be routed to our core switch. >> >> The VM IPs are in the LAN 192.168.143.0/24 range, and obviously don't >> require access to 10.0.0.0/24 >> >> We connect the VMs to vmbr0 and assign VLANs to them by configuring a >> VLAN tag in the proxmox VM config. This works. :-) >> >> However, assigning the IP address to bond0 does NOT work. The IP >> address is ignored. bond0 works, but is IP-less. Adding the IP address >> manually after boot works, using: >>> ip addr add 10.0.0.10/24 dev bond0 >> >> Why is this ip address not assigned to bond0 at boot time? >> >> Is it not possible to have an IP on both bond0 and vmbr0, when bond0 >> is also used as a bridge port? >> > > > No you can not use the ip on the bond and the bridge; while you can run > 2 ip's on bridge, that is a bit ugly. > > the way we do it is running vlan's on the bond, into a vlan aware bridge > > auto ens6f0 > iface ens6f0 inet manual > mtu 9700 > > auto ens6f1 > iface ens6f1 inet manual > mtu 9700 > > auto bond0 > iface bond0 inet manual > slaves ens6f0 ens6f1 > bond_miimon 100 > bond_mode 1 > bond_xmit_hash_policy layer3+4 > mtu 9700 > > auto vmbr0 > iface vmbr0 inet manual > bridge_ports bond0 > bridge_stp off > bridge_maxage 0 > bridge_ageing 0 > bridge_maxwait 0 > bridge_fd 0 > bridge_vlan_aware yes > mtu 9700 > up echo 1 > > /sys/devices/virtual/net/vmbr0/bridge/multicast_querier > up echo 0 > > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping > > then define an vlan interface per subnet > > auto vmbr0.10 > iface vmbr0.10 inet6 static > address 2001:db8:2323::11 > netmask 64 > gateway 2001:bd8:2323::1 > mtu 1500 > > > vm's attach to vmbr0 + the tag for the vlan they should be in. > > good luck > > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >