From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id DB134A2D30 for ; Tue, 20 Jun 2023 16:41:45 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C59EA372A6 for ; Tue, 20 Jun 2023 16:41:45 +0200 (CEST) Received: from ip1-isp1-algemesi.verdnatura.es (ip1-isp1-algemesi.verdnatura.es [195.77.191.178]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 20 Jun 2023 16:41:43 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by ip1-isp1-algemesi.verdnatura.es (Postfix) with ESMTP id 26A9DA0080 for ; Tue, 20 Jun 2023 16:34:45 +0200 (CEST) Authentication-Results: ip1-isp1-algemesi.verdnatura.es (amavisd-new); dkim=pass (1024-bit key) reason="pass (just generated, assumed good)" header.d=verdnatura.es DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=verdnatura.es; h= content-transfer-encoding:content-type:content-type:message-id :subject:subject:from:from:date:date:mime-version:received :received; s=mail; t=1687271683; bh=vTF+srJ45iFCMkaZgRsKjLzGMj9j 48WYtXs4h1CwFuM=; b=h/RAi64slmPIzUfPwO3DUZ9GkVaw4EKKYQ9KDOV35gxx 0VsshUZrWSsjwvv9u7KIxk66OrLM/uZ8IUown1MNk3TBvKO04VlCBPyue1s8+NiR k2vT6bTxvopFFAPaI7uqLhQzRTsdH/zCwEYDNAmSzv43Ja9BCHns2KrvnJT4oXM= X-Virus-Scanned: Debian amavisd-new at verdnatura.es Received: from ip1-isp1-algemesi.verdnatura.es ([127.0.0.1]) by localhost (ip1-isp1-algemesi.verdnatura.es [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 141T4ZdyEhsN; Tue, 20 Jun 2023 16:34:43 +0200 (CEST) Received: from webmail.verdnatura.es (kube-worker1.static.verdnatura.es [10.0.2.121]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ip1-isp1-algemesi.verdnatura.es (Postfix) with ESMTPSA id 94ED0A007F; Tue, 20 Jun 2023 16:34:43 +0200 (CEST) MIME-Version: 1.0 Date: Tue, 20 Jun 2023 16:34:43 +0200 From: nada To: pve-user@lists.proxmox.com Message-ID: X-Sender: nada@verdnatura.es Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL -0.001 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [verdnatura.es, proxmox.com] Subject: [PVE-User] isolate node communication X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Jun 2023 14:41:45 -0000 hi folks Our task is to isolate cluster management from virtuals. Nodes and virtuals (CT/QM) were at the same subnet before. The following was already isolated by different subnet and VLAN before. * 10.19.0.0/16 VLAN19 ... 2nd corosync ring * 10.8.0.0/16 VLAN8 ... independent CEPH (not hyperconv) =3D=3D=3D=3D=3D ISOLATION I used BRAIN from part 3.3.8 at https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_co= nfiguration MANY thanks for these admin guides ! Really helpful source of PROXMOX community knowledge ! status * pve-manager/7.3-3/c3928077 (running kernel: 5.15.74-1-pve) * nodes&virtuals at OLD subnet 10.0.0.0/16 gateway 10.0.0.1 * nodes at NEW subnet=3D10.34.1.91/16 gateway=3D10.34.0.1 VLAN=3D34 (vmbr= 0.34) * virtuals will continue to run in OLD subnet 10.0.0.0/16 (vmbr0) Following config files are from a TESTING cluster 'minimox' with 3 nodes (mox91,mox92,mox93) example of host node isolation with OLD IP 10.0.1.93/16 gateway 10.0.0.1 NEW IP 10.34.1.93/16 gateway 10.34.0.1 VLAN34 before isolation I was able to ping&nmap cluster node from inside of=20 CT/QM after isolation it is NOT possible to ping&nmap cluster node from inside=20 of CT/QM ;-) everything appears to work well BUT I see a lot of rejected packages at syslog so is it good ??? or should I do it different way at PRODUCTION cluster ??? BTW when I restarted corosync at ALL isolated nodes once more there are NO more messages about rejected packets at syslog any comments are appreciated Nada =3D=3D=3D=3D=3D INTERFACES auto lo iface lo inet loopback iface eno1 inet manual auto vmbr1 iface vmbr1 inet static address 10.8.3.93/16 bridge-ports vlan8 bridge-stp off bridge-fd 0 #ceph auto vmbr2 iface vmbr2 inet static address 10.19.0.93/16 bridge-ports vlan19 bridge-stp off bridge-fd 0 #corosync auto vmbr0 iface vmbr0 inet static address 10.0.1.93/16 bridge-ports eno1 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 #LAN4virtuals auto vmbr0.34 iface vmbr0.34 inet static address 10.34.1.93/16 gateway 10.34.0.1 #LAN4management auto vlan8 iface vlan8 inet manual vlan-raw-device eno1 auto vlan19 iface vlan19 inet manual vlan-raw-device eno1 =3D=3D=3D=3D COROSYNC # cat /etc/pve/corosync.conf logging { debug: off to_syslog: yes } nodelist { node { name: mox91 nodeid: 1 quorum_votes: 1 ring0_addr: 10.34.1.91 ring1_addr: 10.19.0.91 } node { name: mox92 nodeid: 2 quorum_votes: 1 ring0_addr: 10.34.1.92 ring1_addr: 10.19.0.92 } node { name: mox93 nodeid: 3 quorum_votes: 1 ring0_addr: 10.34.1.93 ring1_addr: 10.19.0.93 } } quorum { provider: corosync_votequorum } totem { cluster_name: minimox config_version: 16 interface { linknumber: 0 knet_link_priority: 100 } interface { linknumber: 1 knet_link_priority: 25 } ip_version: ipv4 link_mode: passive secauth: on version: 2 } =3D=3D=3D=3D=3D REJECTED packets # systemctl status corosync.service =E2=97=8F corosync.service - Corosync Cluster Engine Loaded: loaded (/lib/systemd/system/corosync.service; enabled;=20 vendor preset: enabled) Active: active (running) since Mon 2023-06-19 19:14:58 CEST; 19h=20 ago Docs: man:corosync man:corosync.conf man:corosync_overview Main PID: 1947 (corosync) Tasks: 9 (limit: 18927) Memory: 136.9M CPU: 10min 38.124s CGroup: /system.slice/corosync.service =E2=94=94=E2=94=801947 /usr/sbin/corosync -f Jun 20 14:45:25 mox93 corosync[1947]: [KNET ] rx: Packet rejected=20 from 10.34.1.92:5405 Jun 20 14:45:25 mox93 corosync[1947]: [KNET ] rx: Packet rejected=20 from 10.34.1.91:5405 Jun 20 14:45:26 mox93 corosync[1947]: [KNET ] rx: Packet rejected=20 from 10.34.1.92:5405 Jun 20 14:45:26 mox93 corosync[1947]: [KNET ] rx: Packet rejected=20 from 10.34.1.91:5405 Jun 20 14:45:27 mox93 corosync[1947]: [KNET ] rx: Packet rejected=20 from 10.34.1.92:5405 Jun 20 14:45:27 mox93 corosync[1947]: [KNET ] rx: Packet rejected=20 from 10.34.1.91:5405 Jun 20 14:45:28 mox93 corosync[1947]: [KNET ] rx: Packet rejected=20 from 10.34.1.92:5405 Jun 20 14:45:28 mox93 corosync[1947]: [KNET ] rx: Packet rejected=20 from 10.34.1.91:5405 Jun 20 14:45:29 mox93 corosync[1947]: [KNET ] rx: Packet rejected=20 from 10.34.1.92:5405 Jun 20 14:45:29 mox93 corosync[1947]: [KNET ] rx: Packet rejected=20 from 10.34.1.91:5405