From: nada <nada@verdnatura.es>
To: pve-user@lists.proxmox.com
Subject: [PVE-User] isolate node communication
Date: Tue, 20 Jun 2023 16:34:43 +0200 [thread overview]
Message-ID: <b359b1b5803b4f9a3f69a5a931f4f59a@verdnatura.es> (raw)
hi folks
Our task is to isolate cluster management from virtuals.
Nodes and virtuals (CT/QM) were at the same subnet before.
The following was already isolated by different subnet and VLAN before.
* 10.19.0.0/16 VLAN19 ... 2nd corosync ring
* 10.8.0.0/16 VLAN8 ... independent CEPH (not hyperconv)
===== ISOLATION
I used BRAIN from part 3.3.8 at
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration
MANY thanks for these admin guides !
Really helpful source of PROXMOX community knowledge !
status
* pve-manager/7.3-3/c3928077 (running kernel: 5.15.74-1-pve)
* nodes&virtuals at OLD subnet 10.0.0.0/16 gateway 10.0.0.1
* nodes at NEW subnet=10.34.1.91/16 gateway=10.34.0.1 VLAN=34 (vmbr0.34)
* virtuals will continue to run in OLD subnet 10.0.0.0/16 (vmbr0)
Following config files are from a TESTING cluster 'minimox'
with 3 nodes (mox91,mox92,mox93)
example of host node isolation with
OLD IP 10.0.1.93/16 gateway 10.0.0.1
NEW IP 10.34.1.93/16 gateway 10.34.0.1 VLAN34
before isolation I was able to ping&nmap cluster node from inside of
CT/QM
after isolation it is NOT possible to ping&nmap cluster node from inside
of CT/QM ;-)
everything appears to work well
BUT I see a lot of rejected packages at syslog
so is it good ???
or should I do it different way at PRODUCTION cluster ???
BTW when I restarted corosync at ALL isolated nodes once more
there are NO more messages about rejected packets at syslog
any comments are appreciated
Nada
===== INTERFACES
auto lo
iface lo inet loopback
iface eno1 inet manual
auto vmbr1
iface vmbr1 inet static
address 10.8.3.93/16
bridge-ports vlan8
bridge-stp off
bridge-fd 0
#ceph
auto vmbr2
iface vmbr2 inet static
address 10.19.0.93/16
bridge-ports vlan19
bridge-stp off
bridge-fd 0
#corosync
auto vmbr0
iface vmbr0 inet static
address 10.0.1.93/16
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#LAN4virtuals
auto vmbr0.34
iface vmbr0.34 inet static
address 10.34.1.93/16
gateway 10.34.0.1
#LAN4management
auto vlan8
iface vlan8 inet manual
vlan-raw-device eno1
auto vlan19
iface vlan19 inet manual
vlan-raw-device eno1
==== COROSYNC
# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}
nodelist {
node {
name: mox91
nodeid: 1
quorum_votes: 1
ring0_addr: 10.34.1.91
ring1_addr: 10.19.0.91
}
node {
name: mox92
nodeid: 2
quorum_votes: 1
ring0_addr: 10.34.1.92
ring1_addr: 10.19.0.92
}
node {
name: mox93
nodeid: 3
quorum_votes: 1
ring0_addr: 10.34.1.93
ring1_addr: 10.19.0.93
}
}
quorum {
provider: corosync_votequorum
}
totem {
cluster_name: minimox
config_version: 16
interface {
linknumber: 0
knet_link_priority: 100
}
interface {
linknumber: 1
knet_link_priority: 25
}
ip_version: ipv4
link_mode: passive
secauth: on
version: 2
}
===== REJECTED packets
# systemctl status corosync.service
● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled;
vendor preset: enabled)
Active: active (running) since Mon 2023-06-19 19:14:58 CEST; 19h
ago
Docs: man:corosync
man:corosync.conf
man:corosync_overview
Main PID: 1947 (corosync)
Tasks: 9 (limit: 18927)
Memory: 136.9M
CPU: 10min 38.124s
CGroup: /system.slice/corosync.service
└─1947 /usr/sbin/corosync -f
Jun 20 14:45:25 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.92:5405
Jun 20 14:45:25 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.91:5405
Jun 20 14:45:26 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.92:5405
Jun 20 14:45:26 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.91:5405
Jun 20 14:45:27 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.92:5405
Jun 20 14:45:27 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.91:5405
Jun 20 14:45:28 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.92:5405
Jun 20 14:45:28 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.91:5405
Jun 20 14:45:29 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.92:5405
Jun 20 14:45:29 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.91:5405
reply other threads:[~2023-06-20 14:41 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b359b1b5803b4f9a3f69a5a931f4f59a@verdnatura.es \
--to=nada@verdnatura.es \
--cc=pve-user@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox