From: nada <nada@verdnatura.es>
To: pve-user@lists.proxmox.com
Subject: [PVE-User] isolate node communication
Date: Tue, 20 Jun 2023 16:34:43 +0200 [thread overview]
Message-ID: <b359b1b5803b4f9a3f69a5a931f4f59a@verdnatura.es> (raw)
hi folks
Our task is to isolate cluster management from virtuals.
Nodes and virtuals (CT/QM) were at the same subnet before.
The following was already isolated by different subnet and VLAN before.
* 10.19.0.0/16 VLAN19 ... 2nd corosync ring
* 10.8.0.0/16 VLAN8 ... independent CEPH (not hyperconv)
===== ISOLATION
I used BRAIN from part 3.3.8 at
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration
MANY thanks for these admin guides !
Really helpful source of PROXMOX community knowledge !
status
* pve-manager/7.3-3/c3928077 (running kernel: 5.15.74-1-pve)
* nodes&virtuals at OLD subnet 10.0.0.0/16 gateway 10.0.0.1
* nodes at NEW subnet=10.34.1.91/16 gateway=10.34.0.1 VLAN=34 (vmbr0.34)
* virtuals will continue to run in OLD subnet 10.0.0.0/16 (vmbr0)
Following config files are from a TESTING cluster 'minimox'
with 3 nodes (mox91,mox92,mox93)
example of host node isolation with
OLD IP 10.0.1.93/16 gateway 10.0.0.1
NEW IP 10.34.1.93/16 gateway 10.34.0.1 VLAN34
before isolation I was able to ping&nmap cluster node from inside of
CT/QM
after isolation it is NOT possible to ping&nmap cluster node from inside
of CT/QM ;-)
everything appears to work well
BUT I see a lot of rejected packages at syslog
so is it good ???
or should I do it different way at PRODUCTION cluster ???
BTW when I restarted corosync at ALL isolated nodes once more
there are NO more messages about rejected packets at syslog
any comments are appreciated
Nada
===== INTERFACES
auto lo
iface lo inet loopback
iface eno1 inet manual
auto vmbr1
iface vmbr1 inet static
address 10.8.3.93/16
bridge-ports vlan8
bridge-stp off
bridge-fd 0
#ceph
auto vmbr2
iface vmbr2 inet static
address 10.19.0.93/16
bridge-ports vlan19
bridge-stp off
bridge-fd 0
#corosync
auto vmbr0
iface vmbr0 inet static
address 10.0.1.93/16
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#LAN4virtuals
auto vmbr0.34
iface vmbr0.34 inet static
address 10.34.1.93/16
gateway 10.34.0.1
#LAN4management
auto vlan8
iface vlan8 inet manual
vlan-raw-device eno1
auto vlan19
iface vlan19 inet manual
vlan-raw-device eno1
==== COROSYNC
# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}
nodelist {
node {
name: mox91
nodeid: 1
quorum_votes: 1
ring0_addr: 10.34.1.91
ring1_addr: 10.19.0.91
}
node {
name: mox92
nodeid: 2
quorum_votes: 1
ring0_addr: 10.34.1.92
ring1_addr: 10.19.0.92
}
node {
name: mox93
nodeid: 3
quorum_votes: 1
ring0_addr: 10.34.1.93
ring1_addr: 10.19.0.93
}
}
quorum {
provider: corosync_votequorum
}
totem {
cluster_name: minimox
config_version: 16
interface {
linknumber: 0
knet_link_priority: 100
}
interface {
linknumber: 1
knet_link_priority: 25
}
ip_version: ipv4
link_mode: passive
secauth: on
version: 2
}
===== REJECTED packets
# systemctl status corosync.service
● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled;
vendor preset: enabled)
Active: active (running) since Mon 2023-06-19 19:14:58 CEST; 19h
ago
Docs: man:corosync
man:corosync.conf
man:corosync_overview
Main PID: 1947 (corosync)
Tasks: 9 (limit: 18927)
Memory: 136.9M
CPU: 10min 38.124s
CGroup: /system.slice/corosync.service
└─1947 /usr/sbin/corosync -f
Jun 20 14:45:25 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.92:5405
Jun 20 14:45:25 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.91:5405
Jun 20 14:45:26 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.92:5405
Jun 20 14:45:26 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.91:5405
Jun 20 14:45:27 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.92:5405
Jun 20 14:45:27 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.91:5405
Jun 20 14:45:28 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.92:5405
Jun 20 14:45:28 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.91:5405
Jun 20 14:45:29 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.92:5405
Jun 20 14:45:29 mox93 corosync[1947]: [KNET ] rx: Packet rejected
from 10.34.1.91:5405
reply other threads:[~2023-06-20 14:41 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b359b1b5803b4f9a3f69a5a931f4f59a@verdnatura.es \
--to=nada@verdnatura.es \
--cc=pve-user@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.