==== general system info ==== # hostname node06 # pveversion --verbose proxmox-ve: 7.0-2 (running kernel: 5.11.22-1-pve) pve-manager: 7.0-5 (running version: 7.0-5/cce9b25f) pve-kernel-5.11: 7.0-3 pve-kernel-helper: 7.0-3 pve-kernel-5.4: 6.4-3 pve-kernel-5.11.22-1-pve: 5.11.22-1 pve-kernel-5.4.119-1-pve: 5.4.119-1 pve-kernel-5.4.106-1-pve: 5.4.106-1 pve-kernel-5.4.34-1-pve: 5.4.34-2 ceph: 15.2.13-pve1 ceph-fuse: 15.2.13-pve1 corosync: 3.1.2-pve2 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown: residual config ifupdown2: 3.0.0-1+pve5 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.21-pve1 libproxmox-acme-perl: 1.1.0 libproxmox-backup-qemu0: 1.0.3-1 libpve-access-control: 7.0-3 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.0-4 libpve-guest-common-perl: 4.0-2 libpve-http-server-perl: 4.0-2 libpve-storage-perl: 7.0-6 libqb0: 1.0.5-1 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 4.0.9-1 lxcfs: 4.0.8-pve1 novnc-pve: 1.2.0-3 proxmox-backup-client: 1.1.10-1 proxmox-backup-file-restore: 1.1.10-1 proxmox-mini-journalreader: 1.2-1 proxmox-widget-toolkit: 3.1-4 pve-cluster: 7.0-2 pve-container: 4.0-3 pve-docs: 7.0-3 pve-edk2-firmware: 3.20200531-1 pve-firewall: 4.2-2 pve-firmware: 3.2-4 pve-ha-manager: 3.2-2 pve-i18n: 2.3-1 pve-qemu-kvm: 6.0.0-2 pve-xtermjs: 4.12.0-1 qemu-server: 7.0-5 smartmontools: 7.2-pve2 spiceterm: 3.2-2 vncterm: 1.7-1 zfsutils-linux: 2.0.4-pve1 # cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 2a03:7900:111::dc:6 node06.demo.customers.tuxis.net node06 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts #2a03:7900:111:0:90d4:a7ff:fe7f:ccf6 pbs.tuxis.net # pvesubscription get message: There is no subscription key serverid: F3A59435D1B87C5A2460F965646A3177 status: NotFound url: https://www.proxmox.com/proxmox-ve/pricing # cat /etc/apt/sources.list ## Managed via Ansible deb http://debmirror.tuxis.nl/debian/ bullseye main contrib non-free deb-src http://debmirror.tuxis.nl/debian/ bullseye main contrib non-free deb http://security.debian.org/ bullseye-security main contrib non-free deb-src http://security.debian.org/ bullseye-security main contrib non-free deb http://debmirror.tuxis.nl/debian/ bullseye-updates main contrib non-free deb-src http://debmirror.tuxis.nl/debian/ bullseye-updates main contrib non-free # cat /etc/apt/sources.list.d/pvetest-for-beta.list deb http://download.proxmox.com/debian/pve bullseye pvetest # cat /etc/apt/sources.list.d/ceph.list deb http://download.proxmox.com/debian/ceph-octopus bullseye main # cat /etc/apt/sources.list.d/apt_tuxis_nl_tuxis.list deb https://apt.tuxis.nl/tuxis/ tuxis-cron main deb https://apt.tuxis.nl/tuxis/ monitoring main deb https://apt.tuxis.nl/tuxis/ pmrb main # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz Stepping: 2 CPU MHz: 1200.000 CPU max MHz: 3600.0000 CPU min MHz: 1200.0000 BogoMIPS: 6999.74 Virtualization: VT-x L1d cache: 128 KiB L1i cache: 128 KiB L2 cache: 1 MiB L3 cache: 10 MiB NUMA node0 CPU(s): 0-7 Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d # pvesh get /cluster/resources --type node --output-format=yaml --- - id: node/node04 node: node04 status: offline type: node - cpu: 0 disk: 2595225600 id: node/node06 level: '' maxcpu: 8 maxdisk: 115451232256 maxmem: 67322179584 mem: 1778470912 node: node06 status: online type: node uptime: 17 - id: node/node05 node: node05 status: offline type: node ==== overall system load info ==== # top -b -c -w512 -n 1 -o TIME | head -n 30 top - 10:30:25 up 1 min, 1 user, load average: 0.56, 0.23, 0.08 Tasks: 316 total, 3 running, 313 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.5 us, 3.1 sy, 0.0 ni, 95.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 64203.4 total, 62239.5 free, 1723.6 used, 240.4 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 61675.9 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 165796 8924 5312 S 0.0 0.0 0:01.96 /sbin/init 1908 root 20 0 333916 125364 18596 S 0.0 0.2 0:00.63 /usr/bin/perl /usr/bin/pvesh --nooutput create /nodes/localhost/startall 1900 www-data 20 0 346652 126348 7264 S 0.0 0.2 0:00.53 pveproxy worker 1684 root rt 0 309108 177892 53256 R 0.0 0.3 0:00.52 /usr/sbin/corosync -f 1680 ceph 20 0 255064 19864 10604 S 0.0 0.0 0:00.35 /usr/bin/ceph-mgr -f --cluster ceph --id node06 --setuser ceph --setgroup ceph 1681 ceph 20 0 409952 32068 18780 S 6.2 0.0 0:00.34 /usr/bin/ceph-mon -f --cluster ceph --id node06 --setuser ceph --setgroup ceph 1679 ceph 20 0 250860 18892 10156 S 0.0 0.0 0:00.33 /usr/bin/ceph-mds -f --cluster ceph --id node06 --setuser ceph --setgroup ceph 1699 ceph 20 0 267296 20544 11224 S 0.0 0.0 0:00.31 /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph 31 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/3] 37 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/4] 43 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/5] 49 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/6] 55 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/7] 1701 ceph 20 0 267292 20304 11036 S 0.0 0.0 0:00.30 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph 19 root rt 0 0 0 0 S 0.0 0.0 0:00.29 [migration/1] 25 root rt 0 0 0 0 S 0.0 0.0 0:00.29 [migration/2] 1787 root 20 0 264744 84344 3988 S 0.0 0.1 0:00.28 pve-firewall 1901 www-data 20 0 346460 126360 7280 S 0.0 0.2 0:00.28 pveproxy worker 1813 root 20 0 345068 123768 6148 S 0.0 0.2 0:00.25 pvedaemon worker 1121 root 20 0 98428 3324 2640 S 0.0 0.0 0:00.23 /usr/sbin/zed -F 734 root 20 0 40140 14964 13888 S 0.0 0.0 0:00.22 /lib/systemd/systemd-journald 780 root 20 0 22124 3568 2380 S 0.0 0.0 0:00.19 /lib/systemd/systemd-udevd 1339 root 20 0 396620 18592 7216 S 0.0 0.0 0:00.17 /usr/bin/python3 /usr/bin/fail2ban-server -xf start # head /proc/pressure/* ==> /proc/pressure/cpu <== some avg10=0.00 avg60=0.24 avg300=0.15 total=727501 ==> /proc/pressure/io <== some avg10=0.00 avg60=0.10 avg300=0.06 total=376684 full avg10=0.00 avg60=0.08 avg300=0.05 total=320128 ==> /proc/pressure/memory <== some avg10=0.00 avg60=0.00 avg300=0.00 total=0 full avg10=0.00 avg60=0.00 avg300=0.00 total=0 ==== info about storage ==== # cat /etc/pve/storage.cfg dir: local path /var/lib/vz content images,backup,iso,vztmpl prune-backups keep-last=1 shared 0 zfspool: local-zfs pool rpool/data content rootdir,images sparse 1 rbd: Ceph content rootdir,images krbd 0 pool Ceph cephfs: CephFS path /mnt/pve/CephFS content iso,snippets,backup,vztmpl prune-backups keep-last=1 dir: Tuxis_Marketplace path /mnt/pve/Tuxis_Marketplace content iso,backup is_mountpoint yes mkdir 0 shared 1 dir: Tuxis_Marketplace_Beta path /mnt/pve/Tuxis_Marketplace_Beta content backup,iso is_mountpoint yes mkdir 0 shared 1 rbd: CephKRBD content images krbd 1 pool Ceph pbs: pbs002.tuxis.nl datastore DB0220_demo server pbs002.tuxis.nl content backup encryption-key 68:d5:89:f6:f1:f4:67:59:1b:74:6a:78:99:11:ad:09:a0:b0:12:db:43:8d:41:19:af:38:90:77:12:c1:6d:f8 fingerprint 45:f8:79:eb:27:96:88:6b:29:ad:21:00:13:c6:bd:b8:30:f6:f3:9b:f0:bf:dd:f3:ad:f0:09:d5:d2:9a:34:79 prune-backups keep-last=1 username DB0220@pbs # pvesm status got timeout ERROR: command 'pvesm status' failed: got timeout # cat /etc/fstab # proc /proc proc defaults 0 0 # findmnt --ascii TARGET SOURCE FSTYPE OPTIONS / rpool/ROOT/pve-1 zfs rw,relatime,xattr,noacl |-/sys sysfs sysfs rw,nosuid,nodev,noexec,relatime | |-/sys/kernel/security securityfs securityfs rw,nosuid,nodev,noexec,relatime | |-/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime | |-/sys/fs/pstore pstore pstore rw,nosuid,nodev,noexec,relatime | |-/sys/fs/bpf none bpf rw,nosuid,nodev,noexec,relatime,mode=700 | |-/sys/kernel/debug debugfs debugfs rw,nosuid,nodev,noexec,relatime | |-/sys/kernel/tracing tracefs tracefs rw,nosuid,nodev,noexec,relatime | |-/sys/fs/fuse/connections fusectl fusectl rw,nosuid,nodev,noexec,relatime | `-/sys/kernel/config configfs configfs rw,nosuid,nodev,noexec,relatime |-/proc proc proc rw,relatime | `-/proc/sys/fs/binfmt_misc systemd-1 autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=23036 |-/dev udev devtmpfs rw,nosuid,relatime,size=32838164k,nr_inodes=8209541,mode=755,inode64 | |-/dev/pts devpts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 | |-/dev/shm tmpfs tmpfs rw,nosuid,nodev,inode64 | |-/dev/hugepages hugetlbfs hugetlbfs rw,relatime,pagesize=2M | `-/dev/mqueue mqueue mqueue rw,nosuid,nodev,noexec,relatime |-/run tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=6574432k,mode=755,inode64 | |-/run/lock tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 | |-/run/rpc_pipefs sunrpc rpc_pipefs rw,relatime | `-/run/user/0 tmpfs tmpfs rw,nosuid,nodev,relatime,size=6574428k,nr_inodes=1643607,mode=700,inode64 |-/rpool rpool zfs rw,noatime,xattr,noacl | |-/rpool/ROOT rpool/ROOT zfs rw,noatime,xattr,noacl | `-/rpool/data rpool/data zfs rw,noatime,xattr,noacl |-/var/lib/ceph/osd/ceph-4 tmpfs tmpfs rw,relatime,inode64 |-/var/lib/ceph/osd/ceph-5 tmpfs tmpfs rw,relatime,inode64 |-/mnt/pve/Tuxis_Marketplace s3fs fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0 |-/mnt/pve/Tuxis_Marketplace_Beta s3fs fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0 |-/etc/pve /dev/fuse fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other `-/var/lib/lxcfs lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other # df --human Filesystem Size Used Avail Use% Mounted on udev 32G 0 32G 0% /dev tmpfs 6.3G 1.3M 6.3G 1% /run rpool/ROOT/pve-1 108G 2.5G 106G 3% / tmpfs 32G 66M 32G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock rpool 106G 128K 106G 1% /rpool rpool/ROOT 106G 128K 106G 1% /rpool/ROOT rpool/data 106G 128K 106G 1% /rpool/data tmpfs 32G 24K 32G 1% /var/lib/ceph/osd/ceph-4 tmpfs 32G 24K 32G 1% /var/lib/ceph/osd/ceph-5 s3fs 256T 0 256T 0% /mnt/pve/Tuxis_Marketplace s3fs 256T 0 256T 0% /mnt/pve/Tuxis_Marketplace_Beta /dev/fuse 30M 40K 30M 1% /etc/pve tmpfs 6.3G 4.0K 6.3G 1% /run/user/0 ==== info about virtual guests ==== # qm list # pct list ==== info about network ==== # ip -details -statistics address 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever RX: bytes packets errors dropped missed mcast 729230 6382 0 0 0 0 TX: bytes packets errors dropped carrier collsns 729230 6382 0 0 0 0 2: eno1: mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000 link/ether 0c:c4:7a:d9:1d:f6 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9216 bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.52:ed:27:6f:7b:f3 designated_root 8000.52:ed:27:6f:7b:f3 hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 altname enp6s0 RX: bytes packets errors dropped missed mcast 507910 4689 0 3 0 270 TX: bytes packets errors dropped carrier collsns 425694 1188 0 0 0 0 3: eno2: mtu 1500 qdisc mq master vmbr999 state UP group default qlen 1000 link/ether 0c:c4:7a:d9:1d:f7 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 9216 bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.16:2d:db:6c:6d:8a designated_root 8000.16:2d:db:6c:6d:8a hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 altname enp7s0 RX: bytes packets errors dropped missed mcast 720 8 0 0 0 8 TX: bytes packets errors dropped carrier collsns 36292 370 0 0 0 0 4: vmbr0: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:ed:27:6f:7b:f3 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.52:ed:27:6f:7b:f3 designated_root 8000.52:ed:27:6f:7b:f3 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 204.43 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet6 2a03:7900:111::dc:6/64 scope global valid_lft forever preferred_lft forever inet6 fe80::50ed:27ff:fe6f:7bf3/64 scope link valid_lft forever preferred_lft forever RX: bytes packets errors dropped missed mcast 439912 4664 0 0 0 267 TX: bytes packets errors dropped carrier collsns 414944 1063 0 0 0 0 5: vmbr999: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 16:2d:db:6c:6d:8a brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 1 vlan_protocol 802.1Q bridge_id 8000.16:2d:db:6c:6d:8a designated_root 8000.16:2d:db:6c:6d:8a root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 204.52 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet6 fdb0:5bd1:dc::6/64 scope global valid_lft forever preferred_lft forever inet6 fe80::142d:dbff:fe6c:6d8a/64 scope link valid_lft forever preferred_lft forever RX: bytes packets errors dropped missed mcast 608 8 0 0 0 8 TX: bytes packets errors dropped carrier collsns 36292 370 0 0 0 0 # ip -details -4 route show # ip -details -6 route show unicast ::1 dev lo proto kernel scope global metric 256 pref medium unicast 2a03:7900:111::/64 dev vmbr0 proto kernel scope global metric 256 pref medium unicast fdb0:5bd1:dc::/64 dev vmbr999 proto kernel scope global metric 256 pref medium unicast fdb0:5bd1:cde::/64 via fdb0:5bd1:dc::ffff dev vmbr999 proto boot scope global metric 1024 pref medium unicast fe80::/64 dev vmbr0 proto kernel scope global metric 256 pref medium unicast fe80::/64 dev vmbr999 proto kernel scope global metric 256 pref medium unicast default via 2a03:7900:111::1 dev vmbr0 proto kernel scope global metric 1024 onlink pref medium # cat /etc/network/interfaces # network interface settings; autogenerated # Please do NOT modify this file directly, unless you know what # you're doing. # # If you want to manage parts of the network configuration manually, # please utilize the 'source' or 'source-directory' directives to do # so. # PVE will preserve these directives, but will NOT read its network # configuration from sourced files, so do not attempt to move any of # the PVE managed interfaces into external files! auto lo iface lo inet loopback iface eno1 inet6 manual iface eno2 inet6 manual auto vmbr0 iface vmbr0 inet6 static address 2a03:7900:111::dc:6/64 gateway 2a03:7900:111::1 bridge-ports eno1 bridge-stp off bridge-fd 0 auto vmbr999 iface vmbr999 inet6 static address fdb0:5bd1:dc::6/64 bridge-ports eno2 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 post-up /usr/sbin/ip ro add fdb0:5bd1:cde::/64 via fdb0:5bd1:dc::ffff ==== info about firewall ==== # cat /etc/pve/local/host.fw cat: /etc/pve/local/host.fw: No such file or directory # iptables-save # Generated by iptables-save v1.8.7 on Tue Jun 29 10:30:36 2021 *raw :PREROUTING ACCEPT [376:131554] :OUTPUT ACCEPT [281:128086] COMMIT # Completed on Tue Jun 29 10:30:36 2021 # Generated by iptables-save v1.8.7 on Tue Jun 29 10:30:36 2021 *filter :INPUT ACCEPT [281:128086] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [282:128126] COMMIT # Completed on Tue Jun 29 10:30:36 2021 ==== info about cluster ==== # pvecm nodes Membership information ---------------------- Nodeid Votes Name 3 1 node06 (local) # pvecm status Cluster information ------------------- Name: Demo Config Version: 3 Transport: knet Secure auth: on Quorum information ------------------ Date: Tue Jun 29 10:30:37 2021 Quorum provider: corosync_votequorum Nodes: 1 Node ID: 0x00000003 Ring ID: 3.f4c Quorate: No Votequorum information ---------------------- Expected votes: 3 Highest expected: 3 Total votes: 1 Quorum: 2 Activity blocked Flags: Membership information ---------------------- Nodeid Votes Name 0x00000003 1 fdb0:5bd1:dc::6%32732 (local) # cat /etc/pve/corosync.conf 2>/dev/null logging { debug: off to_syslog: yes } nodelist { node { name: node04 nodeid: 1 quorum_votes: 1 ring0_addr: fdb0:5bd1:dc::4 } node { name: node05 nodeid: 2 quorum_votes: 1 ring0_addr: fdb0:5bd1:dc::5 } node { name: node06 nodeid: 3 quorum_votes: 1 ring0_addr: fdb0:5bd1:dc::6 } } quorum { provider: corosync_votequorum } totem { cluster_name: Demo config_version: 3 interface { linknumber: 0 } ip_version: ipv4-6 link_mode: passive secauth: on version: 2 } # ha-manager status quorum No quorum on node 'node06'! master node06 (old timestamp - dead?, Tue Jun 9 16:20:27 2020) lrm node04 (old timestamp - dead?, Tue Jun 29 10:27:38 2021) lrm node05 (old timestamp - dead?, Tue Jun 29 10:27:38 2021) lrm node06 (old timestamp - dead?, Tue Jun 29 10:27:37 2021) ==== info about hardware ==== # dmidecode -t bios # dmidecode 3.3 Getting SMBIOS data from sysfs. SMBIOS 3.0 present. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: American Megatrends Inc. Version: 2.0a Release Date: 08/01/2016 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16 MB Characteristics: PCI is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported BIOS ROM is socketed EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) ACPI is supported USB legacy is supported BIOS boot specification is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 5.6 # lspci -nnk 00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [8086:2f00] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [15d9:0832] 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f02] (rev 02) Kernel driver in use: pcieport 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f03] (rev 02) Kernel driver in use: pcieport 00:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02) Kernel driver in use: pcieport 00:03.2 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f0a] (rev 02) Kernel driver in use: pcieport 00:04.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [8086:2f20] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [8086:2f21] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [8086:2f22] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [8086:2f23] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [8086:2f24] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [8086:2f25] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [8086:2f26] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [8086:2f27] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [8086:2f28] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [15d9:0832] 00:05.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [8086:2f29] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [15d9:0832] 00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [8086:2f2a] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [15d9:0832] 00:05.4 PIC [0800]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [8086:2f2c] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [15d9:0832] 00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR [8086:8d7c] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] 00:11.4 SATA controller [0106]: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] [8086:8d62] (rev 05) Subsystem: Super Micro Computer Inc C610/X99 series chipset sSATA Controller [AHCI mode] [15d9:0832] Kernel driver in use: ahci Kernel modules: ahci 00:14.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB xHCI Host Controller [8086:8d31] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: xhci_hcd Kernel modules: xhci_pci 00:16.0 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #1 [8086:8d3a] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel modules: mei_me 00:16.1 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #2 [8086:8d3b] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] 00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 [8086:8d10] (rev d5) Kernel driver in use: pcieport 00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 [8086:8d18] (rev d5) Kernel driver in use: pcieport 00:1c.5 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #6 [8086:8d1a] (rev d5) Kernel driver in use: pcieport 00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 [8086:8d1c] (rev d5) Kernel driver in use: pcieport 00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99 series chipset LPC Controller [8086:8d44] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: lpc_ich Kernel modules: lpc_ich 00:1f.2 SATA controller [0106]: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [8086:8d02] (rev 05) Subsystem: Super Micro Computer Inc C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [15d9:0832] Kernel driver in use: ahci Kernel modules: ahci 00:1f.3 SMBus [0c05]: Intel Corporation C610/X99 series chipset SMBus Controller [8086:8d22] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: i801_smbus Kernel modules: i2c_i801 06:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) DeviceName: Intel Ethernet i210AT #1 Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533] Kernel driver in use: igb Kernel modules: igb 07:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) DeviceName: Intel Ethernet i210AT #2 Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533] Kernel driver in use: igb Kernel modules: igb 08:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03) 09:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30) DeviceName: ASPEED Video AST2400 Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: ast Kernel modules: ast ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f81] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832] ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f36] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832] Kernel driver in use: hswep_uncore ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f37] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832] Kernel driver in use: hswep_uncore ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe1] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe2] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe3] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff8] (rev 02) ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff9] (rev 02) ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffc] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832] ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffd] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832] ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffe] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832] ff:10.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f1d] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [15d9:0832] ff:10.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f34] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [15d9:0832] Kernel driver in use: hswep_uncore ff:10.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1e] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832] ff:10.6 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f7d] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832] ff:10.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1f] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832] ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2fa0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [15d9:0832] ff:12.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2f30] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [15d9:0832] Kernel driver in use: hswep_uncore ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [15d9:0832] ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [15d9:0832] ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2faa] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fab] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fac] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fad] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast [8086:2fae] (rev 02) ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2faf] (rev 02) ff:14.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [8086:2fb0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:14.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [8086:2fb1] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:14.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [8086:2fb2] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [15d9:0832] ff:14.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [8086:2fb3] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [15d9:0832] ff:14.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbc] (rev 02) ff:14.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbd] (rev 02) ff:14.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbe] (rev 02) ff:14.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbf] (rev 02) ff:15.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [8086:2fb4] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:15.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [8086:2fb5] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:15.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [8086:2fb6] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [15d9:0832] ff:15.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [8086:2fb7] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [15d9:0832] ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers [8086... (rev 02) ff:16.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast [8086:2f6e] (rev 02) ff:16.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2f6f] (rev 02) ff:17.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control [8086:2fd0] (rev 02) Kernel driver in use: hswep_uncore ff:17.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb8] (rev 02) ff:17.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb9] (rev 02) ff:17.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fba] (rev 02) ff:17.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fbb] (rev 02) ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f98] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f99] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9a] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2fc0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9c] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f88] (rev 02) ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f8a] (rev 02) ==== info about block devices ==== # lsblk --ascii NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 111.8G 0 disk |-sda1 8:1 0 1007K 0 part |-sda2 8:2 0 512M 0 part `-sda3 8:3 0 111.3G 0 part sdb 8:16 0 111.8G 0 disk |-sdb1 8:17 0 1007K 0 part |-sdb2 8:18 0 512M 0 part `-sdb3 8:19 0 111.3G 0 part sdc 8:32 0 447.1G 0 disk `-ceph--33bdcbd7--07be--4373--97ca--0678dda8888d-osd--block--e2deed6d--596f--4837--b14e--88c9afdbe531 253:0 0 447.1G 0 lvm sdd 8:48 0 447.1G 0 disk `-ceph--97bdf879--bbf1--41ba--8563--81abe42cf617-osd--block--55199458--8b33--44f2--b4d2--3a876072a622 253:1 0 447.1G 0 lvm # ls -l /dev/disk/by-*/ /dev/disk/by-id/: total 0 lrwxrwxrwx 1 root root 9 Jun 29 10:29 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNXAH102524 -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 10:29 ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNXAH102531 -> ../../sdc lrwxrwxrwx 1 root root 10 Jun 29 10:29 dm-name-ceph--33bdcbd7--07be--4373--97ca--0678dda8888d-osd--block--e2deed6d--596f--4837--b14e--88c9afdbe531 -> ../../dm-0 lrwxrwxrwx 1 root root 10 Jun 29 10:29 dm-name-ceph--97bdf879--bbf1--41ba--8563--81abe42cf617-osd--block--55199458--8b33--44f2--b4d2--3a876072a622 -> ../../dm-1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 dm-uuid-LVM-GHM6Bwl9TQ7jv5GJd8ORRD6XDearTRZhgvpxQ22a3TWdlBd9iGk1oHhop5lXn8lL -> ../../dm-1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 dm-uuid-LVM-hoOm4ydEDwKOrnNdVuCBCsY31it5n1ZRDsf4uP4Irce8u2hubaahZCqfMz9IpwhI -> ../../dm-0 lrwxrwxrwx 1 root root 9 Jun 29 10:29 lvm-pv-uuid-AGbSTn-aDmD-AbAR-ngCX-8glc-2KVW-xal2xh -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 10:29 lvm-pv-uuid-QPw8aR-Rbbe-LzZ7-0j3t-n8gn-OeOs-YWPaoV -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 10:29 wwn-0x5002538c00018347 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x5002538c00018347-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x5002538c00018347-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x5002538c00018347-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 wwn-0x5002538c40146ccb -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 10:29 wwn-0x5002538c40146cd2 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 10:29 wwn-0x55cd2e414db345fd -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x55cd2e414db345fd-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x55cd2e414db345fd-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x55cd2e414db345fd-part3 -> ../../sda3 /dev/disk/by-label/: total 0 lrwxrwxrwx 1 root root 10 Jun 29 10:29 rpool -> ../../sda3 /dev/disk/by-partuuid/: total 0 lrwxrwxrwx 1 root root 10 Jun 29 10:29 4f42744a-eef7-49f5-bfa4-5cb3ca1ee4b2 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 70166c71-7a1f-400e-bd39-f8f4be867d3e -> ../../sda3 lrwxrwxrwx 1 root root 10 Jun 29 10:29 87402126-9aa6-4be9-9c13-4704492a974b -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 a52ed3d9-d18c-4d5b-9d8a-c92b235fd9e1 -> ../../sdb3 lrwxrwxrwx 1 root root 10 Jun 29 10:29 de77a2cb-a1df-460e-97a2-3c8c8ae9fad5 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 fb306c92-2607-46a5-a32d-7556b04dd494 -> ../../sda2 /dev/disk/by-path/: total 0 lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:11.4-ata-3 -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:11.4-ata-3.0 -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3.0-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3.0-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3.0-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:11.4-ata-4 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:11.4-ata-4.0 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4.0-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4.0-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4.0-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:1f.2-ata-1 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:1f.2-ata-1.0 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:1f.2-ata-2 -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:1f.2-ata-2.0 -> ../../sdd /dev/disk/by-uuid/: total 0 lrwxrwxrwx 1 root root 10 Jun 29 10:29 17716103480993325194 -> ../../sda3 lrwxrwxrwx 1 root root 10 Jun 29 10:29 B851-E178 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 B852-ACFC -> ../../sdb2 # iscsiadm -m node iscsiadm: No records found # iscsiadm -m session iscsiadm: No active sessions. ==== info about volumes ==== # pvs PV VG Fmt Attr PSize PFree /dev/sdc ceph-33bdcbd7-07be-4373-97ca-0678dda8888d lvm2 a-- <447.13g 0 /dev/sdd ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 lvm2 a-- <447.13g 0 # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd-block-e2deed6d-596f-4837-b14e-88c9afdbe531 ceph-33bdcbd7-07be-4373-97ca-0678dda8888d -wi-a----- <447.13g osd-block-55199458-8b33-44f2-b4d2-3a876072a622 ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 -wi-a----- <447.13g # vgs VG #PV #LV #SN Attr VSize VFree ceph-33bdcbd7-07be-4373-97ca-0678dda8888d 1 1 0 wz--n- <447.13g 0 ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 1 1 0 wz--n- <447.13g 0 # zpool status pool: rpool state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0B in 00:00:19 with 0 errors on Sun Jun 13 00:24:20 2021 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 ONLINE 0 0 0 errors: No known data errors # zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT rpool 111G 2.43G 109G - - 5% 2% 1.00x ONLINE - ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 111G 2.43G 109G - - 5% 2.18% - ONLINE # zfs list NAME USED AVAIL REFER MOUNTPOINT rpool 2.43G 105G 104K /rpool rpool/ROOT 2.42G 105G 96K /rpool/ROOT rpool/ROOT/pve-1 2.42G 105G 2.42G / rpool/data 96K 105G 96K /rpool/data # pveceph status ERROR: command 'pveceph status' failed: got timeout # ceph osd status ERROR: command 'ceph osd status' failed: got timeout # ceph df ERROR: command 'ceph df' failed: got timeout # ceph osd df tree ERROR: command 'ceph osd df tree' failed: got timeout # cat /etc/ceph/ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = fdb0:5bd1:dc::4/64 fsid = 73045ca5-eead-4e44-a0c1-b6796ed3d7d5 mon_allow_pool_delete = true mon_host = fdb0:5bd1:dc::4 fdb0:5bd1:dc::5 fdb0:5bd1:dc::6 ms_bind_ipv4 = false ms_bind_ipv6 = true osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = fdb0:5bd1:dc::4/64 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.node04] host = node04 mds_standby_for_name = pve [mds.node05] host = node05 mds_standby_for_name = pve [mds.node06] host = node06 mds standby for name = pve [mon.node04] public_addr = fdb0:5bd1:dc::4 [mon.node05] public_addr = fdb0:5bd1:dc::5 [mon.node06] public_addr = fdb0:5bd1:dc::6 # ceph config dump ERROR: command 'ceph config dump' failed: got timeout # pveceph pool ls got timeout # ceph versions ERROR: command 'ceph versions' failed: got timeout