==== general system info ==== # hostname node06 # pveversion --verbose proxmox-ve: 7.0-2 (running kernel: 5.11.22-1-pve) pve-manager: 7.0-5 (running version: 7.0-5/cce9b25f) pve-kernel-5.11: 7.0-3 pve-kernel-helper: 7.0-3 pve-kernel-5.4: 6.4-3 pve-kernel-5.11.22-1-pve: 5.11.22-1 pve-kernel-5.4.119-1-pve: 5.4.119-1 pve-kernel-5.4.106-1-pve: 5.4.106-1 pve-kernel-5.4.34-1-pve: 5.4.34-2 ceph: 15.2.13-pve1 ceph-fuse: 15.2.13-pve1 corosync: 3.1.2-pve2 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown: residual config ifupdown2: 3.0.0-1+pve5 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.21-pve1 libproxmox-acme-perl: 1.1.0 libproxmox-backup-qemu0: 1.0.3-1 libpve-access-control: 7.0-3 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.0-4 libpve-guest-common-perl: 4.0-2 libpve-http-server-perl: 4.0-2 libpve-storage-perl: 7.0-6 libqb0: 1.0.5-1 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 4.0.9-1 lxcfs: 4.0.8-pve1 novnc-pve: 1.2.0-3 proxmox-backup-client: 1.1.10-1 proxmox-backup-file-restore: 1.1.10-1 proxmox-mini-journalreader: 1.2-1 proxmox-widget-toolkit: 3.1-4 pve-cluster: 7.0-2 pve-container: 4.0-3 pve-docs: 7.0-3 pve-edk2-firmware: 3.20200531-1 pve-firewall: 4.2-2 pve-firmware: 3.2-4 pve-ha-manager: 3.2-2 pve-i18n: 2.3-1 pve-qemu-kvm: 6.0.0-2 pve-xtermjs: 4.12.0-1 qemu-server: 7.0-5 smartmontools: 7.2-pve2 spiceterm: 3.2-2 vncterm: 1.7-1 zfsutils-linux: 2.0.4-pve1 # cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 2a03:7900:111::dc:6 node06.demo.customers.tuxis.net node06 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts #2a03:7900:111:0:90d4:a7ff:fe7f:ccf6 pbs.tuxis.net # pvesubscription get message: There is no subscription key serverid: F3A59435D1B87C5A2460F965646A3177 status: NotFound url: https://www.proxmox.com/proxmox-ve/pricing # cat /etc/apt/sources.list ## Managed via Ansible deb http://debmirror.tuxis.nl/debian/ bullseye main contrib non-free deb-src http://debmirror.tuxis.nl/debian/ bullseye main contrib non-free deb http://security.debian.org/ bullseye-security main contrib non-free deb-src http://security.debian.org/ bullseye-security main contrib non-free deb http://debmirror.tuxis.nl/debian/ bullseye-updates main contrib non-free deb-src http://debmirror.tuxis.nl/debian/ bullseye-updates main contrib non-free # cat /etc/apt/sources.list.d/pvetest-for-beta.list deb http://download.proxmox.com/debian/pve bullseye pvetest # cat /etc/apt/sources.list.d/ceph.list deb http://download.proxmox.com/debian/ceph-octopus bullseye main # cat /etc/apt/sources.list.d/apt_tuxis_nl_tuxis.list deb https://apt.tuxis.nl/tuxis/ tuxis-cron main deb https://apt.tuxis.nl/tuxis/ monitoring main deb https://apt.tuxis.nl/tuxis/ pmrb main # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz Stepping: 2 CPU MHz: 3600.000 CPU max MHz: 3600.0000 CPU min MHz: 1200.0000 BogoMIPS: 7000.21 Virtualization: VT-x L1d cache: 128 KiB L1i cache: 128 KiB L2 cache: 1 MiB L3 cache: 10 MiB NUMA node0 CPU(s): 0-7 Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d # pvesh get /cluster/resources --type node --output-format=yaml --- - cpu: 0.011763712868024 disk: 2595618816 id: node/node06 level: '' maxcpu: 8 maxdisk: 115451363328 maxmem: 67322179584 mem: 3120074752 node: node06 status: online type: node uptime: 2238 - cpu: 0.188962970750996 disk: 21875785728 id: node/node04 level: '' maxcpu: 8 maxdisk: 101597184000 maxmem: 67331584000 mem: 23079858176 node: node04 status: online type: node uptime: 13969251 - cpu: 0.0102829414157591 disk: 3339059200 id: node/node05 level: '' maxcpu: 8 maxdisk: 115422265344 maxmem: 67331592192 mem: 8491147264 node: node05 status: online type: node uptime: 2830727 ==== overall system load info ==== # top -b -c -w512 -n 1 -o TIME | head -n 30 top - 10:26:43 up 37 min, 1 user, load average: 0.13, 0.08, 0.06 Tasks: 295 total, 1 running, 294 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.8 us, 1.5 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 64203.4 total, 60470.7 free, 2954.7 used, 778.0 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 60187.0 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1647 root rt 0 571448 178096 53256 S 6.2 0.3 0:53.45 /usr/sbin/corosync -f 1663 ceph 20 0 1400308 570700 28140 S 0.0 0.9 0:39.35 /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph 1656 ceph 20 0 1298416 546100 28064 S 0.0 0.8 0:27.35 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph 1644 ceph 20 0 507260 123516 20868 S 0.0 0.2 0:24.52 /usr/bin/ceph-mon -f --cluster ceph --id node06 --setuser ceph --setgroup ceph 1762 root 20 0 270424 95672 9780 S 0.0 0.1 0:23.76 pvestatd 1888 www-data 20 0 346704 127472 8076 S 0.0 0.2 0:09.48 pveproxy worker 1887 www-data 20 0 346736 126700 7424 S 0.0 0.2 0:08.86 pveproxy worker 1889 www-data 20 0 346604 126552 7296 S 0.0 0.2 0:07.14 pveproxy worker 1763 root 20 0 264748 84656 4284 S 0.0 0.1 0:06.55 pve-firewall 1550 root 20 0 618448 55796 50792 S 0.0 0.1 0:05.45 /usr/bin/pmxcfs 1643 ceph 20 0 503992 176144 19884 S 0.0 0.3 0:04.03 /usr/bin/ceph-mgr -f --cluster ceph --id node06 --setuser ceph --setgroup ceph 1789 root 20 0 345064 124220 6524 S 0.0 0.2 0:02.80 pvedaemon worker 13 root 20 0 0 0 0 I 0.0 0.0 0:02.48 [rcu_sched] 1788 root 20 0 345068 123856 6204 S 0.0 0.2 0:02.40 pvedaemon worker 1790 root 20 0 345064 123804 6168 S 0.0 0.2 0:02.40 pvedaemon worker 1 root 20 0 165772 8880 5280 S 0.0 0.0 0:02.17 /sbin/init 1335 Debian-+ 20 0 24352 10532 5780 S 0.0 0.0 0:01.40 /usr/sbin/snmpd -LOw -u Debian-snmp -g Debian-snmp -I -smux mteTrigger mteTriggerConf -f -p /run/snmpd.pid 1642 ceph 20 0 302080 27288 11188 S 0.0 0.0 0:00.85 /usr/bin/ceph-mds -f --cluster ceph --id node06 --setuser ceph --setgroup ceph 421 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss] 422 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss] 423 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss] 424 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss] 425 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss] # head /proc/pressure/* ==> /proc/pressure/cpu <== some avg10=0.00 avg60=0.00 avg300=0.00 total=5430541 ==> /proc/pressure/io <== some avg10=0.00 avg60=0.00 avg300=0.00 total=1300222 full avg10=0.00 avg60=0.00 avg300=0.00 total=1149011 ==> /proc/pressure/memory <== some avg10=0.00 avg60=0.00 avg300=0.00 total=0 full avg10=0.00 avg60=0.00 avg300=0.00 total=0 ==== info about storage ==== # cat /etc/pve/storage.cfg dir: local path /var/lib/vz content images,backup,iso,vztmpl prune-backups keep-last=1 shared 0 zfspool: local-zfs pool rpool/data content rootdir,images sparse 1 rbd: Ceph content rootdir,images krbd 0 pool Ceph cephfs: CephFS path /mnt/pve/CephFS content iso,snippets,backup,vztmpl prune-backups keep-last=1 dir: Tuxis_Marketplace path /mnt/pve/Tuxis_Marketplace content iso,backup is_mountpoint yes mkdir 0 shared 1 dir: Tuxis_Marketplace_Beta path /mnt/pve/Tuxis_Marketplace_Beta content backup,iso is_mountpoint yes mkdir 0 shared 1 rbd: CephKRBD content images krbd 1 pool Ceph pbs: pbs002.tuxis.nl datastore DB0220_demo server pbs002.tuxis.nl content backup encryption-key 68:d5:89:f6:f1:f4:67:59:1b:74:6a:78:99:11:ad:09:a0:b0:12:db:43:8d:41:19:af:38:90:77:12:c1:6d:f8 fingerprint 45:f8:79:eb:27:96:88:6b:29:ad:21:00:13:c6:bd:b8:30:f6:f3:9b:f0:bf:dd:f3:ad:f0:09:d5:d2:9a:34:79 prune-backups keep-last=1 username DB0220@pbs # pvesm status Name Type Status Total Used Available % Ceph rbd active 802501642 337252874 465248768 42.03% CephFS cephfs active 500432896 35184640 465248256 7.03% CephKRBD rbd active 802501642 337252874 465248768 42.03% Tuxis_Marketplace dir active 274877906944 0 274877906944 0.00% Tuxis_Marketplace_Beta dir active 274877906944 0 274877906944 0.00% local dir active 112745472 2534784 110210688 2.25% local-zfs zfspool active 110210824 96 110210728 0.00% pbs002.tuxis.nl pbs active 1701798656 10516096 1691282560 0.62% # cat /etc/fstab # proc /proc proc defaults 0 0 # findmnt --ascii TARGET SOURCE FSTYPE OPTIONS / rpool/ROOT/pve-1 zfs rw,relatime,xattr,noacl |-/sys sysfs sysfs rw,nosuid,nodev,noexec,relatime | |-/sys/kernel/security securityfs securityfs rw,nosuid,nodev,noexec,relatime | |-/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime | |-/sys/fs/pstore pstore pstore rw,nosuid,nodev,noexec,relatime | |-/sys/fs/bpf none bpf rw,nosuid,nodev,noexec,relatime,mode=700 | |-/sys/kernel/debug debugfs debugfs rw,nosuid,nodev,noexec,relatime | |-/sys/kernel/tracing tracefs tracefs rw,nosuid,nodev,noexec,relatime | |-/sys/fs/fuse/connections fusectl fusectl rw,nosuid,nodev,noexec,relatime | `-/sys/kernel/config configfs configfs rw,nosuid,nodev,noexec,relatime |-/proc proc proc rw,relatime | `-/proc/sys/fs/binfmt_misc systemd-1 autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=561 |-/dev udev devtmpfs rw,nosuid,relatime,size=32838164k,nr_inodes=8209541,mode=755,inode64 | |-/dev/pts devpts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 | |-/dev/shm tmpfs tmpfs rw,nosuid,nodev,inode64 | |-/dev/mqueue mqueue mqueue rw,nosuid,nodev,noexec,relatime | `-/dev/hugepages hugetlbfs hugetlbfs rw,relatime,pagesize=2M |-/run tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=6574432k,mode=755,inode64 | |-/run/lock tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 | |-/run/rpc_pipefs sunrpc rpc_pipefs rw,relatime | `-/run/user/0 tmpfs tmpfs rw,nosuid,nodev,relatime,size=6574428k,nr_inodes=1643607,mode=700,inode64 |-/rpool rpool zfs rw,noatime,xattr,noacl | |-/rpool/ROOT rpool/ROOT zfs rw,noatime,xattr,noacl | `-/rpool/data rpool/data zfs rw,noatime,xattr,noacl |-/var/lib/ceph/osd/ceph-4 tmpfs tmpfs rw,relatime,inode64 |-/var/lib/ceph/osd/ceph-5 tmpfs tmpfs rw,relatime,inode64 |-/mnt/pve/Tuxis_Marketplace_Beta s3fs fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0 |-/mnt/pve/Tuxis_Marketplace s3fs fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0 |-/etc/pve /dev/fuse fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other |-/var/lib/lxcfs lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other `-/mnt/pve/CephFS [fdb0:5bd1:dc::4],[fdb0:5bd1:dc::5],[fdb0:5bd1:dc::6]:/ ceph rw,relatime,name=admin,secret=,acl # df --human Filesystem Size Used Avail Use% Mounted on udev 32G 0 32G 0% /dev tmpfs 6.3G 1.3M 6.3G 1% /run rpool/ROOT/pve-1 108G 2.5G 106G 3% / tmpfs 32G 63M 32G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock rpool 106G 128K 106G 1% /rpool rpool/ROOT 106G 128K 106G 1% /rpool/ROOT rpool/data 106G 128K 106G 1% /rpool/data tmpfs 32G 28K 32G 1% /var/lib/ceph/osd/ceph-4 tmpfs 32G 28K 32G 1% /var/lib/ceph/osd/ceph-5 s3fs 256T 0 256T 0% /mnt/pve/Tuxis_Marketplace_Beta s3fs 256T 0 256T 0% /mnt/pve/Tuxis_Marketplace /dev/fuse 30M 40K 30M 1% /etc/pve [fdb0:5bd1:dc::4],[fdb0:5bd1:dc::5],[fdb0:5bd1:dc::6]:/ 478G 34G 444G 8% /mnt/pve/CephFS tmpfs 6.3G 4.0K 6.3G 1% /run/user/0 ==== info about network ==== # ip -details -statistics address 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever RX: bytes packets errors dropped missed mcast 55254397 58681 0 0 0 0 TX: bytes packets errors dropped carrier collsns 55254397 58681 0 0 0 0 2: eno1: mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000 link/ether 0c:c4:7a:d9:1d:f6 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9216 bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.52:ed:27:6f:7b:f3 designated_root 8000.52:ed:27:6f:7b:f3 hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 altname enp6s0 RX: bytes packets errors dropped missed mcast 15982113 105257 0 74 0 5642 TX: bytes packets errors dropped carrier collsns 12624090 36080 0 0 0 0 3: eno2: mtu 1500 qdisc mq master vmbr999 state UP group default qlen 1000 link/ether 0c:c4:7a:d9:1d:f7 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9216 bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.16:2d:db:6c:6d:8a designated_root 8000.16:2d:db:6c:6d:8a hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 altname enp7s0 RX: bytes packets errors dropped missed mcast 489871250 699399 0 0 0 33 TX: bytes packets errors dropped carrier collsns 268886467 574470 0 0 0 0 4: vmbr0: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:ed:27:6f:7b:f3 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.52:ed:27:6f:7b:f3 designated_root 8000.52:ed:27:6f:7b:f3 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 23.76 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet6 2a03:7900:111::dc:6/64 scope global valid_lft forever preferred_lft forever inet6 fe80::50ed:27ff:fe6f:7bf3/64 scope link valid_lft forever preferred_lft forever RX: bytes packets errors dropped missed mcast 14379299 103674 0 0 0 5565 TX: bytes packets errors dropped carrier collsns 12356802 32972 0 0 0 0 5: vmbr999: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 16:2d:db:6c:6d:8a brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.16:2d:db:6c:6d:8a designated_root 8000.16:2d:db:6c:6d:8a root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 160.33 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet6 fdb0:5bd1:dc::6/64 scope global valid_lft forever preferred_lft forever inet6 fe80::142d:dbff:fe6c:6d8a/64 scope link valid_lft forever preferred_lft forever RX: bytes packets errors dropped missed mcast 465681104 499419 0 0 0 33 TX: bytes packets errors dropped carrier collsns 260240217 473931 0 0 0 0 # ip -details -4 route show # ip -details -6 route show unicast ::1 dev lo proto kernel scope global metric 256 pref medium unicast 2a03:7900:111::/64 dev vmbr0 proto kernel scope global metric 256 pref medium unicast fdb0:5bd1:dc::/64 dev vmbr999 proto kernel scope global metric 256 pref medium unicast fdb0:5bd1:cde::/64 via fdb0:5bd1:dc::ffff dev vmbr999 proto boot scope global metric 1024 pref medium unicast fe80::/64 dev vmbr999 proto kernel scope global metric 256 pref medium unicast fe80::/64 dev vmbr0 proto kernel scope global metric 256 pref medium unicast default via 2a03:7900:111::1 dev vmbr0 proto kernel scope global metric 1024 onlink pref medium # cat /etc/network/interfaces # network interface settings; autogenerated # Please do NOT modify this file directly, unless you know what # you're doing. # # If you want to manage parts of the network configuration manually, # please utilize the 'source' or 'source-directory' directives to do # so. # PVE will preserve these directives, but will NOT read its network # configuration from sourced files, so do not attempt to move any of # the PVE managed interfaces into external files! auto lo iface lo inet loopback iface eno1 inet6 manual iface eno2 inet6 manual auto vmbr0 iface vmbr0 inet6 static address 2a03:7900:111::dc:6/64 gateway 2a03:7900:111::1 bridge-ports eno1 bridge-stp off bridge-fd 0 auto vmbr999 iface vmbr999 inet6 static address fdb0:5bd1:dc::6/64 bridge-ports eno2 bridge-stp off bridge-fd 0 #bridge-vlan-aware yes #bridge-vids 2-4094 post-up /usr/sbin/ip ro add fdb0:5bd1:cde::/64 via fdb0:5bd1:dc::ffff ==== info about virtual guests ==== # qm list # pct list ==== info about firewall ==== # cat /etc/pve/local/host.fw cat: /etc/pve/local/host.fw: No such file or directory # iptables-save # Generated by iptables-save v1.8.7 on Tue Jun 29 10:26:45 2021 *raw :PREROUTING ACCEPT [10644:3275736] :OUTPUT ACCEPT [8292:3186298] COMMIT # Completed on Tue Jun 29 10:26:45 2021 # Generated by iptables-save v1.8.7 on Tue Jun 29 10:26:45 2021 *filter :INPUT ACCEPT [8292:3186298] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [8293:3186338] COMMIT # Completed on Tue Jun 29 10:26:45 2021 ==== info about cluster ==== # pvecm nodes Membership information ---------------------- Nodeid Votes Name 1 1 node04 2 1 node05 3 1 node06 (local) # pvecm status Cluster information ------------------- Name: Demo Config Version: 3 Transport: knet Secure auth: on Quorum information ------------------ Date: Tue Jun 29 10:26:46 2021 Quorum provider: corosync_votequorum Nodes: 3 Node ID: 0x00000003 Ring ID: 1.f47 Quorate: Yes Votequorum information ---------------------- Expected votes: 3 Highest expected: 3 Total votes: 3 Quorum: 2 Flags: Quorate Membership information ---------------------- Nodeid Votes Name 0x00000001 1 fdb0:5bd1:dc::4%32617 0x00000002 1 fdb0:5bd1:dc::5%32617 0x00000003 1 fdb0:5bd1:dc::6%32617 (local) # cat /etc/pve/corosync.conf 2>/dev/null logging { debug: off to_syslog: yes } nodelist { node { name: node04 nodeid: 1 quorum_votes: 1 ring0_addr: fdb0:5bd1:dc::4 } node { name: node05 nodeid: 2 quorum_votes: 1 ring0_addr: fdb0:5bd1:dc::5 } node { name: node06 nodeid: 3 quorum_votes: 1 ring0_addr: fdb0:5bd1:dc::6 } } quorum { provider: corosync_votequorum } totem { cluster_name: Demo config_version: 3 interface { linknumber: 0 } ip_version: ipv4-6 link_mode: passive secauth: on version: 2 } # ha-manager status quorum OK master node06 (idle, Tue Jun 9 16:20:27 2020) lrm node04 (idle, Tue Jun 29 10:26:43 2021) lrm node05 (idle, Tue Jun 29 10:26:43 2021) lrm node06 (idle, Tue Jun 29 10:26:41 2021) ==== info about hardware ==== # dmidecode -t bios # dmidecode 3.3 Getting SMBIOS data from sysfs. SMBIOS 3.0 present. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: American Megatrends Inc. Version: 2.0a Release Date: 08/01/2016 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16 MB Characteristics: PCI is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported BIOS ROM is socketed EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) ACPI is supported USB legacy is supported BIOS boot specification is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 5.6 # lspci -nnk 00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [8086:2f00] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [15d9:0832] 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f02] (rev 02) Kernel driver in use: pcieport 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f03] (rev 02) Kernel driver in use: pcieport 00:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02) Kernel driver in use: pcieport 00:03.2 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f0a] (rev 02) Kernel driver in use: pcieport 00:04.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [8086:2f20] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [8086:2f21] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [8086:2f22] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [8086:2f23] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [8086:2f24] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [8086:2f25] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [8086:2f26] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [8086:2f27] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [8086:2f28] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [15d9:0832] 00:05.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [8086:2f29] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [15d9:0832] 00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [8086:2f2a] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [15d9:0832] 00:05.4 PIC [0800]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [8086:2f2c] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [15d9:0832] 00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR [8086:8d7c] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] 00:11.4 SATA controller [0106]: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] [8086:8d62] (rev 05) Subsystem: Super Micro Computer Inc C610/X99 series chipset sSATA Controller [AHCI mode] [15d9:0832] Kernel driver in use: ahci Kernel modules: ahci 00:14.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB xHCI Host Controller [8086:8d31] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: xhci_hcd Kernel modules: xhci_pci 00:16.0 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #1 [8086:8d3a] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel modules: mei_me 00:16.1 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #2 [8086:8d3b] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] 00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 [8086:8d10] (rev d5) Kernel driver in use: pcieport 00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 [8086:8d18] (rev d5) Kernel driver in use: pcieport 00:1c.5 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #6 [8086:8d1a] (rev d5) Kernel driver in use: pcieport 00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 [8086:8d1c] (rev d5) Kernel driver in use: pcieport 00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99 series chipset LPC Controller [8086:8d44] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: lpc_ich Kernel modules: lpc_ich 00:1f.2 SATA controller [0106]: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [8086:8d02] (rev 05) Subsystem: Super Micro Computer Inc C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [15d9:0832] Kernel driver in use: ahci Kernel modules: ahci 00:1f.3 SMBus [0c05]: Intel Corporation C610/X99 series chipset SMBus Controller [8086:8d22] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: i801_smbus Kernel modules: i2c_i801 06:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) DeviceName: Intel Ethernet i210AT #1 Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533] Kernel driver in use: igb Kernel modules: igb 07:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) DeviceName: Intel Ethernet i210AT #2 Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533] Kernel driver in use: igb Kernel modules: igb 08:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03) 09:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30) DeviceName: ASPEED Video AST2400 Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: ast Kernel modules: ast ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f81] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832] ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f36] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832] Kernel driver in use: hswep_uncore ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f37] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832] Kernel driver in use: hswep_uncore ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe1] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe2] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe3] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff8] (rev 02) ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff9] (rev 02) ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffc] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832] ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffd] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832] ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffe] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832] ff:10.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f1d] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [15d9:0832] ff:10.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f34] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [15d9:0832] Kernel driver in use: hswep_uncore ff:10.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1e] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832] ff:10.6 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f7d] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832] ff:10.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1f] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832] ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2fa0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [15d9:0832] ff:12.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2f30] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [15d9:0832] Kernel driver in use: hswep_uncore ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [15d9:0832] ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [15d9:0832] ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2faa] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fab] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fac] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fad] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast [8086:2fae] (rev 02) ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2faf] (rev 02) ff:14.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [8086:2fb0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:14.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [8086:2fb1] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:14.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [8086:2fb2] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [15d9:0832] ff:14.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [8086:2fb3] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [15d9:0832] ff:14.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbc] (rev 02) ff:14.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbd] (rev 02) ff:14.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbe] (rev 02) ff:14.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbf] (rev 02) ff:15.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [8086:2fb4] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:15.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [8086:2fb5] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:15.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [8086:2fb6] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [15d9:0832] ff:15.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [8086:2fb7] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [15d9:0832] ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers [8086... (rev 02) ff:16.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast [8086:2f6e] (rev 02) ff:16.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2f6f] (rev 02) ff:17.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control [8086:2fd0] (rev 02) Kernel driver in use: hswep_uncore ff:17.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb8] (rev 02) ff:17.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb9] (rev 02) ff:17.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fba] (rev 02) ff:17.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fbb] (rev 02) ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f98] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f99] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9a] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2fc0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9c] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f88] (rev 02) ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f8a] (rev 02) ==== info about block devices ==== # lsblk --ascii NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 111.8G 0 disk |-sda1 8:1 0 1007K 0 part |-sda2 8:2 0 512M 0 part `-sda3 8:3 0 111.3G 0 part sdb 8:16 0 111.8G 0 disk |-sdb1 8:17 0 1007K 0 part |-sdb2 8:18 0 512M 0 part `-sdb3 8:19 0 111.3G 0 part sdc 8:32 0 447.1G 0 disk `-ceph--33bdcbd7--07be--4373--97ca--0678dda8888d-osd--block--e2deed6d--596f--4837--b14e--88c9afdbe531 253:0 0 447.1G 0 lvm sdd 8:48 0 447.1G 0 disk `-ceph--97bdf879--bbf1--41ba--8563--81abe42cf617-osd--block--55199458--8b33--44f2--b4d2--3a876072a622 253:1 0 447.1G 0 lvm # ls -l /dev/disk/by-*/ /dev/disk/by-id/: total 0 lrwxrwxrwx 1 root root 9 Jun 29 09:49 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNXAH102524 -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 09:49 ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNXAH102531 -> ../../sdc lrwxrwxrwx 1 root root 10 Jun 29 09:49 dm-name-ceph--33bdcbd7--07be--4373--97ca--0678dda8888d-osd--block--e2deed6d--596f--4837--b14e--88c9afdbe531 -> ../../dm-0 lrwxrwxrwx 1 root root 10 Jun 29 09:49 dm-name-ceph--97bdf879--bbf1--41ba--8563--81abe42cf617-osd--block--55199458--8b33--44f2--b4d2--3a876072a622 -> ../../dm-1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 dm-uuid-LVM-GHM6Bwl9TQ7jv5GJd8ORRD6XDearTRZhgvpxQ22a3TWdlBd9iGk1oHhop5lXn8lL -> ../../dm-1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 dm-uuid-LVM-hoOm4ydEDwKOrnNdVuCBCsY31it5n1ZRDsf4uP4Irce8u2hubaahZCqfMz9IpwhI -> ../../dm-0 lrwxrwxrwx 1 root root 9 Jun 29 09:49 lvm-pv-uuid-AGbSTn-aDmD-AbAR-ngCX-8glc-2KVW-xal2xh -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 09:49 lvm-pv-uuid-QPw8aR-Rbbe-LzZ7-0j3t-n8gn-OeOs-YWPaoV -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 09:49 wwn-0x5002538c00018347 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x5002538c00018347-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x5002538c00018347-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x5002538c00018347-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 wwn-0x5002538c40146ccb -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 09:49 wwn-0x5002538c40146cd2 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 09:49 wwn-0x55cd2e414db345fd -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x55cd2e414db345fd-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x55cd2e414db345fd-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x55cd2e414db345fd-part3 -> ../../sda3 /dev/disk/by-label/: total 0 lrwxrwxrwx 1 root root 10 Jun 29 09:49 rpool -> ../../sda3 /dev/disk/by-partuuid/: total 0 lrwxrwxrwx 1 root root 10 Jun 29 09:49 4f42744a-eef7-49f5-bfa4-5cb3ca1ee4b2 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 70166c71-7a1f-400e-bd39-f8f4be867d3e -> ../../sda3 lrwxrwxrwx 1 root root 10 Jun 29 09:49 87402126-9aa6-4be9-9c13-4704492a974b -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 a52ed3d9-d18c-4d5b-9d8a-c92b235fd9e1 -> ../../sdb3 lrwxrwxrwx 1 root root 10 Jun 29 09:49 de77a2cb-a1df-460e-97a2-3c8c8ae9fad5 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 fb306c92-2607-46a5-a32d-7556b04dd494 -> ../../sda2 /dev/disk/by-path/: total 0 lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:11.4-ata-3 -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:11.4-ata-3.0 -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3.0-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3.0-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3.0-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:11.4-ata-4 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:11.4-ata-4.0 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4.0-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4.0-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4.0-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:1f.2-ata-1 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:1f.2-ata-1.0 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:1f.2-ata-2 -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:1f.2-ata-2.0 -> ../../sdd /dev/disk/by-uuid/: total 0 lrwxrwxrwx 1 root root 10 Jun 29 09:49 17716103480993325194 -> ../../sda3 lrwxrwxrwx 1 root root 10 Jun 29 09:49 B851-E178 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 B852-ACFC -> ../../sdb2 # iscsiadm -m node iscsiadm: No records found # iscsiadm -m session iscsiadm: No active sessions. ==== info about volumes ==== # pvs PV VG Fmt Attr PSize PFree /dev/sdc ceph-33bdcbd7-07be-4373-97ca-0678dda8888d lvm2 a-- <447.13g 0 /dev/sdd ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 lvm2 a-- <447.13g 0 # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd-block-e2deed6d-596f-4837-b14e-88c9afdbe531 ceph-33bdcbd7-07be-4373-97ca-0678dda8888d -wi-ao---- <447.13g osd-block-55199458-8b33-44f2-b4d2-3a876072a622 ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 -wi-ao---- <447.13g # vgs VG #PV #LV #SN Attr VSize VFree ceph-33bdcbd7-07be-4373-97ca-0678dda8888d 1 1 0 wz--n- <447.13g 0 ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 1 1 0 wz--n- <447.13g 0 # zpool status pool: rpool state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0B in 00:00:19 with 0 errors on Sun Jun 13 00:24:20 2021 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 ONLINE 0 0 0 errors: No known data errors # zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT rpool 111G 2.43G 109G - - 5% 2% 1.00x ONLINE - ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 111G 2.43G 109G - - 5% 2.18% - ONLINE # zfs list NAME USED AVAIL REFER MOUNTPOINT rpool 2.43G 105G 104K /rpool rpool/ROOT 2.42G 105G 96K /rpool/ROOT rpool/ROOT/pve-1 2.42G 105G 2.42G / rpool/data 96K 105G 96K /rpool/data # pveceph status cluster: id: 73045ca5-eead-4e44-a0c1-b6796ed3d7d5 health: HEALTH_WARN client is using insecure global_id reclaim mons are allowing insecure global_id reclaim 4 slow ops, oldest one blocked for 4666 sec, mon.node04 has slow ops services: mon: 3 daemons, quorum node04,node05,node06 (age 36m) mgr: node04(active, since 97m), standbys: node05, node06 mds: CephFS:1 {0=node05=up:active} 2 up:standby osd: 6 osds: 6 up (since 37m), 6 in (since 37m) data: pools: 4 pools, 193 pgs objects: 96.62k objects, 365 GiB usage: 1.0 TiB used, 1.6 TiB / 2.6 TiB avail pgs: 193 active+clean io: client: 0 B/s rd, 20 KiB/s wr, 0 op/s rd, 3 op/s wr # ceph osd status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 node04 154G 292G 1 7372 0 0 exists,up 1 node04 202G 244G 1 5733 0 0 exists,up 2 node05 163G 283G 0 7371 0 0 exists,up 3 node05 193G 253G 0 4095 0 0 exists,up 4 node06 180G 266G 0 4095 0 0 exists,up 5 node06 177G 270G 0 0 0 0 exists,up # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 2.6 TiB 1.6 TiB 1.0 TiB 1.0 TiB 39.96 TOTAL 2.6 TiB 1.6 TiB 1.0 TiB 1.0 TiB 39.96 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL Ceph 2 128 322 GiB 87.91k 965 GiB 42.04 444 GiB CephFS_data 3 32 34 GiB 8.68k 101 GiB 7.03 444 GiB CephFS_metadata 4 32 7.9 MiB 24 24 MiB 0 444 GiB device_health_metrics 5 1 28 MiB 6 84 MiB 0 444 GiB # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 2.61960 - 2.6 TiB 1.0 TiB 1.0 TiB 98 MiB 5.9 GiB 1.6 TiB 39.96 1.00 - root default -3 0.87320 - 894 GiB 357 GiB 355 GiB 43 MiB 2.0 GiB 537 GiB 39.96 1.00 - host node04 0 ssd 0.43660 1.00000 447 GiB 154 GiB 153 GiB 5.5 MiB 1018 MiB 293 GiB 34.53 0.86 82 up osd.0 1 ssd 0.43660 1.00000 447 GiB 203 GiB 202 GiB 38 MiB 986 MiB 244 GiB 45.38 1.14 111 up osd.1 -5 0.87320 - 894 GiB 357 GiB 355 GiB 30 MiB 2.0 GiB 537 GiB 39.96 1.00 - host node05 2 ssd 0.43660 1.00000 447 GiB 164 GiB 163 GiB 25 MiB 1.0 GiB 284 GiB 36.58 0.92 94 up osd.2 3 ssd 0.43660 1.00000 447 GiB 194 GiB 193 GiB 4.8 MiB 1019 MiB 253 GiB 43.34 1.08 99 up osd.3 -7 0.87320 - 894 GiB 357 GiB 355 GiB 25 MiB 2.0 GiB 537 GiB 39.96 1.00 - host node06 4 ssd 0.43660 1.00000 447 GiB 180 GiB 179 GiB 22 MiB 1002 MiB 267 GiB 40.32 1.01 97 up osd.4 5 ssd 0.43660 1.00000 447 GiB 177 GiB 176 GiB 2.9 MiB 1021 MiB 270 GiB 39.60 0.99 96 up osd.5 TOTAL 2.6 TiB 1.0 TiB 1.0 TiB 98 MiB 5.9 GiB 1.6 TiB 39.96 MIN/MAX VAR: 0.86/1.14 STDDEV: 3.70 # cat /etc/ceph/ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = fdb0:5bd1:dc::4/64 fsid = 73045ca5-eead-4e44-a0c1-b6796ed3d7d5 mon_allow_pool_delete = true mon_host = fdb0:5bd1:dc::4 fdb0:5bd1:dc::5 fdb0:5bd1:dc::6 ms_bind_ipv4 = false ms_bind_ipv6 = true osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = fdb0:5bd1:dc::4/64 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.node04] host = node04 mds_standby_for_name = pve [mds.node05] host = node05 mds_standby_for_name = pve [mds.node06] host = node06 mds standby for name = pve [mon.node04] public_addr = fdb0:5bd1:dc::4 [mon.node05] public_addr = fdb0:5bd1:dc::5 [mon.node06] public_addr = fdb0:5bd1:dc::6 # ceph config dump WHO MASK LEVEL OPTION VALUE RO mgr unknown mgr/dashboard/server_addr ::1 * mgr unknown mgr/dashboard/ssl false * # pveceph pool ls +-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+ | Name | Size | Min Size | PG Num | min. PG Num | Optimal PG Num | PG Autoscale Mode | PG Autoscale Target Size | PG Autoscale Target Ratio | Crush Rule Name | %-Used | Used | +=======================+======+==========+========+=============+================+===================+==========================+===========================+=================+======================+===============+ | Ceph | 3 | 2 | 128 | | 64 | on | | | replicated_rule | 0.420354932546616 | 1036478722631 | +-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+ | CephFS_data | 3 | 2 | 32 | | 32 | on | | | replicated_rule | 0.0703079700469971 | 108086587392 | +-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+ | CephFS_metadata | 3 | 2 | 32 | 16 | 16 | on | | | replicated_rule | 1.73890748556005e-05 | 24853666 | +-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+ | device_health_metrics | 3 | 2 | 1 | 1 | 1 | on | | | replicated_rule | 6.18295161984861e-05 | 88374925 | +-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+ # ceph versions { "mon": { "ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 2, "ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 1 }, "mgr": { "ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 2, "ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 1 }, "osd": { "ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 4, "ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 2 }, "mds": { "ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 2, "ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 1 }, "overall": { "ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 10, "ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 5 } }