* [pve-devel] corosync bug: cluster break after 1 node clean shutdown
@ 2020-09-03 14:11 Alexandre DERUMIER
2020-09-04 12:29 ` Alexandre DERUMIER
` (2 more replies)
0 siblings, 3 replies; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-03 14:11 UTC (permalink / raw)
To: pve-devel
Hi,
I had a problem this morning with corosync, after shutting down cleanly a node from the cluster.
This is a 14 nodes cluster, node2 was shutted. (with "halt" command)
HA was actived, and all nodes have been rebooted.
(I have already see this problem some months ago, without HA enabled, and it was like half of the nodes didn't see others nodes, like 2 cluster formations)
Some users have reported similar problems when adding a new node
https://forum.proxmox.com/threads/quorum-lost-when-adding-new-7th-node.75197/
Here the full logs of the nodes, for each node I'm seeing quorum with 13 nodes. (so it's seem to be ok).
I didn't have time to look live on server, as HA have restarted them.
So, I don't known if it's a corosync bug or something related to crm,lrm,pmxcs (but I don't see any special logs)
Only node7 have survived a little bit longer,and I had stop lrm/crm to avoid reboot.
libknet1: 1.15-pve1
corosync: 3.0.3-pve1
Any ideas before submiting bug to corosync mailing ? (I'm seeing new libknet && corosync version on their github, I don't have read the changelog yet)
node1
-----
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm1 corosync[3678]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm1 corosync[3678]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm1 corosync[3678]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [dcdb] notice: cpg_send_message retried 1 times
Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: received all states
Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 has no active links
--> reboot
node2 : shutdown log
-----
Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd received signal 15: Terminated
Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_00] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302Y9120CGN.ata.state
Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_01] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302MG120CGN.ata.state
Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd is exiting (exit status 0)
Sep 3 10:39:05 m6kvm2 nrpe[20095]: Caught SIGTERM - shutting down...
Sep 3 10:39:05 m6kvm2 nrpe[20095]: Daemon shutdown
Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> starting task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root@pam:
Sep 3 10:39:07 m6kvm2 pve-guests[38552]: all VMs and CTs stopped
Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> end task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root@pam: OK
Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: received signal TERM
Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server closing
Sep 3 10:39:07 m6kvm2 spiceproxy[36572]: worker exit
Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: worker 36572 finished
Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server stopped
Sep 3 10:39:07 m6kvm2 pvestatd[3786]: received signal TERM
Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server closing
Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server stopped
Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: received signal TERM
Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: got shutdown request with shutdown policy 'conditional'
Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: shutdown LRM, stop all services
Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: received terminate request (signal)
Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: stopping pvefw logger
Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: watchdog closed (disabled)
Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: server stopped
Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: received signal TERM
Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: server received shutdown request
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: received signal TERM
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server closing
Sep 3 10:39:10 m6kvm2 pveproxy[29731]: worker exit
Sep 3 10:39:10 m6kvm2 pveproxy[30873]: worker exit
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 30873 finished
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 31391 finished
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 29731 finished
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server stopped
Sep 3 10:39:14 m6kvm2 pve-ha-crm[12847]: server stopped
node3
-----
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm3 corosync[30580]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm3 corosync[30580]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm3 corosync[30580]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: cpg_send_message retried 1 times
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: received all states
Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 has no active links
node4
-----
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm4 corosync[4085]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm4 corosync[4085]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: received all states
Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: all data is up to date
Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:20 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
node5
-----
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm5 corosync[41830]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm5 corosync[41830]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: received all states
Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
node6
-----
Sep 3 10:39:16 m6kvm6 corosync[36694]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm6 corosync[36694]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 has no active links
node7
-----
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: received all states
Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:20 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
---> here the others nodes reboot almost at the same time
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 3 link: 0 is down
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 14 link: 0 is down
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 has no active links
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 (passive) best link: 0 (pri: 1)
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 has no active links
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 13 link: 0 is down
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 8 link: 0 is down
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 6 link: 0 is down
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 10 link: 0 is down
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 (passive) best link: 0 (pri: 1)
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 has no active links
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 (passive) best link: 0 (pri: 1)
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 has no active links
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 1)
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 has no active links
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 (passive) best link: 0 (pri: 1)
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 has no active links
Sep 3 10:40:28 m6kvm7 corosync[15467]: [TOTEM ] Token has not been received in 4505 ms
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 11 link: 0 is down
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 9 link: 0 is down
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 (passive) best link: 0 (pri: 1)
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 has no active links
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 (passive) best link: 0 (pri: 1)
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 has no active links
Sep 3 10:40:30 m6kvm7 corosync[15467]: [TOTEM ] A processor failed, forming new configuration.
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 4 link: 0 is down
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 12 link: 0 is down
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 1 link: 0 is down
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 has no active links
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 (passive) best link: 0 (pri: 1)
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 has no active links
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 has no active links
Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] link: host: 5 link: 0 is down
Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 (passive) best link: 0 (pri: 1)
Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 has no active links
Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] A new membership (7.82f) was formed. Members left: 1 3 4 5 6 8 9 10 11 12 13 14
Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] Failed to receive the leave message. failed: 1 3 4 5 6 8 9 10 11 12 13 14
Sep 3 10:40:41 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 12 received
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892
Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] Members[1]: 7
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:40:41 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: dfsm_deliver_queue: queue length 51
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: node lost quorum
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: members: 7/15892
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: received write while not quorate - trigger resync
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: leaving CPG group
Sep 3 10:40:41 m6kvm7 pve-ha-crm[16196]: node 'm6kvm2': state changed from 'online' => 'unknown'
Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] notice: start cluster connection
Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: cpg_join failed: 14
Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: can't initialize service
Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892
Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during cfs-locked 'domain-ha' operation: no quorum!
Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs lock update failed - Permission denied
Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - cfs lock update failed - Permission denied
Sep 3 10:40:56 m6kvm7 pve-ha-lrm[16140]: status change active => lost_agent_lock
Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change master => lost_manager_lock
Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: watchdog closed (disabled)
Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change lost_manager_lock => wait_for_quorum
Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] rx: host: 4 link: 0 is up
Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 3 10:43:24 m6kvm7 corosync[15467]: [TOTEM ] A new membership (4.834) was formed. Members joined: 4
Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 4/3965, 7/15892
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: members: 4/3965, 7/15892
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
Sep 3 10:43:24 m6kvm7 corosync[15467]: [QUORUM] Members[2]: 4 7
Sep 3 10:43:24 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 4/3965/00000002)
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 4/3965/00000002)
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 7/15892
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 7/15892
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: start sending inode updates
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: sent all (4) updates
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received all states
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] rx: host: 3 link: 0 is up
Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 3 10:44:05 m6kvm7 corosync[15467]: [TOTEM ] A new membership (3.838) was formed. Members joined: 3
Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 3/3613, 4/3965, 7/15892
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: members: 3/3613, 4/3965, 7/15892
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
Sep 3 10:44:05 m6kvm7 corosync[15467]: [QUORUM] Members[3]: 3 4 7
Sep 3 10:44:05 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 3/3613/00000002)
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 3/3613/00000002)
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 4/3965
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 4/3965, 7/15892
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received all states
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: dfsm_deliver_queue: queue length 1
Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] rx: host: 1 link: 0 is up
Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 3 10:44:31 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.83c) was formed. Members joined: 1
Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3552, 3/3613, 4/3965, 7/15892
Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3552, 3/3613, 4/3965, 7/15892
Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
Sep 3 10:44:31 m6kvm7 corosync[15467]: [QUORUM] Members[4]: 1 3 4 7
Sep 3 10:44:31 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3552/00000002)
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3552/00000002)
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 3/3613
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 3/3613, 4/3965, 7/15892
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received all states
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:49:18 m6kvm7 pmxcfs[15892]: [status] notice: received log
node8
-----
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm8 corosync[24361]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm8 corosync[24361]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: received all states
Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
--> reboot
node9
-----
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm9 corosync[22340]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm9 corosync[22340]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: received all states
Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: all data is up to date
Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:20 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
--> reboot
node10
------
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm10 corosync[41458]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm10 corosync[41458]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: received all states
Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: all data is up to date
Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log
Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log
--> reboot
node11
------
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm11 corosync[12455]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm11 corosync[12455]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: received all states
Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
Sep 3 10:40:20 m6kvm11 pmxcfs[12983]: [status] notice: received log
node12
------
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm12 corosync[43716]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm12 corosync[43716]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [dcdb] notice: cpg_send_message retried 1 times
Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: received all states
Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:20 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
node13
------
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm13 corosync[39182]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm13 corosync[39182]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: cpg_send_message retried 1 times
Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: received all states
Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: all data is up to date
Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
--> reboot
node14
------
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm14 corosync[42413]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm14 corosync[42413]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: received all states
Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 has no active links
--> reboot
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-03 14:11 [pve-devel] corosync bug: cluster break after 1 node clean shutdown Alexandre DERUMIER
@ 2020-09-04 12:29 ` Alexandre DERUMIER
2020-09-04 15:42 ` Dietmar Maurer
2020-12-29 14:21 ` Josef Johansson
2020-09-04 15:46 ` Alexandre DERUMIER
2020-09-30 15:50 ` Thomas Lamprecht
2 siblings, 2 replies; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-04 12:29 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: pve-devel
BTW,
do you think it could be possible to add an extra optionnal layer of security check, not related to corosync ?
I'm still afraid of this corosync bug since years, and still don't use HA. (or I have tried to enable it 2months ago,and this give me a disaster yesterday..)
Something like an extra heartbeat between nodes daemons, and check if we also have quorum with theses heartbeats ?
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Jeudi 3 Septembre 2020 16:11:56
Objet: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Hi,
I had a problem this morning with corosync, after shutting down cleanly a node from the cluster.
This is a 14 nodes cluster, node2 was shutted. (with "halt" command)
HA was actived, and all nodes have been rebooted.
(I have already see this problem some months ago, without HA enabled, and it was like half of the nodes didn't see others nodes, like 2 cluster formations)
Some users have reported similar problems when adding a new node
https://forum.proxmox.com/threads/quorum-lost-when-adding-new-7th-node.75197/
Here the full logs of the nodes, for each node I'm seeing quorum with 13 nodes. (so it's seem to be ok).
I didn't have time to look live on server, as HA have restarted them.
So, I don't known if it's a corosync bug or something related to crm,lrm,pmxcs (but I don't see any special logs)
Only node7 have survived a little bit longer,and I had stop lrm/crm to avoid reboot.
libknet1: 1.15-pve1
corosync: 3.0.3-pve1
Any ideas before submiting bug to corosync mailing ? (I'm seeing new libknet && corosync version on their github, I don't have read the changelog yet)
node1
-----
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm1 corosync[3678]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm1 corosync[3678]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm1 corosync[3678]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [dcdb] notice: cpg_send_message retried 1 times
Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: received all states
Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 has no active links
--> reboot
node2 : shutdown log
-----
Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd received signal 15: Terminated
Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_00] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302Y9120CGN.ata.state
Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_01] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302MG120CGN.ata.state
Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd is exiting (exit status 0)
Sep 3 10:39:05 m6kvm2 nrpe[20095]: Caught SIGTERM - shutting down...
Sep 3 10:39:05 m6kvm2 nrpe[20095]: Daemon shutdown
Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> starting task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root@pam:
Sep 3 10:39:07 m6kvm2 pve-guests[38552]: all VMs and CTs stopped
Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> end task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root@pam: OK
Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: received signal TERM
Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server closing
Sep 3 10:39:07 m6kvm2 spiceproxy[36572]: worker exit
Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: worker 36572 finished
Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server stopped
Sep 3 10:39:07 m6kvm2 pvestatd[3786]: received signal TERM
Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server closing
Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server stopped
Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: received signal TERM
Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: got shutdown request with shutdown policy 'conditional'
Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: shutdown LRM, stop all services
Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: received terminate request (signal)
Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: stopping pvefw logger
Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: watchdog closed (disabled)
Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: server stopped
Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: received signal TERM
Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: server received shutdown request
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: received signal TERM
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server closing
Sep 3 10:39:10 m6kvm2 pveproxy[29731]: worker exit
Sep 3 10:39:10 m6kvm2 pveproxy[30873]: worker exit
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 30873 finished
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 31391 finished
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 29731 finished
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server stopped
Sep 3 10:39:14 m6kvm2 pve-ha-crm[12847]: server stopped
node3
-----
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm3 corosync[30580]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm3 corosync[30580]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm3 corosync[30580]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: cpg_send_message retried 1 times
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: received all states
Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 has no active links
node4
-----
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm4 corosync[4085]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm4 corosync[4085]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: received all states
Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: all data is up to date
Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:20 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
node5
-----
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm5 corosync[41830]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm5 corosync[41830]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: received all states
Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
node6
-----
Sep 3 10:39:16 m6kvm6 corosync[36694]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm6 corosync[36694]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 has no active links
node7
-----
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: received all states
Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:20 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
---> here the others nodes reboot almost at the same time
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 3 link: 0 is down
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 14 link: 0 is down
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 has no active links
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 (passive) best link: 0 (pri: 1)
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 has no active links
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 13 link: 0 is down
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 8 link: 0 is down
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 6 link: 0 is down
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 10 link: 0 is down
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 (passive) best link: 0 (pri: 1)
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 has no active links
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 (passive) best link: 0 (pri: 1)
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 has no active links
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 1)
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 has no active links
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 (passive) best link: 0 (pri: 1)
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 has no active links
Sep 3 10:40:28 m6kvm7 corosync[15467]: [TOTEM ] Token has not been received in 4505 ms
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 11 link: 0 is down
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 9 link: 0 is down
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 (passive) best link: 0 (pri: 1)
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 has no active links
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 (passive) best link: 0 (pri: 1)
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 has no active links
Sep 3 10:40:30 m6kvm7 corosync[15467]: [TOTEM ] A processor failed, forming new configuration.
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 4 link: 0 is down
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 12 link: 0 is down
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 1 link: 0 is down
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 has no active links
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 (passive) best link: 0 (pri: 1)
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 has no active links
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 has no active links
Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] link: host: 5 link: 0 is down
Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 (passive) best link: 0 (pri: 1)
Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 has no active links
Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] A new membership (7.82f) was formed. Members left: 1 3 4 5 6 8 9 10 11 12 13 14
Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] Failed to receive the leave message. failed: 1 3 4 5 6 8 9 10 11 12 13 14
Sep 3 10:40:41 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 12 received
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892
Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] Members[1]: 7
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:40:41 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: dfsm_deliver_queue: queue length 51
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: node lost quorum
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: members: 7/15892
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: received write while not quorate - trigger resync
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: leaving CPG group
Sep 3 10:40:41 m6kvm7 pve-ha-crm[16196]: node 'm6kvm2': state changed from 'online' => 'unknown'
Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] notice: start cluster connection
Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: cpg_join failed: 14
Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: can't initialize service
Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892
Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during cfs-locked 'domain-ha' operation: no quorum!
Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs lock update failed - Permission denied
Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - cfs lock update failed - Permission denied
Sep 3 10:40:56 m6kvm7 pve-ha-lrm[16140]: status change active => lost_agent_lock
Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change master => lost_manager_lock
Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: watchdog closed (disabled)
Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change lost_manager_lock => wait_for_quorum
Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] rx: host: 4 link: 0 is up
Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 3 10:43:24 m6kvm7 corosync[15467]: [TOTEM ] A new membership (4.834) was formed. Members joined: 4
Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 4/3965, 7/15892
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: members: 4/3965, 7/15892
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
Sep 3 10:43:24 m6kvm7 corosync[15467]: [QUORUM] Members[2]: 4 7
Sep 3 10:43:24 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 4/3965/00000002)
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 4/3965/00000002)
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 7/15892
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 7/15892
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: start sending inode updates
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: sent all (4) updates
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received all states
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] rx: host: 3 link: 0 is up
Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 3 10:44:05 m6kvm7 corosync[15467]: [TOTEM ] A new membership (3.838) was formed. Members joined: 3
Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 3/3613, 4/3965, 7/15892
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: members: 3/3613, 4/3965, 7/15892
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
Sep 3 10:44:05 m6kvm7 corosync[15467]: [QUORUM] Members[3]: 3 4 7
Sep 3 10:44:05 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 3/3613/00000002)
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 3/3613/00000002)
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 4/3965
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 4/3965, 7/15892
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received all states
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: dfsm_deliver_queue: queue length 1
Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] rx: host: 1 link: 0 is up
Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 3 10:44:31 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.83c) was formed. Members joined: 1
Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3552, 3/3613, 4/3965, 7/15892
Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3552, 3/3613, 4/3965, 7/15892
Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
Sep 3 10:44:31 m6kvm7 corosync[15467]: [QUORUM] Members[4]: 1 3 4 7
Sep 3 10:44:31 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3552/00000002)
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3552/00000002)
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 3/3613
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 3/3613, 4/3965, 7/15892
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received all states
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:49:18 m6kvm7 pmxcfs[15892]: [status] notice: received log
node8
-----
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm8 corosync[24361]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm8 corosync[24361]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: received all states
Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
--> reboot
node9
-----
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm9 corosync[22340]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm9 corosync[22340]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: received all states
Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: all data is up to date
Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:20 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
--> reboot
node10
------
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm10 corosync[41458]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm10 corosync[41458]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: received all states
Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: all data is up to date
Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log
Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log
--> reboot
node11
------
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm11 corosync[12455]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm11 corosync[12455]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: received all states
Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
Sep 3 10:40:20 m6kvm11 pmxcfs[12983]: [status] notice: received log
node12
------
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm12 corosync[43716]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm12 corosync[43716]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [dcdb] notice: cpg_send_message retried 1 times
Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: received all states
Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:20 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
node13
------
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm13 corosync[39182]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm13 corosync[39182]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: cpg_send_message retried 1 times
Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: received all states
Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: all data is up to date
Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
--> reboot
node14
------
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm14 corosync[42413]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm14 corosync[42413]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: received all states
Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 has no active links
--> reboot
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-04 12:29 ` Alexandre DERUMIER
@ 2020-09-04 15:42 ` Dietmar Maurer
2020-09-05 13:32 ` Alexandre DERUMIER
2020-12-29 14:21 ` Josef Johansson
1 sibling, 1 reply; 84+ messages in thread
From: Dietmar Maurer @ 2020-09-04 15:42 UTC (permalink / raw)
To: Proxmox VE development discussion, Alexandre DERUMIER; +Cc: pve-devel
> do you think it could be possible to add an extra optionnal layer of security check, not related to corosync ?
I would try to find the bug instead.
> I'm still afraid of this corosync bug since years, and still don't use HA. (or I have tried to enable it 2months ago,and this give me a disaster yesterday..)
>
> Something like an extra heartbeat between nodes daemons, and check if we also have quorum with theses heartbeats ?
Was this even related to corosync? What exactly caused the reboot?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-03 14:11 [pve-devel] corosync bug: cluster break after 1 node clean shutdown Alexandre DERUMIER
2020-09-04 12:29 ` Alexandre DERUMIER
@ 2020-09-04 15:46 ` Alexandre DERUMIER
2020-09-30 15:50 ` Thomas Lamprecht
2 siblings, 0 replies; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-04 15:46 UTC (permalink / raw)
To: pve-devel
What I'm not sure, is what was the running libknet version.
package was upgraded everywhere to 1.16, but I have notice that corosync process is not restarted when libknet is upgraded.
so it's quite possible that corosync was still running 1.13 libknet. (I need to find a way to find last corosync or node restart).
I think we should force corosync restart on libknet upgrade. (or maybe bump corosync package version at the same time)
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-04 15:42 ` Dietmar Maurer
@ 2020-09-05 13:32 ` Alexandre DERUMIER
2020-09-05 15:23 ` dietmar
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-05 13:32 UTC (permalink / raw)
To: dietmar; +Cc: Proxmox VE development discussion, pve-devel
> Something like an extra heartbeat between nodes daemons, and check if we also have quorum with theses heartbeats ?
>
>>Was this even related to corosync? What exactly caused the reboot?
Hi Dietmar,
what I'm 100% sure, it that the watchdog have reboot all the servers. (I have watchdog trace in ipmi)
That's happen, just after shutdown the server.
What is strange is that corosync logs on all servers show that they correctly the node down, and see other nodes.
So, I really don't known.
Maybe corosync what hanging ?
I don't have any other logs from crm/lrm/pmxcs...
I'm really blind. :/
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "aderumier" <aderumier@odiso.com>
Cc: "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Vendredi 4 Septembre 2020 17:42:45
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
> do you think it could be possible to add an extra optionnal layer of security check, not related to corosync ?
I would try to find the bug instead.
> I'm still afraid of this corosync bug since years, and still don't use HA. (or I have tried to enable it 2months ago,and this give me a disaster yesterday..)
>
> Something like an extra heartbeat between nodes daemons, and check if we also have quorum with theses heartbeats ?
Was this even related to corosync? What exactly caused the reboot?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-05 13:32 ` Alexandre DERUMIER
@ 2020-09-05 15:23 ` dietmar
2020-09-05 17:30 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: dietmar @ 2020-09-05 15:23 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: Proxmox VE development discussion, pve-devel
> what I'm 100% sure, it that the watchdog have reboot all the servers. (I have watchdog trace in ipmi)
So you are using ipmi hardware watchdog?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-05 15:23 ` dietmar
@ 2020-09-05 17:30 ` Alexandre DERUMIER
2020-09-06 4:21 ` dietmar
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-05 17:30 UTC (permalink / raw)
To: dietmar; +Cc: Proxmox VE development discussion, pve-devel
>>So you are using ipmi hardware watchdog?
yes, I'm using dell idrac ipmi card watchdog.
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Samedi 5 Septembre 2020 17:23:28
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
> what I'm 100% sure, it that the watchdog have reboot all the servers. (I have watchdog trace in ipmi)
So you are using ipmi hardware watchdog?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-05 17:30 ` Alexandre DERUMIER
@ 2020-09-06 4:21 ` dietmar
2020-09-06 5:36 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: dietmar @ 2020-09-06 4:21 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: Proxmox VE development discussion, pve-devel
> >>So you are using ipmi hardware watchdog?
>
> yes, I'm using dell idrac ipmi card watchdog
But the pve logs look ok, and there is no indication
that we stopped updating the watchdog. So why did the
watchdog trigger? Maybe an IPMI bug?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-06 4:21 ` dietmar
@ 2020-09-06 5:36 ` Alexandre DERUMIER
2020-09-06 6:33 ` Alexandre DERUMIER
2020-09-06 8:43 ` Alexandre DERUMIER
0 siblings, 2 replies; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-06 5:36 UTC (permalink / raw)
To: dietmar; +Cc: Proxmox VE development discussion, pve-devel
>>But the pve logs look ok, and there is no indication
>>that we stopped updating the watchdog. So why did the
>>watchdog trigger? Maybe an IPMI bug?
do you mean an ipmi bug on all 13 servers at the same time ?
(I also have 2 supermicro servers in this cluster, but they use same ipmi watchdog driver. (ipmi_watchdog)
I had same kind of with bug once (when stopping a server), on another cluster, 6 months ago.
This was without HA, but different version of corosync, and that time, I was really seeing quorum split in the corosync logs of the servers.
I'll try to reproduce with a virtual cluster with 14 nodes (don't have enough hardware)
Could I be a bug in proxmox HA code, where watchdog is not resetted by LRM anymore?
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Dimanche 6 Septembre 2020 06:21:55
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
> >>So you are using ipmi hardware watchdog?
>
> yes, I'm using dell idrac ipmi card watchdog
But the pve logs look ok, and there is no indication
that we stopped updating the watchdog. So why did the
watchdog trigger? Maybe an IPMI bug?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-06 5:36 ` Alexandre DERUMIER
@ 2020-09-06 6:33 ` Alexandre DERUMIER
2020-09-06 8:43 ` Alexandre DERUMIER
1 sibling, 0 replies; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-06 6:33 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: dietmar, pve-devel
Also, I wonder if it could be possible to not use watchdog fencing at all (as option),
if cluster use only shared storages with native disk lock/reservation.
Like ceph rbd for example, with exclusive-lock, you can't write from 2 clients on same rbd,
so ha will not be able to start qemu on another node.
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "dietmar" <dietmar@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Dimanche 6 Septembre 2020 07:36:10
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>But the pve logs look ok, and there is no indication
>>that we stopped updating the watchdog. So why did the
>>watchdog trigger? Maybe an IPMI bug?
do you mean an ipmi bug on all 13 servers at the same time ?
(I also have 2 supermicro servers in this cluster, but they use same ipmi watchdog driver. (ipmi_watchdog)
I had same kind of with bug once (when stopping a server), on another cluster, 6 months ago.
This was without HA, but different version of corosync, and that time, I was really seeing quorum split in the corosync logs of the servers.
I'll try to reproduce with a virtual cluster with 14 nodes (don't have enough hardware)
Could I be a bug in proxmox HA code, where watchdog is not resetted by LRM anymore?
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Dimanche 6 Septembre 2020 06:21:55
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
> >>So you are using ipmi hardware watchdog?
>
> yes, I'm using dell idrac ipmi card watchdog
But the pve logs look ok, and there is no indication
that we stopped updating the watchdog. So why did the
watchdog trigger? Maybe an IPMI bug?
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-06 5:36 ` Alexandre DERUMIER
2020-09-06 6:33 ` Alexandre DERUMIER
@ 2020-09-06 8:43 ` Alexandre DERUMIER
2020-09-06 12:14 ` dietmar
1 sibling, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-06 8:43 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: dietmar, pve-devel
Maybe something interesting, the only survived node was node7, and it was the crm master
I'm also seein crm disabling watchdog, and also some "loop take too long" messages
(some migration logs from node2 to node1 before the maintenance)
Sep 3 10:36:29 m6kvm7 pve-ha-crm[16196]: service 'vm:992': state changed from 'migrate' to 'started' (node = m6kvm1)
Sep 3 10:36:29 m6kvm7 pve-ha-crm[16196]: service 'vm:993': state changed from 'migrate' to 'started' (node = m6kvm1)
Sep 3 10:36:29 m6kvm7 pve-ha-crm[16196]: service 'vm:997': state changed from 'migrate' to 'started' (node = m6kvm1)
....
Sep 3 10:40:41 m6kvm7 pve-ha-crm[16196]: node 'm6kvm2': state changed from 'online' => 'unknown'
Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during cfs-locked 'domain-ha' operation: no quorum!
Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs lock update failed - Permission denied
Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - cfs lock update failed - Permission denied
Sep 3 10:40:56 m6kvm7 pve-ha-lrm[16140]: status change active => lost_agent_lock
Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change master => lost_manager_lock
Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: watchdog closed (disabled)
Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change lost_manager_lock => wait_for_quorum
others nodes timing
--------------------
10:39:16 -> node2 shutdown, leave coroync
10:40:25 -> other nodes rebooted by watchdog
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "dietmar" <dietmar@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Dimanche 6 Septembre 2020 07:36:10
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>But the pve logs look ok, and there is no indication
>>that we stopped updating the watchdog. So why did the
>>watchdog trigger? Maybe an IPMI bug?
do you mean an ipmi bug on all 13 servers at the same time ?
(I also have 2 supermicro servers in this cluster, but they use same ipmi watchdog driver. (ipmi_watchdog)
I had same kind of with bug once (when stopping a server), on another cluster, 6 months ago.
This was without HA, but different version of corosync, and that time, I was really seeing quorum split in the corosync logs of the servers.
I'll try to reproduce with a virtual cluster with 14 nodes (don't have enough hardware)
Could I be a bug in proxmox HA code, where watchdog is not resetted by LRM anymore?
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Dimanche 6 Septembre 2020 06:21:55
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
> >>So you are using ipmi hardware watchdog?
>
> yes, I'm using dell idrac ipmi card watchdog
But the pve logs look ok, and there is no indication
that we stopped updating the watchdog. So why did the
watchdog trigger? Maybe an IPMI bug?
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-06 8:43 ` Alexandre DERUMIER
@ 2020-09-06 12:14 ` dietmar
2020-09-06 12:19 ` dietmar
2020-09-07 7:19 ` Alexandre DERUMIER
0 siblings, 2 replies; 84+ messages in thread
From: dietmar @ 2020-09-06 12:14 UTC (permalink / raw)
To: Alexandre DERUMIER, Proxmox VE development discussion; +Cc: pve-devel
> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
Indeed, this should not happen. Do you use a spearate network for corosync? Or
was there high traffic on the network? What kind of maintenance was the reason
for the shutdown?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-06 12:14 ` dietmar
@ 2020-09-06 12:19 ` dietmar
2020-09-07 7:00 ` Thomas Lamprecht
2020-09-07 7:19 ` Alexandre DERUMIER
1 sibling, 1 reply; 84+ messages in thread
From: dietmar @ 2020-09-06 12:19 UTC (permalink / raw)
To: Alexandre DERUMIER, Proxmox VE development discussion; +Cc: pve-devel
> On 09/06/2020 2:14 PM dietmar <dietmar@proxmox.com> wrote:
>
>
> > Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
> > Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
>
> Indeed, this should not happen. Do you use a spearate network for corosync? Or
> was there high traffic on the network? What kind of maintenance was the reason
> for the shutdown?
Do you use the default corosync timeout values, or do you have a special setup?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-06 12:19 ` dietmar
@ 2020-09-07 7:00 ` Thomas Lamprecht
0 siblings, 0 replies; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-07 7:00 UTC (permalink / raw)
To: Proxmox VE development discussion, dietmar, Alexandre DERUMIER
On 06.09.20 14:19, dietmar wrote:
>> On 09/06/2020 2:14 PM dietmar <dietmar@proxmox.com> wrote:
>>
>>
>>> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
>>> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
>>
>> Indeed, this should not happen. Do you use a spearate network for corosync? Or
>> was there high traffic on the network? What kind of maintenance was the reason
>> for the shutdown?
>
> Do you use the default corosync timeout values, or do you have a special setup?
>
Can you please post the full corosync config?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-06 12:14 ` dietmar
2020-09-06 12:19 ` dietmar
@ 2020-09-07 7:19 ` Alexandre DERUMIER
2020-09-07 8:18 ` dietmar
1 sibling, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-07 7:19 UTC (permalink / raw)
To: dietmar; +Cc: Proxmox VE development discussion, pve-devel
>>Indeed, this should not happen. Do you use a spearate network for corosync?
No, I use 2x40GB lacp link.
>>was there high traffic on the network?
but I'm far from saturated them. (in pps or througput), (I'm around 3-4gbps)
The cluster is 14 nodes, with around 1000vms (with ha enabled on all vms)
From my understanding, watchdog-mux was still runing as the watchdog have reset only after 1min and not 10s,
so it's like the lrm was blocked and not sending watchdog timer reset to watchdog-mux.
I'll do tests with softdog + soft_noboot=1, so if that happen again,I'll able to debug.
>>What kind of maintenance was the reason for the shutdown?
ram upgrade. (the server was running ok before shutdown, no hardware problem)
(I just shutdown the server, and don't have started it yet when problem occur)
>>Do you use the default corosync timeout values, or do you have a special setup?
no special tuning, default values. (I don't have any retransmit since months in the logs)
>>Can you please post the full corosync config?
(I have verified, the running version was corosync was 3.0.3 with libknet 1.15)
here the config:
"
logging {
debug: off
to_syslog: yes
}
nodelist {
node {
name: m6kvm1
nodeid: 1
quorum_votes: 1
ring0_addr: m6kvm1
}
node {
name: m6kvm10
nodeid: 10
quorum_votes: 1
ring0_addr: m6kvm10
}
node {
name: m6kvm11
nodeid: 11
quorum_votes: 1
ring0_addr: m6kvm11
}
node {
name: m6kvm12
nodeid: 12
quorum_votes: 1
ring0_addr: m6kvm12
}
node {
name: m6kvm13
nodeid: 13
quorum_votes: 1
ring0_addr: m6kvm13
}
node {
name: m6kvm14
nodeid: 14
quorum_votes: 1
ring0_addr: m6kvm14
}
node {
name: m6kvm2
nodeid: 2
quorum_votes: 1
ring0_addr: m6kvm2
}
node {
name: m6kvm3
nodeid: 3
quorum_votes: 1
ring0_addr: m6kvm3
}
node {
name: m6kvm4
nodeid: 4
quorum_votes: 1
ring0_addr: m6kvm4
}
node {
name: m6kvm5
nodeid: 5
quorum_votes: 1
ring0_addr: m6kvm5
}
node {
name: m6kvm6
nodeid: 6
quorum_votes: 1
ring0_addr: m6kvm6
}
node {
name: m6kvm7
nodeid: 7
quorum_votes: 1
ring0_addr: m6kvm7
}
node {
name: m6kvm8
nodeid: 8
quorum_votes: 1
ring0_addr: m6kvm8
}
node {
name: m6kvm9
nodeid: 9
quorum_votes: 1
ring0_addr: m6kvm9
}
}
quorum {
provider: corosync_votequorum
}
totem {
cluster_name: m6kvm
config_version: 19
interface {
bindnetaddr: 10.3.94.89
ringnumber: 0
}
ip_version: ipv4
secauth: on
transport: knet
version: 2
}
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Dimanche 6 Septembre 2020 14:14:06
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
Indeed, this should not happen. Do you use a spearate network for corosync? Or
was there high traffic on the network? What kind of maintenance was the reason
for the shutdown?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-07 7:19 ` Alexandre DERUMIER
@ 2020-09-07 8:18 ` dietmar
2020-09-07 9:32 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: dietmar @ 2020-09-07 8:18 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: Proxmox VE development discussion
There is a similar report in the forum:
https://forum.proxmox.com/threads/cluster-die-after-adding-the-39th-node-proxmox-is-not-stable.75506/#post-336111
No HA involved...
> On 09/07/2020 9:19 AM Alexandre DERUMIER <aderumier@odiso.com> wrote:
>
>
> >>Indeed, this should not happen. Do you use a spearate network for corosync?
>
> No, I use 2x40GB lacp link.
>
> >>was there high traffic on the network?
>
> but I'm far from saturated them. (in pps or througput), (I'm around 3-4gbps)
>
>
> The cluster is 14 nodes, with around 1000vms (with ha enabled on all vms)
>
>
> From my understanding, watchdog-mux was still runing as the watchdog have reset only after 1min and not 10s,
> so it's like the lrm was blocked and not sending watchdog timer reset to watchdog-mux.
>
>
> I'll do tests with softdog + soft_noboot=1, so if that happen again,I'll able to debug.
>
>
>
> >>What kind of maintenance was the reason for the shutdown?
>
> ram upgrade. (the server was running ok before shutdown, no hardware problem)
> (I just shutdown the server, and don't have started it yet when problem occur)
>
>
>
> >>Do you use the default corosync timeout values, or do you have a special setup?
>
>
> no special tuning, default values. (I don't have any retransmit since months in the logs)
>
> >>Can you please post the full corosync config?
>
> (I have verified, the running version was corosync was 3.0.3 with libknet 1.15)
>
>
> here the config:
>
> "
> logging {
> debug: off
> to_syslog: yes
> }
>
> nodelist {
> node {
> name: m6kvm1
> nodeid: 1
> quorum_votes: 1
> ring0_addr: m6kvm1
> }
> node {
> name: m6kvm10
> nodeid: 10
> quorum_votes: 1
> ring0_addr: m6kvm10
> }
> node {
> name: m6kvm11
> nodeid: 11
> quorum_votes: 1
> ring0_addr: m6kvm11
> }
> node {
> name: m6kvm12
> nodeid: 12
> quorum_votes: 1
> ring0_addr: m6kvm12
> }
> node {
> name: m6kvm13
> nodeid: 13
> quorum_votes: 1
> ring0_addr: m6kvm13
> }
> node {
> name: m6kvm14
> nodeid: 14
> quorum_votes: 1
> ring0_addr: m6kvm14
> }
> node {
> name: m6kvm2
> nodeid: 2
> quorum_votes: 1
> ring0_addr: m6kvm2
> }
> node {
> name: m6kvm3
> nodeid: 3
> quorum_votes: 1
> ring0_addr: m6kvm3
> }
> node {
> name: m6kvm4
> nodeid: 4
> quorum_votes: 1
> ring0_addr: m6kvm4
> }
> node {
> name: m6kvm5
> nodeid: 5
> quorum_votes: 1
> ring0_addr: m6kvm5
> }
> node {
> name: m6kvm6
> nodeid: 6
> quorum_votes: 1
> ring0_addr: m6kvm6
> }
> node {
> name: m6kvm7
> nodeid: 7
> quorum_votes: 1
> ring0_addr: m6kvm7
> }
>
> node {
> name: m6kvm8
> nodeid: 8
> quorum_votes: 1
> ring0_addr: m6kvm8
> }
> node {
> name: m6kvm9
> nodeid: 9
> quorum_votes: 1
> ring0_addr: m6kvm9
> }
> }
>
> quorum {
> provider: corosync_votequorum
> }
>
> totem {
> cluster_name: m6kvm
> config_version: 19
> interface {
> bindnetaddr: 10.3.94.89
> ringnumber: 0
> }
> ip_version: ipv4
> secauth: on
> transport: knet
> version: 2
> }
>
>
>
> ----- Mail original -----
> De: "dietmar" <dietmar@proxmox.com>
> À: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
> Cc: "pve-devel" <pve-devel@pve.proxmox.com>
> Envoyé: Dimanche 6 Septembre 2020 14:14:06
> Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>
> > Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
> > Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
>
> Indeed, this should not happen. Do you use a spearate network for corosync? Or
> was there high traffic on the network? What kind of maintenance was the reason
> for the shutdown?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-07 8:18 ` dietmar
@ 2020-09-07 9:32 ` Alexandre DERUMIER
2020-09-07 13:23 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-07 9:32 UTC (permalink / raw)
To: dietmar; +Cc: Proxmox VE development discussion
>>https://forum.proxmox.com/threads/cluster-die-after-adding-the-39th-node-proxmox-is-not-stable.75506/#post-336111
>>
>>No HA involved...
I had already help this user some week ago
https://forum.proxmox.com/threads/proxmox-6-2-4-cluster-die-node-auto-reboot-need-help.74643/#post-333093
HA was actived at this time. (Maybe the watchdog was still running, I'm not sure if you disable HA from all vms if LRM disable the watchdog ?)
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Lundi 7 Septembre 2020 10:18:42
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
There is a similar report in the forum:
https://forum.proxmox.com/threads/cluster-die-after-adding-the-39th-node-proxmox-is-not-stable.75506/#post-336111
No HA involved...
> On 09/07/2020 9:19 AM Alexandre DERUMIER <aderumier@odiso.com> wrote:
>
>
> >>Indeed, this should not happen. Do you use a spearate network for corosync?
>
> No, I use 2x40GB lacp link.
>
> >>was there high traffic on the network?
>
> but I'm far from saturated them. (in pps or througput), (I'm around 3-4gbps)
>
>
> The cluster is 14 nodes, with around 1000vms (with ha enabled on all vms)
>
>
> From my understanding, watchdog-mux was still runing as the watchdog have reset only after 1min and not 10s,
> so it's like the lrm was blocked and not sending watchdog timer reset to watchdog-mux.
>
>
> I'll do tests with softdog + soft_noboot=1, so if that happen again,I'll able to debug.
>
>
>
> >>What kind of maintenance was the reason for the shutdown?
>
> ram upgrade. (the server was running ok before shutdown, no hardware problem)
> (I just shutdown the server, and don't have started it yet when problem occur)
>
>
>
> >>Do you use the default corosync timeout values, or do you have a special setup?
>
>
> no special tuning, default values. (I don't have any retransmit since months in the logs)
>
> >>Can you please post the full corosync config?
>
> (I have verified, the running version was corosync was 3.0.3 with libknet 1.15)
>
>
> here the config:
>
> "
> logging {
> debug: off
> to_syslog: yes
> }
>
> nodelist {
> node {
> name: m6kvm1
> nodeid: 1
> quorum_votes: 1
> ring0_addr: m6kvm1
> }
> node {
> name: m6kvm10
> nodeid: 10
> quorum_votes: 1
> ring0_addr: m6kvm10
> }
> node {
> name: m6kvm11
> nodeid: 11
> quorum_votes: 1
> ring0_addr: m6kvm11
> }
> node {
> name: m6kvm12
> nodeid: 12
> quorum_votes: 1
> ring0_addr: m6kvm12
> }
> node {
> name: m6kvm13
> nodeid: 13
> quorum_votes: 1
> ring0_addr: m6kvm13
> }
> node {
> name: m6kvm14
> nodeid: 14
> quorum_votes: 1
> ring0_addr: m6kvm14
> }
> node {
> name: m6kvm2
> nodeid: 2
> quorum_votes: 1
> ring0_addr: m6kvm2
> }
> node {
> name: m6kvm3
> nodeid: 3
> quorum_votes: 1
> ring0_addr: m6kvm3
> }
> node {
> name: m6kvm4
> nodeid: 4
> quorum_votes: 1
> ring0_addr: m6kvm4
> }
> node {
> name: m6kvm5
> nodeid: 5
> quorum_votes: 1
> ring0_addr: m6kvm5
> }
> node {
> name: m6kvm6
> nodeid: 6
> quorum_votes: 1
> ring0_addr: m6kvm6
> }
> node {
> name: m6kvm7
> nodeid: 7
> quorum_votes: 1
> ring0_addr: m6kvm7
> }
>
> node {
> name: m6kvm8
> nodeid: 8
> quorum_votes: 1
> ring0_addr: m6kvm8
> }
> node {
> name: m6kvm9
> nodeid: 9
> quorum_votes: 1
> ring0_addr: m6kvm9
> }
> }
>
> quorum {
> provider: corosync_votequorum
> }
>
> totem {
> cluster_name: m6kvm
> config_version: 19
> interface {
> bindnetaddr: 10.3.94.89
> ringnumber: 0
> }
> ip_version: ipv4
> secauth: on
> transport: knet
> version: 2
> }
>
>
>
> ----- Mail original -----
> De: "dietmar" <dietmar@proxmox.com>
> À: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
> Cc: "pve-devel" <pve-devel@pve.proxmox.com>
> Envoyé: Dimanche 6 Septembre 2020 14:14:06
> Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>
> > Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
> > Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
>
> Indeed, this should not happen. Do you use a spearate network for corosync? Or
> was there high traffic on the network? What kind of maintenance was the reason
> for the shutdown?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-07 9:32 ` Alexandre DERUMIER
@ 2020-09-07 13:23 ` Alexandre DERUMIER
2020-09-08 4:41 ` dietmar
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-07 13:23 UTC (permalink / raw)
To: Proxmox VE development discussion
Looking at theses logs:
Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs lock update failed - Permission denied
Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - cfs lock update failed - Permission denied
in PVE/HA/Env/PVE2.pm
"
my $ctime = time();
my $last_lock_time = $last->{lock_time} // 0;
my $last_got_lock = $last->{got_lock};
my $retry_timeout = 120; # hardcoded lock lifetime limit from pmxcfs
eval {
mkdir $lockdir;
# pve cluster filesystem not online
die "can't create '$lockdir' (pmxcfs not mounted?)\n" if ! -d $lockdir;
if (($ctime - $last_lock_time) < $retry_timeout) {
# try cfs lock update request (utime)
if (utime(0, $ctime, $filename)) {
$got_lock = 1;
return;
}
die "cfs lock update failed - $!\n";
}
"
If the retry_timeout is = 120, could it explain why I don't have log on others node, if the watchdog trigger after 60s ?
I don't known too much how locks are working in pmxcfs, but when a corosync member leave or join, and a new cluster memership is formed,
could we have some lock lost or hang ?
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "dietmar" <dietmar@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Lundi 7 Septembre 2020 11:32:13
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>https://forum.proxmox.com/threads/cluster-die-after-adding-the-39th-node-proxmox-is-not-stable.75506/#post-336111
>>
>>No HA involved...
I had already help this user some week ago
https://forum.proxmox.com/threads/proxmox-6-2-4-cluster-die-node-auto-reboot-need-help.74643/#post-333093
HA was actived at this time. (Maybe the watchdog was still running, I'm not sure if you disable HA from all vms if LRM disable the watchdog ?)
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Lundi 7 Septembre 2020 10:18:42
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
There is a similar report in the forum:
https://forum.proxmox.com/threads/cluster-die-after-adding-the-39th-node-proxmox-is-not-stable.75506/#post-336111
No HA involved...
> On 09/07/2020 9:19 AM Alexandre DERUMIER <aderumier@odiso.com> wrote:
>
>
> >>Indeed, this should not happen. Do you use a spearate network for corosync?
>
> No, I use 2x40GB lacp link.
>
> >>was there high traffic on the network?
>
> but I'm far from saturated them. (in pps or througput), (I'm around 3-4gbps)
>
>
> The cluster is 14 nodes, with around 1000vms (with ha enabled on all vms)
>
>
> From my understanding, watchdog-mux was still runing as the watchdog have reset only after 1min and not 10s,
> so it's like the lrm was blocked and not sending watchdog timer reset to watchdog-mux.
>
>
> I'll do tests with softdog + soft_noboot=1, so if that happen again,I'll able to debug.
>
>
>
> >>What kind of maintenance was the reason for the shutdown?
>
> ram upgrade. (the server was running ok before shutdown, no hardware problem)
> (I just shutdown the server, and don't have started it yet when problem occur)
>
>
>
> >>Do you use the default corosync timeout values, or do you have a special setup?
>
>
> no special tuning, default values. (I don't have any retransmit since months in the logs)
>
> >>Can you please post the full corosync config?
>
> (I have verified, the running version was corosync was 3.0.3 with libknet 1.15)
>
>
> here the config:
>
> "
> logging {
> debug: off
> to_syslog: yes
> }
>
> nodelist {
> node {
> name: m6kvm1
> nodeid: 1
> quorum_votes: 1
> ring0_addr: m6kvm1
> }
> node {
> name: m6kvm10
> nodeid: 10
> quorum_votes: 1
> ring0_addr: m6kvm10
> }
> node {
> name: m6kvm11
> nodeid: 11
> quorum_votes: 1
> ring0_addr: m6kvm11
> }
> node {
> name: m6kvm12
> nodeid: 12
> quorum_votes: 1
> ring0_addr: m6kvm12
> }
> node {
> name: m6kvm13
> nodeid: 13
> quorum_votes: 1
> ring0_addr: m6kvm13
> }
> node {
> name: m6kvm14
> nodeid: 14
> quorum_votes: 1
> ring0_addr: m6kvm14
> }
> node {
> name: m6kvm2
> nodeid: 2
> quorum_votes: 1
> ring0_addr: m6kvm2
> }
> node {
> name: m6kvm3
> nodeid: 3
> quorum_votes: 1
> ring0_addr: m6kvm3
> }
> node {
> name: m6kvm4
> nodeid: 4
> quorum_votes: 1
> ring0_addr: m6kvm4
> }
> node {
> name: m6kvm5
> nodeid: 5
> quorum_votes: 1
> ring0_addr: m6kvm5
> }
> node {
> name: m6kvm6
> nodeid: 6
> quorum_votes: 1
> ring0_addr: m6kvm6
> }
> node {
> name: m6kvm7
> nodeid: 7
> quorum_votes: 1
> ring0_addr: m6kvm7
> }
>
> node {
> name: m6kvm8
> nodeid: 8
> quorum_votes: 1
> ring0_addr: m6kvm8
> }
> node {
> name: m6kvm9
> nodeid: 9
> quorum_votes: 1
> ring0_addr: m6kvm9
> }
> }
>
> quorum {
> provider: corosync_votequorum
> }
>
> totem {
> cluster_name: m6kvm
> config_version: 19
> interface {
> bindnetaddr: 10.3.94.89
> ringnumber: 0
> }
> ip_version: ipv4
> secauth: on
> transport: knet
> version: 2
> }
>
>
>
> ----- Mail original -----
> De: "dietmar" <dietmar@proxmox.com>
> À: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
> Cc: "pve-devel" <pve-devel@pve.proxmox.com>
> Envoyé: Dimanche 6 Septembre 2020 14:14:06
> Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>
> > Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
> > Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
>
> Indeed, this should not happen. Do you use a spearate network for corosync? Or
> was there high traffic on the network? What kind of maintenance was the reason
> for the shutdown?
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-07 13:23 ` Alexandre DERUMIER
@ 2020-09-08 4:41 ` dietmar
2020-09-08 7:11 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: dietmar @ 2020-09-08 4:41 UTC (permalink / raw)
To: Alexandre DERUMIER, Proxmox VE development discussion
> I don't known too much how locks are working in pmxcfs, but when a corosync member leave or join, and a new cluster memership is formed, could we have some lock lost or hang ?
It would really help if we can reproduce the bug somehow. Do you have and idea how
to trigger the bug?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-08 4:41 ` dietmar
@ 2020-09-08 7:11 ` Alexandre DERUMIER
2020-09-09 20:05 ` Thomas Lamprecht
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-08 7:11 UTC (permalink / raw)
To: dietmar; +Cc: Proxmox VE development discussion
>>It would really help if we can reproduce the bug somehow. Do you have and idea how
>>to trigger the bug?
I really don't known. I'm currently trying to reproduce on the same cluster, with softdog && noboot=1, and rebooting node.
Maybe it's related with the number of vms, or the number of nodes, don't have any clue ...
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 8 Septembre 2020 06:41:10
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
> I don't known too much how locks are working in pmxcfs, but when a corosync member leave or join, and a new cluster memership is formed, could we have some lock lost or hang ?
It would really help if we can reproduce the bug somehow. Do you have and idea how
to trigger the bug?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-08 7:11 ` Alexandre DERUMIER
@ 2020-09-09 20:05 ` Thomas Lamprecht
2020-09-10 4:58 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-09 20:05 UTC (permalink / raw)
To: Proxmox VE development discussion, Alexandre DERUMIER, dietmar
On 08.09.20 09:11, Alexandre DERUMIER wrote:
>>> It would really help if we can reproduce the bug somehow. Do you have and idea how
>>> to trigger the bug?
>
> I really don't known. I'm currently trying to reproduce on the same cluster, with softdog && noboot=1, and rebooting node.
>
>
> Maybe it's related with the number of vms, or the number of nodes, don't have any clue ...
I checked a bit the watchdog code, our user-space mux one and the kernel drivers,
and just noting a few things here (thinking out aloud):
The /dev/watchdog itself is always active, else we could loose it to some
other program and not be able to activate HA dynamically.
But, as long as no HA service got active, it's a simple dummy "wake up every
second and do an ioctl keep-alive update".
This is really simple and efficiently written, so if that fails for over 10s
the systems is really loaded, probably barely responding to anything.
Currently the watchdog-mux runs as normal process, no re-nice, no real-time
scheduling. This is IMO wrong, as it is a critical process which needs to be
run with high priority. I've a patch here which sets it to the highest RR
realtime-scheduling priority available, effectively the same what corosync does.
diff --git a/src/watchdog-mux.c b/src/watchdog-mux.c
index 818ae00..71981d7 100644
--- a/src/watchdog-mux.c
+++ b/src/watchdog-mux.c
@@ -8,2 +8,3 @@
#include <time.h>
+#include <sched.h>
#include <sys/ioctl.h>
@@ -151,2 +177,15 @@ main(void)
+ int sched_priority = sched_get_priority_max (SCHED_RR);
+ if (sched_priority != -1) {
+ struct sched_param global_sched_param;
+ global_sched_param.sched_priority = sched_priority;
+ int res = sched_setscheduler (0, SCHED_RR, &global_sched_param);
+ if (res == -1) {
+ fprintf(stderr, "Could not set SCHED_RR at priority %d\n", sched_priority);
+ } else {
+ fprintf(stderr, "set SCHED_RR at priority %d\n", sched_priority);
+ }
+ }
+
+
if ((watchdog_fd = open(WATCHDOG_DEV, O_WRONLY)) == -1) {
The issue with no HA but watchdog reset due to massively overloaded system
should be avoided already a lot with the scheduling change alone.
Interesting, IMO, is that lots of nodes rebooted at the same time, with no HA active.
This *could* come from a side-effect like ceph rebalacing kicking off and producing
a load spike for >10s, hindering the scheduling of the watchdog-mux.
This is a theory, but with HA off it needs to be something like that, as in HA-off
case there's *no* direct or indirect connection between corosync/pmxcfs and the
watchdog-mux. It simply does not cares, or notices, quorum partition changes at all.
There may be a approach to reserve the watchdog for the mux, but avoid having it
as "ticking time bomb":
Theoretically one could open it, then disable it with an ioctl (it can be queried
if a driver support that) and only enable it for real once the first client connects
to the MUX. This may not work for all watchdog modules, and if, we may want to make
it configurable, as some people actually want a reset if a (future) real-time process
cannot be scheduled for >= 10 seconds.
With HA active, well then there could be something off, either in corosync/knet or
also in how we interface with it in pmxcfs, that could well be, but won't explain the
non-HA issues.
Speaking of pmxcfs, that one runs also with standard priority, we may want to change
that too to a RT scheduler, so that its ensured it can process all corosync events.
I have also a few other small watchdog mux patches around, it should nowadays actually
be able to tell us why a reset happened (can also be over/under voltage, temperature,
...) and I'll repeat doing the ioctl for keep-alive a few times if it fails, can only
win with that after all.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-09 20:05 ` Thomas Lamprecht
@ 2020-09-10 4:58 ` Alexandre DERUMIER
2020-09-10 8:21 ` Thomas Lamprecht
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-10 4:58 UTC (permalink / raw)
To: Thomas Lamprecht; +Cc: Proxmox VE development discussion, dietmar
Thanks Thomas for the investigations.
I'm still trying to reproduce...
I think I have some special case here, because the user of the forum with 30 nodes had corosync cluster split. (Note that I had this bug 6 months ago,when shuting down a node too, and the only way was stop full stop corosync on all nodes, and start corosync again on all nodes).
But this time, corosync logs looks fine. (every node, correctly see node2 down, and see remaning nodes)
surviving node7, was the only node with HA, and LRM didn't have enable watchog (I don't have found any log like "pve-ha-lrm: watchdog active" for the last 6months on this nodes
So, the timing was:
10:39:05 : "halt" command is send to node2
10:39:16 : node2 is leaving corosync / halt -> every node is seeing it and correctly do a new membership with 13 remaining nodes
...don't see any special logs (corosync,pmxcfs,pve-ha-crm,pve-ha-lrm) after the node2 leaving.
But they are still activity on the server, pve-firewall is still logging, vms are running fine
between 10:40:25 - 10:40:34 : watchdog reset nodes, but not node7.
-> so between 70s-80s after the node2 was done, so I think that watchdog-mux was still running fine until that.
(That's sound like lrm was stuck, and client_watchdog_timeout have expired in watchdog-mux)
10:40:41 node7, loose quorum (as all others nodes have reset),
10:40:50: node7 crm/lrm finally log.
Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during cfs-locked 'domain-ha' operation: no quorum!
Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs lock update failed - Permission denied
Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - cfs lock update failed - Permission denied
So, I really think that something have stucked lrm/crm loop, and watchdog was not resetted because of that.
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "aderumier" <aderumier@odiso.com>, "dietmar" <dietmar@proxmox.com>
Envoyé: Mercredi 9 Septembre 2020 22:05:49
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 08.09.20 09:11, Alexandre DERUMIER wrote:
>>> It would really help if we can reproduce the bug somehow. Do you have and idea how
>>> to trigger the bug?
>
> I really don't known. I'm currently trying to reproduce on the same cluster, with softdog && noboot=1, and rebooting node.
>
>
> Maybe it's related with the number of vms, or the number of nodes, don't have any clue ...
I checked a bit the watchdog code, our user-space mux one and the kernel drivers,
and just noting a few things here (thinking out aloud):
The /dev/watchdog itself is always active, else we could loose it to some
other program and not be able to activate HA dynamically.
But, as long as no HA service got active, it's a simple dummy "wake up every
second and do an ioctl keep-alive update".
This is really simple and efficiently written, so if that fails for over 10s
the systems is really loaded, probably barely responding to anything.
Currently the watchdog-mux runs as normal process, no re-nice, no real-time
scheduling. This is IMO wrong, as it is a critical process which needs to be
run with high priority. I've a patch here which sets it to the highest RR
realtime-scheduling priority available, effectively the same what corosync does.
diff --git a/src/watchdog-mux.c b/src/watchdog-mux.c
index 818ae00..71981d7 100644
--- a/src/watchdog-mux.c
+++ b/src/watchdog-mux.c
@@ -8,2 +8,3 @@
#include <time.h>
+#include <sched.h>
#include <sys/ioctl.h>
@@ -151,2 +177,15 @@ main(void)
+ int sched_priority = sched_get_priority_max (SCHED_RR);
+ if (sched_priority != -1) {
+ struct sched_param global_sched_param;
+ global_sched_param.sched_priority = sched_priority;
+ int res = sched_setscheduler (0, SCHED_RR, &global_sched_param);
+ if (res == -1) {
+ fprintf(stderr, "Could not set SCHED_RR at priority %d\n", sched_priority);
+ } else {
+ fprintf(stderr, "set SCHED_RR at priority %d\n", sched_priority);
+ }
+ }
+
+
if ((watchdog_fd = open(WATCHDOG_DEV, O_WRONLY)) == -1) {
The issue with no HA but watchdog reset due to massively overloaded system
should be avoided already a lot with the scheduling change alone.
Interesting, IMO, is that lots of nodes rebooted at the same time, with no HA active.
This *could* come from a side-effect like ceph rebalacing kicking off and producing
a load spike for >10s, hindering the scheduling of the watchdog-mux.
This is a theory, but with HA off it needs to be something like that, as in HA-off
case there's *no* direct or indirect connection between corosync/pmxcfs and the
watchdog-mux. It simply does not cares, or notices, quorum partition changes at all.
There may be a approach to reserve the watchdog for the mux, but avoid having it
as "ticking time bomb":
Theoretically one could open it, then disable it with an ioctl (it can be queried
if a driver support that) and only enable it for real once the first client connects
to the MUX. This may not work for all watchdog modules, and if, we may want to make
it configurable, as some people actually want a reset if a (future) real-time process
cannot be scheduled for >= 10 seconds.
With HA active, well then there could be something off, either in corosync/knet or
also in how we interface with it in pmxcfs, that could well be, but won't explain the
non-HA issues.
Speaking of pmxcfs, that one runs also with standard priority, we may want to change
that too to a RT scheduler, so that its ensured it can process all corosync events.
I have also a few other small watchdog mux patches around, it should nowadays actually
be able to tell us why a reset happened (can also be over/under voltage, temperature,
...) and I'll repeat doing the ioctl for keep-alive a few times if it fails, can only
win with that after all.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-10 4:58 ` Alexandre DERUMIER
@ 2020-09-10 8:21 ` Thomas Lamprecht
2020-09-10 11:34 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-10 8:21 UTC (permalink / raw)
To: Proxmox VE development discussion, Alexandre DERUMIER
On 10.09.20 06:58, Alexandre DERUMIER wrote:
> Thanks Thomas for the investigations.
>
> I'm still trying to reproduce...
> I think I have some special case here, because the user of the forum with 30 nodes had corosync cluster split. (Note that I had this bug 6 months ago,when shuting down a node too, and the only way was stop full stop corosync on all nodes, and start corosync again on all nodes).
>
>
> But this time, corosync logs looks fine. (every node, correctly see node2 down, and see remaning nodes)
>
> surviving node7, was the only node with HA, and LRM didn't have enable watchog (I don't have found any log like "pve-ha-lrm: watchdog active" for the last 6months on this nodes
>
>
> So, the timing was:
>
> 10:39:05 : "halt" command is send to node2
> 10:39:16 : node2 is leaving corosync / halt -> every node is seeing it and correctly do a new membership with 13 remaining nodes
>
> ...don't see any special logs (corosync,pmxcfs,pve-ha-crm,pve-ha-lrm) after the node2 leaving.
> But they are still activity on the server, pve-firewall is still logging, vms are running fine
>
>
> between 10:40:25 - 10:40:34 : watchdog reset nodes, but not node7.
>
> -> so between 70s-80s after the node2 was done, so I think that watchdog-mux was still running fine until that.
> (That's sound like lrm was stuck, and client_watchdog_timeout have expired in watchdog-mux)
as said, if the other nodes where not using HA, the watchdog-mux had no
client which could expire.
>
> 10:40:41 node7, loose quorum (as all others nodes have reset),
> 10:40:50: node7 crm/lrm finally log.
>
> Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during cfs-locked 'domain-ha' operation: no quorum!
> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
above lines also indicate very high load.
Do you have some monitoring which shows the CPU/IO load before/during this event?
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs lock update failed - Permission denied
> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - cfs lock update failed - Permission denied
>
>
>
> So, I really think that something have stucked lrm/crm loop, and watchdog was not resetted because of that.
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-10 8:21 ` Thomas Lamprecht
@ 2020-09-10 11:34 ` Alexandre DERUMIER
2020-09-10 18:21 ` Thomas Lamprecht
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-10 11:34 UTC (permalink / raw)
To: Thomas Lamprecht; +Cc: Proxmox VE development discussion
>>as said, if the other nodes where not using HA, the watchdog-mux had no
>>client which could expire.
sorry, maybe I have wrong explained it,
but all my nodes had HA enabled.
I have double check lrm_status json files from my morning backup 2h before the problem,
they were all in "active" state. ("state":"active","mode":"active" )
I don't why node7 don't have rebooted, the only difference is that is was the crm master.
(I think crm also reset the watchdog counter ? maybe behaviour is different than lrm ?)
>>above lines also indicate very high load.
>>Do you have some monitoring which shows the CPU/IO load before/during this event?
load (1,5,15 ) was: 6 (for 48cores), cpu usage: 23%
no iowait on disk (vms are on a remote ceph, only proxmox services are running on local ssd disk)
so nothing strange here :/
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "Alexandre Derumier" <aderumier@odiso.com>
Envoyé: Jeudi 10 Septembre 2020 10:21:48
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 10.09.20 06:58, Alexandre DERUMIER wrote:
> Thanks Thomas for the investigations.
>
> I'm still trying to reproduce...
> I think I have some special case here, because the user of the forum with 30 nodes had corosync cluster split. (Note that I had this bug 6 months ago,when shuting down a node too, and the only way was stop full stop corosync on all nodes, and start corosync again on all nodes).
>
>
> But this time, corosync logs looks fine. (every node, correctly see node2 down, and see remaning nodes)
>
> surviving node7, was the only node with HA, and LRM didn't have enable watchog (I don't have found any log like "pve-ha-lrm: watchdog active" for the last 6months on this nodes
>
>
> So, the timing was:
>
> 10:39:05 : "halt" command is send to node2
> 10:39:16 : node2 is leaving corosync / halt -> every node is seeing it and correctly do a new membership with 13 remaining nodes
>
> ...don't see any special logs (corosync,pmxcfs,pve-ha-crm,pve-ha-lrm) after the node2 leaving.
> But they are still activity on the server, pve-firewall is still logging, vms are running fine
>
>
> between 10:40:25 - 10:40:34 : watchdog reset nodes, but not node7.
>
> -> so between 70s-80s after the node2 was done, so I think that watchdog-mux was still running fine until that.
> (That's sound like lrm was stuck, and client_watchdog_timeout have expired in watchdog-mux)
as said, if the other nodes where not using HA, the watchdog-mux had no
client which could expire.
>
> 10:40:41 node7, loose quorum (as all others nodes have reset),
> 10:40:50: node7 crm/lrm finally log.
>
> Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during cfs-locked 'domain-ha' operation: no quorum!
> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
above lines also indicate very high load.
Do you have some monitoring which shows the CPU/IO load before/during this event?
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs lock update failed - Permission denied
> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - cfs lock update failed - Permission denied
>
>
>
> So, I really think that something have stucked lrm/crm loop, and watchdog was not resetted because of that.
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-10 11:34 ` Alexandre DERUMIER
@ 2020-09-10 18:21 ` Thomas Lamprecht
2020-09-14 4:54 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-10 18:21 UTC (permalink / raw)
To: Proxmox VE development discussion, Alexandre DERUMIER
On 10.09.20 13:34, Alexandre DERUMIER wrote:
>>> as said, if the other nodes where not using HA, the watchdog-mux had no
>>> client which could expire.
>
> sorry, maybe I have wrong explained it,
> but all my nodes had HA enabled.
>
> I have double check lrm_status json files from my morning backup 2h before the problem,
> they were all in "active" state. ("state":"active","mode":"active" )
>
OK, so all had a connection to the watchdog-mux open. This shifts the suspicion
again over to pmxcfs and/or corosync.
> I don't why node7 don't have rebooted, the only difference is that is was the crm master.
> (I think crm also reset the watchdog counter ? maybe behaviour is different than lrm ?)
The watchdog-mux stops updating the real watchdog as soon any client disconnects or times
out. It does not know which client (daemon) that was.
>>> above lines also indicate very high load.
>>> Do you have some monitoring which shows the CPU/IO load before/during this event?
>
> load (1,5,15 ) was: 6 (for 48cores), cpu usage: 23%
> no iowait on disk (vms are on a remote ceph, only proxmox services are running on local ssd disk)
>
> so nothing strange here :/
Hmm, the long loop times could then be the effect of a pmxcfs read or write
operation being (temporarily) stuck.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-10 18:21 ` Thomas Lamprecht
@ 2020-09-14 4:54 ` Alexandre DERUMIER
2020-09-14 7:14 ` Dietmar Maurer
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-14 4:54 UTC (permalink / raw)
To: Thomas Lamprecht; +Cc: Proxmox VE development discussion
I wonder if something like pacemaker sbd could be implemented in proxmox as extra layer of protection ?
http://manpages.ubuntu.com/manpages/bionic/man8/sbd.8.html
(shared disk heartbeat).
Something like a independent daemon (not using corosync/pmxcfs/...), also connected to watchdog muxer.
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "aderumier" <aderumier@odiso.com>
Envoyé: Jeudi 10 Septembre 2020 20:21:14
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 10.09.20 13:34, Alexandre DERUMIER wrote:
>>> as said, if the other nodes where not using HA, the watchdog-mux had no
>>> client which could expire.
>
> sorry, maybe I have wrong explained it,
> but all my nodes had HA enabled.
>
> I have double check lrm_status json files from my morning backup 2h before the problem,
> they were all in "active" state. ("state":"active","mode":"active" )
>
OK, so all had a connection to the watchdog-mux open. This shifts the suspicion
again over to pmxcfs and/or corosync.
> I don't why node7 don't have rebooted, the only difference is that is was the crm master.
> (I think crm also reset the watchdog counter ? maybe behaviour is different than lrm ?)
The watchdog-mux stops updating the real watchdog as soon any client disconnects or times
out. It does not know which client (daemon) that was.
>>> above lines also indicate very high load.
>>> Do you have some monitoring which shows the CPU/IO load before/during this event?
>
> load (1,5,15 ) was: 6 (for 48cores), cpu usage: 23%
> no iowait on disk (vms are on a remote ceph, only proxmox services are running on local ssd disk)
>
> so nothing strange here :/
Hmm, the long loop times could then be the effect of a pmxcfs read or write
operation being (temporarily) stuck.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-14 4:54 ` Alexandre DERUMIER
@ 2020-09-14 7:14 ` Dietmar Maurer
2020-09-14 8:27 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Dietmar Maurer @ 2020-09-14 7:14 UTC (permalink / raw)
To: Proxmox VE development discussion, Alexandre DERUMIER, Thomas Lamprecht
> I wonder if something like pacemaker sbd could be implemented in proxmox as extra layer of protection ?
AFAIK Thomas already has patches to implement active fencing.
But IMHO this will not solve the corosync problems..
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-14 7:14 ` Dietmar Maurer
@ 2020-09-14 8:27 ` Alexandre DERUMIER
2020-09-14 8:51 ` Thomas Lamprecht
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-14 8:27 UTC (permalink / raw)
To: dietmar; +Cc: Proxmox VE development discussion, Thomas Lamprecht
> I wonder if something like pacemaker sbd could be implemented in proxmox as extra layer of protection ?
>>AFAIK Thomas already has patches to implement active fencing.
>>But IMHO this will not solve the corosync problems..
Yes, sure. I'm really to have to 2 differents sources of verification, with different path/software, to avoid this kind of bug.
(shit happens, murphy law ;)
as we say in French "ceinture & bretelles" -> "belt and braces"
BTW,
a user have reported new corosync problem here:
https://forum.proxmox.com/threads/proxmox-6-2-corosync-3-rare-and-spontaneous-disruptive-udp-5405-storm-flood.75871
(Sound like the bug that I have 6month ago, with corosync bug flooding a lof of udp packets, but not the same bug I have here)
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "aderumier" <aderumier@odiso.com>, "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoyé: Lundi 14 Septembre 2020 09:14:50
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
> I wonder if something like pacemaker sbd could be implemented in proxmox as extra layer of protection ?
AFAIK Thomas already has patches to implement active fencing.
But IMHO this will not solve the corosync problems..
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-14 8:27 ` Alexandre DERUMIER
@ 2020-09-14 8:51 ` Thomas Lamprecht
2020-09-14 15:45 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-14 8:51 UTC (permalink / raw)
To: Proxmox VE development discussion, Alexandre DERUMIER, dietmar
On 9/14/20 10:27 AM, Alexandre DERUMIER wrote:
>> I wonder if something like pacemaker sbd could be implemented in proxmox as extra layer of protection ?
>
>>> AFAIK Thomas already has patches to implement active fencing.
>
>>> But IMHO this will not solve the corosync problems..
>
> Yes, sure. I'm really to have to 2 differents sources of verification, with different path/software, to avoid this kind of bug.
> (shit happens, murphy law ;)
would then need at least three, and if one has a bug flooding the network in
a lot of setups (not having beefy switches like you ;) the other two will be
taken down also, either as memory or the system stack gets overloaded.
>
> as we say in French "ceinture & bretelles" -> "belt and braces"
>
>
> BTW,
> a user have reported new corosync problem here:
> https://forum.proxmox.com/threads/proxmox-6-2-corosync-3-rare-and-spontaneous-disruptive-udp-5405-storm-flood.75871
> (Sound like the bug that I have 6month ago, with corosync bug flooding a lof of udp packets, but not the same bug I have here)
Did you get in contact with knet/corosync devs about this?
Because, it may well be something their stack is better at handling it, maybe
there's also really still a bug, or bad behaviour on some edge cases...
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-14 8:51 ` Thomas Lamprecht
@ 2020-09-14 15:45 ` Alexandre DERUMIER
2020-09-15 5:45 ` dietmar
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-14 15:45 UTC (permalink / raw)
To: Thomas Lamprecht; +Cc: Proxmox VE development discussion, dietmar
>>Did you get in contact with knet/corosync devs about this?
>>Because, it may well be something their stack is better at handling it, maybe
>>there's also really still a bug, or bad behaviour on some edge cases...
not yet, I would like to have more infos to submit, because I'm blind.
I have enabled debug logs on all my cluster if that happen again.
BTW,
I have noticed something,
corosync is stopped after syslog stop, so at shutdown we never have corosync log
I have edit corosync.service
- After=network-online.target
+ After=network-online.target syslog.target
and now It's logging correctly.
Now, that logging work, I'm also seeeing pmxcfs errors when corosync is stopping.
(But no pmxcfs shutdown log)
Do you think it's possible to have a clean shutdown of pmxcfs first, before stopping corosync ?
"
Sep 14 17:23:49 pve corosync[1346]: [MAIN ] Node was shut down by a signal
Sep 14 17:23:49 pve systemd[1]: Stopping Corosync Cluster Engine...
Sep 14 17:23:49 pve corosync[1346]: [SERV ] Unloading all Corosync service engines.
Sep 14 17:23:49 pve corosync[1346]: [QB ] withdrawing server sockets
Sep 14 17:23:49 pve corosync[1346]: [SERV ] Service engine unloaded: corosync vote quorum service v1.0
Sep 14 17:23:49 pve pmxcfs[1132]: [confdb] crit: cmap_dispatch failed: 2
Sep 14 17:23:49 pve corosync[1346]: [QB ] withdrawing server sockets
Sep 14 17:23:49 pve corosync[1346]: [SERV ] Service engine unloaded: corosync configuration map access
Sep 14 17:23:49 pve corosync[1346]: [QB ] withdrawing server sockets
Sep 14 17:23:49 pve corosync[1346]: [SERV ] Service engine unloaded: corosync configuration service
Sep 14 17:23:49 pve pmxcfs[1132]: [status] crit: cpg_dispatch failed: 2
Sep 14 17:23:49 pve pmxcfs[1132]: [status] crit: cpg_leave failed: 2
Sep 14 17:23:49 pve pmxcfs[1132]: [dcdb] crit: cpg_dispatch failed: 2
Sep 14 17:23:49 pve pmxcfs[1132]: [dcdb] crit: cpg_leave failed: 2
Sep 14 17:23:49 pve corosync[1346]: [QB ] withdrawing server sockets
Sep 14 17:23:49 pve corosync[1346]: [SERV ] Service engine unloaded: corosync cluster quorum service v0.1
Sep 14 17:23:49 pve pmxcfs[1132]: [quorum] crit: quorum_dispatch failed: 2
Sep 14 17:23:49 pve pmxcfs[1132]: [status] notice: node lost quorum
Sep 14 17:23:49 pve corosync[1346]: [SERV ] Service engine unloaded: corosync profile loading service
Sep 14 17:23:49 pve corosync[1346]: [SERV ] Service engine unloaded: corosync resource monitoring service
Sep 14 17:23:49 pve corosync[1346]: [SERV ] Service engine unloaded: corosync watchdog service
Sep 14 17:23:49 pve pmxcfs[1132]: [quorum] crit: quorum_initialize failed: 2
Sep 14 17:23:49 pve pmxcfs[1132]: [quorum] crit: can't initialize service
Sep 14 17:23:49 pve pmxcfs[1132]: [confdb] crit: cmap_initialize failed: 2
Sep 14 17:23:49 pve pmxcfs[1132]: [confdb] crit: can't initialize service
Sep 14 17:23:49 pve pmxcfs[1132]: [dcdb] notice: start cluster connection
Sep 14 17:23:49 pve pmxcfs[1132]: [dcdb] crit: cpg_initialize failed: 2
Sep 14 17:23:49 pve pmxcfs[1132]: [dcdb] crit: can't initialize service
Sep 14 17:23:49 pve pmxcfs[1132]: [status] notice: start cluster connection
Sep 14 17:23:49 pve pmxcfs[1132]: [status] crit: cpg_initialize failed: 2
Sep 14 17:23:49 pve pmxcfs[1132]: [status] crit: can't initialize service
Sep 14 17:23:50 pve corosync[1346]: [MAIN ] Corosync Cluster Engine exiting normally
"
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "aderumier" <aderumier@odiso.com>, "dietmar" <dietmar@proxmox.com>
Envoyé: Lundi 14 Septembre 2020 10:51:03
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/14/20 10:27 AM, Alexandre DERUMIER wrote:
>> I wonder if something like pacemaker sbd could be implemented in proxmox as extra layer of protection ?
>
>>> AFAIK Thomas already has patches to implement active fencing.
>
>>> But IMHO this will not solve the corosync problems..
>
> Yes, sure. I'm really to have to 2 differents sources of verification, with different path/software, to avoid this kind of bug.
> (shit happens, murphy law ;)
would then need at least three, and if one has a bug flooding the network in
a lot of setups (not having beefy switches like you ;) the other two will be
taken down also, either as memory or the system stack gets overloaded.
>
> as we say in French "ceinture & bretelles" -> "belt and braces"
>
>
> BTW,
> a user have reported new corosync problem here:
> https://forum.proxmox.com/threads/proxmox-6-2-corosync-3-rare-and-spontaneous-disruptive-udp-5405-storm-flood.75871
> (Sound like the bug that I have 6month ago, with corosync bug flooding a lof of udp packets, but not the same bug I have here)
Did you get in contact with knet/corosync devs about this?
Because, it may well be something their stack is better at handling it, maybe
there's also really still a bug, or bad behaviour on some edge cases...
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-14 15:45 ` Alexandre DERUMIER
@ 2020-09-15 5:45 ` dietmar
2020-09-15 6:27 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: dietmar @ 2020-09-15 5:45 UTC (permalink / raw)
To: Alexandre DERUMIER, Thomas Lamprecht; +Cc: Proxmox VE development discussion
> Now, that logging work, I'm also seeeing pmxcfs errors when corosync is stopping.
> (But no pmxcfs shutdown log)
>
> Do you think it's possible to have a clean shutdown of pmxcfs first, before stopping corosync ?
This is by intention - we do not want to stop pmxcfs only because coorosync service stops.
Unit]
Description=The Proxmox VE cluster filesystem
ConditionFileIsExecutable=/usr/bin/pmxcfs
Wants=corosync.service
Wants=rrdcached.service
Before=corosync.service
Before=ceph.service
Before=cron.service
After=network.target
After=sys-fs-fuse-connections.mount
After=time-sync.target
After=rrdcached.service
DefaultDependencies=no
Before=shutdown.target
Conflicts=shutdown.target
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 5:45 ` dietmar
@ 2020-09-15 6:27 ` Alexandre DERUMIER
2020-09-15 7:13 ` dietmar
2020-09-15 7:58 ` Thomas Lamprecht
0 siblings, 2 replies; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-15 6:27 UTC (permalink / raw)
To: dietmar; +Cc: Thomas Lamprecht, Proxmox VE development discussion
>>This is by intention - we do not want to stop pmxcfs only because coorosync service stops.
Yes, but at shutdown, it could be great to stop pmxcfs before corosync ?
I ask the question, because the 2 times I have problem, it was when shutting down a server.
So maybe some strange behaviour occur with both corosync && pmxcfs are stopped at same time ?
looking at the pve-cluster unit file,
why do we have "Before=corosync.service" and not "After=corosync.service" ?
I have tried to change this, but even with that, both are still shutting down in parallel.
the only way I have found to have clean shutdown, is "Requires=corosync.server" + "After=corosync.service".
But that mean than if you restart corosync, it's restart pmxcfs too first.
I have looked at systemd doc, After= should be enough (as at shutdown it's doing the reverse order),
but I don't known why corosync don't wait than pve-cluster ???
(Also, I think than pmxcfs is also stopping after syslog, because I never see the pmxcfs "teardown filesystem" logs at shutdown)
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "aderumier" <aderumier@odiso.com>, "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 07:45:41
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
> Now, that logging work, I'm also seeeing pmxcfs errors when corosync is stopping.
> (But no pmxcfs shutdown log)
>
> Do you think it's possible to have a clean shutdown of pmxcfs first, before stopping corosync ?
This is by intention - we do not want to stop pmxcfs only because coorosync service stops.
Unit]
Description=The Proxmox VE cluster filesystem
ConditionFileIsExecutable=/usr/bin/pmxcfs
Wants=corosync.service
Wants=rrdcached.service
Before=corosync.service
Before=ceph.service
Before=cron.service
After=network.target
After=sys-fs-fuse-connections.mount
After=time-sync.target
After=rrdcached.service
DefaultDependencies=no
Before=shutdown.target
Conflicts=shutdown.target
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 6:27 ` Alexandre DERUMIER
@ 2020-09-15 7:13 ` dietmar
2020-09-15 8:42 ` Alexandre DERUMIER
2020-09-15 7:58 ` Thomas Lamprecht
1 sibling, 1 reply; 84+ messages in thread
From: dietmar @ 2020-09-15 7:13 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: Thomas Lamprecht, Proxmox VE development discussion
> I ask the question, because the 2 times I have problem, it was when shutting down a server.
> So maybe some strange behaviour occur with both corosync && pmxcfs are stopped at same time ?
pmxcfs cannot send anything in that case, so it is impossible that this has effects on other nodes.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 6:27 ` Alexandre DERUMIER
2020-09-15 7:13 ` dietmar
@ 2020-09-15 7:58 ` Thomas Lamprecht
1 sibling, 0 replies; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-15 7:58 UTC (permalink / raw)
To: Alexandre DERUMIER, dietmar; +Cc: Proxmox VE development discussion
On 9/15/20 8:27 AM, Alexandre DERUMIER wrote:
>>> This is by intention - we do not want to stop pmxcfs only because coorosync service stops.
>
> Yes, but at shutdown, it could be great to stop pmxcfs before corosync ?
> I ask the question, because the 2 times I have problem, it was when shutting down a server.
> So maybe some strange behaviour occur with both corosync && pmxcfs are stopped at same time ?
>
>
> looking at the pve-cluster unit file,
> why do we have "Before=corosync.service" and not "After=corosync.service" ?
We may need to sync over the cluster corosync.conf to the local one, that can
only happen before.
Also, if we shutdown pmxcfs before corosync we may still get corosync events (file writes,
locking, ...) but the node does not sees it locally anymore but still looks quorate for
others, that'd be not good.
>
> I have tried to change this, but even with that, both are still shutting down in parallel.
>
> the only way I have found to have clean shutdown, is "Requires=corosync.server" + "After=corosync.service".
> But that mean than if you restart corosync, it's restart pmxcfs too first.
>
> I have looked at systemd doc, After= should be enough (as at shutdown it's doing the reverse order),
> but I don't known why corosync don't wait than pve-cluster ???
>
>
> (Also, I think than pmxcfs is also stopping after syslog, because I never see the pmxcfs "teardown filesystem" logs at shutdown)
is that true for (persistent) systemd-journald too? IIRC syslog.target is
deprecated and only rsyslog provides it.
As the next Debian will enable persistent journal by default and we already
use it for everything (IIRC) were we provide an interface to logs, we will
probably not enable rsyslog by default with PVE 7.x
But if we can add some ordering for this to be improved I'm open for it.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 7:13 ` dietmar
@ 2020-09-15 8:42 ` Alexandre DERUMIER
2020-09-15 9:35 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-15 8:42 UTC (permalink / raw)
To: dietmar; +Cc: Thomas Lamprecht, Proxmox VE development discussion
>>pmxcfs cannot send anything in that case, so it is impossible that this has effects on other nodes.
yes, I understand that, but I was thinking of the case if corosync is in stopping phase (not totally stopped).
Something racy (I really don't known).
I just send 2 patch to start pve-cluster && corosync after syslog, like this we'll have shutdown logs too.
(I'm currently try to reproduce the problem, with reboot loops, but I still can't reproduce it :/ )
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Thomas Lamprecht" <t.lamprecht@proxmox.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 09:13:53
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
> I ask the question, because the 2 times I have problem, it was when shutting down a server.
> So maybe some strange behaviour occur with both corosync && pmxcfs are stopped at same time ?
pmxcfs cannot send anything in that case, so it is impossible that this has effects on other nodes.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 8:42 ` Alexandre DERUMIER
@ 2020-09-15 9:35 ` Alexandre DERUMIER
2020-09-15 9:46 ` Thomas Lamprecht
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-15 9:35 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: dietmar, Thomas Lamprecht
Hi,
I have finally reproduce it !
But this is with a corosync restart in cron each 1 minute, on node1
Then: lrm was stuck for too long for around 60s and softdog have been triggered on multiple other nodes.
here the logs with full corosync debug at the time of last corosync restart.
node1 (where corosync is restarted each minute)
https://gist.github.com/aderumier/c4f192fbce8e96759f91a61906db514e
node2
https://gist.github.com/aderumier/2d35ea05c1fbff163652e564fc430e67
node5
https://gist.github.com/aderumier/df1d91cddbb6e15bb0d0193ed8df9273
I'll prepare logs from the previous corosync restart, as the lrm seem to be already stuck before.
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "dietmar" <dietmar@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoyé: Mardi 15 Septembre 2020 10:42:15
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>pmxcfs cannot send anything in that case, so it is impossible that this has effects on other nodes.
yes, I understand that, but I was thinking of the case if corosync is in stopping phase (not totally stopped).
Something racy (I really don't known).
I just send 2 patch to start pve-cluster && corosync after syslog, like this we'll have shutdown logs too.
(I'm currently try to reproduce the problem, with reboot loops, but I still can't reproduce it :/ )
----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Thomas Lamprecht" <t.lamprecht@proxmox.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 09:13:53
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
> I ask the question, because the 2 times I have problem, it was when shutting down a server.
> So maybe some strange behaviour occur with both corosync && pmxcfs are stopped at same time ?
pmxcfs cannot send anything in that case, so it is impossible that this has effects on other nodes.
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 9:35 ` Alexandre DERUMIER
@ 2020-09-15 9:46 ` Thomas Lamprecht
2020-09-15 10:15 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-15 9:46 UTC (permalink / raw)
To: Alexandre DERUMIER, Proxmox VE development discussion
On 9/15/20 11:35 AM, Alexandre DERUMIER wrote:
> Hi,
>
> I have finally reproduce it !
>
> But this is with a corosync restart in cron each 1 minute, on node1
>
> Then: lrm was stuck for too long for around 60s and softdog have been triggered on multiple other nodes.
>
> here the logs with full corosync debug at the time of last corosync restart.
>
> node1 (where corosync is restarted each minute)
> https://gist.github.com/aderumier/c4f192fbce8e96759f91a61906db514e
>
> node2
> https://gist.github.com/aderumier/2d35ea05c1fbff163652e564fc430e67
>
> node5
> https://gist.github.com/aderumier/df1d91cddbb6e15bb0d0193ed8df9273
>
> I'll prepare logs from the previous corosync restart, as the lrm seem to be already stuck before.
Yeah that would be good, as yes the lrm seems to get stuck at around 10:46:21
> Sep 15 10:47:26 m6kvm2 pve-ha-lrm[3736]: loop take too long (65 seconds)
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 9:46 ` Thomas Lamprecht
@ 2020-09-15 10:15 ` Alexandre DERUMIER
2020-09-15 11:04 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-15 10:15 UTC (permalink / raw)
To: Thomas Lamprecht; +Cc: Proxmox VE development discussion, dietmar
here the previous restart log
node1 -> corosync restart at 10:46:15
-----
https://gist.github.com/aderumier/0992051d20f51270ceceb5b3431d18d7
node2
-----
https://gist.github.com/aderumier/eea0c50fefc1d8561868576f417191ba
node5
------
https://gist.github.com/aderumier/f2ce1bc5a93827045a5691583bbc7a37
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "dietmar" <dietmar@proxmox.com>
Envoyé: Mardi 15 Septembre 2020 11:46:51
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/15/20 11:35 AM, Alexandre DERUMIER wrote:
> Hi,
>
> I have finally reproduce it !
>
> But this is with a corosync restart in cron each 1 minute, on node1
>
> Then: lrm was stuck for too long for around 60s and softdog have been triggered on multiple other nodes.
>
> here the logs with full corosync debug at the time of last corosync restart.
>
> node1 (where corosync is restarted each minute)
> https://gist.github.com/aderumier/c4f192fbce8e96759f91a61906db514e
>
> node2
> https://gist.github.com/aderumier/2d35ea05c1fbff163652e564fc430e67
>
> node5
> https://gist.github.com/aderumier/df1d91cddbb6e15bb0d0193ed8df9273
>
> I'll prepare logs from the previous corosync restart, as the lrm seem to be already stuck before.
Yeah that would be good, as yes the lrm seems to get stuck at around 10:46:21
> Sep 15 10:47:26 m6kvm2 pve-ha-lrm[3736]: loop take too long (65 seconds)
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 10:15 ` Alexandre DERUMIER
@ 2020-09-15 11:04 ` Alexandre DERUMIER
2020-09-15 12:49 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-15 11:04 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: Thomas Lamprecht
also logs of node14, where the lrm was not too long
https://gist.github.com/aderumier/a2e2d6afc7e04646c923ae6f37cb6c2d
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 12:15:47
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
here the previous restart log
node1 -> corosync restart at 10:46:15
-----
https://gist.github.com/aderumier/0992051d20f51270ceceb5b3431d18d7
node2
-----
https://gist.github.com/aderumier/eea0c50fefc1d8561868576f417191ba
node5
------
https://gist.github.com/aderumier/f2ce1bc5a93827045a5691583bbc7a37
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "dietmar" <dietmar@proxmox.com>
Envoyé: Mardi 15 Septembre 2020 11:46:51
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/15/20 11:35 AM, Alexandre DERUMIER wrote:
> Hi,
>
> I have finally reproduce it !
>
> But this is with a corosync restart in cron each 1 minute, on node1
>
> Then: lrm was stuck for too long for around 60s and softdog have been triggered on multiple other nodes.
>
> here the logs with full corosync debug at the time of last corosync restart.
>
> node1 (where corosync is restarted each minute)
> https://gist.github.com/aderumier/c4f192fbce8e96759f91a61906db514e
>
> node2
> https://gist.github.com/aderumier/2d35ea05c1fbff163652e564fc430e67
>
> node5
> https://gist.github.com/aderumier/df1d91cddbb6e15bb0d0193ed8df9273
>
> I'll prepare logs from the previous corosync restart, as the lrm seem to be already stuck before.
Yeah that would be good, as yes the lrm seems to get stuck at around 10:46:21
> Sep 15 10:47:26 m6kvm2 pve-ha-lrm[3736]: loop take too long (65 seconds)
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 11:04 ` Alexandre DERUMIER
@ 2020-09-15 12:49 ` Alexandre DERUMIER
2020-09-15 13:00 ` Thomas Lamprecht
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-15 12:49 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: Thomas Lamprecht
Hi,
I have produce it again,
now I can't write to /etc/pve/ from any node
I have also added some debug logs to pve-ha-lrm, and it was stuck in:
(but if /etc/pve is locked, this is normal)
if ($fence_request) {
$haenv->log('err', "node need to be fenced - releasing agent_lock\n");
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif (!$self->get_protected_ha_agent_lock()) {
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif ($self->{mode} eq 'maintenance') {
$self->set_local_status({ state => 'maintenance'});
}
corosync quorum is currently ok
I'm currently digging the logs
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoyé: Mardi 15 Septembre 2020 13:04:31
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
also logs of node14, where the lrm was not too long
https://gist.github.com/aderumier/a2e2d6afc7e04646c923ae6f37cb6c2d
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 12:15:47
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
here the previous restart log
node1 -> corosync restart at 10:46:15
-----
https://gist.github.com/aderumier/0992051d20f51270ceceb5b3431d18d7
node2
-----
https://gist.github.com/aderumier/eea0c50fefc1d8561868576f417191ba
node5
------
https://gist.github.com/aderumier/f2ce1bc5a93827045a5691583bbc7a37
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "dietmar" <dietmar@proxmox.com>
Envoyé: Mardi 15 Septembre 2020 11:46:51
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/15/20 11:35 AM, Alexandre DERUMIER wrote:
> Hi,
>
> I have finally reproduce it !
>
> But this is with a corosync restart in cron each 1 minute, on node1
>
> Then: lrm was stuck for too long for around 60s and softdog have been triggered on multiple other nodes.
>
> here the logs with full corosync debug at the time of last corosync restart.
>
> node1 (where corosync is restarted each minute)
> https://gist.github.com/aderumier/c4f192fbce8e96759f91a61906db514e
>
> node2
> https://gist.github.com/aderumier/2d35ea05c1fbff163652e564fc430e67
>
> node5
> https://gist.github.com/aderumier/df1d91cddbb6e15bb0d0193ed8df9273
>
> I'll prepare logs from the previous corosync restart, as the lrm seem to be already stuck before.
Yeah that would be good, as yes the lrm seems to get stuck at around 10:46:21
> Sep 15 10:47:26 m6kvm2 pve-ha-lrm[3736]: loop take too long (65 seconds)
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 12:49 ` Alexandre DERUMIER
@ 2020-09-15 13:00 ` Thomas Lamprecht
2020-09-15 14:09 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-15 13:00 UTC (permalink / raw)
To: Alexandre DERUMIER, Proxmox VE development discussion
On 9/15/20 2:49 PM, Alexandre DERUMIER wrote:
> Hi,
>
> I have produce it again,
>
> now I can't write to /etc/pve/ from any node
>
OK, so seems to really be an issue in pmxcfs or between corosync and pmxcfs,
not the HA LRM or watchdog mux itself.
Can you try to give pmxcfs real time scheduling, e.g., by doing:
# systemctl edit pve-cluster
And then add snippet:
[Service]
CPUSchedulingPolicy=rr
CPUSchedulingPriority=99
And restart pve-cluster
> I have also added some debug logs to pve-ha-lrm, and it was stuck in:
> (but if /etc/pve is locked, this is normal)
>
> if ($fence_request) {
> $haenv->log('err', "node need to be fenced - releasing agent_lock\n");
> $self->set_local_status({ state => 'lost_agent_lock'});
> } elsif (!$self->get_protected_ha_agent_lock()) {
> $self->set_local_status({ state => 'lost_agent_lock'});
> } elsif ($self->{mode} eq 'maintenance') {
> $self->set_local_status({ state => 'maintenance'});
> }
>
>
> corosync quorum is currently ok
>
> I'm currently digging the logs
Is your most simplest/stable reproducer still a periodic restart of corosync in one node?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 13:00 ` Thomas Lamprecht
@ 2020-09-15 14:09 ` Alexandre DERUMIER
2020-09-15 14:19 ` Alexandre DERUMIER
2020-09-15 14:32 ` Thomas Lamprecht
0 siblings, 2 replies; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-15 14:09 UTC (permalink / raw)
To: Thomas Lamprecht; +Cc: Proxmox VE development discussion
>>
>>Can you try to give pmxcfs real time scheduling, e.g., by doing:
>>
>># systemctl edit pve-cluster
>>
>>And then add snippet:
>>
>>
>>[Service]
>>CPUSchedulingPolicy=rr
>>CPUSchedulingPriority=99
yes, sure, I'll do it now
> I'm currently digging the logs
>>Is your most simplest/stable reproducer still a periodic restart of corosync in one node?
yes, a simple "systemctl restart corosync" on 1 node each minute
After 1hour, it's still locked.
on other nodes, I still have pmxfs logs like:
Sep 15 15:36:31 m6kvm2 pmxcfs[3474]: [status] notice: received log
Sep 15 15:46:21 m6kvm2 pmxcfs[3474]: [status] notice: received log
Sep 15 15:46:23 m6kvm2 pmxcfs[3474]: [status] notice: received log
...
on node1, I just restarted the pve-cluster service with systemctl restart pve-cluster,
the pmxcfs process was killed, but not able to start it again
and after that the /etc/pve become writable again on others node.
(I don't have rebooted yet node1, if you want more test on pmxcfs)
root@m6kvm1:~# systemctl status pve-cluster
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2020-09-15 15:52:11 CEST; 3min 29s ago
Process: 12536 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)
Sep 15 15:52:11 m6kvm1 systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Sep 15 15:52:11 m6kvm1 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Sep 15 15:52:11 m6kvm1 systemd[1]: Stopped The Proxmox VE cluster filesystem.
Sep 15 15:52:11 m6kvm1 systemd[1]: pve-cluster.service: Start request repeated too quickly.
Sep 15 15:52:11 m6kvm1 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 15 15:52:11 m6kvm1 systemd[1]: Failed to start The Proxmox VE cluster filesystem.
manual "pmxcfs -d"
https://gist.github.com/aderumier/4cd91d17e1f8847b93ea5f621f257c2e
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_initialize failed: 2
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [quorum] crit: can't initialize service
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_initialize failed: 2
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [confdb] crit: can't initialize service
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [dcdb] notice: start cluster connection
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_initialize failed: 2
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [dcdb] crit: can't initialize service
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [status] notice: start cluster connection
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [status] crit: cpg_initialize failed: 2
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [status] crit: can't initialize service
Sep 15 14:38:30 m6kvm1 pmxcfs[3491]: [status] notice: update cluster info (cluster name m6kvm, version = 20)
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: node has quorum
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: starting data syncronisation
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: starting data syncronisation
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: received sync request (epoch 1/3491/00000064)
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: received sync request (epoch 1/3491/00000063)
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: received all states
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: leader is 2/3474
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: synced members: 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: waiting for updates from leader
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: dfsm_deliver_queue: queue length 23
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: received all states
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: all data is up to date
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: dfsm_deliver_queue: queue length 157
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: update complete - trying to commit (got 4 inode updates)
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: all data is up to date
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: dfsm_deliver_sync_queue: queue length 31
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_dispatch failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [status] crit: cpg_dispatch failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [status] crit: cpg_leave failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_dispatch failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_leave failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_dispatch failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [status] notice: node lost quorum
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_initialize failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [quorum] crit: can't initialize service
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_initialize failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [confdb] crit: can't initialize service
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [dcdb] notice: start cluster connection
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_initialize failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [dcdb] crit: can't initialize service
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [status] notice: start cluster connection
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [status] crit: cpg_initialize failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [status] crit: can't initialize service
Sep 15 14:39:31 m6kvm1 pmxcfs[3491]: [status] notice: update cluster info (cluster name m6kvm, version = 20)
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [status] notice: node has quorum
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: starting data syncronisation
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [status] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [status] notice: starting data syncronisation
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: received sync request (epoch 1/3491/00000065)
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [status] notice: received sync request (epoch 1/3491/00000064)
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: received all states
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: leader is 2/3474
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: synced members: 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: waiting for updates from leader
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: dfsm_deliver_queue: queue length 20
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [status] notice: received all states
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [status] notice: all data is up to date
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: update complete - trying to commit (got 9 inode updates)
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: all data is up to date
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: dfsm_deliver_sync_queue: queue length 25
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_dispatch failed: 2
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [status] crit: cpg_dispatch failed: 2
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [status] crit: cpg_leave failed: 2
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_dispatch failed: 2
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_leave failed: 2
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_dispatch failed: 2
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [status] notice: node lost quorum
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_initialize failed: 2
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [quorum] crit: can't initialize service
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_initialize failed: 2
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [confdb] crit: can't initialize service
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [dcdb] notice: start cluster connection
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_initialize failed: 2
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [dcdb] crit: can't initialize service
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [status] notice: start cluster connection
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [status] crit: cpg_initialize failed: 2
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [status] crit: can't initialize service
Sep 15 14:40:33 m6kvm1 pmxcfs[3491]: [status] notice: update cluster info (cluster name m6kvm, version = 20)
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: node has quorum
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: starting data syncronisation
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: starting data syncronisation
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: received sync request (epoch 1/3491/00000066)
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: received sync request (epoch 1/3491/00000065)
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: received all states
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: leader is 2/3474
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: synced members: 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: waiting for updates from leader
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: dfsm_deliver_queue: queue length 23
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: received all states
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: all data is up to date
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: dfsm_deliver_queue: queue length 87
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: update complete - trying to commit (got 6 inode updates)
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: all data is up to date
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: dfsm_deliver_sync_queue: queue length 33
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_dispatch failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [status] crit: cpg_dispatch failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [status] crit: cpg_leave failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_dispatch failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_leave failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_dispatch failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [status] notice: node lost quorum
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_initialize failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [quorum] crit: can't initialize service
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_initialize failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [confdb] crit: can't initialize service
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [dcdb] notice: start cluster connection
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_initialize failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [dcdb] crit: can't initialize service
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [status] notice: start cluster connection
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [status] crit: cpg_initialize failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [status] crit: can't initialize service
Sep 15 14:41:34 m6kvm1 pmxcfs[3491]: [status] notice: update cluster info (cluster name m6kvm, version = 20)
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [status] notice: node has quorum
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [dcdb] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [dcdb] notice: starting data syncronisation
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [status] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [status] notice: starting data syncronisation
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [dcdb] notice: received sync request (epoch 1/3491/00000067)
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [status] notice: received sync request (epoch 1/3491/00000066)
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [status] notice: received all states
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [status] notice: all data is up to date
Sep 15 14:47:54 m6kvm1 pmxcfs[3491]: [status] notice: received log
Sep 15 15:02:55 m6kvm1 pmxcfs[3491]: [status] notice: received log
Sep 15 15:17:56 m6kvm1 pmxcfs[3491]: [status] notice: received log
Sep 15 15:32:57 m6kvm1 pmxcfs[3491]: [status] notice: received log
Sep 15 15:47:58 m6kvm1 pmxcfs[3491]: [status] notice: received log
----> restart
2352 [ 15/09/2020 15:52:00 ] systemctl restart pve-cluster
Sep 15 15:52:10 m6kvm1 pmxcfs[12438]: [main] crit: fuse_mount error: Transport endpoint is not connected
Sep 15 15:52:10 m6kvm1 pmxcfs[12438]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 15 15:52:10 m6kvm1 pmxcfs[12529]: [main] crit: fuse_mount error: Transport endpoint is not connected
Sep 15 15:52:10 m6kvm1 pmxcfs[12529]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 15 15:52:10 m6kvm1 pmxcfs[12531]: [main] crit: fuse_mount error: Transport endpoint is not connected
Sep 15 15:52:10 m6kvm1 pmxcfs[12531]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 15 15:52:11 m6kvm1 pmxcfs[12533]: [main] crit: fuse_mount error: Transport endpoint is not connected
Sep 15 15:52:11 m6kvm1 pmxcfs[12533]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 15 15:52:11 m6kvm1 pmxcfs[12536]: [main] crit: fuse_mount error: Transport endpoint is not connected
Sep 15 15:52:11 m6kvm1 pmxcfs[12536]: [main] notice: exit proxmox configuration filesystem (-1)
some interesting dmesg about "pvesr"
[Tue Sep 15 14:45:34 2020] INFO: task pvesr:19038 blocked for more than 120 seconds.
[Tue Sep 15 14:45:34 2020] Tainted: P O 5.4.60-1-pve #1
[Tue Sep 15 14:45:34 2020] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Sep 15 14:45:34 2020] pvesr D 0 19038 1 0x00000080
[Tue Sep 15 14:45:34 2020] Call Trace:
[Tue Sep 15 14:45:34 2020] __schedule+0x2e6/0x6f0
[Tue Sep 15 14:45:34 2020] ? filename_parentat.isra.57.part.58+0xf7/0x180
[Tue Sep 15 14:45:34 2020] schedule+0x33/0xa0
[Tue Sep 15 14:45:34 2020] rwsem_down_write_slowpath+0x2ed/0x4a0
[Tue Sep 15 14:45:34 2020] down_write+0x3d/0x40
[Tue Sep 15 14:45:34 2020] filename_create+0x8e/0x180
[Tue Sep 15 14:45:34 2020] do_mkdirat+0x59/0x110
[Tue Sep 15 14:45:34 2020] __x64_sys_mkdir+0x1b/0x20
[Tue Sep 15 14:45:34 2020] do_syscall_64+0x57/0x190
[Tue Sep 15 14:45:34 2020] entry_SYSCALL_64_after_hwframe+0x44/0xa9
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 15:00:03
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/15/20 2:49 PM, Alexandre DERUMIER wrote:
> Hi,
>
> I have produce it again,
>
> now I can't write to /etc/pve/ from any node
>
OK, so seems to really be an issue in pmxcfs or between corosync and pmxcfs,
not the HA LRM or watchdog mux itself.
Can you try to give pmxcfs real time scheduling, e.g., by doing:
# systemctl edit pve-cluster
And then add snippet:
[Service]
CPUSchedulingPolicy=rr
CPUSchedulingPriority=99
And restart pve-cluster
> I have also added some debug logs to pve-ha-lrm, and it was stuck in:
> (but if /etc/pve is locked, this is normal)
>
> if ($fence_request) {
> $haenv->log('err', "node need to be fenced - releasing agent_lock\n");
> $self->set_local_status({ state => 'lost_agent_lock'});
> } elsif (!$self->get_protected_ha_agent_lock()) {
> $self->set_local_status({ state => 'lost_agent_lock'});
> } elsif ($self->{mode} eq 'maintenance') {
> $self->set_local_status({ state => 'maintenance'});
> }
>
>
> corosync quorum is currently ok
>
> I'm currently digging the logs
Is your most simplest/stable reproducer still a periodic restart of corosync in one node?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 14:09 ` Alexandre DERUMIER
@ 2020-09-15 14:19 ` Alexandre DERUMIER
2020-09-15 14:32 ` Thomas Lamprecht
1 sibling, 0 replies; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-15 14:19 UTC (permalink / raw)
To: Thomas Lamprecht; +Cc: Proxmox VE development discussion
about node1: /etc/pve directory seem to be in bad state, that's why it can't be mount
ls -lah /etc/pve:
?????????? ? ? ? ? ? pve
I have forced an lazy umount
umount -l /etc/pve
and now it's working fine.
(so maybe when pmxcfs was killed, it don't have cleanly umount /etc/pve ? )
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 16:09:50
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>
>>Can you try to give pmxcfs real time scheduling, e.g., by doing:
>>
>># systemctl edit pve-cluster
>>
>>And then add snippet:
>>
>>
>>[Service]
>>CPUSchedulingPolicy=rr
>>CPUSchedulingPriority=99
yes, sure, I'll do it now
> I'm currently digging the logs
>>Is your most simplest/stable reproducer still a periodic restart of corosync in one node?
yes, a simple "systemctl restart corosync" on 1 node each minute
After 1hour, it's still locked.
on other nodes, I still have pmxfs logs like:
Sep 15 15:36:31 m6kvm2 pmxcfs[3474]: [status] notice: received log
Sep 15 15:46:21 m6kvm2 pmxcfs[3474]: [status] notice: received log
Sep 15 15:46:23 m6kvm2 pmxcfs[3474]: [status] notice: received log
...
on node1, I just restarted the pve-cluster service with systemctl restart pve-cluster,
the pmxcfs process was killed, but not able to start it again
and after that the /etc/pve become writable again on others node.
(I don't have rebooted yet node1, if you want more test on pmxcfs)
root@m6kvm1:~# systemctl status pve-cluster
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2020-09-15 15:52:11 CEST; 3min 29s ago
Process: 12536 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)
Sep 15 15:52:11 m6kvm1 systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Sep 15 15:52:11 m6kvm1 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Sep 15 15:52:11 m6kvm1 systemd[1]: Stopped The Proxmox VE cluster filesystem.
Sep 15 15:52:11 m6kvm1 systemd[1]: pve-cluster.service: Start request repeated too quickly.
Sep 15 15:52:11 m6kvm1 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 15 15:52:11 m6kvm1 systemd[1]: Failed to start The Proxmox VE cluster filesystem.
manual "pmxcfs -d"
https://gist.github.com/aderumier/4cd91d17e1f8847b93ea5f621f257c2e
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_initialize failed: 2
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [quorum] crit: can't initialize service
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_initialize failed: 2
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [confdb] crit: can't initialize service
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [dcdb] notice: start cluster connection
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_initialize failed: 2
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [dcdb] crit: can't initialize service
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [status] notice: start cluster connection
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [status] crit: cpg_initialize failed: 2
Sep 15 14:38:24 m6kvm1 pmxcfs[3491]: [status] crit: can't initialize service
Sep 15 14:38:30 m6kvm1 pmxcfs[3491]: [status] notice: update cluster info (cluster name m6kvm, version = 20)
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: node has quorum
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: starting data syncronisation
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: starting data syncronisation
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: received sync request (epoch 1/3491/00000064)
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: received sync request (epoch 1/3491/00000063)
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: received all states
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: leader is 2/3474
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: synced members: 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: waiting for updates from leader
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: dfsm_deliver_queue: queue length 23
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: received all states
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: all data is up to date
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [status] notice: dfsm_deliver_queue: queue length 157
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: update complete - trying to commit (got 4 inode updates)
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: all data is up to date
Sep 15 14:38:32 m6kvm1 pmxcfs[3491]: [dcdb] notice: dfsm_deliver_sync_queue: queue length 31
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_dispatch failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [status] crit: cpg_dispatch failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [status] crit: cpg_leave failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_dispatch failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_leave failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_dispatch failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [status] notice: node lost quorum
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_initialize failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [quorum] crit: can't initialize service
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_initialize failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [confdb] crit: can't initialize service
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [dcdb] notice: start cluster connection
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_initialize failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [dcdb] crit: can't initialize service
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [status] notice: start cluster connection
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [status] crit: cpg_initialize failed: 2
Sep 15 14:39:25 m6kvm1 pmxcfs[3491]: [status] crit: can't initialize service
Sep 15 14:39:31 m6kvm1 pmxcfs[3491]: [status] notice: update cluster info (cluster name m6kvm, version = 20)
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [status] notice: node has quorum
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: starting data syncronisation
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [status] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [status] notice: starting data syncronisation
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: received sync request (epoch 1/3491/00000065)
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [status] notice: received sync request (epoch 1/3491/00000064)
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: received all states
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: leader is 2/3474
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: synced members: 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: waiting for updates from leader
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: dfsm_deliver_queue: queue length 20
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [status] notice: received all states
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [status] notice: all data is up to date
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: update complete - trying to commit (got 9 inode updates)
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: all data is up to date
Sep 15 14:39:33 m6kvm1 pmxcfs[3491]: [dcdb] notice: dfsm_deliver_sync_queue: queue length 25
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_dispatch failed: 2
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [status] crit: cpg_dispatch failed: 2
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [status] crit: cpg_leave failed: 2
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_dispatch failed: 2
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_leave failed: 2
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_dispatch failed: 2
Sep 15 14:40:26 m6kvm1 pmxcfs[3491]: [status] notice: node lost quorum
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_initialize failed: 2
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [quorum] crit: can't initialize service
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_initialize failed: 2
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [confdb] crit: can't initialize service
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [dcdb] notice: start cluster connection
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_initialize failed: 2
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [dcdb] crit: can't initialize service
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [status] notice: start cluster connection
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [status] crit: cpg_initialize failed: 2
Sep 15 14:40:27 m6kvm1 pmxcfs[3491]: [status] crit: can't initialize service
Sep 15 14:40:33 m6kvm1 pmxcfs[3491]: [status] notice: update cluster info (cluster name m6kvm, version = 20)
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: node has quorum
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: starting data syncronisation
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: starting data syncronisation
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: received sync request (epoch 1/3491/00000066)
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: received sync request (epoch 1/3491/00000065)
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: received all states
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: leader is 2/3474
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: synced members: 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: waiting for updates from leader
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: dfsm_deliver_queue: queue length 23
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: received all states
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: all data is up to date
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [status] notice: dfsm_deliver_queue: queue length 87
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: update complete - trying to commit (got 6 inode updates)
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: all data is up to date
Sep 15 14:40:34 m6kvm1 pmxcfs[3491]: [dcdb] notice: dfsm_deliver_sync_queue: queue length 33
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_dispatch failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [status] crit: cpg_dispatch failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [status] crit: cpg_leave failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_dispatch failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_leave failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_dispatch failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [status] notice: node lost quorum
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [quorum] crit: quorum_initialize failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [quorum] crit: can't initialize service
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [confdb] crit: cmap_initialize failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [confdb] crit: can't initialize service
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [dcdb] notice: start cluster connection
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [dcdb] crit: cpg_initialize failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [dcdb] crit: can't initialize service
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [status] notice: start cluster connection
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [status] crit: cpg_initialize failed: 2
Sep 15 14:41:28 m6kvm1 pmxcfs[3491]: [status] crit: can't initialize service
Sep 15 14:41:34 m6kvm1 pmxcfs[3491]: [status] notice: update cluster info (cluster name m6kvm, version = 20)
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [status] notice: node has quorum
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [dcdb] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [dcdb] notice: starting data syncronisation
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [status] notice: members: 1/3491, 2/3474, 3/3566, 4/3805, 5/3835, 6/3862, 7/3797, 8/3808, 9/9541, 10/3787, 11/3799, 12/3795, 13/3776, 14/3778
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [status] notice: starting data syncronisation
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [dcdb] notice: received sync request (epoch 1/3491/00000067)
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [status] notice: received sync request (epoch 1/3491/00000066)
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [status] notice: received all states
Sep 15 14:41:35 m6kvm1 pmxcfs[3491]: [status] notice: all data is up to date
Sep 15 14:47:54 m6kvm1 pmxcfs[3491]: [status] notice: received log
Sep 15 15:02:55 m6kvm1 pmxcfs[3491]: [status] notice: received log
Sep 15 15:17:56 m6kvm1 pmxcfs[3491]: [status] notice: received log
Sep 15 15:32:57 m6kvm1 pmxcfs[3491]: [status] notice: received log
Sep 15 15:47:58 m6kvm1 pmxcfs[3491]: [status] notice: received log
----> restart
2352 [ 15/09/2020 15:52:00 ] systemctl restart pve-cluster
Sep 15 15:52:10 m6kvm1 pmxcfs[12438]: [main] crit: fuse_mount error: Transport endpoint is not connected
Sep 15 15:52:10 m6kvm1 pmxcfs[12438]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 15 15:52:10 m6kvm1 pmxcfs[12529]: [main] crit: fuse_mount error: Transport endpoint is not connected
Sep 15 15:52:10 m6kvm1 pmxcfs[12529]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 15 15:52:10 m6kvm1 pmxcfs[12531]: [main] crit: fuse_mount error: Transport endpoint is not connected
Sep 15 15:52:10 m6kvm1 pmxcfs[12531]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 15 15:52:11 m6kvm1 pmxcfs[12533]: [main] crit: fuse_mount error: Transport endpoint is not connected
Sep 15 15:52:11 m6kvm1 pmxcfs[12533]: [main] notice: exit proxmox configuration filesystem (-1)
Sep 15 15:52:11 m6kvm1 pmxcfs[12536]: [main] crit: fuse_mount error: Transport endpoint is not connected
Sep 15 15:52:11 m6kvm1 pmxcfs[12536]: [main] notice: exit proxmox configuration filesystem (-1)
some interesting dmesg about "pvesr"
[Tue Sep 15 14:45:34 2020] INFO: task pvesr:19038 blocked for more than 120 seconds.
[Tue Sep 15 14:45:34 2020] Tainted: P O 5.4.60-1-pve #1
[Tue Sep 15 14:45:34 2020] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Tue Sep 15 14:45:34 2020] pvesr D 0 19038 1 0x00000080
[Tue Sep 15 14:45:34 2020] Call Trace:
[Tue Sep 15 14:45:34 2020] __schedule+0x2e6/0x6f0
[Tue Sep 15 14:45:34 2020] ? filename_parentat.isra.57.part.58+0xf7/0x180
[Tue Sep 15 14:45:34 2020] schedule+0x33/0xa0
[Tue Sep 15 14:45:34 2020] rwsem_down_write_slowpath+0x2ed/0x4a0
[Tue Sep 15 14:45:34 2020] down_write+0x3d/0x40
[Tue Sep 15 14:45:34 2020] filename_create+0x8e/0x180
[Tue Sep 15 14:45:34 2020] do_mkdirat+0x59/0x110
[Tue Sep 15 14:45:34 2020] __x64_sys_mkdir+0x1b/0x20
[Tue Sep 15 14:45:34 2020] do_syscall_64+0x57/0x190
[Tue Sep 15 14:45:34 2020] entry_SYSCALL_64_after_hwframe+0x44/0xa9
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 15:00:03
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/15/20 2:49 PM, Alexandre DERUMIER wrote:
> Hi,
>
> I have produce it again,
>
> now I can't write to /etc/pve/ from any node
>
OK, so seems to really be an issue in pmxcfs or between corosync and pmxcfs,
not the HA LRM or watchdog mux itself.
Can you try to give pmxcfs real time scheduling, e.g., by doing:
# systemctl edit pve-cluster
And then add snippet:
[Service]
CPUSchedulingPolicy=rr
CPUSchedulingPriority=99
And restart pve-cluster
> I have also added some debug logs to pve-ha-lrm, and it was stuck in:
> (but if /etc/pve is locked, this is normal)
>
> if ($fence_request) {
> $haenv->log('err', "node need to be fenced - releasing agent_lock\n");
> $self->set_local_status({ state => 'lost_agent_lock'});
> } elsif (!$self->get_protected_ha_agent_lock()) {
> $self->set_local_status({ state => 'lost_agent_lock'});
> } elsif ($self->{mode} eq 'maintenance') {
> $self->set_local_status({ state => 'maintenance'});
> }
>
>
> corosync quorum is currently ok
>
> I'm currently digging the logs
Is your most simplest/stable reproducer still a periodic restart of corosync in one node?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 14:09 ` Alexandre DERUMIER
2020-09-15 14:19 ` Alexandre DERUMIER
@ 2020-09-15 14:32 ` Thomas Lamprecht
2020-09-15 14:57 ` Alexandre DERUMIER
1 sibling, 1 reply; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-15 14:32 UTC (permalink / raw)
To: Alexandre DERUMIER; +Cc: Proxmox VE development discussion
On 9/15/20 4:09 PM, Alexandre DERUMIER wrote:
>>> Can you try to give pmxcfs real time scheduling, e.g., by doing:
>>>
>>> # systemctl edit pve-cluster
>>>
>>> And then add snippet:
>>>
>>>
>>> [Service]
>>> CPUSchedulingPolicy=rr
>>> CPUSchedulingPriority=99
> yes, sure, I'll do it now
>
>
>> I'm currently digging the logs
>>> Is your most simplest/stable reproducer still a periodic restart of corosync in one node?
> yes, a simple "systemctl restart corosync" on 1 node each minute
>
>
>
> After 1hour, it's still locked.
>
> on other nodes, I still have pmxfs logs like:
>
I mean this is bad, but also great!
Cam you do a coredump of the whole thing and upload it somewhere with the version info
used (for dbgsym package)? That could help a lot.
> manual "pmxcfs -d"
> https://gist.github.com/aderumier/4cd91d17e1f8847b93ea5f621f257c2e
>
Hmm, the fuse connection of the previous one got into a weird state (or something is still
running) but I'd rather say this is a side-effect not directly connected to the real bug.
>
> some interesting dmesg about "pvesr"
>
> [Tue Sep 15 14:45:34 2020] INFO: task pvesr:19038 blocked for more than 120 seconds.
> [Tue Sep 15 14:45:34 2020] Tainted: P O 5.4.60-1-pve #1
> [Tue Sep 15 14:45:34 2020] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Tue Sep 15 14:45:34 2020] pvesr D 0 19038 1 0x00000080
> [Tue Sep 15 14:45:34 2020] Call Trace:
> [Tue Sep 15 14:45:34 2020] __schedule+0x2e6/0x6f0
> [Tue Sep 15 14:45:34 2020] ? filename_parentat.isra.57.part.58+0xf7/0x180
> [Tue Sep 15 14:45:34 2020] schedule+0x33/0xa0
> [Tue Sep 15 14:45:34 2020] rwsem_down_write_slowpath+0x2ed/0x4a0
> [Tue Sep 15 14:45:34 2020] down_write+0x3d/0x40
> [Tue Sep 15 14:45:34 2020] filename_create+0x8e/0x180
> [Tue Sep 15 14:45:34 2020] do_mkdirat+0x59/0x110
> [Tue Sep 15 14:45:34 2020] __x64_sys_mkdir+0x1b/0x20
> [Tue Sep 15 14:45:34 2020] do_syscall_64+0x57/0x190
> [Tue Sep 15 14:45:34 2020] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
hmm, hangs in mkdir (cluster wide locking)
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 14:32 ` Thomas Lamprecht
@ 2020-09-15 14:57 ` Alexandre DERUMIER
2020-09-15 15:58 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-15 14:57 UTC (permalink / raw)
To: Thomas Lamprecht; +Cc: Proxmox VE development discussion
>>I mean this is bad, but also great!
>>Cam you do a coredump of the whole thing and upload it somewhere with the version info
>>used (for dbgsym package)? That could help a lot.
I'll try to reproduce it again (with the full lock everywhere), and do the coredump.
I have tried the real time scheduling,
but I still have been able to reproduce the "lrm too long" for 60s (but as I'm restarting corosync each minute, I think it's unlocking
something at next corosync restart.)
this time it was blocked at the same time on a node in:
work {
...
} elsif ($state eq 'active') {
....
$self->update_lrm_status();
and another node in
if ($fence_request) {
$haenv->log('err', "node need to be fenced - releasing agent_lock\n");
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif (!$self->get_protected_ha_agent_lock()) {
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif ($self->{mode} eq 'maintenance') {
$self->set_local_status({ state => 'maintenance'});
}
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 16:32:52
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/15/20 4:09 PM, Alexandre DERUMIER wrote:
>>> Can you try to give pmxcfs real time scheduling, e.g., by doing:
>>>
>>> # systemctl edit pve-cluster
>>>
>>> And then add snippet:
>>>
>>>
>>> [Service]
>>> CPUSchedulingPolicy=rr
>>> CPUSchedulingPriority=99
> yes, sure, I'll do it now
>
>
>> I'm currently digging the logs
>>> Is your most simplest/stable reproducer still a periodic restart of corosync in one node?
> yes, a simple "systemctl restart corosync" on 1 node each minute
>
>
>
> After 1hour, it's still locked.
>
> on other nodes, I still have pmxfs logs like:
>
I mean this is bad, but also great!
Cam you do a coredump of the whole thing and upload it somewhere with the version info
used (for dbgsym package)? That could help a lot.
> manual "pmxcfs -d"
> https://gist.github.com/aderumier/4cd91d17e1f8847b93ea5f621f257c2e
>
Hmm, the fuse connection of the previous one got into a weird state (or something is still
running) but I'd rather say this is a side-effect not directly connected to the real bug.
>
> some interesting dmesg about "pvesr"
>
> [Tue Sep 15 14:45:34 2020] INFO: task pvesr:19038 blocked for more than 120 seconds.
> [Tue Sep 15 14:45:34 2020] Tainted: P O 5.4.60-1-pve #1
> [Tue Sep 15 14:45:34 2020] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Tue Sep 15 14:45:34 2020] pvesr D 0 19038 1 0x00000080
> [Tue Sep 15 14:45:34 2020] Call Trace:
> [Tue Sep 15 14:45:34 2020] __schedule+0x2e6/0x6f0
> [Tue Sep 15 14:45:34 2020] ? filename_parentat.isra.57.part.58+0xf7/0x180
> [Tue Sep 15 14:45:34 2020] schedule+0x33/0xa0
> [Tue Sep 15 14:45:34 2020] rwsem_down_write_slowpath+0x2ed/0x4a0
> [Tue Sep 15 14:45:34 2020] down_write+0x3d/0x40
> [Tue Sep 15 14:45:34 2020] filename_create+0x8e/0x180
> [Tue Sep 15 14:45:34 2020] do_mkdirat+0x59/0x110
> [Tue Sep 15 14:45:34 2020] __x64_sys_mkdir+0x1b/0x20
> [Tue Sep 15 14:45:34 2020] do_syscall_64+0x57/0x190
> [Tue Sep 15 14:45:34 2020] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
hmm, hangs in mkdir (cluster wide locking)
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 14:57 ` Alexandre DERUMIER
@ 2020-09-15 15:58 ` Alexandre DERUMIER
2020-09-16 7:34 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-15 15:58 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: Thomas Lamprecht
Another small lock at 17:41:09
To be sure, I have done a small loop of write each second in /etc/pve, node node2.
it's hanging at first corosync restart, then, on second corosync restart it's working again.
I'll try to improve this tomorrow to be able to debug corosync process
- restarting corosync
do some write in /etc/pve/
- and if it's hanging don't restart corosync again
node2: echo test > /etc/pve/test loop
--------------------------------------
Current time : 17:41:01
Current time : 17:41:02
Current time : 17:41:03
Current time : 17:41:04
Current time : 17:41:05
Current time : 17:41:06
Current time : 17:41:07
Current time : 17:41:08
Current time : 17:41:09
hang
Current time : 17:42:05
Current time : 17:42:06
Current time : 17:42:07
node1
-----
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: PMTUD completed for host: 6 link: 0 current link mtu: 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: Starting PMTUD for host: 10 link: 0
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] udp: detected kernel MTU: 1500
Sep 15 17:41:08 m6kvm1 corosync[18145]: [TOTEM ] Knet pMTU change: 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: PMTUD link change for host: 10 link: 0 from 469 to 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: PMTUD completed for host: 10 link: 0 current link mtu: 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: Global data MTU changed to: 1397
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-31-zx6KJM/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] lib_init_fn: conn=0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] got quorum_type request on 0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] got trackstart request on 0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] sending initial status to 0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] sending quorum notification to 0x556c2918d5f0, length = 52
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-32-I7ZZ6e/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CMAP ] lib_init_fn: conn=0x556c2918ef20
Sep 15 17:41:09 m6kvm1 pmxcfs[16239]: [status] notice: update cluster info (cluster name m6kvm, version = 20)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-33-6RKbvH/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] lib_init_fn: conn=0x556c2918ad00, cpd=0x556c2918b50c
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-34-GAY5T9/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] lib_init_fn: conn=0x556c2918c740, cpd=0x556c2918ce8c
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Creating commit token because I am the rep.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Saving state aru 5 high seq received 5
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Storing new sequence id for ring 1197
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] entering COMMIT state.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] entering RECOVERY state.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] TRANS [0] member 1:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [0] member 1:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (1.1193)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 5 high delivered 5 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [1] member 2:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [2] member 3:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [3] member 4:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
ep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [4] member 5:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [5] member 6:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [6] member 7:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [7] member 8:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [8] member 9:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [9] member 10:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [10] member 11:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [11] member 12:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [12] member 13:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [13] member 14:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Did not need to originate any messages in recovery.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Sending initial ORF token
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Resetting old ring state
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] recovery to regular 1-0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] waiting_trans_ack changed to 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.90)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.91)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.92)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.93)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.94)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.95)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.96)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.97)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.107)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.108)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.109)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.110)
ep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.111)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] call init for locally known services
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] entering OPERATIONAL state.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] A new membership (1.1197) was formed. Members joined: 2 3 4 5 6 7 8 9 10 11 12 13 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] enter sync process
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] Committing synchronization for corosync configuration map access
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 2
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 3
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 4
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 5
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 6
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 7
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 8
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 9
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 10
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 11
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 12
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 13
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] Committing synchronization for corosync cluster closed process group service v1.01
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] my downlist: members(old:1 left:0)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[0] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[1] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[2] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[3] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[4] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[5] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[6] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[7] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[8] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[9] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[10] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[11] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[12] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[13] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[14] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[15] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[16] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[17] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[18] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[19] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[20] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[21] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[22] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[23] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[24] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[25] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] Sending nodelist callback. ring_id = 1.1197
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[13]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=2, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[14]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=3, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[1]: votes: 1, expected: 14 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=3, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[2]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=4, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 3
....
....
next corosync restart
Sep 15 17:42:03 m6kvm1 corosync[18145]: [MAIN ] Node was shut down by a signal
Sep 15 17:42:03 m6kvm1 corosync[18145]: [SERV ] Unloading all Corosync service engines.
Sep 15 17:42:03 m6kvm1 corosync[18145]: [QB ] withdrawing server sockets
Sep 15 17:42:03 m6kvm1 corosync[18145]: [QB ] qb_ipcs_unref() - destroying
Sep 15 17:42:03 m6kvm1 corosync[18145]: [SERV ] Service engine unloaded: corosync vote quorum service v1.0
Sep 15 17:42:03 m6kvm1 corosync[18145]: [QB ] qb_ipcs_disconnect(/dev/shm/qb-18145-16239-32-I7ZZ6e/qb) state:2
Sep 15 17:42:03 m6kvm1 pmxcfs[16239]: [confdb] crit: cmap_dispatch failed: 2
Sep 15 17:42:03 m6kvm1 corosync[18145]: [MAIN ] cs_ipcs_connection_closed()
Sep 15 17:42:03 m6kvm1 corosync[18145]: [CMAP ] exit_fn for conn=0x556c2918ef20
Sep 15 17:42:03 m6kvm1 corosync[18145]: [MAIN ] cs_ipcs_connection_destroyed()
node2
-----
Sep 15 17:41:05 m6kvm2 corosync[25411]: [KNET ] pmtud: Starting PMTUD for host: 10 link: 0
Sep 15 17:41:05 m6kvm2 corosync[25411]: [KNET ] udp: detected kernel MTU: 1500
Sep 15 17:41:05 m6kvm2 corosync[25411]: [KNET ] pmtud: PMTUD completed for host: 10 link: 0 current link mtu: 1397
Sep 15 17:41:07 m6kvm2 corosync[25411]: [KNET ] rx: host: 1 link: 0 received pong: 2
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [TOTEM ] entering GATHER state from 11(merge during join).
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: host: 1 link: 0 is up
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Knet host change callback. nodeid: 1 reachable: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] pmtud: Starting PMTUD for host: 1 link: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] udp: detected kernel MTU: 1500
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] pmtud: PMTUD completed for host: 1 link: 0 current link mtu: 1397
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Saving state aru 123 high seq received 123
Sep 15 17:41:09 m6kvm2 corosync[25411]: [MAIN ] Storing new sequence id for ring 1197
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] entering COMMIT state.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] entering RECOVERY state.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [0] member 2:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [1] member 3:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [2] member 4:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [3] member 5:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [4] member 6:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [5] member 7:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [6] member 8:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [7] member 9:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [8] member 10:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [9] member 11:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [10] member 12:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [11] member 13:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [12] member 14:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [0] member 1:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (1.1193)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 5 high delivered 5 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [1] member 2:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [2] member 3:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [3] member 4:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [4] member 5:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [5] member 6:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [6] member 7:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [7] member 8:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [8] member 9:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [9] member 10:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [10] member 11:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [11] member 12:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [12] member 13:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [13] member 14:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Did not need to originate any messages in recovery.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru ffffffff
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Resetting old ring state
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] recovery to regular 1-0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] waiting_trans_ack changed to 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [MAIN ] Member joined: r(0) ip(10.3.94.89)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] call init for locally known services
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] entering OPERATIONAL state.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] A new membership (1.1197) was formed. Members joined: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] enter sync process
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] Committing synchronization for corosync configuration map access
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CMAP ] Not first sync -> no action
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 3
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 4
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 5
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 6
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 7
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 8
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 9
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 10
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 11
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 12
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 13
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] Committing synchronization for corosync cluster closed process group service v1.01
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] my downlist: members(old:13 left:0)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[0] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[1] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[2] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[3] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[4] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[5] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[6] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[7] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[8] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[9] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[10] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[11] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[12] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[13] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[14] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[15] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[16] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[17] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[18] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[19] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[20] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[21] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[22] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[23] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[24] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[25] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] Sending nodelist callback. ring_id = 1.1197
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[13]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[14]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[1]: votes: 1, expected: 14 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] total_votes=14, expected_votes=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 3 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 4 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 5 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 6 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 7 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 8 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 9 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 10 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 11 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 12 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] lowest node id: 1 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] highest node id: 14 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[2]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] total_votes=14, expected_votes=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 3 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 4 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 5 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 6 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 7 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 8 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 9 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 10 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 11 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 12 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] lowest node id: 1 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] highest node id: 14 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 3
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[3]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 3
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 4
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[4]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 4
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 5
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[5]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 5
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 6
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[6]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 6
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 7
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[7]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 7
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 8
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[8]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 8
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 9
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[9]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 9
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 10
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[10]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 10
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 11
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[11]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 11
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 12
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[12]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 12
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] Committing synchronization for corosync vote quorum service v1.0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] total_votes=14, expected_votes=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 3 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 4 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 5 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 6 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 7 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 8 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 9 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 10 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 11 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 12 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] lowest node id: 1 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] highest node id: 14 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [QUORUM] Members[14]: 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [QUORUM] sending quorum notification to (nil), length = 104
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] Sending quorum callback, quorate = 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] waiting_trans_ack changed to 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got procjoin message from cluster node 1 (r(0) ip(10.3.94.89) ) for pid 16239
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got procjoin message from cluster node 1 (r(0) ip(10.3.94.89) ) for pid 16239
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 16:57:46
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>I mean this is bad, but also great!
>>Cam you do a coredump of the whole thing and upload it somewhere with the version info
>>used (for dbgsym package)? That could help a lot.
I'll try to reproduce it again (with the full lock everywhere), and do the coredump.
I have tried the real time scheduling,
but I still have been able to reproduce the "lrm too long" for 60s (but as I'm restarting corosync each minute, I think it's unlocking
something at next corosync restart.)
this time it was blocked at the same time on a node in:
work {
...
} elsif ($state eq 'active') {
....
$self->update_lrm_status();
and another node in
if ($fence_request) {
$haenv->log('err', "node need to be fenced - releasing agent_lock\n");
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif (!$self->get_protected_ha_agent_lock()) {
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif ($self->{mode} eq 'maintenance') {
$self->set_local_status({ state => 'maintenance'});
}
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 16:32:52
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/15/20 4:09 PM, Alexandre DERUMIER wrote:
>>> Can you try to give pmxcfs real time scheduling, e.g., by doing:
>>>
>>> # systemctl edit pve-cluster
>>>
>>> And then add snippet:
>>>
>>>
>>> [Service]
>>> CPUSchedulingPolicy=rr
>>> CPUSchedulingPriority=99
> yes, sure, I'll do it now
>
>
>> I'm currently digging the logs
>>> Is your most simplest/stable reproducer still a periodic restart of corosync in one node?
> yes, a simple "systemctl restart corosync" on 1 node each minute
>
>
>
> After 1hour, it's still locked.
>
> on other nodes, I still have pmxfs logs like:
>
I mean this is bad, but also great!
Cam you do a coredump of the whole thing and upload it somewhere with the version info
used (for dbgsym package)? That could help a lot.
> manual "pmxcfs -d"
> https://gist.github.com/aderumier/4cd91d17e1f8847b93ea5f621f257c2e
>
Hmm, the fuse connection of the previous one got into a weird state (or something is still
running) but I'd rather say this is a side-effect not directly connected to the real bug.
>
> some interesting dmesg about "pvesr"
>
> [Tue Sep 15 14:45:34 2020] INFO: task pvesr:19038 blocked for more than 120 seconds.
> [Tue Sep 15 14:45:34 2020] Tainted: P O 5.4.60-1-pve #1
> [Tue Sep 15 14:45:34 2020] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Tue Sep 15 14:45:34 2020] pvesr D 0 19038 1 0x00000080
> [Tue Sep 15 14:45:34 2020] Call Trace:
> [Tue Sep 15 14:45:34 2020] __schedule+0x2e6/0x6f0
> [Tue Sep 15 14:45:34 2020] ? filename_parentat.isra.57.part.58+0xf7/0x180
> [Tue Sep 15 14:45:34 2020] schedule+0x33/0xa0
> [Tue Sep 15 14:45:34 2020] rwsem_down_write_slowpath+0x2ed/0x4a0
> [Tue Sep 15 14:45:34 2020] down_write+0x3d/0x40
> [Tue Sep 15 14:45:34 2020] filename_create+0x8e/0x180
> [Tue Sep 15 14:45:34 2020] do_mkdirat+0x59/0x110
> [Tue Sep 15 14:45:34 2020] __x64_sys_mkdir+0x1b/0x20
> [Tue Sep 15 14:45:34 2020] do_syscall_64+0x57/0x190
> [Tue Sep 15 14:45:34 2020] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
hmm, hangs in mkdir (cluster wide locking)
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-15 15:58 ` Alexandre DERUMIER
@ 2020-09-16 7:34 ` Alexandre DERUMIER
2020-09-16 7:58 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-16 7:34 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: Thomas Lamprecht
Hi,
I have produced the problem now,
and this time I don't have restarted corosync a second after the lock of /etc/pve
so, currently it's readonly.
I don't have used gdb since a long time,
could you tell me how to attach to the running pmxcfs process, and some gbd commands to launch ?
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoyé: Mardi 15 Septembre 2020 17:58:33
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Another small lock at 17:41:09
To be sure, I have done a small loop of write each second in /etc/pve, node node2.
it's hanging at first corosync restart, then, on second corosync restart it's working again.
I'll try to improve this tomorrow to be able to debug corosync process
- restarting corosync
do some write in /etc/pve/
- and if it's hanging don't restart corosync again
node2: echo test > /etc/pve/test loop
--------------------------------------
Current time : 17:41:01
Current time : 17:41:02
Current time : 17:41:03
Current time : 17:41:04
Current time : 17:41:05
Current time : 17:41:06
Current time : 17:41:07
Current time : 17:41:08
Current time : 17:41:09
hang
Current time : 17:42:05
Current time : 17:42:06
Current time : 17:42:07
node1
-----
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: PMTUD completed for host: 6 link: 0 current link mtu: 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: Starting PMTUD for host: 10 link: 0
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] udp: detected kernel MTU: 1500
Sep 15 17:41:08 m6kvm1 corosync[18145]: [TOTEM ] Knet pMTU change: 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: PMTUD link change for host: 10 link: 0 from 469 to 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: PMTUD completed for host: 10 link: 0 current link mtu: 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: Global data MTU changed to: 1397
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-31-zx6KJM/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] lib_init_fn: conn=0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] got quorum_type request on 0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] got trackstart request on 0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] sending initial status to 0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] sending quorum notification to 0x556c2918d5f0, length = 52
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-32-I7ZZ6e/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CMAP ] lib_init_fn: conn=0x556c2918ef20
Sep 15 17:41:09 m6kvm1 pmxcfs[16239]: [status] notice: update cluster info (cluster name m6kvm, version = 20)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-33-6RKbvH/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] lib_init_fn: conn=0x556c2918ad00, cpd=0x556c2918b50c
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-34-GAY5T9/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] lib_init_fn: conn=0x556c2918c740, cpd=0x556c2918ce8c
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Creating commit token because I am the rep.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Saving state aru 5 high seq received 5
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Storing new sequence id for ring 1197
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] entering COMMIT state.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] entering RECOVERY state.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] TRANS [0] member 1:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [0] member 1:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (1.1193)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 5 high delivered 5 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [1] member 2:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [2] member 3:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [3] member 4:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
ep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [4] member 5:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [5] member 6:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [6] member 7:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [7] member 8:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [8] member 9:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [9] member 10:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [10] member 11:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [11] member 12:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [12] member 13:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [13] member 14:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Did not need to originate any messages in recovery.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Sending initial ORF token
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Resetting old ring state
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] recovery to regular 1-0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] waiting_trans_ack changed to 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.90)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.91)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.92)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.93)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.94)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.95)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.96)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.97)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.107)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.108)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.109)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.110)
ep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.111)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] call init for locally known services
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] entering OPERATIONAL state.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] A new membership (1.1197) was formed. Members joined: 2 3 4 5 6 7 8 9 10 11 12 13 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] enter sync process
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] Committing synchronization for corosync configuration map access
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 2
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 3
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 4
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 5
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 6
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 7
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 8
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 9
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 10
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 11
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 12
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 13
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] Committing synchronization for corosync cluster closed process group service v1.01
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] my downlist: members(old:1 left:0)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[0] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[1] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[2] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[3] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[4] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[5] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[6] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[7] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[8] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[9] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[10] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[11] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[12] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[13] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[14] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[15] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[16] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[17] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[18] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[19] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[20] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[21] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[22] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[23] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[24] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[25] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] Sending nodelist callback. ring_id = 1.1197
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[13]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=2, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[14]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=3, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[1]: votes: 1, expected: 14 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=3, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[2]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=4, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 3
....
....
next corosync restart
Sep 15 17:42:03 m6kvm1 corosync[18145]: [MAIN ] Node was shut down by a signal
Sep 15 17:42:03 m6kvm1 corosync[18145]: [SERV ] Unloading all Corosync service engines.
Sep 15 17:42:03 m6kvm1 corosync[18145]: [QB ] withdrawing server sockets
Sep 15 17:42:03 m6kvm1 corosync[18145]: [QB ] qb_ipcs_unref() - destroying
Sep 15 17:42:03 m6kvm1 corosync[18145]: [SERV ] Service engine unloaded: corosync vote quorum service v1.0
Sep 15 17:42:03 m6kvm1 corosync[18145]: [QB ] qb_ipcs_disconnect(/dev/shm/qb-18145-16239-32-I7ZZ6e/qb) state:2
Sep 15 17:42:03 m6kvm1 pmxcfs[16239]: [confdb] crit: cmap_dispatch failed: 2
Sep 15 17:42:03 m6kvm1 corosync[18145]: [MAIN ] cs_ipcs_connection_closed()
Sep 15 17:42:03 m6kvm1 corosync[18145]: [CMAP ] exit_fn for conn=0x556c2918ef20
Sep 15 17:42:03 m6kvm1 corosync[18145]: [MAIN ] cs_ipcs_connection_destroyed()
node2
-----
Sep 15 17:41:05 m6kvm2 corosync[25411]: [KNET ] pmtud: Starting PMTUD for host: 10 link: 0
Sep 15 17:41:05 m6kvm2 corosync[25411]: [KNET ] udp: detected kernel MTU: 1500
Sep 15 17:41:05 m6kvm2 corosync[25411]: [KNET ] pmtud: PMTUD completed for host: 10 link: 0 current link mtu: 1397
Sep 15 17:41:07 m6kvm2 corosync[25411]: [KNET ] rx: host: 1 link: 0 received pong: 2
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [TOTEM ] entering GATHER state from 11(merge during join).
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: host: 1 link: 0 is up
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Knet host change callback. nodeid: 1 reachable: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] pmtud: Starting PMTUD for host: 1 link: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] udp: detected kernel MTU: 1500
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] pmtud: PMTUD completed for host: 1 link: 0 current link mtu: 1397
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Saving state aru 123 high seq received 123
Sep 15 17:41:09 m6kvm2 corosync[25411]: [MAIN ] Storing new sequence id for ring 1197
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] entering COMMIT state.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] entering RECOVERY state.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [0] member 2:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [1] member 3:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [2] member 4:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [3] member 5:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [4] member 6:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [5] member 7:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [6] member 8:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [7] member 9:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [8] member 10:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [9] member 11:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [10] member 12:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [11] member 13:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [12] member 14:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [0] member 1:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (1.1193)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 5 high delivered 5 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [1] member 2:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [2] member 3:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [3] member 4:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [4] member 5:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [5] member 6:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [6] member 7:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [7] member 8:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [8] member 9:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [9] member 10:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [10] member 11:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [11] member 12:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [12] member 13:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [13] member 14:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Did not need to originate any messages in recovery.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru ffffffff
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Resetting old ring state
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] recovery to regular 1-0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] waiting_trans_ack changed to 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [MAIN ] Member joined: r(0) ip(10.3.94.89)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] call init for locally known services
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] entering OPERATIONAL state.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] A new membership (1.1197) was formed. Members joined: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] enter sync process
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] Committing synchronization for corosync configuration map access
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CMAP ] Not first sync -> no action
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 3
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 4
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 5
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 6
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 7
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 8
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 9
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 10
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 11
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 12
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 13
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] Committing synchronization for corosync cluster closed process group service v1.01
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] my downlist: members(old:13 left:0)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[0] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[1] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[2] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[3] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[4] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[5] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[6] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[7] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[8] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[9] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[10] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[11] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[12] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[13] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[14] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[15] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[16] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[17] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[18] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[19] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[20] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[21] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[22] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[23] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[24] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[25] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] Sending nodelist callback. ring_id = 1.1197
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[13]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[14]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[1]: votes: 1, expected: 14 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] total_votes=14, expected_votes=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 3 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 4 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 5 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 6 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 7 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 8 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 9 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 10 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 11 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 12 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] lowest node id: 1 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] highest node id: 14 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[2]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] total_votes=14, expected_votes=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 3 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 4 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 5 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 6 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 7 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 8 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 9 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 10 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 11 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 12 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] lowest node id: 1 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] highest node id: 14 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 3
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[3]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 3
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 4
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[4]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 4
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 5
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[5]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 5
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 6
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[6]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 6
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 7
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[7]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 7
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 8
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[8]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 8
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 9
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[9]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 9
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 10
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[10]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 10
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 11
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[11]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 11
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 12
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[12]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 12
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] Committing synchronization for corosync vote quorum service v1.0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] total_votes=14, expected_votes=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 3 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 4 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 5 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 6 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 7 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 8 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 9 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 10 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 11 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 12 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] lowest node id: 1 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] highest node id: 14 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [QUORUM] Members[14]: 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [QUORUM] sending quorum notification to (nil), length = 104
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] Sending quorum callback, quorate = 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] waiting_trans_ack changed to 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got procjoin message from cluster node 1 (r(0) ip(10.3.94.89) ) for pid 16239
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got procjoin message from cluster node 1 (r(0) ip(10.3.94.89) ) for pid 16239
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 16:57:46
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>I mean this is bad, but also great!
>>Cam you do a coredump of the whole thing and upload it somewhere with the version info
>>used (for dbgsym package)? That could help a lot.
I'll try to reproduce it again (with the full lock everywhere), and do the coredump.
I have tried the real time scheduling,
but I still have been able to reproduce the "lrm too long" for 60s (but as I'm restarting corosync each minute, I think it's unlocking
something at next corosync restart.)
this time it was blocked at the same time on a node in:
work {
...
} elsif ($state eq 'active') {
....
$self->update_lrm_status();
and another node in
if ($fence_request) {
$haenv->log('err', "node need to be fenced - releasing agent_lock\n");
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif (!$self->get_protected_ha_agent_lock()) {
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif ($self->{mode} eq 'maintenance') {
$self->set_local_status({ state => 'maintenance'});
}
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 16:32:52
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/15/20 4:09 PM, Alexandre DERUMIER wrote:
>>> Can you try to give pmxcfs real time scheduling, e.g., by doing:
>>>
>>> # systemctl edit pve-cluster
>>>
>>> And then add snippet:
>>>
>>>
>>> [Service]
>>> CPUSchedulingPolicy=rr
>>> CPUSchedulingPriority=99
> yes, sure, I'll do it now
>
>
>> I'm currently digging the logs
>>> Is your most simplest/stable reproducer still a periodic restart of corosync in one node?
> yes, a simple "systemctl restart corosync" on 1 node each minute
>
>
>
> After 1hour, it's still locked.
>
> on other nodes, I still have pmxfs logs like:
>
I mean this is bad, but also great!
Cam you do a coredump of the whole thing and upload it somewhere with the version info
used (for dbgsym package)? That could help a lot.
> manual "pmxcfs -d"
> https://gist.github.com/aderumier/4cd91d17e1f8847b93ea5f621f257c2e
>
Hmm, the fuse connection of the previous one got into a weird state (or something is still
running) but I'd rather say this is a side-effect not directly connected to the real bug.
>
> some interesting dmesg about "pvesr"
>
> [Tue Sep 15 14:45:34 2020] INFO: task pvesr:19038 blocked for more than 120 seconds.
> [Tue Sep 15 14:45:34 2020] Tainted: P O 5.4.60-1-pve #1
> [Tue Sep 15 14:45:34 2020] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Tue Sep 15 14:45:34 2020] pvesr D 0 19038 1 0x00000080
> [Tue Sep 15 14:45:34 2020] Call Trace:
> [Tue Sep 15 14:45:34 2020] __schedule+0x2e6/0x6f0
> [Tue Sep 15 14:45:34 2020] ? filename_parentat.isra.57.part.58+0xf7/0x180
> [Tue Sep 15 14:45:34 2020] schedule+0x33/0xa0
> [Tue Sep 15 14:45:34 2020] rwsem_down_write_slowpath+0x2ed/0x4a0
> [Tue Sep 15 14:45:34 2020] down_write+0x3d/0x40
> [Tue Sep 15 14:45:34 2020] filename_create+0x8e/0x180
> [Tue Sep 15 14:45:34 2020] do_mkdirat+0x59/0x110
> [Tue Sep 15 14:45:34 2020] __x64_sys_mkdir+0x1b/0x20
> [Tue Sep 15 14:45:34 2020] do_syscall_64+0x57/0x190
> [Tue Sep 15 14:45:34 2020] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
hmm, hangs in mkdir (cluster wide locking)
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-16 7:34 ` Alexandre DERUMIER
@ 2020-09-16 7:58 ` Alexandre DERUMIER
2020-09-16 8:30 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-16 7:58 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: Thomas Lamprecht
here a backtrace with pve-cluster-dbgsym installed
(gdb) bt full
#0 0x00007fce67721896 in do_futex_wait.constprop () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x00007fce67721988 in __new_sem_wait_slow.constprop.0 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x00007fce678c5f98 in fuse_session_loop_mt () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#3 0x00007fce678cb577 in fuse_loop_mt () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#4 0x00005617f6d5ab0e in main (argc=<optimized out>, argv=<optimized out>) at pmxcfs.c:1055
ret = -1
lockfd = <optimized out>
pipefd = {8, 9}
foreground = 0
force_local_mode = 0
wrote_pidfile = 1
memdb = 0x5617f7c563b0
dcdb = 0x5617f8046ca0
status_fsm = 0x5617f806a630
context = <optimized out>
entries = {{long_name = 0x5617f6d7440f "debug", short_name = 100 'd', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x5617f6d82104 <cfs+20>, description = 0x5617f6d742cb "Turn on debug messages", arg_description = 0x0}, {
long_name = 0x5617f6d742e2 "foreground", short_name = 102 'f', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x7ffd672edc38, description = 0x5617f6d742ed "Do not daemonize server", arg_description = 0x0}, {
long_name = 0x5617f6d74305 "local", short_name = 108 'l', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x7ffd672edc3c, description = 0x5617f6d746a0 "Force local mode (ignore corosync.conf, force quorum)",
arg_description = 0x0}, {long_name = 0x0, short_name = 0 '\000', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x0, description = 0x0, arg_description = 0x0}}
err = 0x0
__func__ = "main"
utsname = {sysname = "Linux", '\000' <repeats 59 times>, nodename = "m6kvm1", '\000' <repeats 58 times>, release = "5.4.60-1-pve", '\000' <repeats 52 times>,
version = "#1 SMP PVE 5.4.60-1 (Mon, 31 Aug 2020 10:36:22 +0200)", '\000' <repeats 11 times>, machine = "x86_64", '\000' <repeats 58 times>, __domainname = "(none)", '\000' <repeats 58 times>}
dot = <optimized out>
www_data = <optimized out>
create = <optimized out>
conf_data = 0x5617f80466a0
len = <optimized out>
config = <optimized out>
bplug = <optimized out>
fa = {0x5617f6d7441e "-f", 0x5617f6d74421 "-odefault_permissions", 0x5617f6d74437 "-oallow_other", 0x0}
fuse_args = {argc = 1, argv = 0x5617f8046960, allocated = 1}
fuse_chan = 0x5617f7c560c0
corosync_loop = 0x5617f80481d0
service_quorum = 0x5617f8048460
service_confdb = 0x5617f8069da0
service_dcdb = 0x5617f806a5d0
service_status = 0x5617f806a8e0
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoyé: Mercredi 16 Septembre 2020 09:34:27
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Hi,
I have produced the problem now,
and this time I don't have restarted corosync a second after the lock of /etc/pve
so, currently it's readonly.
I don't have used gdb since a long time,
could you tell me how to attach to the running pmxcfs process, and some gbd commands to launch ?
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoyé: Mardi 15 Septembre 2020 17:58:33
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Another small lock at 17:41:09
To be sure, I have done a small loop of write each second in /etc/pve, node node2.
it's hanging at first corosync restart, then, on second corosync restart it's working again.
I'll try to improve this tomorrow to be able to debug corosync process
- restarting corosync
do some write in /etc/pve/
- and if it's hanging don't restart corosync again
node2: echo test > /etc/pve/test loop
--------------------------------------
Current time : 17:41:01
Current time : 17:41:02
Current time : 17:41:03
Current time : 17:41:04
Current time : 17:41:05
Current time : 17:41:06
Current time : 17:41:07
Current time : 17:41:08
Current time : 17:41:09
hang
Current time : 17:42:05
Current time : 17:42:06
Current time : 17:42:07
node1
-----
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: PMTUD completed for host: 6 link: 0 current link mtu: 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: Starting PMTUD for host: 10 link: 0
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] udp: detected kernel MTU: 1500
Sep 15 17:41:08 m6kvm1 corosync[18145]: [TOTEM ] Knet pMTU change: 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: PMTUD link change for host: 10 link: 0 from 469 to 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: PMTUD completed for host: 10 link: 0 current link mtu: 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: Global data MTU changed to: 1397
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-31-zx6KJM/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] lib_init_fn: conn=0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] got quorum_type request on 0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] got trackstart request on 0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] sending initial status to 0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] sending quorum notification to 0x556c2918d5f0, length = 52
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-32-I7ZZ6e/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CMAP ] lib_init_fn: conn=0x556c2918ef20
Sep 15 17:41:09 m6kvm1 pmxcfs[16239]: [status] notice: update cluster info (cluster name m6kvm, version = 20)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-33-6RKbvH/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] lib_init_fn: conn=0x556c2918ad00, cpd=0x556c2918b50c
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-34-GAY5T9/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] lib_init_fn: conn=0x556c2918c740, cpd=0x556c2918ce8c
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Creating commit token because I am the rep.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Saving state aru 5 high seq received 5
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Storing new sequence id for ring 1197
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] entering COMMIT state.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] entering RECOVERY state.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] TRANS [0] member 1:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [0] member 1:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (1.1193)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 5 high delivered 5 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [1] member 2:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [2] member 3:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [3] member 4:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
ep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [4] member 5:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [5] member 6:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [6] member 7:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [7] member 8:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [8] member 9:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [9] member 10:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [10] member 11:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [11] member 12:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [12] member 13:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [13] member 14:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Did not need to originate any messages in recovery.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Sending initial ORF token
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Resetting old ring state
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] recovery to regular 1-0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] waiting_trans_ack changed to 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.90)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.91)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.92)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.93)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.94)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.95)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.96)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.97)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.107)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.108)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.109)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.110)
ep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.111)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] call init for locally known services
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] entering OPERATIONAL state.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] A new membership (1.1197) was formed. Members joined: 2 3 4 5 6 7 8 9 10 11 12 13 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] enter sync process
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] Committing synchronization for corosync configuration map access
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 2
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 3
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 4
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 5
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 6
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 7
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 8
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 9
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 10
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 11
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 12
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 13
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] Committing synchronization for corosync cluster closed process group service v1.01
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] my downlist: members(old:1 left:0)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[0] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[1] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[2] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[3] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[4] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[5] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[6] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[7] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[8] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[9] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[10] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[11] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[12] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[13] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[14] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[15] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[16] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[17] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[18] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[19] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[20] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[21] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[22] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[23] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[24] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[25] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] Sending nodelist callback. ring_id = 1.1197
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[13]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=2, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[14]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=3, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[1]: votes: 1, expected: 14 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=3, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[2]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=4, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 3
....
....
next corosync restart
Sep 15 17:42:03 m6kvm1 corosync[18145]: [MAIN ] Node was shut down by a signal
Sep 15 17:42:03 m6kvm1 corosync[18145]: [SERV ] Unloading all Corosync service engines.
Sep 15 17:42:03 m6kvm1 corosync[18145]: [QB ] withdrawing server sockets
Sep 15 17:42:03 m6kvm1 corosync[18145]: [QB ] qb_ipcs_unref() - destroying
Sep 15 17:42:03 m6kvm1 corosync[18145]: [SERV ] Service engine unloaded: corosync vote quorum service v1.0
Sep 15 17:42:03 m6kvm1 corosync[18145]: [QB ] qb_ipcs_disconnect(/dev/shm/qb-18145-16239-32-I7ZZ6e/qb) state:2
Sep 15 17:42:03 m6kvm1 pmxcfs[16239]: [confdb] crit: cmap_dispatch failed: 2
Sep 15 17:42:03 m6kvm1 corosync[18145]: [MAIN ] cs_ipcs_connection_closed()
Sep 15 17:42:03 m6kvm1 corosync[18145]: [CMAP ] exit_fn for conn=0x556c2918ef20
Sep 15 17:42:03 m6kvm1 corosync[18145]: [MAIN ] cs_ipcs_connection_destroyed()
node2
-----
Sep 15 17:41:05 m6kvm2 corosync[25411]: [KNET ] pmtud: Starting PMTUD for host: 10 link: 0
Sep 15 17:41:05 m6kvm2 corosync[25411]: [KNET ] udp: detected kernel MTU: 1500
Sep 15 17:41:05 m6kvm2 corosync[25411]: [KNET ] pmtud: PMTUD completed for host: 10 link: 0 current link mtu: 1397
Sep 15 17:41:07 m6kvm2 corosync[25411]: [KNET ] rx: host: 1 link: 0 received pong: 2
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [TOTEM ] entering GATHER state from 11(merge during join).
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: host: 1 link: 0 is up
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Knet host change callback. nodeid: 1 reachable: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] pmtud: Starting PMTUD for host: 1 link: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] udp: detected kernel MTU: 1500
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] pmtud: PMTUD completed for host: 1 link: 0 current link mtu: 1397
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Saving state aru 123 high seq received 123
Sep 15 17:41:09 m6kvm2 corosync[25411]: [MAIN ] Storing new sequence id for ring 1197
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] entering COMMIT state.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] entering RECOVERY state.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [0] member 2:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [1] member 3:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [2] member 4:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [3] member 5:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [4] member 6:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [5] member 7:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [6] member 8:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [7] member 9:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [8] member 10:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [9] member 11:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [10] member 12:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [11] member 13:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [12] member 14:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [0] member 1:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (1.1193)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 5 high delivered 5 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [1] member 2:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [2] member 3:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [3] member 4:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [4] member 5:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [5] member 6:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [6] member 7:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [7] member 8:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [8] member 9:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [9] member 10:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [10] member 11:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [11] member 12:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [12] member 13:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [13] member 14:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Did not need to originate any messages in recovery.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru ffffffff
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Resetting old ring state
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] recovery to regular 1-0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] waiting_trans_ack changed to 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [MAIN ] Member joined: r(0) ip(10.3.94.89)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] call init for locally known services
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] entering OPERATIONAL state.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] A new membership (1.1197) was formed. Members joined: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] enter sync process
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] Committing synchronization for corosync configuration map access
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CMAP ] Not first sync -> no action
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 3
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 4
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 5
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 6
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 7
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 8
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 9
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 10
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 11
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 12
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 13
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] Committing synchronization for corosync cluster closed process group service v1.01
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] my downlist: members(old:13 left:0)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[0] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[1] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[2] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[3] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[4] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[5] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[6] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[7] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[8] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[9] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[10] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[11] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[12] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[13] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[14] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[15] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[16] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[17] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[18] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[19] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[20] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[21] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[22] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[23] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[24] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[25] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] Sending nodelist callback. ring_id = 1.1197
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[13]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[14]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[1]: votes: 1, expected: 14 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] total_votes=14, expected_votes=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 3 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 4 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 5 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 6 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 7 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 8 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 9 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 10 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 11 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 12 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] lowest node id: 1 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] highest node id: 14 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[2]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] total_votes=14, expected_votes=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 3 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 4 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 5 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 6 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 7 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 8 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 9 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 10 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 11 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 12 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] lowest node id: 1 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] highest node id: 14 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 3
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[3]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 3
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 4
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[4]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 4
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 5
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[5]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 5
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 6
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[6]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 6
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 7
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[7]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 7
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 8
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[8]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 8
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 9
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[9]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 9
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 10
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[10]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 10
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 11
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[11]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 11
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 12
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[12]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 12
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] Committing synchronization for corosync vote quorum service v1.0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] total_votes=14, expected_votes=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 3 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 4 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 5 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 6 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 7 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 8 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 9 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 10 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 11 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 12 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] lowest node id: 1 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] highest node id: 14 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [QUORUM] Members[14]: 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [QUORUM] sending quorum notification to (nil), length = 104
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] Sending quorum callback, quorate = 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] waiting_trans_ack changed to 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got procjoin message from cluster node 1 (r(0) ip(10.3.94.89) ) for pid 16239
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got procjoin message from cluster node 1 (r(0) ip(10.3.94.89) ) for pid 16239
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 16:57:46
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>I mean this is bad, but also great!
>>Cam you do a coredump of the whole thing and upload it somewhere with the version info
>>used (for dbgsym package)? That could help a lot.
I'll try to reproduce it again (with the full lock everywhere), and do the coredump.
I have tried the real time scheduling,
but I still have been able to reproduce the "lrm too long" for 60s (but as I'm restarting corosync each minute, I think it's unlocking
something at next corosync restart.)
this time it was blocked at the same time on a node in:
work {
...
} elsif ($state eq 'active') {
....
$self->update_lrm_status();
and another node in
if ($fence_request) {
$haenv->log('err', "node need to be fenced - releasing agent_lock\n");
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif (!$self->get_protected_ha_agent_lock()) {
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif ($self->{mode} eq 'maintenance') {
$self->set_local_status({ state => 'maintenance'});
}
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 16:32:52
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/15/20 4:09 PM, Alexandre DERUMIER wrote:
>>> Can you try to give pmxcfs real time scheduling, e.g., by doing:
>>>
>>> # systemctl edit pve-cluster
>>>
>>> And then add snippet:
>>>
>>>
>>> [Service]
>>> CPUSchedulingPolicy=rr
>>> CPUSchedulingPriority=99
> yes, sure, I'll do it now
>
>
>> I'm currently digging the logs
>>> Is your most simplest/stable reproducer still a periodic restart of corosync in one node?
> yes, a simple "systemctl restart corosync" on 1 node each minute
>
>
>
> After 1hour, it's still locked.
>
> on other nodes, I still have pmxfs logs like:
>
I mean this is bad, but also great!
Cam you do a coredump of the whole thing and upload it somewhere with the version info
used (for dbgsym package)? That could help a lot.
> manual "pmxcfs -d"
> https://gist.github.com/aderumier/4cd91d17e1f8847b93ea5f621f257c2e
>
Hmm, the fuse connection of the previous one got into a weird state (or something is still
running) but I'd rather say this is a side-effect not directly connected to the real bug.
>
> some interesting dmesg about "pvesr"
>
> [Tue Sep 15 14:45:34 2020] INFO: task pvesr:19038 blocked for more than 120 seconds.
> [Tue Sep 15 14:45:34 2020] Tainted: P O 5.4.60-1-pve #1
> [Tue Sep 15 14:45:34 2020] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Tue Sep 15 14:45:34 2020] pvesr D 0 19038 1 0x00000080
> [Tue Sep 15 14:45:34 2020] Call Trace:
> [Tue Sep 15 14:45:34 2020] __schedule+0x2e6/0x6f0
> [Tue Sep 15 14:45:34 2020] ? filename_parentat.isra.57.part.58+0xf7/0x180
> [Tue Sep 15 14:45:34 2020] schedule+0x33/0xa0
> [Tue Sep 15 14:45:34 2020] rwsem_down_write_slowpath+0x2ed/0x4a0
> [Tue Sep 15 14:45:34 2020] down_write+0x3d/0x40
> [Tue Sep 15 14:45:34 2020] filename_create+0x8e/0x180
> [Tue Sep 15 14:45:34 2020] do_mkdirat+0x59/0x110
> [Tue Sep 15 14:45:34 2020] __x64_sys_mkdir+0x1b/0x20
> [Tue Sep 15 14:45:34 2020] do_syscall_64+0x57/0x190
> [Tue Sep 15 14:45:34 2020] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
hmm, hangs in mkdir (cluster wide locking)
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-16 7:58 ` Alexandre DERUMIER
@ 2020-09-16 8:30 ` Alexandre DERUMIER
2020-09-16 8:53 ` Alexandre DERUMIER
[not found] ` <1894376736.864562.1600253445817.JavaMail.zimbra@odiso.com>
0 siblings, 2 replies; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-16 8:30 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: Thomas Lamprecht
backtrace of all threads.
(I don't have libqb or fuse debug symbol)
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7fce63d5f900 (LWP 16239) "pmxcfs" 0x00007fce67721896 in do_futex_wait.constprop () from /lib/x86_64-linux-gnu/libpthread.so.0
2 Thread 0x7fce63ce6700 (LWP 16240) "cfs_loop" 0x00007fce676497ef in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
3 Thread 0x7fce618c9700 (LWP 16246) "server" 0x00007fce676497ef in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
4 Thread 0x7fce46beb700 (LWP 30256) "pmxcfs" 0x00007fce67643f59 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
5 Thread 0x7fce2abf9700 (LWP 43943) "pmxcfs" 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
6 Thread 0x7fce610a6700 (LWP 13346) "pmxcfs" 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
7 Thread 0x7fce2a3f8700 (LWP 8832) "pmxcfs" 0x00007fce67643f59 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
8 Thread 0x7fce28bf5700 (LWP 3464) "pmxcfs" 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
9 Thread 0x7fce453e8700 (LWP 3727) "pmxcfs" 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
10 Thread 0x7fce2bfff700 (LWP 6705) "pmxcfs" 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
11 Thread 0x7fce293f6700 (LWP 41454) "pmxcfs" 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
12 Thread 0x7fce45be9700 (LWP 17734) "pmxcfs" 0x00007fce67643f59 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
13 Thread 0x7fce2b7fe700 (LWP 17762) "pmxcfs" 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
14 Thread 0x7fce463ea700 (LWP 2347) "pmxcfs" 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
15 Thread 0x7fce44be7700 (LWP 11335) "pmxcfs" 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) thread 1
[Switching to thread 1 (Thread 0x7fce63d5f900 (LWP 16239))]
#0 0x00007fce67721896 in do_futex_wait.constprop () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt full
#0 0x00007fce67721896 in do_futex_wait.constprop () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x00007fce67721988 in __new_sem_wait_slow.constprop.0 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x00007fce678c5f98 in fuse_session_loop_mt () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#3 0x00007fce678cb577 in fuse_loop_mt () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#4 0x00005617f6d5ab0e in main (argc=<optimized out>, argv=<optimized out>) at pmxcfs.c:1055
ret = -1
lockfd = <optimized out>
pipefd = {8, 9}
foreground = 0
force_local_mode = 0
wrote_pidfile = 1
memdb = 0x5617f7c563b0
dcdb = 0x5617f8046ca0
status_fsm = 0x5617f806a630
context = <optimized out>
entries = {{long_name = 0x5617f6d7440f "debug", short_name = 100 'd', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x5617f6d82104 <cfs+20>, description = 0x5617f6d742cb "Turn on debug messages", arg_description = 0x0}, {
long_name = 0x5617f6d742e2 "foreground", short_name = 102 'f', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x7ffd672edc38, description = 0x5617f6d742ed "Do not daemonize server", arg_description = 0x0}, {
long_name = 0x5617f6d74305 "local", short_name = 108 'l', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x7ffd672edc3c, description = 0x5617f6d746a0 "Force local mode (ignore corosync.conf, force quorum)",
arg_description = 0x0}, {long_name = 0x0, short_name = 0 '\000', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x0, description = 0x0, arg_description = 0x0}}
err = 0x0
__func__ = "main"
utsname = {sysname = "Linux", '\000' <repeats 59 times>, nodename = "m6kvm1", '\000' <repeats 58 times>, release = "5.4.60-1-pve", '\000' <repeats 52 times>,
version = "#1 SMP PVE 5.4.60-1 (Mon, 31 Aug 2020 10:36:22 +0200)", '\000' <repeats 11 times>, machine = "x86_64", '\000' <repeats 58 times>, __domainname = "(none)", '\000' <repeats 58 times>}
dot = <optimized out>
www_data = <optimized out>
create = <optimized out>
conf_data = 0x5617f80466a0
len = <optimized out>
config = <optimized out>
bplug = <optimized out>
fa = {0x5617f6d7441e "-f", 0x5617f6d74421 "-odefault_permissions", 0x5617f6d74437 "-oallow_other", 0x0}
fuse_args = {argc = 1, argv = 0x5617f8046960, allocated = 1}
fuse_chan = 0x5617f7c560c0
corosync_loop = 0x5617f80481d0
service_quorum = 0x5617f8048460
service_confdb = 0x5617f8069da0
service_dcdb = 0x5617f806a5d0
service_status = 0x5617f806a8e0
(gdb) thread 2
[Switching to thread 2 (Thread 0x7fce63ce6700 (LWP 16240))]
#0 0x00007fce676497ef in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt full
#0 0x00007fce676497ef in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1 0x00007fce67500b2a in ?? () from /lib/x86_64-linux-gnu/libqb.so.0
No symbol table info available.
#2 0x00007fce674f1bb0 in qb_loop_run () from /lib/x86_64-linux-gnu/libqb.so.0
No symbol table info available.
#3 0x00005617f6d5cd31 in cfs_loop_worker_thread (data=0x5617f80481d0) at loop.c:330
__func__ = "cfs_loop_worker_thread"
loop = 0x5617f80481d0
qbloop = 0x5617f8048230
l = <optimized out>
ctime = <optimized out>
th = 7222815479134420993
#4 0x00007fce67968415 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#5 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#6 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) bt full
#0 0x00007fce676497ef in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1 0x00007fce67500b2a in ?? () from /lib/x86_64-linux-gnu/libqb.so.0
No symbol table info available.
#2 0x00007fce674f1bb0 in qb_loop_run () from /lib/x86_64-linux-gnu/libqb.so.0
No symbol table info available.
#3 0x00005617f6d5e453 in worker_thread (data=<optimized out>) at server.c:529
th = 5863405772435619840
__func__ = "worker_thread"
_g_boolean_var_ = <optimized out>
#4 0x00007fce67968415 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#5 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#6 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) bt full
#0 0x00007fce67643f59 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1 0x00007fce67989f9f in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#2 0x00005617f6d67f14 in dfsm_send_message_sync (dfsm=dfsm@entry=0x5617f8046ca0, msgtype=msgtype@entry=5, iov=iov@entry=0x7fce46bea870, len=len@entry=8, rp=rp@entry=0x7fce46bea860) at dfsm.c:339
__func__ = "dfsm_send_message_sync"
msgcount = 189465
header = {base = {type = 0, subtype = 5, protocol_version = 1, time = 1600241342, reserved = 0}, count = 189465}
real_iov = {{iov_base = 0x7fce46bea7e0, iov_len = 24}, {iov_base = 0x7fce46bea84c, iov_len = 4}, {iov_base = 0x7fce46bea940, iov_len = 4}, {iov_base = 0x7fce46bea858, iov_len = 4}, {iov_base = 0x7fce46bea85c, iov_len = 4}, {
iov_base = 0x7fce46bea948, iov_len = 4}, {iov_base = 0x7fce1c002980, iov_len = 34}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}}
result = CS_OK
#3 0x00005617f6d6491e in dcdb_send_fuse_message (dfsm=0x5617f8046ca0, msg_type=msg_type@entry=DCDB_MESSAGE_CFS_CREATE, path=0x7fce1c002980 "nodes/m6kvm1/lrm_status.tmp.42080", to=to@entry=0x0, buf=buf@entry=0x0, size=<optimized out>,
size@entry=0, offset=<optimized out>, flags=<optimized out>) at dcdb.c:157
iov = {{iov_base = 0x7fce46bea84c, iov_len = 4}, {iov_base = 0x7fce46bea940, iov_len = 4}, {iov_base = 0x7fce46bea858, iov_len = 4}, {iov_base = 0x7fce46bea85c, iov_len = 4}, {iov_base = 0x7fce46bea948, iov_len = 4}, {
iov_base = 0x7fce1c002980, iov_len = 34}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}}
pathlen = 34
tolen = 0
rc = {msgcount = 189465, result = -16, processed = 0}
#4 0x00005617f6d6b640 in cfs_plug_memdb_create (mode=<optimized out>, fi=<optimized out>, path=<optimized out>, plug=0x5617f8048b70) at cfs-plug-memdb.c:307
res = <optimized out>
mdb = 0x5617f8048b70
__func__ = "cfs_plug_memdb_create"
res = <optimized out>
mdb = <optimized out>
_g_boolean_var_ = <optimized out>
_g_boolean_var_ = <optimized out>
_g_boolean_var_ = <optimized out>
_g_boolean_var_ = <optimized out>
ctime = <optimized out>
#5 cfs_plug_memdb_create (plug=0x5617f8048b70, path=<optimized out>, mode=<optimized out>, fi=<optimized out>) at cfs-plug-memdb.c:291
res = <optimized out>
mdb = <optimized out>
__func__ = "cfs_plug_memdb_create"
_g_boolean_var_ = <optimized out>
_g_boolean_var_ = <optimized out>
_g_boolean_var_ = <optimized out>
_g_boolean_var_ = <optimized out>
ctime = <optimized out>
#6 0x00005617f6d5ca5d in cfs_fuse_create (path=0x7fce1c000bf0 "/nodes/m6kvm1/lrm_status.tmp.42080", mode=33188, fi=0x7fce46beab30) at pmxcfs.c:415
__func__ = "cfs_fuse_create"
ret = -13
subpath = 0x7fce1c002980 "nodes/m6kvm1/lrm_status.tmp.42080"
plug = <optimized out>
#7 0x00007fce678c152b in fuse_fs_create () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#8 0x00007fce678c165a in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#9 0x00007fce678c7bb7 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#10 0x00007fce678c90b8 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#11 0x00007fce678c5d5c in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#12 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#13 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) thread 5
[Switching to thread 5 (Thread 0x7fce2abf9700 (LWP 43943))]
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt full
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x00007fce678c56d2 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#2 0x00007fce678c7249 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#3 0x00007fce678c5cdf in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#4 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
gdb) thread 6
[Switching to thread 6 (Thread 0x7fce610a6700 (LWP 13346))]
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt full
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x00007fce678c56d2 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#2 0x00007fce678c7249 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#3 0x00007fce678c5cdf in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#4 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) thread 7
[Switching to thread 7 (Thread 0x7fce2a3f8700 (LWP 8832))]
#0 0x00007fce67643f59 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt full
#0 0x00007fce67643f59 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1 0x00007fce67989f9f in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#2 0x00005617f6d67f14 in dfsm_send_message_sync (dfsm=dfsm@entry=0x5617f8046ca0, msgtype=msgtype@entry=5, iov=iov@entry=0x7fce2a3f7870, len=len@entry=8, rp=rp@entry=0x7fce2a3f7860) at dfsm.c:339
__func__ = "dfsm_send_message_sync"
msgcount = 189466
header = {base = {type = 0, subtype = 5, protocol_version = 1, time = 1600242491, reserved = 0}, count = 189466}
real_iov = {{iov_base = 0x7fce2a3f77e0, iov_len = 24}, {iov_base = 0x7fce2a3f784c, iov_len = 4}, {iov_base = 0x7fce2a3f7940, iov_len = 4}, {iov_base = 0x7fce2a3f7858, iov_len = 4}, {iov_base = 0x7fce2a3f785c, iov_len = 4}, {
iov_base = 0x7fce2a3f7948, iov_len = 4}, {iov_base = 0x7fce1406f350, iov_len = 10}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}}
result = CS_OK
#3 0x00005617f6d6491e in dcdb_send_fuse_message (dfsm=0x5617f8046ca0, msg_type=msg_type@entry=DCDB_MESSAGE_CFS_CREATE, path=0x7fce1406f350 "testalex2", to=to@entry=0x0, buf=buf@entry=0x0, size=<optimized out>, size@entry=0,
offset=<optimized out>, flags=<optimized out>) at dcdb.c:157
iov = {{iov_base = 0x7fce2a3f784c, iov_len = 4}, {iov_base = 0x7fce2a3f7940, iov_len = 4}, {iov_base = 0x7fce2a3f7858, iov_len = 4}, {iov_base = 0x7fce2a3f785c, iov_len = 4}, {iov_base = 0x7fce2a3f7948, iov_len = 4}, {
iov_base = 0x7fce1406f350, iov_len = 10}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}}
pathlen = 10
tolen = 0
rc = {msgcount = 189466, result = -16, processed = 0}
#4 0x00005617f6d6b640 in cfs_plug_memdb_create (mode=<optimized out>, fi=<optimized out>, path=<optimized out>, plug=0x5617f8048b70) at cfs-plug-memdb.c:307
res = <optimized out>
mdb = 0x5617f8048b70
__func__ = "cfs_plug_memdb_create"
res = <optimized out>
mdb = <optimized out>
_g_boolean_var_ = <optimized out>
_g_boolean_var_ = <optimized out>
_g_boolean_var_ = <optimized out>
_g_boolean_var_ = <optimized out>
ctime = <optimized out>
#5 cfs_plug_memdb_create (plug=0x5617f8048b70, path=<optimized out>, mode=<optimized out>, fi=<optimized out>) at cfs-plug-memdb.c:291
res = <optimized out>
mdb = <optimized out>
__func__ = "cfs_plug_memdb_create"
_g_boolean_var_ = <optimized out>
_g_boolean_var_ = <optimized out>
_g_boolean_var_ = <optimized out>
_g_boolean_var_ = <optimized out>
ctime = <optimized out>
#6 0x00005617f6d5ca5d in cfs_fuse_create (path=0x7fce14022ff0 "/testalex2", mode=33188, fi=0x7fce2a3f7b30) at pmxcfs.c:415
__func__ = "cfs_fuse_create"
ret = -13
subpath = 0x7fce1406f350 "testalex2"
plug = <optimized out>
#7 0x00007fce678c152b in fuse_fs_create () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#8 0x00007fce678c165a in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#9 0x00007fce678c7bb7 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#10 0x00007fce678c90b8 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#11 0x00007fce678c5d5c in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#12 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#13 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) thread 8
[Switching to thread 8 (Thread 0x7fce28bf5700 (LWP 3464))]
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt full
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x00007fce678c56d2 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#2 0x00007fce678c7249 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#3 0x00007fce678c5cdf in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#4 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) thread 9
[Switching to thread 9 (Thread 0x7fce453e8700 (LWP 3727))]
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt full
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x00007fce678c56d2 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#2 0x00007fce678c7249 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#3 0x00007fce678c5cdf in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#4 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) thread 10
[Switching to thread 10 (Thread 0x7fce2bfff700 (LWP 6705))]
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt full
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x00007fce678c56d2 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#2 0x00007fce678c7249 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#3 0x00007fce678c5cdf in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#4 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) thread 11
[Switching to thread 11 (Thread 0x7fce293f6700 (LWP 41454))]
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt full
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x00007fce678c56d2 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#2 0x00007fce678c7249 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#3 0x00007fce678c5cdf in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#4 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) thread 12
[Switching to thread 12 (Thread 0x7fce45be9700 (LWP 17734))]
#0 0x00007fce67643f59 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt full
#0 0x00007fce67643f59 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1 0x00007fce67989f9f in g_cond_wait () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
No symbol table info available.
#2 0x00005617f6d67f14 in dfsm_send_message_sync (dfsm=dfsm@entry=0x5617f8046ca0, msgtype=msgtype@entry=2, iov=iov@entry=0x7fce45be88e0, len=len@entry=8, rp=rp@entry=0x7fce45be88d0) at dfsm.c:339
__func__ = "dfsm_send_message_sync"
msgcount = 189464
header = {base = {type = 0, subtype = 2, protocol_version = 1, time = 1600241340, reserved = 0}, count = 189464}
real_iov = {{iov_base = 0x7fce45be8850, iov_len = 24}, {iov_base = 0x7fce45be88bc, iov_len = 4}, {iov_base = 0x7fce45be89b0, iov_len = 4}, {iov_base = 0x7fce45be88c8, iov_len = 4}, {iov_base = 0x7fce45be88cc, iov_len = 4}, {
iov_base = 0x7fce45be89b8, iov_len = 4}, {iov_base = 0x7fce34001dd0, iov_len = 31}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}}
result = CS_OK
#3 0x00005617f6d6491e in dcdb_send_fuse_message (dfsm=0x5617f8046ca0, msg_type=msg_type@entry=DCDB_MESSAGE_CFS_MKDIR, path=0x7fce34001dd0 "priv/lock/file-replication_cfg", to=to@entry=0x0, buf=buf@entry=0x0, size=<optimized out>,
size@entry=0, offset=<optimized out>, flags=<optimized out>) at dcdb.c:157
iov = {{iov_base = 0x7fce45be88bc, iov_len = 4}, {iov_base = 0x7fce45be89b0, iov_len = 4}, {iov_base = 0x7fce45be88c8, iov_len = 4}, {iov_base = 0x7fce45be88cc, iov_len = 4}, {iov_base = 0x7fce45be89b8, iov_len = 4}, {
iov_base = 0x7fce34001dd0, iov_len = 31}, {iov_base = 0x0, iov_len = 0}, {iov_base = 0x0, iov_len = 0}}
pathlen = 31
tolen = 0
rc = {msgcount = 189464, result = -16, processed = 0}
#4 0x00005617f6d6b0e7 in cfs_plug_memdb_mkdir (plug=0x5617f8048b70, path=<optimized out>, mode=<optimized out>) at cfs-plug-memdb.c:329
__func__ = "cfs_plug_memdb_mkdir"
res = <optimized out>
mdb = 0x5617f8048b70
#5 0x00005617f6d5bc70 in cfs_fuse_mkdir (path=0x7fce34002240 "/priv/lock/file-replication_cfg", mode=493) at pmxcfs.c:238
__func__ = "cfs_fuse_mkdir"
ret = -13
subpath = 0x7fce34001dd0 "priv/lock/file-replication_cfg"
plug = <optimized out>
#6 0x00007fce678c2ccb in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#7 0x00007fce678c90b8 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#8 0x00007fce678c5d5c in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#9 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#10 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) thread 13
[Switching to thread 13 (Thread 0x7fce2b7fe700 (LWP 17762))]
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt full
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x00007fce678c56d2 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#2 0x00007fce678c7249 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#3 0x00007fce678c5cdf in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#4 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) thread 14
[Switching to thread 14 (Thread 0x7fce463ea700 (LWP 2347))]
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt full
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x00007fce678c56d2 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#2 0x00007fce678c7249 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#3 0x00007fce678c5cdf in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#4 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
(gdb) thread 15
[Switching to thread 15 (Thread 0x7fce44be7700 (LWP 11335))]
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) bt full
#0 0x00007fce67722544 in read () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x00007fce678c56d2 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#2 0x00007fce678c7249 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#3 0x00007fce678c5cdf in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#4 0x00007fce67718fa3 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#5 0x00007fce676494cf in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoyé: Mercredi 16 Septembre 2020 09:58:02
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
here a backtrace with pve-cluster-dbgsym installed
(gdb) bt full
#0 0x00007fce67721896 in do_futex_wait.constprop () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#1 0x00007fce67721988 in __new_sem_wait_slow.constprop.0 () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#2 0x00007fce678c5f98 in fuse_session_loop_mt () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#3 0x00007fce678cb577 in fuse_loop_mt () from /lib/x86_64-linux-gnu/libfuse.so.2
No symbol table info available.
#4 0x00005617f6d5ab0e in main (argc=<optimized out>, argv=<optimized out>) at pmxcfs.c:1055
ret = -1
lockfd = <optimized out>
pipefd = {8, 9}
foreground = 0
force_local_mode = 0
wrote_pidfile = 1
memdb = 0x5617f7c563b0
dcdb = 0x5617f8046ca0
status_fsm = 0x5617f806a630
context = <optimized out>
entries = {{long_name = 0x5617f6d7440f "debug", short_name = 100 'd', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x5617f6d82104 <cfs+20>, description = 0x5617f6d742cb "Turn on debug messages", arg_description = 0x0}, {
long_name = 0x5617f6d742e2 "foreground", short_name = 102 'f', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x7ffd672edc38, description = 0x5617f6d742ed "Do not daemonize server", arg_description = 0x0}, {
long_name = 0x5617f6d74305 "local", short_name = 108 'l', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x7ffd672edc3c, description = 0x5617f6d746a0 "Force local mode (ignore corosync.conf, force quorum)",
arg_description = 0x0}, {long_name = 0x0, short_name = 0 '\000', flags = 0, arg = G_OPTION_ARG_NONE, arg_data = 0x0, description = 0x0, arg_description = 0x0}}
err = 0x0
__func__ = "main"
utsname = {sysname = "Linux", '\000' <repeats 59 times>, nodename = "m6kvm1", '\000' <repeats 58 times>, release = "5.4.60-1-pve", '\000' <repeats 52 times>,
version = "#1 SMP PVE 5.4.60-1 (Mon, 31 Aug 2020 10:36:22 +0200)", '\000' <repeats 11 times>, machine = "x86_64", '\000' <repeats 58 times>, __domainname = "(none)", '\000' <repeats 58 times>}
dot = <optimized out>
www_data = <optimized out>
create = <optimized out>
conf_data = 0x5617f80466a0
len = <optimized out>
config = <optimized out>
bplug = <optimized out>
fa = {0x5617f6d7441e "-f", 0x5617f6d74421 "-odefault_permissions", 0x5617f6d74437 "-oallow_other", 0x0}
fuse_args = {argc = 1, argv = 0x5617f8046960, allocated = 1}
fuse_chan = 0x5617f7c560c0
corosync_loop = 0x5617f80481d0
service_quorum = 0x5617f8048460
service_confdb = 0x5617f8069da0
service_dcdb = 0x5617f806a5d0
service_status = 0x5617f806a8e0
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoyé: Mercredi 16 Septembre 2020 09:34:27
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Hi,
I have produced the problem now,
and this time I don't have restarted corosync a second after the lock of /etc/pve
so, currently it's readonly.
I don't have used gdb since a long time,
could you tell me how to attach to the running pmxcfs process, and some gbd commands to launch ?
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoyé: Mardi 15 Septembre 2020 17:58:33
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Another small lock at 17:41:09
To be sure, I have done a small loop of write each second in /etc/pve, node node2.
it's hanging at first corosync restart, then, on second corosync restart it's working again.
I'll try to improve this tomorrow to be able to debug corosync process
- restarting corosync
do some write in /etc/pve/
- and if it's hanging don't restart corosync again
node2: echo test > /etc/pve/test loop
--------------------------------------
Current time : 17:41:01
Current time : 17:41:02
Current time : 17:41:03
Current time : 17:41:04
Current time : 17:41:05
Current time : 17:41:06
Current time : 17:41:07
Current time : 17:41:08
Current time : 17:41:09
hang
Current time : 17:42:05
Current time : 17:42:06
Current time : 17:42:07
node1
-----
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: PMTUD completed for host: 6 link: 0 current link mtu: 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: Starting PMTUD for host: 10 link: 0
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] udp: detected kernel MTU: 1500
Sep 15 17:41:08 m6kvm1 corosync[18145]: [TOTEM ] Knet pMTU change: 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: PMTUD link change for host: 10 link: 0 from 469 to 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: PMTUD completed for host: 10 link: 0 current link mtu: 1397
Sep 15 17:41:08 m6kvm1 corosync[18145]: [KNET ] pmtud: Global data MTU changed to: 1397
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-31-zx6KJM/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] lib_init_fn: conn=0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] got quorum_type request on 0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] got trackstart request on 0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] sending initial status to 0x556c2918d5f0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QUORUM] sending quorum notification to 0x556c2918d5f0, length = 52
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-32-I7ZZ6e/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CMAP ] lib_init_fn: conn=0x556c2918ef20
Sep 15 17:41:09 m6kvm1 pmxcfs[16239]: [status] notice: update cluster info (cluster name m6kvm, version = 20)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-33-6RKbvH/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] lib_init_fn: conn=0x556c2918ad00, cpd=0x556c2918b50c
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] IPC credentials authenticated (/dev/shm/qb-18145-16239-34-GAY5T9/qb)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] connecting to client [16239]
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] connection created
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] lib_init_fn: conn=0x556c2918c740, cpd=0x556c2918ce8c
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Creating commit token because I am the rep.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Saving state aru 5 high seq received 5
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Storing new sequence id for ring 1197
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] entering COMMIT state.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] entering RECOVERY state.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] TRANS [0] member 1:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [0] member 1:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (1.1193)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 5 high delivered 5 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [1] member 2:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [2] member 3:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [3] member 4:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
ep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [4] member 5:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [5] member 6:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [6] member 7:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [7] member 8:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [8] member 9:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [9] member 10:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [10] member 11:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [11] member 12:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [12] member 13:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] position [13] member 14:
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Did not need to originate any messages in recovery.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Sending initial ORF token
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] Resetting old ring state
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] recovery to regular 1-0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] waiting_trans_ack changed to 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.90)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.91)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.92)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.93)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.94)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.95)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.96)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.97)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.107)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.108)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.109)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.110)
ep 15 17:41:09 m6kvm1 corosync[18145]: [MAIN ] Member joined: r(0) ip(10.3.94.111)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] call init for locally known services
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] entering OPERATIONAL state.
Sep 15 17:41:09 m6kvm1 corosync[18145]: [TOTEM ] A new membership (1.1197) was formed. Members joined: 2 3 4 5 6 7 8 9 10 11 12 13 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] enter sync process
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] Committing synchronization for corosync configuration map access
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 2
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 3
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 4
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 5
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 6
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 7
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 8
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 9
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 10
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 11
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 12
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] got joinlist message from node 13
Sep 15 17:41:09 m6kvm1 corosync[18145]: [SYNC ] Committing synchronization for corosync cluster closed process group service v1.01
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] my downlist: members(old:1 left:0)
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[0] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[1] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[2] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[3] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[4] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[5] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[6] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[7] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[8] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[9] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[10] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[11] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[12] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[13] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[14] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[15] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[16] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[17] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[18] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[19] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[20] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[21] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[22] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[23] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[24] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm1 corosync[18145]: [CPG ] joinlist_messages[25] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] Sending nodelist callback. ring_id = 1.1197
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[13]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=2, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[14]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=3, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[1]: votes: 1, expected: 14 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=3, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[2]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] total_votes=4, expected_votes=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm1 corosync[18145]: [VOTEQ ] got nodeinfo message from cluster node 3
....
....
next corosync restart
Sep 15 17:42:03 m6kvm1 corosync[18145]: [MAIN ] Node was shut down by a signal
Sep 15 17:42:03 m6kvm1 corosync[18145]: [SERV ] Unloading all Corosync service engines.
Sep 15 17:42:03 m6kvm1 corosync[18145]: [QB ] withdrawing server sockets
Sep 15 17:42:03 m6kvm1 corosync[18145]: [QB ] qb_ipcs_unref() - destroying
Sep 15 17:42:03 m6kvm1 corosync[18145]: [SERV ] Service engine unloaded: corosync vote quorum service v1.0
Sep 15 17:42:03 m6kvm1 corosync[18145]: [QB ] qb_ipcs_disconnect(/dev/shm/qb-18145-16239-32-I7ZZ6e/qb) state:2
Sep 15 17:42:03 m6kvm1 pmxcfs[16239]: [confdb] crit: cmap_dispatch failed: 2
Sep 15 17:42:03 m6kvm1 corosync[18145]: [MAIN ] cs_ipcs_connection_closed()
Sep 15 17:42:03 m6kvm1 corosync[18145]: [CMAP ] exit_fn for conn=0x556c2918ef20
Sep 15 17:42:03 m6kvm1 corosync[18145]: [MAIN ] cs_ipcs_connection_destroyed()
node2
-----
Sep 15 17:41:05 m6kvm2 corosync[25411]: [KNET ] pmtud: Starting PMTUD for host: 10 link: 0
Sep 15 17:41:05 m6kvm2 corosync[25411]: [KNET ] udp: detected kernel MTU: 1500
Sep 15 17:41:05 m6kvm2 corosync[25411]: [KNET ] pmtud: PMTUD completed for host: 10 link: 0 current link mtu: 1397
Sep 15 17:41:07 m6kvm2 corosync[25411]: [KNET ] rx: host: 1 link: 0 received pong: 2
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [TOTEM ] entering GATHER state from 11(merge during join).
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:08 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: Source host 1 not reachable yet. Discarding packet.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] rx: host: 1 link: 0 is up
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Knet host change callback. nodeid: 1 reachable: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] pmtud: Starting PMTUD for host: 1 link: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] udp: detected kernel MTU: 1500
Sep 15 17:41:09 m6kvm2 corosync[25411]: [KNET ] pmtud: PMTUD completed for host: 1 link: 0 current link mtu: 1397
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Saving state aru 123 high seq received 123
Sep 15 17:41:09 m6kvm2 corosync[25411]: [MAIN ] Storing new sequence id for ring 1197
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] entering COMMIT state.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] got commit token
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] entering RECOVERY state.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [0] member 2:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [1] member 3:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [2] member 4:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [3] member 5:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [4] member 6:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [5] member 7:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [6] member 8:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [7] member 9:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [8] member 10:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [9] member 11:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [10] member 12:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [11] member 13:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] TRANS [12] member 14:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [0] member 1:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (1.1193)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 5 high delivered 5 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [1] member 2:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [2] member 3:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [3] member 4:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [4] member 5:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [5] member 6:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [6] member 7:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [7] member 8:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [8] member 9:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [9] member 10:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [10] member 11:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [11] member 12:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [12] member 13:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] position [13] member 14:
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] previous ringid (2.1192)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] aru 123 high delivered 123 received flag 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Did not need to originate any messages in recovery.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru ffffffff
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] install seq 0 aru 0 high seq received 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] retrans flag count 4 token aru 0 install seq 0 aru 0 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] Resetting old ring state
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] recovery to regular 1-0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] waiting_trans_ack changed to 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [MAIN ] Member joined: r(0) ip(10.3.94.89)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] call init for locally known services
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] entering OPERATIONAL state.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] A new membership (1.1197) was formed. Members joined: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] enter sync process
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] Committing synchronization for corosync configuration map access
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CMAP ] Not first sync -> no action
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 3
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 4
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 5
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 6
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 7
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 8
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 9
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 10
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 11
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 12
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] downlist left_list: 0 received
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got joinlist message from node 13
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] Committing synchronization for corosync cluster closed process group service v1.01
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] my downlist: members(old:13 left:0)
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[0] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[1] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.110) , pid:30209
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[2] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[3] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.109) , pid:31350
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[4] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[5] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.108) , pid:3569
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[6] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[7] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.107) , pid:19504
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[8] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[9] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.97) , pid:11947
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[10] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[11] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.96) , pid:20814
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[12] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[13] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.95) , pid:39420
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[14] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[15] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.94) , pid:12452
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[16] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[17] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.93) , pid:44300
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[18] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[19] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.92) , pid:42259
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[20] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[21] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.91) , pid:40630
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[22] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[23] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.90) , pid:25870
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[24] group:pve_kvstore_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] joinlist_messages[25] group:pve_dcdb_v1\x00, ip:r(0) ip(10.3.94.111) , pid:25634
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] Sending nodelist callback. ring_id = 1.1197
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[13]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 13
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[14]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[1]: votes: 1, expected: 14 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: No Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] total_votes=14, expected_votes=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 3 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 4 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 5 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 6 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 7 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 8 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 9 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 10 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 11 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 12 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] lowest node id: 1 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] highest node id: 14 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[2]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] total_votes=14, expected_votes=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 3 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 4 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 5 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 6 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 7 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 8 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 9 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 10 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 11 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 12 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] lowest node id: 1 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] highest node id: 14 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 3
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[3]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 3
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 4
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[4]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 4
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 5
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[5]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 5
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 6
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[6]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 6
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 7
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[7]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 7
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 8
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[8]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 8
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 9
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[9]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 9
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 10
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[10]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 10
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 11
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[11]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 11
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 12
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[12]: votes: 1, expected: 14 flags: 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] flags: quorate: Yes Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No QdeviceCastVote: No QdeviceMasterWins: No
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] got nodeinfo message from cluster node 12
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] nodeinfo message[0]: votes: 0, expected: 0 flags: 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [SYNC ] Committing synchronization for corosync vote quorum service v1.0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] total_votes=14, expected_votes=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 1 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 3 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 4 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 5 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 6 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 7 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 8 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 9 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 10 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 11 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 12 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 13 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 14 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] node 2 state=1, votes=1, expected=14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] lowest node id: 1 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] highest node id: 14 us: 2
Sep 15 17:41:09 m6kvm2 corosync[25411]: [QUORUM] Members[14]: 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Sep 15 17:41:09 m6kvm2 corosync[25411]: [QUORUM] sending quorum notification to (nil), length = 104
Sep 15 17:41:09 m6kvm2 corosync[25411]: [VOTEQ ] Sending quorum callback, quorate = 1
Sep 15 17:41:09 m6kvm2 corosync[25411]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 15 17:41:09 m6kvm2 corosync[25411]: [TOTEM ] waiting_trans_ack changed to 0
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got procjoin message from cluster node 1 (r(0) ip(10.3.94.89) ) for pid 16239
Sep 15 17:41:09 m6kvm2 corosync[25411]: [CPG ] got procjoin message from cluster node 1 (r(0) ip(10.3.94.89) ) for pid 16239
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 16:57:46
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>I mean this is bad, but also great!
>>Cam you do a coredump of the whole thing and upload it somewhere with the version info
>>used (for dbgsym package)? That could help a lot.
I'll try to reproduce it again (with the full lock everywhere), and do the coredump.
I have tried the real time scheduling,
but I still have been able to reproduce the "lrm too long" for 60s (but as I'm restarting corosync each minute, I think it's unlocking
something at next corosync restart.)
this time it was blocked at the same time on a node in:
work {
...
} elsif ($state eq 'active') {
....
$self->update_lrm_status();
and another node in
if ($fence_request) {
$haenv->log('err', "node need to be fenced - releasing agent_lock\n");
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif (!$self->get_protected_ha_agent_lock()) {
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif ($self->{mode} eq 'maintenance') {
$self->set_local_status({ state => 'maintenance'});
}
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "aderumier" <aderumier@odiso.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 15 Septembre 2020 16:32:52
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/15/20 4:09 PM, Alexandre DERUMIER wrote:
>>> Can you try to give pmxcfs real time scheduling, e.g., by doing:
>>>
>>> # systemctl edit pve-cluster
>>>
>>> And then add snippet:
>>>
>>>
>>> [Service]
>>> CPUSchedulingPolicy=rr
>>> CPUSchedulingPriority=99
> yes, sure, I'll do it now
>
>
>> I'm currently digging the logs
>>> Is your most simplest/stable reproducer still a periodic restart of corosync in one node?
> yes, a simple "systemctl restart corosync" on 1 node each minute
>
>
>
> After 1hour, it's still locked.
>
> on other nodes, I still have pmxfs logs like:
>
I mean this is bad, but also great!
Cam you do a coredump of the whole thing and upload it somewhere with the version info
used (for dbgsym package)? That could help a lot.
> manual "pmxcfs -d"
> https://gist.github.com/aderumier/4cd91d17e1f8847b93ea5f621f257c2e
>
Hmm, the fuse connection of the previous one got into a weird state (or something is still
running) but I'd rather say this is a side-effect not directly connected to the real bug.
>
> some interesting dmesg about "pvesr"
>
> [Tue Sep 15 14:45:34 2020] INFO: task pvesr:19038 blocked for more than 120 seconds.
> [Tue Sep 15 14:45:34 2020] Tainted: P O 5.4.60-1-pve #1
> [Tue Sep 15 14:45:34 2020] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Tue Sep 15 14:45:34 2020] pvesr D 0 19038 1 0x00000080
> [Tue Sep 15 14:45:34 2020] Call Trace:
> [Tue Sep 15 14:45:34 2020] __schedule+0x2e6/0x6f0
> [Tue Sep 15 14:45:34 2020] ? filename_parentat.isra.57.part.58+0xf7/0x180
> [Tue Sep 15 14:45:34 2020] schedule+0x33/0xa0
> [Tue Sep 15 14:45:34 2020] rwsem_down_write_slowpath+0x2ed/0x4a0
> [Tue Sep 15 14:45:34 2020] down_write+0x3d/0x40
> [Tue Sep 15 14:45:34 2020] filename_create+0x8e/0x180
> [Tue Sep 15 14:45:34 2020] do_mkdirat+0x59/0x110
> [Tue Sep 15 14:45:34 2020] __x64_sys_mkdir+0x1b/0x20
> [Tue Sep 15 14:45:34 2020] do_syscall_64+0x57/0x190
> [Tue Sep 15 14:45:34 2020] entry_SYSCALL_64_after_hwframe+0x44/0xa9
>
hmm, hangs in mkdir (cluster wide locking)
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-16 8:30 ` Alexandre DERUMIER
@ 2020-09-16 8:53 ` Alexandre DERUMIER
[not found] ` <1894376736.864562.1600253445817.JavaMail.zimbra@odiso.com>
1 sibling, 0 replies; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-16 8:53 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: Thomas Lamprecht
my last mail was too long, here the backtrace of pmxcfs in gist
node1-bt -> where the corosync is restarted each minute
--------
https://gist.github.com/aderumier/ed21f22aa6ed9099ec0199255112f6b6
node2-bt -> other node hanging too
--------
https://gist.github.com/aderumier/31fb72b93e77a93fbaec975bc54dfb3a
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
[not found] ` <1894376736.864562.1600253445817.JavaMail.zimbra@odiso.com>
@ 2020-09-16 13:15 ` Alexandre DERUMIER
2020-09-16 14:45 ` Thomas Lamprecht
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-16 13:15 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: Thomas Lamprecht
I have reproduce it again, with pmxcfs in debug mode
corosync restart at 15:02:10, and it was already block on other nodes at 15:02:12
The pmxcfs was still logging after the lock.
here the log on node1 where corosync has been restarted
http://odisoweb1.odiso.net/pmxcfs-corosync.log
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-16 13:15 ` Alexandre DERUMIER
@ 2020-09-16 14:45 ` Thomas Lamprecht
2020-09-16 15:17 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-16 14:45 UTC (permalink / raw)
To: Alexandre DERUMIER, Proxmox VE development discussion
On 9/16/20 3:15 PM, Alexandre DERUMIER wrote:
> I have reproduce it again, with pmxcfs in debug mode
>
> corosync restart at 15:02:10, and it was already block on other nodes at 15:02:12
>
> The pmxcfs was still logging after the lock.
>
>
> here the log on node1 where corosync has been restarted
>
> http://odisoweb1.odiso.net/pmxcfs-corosync.log
>
thanks for those, I need a bit to sift through them. Seem like either dfsm gets
out of sync or we do not get a ACK reply from cpg_send.
A full core dump would be still nice, in gdb:
generate-core-file
PS: instead of manually switching to threads you can do:
thread apply all bt full
to get a backtrace for all threads in one command
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-16 14:45 ` Thomas Lamprecht
@ 2020-09-16 15:17 ` Alexandre DERUMIER
2020-09-17 9:21 ` Fabian Grünbichler
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-16 15:17 UTC (permalink / raw)
To: Thomas Lamprecht; +Cc: Proxmox VE development discussion
I have produce it again, with the coredump this time
restart corosync : 17:05:27
http://odisoweb1.odiso.net/pmxcfs-corosync2.log
bt full
https://gist.github.com/aderumier/466dcc4aedb795aaf0f308de0d1c652b
coredump
http://odisoweb1.odiso.net/core.7761.gz
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mercredi 16 Septembre 2020 16:45:12
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/16/20 3:15 PM, Alexandre DERUMIER wrote:
> I have reproduce it again, with pmxcfs in debug mode
>
> corosync restart at 15:02:10, and it was already block on other nodes at 15:02:12
>
> The pmxcfs was still logging after the lock.
>
>
> here the log on node1 where corosync has been restarted
>
> http://odisoweb1.odiso.net/pmxcfs-corosync.log
>
thanks for those, I need a bit to sift through them. Seem like either dfsm gets
out of sync or we do not get a ACK reply from cpg_send.
A full core dump would be still nice, in gdb:
generate-core-file
PS: instead of manually switching to threads you can do:
thread apply all bt full
to get a backtrace for all threads in one command
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-16 15:17 ` Alexandre DERUMIER
@ 2020-09-17 9:21 ` Fabian Grünbichler
2020-09-17 9:59 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Fabian Grünbichler @ 2020-09-17 9:21 UTC (permalink / raw)
To: Proxmox VE development discussion, Thomas Lamprecht
On September 16, 2020 5:17 pm, Alexandre DERUMIER wrote:
> I have produce it again, with the coredump this time
>
>
> restart corosync : 17:05:27
>
> http://odisoweb1.odiso.net/pmxcfs-corosync2.log
>
>
> bt full
>
> https://gist.github.com/aderumier/466dcc4aedb795aaf0f308de0d1c652b
>
>
> coredump
>
>
> http://odisoweb1.odiso.net/core.7761.gz
just a short update on this:
dcdb is stuck in START_SYNC mode, but nodeid 13 hasn't sent a STATE msg
(yet). this looks like either the START_SYNC message to node 13, or the
STATE response from it got lost or processed wrong. until the mode
switches to SYNCED (after all states have been received and the state
update went through), regular/normal messages can be sent, but the
incoming normal messages are queued and not processed. this is why the
fuse access blocks, it sends the request out, but the response ends up
in the queue.
status (the other thing running on top of dfsm) got correctly synced up
at the same time, so it's either a dcdb specific bug, or just bad luck
that one was affected and the other wasn't.
unfortunately even with debug enabled the logs don't contain much
information that would help (e.g., we don't log sending/receiving STATE
messages except when they look 'wrong'), so Thomas is trying to
reproduce this using your scenario here to improve turn around time. if
we can't reproduce it, we'll have to send you patches/patched debs with
increased logging to narrow down what is going on. if we can, than we
can hopefully find and fix the issue fast.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-17 9:21 ` Fabian Grünbichler
@ 2020-09-17 9:59 ` Alexandre DERUMIER
2020-09-17 10:02 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-17 9:59 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: Thomas Lamprecht
Thanks for the update.
>> if
>>we can't reproduce it, we'll have to send you patches/patched debs with
>>increased logging to narrow down what is going on. if we can, than we
>>can hopefully find and fix the issue fast.
No problem, I can install the patched deb if needed.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoyé: Jeudi 17 Septembre 2020 11:21:45
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 16, 2020 5:17 pm, Alexandre DERUMIER wrote:
> I have produce it again, with the coredump this time
>
>
> restart corosync : 17:05:27
>
> http://odisoweb1.odiso.net/pmxcfs-corosync2.log
>
>
> bt full
>
> https://gist.github.com/aderumier/466dcc4aedb795aaf0f308de0d1c652b
>
>
> coredump
>
>
> http://odisoweb1.odiso.net/core.7761.gz
just a short update on this:
dcdb is stuck in START_SYNC mode, but nodeid 13 hasn't sent a STATE msg
(yet). this looks like either the START_SYNC message to node 13, or the
STATE response from it got lost or processed wrong. until the mode
switches to SYNCED (after all states have been received and the state
update went through), regular/normal messages can be sent, but the
incoming normal messages are queued and not processed. this is why the
fuse access blocks, it sends the request out, but the response ends up
in the queue.
status (the other thing running on top of dfsm) got correctly synced up
at the same time, so it's either a dcdb specific bug, or just bad luck
that one was affected and the other wasn't.
unfortunately even with debug enabled the logs don't contain much
information that would help (e.g., we don't log sending/receiving STATE
messages except when they look 'wrong'), so Thomas is trying to
reproduce this using your scenario here to improve turn around time. if
we can't reproduce it, we'll have to send you patches/patched debs with
increased logging to narrow down what is going on. if we can, than we
can hopefully find and fix the issue fast.
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-17 9:59 ` Alexandre DERUMIER
@ 2020-09-17 10:02 ` Alexandre DERUMIER
2020-09-17 11:35 ` Thomas Lamprecht
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-17 10:02 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: Thomas Lamprecht
if needed, here my test script to reproduce it
node1 (restart corosync until node2 don't send the timestamp anymore)
-----
#!/bin/bash
for i in `seq 10000`; do
now=$(date +"%T")
echo "restart corosync : $now"
systemctl restart corosync
for j in {1..59}; do
last=$(cat /tmp/timestamp)
curr=`date '+%s'`
diff=$(($curr - $last))
if [ $diff -gt 20 ]; then
echo "too old"
exit 0
fi
sleep 1
done
done
node2 (write to /etc/pve/test each second, then send the last timestamp to node1)
-----
#!/bin/bash
for i in {1..10000};
do
now=$(date +"%T")
echo "Current time : $now"
curr=`date '+%s'`
ssh root@node1 "echo $curr > /tmp/timestamp"
echo "test" > /etc/pve/test
sleep 1
done
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoyé: Jeudi 17 Septembre 2020 11:59:32
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Thanks for the update.
>> if
>>we can't reproduce it, we'll have to send you patches/patched debs with
>>increased logging to narrow down what is going on. if we can, than we
>>can hopefully find and fix the issue fast.
No problem, I can install the patched deb if needed.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoyé: Jeudi 17 Septembre 2020 11:21:45
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 16, 2020 5:17 pm, Alexandre DERUMIER wrote:
> I have produce it again, with the coredump this time
>
>
> restart corosync : 17:05:27
>
> http://odisoweb1.odiso.net/pmxcfs-corosync2.log
>
>
> bt full
>
> https://gist.github.com/aderumier/466dcc4aedb795aaf0f308de0d1c652b
>
>
> coredump
>
>
> http://odisoweb1.odiso.net/core.7761.gz
just a short update on this:
dcdb is stuck in START_SYNC mode, but nodeid 13 hasn't sent a STATE msg
(yet). this looks like either the START_SYNC message to node 13, or the
STATE response from it got lost or processed wrong. until the mode
switches to SYNCED (after all states have been received and the state
update went through), regular/normal messages can be sent, but the
incoming normal messages are queued and not processed. this is why the
fuse access blocks, it sends the request out, but the response ends up
in the queue.
status (the other thing running on top of dfsm) got correctly synced up
at the same time, so it's either a dcdb specific bug, or just bad luck
that one was affected and the other wasn't.
unfortunately even with debug enabled the logs don't contain much
information that would help (e.g., we don't log sending/receiving STATE
messages except when they look 'wrong'), so Thomas is trying to
reproduce this using your scenario here to improve turn around time. if
we can't reproduce it, we'll have to send you patches/patched debs with
increased logging to narrow down what is going on. if we can, than we
can hopefully find and fix the issue fast.
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-17 10:02 ` Alexandre DERUMIER
@ 2020-09-17 11:35 ` Thomas Lamprecht
2020-09-20 23:54 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-17 11:35 UTC (permalink / raw)
To: Proxmox VE development discussion, Alexandre DERUMIER
On 9/17/20 12:02 PM, Alexandre DERUMIER wrote:
> if needed, here my test script to reproduce it
thanks, I'm now using this specific one, had a similar (but all nodes writes)
running here since ~ two hours without luck yet, lets see how this behaves.
>
> node1 (restart corosync until node2 don't send the timestamp anymore)
> -----
>
> #!/bin/bash
>
> for i in `seq 10000`; do
> now=$(date +"%T")
> echo "restart corosync : $now"
> systemctl restart corosync
> for j in {1..59}; do
> last=$(cat /tmp/timestamp)
> curr=`date '+%s'`
> diff=$(($curr - $last))
> if [ $diff -gt 20 ]; then
> echo "too old"
> exit 0
> fi
> sleep 1
> done
> done
>
>
>
> node2 (write to /etc/pve/test each second, then send the last timestamp to node1)
> -----
> #!/bin/bash
> for i in {1..10000};
> do
> now=$(date +"%T")
> echo "Current time : $now"
> curr=`date '+%s'`
> ssh root@node1 "echo $curr > /tmp/timestamp"
> echo "test" > /etc/pve/test
> sleep 1
> done
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-17 11:35 ` Thomas Lamprecht
@ 2020-09-20 23:54 ` Alexandre DERUMIER
2020-09-22 5:43 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-20 23:54 UTC (permalink / raw)
To: Thomas Lamprecht; +Cc: Proxmox VE development discussion
Hi,
I have done a new test, this time with "systemctl stop corosync", wait 15s, "systemctl start corosync", wait 15s.
I was able to reproduce it at corosync stop on node1, 1second later /etc/pve was locked on all other nodes.
I have started corosync 10min later on node1, and /etc/pve has become writeable again on all nodes
node1: corosync stop: 01:26:50
node2 : /etc/pve locked : 01:26:51
http://odisoweb1.odiso.net/corosync-stop.log
pmxcfs : bt full all threads:
https://gist.github.com/aderumier/c45af4ee73b80330367e416af858bc65
pmxcfs: coredump :http://odisoweb1.odiso.net/core.17995.gz
node1:corosync start: 01:35:36
http://odisoweb1.odiso.net/corosync-start.log
BTW, I have been contacted in pm on the forum by a user following this mailing thread,
and he had exactly the same problem with a 7 nodes cluster recently.
(shutting down 1 node, /etc/pve was locked until the node was restarted)
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "aderumier" <aderumier@odiso.com>
Envoyé: Jeudi 17 Septembre 2020 13:35:55
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/17/20 12:02 PM, Alexandre DERUMIER wrote:
> if needed, here my test script to reproduce it
thanks, I'm now using this specific one, had a similar (but all nodes writes)
running here since ~ two hours without luck yet, lets see how this behaves.
>
> node1 (restart corosync until node2 don't send the timestamp anymore)
> -----
>
> #!/bin/bash
>
> for i in `seq 10000`; do
> now=$(date +"%T")
> echo "restart corosync : $now"
> systemctl restart corosync
> for j in {1..59}; do
> last=$(cat /tmp/timestamp)
> curr=`date '+%s'`
> diff=$(($curr - $last))
> if [ $diff -gt 20 ]; then
> echo "too old"
> exit 0
> fi
> sleep 1
> done
> done
>
>
>
> node2 (write to /etc/pve/test each second, then send the last timestamp to node1)
> -----
> #!/bin/bash
> for i in {1..10000};
> do
> now=$(date +"%T")
> echo "Current time : $now"
> curr=`date '+%s'`
> ssh root@node1 "echo $curr > /tmp/timestamp"
> echo "test" > /etc/pve/test
> sleep 1
> done
>
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-20 23:54 ` Alexandre DERUMIER
@ 2020-09-22 5:43 ` Alexandre DERUMIER
2020-09-24 14:02 ` Fabian Grünbichler
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-22 5:43 UTC (permalink / raw)
To: Proxmox VE development discussion; +Cc: Thomas Lamprecht
I have done test with "kill -9 <pidofcorosync", and I have around 20s hang on other nodes,
but after that it's become available again.
So, it's really something when corosync is in shutdown phase, and pmxcfs is running.
So, for now, as workaround, I have changed
/lib/systemd/system/pve-cluster.service
#Wants=corosync.service
#Before=corosync.service
Requires=corosync.service
After=corosync.service
Like this, at shutdown, pve-cluster is stopped before corosync, and if I restart corosync, pve-cluster is stopped first.
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Lundi 21 Septembre 2020 01:54:59
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Hi,
I have done a new test, this time with "systemctl stop corosync", wait 15s, "systemctl start corosync", wait 15s.
I was able to reproduce it at corosync stop on node1, 1second later /etc/pve was locked on all other nodes.
I have started corosync 10min later on node1, and /etc/pve has become writeable again on all nodes
node1: corosync stop: 01:26:50
node2 : /etc/pve locked : 01:26:51
http://odisoweb1.odiso.net/corosync-stop.log
pmxcfs : bt full all threads:
https://gist.github.com/aderumier/c45af4ee73b80330367e416af858bc65
pmxcfs: coredump :http://odisoweb1.odiso.net/core.17995.gz
node1:corosync start: 01:35:36
http://odisoweb1.odiso.net/corosync-start.log
BTW, I have been contacted in pm on the forum by a user following this mailing thread,
and he had exactly the same problem with a 7 nodes cluster recently.
(shutting down 1 node, /etc/pve was locked until the node was restarted)
----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "aderumier" <aderumier@odiso.com>
Envoyé: Jeudi 17 Septembre 2020 13:35:55
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On 9/17/20 12:02 PM, Alexandre DERUMIER wrote:
> if needed, here my test script to reproduce it
thanks, I'm now using this specific one, had a similar (but all nodes writes)
running here since ~ two hours without luck yet, lets see how this behaves.
>
> node1 (restart corosync until node2 don't send the timestamp anymore)
> -----
>
> #!/bin/bash
>
> for i in `seq 10000`; do
> now=$(date +"%T")
> echo "restart corosync : $now"
> systemctl restart corosync
> for j in {1..59}; do
> last=$(cat /tmp/timestamp)
> curr=`date '+%s'`
> diff=$(($curr - $last))
> if [ $diff -gt 20 ]; then
> echo "too old"
> exit 0
> fi
> sleep 1
> done
> done
>
>
>
> node2 (write to /etc/pve/test each second, then send the last timestamp to node1)
> -----
> #!/bin/bash
> for i in {1..10000};
> do
> now=$(date +"%T")
> echo "Current time : $now"
> curr=`date '+%s'`
> ssh root@node1 "echo $curr > /tmp/timestamp"
> echo "test" > /etc/pve/test
> sleep 1
> done
>
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-22 5:43 ` Alexandre DERUMIER
@ 2020-09-24 14:02 ` Fabian Grünbichler
2020-09-24 14:29 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Fabian Grünbichler @ 2020-09-24 14:02 UTC (permalink / raw)
To: Proxmox VE development discussion
On September 22, 2020 7:43 am, Alexandre DERUMIER wrote:
> I have done test with "kill -9 <pidofcorosync", and I have around 20s hang on other nodes,
> but after that it's become available again.
>
>
> So, it's really something when corosync is in shutdown phase, and pmxcfs is running.
>
> So, for now, as workaround, I have changed
>
> /lib/systemd/system/pve-cluster.service
>
> #Wants=corosync.service
> #Before=corosync.service
> Requires=corosync.service
> After=corosync.service
>
>
> Like this, at shutdown, pve-cluster is stopped before corosync, and if I restart corosync, pve-cluster is stopped first.
if you are still able to test, it would be great if you could give the
following packages a spin (they only contain some extra debug prints
on message processing/sending):
http://download.proxmox.com/temp/pmxcfs-dbg/
64eb9dbd2f60fe319abaeece89a84fd5de1f05a8c38cb9871058ec1f55025486ec15b7c0053976159fe5c2518615fd80084925abf4d3a5061ea7d6edef264c36 pve-cluster_6.1-8_amd64.deb
04b557c7f0dc1aa2846b534d6afab70c2b8d4720ac307364e015d885e2e997b6dcaa54cad673b22d626d27cb053e5723510fde15d078d5fe1f262fc5486e6cef pve-cluster-dbgsym_6.1-8_amd64.deb
ideally, you could get the debug logs from all nodes, and the
coredump/bt from the node where pmxcfs hangs. thanks!
diff --git a/data/src/dfsm.c b/data/src/dfsm.c
index 529c7f9..e0bd93f 100644
--- a/data/src/dfsm.c
+++ b/data/src/dfsm.c
@@ -162,8 +162,8 @@ static void
dfsm_send_sync_message_abort(dfsm_t *dfsm)
{
g_return_if_fail(dfsm != NULL);
-
g_mutex_lock (&dfsm->sync_mutex);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_sync_message_abort - %" PRIu64" / %" PRIu64, dfsm->msgcount_rcvd, dfsm->msgcount);
dfsm->msgcount_rcvd = dfsm->msgcount;
g_cond_broadcast (&dfsm->sync_cond);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -181,6 +181,7 @@ dfsm_record_local_result(
g_mutex_lock (&dfsm->sync_mutex);
dfsm_result_t *rp = (dfsm_result_t *)g_hash_table_lookup(dfsm->results, &msg_count);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_record_local_result - %" PRIu64": %d", msg_count, msg_result);
if (rp) {
rp->result = msg_result;
rp->processed = processed;
@@ -235,6 +236,8 @@ dfsm_send_state_message_full(
g_return_val_if_fail(DFSM_VALID_STATE_MESSAGE(type), CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_state_message_full: type %d len %d", type, len);
+
dfsm_message_state_header_t header;
header.base.type = type;
header.base.subtype = 0;
@@ -317,6 +320,7 @@ dfsm_send_message_sync(
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_messag_sync: type NORMAL, msgtype %d, len %d", msgtype, len);
cs_error_t result = dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -335,10 +339,12 @@ dfsm_send_message_sync(
if (rp) {
g_mutex_lock (&dfsm->sync_mutex);
- while (dfsm->msgcount_rcvd < msgcount)
+ while (dfsm->msgcount_rcvd < msgcount) {
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: waiting for received messages %" PRIu64 " / %" PRIu64, dfsm->msgcount_rcvd, msgcount);
g_cond_wait (&dfsm->sync_cond, &dfsm->sync_mutex);
+ }
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: done waiting for received messages!");
-
g_hash_table_remove(dfsm->results, &rp->msgcount);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -685,9 +691,13 @@ dfsm_cpg_deliver_callback(
return;
}
+ cfs_dom_debug(dfsm->log_domain, "received message's header type is %d", base_header->type);
+
if (base_header->type == DFSM_MESSAGE_NORMAL) {
dfsm_message_normal_header_t *header = (dfsm_message_normal_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received normal message (type = %d, subtype = %d, %zd bytes)",
+ base_header->type, base_header->subtype, msg_len);
if (msg_len < sizeof(dfsm_message_normal_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short message (type = %d, subtype = %d, %zd bytes)",
@@ -704,6 +714,8 @@ dfsm_cpg_deliver_callback(
} else {
int msg_res = -1;
+ cfs_dom_debug(dfsm->log_domain, "deliver message %" PRIu64 " (subtype = %d, length = %zd)",
+ header->count, base_header->subtype, msg_len);
int res = dfsm->dfsm_callbacks->dfsm_deliver_fn(
dfsm, dfsm->data, &msg_res, nodeid, pid, base_header->subtype,
base_header->time, (uint8_t *)msg + sizeof(dfsm_message_normal_header_t),
@@ -724,6 +736,8 @@ dfsm_cpg_deliver_callback(
*/
dfsm_message_state_header_t *header = (dfsm_message_state_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received state message (type = %d, subtype = %d, %zd bytes), mode is %d",
+ base_header->type, base_header->subtype, msg_len, mode);
if (msg_len < sizeof(dfsm_message_state_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short state message (type = %d, subtype = %d, %zd bytes)",
@@ -744,6 +758,7 @@ dfsm_cpg_deliver_callback(
if (mode == DFSM_MODE_SYNCED) {
if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
for (int i = 0; i < dfsm->sync_info->node_count; i++)
dfsm->sync_info->nodes[i].synced = 1;
@@ -754,6 +769,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_VERIFY_REQUEST) {
+ cfs_dom_debug(dfsm->log_domain, "received verify request message");
if (msg_len != sizeof(dfsm->csum_counter)) {
cfs_dom_critical(dfsm->log_domain, "cpg received verify request with wrong length (%zd bytes) form node %d/%d", msg_len, nodeid, pid);
@@ -823,7 +839,6 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_START_SYNC) {
if (base_header->type == DFSM_MESSAGE_SYNC_START) {
-
if (nodeid != dfsm->lowest_nodeid) {
cfs_dom_critical(dfsm->log_domain, "ignore sync request from wrong member %d/%d",
nodeid, pid);
@@ -861,6 +876,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_STATE) {
+ cfs_dom_debug(dfsm->log_domain, "received state message for %d/%d", nodeid, pid);
dfsm_node_info_t *ni;
@@ -906,6 +922,8 @@ dfsm_cpg_deliver_callback(
goto leave;
}
+ } else {
+ cfs_dom_debug(dfsm->log_domain, "haven't received all states, waiting for more");
}
return;
@@ -914,6 +932,7 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_UPDATE) {
if (base_header->type == DFSM_MESSAGE_UPDATE) {
+ cfs_dom_debug(dfsm->log_domain, "received update message");
int res = dfsm->dfsm_callbacks->dfsm_process_update_fn(
dfsm, dfsm->data, dfsm->sync_info, nodeid, pid, msg, msg_len);
@@ -925,6 +944,7 @@ dfsm_cpg_deliver_callback(
} else if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
int res = dfsm->dfsm_callbacks->dfsm_commit_fn(dfsm, dfsm->data, dfsm->sync_info);
@@ -1104,6 +1124,7 @@ dfsm_cpg_confchg_callback(
size_t joined_list_entries)
{
cs_error_t result;
+ cfs_debug("dfsm_cpg_confchg_callback called");
dfsm_t *dfsm = NULL;
result = cpg_context_get(handle, (gpointer *)&dfsm);
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-24 14:02 ` Fabian Grünbichler
@ 2020-09-24 14:29 ` Alexandre DERUMIER
2020-09-24 18:07 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-24 14:29 UTC (permalink / raw)
To: Proxmox VE development discussion
Hi fabian,
>>if you are still able to test, it would be great if you could give the
>>following packages a spin (they only contain some extra debug prints
>>on message processing/sending):
Sure, no problem, I'm going to test it tonight.
>>ideally, you could get the debug logs from all nodes, and the
>>coredump/bt from the node where pmxcfs hangs. thanks!
ok,no problem.
I'll keep you in touch tomorrow.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Jeudi 24 Septembre 2020 16:02:04
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 22, 2020 7:43 am, Alexandre DERUMIER wrote:
> I have done test with "kill -9 <pidofcorosync", and I have around 20s hang on other nodes,
> but after that it's become available again.
>
>
> So, it's really something when corosync is in shutdown phase, and pmxcfs is running.
>
> So, for now, as workaround, I have changed
>
> /lib/systemd/system/pve-cluster.service
>
> #Wants=corosync.service
> #Before=corosync.service
> Requires=corosync.service
> After=corosync.service
>
>
> Like this, at shutdown, pve-cluster is stopped before corosync, and if I restart corosync, pve-cluster is stopped first.
if you are still able to test, it would be great if you could give the
following packages a spin (they only contain some extra debug prints
on message processing/sending):
http://download.proxmox.com/temp/pmxcfs-dbg/
64eb9dbd2f60fe319abaeece89a84fd5de1f05a8c38cb9871058ec1f55025486ec15b7c0053976159fe5c2518615fd80084925abf4d3a5061ea7d6edef264c36 pve-cluster_6.1-8_amd64.deb
04b557c7f0dc1aa2846b534d6afab70c2b8d4720ac307364e015d885e2e997b6dcaa54cad673b22d626d27cb053e5723510fde15d078d5fe1f262fc5486e6cef pve-cluster-dbgsym_6.1-8_amd64.deb
ideally, you could get the debug logs from all nodes, and the
coredump/bt from the node where pmxcfs hangs. thanks!
diff --git a/data/src/dfsm.c b/data/src/dfsm.c
index 529c7f9..e0bd93f 100644
--- a/data/src/dfsm.c
+++ b/data/src/dfsm.c
@@ -162,8 +162,8 @@ static void
dfsm_send_sync_message_abort(dfsm_t *dfsm)
{
g_return_if_fail(dfsm != NULL);
-
g_mutex_lock (&dfsm->sync_mutex);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_sync_message_abort - %" PRIu64" / %" PRIu64, dfsm->msgcount_rcvd, dfsm->msgcount);
dfsm->msgcount_rcvd = dfsm->msgcount;
g_cond_broadcast (&dfsm->sync_cond);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -181,6 +181,7 @@ dfsm_record_local_result(
g_mutex_lock (&dfsm->sync_mutex);
dfsm_result_t *rp = (dfsm_result_t *)g_hash_table_lookup(dfsm->results, &msg_count);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_record_local_result - %" PRIu64": %d", msg_count, msg_result);
if (rp) {
rp->result = msg_result;
rp->processed = processed;
@@ -235,6 +236,8 @@ dfsm_send_state_message_full(
g_return_val_if_fail(DFSM_VALID_STATE_MESSAGE(type), CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_state_message_full: type %d len %d", type, len);
+
dfsm_message_state_header_t header;
header.base.type = type;
header.base.subtype = 0;
@@ -317,6 +320,7 @@ dfsm_send_message_sync(
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_messag_sync: type NORMAL, msgtype %d, len %d", msgtype, len);
cs_error_t result = dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -335,10 +339,12 @@ dfsm_send_message_sync(
if (rp) {
g_mutex_lock (&dfsm->sync_mutex);
- while (dfsm->msgcount_rcvd < msgcount)
+ while (dfsm->msgcount_rcvd < msgcount) {
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: waiting for received messages %" PRIu64 " / %" PRIu64, dfsm->msgcount_rcvd, msgcount);
g_cond_wait (&dfsm->sync_cond, &dfsm->sync_mutex);
+ }
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: done waiting for received messages!");
-
g_hash_table_remove(dfsm->results, &rp->msgcount);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -685,9 +691,13 @@ dfsm_cpg_deliver_callback(
return;
}
+ cfs_dom_debug(dfsm->log_domain, "received message's header type is %d", base_header->type);
+
if (base_header->type == DFSM_MESSAGE_NORMAL) {
dfsm_message_normal_header_t *header = (dfsm_message_normal_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received normal message (type = %d, subtype = %d, %zd bytes)",
+ base_header->type, base_header->subtype, msg_len);
if (msg_len < sizeof(dfsm_message_normal_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short message (type = %d, subtype = %d, %zd bytes)",
@@ -704,6 +714,8 @@ dfsm_cpg_deliver_callback(
} else {
int msg_res = -1;
+ cfs_dom_debug(dfsm->log_domain, "deliver message %" PRIu64 " (subtype = %d, length = %zd)",
+ header->count, base_header->subtype, msg_len);
int res = dfsm->dfsm_callbacks->dfsm_deliver_fn(
dfsm, dfsm->data, &msg_res, nodeid, pid, base_header->subtype,
base_header->time, (uint8_t *)msg + sizeof(dfsm_message_normal_header_t),
@@ -724,6 +736,8 @@ dfsm_cpg_deliver_callback(
*/
dfsm_message_state_header_t *header = (dfsm_message_state_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received state message (type = %d, subtype = %d, %zd bytes), mode is %d",
+ base_header->type, base_header->subtype, msg_len, mode);
if (msg_len < sizeof(dfsm_message_state_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short state message (type = %d, subtype = %d, %zd bytes)",
@@ -744,6 +758,7 @@ dfsm_cpg_deliver_callback(
if (mode == DFSM_MODE_SYNCED) {
if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
for (int i = 0; i < dfsm->sync_info->node_count; i++)
dfsm->sync_info->nodes[i].synced = 1;
@@ -754,6 +769,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_VERIFY_REQUEST) {
+ cfs_dom_debug(dfsm->log_domain, "received verify request message");
if (msg_len != sizeof(dfsm->csum_counter)) {
cfs_dom_critical(dfsm->log_domain, "cpg received verify request with wrong length (%zd bytes) form node %d/%d", msg_len, nodeid, pid);
@@ -823,7 +839,6 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_START_SYNC) {
if (base_header->type == DFSM_MESSAGE_SYNC_START) {
-
if (nodeid != dfsm->lowest_nodeid) {
cfs_dom_critical(dfsm->log_domain, "ignore sync request from wrong member %d/%d",
nodeid, pid);
@@ -861,6 +876,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_STATE) {
+ cfs_dom_debug(dfsm->log_domain, "received state message for %d/%d", nodeid, pid);
dfsm_node_info_t *ni;
@@ -906,6 +922,8 @@ dfsm_cpg_deliver_callback(
goto leave;
}
+ } else {
+ cfs_dom_debug(dfsm->log_domain, "haven't received all states, waiting for more");
}
return;
@@ -914,6 +932,7 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_UPDATE) {
if (base_header->type == DFSM_MESSAGE_UPDATE) {
+ cfs_dom_debug(dfsm->log_domain, "received update message");
int res = dfsm->dfsm_callbacks->dfsm_process_update_fn(
dfsm, dfsm->data, dfsm->sync_info, nodeid, pid, msg, msg_len);
@@ -925,6 +944,7 @@ dfsm_cpg_deliver_callback(
} else if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
int res = dfsm->dfsm_callbacks->dfsm_commit_fn(dfsm, dfsm->data, dfsm->sync_info);
@@ -1104,6 +1124,7 @@ dfsm_cpg_confchg_callback(
size_t joined_list_entries)
{
cs_error_t result;
+ cfs_debug("dfsm_cpg_confchg_callback called");
dfsm_t *dfsm = NULL;
result = cpg_context_get(handle, (gpointer *)&dfsm);
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-24 14:29 ` Alexandre DERUMIER
@ 2020-09-24 18:07 ` Alexandre DERUMIER
2020-09-25 6:44 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-24 18:07 UTC (permalink / raw)
To: Proxmox VE development discussion
I was able to reproduce
stop corosync on node1 : 18:12:29
/etc/pve locked at 18:12:30
logs of all nodes are here:
http://odisoweb1.odiso.net/test1/
I don't have coredump, as my coworker have restarted pmxcfs too fast :/ . Sorry.
I'm going to launch another test with coredump this time
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Jeudi 24 Septembre 2020 16:29:17
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Hi fabian,
>>if you are still able to test, it would be great if you could give the
>>following packages a spin (they only contain some extra debug prints
>>on message processing/sending):
Sure, no problem, I'm going to test it tonight.
>>ideally, you could get the debug logs from all nodes, and the
>>coredump/bt from the node where pmxcfs hangs. thanks!
ok,no problem.
I'll keep you in touch tomorrow.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Jeudi 24 Septembre 2020 16:02:04
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 22, 2020 7:43 am, Alexandre DERUMIER wrote:
> I have done test with "kill -9 <pidofcorosync", and I have around 20s hang on other nodes,
> but after that it's become available again.
>
>
> So, it's really something when corosync is in shutdown phase, and pmxcfs is running.
>
> So, for now, as workaround, I have changed
>
> /lib/systemd/system/pve-cluster.service
>
> #Wants=corosync.service
> #Before=corosync.service
> Requires=corosync.service
> After=corosync.service
>
>
> Like this, at shutdown, pve-cluster is stopped before corosync, and if I restart corosync, pve-cluster is stopped first.
if you are still able to test, it would be great if you could give the
following packages a spin (they only contain some extra debug prints
on message processing/sending):
http://download.proxmox.com/temp/pmxcfs-dbg/
64eb9dbd2f60fe319abaeece89a84fd5de1f05a8c38cb9871058ec1f55025486ec15b7c0053976159fe5c2518615fd80084925abf4d3a5061ea7d6edef264c36 pve-cluster_6.1-8_amd64.deb
04b557c7f0dc1aa2846b534d6afab70c2b8d4720ac307364e015d885e2e997b6dcaa54cad673b22d626d27cb053e5723510fde15d078d5fe1f262fc5486e6cef pve-cluster-dbgsym_6.1-8_amd64.deb
ideally, you could get the debug logs from all nodes, and the
coredump/bt from the node where pmxcfs hangs. thanks!
diff --git a/data/src/dfsm.c b/data/src/dfsm.c
index 529c7f9..e0bd93f 100644
--- a/data/src/dfsm.c
+++ b/data/src/dfsm.c
@@ -162,8 +162,8 @@ static void
dfsm_send_sync_message_abort(dfsm_t *dfsm)
{
g_return_if_fail(dfsm != NULL);
-
g_mutex_lock (&dfsm->sync_mutex);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_sync_message_abort - %" PRIu64" / %" PRIu64, dfsm->msgcount_rcvd, dfsm->msgcount);
dfsm->msgcount_rcvd = dfsm->msgcount;
g_cond_broadcast (&dfsm->sync_cond);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -181,6 +181,7 @@ dfsm_record_local_result(
g_mutex_lock (&dfsm->sync_mutex);
dfsm_result_t *rp = (dfsm_result_t *)g_hash_table_lookup(dfsm->results, &msg_count);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_record_local_result - %" PRIu64": %d", msg_count, msg_result);
if (rp) {
rp->result = msg_result;
rp->processed = processed;
@@ -235,6 +236,8 @@ dfsm_send_state_message_full(
g_return_val_if_fail(DFSM_VALID_STATE_MESSAGE(type), CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_state_message_full: type %d len %d", type, len);
+
dfsm_message_state_header_t header;
header.base.type = type;
header.base.subtype = 0;
@@ -317,6 +320,7 @@ dfsm_send_message_sync(
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_messag_sync: type NORMAL, msgtype %d, len %d", msgtype, len);
cs_error_t result = dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -335,10 +339,12 @@ dfsm_send_message_sync(
if (rp) {
g_mutex_lock (&dfsm->sync_mutex);
- while (dfsm->msgcount_rcvd < msgcount)
+ while (dfsm->msgcount_rcvd < msgcount) {
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: waiting for received messages %" PRIu64 " / %" PRIu64, dfsm->msgcount_rcvd, msgcount);
g_cond_wait (&dfsm->sync_cond, &dfsm->sync_mutex);
+ }
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: done waiting for received messages!");
-
g_hash_table_remove(dfsm->results, &rp->msgcount);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -685,9 +691,13 @@ dfsm_cpg_deliver_callback(
return;
}
+ cfs_dom_debug(dfsm->log_domain, "received message's header type is %d", base_header->type);
+
if (base_header->type == DFSM_MESSAGE_NORMAL) {
dfsm_message_normal_header_t *header = (dfsm_message_normal_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received normal message (type = %d, subtype = %d, %zd bytes)",
+ base_header->type, base_header->subtype, msg_len);
if (msg_len < sizeof(dfsm_message_normal_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short message (type = %d, subtype = %d, %zd bytes)",
@@ -704,6 +714,8 @@ dfsm_cpg_deliver_callback(
} else {
int msg_res = -1;
+ cfs_dom_debug(dfsm->log_domain, "deliver message %" PRIu64 " (subtype = %d, length = %zd)",
+ header->count, base_header->subtype, msg_len);
int res = dfsm->dfsm_callbacks->dfsm_deliver_fn(
dfsm, dfsm->data, &msg_res, nodeid, pid, base_header->subtype,
base_header->time, (uint8_t *)msg + sizeof(dfsm_message_normal_header_t),
@@ -724,6 +736,8 @@ dfsm_cpg_deliver_callback(
*/
dfsm_message_state_header_t *header = (dfsm_message_state_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received state message (type = %d, subtype = %d, %zd bytes), mode is %d",
+ base_header->type, base_header->subtype, msg_len, mode);
if (msg_len < sizeof(dfsm_message_state_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short state message (type = %d, subtype = %d, %zd bytes)",
@@ -744,6 +758,7 @@ dfsm_cpg_deliver_callback(
if (mode == DFSM_MODE_SYNCED) {
if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
for (int i = 0; i < dfsm->sync_info->node_count; i++)
dfsm->sync_info->nodes[i].synced = 1;
@@ -754,6 +769,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_VERIFY_REQUEST) {
+ cfs_dom_debug(dfsm->log_domain, "received verify request message");
if (msg_len != sizeof(dfsm->csum_counter)) {
cfs_dom_critical(dfsm->log_domain, "cpg received verify request with wrong length (%zd bytes) form node %d/%d", msg_len, nodeid, pid);
@@ -823,7 +839,6 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_START_SYNC) {
if (base_header->type == DFSM_MESSAGE_SYNC_START) {
-
if (nodeid != dfsm->lowest_nodeid) {
cfs_dom_critical(dfsm->log_domain, "ignore sync request from wrong member %d/%d",
nodeid, pid);
@@ -861,6 +876,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_STATE) {
+ cfs_dom_debug(dfsm->log_domain, "received state message for %d/%d", nodeid, pid);
dfsm_node_info_t *ni;
@@ -906,6 +922,8 @@ dfsm_cpg_deliver_callback(
goto leave;
}
+ } else {
+ cfs_dom_debug(dfsm->log_domain, "haven't received all states, waiting for more");
}
return;
@@ -914,6 +932,7 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_UPDATE) {
if (base_header->type == DFSM_MESSAGE_UPDATE) {
+ cfs_dom_debug(dfsm->log_domain, "received update message");
int res = dfsm->dfsm_callbacks->dfsm_process_update_fn(
dfsm, dfsm->data, dfsm->sync_info, nodeid, pid, msg, msg_len);
@@ -925,6 +944,7 @@ dfsm_cpg_deliver_callback(
} else if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
int res = dfsm->dfsm_callbacks->dfsm_commit_fn(dfsm, dfsm->data, dfsm->sync_info);
@@ -1104,6 +1124,7 @@ dfsm_cpg_confchg_callback(
size_t joined_list_entries)
{
cs_error_t result;
+ cfs_debug("dfsm_cpg_confchg_callback called");
dfsm_t *dfsm = NULL;
result = cpg_context_get(handle, (gpointer *)&dfsm);
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-24 18:07 ` Alexandre DERUMIER
@ 2020-09-25 6:44 ` Alexandre DERUMIER
2020-09-25 7:15 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-25 6:44 UTC (permalink / raw)
To: Proxmox VE development discussion
Another test this morning with the coredump available
http://odisoweb1.odiso.net/test2/
Something different this time, it has happened on corosync start
node1 (corosync start)
------
start corosync : 08:06:56
node2 (/etc/pve locked)
-----
Current time : 08:07:01
I had a warning on coredump
(gdb) generate-core-file
warning: target file /proc/35248/cmdline contained unexpected null characters
warning: Memory read failed for corefile section, 4096 bytes at 0xffffffffff600000.
Saved corefile core.35248
I hope it's ok.
I'll do another test this morning
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Jeudi 24 Septembre 2020 20:07:43
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
I was able to reproduce
stop corosync on node1 : 18:12:29
/etc/pve locked at 18:12:30
logs of all nodes are here:
http://odisoweb1.odiso.net/test1/
I don't have coredump, as my coworker have restarted pmxcfs too fast :/ . Sorry.
I'm going to launch another test with coredump this time
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Jeudi 24 Septembre 2020 16:29:17
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Hi fabian,
>>if you are still able to test, it would be great if you could give the
>>following packages a spin (they only contain some extra debug prints
>>on message processing/sending):
Sure, no problem, I'm going to test it tonight.
>>ideally, you could get the debug logs from all nodes, and the
>>coredump/bt from the node where pmxcfs hangs. thanks!
ok,no problem.
I'll keep you in touch tomorrow.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Jeudi 24 Septembre 2020 16:02:04
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 22, 2020 7:43 am, Alexandre DERUMIER wrote:
> I have done test with "kill -9 <pidofcorosync", and I have around 20s hang on other nodes,
> but after that it's become available again.
>
>
> So, it's really something when corosync is in shutdown phase, and pmxcfs is running.
>
> So, for now, as workaround, I have changed
>
> /lib/systemd/system/pve-cluster.service
>
> #Wants=corosync.service
> #Before=corosync.service
> Requires=corosync.service
> After=corosync.service
>
>
> Like this, at shutdown, pve-cluster is stopped before corosync, and if I restart corosync, pve-cluster is stopped first.
if you are still able to test, it would be great if you could give the
following packages a spin (they only contain some extra debug prints
on message processing/sending):
http://download.proxmox.com/temp/pmxcfs-dbg/
64eb9dbd2f60fe319abaeece89a84fd5de1f05a8c38cb9871058ec1f55025486ec15b7c0053976159fe5c2518615fd80084925abf4d3a5061ea7d6edef264c36 pve-cluster_6.1-8_amd64.deb
04b557c7f0dc1aa2846b534d6afab70c2b8d4720ac307364e015d885e2e997b6dcaa54cad673b22d626d27cb053e5723510fde15d078d5fe1f262fc5486e6cef pve-cluster-dbgsym_6.1-8_amd64.deb
ideally, you could get the debug logs from all nodes, and the
coredump/bt from the node where pmxcfs hangs. thanks!
diff --git a/data/src/dfsm.c b/data/src/dfsm.c
index 529c7f9..e0bd93f 100644
--- a/data/src/dfsm.c
+++ b/data/src/dfsm.c
@@ -162,8 +162,8 @@ static void
dfsm_send_sync_message_abort(dfsm_t *dfsm)
{
g_return_if_fail(dfsm != NULL);
-
g_mutex_lock (&dfsm->sync_mutex);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_sync_message_abort - %" PRIu64" / %" PRIu64, dfsm->msgcount_rcvd, dfsm->msgcount);
dfsm->msgcount_rcvd = dfsm->msgcount;
g_cond_broadcast (&dfsm->sync_cond);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -181,6 +181,7 @@ dfsm_record_local_result(
g_mutex_lock (&dfsm->sync_mutex);
dfsm_result_t *rp = (dfsm_result_t *)g_hash_table_lookup(dfsm->results, &msg_count);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_record_local_result - %" PRIu64": %d", msg_count, msg_result);
if (rp) {
rp->result = msg_result;
rp->processed = processed;
@@ -235,6 +236,8 @@ dfsm_send_state_message_full(
g_return_val_if_fail(DFSM_VALID_STATE_MESSAGE(type), CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_state_message_full: type %d len %d", type, len);
+
dfsm_message_state_header_t header;
header.base.type = type;
header.base.subtype = 0;
@@ -317,6 +320,7 @@ dfsm_send_message_sync(
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_messag_sync: type NORMAL, msgtype %d, len %d", msgtype, len);
cs_error_t result = dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -335,10 +339,12 @@ dfsm_send_message_sync(
if (rp) {
g_mutex_lock (&dfsm->sync_mutex);
- while (dfsm->msgcount_rcvd < msgcount)
+ while (dfsm->msgcount_rcvd < msgcount) {
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: waiting for received messages %" PRIu64 " / %" PRIu64, dfsm->msgcount_rcvd, msgcount);
g_cond_wait (&dfsm->sync_cond, &dfsm->sync_mutex);
+ }
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: done waiting for received messages!");
-
g_hash_table_remove(dfsm->results, &rp->msgcount);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -685,9 +691,13 @@ dfsm_cpg_deliver_callback(
return;
}
+ cfs_dom_debug(dfsm->log_domain, "received message's header type is %d", base_header->type);
+
if (base_header->type == DFSM_MESSAGE_NORMAL) {
dfsm_message_normal_header_t *header = (dfsm_message_normal_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received normal message (type = %d, subtype = %d, %zd bytes)",
+ base_header->type, base_header->subtype, msg_len);
if (msg_len < sizeof(dfsm_message_normal_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short message (type = %d, subtype = %d, %zd bytes)",
@@ -704,6 +714,8 @@ dfsm_cpg_deliver_callback(
} else {
int msg_res = -1;
+ cfs_dom_debug(dfsm->log_domain, "deliver message %" PRIu64 " (subtype = %d, length = %zd)",
+ header->count, base_header->subtype, msg_len);
int res = dfsm->dfsm_callbacks->dfsm_deliver_fn(
dfsm, dfsm->data, &msg_res, nodeid, pid, base_header->subtype,
base_header->time, (uint8_t *)msg + sizeof(dfsm_message_normal_header_t),
@@ -724,6 +736,8 @@ dfsm_cpg_deliver_callback(
*/
dfsm_message_state_header_t *header = (dfsm_message_state_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received state message (type = %d, subtype = %d, %zd bytes), mode is %d",
+ base_header->type, base_header->subtype, msg_len, mode);
if (msg_len < sizeof(dfsm_message_state_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short state message (type = %d, subtype = %d, %zd bytes)",
@@ -744,6 +758,7 @@ dfsm_cpg_deliver_callback(
if (mode == DFSM_MODE_SYNCED) {
if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
for (int i = 0; i < dfsm->sync_info->node_count; i++)
dfsm->sync_info->nodes[i].synced = 1;
@@ -754,6 +769,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_VERIFY_REQUEST) {
+ cfs_dom_debug(dfsm->log_domain, "received verify request message");
if (msg_len != sizeof(dfsm->csum_counter)) {
cfs_dom_critical(dfsm->log_domain, "cpg received verify request with wrong length (%zd bytes) form node %d/%d", msg_len, nodeid, pid);
@@ -823,7 +839,6 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_START_SYNC) {
if (base_header->type == DFSM_MESSAGE_SYNC_START) {
-
if (nodeid != dfsm->lowest_nodeid) {
cfs_dom_critical(dfsm->log_domain, "ignore sync request from wrong member %d/%d",
nodeid, pid);
@@ -861,6 +876,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_STATE) {
+ cfs_dom_debug(dfsm->log_domain, "received state message for %d/%d", nodeid, pid);
dfsm_node_info_t *ni;
@@ -906,6 +922,8 @@ dfsm_cpg_deliver_callback(
goto leave;
}
+ } else {
+ cfs_dom_debug(dfsm->log_domain, "haven't received all states, waiting for more");
}
return;
@@ -914,6 +932,7 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_UPDATE) {
if (base_header->type == DFSM_MESSAGE_UPDATE) {
+ cfs_dom_debug(dfsm->log_domain, "received update message");
int res = dfsm->dfsm_callbacks->dfsm_process_update_fn(
dfsm, dfsm->data, dfsm->sync_info, nodeid, pid, msg, msg_len);
@@ -925,6 +944,7 @@ dfsm_cpg_deliver_callback(
} else if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
int res = dfsm->dfsm_callbacks->dfsm_commit_fn(dfsm, dfsm->data, dfsm->sync_info);
@@ -1104,6 +1124,7 @@ dfsm_cpg_confchg_callback(
size_t joined_list_entries)
{
cs_error_t result;
+ cfs_debug("dfsm_cpg_confchg_callback called");
dfsm_t *dfsm = NULL;
result = cpg_context_get(handle, (gpointer *)&dfsm);
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-25 6:44 ` Alexandre DERUMIER
@ 2020-09-25 7:15 ` Alexandre DERUMIER
2020-09-25 9:19 ` Fabian Grünbichler
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-25 7:15 UTC (permalink / raw)
To: Proxmox VE development discussion
Another hang, this time on corosync stop, coredump available
http://odisoweb1.odiso.net/test3/
node1
----
stop corosync : 09:03:10
node2: /etc/pve locked
------
Current time : 09:03:10
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Vendredi 25 Septembre 2020 08:44:24
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Another test this morning with the coredump available
http://odisoweb1.odiso.net/test2/
Something different this time, it has happened on corosync start
node1 (corosync start)
------
start corosync : 08:06:56
node2 (/etc/pve locked)
-----
Current time : 08:07:01
I had a warning on coredump
(gdb) generate-core-file
warning: target file /proc/35248/cmdline contained unexpected null characters
warning: Memory read failed for corefile section, 4096 bytes at 0xffffffffff600000.
Saved corefile core.35248
I hope it's ok.
I'll do another test this morning
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Jeudi 24 Septembre 2020 20:07:43
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
I was able to reproduce
stop corosync on node1 : 18:12:29
/etc/pve locked at 18:12:30
logs of all nodes are here:
http://odisoweb1.odiso.net/test1/
I don't have coredump, as my coworker have restarted pmxcfs too fast :/ . Sorry.
I'm going to launch another test with coredump this time
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Jeudi 24 Septembre 2020 16:29:17
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Hi fabian,
>>if you are still able to test, it would be great if you could give the
>>following packages a spin (they only contain some extra debug prints
>>on message processing/sending):
Sure, no problem, I'm going to test it tonight.
>>ideally, you could get the debug logs from all nodes, and the
>>coredump/bt from the node where pmxcfs hangs. thanks!
ok,no problem.
I'll keep you in touch tomorrow.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Jeudi 24 Septembre 2020 16:02:04
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 22, 2020 7:43 am, Alexandre DERUMIER wrote:
> I have done test with "kill -9 <pidofcorosync", and I have around 20s hang on other nodes,
> but after that it's become available again.
>
>
> So, it's really something when corosync is in shutdown phase, and pmxcfs is running.
>
> So, for now, as workaround, I have changed
>
> /lib/systemd/system/pve-cluster.service
>
> #Wants=corosync.service
> #Before=corosync.service
> Requires=corosync.service
> After=corosync.service
>
>
> Like this, at shutdown, pve-cluster is stopped before corosync, and if I restart corosync, pve-cluster is stopped first.
if you are still able to test, it would be great if you could give the
following packages a spin (they only contain some extra debug prints
on message processing/sending):
http://download.proxmox.com/temp/pmxcfs-dbg/
64eb9dbd2f60fe319abaeece89a84fd5de1f05a8c38cb9871058ec1f55025486ec15b7c0053976159fe5c2518615fd80084925abf4d3a5061ea7d6edef264c36 pve-cluster_6.1-8_amd64.deb
04b557c7f0dc1aa2846b534d6afab70c2b8d4720ac307364e015d885e2e997b6dcaa54cad673b22d626d27cb053e5723510fde15d078d5fe1f262fc5486e6cef pve-cluster-dbgsym_6.1-8_amd64.deb
ideally, you could get the debug logs from all nodes, and the
coredump/bt from the node where pmxcfs hangs. thanks!
diff --git a/data/src/dfsm.c b/data/src/dfsm.c
index 529c7f9..e0bd93f 100644
--- a/data/src/dfsm.c
+++ b/data/src/dfsm.c
@@ -162,8 +162,8 @@ static void
dfsm_send_sync_message_abort(dfsm_t *dfsm)
{
g_return_if_fail(dfsm != NULL);
-
g_mutex_lock (&dfsm->sync_mutex);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_sync_message_abort - %" PRIu64" / %" PRIu64, dfsm->msgcount_rcvd, dfsm->msgcount);
dfsm->msgcount_rcvd = dfsm->msgcount;
g_cond_broadcast (&dfsm->sync_cond);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -181,6 +181,7 @@ dfsm_record_local_result(
g_mutex_lock (&dfsm->sync_mutex);
dfsm_result_t *rp = (dfsm_result_t *)g_hash_table_lookup(dfsm->results, &msg_count);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_record_local_result - %" PRIu64": %d", msg_count, msg_result);
if (rp) {
rp->result = msg_result;
rp->processed = processed;
@@ -235,6 +236,8 @@ dfsm_send_state_message_full(
g_return_val_if_fail(DFSM_VALID_STATE_MESSAGE(type), CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_state_message_full: type %d len %d", type, len);
+
dfsm_message_state_header_t header;
header.base.type = type;
header.base.subtype = 0;
@@ -317,6 +320,7 @@ dfsm_send_message_sync(
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_messag_sync: type NORMAL, msgtype %d, len %d", msgtype, len);
cs_error_t result = dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -335,10 +339,12 @@ dfsm_send_message_sync(
if (rp) {
g_mutex_lock (&dfsm->sync_mutex);
- while (dfsm->msgcount_rcvd < msgcount)
+ while (dfsm->msgcount_rcvd < msgcount) {
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: waiting for received messages %" PRIu64 " / %" PRIu64, dfsm->msgcount_rcvd, msgcount);
g_cond_wait (&dfsm->sync_cond, &dfsm->sync_mutex);
+ }
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: done waiting for received messages!");
-
g_hash_table_remove(dfsm->results, &rp->msgcount);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -685,9 +691,13 @@ dfsm_cpg_deliver_callback(
return;
}
+ cfs_dom_debug(dfsm->log_domain, "received message's header type is %d", base_header->type);
+
if (base_header->type == DFSM_MESSAGE_NORMAL) {
dfsm_message_normal_header_t *header = (dfsm_message_normal_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received normal message (type = %d, subtype = %d, %zd bytes)",
+ base_header->type, base_header->subtype, msg_len);
if (msg_len < sizeof(dfsm_message_normal_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short message (type = %d, subtype = %d, %zd bytes)",
@@ -704,6 +714,8 @@ dfsm_cpg_deliver_callback(
} else {
int msg_res = -1;
+ cfs_dom_debug(dfsm->log_domain, "deliver message %" PRIu64 " (subtype = %d, length = %zd)",
+ header->count, base_header->subtype, msg_len);
int res = dfsm->dfsm_callbacks->dfsm_deliver_fn(
dfsm, dfsm->data, &msg_res, nodeid, pid, base_header->subtype,
base_header->time, (uint8_t *)msg + sizeof(dfsm_message_normal_header_t),
@@ -724,6 +736,8 @@ dfsm_cpg_deliver_callback(
*/
dfsm_message_state_header_t *header = (dfsm_message_state_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received state message (type = %d, subtype = %d, %zd bytes), mode is %d",
+ base_header->type, base_header->subtype, msg_len, mode);
if (msg_len < sizeof(dfsm_message_state_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short state message (type = %d, subtype = %d, %zd bytes)",
@@ -744,6 +758,7 @@ dfsm_cpg_deliver_callback(
if (mode == DFSM_MODE_SYNCED) {
if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
for (int i = 0; i < dfsm->sync_info->node_count; i++)
dfsm->sync_info->nodes[i].synced = 1;
@@ -754,6 +769,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_VERIFY_REQUEST) {
+ cfs_dom_debug(dfsm->log_domain, "received verify request message");
if (msg_len != sizeof(dfsm->csum_counter)) {
cfs_dom_critical(dfsm->log_domain, "cpg received verify request with wrong length (%zd bytes) form node %d/%d", msg_len, nodeid, pid);
@@ -823,7 +839,6 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_START_SYNC) {
if (base_header->type == DFSM_MESSAGE_SYNC_START) {
-
if (nodeid != dfsm->lowest_nodeid) {
cfs_dom_critical(dfsm->log_domain, "ignore sync request from wrong member %d/%d",
nodeid, pid);
@@ -861,6 +876,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_STATE) {
+ cfs_dom_debug(dfsm->log_domain, "received state message for %d/%d", nodeid, pid);
dfsm_node_info_t *ni;
@@ -906,6 +922,8 @@ dfsm_cpg_deliver_callback(
goto leave;
}
+ } else {
+ cfs_dom_debug(dfsm->log_domain, "haven't received all states, waiting for more");
}
return;
@@ -914,6 +932,7 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_UPDATE) {
if (base_header->type == DFSM_MESSAGE_UPDATE) {
+ cfs_dom_debug(dfsm->log_domain, "received update message");
int res = dfsm->dfsm_callbacks->dfsm_process_update_fn(
dfsm, dfsm->data, dfsm->sync_info, nodeid, pid, msg, msg_len);
@@ -925,6 +944,7 @@ dfsm_cpg_deliver_callback(
} else if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
int res = dfsm->dfsm_callbacks->dfsm_commit_fn(dfsm, dfsm->data, dfsm->sync_info);
@@ -1104,6 +1124,7 @@ dfsm_cpg_confchg_callback(
size_t joined_list_entries)
{
cs_error_t result;
+ cfs_debug("dfsm_cpg_confchg_callback called");
dfsm_t *dfsm = NULL;
result = cpg_context_get(handle, (gpointer *)&dfsm);
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-25 7:15 ` Alexandre DERUMIER
@ 2020-09-25 9:19 ` Fabian Grünbichler
2020-09-25 9:46 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Fabian Grünbichler @ 2020-09-25 9:19 UTC (permalink / raw)
To: Proxmox VE development discussion
On September 25, 2020 9:15 am, Alexandre DERUMIER wrote:
>
> Another hang, this time on corosync stop, coredump available
>
> http://odisoweb1.odiso.net/test3/
>
>
> node1
> ----
> stop corosync : 09:03:10
>
> node2: /etc/pve locked
> ------
> Current time : 09:03:10
thanks, these all indicate the same symptoms:
1. cluster config changes (corosync goes down/comes back up in this case)
2. pmxcfs starts sync process
3. all (online) nodes receive sync request for dcdb and status
4. all nodes send state for dcdb and status via CPG
5. all nodes receive state for dcdb and status from all nodes except one (13 in test 2, 10 in test 3)
in step 5, there is no trace of the message on the receiving side, even
though the sending node does not log an error. as before, the hang is
just a side-effect of the state machine ending up in a state that should
be short-lived (syncing, waiting for state from all nodes) with no
progress. the code and theory say that this should not happen, as either
sending the state fails triggering the node to leave the CPG (restarting
the sync), or a node drops out of quorum (triggering a config change,
which triggers restarting the sync), or we get all states from all nodes
and the sync proceeds. this looks to me like a fundamental
assumption/guarantee does not hold..
I will rebuild once more modifying the send code a bit to log a lot more
details when sending state messages, it would be great if you could
repeat with that as we are still unable to reproduce the issue.
hopefully those logs will then indicate whether this is a corosync/knet
bug, or if the issue is in our state machine code somewhere. so far it
looks more like the former..
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-25 9:19 ` Fabian Grünbichler
@ 2020-09-25 9:46 ` Alexandre DERUMIER
2020-09-25 12:51 ` Fabian Grünbichler
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-25 9:46 UTC (permalink / raw)
To: Proxmox VE development discussion
>>I will rebuild once more modifying the send code a bit to log a lot more
>>details when sending state messages, it would be great if you could
>>repeat with that as we are still unable to reproduce the issue.
ok, no problem, I'm able to easily reproduce it, I'll do new test when you'll
send the new version.
(and thanks again to debugging this, because It's really beyond my competence)
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Vendredi 25 Septembre 2020 11:19:04
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 25, 2020 9:15 am, Alexandre DERUMIER wrote:
>
> Another hang, this time on corosync stop, coredump available
>
> http://odisoweb1.odiso.net/test3/
>
>
> node1
> ----
> stop corosync : 09:03:10
>
> node2: /etc/pve locked
> ------
> Current time : 09:03:10
thanks, these all indicate the same symptoms:
1. cluster config changes (corosync goes down/comes back up in this case)
2. pmxcfs starts sync process
3. all (online) nodes receive sync request for dcdb and status
4. all nodes send state for dcdb and status via CPG
5. all nodes receive state for dcdb and status from all nodes except one (13 in test 2, 10 in test 3)
in step 5, there is no trace of the message on the receiving side, even
though the sending node does not log an error. as before, the hang is
just a side-effect of the state machine ending up in a state that should
be short-lived (syncing, waiting for state from all nodes) with no
progress. the code and theory say that this should not happen, as either
sending the state fails triggering the node to leave the CPG (restarting
the sync), or a node drops out of quorum (triggering a config change,
which triggers restarting the sync), or we get all states from all nodes
and the sync proceeds. this looks to me like a fundamental
assumption/guarantee does not hold..
I will rebuild once more modifying the send code a bit to log a lot more
details when sending state messages, it would be great if you could
repeat with that as we are still unable to reproduce the issue.
hopefully those logs will then indicate whether this is a corosync/knet
bug, or if the issue is in our state machine code somewhere. so far it
looks more like the former..
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-25 9:46 ` Alexandre DERUMIER
@ 2020-09-25 12:51 ` Fabian Grünbichler
2020-09-25 16:29 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Fabian Grünbichler @ 2020-09-25 12:51 UTC (permalink / raw)
To: Proxmox VE development discussion
On September 25, 2020 11:46 am, Alexandre DERUMIER wrote:
>>>I will rebuild once more modifying the send code a bit to log a lot more
>>>details when sending state messages, it would be great if you could
>>>repeat with that as we are still unable to reproduce the issue.
>
> ok, no problem, I'm able to easily reproduce it, I'll do new test when you'll
> send the new version.
>
> (and thanks again to debugging this, because It's really beyond my competence)
same procedure as last time, same place, new checksums:
6b5e2defe543a874e0f7a883e40b279c997438dde158566c4c93c11ea531aef924ed2e4eb2506b5064e49ec9bdd4ebe7acd0b9278e9286eac527b0b15a43d8d7 pve-cluster_6.1-8_amd64.deb
d57ddc08824055826ee15c9c255690d9140e43f8d5164949108f0dc483a2d181b2bda76f0e7f47202699a062342c0cf0bba8f2ae0f7c5411af9967ef051050a0 pve-cluster-dbgsym_6.1-8_amd64.deb
I found one (unfortunately unrelated) bug in our error handling, so
there's that at least ;)
diff --git a/data/src/dfsm.c b/data/src/dfsm.c
index 529c7f9..f3397a0 100644
--- a/data/src/dfsm.c
+++ b/data/src/dfsm.c
@@ -162,8 +162,8 @@ static void
dfsm_send_sync_message_abort(dfsm_t *dfsm)
{
g_return_if_fail(dfsm != NULL);
-
g_mutex_lock (&dfsm->sync_mutex);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_sync_message_abort - %" PRIu64" / %" PRIu64, dfsm->msgcount_rcvd, dfsm->msgcount);
dfsm->msgcount_rcvd = dfsm->msgcount;
g_cond_broadcast (&dfsm->sync_cond);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -181,6 +181,7 @@ dfsm_record_local_result(
g_mutex_lock (&dfsm->sync_mutex);
dfsm_result_t *rp = (dfsm_result_t *)g_hash_table_lookup(dfsm->results, &msg_count);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_record_local_result - %" PRIu64": %d", msg_count, msg_result);
if (rp) {
rp->result = msg_result;
rp->processed = processed;
@@ -224,6 +225,48 @@ loop:
return result;
}
+static cs_error_t
+dfsm_send_message_full_debug_state(
+ dfsm_t *dfsm,
+ struct iovec *iov,
+ unsigned int len,
+ int retry)
+{
+ g_return_val_if_fail(dfsm != NULL, CS_ERR_INVALID_PARAM);
+ g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
+
+ struct timespec tvreq = { .tv_sec = 0, .tv_nsec = 100000000 };
+ cs_error_t result;
+ int retries = 0;
+ cfs_dom_message(dfsm->log_domain, "send state message debug");
+ cfs_dom_message(dfsm->log_domain, "cpg_handle %" PRIu64, dfsm->cpg_handle);
+ for (int i = 0; i < len; i++)
+ cfs_dom_message(dfsm->log_domain, "iov[%d] len %zd", i, iov[i].iov_len);
+loop:
+ cfs_dom_message(dfsm->log_domain, "send state message loop body");
+
+ result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len);
+
+ cfs_dom_message(dfsm->log_domain, "send state message result: %d", result);
+ if (retry && result == CS_ERR_TRY_AGAIN) {
+ nanosleep(&tvreq, NULL);
+ ++retries;
+ if ((retries % 10) == 0)
+ cfs_dom_message(dfsm->log_domain, "cpg_send_message retry %d", retries);
+ if (retries < 100)
+ goto loop;
+ }
+
+ if (retries)
+ cfs_dom_message(dfsm->log_domain, "cpg_send_message retried %d times", retries);
+
+ if (result != CS_OK &&
+ (!retry || result != CS_ERR_TRY_AGAIN))
+ cfs_dom_critical(dfsm->log_domain, "cpg_send_message failed: %d", result);
+
+ return result;
+}
+
static cs_error_t
dfsm_send_state_message_full(
dfsm_t *dfsm,
@@ -235,6 +278,8 @@ dfsm_send_state_message_full(
g_return_val_if_fail(DFSM_VALID_STATE_MESSAGE(type), CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_state_message_full: type %d len %d", type, len);
+
dfsm_message_state_header_t header;
header.base.type = type;
header.base.subtype = 0;
@@ -252,7 +297,7 @@ dfsm_send_state_message_full(
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
- return dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
+ return type == DFSM_MESSAGE_STATE ? dfsm_send_message_full_debug_state(dfsm, real_iov, len + 1, 1) : dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
}
cs_error_t
@@ -317,6 +362,7 @@ dfsm_send_message_sync(
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_messag_sync: type NORMAL, count %" PRIu64 ", msgtype %d, len %d", msgcount, msgtype, len);
cs_error_t result = dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -335,10 +381,12 @@ dfsm_send_message_sync(
if (rp) {
g_mutex_lock (&dfsm->sync_mutex);
- while (dfsm->msgcount_rcvd < msgcount)
+ while (dfsm->msgcount_rcvd < msgcount) {
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: waiting for received messages %" PRIu64 " / %" PRIu64, dfsm->msgcount_rcvd, msgcount);
g_cond_wait (&dfsm->sync_cond, &dfsm->sync_mutex);
+ }
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: done waiting for received messages!");
-
g_hash_table_remove(dfsm->results, &rp->msgcount);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -685,9 +733,13 @@ dfsm_cpg_deliver_callback(
return;
}
+ cfs_dom_debug(dfsm->log_domain, "received message's header type is %d", base_header->type);
+
if (base_header->type == DFSM_MESSAGE_NORMAL) {
dfsm_message_normal_header_t *header = (dfsm_message_normal_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received normal message (type = %d, subtype = %d, %zd bytes)",
+ base_header->type, base_header->subtype, msg_len);
if (msg_len < sizeof(dfsm_message_normal_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short message (type = %d, subtype = %d, %zd bytes)",
@@ -704,6 +756,8 @@ dfsm_cpg_deliver_callback(
} else {
int msg_res = -1;
+ cfs_dom_debug(dfsm->log_domain, "deliver message %" PRIu64 " (subtype = %d, length = %zd)",
+ header->count, base_header->subtype, msg_len);
int res = dfsm->dfsm_callbacks->dfsm_deliver_fn(
dfsm, dfsm->data, &msg_res, nodeid, pid, base_header->subtype,
base_header->time, (uint8_t *)msg + sizeof(dfsm_message_normal_header_t),
@@ -724,6 +778,8 @@ dfsm_cpg_deliver_callback(
*/
dfsm_message_state_header_t *header = (dfsm_message_state_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received state message (type = %d, subtype = %d, %zd bytes), mode is %d",
+ base_header->type, base_header->subtype, msg_len, mode);
if (msg_len < sizeof(dfsm_message_state_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short state message (type = %d, subtype = %d, %zd bytes)",
@@ -744,6 +800,7 @@ dfsm_cpg_deliver_callback(
if (mode == DFSM_MODE_SYNCED) {
if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
for (int i = 0; i < dfsm->sync_info->node_count; i++)
dfsm->sync_info->nodes[i].synced = 1;
@@ -754,6 +811,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_VERIFY_REQUEST) {
+ cfs_dom_debug(dfsm->log_domain, "received verify request message");
if (msg_len != sizeof(dfsm->csum_counter)) {
cfs_dom_critical(dfsm->log_domain, "cpg received verify request with wrong length (%zd bytes) form node %d/%d", msg_len, nodeid, pid);
@@ -823,7 +881,6 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_START_SYNC) {
if (base_header->type == DFSM_MESSAGE_SYNC_START) {
-
if (nodeid != dfsm->lowest_nodeid) {
cfs_dom_critical(dfsm->log_domain, "ignore sync request from wrong member %d/%d",
nodeid, pid);
@@ -861,6 +918,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_STATE) {
+ cfs_dom_debug(dfsm->log_domain, "received state message for %d/%d", nodeid, pid);
dfsm_node_info_t *ni;
@@ -906,6 +964,8 @@ dfsm_cpg_deliver_callback(
goto leave;
}
+ } else {
+ cfs_dom_debug(dfsm->log_domain, "haven't received all states, waiting for more");
}
return;
@@ -914,6 +974,7 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_UPDATE) {
if (base_header->type == DFSM_MESSAGE_UPDATE) {
+ cfs_dom_debug(dfsm->log_domain, "received update message");
int res = dfsm->dfsm_callbacks->dfsm_process_update_fn(
dfsm, dfsm->data, dfsm->sync_info, nodeid, pid, msg, msg_len);
@@ -925,6 +986,7 @@ dfsm_cpg_deliver_callback(
} else if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
int res = dfsm->dfsm_callbacks->dfsm_commit_fn(dfsm, dfsm->data, dfsm->sync_info);
@@ -1104,6 +1166,7 @@ dfsm_cpg_confchg_callback(
size_t joined_list_entries)
{
cs_error_t result;
+ cfs_debug("dfsm_cpg_confchg_callback called");
dfsm_t *dfsm = NULL;
result = cpg_context_get(handle, (gpointer *)&dfsm);
@@ -1190,7 +1253,7 @@ dfsm_cpg_confchg_callback(
dfsm_set_mode(dfsm, DFSM_MODE_START_SYNC);
if (lowest_nodeid == dfsm->nodeid) {
- if (!dfsm_send_state_message_full(dfsm, DFSM_MESSAGE_SYNC_START, NULL, 0)) {
+ if (dfsm_send_state_message_full(dfsm, DFSM_MESSAGE_SYNC_START, NULL, 0) != CS_OK) {
cfs_dom_critical(dfsm->log_domain, "failed to send SYNC_START message");
goto leave;
}
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-25 12:51 ` Fabian Grünbichler
@ 2020-09-25 16:29 ` Alexandre DERUMIER
2020-09-28 9:17 ` Fabian Grünbichler
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-25 16:29 UTC (permalink / raw)
To: Proxmox VE development discussion
here a new hang:
http://odisoweb1.odiso.net/test4/
This time on corosync start.
node1:
-----
start corosync : 17:22:02
node2
-----
/etc/pve locked 17:22:07
Something new: where doing coredump or bt-full on pmxcfs on node1,
this have unlocked /etc/pve on other nodes
/etc/pve unlocked(with coredump or bt-full): 17:57:40
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Vendredi 25 Septembre 2020 14:51:30
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 25, 2020 11:46 am, Alexandre DERUMIER wrote:
>>>I will rebuild once more modifying the send code a bit to log a lot more
>>>details when sending state messages, it would be great if you could
>>>repeat with that as we are still unable to reproduce the issue.
>
> ok, no problem, I'm able to easily reproduce it, I'll do new test when you'll
> send the new version.
>
> (and thanks again to debugging this, because It's really beyond my competence)
same procedure as last time, same place, new checksums:
6b5e2defe543a874e0f7a883e40b279c997438dde158566c4c93c11ea531aef924ed2e4eb2506b5064e49ec9bdd4ebe7acd0b9278e9286eac527b0b15a43d8d7 pve-cluster_6.1-8_amd64.deb
d57ddc08824055826ee15c9c255690d9140e43f8d5164949108f0dc483a2d181b2bda76f0e7f47202699a062342c0cf0bba8f2ae0f7c5411af9967ef051050a0 pve-cluster-dbgsym_6.1-8_amd64.deb
I found one (unfortunately unrelated) bug in our error handling, so
there's that at least ;)
diff --git a/data/src/dfsm.c b/data/src/dfsm.c
index 529c7f9..f3397a0 100644
--- a/data/src/dfsm.c
+++ b/data/src/dfsm.c
@@ -162,8 +162,8 @@ static void
dfsm_send_sync_message_abort(dfsm_t *dfsm)
{
g_return_if_fail(dfsm != NULL);
-
g_mutex_lock (&dfsm->sync_mutex);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_sync_message_abort - %" PRIu64" / %" PRIu64, dfsm->msgcount_rcvd, dfsm->msgcount);
dfsm->msgcount_rcvd = dfsm->msgcount;
g_cond_broadcast (&dfsm->sync_cond);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -181,6 +181,7 @@ dfsm_record_local_result(
g_mutex_lock (&dfsm->sync_mutex);
dfsm_result_t *rp = (dfsm_result_t *)g_hash_table_lookup(dfsm->results, &msg_count);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_record_local_result - %" PRIu64": %d", msg_count, msg_result);
if (rp) {
rp->result = msg_result;
rp->processed = processed;
@@ -224,6 +225,48 @@ loop:
return result;
}
+static cs_error_t
+dfsm_send_message_full_debug_state(
+ dfsm_t *dfsm,
+ struct iovec *iov,
+ unsigned int len,
+ int retry)
+{
+ g_return_val_if_fail(dfsm != NULL, CS_ERR_INVALID_PARAM);
+ g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
+
+ struct timespec tvreq = { .tv_sec = 0, .tv_nsec = 100000000 };
+ cs_error_t result;
+ int retries = 0;
+ cfs_dom_message(dfsm->log_domain, "send state message debug");
+ cfs_dom_message(dfsm->log_domain, "cpg_handle %" PRIu64, dfsm->cpg_handle);
+ for (int i = 0; i < len; i++)
+ cfs_dom_message(dfsm->log_domain, "iov[%d] len %zd", i, iov[i].iov_len);
+loop:
+ cfs_dom_message(dfsm->log_domain, "send state message loop body");
+
+ result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len);
+
+ cfs_dom_message(dfsm->log_domain, "send state message result: %d", result);
+ if (retry && result == CS_ERR_TRY_AGAIN) {
+ nanosleep(&tvreq, NULL);
+ ++retries;
+ if ((retries % 10) == 0)
+ cfs_dom_message(dfsm->log_domain, "cpg_send_message retry %d", retries);
+ if (retries < 100)
+ goto loop;
+ }
+
+ if (retries)
+ cfs_dom_message(dfsm->log_domain, "cpg_send_message retried %d times", retries);
+
+ if (result != CS_OK &&
+ (!retry || result != CS_ERR_TRY_AGAIN))
+ cfs_dom_critical(dfsm->log_domain, "cpg_send_message failed: %d", result);
+
+ return result;
+}
+
static cs_error_t
dfsm_send_state_message_full(
dfsm_t *dfsm,
@@ -235,6 +278,8 @@ dfsm_send_state_message_full(
g_return_val_if_fail(DFSM_VALID_STATE_MESSAGE(type), CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_state_message_full: type %d len %d", type, len);
+
dfsm_message_state_header_t header;
header.base.type = type;
header.base.subtype = 0;
@@ -252,7 +297,7 @@ dfsm_send_state_message_full(
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
- return dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
+ return type == DFSM_MESSAGE_STATE ? dfsm_send_message_full_debug_state(dfsm, real_iov, len + 1, 1) : dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
}
cs_error_t
@@ -317,6 +362,7 @@ dfsm_send_message_sync(
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_messag_sync: type NORMAL, count %" PRIu64 ", msgtype %d, len %d", msgcount, msgtype, len);
cs_error_t result = dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -335,10 +381,12 @@ dfsm_send_message_sync(
if (rp) {
g_mutex_lock (&dfsm->sync_mutex);
- while (dfsm->msgcount_rcvd < msgcount)
+ while (dfsm->msgcount_rcvd < msgcount) {
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: waiting for received messages %" PRIu64 " / %" PRIu64, dfsm->msgcount_rcvd, msgcount);
g_cond_wait (&dfsm->sync_cond, &dfsm->sync_mutex);
+ }
+ cfs_dom_debug(dfsm->log_domain, "dfsm_send_message_sync: done waiting for received messages!");
-
g_hash_table_remove(dfsm->results, &rp->msgcount);
g_mutex_unlock (&dfsm->sync_mutex);
@@ -685,9 +733,13 @@ dfsm_cpg_deliver_callback(
return;
}
+ cfs_dom_debug(dfsm->log_domain, "received message's header type is %d", base_header->type);
+
if (base_header->type == DFSM_MESSAGE_NORMAL) {
dfsm_message_normal_header_t *header = (dfsm_message_normal_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received normal message (type = %d, subtype = %d, %zd bytes)",
+ base_header->type, base_header->subtype, msg_len);
if (msg_len < sizeof(dfsm_message_normal_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short message (type = %d, subtype = %d, %zd bytes)",
@@ -704,6 +756,8 @@ dfsm_cpg_deliver_callback(
} else {
int msg_res = -1;
+ cfs_dom_debug(dfsm->log_domain, "deliver message %" PRIu64 " (subtype = %d, length = %zd)",
+ header->count, base_header->subtype, msg_len);
int res = dfsm->dfsm_callbacks->dfsm_deliver_fn(
dfsm, dfsm->data, &msg_res, nodeid, pid, base_header->subtype,
base_header->time, (uint8_t *)msg + sizeof(dfsm_message_normal_header_t),
@@ -724,6 +778,8 @@ dfsm_cpg_deliver_callback(
*/
dfsm_message_state_header_t *header = (dfsm_message_state_header_t *)msg;
+ cfs_dom_debug(dfsm->log_domain, "received state message (type = %d, subtype = %d, %zd bytes), mode is %d",
+ base_header->type, base_header->subtype, msg_len, mode);
if (msg_len < sizeof(dfsm_message_state_header_t)) {
cfs_dom_critical(dfsm->log_domain, "received short state message (type = %d, subtype = %d, %zd bytes)",
@@ -744,6 +800,7 @@ dfsm_cpg_deliver_callback(
if (mode == DFSM_MODE_SYNCED) {
if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
for (int i = 0; i < dfsm->sync_info->node_count; i++)
dfsm->sync_info->nodes[i].synced = 1;
@@ -754,6 +811,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_VERIFY_REQUEST) {
+ cfs_dom_debug(dfsm->log_domain, "received verify request message");
if (msg_len != sizeof(dfsm->csum_counter)) {
cfs_dom_critical(dfsm->log_domain, "cpg received verify request with wrong length (%zd bytes) form node %d/%d", msg_len, nodeid, pid);
@@ -823,7 +881,6 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_START_SYNC) {
if (base_header->type == DFSM_MESSAGE_SYNC_START) {
-
if (nodeid != dfsm->lowest_nodeid) {
cfs_dom_critical(dfsm->log_domain, "ignore sync request from wrong member %d/%d",
nodeid, pid);
@@ -861,6 +918,7 @@ dfsm_cpg_deliver_callback(
return;
} else if (base_header->type == DFSM_MESSAGE_STATE) {
+ cfs_dom_debug(dfsm->log_domain, "received state message for %d/%d", nodeid, pid);
dfsm_node_info_t *ni;
@@ -906,6 +964,8 @@ dfsm_cpg_deliver_callback(
goto leave;
}
+ } else {
+ cfs_dom_debug(dfsm->log_domain, "haven't received all states, waiting for more");
}
return;
@@ -914,6 +974,7 @@ dfsm_cpg_deliver_callback(
} else if (mode == DFSM_MODE_UPDATE) {
if (base_header->type == DFSM_MESSAGE_UPDATE) {
+ cfs_dom_debug(dfsm->log_domain, "received update message");
int res = dfsm->dfsm_callbacks->dfsm_process_update_fn(
dfsm, dfsm->data, dfsm->sync_info, nodeid, pid, msg, msg_len);
@@ -925,6 +986,7 @@ dfsm_cpg_deliver_callback(
} else if (base_header->type == DFSM_MESSAGE_UPDATE_COMPLETE) {
+ cfs_dom_debug(dfsm->log_domain, "received update complete message");
int res = dfsm->dfsm_callbacks->dfsm_commit_fn(dfsm, dfsm->data, dfsm->sync_info);
@@ -1104,6 +1166,7 @@ dfsm_cpg_confchg_callback(
size_t joined_list_entries)
{
cs_error_t result;
+ cfs_debug("dfsm_cpg_confchg_callback called");
dfsm_t *dfsm = NULL;
result = cpg_context_get(handle, (gpointer *)&dfsm);
@@ -1190,7 +1253,7 @@ dfsm_cpg_confchg_callback(
dfsm_set_mode(dfsm, DFSM_MODE_START_SYNC);
if (lowest_nodeid == dfsm->nodeid) {
- if (!dfsm_send_state_message_full(dfsm, DFSM_MESSAGE_SYNC_START, NULL, 0)) {
+ if (dfsm_send_state_message_full(dfsm, DFSM_MESSAGE_SYNC_START, NULL, 0) != CS_OK) {
cfs_dom_critical(dfsm->log_domain, "failed to send SYNC_START message");
goto leave;
}
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-25 16:29 ` Alexandre DERUMIER
@ 2020-09-28 9:17 ` Fabian Grünbichler
2020-09-28 9:35 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Fabian Grünbichler @ 2020-09-28 9:17 UTC (permalink / raw)
To: Proxmox VE development discussion
On September 25, 2020 6:29 pm, Alexandre DERUMIER wrote:
> here a new hang:
>
> http://odisoweb1.odiso.net/test4/
okay, so at least we now know something strange inside pmxcfs is going
on, and not inside corosync - we never reach the part where the broken
node (again #13 in this hang!) sends out the state message:
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: dfsm mode is 1 (dfsm.c:706:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: received message's header type is 1 (dfsm.c:736:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: received state message (type = 1, subtype = 0, 32 bytes), mode is 1 (dfsm.c:782:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] notice: received sync request (epoch 1/23389/000000A8) (dfsm.c:890:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [main] debug: enter dfsm_release_sync_resources (dfsm.c:640:dfsm_release_sync_resources)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: enter dcdb_get_state 00000000365345C2 5F6E0B1F (dcdb.c:444:dcdb_get_state)
[...]
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: dfsm_send_state_message_full: type 2 len 1 (dfsm.c:281:dfsm_send_state_message_full)
this should be followed by output like this (from the -unblock log,
where the sync went through):
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: send state message debug (dfsm.c:241:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: cpg_handle 5109470997161967616 (dfsm.c:242:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: iov[0] len 32 (dfsm.c:244:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: iov[1] len 64792 (dfsm.c:244:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: send state message loop body (dfsm.c:246:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: send state message result: 1 (dfsm.c:250:dfsm_send_message_full_debug_state)
but this never happens. the relevant code from our patched dfsm.c
(reordered):
static cs_error_t
dfsm_send_state_message_full(
dfsm_t *dfsm,
uint16_t type,
struct iovec *iov,
unsigned int len)
{
g_return_val_if_fail(dfsm != NULL, CS_ERR_INVALID_PARAM);
g_return_val_if_fail(DFSM_VALID_STATE_MESSAGE(type), CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
// this message is still logged
cfs_dom_debug(dfsm->log_domain, "dfsm_send_state_message_full: type %d len %d", type, len);
// everything below this point might not have happened anymore
dfsm_message_state_header_t header;
header.base.type = type;
header.base.subtype = 0;
header.base.protocol_version = dfsm->protocol_version;
header.base.time = time(NULL);
header.base.reserved = 0;
header.epoch = dfsm->sync_epoch;
struct iovec real_iov[len + 1];
real_iov[0].iov_base = (char *)&header;
real_iov[0].iov_len = sizeof(header);
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
return type == DFSM_MESSAGE_STATE ? dfsm_send_message_full_debug_state(dfsm, real_iov, len + 1, 1) : dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
}
static cs_error_t
dfsm_send_message_full_debug_state(
dfsm_t *dfsm,
struct iovec *iov,
unsigned int len,
int retry)
{
g_return_val_if_fail(dfsm != NULL, CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
struct timespec tvreq = { .tv_sec = 0, .tv_nsec = 100000000 };
cs_error_t result;
int retries = 0;
// this message is not visible in log
// we don't know how far above this we managed to run
cfs_dom_message(dfsm->log_domain, "send state message debug");
cfs_dom_message(dfsm->log_domain, "cpg_handle %" PRIu64, dfsm->cpg_handle);
for (int i = 0; i < len; i++)
cfs_dom_message(dfsm->log_domain, "iov[%d] len %zd", i, iov[i].iov_len);
loop:
cfs_dom_message(dfsm->log_domain, "send state message loop body");
result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len);
cfs_dom_message(dfsm->log_domain, "send state message result: %d", result);
if (retry && result == CS_ERR_TRY_AGAIN) {
nanosleep(&tvreq, NULL);
++retries;
if ((retries % 10) == 0)
cfs_dom_message(dfsm->log_domain, "cpg_send_message retry %d", retries);
if (retries < 100)
goto loop;
}
if (retries)
cfs_dom_message(dfsm->log_domain, "cpg_send_message retried %d times", retries);
if (result != CS_OK &&
(!retry || result != CS_ERR_TRY_AGAIN))
cfs_dom_critical(dfsm->log_domain, "cpg_send_message failed: %d", result);
return result;
}
I don't see much that could go wrong inside dfsm_send_state_message_full
after the debug log statement (it's just filling out the header and iov
structs, and calling 'dfsm_send_message_full_debug_state' since type ==
2 == DFSM_MESSAGE_STATE).
inside dfsm_send_message_full_debug, before the first log statement we
only check for dfsm or the message length/content being NULL/0, all of
which can't really happen with that call path. also, in that case we'd
return CS_ERR_INVALID_PARAM , which would bubble up into the delivery
callback and cause us to leave CPG, which would again be visible in the
logs..
but, just to make sure, could you reproduce the issue once more, and
then (with debug symbols installed) run
$ gdb -ex 'set pagination 0' -ex 'thread apply all bt' --batch -p $(pidof pmxcfs)
on all nodes at the same time? this should minimize the fallout and show
us whether the thread that logged the first part of sending the state is
still around on the node that triggers the hang..
> Something new: where doing coredump or bt-full on pmxcfs on node1,
> this have unlocked /etc/pve on other nodes
>
> /etc/pve unlocked(with coredump or bt-full): 17:57:40
this looks like you bt-ed corosync this time around? if so, than this is
expected:
- attach gdb to corosync
- corosync blocks
- other nodes notice corosync is gone on node X
- config change triggered
- sync restarts on all nodes, does not trigger bug this time
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-28 9:17 ` Fabian Grünbichler
@ 2020-09-28 9:35 ` Alexandre DERUMIER
2020-09-28 15:59 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-28 9:35 UTC (permalink / raw)
To: Proxmox VE development discussion
>>but, just to make sure, could you reproduce the issue once more, and
>>then (with debug symbols installed) run
>>
>>$ gdb -ex 'set pagination 0' -ex 'thread apply all bt' --batch -p $(pidof pmxcfs)
>>
>>on all nodes at the same time? this should minimize the fallout and show
>>us whether the thread that logged the first part of sending the state is
>>still around on the node that triggers the hang..
ok, no problem, I'll do a new test today
>>this looks like you bt-ed corosync this time around? if so, than this is
>>expected:
>>
>>- attach gdb to corosync
>>- corosync blocks
>>- other nodes notice corosync is gone on node X
>>- config change triggered
>>- sync restarts on all nodes, does not trigger bug this time
ok, thanks, make sense. I didn't notice this previously, but this time it was with corosync started.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Lundi 28 Septembre 2020 11:17:37
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 25, 2020 6:29 pm, Alexandre DERUMIER wrote:
> here a new hang:
>
> http://odisoweb1.odiso.net/test4/
okay, so at least we now know something strange inside pmxcfs is going
on, and not inside corosync - we never reach the part where the broken
node (again #13 in this hang!) sends out the state message:
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: dfsm mode is 1 (dfsm.c:706:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: received message's header type is 1 (dfsm.c:736:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: received state message (type = 1, subtype = 0, 32 bytes), mode is 1 (dfsm.c:782:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] notice: received sync request (epoch 1/23389/000000A8) (dfsm.c:890:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [main] debug: enter dfsm_release_sync_resources (dfsm.c:640:dfsm_release_sync_resources)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: enter dcdb_get_state 00000000365345C2 5F6E0B1F (dcdb.c:444:dcdb_get_state)
[...]
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: dfsm_send_state_message_full: type 2 len 1 (dfsm.c:281:dfsm_send_state_message_full)
this should be followed by output like this (from the -unblock log,
where the sync went through):
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: send state message debug (dfsm.c:241:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: cpg_handle 5109470997161967616 (dfsm.c:242:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: iov[0] len 32 (dfsm.c:244:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: iov[1] len 64792 (dfsm.c:244:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: send state message loop body (dfsm.c:246:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: send state message result: 1 (dfsm.c:250:dfsm_send_message_full_debug_state)
but this never happens. the relevant code from our patched dfsm.c
(reordered):
static cs_error_t
dfsm_send_state_message_full(
dfsm_t *dfsm,
uint16_t type,
struct iovec *iov,
unsigned int len)
{
g_return_val_if_fail(dfsm != NULL, CS_ERR_INVALID_PARAM);
g_return_val_if_fail(DFSM_VALID_STATE_MESSAGE(type), CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
// this message is still logged
cfs_dom_debug(dfsm->log_domain, "dfsm_send_state_message_full: type %d len %d", type, len);
// everything below this point might not have happened anymore
dfsm_message_state_header_t header;
header.base.type = type;
header.base.subtype = 0;
header.base.protocol_version = dfsm->protocol_version;
header.base.time = time(NULL);
header.base.reserved = 0;
header.epoch = dfsm->sync_epoch;
struct iovec real_iov[len + 1];
real_iov[0].iov_base = (char *)&header;
real_iov[0].iov_len = sizeof(header);
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
return type == DFSM_MESSAGE_STATE ? dfsm_send_message_full_debug_state(dfsm, real_iov, len + 1, 1) : dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
}
static cs_error_t
dfsm_send_message_full_debug_state(
dfsm_t *dfsm,
struct iovec *iov,
unsigned int len,
int retry)
{
g_return_val_if_fail(dfsm != NULL, CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
struct timespec tvreq = { .tv_sec = 0, .tv_nsec = 100000000 };
cs_error_t result;
int retries = 0;
// this message is not visible in log
// we don't know how far above this we managed to run
cfs_dom_message(dfsm->log_domain, "send state message debug");
cfs_dom_message(dfsm->log_domain, "cpg_handle %" PRIu64, dfsm->cpg_handle);
for (int i = 0; i < len; i++)
cfs_dom_message(dfsm->log_domain, "iov[%d] len %zd", i, iov[i].iov_len);
loop:
cfs_dom_message(dfsm->log_domain, "send state message loop body");
result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len);
cfs_dom_message(dfsm->log_domain, "send state message result: %d", result);
if (retry && result == CS_ERR_TRY_AGAIN) {
nanosleep(&tvreq, NULL);
++retries;
if ((retries % 10) == 0)
cfs_dom_message(dfsm->log_domain, "cpg_send_message retry %d", retries);
if (retries < 100)
goto loop;
}
if (retries)
cfs_dom_message(dfsm->log_domain, "cpg_send_message retried %d times", retries);
if (result != CS_OK &&
(!retry || result != CS_ERR_TRY_AGAIN))
cfs_dom_critical(dfsm->log_domain, "cpg_send_message failed: %d", result);
return result;
}
I don't see much that could go wrong inside dfsm_send_state_message_full
after the debug log statement (it's just filling out the header and iov
structs, and calling 'dfsm_send_message_full_debug_state' since type ==
2 == DFSM_MESSAGE_STATE).
inside dfsm_send_message_full_debug, before the first log statement we
only check for dfsm or the message length/content being NULL/0, all of
which can't really happen with that call path. also, in that case we'd
return CS_ERR_INVALID_PARAM , which would bubble up into the delivery
callback and cause us to leave CPG, which would again be visible in the
logs..
but, just to make sure, could you reproduce the issue once more, and
then (with debug symbols installed) run
$ gdb -ex 'set pagination 0' -ex 'thread apply all bt' --batch -p $(pidof pmxcfs)
on all nodes at the same time? this should minimize the fallout and show
us whether the thread that logged the first part of sending the state is
still around on the node that triggers the hang..
> Something new: where doing coredump or bt-full on pmxcfs on node1,
> this have unlocked /etc/pve on other nodes
>
> /etc/pve unlocked(with coredump or bt-full): 17:57:40
this looks like you bt-ed corosync this time around? if so, than this is
expected:
- attach gdb to corosync
- corosync blocks
- other nodes notice corosync is gone on node X
- config change triggered
- sync restarts on all nodes, does not trigger bug this time
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-28 9:35 ` Alexandre DERUMIER
@ 2020-09-28 15:59 ` Alexandre DERUMIER
2020-09-29 5:30 ` Alexandre DERUMIER
2020-09-29 8:51 ` Fabian Grünbichler
0 siblings, 2 replies; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-28 15:59 UTC (permalink / raw)
To: Proxmox VE development discussion
Here a new test http://odisoweb1.odiso.net/test5
This has occured at corosync start
node1:
-----
start corosync : 17:30:19
node2: /etc/pve locked
--------------
Current time : 17:30:24
I have done backtrace of all nodes at same time with parallel ssh at 17:35:22
and a coredump of all nodes at same time with parallel ssh at 17:42:26
(Note that this time, /etc/pve was still locked after backtrace/coredump)
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Lundi 28 Septembre 2020 11:35:00
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>but, just to make sure, could you reproduce the issue once more, and
>>then (with debug symbols installed) run
>>
>>$ gdb -ex 'set pagination 0' -ex 'thread apply all bt' --batch -p $(pidof pmxcfs)
>>
>>on all nodes at the same time? this should minimize the fallout and show
>>us whether the thread that logged the first part of sending the state is
>>still around on the node that triggers the hang..
ok, no problem, I'll do a new test today
>>this looks like you bt-ed corosync this time around? if so, than this is
>>expected:
>>
>>- attach gdb to corosync
>>- corosync blocks
>>- other nodes notice corosync is gone on node X
>>- config change triggered
>>- sync restarts on all nodes, does not trigger bug this time
ok, thanks, make sense. I didn't notice this previously, but this time it was with corosync started.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Lundi 28 Septembre 2020 11:17:37
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 25, 2020 6:29 pm, Alexandre DERUMIER wrote:
> here a new hang:
>
> http://odisoweb1.odiso.net/test4/
okay, so at least we now know something strange inside pmxcfs is going
on, and not inside corosync - we never reach the part where the broken
node (again #13 in this hang!) sends out the state message:
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: dfsm mode is 1 (dfsm.c:706:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: received message's header type is 1 (dfsm.c:736:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: received state message (type = 1, subtype = 0, 32 bytes), mode is 1 (dfsm.c:782:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] notice: received sync request (epoch 1/23389/000000A8) (dfsm.c:890:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [main] debug: enter dfsm_release_sync_resources (dfsm.c:640:dfsm_release_sync_resources)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: enter dcdb_get_state 00000000365345C2 5F6E0B1F (dcdb.c:444:dcdb_get_state)
[...]
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: dfsm_send_state_message_full: type 2 len 1 (dfsm.c:281:dfsm_send_state_message_full)
this should be followed by output like this (from the -unblock log,
where the sync went through):
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: send state message debug (dfsm.c:241:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: cpg_handle 5109470997161967616 (dfsm.c:242:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: iov[0] len 32 (dfsm.c:244:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: iov[1] len 64792 (dfsm.c:244:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: send state message loop body (dfsm.c:246:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: send state message result: 1 (dfsm.c:250:dfsm_send_message_full_debug_state)
but this never happens. the relevant code from our patched dfsm.c
(reordered):
static cs_error_t
dfsm_send_state_message_full(
dfsm_t *dfsm,
uint16_t type,
struct iovec *iov,
unsigned int len)
{
g_return_val_if_fail(dfsm != NULL, CS_ERR_INVALID_PARAM);
g_return_val_if_fail(DFSM_VALID_STATE_MESSAGE(type), CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
// this message is still logged
cfs_dom_debug(dfsm->log_domain, "dfsm_send_state_message_full: type %d len %d", type, len);
// everything below this point might not have happened anymore
dfsm_message_state_header_t header;
header.base.type = type;
header.base.subtype = 0;
header.base.protocol_version = dfsm->protocol_version;
header.base.time = time(NULL);
header.base.reserved = 0;
header.epoch = dfsm->sync_epoch;
struct iovec real_iov[len + 1];
real_iov[0].iov_base = (char *)&header;
real_iov[0].iov_len = sizeof(header);
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
return type == DFSM_MESSAGE_STATE ? dfsm_send_message_full_debug_state(dfsm, real_iov, len + 1, 1) : dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
}
static cs_error_t
dfsm_send_message_full_debug_state(
dfsm_t *dfsm,
struct iovec *iov,
unsigned int len,
int retry)
{
g_return_val_if_fail(dfsm != NULL, CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
struct timespec tvreq = { .tv_sec = 0, .tv_nsec = 100000000 };
cs_error_t result;
int retries = 0;
// this message is not visible in log
// we don't know how far above this we managed to run
cfs_dom_message(dfsm->log_domain, "send state message debug");
cfs_dom_message(dfsm->log_domain, "cpg_handle %" PRIu64, dfsm->cpg_handle);
for (int i = 0; i < len; i++)
cfs_dom_message(dfsm->log_domain, "iov[%d] len %zd", i, iov[i].iov_len);
loop:
cfs_dom_message(dfsm->log_domain, "send state message loop body");
result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len);
cfs_dom_message(dfsm->log_domain, "send state message result: %d", result);
if (retry && result == CS_ERR_TRY_AGAIN) {
nanosleep(&tvreq, NULL);
++retries;
if ((retries % 10) == 0)
cfs_dom_message(dfsm->log_domain, "cpg_send_message retry %d", retries);
if (retries < 100)
goto loop;
}
if (retries)
cfs_dom_message(dfsm->log_domain, "cpg_send_message retried %d times", retries);
if (result != CS_OK &&
(!retry || result != CS_ERR_TRY_AGAIN))
cfs_dom_critical(dfsm->log_domain, "cpg_send_message failed: %d", result);
return result;
}
I don't see much that could go wrong inside dfsm_send_state_message_full
after the debug log statement (it's just filling out the header and iov
structs, and calling 'dfsm_send_message_full_debug_state' since type ==
2 == DFSM_MESSAGE_STATE).
inside dfsm_send_message_full_debug, before the first log statement we
only check for dfsm or the message length/content being NULL/0, all of
which can't really happen with that call path. also, in that case we'd
return CS_ERR_INVALID_PARAM , which would bubble up into the delivery
callback and cause us to leave CPG, which would again be visible in the
logs..
but, just to make sure, could you reproduce the issue once more, and
then (with debug symbols installed) run
$ gdb -ex 'set pagination 0' -ex 'thread apply all bt' --batch -p $(pidof pmxcfs)
on all nodes at the same time? this should minimize the fallout and show
us whether the thread that logged the first part of sending the state is
still around on the node that triggers the hang..
> Something new: where doing coredump or bt-full on pmxcfs on node1,
> this have unlocked /etc/pve on other nodes
>
> /etc/pve unlocked(with coredump or bt-full): 17:57:40
this looks like you bt-ed corosync this time around? if so, than this is
expected:
- attach gdb to corosync
- corosync blocks
- other nodes notice corosync is gone on node X
- config change triggered
- sync restarts on all nodes, does not trigger bug this time
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-28 15:59 ` Alexandre DERUMIER
@ 2020-09-29 5:30 ` Alexandre DERUMIER
2020-09-29 8:51 ` Fabian Grünbichler
1 sibling, 0 replies; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-29 5:30 UTC (permalink / raw)
To: Proxmox VE development discussion
also for test5,
I have restarted corosync on node1 at 17:54:05, this have unlocked /etc/pve on other nodes
I have submitted logs too : "corosync-restart-nodeX.log"
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Lundi 28 Septembre 2020 17:59:20
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Here a new test http://odisoweb1.odiso.net/test5
This has occured at corosync start
node1:
-----
start corosync : 17:30:19
node2: /etc/pve locked
--------------
Current time : 17:30:24
I have done backtrace of all nodes at same time with parallel ssh at 17:35:22
and a coredump of all nodes at same time with parallel ssh at 17:42:26
(Note that this time, /etc/pve was still locked after backtrace/coredump)
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Lundi 28 Septembre 2020 11:35:00
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>but, just to make sure, could you reproduce the issue once more, and
>>then (with debug symbols installed) run
>>
>>$ gdb -ex 'set pagination 0' -ex 'thread apply all bt' --batch -p $(pidof pmxcfs)
>>
>>on all nodes at the same time? this should minimize the fallout and show
>>us whether the thread that logged the first part of sending the state is
>>still around on the node that triggers the hang..
ok, no problem, I'll do a new test today
>>this looks like you bt-ed corosync this time around? if so, than this is
>>expected:
>>
>>- attach gdb to corosync
>>- corosync blocks
>>- other nodes notice corosync is gone on node X
>>- config change triggered
>>- sync restarts on all nodes, does not trigger bug this time
ok, thanks, make sense. I didn't notice this previously, but this time it was with corosync started.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Lundi 28 Septembre 2020 11:17:37
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 25, 2020 6:29 pm, Alexandre DERUMIER wrote:
> here a new hang:
>
> http://odisoweb1.odiso.net/test4/
okay, so at least we now know something strange inside pmxcfs is going
on, and not inside corosync - we never reach the part where the broken
node (again #13 in this hang!) sends out the state message:
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: dfsm mode is 1 (dfsm.c:706:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: received message's header type is 1 (dfsm.c:736:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: received state message (type = 1, subtype = 0, 32 bytes), mode is 1 (dfsm.c:782:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] notice: received sync request (epoch 1/23389/000000A8) (dfsm.c:890:dfsm_cpg_deliver_callback)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [main] debug: enter dfsm_release_sync_resources (dfsm.c:640:dfsm_release_sync_resources)
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: enter dcdb_get_state 00000000365345C2 5F6E0B1F (dcdb.c:444:dcdb_get_state)
[...]
Sep 25 17:22:08 m6kvm13 pmxcfs[10086]: [dcdb] debug: dfsm_send_state_message_full: type 2 len 1 (dfsm.c:281:dfsm_send_state_message_full)
this should be followed by output like this (from the -unblock log,
where the sync went through):
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: send state message debug (dfsm.c:241:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: cpg_handle 5109470997161967616 (dfsm.c:242:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: iov[0] len 32 (dfsm.c:244:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: iov[1] len 64792 (dfsm.c:244:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: send state message loop body (dfsm.c:246:dfsm_send_message_full_debug_state)
Sep 25 17:57:39 m6kvm13 pmxcfs[10086]: [dcdb] notice: send state message result: 1 (dfsm.c:250:dfsm_send_message_full_debug_state)
but this never happens. the relevant code from our patched dfsm.c
(reordered):
static cs_error_t
dfsm_send_state_message_full(
dfsm_t *dfsm,
uint16_t type,
struct iovec *iov,
unsigned int len)
{
g_return_val_if_fail(dfsm != NULL, CS_ERR_INVALID_PARAM);
g_return_val_if_fail(DFSM_VALID_STATE_MESSAGE(type), CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
// this message is still logged
cfs_dom_debug(dfsm->log_domain, "dfsm_send_state_message_full: type %d len %d", type, len);
// everything below this point might not have happened anymore
dfsm_message_state_header_t header;
header.base.type = type;
header.base.subtype = 0;
header.base.protocol_version = dfsm->protocol_version;
header.base.time = time(NULL);
header.base.reserved = 0;
header.epoch = dfsm->sync_epoch;
struct iovec real_iov[len + 1];
real_iov[0].iov_base = (char *)&header;
real_iov[0].iov_len = sizeof(header);
for (int i = 0; i < len; i++)
real_iov[i + 1] = iov[i];
return type == DFSM_MESSAGE_STATE ? dfsm_send_message_full_debug_state(dfsm, real_iov, len + 1, 1) : dfsm_send_message_full(dfsm, real_iov, len + 1, 1);
}
static cs_error_t
dfsm_send_message_full_debug_state(
dfsm_t *dfsm,
struct iovec *iov,
unsigned int len,
int retry)
{
g_return_val_if_fail(dfsm != NULL, CS_ERR_INVALID_PARAM);
g_return_val_if_fail(!len || iov != NULL, CS_ERR_INVALID_PARAM);
struct timespec tvreq = { .tv_sec = 0, .tv_nsec = 100000000 };
cs_error_t result;
int retries = 0;
// this message is not visible in log
// we don't know how far above this we managed to run
cfs_dom_message(dfsm->log_domain, "send state message debug");
cfs_dom_message(dfsm->log_domain, "cpg_handle %" PRIu64, dfsm->cpg_handle);
for (int i = 0; i < len; i++)
cfs_dom_message(dfsm->log_domain, "iov[%d] len %zd", i, iov[i].iov_len);
loop:
cfs_dom_message(dfsm->log_domain, "send state message loop body");
result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len);
cfs_dom_message(dfsm->log_domain, "send state message result: %d", result);
if (retry && result == CS_ERR_TRY_AGAIN) {
nanosleep(&tvreq, NULL);
++retries;
if ((retries % 10) == 0)
cfs_dom_message(dfsm->log_domain, "cpg_send_message retry %d", retries);
if (retries < 100)
goto loop;
}
if (retries)
cfs_dom_message(dfsm->log_domain, "cpg_send_message retried %d times", retries);
if (result != CS_OK &&
(!retry || result != CS_ERR_TRY_AGAIN))
cfs_dom_critical(dfsm->log_domain, "cpg_send_message failed: %d", result);
return result;
}
I don't see much that could go wrong inside dfsm_send_state_message_full
after the debug log statement (it's just filling out the header and iov
structs, and calling 'dfsm_send_message_full_debug_state' since type ==
2 == DFSM_MESSAGE_STATE).
inside dfsm_send_message_full_debug, before the first log statement we
only check for dfsm or the message length/content being NULL/0, all of
which can't really happen with that call path. also, in that case we'd
return CS_ERR_INVALID_PARAM , which would bubble up into the delivery
callback and cause us to leave CPG, which would again be visible in the
logs..
but, just to make sure, could you reproduce the issue once more, and
then (with debug symbols installed) run
$ gdb -ex 'set pagination 0' -ex 'thread apply all bt' --batch -p $(pidof pmxcfs)
on all nodes at the same time? this should minimize the fallout and show
us whether the thread that logged the first part of sending the state is
still around on the node that triggers the hang..
> Something new: where doing coredump or bt-full on pmxcfs on node1,
> this have unlocked /etc/pve on other nodes
>
> /etc/pve unlocked(with coredump or bt-full): 17:57:40
this looks like you bt-ed corosync this time around? if so, than this is
expected:
- attach gdb to corosync
- corosync blocks
- other nodes notice corosync is gone on node X
- config change triggered
- sync restarts on all nodes, does not trigger bug this time
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-28 15:59 ` Alexandre DERUMIER
2020-09-29 5:30 ` Alexandre DERUMIER
@ 2020-09-29 8:51 ` Fabian Grünbichler
2020-09-29 9:37 ` Alexandre DERUMIER
1 sibling, 1 reply; 84+ messages in thread
From: Fabian Grünbichler @ 2020-09-29 8:51 UTC (permalink / raw)
To: Proxmox VE development discussion
On September 28, 2020 5:59 pm, Alexandre DERUMIER wrote:
> Here a new test http://odisoweb1.odiso.net/test5
>
> This has occured at corosync start
>
>
> node1:
> -----
> start corosync : 17:30:19
>
>
> node2: /etc/pve locked
> --------------
> Current time : 17:30:24
>
>
> I have done backtrace of all nodes at same time with parallel ssh at 17:35:22
>
> and a coredump of all nodes at same time with parallel ssh at 17:42:26
>
>
> (Note that this time, /etc/pve was still locked after backtrace/coredump)
okay, so this time two more log lines got printed on the (again) problem
causing node #13, but it still stops logging at a point where this makes
no sense.
I rebuilt the packages:
f318f12e5983cb09d186c2ee37743203f599d103b6abb2d00c78d312b4f12df942d8ed1ff5de6e6c194785d0a81eb881e80f7bbfd4865ca1a5a509acd40f64aa pve-cluster_6.1-8_amd64.deb
b220ee95303e22704793412e83ac5191ba0e53c2f41d85358a247c248d2a6856e5b791b1d12c36007a297056388224acf4e5a1250ef1dd019aee97e8ac4bcac7 pve-cluster-dbgsym_6.1-8_amd64.deb
with a change of how the logging is set up (I now suspect that some
messages might get dropped if the logging throughput is high enough),
let's hope this gets us the information we need. please repeat the test5
again with these packages.
is there anything special about node 13? network topology, slower
hardware, ... ?
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-29 8:51 ` Fabian Grünbichler
@ 2020-09-29 9:37 ` Alexandre DERUMIER
2020-09-29 10:52 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-29 9:37 UTC (permalink / raw)
To: Proxmox VE development discussion
>>with a change of how the logging is set up (I now suspect that some
>>messages might get dropped if the logging throughput is high enough),
>>let's hope this gets us the information we need. please repeat the test5
>>again with these packages.
I'll test this afternoon
>>is there anything special about node 13? network topology, slower
>>hardware, ... ?
no nothing special, all nodes have exactly same hardware/cpu (24cores/48threads 3ghz)/memory/disk.
this node is around 10% cpu usage, load is around 5.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 10:51:32
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 28, 2020 5:59 pm, Alexandre DERUMIER wrote:
> Here a new test http://odisoweb1.odiso.net/test5
>
> This has occured at corosync start
>
>
> node1:
> -----
> start corosync : 17:30:19
>
>
> node2: /etc/pve locked
> --------------
> Current time : 17:30:24
>
>
> I have done backtrace of all nodes at same time with parallel ssh at 17:35:22
>
> and a coredump of all nodes at same time with parallel ssh at 17:42:26
>
>
> (Note that this time, /etc/pve was still locked after backtrace/coredump)
okay, so this time two more log lines got printed on the (again) problem
causing node #13, but it still stops logging at a point where this makes
no sense.
I rebuilt the packages:
f318f12e5983cb09d186c2ee37743203f599d103b6abb2d00c78d312b4f12df942d8ed1ff5de6e6c194785d0a81eb881e80f7bbfd4865ca1a5a509acd40f64aa pve-cluster_6.1-8_amd64.deb
b220ee95303e22704793412e83ac5191ba0e53c2f41d85358a247c248d2a6856e5b791b1d12c36007a297056388224acf4e5a1250ef1dd019aee97e8ac4bcac7 pve-cluster-dbgsym_6.1-8_amd64.deb
with a change of how the logging is set up (I now suspect that some
messages might get dropped if the logging throughput is high enough),
let's hope this gets us the information we need. please repeat the test5
again with these packages.
is there anything special about node 13? network topology, slower
hardware, ... ?
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-29 9:37 ` Alexandre DERUMIER
@ 2020-09-29 10:52 ` Alexandre DERUMIER
2020-09-29 11:43 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-29 10:52 UTC (permalink / raw)
To: Proxmox VE development discussion
here a new test:
http://odisoweb1.odiso.net/test6/
node1
-----
start corosync : 12:08:33
node2 (/etc/pve lock)
-----
Current time : 12:08:39
node1 (stop corosync : unlock /etc/pve)
-----
12:28:11 : systemctl stop corosync
backtraces: 12:26:30
coredump : 12:27:21
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 11:37:41
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>with a change of how the logging is set up (I now suspect that some
>>messages might get dropped if the logging throughput is high enough),
>>let's hope this gets us the information we need. please repeat the test5
>>again with these packages.
I'll test this afternoon
>>is there anything special about node 13? network topology, slower
>>hardware, ... ?
no nothing special, all nodes have exactly same hardware/cpu (24cores/48threads 3ghz)/memory/disk.
this node is around 10% cpu usage, load is around 5.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 10:51:32
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 28, 2020 5:59 pm, Alexandre DERUMIER wrote:
> Here a new test http://odisoweb1.odiso.net/test5
>
> This has occured at corosync start
>
>
> node1:
> -----
> start corosync : 17:30:19
>
>
> node2: /etc/pve locked
> --------------
> Current time : 17:30:24
>
>
> I have done backtrace of all nodes at same time with parallel ssh at 17:35:22
>
> and a coredump of all nodes at same time with parallel ssh at 17:42:26
>
>
> (Note that this time, /etc/pve was still locked after backtrace/coredump)
okay, so this time two more log lines got printed on the (again) problem
causing node #13, but it still stops logging at a point where this makes
no sense.
I rebuilt the packages:
f318f12e5983cb09d186c2ee37743203f599d103b6abb2d00c78d312b4f12df942d8ed1ff5de6e6c194785d0a81eb881e80f7bbfd4865ca1a5a509acd40f64aa pve-cluster_6.1-8_amd64.deb
b220ee95303e22704793412e83ac5191ba0e53c2f41d85358a247c248d2a6856e5b791b1d12c36007a297056388224acf4e5a1250ef1dd019aee97e8ac4bcac7 pve-cluster-dbgsym_6.1-8_amd64.deb
with a change of how the logging is set up (I now suspect that some
messages might get dropped if the logging throughput is high enough),
let's hope this gets us the information we need. please repeat the test5
again with these packages.
is there anything special about node 13? network topology, slower
hardware, ... ?
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-29 10:52 ` Alexandre DERUMIER
@ 2020-09-29 11:43 ` Alexandre DERUMIER
2020-09-29 11:50 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-29 11:43 UTC (permalink / raw)
To: Proxmox VE development discussion
>>
>>node1 (stop corosync : unlock /etc/pve)
>>-----
>>12:28:11 : systemctl stop corosync
sorry, this was wrong,I need to start corosync after the stop to get it working again
I'll reupload theses logs
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 12:52:44
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
here a new test:
http://odisoweb1.odiso.net/test6/
node1
-----
start corosync : 12:08:33
node2 (/etc/pve lock)
-----
Current time : 12:08:39
node1 (stop corosync : unlock /etc/pve)
-----
12:28:11 : systemctl stop corosync
backtraces: 12:26:30
coredump : 12:27:21
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 11:37:41
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>with a change of how the logging is set up (I now suspect that some
>>messages might get dropped if the logging throughput is high enough),
>>let's hope this gets us the information we need. please repeat the test5
>>again with these packages.
I'll test this afternoon
>>is there anything special about node 13? network topology, slower
>>hardware, ... ?
no nothing special, all nodes have exactly same hardware/cpu (24cores/48threads 3ghz)/memory/disk.
this node is around 10% cpu usage, load is around 5.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 10:51:32
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 28, 2020 5:59 pm, Alexandre DERUMIER wrote:
> Here a new test http://odisoweb1.odiso.net/test5
>
> This has occured at corosync start
>
>
> node1:
> -----
> start corosync : 17:30:19
>
>
> node2: /etc/pve locked
> --------------
> Current time : 17:30:24
>
>
> I have done backtrace of all nodes at same time with parallel ssh at 17:35:22
>
> and a coredump of all nodes at same time with parallel ssh at 17:42:26
>
>
> (Note that this time, /etc/pve was still locked after backtrace/coredump)
okay, so this time two more log lines got printed on the (again) problem
causing node #13, but it still stops logging at a point where this makes
no sense.
I rebuilt the packages:
f318f12e5983cb09d186c2ee37743203f599d103b6abb2d00c78d312b4f12df942d8ed1ff5de6e6c194785d0a81eb881e80f7bbfd4865ca1a5a509acd40f64aa pve-cluster_6.1-8_amd64.deb
b220ee95303e22704793412e83ac5191ba0e53c2f41d85358a247c248d2a6856e5b791b1d12c36007a297056388224acf4e5a1250ef1dd019aee97e8ac4bcac7 pve-cluster-dbgsym_6.1-8_amd64.deb
with a change of how the logging is set up (I now suspect that some
messages might get dropped if the logging throughput is high enough),
let's hope this gets us the information we need. please repeat the test5
again with these packages.
is there anything special about node 13? network topology, slower
hardware, ... ?
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-29 11:43 ` Alexandre DERUMIER
@ 2020-09-29 11:50 ` Alexandre DERUMIER
2020-09-29 13:28 ` Fabian Grünbichler
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-29 11:50 UTC (permalink / raw)
To: Proxmox VE development discussion
I have reuploaded the logs
node1
-----
start corosync : 12:08:33 (corosync.log)
node2 (/etc/pve lock)
-----
Current time : 12:08:39
node1 (stop corosync : ---> not unlocked) (corosync-stop.log)
-----
12:28:11 : systemctl stop corosync
node2 (start corosync: ----> /etc/pve unlocked (corosync-start.log)
------------------------------------------------
13:41:16 : systemctl start corosync
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 13:43:08
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>
>>node1 (stop corosync : unlock /etc/pve)
>>-----
>>12:28:11 : systemctl stop corosync
sorry, this was wrong,I need to start corosync after the stop to get it working again
I'll reupload theses logs
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 12:52:44
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
here a new test:
http://odisoweb1.odiso.net/test6/
node1
-----
start corosync : 12:08:33
node2 (/etc/pve lock)
-----
Current time : 12:08:39
node1 (stop corosync : unlock /etc/pve)
-----
12:28:11 : systemctl stop corosync
backtraces: 12:26:30
coredump : 12:27:21
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 11:37:41
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>with a change of how the logging is set up (I now suspect that some
>>messages might get dropped if the logging throughput is high enough),
>>let's hope this gets us the information we need. please repeat the test5
>>again with these packages.
I'll test this afternoon
>>is there anything special about node 13? network topology, slower
>>hardware, ... ?
no nothing special, all nodes have exactly same hardware/cpu (24cores/48threads 3ghz)/memory/disk.
this node is around 10% cpu usage, load is around 5.
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 10:51:32
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
On September 28, 2020 5:59 pm, Alexandre DERUMIER wrote:
> Here a new test http://odisoweb1.odiso.net/test5
>
> This has occured at corosync start
>
>
> node1:
> -----
> start corosync : 17:30:19
>
>
> node2: /etc/pve locked
> --------------
> Current time : 17:30:24
>
>
> I have done backtrace of all nodes at same time with parallel ssh at 17:35:22
>
> and a coredump of all nodes at same time with parallel ssh at 17:42:26
>
>
> (Note that this time, /etc/pve was still locked after backtrace/coredump)
okay, so this time two more log lines got printed on the (again) problem
causing node #13, but it still stops logging at a point where this makes
no sense.
I rebuilt the packages:
f318f12e5983cb09d186c2ee37743203f599d103b6abb2d00c78d312b4f12df942d8ed1ff5de6e6c194785d0a81eb881e80f7bbfd4865ca1a5a509acd40f64aa pve-cluster_6.1-8_amd64.deb
b220ee95303e22704793412e83ac5191ba0e53c2f41d85358a247c248d2a6856e5b791b1d12c36007a297056388224acf4e5a1250ef1dd019aee97e8ac4bcac7 pve-cluster-dbgsym_6.1-8_amd64.deb
with a change of how the logging is set up (I now suspect that some
messages might get dropped if the logging throughput is high enough),
let's hope this gets us the information we need. please repeat the test5
again with these packages.
is there anything special about node 13? network topology, slower
hardware, ... ?
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-29 11:50 ` Alexandre DERUMIER
@ 2020-09-29 13:28 ` Fabian Grünbichler
2020-09-29 13:52 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Fabian Grünbichler @ 2020-09-29 13:28 UTC (permalink / raw)
To: Proxmox VE development discussion
huge thanks for all the work on this btw!
I think I've found a likely culprit (a missing lock around a
non-thread-safe corosync library call) based on the last logs (which
were now finally complete!).
rebuilt packages with a proof-of-concept-fix:
23b03a48d3aa9c14e86fe8cf9bbb7b00bd8fe9483084b9e0fd75fd67f29f10bec00e317e2a66758713050f36c165d72f107ee3449f9efeb842d3a57c25f8bca7 pve-cluster_6.1-8_amd64.deb
9e1addd676513b176f5afb67cc6d85630e7da9bbbf63562421b4fd2a3916b3b2af922df555059b99f8b0b9e64171101a1c9973846e25f9144ded9d487450baef pve-cluster-dbgsym_6.1-8_amd64.deb
I removed some logging statements which are no longer needed, so output
is a bit less verbose again. if you are not able to trigger the issue
with this package, feel free to remove the -debug and let it run for a
little longer without the massive logs.
if feedback from your end is positive, I'll whip up a proper patch
tomorrow or on Thursday.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-29 13:28 ` Fabian Grünbichler
@ 2020-09-29 13:52 ` Alexandre DERUMIER
2020-09-30 6:09 ` Alexandre DERUMIER
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-29 13:52 UTC (permalink / raw)
To: Proxmox VE development discussion
>>huge thanks for all the work on this btw!
huge thanks to you ! ;)
>>I think I've found a likely culprit (a missing lock around a
>>non-thread-safe corosync library call) based on the last logs (which
>>were now finally complete!).
YES :)
>>if feedback from your end is positive, I'll whip up a proper patch
>>tomorrow or on Thursday.
I'm going to launch a new test right now !
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 15:28:19
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
huge thanks for all the work on this btw!
I think I've found a likely culprit (a missing lock around a
non-thread-safe corosync library call) based on the last logs (which
were now finally complete!).
rebuilt packages with a proof-of-concept-fix:
23b03a48d3aa9c14e86fe8cf9bbb7b00bd8fe9483084b9e0fd75fd67f29f10bec00e317e2a66758713050f36c165d72f107ee3449f9efeb842d3a57c25f8bca7 pve-cluster_6.1-8_amd64.deb
9e1addd676513b176f5afb67cc6d85630e7da9bbbf63562421b4fd2a3916b3b2af922df555059b99f8b0b9e64171101a1c9973846e25f9144ded9d487450baef pve-cluster-dbgsym_6.1-8_amd64.deb
I removed some logging statements which are no longer needed, so output
is a bit less verbose again. if you are not able to trigger the issue
with this package, feel free to remove the -debug and let it run for a
little longer without the massive logs.
if feedback from your end is positive, I'll whip up a proper patch
tomorrow or on Thursday.
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-29 13:52 ` Alexandre DERUMIER
@ 2020-09-30 6:09 ` Alexandre DERUMIER
2020-09-30 6:26 ` Thomas Lamprecht
0 siblings, 1 reply; 84+ messages in thread
From: Alexandre DERUMIER @ 2020-09-30 6:09 UTC (permalink / raw)
To: Proxmox VE development discussion
Hi,
some news, my last test is running for 14h now, and I don't have had any problem :)
So, it seem that is indeed fixed ! Congratulations !
I wonder if it could be related to this forum user
https://forum.proxmox.com/threads/proxmox-6-2-corosync-3-rare-and-spontaneous-disruptive-udp-5405-storm-flood.75871/
His problem is that after corosync lag (he's have 1 cluster stretch on 2DC with 10km distance, so I think sometimes he's having some small lag,
1 node is flooding other nodes with a lot of udp packets. (and making things worst, as corosync cpu is going to 100% / overloaded, and then can't see other onodes
I had this problem 6month ago after shutting down a node, that's why I'm thinking it could "maybe" related.
So, I wonder if it could be same pmxcfs bug, when something looping or send again again packets.
The forum user seem to have the problem multiple times in some week, so maybe he'll be able to test the new fixed pmxcs, and tell us if it's fixing this bug too.
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 15:52:18
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>>huge thanks for all the work on this btw!
huge thanks to you ! ;)
>>I think I've found a likely culprit (a missing lock around a
>>non-thread-safe corosync library call) based on the last logs (which
>>were now finally complete!).
YES :)
>>if feedback from your end is positive, I'll whip up a proper patch
>>tomorrow or on Thursday.
I'm going to launch a new test right now !
----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler@proxmox.com>
À: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Envoyé: Mardi 29 Septembre 2020 15:28:19
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
huge thanks for all the work on this btw!
I think I've found a likely culprit (a missing lock around a
non-thread-safe corosync library call) based on the last logs (which
were now finally complete!).
rebuilt packages with a proof-of-concept-fix:
23b03a48d3aa9c14e86fe8cf9bbb7b00bd8fe9483084b9e0fd75fd67f29f10bec00e317e2a66758713050f36c165d72f107ee3449f9efeb842d3a57c25f8bca7 pve-cluster_6.1-8_amd64.deb
9e1addd676513b176f5afb67cc6d85630e7da9bbbf63562421b4fd2a3916b3b2af922df555059b99f8b0b9e64171101a1c9973846e25f9144ded9d487450baef pve-cluster-dbgsym_6.1-8_amd64.deb
I removed some logging statements which are no longer needed, so output
is a bit less verbose again. if you are not able to trigger the issue
with this package, feel free to remove the -debug and let it run for a
little longer without the massive logs.
if feedback from your end is positive, I'll whip up a proper patch
tomorrow or on Thursday.
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-30 6:09 ` Alexandre DERUMIER
@ 2020-09-30 6:26 ` Thomas Lamprecht
0 siblings, 0 replies; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-30 6:26 UTC (permalink / raw)
To: Proxmox VE development discussion, Alexandre DERUMIER
Hi,
On 30.09.20 08:09, Alexandre DERUMIER wrote:
> some news, my last test is running for 14h now, and I don't have had any problem :)
>
great! Thanks for all your testing time, this would have been much harder,
if even possible at all, without you probiving so much testing effort on a
production(!) cluster - appreciated!
Naturally many thanks to Fabian too, for reading so many logs without going
insane :-)
> So, it seem that is indeed fixed ! Congratulations !
>
honza comfirmed Fabians suspicion about lacking guarantees of thread safety
for cpg_mcast_joined, which was sadly not documented, so this is surely
a bug, let's hope the last of such hard to reproduce ones.
>
>
> I wonder if it could be related to this forum user
> https://forum.proxmox.com/threads/proxmox-6-2-corosync-3-rare-and-spontaneous-disruptive-udp-5405-storm-flood.75871/
>
> His problem is that after corosync lag (he's have 1 cluster stretch on 2DC with 10km distance, so I think sometimes he's having some small lag,
> 1 node is flooding other nodes with a lot of udp packets. (and making things worst, as corosync cpu is going to 100% / overloaded, and then can't see other onodes
I can imagine this problem showing up as a a side effect of a flood where partition
changes happen. Not so sure that this can be the cause of that directly.
>
> I had this problem 6month ago after shutting down a node, that's why I'm thinking it could "maybe" related.
>
> So, I wonder if it could be same pmxcfs bug, when something looping or send again again packets.
>
> The forum user seem to have the problem multiple times in some week, so maybe he'll be able to test the new fixed pmxcs, and tell us if it's fixing this bug too.
Testing once available would be sure a good idea for them.
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-03 14:11 [pve-devel] corosync bug: cluster break after 1 node clean shutdown Alexandre DERUMIER
2020-09-04 12:29 ` Alexandre DERUMIER
2020-09-04 15:46 ` Alexandre DERUMIER
@ 2020-09-30 15:50 ` Thomas Lamprecht
2020-10-15 9:16 ` Eneko Lacunza
2 siblings, 1 reply; 84+ messages in thread
From: Thomas Lamprecht @ 2020-09-30 15:50 UTC (permalink / raw)
To: Proxmox VE development discussion, Alexandre DERUMIER
Hi,
FYI: pve-cluster 6.2-1 is now available on pvetest, it includes the slightly modified
patch from Fabian.
cheers,
Thomas
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-30 15:50 ` Thomas Lamprecht
@ 2020-10-15 9:16 ` Eneko Lacunza
0 siblings, 0 replies; 84+ messages in thread
From: Eneko Lacunza @ 2020-10-15 9:16 UTC (permalink / raw)
To: pve-devel
Hi all,
I'm just a lurker on this list, but wanted to send a big THANK YOU to
all involved in this fix, even with this 15-day lag. :-)
I think some of our clients have been affected by this on production
clusters, so I hope this will improve their cluster stability.
Normally new features are the prettiest things, but having a robust and
dependable system - that has no price!!
Thanks a lot!!
Eneko
El 30/9/20 a las 17:50, Thomas Lamprecht escribió:
> Hi,
>
> FYI: pve-cluster 6.2-1 is now available on pvetest, it includes the slightly modified
> patch from Fabian.
>
> cheers,
> Thomas
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
--
Eneko Lacunza | +34 943 569 206
| elacunza@binovo.es
Zuzendari teknikoa | https://www.binovo.es
Director técnico | Astigarragako Bidea, 2 - 2º izda.
BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun
^ permalink raw reply [flat|nested] 84+ messages in thread
* Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
2020-09-04 12:29 ` Alexandre DERUMIER
2020-09-04 15:42 ` Dietmar Maurer
@ 2020-12-29 14:21 ` Josef Johansson
1 sibling, 0 replies; 84+ messages in thread
From: Josef Johansson @ 2020-12-29 14:21 UTC (permalink / raw)
To: pve-devel
Hi,
On 9/4/20 2:29 PM, Alexandre DERUMIER wrote:
> BTW,
>
> do you think it could be possible to add an extra optionnal layer of security check, not related to corosync ?
>
> I'm still afraid of this corosync bug since years, and still don't use HA. (or I have tried to enable it 2months ago,and this give me a disaster yesterday..)
Using corosync to enable HA is a bit scary TBH, not sure how people
solve this, but I'd rather have the logic outside the cluster so it can
make a solid decision whether a node should be rebooted or not, and
choose to reboot via iDRAC if all checks fail. Maybe use the metrics
support to facilitate this?
> Something like an extra heartbeat between nodes daemons, and check if we also have quorum with theses heartbeats ?
>
>
>
> ----- Mail original -----
> De: "aderumier" <aderumier@odiso.com>
> À: "pve-devel" <pve-devel@pve.proxmox.com>
> Envoyé: Jeudi 3 Septembre 2020 16:11:56
> Objet: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>
> Hi,
>
> I had a problem this morning with corosync, after shutting down cleanly a node from the cluster.
> This is a 14 nodes cluster, node2 was shutted. (with "halt" command)
>
> HA was actived, and all nodes have been rebooted.
> (I have already see this problem some months ago, without HA enabled, and it was like half of the nodes didn't see others nodes, like 2 cluster formations)
>
> Some users have reported similar problems when adding a new node
> https://forum.proxmox.com/threads/quorum-lost-when-adding-new-7th-node.75197/
>
>
> Here the full logs of the nodes, for each node I'm seeing quorum with 13 nodes. (so it's seem to be ok).
> I didn't have time to look live on server, as HA have restarted them.
> So, I don't known if it's a corosync bug or something related to crm,lrm,pmxcs (but I don't see any special logs)
>
> Only node7 have survived a little bit longer,and I had stop lrm/crm to avoid reboot.
>
> libknet1: 1.15-pve1
> corosync: 3.0.3-pve1
>
>
> Any ideas before submiting bug to corosync mailing ? (I'm seeing new libknet && corosync version on their github, I don't have read the changelog yet)
>
>
>
>
> node1
> -----
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm1 corosync[3678]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm1 corosync[3678]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm1 corosync[3678]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [dcdb] notice: cpg_send_message retried 1 times
> Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 has no active links
>
> --> reboot
>
> node2 : shutdown log
> -----
> Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd received signal 15: Terminated
> Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_00] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302Y9120CGN.ata.state
> Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_01] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302MG120CGN.ata.state
> Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd is exiting (exit status 0)
> Sep 3 10:39:05 m6kvm2 nrpe[20095]: Caught SIGTERM - shutting down...
> Sep 3 10:39:05 m6kvm2 nrpe[20095]: Daemon shutdown
> Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> starting task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root@pam:
> Sep 3 10:39:07 m6kvm2 pve-guests[38552]: all VMs and CTs stopped
> Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> end task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root@pam: OK
> Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: received signal TERM
> Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server closing
> Sep 3 10:39:07 m6kvm2 spiceproxy[36572]: worker exit
> Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: worker 36572 finished
> Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server stopped
> Sep 3 10:39:07 m6kvm2 pvestatd[3786]: received signal TERM
> Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server closing
> Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server stopped
> Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: received signal TERM
> Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: got shutdown request with shutdown policy 'conditional'
> Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: shutdown LRM, stop all services
> Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: received terminate request (signal)
> Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: stopping pvefw logger
> Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: watchdog closed (disabled)
> Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: server stopped
> Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: received signal TERM
> Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: server received shutdown request
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: received signal TERM
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server closing
> Sep 3 10:39:10 m6kvm2 pveproxy[29731]: worker exit
> Sep 3 10:39:10 m6kvm2 pveproxy[30873]: worker exit
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 30873 finished
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 31391 finished
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 29731 finished
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server stopped
> Sep 3 10:39:14 m6kvm2 pve-ha-crm[12847]: server stopped
>
>
> node3
> -----
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm3 corosync[30580]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm3 corosync[30580]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm3 corosync[30580]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: cpg_send_message retried 1 times
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 has no active links
>
>
> node4
> -----
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: all data is up to date
> Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:20 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
>
> node5
> -----
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
>
>
>
>
>
> node6
> -----
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 has no active links
>
>
>
> node7
> -----
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
> Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:20 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> ---> here the others nodes reboot almost at the same time
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 3 link: 0 is down
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 14 link: 0 is down
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 has no active links
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 has no active links
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 13 link: 0 is down
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 8 link: 0 is down
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 6 link: 0 is down
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 10 link: 0 is down
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 has no active links
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 has no active links
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 has no active links
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 has no active links
> Sep 3 10:40:28 m6kvm7 corosync[15467]: [TOTEM ] Token has not been received in 4505 ms
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 11 link: 0 is down
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 9 link: 0 is down
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 has no active links
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 has no active links
> Sep 3 10:40:30 m6kvm7 corosync[15467]: [TOTEM ] A processor failed, forming new configuration.
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 4 link: 0 is down
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 12 link: 0 is down
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 1 link: 0 is down
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 has no active links
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 has no active links
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 has no active links
> Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] link: host: 5 link: 0 is down
> Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 has no active links
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] A new membership (7.82f) was formed. Members left: 1 3 4 5 6 8 9 10 11 12 13 14
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] Failed to receive the leave message. failed: 1 3 4 5 6 8 9 10 11 12 13 14
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 12 received
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] Members[1]: 7
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: dfsm_deliver_queue: queue length 51
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: node lost quorum
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: members: 7/15892
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: received write while not quorate - trigger resync
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: leaving CPG group
> Sep 3 10:40:41 m6kvm7 pve-ha-crm[16196]: node 'm6kvm2': state changed from 'online' => 'unknown'
> Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] notice: start cluster connection
> Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: cpg_join failed: 14
> Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: can't initialize service
> Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892
> Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during cfs-locked 'domain-ha' operation: no quorum!
> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs lock update failed - Permission denied
> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - cfs lock update failed - Permission denied
> Sep 3 10:40:56 m6kvm7 pve-ha-lrm[16140]: status change active => lost_agent_lock
> Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change master => lost_manager_lock
> Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: watchdog closed (disabled)
> Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change lost_manager_lock => wait_for_quorum
> Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] rx: host: 4 link: 0 is up
> Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [TOTEM ] A new membership (4.834) was formed. Members joined: 4
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 4/3965, 7/15892
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: members: 4/3965, 7/15892
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [QUORUM] Members[2]: 4 7
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 4/3965/00000002)
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 4/3965/00000002)
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 7/15892
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 7/15892
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: start sending inode updates
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: sent all (4) updates
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received all states
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] rx: host: 3 link: 0 is up
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [TOTEM ] A new membership (3.838) was formed. Members joined: 3
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 3/3613, 4/3965, 7/15892
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: members: 3/3613, 4/3965, 7/15892
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [QUORUM] Members[3]: 3 4 7
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 3/3613/00000002)
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 3/3613/00000002)
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 4/3965
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 4/3965, 7/15892
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received all states
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: dfsm_deliver_queue: queue length 1
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] rx: host: 1 link: 0 is up
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.83c) was formed. Members joined: 1
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3552, 3/3613, 4/3965, 7/15892
> Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
> Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3552, 3/3613, 4/3965, 7/15892
> Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [QUORUM] Members[4]: 1 3 4 7
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3552/00000002)
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3552/00000002)
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 3/3613
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 3/3613, 4/3965, 7/15892
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received all states
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
> Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:49:18 m6kvm7 pmxcfs[15892]: [status] notice: received log
>
>
>
>
> node8
> -----
>
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
> Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
> Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
> Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
>
>
> --> reboot
>
>
>
> node9
> -----
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: all data is up to date
> Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:20 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
>
>
> --> reboot
>
>
>
> node10
> ------
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: all data is up to date
> Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log
> Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log
>
> --> reboot
>
>
> node11
> ------
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
> Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
> Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
> Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
> Sep 3 10:40:20 m6kvm11 pmxcfs[12983]: [status] notice: received log
>
>
>
>
> node12
> ------
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [dcdb] notice: cpg_send_message retried 1 times
> Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:20 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
>
>
>
>
>
> node13
> ------
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: cpg_send_message retried 1 times
> Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: all data is up to date
> Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
> Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
> Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
> Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
> --> reboot
>
> node14
> ------
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 has no active links
>
> --> reboot
>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
--
Med vänliga hälsningar
Josef Johansson
^ permalink raw reply [flat|nested] 84+ messages in thread
end of thread, other threads:[~2020-12-29 14:51 UTC | newest]
Thread overview: 84+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-09-03 14:11 [pve-devel] corosync bug: cluster break after 1 node clean shutdown Alexandre DERUMIER
2020-09-04 12:29 ` Alexandre DERUMIER
2020-09-04 15:42 ` Dietmar Maurer
2020-09-05 13:32 ` Alexandre DERUMIER
2020-09-05 15:23 ` dietmar
2020-09-05 17:30 ` Alexandre DERUMIER
2020-09-06 4:21 ` dietmar
2020-09-06 5:36 ` Alexandre DERUMIER
2020-09-06 6:33 ` Alexandre DERUMIER
2020-09-06 8:43 ` Alexandre DERUMIER
2020-09-06 12:14 ` dietmar
2020-09-06 12:19 ` dietmar
2020-09-07 7:00 ` Thomas Lamprecht
2020-09-07 7:19 ` Alexandre DERUMIER
2020-09-07 8:18 ` dietmar
2020-09-07 9:32 ` Alexandre DERUMIER
2020-09-07 13:23 ` Alexandre DERUMIER
2020-09-08 4:41 ` dietmar
2020-09-08 7:11 ` Alexandre DERUMIER
2020-09-09 20:05 ` Thomas Lamprecht
2020-09-10 4:58 ` Alexandre DERUMIER
2020-09-10 8:21 ` Thomas Lamprecht
2020-09-10 11:34 ` Alexandre DERUMIER
2020-09-10 18:21 ` Thomas Lamprecht
2020-09-14 4:54 ` Alexandre DERUMIER
2020-09-14 7:14 ` Dietmar Maurer
2020-09-14 8:27 ` Alexandre DERUMIER
2020-09-14 8:51 ` Thomas Lamprecht
2020-09-14 15:45 ` Alexandre DERUMIER
2020-09-15 5:45 ` dietmar
2020-09-15 6:27 ` Alexandre DERUMIER
2020-09-15 7:13 ` dietmar
2020-09-15 8:42 ` Alexandre DERUMIER
2020-09-15 9:35 ` Alexandre DERUMIER
2020-09-15 9:46 ` Thomas Lamprecht
2020-09-15 10:15 ` Alexandre DERUMIER
2020-09-15 11:04 ` Alexandre DERUMIER
2020-09-15 12:49 ` Alexandre DERUMIER
2020-09-15 13:00 ` Thomas Lamprecht
2020-09-15 14:09 ` Alexandre DERUMIER
2020-09-15 14:19 ` Alexandre DERUMIER
2020-09-15 14:32 ` Thomas Lamprecht
2020-09-15 14:57 ` Alexandre DERUMIER
2020-09-15 15:58 ` Alexandre DERUMIER
2020-09-16 7:34 ` Alexandre DERUMIER
2020-09-16 7:58 ` Alexandre DERUMIER
2020-09-16 8:30 ` Alexandre DERUMIER
2020-09-16 8:53 ` Alexandre DERUMIER
[not found] ` <1894376736.864562.1600253445817.JavaMail.zimbra@odiso.com>
2020-09-16 13:15 ` Alexandre DERUMIER
2020-09-16 14:45 ` Thomas Lamprecht
2020-09-16 15:17 ` Alexandre DERUMIER
2020-09-17 9:21 ` Fabian Grünbichler
2020-09-17 9:59 ` Alexandre DERUMIER
2020-09-17 10:02 ` Alexandre DERUMIER
2020-09-17 11:35 ` Thomas Lamprecht
2020-09-20 23:54 ` Alexandre DERUMIER
2020-09-22 5:43 ` Alexandre DERUMIER
2020-09-24 14:02 ` Fabian Grünbichler
2020-09-24 14:29 ` Alexandre DERUMIER
2020-09-24 18:07 ` Alexandre DERUMIER
2020-09-25 6:44 ` Alexandre DERUMIER
2020-09-25 7:15 ` Alexandre DERUMIER
2020-09-25 9:19 ` Fabian Grünbichler
2020-09-25 9:46 ` Alexandre DERUMIER
2020-09-25 12:51 ` Fabian Grünbichler
2020-09-25 16:29 ` Alexandre DERUMIER
2020-09-28 9:17 ` Fabian Grünbichler
2020-09-28 9:35 ` Alexandre DERUMIER
2020-09-28 15:59 ` Alexandre DERUMIER
2020-09-29 5:30 ` Alexandre DERUMIER
2020-09-29 8:51 ` Fabian Grünbichler
2020-09-29 9:37 ` Alexandre DERUMIER
2020-09-29 10:52 ` Alexandre DERUMIER
2020-09-29 11:43 ` Alexandre DERUMIER
2020-09-29 11:50 ` Alexandre DERUMIER
2020-09-29 13:28 ` Fabian Grünbichler
2020-09-29 13:52 ` Alexandre DERUMIER
2020-09-30 6:09 ` Alexandre DERUMIER
2020-09-30 6:26 ` Thomas Lamprecht
2020-09-15 7:58 ` Thomas Lamprecht
2020-12-29 14:21 ` Josef Johansson
2020-09-04 15:46 ` Alexandre DERUMIER
2020-09-30 15:50 ` Thomas Lamprecht
2020-10-15 9:16 ` Eneko Lacunza
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox