From: Josef Johansson <josef@oderland.se>
To: pve-devel@lists.proxmox.com
Subject: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Date: Tue, 29 Dec 2020 15:21:07 +0100 [thread overview]
Message-ID: <a67bdb3c-f2a7-dbcd-f2bf-9ea49db7db14@oderland.se> (raw)
In-Reply-To: <1044807310.366666.1599222580644.JavaMail.zimbra@odiso.com>
Hi,
On 9/4/20 2:29 PM, Alexandre DERUMIER wrote:
> BTW,
>
> do you think it could be possible to add an extra optionnal layer of security check, not related to corosync ?
>
> I'm still afraid of this corosync bug since years, and still don't use HA. (or I have tried to enable it 2months ago,and this give me a disaster yesterday..)
Using corosync to enable HA is a bit scary TBH, not sure how people
solve this, but I'd rather have the logic outside the cluster so it can
make a solid decision whether a node should be rebooted or not, and
choose to reboot via iDRAC if all checks fail. Maybe use the metrics
support to facilitate this?
> Something like an extra heartbeat between nodes daemons, and check if we also have quorum with theses heartbeats ?
>
>
>
> ----- Mail original -----
> De: "aderumier" <aderumier@odiso.com>
> À: "pve-devel" <pve-devel@pve.proxmox.com>
> Envoyé: Jeudi 3 Septembre 2020 16:11:56
> Objet: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>
> Hi,
>
> I had a problem this morning with corosync, after shutting down cleanly a node from the cluster.
> This is a 14 nodes cluster, node2 was shutted. (with "halt" command)
>
> HA was actived, and all nodes have been rebooted.
> (I have already see this problem some months ago, without HA enabled, and it was like half of the nodes didn't see others nodes, like 2 cluster formations)
>
> Some users have reported similar problems when adding a new node
> https://forum.proxmox.com/threads/quorum-lost-when-adding-new-7th-node.75197/
>
>
> Here the full logs of the nodes, for each node I'm seeing quorum with 13 nodes. (so it's seem to be ok).
> I didn't have time to look live on server, as HA have restarted them.
> So, I don't known if it's a corosync bug or something related to crm,lrm,pmxcs (but I don't see any special logs)
>
> Only node7 have survived a little bit longer,and I had stop lrm/crm to avoid reboot.
>
> libknet1: 1.15-pve1
> corosync: 3.0.3-pve1
>
>
> Any ideas before submiting bug to corosync mailing ? (I'm seeing new libknet && corosync version on their github, I don't have read the changelog yet)
>
>
>
>
> node1
> -----
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm1 corosync[3678]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm1 corosync[3678]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm1 corosync[3678]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [dcdb] notice: cpg_send_message retried 1 times
> Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 has no active links
>
> --> reboot
>
> node2 : shutdown log
> -----
> Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd received signal 15: Terminated
> Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_00] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302Y9120CGN.ata.state
> Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_01] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302MG120CGN.ata.state
> Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd is exiting (exit status 0)
> Sep 3 10:39:05 m6kvm2 nrpe[20095]: Caught SIGTERM - shutting down...
> Sep 3 10:39:05 m6kvm2 nrpe[20095]: Daemon shutdown
> Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> starting task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root@pam:
> Sep 3 10:39:07 m6kvm2 pve-guests[38552]: all VMs and CTs stopped
> Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> end task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root@pam: OK
> Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: received signal TERM
> Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server closing
> Sep 3 10:39:07 m6kvm2 spiceproxy[36572]: worker exit
> Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: worker 36572 finished
> Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server stopped
> Sep 3 10:39:07 m6kvm2 pvestatd[3786]: received signal TERM
> Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server closing
> Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server stopped
> Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: received signal TERM
> Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: got shutdown request with shutdown policy 'conditional'
> Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: shutdown LRM, stop all services
> Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: received terminate request (signal)
> Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: stopping pvefw logger
> Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: watchdog closed (disabled)
> Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: server stopped
> Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: received signal TERM
> Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: server received shutdown request
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: received signal TERM
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server closing
> Sep 3 10:39:10 m6kvm2 pveproxy[29731]: worker exit
> Sep 3 10:39:10 m6kvm2 pveproxy[30873]: worker exit
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 30873 finished
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 31391 finished
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 29731 finished
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server stopped
> Sep 3 10:39:14 m6kvm2 pve-ha-crm[12847]: server stopped
>
>
> node3
> -----
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm3 corosync[30580]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm3 corosync[30580]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm3 corosync[30580]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: cpg_send_message retried 1 times
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 has no active links
>
>
> node4
> -----
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: all data is up to date
> Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:20 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
>
> node5
> -----
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
>
>
>
>
>
> node6
> -----
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 has no active links
>
>
>
> node7
> -----
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
> Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:20 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> ---> here the others nodes reboot almost at the same time
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 3 link: 0 is down
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 14 link: 0 is down
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 has no active links
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 has no active links
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 13 link: 0 is down
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 8 link: 0 is down
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 6 link: 0 is down
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 10 link: 0 is down
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 has no active links
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 has no active links
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 has no active links
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 has no active links
> Sep 3 10:40:28 m6kvm7 corosync[15467]: [TOTEM ] Token has not been received in 4505 ms
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 11 link: 0 is down
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 9 link: 0 is down
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 has no active links
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 has no active links
> Sep 3 10:40:30 m6kvm7 corosync[15467]: [TOTEM ] A processor failed, forming new configuration.
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 4 link: 0 is down
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 12 link: 0 is down
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 1 link: 0 is down
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 has no active links
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 has no active links
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 has no active links
> Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] link: host: 5 link: 0 is down
> Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 has no active links
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] A new membership (7.82f) was formed. Members left: 1 3 4 5 6 8 9 10 11 12 13 14
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] Failed to receive the leave message. failed: 1 3 4 5 6 8 9 10 11 12 13 14
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 12 received
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] Members[1]: 7
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: dfsm_deliver_queue: queue length 51
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: node lost quorum
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: members: 7/15892
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: received write while not quorate - trigger resync
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: leaving CPG group
> Sep 3 10:40:41 m6kvm7 pve-ha-crm[16196]: node 'm6kvm2': state changed from 'online' => 'unknown'
> Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] notice: start cluster connection
> Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: cpg_join failed: 14
> Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: can't initialize service
> Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892
> Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during cfs-locked 'domain-ha' operation: no quorum!
> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs lock update failed - Permission denied
> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - cfs lock update failed - Permission denied
> Sep 3 10:40:56 m6kvm7 pve-ha-lrm[16140]: status change active => lost_agent_lock
> Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change master => lost_manager_lock
> Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: watchdog closed (disabled)
> Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change lost_manager_lock => wait_for_quorum
> Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] rx: host: 4 link: 0 is up
> Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [TOTEM ] A new membership (4.834) was formed. Members joined: 4
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 4/3965, 7/15892
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: members: 4/3965, 7/15892
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [QUORUM] Members[2]: 4 7
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 4/3965/00000002)
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 4/3965/00000002)
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 7/15892
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 7/15892
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: start sending inode updates
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: sent all (4) updates
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received all states
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] rx: host: 3 link: 0 is up
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [TOTEM ] A new membership (3.838) was formed. Members joined: 3
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 3/3613, 4/3965, 7/15892
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: members: 3/3613, 4/3965, 7/15892
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [QUORUM] Members[3]: 3 4 7
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 3/3613/00000002)
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 3/3613/00000002)
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 4/3965
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 4/3965, 7/15892
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received all states
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: dfsm_deliver_queue: queue length 1
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] rx: host: 1 link: 0 is up
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.83c) was formed. Members joined: 1
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3552, 3/3613, 4/3965, 7/15892
> Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
> Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3552, 3/3613, 4/3965, 7/15892
> Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [QUORUM] Members[4]: 1 3 4 7
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3552/00000002)
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3552/00000002)
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 3/3613
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 3/3613, 4/3965, 7/15892
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received all states
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
> Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:49:18 m6kvm7 pmxcfs[15892]: [status] notice: received log
>
>
>
>
> node8
> -----
>
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
> Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
> Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
> Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
>
>
> --> reboot
>
>
>
> node9
> -----
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: all data is up to date
> Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:20 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
>
>
> --> reboot
>
>
>
> node10
> ------
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: all data is up to date
> Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log
> Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log
>
> --> reboot
>
>
> node11
> ------
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
> Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
> Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
> Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
> Sep 3 10:40:20 m6kvm11 pmxcfs[12983]: [status] notice: received log
>
>
>
>
> node12
> ------
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [dcdb] notice: cpg_send_message retried 1 times
> Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:20 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
>
>
>
>
>
> node13
> ------
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: cpg_send_message retried 1 times
> Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: all data is up to date
> Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
> Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
> Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
> Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
> --> reboot
>
> node14
> ------
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 has no active links
>
> --> reboot
>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
--
Med vänliga hälsningar
Josef Johansson
next prev parent reply other threads:[~2020-12-29 14:51 UTC|newest]
Thread overview: 84+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-03 14:11 Alexandre DERUMIER
2020-09-04 12:29 ` Alexandre DERUMIER
2020-09-04 15:42 ` Dietmar Maurer
2020-09-05 13:32 ` Alexandre DERUMIER
2020-09-05 15:23 ` dietmar
2020-09-05 17:30 ` Alexandre DERUMIER
2020-09-06 4:21 ` dietmar
2020-09-06 5:36 ` Alexandre DERUMIER
2020-09-06 6:33 ` Alexandre DERUMIER
2020-09-06 8:43 ` Alexandre DERUMIER
2020-09-06 12:14 ` dietmar
2020-09-06 12:19 ` dietmar
2020-09-07 7:00 ` Thomas Lamprecht
2020-09-07 7:19 ` Alexandre DERUMIER
2020-09-07 8:18 ` dietmar
2020-09-07 9:32 ` Alexandre DERUMIER
2020-09-07 13:23 ` Alexandre DERUMIER
2020-09-08 4:41 ` dietmar
2020-09-08 7:11 ` Alexandre DERUMIER
2020-09-09 20:05 ` Thomas Lamprecht
2020-09-10 4:58 ` Alexandre DERUMIER
2020-09-10 8:21 ` Thomas Lamprecht
2020-09-10 11:34 ` Alexandre DERUMIER
2020-09-10 18:21 ` Thomas Lamprecht
2020-09-14 4:54 ` Alexandre DERUMIER
2020-09-14 7:14 ` Dietmar Maurer
2020-09-14 8:27 ` Alexandre DERUMIER
2020-09-14 8:51 ` Thomas Lamprecht
2020-09-14 15:45 ` Alexandre DERUMIER
2020-09-15 5:45 ` dietmar
2020-09-15 6:27 ` Alexandre DERUMIER
2020-09-15 7:13 ` dietmar
2020-09-15 8:42 ` Alexandre DERUMIER
2020-09-15 9:35 ` Alexandre DERUMIER
2020-09-15 9:46 ` Thomas Lamprecht
2020-09-15 10:15 ` Alexandre DERUMIER
2020-09-15 11:04 ` Alexandre DERUMIER
2020-09-15 12:49 ` Alexandre DERUMIER
2020-09-15 13:00 ` Thomas Lamprecht
2020-09-15 14:09 ` Alexandre DERUMIER
2020-09-15 14:19 ` Alexandre DERUMIER
2020-09-15 14:32 ` Thomas Lamprecht
2020-09-15 14:57 ` Alexandre DERUMIER
2020-09-15 15:58 ` Alexandre DERUMIER
2020-09-16 7:34 ` Alexandre DERUMIER
2020-09-16 7:58 ` Alexandre DERUMIER
2020-09-16 8:30 ` Alexandre DERUMIER
2020-09-16 8:53 ` Alexandre DERUMIER
[not found] ` <1894376736.864562.1600253445817.JavaMail.zimbra@odiso.com>
2020-09-16 13:15 ` Alexandre DERUMIER
2020-09-16 14:45 ` Thomas Lamprecht
2020-09-16 15:17 ` Alexandre DERUMIER
2020-09-17 9:21 ` Fabian Grünbichler
2020-09-17 9:59 ` Alexandre DERUMIER
2020-09-17 10:02 ` Alexandre DERUMIER
2020-09-17 11:35 ` Thomas Lamprecht
2020-09-20 23:54 ` Alexandre DERUMIER
2020-09-22 5:43 ` Alexandre DERUMIER
2020-09-24 14:02 ` Fabian Grünbichler
2020-09-24 14:29 ` Alexandre DERUMIER
2020-09-24 18:07 ` Alexandre DERUMIER
2020-09-25 6:44 ` Alexandre DERUMIER
2020-09-25 7:15 ` Alexandre DERUMIER
2020-09-25 9:19 ` Fabian Grünbichler
2020-09-25 9:46 ` Alexandre DERUMIER
2020-09-25 12:51 ` Fabian Grünbichler
2020-09-25 16:29 ` Alexandre DERUMIER
2020-09-28 9:17 ` Fabian Grünbichler
2020-09-28 9:35 ` Alexandre DERUMIER
2020-09-28 15:59 ` Alexandre DERUMIER
2020-09-29 5:30 ` Alexandre DERUMIER
2020-09-29 8:51 ` Fabian Grünbichler
2020-09-29 9:37 ` Alexandre DERUMIER
2020-09-29 10:52 ` Alexandre DERUMIER
2020-09-29 11:43 ` Alexandre DERUMIER
2020-09-29 11:50 ` Alexandre DERUMIER
2020-09-29 13:28 ` Fabian Grünbichler
2020-09-29 13:52 ` Alexandre DERUMIER
2020-09-30 6:09 ` Alexandre DERUMIER
2020-09-30 6:26 ` Thomas Lamprecht
2020-09-15 7:58 ` Thomas Lamprecht
2020-12-29 14:21 ` Josef Johansson [this message]
2020-09-04 15:46 ` Alexandre DERUMIER
2020-09-30 15:50 ` Thomas Lamprecht
2020-10-15 9:16 ` Eneko Lacunza
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a67bdb3c-f2a7-dbcd-f2bf-9ea49db7db14@oderland.se \
--to=josef@oderland.se \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal