From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <josef@oderland.se> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 7835C646D5 for <pve-devel@lists.proxmox.com>; Tue, 29 Dec 2020 15:51:44 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5E1F319B91 for <pve-devel@lists.proxmox.com>; Tue, 29 Dec 2020 15:51:44 +0100 (CET) Received: from office.oderland.com (office.oderland.com [91.201.60.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 09D8B19B46 for <pve-devel@lists.proxmox.com>; Tue, 29 Dec 2020 15:51:40 +0100 (CET) Received: from [193.180.18.13] (port=33480 helo=[10.137.0.13]) by office.oderland.com with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from <josef@oderland.se>) id 1kuFs4-009yd2-Oh for pve-devel@lists.proxmox.com; Tue, 29 Dec 2020 15:21:09 +0100 To: pve-devel@lists.proxmox.com References: <216436814.339545.1599142316781.JavaMail.zimbra@odiso.com> <1044807310.366666.1599222580644.JavaMail.zimbra@odiso.com> From: Josef Johansson <josef@oderland.se> Message-ID: <a67bdb3c-f2a7-dbcd-f2bf-9ea49db7db14@oderland.se> Date: Tue, 29 Dec 2020 15:21:07 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Thunderbird/83.0 MIME-Version: 1.0 In-Reply-To: <1044807310.366666.1599222580644.JavaMail.zimbra@odiso.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - office.oderland.com X-AntiAbuse: Original Domain - lists.proxmox.com X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - oderland.se X-Get-Message-Sender-Via: office.oderland.com: authenticated_id: josjoh@oderland.se X-Authenticated-Sender: office.oderland.com: josjoh@oderland.se X-SPAM-LEVEL: Spam detection results: 0 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -2.698 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> List-Post: <mailto:pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Tue, 29 Dec 2020 14:51:44 -0000 Hi, On 9/4/20 2:29 PM, Alexandre DERUMIER wrote: > BTW, > > do you think it could be possible to add an extra optionnal layer of security check, not related to corosync ? > > I'm still afraid of this corosync bug since years, and still don't use HA. (or I have tried to enable it 2months ago,and this give me a disaster yesterday..) Using corosync to enable HA is a bit scary TBH, not sure how people solve this, but I'd rather have the logic outside the cluster so it can make a solid decision whether a node should be rebooted or not, and choose to reboot via iDRAC if all checks fail. Maybe use the metrics support to facilitate this? > Something like an extra heartbeat between nodes daemons, and check if we also have quorum with theses heartbeats ? > > > > ----- Mail original ----- > De: "aderumier" <aderumier@odiso.com> > À: "pve-devel" <pve-devel@pve.proxmox.com> > Envoyé: Jeudi 3 Septembre 2020 16:11:56 > Objet: [pve-devel] corosync bug: cluster break after 1 node clean shutdown > > Hi, > > I had a problem this morning with corosync, after shutting down cleanly a node from the cluster. > This is a 14 nodes cluster, node2 was shutted. (with "halt" command) > > HA was actived, and all nodes have been rebooted. > (I have already see this problem some months ago, without HA enabled, and it was like half of the nodes didn't see others nodes, like 2 cluster formations) > > Some users have reported similar problems when adding a new node > https://forum.proxmox.com/threads/quorum-lost-when-adding-new-7th-node.75197/ > > > Here the full logs of the nodes, for each node I'm seeing quorum with 13 nodes. (so it's seem to be ok). > I didn't have time to look live on server, as HA have restarted them. > So, I don't known if it's a corosync bug or something related to crm,lrm,pmxcs (but I don't see any special logs) > > Only node7 have survived a little bit longer,and I had stop lrm/crm to avoid reboot. > > libknet1: 1.15-pve1 > corosync: 3.0.3-pve1 > > > Any ideas before submiting bug to corosync mailing ? (I'm seeing new libknet && corosync version on their github, I don't have read the changelog yet) > > > > > node1 > ----- > Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm1 corosync[3678]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm1 corosync[3678]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm1 corosync[3678]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [dcdb] notice: cpg_send_message retried 1 times > Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: received all states > Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 has no active links > > --> reboot > > node2 : shutdown log > ----- > Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd received signal 15: Terminated > Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_00] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302Y9120CGN.ata.state > Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_01] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302MG120CGN.ata.state > Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd is exiting (exit status 0) > Sep 3 10:39:05 m6kvm2 nrpe[20095]: Caught SIGTERM - shutting down... > Sep 3 10:39:05 m6kvm2 nrpe[20095]: Daemon shutdown > Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> starting task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root@pam: > Sep 3 10:39:07 m6kvm2 pve-guests[38552]: all VMs and CTs stopped > Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> end task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root@pam: OK > Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: received signal TERM > Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server closing > Sep 3 10:39:07 m6kvm2 spiceproxy[36572]: worker exit > Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: worker 36572 finished > Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server stopped > Sep 3 10:39:07 m6kvm2 pvestatd[3786]: received signal TERM > Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server closing > Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server stopped > Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: received signal TERM > Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: got shutdown request with shutdown policy 'conditional' > Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: shutdown LRM, stop all services > Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: received terminate request (signal) > Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: stopping pvefw logger > Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: watchdog closed (disabled) > Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: server stopped > Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: received signal TERM > Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: server received shutdown request > Sep 3 10:39:10 m6kvm2 pveproxy[24735]: received signal TERM > Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server closing > Sep 3 10:39:10 m6kvm2 pveproxy[29731]: worker exit > Sep 3 10:39:10 m6kvm2 pveproxy[30873]: worker exit > Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 30873 finished > Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 31391 finished > Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 29731 finished > Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server stopped > Sep 3 10:39:14 m6kvm2 pve-ha-crm[12847]: server stopped > > > node3 > ----- > Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm3 corosync[30580]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm3 corosync[30580]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm3 corosync[30580]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: cpg_send_message retried 1 times > Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: received all states > Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 has no active links > > > node4 > ----- > Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm4 corosync[4085]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm4 corosync[4085]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm4 corosync[4085]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: received all states > Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: all data is up to date > Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 has no active links > Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:20 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log > > node5 > ----- > Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm5 corosync[41830]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm5 corosync[41830]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm5 corosync[41830]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: received all states > Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 has no active links > Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log > > > > > > node6 > ----- > Sep 3 10:39:16 m6kvm6 corosync[36694]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm6 corosync[36694]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm6 corosync[36694]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 has no active links > > > > node7 > ----- > Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm7 corosync[15467]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: received all states > Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date > Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 has no active links > Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:20 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log > ---> here the others nodes reboot almost at the same time > Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 3 link: 0 is down > Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 14 link: 0 is down > Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) > Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 has no active links > Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 (passive) best link: 0 (pri: 1) > Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 has no active links > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 13 link: 0 is down > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 8 link: 0 is down > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 6 link: 0 is down > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 10 link: 0 is down > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 (passive) best link: 0 (pri: 1) > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 has no active links > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 (passive) best link: 0 (pri: 1) > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 has no active links > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 1) > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 has no active links > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 (passive) best link: 0 (pri: 1) > Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 has no active links > Sep 3 10:40:28 m6kvm7 corosync[15467]: [TOTEM ] Token has not been received in 4505 ms > Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 11 link: 0 is down > Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 9 link: 0 is down > Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 (passive) best link: 0 (pri: 1) > Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 has no active links > Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 (passive) best link: 0 (pri: 1) > Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 has no active links > Sep 3 10:40:30 m6kvm7 corosync[15467]: [TOTEM ] A processor failed, forming new configuration. > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 4 link: 0 is down > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 12 link: 0 is down > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 1 link: 0 is down > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1) > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 has no active links > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 (passive) best link: 0 (pri: 1) > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 has no active links > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1) > Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 has no active links > Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] link: host: 5 link: 0 is down > Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 (passive) best link: 0 (pri: 1) > Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 has no active links > Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] A new membership (7.82f) was formed. Members left: 1 3 4 5 6 8 9 10 11 12 13 14 > Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] Failed to receive the leave message. failed: 1 3 4 5 6 8 9 10 11 12 13 14 > Sep 3 10:40:41 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 12 received > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892 > Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] This node is within the non-primary component and will NOT provide any services. > Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] Members[1]: 7 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date > Sep 3 10:40:41 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: dfsm_deliver_queue: queue length 51 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: node lost quorum > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: members: 7/15892 > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: received write while not quorate - trigger resync > Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: leaving CPG group > Sep 3 10:40:41 m6kvm7 pve-ha-crm[16196]: node 'm6kvm2': state changed from 'online' => 'unknown' > Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] notice: start cluster connection > Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: cpg_join failed: 14 > Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: can't initialize service > Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892 > Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date > Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during cfs-locked 'domain-ha' operation: no quorum! > Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds) > Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds) > Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs lock update failed - Permission denied > Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - cfs lock update failed - Permission denied > Sep 3 10:40:56 m6kvm7 pve-ha-lrm[16140]: status change active => lost_agent_lock > Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change master => lost_manager_lock > Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: watchdog closed (disabled) > Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change lost_manager_lock => wait_for_quorum > Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] rx: host: 4 link: 0 is up > Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1) > Sep 3 10:43:24 m6kvm7 corosync[15467]: [TOTEM ] A new membership (4.834) was formed. Members joined: 4 > Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 4/3965, 7/15892 > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: members: 4/3965, 7/15892 > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation > Sep 3 10:43:24 m6kvm7 corosync[15467]: [QUORUM] Members[2]: 4 7 > Sep 3 10:43:24 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 4/3965/00000002) > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 4/3965/00000002) > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 7/15892 > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 7/15892 > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: start sending inode updates > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: sent all (4) updates > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received all states > Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date > Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] rx: host: 3 link: 0 is up > Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) > Sep 3 10:44:05 m6kvm7 corosync[15467]: [TOTEM ] A new membership (3.838) was formed. Members joined: 3 > Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 3/3613, 4/3965, 7/15892 > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: members: 3/3613, 4/3965, 7/15892 > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation > Sep 3 10:44:05 m6kvm7 corosync[15467]: [QUORUM] Members[3]: 3 4 7 > Sep 3 10:44:05 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 3/3613/00000002) > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 3/3613/00000002) > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 4/3965 > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 4/3965, 7/15892 > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received all states > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date > Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: dfsm_deliver_queue: queue length 1 > Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] rx: host: 1 link: 0 is up > Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1) > Sep 3 10:44:31 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.83c) was formed. Members joined: 1 > Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received > Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3552, 3/3613, 4/3965, 7/15892 > Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation > Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3552, 3/3613, 4/3965, 7/15892 > Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation > Sep 3 10:44:31 m6kvm7 corosync[15467]: [QUORUM] Members[4]: 1 3 4 7 > Sep 3 10:44:31 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3552/00000002) > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3552/00000002) > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 3/3613 > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 3/3613, 4/3965, 7/15892 > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received all states > Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date > Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log > Sep 3 10:49:18 m6kvm7 pmxcfs[15892]: [status] notice: received log > > > > > node8 > ----- > > Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm8 corosync[24361]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm8 corosync[24361]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm8 corosync[24361]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: received all states > Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 has no active links > Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log > Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log > Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log > Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log > > > --> reboot > > > > node9 > ----- > Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm9 corosync[22340]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm9 corosync[22340]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm9 corosync[22340]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: received all states > Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: all data is up to date > Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 has no active links > Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:20 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log > > > --> reboot > > > > node10 > ------ > Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm10 corosync[41458]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm10 corosync[41458]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm10 corosync[41458]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: received all states > Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: all data is up to date > Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 has no active links > Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log > Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log > > --> reboot > > > node11 > ------ > Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm11 corosync[12455]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm11 corosync[12455]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm11 corosync[12455]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: received all states > Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 has no active links > Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log > Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log > Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log > Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log > Sep 3 10:40:20 m6kvm11 pmxcfs[12983]: [status] notice: received log > > > > > node12 > ------ > Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm12 corosync[43716]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm12 corosync[43716]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm12 corosync[43716]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [dcdb] notice: cpg_send_message retried 1 times > Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: received all states > Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 has no active links > Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:20 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log > > > > > > node13 > ------ > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm13 corosync[39182]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm13 corosync[39182]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm13 corosync[39182]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: cpg_send_message retried 1 times > Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: received all states > Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: all data is up to date > Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 has no active links > Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log > Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log > Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log > Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log > --> reboot > > node14 > ------ > Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930 > Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: starting data syncronisation > Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: received sync request (epoch 1/3527/00000002) > Sep 3 10:39:16 m6kvm14 corosync[42413]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2 > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received > Sep 3 10:39:16 m6kvm14 corosync[42413]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14 > Sep 3 10:39:16 m6kvm14 corosync[42413]: [MAIN ] Completed service synchronization, ready to provide service. > Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: received all states > Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: all data is up to date > Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] link: host: 2 link: 0 is down > Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1) > Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 has no active links > > --> reboot > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel -- Med vänliga hälsningar Josef Johansson