From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <aderumier@odiso.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id BFCA3610A2 for <pve-devel@lists.proxmox.com>; Fri, 4 Sep 2020 14:30:21 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B46AC20A6C for <pve-devel@lists.proxmox.com>; Fri, 4 Sep 2020 14:29:51 +0200 (CEST) Received: from mailpro.odiso.net (mailpro.odiso.net [89.248.211.110]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 6C65D20A59 for <pve-devel@lists.proxmox.com>; Fri, 4 Sep 2020 14:29:41 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id 2D5B5181EC40; Fri, 4 Sep 2020 14:29:41 +0200 (CEST) Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id zCuI2SQmBx2J; Fri, 4 Sep 2020 14:29:41 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id 06B73181EC5C; Fri, 4 Sep 2020 14:29:41 +0200 (CEST) X-Virus-Scanned: amavisd-new at mailpro.odiso.com Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id aWOqDB75AUx2; Fri, 4 Sep 2020 14:29:40 +0200 (CEST) Received: from mailpro.odiso.net (mailpro.odiso.net [10.1.31.111]) by mailpro.odiso.net (Postfix) with ESMTP id D692B181EC5B; Fri, 4 Sep 2020 14:29:40 +0200 (CEST) Date: Fri, 4 Sep 2020 14:29:40 +0200 (CEST) From: Alexandre DERUMIER <aderumier@odiso.com> To: Proxmox VE development discussion <pve-devel@lists.proxmox.com> Cc: pve-devel <pve-devel@pve.proxmox.com> Message-ID: <1044807310.366666.1599222580644.JavaMail.zimbra@odiso.com> In-Reply-To: <216436814.339545.1599142316781.JavaMail.zimbra@odiso.com> References: <216436814.339545.1599142316781.JavaMail.zimbra@odiso.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Mailer: Zimbra 8.8.12_GA_3866 (ZimbraWebClient - GC83 (Linux)/8.8.12_GA_3844) Thread-Topic: corosync bug: cluster break after 1 node clean shutdown Thread-Index: BDCUNDa2t04O78rYKqmARZV+ZCSDikcj1/Ig X-SPAM-LEVEL: Spam detection results: 0 AWL 0.036 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> List-Post: <mailto:pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Fri, 04 Sep 2020 12:30:21 -0000 BTW, do you think it could be possible to add an extra optionnal layer of securi= ty check, not related to corosync ? I'm still afraid of this corosync bug since years, and still don't use HA. = (or I have tried to enable it 2months ago,and this give me a disaster yeste= rday..) Something like an extra heartbeat between nodes daemons, and check if we al= so have quorum with theses heartbeats ? ----- Mail original ----- De: "aderumier" <aderumier@odiso.com> =C3=80: "pve-devel" <pve-devel@pve.proxmox.com> Envoy=C3=A9: Jeudi 3 Septembre 2020 16:11:56 Objet: [pve-devel] corosync bug: cluster break after 1 node clean shutdown Hi,=20 I had a problem this morning with corosync, after shutting down cleanly a n= ode from the cluster.=20 This is a 14 nodes cluster, node2 was shutted. (with "halt" command)=20 HA was actived, and all nodes have been rebooted.=20 (I have already see this problem some months ago, without HA enabled, and i= t was like half of the nodes didn't see others nodes, like 2 cluster format= ions)=20 Some users have reported similar problems when adding a new node=20 https://forum.proxmox.com/threads/quorum-lost-when-adding-new-7th-node.7519= 7/=20 Here the full logs of the nodes, for each node I'm seeing quorum with 13 no= des. (so it's seem to be ok).=20 I didn't have time to look live on server, as HA have restarted them.=20 So, I don't known if it's a corosync bug or something related to crm,lrm,pm= xcs (but I don't see any special logs)=20 Only node7 have survived a little bit longer,and I had stop lrm/crm to avoi= d reboot.=20 libknet1: 1.15-pve1=20 corosync: 3.0.3-pve1=20 Any ideas before submiting bug to corosync mailing ? (I'm seeing new libkne= t && corosync version on their github, I don't have read the changelog yet)= =20 node1=20 -----=20 Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: members: 1/3527, 3/273= 73, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983= , 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: starting data syncroni= sation=20 Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: members: 1/3527, 3/27373= , 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, = 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: starting data syncronisa= tion=20 Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: received sync request = (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: received sync request (e= poch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm1 corosync[3678]: [TOTEM ] A new membership (1.82b) was= formed. Members left: 2=20 Sep 3 10:39:16 m6kvm1 corosync[3678]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9= 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm1 corosync[3678]: [MAIN ] Completed service synchroniza= tion, ready to provide service.=20 Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [dcdb] notice: cpg_send_message retried= 1 times=20 Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: received all states=20 Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: all data is up to date= =20 Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] link: host: 2 link: 0 is down= =20 Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 (passive) best = link: 0 (pri: 1)=20 Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 has no active l= inks=20 --> reboot=20 node2 : shutdown log=20 -----=20 Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd received signal 15: Terminated= =20 Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_00] = [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-= PHWA640302Y9120CGN.ata.state=20 Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_01] = [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-= PHWA640302MG120CGN.ata.state=20 Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd is exiting (exit status 0)=20 Sep 3 10:39:05 m6kvm2 nrpe[20095]: Caught SIGTERM - shutting down...=20 Sep 3 10:39:05 m6kvm2 nrpe[20095]: Daemon shutdown=20 Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> starting task UPID:m6kv= m2:00009698:93A98FFD:5F50ABAB:stopall::root@pam:=20 Sep 3 10:39:07 m6kvm2 pve-guests[38552]: all VMs and CTs stopped=20 Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root@pam> end task UPID:m6kvm2:00= 009698:93A98FFD:5F50ABAB:stopall::root@pam: OK=20 Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: received signal TERM=20 Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server closing=20 Sep 3 10:39:07 m6kvm2 spiceproxy[36572]: worker exit=20 Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: worker 36572 finished=20 Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server stopped=20 Sep 3 10:39:07 m6kvm2 pvestatd[3786]: received signal TERM=20 Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server closing=20 Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server stopped=20 Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: received signal TERM=20 Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: got shutdown request with shutdown= policy 'conditional'=20 Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: shutdown LRM, stop all services=20 Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: received terminate request (sign= al)=20 Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: stopping pvefw logger=20 Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: watchdog closed (disabled)=20 Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: server stopped=20 Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: received signal TERM=20 Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: server received shutdown request= =20 Sep 3 10:39:10 m6kvm2 pveproxy[24735]: received signal TERM=20 Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server closing=20 Sep 3 10:39:10 m6kvm2 pveproxy[29731]: worker exit=20 Sep 3 10:39:10 m6kvm2 pveproxy[30873]: worker exit=20 Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 30873 finished=20 Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 31391 finished=20 Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 29731 finished=20 Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server stopped=20 Sep 3 10:39:14 m6kvm2 pve-ha-crm[12847]: server stopped=20 node3=20 -----=20 Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: starting data syncronis= ation=20 Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: members: 1/3527, 3/27= 373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/1298= 3, 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: starting data syncron= isation=20 Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: received sync request (= epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm3 corosync[30580]: [TOTEM ] A new membership (1.82b) wa= s formed. Members left: 2=20 Sep 3 10:39:16 m6kvm3 corosync[30580]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 = 9 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm3 corosync[30580]: [MAIN ] Completed service synchroniz= ation, ready to provide service.=20 Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: cpg_send_message retrie= d 1 times=20 Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: received sync request= (epoch 1/3527/00000002)=20 Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: received all states= =20 Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: all data is up to dat= e=20 Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] link: host: 2 link: 0 is dow= n=20 Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 has no active = links=20 node4=20 -----=20 Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: members: 1/3527, 3/27373= , 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, = 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: starting data syncronisa= tion=20 Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: members: 1/3527, 3/273= 73, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983= , 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: starting data syncroni= sation=20 Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: received sync request (e= poch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: received sync request = (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [TOTEM ] A new membership (1.82b) was= formed. Members left: 2=20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received= =20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9= 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm4 corosync[4085]: [MAIN ] Completed service synchroniza= tion, ready to provide service.=20 Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: received all states=20 Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: all data is up to date= =20 Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] link: host: 2 link: 0 is down= =20 Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 (passive) best = link: 0 (pri: 1)=20 Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 has no active l= inks=20 Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:20 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log=20 node5=20 -----=20 Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: members: 1/3527, 3/2737= 3, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983,= 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: starting data syncronis= ation=20 Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: members: 1/3527, 3/27= 373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/1298= 3, 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: starting data syncron= isation=20 Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: received sync request (= epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: received sync request= (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [TOTEM ] A new membership (1.82b) wa= s formed. Members left: 2=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 = 9 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm5 corosync[41830]: [MAIN ] Completed service synchroniz= ation, ready to provide service.=20 Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: received all states= =20 Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: all data is up to dat= e=20 Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] link: host: 2 link: 0 is dow= n=20 Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 has no active = links=20 Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log=20 node6=20 -----=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [TOTEM ] A new membership (1.82b) wa= s formed. Members left: 2=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 = 9 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm6 corosync[36694]: [MAIN ] Completed service synchroniz= ation, ready to provide service.=20 Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] link: host: 2 link: 0 is dow= n=20 Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 has no active = links=20 node7=20 -----=20 Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3527, 3/2737= 3, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983,= 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronis= ation=20 Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3527, 3/27= 373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/1298= 3, 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncron= isation=20 Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (= epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: received sync request= (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.82b) wa= s formed. Members left: 2=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 = 9 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm7 corosync[15467]: [MAIN ] Completed service synchroniz= ation, ready to provide service.=20 Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: received all states= =20 Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to dat= e=20 Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] link: host: 2 link: 0 is dow= n=20 Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 has no active = links=20 Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:20 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 ---> here the others nodes reboot almost at the same time=20 Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 3 link: 0 is dow= n=20 Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 14 link: 0 is do= wn=20 Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 has no active = links=20 Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 (passive) bes= t link: 0 (pri: 1)=20 Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 has no active= links=20 Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 13 link: 0 is do= wn=20 Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 8 link: 0 is dow= n=20 Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 6 link: 0 is dow= n=20 Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 10 link: 0 is do= wn=20 Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 (passive) bes= t link: 0 (pri: 1)=20 Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 has no active= links=20 Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 has no active = links=20 Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 has no active = links=20 Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 (passive) bes= t link: 0 (pri: 1)=20 Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 has no active= links=20 Sep 3 10:40:28 m6kvm7 corosync[15467]: [TOTEM ] Token has not been received= in 4505 ms=20 Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 11 link: 0 is do= wn=20 Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 9 link: 0 is dow= n=20 Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 (passive) bes= t link: 0 (pri: 1)=20 Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 has no active= links=20 Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 has no active = links=20 Sep 3 10:40:30 m6kvm7 corosync[15467]: [TOTEM ] A processor failed, forming= new configuration.=20 Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 4 link: 0 is dow= n=20 Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 12 link: 0 is do= wn=20 Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 1 link: 0 is dow= n=20 Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 has no active = links=20 Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 (passive) bes= t link: 0 (pri: 1)=20 Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 has no active= links=20 Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 has no active = links=20 Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] link: host: 5 link: 0 is dow= n=20 Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 has no active = links=20 Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] A new membership (7.82f) wa= s formed. Members left: 1 3 4 5 6 8 9 10 11 12 13 14=20 Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] Failed to receive the leave= message. failed: 1 3 4 5 6 8 9 10 11 12 13 14=20 Sep 3 10:40:41 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 12 receiv= ed=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892=20 Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] This node is within the non= -primary component and will NOT provide any services.=20 Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] Members[1]: 7=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date= =20 Sep 3 10:40:41 m6kvm7 corosync[15467]: [MAIN ] Completed service synchroniz= ation, ready to provide service.=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: dfsm_deliver_queue: que= ue length 51=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 1/3527=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 1/3527=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 1/3527=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 1/3527=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 4/3903=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 4/3903=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 4/3903=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 4/3903=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 9/22810=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 9/22810=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 9/22810=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 9/22810=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 6/37248=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 6/37248=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 6/37248=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 6/37248=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 11/12983=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 11/12983=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 11/12983=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 11/12983=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 10/41940=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 10/41940=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 10/41940=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 10/41940=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 5/42272=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 5/42272=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 5/42272=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 13/39678=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 13/39678=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 13/39678=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 13/39678=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 8/24790=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 8/24790=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 8/24790=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 8/24790=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 3/27373=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 3/27373=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 3/27373=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 3/27373=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 3/27373=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 12/44214=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 12/44214=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 12/44214=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 12/44214=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 14/42930=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 14/42930=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 14/42930=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non= -member 14/42930=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: node lost quorum=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: members: 7/15892=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: received write while not = quorate - trigger resync=20 Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: leaving CPG group=20 Sep 3 10:40:41 m6kvm7 pve-ha-crm[16196]: node 'm6kvm2': state changed from = 'online' =3D> 'unknown'=20 Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] notice: start cluster connectio= n=20 Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: cpg_join failed: 14=20 Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: can't initialize service= =20 Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892=20 Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date= =20 Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error durin= g cfs-locked 'domain-ha' operation: no quorum!=20 Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)=20 Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)=20 Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs l= ock update failed - Permission denied=20 Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - = cfs lock update failed - Permission denied=20 Sep 3 10:40:56 m6kvm7 pve-ha-lrm[16140]: status change active =3D> lost_age= nt_lock=20 Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change master =3D> lost_man= ager_lock=20 Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: watchdog closed (disabled)=20 Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change lost_manager_lock = =3D> wait_for_quorum=20 Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] rx: host: 4 link: 0 is up=20 Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:43:24 m6kvm7 corosync[15467]: [TOTEM ] A new membership (4.834) wa= s formed. Members joined: 4=20 Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 receive= d=20 Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 receive= d=20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 4/3965, 7/1589= 2=20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronis= ation=20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: members: 4/3965, 7/15= 892=20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncron= isation=20 Sep 3 10:43:24 m6kvm7 corosync[15467]: [QUORUM] Members[2]: 4 7=20 Sep 3 10:43:24 m6kvm7 corosync[15467]: [MAIN ] Completed service synchroniz= ation, ready to provide service.=20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (= epoch 4/3965/00000002)=20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received sync request= (epoch 4/3965/00000002)=20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states=20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 7/15892=20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 7/15892= =20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: start sending inode upd= ates=20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: sent all (4) updates=20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date= =20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received all states= =20 Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to dat= e=20 Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] rx: host: 3 link: 0 is up=20 Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:44:05 m6kvm7 corosync[15467]: [TOTEM ] A new membership (3.838) wa= s formed. Members joined: 3=20 Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 receive= d=20 Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 receive= d=20 Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 receive= d=20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 3/3613, 4/3965= , 7/15892=20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronis= ation=20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: members: 3/3613, 4/39= 65, 7/15892=20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncron= isation=20 Sep 3 10:44:05 m6kvm7 corosync[15467]: [QUORUM] Members[3]: 3 4 7=20 Sep 3 10:44:05 m6kvm7 corosync[15467]: [MAIN ] Completed service synchroniz= ation, ready to provide service.=20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (= epoch 3/3613/00000002)=20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received sync request= (epoch 3/3613/00000002)=20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states=20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 4/3965=20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 4/3965,= 7/15892=20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date= =20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received all states= =20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to dat= e=20 Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: dfsm_deliver_queue: q= ueue length 1=20 Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] rx: host: 1 link: 0 is up=20 Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:44:31 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.83c) wa= s formed. Members joined: 1=20 Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 receive= d=20 Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 receive= d=20 Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 receive= d=20 Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 receive= d=20 Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3552, 3/3613= , 4/3965, 7/15892=20 Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronis= ation=20 Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3552, 3/36= 13, 4/3965, 7/15892=20 Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncron= isation=20 Sep 3 10:44:31 m6kvm7 corosync[15467]: [QUORUM] Members[4]: 1 3 4 7=20 Sep 3 10:44:31 m6kvm7 corosync[15467]: [MAIN ] Completed service synchroniz= ation, ready to provide service.=20 Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (= epoch 1/3552/00000002)=20 Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received sync request= (epoch 1/3552/00000002)=20 Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states=20 Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 3/3613=20 Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 3/3613,= 4/3965, 7/15892=20 Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date= =20 Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received all states= =20 Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to dat= e=20 Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 Sep 3 10:49:18 m6kvm7 pmxcfs[15892]: [status] notice: received log=20 node8=20 -----=20 Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: members: 1/3527, 3/2737= 3, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983,= 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: starting data syncronis= ation=20 Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: members: 1/3527, 3/27= 373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/1298= 3, 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: starting data syncron= isation=20 Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: received sync request (= epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: received sync request= (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [TOTEM ] A new membership (1.82b) wa= s formed. Members left: 2=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 = 9 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm8 corosync[24361]: [MAIN ] Completed service synchroniz= ation, ready to provide service.=20 Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: received all states= =20 Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: all data is up to dat= e=20 Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] link: host: 2 link: 0 is dow= n=20 Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 has no active = links=20 Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log=20 --> reboot=20 node9=20 -----=20 Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: members: 1/3527, 3/2737= 3, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983,= 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: starting data syncronis= ation=20 Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: members: 1/3527, 3/27= 373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/1298= 3, 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: starting data syncron= isation=20 Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: received sync request (= epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: received sync request= (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [TOTEM ] A new membership (1.82b) wa= s formed. Members left: 2=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 receive= d=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 = 9 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm9 corosync[22340]: [MAIN ] Completed service synchroniz= ation, ready to provide service.=20 Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: received all states= =20 Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: all data is up to dat= e=20 Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] link: host: 2 link: 0 is dow= n=20 Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 (passive) best= link: 0 (pri: 1)=20 Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 has no active = links=20 Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:20 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log=20 --> reboot=20 node10=20 ------=20 Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: members: 1/3527, 3/2= 7373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/129= 83, 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: starting data syncro= nisation=20 Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: members: 1/3527, 3/273= 73, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983= , 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: starting data syncroni= sation=20 Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: received sync request = (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: received sync reques= t (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [TOTEM ] A new membership (1.82b) w= as formed. Members left: 2=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [QUORUM] Members[13]: 1 3 4 5 6 7 8= 9 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm10 corosync[41458]: [MAIN ] Completed service synchroni= zation, ready to provide service.=20 Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: received all states= =20 Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: all data is up to da= te=20 Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] link: host: 2 link: 0 is do= wn=20 Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 (passive) bes= t link: 0 (pri: 1)=20 Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 has no active= links=20 Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log=20 --> reboot=20 node11=20 ------=20 Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: members: 1/3527, 3/273= 73, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983= , 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: starting data syncroni= sation=20 Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: members: 1/3527, 3/2= 7373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/129= 83, 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: starting data syncro= nisation=20 Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: received sync request = (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: received sync reques= t (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [TOTEM ] A new membership (1.82b) w= as formed. Members left: 2=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [QUORUM] Members[13]: 1 3 4 5 6 7 8= 9 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm11 corosync[12455]: [MAIN ] Completed service synchroni= zation, ready to provide service.=20 Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: received all states= =20 Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: all data is up to da= te=20 Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] link: host: 2 link: 0 is do= wn=20 Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 (passive) bes= t link: 0 (pri: 1)=20 Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 has no active= links=20 Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log=20 Sep 3 10:40:20 m6kvm11 pmxcfs[12983]: [status] notice: received log=20 node12=20 ------=20 Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: members: 1/3527, 3/2= 7373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/129= 83, 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: starting data syncro= nisation=20 Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: members: 1/3527, 3/273= 73, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983= , 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: starting data syncroni= sation=20 Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: received sync reques= t (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: received sync request = (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [TOTEM ] A new membership (1.82b) w= as formed. Members left: 2=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [QUORUM] Members[13]: 1 3 4 5 6 7 8= 9 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm12 corosync[43716]: [MAIN ] Completed service synchroni= zation, ready to provide service.=20 Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [dcdb] notice: cpg_send_message retri= ed 1 times=20 Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: received all states= =20 Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: all data is up to da= te=20 Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] link: host: 2 link: 0 is do= wn=20 Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 (passive) bes= t link: 0 (pri: 1)=20 Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 has no active= links=20 Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:20 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log=20 node13=20 ------=20 Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: members: 1/3527, 3/2= 7373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/129= 83, 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: starting data syncro= nisation=20 Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: members: 1/3527, 3/273= 73, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983= , 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: starting data syncroni= sation=20 Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: received sync reques= t (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: received sync request = (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [TOTEM ] A new membership (1.82b) w= as formed. Members left: 2=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [QUORUM] Members[13]: 1 3 4 5 6 7 8= 9 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm13 corosync[39182]: [MAIN ] Completed service synchroni= zation, ready to provide service.=20 Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: cpg_send_message retri= ed 1 times=20 Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: received all states= =20 Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: all data is up to da= te=20 Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] link: host: 2 link: 0 is do= wn=20 Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 (passive) bes= t link: 0 (pri: 1)=20 Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 has no active= links=20 Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log=20 Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log=20 --> reboot=20 node14=20 ------=20 Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: members: 1/3527, 3/273= 73, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983= , 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: starting data syncroni= sation=20 Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: members: 1/3527, 3/2= 7373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/129= 83, 12/44214, 13/39678, 14/42930=20 Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: starting data syncro= nisation=20 Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: received sync request = (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: received sync reques= t (epoch 1/3527/00000002)=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [TOTEM ] A new membership (1.82b) w= as formed. Members left: 2=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 receiv= ed=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [QUORUM] Members[13]: 1 3 4 5 6 7 8= 9 10 11 12 13 14=20 Sep 3 10:39:16 m6kvm14 corosync[42413]: [MAIN ] Completed service synchroni= zation, ready to provide service.=20 Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: received all states= =20 Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: all data is up to da= te=20 Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] link: host: 2 link: 0 is do= wn=20 Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 (passive) bes= t link: 0 (pri: 1)=20 Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 has no active= links=20 --> reboot=20 _______________________________________________=20 pve-devel mailing list=20 pve-devel@lists.proxmox.com=20 https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel=20