From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 4373A61E57 for ; Tue, 15 Sep 2020 12:15:51 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3794319452 for ; Tue, 15 Sep 2020 12:15:51 +0200 (CEST) Received: from mailpro.odiso.net (mailpro.odiso.net [89.248.211.110]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id E303019445 for ; Tue, 15 Sep 2020 12:15:47 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id A4F891A5D37B; Tue, 15 Sep 2020 12:15:47 +0200 (CEST) Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id smoPoy8SeRaG; Tue, 15 Sep 2020 12:15:47 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id 81DBE1A5D377; Tue, 15 Sep 2020 12:15:47 +0200 (CEST) X-Virus-Scanned: amavisd-new at mailpro.odiso.com Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id VN4wBaku_I91; Tue, 15 Sep 2020 12:15:47 +0200 (CEST) Received: from mailpro.odiso.net (mailpro.odiso.net [10.1.31.111]) by mailpro.odiso.net (Postfix) with ESMTP id 6A3411A5D371; Tue, 15 Sep 2020 12:15:47 +0200 (CEST) Date: Tue, 15 Sep 2020 12:15:47 +0200 (CEST) From: Alexandre DERUMIER To: Thomas Lamprecht Cc: Proxmox VE development discussion , dietmar Message-ID: <1282130277.831843.1600164947209.JavaMail.zimbra@odiso.com> In-Reply-To: <98e79e8d-9001-db77-c032-bdfcdb3698a6@proxmox.com> References: <216436814.339545.1599142316781.JavaMail.zimbra@odiso.com> <1775665592.735772.1600098305930.JavaMail.zimbra@odiso.com> <487514223.9.1600148741895@webmail.proxmox.com> <295606419.745430.1600151269212.JavaMail.zimbra@odiso.com> <1227203309.12.1600154034412@webmail.proxmox.com> <1746620611.752896.1600159335616.JavaMail.zimbra@odiso.com> <1464606394.823230.1600162557186.JavaMail.zimbra@odiso.com> <98e79e8d-9001-db77-c032-bdfcdb3698a6@proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Mailer: Zimbra 8.8.12_GA_3866 (ZimbraWebClient - GC83 (Linux)/8.8.12_GA_3844) Thread-Topic: corosync bug: cluster break after 1 node clean shutdown Thread-Index: rEZm9g7m7mPg533DJU/Q77HVZFIsRQ== X-SPAM-LEVEL: Spam detection results: 0 AWL 0.090 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Sep 2020 10:15:51 -0000 here the previous restart log node1 -> corosync restart at 10:46:15 ----- https://gist.github.com/aderumier/0992051d20f51270ceceb5b3431d18d7 node2 ----- https://gist.github.com/aderumier/eea0c50fefc1d8561868576f417191ba node5 ------ https://gist.github.com/aderumier/f2ce1bc5a93827045a5691583bbc7a37 ----- Mail original ----- De: "Thomas Lamprecht" =C3=80: "aderumier" , "Proxmox VE development discussi= on" Cc: "dietmar" Envoy=C3=A9: Mardi 15 Septembre 2020 11:46:51 Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutd= own On 9/15/20 11:35 AM, Alexandre DERUMIER wrote:=20 > Hi,=20 >=20 > I have finally reproduce it !=20 >=20 > But this is with a corosync restart in cron each 1 minute, on node1=20 >=20 > Then: lrm was stuck for too long for around 60s and softdog have been tri= ggered on multiple other nodes.=20 >=20 > here the logs with full corosync debug at the time of last corosync resta= rt.=20 >=20 > node1 (where corosync is restarted each minute)=20 > https://gist.github.com/aderumier/c4f192fbce8e96759f91a61906db514e=20 >=20 > node2=20 > https://gist.github.com/aderumier/2d35ea05c1fbff163652e564fc430e67=20 >=20 > node5=20 > https://gist.github.com/aderumier/df1d91cddbb6e15bb0d0193ed8df9273=20 >=20 > I'll prepare logs from the previous corosync restart, as the lrm seem to = be already stuck before.=20 Yeah that would be good, as yes the lrm seems to get stuck at around 10:46:= 21=20 > Sep 15 10:47:26 m6kvm2 pve-ha-lrm[3736]: loop take too long (65 seconds)= =20