From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 4B3E061EF0 for ; Tue, 15 Sep 2020 13:04:34 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3A66619D29 for ; Tue, 15 Sep 2020 13:04:34 +0200 (CEST) Received: from mailpro.odiso.net (mailpro.odiso.net [89.248.211.110]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 71F3C19D1C for ; Tue, 15 Sep 2020 13:04:32 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id 3E9191A5D64E; Tue, 15 Sep 2020 13:04:32 +0200 (CEST) Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id gRSTfYnz9cgD; Tue, 15 Sep 2020 13:04:32 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id 225931A5D64F; Tue, 15 Sep 2020 13:04:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at mailpro.odiso.com Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id oMmXqLngwC38; Tue, 15 Sep 2020 13:04:32 +0200 (CEST) Received: from mailpro.odiso.net (mailpro.odiso.net [10.1.31.111]) by mailpro.odiso.net (Postfix) with ESMTP id 0BF0A1A5D64E; Tue, 15 Sep 2020 13:04:32 +0200 (CEST) Date: Tue, 15 Sep 2020 13:04:31 +0200 (CEST) From: Alexandre DERUMIER To: Proxmox VE development discussion Cc: Thomas Lamprecht Message-ID: <1732268946.834480.1600167871823.JavaMail.zimbra@odiso.com> In-Reply-To: <1282130277.831843.1600164947209.JavaMail.zimbra@odiso.com> References: <216436814.339545.1599142316781.JavaMail.zimbra@odiso.com> <487514223.9.1600148741895@webmail.proxmox.com> <295606419.745430.1600151269212.JavaMail.zimbra@odiso.com> <1227203309.12.1600154034412@webmail.proxmox.com> <1746620611.752896.1600159335616.JavaMail.zimbra@odiso.com> <1464606394.823230.1600162557186.JavaMail.zimbra@odiso.com> <98e79e8d-9001-db77-c032-bdfcdb3698a6@proxmox.com> <1282130277.831843.1600164947209.JavaMail.zimbra@odiso.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Mailer: Zimbra 8.8.12_GA_3866 (ZimbraWebClient - GC83 (Linux)/8.8.12_GA_3844) Thread-Topic: corosync bug: cluster break after 1 node clean shutdown Thread-Index: rEZm9g7m7mPg533DJU/Q77HVZFIsRWEAvWvF X-SPAM-LEVEL: Spam detection results: 0 AWL 0.088 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Sep 2020 11:04:34 -0000 also logs of node14, where the lrm was not too long https://gist.github.com/aderumier/a2e2d6afc7e04646c923ae6f37cb6c2d ----- Mail original ----- De: "aderumier" =C3=80: "Thomas Lamprecht" Cc: "Proxmox VE development discussion" Envoy=C3=A9: Mardi 15 Septembre 2020 12:15:47 Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutd= own here the previous restart log=20 node1 -> corosync restart at 10:46:15=20 -----=20 https://gist.github.com/aderumier/0992051d20f51270ceceb5b3431d18d7=20 node2=20 -----=20 https://gist.github.com/aderumier/eea0c50fefc1d8561868576f417191ba=20 node5=20 ------=20 https://gist.github.com/aderumier/f2ce1bc5a93827045a5691583bbc7a37=20 ----- Mail original -----=20 De: "Thomas Lamprecht" =20 =C3=80: "aderumier" , "Proxmox VE development discussi= on" =20 Cc: "dietmar" =20 Envoy=C3=A9: Mardi 15 Septembre 2020 11:46:51=20 Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutd= own=20 On 9/15/20 11:35 AM, Alexandre DERUMIER wrote:=20 > Hi,=20 >=20 > I have finally reproduce it !=20 >=20 > But this is with a corosync restart in cron each 1 minute, on node1=20 >=20 > Then: lrm was stuck for too long for around 60s and softdog have been tri= ggered on multiple other nodes.=20 >=20 > here the logs with full corosync debug at the time of last corosync resta= rt.=20 >=20 > node1 (where corosync is restarted each minute)=20 > https://gist.github.com/aderumier/c4f192fbce8e96759f91a61906db514e=20 >=20 > node2=20 > https://gist.github.com/aderumier/2d35ea05c1fbff163652e564fc430e67=20 >=20 > node5=20 > https://gist.github.com/aderumier/df1d91cddbb6e15bb0d0193ed8df9273=20 >=20 > I'll prepare logs from the previous corosync restart, as the lrm seem to = be already stuck before.=20 Yeah that would be good, as yes the lrm seems to get stuck at around 10:46:= 21=20 > Sep 15 10:47:26 m6kvm2 pve-ha-lrm[3736]: loop take too long (65 seconds)= =20 _______________________________________________=20 pve-devel mailing list=20 pve-devel@lists.proxmox.com=20 https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel=20