From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <aderumier@odiso.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id E6E8861BEA for <pve-devel@lists.proxmox.com>; Tue, 15 Sep 2020 11:36:28 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id CB847184A2 for <pve-devel@lists.proxmox.com>; Tue, 15 Sep 2020 11:35:58 +0200 (CEST) Received: from mailpro.odiso.net (mailpro.odiso.net [89.248.211.110]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id E78B718498 for <pve-devel@lists.proxmox.com>; Tue, 15 Sep 2020 11:35:57 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id BB3401A5CBB4; Tue, 15 Sep 2020 11:35:57 +0200 (CEST) Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 8FePgWrrNolm; Tue, 15 Sep 2020 11:35:57 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id 9C3B91A5CBB5; Tue, 15 Sep 2020 11:35:57 +0200 (CEST) X-Virus-Scanned: amavisd-new at mailpro.odiso.com Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 0HBo8UPHorgh; Tue, 15 Sep 2020 11:35:57 +0200 (CEST) Received: from mailpro.odiso.net (mailpro.odiso.net [10.1.31.111]) by mailpro.odiso.net (Postfix) with ESMTP id 831521A5CBB4; Tue, 15 Sep 2020 11:35:57 +0200 (CEST) Date: Tue, 15 Sep 2020 11:35:57 +0200 (CEST) From: Alexandre DERUMIER <aderumier@odiso.com> To: Proxmox VE development discussion <pve-devel@lists.proxmox.com> Cc: dietmar <dietmar@proxmox.com>, Thomas Lamprecht <t.lamprecht@proxmox.com> Message-ID: <1464606394.823230.1600162557186.JavaMail.zimbra@odiso.com> In-Reply-To: <1746620611.752896.1600159335616.JavaMail.zimbra@odiso.com> References: <216436814.339545.1599142316781.JavaMail.zimbra@odiso.com> <2093781647.723563.1600072074707.JavaMail.zimbra@odiso.com> <88fe5075-870d-9197-7c84-71ae8a25e9dd@proxmox.com> <1775665592.735772.1600098305930.JavaMail.zimbra@odiso.com> <487514223.9.1600148741895@webmail.proxmox.com> <295606419.745430.1600151269212.JavaMail.zimbra@odiso.com> <1227203309.12.1600154034412@webmail.proxmox.com> <1746620611.752896.1600159335616.JavaMail.zimbra@odiso.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Mailer: Zimbra 8.8.12_GA_3866 (ZimbraWebClient - GC83 (Linux)/8.8.12_GA_3844) Thread-Topic: corosync bug: cluster break after 1 node clean shutdown Thread-Index: lmWFdNPil+vFbWG2X6xcUMRyNAyc79zHyyMe X-SPAM-LEVEL: Spam detection results: 0 AWL 0.092 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> List-Post: <mailto:pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Tue, 15 Sep 2020 09:36:29 -0000 Hi, I have finally reproduce it ! But this is with a corosync restart in cron each 1 minute, on node1 Then: lrm was stuck for too long for around 60s and softdog have been trigg= ered on multiple other nodes. here the logs with full corosync debug at the time of last corosync restart= .=20 node1 (where corosync is restarted each minute) https://gist.github.com/aderumier/c4f192fbce8e96759f91a61906db514e node2 https://gist.github.com/aderumier/2d35ea05c1fbff163652e564fc430e67 node5 https://gist.github.com/aderumier/df1d91cddbb6e15bb0d0193ed8df9273 I'll prepare logs from the previous corosync restart, as the lrm seem to be= already stuck before. ----- Mail original ----- De: "aderumier" <aderumier@odiso.com> =C3=80: "dietmar" <dietmar@proxmox.com> Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, "Tho= mas Lamprecht" <t.lamprecht@proxmox.com> Envoy=C3=A9: Mardi 15 Septembre 2020 10:42:15 Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutd= own >>pmxcfs cannot send anything in that case, so it is impossible that this h= as effects on other nodes.=20 yes, I understand that, but I was thinking of the case if corosync is in st= opping phase (not totally stopped).=20 Something racy (I really don't known).=20 I just send 2 patch to start pve-cluster && corosync after syslog, like thi= s we'll have shutdown logs too.=20 (I'm currently try to reproduce the problem, with reboot loops, but I still= can't reproduce it :/ )=20 ----- Mail original -----=20 De: "dietmar" <dietmar@proxmox.com>=20 =C3=80: "aderumier" <aderumier@odiso.com>=20 Cc: "Thomas Lamprecht" <t.lamprecht@proxmox.com>, "Proxmox VE development d= iscussion" <pve-devel@lists.proxmox.com>=20 Envoy=C3=A9: Mardi 15 Septembre 2020 09:13:53=20 Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutd= own=20 > I ask the question, because the 2 times I have problem, it was when shutt= ing down a server.=20 > So maybe some strange behaviour occur with both corosync && pmxcfs are st= opped at same time ?=20 pmxcfs cannot send anything in that case, so it is impossible that this has= effects on other nodes.=20 _______________________________________________=20 pve-devel mailing list=20 pve-devel@lists.proxmox.com=20 https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel=20