From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 134BF61FE6 for ; Tue, 15 Sep 2020 14:49:59 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 06EC01AAB4 for ; Tue, 15 Sep 2020 14:49:59 +0200 (CEST) Received: from mailpro.odiso.net (mailpro.odiso.net [89.248.211.110]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 8AD5B1AAA8 for ; Tue, 15 Sep 2020 14:49:55 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id 2F2001A5F10D; Tue, 15 Sep 2020 14:49:55 +0200 (CEST) Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id movbeiVPoxUb; Tue, 15 Sep 2020 14:49:55 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id 14A031A5F10E; Tue, 15 Sep 2020 14:49:55 +0200 (CEST) X-Virus-Scanned: amavisd-new at mailpro.odiso.com Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id hqzsprJ2DhRD; Tue, 15 Sep 2020 14:49:55 +0200 (CEST) Received: from mailpro.odiso.net (mailpro.odiso.net [10.1.31.111]) by mailpro.odiso.net (Postfix) with ESMTP id EF9FC1A5F10D; Tue, 15 Sep 2020 14:49:54 +0200 (CEST) Date: Tue, 15 Sep 2020 14:49:54 +0200 (CEST) From: Alexandre DERUMIER To: Proxmox VE development discussion Cc: Thomas Lamprecht Message-ID: <1800811328.836757.1600174194769.JavaMail.zimbra@odiso.com> In-Reply-To: <1732268946.834480.1600167871823.JavaMail.zimbra@odiso.com> References: <216436814.339545.1599142316781.JavaMail.zimbra@odiso.com> <295606419.745430.1600151269212.JavaMail.zimbra@odiso.com> <1227203309.12.1600154034412@webmail.proxmox.com> <1746620611.752896.1600159335616.JavaMail.zimbra@odiso.com> <1464606394.823230.1600162557186.JavaMail.zimbra@odiso.com> <98e79e8d-9001-db77-c032-bdfcdb3698a6@proxmox.com> <1282130277.831843.1600164947209.JavaMail.zimbra@odiso.com> <1732268946.834480.1600167871823.JavaMail.zimbra@odiso.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Mailer: Zimbra 8.8.12_GA_3866 (ZimbraWebClient - GC83 (Linux)/8.8.12_GA_3844) Thread-Topic: corosync bug: cluster break after 1 node clean shutdown Thread-Index: rEZm9g7m7mPg533DJU/Q77HVZFIsRWEAvWvFe8kxWQU= X-SPAM-LEVEL: Spam detection results: 0 AWL -0.313 Adjusted score from AWL reputation of From: address KAM_ASCII_DIVIDERS 0.8 Spam that uses ascii formatting tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Sep 2020 12:49:59 -0000 Hi, I have produce it again,=20 now I can't write to /etc/pve/ from any node I have also added some debug logs to pve-ha-lrm, and it was stuck in: (but if /etc/pve is locked, this is normal) if ($fence_request) { $haenv->log('err', "node need to be fenced - releasing agent_lo= ck\n"); $self->set_local_status({ state =3D> 'lost_agent_lock'}); } elsif (!$self->get_protected_ha_agent_lock()) { $self->set_local_status({ state =3D> 'lost_agent_lock'}); } elsif ($self->{mode} eq 'maintenance') { $self->set_local_status({ state =3D> 'maintenance'}); } corosync quorum is currently ok I'm currently digging the logs ----- Mail original ----- De: "aderumier" =C3=80: "Proxmox VE development discussion" Cc: "Thomas Lamprecht" Envoy=C3=A9: Mardi 15 Septembre 2020 13:04:31 Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutd= own also logs of node14, where the lrm was not too long=20 https://gist.github.com/aderumier/a2e2d6afc7e04646c923ae6f37cb6c2d=20 ----- Mail original -----=20 De: "aderumier" =20 =C3=80: "Thomas Lamprecht" =20 Cc: "Proxmox VE development discussion" =20 Envoy=C3=A9: Mardi 15 Septembre 2020 12:15:47=20 Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutd= own=20 here the previous restart log=20 node1 -> corosync restart at 10:46:15=20 -----=20 https://gist.github.com/aderumier/0992051d20f51270ceceb5b3431d18d7=20 node2=20 -----=20 https://gist.github.com/aderumier/eea0c50fefc1d8561868576f417191ba=20 node5=20 ------=20 https://gist.github.com/aderumier/f2ce1bc5a93827045a5691583bbc7a37=20 ----- Mail original -----=20 De: "Thomas Lamprecht" =20 =C3=80: "aderumier" , "Proxmox VE development discussi= on" =20 Cc: "dietmar" =20 Envoy=C3=A9: Mardi 15 Septembre 2020 11:46:51=20 Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutd= own=20 On 9/15/20 11:35 AM, Alexandre DERUMIER wrote:=20 > Hi,=20 >=20 > I have finally reproduce it !=20 >=20 > But this is with a corosync restart in cron each 1 minute, on node1=20 >=20 > Then: lrm was stuck for too long for around 60s and softdog have been tri= ggered on multiple other nodes.=20 >=20 > here the logs with full corosync debug at the time of last corosync resta= rt.=20 >=20 > node1 (where corosync is restarted each minute)=20 > https://gist.github.com/aderumier/c4f192fbce8e96759f91a61906db514e=20 >=20 > node2=20 > https://gist.github.com/aderumier/2d35ea05c1fbff163652e564fc430e67=20 >=20 > node5=20 > https://gist.github.com/aderumier/df1d91cddbb6e15bb0d0193ed8df9273=20 >=20 > I'll prepare logs from the previous corosync restart, as the lrm seem to = be already stuck before.=20 Yeah that would be good, as yes the lrm seems to get stuck at around 10:46:= 21=20 > Sep 15 10:47:26 m6kvm2 pve-ha-lrm[3736]: loop take too long (65 seconds)= =20 _______________________________________________=20 pve-devel mailing list=20 pve-devel@lists.proxmox.com=20 https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel=20 _______________________________________________=20 pve-devel mailing list=20 pve-devel@lists.proxmox.com=20 https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel=20