From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <aderumier@odiso.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 134BF61FE6
for <pve-devel@lists.proxmox.com>; Tue, 15 Sep 2020 14:49:59 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 06EC01AAB4
for <pve-devel@lists.proxmox.com>; Tue, 15 Sep 2020 14:49:59 +0200 (CEST)
Received: from mailpro.odiso.net (mailpro.odiso.net [89.248.211.110])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS id 8AD5B1AAA8
for <pve-devel@lists.proxmox.com>; Tue, 15 Sep 2020 14:49:55 +0200 (CEST)
Received: from localhost (localhost [127.0.0.1])
by mailpro.odiso.net (Postfix) with ESMTP id 2F2001A5F10D;
Tue, 15 Sep 2020 14:49:55 +0200 (CEST)
Received: from mailpro.odiso.net ([127.0.0.1])
by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10032)
with ESMTP id movbeiVPoxUb; Tue, 15 Sep 2020 14:49:55 +0200 (CEST)
Received: from localhost (localhost [127.0.0.1])
by mailpro.odiso.net (Postfix) with ESMTP id 14A031A5F10E;
Tue, 15 Sep 2020 14:49:55 +0200 (CEST)
X-Virus-Scanned: amavisd-new at mailpro.odiso.com
Received: from mailpro.odiso.net ([127.0.0.1])
by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10026)
with ESMTP id hqzsprJ2DhRD; Tue, 15 Sep 2020 14:49:55 +0200 (CEST)
Received: from mailpro.odiso.net (mailpro.odiso.net [10.1.31.111])
by mailpro.odiso.net (Postfix) with ESMTP id EF9FC1A5F10D;
Tue, 15 Sep 2020 14:49:54 +0200 (CEST)
Date: Tue, 15 Sep 2020 14:49:54 +0200 (CEST)
From: Alexandre DERUMIER <aderumier@odiso.com>
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
Cc: Thomas Lamprecht <t.lamprecht@proxmox.com>
Message-ID: <1800811328.836757.1600174194769.JavaMail.zimbra@odiso.com>
In-Reply-To: <1732268946.834480.1600167871823.JavaMail.zimbra@odiso.com>
References: <216436814.339545.1599142316781.JavaMail.zimbra@odiso.com>
<295606419.745430.1600151269212.JavaMail.zimbra@odiso.com>
<1227203309.12.1600154034412@webmail.proxmox.com>
<1746620611.752896.1600159335616.JavaMail.zimbra@odiso.com>
<1464606394.823230.1600162557186.JavaMail.zimbra@odiso.com>
<98e79e8d-9001-db77-c032-bdfcdb3698a6@proxmox.com>
<1282130277.831843.1600164947209.JavaMail.zimbra@odiso.com>
<1732268946.834480.1600167871823.JavaMail.zimbra@odiso.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Mailer: Zimbra 8.8.12_GA_3866 (ZimbraWebClient - GC83 (Linux)/8.8.12_GA_3844)
Thread-Topic: corosync bug: cluster break after 1 node clean shutdown
Thread-Index: rEZm9g7m7mPg533DJU/Q77HVZFIsRWEAvWvFe8kxWQU=
X-SPAM-LEVEL: Spam detection results: 0
AWL -0.313 Adjusted score from AWL reputation of From: address
KAM_ASCII_DIVIDERS 0.8 Spam that uses ascii formatting tricks
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/,
no trust
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [proxmox.com]
Subject: Re: [pve-devel] corosync bug: cluster break after 1 node clean
shutdown
X-BeenThere: pve-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
List-Post: <mailto:pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Tue, 15 Sep 2020 12:49:59 -0000
Hi,
I have produce it again,=20
now I can't write to /etc/pve/ from any node
I have also added some debug logs to pve-ha-lrm, and it was stuck in:
(but if /etc/pve is locked, this is normal)
if ($fence_request) {
$haenv->log('err', "node need to be fenced - releasing agent_lo=
ck\n");
$self->set_local_status({ state =3D> 'lost_agent_lock'});
} elsif (!$self->get_protected_ha_agent_lock()) {
$self->set_local_status({ state =3D> 'lost_agent_lock'});
} elsif ($self->{mode} eq 'maintenance') {
$self->set_local_status({ state =3D> 'maintenance'});
}
corosync quorum is currently ok
I'm currently digging the logs
----- Mail original -----
De: "aderumier" <aderumier@odiso.com>
=C3=80: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "Thomas Lamprecht" <t.lamprecht@proxmox.com>
Envoy=C3=A9: Mardi 15 Septembre 2020 13:04:31
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutd=
own
also logs of node14, where the lrm was not too long=20
https://gist.github.com/aderumier/a2e2d6afc7e04646c923ae6f37cb6c2d=20
----- Mail original -----=20
De: "aderumier" <aderumier@odiso.com>=20
=C3=80: "Thomas Lamprecht" <t.lamprecht@proxmox.com>=20
Cc: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>=20
Envoy=C3=A9: Mardi 15 Septembre 2020 12:15:47=20
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutd=
own=20
here the previous restart log=20
node1 -> corosync restart at 10:46:15=20
-----=20
https://gist.github.com/aderumier/0992051d20f51270ceceb5b3431d18d7=20
node2=20
-----=20
https://gist.github.com/aderumier/eea0c50fefc1d8561868576f417191ba=20
node5=20
------=20
https://gist.github.com/aderumier/f2ce1bc5a93827045a5691583bbc7a37=20
----- Mail original -----=20
De: "Thomas Lamprecht" <t.lamprecht@proxmox.com>=20
=C3=80: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussi=
on" <pve-devel@lists.proxmox.com>=20
Cc: "dietmar" <dietmar@proxmox.com>=20
Envoy=C3=A9: Mardi 15 Septembre 2020 11:46:51=20
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutd=
own=20
On 9/15/20 11:35 AM, Alexandre DERUMIER wrote:=20
> Hi,=20
>=20
> I have finally reproduce it !=20
>=20
> But this is with a corosync restart in cron each 1 minute, on node1=20
>=20
> Then: lrm was stuck for too long for around 60s and softdog have been tri=
ggered on multiple other nodes.=20
>=20
> here the logs with full corosync debug at the time of last corosync resta=
rt.=20
>=20
> node1 (where corosync is restarted each minute)=20
> https://gist.github.com/aderumier/c4f192fbce8e96759f91a61906db514e=20
>=20
> node2=20
> https://gist.github.com/aderumier/2d35ea05c1fbff163652e564fc430e67=20
>=20
> node5=20
> https://gist.github.com/aderumier/df1d91cddbb6e15bb0d0193ed8df9273=20
>=20
> I'll prepare logs from the previous corosync restart, as the lrm seem to =
be already stuck before.=20
Yeah that would be good, as yes the lrm seems to get stuck at around 10:46:=
21=20
> Sep 15 10:47:26 m6kvm2 pve-ha-lrm[3736]: loop take too long (65 seconds)=
=20
_______________________________________________=20
pve-devel mailing list=20
pve-devel@lists.proxmox.com=20
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel=20
_______________________________________________=20
pve-devel mailing list=20
pve-devel@lists.proxmox.com=20
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel=20