From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <aderumier@odiso.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id BD306628ED for <pve-devel@lists.proxmox.com>; Thu, 17 Sep 2020 12:00:10 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B150EBD4D for <pve-devel@lists.proxmox.com>; Thu, 17 Sep 2020 11:59:40 +0200 (CEST) Received: from mailpro.odiso.net (mailpro.odiso.net [89.248.211.110]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id E276FBD41 for <pve-devel@lists.proxmox.com>; Thu, 17 Sep 2020 11:59:38 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id A3FB31A9F8D2; Thu, 17 Sep 2020 11:59:32 +0200 (CEST) Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id xIRMsJf5Hgth; Thu, 17 Sep 2020 11:59:32 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id 898321A9F8D3; Thu, 17 Sep 2020 11:59:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at mailpro.odiso.com Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id lTcB7CVPQq3d; Thu, 17 Sep 2020 11:59:32 +0200 (CEST) Received: from mailpro.odiso.net (mailpro.odiso.net [10.1.31.111]) by mailpro.odiso.net (Postfix) with ESMTP id 744331A9F8D2; Thu, 17 Sep 2020 11:59:32 +0200 (CEST) Date: Thu, 17 Sep 2020 11:59:32 +0200 (CEST) From: Alexandre DERUMIER <aderumier@odiso.com> To: Proxmox VE development discussion <pve-devel@lists.proxmox.com> Cc: Thomas Lamprecht <t.lamprecht@proxmox.com> Message-ID: <475756962.894651.1600336772315.JavaMail.zimbra@odiso.com> In-Reply-To: <1600333910.bmtyynl8cl.astroid@nora.none> References: <216436814.339545.1599142316781.JavaMail.zimbra@odiso.com> <602718914.852368.1600243082185.JavaMail.zimbra@odiso.com> <1767271081.853403.1600245029802.JavaMail.zimbra@odiso.com> <1894376736.864562.1600253445817.JavaMail.zimbra@odiso.com> <2054513461.868164.1600262132255.JavaMail.zimbra@odiso.com> <2bdde345-b966-d393-44d1-e5385821fbad@proxmox.com> <65105078.871552.1600269422383.JavaMail.zimbra@odiso.com> <1600333910.bmtyynl8cl.astroid@nora.none> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Mailer: Zimbra 8.8.12_GA_3866 (ZimbraWebClient - GC83 (Linux)/8.8.12_GA_3844) Thread-Topic: corosync bug: cluster break after 1 node clean shutdown Thread-Index: kvc1NL2TQsyEIcFjTEwozOCRk3+yGw== X-SPAM-LEVEL: Spam detection results: 0 AWL 0.078 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com, odiso.net] Subject: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> List-Post: <mailto:pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Thu, 17 Sep 2020 10:00:10 -0000 Thanks for the update. >> if >>we can't reproduce it, we'll have to send you patches/patched debs with >>increased logging to narrow down what is going on. if we can, than we >>can hopefully find and fix the issue fast. No problem, I can install the patched deb if needed. ----- Mail original ----- De: "Fabian Gr=C3=BCnbichler" <f.gruenbichler@proxmox.com> =C3=80: "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>, = "Thomas Lamprecht" <t.lamprecht@proxmox.com> Envoy=C3=A9: Jeudi 17 Septembre 2020 11:21:45 Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutd= own On September 16, 2020 5:17 pm, Alexandre DERUMIER wrote:=20 > I have produce it again, with the coredump this time=20 >=20 >=20 > restart corosync : 17:05:27=20 >=20 > http://odisoweb1.odiso.net/pmxcfs-corosync2.log=20 >=20 >=20 > bt full=20 >=20 > https://gist.github.com/aderumier/466dcc4aedb795aaf0f308de0d1c652b=20 >=20 >=20 > coredump=20 >=20 >=20 > http://odisoweb1.odiso.net/core.7761.gz=20 just a short update on this:=20 dcdb is stuck in START_SYNC mode, but nodeid 13 hasn't sent a STATE msg=20 (yet). this looks like either the START_SYNC message to node 13, or the=20 STATE response from it got lost or processed wrong. until the mode=20 switches to SYNCED (after all states have been received and the state=20 update went through), regular/normal messages can be sent, but the=20 incoming normal messages are queued and not processed. this is why the=20 fuse access blocks, it sends the request out, but the response ends up=20 in the queue.=20 status (the other thing running on top of dfsm) got correctly synced up=20 at the same time, so it's either a dcdb specific bug, or just bad luck=20 that one was affected and the other wasn't.=20 unfortunately even with debug enabled the logs don't contain much=20 information that would help (e.g., we don't log sending/receiving STATE=20 messages except when they look 'wrong'), so Thomas is trying to=20 reproduce this using your scenario here to improve turn around time. if=20 we can't reproduce it, we'll have to send you patches/patched debs with=20 increased logging to narrow down what is going on. if we can, than we=20 can hopefully find and fix the issue fast.=20 _______________________________________________=20 pve-devel mailing list=20 pve-devel@lists.proxmox.com=20 https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel=20