From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <t.lamprecht@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 5C31760899 for <pve-user@lists.proxmox.com>; Thu, 26 Nov 2020 17:15:12 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 4B94E1397A for <pve-user@lists.proxmox.com>; Thu, 26 Nov 2020 17:14:42 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 675EB1396F for <pve-user@lists.proxmox.com>; Thu, 26 Nov 2020 17:14:41 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 322EE45CAB; Thu, 26 Nov 2020 17:14:41 +0100 (CET) To: Proxmox VE user list <pve-user@lists.proxmox.com>, Lindsay Mathieson <lindsay.mathieson@gmail.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <15f5893d-83cf-ee8c-11ff-0a67e376dc41@gmail.com> <9c0ceb9e-ff96-b12f-a4ed-1e1ac250fbb6@gmail.com> <11d60eb0-9b17-53ba-cc21-79728e1296b0@proxmox.com> <269aba60-412c-578b-9757-6a0567d270e5@gmail.com> From: Thomas Lamprecht <t.lamprecht@proxmox.com> Message-ID: <72fbd5d5-1c9b-5d59-5256-4cf1dd1f79cb@proxmox.com> Date: Thu, 26 Nov 2020 17:14:40 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:83.0) Gecko/20100101 Thunderbird/83.0 MIME-Version: 1.0 In-Reply-To: <269aba60-412c-578b-9757-6a0567d270e5@gmail.com> Content-Type: text/plain; charset=UTF-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL -0.078 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.001 Looks like a legit reply (A) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [default.target, anyevent.pm] Subject: Re: [PVE-User] Caution: ceph-mon service does not start after today's updates X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list <pve-user.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-user/> List-Post: <mailto:pve-user@lists.proxmox.com> List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>, <mailto:pve-user-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Thu, 26 Nov 2020 16:15:12 -0000 On 26.11.20 16:03, Lindsay Mathieson wrote: > On 27/11/2020 12:46 am, Thomas Lamprecht wrote: >> Note, it's always good idea to check if all services are running OK ag= ain before >> continuing with upgrading the next host, not just on this update:-) >> >> Also, ceph monitors can be nicely restarted over the web interface, th= ere's a >> visible status about which services run outdated versions/need a resta= rt. >> >> >> Anyway, do you have any logs which could give more details for possibl= e issues? >=20 > I have a node that is just failing to rejoin the cluster and the ceph m= on & mgr fail to start. >=20 >=20 > Seeing this repeated in syslog >=20 > =C2=A0=C2=A0 Nov 27 00:58:23 vnh pveproxy[2903]: /etc/pve/local/pve-ssl= =2Ekey: > =C2=A0=C2=A0 failed to load local private key (key_file or key) at > =C2=A0=C2=A0 /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737. > =C2=A0=C2=A0 Nov 27 00:58:23 vnh pveproxy[2904]: /etc/pve/local/pve-ssl= =2Ekey: > =C2=A0=C2=A0 failed to load local private key (key_file or key) at > =C2=A0=C2=A0 /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737. > =C2=A0=C2=A0 Nov 27 00:58:23 vnh pveproxy[2905]: /etc/pve/local/pve-ssl= =2Ekey: > =C2=A0=C2=A0 failed to load local private key (key_file or key) at > =C2=A0=C2=A0 /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737. > =C2=A0=C2=A0 Nov 27 00:58:26 vnh ceph-mon[2073]: 2020-11-27 00:58:26.37= 8 > =C2=A0=C2=A0 7fb182935700 -1 mon.vnh@0(probing) e9 handle_auth_bad_meth= od hmm, > =C2=A0=C2=A0 they didn't like 2 result (95) Operation not supported > =C2=A0=C2=A0 Nov 27 00:58:26 vnh ceph-mon[2073]: 2020-11-27 00:58:26.39= 0 > =C2=A0=C2=A0 7fb17d92b700 -1 mon.vnh@0(probing) e9 handle_auth_bad_meth= od hmm, > =C2=A0=C2=A0 they didn't like 2 result (95) Operation not supported > =C2=A0=C2=A0 Nov 27 00:58:26 vnh ceph-mon[2073]: 2020-11-27 00:58:26.52= 6 > =C2=A0=C2=A0 7fb183136700 -1 mon.vnh@0(probing) e9 handle_auth_bad_meth= od hmm, > =C2=A0=C2=A0 they didn't like 2 result (95) Operation not supported > =C2=A0=C2=A0 Nov 27 00:58:27 vnh ceph-mon[2073]: 2020-11-27 00:58:27.70= 2 > =C2=A0=C2=A0 7fb182935700 -1 mon.vnh@0(probing) e9 handle_auth_request = no > =C2=A0=C2=A0 AuthAuthorizeHandler found for auth method 1 >=20 the errors seems like being the result of pve-cluster not coming up, which seems the actual problem. >=20 > The following gets the node back on the cluster: >=20 > systemctl start pve-cluster.service Anything of pve-cluster service in the log? What does: # systemd-analyze verify default.target outputs? cheers, Thomas