From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 6C23D6086A for ; Thu, 26 Nov 2020 16:03:24 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 6332212B54 for ; Thu, 26 Nov 2020 16:03:24 +0100 (CET) Received: from mail-pg1-x530.google.com (mail-pg1-x530.google.com [IPv6:2607:f8b0:4864:20::530]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 59E5212B3B for ; Thu, 26 Nov 2020 16:03:23 +0100 (CET) Received: by mail-pg1-x530.google.com with SMTP id 81so1969490pgf.0 for ; Thu, 26 Nov 2020 07:03:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=QA+3zM0Bku9uCf2iZDcA7auJ/aFtK0T1Yn1nkejnrUA=; b=UX3Cej95oVc94l9K6KvUXqWE6o/dM7E59llCwidXHMb46Yt3SjH6KLhsJVZcSaV0NR an8Cp39xEu9O+ZuK9haloN8pABjd3Vjti2NhX1YApCUOXvGOmgcI+bG6BkLmgScNdpJg eUTboHm0qYcg/7j9iZk03oTEDQvgGBHW+333hVdlnKw7vfpLEC7gY0zrcVyZy/jBEQRE mDfrRAcH8LGQmCVYwgeCURnIYNzr/8tunBzdIpn7pUDFxfzqnWTDwM8F1pwEbK06izMK wTIRyOP6Wh46C/lroXiq5Vj7prgvKs+dF36l7kGLoNqvzgMJTs6Av7wvZlmFbaaVRkPG IM2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=QA+3zM0Bku9uCf2iZDcA7auJ/aFtK0T1Yn1nkejnrUA=; b=OxoeDKRePaYkAJlZkrAaZQ/F4nf2St+KFCsfBRgP/J/kT0mNJzBWnu25/2c5uetmzF Tc/xFZeaXCwuZF3g90ETkmOyboK0RX2nBX3Q1Ixhw173i6hHfWQU9Uj3dMz/V8yWG//B cyWEzOZIox53emuQ1EKyJ/JWaBDzy9qyhbiKeNz0wunzyez0Q0Jv/DjYX72dqK0LKfT2 wHQ9pa3HiAUjM7mpf8z/CsTzApL4/DNmL0XmouVSahn+ZxvVKzHyZxpzMjoSiit6rBeY XMWUt8jnecnm6iU/jPR6/HcKIzoJlWkPSjZ77+7kqoqkSK8lSUo37e+BePMNHDz51ssI W3Lg== X-Gm-Message-State: AOAM533rEr4dKuZa9Ircc1CAxAZK7VDygCs29AKPkGJ7tjrpFJmyEDj/ ukcQx/HRMAXPdJb6g/i0qOZ3iAqV1cn1WA== X-Google-Smtp-Source: ABdhPJwLcz66+JuTN0zmSrgjbaFgCYw7kaGrIohjUXYYVx9UdvcXgQR5b4NFr+Ju4A/Y+mtduB2tlQ== X-Received: by 2002:a17:90b:4b07:: with SMTP id lx7mr4161408pjb.230.1606402995175; Thu, 26 Nov 2020 07:03:15 -0800 (PST) Received: from [192.168.1.11] (167-179-176-9.a7b3b0.bne.nbn.aussiebb.net. [167.179.176.9]) by smtp.gmail.com with ESMTPSA id r11sm7123682pjd.52.2020.11.26.07.03.12 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 26 Nov 2020 07:03:13 -0800 (PST) To: pve-user@lists.proxmox.com References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <15f5893d-83cf-ee8c-11ff-0a67e376dc41@gmail.com> <9c0ceb9e-ff96-b12f-a4ed-1e1ac250fbb6@gmail.com> <11d60eb0-9b17-53ba-cc21-79728e1296b0@proxmox.com> From: Lindsay Mathieson Message-ID: <269aba60-412c-578b-9757-6a0567d270e5@gmail.com> Date: Fri, 27 Nov 2020 01:03:10 +1000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: <11d60eb0-9b17-53ba-cc21-79728e1296b0@proxmox.com> Content-Language: en-US X-SPAM-LEVEL: Spam detection results: 0 AWL -0.036 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider HTML_MESSAGE 0.001 HTML included in message NICE_REPLY_A -0.001 Looks like a legit reply (A) RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [anyevent.pm] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] Caution: ceph-mon service does not start after today's updates X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Nov 2020 15:03:24 -0000 On 27/11/2020 12:46 am, Thomas Lamprecht wrote: > Note, it's always good idea to check if all services are running OK again before > continuing with upgrading the next host, not just on this update:-) > > Also, ceph monitors can be nicely restarted over the web interface, there's a > visible status about which services run outdated versions/need a restart. > > > Anyway, do you have any logs which could give more details for possible issues? I have a node that is just failing to rejoin the cluster and the ceph mon & mgr fail to start. Seeing this repeated in syslog Nov 27 00:58:23 vnh pveproxy[2903]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737. Nov 27 00:58:23 vnh pveproxy[2904]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737. Nov 27 00:58:23 vnh pveproxy[2905]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737. Nov 27 00:58:26 vnh ceph-mon[2073]: 2020-11-27 00:58:26.378 7fb182935700 -1 mon.vnh@0(probing) e9 handle_auth_bad_method hmm, they didn't like 2 result (95) Operation not supported Nov 27 00:58:26 vnh ceph-mon[2073]: 2020-11-27 00:58:26.390 7fb17d92b700 -1 mon.vnh@0(probing) e9 handle_auth_bad_method hmm, they didn't like 2 result (95) Operation not supported Nov 27 00:58:26 vnh ceph-mon[2073]: 2020-11-27 00:58:26.526 7fb183136700 -1 mon.vnh@0(probing) e9 handle_auth_bad_method hmm, they didn't like 2 result (95) Operation not supported Nov 27 00:58:27 vnh ceph-mon[2073]: 2020-11-27 00:58:27.702 7fb182935700 -1 mon.vnh@0(probing) e9 handle_auth_request no AuthAuthorizeHandler found for auth method 1 The following gets the node back on the cluster: systemctl start pve-cluster.service systemctl restart pvestatd.service But I can't get the mon, mgr or osd services to start. -- Lindsay