From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id EA3E465CB1 for ; Mon, 4 Jan 2021 12:45:06 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id E2DDC1D203 for ; Mon, 4 Jan 2021 12:45:06 +0100 (CET) Received: from mx-ext.inet.dkfz-heidelberg.de (mx-ext.inet.dkfz-heidelberg.de [192.54.49.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 67E3B1D26A for ; Mon, 4 Jan 2021 12:45:06 +0100 (CET) X-Virus-Scanned-DKFZ: amavisd-new at dkfz-heidelberg.de Received: from [194.94.115.235] (dkfz-vpn235.inet.dkfz-heidelberg.de [194.94.115.235]) (authenticated bits=0) by mx-ext.inet.dkfz-heidelberg.de (8.14.7/8.14.7/smtpin) with ESMTP id 104Biv0A020751 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Mon, 4 Jan 2021 12:44:58 +0100 DKIM-Filter: OpenDKIM Filter v2.11.0 mx-ext.inet.dkfz-heidelberg.de 104Biv0A020751 To: Proxmox VE user list From: Frank Thommen Organization: DKFZ Heidelberg, Omics IT and Data Management Core Facility (ODCF) Message-ID: <21dec802-c6e8-d395-1444-7b30df5620cd@dkfz-heidelberg.de> Date: Mon, 4 Jan 2021 12:44:57 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.6.2 (mx-ext.inet.dkfz-heidelberg.de [192.54.49.101]); Mon, 04 Jan 2021 12:44:58 +0100 (CET) X-Spam-Status: No, score=-100.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mx-ext.inet.dkfz-heidelberg.de X-SPAM-LEVEL: Spam detection results: 0 AWL 0.350 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [PVE-User] After update Ceph monitor shows wrong version in UI and is down and out of quorum X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Jan 2021 11:45:07 -0000 Dear all, one of our three PVE hypervisors in the cluster crashed (it was fenced successfully) and rebooted automatically. I took the chance to do a complete dist-upgrade and rebooted again. The PVE Ceph dashboard now reports, that * the monitor on the host is down (out of quorum), and * "A newer version was installed but old version still running, please restart" The Ceph UI reports monitor version 14.2.11 while in fact 14.2.16 is installed. The hypervisor has been rebooted twice since the upgrade, so it should be basically impossible that the old version is still running. `systemctl restart ceph.target` and restarting the monitor through the PVE Ceph UI didn't help. The hypervisor is running PVE 6.3-3 (the other two are running 6.3-2 with monitor 14.2.15) What to do in this situation? I am happy with either UI or commandline instructions, but I have no Ceph experience besides setting up it up following the PVE instructions. Any help or hint is appreciated. Cheers, Frank