From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id D592E690BF for ; Sat, 16 Jan 2021 13:26:31 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C94242D2CD for ; Sat, 16 Jan 2021 13:26:31 +0100 (CET) Received: from mx-ext.inet.dkfz-heidelberg.de (mx-ext.inet.dkfz-heidelberg.de [192.54.49.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 09D272D2C1 for ; Sat, 16 Jan 2021 13:26:29 +0100 (CET) X-Virus-Scanned-DKFZ: amavisd-new at dkfz-heidelberg.de Received: from w610-mb05.fritz.box ([78.43.207.0]) (authenticated bits=0) by mx-ext.inet.dkfz-heidelberg.de (8.14.7/8.14.7/smtpin) with ESMTP id 10GCQIoo030619 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 16 Jan 2021 13:26:21 +0100 DKIM-Filter: OpenDKIM Filter v2.11.0 mx-ext.inet.dkfz-heidelberg.de 10GCQIoo030619 To: pve-user@lists.proxmox.com References: <21dec802-c6e8-d395-1444-7b30df5620cd@dkfz-heidelberg.de> <255b8af8-8834-0f24-d9a6-819f2d2cf8c8@dkfz-heidelberg.de> <9811d98a-ebf2-8590-ddd0-3b707ede4a4e@dkfz-heidelberg.de> <89a1ad57-6f99-d422-08df-d110f10aa3b9@dkfz-heidelberg.de> <4bdfeb73-582e-2c25-e300-166283e40dc5@dkfz-heidelberg.de> <1073f776-3910-dbde-8304-e86b5f6ed4fb@web.de> <8599b7ad-0ea0-b836-4492-d80b9d43cfcb@dkfz-heidelberg.de> From: Frank Thommen Organization: DKFZ Heidelberg, Omics IT and Data Management Core Facility (ODCF) Message-ID: <8d04daad-107b-4497-dccf-84b95d933a39@dkfz-heidelberg.de> Date: Sat, 16 Jan 2021 13:26:17 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:78.0) Gecko/20100101 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 8bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.6.2 (mx-ext.inet.dkfz-heidelberg.de [192.54.49.101]); Sat, 16 Jan 2021 13:26:21 +0100 (CET) X-Spam-Status: No, score=-100.0 required=5.0 tests=ALL_TRUSTED,NICE_REPLY_A, URIBL_BLOCKED autolearn=disabled version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mx-ext.inet.dkfz-heidelberg.de X-SPAM-LEVEL: Spam detection results: 0 KAM_ASCII_DIVIDERS 0.8 Spam that uses ascii formatting tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.039 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [PVE-User] After update Ceph monitor shows wrong version in UI and is down and out of quorum X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jan 2021 12:26:31 -0000 Just to close this thread on the maillist: I finally made this a support request @proxmox and we are still working on it. It's not an easy case to solve :-) Frank On 08.01.21 13:01, Frank Thommen wrote: > Could this entry be the result of the fencing which happened when the > host initially crashed?  I assumed, that it would automatically be > unfenced when it comes up again.  I never run some manual "unfencing" (I > wouldn't know how). > > Frank > > > > On 08.01.21 12:44, Frank Thommen wrote: >> yes /etc/ceph/ceph.conf is identical on all three hosts and there is a >> mon_host line with the correct IPs.  Interestingly there is a special >> section for odcf-pve02: >> >> ----------- >> [mon.odcf-pve02] >>       public_addr = 192.168.255.2 >> ----------- >> >> This is the same IP as in the mon_host line.  However there is no >> equivalent section for the other two nodes. >> >> Frank >> >> >> On 08.01.21 12:27, Peter Simon wrote: >>> Hi Frank, >>> >>> your /etc/ceph/ceph.conf is the same on all hosts ? >>> >>> is there mon host = ip1, ip2, ip3 >>> >>> and seperate sections with [mon.x] >>> host = hostname >>> mon addr = ip:6789 >>> >>> Cheers >>> Peter >>> >>> Am 08.01.21 um 12:05 schrieb Frank Thommen: >>>> >>>> >>>> On 08.01.21 11:45, Uwe Sauter wrote: >>>>> >>>>> >>>>> Am 08.01.21 um 11:36 schrieb Frank Thommen: >>>>>> >>>>>> On 05.01.21 21:17, Frank Thommen wrote: >>>>>>> On 05.01.21 21:02, Uwe Sauter wrote: >>>>>>>> There's a paragraph about probing mons on >>>>>>>> >>>>>>>> https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/ >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> I will check that (tomorrow :-) >>>>>> >>>>>> >>>>>> using the monitor's admin socket on either of the three nodes I can >>>>>> query the monitors of 01 and 03 (the good ones) but not of 02 (the >>>>>> problematic one): >>>>>> >>>>>> root@odcf-pve01:~# ceph tell mon.odcf-pve02 mon_status >>>>>> Error ENOENT: problem getting command descriptions from >>>>>> mon.odcf-pve02 >>>>>> root@odcf-pve01:~# >>>>>> >>>>>> The monitor daemon is running on all three and the ports are open. >>>>>> >>>>>> Any other ideas? >>>>> >>>>> You could check the permissions on the socket: >>>>> >>>>> ss -xln | grep ceph-mon >>>>> SOCK=$(ss -xln | awk '/ceph-mon/ {print $5}') >>>>> ls -la ${SOCK} >>>>> >>>>> On my host, this shows >>>>> >>>>> srwxr-xr-x 1 ceph ceph 0 Dec 20 23:47 >>>>> /var/run/ceph/ceph-mon.px-alpha-cluster.asok >>>> >>>> same here >>>> >>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user@lists.proxmox.com >>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>> >>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user@lists.proxmox.com >>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >> >> _______________________________________________ >> pve-user mailing list >> pve-user@lists.proxmox.com >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user