From: Frank Thommen <f.thommen@dkfz-heidelberg.de>
To: pve-user@lists.proxmox.com
Subject: Re: [PVE-User] After update Ceph monitor shows wrong version in UI and is down and out of quorum
Date: Tue, 5 Jan 2021 20:08:09 +0100 [thread overview]
Message-ID: <cd27dc6a-a3c3-caef-45e0-020089ca6e9a@dkfz-heidelberg.de> (raw)
In-Reply-To: <255b8af8-8834-0f24-d9a6-819f2d2cf8c8@dkfz-heidelberg.de>
On 05.01.21 20:01, Frank Thommen wrote:
>
> On 04.01.21 12:44, Frank Thommen wrote:
>>
>> Dear all,
>>
>> one of our three PVE hypervisors in the cluster crashed (it was fenced
>> successfully) and rebooted automatically. I took the chance to do a
>> complete dist-upgrade and rebooted again.
>>
>> The PVE Ceph dashboard now reports, that
>>
>> * the monitor on the host is down (out of quorum), and
>> * "A newer version was installed but old version still running,
>> please restart"
>>
>> The Ceph UI reports monitor version 14.2.11 while in fact 14.2.16 is
>> installed. The hypervisor has been rebooted twice since the upgrade,
>> so it should be basically impossible that the old version is still
>> running.
>>
>> `systemctl restart ceph.target` and restarting the monitor through the
>> PVE Ceph UI didn't help. The hypervisor is running PVE 6.3-3 (the
>> other two are running 6.3-2 with monitor 14.2.15)
>>
>> What to do in this situation?
>>
>> I am happy with either UI or commandline instructions, but I have no
>> Ceph experience besides setting up it up following the PVE instructions.
>>
>> Any help or hint is appreciated.
>> Cheers, Frank
>
> In an attempt to fix the issue I destroyed the monitor through the UI
> and recreated it. Unfortunately it can still not be started. A popup
> tells me that the monitor has been started, but the overview still shows
> "stopped" and there is no version number any more.
>
> Then I stopped and started Ceph on the node (`pveceph stop; pveceph
> start`) which resulted in a degraded cluster (1 host down, 7 of 21 OSDs
> down). OSDs cannot be started through the UI either.
>
> I feel extremely uncomfortable with this situation and would appreciate
> any hint as to how I should proceed with the problem.
>
> Cheers, Frank
OSDs and MDSs just took a bit to start, so from this side it looks ok
now. But the monitor still refuses to start.
Frank
next prev parent reply other threads:[~2021-01-05 19:08 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-04 11:44 Frank Thommen
2021-01-05 19:01 ` Frank Thommen
2021-01-05 19:08 ` Frank Thommen [this message]
2021-01-05 19:10 ` Uwe Sauter
2021-01-05 19:24 ` Frank Thommen
2021-01-05 19:29 ` Uwe Sauter
2021-01-05 19:44 ` Frank Thommen
[not found] ` <f3ca5f88-5cbd-9807-02d7-d8f24fcbefdb@gmail.com>
2021-01-05 20:17 ` Frank Thommen
2021-01-08 10:36 ` Frank Thommen
2021-01-08 10:45 ` Uwe Sauter
2021-01-08 11:05 ` Frank Thommen
2021-01-08 11:27 ` Peter Simon
2021-01-08 11:44 ` Frank Thommen
2021-01-08 11:57 ` Peter Simon
2021-01-08 12:01 ` Frank Thommen
2021-01-16 12:26 ` Frank Thommen
[not found] ` <058f3eca-2e6f-eead-365a-4d451fa160d3@gmail.com>
2021-01-05 20:18 ` Frank Thommen
2021-01-05 19:35 ` Frank Thommen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cd27dc6a-a3c3-caef-45e0-020089ca6e9a@dkfz-heidelberg.de \
--to=f.thommen@dkfz-heidelberg.de \
--cc=pve-user@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox