From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 33A2999E54 for ; Thu, 4 May 2023 09:29:53 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 1C8D61AB21 for ; Thu, 4 May 2023 09:29:53 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 4 May 2023 09:29:52 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 2C1834724F; Thu, 4 May 2023 09:29:52 +0200 (CEST) Message-ID: <947b91ff-efa5-1ce7-7f63-c2f211eced1e@proxmox.com> Date: Thu, 4 May 2023 09:29:51 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.10.1 To: Proxmox VE user list , Joseph John References: Content-Language: en-US From: Aaron Lauterer In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 2.051 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -4.28 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [PVE-User] HEALTH_ERR , showing x 1 Full osd(s) , guidance requested X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 May 2023 07:29:53 -0000 As already mentioned in another reply, you will have to make space by either reweighting the OSD or adding more OSDs. Another option, though quite a bit more radical, would be to reduce the size of the pool. Right now, you hopefully have a size/min_size of 3/2. ceph osd pool get {pool} size ceph osd pool get {pool} min_size By reducing the size to 2, you will gain about 1/3 of space which can help you get out of the situation. But the pool will become IO blocked once a single OSD is down. So that should only be done as an emergency measure to get operational again. But then you need to address the actual issue ASAP (get more space) so that you can increase the size back to 3 again. ceph osd pool set {pool} size 2 And please plan an upgrade of the cluster soon! Proxmox VE 6 is EOL since last summer and the intricacies of that version and the Ceph versions along with it are fading away ;) https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0 Cheers, Aaron On 5/4/23 08:39, Joseph John wrote: > Dear All, > Good morning > We have a proxmox setup, with 4 nodes, with 6.3-3 > We have Node 1 and Node 2 running with 6.3-3 > and Node 3 and Node 4 running with 6.4-15 > > today we noticed that , we were not able to ssh to the virtual instance , > or not able to login using the console based option > when I checked the summary, we could see that in "HEALTH_ERR" we are > getting message that 1 full osd(s) > > > > Thanks > Joseph John > 00971-50-7451809 > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >