From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id D073860741 for ; Thu, 13 Jan 2022 09:14:34 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id BCACA1AD91 for ; Thu, 13 Jan 2022 09:14:04 +0100 (CET) Received: from relay163.nicmail.ru (relay163.nicmail.ru [91.189.117.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 5786D1AD80 for ; Thu, 13 Jan 2022 09:14:01 +0100 (CET) Received: from [10.28.138.151] (port=50948 helo=[192.168.8.155]) by relay.hosting.mail.nic.ru with esmtp (Exim 5.55) (envelope-from ) id 1n7vF4-0007hI-Am for pve-user@lists.proxmox.com; Thu, 13 Jan 2022 11:13:54 +0300 Received: from [62.105.41.93] (account tsabolov@t8.ru HELO [192.168.8.155]) by incarp1103.int.hosting.nic.ru (Exim 5.55) with id 1n7vF4-00007x-8s for pve-user@lists.proxmox.com; Thu, 13 Jan 2022 11:13:54 +0300 To: Proxmox VE user list From: =?UTF-8?B?0KHQtdGA0LPQtdC5INCm0LDQsdC+0LvQvtCy?= Message-ID: <37cd7904-a225-76f6-7bd3-973a75be2c74@t8.ru> Date: Thu, 13 Jan 2022 11:13:53 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-KLMS-AntiSpam-Auth: dkim=none X-MS-Exchange-Organization-SCL: -1 X-SPAM-LEVEL: Spam detection results: 0 AWL -1.109 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [PVE-User] Remove 1-2 OSD from PVE Cluster X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Jan 2022 08:14:34 -0000 Hello to all. I have cluster with 7 node. Storage for VM disk and others pool data is on ceph version 15.2.15 (4b7a17f73998a0b4d9bd233cda1db482107e5908) octopus (stable) On pve-7 I have 10 OSD and for test I want to remove 2 osd from this node. I write some steps command how I remove this OSD from pve-7 cluster and ceph storage. 1) root@pve-7 ~ # ceph osd tree 2) ceph osd reweight 2.1) ceph osd reweight osd.${ID} 0.98 2.2) ceph osd reweight osd.${ID} 0.0 can I set the 0.0 to clean osd.X before remove it? 3) When osd is clean from data can I? ceph osd down osd.${ID} 4) Remove the osd from cluster ceph osd out osd.${ID} 5) Stop the OSD and umount the osd systemctl stop ceph-osd@${ID} umount /var/lib/ceph/osd/ceph-${ID} 6) Remove the osd from CRUSH map: ceph osd crush remove osd.${ID} 7) Remove the user of OSD: ceph auth del osd.${ID} 8) And now full delete the OSD: ceph osd rm osd.${ID} 9) Last command ceph osd tree (removed OSD not showing Can some one suggest me my steps command is correct ? Or I need change some steps? Thanks . -- ------------------------- Sergey TS The best Regard