From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id DA1401FF0AD for ; Sun, 20 Sep 2026 10:37:45 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 5DDC2214A6; Sun, 20 Sep 2026 10:37:45 +0200 (CEST) From: Kefu Chai To: pve-devel@lists.proxmox.com Subject: [PATCH manager] ceph: osd: remove the lockbox key when destroying an OSD Date: Sun, 20 Sep 2026 16:37:24 +0800 Message-ID: <20260920083724.2054057-1-k.chai@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1789893448863 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.885 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_MAILER 2 Automated Mailer Tag Left in Email RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: SQPJV3WK2KJLVL2CK4V67NGEZPRDXGMX X-Message-ID-Hash: SQPJV3WK2KJLVL2CK4V67NGEZPRDXGMX X-MailFrom: k.chai@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: The destroy worker runs 'osd crush remove', 'auth del' and 'osd rm'. It never asks the monitors for the cleanup that is keyed by the OSD's UUID. So every destroyed OSD leaves behind its lockbox entity 'client.osd-lockbox.', the dm-crypt key of an encrypted OSD in 'dm-crypt/osd//luks' and its daemon-private config keys. The old lockbox entities are then reported as insecure keys when a cluster is checked for the aes256k migration [0]. And the LUKS passphrase of a disk that is long gone stays readable in the mon store. Ask the monitors to destroy the OSD before removing it. They look the UUID up in the OSD map, so the call has to come before 'osd rm'. The monitors also refuse it while the OSD is up, but the endpoint already stops earlier in that case. Move the monitor commands into their own sub, so a test can check them. [0]: https://forum.proxmox.com/threads/186359/ Signed-off-by: Kefu Chai --- PVE/API2/Ceph/OSD.pm | 86 +++++++++++++++++++++++++++----------------- test/OSD_test.pl | 42 +++++++++++++++++++++- 2 files changed, 94 insertions(+), 34 deletions(-) diff --git a/PVE/API2/Ceph/OSD.pm b/PVE/API2/Ceph/OSD.pm index 5df529aa..24b8e9ff 100644 --- a/PVE/API2/Ceph/OSD.pm +++ b/PVE/API2/Ceph/OSD.pm @@ -958,6 +958,58 @@ sub osd_belongs_to_node { return grep($_ == $osdid, @$osds); } +# Drop everything the monitors still hold for $osdid. +# +# Note: 'osd destroy' does the cleanup that is keyed by the OSD's UUID, namely the lockbox +# entity and the dm-crypt key of an encrypted OSD, and the daemon-private config keys. It +# reads that UUID from the OSD map, so it has to run before 'osd rm'. +sub remove_osd_from_monitors { + my ($rados, $osdid) = @_; + + my $osdsection = "osd.$osdid"; + + print "Remove $osdsection from the CRUSH map\n"; + $rados->mon_command({ prefix => "osd crush remove", name => $osdsection, format => 'plain' }); + + print "Remove the $osdsection authentication key.\n"; + $rados->mon_command({ + prefix => "auth del", + entity => $osdsection, + format => 'plain', + }); + + print "Remove the $osdsection lockbox key and private config keys\n"; + $rados->mon_command({ + prefix => "osd destroy-actual", + id => $osdid, + yes_i_really_mean_it => JSON::true, + format => 'plain', + }); + + print "Remove OSD $osdsection\n"; + $rados->mon_command({ + prefix => "osd rm", + ids => [$osdsection], + format => 'plain', + }); + + print "Remove $osdsection mclock max capacity iops settings from config\n"; + $rados->mon_command( + { + prefix => "config rm", + who => $osdsection, + name => 'osd_mclock_max_capacity_iops_ssd', + }, + ); + $rados->mon_command( + { + prefix => "config rm", + who => $osdsection, + name => 'osd_mclock_max_capacity_iops_hdd', + }, + ); +} + __PACKAGE__->register_method({ name => 'destroyosd', path => '{osdid}', @@ -1038,39 +1090,7 @@ __PACKAGE__->register_method({ }; warn $@ if $@; - print "Remove $osdsection from the CRUSH map\n"; - $rados->mon_command( - { prefix => "osd crush remove", name => $osdsection, format => 'plain' }); - - print "Remove the $osdsection authentication key.\n"; - $rados->mon_command({ - prefix => "auth del", - entity => $osdsection, - format => 'plain', - }); - - print "Remove OSD $osdsection\n"; - $rados->mon_command({ - prefix => "osd rm", - ids => [$osdsection], - format => 'plain', - }); - - print "Remove $osdsection mclock max capacity iops settings from config\n"; - $rados->mon_command( - { - prefix => "config rm", - who => $osdsection, - name => 'osd_mclock_max_capacity_iops_ssd', - }, - ); - $rados->mon_command( - { - prefix => "config rm", - who => $osdsection, - name => 'osd_mclock_max_capacity_iops_hdd', - }, - ); + remove_osd_from_monitors($rados, $osdid); # try to unmount from standard mount point my $mountpoint = "/var/lib/ceph/osd/ceph-$osdid"; diff --git a/test/OSD_test.pl b/test/OSD_test.pl index f7bab3a8..83b28ea1 100755 --- a/test/OSD_test.pl +++ b/test/OSD_test.pl @@ -74,4 +74,44 @@ is( "Early-return false if there's no/empty node tree", ); -done_testing(@belong_to_B + @not_belong_to_B + 2); +# Destroying an OSD has to leave nothing of it behind on the monitors. + +{ + + package FakeRados; + + sub new { + my ($class) = @_; + return bless { commands => [] }, $class; + } + + sub mon_command { + my ($self, $cmd) = @_; + push $self->{commands}->@*, $cmd; + return {}; + } +} + +my $rados = FakeRados->new(); +{ + my $out = ''; + open(my $stdout, '>', \$out) or die $!; + local *STDOUT = $stdout; + PVE::API2::Ceph::OSD::remove_osd_from_monitors($rados, 7); +} + +my $commands = $rados->{commands}; + +# The monitors look up the OSD's UUID in the OSD map to find its lockbox entity and its +# dm-crypt key, so the destroy has to come before the 'osd rm'. +is_deeply( + [map { $_->{prefix} } @$commands], + ['osd crush remove', 'auth del', 'osd destroy-actual', 'osd rm', 'config rm', 'config rm'], + 'the monitors drop the CRUSH entry, the keys and the OSD itself, in that order', +); + +my ($destroy) = grep { $_->{prefix} eq 'osd destroy-actual' } @$commands; +is($destroy->{id}, 7, "'osd destroy-actual' names the OSD"); +ok($destroy->{yes_i_really_mean_it}, 'and confirms, as the monitors refuse it otherwise'); + +done_testing(); -- 2.47.3