* [pve-devel] [PATCH pve-manager master v1 0/2] Fix #6652: LVM Autoactivation Missing for Ceph OSD LVs @ 2025-08-12 16:46 Max R. Carrara 2025-08-12 16:46 ` [pve-devel] [PATCH pve-manager master v1 1/2] fix #6652: ceph: osd: enable autoactivation for OSD LVs on creation Max R. Carrara ` (2 more replies) 0 siblings, 3 replies; 12+ messages in thread From: Max R. Carrara @ 2025-08-12 16:46 UTC (permalink / raw) To: pve-devel Fix #6652: LVM Autoactivation Missing for Ceph OSD LVs - v1 =========================================================== When creating an OSD via the API, the logical volumes backing the OSD's DB and WAL do not have autoactivation enabled. Ceph requires autoactivation on LVs, as it otherwise never activates them directly itself. Fix this by setting autoactivation when creating those LVs as well as providing a helper script that enables autoactivation for LVs used by OSDs. The script also activates any inactive OSD LVs and then also attempts to bring the OSDs back online via ceph-volume. This regression is caused by a recent change in pve-storage [0] --unfortunately, the helpers for the LVM storage plugin are used in pve-manager directly for configuring OSDs, which isn't something one considers when working on a completely different package. I'm in the process of cleaning up those helpers anyway, so expect a another series soon-ish that removes their usage from pve-manager and attempts to confine them to pve-storage as much as possible. References ---------- [0]: https://git.proxmox.com/?p=pve-storage.git;a=commitdiff;h=f296ffc4e4d64b574c3001dc7cc6af3da1406441 Summary of Changes ------------------ Max R. Carrara (2): fix #6652: ceph: osd: enable autoactivation for OSD LVs on creation fix #6652: d/postinst: enable autoactivation for Ceph OSD LVs PVE/API2/Ceph/OSD.pm | 20 ++- bin/Makefile | 3 +- bin/pve-osd-lvm-enable-autoactivation | 195 ++++++++++++++++++++++++++ debian/postinst | 16 +++ 4 files changed, 232 insertions(+), 2 deletions(-) create mode 100644 bin/pve-osd-lvm-enable-autoactivation -- 2.47.2 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* [pve-devel] [PATCH pve-manager master v1 1/2] fix #6652: ceph: osd: enable autoactivation for OSD LVs on creation 2025-08-12 16:46 [pve-devel] [PATCH pve-manager master v1 0/2] Fix #6652: LVM Autoactivation Missing for Ceph OSD LVs Max R. Carrara @ 2025-08-12 16:46 ` Max R. Carrara 2025-08-13 7:48 ` Fabian Grünbichler 2025-08-12 16:46 ` [pve-devel] [PATCH pve-manager master v1 2/2] fix #6652: d/postinst: enable autoactivation for Ceph OSD LVs Max R. Carrara 2025-08-13 13:43 ` [pve-devel] superseded: [PATCH pve-manager master v1 0/2] Fix #6652: LVM Autoactivation Missing " Max R. Carrara 2 siblings, 1 reply; 12+ messages in thread From: Max R. Carrara @ 2025-08-12 16:46 UTC (permalink / raw) To: pve-devel ... by directly calling lvcreate instead of using the LVM storage plugin's helper sub. Autoactivation is required for LVs used by Ceph OSDs, as Ceph otherwise doesn't activate them by itself. This is a regression from f296ffc4e4d in pve-storage [0]. [0]: https://git.proxmox.com/?p=pve-storage.git;a=commitdiff;h=f296ffc4e4d64b574c3001dc7cc6af3da1406441 Fixes: #6652 Signed-off-by: Max R. Carrara <m.carrara@proxmox.com> --- PVE/API2/Ceph/OSD.pm | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/PVE/API2/Ceph/OSD.pm b/PVE/API2/Ceph/OSD.pm index 23e187ce..4801f8dd 100644 --- a/PVE/API2/Ceph/OSD.pm +++ b/PVE/API2/Ceph/OSD.pm @@ -443,7 +443,25 @@ __PACKAGE__->register_method({ my $lv = $type . "-" . UUID::uuid(); PVE::Storage::LVMPlugin::lvm_create_volume_group($dev->{devpath}, $vg); - PVE::Storage::LVMPlugin::lvcreate($vg, $lv, "${size}k"); + + my $cmd = [ + '/sbin/lvcreate', + '-aly', + '-Wy', + '--yes', + '--size', + # size in kilobytes + $size . "k", + '--name', + $lv, + # explicitly enable autoactivation, + # otherwise ceph-volume cannot bring the LV online + '--setautoactivation', + 'y', + $vg, + ]; + + run_command($cmd, errmsg => "lvcreate '$vg/$lv' error"); if (PVE::Diskmanage::is_partition($dev->{devpath})) { eval { PVE::Diskmanage::change_parttype($dev->{devpath}, '8E00'); }; -- 2.47.2 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [pve-devel] [PATCH pve-manager master v1 1/2] fix #6652: ceph: osd: enable autoactivation for OSD LVs on creation 2025-08-12 16:46 ` [pve-devel] [PATCH pve-manager master v1 1/2] fix #6652: ceph: osd: enable autoactivation for OSD LVs on creation Max R. Carrara @ 2025-08-13 7:48 ` Fabian Grünbichler 2025-08-13 8:28 ` Max R. Carrara 0 siblings, 1 reply; 12+ messages in thread From: Fabian Grünbichler @ 2025-08-13 7:48 UTC (permalink / raw) To: Proxmox VE development discussion On August 12, 2025 6:46 pm, Max R. Carrara wrote: > ... by directly calling lvcreate instead of using the LVM storage > plugin's helper sub. > > Autoactivation is required for LVs used by Ceph OSDs, as Ceph > otherwise doesn't activate them by itself. > > This is a regression from f296ffc4e4d in pve-storage [0]. > > [0]: https://git.proxmox.com/?p=pve-storage.git;a=commitdiff;h=f296ffc4e4d64b574c3001dc7cc6af3da1406441 > > Fixes: #6652 > Signed-off-by: Max R. Carrara <m.carrara@proxmox.com> > --- > PVE/API2/Ceph/OSD.pm | 20 +++++++++++++++++++- > 1 file changed, 19 insertions(+), 1 deletion(-) > > diff --git a/PVE/API2/Ceph/OSD.pm b/PVE/API2/Ceph/OSD.pm > index 23e187ce..4801f8dd 100644 > --- a/PVE/API2/Ceph/OSD.pm > +++ b/PVE/API2/Ceph/OSD.pm > @@ -443,7 +443,25 @@ __PACKAGE__->register_method({ > my $lv = $type . "-" . UUID::uuid(); > > PVE::Storage::LVMPlugin::lvm_create_volume_group($dev->{devpath}, $vg); > - PVE::Storage::LVMPlugin::lvcreate($vg, $lv, "${size}k"); > + > + my $cmd = [ > + '/sbin/lvcreate', > + '-aly', > + '-Wy', > + '--yes', > + '--size', > + # size in kilobytes > + $size . "k", > + '--name', > + $lv, > + # explicitly enable autoactivation, > + # otherwise ceph-volume cannot bring the LV online > + '--setautoactivation', > + 'y', > + $vg, > + ]; > + > + run_command($cmd, errmsg => "lvcreate '$vg/$lv' error"); there's a second call to PVE::Storage::LVMPlugin::lvcreate a few lines below, is that not also problematic? > > if (PVE::Diskmanage::is_partition($dev->{devpath})) { > eval { PVE::Diskmanage::change_parttype($dev->{devpath}, '8E00'); }; > -- > 2.47.2 > > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > > _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [pve-devel] [PATCH pve-manager master v1 1/2] fix #6652: ceph: osd: enable autoactivation for OSD LVs on creation 2025-08-13 7:48 ` Fabian Grünbichler @ 2025-08-13 8:28 ` Max R. Carrara 0 siblings, 0 replies; 12+ messages in thread From: Max R. Carrara @ 2025-08-13 8:28 UTC (permalink / raw) To: Proxmox VE development discussion On Wed Aug 13, 2025 at 9:48 AM CEST, Fabian Grünbichler wrote: > On August 12, 2025 6:46 pm, Max R. Carrara wrote: > > ... by directly calling lvcreate instead of using the LVM storage > > plugin's helper sub. > > > > Autoactivation is required for LVs used by Ceph OSDs, as Ceph > > otherwise doesn't activate them by itself. > > > > This is a regression from f296ffc4e4d in pve-storage [0]. > > > > [0]: https://git.proxmox.com/?p=pve-storage.git;a=commitdiff;h=f296ffc4e4d64b574c3001dc7cc6af3da1406441 > > > > Fixes: #6652 > > Signed-off-by: Max R. Carrara <m.carrara@proxmox.com> > > --- > > PVE/API2/Ceph/OSD.pm | 20 +++++++++++++++++++- > > 1 file changed, 19 insertions(+), 1 deletion(-) > > > > diff --git a/PVE/API2/Ceph/OSD.pm b/PVE/API2/Ceph/OSD.pm > > index 23e187ce..4801f8dd 100644 > > --- a/PVE/API2/Ceph/OSD.pm > > +++ b/PVE/API2/Ceph/OSD.pm > > @@ -443,7 +443,25 @@ __PACKAGE__->register_method({ > > my $lv = $type . "-" . UUID::uuid(); > > > > PVE::Storage::LVMPlugin::lvm_create_volume_group($dev->{devpath}, $vg); > > - PVE::Storage::LVMPlugin::lvcreate($vg, $lv, "${size}k"); > > + > > + my $cmd = [ > > + '/sbin/lvcreate', > > + '-aly', > > + '-Wy', > > + '--yes', > > + '--size', > > + # size in kilobytes > > + $size . "k", > > + '--name', > > + $lv, > > + # explicitly enable autoactivation, > > + # otherwise ceph-volume cannot bring the LV online > > + '--setautoactivation', > > + 'y', > > + $vg, > > + ]; > > + > > + run_command($cmd, errmsg => "lvcreate '$vg/$lv' error"); > > there's a second call to PVE::Storage::LVMPlugin::lvcreate a few lines > below, is that not also problematic? Mea culpa; missed that one. Will send a v2. > > > > > if (PVE::Diskmanage::is_partition($dev->{devpath})) { > > eval { PVE::Diskmanage::change_parttype($dev->{devpath}, '8E00'); }; > > -- > > 2.47.2 > > > > > > > > _______________________________________________ > > pve-devel mailing list > > pve-devel@lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > > > > > > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* [pve-devel] [PATCH pve-manager master v1 2/2] fix #6652: d/postinst: enable autoactivation for Ceph OSD LVs 2025-08-12 16:46 [pve-devel] [PATCH pve-manager master v1 0/2] Fix #6652: LVM Autoactivation Missing for Ceph OSD LVs Max R. Carrara 2025-08-12 16:46 ` [pve-devel] [PATCH pve-manager master v1 1/2] fix #6652: ceph: osd: enable autoactivation for OSD LVs on creation Max R. Carrara @ 2025-08-12 16:46 ` Max R. Carrara 2025-08-13 7:52 ` Fabian Grünbichler 2025-08-13 13:43 ` [pve-devel] superseded: [PATCH pve-manager master v1 0/2] Fix #6652: LVM Autoactivation Missing " Max R. Carrara 2 siblings, 1 reply; 12+ messages in thread From: Max R. Carrara @ 2025-08-12 16:46 UTC (permalink / raw) To: pve-devel Introduce a new helper command pve-osd-lvm-enable-autoactivation, which gracefully tries to enable autoactivation for all logical volumes used by Ceph OSDs while also activating any LVs that aren't active yet. Afterwards, the helper attempts to bring all OSDs online. Fixes: #6652 Signed-off-by: Max R. Carrara <m.carrara@proxmox.com> --- bin/Makefile | 3 +- bin/pve-osd-lvm-enable-autoactivation | 195 ++++++++++++++++++++++++++ debian/postinst | 16 +++ 3 files changed, 213 insertions(+), 1 deletion(-) create mode 100644 bin/pve-osd-lvm-enable-autoactivation diff --git a/bin/Makefile b/bin/Makefile index 777e6759..0a0df34d 100644 --- a/bin/Makefile +++ b/bin/Makefile @@ -32,7 +32,8 @@ HELPERS = \ pve-startall-delay \ pve-init-ceph-crash \ pve-firewall-commit \ - pve-sdn-commit + pve-sdn-commit \ + pve-osd-lvm-enable-autoactivation MIGRATIONS = \ pve-lvm-disable-autoactivation \ diff --git a/bin/pve-osd-lvm-enable-autoactivation b/bin/pve-osd-lvm-enable-autoactivation new file mode 100644 index 00000000..acdc91e8 --- /dev/null +++ b/bin/pve-osd-lvm-enable-autoactivation @@ -0,0 +1,195 @@ +#!/usr/bin/perl + +use v5.36; + +use JSON qw(decode_json); + +use PVE::Tools; + +my sub ceph_volume_lvm_osd_info : prototype() () { + my $cmd = [ + "/usr/sbin/ceph-volume", "lvm", "list", "--format", "json", + ]; + + my $stdout = ''; + my $outfunc = sub($line) { + $stdout .= "$line\n"; + }; + + PVE::Tools::run_command($cmd, timeout => 10, outfunc => $outfunc); + my $osd_info = decode_json($stdout); + + return $osd_info; +} + +my sub lvs_from_osd_info : prototype($) ($osd_info) { + my @lvs_for_osds = (); + + for my $osd (keys $osd_info->%*) { + my $osd_lvs = $osd_info->{$osd}; + + for my $osd_lv ($osd_lvs->@*) { + my ($lv_name, $vg_name) = $osd_lv->@{qw(lv_name vg_name)}; + push(@lvs_for_osds, "$vg_name/$lv_name"); + } + } + + return \@lvs_for_osds; +} + +my sub lvs : prototype() () { + my $cmd = [ + "/usr/sbin/lvs", + "--noheadings", + "--separator", + ":", + "--options", + "lv_name,vg_name,autoactivation,active", + ]; + + my $all_lvs = {}; + + my $outfunc = sub($line) { + $line = PVE::Tools::trim($line); + + my ($lv_name, $vg_name, $autoactivation, $active) = split(':', $line, -1); + + return undef if ($lv_name eq '' || $vg_name eq ''); + + $all_lvs->{"$vg_name/$lv_name"} = { + autoactivation => $autoactivation, + active => $active, + }; + }; + + PVE::Tools::run_command( + $cmd, + timeout => 10, + outfunc => $outfunc, + ); + + return $all_lvs; +} + +my sub main : prototype() () { + my $osd_info = ceph_volume_lvm_osd_info(); + my $all_lvs = lvs(); + + my @osd_lvs_no_autoactivation = (); + my @osd_lvs_inactive = (); + + for my $osd (keys $osd_info->%*) { + for my $osd_lv ($osd_info->{$osd}->@*) { + my ($lv_name, $vg_name) = $osd_lv->@{qw(lv_name vg_name)}; + + my $osd_lv = "$vg_name/$lv_name"; + + push(@osd_lvs_no_autoactivation, $osd_lv) if !$all_lvs->{$osd_lv}->{autoactivation}; + push(@osd_lvs_inactive, $osd_lv) if !$all_lvs->{$osd_lv}->{active}; + } + } + + my $has_set_autoactivation_err = 0; + + # Logical volumes are formatted as "vg_name/lv_name", which is necessary for lvchange + for my $lv (@osd_lvs_no_autoactivation) { + print("Enabling autoactivation for OSD logical volume '$lv' ...\n"); + + eval { + my $cmd = [ + '/usr/sbin/lvchange', '--setautoactivation', 'y', $lv, + ]; + + PVE::Tools::run_command( + $cmd, + quiet => 1, + timeout => 10, + ); + }; + if (my $err = $@) { + $has_set_autoactivation_err = 1; + + warn("Error: Failed to enable autoactivation for OSD LV '$lv'\n"); + warn("$@\n"); + + next; + } + + } + + my $has_activation_err = 0; + + # Activate any inactive OSD LVs so that ceph-volume can later bring up any failed OSDs + for my $lv (@osd_lvs_inactive) { + print("Activating OSD logical volume '$lv' ...\n"); + + eval { + my $cmd = [ + '/usr/sbin/lvchange', '--activate', 'y', $lv, + ]; + + PVE::Tools::run_command( + $cmd, + quiet => 1, + timeout => 10, + ); + }; + if (my $err = $@) { + $has_activation_err = 1; + + warn("Error: Failed to activate OSD LV '$lv'\n"); + warn("$@\n"); + + next; + } + } + + # ceph-volume requires all LVs used by OSDs to be active, + # so exit in case there are any we couldn't activate + if ($has_set_autoactivation_err || $has_activation_err) { + if ($has_set_autoactivation_err) { + warn("Couldn't enable autoactivation for all OSD LVs.\n"); + } + + if ($has_activation_err) { + warn("Couldn't activate all OSD LVs.\n"); + } + + exit 1; + } + + print("Activating OSDs ...\n"); + eval { + my $cmd = [ + "/usr/sbin/ceph-volume", "lvm", "activate", "--all", + ]; + + # ceph-volume prints everything to stderr for whatever reason and prefixes + # what actually goes to stderr with " stderr:", so separate the output here. + # Lines starting with "Running command:" are removed because they're overly + # verbose. + my $logfunc = sub ($line) { + return if $line =~ m/^Running command:/; + + if ($line =~ s/^\s*stderr:\s*//) { + print STDERR "$line\n"; + } else { + print STDOUT "$line\n"; + } + }; + + PVE::Tools::run_command( + $cmd, + logfunc => $logfunc, + timeout => 30, + ); + }; + if (my $err = $@) { + warn("Error while activating all Ceph LVM volumes: $@\n"); + exit 1; + } + + return undef; +} + +main(); diff --git a/debian/postinst b/debian/postinst index b6e07fd9..f550e7bb 100755 --- a/debian/postinst +++ b/debian/postinst @@ -133,6 +133,18 @@ migrate_apt_auth_conf() { fi } +ceph_osd_lvm_enable_autoactivation() { + if ! test -e /usr/sbin/ceph-volume; then + return + fi + + if ! /usr/share/pve-manager/helpers/pve-osd-lvm-enable-autoactivation; then + printf "\nEnabling autoactivation for logical volumes used by Ceph OSDs failed."; + printf " Check the output above for errors and try to enable autoactivation for OSD LVs"; + printf " manually by running '/usr/share/pve-manager/helpers/pve-osd-lvm-enable-autoactivation'"; + fi +} + # Copied from dh_installtmpfiles/13.24.2 if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then if [ -x "$(command -v systemd-tmpfiles)" ]; then @@ -246,6 +258,10 @@ case "$1" in fi fi + if test -n "$2" && dpkg --compare-versions "$2" 'lt' '9.0.5'; then + ceph_osd_lvm_enable_autoactivation + fi + ;; abort-upgrade|abort-remove|abort-deconfigure) -- 2.47.2 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [pve-devel] [PATCH pve-manager master v1 2/2] fix #6652: d/postinst: enable autoactivation for Ceph OSD LVs 2025-08-12 16:46 ` [pve-devel] [PATCH pve-manager master v1 2/2] fix #6652: d/postinst: enable autoactivation for Ceph OSD LVs Max R. Carrara @ 2025-08-13 7:52 ` Fabian Grünbichler 2025-08-13 8:50 ` Max R. Carrara 0 siblings, 1 reply; 12+ messages in thread From: Fabian Grünbichler @ 2025-08-13 7:52 UTC (permalink / raw) To: Proxmox VE development discussion On August 12, 2025 6:46 pm, Max R. Carrara wrote: > Introduce a new helper command pve-osd-lvm-enable-autoactivation, > which gracefully tries to enable autoactivation for all logical > volumes used by Ceph OSDs while also activating any LVs that aren't > active yet. Afterwards, the helper attempts to bring all OSDs online. I think this is probably overkill - this only affects a specific non standard setup, the breakage is really obvious, and the fix is easy: either run lvchange on all those LVs, or recreate the OSDs after the fix for creation is rolled out.. i.e., the fallout from some edge cases not being handled correctly in the 200 line helper script here is probably worse than the few setups that run into the original issue that we can easily help along manually.. > > Fixes: #6652 > Signed-off-by: Max R. Carrara <m.carrara@proxmox.com> > --- > bin/Makefile | 3 +- > bin/pve-osd-lvm-enable-autoactivation | 195 ++++++++++++++++++++++++++ > debian/postinst | 16 +++ > 3 files changed, 213 insertions(+), 1 deletion(-) > create mode 100644 bin/pve-osd-lvm-enable-autoactivation > > diff --git a/bin/Makefile b/bin/Makefile > index 777e6759..0a0df34d 100644 > --- a/bin/Makefile > +++ b/bin/Makefile > @@ -32,7 +32,8 @@ HELPERS = \ > pve-startall-delay \ > pve-init-ceph-crash \ > pve-firewall-commit \ > - pve-sdn-commit > + pve-sdn-commit \ > + pve-osd-lvm-enable-autoactivation > > MIGRATIONS = \ > pve-lvm-disable-autoactivation \ > diff --git a/bin/pve-osd-lvm-enable-autoactivation b/bin/pve-osd-lvm-enable-autoactivation > new file mode 100644 > index 00000000..acdc91e8 > --- /dev/null > +++ b/bin/pve-osd-lvm-enable-autoactivation > @@ -0,0 +1,195 @@ > +#!/usr/bin/perl > + > +use v5.36; > + > +use JSON qw(decode_json); > + > +use PVE::Tools; > + > +my sub ceph_volume_lvm_osd_info : prototype() () { > + my $cmd = [ > + "/usr/sbin/ceph-volume", "lvm", "list", "--format", "json", > + ]; > + > + my $stdout = ''; > + my $outfunc = sub($line) { > + $stdout .= "$line\n"; > + }; > + > + PVE::Tools::run_command($cmd, timeout => 10, outfunc => $outfunc); > + my $osd_info = decode_json($stdout); > + > + return $osd_info; > +} > + > +my sub lvs_from_osd_info : prototype($) ($osd_info) { > + my @lvs_for_osds = (); > + > + for my $osd (keys $osd_info->%*) { > + my $osd_lvs = $osd_info->{$osd}; > + > + for my $osd_lv ($osd_lvs->@*) { > + my ($lv_name, $vg_name) = $osd_lv->@{qw(lv_name vg_name)}; > + push(@lvs_for_osds, "$vg_name/$lv_name"); > + } > + } > + > + return \@lvs_for_osds; > +} > + > +my sub lvs : prototype() () { > + my $cmd = [ > + "/usr/sbin/lvs", > + "--noheadings", > + "--separator", > + ":", > + "--options", > + "lv_name,vg_name,autoactivation,active", > + ]; > + > + my $all_lvs = {}; > + > + my $outfunc = sub($line) { > + $line = PVE::Tools::trim($line); > + > + my ($lv_name, $vg_name, $autoactivation, $active) = split(':', $line, -1); > + > + return undef if ($lv_name eq '' || $vg_name eq ''); > + > + $all_lvs->{"$vg_name/$lv_name"} = { > + autoactivation => $autoactivation, > + active => $active, > + }; > + }; > + > + PVE::Tools::run_command( > + $cmd, > + timeout => 10, > + outfunc => $outfunc, > + ); > + > + return $all_lvs; > +} > + > +my sub main : prototype() () { > + my $osd_info = ceph_volume_lvm_osd_info(); > + my $all_lvs = lvs(); > + > + my @osd_lvs_no_autoactivation = (); > + my @osd_lvs_inactive = (); > + > + for my $osd (keys $osd_info->%*) { > + for my $osd_lv ($osd_info->{$osd}->@*) { > + my ($lv_name, $vg_name) = $osd_lv->@{qw(lv_name vg_name)}; > + > + my $osd_lv = "$vg_name/$lv_name"; > + > + push(@osd_lvs_no_autoactivation, $osd_lv) if !$all_lvs->{$osd_lv}->{autoactivation}; > + push(@osd_lvs_inactive, $osd_lv) if !$all_lvs->{$osd_lv}->{active}; > + } > + } > + > + my $has_set_autoactivation_err = 0; > + > + # Logical volumes are formatted as "vg_name/lv_name", which is necessary for lvchange > + for my $lv (@osd_lvs_no_autoactivation) { > + print("Enabling autoactivation for OSD logical volume '$lv' ...\n"); > + > + eval { > + my $cmd = [ > + '/usr/sbin/lvchange', '--setautoactivation', 'y', $lv, > + ]; > + > + PVE::Tools::run_command( > + $cmd, > + quiet => 1, > + timeout => 10, > + ); > + }; > + if (my $err = $@) { > + $has_set_autoactivation_err = 1; > + > + warn("Error: Failed to enable autoactivation for OSD LV '$lv'\n"); > + warn("$@\n"); > + > + next; > + } > + > + } > + > + my $has_activation_err = 0; > + > + # Activate any inactive OSD LVs so that ceph-volume can later bring up any failed OSDs > + for my $lv (@osd_lvs_inactive) { > + print("Activating OSD logical volume '$lv' ...\n"); > + > + eval { > + my $cmd = [ > + '/usr/sbin/lvchange', '--activate', 'y', $lv, > + ]; > + > + PVE::Tools::run_command( > + $cmd, > + quiet => 1, > + timeout => 10, > + ); > + }; > + if (my $err = $@) { > + $has_activation_err = 1; > + > + warn("Error: Failed to activate OSD LV '$lv'\n"); > + warn("$@\n"); > + > + next; > + } > + } > + > + # ceph-volume requires all LVs used by OSDs to be active, > + # so exit in case there are any we couldn't activate > + if ($has_set_autoactivation_err || $has_activation_err) { > + if ($has_set_autoactivation_err) { > + warn("Couldn't enable autoactivation for all OSD LVs.\n"); > + } > + > + if ($has_activation_err) { > + warn("Couldn't activate all OSD LVs.\n"); > + } > + > + exit 1; > + } > + > + print("Activating OSDs ...\n"); > + eval { > + my $cmd = [ > + "/usr/sbin/ceph-volume", "lvm", "activate", "--all", > + ]; > + > + # ceph-volume prints everything to stderr for whatever reason and prefixes > + # what actually goes to stderr with " stderr:", so separate the output here. > + # Lines starting with "Running command:" are removed because they're overly > + # verbose. > + my $logfunc = sub ($line) { > + return if $line =~ m/^Running command:/; > + > + if ($line =~ s/^\s*stderr:\s*//) { > + print STDERR "$line\n"; > + } else { > + print STDOUT "$line\n"; > + } > + }; > + > + PVE::Tools::run_command( > + $cmd, > + logfunc => $logfunc, > + timeout => 30, > + ); > + }; > + if (my $err = $@) { > + warn("Error while activating all Ceph LVM volumes: $@\n"); > + exit 1; > + } > + > + return undef; > +} > + > +main(); > diff --git a/debian/postinst b/debian/postinst > index b6e07fd9..f550e7bb 100755 > --- a/debian/postinst > +++ b/debian/postinst > @@ -133,6 +133,18 @@ migrate_apt_auth_conf() { > fi > } > > +ceph_osd_lvm_enable_autoactivation() { > + if ! test -e /usr/sbin/ceph-volume; then > + return > + fi > + > + if ! /usr/share/pve-manager/helpers/pve-osd-lvm-enable-autoactivation; then > + printf "\nEnabling autoactivation for logical volumes used by Ceph OSDs failed."; > + printf " Check the output above for errors and try to enable autoactivation for OSD LVs"; > + printf " manually by running '/usr/share/pve-manager/helpers/pve-osd-lvm-enable-autoactivation'"; > + fi > +} > + > # Copied from dh_installtmpfiles/13.24.2 > if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then > if [ -x "$(command -v systemd-tmpfiles)" ]; then > @@ -246,6 +258,10 @@ case "$1" in > fi > fi > > + if test -n "$2" && dpkg --compare-versions "$2" 'lt' '9.0.5'; then > + ceph_osd_lvm_enable_autoactivation > + fi > + > ;; > > abort-upgrade|abort-remove|abort-deconfigure) > -- > 2.47.2 > > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > > _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [pve-devel] [PATCH pve-manager master v1 2/2] fix #6652: d/postinst: enable autoactivation for Ceph OSD LVs 2025-08-13 7:52 ` Fabian Grünbichler @ 2025-08-13 8:50 ` Max R. Carrara 2025-08-13 9:01 ` Friedrich Weber 2025-08-13 9:02 ` Fabian Grünbichler 0 siblings, 2 replies; 12+ messages in thread From: Max R. Carrara @ 2025-08-13 8:50 UTC (permalink / raw) To: Proxmox VE development discussion On Wed Aug 13, 2025 at 9:52 AM CEST, Fabian Grünbichler wrote: > On August 12, 2025 6:46 pm, Max R. Carrara wrote: > > Introduce a new helper command pve-osd-lvm-enable-autoactivation, > > which gracefully tries to enable autoactivation for all logical > > volumes used by Ceph OSDs while also activating any LVs that aren't > > active yet. Afterwards, the helper attempts to bring all OSDs online. > > I think this is probably overkill - this only affects a specific non > standard setup, the breakage is really obvious, and the fix is easy: > either run lvchange on all those LVs, or recreate the OSDs after the fix > for creation is rolled out.. > > i.e., the fallout from some edge cases not being handled correctly in > the 200 line helper script here is probably worse than the few setups > that run into the original issue that we can easily help along > manually.. I mean, this script doesn't really do much, and the LVs themselves are fetched via `ceph-volume` ... But then again, you're probably right that it might just break somebody else's arcane setup somewhere. As an alternative, I wouldn't mind writing something for the release notes' known issues section (or some other place). Assuming a standard setup*, all that the user would have to do is identical to what the script does, so nothing too complicated. (*OSDs with WAL + DB on disks / partitions without anything else inbetween) > > > > > Fixes: #6652 > > Signed-off-by: Max R. Carrara <m.carrara@proxmox.com> > > --- > > bin/Makefile | 3 +- > > bin/pve-osd-lvm-enable-autoactivation | 195 ++++++++++++++++++++++++++ > > debian/postinst | 16 +++ > > 3 files changed, 213 insertions(+), 1 deletion(-) > > create mode 100644 bin/pve-osd-lvm-enable-autoactivation > > > > diff --git a/bin/Makefile b/bin/Makefile > > index 777e6759..0a0df34d 100644 > > --- a/bin/Makefile > > +++ b/bin/Makefile > > @@ -32,7 +32,8 @@ HELPERS = \ > > pve-startall-delay \ > > pve-init-ceph-crash \ > > pve-firewall-commit \ > > - pve-sdn-commit > > + pve-sdn-commit \ > > + pve-osd-lvm-enable-autoactivation > > > > MIGRATIONS = \ > > pve-lvm-disable-autoactivation \ > > diff --git a/bin/pve-osd-lvm-enable-autoactivation b/bin/pve-osd-lvm-enable-autoactivation > > new file mode 100644 > > index 00000000..acdc91e8 > > --- /dev/null > > +++ b/bin/pve-osd-lvm-enable-autoactivation > > @@ -0,0 +1,195 @@ > > +#!/usr/bin/perl > > + > > +use v5.36; > > + > > +use JSON qw(decode_json); > > + > > +use PVE::Tools; > > + > > +my sub ceph_volume_lvm_osd_info : prototype() () { > > + my $cmd = [ > > + "/usr/sbin/ceph-volume", "lvm", "list", "--format", "json", > > + ]; > > + > > + my $stdout = ''; > > + my $outfunc = sub($line) { > > + $stdout .= "$line\n"; > > + }; > > + > > + PVE::Tools::run_command($cmd, timeout => 10, outfunc => $outfunc); > > + my $osd_info = decode_json($stdout); > > + > > + return $osd_info; > > +} > > + > > +my sub lvs_from_osd_info : prototype($) ($osd_info) { > > + my @lvs_for_osds = (); > > + > > + for my $osd (keys $osd_info->%*) { > > + my $osd_lvs = $osd_info->{$osd}; > > + > > + for my $osd_lv ($osd_lvs->@*) { > > + my ($lv_name, $vg_name) = $osd_lv->@{qw(lv_name vg_name)}; > > + push(@lvs_for_osds, "$vg_name/$lv_name"); > > + } > > + } > > + > > + return \@lvs_for_osds; > > +} > > + > > +my sub lvs : prototype() () { > > + my $cmd = [ > > + "/usr/sbin/lvs", > > + "--noheadings", > > + "--separator", > > + ":", > > + "--options", > > + "lv_name,vg_name,autoactivation,active", > > + ]; > > + > > + my $all_lvs = {}; > > + > > + my $outfunc = sub($line) { > > + $line = PVE::Tools::trim($line); > > + > > + my ($lv_name, $vg_name, $autoactivation, $active) = split(':', $line, -1); > > + > > + return undef if ($lv_name eq '' || $vg_name eq ''); > > + > > + $all_lvs->{"$vg_name/$lv_name"} = { > > + autoactivation => $autoactivation, > > + active => $active, > > + }; > > + }; > > + > > + PVE::Tools::run_command( > > + $cmd, > > + timeout => 10, > > + outfunc => $outfunc, > > + ); > > + > > + return $all_lvs; > > +} > > + > > +my sub main : prototype() () { > > + my $osd_info = ceph_volume_lvm_osd_info(); > > + my $all_lvs = lvs(); > > + > > + my @osd_lvs_no_autoactivation = (); > > + my @osd_lvs_inactive = (); > > + > > + for my $osd (keys $osd_info->%*) { > > + for my $osd_lv ($osd_info->{$osd}->@*) { > > + my ($lv_name, $vg_name) = $osd_lv->@{qw(lv_name vg_name)}; > > + > > + my $osd_lv = "$vg_name/$lv_name"; > > + > > + push(@osd_lvs_no_autoactivation, $osd_lv) if !$all_lvs->{$osd_lv}->{autoactivation}; > > + push(@osd_lvs_inactive, $osd_lv) if !$all_lvs->{$osd_lv}->{active}; > > + } > > + } > > + > > + my $has_set_autoactivation_err = 0; > > + > > + # Logical volumes are formatted as "vg_name/lv_name", which is necessary for lvchange > > + for my $lv (@osd_lvs_no_autoactivation) { > > + print("Enabling autoactivation for OSD logical volume '$lv' ...\n"); > > + > > + eval { > > + my $cmd = [ > > + '/usr/sbin/lvchange', '--setautoactivation', 'y', $lv, > > + ]; > > + > > + PVE::Tools::run_command( > > + $cmd, > > + quiet => 1, > > + timeout => 10, > > + ); > > + }; > > + if (my $err = $@) { > > + $has_set_autoactivation_err = 1; > > + > > + warn("Error: Failed to enable autoactivation for OSD LV '$lv'\n"); > > + warn("$@\n"); > > + > > + next; > > + } > > + > > + } > > + > > + my $has_activation_err = 0; > > + > > + # Activate any inactive OSD LVs so that ceph-volume can later bring up any failed OSDs > > + for my $lv (@osd_lvs_inactive) { > > + print("Activating OSD logical volume '$lv' ...\n"); > > + > > + eval { > > + my $cmd = [ > > + '/usr/sbin/lvchange', '--activate', 'y', $lv, > > + ]; > > + > > + PVE::Tools::run_command( > > + $cmd, > > + quiet => 1, > > + timeout => 10, > > + ); > > + }; > > + if (my $err = $@) { > > + $has_activation_err = 1; > > + > > + warn("Error: Failed to activate OSD LV '$lv'\n"); > > + warn("$@\n"); > > + > > + next; > > + } > > + } > > + > > + # ceph-volume requires all LVs used by OSDs to be active, > > + # so exit in case there are any we couldn't activate > > + if ($has_set_autoactivation_err || $has_activation_err) { > > + if ($has_set_autoactivation_err) { > > + warn("Couldn't enable autoactivation for all OSD LVs.\n"); > > + } > > + > > + if ($has_activation_err) { > > + warn("Couldn't activate all OSD LVs.\n"); > > + } > > + > > + exit 1; > > + } > > + > > + print("Activating OSDs ...\n"); > > + eval { > > + my $cmd = [ > > + "/usr/sbin/ceph-volume", "lvm", "activate", "--all", > > + ]; > > + > > + # ceph-volume prints everything to stderr for whatever reason and prefixes > > + # what actually goes to stderr with " stderr:", so separate the output here. > > + # Lines starting with "Running command:" are removed because they're overly > > + # verbose. > > + my $logfunc = sub ($line) { > > + return if $line =~ m/^Running command:/; > > + > > + if ($line =~ s/^\s*stderr:\s*//) { > > + print STDERR "$line\n"; > > + } else { > > + print STDOUT "$line\n"; > > + } > > + }; > > + > > + PVE::Tools::run_command( > > + $cmd, > > + logfunc => $logfunc, > > + timeout => 30, > > + ); > > + }; > > + if (my $err = $@) { > > + warn("Error while activating all Ceph LVM volumes: $@\n"); > > + exit 1; > > + } > > + > > + return undef; > > +} > > + > > +main(); > > diff --git a/debian/postinst b/debian/postinst > > index b6e07fd9..f550e7bb 100755 > > --- a/debian/postinst > > +++ b/debian/postinst > > @@ -133,6 +133,18 @@ migrate_apt_auth_conf() { > > fi > > } > > > > +ceph_osd_lvm_enable_autoactivation() { > > + if ! test -e /usr/sbin/ceph-volume; then > > + return > > + fi > > + > > + if ! /usr/share/pve-manager/helpers/pve-osd-lvm-enable-autoactivation; then > > + printf "\nEnabling autoactivation for logical volumes used by Ceph OSDs failed."; > > + printf " Check the output above for errors and try to enable autoactivation for OSD LVs"; > > + printf " manually by running '/usr/share/pve-manager/helpers/pve-osd-lvm-enable-autoactivation'"; > > + fi > > +} > > + > > # Copied from dh_installtmpfiles/13.24.2 > > if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then > > if [ -x "$(command -v systemd-tmpfiles)" ]; then > > @@ -246,6 +258,10 @@ case "$1" in > > fi > > fi > > > > + if test -n "$2" && dpkg --compare-versions "$2" 'lt' '9.0.5'; then > > + ceph_osd_lvm_enable_autoactivation > > + fi > > + > > ;; > > > > abort-upgrade|abort-remove|abort-deconfigure) > > -- > > 2.47.2 > > > > > > > > _______________________________________________ > > pve-devel mailing list > > pve-devel@lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > > > > > > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [pve-devel] [PATCH pve-manager master v1 2/2] fix #6652: d/postinst: enable autoactivation for Ceph OSD LVs 2025-08-13 8:50 ` Max R. Carrara @ 2025-08-13 9:01 ` Friedrich Weber 2025-08-13 9:02 ` Fabian Grünbichler 1 sibling, 0 replies; 12+ messages in thread From: Friedrich Weber @ 2025-08-13 9:01 UTC (permalink / raw) To: Proxmox VE development discussion, Max R. Carrara Thanks for tackling this issue! On 13/08/2025 10:50, Max R. Carrara wrote: > On Wed Aug 13, 2025 at 9:52 AM CEST, Fabian Grünbichler wrote: >> On August 12, 2025 6:46 pm, Max R. Carrara wrote: >>> Introduce a new helper command pve-osd-lvm-enable-autoactivation, >>> which gracefully tries to enable autoactivation for all logical >>> volumes used by Ceph OSDs while also activating any LVs that aren't >>> active yet. Afterwards, the helper attempts to bring all OSDs online. >> >> I think this is probably overkill - this only affects a specific non >> standard setup, the breakage is really obvious, and the fix is easy: >> either run lvchange on all those LVs, or recreate the OSDs after the fix >> for creation is rolled out.. >> >> i.e., the fallout from some edge cases not being handled correctly in >> the 200 line helper script here is probably worse than the few setups >> that run into the original issue that we can easily help along >> manually.. > > I mean, this script doesn't really do much, and the LVs themselves are > fetched via `ceph-volume` ... But then again, you're probably right that > it might just break somebody else's arcane setup somewhere. > > As an alternative, I wouldn't mind writing something for the release > notes' known issues section (or some other place). Assuming a standard > setup*, all that the user would have to do is identical to what the > script does, so nothing too complicated. Yes, I think the "Known issues" section of the release notes is a good start (with an one-liner that re-enables autoactivation for all affected LVs). _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [pve-devel] [PATCH pve-manager master v1 2/2] fix #6652: d/postinst: enable autoactivation for Ceph OSD LVs 2025-08-13 8:50 ` Max R. Carrara 2025-08-13 9:01 ` Friedrich Weber @ 2025-08-13 9:02 ` Fabian Grünbichler 2025-08-13 9:40 ` Max R. Carrara 1 sibling, 1 reply; 12+ messages in thread From: Fabian Grünbichler @ 2025-08-13 9:02 UTC (permalink / raw) To: Proxmox VE development discussion On August 13, 2025 10:50 am, Max R. Carrara wrote: > On Wed Aug 13, 2025 at 9:52 AM CEST, Fabian Grünbichler wrote: >> On August 12, 2025 6:46 pm, Max R. Carrara wrote: >> > Introduce a new helper command pve-osd-lvm-enable-autoactivation, >> > which gracefully tries to enable autoactivation for all logical >> > volumes used by Ceph OSDs while also activating any LVs that aren't >> > active yet. Afterwards, the helper attempts to bring all OSDs online. >> >> I think this is probably overkill - this only affects a specific non >> standard setup, the breakage is really obvious, and the fix is easy: >> either run lvchange on all those LVs, or recreate the OSDs after the fix >> for creation is rolled out.. >> >> i.e., the fallout from some edge cases not being handled correctly in >> the 200 line helper script here is probably worse than the few setups >> that run into the original issue that we can easily help along >> manually.. > > I mean, this script doesn't really do much, and the LVs themselves are > fetched via `ceph-volume` ... But then again, you're probably right that > it might just break somebody else's arcane setup somewhere. > > As an alternative, I wouldn't mind writing something for the release > notes' known issues section (or some other place). Assuming a standard > setup*, all that the user would have to do is identical to what the > script does, so nothing too complicated. but the known issue will be gone, except for the handful of users that ran into it before the fix was rolled out.. this is not something you noticed 5 months later? > (*OSDs with WAL + DB on disks / partitions without anything else inbetween) I am not worried about the script breaking things, but about it printing spurious errors/warnings for unaffected setups. _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [pve-devel] [PATCH pve-manager master v1 2/2] fix #6652: d/postinst: enable autoactivation for Ceph OSD LVs 2025-08-13 9:02 ` Fabian Grünbichler @ 2025-08-13 9:40 ` Max R. Carrara 2025-08-13 10:05 ` Fabian Grünbichler 0 siblings, 1 reply; 12+ messages in thread From: Max R. Carrara @ 2025-08-13 9:40 UTC (permalink / raw) To: Proxmox VE development discussion On Wed Aug 13, 2025 at 11:02 AM CEST, Fabian Grünbichler wrote: > On August 13, 2025 10:50 am, Max R. Carrara wrote: > > On Wed Aug 13, 2025 at 9:52 AM CEST, Fabian Grünbichler wrote: > >> On August 12, 2025 6:46 pm, Max R. Carrara wrote: > >> > Introduce a new helper command pve-osd-lvm-enable-autoactivation, > >> > which gracefully tries to enable autoactivation for all logical > >> > volumes used by Ceph OSDs while also activating any LVs that aren't > >> > active yet. Afterwards, the helper attempts to bring all OSDs online. > >> > >> I think this is probably overkill - this only affects a specific non > >> standard setup, the breakage is really obvious, and the fix is easy: > >> either run lvchange on all those LVs, or recreate the OSDs after the fix > >> for creation is rolled out.. > >> > >> i.e., the fallout from some edge cases not being handled correctly in > >> the 200 line helper script here is probably worse than the few setups > >> that run into the original issue that we can easily help along > >> manually.. > > > > I mean, this script doesn't really do much, and the LVs themselves are > > fetched via `ceph-volume` ... But then again, you're probably right that > > it might just break somebody else's arcane setup somewhere. > > > > As an alternative, I wouldn't mind writing something for the release > > notes' known issues section (or some other place). Assuming a standard > > setup*, all that the user would have to do is identical to what the > > script does, so nothing too complicated. > > but the known issue will be gone, except for the handful of users that > ran into it before the fix was rolled out.. this is not something you > noticed 5 months later? Are you sure, though? There might very well be setups out there that are only rebooted every couple months or so--not everybody is as diligent with their maintenance, unfortunately. We don't really know how common / rare it is to set up a DB / WAL disk for OSDs. > > > (*OSDs with WAL + DB on disks / partitions without anything else inbetween) > > I am not worried about the script breaking things, but about it printing > spurious errors/warnings for unaffected setups. Well, unaffected here would mean that all OSD LVs have autoactivation enabled (and are also activated). Additionally, if the OSDs are already up, `ceph-volume` doesn't do anything either. FWIW we could suppress LVM "No medium found" warnings in the LVM commands the script runs, like we do in pve-storage [0]. Additionally, we could also short-circuit and silently exit early if no changes are necessary; e.g. if autoactivation is enabled for all OSD LVs, we can assume that they're activated as well, and that the OSDs themselves are therefore running, too. So, I would really prefer having either (an improved version of) the script, or some documentation regarding this *somewhere*, just to not leave any users in the dark. [0]: https://git.proxmox.com/?p=pve-storage.git;a=blob;f=src/PVE/Storage/LVMPlugin.pm;h=0416c9e02a1d8255c940d2cd9f5e0111b784fe7c;hb=refs/heads/master#l21 > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [pve-devel] [PATCH pve-manager master v1 2/2] fix #6652: d/postinst: enable autoactivation for Ceph OSD LVs 2025-08-13 9:40 ` Max R. Carrara @ 2025-08-13 10:05 ` Fabian Grünbichler 0 siblings, 0 replies; 12+ messages in thread From: Fabian Grünbichler @ 2025-08-13 10:05 UTC (permalink / raw) To: Proxmox VE development discussion On August 13, 2025 11:40 am, Max R. Carrara wrote: > On Wed Aug 13, 2025 at 11:02 AM CEST, Fabian Grünbichler wrote: >> On August 13, 2025 10:50 am, Max R. Carrara wrote: >> > On Wed Aug 13, 2025 at 9:52 AM CEST, Fabian Grünbichler wrote: >> >> On August 12, 2025 6:46 pm, Max R. Carrara wrote: >> >> > Introduce a new helper command pve-osd-lvm-enable-autoactivation, >> >> > which gracefully tries to enable autoactivation for all logical >> >> > volumes used by Ceph OSDs while also activating any LVs that aren't >> >> > active yet. Afterwards, the helper attempts to bring all OSDs online. >> >> >> >> I think this is probably overkill - this only affects a specific non >> >> standard setup, the breakage is really obvious, and the fix is easy: >> >> either run lvchange on all those LVs, or recreate the OSDs after the fix >> >> for creation is rolled out.. >> >> >> >> i.e., the fallout from some edge cases not being handled correctly in >> >> the 200 line helper script here is probably worse than the few setups >> >> that run into the original issue that we can easily help along >> >> manually.. >> > >> > I mean, this script doesn't really do much, and the LVs themselves are >> > fetched via `ceph-volume` ... But then again, you're probably right that >> > it might just break somebody else's arcane setup somewhere. >> > >> > As an alternative, I wouldn't mind writing something for the release >> > notes' known issues section (or some other place). Assuming a standard >> > setup*, all that the user would have to do is identical to what the >> > script does, so nothing too complicated. >> >> but the known issue will be gone, except for the handful of users that >> ran into it before the fix was rolled out.. this is not something you >> noticed 5 months later? > > Are you sure, though? There might very well be setups out there that are > only rebooted every couple months or so--not everybody is as diligent > with their maintenance, unfortunately. We don't really know how common / > rare it is to set up a DB / WAL disk for OSDs. I am not opposed to adding it to the known issues, I just think it's not a very common issue to run into in the first place. >> > (*OSDs with WAL + DB on disks / partitions without anything else inbetween) >> >> I am not worried about the script breaking things, but about it printing >> spurious errors/warnings for unaffected setups. > > Well, unaffected here would mean that all OSD LVs have autoactivation > enabled (and are also activated). Additionally, if the OSDs are already > up, `ceph-volume` doesn't do anything either. no, unaffected here means 99% of the systems. and as soon as you run commands on any system, there will be a small percentage where those commands fail in noisy fashion for whatever reason. so the tradeoff needs to be there - if the issue is big enough and the commands are not very involved, then yes, fixing in postinst like this is a good idea. for an issue that will affect almost nobody, and that requires running commands with lots of storage interaction(!), that tradeoff looks rather different.. > FWIW we could suppress LVM "No medium found" warnings in the LVM > commands the script runs, like we do in pve-storage [0]. Additionally, > we could also short-circuit and silently exit early if no changes are > necessary; e.g. if autoactivation is enabled for all OSD LVs, we can > assume that they're activated as well, and that the OSDs themselves are > therefore running, too. > > So, I would really prefer having either (an improved version of) the > script, or some documentation regarding this *somewhere*, just to not > leave any users in the dark. > > [0]: https://git.proxmox.com/?p=pve-storage.git;a=blob;f=src/PVE/Storage/LVMPlugin.pm;h=0416c9e02a1d8255c940d2cd9f5e0111b784fe7c;hb=refs/heads/master#l21 > >> >> >> _______________________________________________ >> pve-devel mailing list >> pve-devel@lists.proxmox.com >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
* [pve-devel] superseded: [PATCH pve-manager master v1 0/2] Fix #6652: LVM Autoactivation Missing for Ceph OSD LVs 2025-08-12 16:46 [pve-devel] [PATCH pve-manager master v1 0/2] Fix #6652: LVM Autoactivation Missing for Ceph OSD LVs Max R. Carrara 2025-08-12 16:46 ` [pve-devel] [PATCH pve-manager master v1 1/2] fix #6652: ceph: osd: enable autoactivation for OSD LVs on creation Max R. Carrara 2025-08-12 16:46 ` [pve-devel] [PATCH pve-manager master v1 2/2] fix #6652: d/postinst: enable autoactivation for Ceph OSD LVs Max R. Carrara @ 2025-08-13 13:43 ` Max R. Carrara 2 siblings, 0 replies; 12+ messages in thread From: Max R. Carrara @ 2025-08-13 13:43 UTC (permalink / raw) To: Proxmox VE development discussion On Tue Aug 12, 2025 at 6:46 PM CEST, Max R. Carrara wrote: > Fix #6652: LVM Autoactivation Missing for Ceph OSD LVs - v1 > =========================================================== superseded-by: https://lore.proxmox.com/pve-devel/20250813134028.292213-1-m.carrara@proxmox.com/T _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2025-08-13 13:42 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2025-08-12 16:46 [pve-devel] [PATCH pve-manager master v1 0/2] Fix #6652: LVM Autoactivation Missing for Ceph OSD LVs Max R. Carrara 2025-08-12 16:46 ` [pve-devel] [PATCH pve-manager master v1 1/2] fix #6652: ceph: osd: enable autoactivation for OSD LVs on creation Max R. Carrara 2025-08-13 7:48 ` Fabian Grünbichler 2025-08-13 8:28 ` Max R. Carrara 2025-08-12 16:46 ` [pve-devel] [PATCH pve-manager master v1 2/2] fix #6652: d/postinst: enable autoactivation for Ceph OSD LVs Max R. Carrara 2025-08-13 7:52 ` Fabian Grünbichler 2025-08-13 8:50 ` Max R. Carrara 2025-08-13 9:01 ` Friedrich Weber 2025-08-13 9:02 ` Fabian Grünbichler 2025-08-13 9:40 ` Max R. Carrara 2025-08-13 10:05 ` Fabian Grünbichler 2025-08-13 13:43 ` [pve-devel] superseded: [PATCH pve-manager master v1 0/2] Fix #6652: LVM Autoactivation Missing " Max R. Carrara
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.