* [pve-devel] [PATCH pve-manager v2 1/6] ceph: tools: add helper sub for creating or updating keyring files
2025-12-22 14:18 [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2 Max R. Carrara
@ 2025-12-22 14:18 ` Max R. Carrara
2025-12-22 14:19 ` [pve-devel] [PATCH pve-manager v2 2/6] fix #6816: api: ceph: create 'client.exporter' w/ keyring Max R. Carrara
` (6 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2025-12-22 14:18 UTC (permalink / raw)
To: pve-devel
Add `create_or_update_keyring_file()`, a more generic version of
`create_or_update_crash_keyring_file()`, in order to avoid duplicating
the underlying logic for other kinds of keyrings / Ceph auth entities.
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
PVE/Ceph/Tools.pm | 80 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 80 insertions(+)
diff --git a/PVE/Ceph/Tools.pm b/PVE/Ceph/Tools.pm
index f50d2272..dce9156a 100644
--- a/PVE/Ceph/Tools.pm
+++ b/PVE/Ceph/Tools.pm
@@ -3,6 +3,7 @@ package PVE::Ceph::Tools;
use strict;
use warnings;
+use Carp qw(croak);
use File::Path;
use File::Basename;
use IO::File;
@@ -455,6 +456,85 @@ sub get_or_create_admin_keyring {
return $pve_ckeyring_path;
}
+=head3 create_or_update_keyring_file($dest_file, $entity, $caps [, $rados])
+
+Creates or updates a keyring file C<$dest_file> for C<$entity>. If the
+C<$entity> is created, it gains the capabilities provided with C<$caps>.
+Otherwise, capabilities are not updated.
+
+B<NOTE:> The caller is responsible for ensuring that the provided C<$dest_file>
+is in fact for the given C<$entity>.
+
+Returns C<1> if C<$dest_file> was created or updated, C<0> otherwise.
+
+=over
+
+=item * C<$dest_file>
+
+The path of the keyring file, for example C</etc/pve/ceph/ceph.client.crash.keyring>.
+
+=item * C<$entity>
+
+The entity for which to create the authentication entry and corresponding
+keyring, for example C<client.crash>. If the entity already exists, its
+capabilities are not updated.
+
+=item * C<$caps>
+
+The capabilities to set when creating C<$entity>, for example:
+
+ my $caps = [
+ mgr => 'allow profile osd',
+ mon => 'allow profile osd',
+ osd => 'allow *',
+ ];
+
+=item * C<$rados> (optional)
+
+An existing C<L<PVE::RADOS>> object. If not provided, a new object will be
+created instead.
+
+=back
+
+For an explanation on Ceph capabilities, see:
+L<https://docs.ceph.com/en/latest/rados/operations/user-management/#authorization-capabilities>
+
+=cut
+
+my sub create_or_update_keyring_file {
+ my ($dest_file, $entity, $caps, $rados) = @_;
+
+ croak '$dest_file is undef' if !defined($dest_file);
+ croak '$entity is undef' if !defined($entity);
+ croak '$caps is undef' if !defined($caps);
+
+ $rados = PVE::RADOS->new() if !defined($rados);
+
+ my $output = $rados->mon_command({
+ prefix => 'auth get-or-create',
+ entity => "$entity",
+ caps => $caps,
+ format => 'plain',
+ });
+
+ if (-f $dest_file) {
+ my $contents = PVE::Tools::file_get_contents($dest_file);
+
+ if ($contents ne $output) {
+ PVE::Tools::file_set_contents($dest_file, $output);
+ return 1;
+ }
+
+ return 0;
+
+ } else {
+ PVE::Tools::file_set_contents($dest_file, $output);
+ return 1;
+ }
+
+ return 0;
+}
+
# is also used in `pve-init-ceph-crash` helper
sub create_or_update_crash_keyring_file {
my ($rados) = @_;
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 10+ messages in thread* [pve-devel] [PATCH pve-manager v2 2/6] fix #6816: api: ceph: create 'client.exporter' w/ keyring
2025-12-22 14:18 [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2 Max R. Carrara
2025-12-22 14:18 ` [pve-devel] [PATCH pve-manager v2 1/6] ceph: tools: add helper sub for creating or updating keyring files Max R. Carrara
@ 2025-12-22 14:19 ` Max R. Carrara
2025-12-22 14:19 ` [pve-devel] [PATCH pve-manager v2 3/6] fix #6816: bin: add pve-ceph-keyring helper and call it in postinst Max R. Carrara
` (5 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2025-12-22 14:19 UTC (permalink / raw)
To: pve-devel
... when creating the first MON.
Similar to `ceph-crash.service` [0], create or update the keyring for
the `ceph-exporter` daemon using a new Ceph auth entity called
`client.exporter` when creating the first MON.
Its keyring is placed at `/etc/pve/ceph/ceph.client.exporter.keyring`.
The `ceph-exporter` daemon only needs read access to MONs. Found this
out during testing; this isn't explicitly documented anywhere AFAIK.
In case the daemon requires more capabilities in the future, I
recommend updating them via a separate helper that wraps the `ceph
auth caps` command via `PVE::RADOS`. However, because all this daemon
does is expose a metrics endpoint `http://$HOST:9926/metrics`, I doubt
that it needs any additional capabilities any time soon.
[0]: https://lore.proxmox.com/pve-devel/20240402145523.683008-11-m.carrara@proxmox.com/
Fixes: #6816
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
PVE/API2/Ceph/MON.pm | 9 +++++++++
PVE/Ceph/Tools.pm | 15 +++++++++++++++
2 files changed, 24 insertions(+)
diff --git a/PVE/API2/Ceph/MON.pm b/PVE/API2/Ceph/MON.pm
index 70fc158d..422f107a 100644
--- a/PVE/API2/Ceph/MON.pm
+++ b/PVE/API2/Ceph/MON.pm
@@ -480,6 +480,15 @@ __PACKAGE__->register_method({
cfs_write_file('ceph.conf', $cfg);
};
warn "Unable to configure keyring for ceph-crash.service: $@" if $@;
+
+ print "Configuring keyring for ceph-exporter.service\n";
+ eval {
+ PVE::Ceph::Tools::create_or_update_exporter_keyring_file();
+ $cfg->{'client.exporter'}->{keyring} =
+ '/etc/pve/ceph/$cluster.$name.keyring';
+ cfs_write_file('ceph.conf', $cfg);
+ };
+ warn "Unable to configure keyring for ceph-exporter.service: $@" if $@;
}
eval { PVE::Ceph::Services::ceph_service_cmd('enable', $monsection) };
diff --git a/PVE/Ceph/Tools.pm b/PVE/Ceph/Tools.pm
index dce9156a..8ddce759 100644
--- a/PVE/Ceph/Tools.pm
+++ b/PVE/Ceph/Tools.pm
@@ -22,6 +22,7 @@ my $ceph_cfgpath = "$ceph_cfgdir/$ccname.conf";
my $pve_ceph_cfgdir = "/etc/pve/ceph";
my $pve_ceph_crash_key_path = "$pve_ceph_cfgdir/$ccname.client.crash.keyring";
+my $pve_ceph_exporter_key_path = "$pve_ceph_cfgdir/$ccname.client.exporter.keyring";
my $pve_mon_key_path = "/etc/pve/priv/$ccname.mon.keyring";
my $pve_ckeyring_path = "/etc/pve/priv/$ccname.client.admin.keyring";
my $ckeyring_path = "/etc/ceph/ceph.client.admin.keyring";
@@ -48,6 +49,7 @@ my $config_values = {
my $config_files = {
pve_ceph_cfgpath => $pve_ceph_cfgpath,
pve_ceph_crash_key_path => $pve_ceph_crash_key_path,
+ pve_ceph_exporter_key_path => $pve_ceph_exporter_key_path,
pve_mon_key_path => $pve_mon_key_path,
pve_ckeyring_path => $pve_ckeyring_path,
ceph_bootstrap_osd_keyring => $ceph_bootstrap_osd_keyring,
@@ -568,6 +570,19 @@ sub create_or_update_crash_keyring_file {
return 0;
}
+sub create_or_update_exporter_keyring_file {
+ my ($rados) = @_;
+
+ my $entity = 'client.exporter';
+ my $caps = [
+ mon => 'allow r',
+ ];
+
+ return create_or_update_keyring_file(
+ $pve_ceph_exporter_key_path, $entity, $caps, $rados,
+ );
+}
+
# get ceph-volume managed osds
sub ceph_volume_list {
my $result = {};
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 10+ messages in thread* [pve-devel] [PATCH pve-manager v2 3/6] fix #6816: bin: add pve-ceph-keyring helper and call it in postinst
2025-12-22 14:18 [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2 Max R. Carrara
2025-12-22 14:18 ` [pve-devel] [PATCH pve-manager v2 1/6] ceph: tools: add helper sub for creating or updating keyring files Max R. Carrara
2025-12-22 14:19 ` [pve-devel] [PATCH pve-manager v2 2/6] fix #6816: api: ceph: create 'client.exporter' w/ keyring Max R. Carrara
@ 2025-12-22 14:19 ` Max R. Carrara
2025-12-22 14:19 ` [pve-devel] [PATCH pve-manager v2 4/6] ceph: tools: simplify helper sub for crash keyring file Max R. Carrara
` (4 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2025-12-22 14:19 UTC (permalink / raw)
To: pve-devel
... in order to update the configuration for the 'client.exporter'
Ceph auth entity, its keyring file and the corresponding config
section in /etc/pve/ceph.conf for existing Ceph installations.
The `pve-ceph-keyring` helper is a more generic version of the
`pve-init-ceph-crash` script.
The core logic [0] is retained, while the code organization and
(error) messages are improved a little.
Only the custom 'client.exporter' Ceph auth entity is supported at the
moment. Other auth entities can be added declaratively.
The helper is intentionally designed to be a little more open to
changes in case additional keyring files need to be added in the
future.
The main motivation for adding this helper is to avoid duplicating any
logic for additional Ceph auth entities and keyring files, especially
because the main purpose of this helper is to be called in
`debian/postinst`.
[0]: https://lore.proxmox.com/pve-devel/20240402145523.683008-11-m.carrara@proxmox.com/
Fixes: #6816
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
PVE/Ceph/Tools.pm | 1 +
bin/Makefile | 1 +
bin/pve-ceph-keyring | 282 +++++++++++++++++++++++++++++++++++++++++++
debian/postinst | 24 ++++
4 files changed, 308 insertions(+)
create mode 100755 bin/pve-ceph-keyring
diff --git a/PVE/Ceph/Tools.pm b/PVE/Ceph/Tools.pm
index 8ddce759..afff1a0a 100644
--- a/PVE/Ceph/Tools.pm
+++ b/PVE/Ceph/Tools.pm
@@ -570,6 +570,7 @@ sub create_or_update_crash_keyring_file {
return 0;
}
+# is also used in `pve-ceph-keyring` helper
sub create_or_update_exporter_keyring_file {
my ($rados) = @_;
diff --git a/bin/Makefile b/bin/Makefile
index 777e6759..dd4c53b5 100644
--- a/bin/Makefile
+++ b/bin/Makefile
@@ -31,6 +31,7 @@ SCRIPTS = \
HELPERS = \
pve-startall-delay \
pve-init-ceph-crash \
+ pve-ceph-keyring \
pve-firewall-commit \
pve-sdn-commit
diff --git a/bin/pve-ceph-keyring b/bin/pve-ceph-keyring
new file mode 100755
index 00000000..9d6e89d4
--- /dev/null
+++ b/bin/pve-ceph-keyring
@@ -0,0 +1,282 @@
+#!/usr/bin/perl
+
+use v5.36;
+
+use Getopt::Long qw(GetOptions);
+use List::Util qw(first max);
+
+use PVE::Ceph::Tools;
+use PVE::Cluster;
+use PVE::RADOS;
+use PVE::RPCEnvironment;
+
+my $CEPH_CFG_FILE = 'ceph.conf';
+my $CEPH_CFG_FILE_PATH = PVE::Ceph::Tools::get_config('pve_ceph_cfgpath');
+my $KEYRING_PATH = '/etc/pve/ceph/$cluster.$name.keyring';
+
+my $SUPPORTED_ENTITIES = {
+ 'client.exporter' => {
+ 'keyring-func' => \&PVE::Ceph::Tools::create_or_update_exporter_keyring_file,
+ 'keyring-path' => PVE::Ceph::Tools::get_config('pve_ceph_exporter_key_path'),
+ },
+};
+
+my sub parse_cli_opts : prototype() () {
+ my $init = '';
+ my $needs_help = '';
+
+ GetOptions(
+ 'init!' => \$init,
+ 'help|h!' => \$needs_help,
+ );
+
+ return {
+ init => $init,
+ 'needs-help' => $needs_help,
+ entities => [@ARGV],
+ };
+}
+
+my sub print_usage : prototype() () {
+ my @cmd_path_elements = split('/', $0);
+ my $cmd = $cmd_path_elements[-1] || $0;
+
+ my $supported_entities = join("\n ", sort keys $SUPPORTED_ENTITIES->%*);
+
+ my $actions = [
+ ["--init", "Initialize or update Ceph auth entities and keyrings"],
+ ];
+
+ my $opts_misc = [
+ ["--help, -h", "Print help"],
+ ];
+
+ my $ljust = max map { length($_->[0]) } ($actions->@*, $opts_misc->@*);
+
+ my $format_opts = sub ($opt_pairs) {
+ my @opts_formatted = map { sprintf('%-*s %s', $ljust, $_->[0], $_->[1]) } $opt_pairs->@*;
+ return join("\n ", @opts_formatted);
+ };
+
+ my $actions_str = $format_opts->($actions);
+ my $opts_misc_str = $format_opts->($opts_misc);
+
+ my $usage = <<~EOF;
+ $cmd - Ceph authentication entity and keyring helper
+
+ USAGE:
+ $cmd <ACTION> [OPTIONS] <CEPH AUTH ENTITY ...>
+
+ SUPPORTED ENTITIES:
+ $supported_entities
+
+ ACTIONS:
+ $actions_str
+
+ OTHER OPTIONS:
+ $opts_misc_str
+ EOF
+
+ print STDERR $usage;
+
+ return;
+}
+
+my sub print_usage_short : prototype() () {
+ my @cmd_path_elements = split('/', $0);
+ my $cmd = $cmd_path_elements[-1] || $0;
+
+ my $usage = <<~EOF;
+ USAGE:
+ $cmd <ACTION> [OPTIONS] <CEPH AUTH ENTITY ...>
+
+ Use -h or --help for more details.
+ EOF
+
+ print STDERR $usage;
+
+ return;
+}
+
+my sub try_adapt_cfg : prototype($$) ($cfg, $entity) {
+ my $removed_key = 0;
+
+ print("$entity: Checking configuration.\n");
+
+ my $add_keyring = sub () {
+ print("$entity: Setting keyring path to '$KEYRING_PATH'.\n");
+ $cfg->{$entity}->{keyring} = $KEYRING_PATH;
+ return;
+ };
+
+ if (!exists($cfg->{$entity})) {
+ print("$entity: Adding missing section.\n");
+ $add_keyring->();
+ return 1;
+ }
+
+ if (exists($cfg->{$entity}->{key})) {
+ print("$entity: Removing existing usage of key.\n");
+ delete($cfg->{$entity}->{key});
+ $removed_key = 1;
+ }
+
+ if (!exists($cfg->{$entity}->{keyring})) {
+ print("$entity: Keyring path is missing from configuration.\n");
+ $add_keyring->();
+ return 1;
+ }
+
+ my $current_keyring_value = $cfg->{$entity}->{keyring};
+ if ($current_keyring_value ne $KEYRING_PATH) {
+ print("$entity: Current keyring path differs from expected path.\n");
+ $add_keyring->();
+ return 1;
+ }
+
+ return $removed_key;
+}
+
+my sub update_cluster_cfg : prototype($$$) ($cfg, $entities, $rados) {
+ if (!defined($rados)) {
+ my $has_mon_host = defined($cfg->{global}) && defined($cfg->{global}->{mon_host});
+
+ if ($has_mon_host && $cfg->{global}->{mon_host} ne '') {
+ die "Connection to RADOS failed even though a monitor is configured.\n"
+ . "Please verify whether your configuration in '$CEPH_CFG_FILE' is correct.\n";
+ }
+
+ print("Connection to RADOS failed and no monitor is configured in '$CEPH_CFG_FILE'.\n"
+ . "Assuming that things are fine. No action required.\n");
+ return;
+ }
+
+ my @changed_entities;
+
+ for my $entity ($entities->@*) {
+ my ($keyring_func, $keyring_path) =
+ $SUPPORTED_ENTITIES->{$entity}->@{qw(keyring-func keyring-path)};
+
+ my $has_updated_keyring = eval { $keyring_func->() };
+ die "Failed to configure keyring for Ceph auth entity '$entity': $@"
+ if $@;
+
+ print("$entity: Keyring file '$keyring_path' was updated.\n")
+ if $has_updated_keyring;
+
+ my $changed = try_adapt_cfg($cfg, $entity);
+ push(@changed_entities, $entity) if $changed;
+ }
+
+ return @changed_entities;
+}
+
+sub main : prototype() () {
+ my $opts = parse_cli_opts();
+
+ if ($opts->{'needs-help'}) {
+ print_usage();
+ exit 0;
+ }
+
+ if (!scalar($opts->{entities}->@*)) {
+ print STDERR "Error: No Ceph auth entities specified.\n\n";
+ print_usage_short();
+
+ exit 1;
+ }
+
+ my @unknown_entities = grep { !exists($SUPPORTED_ENTITIES->{$_}) } $opts->{entities}->@*;
+
+ if (scalar(@unknown_entities)) {
+ my $err_entities = join(', ', @unknown_entities);
+
+ print STDERR "Error: Unknown entities: $err_entities\n\n";
+ print_usage_short();
+
+ exit 1;
+ }
+
+ if (!$opts->{init}) {
+ print STDERR "Error: No action specified.\n\n";
+ print_usage_short();
+
+ exit 1;
+ }
+
+ # PVE::RADOS expects an active RPC Environment because it forks itself
+ # and may want to clean up after
+ my $rpcenv = PVE::RPCEnvironment->setup_default_cli_env();
+
+ if (!PVE::Ceph::Tools::check_ceph_installed('ceph_bin', 1)) {
+ print("Ceph is not installed. No action required.\n");
+ exit 0;
+ }
+
+ my $ceph_cfg_path = PVE::Ceph::Tools::get_config('pve_ceph_cfgpath');
+ if (PVE::Ceph::Tools::check_ceph_installed('ceph_mon', 1) && -f $ceph_cfg_path) {
+ my $pve_ceph_cfgdir = PVE::Ceph::Tools::get_config('pve_ceph_cfgdir');
+ if (!-d $pve_ceph_cfgdir) {
+ File::Path::make_path($pve_ceph_cfgdir);
+ }
+ }
+
+ eval { PVE::Ceph::Tools::check_ceph_inited(); };
+ if ($@) {
+ print("Ceph is not initialized. No action required.\n");
+ exit 0;
+ }
+
+ my $rados = eval { PVE::RADOS->new() };
+ my $inner_err = '';
+
+ my $rval = PVE::Cluster::cfs_lock_file(
+ $CEPH_CFG_FILE,
+ undef,
+ sub {
+ eval {
+ my $cfg = PVE::Cluster::cfs_read_file($CEPH_CFG_FILE);
+
+ my @changed_entities = update_cluster_cfg($cfg, $opts->{entities}, $rados);
+
+ if (scalar(@changed_entities)) {
+ print("Committing updated configuration to '$CEPH_CFG_FILE'.\n");
+ PVE::Cluster::cfs_write_file($CEPH_CFG_FILE, $cfg);
+ print(
+ "Successfully updated configuration for the following Ceph auth entities: ",
+ join(', ', @changed_entities),
+ "\n",
+ );
+ } else {
+ print(
+ "Configuration in '$CEPH_CFG_FILE_PATH' does not need to be updated.\n"
+ );
+ }
+ };
+ $inner_err = $@;
+
+ return 1;
+ },
+ );
+
+ # cfs_lock_file sets $@ explicitly to undef
+ my $err = $@ // '';
+
+ my $has_err = !defined($rval) || $inner_err || $err;
+
+ if ($has_err) {
+ $err =~ s/\n*$//;
+ $inner_err =~ s/\n*$//;
+
+ if (!defined($rval)) {
+ warn("Error while acquiring or releasing lock for '$CEPH_CFG_FILE_PATH'.\n");
+ warn("Error: $err\n") if $err ne '';
+ }
+
+ warn("Error: $inner_err\n") if $inner_err ne '';
+
+ exit 1;
+ }
+}
+
+main();
diff --git a/debian/postinst b/debian/postinst
index 924e078d..2bf6752f 100755
--- a/debian/postinst
+++ b/debian/postinst
@@ -103,6 +103,26 @@ update_ceph_conf() {
fi
}
+ceph_init_exporter_keyring() {
+ UNIT='ceph-exporter.service'
+
+ # Don't fail in case user has "exotic" configuration where RADOS
+ # isn't available on all nodes for some reason
+ /usr/share/pve-manager/helpers/pve-ceph-keyring --init client.exporter || true
+
+ if systemctl -q is-enabled "$UNIT" 2> /dev/null; then
+ # If the ceph-exporter package was installed previously, the unit will
+ # most likely have failed, unless the user adapted its configuration.
+ # Therefore check if it failed and reset it if it did, so that we can
+ # actually restart it below.
+ if systemctl -q is-failed "$UNIT" 2> /dev/null; then
+ systemctl reset-failed "$UNIT" || true
+ fi
+
+ deb-systemd-invoke restart "$UNIT" || true
+ fi
+}
+
# Copied from dh_installtmpfiles/13.24.2
if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" = "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then
if [ -x "$(command -v systemd-tmpfiles)" ]; then
@@ -183,6 +203,10 @@ case "$1" in
update_ceph_conf
fi
+ if test -n "$2" && dpkg --compare-versions "$2" 'lt' '9.1.5'; then
+ ceph_init_exporter_keyring
+ fi
+
if test ! -e /proxmox_install_mode; then
# modeled after code generated by dh_start
for unit in ${UNITS}; do
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 10+ messages in thread* [pve-devel] [PATCH pve-manager v2 4/6] ceph: tools: simplify helper sub for crash keyring file
2025-12-22 14:18 [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2 Max R. Carrara
` (2 preceding siblings ...)
2025-12-22 14:19 ` [pve-devel] [PATCH pve-manager v2 3/6] fix #6816: bin: add pve-ceph-keyring helper and call it in postinst Max R. Carrara
@ 2025-12-22 14:19 ` Max R. Carrara
2025-12-22 14:19 ` [pve-devel] [PATCH pve-manager v2 5/6] bin: make pve-init-ceph-crash call pve-ceph-keyring Max R. Carrara
` (3 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2025-12-22 14:19 UTC (permalink / raw)
To: pve-devel
... by using the new, more generic `create_or_update_keyring_file()`
helper sub.
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
PVE/Ceph/Tools.pm | 34 ++++++++--------------------------
1 file changed, 8 insertions(+), 26 deletions(-)
diff --git a/PVE/Ceph/Tools.pm b/PVE/Ceph/Tools.pm
index afff1a0a..b6430f79 100644
--- a/PVE/Ceph/Tools.pm
+++ b/PVE/Ceph/Tools.pm
@@ -541,33 +541,15 @@ my sub create_or_update_keyring_file {
sub create_or_update_crash_keyring_file {
my ($rados) = @_;
- if (!defined($rados)) {
- $rados = PVE::RADOS->new();
- }
+ my $entity = 'client.crash';
+ my $caps = [
+ mon => 'profile crash',
+ mgr => 'profile crash',
+ ];
- my $output = $rados->mon_command({
- prefix => 'auth get-or-create',
- entity => 'client.crash',
- caps => [
- mon => 'profile crash',
- mgr => 'profile crash',
- ],
- format => 'plain',
- });
-
- if (-f $pve_ceph_crash_key_path) {
- my $contents = PVE::Tools::file_get_contents($pve_ceph_crash_key_path);
-
- if ($contents ne $output) {
- PVE::Tools::file_set_contents($pve_ceph_crash_key_path, $output);
- return 1;
- }
- } else {
- PVE::Tools::file_set_contents($pve_ceph_crash_key_path, $output);
- return 1;
- }
-
- return 0;
+ return create_or_update_keyring_file(
+ $pve_ceph_crash_key_path, $entity, $caps, $rados,
+ );
}
# is also used in `pve-ceph-keyring` helper
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 10+ messages in thread* [pve-devel] [PATCH pve-manager v2 5/6] bin: make pve-init-ceph-crash call pve-ceph-keyring
2025-12-22 14:18 [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2 Max R. Carrara
` (3 preceding siblings ...)
2025-12-22 14:19 ` [pve-devel] [PATCH pve-manager v2 4/6] ceph: tools: simplify helper sub for crash keyring file Max R. Carrara
@ 2025-12-22 14:19 ` Max R. Carrara
2025-12-22 14:19 ` [pve-devel] [PATCH ceph v2 6/6] fix #6816: patches: make ceph-exporter use custom keyring Max R. Carrara
` (2 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2025-12-22 14:19 UTC (permalink / raw)
To: pve-devel
Support 'client.crash' in the new `pve-ceph-keyring` helper and let
`pve-init-ceph-crash` call `pve-ceph-keyring`.
This avoids having duplicate logic lying around and retains the
original behavior, except that the log output is slightly different.
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
PVE/Ceph/Tools.pm | 2 +-
bin/pve-ceph-keyring | 4 ++
bin/pve-init-ceph-crash | 152 +---------------------------------------
3 files changed, 7 insertions(+), 151 deletions(-)
diff --git a/PVE/Ceph/Tools.pm b/PVE/Ceph/Tools.pm
index b6430f79..dd146f6d 100644
--- a/PVE/Ceph/Tools.pm
+++ b/PVE/Ceph/Tools.pm
@@ -537,7 +537,7 @@ my sub create_or_update_keyring_file {
return 0;
}
-# is also used in `pve-init-ceph-crash` helper
+# is also used in `pve-ceph-keyring` helper
sub create_or_update_crash_keyring_file {
my ($rados) = @_;
diff --git a/bin/pve-ceph-keyring b/bin/pve-ceph-keyring
index 9d6e89d4..cf15776a 100755
--- a/bin/pve-ceph-keyring
+++ b/bin/pve-ceph-keyring
@@ -15,6 +15,10 @@ my $CEPH_CFG_FILE_PATH = PVE::Ceph::Tools::get_config('pve_ceph_cfgpath');
my $KEYRING_PATH = '/etc/pve/ceph/$cluster.$name.keyring';
my $SUPPORTED_ENTITIES = {
+ 'client.crash' => {
+ 'keyring-func' => \&PVE::Ceph::Tools::create_or_update_crash_keyring_file,
+ 'keyring-path' => PVE::Ceph::Tools::get_config('pve_ceph_crash_key_path'),
+ },
'client.exporter' => {
'keyring-func' => \&PVE::Ceph::Tools::create_or_update_exporter_keyring_file,
'keyring-path' => PVE::Ceph::Tools::get_config('pve_ceph_exporter_key_path'),
diff --git a/bin/pve-init-ceph-crash b/bin/pve-init-ceph-crash
index d25201d9..8945ed88 100755
--- a/bin/pve-init-ceph-crash
+++ b/bin/pve-init-ceph-crash
@@ -1,151 +1,3 @@
-#!/usr/bin/perl
+#!/bin/bash
-use strict;
-use warnings;
-
-use List::Util qw(first);
-
-use PVE::Ceph::Tools;
-use PVE::Cluster;
-use PVE::RADOS;
-use PVE::RPCEnvironment;
-
-my $ceph_cfg_file = 'ceph.conf';
-my $keyring_value = '/etc/pve/ceph/$cluster.$name.keyring';
-
-sub try_adapt_cfg {
- my ($cfg) = @_;
-
- my $entity = 'client.crash';
- my $removed_key = 0;
-
- print("Checking whether the configuration for '$entity' needs to be updated.\n");
-
- my $add_keyring = sub {
- print("Setting keyring path to '$keyring_value'.\n");
- $cfg->{$entity}->{keyring} = $keyring_value;
- };
-
- if (!exists($cfg->{$entity})) {
- print("Adding missing section for '$entity'.\n");
- $add_keyring->();
- return 1;
- }
-
- if (exists($cfg->{$entity}->{key})) {
- print("Removing existing usage of key.\n");
- delete($cfg->{$entity}->{key});
- $removed_key = 1;
- }
-
- if (!exists($cfg->{$entity}->{keyring})) {
- print("Keyring path is missing from configuration.\n");
- $add_keyring->();
- return 1;
- }
-
- my $current_keyring_value = $cfg->{$entity}->{keyring};
- if ($current_keyring_value ne $keyring_value) {
- print("Current keyring path differs from expected path.\n");
- $add_keyring->();
- return 1;
- }
-
- return $removed_key;
-}
-
-sub main {
- # PVE::RADOS expects an active RPC Environment because it forks itself
- # and may want to clean up after
- my $rpcenv = PVE::RPCEnvironment->setup_default_cli_env();
-
- if (!PVE::Ceph::Tools::check_ceph_installed('ceph_bin', 1)) {
- print("Ceph is not installed. No action required.\n");
- exit 0;
- }
-
- my $ceph_cfg_path = PVE::Ceph::Tools::get_config('pve_ceph_cfgpath');
- if (PVE::Ceph::Tools::check_ceph_installed('ceph_mon', 1) && -f $ceph_cfg_path) {
- my $pve_ceph_cfgdir = PVE::Ceph::Tools::get_config('pve_ceph_cfgdir');
- if (!-d $pve_ceph_cfgdir) {
- File::Path::make_path($pve_ceph_cfgdir);
- }
- }
-
- eval { PVE::Ceph::Tools::check_ceph_inited(); };
- if ($@) {
- print("Ceph is not initialized. No action required.\n");
- exit 0;
- }
-
- my $rados = eval { PVE::RADOS->new() };
- my $ceph_crash_key_path = PVE::Ceph::Tools::get_config('pve_ceph_crash_key_path');
-
- my $inner_err = '';
-
- my $rval = PVE::Cluster::cfs_lock_file(
- $ceph_cfg_file,
- undef,
- sub {
- eval {
- my $cfg = PVE::Cluster::cfs_read_file($ceph_cfg_file);
-
- if (!defined($rados)) {
- my $has_mon_host =
- defined($cfg->{global}) && defined($cfg->{global}->{mon_host});
- if ($has_mon_host && $cfg->{global}->{mon_host} ne '') {
- die "Connection to RADOS failed even though a monitor is configured.\n"
- . "Please verify whether your configuration in '$ceph_cfg_file' is correct.\n";
- }
-
- print(
- "Connection to RADOS failed and no monitor is configured in '$ceph_cfg_file'.\n"
- . "Assuming that things are fine. No action required.\n");
- return;
- }
-
- my $updated_keyring =
- PVE::Ceph::Tools::create_or_update_crash_keyring_file($rados);
-
- if ($updated_keyring) {
- print("Keyring file '$ceph_crash_key_path' was updated.\n");
- }
-
- my $changed = try_adapt_cfg($cfg);
-
- if ($changed) {
- print("Committing updated configuration to '$ceph_cfg_file'.\n");
- PVE::Cluster::cfs_write_file($ceph_cfg_file, $cfg);
- print("Successfully updated configuration for 'ceph-crash.service'.\n");
- } else {
- print("Configuration in '$ceph_cfg_file' does not need to be updated.\n");
- }
- };
- $inner_err = $@;
-
- return 1;
- },
- );
-
- # cfs_lock_file sets $@ explicitly to undef
- my $err = $@ // '';
-
- my $has_err = !defined($rval) || $inner_err || $err;
-
- if ($has_err) {
- $err =~ s/\n*$//;
- $inner_err =~ s/\n*$//;
-
- if (!defined($rval)) {
- warn("Error while acquiring or releasing lock for '$ceph_cfg_file'.\n");
- warn("Error: $err\n") if $err ne '';
- }
-
- warn("Failed to configure keyring for 'ceph-crash.service'.\nError: $inner_err\n")
- if $inner_err ne '';
-
- exit 1;
- }
-}
-
-main();
+/usr/share/pve-manager/helpers/pve-ceph-keyring --init client.crash
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 10+ messages in thread* [pve-devel] [PATCH ceph v2 6/6] fix #6816: patches: make ceph-exporter use custom keyring
2025-12-22 14:18 [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2 Max R. Carrara
` (4 preceding siblings ...)
2025-12-22 14:19 ` [pve-devel] [PATCH pve-manager v2 5/6] bin: make pve-init-ceph-crash call pve-ceph-keyring Max R. Carrara
@ 2025-12-22 14:19 ` Max R. Carrara
2025-12-23 12:43 ` [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2 Maximiliano Sandoval
2025-12-23 15:51 ` [pve-devel] superseded: " Max R. Carrara
7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2025-12-22 14:19 UTC (permalink / raw)
To: pve-devel
Add a patch that sets the CEPH_KEYRING env var for the ceph-exporter
daemon, telling it to use our custom keyring file. The patch also
sets the group of the daemon to `www-data` so that the daemon can
access the custom keyring.
Fixes: #6816
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
...orter-use-custom-keyring-and-set-gro.patch | 32 +++++++++++++++++++
patches/series | 1 +
2 files changed, 33 insertions(+)
create mode 100644 patches/0042-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch
diff --git a/patches/0042-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch b/patches/0042-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch
new file mode 100644
index 0000000000..f939b1d5af
--- /dev/null
+++ b/patches/0042-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch
@@ -0,0 +1,32 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: "Max R. Carrara" <m.carrara@proxmox.com>
+Date: Tue, 16 Sep 2025 16:34:51 +0200
+Subject: [PATCH] systemd: ceph-exporter: use custom keyring and set group to
+ www-data
+
+With the help of `Environment=`, set the CEPH_KEYRING env var to point
+to our custom keyring file location for the `ceph-exporter` daemon.
+
+Additionally, set the group of the `ceph-exporter` daemon to
+`www-data`, so that it can access this keyring.
+
+Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
+---
+ systemd/ceph-exporter.service.in | 3 ++-
+ 1 file changed, 2 insertions(+), 1 deletion(-)
+
+diff --git a/systemd/ceph-exporter.service.in b/systemd/ceph-exporter.service.in
+index f4f6d05c4b4..4e55939d201 100644
+--- a/systemd/ceph-exporter.service.in
++++ b/systemd/ceph-exporter.service.in
+@@ -6,8 +6,9 @@ Before=ceph.target
+ Wants=network-online.target local-fs.target ceph.target ceph-mon.target
+
+ [Service]
++Environment="CEPH_KEYRING=/etc/pve/ceph/ceph.client.exporter.keyring"
+ ExecReload=/bin/kill -HUP $MAINPID
+-ExecStart=@CMAKE_INSTALL_PREFIX@/bin/ceph-exporter -f --id %i --setuser ceph --setgroup ceph
++ExecStart=@CMAKE_INSTALL_PREFIX@/bin/ceph-exporter -f --id exporter --setuser ceph --setgroup www-data
+ LockPersonality=true
+ NoNewPrivileges=true
+ PrivateDevices=yes
diff --git a/patches/series b/patches/series
index d775a8a068..f5c7b9eafa 100644
--- a/patches/series
+++ b/patches/series
@@ -39,3 +39,4 @@
0039-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch
0040-pybind-rbd-disable-on_progress-callbacks-to-prevent-.patch
0041-systemd-services-fix-installing-ceph-volume.patch
+0042-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch
--
2.47.3
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2
2025-12-22 14:18 [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2 Max R. Carrara
` (5 preceding siblings ...)
2025-12-22 14:19 ` [pve-devel] [PATCH ceph v2 6/6] fix #6816: patches: make ceph-exporter use custom keyring Max R. Carrara
@ 2025-12-23 12:43 ` Maximiliano Sandoval
2025-12-23 15:51 ` Max R. Carrara
2025-12-23 15:51 ` [pve-devel] superseded: " Max R. Carrara
7 siblings, 1 reply; 10+ messages in thread
From: Maximiliano Sandoval @ 2025-12-23 12:43 UTC (permalink / raw)
To: Max R. Carrara; +Cc: pve-devel
"Max R. Carrara" <m.carrara@proxmox.com> writes:
> Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2
> =====================================================================
>
> tl;dr: Stop ceph-exporter.service from ending up in a crash loop by
> handing it a custom keyring file and setting its group to `www-data`,
> similar to what we did for ceph-crash.service [0] before.
>
> This is a refresh of a somewhat older series that has been rebased, with
> the version guard in `debian/postinst` adapted. The description from the
> previous version is provided here again for the reader's convenience.
>
> Currently, the `ceph-exporter` daemon ends up in a short startup crash
> loop before ultimately failing to start at all, because it tries to
> access the keyring file at `/etc/pve/priv/ceph.client.admin.keyring`,
> for which it doesn't have the permissions to do so.
>
> Instead of giving it access to the admin ring, give it its own keyring
> located at `/etc/pve/ceph/ceph.client.exporter.keyring`. This file and
> its corresponding section in `/etc/pve/ceph.conf` is created when the
> first MON is created via the API. If the cluster has already been set
> up, a postinst hook creates the keyring file and adapts
> `/etc/pve/ceph.conf` instead.
>
> The core logic of all of this was already added for `ceph-crash` a while
> ago [0] and is reused throughout the series, with some alterations to
> the original code in order to make it a little more generic.
I tested this series and it works as advertised modulo a race condition:
When the ceph-exporter unit is started before installing this series it
will fail and systemd will retry a handful of times, during this time
`systemctl is-failed ceph-exporter.service` returns 'activating' instead
of 'failed'. This might explain that then the reset-failed is never
called. This results in ceph-exporter being restarted as part of the
postinst script but failing because the reset-failed was never called
and there have been too many attempts already.
Otherwise, it works as expected. Thanks!
Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
--
Maximiliano
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2
2025-12-23 12:43 ` [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2 Maximiliano Sandoval
@ 2025-12-23 15:51 ` Max R. Carrara
0 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2025-12-23 15:51 UTC (permalink / raw)
To: Maximiliano Sandoval; +Cc: pve-devel
On Tue Dec 23, 2025 at 1:43 PM CET, Maximiliano Sandoval wrote:
> "Max R. Carrara" <m.carrara@proxmox.com> writes:
>
> > Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2
> > =====================================================================
> >
> > tl;dr: Stop ceph-exporter.service from ending up in a crash loop by
> > handing it a custom keyring file and setting its group to `www-data`,
> > similar to what we did for ceph-crash.service [0] before.
> >
> > This is a refresh of a somewhat older series that has been rebased, with
> > the version guard in `debian/postinst` adapted. The description from the
> > previous version is provided here again for the reader's convenience.
> >
> > Currently, the `ceph-exporter` daemon ends up in a short startup crash
> > loop before ultimately failing to start at all, because it tries to
> > access the keyring file at `/etc/pve/priv/ceph.client.admin.keyring`,
> > for which it doesn't have the permissions to do so.
> >
> > Instead of giving it access to the admin ring, give it its own keyring
> > located at `/etc/pve/ceph/ceph.client.exporter.keyring`. This file and
> > its corresponding section in `/etc/pve/ceph.conf` is created when the
> > first MON is created via the API. If the cluster has already been set
> > up, a postinst hook creates the keyring file and adapts
> > `/etc/pve/ceph.conf` instead.
> >
> > The core logic of all of this was already added for `ceph-crash` a while
> > ago [0] and is reused throughout the series, with some alterations to
> > the original code in order to make it a little more generic.
>
> I tested this series and it works as advertised modulo a race condition:
>
> When the ceph-exporter unit is started before installing this series it
> will fail and systemd will retry a handful of times, during this time
> `systemctl is-failed ceph-exporter.service` returns 'activating' instead
> of 'failed'. This might explain that then the reset-failed is never
> called. This results in ceph-exporter being restarted as part of the
> postinst script but failing because the reset-failed was never called
> and there have been too many attempts already.
>
> Otherwise, it works as expected. Thanks!
>
> Tested-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
Thanks a ton for testing this! That's a really good catch.
As discussed off-list, `ceph-exporter` won't be reset and restarted
anymore in debian/postinst. See v3 [0] for an update.
[0]: https://lore.proxmox.com/pve-devel/20251223153419.507507-1-m.carrara@proxmox.com/
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* [pve-devel] superseded: [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2
2025-12-22 14:18 [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2 Max R. Carrara
` (6 preceding siblings ...)
2025-12-23 12:43 ` [pve-devel] [PATCH pve-manager v2 0/6] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2 Maximiliano Sandoval
@ 2025-12-23 15:51 ` Max R. Carrara
7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2025-12-23 15:51 UTC (permalink / raw)
To: Proxmox VE development discussion
On Mon Dec 22, 2025 at 3:18 PM CET, Max R. Carrara wrote:
> Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v2
> =====================================================================
>
> tl;dr: Stop ceph-exporter.service from ending up in a crash loop by
> handing it a custom keyring file and setting its group to `www-data`,
> similar to what we did for ceph-crash.service [0] before.
>
> [...]
Superseded-by: https://lore.proxmox.com/pve-devel/20251223153419.507507-1-m.carrara@proxmox.com/
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 10+ messages in thread