public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup
@ 2026-06-23 13:44 Max R. Carrara
  2026-06-23 13:44 ` [PATCH pve-manager v4 1/7] ceph: tools: add helper sub for creating or updating keyring files Max R. Carrara
                   ` (7 more replies)
  0 siblings, 8 replies; 10+ messages in thread
From: Max R. Carrara @ 2026-06-23 13:44 UTC (permalink / raw)
  To: pve-devel

Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup - v4
=====================================================================

tl;dr: Stop ceph-exporter.service from ending up in a crash loop by
handing it a custom keyring file and setting its group to `www-data`,
similar to what we did for ceph-crash.service [0] before.

v2 contains the entire description; see below for a link.

Changes Since v3
----------------

- Rebase the patches
- Change the PVE version in the debian/postinst changes (patch #03)
  to 9.2.4
- Add patch for Ceph Squid (patch #07)

Tested this for both Ceph Tentacle and Ceph Squid, but a quick smoke test here
would be much appreciated nevertheless.

Previous Versions
-----------------

v1: https://lore.proxmox.com/pve-devel/20250916172012.739807-1-m.carrara@proxmox.com/
v2: https://lore.proxmox.com/pve-devel/20251222141907.400926-1-m.carrara@proxmox.com/
v3: https://lore.proxmox.com/pve-devel/20251223153419.507507-1-m.carrara@proxmox.com/

References
----------

[0]: https://lore.proxmox.com/pve-devel/20240402145523.683008-11-m.carrara@proxmox.com/

Summary of Changes
------------------


pve-manager:

Max R. Carrara (5):
  ceph: tools: add helper sub for creating or updating keyring files
  fix #6816: api: ceph: create 'client.exporter' w/ keyring
  fix #6816: bin: add pve-ceph-keyring helper and call it in postinst
  ceph: tools: simplify helper sub for crash keyring file
  bin: make pve-init-ceph-crash call pve-ceph-keyring

 PVE/API2/Ceph/MON.pm    |   9 ++
 PVE/Ceph/Tools.pm       | 108 ++++++++++++---
 bin/Makefile            |   1 +
 bin/pve-ceph-keyring    | 286 ++++++++++++++++++++++++++++++++++++++++
 bin/pve-init-ceph-crash | 152 +--------------------
 debian/postinst         |   6 +
 6 files changed, 397 insertions(+), 165 deletions(-)
 create mode 100755 bin/pve-ceph-keyring


ceph master:

Max R. Carrara (1):
  fix #6816: patches: make ceph-exporter use custom keyring

 ...orter-use-custom-keyring-and-set-gro.patch | 32 +++++++++++++++++++
 patches/series                                |  1 +
 2 files changed, 33 insertions(+)
 create mode 100644 patches/0014-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch


ceph squid:

Max R. Carrara (1):
  fix #6816: patches: make ceph-exporter use custom keyring

 ...orter-use-custom-keyring-and-set-gro.patch | 32 +++++++++++++++++++
 patches/series                                |  1 +
 2 files changed, 33 insertions(+)
 create mode 100644 patches/0053-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch


Summary over all repositories:
  10 files changed, 463 insertions(+), 165 deletions(-)

--
Generated by murpp 0.11.0




^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH pve-manager v4 1/7] ceph: tools: add helper sub for creating or updating keyring files
  2026-06-23 13:44 [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Max R. Carrara
@ 2026-06-23 13:44 ` Max R. Carrara
  2026-06-23 13:44 ` [PATCH pve-manager v4 2/7] fix #6816: api: ceph: create 'client.exporter' w/ keyring Max R. Carrara
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2026-06-23 13:44 UTC (permalink / raw)
  To: pve-devel

Add `create_or_update_keyring_file()`, a more generic version of
`create_or_update_crash_keyring_file()`, in order to avoid duplicating
the underlying logic for other kinds of keyrings / Ceph auth entities.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 PVE/Ceph/Tools.pm | 80 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)

diff --git a/PVE/Ceph/Tools.pm b/PVE/Ceph/Tools.pm
index 1eed2a2d..206921c1 100644
--- a/PVE/Ceph/Tools.pm
+++ b/PVE/Ceph/Tools.pm
@@ -3,6 +3,7 @@ package PVE::Ceph::Tools;
 use strict;
 use warnings;
 
+use Carp qw(croak);
 use File::Path;
 use File::Basename;
 use IO::File;
@@ -456,6 +457,85 @@ sub get_or_create_admin_keyring {
     return $pve_ckeyring_path;
 }
 
+=head3 create_or_update_keyring_file($dest_file, $entity, $caps [, $rados])
+
+Creates or updates a keyring file C<$dest_file> for C<$entity>. If the
+C<$entity> is created, it gains the capabilities provided with C<$caps>.
+Otherwise, capabilities are not updated.
+
+B<NOTE:> The caller is responsible for ensuring that the provided C<$dest_file>
+is in fact for the given C<$entity>.
+
+Returns C<1> if C<$dest_file> was created or updated, C<0> otherwise.
+
+=over
+
+=item * C<$dest_file>
+
+The path of the keyring file, for example C</etc/pve/ceph/ceph.client.crash.keyring>.
+
+=item * C<$entity>
+
+The entity for which to create the authentication entry and corresponding
+keyring, for example C<client.crash>. If the entity already exists, its
+capabilities are not updated.
+
+=item * C<$caps>
+
+The capabilities to set when creating C<$entity>, for example:
+
+    my $caps = [
+        mgr => 'allow profile osd',
+        mon => 'allow profile osd',
+        osd => 'allow *',
+    ];
+
+=item * C<$rados> (optional)
+
+An existing C<L<PVE::RADOS>> object. If not provided, a new object will be
+created instead.
+
+=back
+
+For an explanation on Ceph capabilities, see:
+L<https://docs.ceph.com/en/latest/rados/operations/user-management/#authorization-capabilities>
+
+=cut
+
+my sub create_or_update_keyring_file {
+    my ($dest_file, $entity, $caps, $rados) = @_;
+
+    croak '$dest_file is undef' if !defined($dest_file);
+    croak '$entity is undef' if !defined($entity);
+    croak '$caps is undef' if !defined($caps);
+
+    $rados = PVE::RADOS->new() if !defined($rados);
+
+    my $output = $rados->mon_command({
+        prefix => 'auth get-or-create',
+        entity => "$entity",
+        caps => $caps,
+        format => 'plain',
+    });
+
+    if (-f $dest_file) {
+        my $contents = PVE::Tools::file_get_contents($dest_file);
+
+        if ($contents ne $output) {
+            PVE::Tools::file_set_contents($dest_file, $output);
+            return 1;
+        }
+
+        return 0;
+
+    } else {
+        PVE::Tools::file_set_contents($dest_file, $output);
+        return 1;
+    }
+
+    return 0;
+}
+
 # is also used in `pve-init-ceph-crash` helper
 sub create_or_update_crash_keyring_file {
     my ($rados) = @_;
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [PATCH pve-manager v4 2/7] fix #6816: api: ceph: create 'client.exporter' w/ keyring
  2026-06-23 13:44 [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Max R. Carrara
  2026-06-23 13:44 ` [PATCH pve-manager v4 1/7] ceph: tools: add helper sub for creating or updating keyring files Max R. Carrara
@ 2026-06-23 13:44 ` Max R. Carrara
  2026-06-23 13:44 ` [PATCH pve-manager v4 3/7] fix #6816: bin: add pve-ceph-keyring helper and call it in postinst Max R. Carrara
                   ` (5 subsequent siblings)
  7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2026-06-23 13:44 UTC (permalink / raw)
  To: pve-devel

... when creating the first MON.

Similar to `ceph-crash.service` [0], create or update the keyring for
the `ceph-exporter` daemon using a new Ceph auth entity called
`client.exporter` when creating the first MON.

Its keyring is placed at `/etc/pve/ceph/ceph.client.exporter.keyring`.

The `ceph-exporter` daemon only needs read access to MONs. Found this
out during testing; this isn't explicitly documented anywhere AFAIK.

In case the daemon requires more capabilities in the future, I
recommend updating them via a separate helper that wraps the `ceph
auth caps` command via `PVE::RADOS`. However, because all this daemon
does is expose a metrics endpoint `http://$HOST:9926/metrics`, I doubt
that it needs any additional capabilities any time soon.

[0]: https://lore.proxmox.com/pve-devel/20240402145523.683008-11-m.carrara@proxmox.com/

Fixes: #6816
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 PVE/API2/Ceph/MON.pm |  9 +++++++++
 PVE/Ceph/Tools.pm    | 15 +++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/PVE/API2/Ceph/MON.pm b/PVE/API2/Ceph/MON.pm
index 68abd502..04cfc7f1 100644
--- a/PVE/API2/Ceph/MON.pm
+++ b/PVE/API2/Ceph/MON.pm
@@ -536,6 +536,15 @@ __PACKAGE__->register_method({
                             cfs_write_file('ceph.conf', $cfg);
                         };
                         warn "Unable to configure keyring for ceph-crash.service: $@" if $@;
+
+                        print "Configuring keyring for ceph-exporter.service\n";
+                        eval {
+                            PVE::Ceph::Tools::create_or_update_exporter_keyring_file();
+                            $cfg->{'client.exporter'}->{keyring} =
+                                '/etc/pve/ceph/$cluster.$name.keyring';
+                            cfs_write_file('ceph.conf', $cfg);
+                        };
+                        warn "Unable to configure keyring for ceph-exporter.service: $@" if $@;
                     }
 
                     eval { PVE::Ceph::Services::ceph_service_cmd('enable', $monsection) };
diff --git a/PVE/Ceph/Tools.pm b/PVE/Ceph/Tools.pm
index 206921c1..32c1a989 100644
--- a/PVE/Ceph/Tools.pm
+++ b/PVE/Ceph/Tools.pm
@@ -23,6 +23,7 @@ my $ceph_cfgpath = "$ceph_cfgdir/$ccname.conf";
 my $pve_ceph_cfgdir = "/etc/pve/ceph";
 
 my $pve_ceph_crash_key_path = "$pve_ceph_cfgdir/$ccname.client.crash.keyring";
+my $pve_ceph_exporter_key_path = "$pve_ceph_cfgdir/$ccname.client.exporter.keyring";
 my $pve_mon_key_path = "/etc/pve/priv/$ccname.mon.keyring";
 my $pve_ckeyring_path = "/etc/pve/priv/$ccname.client.admin.keyring";
 my $ckeyring_path = "/etc/ceph/ceph.client.admin.keyring";
@@ -49,6 +50,7 @@ my $config_values = {
 my $config_files = {
     pve_ceph_cfgpath => $pve_ceph_cfgpath,
     pve_ceph_crash_key_path => $pve_ceph_crash_key_path,
+    pve_ceph_exporter_key_path => $pve_ceph_exporter_key_path,
     pve_mon_key_path => $pve_mon_key_path,
     pve_ckeyring_path => $pve_ckeyring_path,
     ceph_bootstrap_osd_keyring => $ceph_bootstrap_osd_keyring,
@@ -569,6 +571,19 @@ sub create_or_update_crash_keyring_file {
     return 0;
 }
 
+sub create_or_update_exporter_keyring_file {
+    my ($rados) = @_;
+
+    my $entity = 'client.exporter';
+    my $caps = [
+        mon => 'allow r',
+    ];
+
+    return create_or_update_keyring_file(
+        $pve_ceph_exporter_key_path, $entity, $caps, $rados,
+    );
+}
+
 # get ceph-volume managed osds
 sub ceph_volume_list {
     my $result = {};
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [PATCH pve-manager v4 3/7] fix #6816: bin: add pve-ceph-keyring helper and call it in postinst
  2026-06-23 13:44 [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Max R. Carrara
  2026-06-23 13:44 ` [PATCH pve-manager v4 1/7] ceph: tools: add helper sub for creating or updating keyring files Max R. Carrara
  2026-06-23 13:44 ` [PATCH pve-manager v4 2/7] fix #6816: api: ceph: create 'client.exporter' w/ keyring Max R. Carrara
@ 2026-06-23 13:44 ` Max R. Carrara
  2026-06-23 13:44 ` [PATCH pve-manager v4 4/7] ceph: tools: simplify helper sub for crash keyring file Max R. Carrara
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2026-06-23 13:44 UTC (permalink / raw)
  To: pve-devel

... in order to update the configuration for the 'client.exporter'
Ceph auth entity, its keyring file and the corresponding config
section in /etc/pve/ceph.conf for existing Ceph installations.

The `pve-ceph-keyring` helper is a more generic version of the
`pve-init-ceph-crash` script.

The core logic [0] is retained, while the code organization and
(error) messages are improved a little.

Only the custom 'client.exporter' Ceph auth entity is supported at the
moment. Other auth entities can be added declaratively.

The helper is intentionally designed to be a little more open to
changes in case additional keyring files need to be added in the
future.

The main motivation for adding this helper is to avoid duplicating any
logic for additional Ceph auth entities and keyring files, especially
because the main purpose of this helper is to be called in
`debian/postinst`.

Note that `ceph-exporter.service` is not reset or restarted after the
invovation of `pve-ceph-keyring` in `debian/postinst`, thereby leaving
the choice of whether the service shall be restarted to admins.

[0]: https://lore.proxmox.com/pve-devel/20240402145523.683008-11-m.carrara@proxmox.com/

Fixes: #6816
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 PVE/Ceph/Tools.pm    |   1 +
 bin/Makefile         |   1 +
 bin/pve-ceph-keyring | 282 +++++++++++++++++++++++++++++++++++++++++++
 debian/postinst      |   6 +
 4 files changed, 290 insertions(+)
 create mode 100755 bin/pve-ceph-keyring

diff --git a/PVE/Ceph/Tools.pm b/PVE/Ceph/Tools.pm
index 32c1a989..5ea002e4 100644
--- a/PVE/Ceph/Tools.pm
+++ b/PVE/Ceph/Tools.pm
@@ -571,6 +571,7 @@ sub create_or_update_crash_keyring_file {
     return 0;
 }
 
+# is also used in `pve-ceph-keyring` helper
 sub create_or_update_exporter_keyring_file {
     my ($rados) = @_;
 
diff --git a/bin/Makefile b/bin/Makefile
index 5d225d17..aa685e18 100644
--- a/bin/Makefile
+++ b/bin/Makefile
@@ -31,6 +31,7 @@ SCRIPTS =  			\
 HELPERS =			\
 	pve-startall-delay	\
 	pve-init-ceph-crash	\
+	pve-ceph-keyring	\
 	pve-firewall-commit	\
 	pve-sdn-commit
 
diff --git a/bin/pve-ceph-keyring b/bin/pve-ceph-keyring
new file mode 100755
index 00000000..9d6e89d4
--- /dev/null
+++ b/bin/pve-ceph-keyring
@@ -0,0 +1,282 @@
+#!/usr/bin/perl
+
+use v5.36;
+
+use Getopt::Long qw(GetOptions);
+use List::Util qw(first max);
+
+use PVE::Ceph::Tools;
+use PVE::Cluster;
+use PVE::RADOS;
+use PVE::RPCEnvironment;
+
+my $CEPH_CFG_FILE = 'ceph.conf';
+my $CEPH_CFG_FILE_PATH = PVE::Ceph::Tools::get_config('pve_ceph_cfgpath');
+my $KEYRING_PATH = '/etc/pve/ceph/$cluster.$name.keyring';
+
+my $SUPPORTED_ENTITIES = {
+    'client.exporter' => {
+        'keyring-func' => \&PVE::Ceph::Tools::create_or_update_exporter_keyring_file,
+        'keyring-path' => PVE::Ceph::Tools::get_config('pve_ceph_exporter_key_path'),
+    },
+};
+
+my sub parse_cli_opts : prototype() () {
+    my $init = '';
+    my $needs_help = '';
+
+    GetOptions(
+        'init!' => \$init,
+        'help|h!' => \$needs_help,
+    );
+
+    return {
+        init => $init,
+        'needs-help' => $needs_help,
+        entities => [@ARGV],
+    };
+}
+
+my sub print_usage : prototype() () {
+    my @cmd_path_elements = split('/', $0);
+    my $cmd = $cmd_path_elements[-1] || $0;
+
+    my $supported_entities = join("\n  ", sort keys $SUPPORTED_ENTITIES->%*);
+
+    my $actions = [
+        ["--init", "Initialize or update Ceph auth entities and keyrings"],
+    ];
+
+    my $opts_misc = [
+        ["--help, -h", "Print help"],
+    ];
+
+    my $ljust = max map { length($_->[0]) } ($actions->@*, $opts_misc->@*);
+
+    my $format_opts = sub ($opt_pairs) {
+        my @opts_formatted = map { sprintf('%-*s    %s', $ljust, $_->[0], $_->[1]) } $opt_pairs->@*;
+        return join("\n  ", @opts_formatted);
+    };
+
+    my $actions_str = $format_opts->($actions);
+    my $opts_misc_str = $format_opts->($opts_misc);
+
+    my $usage = <<~EOF;
+    $cmd - Ceph authentication entity and keyring helper
+
+    USAGE:
+      $cmd <ACTION> [OPTIONS] <CEPH AUTH ENTITY ...>
+
+    SUPPORTED ENTITIES:
+      $supported_entities
+
+    ACTIONS:
+      $actions_str
+
+    OTHER OPTIONS:
+      $opts_misc_str
+    EOF
+
+    print STDERR $usage;
+
+    return;
+}
+
+my sub print_usage_short : prototype() () {
+    my @cmd_path_elements = split('/', $0);
+    my $cmd = $cmd_path_elements[-1] || $0;
+
+    my $usage = <<~EOF;
+    USAGE:
+      $cmd <ACTION> [OPTIONS] <CEPH AUTH ENTITY ...>
+
+    Use -h or --help for more details.
+    EOF
+
+    print STDERR $usage;
+
+    return;
+}
+
+my sub try_adapt_cfg : prototype($$) ($cfg, $entity) {
+    my $removed_key = 0;
+
+    print("$entity: Checking configuration.\n");
+
+    my $add_keyring = sub () {
+        print("$entity: Setting keyring path to '$KEYRING_PATH'.\n");
+        $cfg->{$entity}->{keyring} = $KEYRING_PATH;
+        return;
+    };
+
+    if (!exists($cfg->{$entity})) {
+        print("$entity: Adding missing section.\n");
+        $add_keyring->();
+        return 1;
+    }
+
+    if (exists($cfg->{$entity}->{key})) {
+        print("$entity: Removing existing usage of key.\n");
+        delete($cfg->{$entity}->{key});
+        $removed_key = 1;
+    }
+
+    if (!exists($cfg->{$entity}->{keyring})) {
+        print("$entity: Keyring path is missing from configuration.\n");
+        $add_keyring->();
+        return 1;
+    }
+
+    my $current_keyring_value = $cfg->{$entity}->{keyring};
+    if ($current_keyring_value ne $KEYRING_PATH) {
+        print("$entity: Current keyring path differs from expected path.\n");
+        $add_keyring->();
+        return 1;
+    }
+
+    return $removed_key;
+}
+
+my sub update_cluster_cfg : prototype($$$) ($cfg, $entities, $rados) {
+    if (!defined($rados)) {
+        my $has_mon_host = defined($cfg->{global}) && defined($cfg->{global}->{mon_host});
+
+        if ($has_mon_host && $cfg->{global}->{mon_host} ne '') {
+            die "Connection to RADOS failed even though a monitor is configured.\n"
+                . "Please verify whether your configuration in '$CEPH_CFG_FILE' is correct.\n";
+        }
+
+        print("Connection to RADOS failed and no monitor is configured in '$CEPH_CFG_FILE'.\n"
+            . "Assuming that things are fine. No action required.\n");
+        return;
+    }
+
+    my @changed_entities;
+
+    for my $entity ($entities->@*) {
+        my ($keyring_func, $keyring_path) =
+            $SUPPORTED_ENTITIES->{$entity}->@{qw(keyring-func keyring-path)};
+
+        my $has_updated_keyring = eval { $keyring_func->() };
+        die "Failed to configure keyring for Ceph auth entity '$entity': $@"
+            if $@;
+
+        print("$entity: Keyring file '$keyring_path' was updated.\n")
+            if $has_updated_keyring;
+
+        my $changed = try_adapt_cfg($cfg, $entity);
+        push(@changed_entities, $entity) if $changed;
+    }
+
+    return @changed_entities;
+}
+
+sub main : prototype() () {
+    my $opts = parse_cli_opts();
+
+    if ($opts->{'needs-help'}) {
+        print_usage();
+        exit 0;
+    }
+
+    if (!scalar($opts->{entities}->@*)) {
+        print STDERR "Error: No Ceph auth entities specified.\n\n";
+        print_usage_short();
+
+        exit 1;
+    }
+
+    my @unknown_entities = grep { !exists($SUPPORTED_ENTITIES->{$_}) } $opts->{entities}->@*;
+
+    if (scalar(@unknown_entities)) {
+        my $err_entities = join(', ', @unknown_entities);
+
+        print STDERR "Error: Unknown entities: $err_entities\n\n";
+        print_usage_short();
+
+        exit 1;
+    }
+
+    if (!$opts->{init}) {
+        print STDERR "Error: No action specified.\n\n";
+        print_usage_short();
+
+        exit 1;
+    }
+
+    # PVE::RADOS expects an active RPC Environment because it forks itself
+    # and may want to clean up after
+    my $rpcenv = PVE::RPCEnvironment->setup_default_cli_env();
+
+    if (!PVE::Ceph::Tools::check_ceph_installed('ceph_bin', 1)) {
+        print("Ceph is not installed. No action required.\n");
+        exit 0;
+    }
+
+    my $ceph_cfg_path = PVE::Ceph::Tools::get_config('pve_ceph_cfgpath');
+    if (PVE::Ceph::Tools::check_ceph_installed('ceph_mon', 1) && -f $ceph_cfg_path) {
+        my $pve_ceph_cfgdir = PVE::Ceph::Tools::get_config('pve_ceph_cfgdir');
+        if (!-d $pve_ceph_cfgdir) {
+            File::Path::make_path($pve_ceph_cfgdir);
+        }
+    }
+
+    eval { PVE::Ceph::Tools::check_ceph_inited(); };
+    if ($@) {
+        print("Ceph is not initialized. No action required.\n");
+        exit 0;
+    }
+
+    my $rados = eval { PVE::RADOS->new() };
+    my $inner_err = '';
+
+    my $rval = PVE::Cluster::cfs_lock_file(
+        $CEPH_CFG_FILE,
+        undef,
+        sub {
+            eval {
+                my $cfg = PVE::Cluster::cfs_read_file($CEPH_CFG_FILE);
+
+                my @changed_entities = update_cluster_cfg($cfg, $opts->{entities}, $rados);
+
+                if (scalar(@changed_entities)) {
+                    print("Committing updated configuration to '$CEPH_CFG_FILE'.\n");
+                    PVE::Cluster::cfs_write_file($CEPH_CFG_FILE, $cfg);
+                    print(
+                        "Successfully updated configuration for the following Ceph auth entities: ",
+                        join(', ', @changed_entities),
+                        "\n",
+                    );
+                } else {
+                    print(
+                        "Configuration in '$CEPH_CFG_FILE_PATH' does not need to be updated.\n"
+                    );
+                }
+            };
+            $inner_err = $@;
+
+            return 1;
+        },
+    );
+
+    # cfs_lock_file sets $@ explicitly to undef
+    my $err = $@ // '';
+
+    my $has_err = !defined($rval) || $inner_err || $err;
+
+    if ($has_err) {
+        $err =~ s/\n*$//;
+        $inner_err =~ s/\n*$//;
+
+        if (!defined($rval)) {
+            warn("Error while acquiring or releasing lock for '$CEPH_CFG_FILE_PATH'.\n");
+            warn("Error: $err\n") if $err ne '';
+        }
+
+        warn("Error: $inner_err\n") if $inner_err ne '';
+
+        exit 1;
+    }
+}
+
+main();
diff --git a/debian/postinst b/debian/postinst
index 924e078d..e8a6bb71 100755
--- a/debian/postinst
+++ b/debian/postinst
@@ -183,6 +183,12 @@ case "$1" in
         update_ceph_conf
     fi
 
+    if test -n "$2" && dpkg --compare-versions "$2" 'lt' '9.2.4'; then
+        # Don't fail in case user has "exotic" configuration where RADOS
+        # isn't available on all nodes for some reason
+        /usr/share/pve-manager/helpers/pve-ceph-keyring --init client.exporter || true
+    fi
+
     if test ! -e /proxmox_install_mode; then
         # modeled after code generated by dh_start
         for unit in ${UNITS}; do
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [PATCH pve-manager v4 4/7] ceph: tools: simplify helper sub for crash keyring file
  2026-06-23 13:44 [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Max R. Carrara
                   ` (2 preceding siblings ...)
  2026-06-23 13:44 ` [PATCH pve-manager v4 3/7] fix #6816: bin: add pve-ceph-keyring helper and call it in postinst Max R. Carrara
@ 2026-06-23 13:44 ` Max R. Carrara
  2026-06-23 13:44 ` [PATCH pve-manager v4 5/7] bin: make pve-init-ceph-crash call pve-ceph-keyring Max R. Carrara
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2026-06-23 13:44 UTC (permalink / raw)
  To: pve-devel

... by using the new, more generic `create_or_update_keyring_file()`
helper sub.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 PVE/Ceph/Tools.pm | 34 ++++++++--------------------------
 1 file changed, 8 insertions(+), 26 deletions(-)

diff --git a/PVE/Ceph/Tools.pm b/PVE/Ceph/Tools.pm
index 5ea002e4..07284661 100644
--- a/PVE/Ceph/Tools.pm
+++ b/PVE/Ceph/Tools.pm
@@ -542,33 +542,15 @@ my sub create_or_update_keyring_file {
 sub create_or_update_crash_keyring_file {
     my ($rados) = @_;
 
-    if (!defined($rados)) {
-        $rados = PVE::RADOS->new();
-    }
+    my $entity = 'client.crash';
+    my $caps = [
+        mon => 'profile crash',
+        mgr => 'profile crash',
+    ];
 
-    my $output = $rados->mon_command({
-        prefix => 'auth get-or-create',
-        entity => 'client.crash',
-        caps => [
-            mon => 'profile crash',
-            mgr => 'profile crash',
-        ],
-        format => 'plain',
-    });
-
-    if (-f $pve_ceph_crash_key_path) {
-        my $contents = PVE::Tools::file_get_contents($pve_ceph_crash_key_path);
-
-        if ($contents ne $output) {
-            PVE::Tools::file_set_contents($pve_ceph_crash_key_path, $output);
-            return 1;
-        }
-    } else {
-        PVE::Tools::file_set_contents($pve_ceph_crash_key_path, $output);
-        return 1;
-    }
-
-    return 0;
+    return create_or_update_keyring_file(
+        $pve_ceph_crash_key_path, $entity, $caps, $rados,
+    );
 }
 
 # is also used in `pve-ceph-keyring` helper
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [PATCH pve-manager v4 5/7] bin: make pve-init-ceph-crash call pve-ceph-keyring
  2026-06-23 13:44 [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Max R. Carrara
                   ` (3 preceding siblings ...)
  2026-06-23 13:44 ` [PATCH pve-manager v4 4/7] ceph: tools: simplify helper sub for crash keyring file Max R. Carrara
@ 2026-06-23 13:44 ` Max R. Carrara
  2026-06-23 13:44 ` [PATCH ceph master v4 6/7] fix #6816: patches: make ceph-exporter use custom keyring Max R. Carrara
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2026-06-23 13:44 UTC (permalink / raw)
  To: pve-devel

Support 'client.crash' in the new `pve-ceph-keyring` helper and let
`pve-init-ceph-crash` call `pve-ceph-keyring`.

This avoids having duplicate logic lying around and retains the
original behavior, except that the log output is slightly different.

Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 PVE/Ceph/Tools.pm       |   2 +-
 bin/pve-ceph-keyring    |   4 ++
 bin/pve-init-ceph-crash | 152 +---------------------------------------
 3 files changed, 7 insertions(+), 151 deletions(-)

diff --git a/PVE/Ceph/Tools.pm b/PVE/Ceph/Tools.pm
index 07284661..9a9f94c9 100644
--- a/PVE/Ceph/Tools.pm
+++ b/PVE/Ceph/Tools.pm
@@ -538,7 +538,7 @@ my sub create_or_update_keyring_file {
     return 0;
 }
 
-# is also used in `pve-init-ceph-crash` helper
+# is also used in `pve-ceph-keyring` helper
 sub create_or_update_crash_keyring_file {
     my ($rados) = @_;
 
diff --git a/bin/pve-ceph-keyring b/bin/pve-ceph-keyring
index 9d6e89d4..cf15776a 100755
--- a/bin/pve-ceph-keyring
+++ b/bin/pve-ceph-keyring
@@ -15,6 +15,10 @@ my $CEPH_CFG_FILE_PATH = PVE::Ceph::Tools::get_config('pve_ceph_cfgpath');
 my $KEYRING_PATH = '/etc/pve/ceph/$cluster.$name.keyring';
 
 my $SUPPORTED_ENTITIES = {
+    'client.crash' => {
+        'keyring-func' => \&PVE::Ceph::Tools::create_or_update_crash_keyring_file,
+        'keyring-path' => PVE::Ceph::Tools::get_config('pve_ceph_crash_key_path'),
+    },
     'client.exporter' => {
         'keyring-func' => \&PVE::Ceph::Tools::create_or_update_exporter_keyring_file,
         'keyring-path' => PVE::Ceph::Tools::get_config('pve_ceph_exporter_key_path'),
diff --git a/bin/pve-init-ceph-crash b/bin/pve-init-ceph-crash
index d25201d9..8945ed88 100755
--- a/bin/pve-init-ceph-crash
+++ b/bin/pve-init-ceph-crash
@@ -1,151 +1,3 @@
-#!/usr/bin/perl
+#!/bin/bash
 
-use strict;
-use warnings;
-
-use List::Util qw(first);
-
-use PVE::Ceph::Tools;
-use PVE::Cluster;
-use PVE::RADOS;
-use PVE::RPCEnvironment;
-
-my $ceph_cfg_file = 'ceph.conf';
-my $keyring_value = '/etc/pve/ceph/$cluster.$name.keyring';
-
-sub try_adapt_cfg {
-    my ($cfg) = @_;
-
-    my $entity = 'client.crash';
-    my $removed_key = 0;
-
-    print("Checking whether the configuration for '$entity' needs to be updated.\n");
-
-    my $add_keyring = sub {
-        print("Setting keyring path to '$keyring_value'.\n");
-        $cfg->{$entity}->{keyring} = $keyring_value;
-    };
-
-    if (!exists($cfg->{$entity})) {
-        print("Adding missing section for '$entity'.\n");
-        $add_keyring->();
-        return 1;
-    }
-
-    if (exists($cfg->{$entity}->{key})) {
-        print("Removing existing usage of key.\n");
-        delete($cfg->{$entity}->{key});
-        $removed_key = 1;
-    }
-
-    if (!exists($cfg->{$entity}->{keyring})) {
-        print("Keyring path is missing from configuration.\n");
-        $add_keyring->();
-        return 1;
-    }
-
-    my $current_keyring_value = $cfg->{$entity}->{keyring};
-    if ($current_keyring_value ne $keyring_value) {
-        print("Current keyring path differs from expected path.\n");
-        $add_keyring->();
-        return 1;
-    }
-
-    return $removed_key;
-}
-
-sub main {
-    # PVE::RADOS expects an active RPC Environment because it forks itself
-    # and may want to clean up after
-    my $rpcenv = PVE::RPCEnvironment->setup_default_cli_env();
-
-    if (!PVE::Ceph::Tools::check_ceph_installed('ceph_bin', 1)) {
-        print("Ceph is not installed. No action required.\n");
-        exit 0;
-    }
-
-    my $ceph_cfg_path = PVE::Ceph::Tools::get_config('pve_ceph_cfgpath');
-    if (PVE::Ceph::Tools::check_ceph_installed('ceph_mon', 1) && -f $ceph_cfg_path) {
-        my $pve_ceph_cfgdir = PVE::Ceph::Tools::get_config('pve_ceph_cfgdir');
-        if (!-d $pve_ceph_cfgdir) {
-            File::Path::make_path($pve_ceph_cfgdir);
-        }
-    }
-
-    eval { PVE::Ceph::Tools::check_ceph_inited(); };
-    if ($@) {
-        print("Ceph is not initialized. No action required.\n");
-        exit 0;
-    }
-
-    my $rados = eval { PVE::RADOS->new() };
-    my $ceph_crash_key_path = PVE::Ceph::Tools::get_config('pve_ceph_crash_key_path');
-
-    my $inner_err = '';
-
-    my $rval = PVE::Cluster::cfs_lock_file(
-        $ceph_cfg_file,
-        undef,
-        sub {
-            eval {
-                my $cfg = PVE::Cluster::cfs_read_file($ceph_cfg_file);
-
-                if (!defined($rados)) {
-                    my $has_mon_host =
-                        defined($cfg->{global}) && defined($cfg->{global}->{mon_host});
-                    if ($has_mon_host && $cfg->{global}->{mon_host} ne '') {
-                        die "Connection to RADOS failed even though a monitor is configured.\n"
-                            . "Please verify whether your configuration in '$ceph_cfg_file' is correct.\n";
-                    }
-
-                    print(
-                        "Connection to RADOS failed and no monitor is configured in '$ceph_cfg_file'.\n"
-                            . "Assuming that things are fine. No action required.\n");
-                    return;
-                }
-
-                my $updated_keyring =
-                    PVE::Ceph::Tools::create_or_update_crash_keyring_file($rados);
-
-                if ($updated_keyring) {
-                    print("Keyring file '$ceph_crash_key_path' was updated.\n");
-                }
-
-                my $changed = try_adapt_cfg($cfg);
-
-                if ($changed) {
-                    print("Committing updated configuration to '$ceph_cfg_file'.\n");
-                    PVE::Cluster::cfs_write_file($ceph_cfg_file, $cfg);
-                    print("Successfully updated configuration for 'ceph-crash.service'.\n");
-                } else {
-                    print("Configuration in '$ceph_cfg_file' does not need to be updated.\n");
-                }
-            };
-            $inner_err = $@;
-
-            return 1;
-        },
-    );
-
-    # cfs_lock_file sets $@ explicitly to undef
-    my $err = $@ // '';
-
-    my $has_err = !defined($rval) || $inner_err || $err;
-
-    if ($has_err) {
-        $err =~ s/\n*$//;
-        $inner_err =~ s/\n*$//;
-
-        if (!defined($rval)) {
-            warn("Error while acquiring or releasing lock for '$ceph_cfg_file'.\n");
-            warn("Error: $err\n") if $err ne '';
-        }
-
-        warn("Failed to configure keyring for 'ceph-crash.service'.\nError: $inner_err\n")
-            if $inner_err ne '';
-
-        exit 1;
-    }
-}
-
-main();
+/usr/share/pve-manager/helpers/pve-ceph-keyring --init client.crash
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [PATCH ceph master v4 6/7] fix #6816: patches: make ceph-exporter use custom keyring
  2026-06-23 13:44 [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Max R. Carrara
                   ` (4 preceding siblings ...)
  2026-06-23 13:44 ` [PATCH pve-manager v4 5/7] bin: make pve-init-ceph-crash call pve-ceph-keyring Max R. Carrara
@ 2026-06-23 13:44 ` Max R. Carrara
  2026-06-23 13:44 ` [PATCH ceph squid v4 7/7] " Max R. Carrara
  2026-11-16  9:18 ` [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Kefu Chai
  7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2026-06-23 13:44 UTC (permalink / raw)
  To: pve-devel

Add a patch that sets the CEPH_KEYRING env var for the ceph-exporter
daemon, telling it to use our custom keyring file. The patch also sets
the group of the daemon to `www-data` so that the daemon can access
the custom keyring.

Fixes: #6816
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 ...orter-use-custom-keyring-and-set-gro.patch | 32 +++++++++++++++++++
 patches/series                                |  1 +
 2 files changed, 33 insertions(+)
 create mode 100644 patches/0014-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch

diff --git a/patches/0014-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch b/patches/0014-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch
new file mode 100644
index 0000000000..f939b1d5af
--- /dev/null
+++ b/patches/0014-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch
@@ -0,0 +1,32 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: "Max R. Carrara" <m.carrara@proxmox.com>
+Date: Tue, 16 Sep 2025 16:34:51 +0200
+Subject: [PATCH] systemd: ceph-exporter: use custom keyring and set group to
+ www-data
+
+With the help of `Environment=`, set the CEPH_KEYRING env var to point
+to our custom keyring file location for the `ceph-exporter` daemon.
+
+Additionally, set the group of the `ceph-exporter` daemon to
+`www-data`, so that it can access this keyring.
+
+Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
+---
+ systemd/ceph-exporter.service.in | 3 ++-
+ 1 file changed, 2 insertions(+), 1 deletion(-)
+
+diff --git a/systemd/ceph-exporter.service.in b/systemd/ceph-exporter.service.in
+index f4f6d05c4b4..4e55939d201 100644
+--- a/systemd/ceph-exporter.service.in
++++ b/systemd/ceph-exporter.service.in
+@@ -6,8 +6,9 @@ Before=ceph.target
+ Wants=network-online.target local-fs.target ceph.target ceph-mon.target
+ 
+ [Service]
++Environment="CEPH_KEYRING=/etc/pve/ceph/ceph.client.exporter.keyring"
+ ExecReload=/bin/kill -HUP $MAINPID
+-ExecStart=@CMAKE_INSTALL_PREFIX@/bin/ceph-exporter -f --id %i --setuser ceph --setgroup ceph
++ExecStart=@CMAKE_INSTALL_PREFIX@/bin/ceph-exporter -f --id exporter --setuser ceph --setgroup www-data
+ LockPersonality=true
+ NoNewPrivileges=true
+ PrivateDevices=yes
diff --git a/patches/series b/patches/series
index db38f90574..daeb460654 100644
--- a/patches/series
+++ b/patches/series
@@ -11,3 +11,4 @@
 0011-debian-do-not-ship-legacy-init.d-ceph-script-anymore.patch
 0012-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch
 0013-pybind-rbd-disable-on_progress-callbacks-to-prevent-.patch
+0014-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 10+ messages in thread

* [PATCH ceph squid v4 7/7] fix #6816: patches: make ceph-exporter use custom keyring
  2026-06-23 13:44 [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Max R. Carrara
                   ` (5 preceding siblings ...)
  2026-06-23 13:44 ` [PATCH ceph master v4 6/7] fix #6816: patches: make ceph-exporter use custom keyring Max R. Carrara
@ 2026-06-23 13:44 ` Max R. Carrara
  2026-11-16  9:18 ` [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Kefu Chai
  7 siblings, 0 replies; 10+ messages in thread
From: Max R. Carrara @ 2026-06-23 13:44 UTC (permalink / raw)
  To: pve-devel

Add a patch that sets the CEPH_KEYRING env var for the ceph-exporter
daemon, telling it to use our custom keyring file. The patch also sets
the group of the daemon to `www-data` so that the daemon can access
the custom keyring.

Fixes: #6816
Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
---
 ...orter-use-custom-keyring-and-set-gro.patch | 32 +++++++++++++++++++
 patches/series                                |  1 +
 2 files changed, 33 insertions(+)
 create mode 100644 patches/0053-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch

diff --git a/patches/0053-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch b/patches/0053-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch
new file mode 100644
index 0000000000..2603dbf0e7
--- /dev/null
+++ b/patches/0053-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch
@@ -0,0 +1,32 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: "Max R. Carrara" <m.carrara@proxmox.com>
+Date: Tue, 16 Sep 2025 16:34:51 +0200
+Subject: [PATCH] systemd: ceph-exporter: use custom keyring and set group to
+ www-data
+
+With the help of `Environment=`, set the CEPH_KEYRING env var to point
+to our custom keyring file location for the `ceph-exporter` daemon.
+
+Additionally, set the group of the `ceph-exporter` daemon to
+`www-data`, so that it can access this keyring.
+
+Signed-off-by: Max R. Carrara <m.carrara@proxmox.com>
+---
+ systemd/ceph-exporter.service.in | 3 ++-
+ 1 file changed, 2 insertions(+), 1 deletion(-)
+
+diff --git a/systemd/ceph-exporter.service.in b/systemd/ceph-exporter.service.in
+index f4f6d05c4b4..4e55939d201 100644
+--- a/systemd/ceph-exporter.service.in
++++ b/systemd/ceph-exporter.service.in
+@@ -6,8 +6,9 @@ Before=ceph.target
+ Wants=network-online.target local-fs.target ceph.target ceph-mon.target
+ 
+ [Service]
++Environment="CEPH_KEYRING=/etc/pve/ceph/ceph.client.exporter.keyring"
+ ExecReload=/bin/kill -HUP $MAINPID
+-ExecStart=/usr/bin/ceph-exporter -f --id %i --setuser ceph --setgroup ceph
++ExecStart=/usr/bin/ceph-exporter -f --id exporter --setuser ceph --setgroup www-data
+ LockPersonality=true
+ NoNewPrivileges=true
+ PrivateDevices=yes
diff --git a/patches/series b/patches/series
index b877aacfe5..0ae73e3a8f 100644
--- a/patches/series
+++ b/patches/series
@@ -50,3 +50,4 @@
 0050-mgr-osd_perf_query-fix-ivalid-escape-sequence.patch
 0051-mgr-zabbix-fix-invalid-escape-sequences.patch
 0052-pybind-rbd-disable-on_progress-callbacks-to-prevent-.patch
+0053-systemd-ceph-exporter-use-custom-keyring-and-set-gro.patch
-- 
2.47.3





^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup
  2026-06-23 13:44 [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Max R. Carrara
                   ` (6 preceding siblings ...)
  2026-06-23 13:44 ` [PATCH ceph squid v4 7/7] " Max R. Carrara
@ 2026-11-16  9:18 ` Kefu Chai
  2026-11-16 11:15   ` Kefu Chai
  7 siblings, 1 reply; 10+ messages in thread
From: Kefu Chai @ 2026-11-16  9:18 UTC (permalink / raw)
  To: m.carrara; +Cc: pve-devel

Hi Max,

finally got around to this, sorry for the delay. I applied the
pve-manager patches to current master and they still apply cleanly,
which after five months is a pleasant surprise. I also like that the
keyring is only rewritten when its contents actually changed, and
that pve-init-ceph-crash comes down to a one-liner now.

Most of the comments below are about the ceph patches.

patch 3/7:

> +    if test -n "$2" && dpkg --compare-versions "$2" 'lt' '9.2.4'; then
> +        # Don't fail in case user has "exotic" configuration where RADOS
> +        # isn't available on all nodes for some reason
> +        /usr/share/pve-manager/helpers/pve-ceph-keyring --init client.exporter || true
> +    fi

9.2.4 is long out by now, so for most people upgrading this branch
never triggers. I guess this wants a bump to whatever version the
series lands in.

Also, update_ceph_conf() restarts ceph-crash.service when it is
enabled. Maybe do the same here, otherwise an exporter that already
crash-looped into its start limit stays down even once the keyring is
there.

patch 6/7:

> +diff --git a/systemd/ceph-exporter.service.in b/systemd/ceph-exporter.service.in

Upstream renamed this to a template, ceph-exporter@.service.in, with
the Tentacle 20.2.4 import, so on current master this does not apply
anymore. The patches/series context moved as well, but that part is
just appending at the end again. The squid patch still has the right
file name.

Since the branch exists now, the series could use a tentacle-stable-9
patch too. With the template unit, --id being hardcoded to 'exporter'
means the instance name does not matter for the arguments, but
something still has to enable an instance. Does the packaging do
that? I did not dig into how the unit gets enabled these days.

One tiny nit on 5/7: #!/bin/bash can be #!/bin/sh for the one-liner.

And maybe a line for the commit message: the in-tree exporter never
talks to the cluster at all, it only scans the admin sockets, so the
mon 'allow r' cap is mostly there to have a valid key. Which I think
is fine, just good to know when someone later wonders about the caps.




^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup
  2026-11-16  9:18 ` [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Kefu Chai
@ 2026-11-16 11:15   ` Kefu Chai
  0 siblings, 0 replies; 10+ messages in thread
From: Kefu Chai @ 2026-11-16 11:15 UTC (permalink / raw)
  To: m.carrara; +Cc: pve-devel

Hi Max,

first a correction to my review, sorry: I claimed the mon 'allow r'
cap is mostly a formality because the exporter's own code has no
MonClient, and that was wrong. I hadn't tested it. global_init()
runs the mon config bootstrap for the exporter anyway: as a
CODE_ENVIRONMENT_DAEMON without --no-mon-config it fetches the
monmap and config from the mons at startup, and that mini session
does authenticate with the keyring. So the cap is real, and your
bug report was spot on. Started by systemd as root, the exporter
reads ceph.conf before dropping its privileges, so the keyring
really is the first thing it trips over.

Chasing my own mistake, I ran into the escape hatch that's printed
right in the error message: --no-mon-config. It's a generic CLI
flag, every ceph binary parses it, and with it the exporter needs
nothing from the cluster at startup, no mon connection, no CephX,
and no keyring. Its own options (sock dir, port, and so on) all
have compiled defaults and can be passed as arguments, and I can't
think of anything we'd want to tune via the mon config for it.

Quick test on our lab, as a service: the stock unit crash loops as
described. A drop-in that only appends --no-mon-config to ExecStart,
nothing else, and the service comes up as plain ceph:ceph and
happily serves the local daemon metrics on port 9926.

If you agree this is the better route, that one word could replace
a good chunk of the series: no client.exporter, no ceph.conf
section for it, no pve-ceph-keyring call in postinst, and even the
group change becomes optional (it just keeps a config warning out
of the journal). Not trying to talk you out of the keyring approach
though, that part of the series is well built either way.

One more thing the flag buys us: I also started the exporter with a
readable config and a valid keyring but a dead mon address, and it
just sat in the bootstrap for the four minutes I gave it, printing
nothing, serving nothing. With the flag it starts serving right
away. A mon outage at boot time would otherwise leave you without
node metrics, and those are exactly the moments I'd want them for.

The one downside I can see: options set with 'ceph config set' no
longer reach the exporter. I don't think we use that, and cephadm
style deployments could keep the old behavior by not passing the
flag. Maybe upstream could make the bootstrap best effort for this
daemon some day, but that feels like a separate discussion.

The pve-init-ceph-crash cleanup is unrelated to all this and worth
keeping either way.

Happy to test a v5 either way!




^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2026-11-16 11:15 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-06-23 13:44 [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Max R. Carrara
2026-06-23 13:44 ` [PATCH pve-manager v4 1/7] ceph: tools: add helper sub for creating or updating keyring files Max R. Carrara
2026-06-23 13:44 ` [PATCH pve-manager v4 2/7] fix #6816: api: ceph: create 'client.exporter' w/ keyring Max R. Carrara
2026-06-23 13:44 ` [PATCH pve-manager v4 3/7] fix #6816: bin: add pve-ceph-keyring helper and call it in postinst Max R. Carrara
2026-06-23 13:44 ` [PATCH pve-manager v4 4/7] ceph: tools: simplify helper sub for crash keyring file Max R. Carrara
2026-06-23 13:44 ` [PATCH pve-manager v4 5/7] bin: make pve-init-ceph-crash call pve-ceph-keyring Max R. Carrara
2026-06-23 13:44 ` [PATCH ceph master v4 6/7] fix #6816: patches: make ceph-exporter use custom keyring Max R. Carrara
2026-06-23 13:44 ` [PATCH ceph squid v4 7/7] " Max R. Carrara
2026-11-16  9:18 ` [PATCH ceph master/ceph squid/manager v4 0/7] Fix #6816: Prevent ceph-exporter Daemon from Crashing on Startup Kefu Chai
2026-11-16 11:15   ` Kefu Chai

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal