public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH v4 storage 15/36] api: multipath: add the cluster-wide health status
Date: Fri, 13 Nov 2026 04:00:33 +0100	[thread overview]
Message-ID: <20261113030146.1559030-16-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261113030146.1559030-1-t.lamprecht@proxmox.com>

Combine the reports of the nodes into one view: the state of each
node (ok, multipath unavailable with the reason, or no report within
three minutes) and, per LUN, the state of its map on each node with
the worst one as summary. A LUN is listed if a node reports it, if it
has an explicit entry or if an LVM storage uses it as base volume.

A node should have a LUN if an explicit entry, the iSCSI storage that
requests the map or a shared LVM storage on it covers the node. One
that does not report it counts as missing, one without a current
report as unknown. Storage admins can read it as well, they need the
initiator names for the array.

Each LUN also names the enabled iSCSI storage with multipath that
requests its map on one of its nodes, so that a map without an explicit
entry can be told apart from one that nothing in Proxmox VE asked for. A
node reports the number of paths it expects, so that one with fewer
paths can show how many are missing.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---
 src/PVE/API2/Multipath.pm           |  88 ++++++++++++++++++++-
 src/PVE/Multipath/Status.pm         | 109 +++++++++++++++++++++++++
 src/test/run_multipath_api_tests.pl | 118 +++++++++++++++++++++++++++-
 3 files changed, 312 insertions(+), 3 deletions(-)

diff --git a/src/PVE/API2/Multipath.pm b/src/PVE/API2/Multipath.pm
index 667c2340..d5d2df9d 100644
--- a/src/PVE/API2/Multipath.pm
+++ b/src/PVE/API2/Multipath.pm
@@ -10,6 +10,7 @@ use PVE::RESTHandler;
 use PVE::Tools qw(extract_param);
 
 use PVE::Multipath::Config;
+use PVE::Multipath::Status;
 
 use base qw(PVE::RESTHandler);
 
@@ -60,7 +61,92 @@ __PACKAGE__->register_method({
         links => [{ rel => 'child', href => "{name}" }],
     },
     code => sub {
-        return [map { +{ name => $_ } } qw(overrides wwid)];
+        return [map { +{ name => $_ } } qw(overrides status wwid)];
+    },
+});
+
+my $state_schema = {
+    type => 'string',
+    enum => ['optimal', 'degraded', 'failed', 'missing', 'unknown'],
+};
+my $lun_node_schema = {
+    type => 'object',
+    additionalProperties => 0,
+    properties => {
+        state => $state_schema,
+        name => { type => 'string', optional => 1 },
+        map { $_ => { type => 'integer', minimum => 0, optional => 1 } }
+            qw(size paths-active paths-total paths-expected),
+    },
+};
+my $node_schema = {
+    type => 'object',
+    additionalProperties => 0,
+    properties => {
+        state => { type => 'string', enum => ['ok', 'unavailable', 'no-report'] },
+        time => { type => 'integer', optional => 1 },
+        map { $_ => { type => 'string', optional => 1 } }
+            qw(error apply-error iscsi-initiator-name),
+    },
+};
+
+__PACKAGE__->register_method({
+    name => 'status',
+    path => 'status',
+    method => 'GET',
+    description => "Report the multipath health of each LUN on each node, and the worst"
+        . " state among the nodes that should see or report the LUN.",
+    # Storage administrators need path health and initiator names to configure SAN target ACLs.
+    permissions => {
+        check => [
+            'or', ['perm', '/', ['Sys.Audit']], ['perm', '/storage', ['Datastore.Allocate']],
+        ],
+    },
+    parameters => {
+        additionalProperties => 0,
+        properties => {},
+    },
+    returns => {
+        type => 'object',
+        properties => {
+            luns => {
+                type => 'array',
+                description =>
+                    "One entry per LUN, identified by its WWID, with the state of its"
+                    . " map on each node and the overall cluster state.",
+                items => {
+                    type => 'object',
+                    additionalProperties => 0,
+                    properties => {
+                        wwid => { type => 'string', description => 'The LUN WWID.' },
+                        explicit =>
+                            { type => 'boolean', description => 'An explicit entry exists.' },
+                        name => { type => 'string', optional => 1 },
+                        'used-by-storage' => { type => 'string', optional => 1 },
+                        'multipath-storage' => {
+                            type => 'string',
+                            optional => 1,
+                            description => 'The storage that requests the map of the LUN.',
+                        },
+                        size => { type => 'integer', minimum => 0, optional => 1 },
+                        state => $state_schema,
+                        nodes => {
+                            type => 'object',
+                            description => "Map health keyed by node name.",
+                            additionalProperties => $lun_node_schema,
+                        },
+                    },
+                },
+            },
+            nodes => {
+                type => 'object',
+                description => "Available node reports, keyed by node name.",
+                additionalProperties => $node_schema,
+            },
+        },
+    },
+    code => sub {
+        return PVE::Multipath::Status::cluster_status();
     },
 });
 
diff --git a/src/PVE/Multipath/Status.pm b/src/PVE/Multipath/Status.pm
index 91b699c4..58d11f5d 100644
--- a/src/PVE/Multipath/Status.pm
+++ b/src/PVE/Multipath/Status.pm
@@ -5,11 +5,22 @@ use JSON qw(decode_json);
 use PVE::Cluster;
 use PVE::INotify;
 use PVE::Multipath;
+use PVE::Multipath::Config;
 use PVE::Multipath::Generator;
 
 my $json = JSON->new->utf8->canonical;
 my $KEY = 'multipath';
 
+sub worst_state(@states) {
+    my %rank = (unknown => -1, optimal => 0, degraded => 1, missing => 2, failed => 3);
+    my $worst = 'unknown';
+    for my $state (@states) {
+        $worst = $state
+            if defined($state) && exists($rank{$state}) && $rank{$state} > $rank{$worst};
+    }
+    return $worst;
+}
+
 sub publish($report) {
     my $node = PVE::INotify::nodename();
     my $now = time();
@@ -58,6 +69,8 @@ sub update_node_status() {
             $lun->{size} = $map->{size} if defined($map->{size});
             $lun->{name} = $map->{name} if defined($map->{name}) && $map->{name} ne $wwid;
             $lun->{storage} = $info->{storages}->[0] if $info->{storages};
+            $lun->{'paths-expected'} = $info->{'paths-expected'}
+                if defined($info->{'paths-expected'});
             my $vg = defined($map->{sysfs}) ? $pv_vgs->{"/dev/$map->{sysfs}"} : undef;
             $lun->{vgname} = $vg if defined($vg);
             push $report->{luns}->@*, $lun;
@@ -72,4 +85,100 @@ sub update_node_status() {
     warn "multipath: cannot publish node status - $@" if $@;
 }
 
+sub cluster_status() {
+    require PVE::Storage;
+
+    my $cfg = PVE::Multipath::Config::read_config();
+    my $storage = PVE::Storage::config();
+    my $all_nodes = PVE::Cluster::get_nodelist();
+    my $members = PVE::Cluster::get_members();
+    my $raw = PVE::Cluster::get_node_kv($KEY);
+    my (%nodes, %luns, %expected, %observed, %vg_owner);
+    my $storage_nodes = sub($scfg) {
+        return $scfg->{nodes} && $scfg->{nodes}->%* ? [keys $scfg->{nodes}->%*] : $all_nodes;
+    };
+    # only an enabled iSCSI storage with the option requests the maps of its LUNs, on its nodes
+    my %current = map { $_ => 1 } @$all_nodes;
+    my $requests_maps = sub($sid, $node = undef) {
+        my $scfg = defined($sid) ? $storage->{ids}->{$sid} : undef;
+        return 0
+            if !$scfg || $scfg->{type} ne 'iscsi' || !$scfg->{multipath} || $scfg->{disable};
+        my @nodes = grep { $current{$_} } $storage_nodes->($scfg)->@*;
+        return defined($node) ? !!grep({ $_ eq $node } @nodes) : !!@nodes;
+    };
+    for my $wwid (PVE::Multipath::Config::wwid_list($cfg)->@*) {
+        my $entry = $cfg->{ids}->{$wwid};
+        $luns{$wwid} = { wwid => $wwid, explicit => 1 };
+        $luns{$wwid}->{name} = $entry->{alias} if defined($entry->{alias});
+        $expected{$wwid}->{$_} = 1 for $storage_nodes->($entry)->@*;
+    }
+    for my $sid (sort keys $storage->{ids}->%*) {
+        my $scfg = $storage->{ids}->{$sid};
+        $vg_owner{ $scfg->{vgname} } //= $sid if defined($scfg->{vgname});
+        next if $scfg->{type} ne 'lvm';
+        my ($wwid) = ($scfg->{base} // '') =~ /(?:^|\.)dm-uuid-mpath-(\S+)$/;
+        next if !defined($wwid);
+        $luns{$wwid} //= { wwid => $wwid, explicit => 0 };
+        $luns{$wwid}->{'used-by-storage'} //= $sid;
+        my ($base) = $scfg->{base} =~ /^([^:]+):/;
+        $luns{$wwid}->{'multipath-storage'} //= $base if $requests_maps->($base);
+        $expected{$wwid}->{$_} = 1 for $storage_nodes->($scfg)->@*;
+    }
+    for my $node (sort @$all_nodes) {
+        my $report = eval { decode_json($raw->{$node} // '') };
+        $report = {} if ref($report) ne 'HASH';
+        my $entry = $nodes{$node} = { state => 'no-report' };
+        $entry->{time} = $report->{time} if defined($report->{time});
+        next
+            if (exists($members->{$node}) && !$members->{$node}->{online})
+            || !defined($report->{time})
+            || time() - $report->{time} > 180;
+        $entry->{$_} = $report->{$_}
+            for grep { defined($report->{$_}) } qw(error apply-error iscsi-initiator-name);
+        $entry->{state} = defined($report->{error}) ? 'unavailable' : 'ok';
+        next if $entry->{state} ne 'ok';
+
+        for my $lun (($report->{luns} // [])->@*) {
+            my $wwid = $lun->{wwid} // next;
+            my $row = $luns{$wwid} //= { wwid => $wwid, explicit => 0 };
+            $row->{$_} //= $lun->{$_} for grep { defined($lun->{$_}) } qw(name size);
+            my $owner = defined($lun->{vgname}) ? $vg_owner{ $lun->{vgname} } : undef;
+            $row->{'used-by-storage'} //= $owner if defined($owner);
+            # a shared LVM storage on the map needs it on its nodes, also without a base volume
+            my $lvm = defined($owner) ? $storage->{ids}->{$owner} : undef;
+            if ($lvm && $lvm->{type} eq 'lvm' && $lvm->{shared} && !$lvm->{disable}) {
+                $expected{$wwid}->{$_} = 1 for $storage_nodes->($lvm)->@*;
+            }
+            my $sid = $lun->{storage};
+            if (defined($sid) && (my $scfg = $storage->{ids}->{$sid})) {
+                $row->{'multipath-storage'} //= $sid if $requests_maps->($sid, $node);
+                $row->{'used-by-storage'} //= $sid
+                    if PVE::Storage::san_storage_serves_images($scfg);
+                $expected{$wwid}->{$_} = 1
+                    for !$scfg->{disable} ? $storage_nodes->($scfg)->@* : ();
+            }
+            $observed{$wwid}->{$node} = {
+                state => worst_state($lun->{state}),
+                map { $_ => $lun->{$_} }
+                    grep { defined($lun->{$_}) }
+                    qw(name size paths-active paths-total paths-expected),
+            };
+        }
+    }
+    for my $wwid (sort keys %luns) {
+        my $row = $luns{$wwid};
+        $row->{nodes} = {};
+        for my $node (sort @$all_nodes) {
+            if ($observed{$wwid}->{$node}) {
+                $row->{nodes}->{$node} = $observed{$wwid}->{$node};
+            } elsif ($expected{$wwid}->{$node}) {
+                $row->{nodes}->{$node} =
+                    { state => $nodes{$node}->{state} eq 'ok' ? 'missing' : 'unknown' };
+            }
+        }
+        $row->{state} = worst_state(map { $_->{state} } values $row->{nodes}->%*);
+    }
+    return { nodes => \%nodes, luns => [map { $luns{$_} } sort keys %luns] };
+}
+
 1;
diff --git a/src/test/run_multipath_api_tests.pl b/src/test/run_multipath_api_tests.pl
index aac5e250..77ee3b9f 100755
--- a/src/test/run_multipath_api_tests.pl
+++ b/src/test/run_multipath_api_tests.pl
@@ -8,7 +8,7 @@ BEGIN {
 }
 use lib ('.', '..');
 use File::Temp qw(tempdir);
-use JSON qw(decode_json);
+use JSON qw(encode_json decode_json);
 use Storable qw(dclone);
 use Test::More;
 use Test::MockModule;
@@ -45,7 +45,7 @@ subtest 'explicit entries are ordinary configuration writes' => sub {
     }
     is_deeply(
         PVE::API2::Multipath->index({}),
-        [map { { name => $_ } } qw(overrides wwid)],
+        [map { { name => $_ } } qw(overrides status wwid)],
         'index',
     );
 };
@@ -232,7 +232,121 @@ subtest 'one entry point applies configuration and reports unavailable reasons'
         sub { [{ wwid => '3600a', name => 'data', 'paths-active' => 1, 'paths-total' => 2 }] });
     PVE::Multipath::Status::update_node_status();
     is($report->{luns}->[0]->{state}, 'degraded', 'maps without PVE entries are included');
+    ok(!exists($report->{luns}->[0]->{'paths-expected'}), 'no expected count without a source');
+    $gen->mock(sync => sub { { wwids => { '3600a' => { 'paths-expected' => 3 } } } });
+    $mp->mock(get_maps =>
+        sub { [{ wwid => '3600a', name => 'data', 'paths-active' => 2, 'paths-total' => 2 }] });
+    PVE::Multipath::Status::update_node_status();
+    is($report->{luns}->[0]->{'paths-expected'}, 3, 'expected count reported');
+    is($report->{luns}->[0]->{state}, 'degraded', 'fewer paths than expected');
     ok(!exists($report->{supported}), 'no separate capability state');
 };
 
+subtest 'API returns final node and LUN states' => sub {
+    my $cluster = Test::MockModule->new('PVE::Cluster');
+    my $config = Test::MockModule->new('PVE::Multipath::Config');
+    my $storage = Test::MockModule->new('PVE::Storage');
+    $cluster->mock(get_nodelist => sub { [qw(n1 n2 n3 n4 n5)] });
+    $cluster->mock(get_members => sub { { n4 => { online => 0 } } });
+    my %raw = (
+        n1 => encode_json({
+            time => $NOW,
+            luns => [
+                {
+                    wwid => '3600a',
+                    state => 'optimal',
+                    'paths-total' => 2,
+                    'paths-active' => 2,
+                    'paths-expected' => 2,
+                    storage => 'iscsi',
+                    vgname => 'vgfc',
+                },
+                {
+                    wwid => '3600d',
+                    state => 'optimal',
+                    'paths-total' => 2,
+                    'paths-active' => 2,
+                    storage => 'off',
+                    vgname => 'vgshared',
+                },
+                {
+                    wwid => '3600f',
+                    state => 'optimal',
+                    'paths-total' => 2,
+                    'paths-active' => 2,
+                    storage => 'scoped',
+                },
+            ],
+        }),
+        n2 => encode_json({ time => $NOW, luns => [] }),
+        n3 => encode_json({ time => $NOW - 181, luns => [] }),
+        n4 => encode_json({ time => $NOW, luns => [] }),
+        n5 => encode_json({ time => $NOW, error => 'multipath-tools is not installed' }),
+    );
+    $cluster->mock(get_node_kv => sub { \%raw });
+    $config->mock(read_config => sub { { ids => { '3600a' => { type => 'wwid' } } } });
+    $storage->mock(
+        config => sub { {
+            ids => {
+                iscsi => { type => 'iscsi', multipath => 1 },
+                lvm => { type => 'lvm', base => 'iscsi:0.0.0.dm-uuid-mpath-3600b' },
+                fc => { type => 'lvm', vgname => 'vgfc' },
+                off => { type => 'iscsi' },
+                lvmoff => { type => 'lvm', base => 'off:0.0.0.dm-uuid-mpath-3600c' },
+                shared => { type => 'lvm', vgname => 'vgshared', shared => 1 },
+                gone => { type => 'iscsi', multipath => 1, nodes => { departed => 1 } },
+                lvmgone => { type => 'lvm', base => 'gone:0.0.0.dm-uuid-mpath-3600e' },
+                scoped => { type => 'iscsi', multipath => 1, nodes => { n2 => 1 } },
+            },
+        } },
+    );
+    my $res = PVE::API2::Multipath->status({});
+    is_deeply(
+        [map { $res->{nodes}->{$_}->{state} } qw(n1 n2 n3 n4 n5)],
+        [qw(ok ok no-report no-report unavailable)],
+        'final node states',
+    );
+    is($res->{nodes}->{n3}->{time}, $NOW - 181, 'old report time retained');
+    my $lun = $res->{luns}->[0];
+    is($lun->{state}, 'missing', 'worst known state');
+    is($lun->{nodes}->{n1}->{state}, 'optimal', 'observed healthy map');
+    is($lun->{nodes}->{n2}->{state}, 'missing', 'expected map absent from current report');
+    is($lun->{nodes}->{n3}->{state}, 'unknown', 'stale observation');
+    is($lun->{'used-by-storage'}, 'fc', 'storage attributed through the reported volume group');
+    is($res->{luns}->[1]->{wwid}, '3600b', 'LVM base included without observation');
+    is($res->{luns}->[1]->{'used-by-storage'}, 'lvm', 'LVM consumer attributed');
+    is($lun->{'multipath-storage'}, 'iscsi', 'reported storage requests the map');
+    is($res->{luns}->[1]->{'multipath-storage'}, 'iscsi', 'base storage requests the map');
+    is($lun->{nodes}->{n1}->{'paths-expected'}, 2, 'expected path count of the node');
+    my %lun = map { $_->{wwid} => $_ } $res->{luns}->@*;
+    ok(
+        !exists($lun{'3600c'}->{'multipath-storage'}),
+        'a base without multipath requests nothing',
+    );
+    ok(!exists($lun{'3600d'}->{'multipath-storage'}), 'neither does a reported one');
+    ok(
+        !exists($lun{'3600e'}->{'multipath-storage'}),
+        'a base for no current node requests nothing',
+    );
+    ok(!exists($lun{'3600f'}->{'multipath-storage'}), 'a storage requests on its nodes only');
+    is(
+        $lun{'3600d'}->{nodes}->{n2}->{state},
+        'missing',
+        'the nodes of a shared LVM expect the map',
+    );
+    is($lun{'3600d'}->{'used-by-storage'}, 'shared', 'shared LVM attributed by volume group');
+    eval {
+        PVE::JSONSchema::validate(
+            $res,
+            PVE::API2::Multipath->map_method_by_name('status')->{returns},
+        );
+    };
+    is($@, '', 'result satisfies the API schema');
+    is(
+        PVE::Multipath::Status::worst_state(qw(unknown optimal failed degraded missing)),
+        'failed',
+        'one ranking',
+    );
+};
+
 done_testing();




  parent reply	other threads:[~2026-11-13  3:14 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-13  3:00 [PATCH v4 cluster,storage,docs,manager 00/36] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 cluster 01/36] cfs: observe the multipath configuration files Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 02/36] diskmanage: collect disk transport type from lsblk Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 03/36] diskmanage: qualify NVMe over fabrics transport Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 04/36] diskmanage: optionally include iSCSI session devices in disk enumeration Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 05/36] diskmanage: collect native NVMe multipath path state Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 06/36] iscsi: factor out session device map from device list Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 07/36] multipath: add a library around multipathd Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 08/36] diskmanage: add helper to list multipath devices Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 09/36] multipath: add a cluster-wide configuration Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 10/36] iscsi: add an option to manage storage LUNs with multipath Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 11/36] lvm: activate a missing base volume before using the volume group Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 12/36] multipath: apply the configuration on every node Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 13/36] multipath: publish per-node status Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 14/36] api: add endpoints for the cluster-wide multipath configuration Thomas Lamprecht
2026-11-13  3:00 ` Thomas Lamprecht [this message]
2026-11-13  3:00 ` [PATCH v4 storage 16/36] api: scan: add a SAN LUN inventory Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 17/36] pvesm: expose the " Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 18/36] api: add node-level iSCSI status Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 19/36] disks: lvm: drop unused imports Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 20/36] disks: lvm: allow volume groups on SAN LUNs and multipath maps Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 21/36] disks: lvm: refuse to destroy the volume group of a shared storage Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 storage 22/36] disks: allow wiping a SAN LUN that no storage uses Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 docs 23/36] storage: document the cluster-wide multipath management Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 24/36] pvestatd: apply the multipath configuration and report the node status Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 25/36] report: add the managed multipath configuration Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 26/36] api: cluster: add the multipath endpoints Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 27/36] api: nodes: expose the iSCSI status Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 28/36] ui: dc: add a SAN view Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 29/36] ui: storage: iscsi: add the multipath option Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 30/36] ui: lvm: enable shared when a base storage is selected Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 31/36] ui: node: add SAN LUNs panel Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 32/36] ui: node: san: wipe a LUN that no storage uses Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 33/36] ui: node: san: show iSCSI targets and sessions Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 34/36] ui: dc: san: add a guided setup for shared LVM on iSCSI Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 35/36] ui: dc: san: add a guided setup for shared LVM on Fibre Channel Thomas Lamprecht
2026-11-13  3:00 ` [PATCH v4 manager 36/36] ui: storage: point the iSCSI and LVM add dialogs to the SAN view Thomas Lamprecht
2026-11-16  1:28 ` superseded: [PATCH v4 cluster,storage,docs,manager 00/36] SAN: cluster-wide multipath, SAN views and a guided setup for shared LVM Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261113030146.1559030-16-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal