all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH cluster v3 06/38] pmxcfs: dcdb: generate the cluster ID once the cluster is in sync
Date: Wed, 18 Nov 2026 05:37:18 +0100	[thread overview]
Message-ID: <20261118043817.1659212-7-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261118043817.1659212-1-t.lamprecht@proxmox.com>

Generate a missing or invalid cluster ID in pmxcfs, including for
upgraded clusters. Only the member with the lowest node ID generates
it, preventing competing IDs within a group.

Quorum alone is insufficient. During a cold boot, corosync can be
quorate before all pmxcfs instances have joined their group. A node
with a stale database could generate an ID before learning the
existing one. At equal database versions, its newer mtime could
then make that ID win the next synchronization.

Require quorum, more than half of the configured nodes in the corosync
membership, and every member of that membership synchronized in the
pmxcfs group. Majorities of the same configured node set overlap;
together with normal database transfer on join, this prevents a stale
minority from generating before synchronizing with the other nodes.
Count nodes rather than votes so this also holds with a QDevice,
two_node, last_man_standing, or lowered expected votes. The tradeoff is
that no ID is generated while half or more of the nodes are down.

This does not prevent ordinary database synchronization from selecting
a database that lacks the ID for other reasons. An ID is generated
again in that case. Log why generation is blocked whenever the reason
changes, naming the nodes whose pmxcfs is missing.

Retry after synchronization, after quorum, membership or configuration
changes, and on the hourly dcdb timer. Generation uses the existing
ordered file messages without waiting on the loop thread. Interrupted
creation can leave an empty file, which a later attempt fills. Reuse the
generated ID until a valid one is observed, so retries before delivery
do not replace it. Leave a directory at the path untouched and log it.

Older nodes can replicate the file without a new message type. If the
lowest member runs an older version, generation waits until that node
is upgraded or leaves.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
---

changes v2 -> v3:
* check again when the configured nodes get published or change, with
  the new confdb listener, instead of waiting for the next quorum or
  group change or the hourly timer; test for nodes removed from the
  configuration
* comments: quorum is still required next to the node majority, and the
  test instances share the per-process state

changes v1 -> v2:
* rebase onto the dfsm rework: the test nodes have to count as joined to
  send

 src/pmxcfs/Makefile     |  10 +-
 src/pmxcfs/check_dcdb.c | 563 ++++++++++++++++++++++++++++++++++++++++
 src/pmxcfs/dcdb.c       | 153 +++++++++++
 src/pmxcfs/dcdb.h       |   8 +
 src/pmxcfs/pmxcfs.c     |   4 +-
 src/pmxcfs/status.c     |  11 +
 src/pmxcfs/status.h     |   3 +
 7 files changed, 747 insertions(+), 5 deletions(-)
 create mode 100644 src/pmxcfs/check_dcdb.c

diff --git a/src/pmxcfs/Makefile b/src/pmxcfs/Makefile
index cd93e72..64d6d6f 100644
--- a/src/pmxcfs/Makefile
+++ b/src/pmxcfs/Makefile
@@ -17,7 +17,7 @@ ARFLAGS = crs
 .c.o:
 	$(CC) $(CFLAGS) -c -o $@ $< -MMD -MT $@ -MF $@.d
 
-all: pmxcfs create_pmxcfs_db logtest check_memdb
+all: pmxcfs create_pmxcfs_db logtest check_memdb check_dcdb
 
 libpmxcfs.a: logger.o loop.o server.o status.o confdb.o quorum.o dcdb.o dfsm.o
 libpmxcfs.a: cfs-plug.o cfs-plug-memdb.o cfs-plug-link.o cfs-plug-func.o
@@ -41,6 +41,9 @@ logtest2: logtest2.o libpmxcfs.a
 check_memdb: check_memdb.o libpmxcfs.a
 	$(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS) $(shell pkg-config --libs check)
 
+check_dcdb: check_dcdb.o libpmxcfs.a
+	$(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS) $(shell pkg-config --libs check)
+
 pmxcfs.8:
 
 -include /usr/share/pve-doc-generator/pve-doc-generator.mk
@@ -57,11 +60,12 @@ format:
 	git ls-files ':/src/pmxcfs/*.[ch]' | xargs -n2 -P0 -- clang-format -i
 
 .PHONY: check
-check: check_memdb
+check: check_memdb check_dcdb
 	./check_memdb
+	./check_dcdb
 
 .PHONY: clean
 clean:
-	rm -f *.o *.o.d *.a pmxcfs create_pmxcfs_db check_memdb logtest pmxcfs.8
+	rm -f *.o *.o.d *.a pmxcfs create_pmxcfs_db check_memdb check_dcdb logtest pmxcfs.8
 
 -include *.o.d
diff --git a/src/pmxcfs/check_dcdb.c b/src/pmxcfs/check_dcdb.c
new file mode 100644
index 0000000..81909d3
--- /dev/null
+++ b/src/pmxcfs/check_dcdb.c
@@ -0,0 +1,563 @@
+/*
+  Copyright (C) 2026 Proxmox Server Solutions GmbH
+
+  This program is free software: you can redistribute it and/or modify
+  it under the terms of the GNU Affero General Public License as published by
+  the Free Software Foundation, either version 3 of the License, or
+  (at your option) any later version.
+
+  This program is distributed in the hope that it will be useful,
+  but WITHOUT ANY WARRANTY; without even the implied warranty of
+  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+  GNU Affero General Public License for more details.
+
+  You should have received a copy of the GNU Affero General Public License
+  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+  Author: Thomas Lamprecht <t.lamprecht@proxmox.com>
+
+*/
+
+/* Runs several dcdb instances in one process, each with its own memdb, and replaces the CPG
+ * transport with an ordered in-process queue, so that membership changes, syncs and the cluster ID
+ * generation can be tested without corosync. The sources are included to reach the static CPG
+ * callbacks. The quorum view and the pending cluster ID of dcdb.c are per process, so all instances
+ * share them and the tests keep node 1 as the lowest node. */
+
+#include <check.h>
+#include <stdint.h>
+
+#include "dfsm.c"
+#undef G_LOG_DOMAIN
+#include "dcdb.c"
+
+#if CHECK_MAJOR_VERSION == 0 && CHECK_MINOR_VERSION <= 10
+#define _TEST_TYPE TFun
+#else
+#define _TEST_TYPE const TTest *
+#endif
+
+cfs_t cfs = {
+    .debug = 0,
+    .nodename = "testnode",
+};
+
+typedef struct {
+    uint32_t nodeid;
+    uint32_t pid;
+    size_t len;
+    void *data;
+} packet_t;
+
+static GQueue packets = G_QUEUE_INIT;
+
+cs_error_t cpg_context_get(cpg_handle_t handle, void **context) {
+    *context = (void *)(uintptr_t)handle;
+    return CS_OK;
+}
+
+cs_error_t cpg_mcast_joined(
+    cpg_handle_t handle, cpg_guarantee_t guarantee, const struct iovec *iov, unsigned int iov_len
+) {
+    dfsm_t *sender = (dfsm_t *)(uintptr_t)handle;
+    packet_t *p = g_new0(packet_t, 1);
+
+    p->nodeid = sender->nodeid;
+    p->pid = sender->pid;
+
+    for (unsigned int i = 0; i < iov_len; i++) {
+        p->len += iov[i].iov_len;
+    }
+
+    p->data = g_malloc(p->len);
+
+    size_t pos = 0;
+    for (unsigned int i = 0; i < iov_len; i++) {
+        if (iov[i].iov_len) {
+            memcpy((char *)p->data + pos, iov[i].iov_base, iov[i].iov_len);
+        }
+        pos += iov[i].iov_len;
+    }
+
+    g_queue_push_tail(&packets, p);
+
+    return CS_OK;
+}
+
+/* delivers all queued messages, including those sent meanwhile, to the given nodes in order */
+static void pump(dfsm_t **nodes, int count) {
+    int rounds = 0;
+
+    while (!g_queue_is_empty(&packets)) {
+        ck_assert(++rounds < 1000);
+
+        packet_t *p = g_queue_pop_head(&packets);
+        for (int i = 0; i < count; i++) {
+            dfsm_cpg_deliver_callback(
+                nodes[i]->cpg_handle, &nodes[i]->cpg_group_name, p->nodeid, p->pid, p->data, p->len
+            );
+            ck_assert(dfsm_get_mode(nodes[i]) < DFSM_ERROR_MODE_START);
+        }
+
+        g_free(p->data);
+        g_free(p);
+    }
+}
+
+/* the databases of all tests, removed at the end whatever the tests left behind */
+static char *tmpdir;
+
+static memdb_t *open_db(const char *name) {
+    char *filename = g_build_filename(tmpdir, name, NULL);
+    unlink(filename);
+    memdb_t *memdb = memdb_open(filename);
+    g_free(filename);
+    ck_assert(memdb != NULL);
+
+    /* a node in cluster mode always has a corosync.conf */
+    ck_assert(memdb_create(memdb, "corosync.conf", 0, 100) == 0);
+
+    return memdb;
+}
+
+static dfsm_t *start_node(memdb_t *memdb, uint32_t nodeid) {
+    dfsm_t *dfsm = dcdb_new(memdb);
+    ck_assert(dfsm != NULL);
+
+    dfsm->cpg_handle = (uintptr_t)dfsm;
+    dfsm->joined = TRUE; /* messages are only sent while joined */
+    dfsm->nodeid = nodeid;
+    dfsm->pid = 100 + nodeid;
+
+    return dfsm;
+}
+
+static void stop_nodes(dfsm_t **nodes, memdb_t **dbs, int count) {
+    for (int i = 0; i < count; i++) {
+        nodes[i]->cpg_handle = 0;
+        dfsm_destroy(nodes[i]);
+        memdb_close(dbs[i]);
+    }
+}
+
+/* the nodes in the corosync configuration, as the confdb service reads them */
+static void set_config_nodes(uint32_t count) {
+    cfs_clinfo_t *clinfo = cfs_clinfo_new("test", 1);
+
+    for (uint32_t i = 1; i <= count; i++) {
+        char *name = g_strdup_printf("node%u", i);
+        cfs_clinfo_add_node(clinfo, cfs_clnode_new(name, i, 1));
+        g_free(name);
+    }
+
+    cfs_status_set_clinfo(clinfo);
+}
+
+static void pump(dfsm_t **nodes, int count);
+
+static void publish_config_nodes(dfsm_t **nodes, int count, uint32_t config_count) {
+    set_config_nodes(config_count);
+
+    for (int i = 0; i < count; i++) {
+        dcdb_config_changed(nodes[i]);
+    }
+
+    pump(nodes, count);
+}
+
+/* a change of the dcdb group, as seen by its members, without delivering what it causes */
+static void change_group(dfsm_t **nodes, int count) {
+    struct cpg_address members[count];
+
+    for (int i = 0; i < count; i++) {
+        members[i].nodeid = nodes[i]->nodeid;
+        members[i].pid = nodes[i]->pid;
+        members[i].reason = 0;
+    }
+
+    for (int i = 0; i < count; i++) {
+        dfsm_cpg_confchg_callback(
+            nodes[i]->cpg_handle, &nodes[i]->cpg_group_name, members, count, NULL, 0, members, count
+        );
+    }
+}
+
+static void set_group(dfsm_t **nodes, int count) {
+    change_group(nodes, count);
+    pump(nodes, count);
+}
+
+/* a change of the corosync membership, as the quorum service reports it */
+static void
+set_quorum(dfsm_t **nodes, int count, uint32_t quorate, uint32_t *view, size_t view_len) {
+    cfs_set_quorate(quorate, TRUE);
+
+    for (int i = 0; i < count; i++) {
+        dcdb_quorum_view_changed(nodes[i], quorate, view, view_len);
+    }
+
+    pump(nodes, count);
+}
+
+static void read_id(memdb_t *memdb, char id[CLUSTER_ID_FILE_SIZE]) {
+    gpointer data = NULL;
+
+    ck_assert(memdb_read(memdb, "cluster-id", &data) == CLUSTER_ID_FILE_SIZE);
+    ck_assert(cluster_id_is_valid(data, CLUSTER_ID_FILE_SIZE));
+    memcpy(id, data, CLUSTER_ID_FILE_SIZE);
+
+    g_free(data);
+}
+
+static void assert_id(memdb_t *memdb, const char *expected) {
+    char id[CLUSTER_ID_FILE_SIZE];
+
+    read_id(memdb, id);
+    ck_assert(memcmp(id, expected, CLUSTER_ID_FILE_SIZE) == 0);
+}
+
+static void write_id(memdb_t *memdb, const char *id, guint32 writer, guint32 mtime) {
+    ck_assert(memdb_create(memdb, "cluster-id", writer, mtime) == 0);
+    ck_assert(memdb_write(memdb, "cluster-id", writer, mtime, id, strlen(id), 0, 1) == strlen(id));
+}
+
+static void other_writes(memdb_t *memdb, int count) {
+    ck_assert(memdb_create(memdb, "other", 2, 100) == 0);
+    for (int i = 0; i < count; i++) {
+        ck_assert(memdb_write(memdb, "other", 2, 100, "x", 1, 0, 1) == 1);
+    }
+}
+
+START_TEST(test_single_node_cluster) {
+    memdb_t *dbs[] = {open_db("1.db")};
+    dfsm_t *nodes[] = {start_node(dbs[0], 1)};
+    uint32_t view[] = {1};
+    char id[CLUSTER_ID_FILE_SIZE];
+
+    /* without the configured nodes known, nothing happens */
+    set_quorum(nodes, 1, 1, view, 1);
+    set_group(nodes, 1);
+    ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+    publish_config_nodes(nodes, 1, 1);
+    read_id(dbs[0], id);
+
+    stop_nodes(nodes, dbs, 1);
+}
+END_TEST
+
+/* removing nodes from the configuration can lift the node majority blocker */
+START_TEST(test_config_nodes_removed) {
+    memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+    dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+    uint32_t view[] = {1, 2};
+    char id[CLUSTER_ID_FILE_SIZE];
+
+    publish_config_nodes(nodes, 2, 4);
+    set_quorum(nodes, 2, 1, view, 2);
+    set_group(nodes, 2);
+    ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+    publish_config_nodes(nodes, 2, 2);
+    read_id(dbs[0], id);
+    assert_id(dbs[1], id);
+
+    stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+START_TEST(test_only_lowest_generates) {
+    memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+    dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+    uint32_t view[] = {1, 2};
+    char id1[CLUSTER_ID_FILE_SIZE], id2[CLUSTER_ID_FILE_SIZE];
+
+    set_config_nodes(2);
+    set_quorum(nodes, 2, 1, view, 2);
+    set_group(nodes, 2);
+
+    read_id(dbs[0], id1);
+    read_id(dbs[1], id2);
+    ck_assert(memcmp(id1, id2, CLUSTER_ID_FILE_SIZE) == 0);
+
+    /* written by node 1, the lowest */
+    memdb_tree_entry_t *te = memdb_getattr(dbs[1], "cluster-id");
+    ck_assert(te != NULL);
+    ck_assert(te->writer == 1);
+    memdb_tree_entry_free(te);
+
+    stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* A node with an outdated database starts first and is alone in the group, while corosync, with
+ * the other node running, makes it quorate. It must not publish an ID of its own. */
+START_TEST(test_stale_node_alone) {
+    memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+    write_id(dbs[1], "abcdefghijklmnopqrst\n", 2, 100);
+    other_writes(dbs[1], 10);
+
+    dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+    uint32_t view[] = {1, 2};
+
+    set_config_nodes(2);
+    set_quorum(nodes, 1, 1, view, 2);
+    set_group(nodes, 1);
+    ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+    set_group(nodes, 2);
+    assert_id(dbs[0], "abcdefghijklmnopqrst\n");
+    assert_id(dbs[1], "abcdefghijklmnopqrst\n");
+
+    stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* Like above, but the outdated node holds the empty file of an interrupted creation. Filling it
+ * would make its database the newest one, so its ID would replace the established one. */
+START_TEST(test_stale_node_alone_with_empty_file) {
+    memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+    ck_assert(memdb_create(dbs[0], "cluster-id", 1, 100) == 0);
+    write_id(dbs[1], "abcdefghijklmnopqrst\n", 1, 100);
+
+    dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+    uint32_t view[] = {1, 2};
+
+    set_config_nodes(2);
+    set_quorum(nodes, 1, 1, view, 2);
+    set_group(nodes, 1);
+    ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_INVALID);
+
+    set_group(nodes, 2);
+    assert_id(dbs[0], "abcdefghijklmnopqrst\n");
+    assert_id(dbs[1], "abcdefghijklmnopqrst\n");
+
+    stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* Quorate through a QDevice, or two_node, last_man_standing or lowered expected votes, the
+ * corosync membership of a lone outdated node is just itself. */
+START_TEST(test_lone_node_without_node_majority) {
+    memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+    write_id(dbs[1], "abcdefghijklmnopqrst\n", 2, 100);
+
+    dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+    uint32_t view1[] = {1}, view12[] = {1, 2};
+
+    set_config_nodes(2);
+    set_quorum(nodes, 1, 1, view1, 1);
+    set_group(nodes, 1);
+    ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+    set_quorum(nodes, 2, 1, view12, 2);
+    set_group(nodes, 2);
+    assert_id(dbs[0], "abcdefghijklmnopqrst\n");
+    assert_id(dbs[1], "abcdefghijklmnopqrst\n");
+
+    stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+START_TEST(test_half_of_the_nodes) {
+    memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+    dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+    uint32_t view[] = {1, 2};
+
+    set_config_nodes(4);
+    set_quorum(nodes, 2, 1, view, 2);
+    set_group(nodes, 2);
+    ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+    ck_assert(cluster_id_state(dbs[1]) == CLUSTER_ID_MISSING);
+
+    stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* Neither gaining quorum nor a corosync membership change causes a sync of the group, so the
+ * generation must not wait for the next one. */
+START_TEST(test_quorum_change_triggers) {
+    memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+    dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+    uint32_t view12[] = {1, 2}, view123[] = {1, 2, 3};
+    char id[CLUSTER_ID_FILE_SIZE];
+
+    set_config_nodes(3);
+    set_quorum(nodes, 2, 0, view12, 2);
+    set_group(nodes, 2);
+    ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+    /* node 3 only runs corosync */
+    set_quorum(nodes, 2, 1, view123, 3);
+    ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+    /* node 3 left corosync again, the group did not change */
+    set_quorum(nodes, 2, 1, view12, 2);
+    read_id(dbs[0], id);
+    assert_id(dbs[1], id);
+
+    stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* node 3 joins corosync and the group, in both orders the notifications can arrive */
+static void third_node_joins(gboolean quorum_first) {
+    memdb_t *dbs[] = {open_db("1.db"), open_db("2.db"), open_db("3.db")};
+    dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2), start_node(dbs[2], 3)};
+    uint32_t view12[] = {1, 2}, view123[] = {1, 2, 3};
+    char id[CLUSTER_ID_FILE_SIZE];
+
+    set_config_nodes(5);
+    set_quorum(nodes, 2, 0, view12, 2);
+    set_group(nodes, 2);
+
+    if (quorum_first) {
+        set_quorum(nodes, 3, 1, view123, 3);
+        ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+        set_group(nodes, 3);
+    } else {
+        set_group(nodes, 3);
+        ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+        set_quorum(nodes, 3, 1, view123, 3);
+    }
+
+    read_id(dbs[0], id);
+    assert_id(dbs[1], id);
+    assert_id(dbs[2], id);
+
+    stop_nodes(nodes, dbs, 3);
+}
+
+START_TEST(test_third_node_joins_quorum_first) { third_node_joins(TRUE); }
+END_TEST
+
+START_TEST(test_third_node_joins_group_first) { third_node_joins(FALSE); }
+END_TEST
+
+/* attempts made before the messages of the first one got delivered must write the same ID */
+START_TEST(test_pending_id_reused) {
+    memdb_t *dbs[] = {open_db("1.db")};
+    dfsm_t *nodes[] = {start_node(dbs[0], 1)};
+    uint32_t view[] = {1};
+    char first[CLUSTER_ID_FILE_SIZE];
+
+    set_config_nodes(1);
+    set_quorum(nodes, 1, 1, view, 1);
+
+    change_group(nodes, 1);
+    ck_assert(g_queue_get_length(&packets) == 2);
+    ck_assert(have_pending_cluster_id);
+    memcpy(first, pending_cluster_id, CLUSTER_ID_FILE_SIZE);
+
+    dfsm_check_in_sync(nodes[0]);
+    dcdb_quorum_view_changed(nodes[0], 1, view, 1);
+    ck_assert(g_queue_get_length(&packets) == 6);
+
+    pump(nodes, 1);
+    assert_id(dbs[0], first);
+
+    /* once delivered, the next check finds it valid and forgets the pending one */
+    dfsm_check_in_sync(nodes[0]);
+    ck_assert(g_queue_is_empty(&packets));
+    ck_assert(!have_pending_cluster_id);
+
+    stop_nodes(nodes, dbs, 1);
+}
+END_TEST
+
+START_TEST(test_lowest_repairs_empty_file) {
+    memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+    ck_assert(memdb_create(dbs[0], "cluster-id", 3, 100) == 0);
+    ck_assert(memdb_create(dbs[1], "cluster-id", 3, 100) == 0);
+
+    dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+    uint32_t view[] = {1, 2};
+    char id[CLUSTER_ID_FILE_SIZE];
+
+    set_config_nodes(2);
+    set_quorum(nodes, 2, 1, view, 2);
+    set_group(nodes, 2);
+
+    read_id(dbs[0], id);
+    assert_id(dbs[1], id);
+
+    stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* an ID with only 16 characters is malformed like anything else */
+START_TEST(test_lowest_replaces_short_id) {
+    memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+    write_id(dbs[0], "abcdefghijklmnop\n", 1, 100);
+    write_id(dbs[1], "abcdefghijklmnop\n", 1, 100);
+
+    dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+    uint32_t view[] = {1, 2};
+    char id[CLUSTER_ID_FILE_SIZE];
+
+    set_config_nodes(2);
+    set_quorum(nodes, 2, 1, view, 2);
+    set_group(nodes, 2);
+
+    read_id(dbs[0], id);
+    assert_id(dbs[1], id);
+
+    stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+static void add_test(Suite *s, _TEST_TYPE tf, const char *name) {
+    TCase *tc = tcase_create(name);
+    tcase_add_test(tc, tf);
+    suite_add_tcase(s, tc);
+}
+
+static void remove_tmpdir(void) {
+    GDir *dir = g_dir_open(tmpdir, 0, NULL);
+    if (dir) {
+        const char *name;
+        while ((name = g_dir_read_name(dir))) {
+            char *filename = g_build_filename(tmpdir, name, NULL);
+            unlink(filename);
+            g_free(filename);
+        }
+        g_dir_close(dir);
+    }
+
+    rmdir(tmpdir);
+    g_free(tmpdir);
+}
+
+int main(void) {
+    cfs_status_init();
+
+    if (!(tmpdir = g_dir_make_tmp("check_dcdb-XXXXXX", NULL))) {
+        fprintf(stderr, "unable to create a temporary directory\n");
+        return EXIT_FAILURE;
+    }
+
+    Suite *s = suite_create("dcdb");
+
+    add_test(s, test_single_node_cluster, "single_node_cluster");
+    add_test(s, test_config_nodes_removed, "config_nodes_removed");
+    add_test(s, test_only_lowest_generates, "only_lowest_generates");
+    add_test(s, test_stale_node_alone, "stale_node_alone");
+    add_test(s, test_stale_node_alone_with_empty_file, "stale_node_alone_with_empty_file");
+    add_test(s, test_lone_node_without_node_majority, "lone_node_without_node_majority");
+    add_test(s, test_half_of_the_nodes, "half_of_the_nodes");
+    add_test(s, test_quorum_change_triggers, "quorum_change_triggers");
+    add_test(s, test_third_node_joins_quorum_first, "third_node_joins_quorum_first");
+    add_test(s, test_third_node_joins_group_first, "third_node_joins_group_first");
+    add_test(s, test_pending_id_reused, "pending_id_reused");
+    add_test(s, test_lowest_repairs_empty_file, "lowest_repairs_empty_file");
+    add_test(s, test_lowest_replaces_short_id, "lowest_replaces_short_id");
+
+    SRunner *sr = srunner_create(s);
+    srunner_run_all(sr, CK_NORMAL);
+    int number_failed = srunner_ntests_failed(sr);
+    srunner_free(sr);
+
+    remove_tmpdir();
+
+    return (number_failed == 0) ? EXIT_SUCCESS : EXIT_FAILURE;
+}
diff --git a/src/pmxcfs/dcdb.c b/src/pmxcfs/dcdb.c
index 0965331..7a3693b 100644
--- a/src/pmxcfs/dcdb.c
+++ b/src/pmxcfs/dcdb.c
@@ -38,6 +38,7 @@
 #include <unistd.h>
 
 #include "cfs-utils.h"
+#include "cluster-id.h"
 #include "dcdb.h"
 #include "loop.h"
 #include "status.h"
@@ -985,6 +986,157 @@ leave:
     goto ret;
 }
 
+/* node IDs of the current corosync membership while quorate, loop thread only */
+static uint32_t *quorum_view;
+static size_t quorum_view_len;
+
+/* Kept until the file holds a valid ID, so that an attempt made before the messages of an earlier
+ * one got delivered writes the same ID again instead of replacing it. Loop thread only. */
+static char pending_cluster_id[CLUSTER_ID_FILE_SIZE];
+static gboolean have_pending_cluster_id;
+
+/* why the last check did not generate, to log only changes, loop thread only */
+static char *blocked_reason;
+
+static void set_blocked_reason(char *reason) {
+    if (reason && g_strcmp0(reason, blocked_reason) != 0) {
+        cfs_message("not generating a cluster ID yet, %s", reason);
+    }
+
+    g_free(blocked_reason);
+    blocked_reason = reason;
+}
+
+/* An ID is only generated while more than half of the configured nodes are in the corosync
+ * membership and all of those are in sync in this group. Majorities of the same configured node
+ * set overlap. Together with normal database transfer on join, this prevents a stale minority from
+ * generating an ID before it synchronizes with the other nodes. Counting nodes rather than relying
+ * on quorum alone keeps this with a QDevice, two_node, last_man_standing or lowered expected votes,
+ * at the price of not generating while half of the nodes or more are down. Returns why generating
+ * is blocked. */
+static char *generation_blocker(dfsm_t *dfsm) {
+    if (!quorum_view_len || !cfs_is_quorate()) {
+        return g_strdup("not quorate");
+    }
+
+    guint nodes = cfs_clinfo_node_count();
+    if (!nodes) {
+        return g_strdup("the configured nodes are not known");
+    }
+
+    if (quorum_view_len * 2 <= nodes) {
+        return g_strdup_printf(
+            "only %zu of %u configured nodes are in the corosync membership", quorum_view_len, nodes
+        );
+    }
+
+    GString *missing = NULL;
+    for (size_t i = 0; i < quorum_view_len; i++) {
+        if (!dfsm_nodeid_is_member(dfsm, quorum_view[i])) {
+            if (!missing) {
+                missing = g_string_new(NULL);
+            } else {
+                g_string_append(missing, ", ");
+            }
+            g_string_append_printf(missing, "%u", quorum_view[i]);
+        }
+    }
+
+    if (missing) {
+        char *reason = g_strdup_printf(
+            "node ID(s) %s of the corosync membership are not in the pmxcfs group", missing->str
+        );
+        g_string_free(missing, TRUE);
+        return reason;
+    }
+
+    return NULL;
+}
+
+/* Only the node with the lowest ID generates the cluster ID, like only that node expires locks, so
+ * that no two nodes write different IDs. The loop thread cannot wait for the result of its own
+ * messages, so CREATE and WRITE go out back to back, the messages of one sender stay in order. */
+static void dcdb_in_sync(dfsm_t *dfsm, gpointer data) {
+    g_return_if_fail(dfsm != NULL);
+    g_return_if_fail(data != NULL);
+
+    memdb_t *memdb = (memdb_t *)data;
+
+    if (!dfsm_lowest_nodeid(dfsm)) {
+        return;
+    }
+
+    cluster_id_state_t state = cluster_id_state(memdb);
+    if (state == CLUSTER_ID_VALID) {
+        have_pending_cluster_id = FALSE;
+        set_blocked_reason(NULL);
+        return;
+    }
+
+    if (state == CLUSTER_ID_NOT_A_FILE) {
+        set_blocked_reason(g_strdup_printf("'%s' is not a regular file", CFS_CLUSTER_ID_FN));
+        return;
+    }
+
+    char *reason = generation_blocker(dfsm);
+    set_blocked_reason(reason);
+    if (reason) {
+        return;
+    }
+
+    if (!have_pending_cluster_id) {
+        if (!cluster_id_generate(pending_cluster_id)) {
+            return;
+        }
+        have_pending_cluster_id = TRUE;
+    }
+
+    if (state == CLUSTER_ID_MISSING &&
+        dcdb_send_fuse_message_full(
+            dfsm, DCDB_MESSAGE_CFS_CREATE, CFS_CLUSTER_ID_FN, NULL, NULL, 0, 0, 0, NULL
+        ) != CS_OK) {
+        return;
+    }
+
+    if (dcdb_send_fuse_message_full(
+            dfsm, DCDB_MESSAGE_CFS_WRITE, CFS_CLUSTER_ID_FN, NULL, pending_cluster_id,
+            CLUSTER_ID_FILE_SIZE, 0, 1, NULL
+        ) != CS_OK) {
+        return;
+    }
+
+    cfs_message("generating cluster ID %.*s", CLUSTER_ID_LEN, pending_cluster_id);
+}
+
+void dcdb_quorum_view_changed(
+    gpointer data, uint32_t quorate, const uint32_t *nodeids, size_t count
+) {
+    dfsm_t *dfsm = (dfsm_t *)data;
+
+    g_return_if_fail(dfsm != NULL);
+
+    g_free(quorum_view);
+    quorum_view = NULL;
+    quorum_view_len = 0;
+
+    if (quorate && count) {
+        quorum_view = g_memdup2(nodeids, count * sizeof(*nodeids));
+        quorum_view_len = count;
+    }
+
+    /* neither gaining quorum nor a node that only joins corosync causes a sync of the group */
+    dfsm_check_in_sync(dfsm);
+}
+
+void dcdb_config_changed(gpointer data) {
+    dfsm_t *dfsm = (dfsm_t *)data;
+
+    g_return_if_fail(dfsm != NULL);
+
+    /* the configured nodes can become known or fewer without any change of the group */
+    dfsm_check_in_sync(dfsm);
+}
+
 static dfsm_callbacks_t dcdb_dfsm_callbacks = {
     .dfsm_deliver_fn = dcdb_deliver,
     .dfsm_get_state_fn = dcdb_get_state,
@@ -993,6 +1145,7 @@ static dfsm_callbacks_t dcdb_dfsm_callbacks = {
     .dfsm_commit_fn = dcdb_commit,
     .dfsm_cleanup_fn = dcdb_cleanup,
     .dfsm_checksum_fn = dcdb_checksum,
+    .dfsm_in_sync_fn = dcdb_in_sync,
 };
 
 dfsm_t *dcdb_new(memdb_t *memdb) {
diff --git a/src/pmxcfs/dcdb.h b/src/pmxcfs/dcdb.h
index d637cc7..de56fc2 100644
--- a/src/pmxcfs/dcdb.h
+++ b/src/pmxcfs/dcdb.h
@@ -61,4 +61,12 @@ int dcdb_send_fuse_message(
 
 void dcdb_send_unlock(dfsm_t *dfsm, const char *path, const guchar csum[32], gboolean request);
 
+/* quorum view listener, data is the dcdb dfsm */
+void dcdb_quorum_view_changed(
+    gpointer data, uint32_t quorate, const uint32_t *nodeids, size_t count
+);
+
+/* confdb listener, data is the dcdb dfsm */
+void dcdb_config_changed(gpointer data);
+
 #endif /* _PVE_DCDB_H_ */
diff --git a/src/pmxcfs/pmxcfs.c b/src/pmxcfs/pmxcfs.c
index 2ab48dc..ce9d650 100644
--- a/src/pmxcfs/pmxcfs.c
+++ b/src/pmxcfs/pmxcfs.c
@@ -1019,11 +1019,11 @@ int main(int argc, char *argv[]) {
 
     if (dcdb) {
 
-        service_quorum = service_quorum_new(NULL, NULL);
+        service_quorum = service_quorum_new(dcdb_quorum_view_changed, dcdb);
 
         cfs_loop_add_service(corosync_loop, service_quorum, QB_LOOP_HIGH);
 
-        service_confdb = service_confdb_new(NULL, NULL);
+        service_confdb = service_confdb_new(dcdb_config_changed, dcdb);
 
         cfs_loop_add_service(corosync_loop, service_confdb, QB_LOOP_MED);
 
diff --git a/src/pmxcfs/status.c b/src/pmxcfs/status.c
index 8538d8b..f16dc20 100644
--- a/src/pmxcfs/status.c
+++ b/src/pmxcfs/status.c
@@ -2000,6 +2000,17 @@ gboolean cfs_is_quorate(void) {
     return res;
 }
 
+guint cfs_clinfo_node_count(void) {
+    g_mutex_lock(&mutex);
+    guint count = 0;
+    if (cfs_status.clinfo && cfs_status.clinfo->nodes_byid) {
+        count = g_hash_table_size(cfs_status.clinfo->nodes_byid);
+    }
+    g_mutex_unlock(&mutex);
+
+    return count;
+}
+
 void cfs_set_quorate(uint32_t quorate, gboolean quiet) {
     g_mutex_lock(&mutex);
 
diff --git a/src/pmxcfs/status.h b/src/pmxcfs/status.h
index 709ef64..dfc134f 100644
--- a/src/pmxcfs/status.h
+++ b/src/pmxcfs/status.h
@@ -69,6 +69,9 @@ void cfs_set_quorate(uint32_t quorate, gboolean quiet);
 
 gboolean cfs_is_quorate(void);
 
+/* number of nodes in the corosync configuration, 0 while not known yet */
+guint cfs_clinfo_node_count(void);
+
 GHashTable *vmlist_hash_new(void);
 
 gboolean vmlist_hash_insert_vm(
-- 
2.47.3





  parent reply	other threads:[~2026-11-18  4:41 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18  4:37 [PATCH cluster/guest-common/qemu-server/container/manager/docs v3 00/38] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 01/38] pmxcfs: reject changes to the cluster ID file from user space Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 02/38] pmxcfs: generate a cluster ID on standalone nodes Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 03/38] pmxcfs: dfsm: add a callback that runs once all members are in sync Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 04/38] pmxcfs: quorum: pass the corosync membership to an optional listener Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 05/38] pmxcfs: confdb: call an optional listener when the nodes get published Thomas Lamprecht
2026-11-18  4:37 ` Thomas Lamprecht [this message]
2026-11-18  4:37 ` [PATCH cluster v3 07/38] cluster: add helper to read the cluster ID Thomas Lamprecht
2026-11-18  4:37 ` [PATCH cluster v3 08/38] pmxcfs: allow removing the cluster ID file in forced local mode Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 09/38] guest identity: add helpers for the guest uid and registration time Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 10/38] abstract config: assign a guest identity and keep it on config writes Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 11/38] abstract config: let the VMID reservation return the previous state Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 12/38] guest identity: allow taking over a uid that is unused in the cluster Thomas Lamprecht
2026-11-18  4:37 ` [PATCH guest-common v3 13/38] migrate: add shared helpers for the identity command of the migration tunnel Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 14/38] api: restore: release the VMID if a restore to an unused VMID fails Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 15/38] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 16/38] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 17/38] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 18/38] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 19/38] api: status: return the guest uid Thomas Lamprecht
2026-11-18  4:37 ` [PATCH qemu-server v3 20/38] tests: remote migration: check a failed command and an early tunnel end Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 21/38] config: add read-only meta property with the creation time Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 22/38] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 23/38] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 24/38] restore: take over the guest uid from the backup where possible Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 25/38] restore: refuse overwriting a container with another guest's backup Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 26/38] test: remote migration: check the cleanup of the target after failures Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 27/38] api: remote migration: take over the guest uid from the source Thomas Lamprecht
2026-11-18  4:37 ` [PATCH container v3 28/38] api: status: return the guest uid Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 29/38] api: cluster resources: add guest uid and registration time Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 30/38] api: cluster status: add cluster ID Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 31/38] ui: restore: allow overwriting a guest with a backup of another guest Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 32/38] ui: migrate: refuse target nodes that would drop the guest identity Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 33/38] ui: guest summary: show " Thomas Lamprecht
2026-11-18  4:37 ` [PATCH manager v3 34/38] api: bulk migrate: refuse targets that would drop " Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 35/38] vzdump: describe the guest identity of VMs and containers Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 36/38] pvecm: document the cluster ID and how to get a new one Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 37/38] qm: configuration: describe the read-only meta property Thomas Lamprecht
2026-11-18  4:37 ` [PATCH docs v3 38/38] pct: " Thomas Lamprecht
2026-11-18 13:49 ` [PATCH cluster/guest-common/qemu-server/container/manager/docs v3 00/38] fix #7846: guest identity (uid) and cluster ID Dominik Csapak
2026-11-18 14:13   ` Thomas Lamprecht
2026-11-19 10:38 ` superseded: " Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118043817.1659212-7-t.lamprecht@proxmox.com \
    --to=t.lamprecht@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal