From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH cluster v5 05/41] pmxcfs: dcdb: generate the cluster ID once the cluster is in sync
Date: Thu, 19 Nov 2026 20:18:37 +0100 [thread overview]
Message-ID: <20261119191914.2336934-6-t.lamprecht@proxmox.com> (raw)
In-Reply-To: <20261119191914.2336934-1-t.lamprecht@proxmox.com>
Let pmxcfs generate the cluster ID if it is missing or invalid, also for
clusters that get upgraded. Only the member with the lowest node ID
generates it, so that the members of a group do not generate competing
IDs. pmxcfs owns /etc/pve and reserves this path, so replacing other
content found in a file there on an upgrade is intended.
Being quorate is not enough as a condition. During a cold boot, corosync
can be quorate before all pmxcfs instances have joined their group. A
node with an outdated database could then generate an ID before it
learns the existing one. If the database versions are equal, the newer
modification time of its database could make that ID win the next
synchronization.
So require all of the following: the cluster is quorate, more than half
of the configured nodes are in the corosync membership, and every node
of that membership is a synchronized member of the pmxcfs group. Match
the configured nodes by their ID, because the membership can still
contain a node that just got removed from the configuration.
Two majorities of the same set of configured nodes always have a node in
common. Together with the normal database transfer on join, this keeps
an outdated minority from generating an ID before it has synchronized
with the other nodes. Count nodes and not votes, so that this also holds
with a QDevice, two_node, last_man_standing or lowered expected votes.
The downside is that no ID gets generated while half or more of the
nodes are down.
This does not keep the ordinary database synchronization from selecting
a database that lacks the ID for other reasons. The ID gets generated
again in that case.
The generator can get cut off from the other nodes right after its
write, when only its own database has the ID. It then shows that ID to
local readers, while the other nodes generate another one. When the
generator rejoins, it normally gets the ID of the others. Its own ID
replaces theirs only if nothing else changed in the meantime and its
clock is ahead by the seconds that the others needed to notice the cut.
Its database then wins the tie between equal versions.
Log why the generation is blocked whenever the reason changes, and name
the nodes whose pmxcfs is missing.
Run the check right after each synchronization, and then every ten
seconds for as long as the group stays in sync. For that, the timer of
the dcdb service now ticks every ten seconds. The check only looks at
the current state. So the next tick notices every change of the quorum,
the corosync membership, the configured nodes or the file, and also a
send that failed. None of these needs a notification of its own, which
could be forgotten. Nothing needs the ID sooner, as readers handle a
missing ID as not generated yet. A tick costs two database lookups on
the lowest node and one comparison on the others, and the event loop
wakes up every second anyway.
The generation sends the existing ordered file messages and does not
wait for their result on the loop thread. An interrupted creation can
leave an empty file, which a later attempt fills. The generator uses the
same ID again until a check sees a valid ID in the file. A tick that
comes before the messages of an earlier attempt got delivered therefore
does not replace that ID with another one. A directory at the path is
left untouched and logged.
Older nodes can replicate the file, as this needs no new message type.
If the lowest member runs an older version, the generation waits until
that node is upgraded or leaves.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
---
changes v4 -> v5:
* comment of generation_blocker(): name what counting nodes keeps, and
that it returns NULL if nothing blocks
changes v3 -> v4:
* check right after a sync and then every ten seconds while the group
stays in sync, by the current state alone, instead of on quorum,
membership and configuration events: a failed send or a removed or
overwritten file caused no event and waited for the hourly timer
* the confdb listener patch of v3 is gone with that, and the membership
comes from the status
* count only members of the corosync membership that are configured
nodes, as the membership can still have a node that just got removed
from the configuration
* tests: timer ticks instead of events, with new cases for a member that
is not configured, a tick during a sync, failed sends, a generator
that leaves between creating and filling the file, a removed directory
and an overwritten ID; each test instance has generation state of its
own
* commit message: other content at the path gets replaced on purpose,
and what happens to the ID of a generator that got cut off
* commit message and comments: plainer wording, same content
changes v2 -> v3:
* check again when the configured nodes get published or change, with
the new confdb listener, instead of waiting for the next quorum or
group change or the hourly timer; test for nodes removed from the
configuration
* comments: quorum is still required next to the node majority, and the
test instances share the per-process state
changes v1 -> v2:
* rebase onto the dfsm rework: the test nodes have to count as joined to
send
src/pmxcfs/Makefile | 10 +-
src/pmxcfs/check_dcdb.c | 876 ++++++++++++++++++++++++++++++++++++++++
src/pmxcfs/dcdb.c | 141 +++++++
src/pmxcfs/dcdb.h | 5 +-
src/pmxcfs/pmxcfs.c | 2 +-
src/pmxcfs/status.c | 20 +
src/pmxcfs/status.h | 6 +
7 files changed, 1055 insertions(+), 5 deletions(-)
create mode 100644 src/pmxcfs/check_dcdb.c
diff --git a/src/pmxcfs/Makefile b/src/pmxcfs/Makefile
index cd93e72..64d6d6f 100644
--- a/src/pmxcfs/Makefile
+++ b/src/pmxcfs/Makefile
@@ -17,7 +17,7 @@ ARFLAGS = crs
.c.o:
$(CC) $(CFLAGS) -c -o $@ $< -MMD -MT $@ -MF $@.d
-all: pmxcfs create_pmxcfs_db logtest check_memdb
+all: pmxcfs create_pmxcfs_db logtest check_memdb check_dcdb
libpmxcfs.a: logger.o loop.o server.o status.o confdb.o quorum.o dcdb.o dfsm.o
libpmxcfs.a: cfs-plug.o cfs-plug-memdb.o cfs-plug-link.o cfs-plug-func.o
@@ -41,6 +41,9 @@ logtest2: logtest2.o libpmxcfs.a
check_memdb: check_memdb.o libpmxcfs.a
$(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS) $(shell pkg-config --libs check)
+check_dcdb: check_dcdb.o libpmxcfs.a
+ $(CC) $(CFLAGS) -o $@ $^ $(LDFLAGS) $(shell pkg-config --libs check)
+
pmxcfs.8:
-include /usr/share/pve-doc-generator/pve-doc-generator.mk
@@ -57,11 +60,12 @@ format:
git ls-files ':/src/pmxcfs/*.[ch]' | xargs -n2 -P0 -- clang-format -i
.PHONY: check
-check: check_memdb
+check: check_memdb check_dcdb
./check_memdb
+ ./check_dcdb
.PHONY: clean
clean:
- rm -f *.o *.o.d *.a pmxcfs create_pmxcfs_db check_memdb logtest pmxcfs.8
+ rm -f *.o *.o.d *.a pmxcfs create_pmxcfs_db check_memdb check_dcdb logtest pmxcfs.8
-include *.o.d
diff --git a/src/pmxcfs/check_dcdb.c b/src/pmxcfs/check_dcdb.c
new file mode 100644
index 0000000..fec0044
--- /dev/null
+++ b/src/pmxcfs/check_dcdb.c
@@ -0,0 +1,876 @@
+/*
+ Copyright (C) 2026 Proxmox Server Solutions GmbH
+
+ This program is free software: you can redistribute it and/or modify
+ it under the terms of the GNU Affero General Public License as published by
+ the Free Software Foundation, either version 3 of the License, or
+ (at your option) any later version.
+
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ GNU Affero General Public License for more details.
+
+ You should have received a copy of the GNU Affero General Public License
+ along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+ Author: Thomas Lamprecht <t.lamprecht@proxmox.com>
+
+*/
+
+/* Runs several dcdb instances in one process, each with its own memdb, and replaces the CPG
+ * transport with an ordered in-process queue, so that membership changes, syncs and the cluster ID
+ * generation can be tested without corosync. The sources are included to reach the static CPG
+ * callbacks and the generation state of dcdb.c. That state exists once per pmxcfs process, so each
+ * call into an instance swaps in a copy of its own. The quorate flag, the corosync membership and
+ * the configured nodes of status.c stay common to all instances. */
+
+#include <check.h>
+#include <stdint.h>
+
+#include "dfsm.c"
+#undef G_LOG_DOMAIN
+#include "dcdb.c"
+
+#if CHECK_MAJOR_VERSION == 0 && CHECK_MINOR_VERSION <= 10
+#define _TEST_TYPE TFun
+#else
+#define _TEST_TYPE const TTest *
+#endif
+
+cfs_t cfs = {
+ .debug = 0,
+ .nodename = "testnode",
+};
+
+typedef struct {
+ uint32_t nodeid;
+ uint32_t pid;
+ size_t len;
+ void *data;
+} packet_t;
+
+static GQueue packets = G_QUEUE_INIT;
+
+/* the send with this number fails, counted from 1 since send_nr got reset, 0 for none */
+static int fail_send_nr;
+static int send_nr;
+
+typedef struct {
+ dfsm_t *dfsm;
+ cluster_id_gen_t gen;
+} member_state_t;
+
+static member_state_t member_states[8];
+/* the instance whose state the globals of dcdb.c hold */
+static member_state_t *current_state;
+
+static void save_state(member_state_t *state) { state->gen = cluster_id_gen; }
+
+static void load_state(const member_state_t *state) { cluster_id_gen = state->gen; }
+
+static member_state_t *find_state(dfsm_t *dfsm) {
+ for (size_t i = 0; i < G_N_ELEMENTS(member_states); i++) {
+ if (member_states[i].dfsm == dfsm) {
+ return &member_states[i];
+ }
+ }
+
+ ck_abort_msg("no state for this instance");
+ return NULL;
+}
+
+/* to call before anything that can run code of dcdb.c for this instance */
+static void enter(dfsm_t *dfsm) {
+ member_state_t *state = find_state(dfsm);
+
+ if (state != current_state) {
+ if (current_state) {
+ save_state(current_state);
+ }
+ load_state(state);
+ current_state = state;
+ }
+}
+
+/* the generation state of an instance as of its last call */
+static const member_state_t *state_of(dfsm_t *dfsm) {
+ if (current_state) {
+ save_state(current_state);
+ }
+
+ return find_state(dfsm);
+}
+
+cs_error_t cpg_context_get(cpg_handle_t handle, void **context) {
+ *context = (void *)(uintptr_t)handle;
+ return CS_OK;
+}
+
+cs_error_t cpg_mcast_joined(
+ cpg_handle_t handle, cpg_guarantee_t guarantee, const struct iovec *iov, unsigned int iov_len
+) {
+ dfsm_t *sender = (dfsm_t *)(uintptr_t)handle;
+
+ if (++send_nr == fail_send_nr) {
+ return CS_ERR_LIBRARY;
+ }
+
+ packet_t *p = g_new0(packet_t, 1);
+
+ p->nodeid = sender->nodeid;
+ p->pid = sender->pid;
+
+ for (unsigned int i = 0; i < iov_len; i++) {
+ p->len += iov[i].iov_len;
+ }
+
+ p->data = g_malloc(p->len);
+
+ size_t pos = 0;
+ for (unsigned int i = 0; i < iov_len; i++) {
+ if (iov[i].iov_len) {
+ memcpy((char *)p->data + pos, iov[i].iov_base, iov[i].iov_len);
+ }
+ pos += iov[i].iov_len;
+ }
+
+ g_queue_push_tail(&packets, p);
+
+ return CS_OK;
+}
+
+/* delivers the next queued message to the given nodes, FALSE if there is none */
+static gboolean deliver_next(dfsm_t **nodes, int count) {
+ packet_t *p = g_queue_pop_head(&packets);
+ if (!p) {
+ return FALSE;
+ }
+
+ for (int i = 0; i < count; i++) {
+ enter(nodes[i]);
+ dfsm_cpg_deliver_callback(
+ nodes[i]->cpg_handle, &nodes[i]->cpg_group_name, p->nodeid, p->pid, p->data, p->len
+ );
+ ck_assert(dfsm_get_mode(nodes[i]) < DFSM_ERROR_MODE_START);
+ }
+
+ g_free(p->data);
+ g_free(p);
+
+ return TRUE;
+}
+
+/* delivers all queued messages, including those sent meanwhile, to the given nodes in order */
+static void pump(dfsm_t **nodes, int count) {
+ int rounds = 0;
+
+ while (deliver_next(nodes, count)) {
+ ck_assert(++rounds < 1000);
+ }
+}
+
+/* the messages of a node that left before they got delivered */
+static void drop_packets(void) {
+ packet_t *p;
+
+ while ((p = g_queue_pop_head(&packets))) {
+ g_free(p->data);
+ g_free(p);
+ }
+}
+
+/* the databases of all tests, removed at the end whatever the tests left behind */
+static char *tmpdir;
+
+static memdb_t *open_db(const char *name) {
+ char *filename = g_build_filename(tmpdir, name, NULL);
+ unlink(filename);
+ memdb_t *memdb = memdb_open(filename);
+ g_free(filename);
+ ck_assert(memdb != NULL);
+
+ /* a node in cluster mode always has a corosync.conf */
+ ck_assert(memdb_create(memdb, "corosync.conf", 0, 100) == 0);
+
+ return memdb;
+}
+
+static dfsm_t *start_node(memdb_t *memdb, uint32_t nodeid) {
+ dfsm_t *dfsm = dcdb_new(memdb);
+ ck_assert(dfsm != NULL);
+
+ dfsm->cpg_handle = (uintptr_t)dfsm;
+ dfsm->joined = TRUE; /* messages are only sent while joined */
+ dfsm->nodeid = nodeid;
+ dfsm->pid = 100 + nodeid;
+
+ member_state_t *state = find_state(NULL); /* a free slot */
+ state->dfsm = dfsm;
+
+ return dfsm;
+}
+
+static void stop_nodes(dfsm_t **nodes, memdb_t **dbs, int count) {
+ for (int i = 0; i < count; i++) {
+ member_state_t *state = find_state(nodes[i]);
+ if (state == current_state) {
+ save_state(state);
+ load_state(&(member_state_t){0});
+ current_state = NULL;
+ }
+ g_free(state->gen.blocked_reason);
+ memset(state, 0, sizeof(*state));
+
+ nodes[i]->cpg_handle = 0;
+ dfsm_destroy(nodes[i]);
+ memdb_close(dbs[i]);
+ }
+}
+
+/* the nodes in the corosync configuration, as the confdb service reads them */
+static void set_config_node_ids(const uint32_t *ids, size_t count) {
+ cfs_clinfo_t *clinfo = cfs_clinfo_new("test", 1);
+
+ for (size_t i = 0; i < count; i++) {
+ char *name = g_strdup_printf("node%u", ids[i]);
+ cfs_clinfo_add_node(clinfo, cfs_clnode_new(name, ids[i], 1));
+ g_free(name);
+ }
+
+ cfs_status_set_clinfo(clinfo);
+}
+
+/* nodes 1 to count */
+static void set_config_nodes(uint32_t count) {
+ uint32_t ids[count];
+
+ for (uint32_t i = 0; i < count; i++) {
+ ids[i] = i + 1;
+ }
+
+ set_config_node_ids(ids, count);
+}
+
+/* a timer tick of the dcdb service on each node, without delivering what it sends */
+static void tick_only(dfsm_t **nodes, int count) {
+ for (int i = 0; i < count; i++) {
+ enter(nodes[i]);
+ dfsm_check_in_sync(nodes[i]);
+ }
+}
+
+static void tick(dfsm_t **nodes, int count) {
+ tick_only(nodes, count);
+ pump(nodes, count);
+}
+
+/* a change of the dcdb group, as seen by its members, without delivering what it causes */
+static void change_group(dfsm_t **nodes, int count) {
+ struct cpg_address members[count];
+
+ for (int i = 0; i < count; i++) {
+ members[i].nodeid = nodes[i]->nodeid;
+ members[i].pid = nodes[i]->pid;
+ members[i].reason = 0;
+ }
+
+ for (int i = 0; i < count; i++) {
+ enter(nodes[i]);
+ dfsm_cpg_confchg_callback(
+ nodes[i]->cpg_handle, &nodes[i]->cpg_group_name, members, count, NULL, 0, members, count
+ );
+ }
+}
+
+static void set_group(dfsm_t **nodes, int count) {
+ change_group(nodes, count);
+ pump(nodes, count);
+}
+
+/* a change of the corosync membership, as the quorum service records it */
+static void set_quorum(uint32_t quorate, uint32_t *view, size_t view_len) {
+ cfs_set_quorate(quorate, TRUE);
+ cfs_set_quorum_members(view, view_len);
+}
+
+static void read_id(memdb_t *memdb, char id[CLUSTER_ID_FILE_SIZE]) {
+ gpointer data = NULL;
+
+ ck_assert(memdb_read(memdb, "cluster-id", &data) == CLUSTER_ID_FILE_SIZE);
+ ck_assert(cluster_id_is_valid(data, CLUSTER_ID_FILE_SIZE));
+ memcpy(id, data, CLUSTER_ID_FILE_SIZE);
+
+ g_free(data);
+}
+
+static void assert_id(memdb_t *memdb, const char *expected) {
+ char id[CLUSTER_ID_FILE_SIZE];
+
+ read_id(memdb, id);
+ ck_assert(memcmp(id, expected, CLUSTER_ID_FILE_SIZE) == 0);
+}
+
+static void write_id(memdb_t *memdb, const char *id, guint32 writer, guint32 mtime) {
+ ck_assert(memdb_create(memdb, "cluster-id", writer, mtime) == 0);
+ ck_assert(memdb_write(memdb, "cluster-id", writer, mtime, id, strlen(id), 0, 1) == strlen(id));
+}
+
+static void other_writes(memdb_t *memdb, int count) {
+ ck_assert(memdb_create(memdb, "other", 2, 100) == 0);
+ for (int i = 0; i < count; i++) {
+ ck_assert(memdb_write(memdb, "other", 2, 100, "x", 1, 0, 1) == 1);
+ }
+}
+
+START_TEST(test_single_node_cluster) {
+ memdb_t *dbs[] = {open_db("1.db")};
+ dfsm_t *nodes[] = {start_node(dbs[0], 1)};
+ uint32_t view[] = {1};
+ char id[CLUSTER_ID_FILE_SIZE];
+
+ /* without the configured nodes known, nothing happens */
+ set_quorum(1, view, 1);
+ set_group(nodes, 1);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+ set_config_nodes(1);
+ tick(nodes, 1);
+ read_id(dbs[0], id);
+
+ stop_nodes(nodes, dbs, 1);
+}
+END_TEST
+
+/* removing nodes from the configuration can lift the node majority blocker */
+START_TEST(test_config_nodes_removed) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+ uint32_t view[] = {1, 2};
+ char id[CLUSTER_ID_FILE_SIZE];
+
+ set_config_nodes(4);
+ set_quorum(1, view, 2);
+ set_group(nodes, 2);
+ tick(nodes, 2);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+ set_config_nodes(2);
+ tick(nodes, 2);
+ read_id(dbs[0], id);
+ assert_id(dbs[1], id);
+
+ stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+START_TEST(test_only_lowest_generates) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+ uint32_t view[] = {1, 2};
+ char id1[CLUSTER_ID_FILE_SIZE], id2[CLUSTER_ID_FILE_SIZE];
+
+ set_config_nodes(2);
+ set_quorum(1, view, 2);
+ set_group(nodes, 2);
+
+ read_id(dbs[0], id1);
+ read_id(dbs[1], id2);
+ ck_assert(memcmp(id1, id2, CLUSTER_ID_FILE_SIZE) == 0);
+
+ /* written by node 1, the lowest */
+ memdb_tree_entry_t *te = memdb_getattr(dbs[1], "cluster-id");
+ ck_assert(te != NULL);
+ ck_assert(te->writer == 1);
+ memdb_tree_entry_free(te);
+
+ stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* A node with an outdated database starts first and is alone in the group, while corosync, with
+ * the other node running, makes it quorate. It must not publish an ID of its own. */
+START_TEST(test_stale_node_alone) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+ write_id(dbs[1], "abcdefghijklmnopqrst\n", 2, 100);
+ other_writes(dbs[1], 10);
+
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+ uint32_t view[] = {1, 2};
+
+ set_config_nodes(2);
+ set_quorum(1, view, 2);
+ set_group(nodes, 1);
+ tick(nodes, 1);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+ set_group(nodes, 2);
+ assert_id(dbs[0], "abcdefghijklmnopqrst\n");
+ assert_id(dbs[1], "abcdefghijklmnopqrst\n");
+
+ stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* Like above, but the outdated node holds the empty file of an interrupted creation. Filling it
+ * would make its database the newest one, so its ID would replace the established one. */
+START_TEST(test_stale_node_alone_with_empty_file) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+ ck_assert(memdb_create(dbs[0], "cluster-id", 1, 100) == 0);
+ write_id(dbs[1], "abcdefghijklmnopqrst\n", 1, 100);
+
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+ uint32_t view[] = {1, 2};
+
+ set_config_nodes(2);
+ set_quorum(1, view, 2);
+ set_group(nodes, 1);
+ tick(nodes, 1);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_INVALID);
+
+ set_group(nodes, 2);
+ assert_id(dbs[0], "abcdefghijklmnopqrst\n");
+ assert_id(dbs[1], "abcdefghijklmnopqrst\n");
+
+ stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* Quorate through a QDevice, or two_node, last_man_standing or lowered expected votes, the
+ * corosync membership of a lone outdated node is just itself. */
+START_TEST(test_lone_node_without_node_majority) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+ write_id(dbs[1], "abcdefghijklmnopqrst\n", 2, 100);
+
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+ uint32_t view1[] = {1}, view12[] = {1, 2};
+
+ set_config_nodes(2);
+ set_quorum(1, view1, 1);
+ set_group(nodes, 1);
+ tick(nodes, 1);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+ set_quorum(1, view12, 2);
+ set_group(nodes, 2);
+ assert_id(dbs[0], "abcdefghijklmnopqrst\n");
+ assert_id(dbs[1], "abcdefghijklmnopqrst\n");
+
+ stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+START_TEST(test_half_of_the_nodes) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+ uint32_t view[] = {1, 2};
+
+ set_config_nodes(4);
+ set_quorum(1, view, 2);
+ set_group(nodes, 2);
+ tick(nodes, 2);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+ ck_assert(cluster_id_state(dbs[1]) == CLUSTER_ID_MISSING);
+
+ stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* Right after node 4 got removed from the configuration, the corosync membership and the group can
+ * still contain it. Of the configured nodes 1, 2 and 3, only node 1 is there. */
+START_TEST(test_member_not_configured) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("4.db")};
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 4)};
+ uint32_t view[] = {1, 4}, config[] = {1, 2, 3};
+
+ set_config_nodes(5);
+ set_quorum(1, view, 2);
+ set_group(nodes, 2);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+ set_config_node_ids(config, 3);
+ tick(nodes, 2);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+ stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* Neither gaining quorum nor a corosync membership change causes a sync of the group, the next
+ * timer tick acts on them. */
+START_TEST(test_quorum_change) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+ uint32_t view12[] = {1, 2}, view123[] = {1, 2, 3};
+ char id[CLUSTER_ID_FILE_SIZE];
+
+ set_config_nodes(3);
+ set_quorum(0, view12, 2);
+ set_group(nodes, 2);
+ tick(nodes, 2);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+ /* node 3 only runs corosync */
+ set_quorum(1, view123, 3);
+ tick(nodes, 2);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+ /* node 3 left corosync again, the group did not change */
+ set_quorum(1, view12, 2);
+ tick(nodes, 2);
+ read_id(dbs[0], id);
+ assert_id(dbs[1], id);
+
+ stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* node 3 joins corosync and the group, in both orders the notifications can arrive */
+static void third_node_joins(gboolean quorum_first) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db"), open_db("3.db")};
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2), start_node(dbs[2], 3)};
+ uint32_t view12[] = {1, 2}, view123[] = {1, 2, 3};
+ char id[CLUSTER_ID_FILE_SIZE];
+
+ set_config_nodes(5);
+ set_quorum(0, view12, 2);
+ set_group(nodes, 2);
+
+ if (quorum_first) {
+ set_quorum(1, view123, 3);
+ tick(nodes, 2);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+ /* the sync runs the check right away */
+ set_group(nodes, 3);
+ } else {
+ set_group(nodes, 3);
+ set_quorum(1, view123, 3);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+ tick(nodes, 3);
+ }
+
+ read_id(dbs[0], id);
+ assert_id(dbs[1], id);
+ assert_id(dbs[2], id);
+
+ stop_nodes(nodes, dbs, 3);
+}
+
+START_TEST(test_third_node_joins_quorum_first) { third_node_joins(TRUE); }
+END_TEST
+
+START_TEST(test_third_node_joins_group_first) { third_node_joins(FALSE); }
+END_TEST
+
+/* a timer tick during a sync does nothing, the databases might still differ */
+START_TEST(test_tick_during_sync) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+ write_id(dbs[1], "abcdefghijklmnopqrst\n", 2, 100);
+ other_writes(dbs[1], 10);
+
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+ uint32_t view[] = {1, 2};
+
+ set_config_nodes(2);
+ set_quorum(1, view, 2);
+
+ change_group(nodes, 2);
+ do {
+ guint queued = g_queue_get_length(&packets);
+ tick_only(nodes, 2);
+ ck_assert(g_queue_get_length(&packets) == queued);
+ } while (deliver_next(nodes, 2));
+
+ assert_id(dbs[0], "abcdefghijklmnopqrst\n");
+ assert_id(dbs[1], "abcdefghijklmnopqrst\n");
+
+ stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* ticks before the messages of the first attempt got delivered must write the same ID */
+START_TEST(test_pending_id_reused) {
+ memdb_t *dbs[] = {open_db("1.db")};
+ dfsm_t *nodes[] = {start_node(dbs[0], 1)};
+ uint32_t view[] = {1};
+ char first[CLUSTER_ID_FILE_SIZE];
+
+ set_config_nodes(1);
+ set_quorum(1, view, 1);
+
+ change_group(nodes, 1);
+ ck_assert(g_queue_get_length(&packets) == 2);
+ ck_assert(state_of(nodes[0])->gen.have_pending_id);
+ memcpy(first, state_of(nodes[0])->gen.pending_id, CLUSTER_ID_FILE_SIZE);
+
+ tick_only(nodes, 1);
+ tick_only(nodes, 1);
+ ck_assert(g_queue_get_length(&packets) == 6);
+
+ pump(nodes, 1);
+ assert_id(dbs[0], first);
+
+ /* once delivered, the next check finds it valid and forgets the pending one */
+ tick_only(nodes, 1);
+ ck_assert(g_queue_is_empty(&packets));
+ ck_assert(!state_of(nodes[0])->gen.have_pending_id);
+
+ stop_nodes(nodes, dbs, 1);
+}
+END_TEST
+
+/* A failed send leaves no ID, or the empty file of an interrupted creation, and causes no event.
+ * The next timer tick tries again. fail_nr is the send of the attempt that fails. */
+static void send_fails(int fail_nr, gboolean with_empty_file) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+ if (with_empty_file) {
+ ck_assert(memdb_create(dbs[0], "cluster-id", 3, 100) == 0);
+ ck_assert(memdb_create(dbs[1], "cluster-id", 3, 100) == 0);
+ }
+
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+ uint32_t view[] = {1, 2};
+ char id[CLUSTER_ID_FILE_SIZE];
+
+ set_config_nodes(2);
+ set_quorum(0, view, 2);
+ set_group(nodes, 2);
+
+ set_quorum(1, view, 2);
+ send_nr = 0;
+ fail_send_nr = fail_nr;
+ tick(nodes, 2);
+ fail_send_nr = 0;
+ ck_assert(send_nr == fail_nr);
+ ck_assert(
+ cluster_id_state(dbs[1]) ==
+ (fail_nr == 1 && !with_empty_file ? CLUSTER_ID_MISSING : CLUSTER_ID_INVALID)
+ );
+
+ tick(nodes, 2);
+ read_id(dbs[0], id);
+ assert_id(dbs[1], id);
+
+ stop_nodes(nodes, dbs, 2);
+}
+
+START_TEST(test_create_send_fails) { send_fails(1, FALSE); }
+END_TEST
+
+START_TEST(test_write_send_fails) { send_fails(2, FALSE); }
+END_TEST
+
+START_TEST(test_write_send_fails_with_empty_file) { send_fails(1, TRUE); }
+END_TEST
+
+/* The lowest node leaves after its CREATE got delivered, but before its WRITE. The remaining
+ * majority fills the file with an ID of its own, which stays when the former generator rejoins with
+ * its outdated database and its pending ID. */
+START_TEST(test_generator_leaves_before_write) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db"), open_db("3.db")};
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2), start_node(dbs[2], 3)};
+ dfsm_t **rest = nodes + 1;
+ uint32_t view1[] = {1}, view23[] = {2, 3}, view123[] = {1, 2, 3};
+ char first[CLUSTER_ID_FILE_SIZE], id[CLUSTER_ID_FILE_SIZE];
+
+ set_config_nodes(3);
+ set_quorum(1, view123, 3);
+ change_group(nodes, 3);
+ while (cluster_id_state(dbs[2]) == CLUSTER_ID_MISSING) {
+ ck_assert(deliver_next(nodes, 3));
+ }
+ ck_assert(cluster_id_state(dbs[2]) == CLUSTER_ID_INVALID);
+ ck_assert(g_queue_get_length(&packets) == 1);
+ ck_assert(state_of(nodes[0])->gen.have_pending_id);
+ memcpy(first, state_of(nodes[0])->gen.pending_id, CLUSTER_ID_FILE_SIZE);
+ drop_packets();
+
+ set_quorum(1, view23, 2);
+ set_group(rest, 2);
+ read_id(dbs[1], id);
+ assert_id(dbs[2], id);
+ ck_assert(memcmp(id, first, CLUSTER_ID_FILE_SIZE) != 0);
+
+ /* node 1 meanwhile, alone and without quorum */
+ set_quorum(0, view1, 1);
+ set_group(nodes, 1);
+ tick(nodes, 1);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_INVALID);
+
+ /* it learns about the others from corosync before their group merges with its own */
+ set_quorum(1, view123, 3);
+ tick(nodes, 1);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_INVALID);
+ set_group(nodes, 3);
+ for (int i = 0; i < 3; i++) {
+ assert_id(dbs[i], id);
+ }
+ ck_assert(!state_of(nodes[0])->gen.have_pending_id);
+
+ stop_nodes(nodes, dbs, 3);
+}
+END_TEST
+
+START_TEST(test_lowest_repairs_empty_file) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+ ck_assert(memdb_create(dbs[0], "cluster-id", 3, 100) == 0);
+ ck_assert(memdb_create(dbs[1], "cluster-id", 3, 100) == 0);
+
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+ uint32_t view[] = {1, 2};
+ char id[CLUSTER_ID_FILE_SIZE];
+
+ set_config_nodes(2);
+ set_quorum(1, view, 2);
+ set_group(nodes, 2);
+
+ read_id(dbs[0], id);
+ assert_id(dbs[1], id);
+
+ stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* an ID with only 16 characters is malformed like anything else */
+START_TEST(test_lowest_replaces_short_id) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+ write_id(dbs[0], "abcdefghijklmnop\n", 1, 100);
+ write_id(dbs[1], "abcdefghijklmnop\n", 1, 100);
+
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+ uint32_t view[] = {1, 2};
+ char id[CLUSTER_ID_FILE_SIZE];
+
+ set_config_nodes(2);
+ set_quorum(1, view, 2);
+ set_group(nodes, 2);
+
+ read_id(dbs[0], id);
+ assert_id(dbs[1], id);
+
+ stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+/* after the removal of a directory at the path, the next timer tick generates the ID */
+START_TEST(test_removed_directory) {
+ memdb_t *dbs[] = {open_db("1.db")};
+ ck_assert(memdb_mkdir(dbs[0], "cluster-id", 1, 100) == 0);
+ dfsm_t *nodes[] = {start_node(dbs[0], 1)};
+ uint32_t view[] = {1};
+
+ set_config_nodes(1);
+ set_quorum(1, view, 1);
+ set_group(nodes, 1);
+ tick(nodes, 1);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_NOT_A_FILE);
+
+ enter(nodes[0]);
+ ck_assert(
+ dcdb_send_fuse_message_full(
+ nodes[0], DCDB_MESSAGE_CFS_DELETE, "cluster-id", NULL, NULL, 0, 0, 0, NULL
+ ) == CS_OK
+ );
+ pump(nodes, 1);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_MISSING);
+
+ tick(nodes, 1);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_VALID);
+
+ stop_nodes(nodes, dbs, 1);
+}
+END_TEST
+
+/* content that an older node writes over the ID gets replaced on the next timer tick */
+START_TEST(test_overwritten_id) {
+ memdb_t *dbs[] = {open_db("1.db"), open_db("2.db")};
+ dfsm_t *nodes[] = {start_node(dbs[0], 1), start_node(dbs[1], 2)};
+ uint32_t view[] = {1, 2};
+ char id[CLUSTER_ID_FILE_SIZE];
+
+ set_config_nodes(2);
+ set_quorum(1, view, 2);
+ set_group(nodes, 2);
+ tick(nodes, 2);
+ read_id(dbs[0], id);
+
+ enter(nodes[1]);
+ ck_assert(
+ dcdb_send_fuse_message_full(
+ nodes[1], DCDB_MESSAGE_CFS_WRITE, "cluster-id", NULL, "bogus\n", 6, 0, 1, NULL
+ ) == CS_OK
+ );
+ pump(nodes, 2);
+ ck_assert(cluster_id_state(dbs[0]) == CLUSTER_ID_INVALID);
+
+ tick(nodes, 2);
+ read_id(dbs[0], id);
+ assert_id(dbs[1], id);
+
+ stop_nodes(nodes, dbs, 2);
+}
+END_TEST
+
+static void add_test(Suite *s, _TEST_TYPE tf, const char *name) {
+ TCase *tc = tcase_create(name);
+ tcase_add_test(tc, tf);
+ suite_add_tcase(s, tc);
+}
+
+static void remove_tmpdir(void) {
+ GDir *dir = g_dir_open(tmpdir, 0, NULL);
+ if (dir) {
+ const char *name;
+ while ((name = g_dir_read_name(dir))) {
+ char *filename = g_build_filename(tmpdir, name, NULL);
+ unlink(filename);
+ g_free(filename);
+ }
+ g_dir_close(dir);
+ }
+
+ rmdir(tmpdir);
+ g_free(tmpdir);
+}
+
+int main(void) {
+ cfs_status_init();
+
+ if (!(tmpdir = g_dir_make_tmp("check_dcdb-XXXXXX", NULL))) {
+ fprintf(stderr, "unable to create a temporary directory\n");
+ return EXIT_FAILURE;
+ }
+
+ Suite *s = suite_create("dcdb");
+
+ add_test(s, test_single_node_cluster, "single_node_cluster");
+ add_test(s, test_config_nodes_removed, "config_nodes_removed");
+ add_test(s, test_only_lowest_generates, "only_lowest_generates");
+ add_test(s, test_stale_node_alone, "stale_node_alone");
+ add_test(s, test_stale_node_alone_with_empty_file, "stale_node_alone_with_empty_file");
+ add_test(s, test_lone_node_without_node_majority, "lone_node_without_node_majority");
+ add_test(s, test_half_of_the_nodes, "half_of_the_nodes");
+ add_test(s, test_member_not_configured, "member_not_configured");
+ add_test(s, test_quorum_change, "quorum_change");
+ add_test(s, test_third_node_joins_quorum_first, "third_node_joins_quorum_first");
+ add_test(s, test_third_node_joins_group_first, "third_node_joins_group_first");
+ add_test(s, test_tick_during_sync, "tick_during_sync");
+ add_test(s, test_pending_id_reused, "pending_id_reused");
+ add_test(s, test_create_send_fails, "create_send_fails");
+ add_test(s, test_write_send_fails, "write_send_fails");
+ add_test(s, test_write_send_fails_with_empty_file, "write_send_fails_with_empty_file");
+ add_test(s, test_generator_leaves_before_write, "generator_leaves_before_write");
+ add_test(s, test_lowest_repairs_empty_file, "lowest_repairs_empty_file");
+ add_test(s, test_lowest_replaces_short_id, "lowest_replaces_short_id");
+ add_test(s, test_removed_directory, "removed_directory");
+ add_test(s, test_overwritten_id, "overwritten_id");
+
+ SRunner *sr = srunner_create(s);
+ srunner_run_all(sr, CK_NORMAL);
+ int number_failed = srunner_ntests_failed(sr);
+ srunner_free(sr);
+
+ remove_tmpdir();
+
+ return (number_failed == 0) ? EXIT_SUCCESS : EXIT_FAILURE;
+}
diff --git a/src/pmxcfs/dcdb.c b/src/pmxcfs/dcdb.c
index 0965331..3cb118d 100644
--- a/src/pmxcfs/dcdb.c
+++ b/src/pmxcfs/dcdb.c
@@ -38,6 +38,7 @@
#include <unistd.h>
#include "cfs-utils.h"
+#include "cluster-id.h"
#include "dcdb.h"
#include "loop.h"
#include "status.h"
@@ -985,6 +986,145 @@ leave:
goto ret;
}
+/* state of the cluster ID generation, loop thread only */
+typedef struct {
+ /* Kept until the file holds a valid ID. An attempt can run before the messages of an earlier
+ * attempt got delivered, it then sends the same ID again and not a new one. */
+ char pending_id[CLUSTER_ID_FILE_SIZE];
+ gboolean have_pending_id;
+ /* why the last check did not generate, to log only changes */
+ char *blocked_reason;
+} cluster_id_gen_t;
+
+static cluster_id_gen_t cluster_id_gen;
+
+static void set_blocked_reason(char *reason) {
+ if (reason && g_strcmp0(reason, cluster_id_gen.blocked_reason) != 0) {
+ cfs_message("not generating a cluster ID yet, %s", reason);
+ }
+
+ g_free(cluster_id_gen.blocked_reason);
+ cluster_id_gen.blocked_reason = reason;
+}
+
+/* An ID is only generated while more than half of the configured nodes are in the corosync
+ * membership and all of those are in sync in this group. Majorities of the same configured node
+ * set overlap. Together with normal database transfer on join, this prevents a stale minority from
+ * generating an ID before it synchronizes with the other nodes. Counting nodes rather than relying
+ * on quorum alone keeps this guarantee with a QDevice, two_node, last_man_standing or lowered
+ * expected votes, at the price of not generating while half of the nodes or more are down. Returns
+ * why generating is blocked, NULL if it is not. */
+static char *generation_blocker(dfsm_t *dfsm) {
+ size_t members_len;
+ g_autofree uint32_t *members = cfs_get_quorum_members(&members_len);
+
+ if (!members_len || !cfs_is_quorate()) {
+ return g_strdup("not quorate");
+ }
+
+ guint nodes = cfs_clinfo_node_count();
+ if (!nodes) {
+ return g_strdup("the configured nodes are not known");
+ }
+
+ /* by node ID, the membership can still hold a node that got removed from the configuration */
+ size_t configured = 0;
+ for (size_t i = 0; i < members_len; i++) {
+ if (cfs_clinfo_has_node(members[i])) {
+ configured++;
+ }
+ }
+
+ if (configured * 2 <= nodes) {
+ return g_strdup_printf(
+ "only %zu of %u configured nodes are in the corosync membership", configured, nodes
+ );
+ }
+
+ GString *missing = NULL;
+ for (size_t i = 0; i < members_len; i++) {
+ if (!dfsm_nodeid_is_member(dfsm, members[i])) {
+ if (!missing) {
+ missing = g_string_new(NULL);
+ } else {
+ g_string_append(missing, ", ");
+ }
+ g_string_append_printf(missing, "%u", members[i]);
+ }
+ }
+
+ if (missing) {
+ char *reason = g_strdup_printf(
+ "node ID(s) %s of the corosync membership are not in the pmxcfs group", missing->str
+ );
+ g_string_free(missing, TRUE);
+ return reason;
+ }
+
+ return NULL;
+}
+
+/* Generates the cluster ID if it is missing or invalid and nothing blocks that. This is the only
+ * place that decides it, and it only looks at the current state: it runs right after each sync and
+ * then on every timer tick while in sync, so nothing has to call it when the quorum, the
+ * membership, the configured nodes or the file change, or when sending failed. It can run again
+ * before the messages of an earlier run got delivered.
+ *
+ * Only the node with the lowest ID generates the cluster ID, like only that node expires locks, so
+ * that no two nodes write different IDs. The loop thread cannot wait for the result of its own
+ * messages, so CREATE and WRITE go out back to back, the messages of one sender stay in order. */
+static void dcdb_ensure_cluster_id(dfsm_t *dfsm, gpointer data) {
+ g_return_if_fail(dfsm != NULL);
+ g_return_if_fail(data != NULL);
+
+ memdb_t *memdb = (memdb_t *)data;
+
+ if (!dfsm_lowest_nodeid(dfsm)) {
+ return;
+ }
+
+ cluster_id_state_t state = cluster_id_state(memdb);
+ if (state == CLUSTER_ID_VALID) {
+ cluster_id_gen.have_pending_id = FALSE;
+ set_blocked_reason(NULL);
+ return;
+ }
+
+ if (state == CLUSTER_ID_NOT_A_FILE) {
+ set_blocked_reason(g_strdup_printf("'%s' is not a regular file", CFS_CLUSTER_ID_FN));
+ return;
+ }
+
+ char *reason = generation_blocker(dfsm);
+ set_blocked_reason(reason);
+ if (reason) {
+ return;
+ }
+
+ if (!cluster_id_gen.have_pending_id) {
+ if (!cluster_id_generate(cluster_id_gen.pending_id)) {
+ return;
+ }
+ cluster_id_gen.have_pending_id = TRUE;
+ }
+
+ if (state == CLUSTER_ID_MISSING &&
+ dcdb_send_fuse_message_full(
+ dfsm, DCDB_MESSAGE_CFS_CREATE, CFS_CLUSTER_ID_FN, NULL, NULL, 0, 0, 0, NULL
+ ) != CS_OK) {
+ return;
+ }
+
+ if (dcdb_send_fuse_message_full(
+ dfsm, DCDB_MESSAGE_CFS_WRITE, CFS_CLUSTER_ID_FN, NULL, cluster_id_gen.pending_id,
+ CLUSTER_ID_FILE_SIZE, 0, 1, NULL
+ ) != CS_OK) {
+ return;
+ }
+
+ cfs_message("generating cluster ID %.*s", CLUSTER_ID_LEN, cluster_id_gen.pending_id);
+}
+
static dfsm_callbacks_t dcdb_dfsm_callbacks = {
.dfsm_deliver_fn = dcdb_deliver,
.dfsm_get_state_fn = dcdb_get_state,
@@ -993,6 +1133,7 @@ static dfsm_callbacks_t dcdb_dfsm_callbacks = {
.dfsm_commit_fn = dcdb_commit,
.dfsm_cleanup_fn = dcdb_cleanup,
.dfsm_checksum_fn = dcdb_checksum,
+ .dfsm_in_sync_fn = dcdb_ensure_cluster_id,
};
dfsm_t *dcdb_new(memdb_t *memdb) {
diff --git a/src/pmxcfs/dcdb.h b/src/pmxcfs/dcdb.h
index d637cc7..efb23b2 100644
--- a/src/pmxcfs/dcdb.h
+++ b/src/pmxcfs/dcdb.h
@@ -29,7 +29,10 @@
#define DCDB_CPG_GROUP_NAME "pve_dcdb_v1"
/* please increase protocol version if you want to stop older nodes */
#define DCDB_PROTOCOL_VERSION 1
-#define DCDB_VERIFY_TIME (60 * 60)
+/* Seconds between the timer ticks of the dcdb service. Only a completed sync runs the in-sync check
+ * right away, any other change it depends on waits for the next tick. Long compared to the delivery
+ * of a message, so that a tick rarely repeats what the previous one sent. */
+#define DCDB_CHECK_TIME 10
typedef enum {
DCDB_MESSAGE_CFS_WRITE = 1,
diff --git a/src/pmxcfs/pmxcfs.c b/src/pmxcfs/pmxcfs.c
index 1b470b1..21a77a0 100644
--- a/src/pmxcfs/pmxcfs.c
+++ b/src/pmxcfs/pmxcfs.c
@@ -1026,7 +1026,7 @@ int main(int argc, char *argv[]) {
cfs_loop_add_service(corosync_loop, service_confdb, QB_LOOP_MED);
service_dcdb = service_dfsm_new(dcdb);
- cfs_service_set_timer(service_dcdb, DCDB_VERIFY_TIME);
+ cfs_service_set_timer(service_dcdb, DCDB_CHECK_TIME);
cfs_loop_add_service(corosync_loop, service_dcdb, QB_LOOP_MED);
diff --git a/src/pmxcfs/status.c b/src/pmxcfs/status.c
index bbfa88e..afffa96 100644
--- a/src/pmxcfs/status.c
+++ b/src/pmxcfs/status.c
@@ -2002,6 +2002,26 @@ gboolean cfs_is_quorate(void) {
return res;
}
+guint cfs_clinfo_node_count(void) {
+ g_mutex_lock(&mutex);
+ guint count = 0;
+ if (cfs_status.clinfo && cfs_status.clinfo->nodes_byid) {
+ count = g_hash_table_size(cfs_status.clinfo->nodes_byid);
+ }
+ g_mutex_unlock(&mutex);
+
+ return count;
+}
+
+gboolean cfs_clinfo_has_node(uint32_t nodeid) {
+ g_mutex_lock(&mutex);
+ gboolean found = cfs_status.clinfo && cfs_status.clinfo->nodes_byid &&
+ g_hash_table_contains(cfs_status.clinfo->nodes_byid, &nodeid);
+ g_mutex_unlock(&mutex);
+
+ return found;
+}
+
void cfs_set_quorum_members(const uint32_t *nodeids, size_t count) {
g_return_if_fail(!count || nodeids != NULL);
diff --git a/src/pmxcfs/status.h b/src/pmxcfs/status.h
index a757326..45dd4b6 100644
--- a/src/pmxcfs/status.h
+++ b/src/pmxcfs/status.h
@@ -76,6 +76,12 @@ void cfs_set_quorum_members(const uint32_t *nodeids, size_t count);
/* a copy of them, to free with g_free() */
uint32_t *cfs_get_quorum_members(size_t *count);
+/* number of nodes in the corosync configuration, 0 while not known yet */
+guint cfs_clinfo_node_count(void);
+
+/* whether the node is in the corosync configuration */
+gboolean cfs_clinfo_has_node(uint32_t nodeid);
+
GHashTable *vmlist_hash_new(void);
gboolean vmlist_hash_insert_vm(
--
2.47.3
next prev parent reply other threads:[~2026-11-19 19:25 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 19:18 [PATCH cluster/guest-common/qemu-server/container/manager/docs v5 00/41] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
2026-11-19 19:18 ` [PATCH cluster v5 01/41] pmxcfs: reject changes to the cluster ID file from user space Thomas Lamprecht
2026-11-19 19:18 ` [PATCH cluster v5 02/41] pmxcfs: generate a cluster ID on standalone nodes Thomas Lamprecht
2026-11-19 19:18 ` [PATCH cluster v5 03/41] pmxcfs: dfsm: add a callback that runs once all members are in sync Thomas Lamprecht
2026-11-19 19:18 ` [PATCH cluster v5 04/41] pmxcfs: quorum: record the corosync membership Thomas Lamprecht
2026-11-19 19:18 ` Thomas Lamprecht [this message]
2026-11-19 19:18 ` [PATCH cluster v5 06/41] cluster: add helper to read the cluster ID Thomas Lamprecht
2026-11-19 19:18 ` [PATCH cluster v5 07/41] pmxcfs: allow removing the cluster ID file in forced local mode Thomas Lamprecht
2026-11-19 19:18 ` [PATCH guest-common v5 08/41] guest identity: add helpers for the guest uid and registration time Thomas Lamprecht
2026-11-19 19:18 ` [PATCH guest-common v5 09/41] abstract config: assign a guest identity and keep it on config writes Thomas Lamprecht
2026-11-19 19:18 ` [PATCH guest-common v5 10/41] abstract config: let the VMID reservation return the previous state Thomas Lamprecht
2026-11-19 19:18 ` [PATCH guest-common v5 11/41] guest identity: allow adopting a uid that is unused in the cluster Thomas Lamprecht
2026-11-19 19:18 ` [PATCH guest-common v5 12/41] migrate: add shared helpers for the identity command of the migration tunnel Thomas Lamprecht
2026-11-19 19:18 ` [PATCH guest-common v5 13/41] vzdump: notes template: list the uid variable Thomas Lamprecht
2026-11-19 19:18 ` [PATCH qemu-server v5 14/41] api: restore: release the VMID if a restore to an unused VMID fails Thomas Lamprecht
2026-11-19 19:18 ` [PATCH qemu-server v5 15/41] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-19 19:18 ` [PATCH qemu-server v5 16/41] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-19 19:18 ` [PATCH qemu-server v5 17/41] restore: adopt the guest uid from the backup where possible Thomas Lamprecht
2026-11-19 19:18 ` [PATCH qemu-server v5 18/41] api: remote migration: adopt the guest uid from the source Thomas Lamprecht
2026-11-19 19:18 ` [PATCH qemu-server v5 19/41] api: status: return the guest uid Thomas Lamprecht
2026-11-19 19:18 ` [PATCH container v5 20/41] restore: decide by the reservation whether the container is new Thomas Lamprecht
2026-11-19 19:18 ` [PATCH container v5 21/41] config: add read-only meta property with the creation time Thomas Lamprecht
2026-11-19 19:18 ` [PATCH container v5 22/41] fix #7846: config: add a guest identity to the 'meta' property Thomas Lamprecht
2026-11-19 19:18 ` [PATCH container v5 23/41] api: migrate, clone: refuse targets that would drop the guest identity Thomas Lamprecht
2026-11-19 19:18 ` [PATCH container v5 24/41] restore: adopt the guest uid from the backup where possible Thomas Lamprecht
2026-11-19 19:18 ` [PATCH container v5 25/41] restore: refuse overwriting a container with another guest's backup Thomas Lamprecht
2026-11-19 19:18 ` [PATCH container v5 26/41] api: remote migration: adopt the guest uid from the source Thomas Lamprecht
2026-11-19 19:18 ` [PATCH container v5 27/41] api: status: return the guest uid Thomas Lamprecht
2026-11-19 19:19 ` [PATCH manager v5 28/41] api: cluster resources: add guest uid and registration time Thomas Lamprecht
2026-11-19 19:19 ` [PATCH manager v5 29/41] api: cluster status: add cluster ID Thomas Lamprecht
2026-11-19 19:19 ` [PATCH manager v5 30/41] ui: datacenter summary: show the " Thomas Lamprecht
2026-11-19 19:19 ` [PATCH manager v5 31/41] ui: restore: allow overwriting a guest with a backup of another guest Thomas Lamprecht
2026-11-19 19:19 ` [PATCH manager v5 32/41] ui: migrate: refuse target nodes that would drop the guest identity Thomas Lamprecht
2026-11-19 19:19 ` [PATCH manager v5 33/41] ui: guest summary: show " Thomas Lamprecht
2026-11-19 19:19 ` [PATCH manager v5 34/41] api: bulk migrate: refuse targets that would drop " Thomas Lamprecht
2026-11-19 19:19 ` [PATCH manager v5 35/41] vzdump: notes template: add the guest uid as variable Thomas Lamprecht
2026-11-19 19:19 ` [PATCH docs v5 36/41] vzdump: restore: describe what a failed restore can leave behind Thomas Lamprecht
2026-11-19 19:19 ` [PATCH docs v5 37/41] vzdump: describe the guest identity of VMs and containers Thomas Lamprecht
2026-11-19 19:19 ` [PATCH docs v5 38/41] pvecm: document the cluster ID and how to get a new one Thomas Lamprecht
2026-11-19 19:19 ` [PATCH docs v5 39/41] qm: configuration: describe the read-only meta property Thomas Lamprecht
2026-11-19 19:19 ` [PATCH docs v5 40/41] pct: " Thomas Lamprecht
2026-11-19 19:19 ` [PATCH docs v5 41/41] vzdump: notes: document the uid template variable Thomas Lamprecht
2026-11-20 3:07 ` [PATCH cluster/guest-common/qemu-server/container/manager/docs v5 00/41] fix #7846: guest identity (uid) and cluster ID Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261119191914.2336934-6-t.lamprecht@proxmox.com \
--to=t.lamprecht@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.