From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id D61E71FF0AA for ; Tue, 17 Nov 2026 23:40:49 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 059CE21763; Tue, 17 Nov 2026 23:40:06 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH cluster v2 02/36] pmxcfs: generate a cluster ID on standalone nodes Date: Tue, 17 Nov 2026 23:34:12 +0100 Message-ID: <20261117223945.862914-3-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261117223945.862914-1-t.lamprecht@proxmox.com> References: <20261117223945.862914-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794955190474 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.200 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_SHORT 0.001 Use of a URL Shortener for very short URL RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: W3BYVWM7AYOIZM64BF4HEKR7VLNAROCI X-Message-ID-Hash: W3BYVWM7AYOIZM64BF4HEKR7VLNAROCI X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Standalone nodes need a cluster ID to distinguish them from other installations. Generate and persist it before mounting the filesystem or starting the IPC server, when no other writer or reader can access the database. Use 20 random characters of the lowercase RFC 4648 base32 alphabet, each from a random byte of /dev/urandom, followed by a newline. These 100 random bits make a duplicate between installations practically impossible, fit into a 128-bit value with bits to spare, and stay short and free of characters that need quoting. Keep a valid existing ID, with or without its newline. Replace invalid content, including an empty file left by interrupted initialization. Log a directory at the path and leave it for an administrator to remove with rmdir. Abort startup if storing the ID fails: the in-memory database could otherwise expose an ID that is lost or replaced after a restart. A failure to obtain random data only logs an error, since nothing was written and the next startup can retry. Do not generate an ID in forced local mode if corosync.conf exists. That node belongs to a cluster and must not create a conflicting ID before rejoining it. Signed-off-by: Thomas Lamprecht --- changes v1 -> v2: * take each character from a random byte of its own instead of encoding 13 random bytes, which drops the encoder and its test vectors (Dominik) * give the reasons for the format in the commit message src/pmxcfs/Makefile | 2 +- src/pmxcfs/check_memdb.c | 165 +++++++++++++++++++++++++++++++++++++++ src/pmxcfs/cluster-id.c | 142 +++++++++++++++++++++++++++++++++ src/pmxcfs/cluster-id.h | 50 ++++++++++++ src/pmxcfs/pmxcfs.c | 6 ++ 5 files changed, 364 insertions(+), 1 deletion(-) create mode 100644 src/pmxcfs/cluster-id.c create mode 100644 src/pmxcfs/cluster-id.h diff --git a/src/pmxcfs/Makefile b/src/pmxcfs/Makefile index 547f258..cd93e72 100644 --- a/src/pmxcfs/Makefile +++ b/src/pmxcfs/Makefile @@ -21,7 +21,7 @@ all: pmxcfs create_pmxcfs_db logtest check_memdb libpmxcfs.a: logger.o loop.o server.o status.o confdb.o quorum.o dcdb.o dfsm.o libpmxcfs.a: cfs-plug.o cfs-plug-memdb.o cfs-plug-link.o cfs-plug-func.o -libpmxcfs.a: cfs-utils.o memdb.o database.o +libpmxcfs.a: cfs-utils.o memdb.o database.o cluster-id.o libpmxcfs.a: $(AR) $(ARFLAGS) $@ $^ diff --git a/src/pmxcfs/check_memdb.c b/src/pmxcfs/check_memdb.c index caac740..db85cbd 100644 --- a/src/pmxcfs/check_memdb.c +++ b/src/pmxcfs/check_memdb.c @@ -19,8 +19,10 @@ */ #include +#include #include #include +#include #include #include #include @@ -28,6 +30,7 @@ #include "cfs-plug-memdb.h" #include "cfs-utils.h" +#include "cluster-id.h" #include "memdb.h" #include "status.h" @@ -302,6 +305,158 @@ START_TEST(test_cluster_id_fuse_read_only) { } END_TEST +START_TEST(test_cluster_id_is_valid) { + ck_assert(cluster_id_is_valid("abcdefghijklmnopqrst\n", 21)); + ck_assert(cluster_id_is_valid("uvwxyz234567abcdefgh\n", 21)); + ck_assert(cluster_id_is_valid("abcdefghijklmnopqrst", 20)); + + ck_assert(!cluster_id_is_valid(NULL, 21)); + ck_assert(!cluster_id_is_valid("", 0)); + ck_assert(!cluster_id_is_valid("abcdefghijklmno", 15)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnop", 16)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnop\n", 17)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrs\n", 20)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrstu", 21)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrstu\n", 22)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrst\n\n", 22)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrst ", 21)); + ck_assert(!cluster_id_is_valid("Abcdefghijklmnopqrst\n", 21)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrs1\n", 21)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrs8\n", 21)); + ck_assert(!cluster_id_is_valid("abcdefgh\0jklmnopqrst\n", 21)); +} +END_TEST + +START_TEST(test_cluster_id_generate) { + char id1[CLUSTER_ID_FILE_SIZE], id2[CLUSTER_ID_FILE_SIZE]; + + ck_assert(cluster_id_generate(id1)); + ck_assert(cluster_id_generate(id2)); + + ck_assert(cluster_id_is_valid(id1, sizeof(id1))); + ck_assert(cluster_id_is_valid(id2, sizeof(id2))); + ck_assert(memcmp(id1, id2, sizeof(id1)) != 0); +} +END_TEST + +START_TEST(test_cluster_id_state) { + time_t ctime = 1234; + + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_MISSING); + + ck_assert(memdb_create(memdb, "cluster-id", 0, ctime) == 0); + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_INVALID); + + ck_assert(memdb_write(memdb, "cluster-id", 0, ctime, "abcdefghijklmnopqrst\n", 21, 0, 1) == 21); + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_VALID); + + ck_assert(memdb_write(memdb, "cluster-id", 0, ctime, "foo", 3, 0, 1) == 3); + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_INVALID); + + ck_assert(memdb_delete(memdb, "cluster-id", 0, ctime) == 0); + ck_assert(memdb_mkdir(memdb, "cluster-id", 0, ctime) == 0); + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_NOT_A_FILE); +} +END_TEST + +START_TEST(test_cluster_id_standalone) { + time_t ctime = 1234; + gpointer data; + int len; + + ck_assert(cluster_id_ensure_standalone(memdb)); + len = memdb_read(memdb, "cluster-id", &data); + ck_assert(len == CLUSTER_ID_FILE_SIZE); + ck_assert(cluster_id_is_valid(data, len)); + + /* an existing ID is kept */ + char id[CLUSTER_ID_FILE_SIZE]; + memcpy(id, data, sizeof(id)); + g_free(data); + + ck_assert(cluster_id_ensure_standalone(memdb)); + ck_assert(memdb_read(memdb, "cluster-id", &data) == CLUSTER_ID_FILE_SIZE); + ck_assert(memcmp(data, id, sizeof(id)) == 0); + g_free(data); + + /* an empty file is what a crash between creating and writing leaves behind, it gets filled */ + ck_assert(memdb_delete(memdb, "cluster-id", 0, ctime) == 0); + ck_assert(memdb_create(memdb, "cluster-id", 0, ctime) == 0); + + ck_assert(cluster_id_ensure_standalone(memdb)); + len = memdb_read(memdb, "cluster-id", &data); + ck_assert(len == CLUSTER_ID_FILE_SIZE); + ck_assert(cluster_id_is_valid(data, len)); + g_free(data); + + /* so is invalid content */ + ck_assert(memdb_write(memdb, "cluster-id", 0, ctime, "foo\n", 4, 0, 1) == 4); + + ck_assert(cluster_id_ensure_standalone(memdb)); + len = memdb_read(memdb, "cluster-id", &data); + ck_assert(len == CLUSTER_ID_FILE_SIZE); + ck_assert(cluster_id_is_valid(data, len)); + g_free(data); + + /* a valid ID without newline is kept as is */ + ck_assert(memdb_write(memdb, "cluster-id", 0, ctime, "abcdefghijklmnopqrst", 20, 0, 1) == 20); + + ck_assert(cluster_id_ensure_standalone(memdb)); + ck_assert(memdb_read(memdb, "cluster-id", &data) == 20); + ck_assert(memcmp(data, "abcdefghijklmnopqrst", 20) == 0); + g_free(data); + + /* an ID with only 16 characters is malformed like anything else and gets replaced */ + ck_assert(memdb_write(memdb, "cluster-id", 0, ctime, "abcdefghijklmnop\n", 17, 0, 1) == 17); + + ck_assert(cluster_id_ensure_standalone(memdb)); + len = memdb_read(memdb, "cluster-id", &data); + ck_assert(len == CLUSTER_ID_FILE_SIZE); + ck_assert(cluster_id_is_valid(data, len)); + g_free(data); + + /* a directory is left for the admin and does not stop the start */ + ck_assert(memdb_delete(memdb, "cluster-id", 0, ctime) == 0); + ck_assert(memdb_mkdir(memdb, "cluster-id", 0, ctime) == 0); + + ck_assert(cluster_id_ensure_standalone(memdb)); + memdb_tree_entry_t *te = memdb_getattr(memdb, "cluster-id"); + ck_assert(te != NULL); + ck_assert(te->type == DT_DIR); + memdb_tree_entry_free(te); +} +END_TEST + +static void sql_exec(const char *statement) { + sqlite3 *db = NULL; + ck_assert(sqlite3_open(TESTDB, &db) == SQLITE_OK); + ck_assert(sqlite3_exec(db, statement, NULL, NULL, NULL) == SQLITE_OK); + ck_assert(sqlite3_close(db) == SQLITE_OK); +} + +START_TEST(test_cluster_id_standalone_write_failure) { + sql_exec( + "CREATE TRIGGER fail_cluster_id BEFORE INSERT ON tree " + "WHEN NEW.name = 'cluster-id' AND length(NEW.data) > 0 " + "BEGIN SELECT RAISE(FAIL, 'injected failure'); END;" + ); + + /* creating the empty file still works, storing the content fails */ + ck_assert(!cluster_id_ensure_standalone(memdb)); + ck_assert(memdb->errors); + + memdb_close(memdb); + sql_exec("DROP TRIGGER fail_cluster_id;"); + memdb = memdb_open(TESTDB); + ck_assert(memdb != NULL); + + /* the next start finds the empty file and fills it */ + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_INVALID); + ck_assert(cluster_id_ensure_standalone(memdb)); + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_VALID); +} +END_TEST + START_TEST(test_truncate_grow) { const char *fn = "/f1"; time_t ctime = 1234; @@ -590,6 +745,16 @@ static Suite *memdb_suite(void) { add_test(s, test_cluster_id_fuse_read_only, "cluster_id_fuse_read_only"); + add_test(s, test_cluster_id_is_valid, "cluster_id_is_valid"); + + add_test(s, test_cluster_id_generate, "cluster_id_generate"); + + add_test(s, test_cluster_id_state, "cluster_id_state"); + + add_test(s, test_cluster_id_standalone, "cluster_id_standalone"); + + add_test(s, test_cluster_id_standalone_write_failure, "cluster_id_standalone_write_failure"); + add_test(s, test_indextest1, "indextest1"); add_test(s, test_loaddb1, "loaddb1"); diff --git a/src/pmxcfs/cluster-id.c b/src/pmxcfs/cluster-id.c new file mode 100644 index 0000000..b9a5a6b --- /dev/null +++ b/src/pmxcfs/cluster-id.c @@ -0,0 +1,142 @@ +/* + Copyright (C) 2026 Proxmox Server Solutions GmbH + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see . + + Author: Thomas Lamprecht + +*/ + +#ifdef HAVE_CONFIG_H +#include +#endif /* HAVE_CONFIG_H */ + +#include +#include +#include +#include +#include +#include + +#include "cfs-utils.h" +#include "cluster-id.h" + +static const char base32_alphabet[] = "abcdefghijklmnopqrstuvwxyz234567"; + +gboolean cluster_id_generate(char id[CLUSTER_ID_FILE_SIZE]) { + guint8 raw[CLUSTER_ID_LEN]; + + int fd = open("/dev/urandom", O_RDONLY | O_CLOEXEC); + if (fd == -1) { + cfs_critical("unable to open /dev/urandom - %s", g_strerror(errno)); + return FALSE; + } + + ssize_t res = safe_read(fd, (char *)raw, sizeof(raw)); + int saved_errno = errno; + close(fd); + + if (res != (ssize_t)sizeof(raw)) { + cfs_critical( + "unable to read from /dev/urandom - %s", + res < 0 ? g_strerror(saved_errno) : "short read" + ); + return FALSE; + } + + /* five bits of a random byte per character, so each one is uniformly distributed */ + for (int i = 0; i < CLUSTER_ID_LEN; i++) { + id[i] = base32_alphabet[raw[i] & 0x1f]; + } + id[CLUSTER_ID_LEN] = '\n'; + + return TRUE; +} + +gboolean cluster_id_is_valid(gconstpointer data, size_t len) { + const char *id = data; + + if (!id || + !(len == CLUSTER_ID_LEN || (len == CLUSTER_ID_FILE_SIZE && id[CLUSTER_ID_LEN] == '\n'))) { + return FALSE; + } + + for (int i = 0; i < CLUSTER_ID_LEN; i++) { + if (!((id[i] >= 'a' && id[i] <= 'z') || (id[i] >= '2' && id[i] <= '7'))) { + return FALSE; + } + } + + return TRUE; +} + +cluster_id_state_t cluster_id_state(memdb_t *memdb) { + g_return_val_if_fail(memdb != NULL, CLUSTER_ID_NOT_A_FILE); + + memdb_tree_entry_t *te = memdb_getattr(memdb, CFS_CLUSTER_ID_FN); + if (!te) { + return CLUSTER_ID_MISSING; + } + + gboolean is_file = te->type == DT_REG; + memdb_tree_entry_free(te); + if (!is_file) { + return CLUSTER_ID_NOT_A_FILE; + } + + gpointer data = NULL; + int len = memdb_read(memdb, CFS_CLUSTER_ID_FN, &data); + gboolean valid = len >= 0 && cluster_id_is_valid(data, len); + g_free(data); + + return valid ? CLUSTER_ID_VALID : CLUSTER_ID_INVALID; +} + +gboolean cluster_id_ensure_standalone(memdb_t *memdb) { + g_return_val_if_fail(memdb != NULL, FALSE); + + cluster_id_state_t state = cluster_id_state(memdb); + if (state == CLUSTER_ID_VALID) { + return TRUE; + } + + if (state == CLUSTER_ID_NOT_A_FILE) { + cfs_message("not generating a cluster ID, '%s' is not a regular file", CFS_CLUSTER_ID_FN); + return TRUE; + } + + /* nothing got written, so the start can go on, the next one tries again */ + char id[CLUSTER_ID_FILE_SIZE]; + if (!cluster_id_generate(id)) { + return TRUE; + } + + /* nothing is mounted yet, so no reader can see the file between creating and writing it, and + * a crash in between leaves an empty file that the next start fills */ + guint32 mtime = time(NULL); + int res = state == CLUSTER_ID_MISSING ? memdb_create(memdb, CFS_CLUSTER_ID_FN, 0, mtime) : 0; + if (res == 0) { + res = memdb_write(memdb, CFS_CLUSTER_ID_FN, 0, mtime, id, CLUSTER_ID_FILE_SIZE, 0, TRUE); + } + + /* the memdb already shows the new content if only storing it failed, so it must not be used */ + if (res != CLUSTER_ID_FILE_SIZE) { + cfs_critical("unable to write '%s' - %s", CFS_CLUSTER_ID_FN, g_strerror(-res)); + return FALSE; + } + + cfs_message("generated cluster ID %.*s", CLUSTER_ID_LEN, id); + + return TRUE; +} diff --git a/src/pmxcfs/cluster-id.h b/src/pmxcfs/cluster-id.h new file mode 100644 index 0000000..2715f7d --- /dev/null +++ b/src/pmxcfs/cluster-id.h @@ -0,0 +1,50 @@ +/* + Copyright (C) 2026 Proxmox Server Solutions GmbH + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see . + + Author: Thomas Lamprecht + +*/ + +#ifndef _PVE_CLUSTER_ID_H_ +#define _PVE_CLUSTER_ID_H_ + +#include + +#include "memdb.h" + +/* the ID is 20 random characters of the lowercase RFC 4648 base32 alphabet, so 100 random bits, + * pmxcfs writes it plus a newline */ +#define CLUSTER_ID_LEN 20 +#define CLUSTER_ID_FILE_SIZE (CLUSTER_ID_LEN + 1) + +gboolean cluster_id_generate(char id[CLUSTER_ID_FILE_SIZE]); + +gboolean cluster_id_is_valid(gconstpointer data, size_t len); + +typedef enum { + CLUSTER_ID_VALID, + CLUSTER_ID_MISSING, + CLUSTER_ID_INVALID, + /* a directory, which only rmdir can remove */ + CLUSTER_ID_NOT_A_FILE, +} cluster_id_state_t; + +cluster_id_state_t cluster_id_state(memdb_t *memdb); + +/* FALSE only if storing the ID failed, the memdb must not be used then */ +gboolean cluster_id_ensure_standalone(memdb_t *memdb); + +#endif /* _PVE_CLUSTER_ID_H_ */ diff --git a/src/pmxcfs/pmxcfs.c b/src/pmxcfs/pmxcfs.c index b8a212d..2bf9145 100644 --- a/src/pmxcfs/pmxcfs.c +++ b/src/pmxcfs/pmxcfs.c @@ -48,6 +48,7 @@ #include "cfs-plug-memdb.h" #include "cfs-plug.h" #include "cfs-utils.h" +#include "cluster-id.h" #include "confdb.h" #include "dcdb.h" #include "dfsm.h" @@ -912,6 +913,11 @@ int main(int argc, char *argv[]) { } else { cfs_debug("using local mode (corosync.conf does not exist)"); cfs_set_quorate(1, TRUE); + /* only without a corosync.conf, forced local mode on a cluster node must not generate an ID + * of its own */ + if (!cluster_id_ensure_standalone(memdb)) { + goto err; + } } if (conf_data) { g_free(conf_data); -- 2.47.3