From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id A9F1C1FF0E1 for ; Mon, 16 Nov 2026 01:48:24 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id B66C22179F; Mon, 16 Nov 2026 01:47:21 +0100 (CET) From: Thomas Lamprecht To: pve-devel@lists.proxmox.com Subject: [PATCH cluster 02/34] pmxcfs: generate a cluster ID on standalone nodes Date: Mon, 16 Nov 2026 01:46:14 +0100 Message-ID: <20261116004703.2908294-3-t.lamprecht@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261116004703.2908294-1-t.lamprecht@proxmox.com> References: <20261116004703.2908294-1-t.lamprecht@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794790028854 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.219 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) KAM_SHORT 0.001 Use of a URL Shortener for very short URL RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 74V7BFAG6EXS2T24PL5RLM6L75SUWX6G X-Message-ID-Hash: 74V7BFAG6EXS2T24PL5RLM6L75SUWX6G X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Standalone nodes need a cluster ID to distinguish them from other installations. Generate and persist it before mounting the filesystem or starting the IPC server, when no other writer or reader can access the database. Use 100 random bits from /dev/urandom, encoded as 20 lowercase RFC 4648 base32 characters without padding, followed by a newline. Keep a valid existing ID, with or without its newline. Replace invalid content, including an empty file left by interrupted initialization. Log a directory at the path and leave it for an administrator to remove with rmdir. Abort startup if storing the ID fails: the in-memory database could otherwise expose an ID that is lost or replaced after a restart. A failure to obtain random data only logs an error, since nothing was written and the next startup can retry. Do not generate an ID in forced local mode if corosync.conf exists. That node belongs to a cluster and must not create a conflicting ID before rejoining it. Signed-off-by: Thomas Lamprecht --- src/pmxcfs/Makefile | 2 +- src/pmxcfs/check_memdb.c | 193 +++++++++++++++++++++++++++++++++++++++ src/pmxcfs/cluster-id.c | 158 ++++++++++++++++++++++++++++++++ src/pmxcfs/cluster-id.h | 52 +++++++++++ src/pmxcfs/pmxcfs.c | 6 ++ 5 files changed, 410 insertions(+), 1 deletion(-) create mode 100644 src/pmxcfs/cluster-id.c create mode 100644 src/pmxcfs/cluster-id.h diff --git a/src/pmxcfs/Makefile b/src/pmxcfs/Makefile index 547f258..cd93e72 100644 --- a/src/pmxcfs/Makefile +++ b/src/pmxcfs/Makefile @@ -21,7 +21,7 @@ all: pmxcfs create_pmxcfs_db logtest check_memdb libpmxcfs.a: logger.o loop.o server.o status.o confdb.o quorum.o dcdb.o dfsm.o libpmxcfs.a: cfs-plug.o cfs-plug-memdb.o cfs-plug-link.o cfs-plug-func.o -libpmxcfs.a: cfs-utils.o memdb.o database.o +libpmxcfs.a: cfs-utils.o memdb.o database.o cluster-id.o libpmxcfs.a: $(AR) $(ARFLAGS) $@ $^ diff --git a/src/pmxcfs/check_memdb.c b/src/pmxcfs/check_memdb.c index 1ca1372..bca6443 100644 --- a/src/pmxcfs/check_memdb.c +++ b/src/pmxcfs/check_memdb.c @@ -19,8 +19,10 @@ */ #include +#include #include #include +#include #include #include #include @@ -28,6 +30,7 @@ #include "cfs-plug-memdb.h" #include "cfs-utils.h" +#include "cluster-id.h" #include "memdb.h" #include "status.h" @@ -302,6 +305,184 @@ START_TEST(test_cluster_id_fuse_read_only) { } END_TEST +START_TEST(test_cluster_id_encode) { + struct { + guint8 raw[CLUSTER_ID_RAW_SIZE]; + const char *id; + } cases[] = { + {{0}, "aaaaaaaaaaaaaaaaaaaa\n"}, + {{0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff}, + "77777777777777777777\n"}, + {{'f', 'o', 'o', 'b', 'a', 'f', 'o', 'o', 'b', 'a', 'f', 'o', 'o'}, "mzxw6ytbmzxw6ytbmzxw\n" + }, + {{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}, "aaaqeayeaudaocajbifq\n"}, + /* the last character takes bit 95 and the upper half of the last byte */ + {{0x80, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x01, 0xf0}, "qaaaaaaaaaaaaaaaaaa7\n"}, + /* the lower half of the last byte is not used */ + {{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x0f}, "aaaaaaaaaaaaaaaaaaaa\n"}, + }; + + for (size_t i = 0; i < G_N_ELEMENTS(cases); i++) { + char id[CLUSTER_ID_FILE_SIZE]; + cluster_id_encode(cases[i].raw, id); + ck_assert(memcmp(id, cases[i].id, CLUSTER_ID_FILE_SIZE) == 0); + ck_assert(cluster_id_is_valid(id, sizeof(id))); + } +} +END_TEST + +START_TEST(test_cluster_id_is_valid) { + ck_assert(cluster_id_is_valid("abcdefghijklmnopqrst\n", 21)); + ck_assert(cluster_id_is_valid("uvwxyz234567abcdefgh\n", 21)); + ck_assert(cluster_id_is_valid("abcdefghijklmnopqrst", 20)); + + ck_assert(!cluster_id_is_valid(NULL, 21)); + ck_assert(!cluster_id_is_valid("", 0)); + ck_assert(!cluster_id_is_valid("abcdefghijklmno", 15)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnop", 16)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnop\n", 17)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrs\n", 20)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrstu", 21)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrstu\n", 22)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrst\n\n", 22)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrst ", 21)); + ck_assert(!cluster_id_is_valid("Abcdefghijklmnopqrst\n", 21)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrs1\n", 21)); + ck_assert(!cluster_id_is_valid("abcdefghijklmnopqrs8\n", 21)); + ck_assert(!cluster_id_is_valid("abcdefgh\0jklmnopqrst\n", 21)); +} +END_TEST + +START_TEST(test_cluster_id_generate) { + char id1[CLUSTER_ID_FILE_SIZE], id2[CLUSTER_ID_FILE_SIZE]; + + ck_assert(cluster_id_generate(id1)); + ck_assert(cluster_id_generate(id2)); + + ck_assert(cluster_id_is_valid(id1, sizeof(id1))); + ck_assert(cluster_id_is_valid(id2, sizeof(id2))); + ck_assert(memcmp(id1, id2, sizeof(id1)) != 0); +} +END_TEST + +START_TEST(test_cluster_id_state) { + time_t ctime = 1234; + + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_MISSING); + + ck_assert(memdb_create(memdb, "cluster-id", 0, ctime) == 0); + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_INVALID); + + ck_assert(memdb_write(memdb, "cluster-id", 0, ctime, "abcdefghijklmnopqrst\n", 21, 0, 1) == 21); + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_VALID); + + ck_assert(memdb_write(memdb, "cluster-id", 0, ctime, "foo", 3, 0, 1) == 3); + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_INVALID); + + ck_assert(memdb_delete(memdb, "cluster-id", 0, ctime) == 0); + ck_assert(memdb_mkdir(memdb, "cluster-id", 0, ctime) == 0); + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_NOT_A_FILE); +} +END_TEST + +START_TEST(test_cluster_id_standalone) { + time_t ctime = 1234; + gpointer data; + int len; + + ck_assert(cluster_id_ensure_standalone(memdb)); + len = memdb_read(memdb, "cluster-id", &data); + ck_assert(len == CLUSTER_ID_FILE_SIZE); + ck_assert(cluster_id_is_valid(data, len)); + + /* an existing ID is kept */ + char id[CLUSTER_ID_FILE_SIZE]; + memcpy(id, data, sizeof(id)); + g_free(data); + + ck_assert(cluster_id_ensure_standalone(memdb)); + ck_assert(memdb_read(memdb, "cluster-id", &data) == CLUSTER_ID_FILE_SIZE); + ck_assert(memcmp(data, id, sizeof(id)) == 0); + g_free(data); + + /* an empty file is what a crash between creating and writing leaves behind, it gets filled */ + ck_assert(memdb_delete(memdb, "cluster-id", 0, ctime) == 0); + ck_assert(memdb_create(memdb, "cluster-id", 0, ctime) == 0); + + ck_assert(cluster_id_ensure_standalone(memdb)); + len = memdb_read(memdb, "cluster-id", &data); + ck_assert(len == CLUSTER_ID_FILE_SIZE); + ck_assert(cluster_id_is_valid(data, len)); + g_free(data); + + /* so is invalid content */ + ck_assert(memdb_write(memdb, "cluster-id", 0, ctime, "foo\n", 4, 0, 1) == 4); + + ck_assert(cluster_id_ensure_standalone(memdb)); + len = memdb_read(memdb, "cluster-id", &data); + ck_assert(len == CLUSTER_ID_FILE_SIZE); + ck_assert(cluster_id_is_valid(data, len)); + g_free(data); + + /* a valid ID without newline is kept as is */ + ck_assert(memdb_write(memdb, "cluster-id", 0, ctime, "abcdefghijklmnopqrst", 20, 0, 1) == 20); + + ck_assert(cluster_id_ensure_standalone(memdb)); + ck_assert(memdb_read(memdb, "cluster-id", &data) == 20); + ck_assert(memcmp(data, "abcdefghijklmnopqrst", 20) == 0); + g_free(data); + + /* an ID with only 16 characters is malformed like anything else and gets replaced */ + ck_assert(memdb_write(memdb, "cluster-id", 0, ctime, "abcdefghijklmnop\n", 17, 0, 1) == 17); + + ck_assert(cluster_id_ensure_standalone(memdb)); + len = memdb_read(memdb, "cluster-id", &data); + ck_assert(len == CLUSTER_ID_FILE_SIZE); + ck_assert(cluster_id_is_valid(data, len)); + g_free(data); + + /* a directory is left for the admin and does not stop the start */ + ck_assert(memdb_delete(memdb, "cluster-id", 0, ctime) == 0); + ck_assert(memdb_mkdir(memdb, "cluster-id", 0, ctime) == 0); + + ck_assert(cluster_id_ensure_standalone(memdb)); + memdb_tree_entry_t *te = memdb_getattr(memdb, "cluster-id"); + ck_assert(te != NULL); + ck_assert(te->type == DT_DIR); + memdb_tree_entry_free(te); +} +END_TEST + +static void sql_exec(const char *statement) { + sqlite3 *db = NULL; + ck_assert(sqlite3_open(TESTDB, &db) == SQLITE_OK); + ck_assert(sqlite3_exec(db, statement, NULL, NULL, NULL) == SQLITE_OK); + ck_assert(sqlite3_close(db) == SQLITE_OK); +} + +START_TEST(test_cluster_id_standalone_write_failure) { + sql_exec( + "CREATE TRIGGER fail_cluster_id BEFORE INSERT ON tree " + "WHEN NEW.name = 'cluster-id' AND length(NEW.data) > 0 " + "BEGIN SELECT RAISE(FAIL, 'injected failure'); END;" + ); + + /* creating the empty file still works, storing the content fails */ + ck_assert(!cluster_id_ensure_standalone(memdb)); + ck_assert(memdb->errors); + + memdb_close(memdb); + sql_exec("DROP TRIGGER fail_cluster_id;"); + memdb = memdb_open(TESTDB); + ck_assert(memdb != NULL); + + /* the next start finds the empty file and fills it */ + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_INVALID); + ck_assert(cluster_id_ensure_standalone(memdb)); + ck_assert(cluster_id_state(memdb) == CLUSTER_ID_VALID); +} +END_TEST + START_TEST(test_truncate_grow) { const char *fn = "/f1"; time_t ctime = 1234; @@ -590,6 +771,18 @@ static Suite *memdb_suite(void) { add_test(s, test_cluster_id_fuse_read_only, "cluster_id_fuse_read_only"); + add_test(s, test_cluster_id_encode, "cluster_id_encode"); + + add_test(s, test_cluster_id_is_valid, "cluster_id_is_valid"); + + add_test(s, test_cluster_id_generate, "cluster_id_generate"); + + add_test(s, test_cluster_id_state, "cluster_id_state"); + + add_test(s, test_cluster_id_standalone, "cluster_id_standalone"); + + add_test(s, test_cluster_id_standalone_write_failure, "cluster_id_standalone_write_failure"); + add_test(s, test_indextest1, "indextest1"); add_test(s, test_loaddb1, "loaddb1"); diff --git a/src/pmxcfs/cluster-id.c b/src/pmxcfs/cluster-id.c new file mode 100644 index 0000000..57907d1 --- /dev/null +++ b/src/pmxcfs/cluster-id.c @@ -0,0 +1,158 @@ +/* + Copyright (C) 2026 Proxmox Server Solutions GmbH + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see . + + Author: Thomas Lamprecht + +*/ + +#ifdef HAVE_CONFIG_H +#include +#endif /* HAVE_CONFIG_H */ + +#include +#include +#include +#include +#include +#include + +#include "cfs-utils.h" +#include "cluster-id.h" + +static const char base32_alphabet[] = "abcdefghijklmnopqrstuvwxyz234567"; + +/* Every character takes the next five bits, so each one is uniformly distributed, and the bits + * left over after the last character are dropped instead of padded. */ +void cluster_id_encode(const guint8 raw[CLUSTER_ID_RAW_SIZE], char id[CLUSTER_ID_FILE_SIZE]) { + guint32 bits = 0; + int pending = 0; + int pos = 0; + + for (int i = 0; i < CLUSTER_ID_RAW_SIZE && pos < CLUSTER_ID_LEN; i++) { + bits = (bits << 8) | raw[i]; + pending += 8; + + while (pending >= 5 && pos < CLUSTER_ID_LEN) { + pending -= 5; + id[pos++] = base32_alphabet[(bits >> pending) & 0x1f]; + } + } + + id[pos] = '\n'; +} + +gboolean cluster_id_generate(char id[CLUSTER_ID_FILE_SIZE]) { + guint8 raw[CLUSTER_ID_RAW_SIZE]; + + int fd = open("/dev/urandom", O_RDONLY | O_CLOEXEC); + if (fd == -1) { + cfs_critical("unable to open /dev/urandom - %s", g_strerror(errno)); + return FALSE; + } + + ssize_t res = safe_read(fd, (char *)raw, sizeof(raw)); + int saved_errno = errno; + close(fd); + + if (res != (ssize_t)sizeof(raw)) { + cfs_critical( + "unable to read from /dev/urandom - %s", + res < 0 ? g_strerror(saved_errno) : "short read" + ); + return FALSE; + } + + cluster_id_encode(raw, id); + + return TRUE; +} + +gboolean cluster_id_is_valid(gconstpointer data, size_t len) { + const char *id = data; + + if (!id || + !(len == CLUSTER_ID_LEN || (len == CLUSTER_ID_FILE_SIZE && id[CLUSTER_ID_LEN] == '\n'))) { + return FALSE; + } + + for (int i = 0; i < CLUSTER_ID_LEN; i++) { + if (!((id[i] >= 'a' && id[i] <= 'z') || (id[i] >= '2' && id[i] <= '7'))) { + return FALSE; + } + } + + return TRUE; +} + +cluster_id_state_t cluster_id_state(memdb_t *memdb) { + g_return_val_if_fail(memdb != NULL, CLUSTER_ID_NOT_A_FILE); + + memdb_tree_entry_t *te = memdb_getattr(memdb, CFS_CLUSTER_ID_FN); + if (!te) { + return CLUSTER_ID_MISSING; + } + + gboolean is_file = te->type == DT_REG; + memdb_tree_entry_free(te); + if (!is_file) { + return CLUSTER_ID_NOT_A_FILE; + } + + gpointer data = NULL; + int len = memdb_read(memdb, CFS_CLUSTER_ID_FN, &data); + gboolean valid = len >= 0 && cluster_id_is_valid(data, len); + g_free(data); + + return valid ? CLUSTER_ID_VALID : CLUSTER_ID_INVALID; +} + +gboolean cluster_id_ensure_standalone(memdb_t *memdb) { + g_return_val_if_fail(memdb != NULL, FALSE); + + cluster_id_state_t state = cluster_id_state(memdb); + if (state == CLUSTER_ID_VALID) { + return TRUE; + } + + if (state == CLUSTER_ID_NOT_A_FILE) { + cfs_message("not generating a cluster ID, '%s' is not a regular file", CFS_CLUSTER_ID_FN); + return TRUE; + } + + /* nothing got written, so the start can go on, the next one tries again */ + char id[CLUSTER_ID_FILE_SIZE]; + if (!cluster_id_generate(id)) { + return TRUE; + } + + /* nothing is mounted yet, so no reader can see the file between creating and writing it, and + * a crash in between leaves an empty file that the next start fills */ + guint32 mtime = time(NULL); + int res = state == CLUSTER_ID_MISSING ? memdb_create(memdb, CFS_CLUSTER_ID_FN, 0, mtime) : 0; + if (res == 0) { + res = memdb_write(memdb, CFS_CLUSTER_ID_FN, 0, mtime, id, CLUSTER_ID_FILE_SIZE, 0, TRUE); + } + + /* the memdb already shows the new content if only storing it failed, so it must not be used */ + if (res != CLUSTER_ID_FILE_SIZE) { + cfs_critical("unable to write '%s' - %s", CFS_CLUSTER_ID_FN, g_strerror(-res)); + return FALSE; + } + + cfs_message("generated cluster ID %.*s", CLUSTER_ID_LEN, id); + + return TRUE; +} diff --git a/src/pmxcfs/cluster-id.h b/src/pmxcfs/cluster-id.h new file mode 100644 index 0000000..996cc71 --- /dev/null +++ b/src/pmxcfs/cluster-id.h @@ -0,0 +1,52 @@ +/* + Copyright (C) 2026 Proxmox Server Solutions GmbH + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU Affero General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU Affero General Public License for more details. + + You should have received a copy of the GNU Affero General Public License + along with this program. If not, see . + + Author: Thomas Lamprecht + +*/ + +#ifndef _PVE_CLUSTER_ID_H_ +#define _PVE_CLUSTER_ID_H_ + +#include + +#include "memdb.h" + +/* the ID is 100 random bits in lowercase RFC 4648 base32, pmxcfs writes it plus a newline */ +#define CLUSTER_ID_RAW_SIZE 13 +#define CLUSTER_ID_LEN 20 +#define CLUSTER_ID_FILE_SIZE (CLUSTER_ID_LEN + 1) + +void cluster_id_encode(const guint8 raw[CLUSTER_ID_RAW_SIZE], char id[CLUSTER_ID_FILE_SIZE]); + +gboolean cluster_id_generate(char id[CLUSTER_ID_FILE_SIZE]); + +gboolean cluster_id_is_valid(gconstpointer data, size_t len); + +typedef enum { + CLUSTER_ID_VALID, + CLUSTER_ID_MISSING, + CLUSTER_ID_INVALID, + /* a directory, which only rmdir can remove */ + CLUSTER_ID_NOT_A_FILE, +} cluster_id_state_t; + +cluster_id_state_t cluster_id_state(memdb_t *memdb); + +/* FALSE only if storing the ID failed, the memdb must not be used then */ +gboolean cluster_id_ensure_standalone(memdb_t *memdb); + +#endif /* _PVE_CLUSTER_ID_H_ */ diff --git a/src/pmxcfs/pmxcfs.c b/src/pmxcfs/pmxcfs.c index 14f1168..0924eb8 100644 --- a/src/pmxcfs/pmxcfs.c +++ b/src/pmxcfs/pmxcfs.c @@ -48,6 +48,7 @@ #include "cfs-plug-memdb.h" #include "cfs-plug.h" #include "cfs-utils.h" +#include "cluster-id.h" #include "confdb.h" #include "dcdb.h" #include "dfsm.h" @@ -912,6 +913,11 @@ int main(int argc, char *argv[]) { } else { cfs_debug("using local mode (corosync.conf does not exist)"); cfs_set_quorate(1, TRUE); + /* only without a corosync.conf, forced local mode on a cluster node must not generate an ID + * of its own */ + if (!cluster_id_ensure_standalone(memdb)) { + goto err; + } } if (conf_data) { g_free(conf_data); -- 2.47.3