From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 2850C1FF0BC for ; Mon, 16 Nov 2026 17:02:08 +0100 (CET) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id B16BE2169B; Mon, 16 Nov 2026 17:01:43 +0100 (CET) From: Shannon Sterz To: pve-devel@lists.proxmox.com Subject: [PATCH cluster 2/4] fix #8325: pmxcfs: dfsm: abort send if dfsm is not joined via a handle Date: Mon, 16 Nov 2026 17:01:29 +0100 Message-ID: <20261116160131.344857-3-s.sterz@proxmox.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20261116160131.344857-1-s.sterz@proxmox.com> References: <20261116160131.344857-1-s.sterz@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1794844894128 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.600 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: DWYXZP7ZH2EUPVGWUBOMAUCLCHQCYWVF X-Message-ID-Hash: DWYXZP7ZH2EUPVGWUBOMAUCLCHQCYWVF X-MailFrom: s.sterz@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: `cpg_mcast_joined()` seems to return `CS_OK` for unfragmented messages even if `cpg_join()` has not joined a group yet. the message will be dropped and an error message is logged by Corosync ("corosync[1028]: [CPG ] *** 0xXXXXXXXXXXXXX can't mcast to group state:0, error:12"). this leads to the following situation in which a thread can get stuck: 1. a new dfsm is initialized via `dfsm_initialize()` 2. a FUSE thread tries to issue a write (`dcdb_send_fuse_message_full()`) -> this triggers a `dfsm_send_message_sync()` which in turn triggers `dfsm_send_message_full()` and `cpg_mcast_joined()`. the message is dropped as `cpg_join()` did not successfully complete yet. 3. only now `dfsm_join()` successfully returns from `cpg_join()` (note that this step can take longer if the cluster is unstable as it will be indefinitely retried). 4. the thread handling `dfsm_send_message_sync()` is stuck waiting on `dfsm->sync_cond` in theory a later message from the same node could wake the thread again. however, if the FUSE thread is trying to acquire a lock, it is highly likely that all other requests get stuck on trying to acquire the same lock before ever sending a message out via dfsm. so threads can get stuck. Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=8325 Signed-off-by: Shannon Sterz --- Notes: to reproduce run the following: - set up at least a quorate two node cluster. - install pve-cluster debug symbols via the debug repo [1] and gdb (`apt install pve-cluster-dbgsym gdb`) - save the following gdb script in `reproducer.gdb`: ```gdb tbreak dfsm_dispatch if $_streq(dfsm->log_domain, "dcdb") commands return CS_ERR_LIBRARY continue end break dfsm_join if $_streq(dfsm->log_domain, "dcdb") ``` this sets two break points: 1. a temporary breakpoint that will return an error on the next `dfsm_dispatch()` call to simulate a dropped corosync connection. this will trigger pmxcfs to re-initialize dfsm. 2. a break point on `dfsm_join()` that will make the next `dcdb` join take as long as needed. - open three shells to the same node of the cluster 1. run `journalctl -u pve-cluster.service -u corosync.service -f` 2. run `gdb -iex 'set non-stop on' -iex 'set pagination off' -iex\ 'set confirm off' -x reproducer.gdb -p "$(pidof pmxcfs)"` this will attach gdb to pmxcfs in non-stop mode and execute the script from the previous step. type `continue -a` and hit enter. wait until gdb reports a stop at `dfsm_join`. note that the temporary breakpoint at `dfsm_dispatch()` may need an event on another node, however by default `pvestatd` is likely to do this for you anyway. 3. run `mkdir /etc/pve/priv/lock/repro &` - run `del 2` to remove the second breakpoint (the first one should be removed automatically since its a temporary one) and then execute `continue -a` - the `mkdir` process in terminal 3. should be stuck in D state now and the journal in terminal 1. should contain log lines such as: ``` corosync[1208]: [CPG ] *** 0xXXXXXXXXXXXXX can't mcast to group state:0, error:12 ``` with this patch applied the reproducer will stop working, the fuse request will simply fail (it should log `cpg_send_message failed: CS_ERR_NOT_EXIST`). nothing about a failed mcast will be logged and processes should not be stuck in D state. [1]: https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_dbgsym_repo src/pmxcfs/dfsm.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/src/pmxcfs/dfsm.c b/src/pmxcfs/dfsm.c index b9d07d0..741db54 100644 --- a/src/pmxcfs/dfsm.c +++ b/src/pmxcfs/dfsm.c @@ -187,7 +187,12 @@ dfsm_send_message_full(dfsm_t *dfsm, struct iovec *iov, unsigned int len, int re int retries = 0; loop: g_mutex_lock(&dfsm->cpg_mutex); - result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len); + /* only send when joined, Corosync will otherwise silently drop messages. */ + if (!dfsm->joined) { + result = CS_ERR_NOT_EXIST; + } else { + result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len); + } g_mutex_unlock(&dfsm->cpg_mutex); if (retry && result == CS_ERR_TRY_AGAIN) { nanosleep(&tvreq, NULL); @@ -1362,7 +1367,9 @@ cs_error_t dfsm_initialize(dfsm_t *dfsm, int *fd) { dfsm_free_message_queue(dfsm); dfsm_send_sync_message_abort(dfsm); + g_mutex_lock(&dfsm->cpg_mutex); dfsm->joined = FALSE; + g_mutex_unlock(&dfsm->cpg_mutex); dfsm->we_are_member = 0; dfsm_set_mode(dfsm, DFSM_MODE_START); @@ -1425,6 +1432,9 @@ cs_error_t dfsm_join(dfsm_t *dfsm) { loop: g_mutex_lock(&dfsm->cpg_mutex); result = cpg_join(dfsm->cpg_handle, &dfsm->cpg_group_name); + if (result == CS_OK) { + dfsm->joined = TRUE; + } g_mutex_unlock(&dfsm->cpg_mutex); if (result == CS_ERR_TRY_AGAIN) { nanosleep(&tvreq, NULL); @@ -1440,7 +1450,6 @@ loop: return result; } - dfsm->joined = TRUE; return TRUE; } @@ -1455,6 +1464,9 @@ cs_error_t dfsm_leave(dfsm_t *dfsm) { loop: g_mutex_lock(&dfsm->cpg_mutex); result = cpg_leave(dfsm->cpg_handle, &dfsm->cpg_group_name); + if (result == CS_OK) { + dfsm->joined = FALSE; + } g_mutex_unlock(&dfsm->cpg_mutex); if (result == CS_ERR_TRY_AGAIN) { nanosleep(&tvreq, NULL); @@ -1470,8 +1482,6 @@ loop: return result; } - dfsm->joined = FALSE; - return TRUE; } @@ -1485,9 +1495,11 @@ gboolean dfsm_finalize(dfsm_t *dfsm) { } if (dfsm->cpg_handle) { + g_mutex_lock(&dfsm->cpg_mutex); cpg_finalize(dfsm->cpg_handle); dfsm->cpg_handle = 0; dfsm->joined = FALSE; + g_mutex_unlock(&dfsm->cpg_mutex); dfsm->we_are_member = 0; } -- 2.47.3