From: Shannon Sterz <s.sterz@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH cluster 2/4] fix #8325: pmxcfs: dfsm: abort send if dfsm is not joined via a handle
Date: Mon, 16 Nov 2026 17:01:29 +0100 [thread overview]
Message-ID: <20261116160131.344857-3-s.sterz@proxmox.com> (raw)
In-Reply-To: <20261116160131.344857-1-s.sterz@proxmox.com>
`cpg_mcast_joined()` seems to return `CS_OK` for unfragmented messages
even if `cpg_join()` has not joined a group yet. the message will be
dropped and an error message is logged by Corosync ("corosync[1028]:
[CPG ] *** 0xXXXXXXXXXXXXX can't mcast to group state:0, error:12").
this leads to the following situation in which a thread can get stuck:
1. a new dfsm is initialized via `dfsm_initialize()`
2. a FUSE thread tries to issue a write
(`dcdb_send_fuse_message_full()`) -> this triggers a
`dfsm_send_message_sync()` which in turn triggers
`dfsm_send_message_full()` and `cpg_mcast_joined()`. the message is
dropped as `cpg_join()` did not successfully complete yet.
3. only now `dfsm_join()` successfully returns from `cpg_join()` (note
that this step can take longer if the cluster is unstable as it
will be indefinitely retried).
4. the thread handling `dfsm_send_message_sync()` is stuck waiting on
`dfsm->sync_cond`
in theory a later message from the same node could wake the thread
again. however, if the FUSE thread is trying to acquire a lock, it is
highly likely that all other requests get stuck on trying to acquire
the same lock before ever sending a message out via dfsm. so threads
can get stuck.
Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=8325
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
---
Notes:
to reproduce run the following:
- set up at least a quorate two node cluster.
- install pve-cluster debug symbols via the debug repo [1] and gdb
(`apt install pve-cluster-dbgsym gdb`)
- save the following gdb script in `reproducer.gdb`:
```gdb
tbreak dfsm_dispatch if $_streq(dfsm->log_domain, "dcdb")
commands
return CS_ERR_LIBRARY
continue
end
break dfsm_join if $_streq(dfsm->log_domain, "dcdb")
```
this sets two break points:
1. a temporary breakpoint that will return an error on the next
`dfsm_dispatch()` call to simulate a dropped corosync
connection. this will trigger pmxcfs to re-initialize dfsm.
2. a break point on `dfsm_join()` that will make the next `dcdb`
join take as long as needed.
- open three shells to the same node of the cluster
1. run `journalctl -u pve-cluster.service -u corosync.service -f`
2. run `gdb -iex 'set non-stop on' -iex 'set pagination off' -iex\
'set confirm off' -x reproducer.gdb -p "$(pidof pmxcfs)"`
this will attach gdb to pmxcfs in non-stop mode and execute the
script from the previous step. type `continue -a` and hit
enter. wait until gdb reports a stop at `dfsm_join`. note that
the temporary breakpoint at `dfsm_dispatch()` may need an event
on another node, however by default `pvestatd` is likely to do
this for you anyway.
3. run `mkdir /etc/pve/priv/lock/repro &`
- run `del 2` to remove the second breakpoint (the first one should be
removed automatically since its a temporary one) and then execute
`continue -a`
- the `mkdir` process in terminal 3. should be stuck in D state now
and the journal in terminal 1. should contain log lines such as:
```
corosync[1208]: [CPG ] *** 0xXXXXXXXXXXXXX can't mcast to group state:0, error:12
```
with this patch applied the reproducer will stop working, the fuse
request will simply fail (it should log `cpg_send_message failed:
CS_ERR_NOT_EXIST`). nothing about a failed mcast will be logged and
processes should not be stuck in D state.
[1]: https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_dbgsym_repo
src/pmxcfs/dfsm.c | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
diff --git a/src/pmxcfs/dfsm.c b/src/pmxcfs/dfsm.c
index b9d07d0..741db54 100644
--- a/src/pmxcfs/dfsm.c
+++ b/src/pmxcfs/dfsm.c
@@ -187,7 +187,12 @@ dfsm_send_message_full(dfsm_t *dfsm, struct iovec *iov, unsigned int len, int re
int retries = 0;
loop:
g_mutex_lock(&dfsm->cpg_mutex);
- result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len);
+ /* only send when joined, Corosync will otherwise silently drop messages. */
+ if (!dfsm->joined) {
+ result = CS_ERR_NOT_EXIST;
+ } else {
+ result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len);
+ }
g_mutex_unlock(&dfsm->cpg_mutex);
if (retry && result == CS_ERR_TRY_AGAIN) {
nanosleep(&tvreq, NULL);
@@ -1362,7 +1367,9 @@ cs_error_t dfsm_initialize(dfsm_t *dfsm, int *fd) {
dfsm_free_message_queue(dfsm);
dfsm_send_sync_message_abort(dfsm);
+ g_mutex_lock(&dfsm->cpg_mutex);
dfsm->joined = FALSE;
+ g_mutex_unlock(&dfsm->cpg_mutex);
dfsm->we_are_member = 0;
dfsm_set_mode(dfsm, DFSM_MODE_START);
@@ -1425,6 +1432,9 @@ cs_error_t dfsm_join(dfsm_t *dfsm) {
loop:
g_mutex_lock(&dfsm->cpg_mutex);
result = cpg_join(dfsm->cpg_handle, &dfsm->cpg_group_name);
+ if (result == CS_OK) {
+ dfsm->joined = TRUE;
+ }
g_mutex_unlock(&dfsm->cpg_mutex);
if (result == CS_ERR_TRY_AGAIN) {
nanosleep(&tvreq, NULL);
@@ -1440,7 +1450,6 @@ loop:
return result;
}
- dfsm->joined = TRUE;
return TRUE;
}
@@ -1455,6 +1464,9 @@ cs_error_t dfsm_leave(dfsm_t *dfsm) {
loop:
g_mutex_lock(&dfsm->cpg_mutex);
result = cpg_leave(dfsm->cpg_handle, &dfsm->cpg_group_name);
+ if (result == CS_OK) {
+ dfsm->joined = FALSE;
+ }
g_mutex_unlock(&dfsm->cpg_mutex);
if (result == CS_ERR_TRY_AGAIN) {
nanosleep(&tvreq, NULL);
@@ -1470,8 +1482,6 @@ loop:
return result;
}
- dfsm->joined = FALSE;
-
return TRUE;
}
@@ -1485,9 +1495,11 @@ gboolean dfsm_finalize(dfsm_t *dfsm) {
}
if (dfsm->cpg_handle) {
+ g_mutex_lock(&dfsm->cpg_mutex);
cpg_finalize(dfsm->cpg_handle);
dfsm->cpg_handle = 0;
dfsm->joined = FALSE;
+ g_mutex_unlock(&dfsm->cpg_mutex);
dfsm->we_are_member = 0;
}
--
2.47.3
next prev parent reply other threads:[~2026-11-16 16:02 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 16:01 [PATCH cluster 0/4] Fix 8325 and minor pmxcfs Clean-Up Shannon Sterz
2026-11-16 16:01 ` [PATCH cluster 1/4] tree wide: update copyright notice to current year Shannon Sterz
2026-11-16 16:01 ` Shannon Sterz [this message]
2026-11-16 16:01 ` [PATCH cluster 3/4] pmxcfs: dfsm: wake waiters in dfsm_send_message_sync() on error mode Shannon Sterz
2026-11-16 16:01 ` [PATCH cluster 4/4] pmxcfs: dfsm: make sure error results are processed correctly Shannon Sterz
2026-11-17 1:32 ` applied: [PATCH cluster 0/4] Fix 8325 and minor pmxcfs Clean-Up Thomas Lamprecht
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261116160131.344857-3-s.sterz@proxmox.com \
--to=s.sterz@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox