public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Shannon Sterz <s.sterz@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH cluster 2/4] fix #8325: pmxcfs: dfsm: abort send if dfsm is not joined via a handle
Date: Mon, 16 Nov 2026 17:01:29 +0100	[thread overview]
Message-ID: <20261116160131.344857-3-s.sterz@proxmox.com> (raw)
In-Reply-To: <20261116160131.344857-1-s.sterz@proxmox.com>

`cpg_mcast_joined()` seems to return `CS_OK` for unfragmented messages
even if `cpg_join()` has not joined a group yet. the message will be
dropped and an error message is logged by Corosync ("corosync[1028]:
[CPG   ] *** 0xXXXXXXXXXXXXX can't mcast to group state:0, error:12").

this leads to the following situation in which a thread can get stuck:

1. a new dfsm is initialized via `dfsm_initialize()`
2. a FUSE thread tries to issue a write
   (`dcdb_send_fuse_message_full()`) -> this triggers a
   `dfsm_send_message_sync()` which in turn triggers
   `dfsm_send_message_full()` and `cpg_mcast_joined()`. the message is
   dropped as `cpg_join()` did not successfully complete yet.
3. only now `dfsm_join()` successfully returns from `cpg_join()` (note
   that this step can take longer if the cluster is unstable as it
   will be indefinitely retried).
4. the thread handling `dfsm_send_message_sync()` is stuck waiting on
   `dfsm->sync_cond`

in theory a later message from the same node could wake the thread
again. however, if the FUSE thread is trying to acquire a lock, it is
highly likely that all other requests get stuck on trying to acquire
the same lock before ever sending a message out via dfsm. so threads
can get stuck.

Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=8325
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
---

Notes:
    to reproduce run the following:
    
    - set up at least a quorate two node cluster.
    - install pve-cluster debug symbols via the debug repo [1] and gdb
      (`apt install pve-cluster-dbgsym gdb`)
    - save the following gdb script in `reproducer.gdb`:
      ```gdb
      tbreak dfsm_dispatch if $_streq(dfsm->log_domain, "dcdb")
      commands
        return CS_ERR_LIBRARY
        continue
      end
    
      break dfsm_join if $_streq(dfsm->log_domain, "dcdb")
      ```
        this sets two break points:
        1. a temporary breakpoint that will return an error on the next
           `dfsm_dispatch()` call to simulate a dropped corosync
           connection. this will trigger pmxcfs to re-initialize dfsm.
        2. a break point on `dfsm_join()` that will make the next `dcdb`
           join take as long as needed.
    - open three shells to the same node of the cluster
        1. run `journalctl -u pve-cluster.service -u corosync.service -f`
        2. run `gdb -iex 'set non-stop on' -iex 'set pagination off' -iex\
           'set confirm off' -x reproducer.gdb -p "$(pidof pmxcfs)"`
           this will attach gdb to pmxcfs in non-stop mode and execute the
           script from the previous step. type `continue -a` and hit
           enter. wait until gdb reports a stop at `dfsm_join`. note that
           the temporary breakpoint at `dfsm_dispatch()` may need an event
           on another node, however by default `pvestatd` is likely to do
           this for you anyway.
        3. run `mkdir /etc/pve/priv/lock/repro &`
    - run `del 2` to remove the second breakpoint (the first one should be
      removed automatically since its a temporary one) and then execute
      `continue -a`
    - the `mkdir` process in terminal 3. should be stuck in D state now
      and the journal in terminal 1. should contain log lines such as:
      ```
      corosync[1208]:   [CPG   ] *** 0xXXXXXXXXXXXXX can't mcast to group  state:0, error:12
      ```
    
    with this patch applied the reproducer will stop working, the fuse
    request will simply fail (it should log `cpg_send_message failed:
    CS_ERR_NOT_EXIST`). nothing about a failed mcast will be logged and
    processes should not be stuck in D state.
    
    [1]: https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_dbgsym_repo

 src/pmxcfs/dfsm.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/src/pmxcfs/dfsm.c b/src/pmxcfs/dfsm.c
index b9d07d0..741db54 100644
--- a/src/pmxcfs/dfsm.c
+++ b/src/pmxcfs/dfsm.c
@@ -187,7 +187,12 @@ dfsm_send_message_full(dfsm_t *dfsm, struct iovec *iov, unsigned int len, int re
     int retries = 0;
 loop:
     g_mutex_lock(&dfsm->cpg_mutex);
-    result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len);
+    /* only send when joined, Corosync will otherwise silently drop messages. */
+    if (!dfsm->joined) {
+        result = CS_ERR_NOT_EXIST;
+    } else {
+        result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len);
+    }
     g_mutex_unlock(&dfsm->cpg_mutex);
     if (retry && result == CS_ERR_TRY_AGAIN) {
         nanosleep(&tvreq, NULL);
@@ -1362,7 +1367,9 @@ cs_error_t dfsm_initialize(dfsm_t *dfsm, int *fd) {
     dfsm_free_message_queue(dfsm);
     dfsm_send_sync_message_abort(dfsm);
 
+    g_mutex_lock(&dfsm->cpg_mutex);
     dfsm->joined = FALSE;
+    g_mutex_unlock(&dfsm->cpg_mutex);
     dfsm->we_are_member = 0;
 
     dfsm_set_mode(dfsm, DFSM_MODE_START);
@@ -1425,6 +1432,9 @@ cs_error_t dfsm_join(dfsm_t *dfsm) {
 loop:
     g_mutex_lock(&dfsm->cpg_mutex);
     result = cpg_join(dfsm->cpg_handle, &dfsm->cpg_group_name);
+    if (result == CS_OK) {
+        dfsm->joined = TRUE;
+    }
     g_mutex_unlock(&dfsm->cpg_mutex);
     if (result == CS_ERR_TRY_AGAIN) {
         nanosleep(&tvreq, NULL);
@@ -1440,7 +1450,6 @@ loop:
         return result;
     }
 
-    dfsm->joined = TRUE;
     return TRUE;
 }
 
@@ -1455,6 +1464,9 @@ cs_error_t dfsm_leave(dfsm_t *dfsm) {
 loop:
     g_mutex_lock(&dfsm->cpg_mutex);
     result = cpg_leave(dfsm->cpg_handle, &dfsm->cpg_group_name);
+    if (result == CS_OK) {
+        dfsm->joined = FALSE;
+    }
     g_mutex_unlock(&dfsm->cpg_mutex);
     if (result == CS_ERR_TRY_AGAIN) {
         nanosleep(&tvreq, NULL);
@@ -1470,8 +1482,6 @@ loop:
         return result;
     }
 
-    dfsm->joined = FALSE;
-
     return TRUE;
 }
 
@@ -1485,9 +1495,11 @@ gboolean dfsm_finalize(dfsm_t *dfsm) {
     }
 
     if (dfsm->cpg_handle) {
+        g_mutex_lock(&dfsm->cpg_mutex);
         cpg_finalize(dfsm->cpg_handle);
         dfsm->cpg_handle = 0;
         dfsm->joined = FALSE;
+        g_mutex_unlock(&dfsm->cpg_mutex);
         dfsm->we_are_member = 0;
     }
 
-- 
2.47.3





  parent reply	other threads:[~2026-11-16 16:02 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 16:01 [PATCH cluster 0/4] Fix 8325 and minor pmxcfs Clean-Up Shannon Sterz
2026-11-16 16:01 ` [PATCH cluster 1/4] tree wide: update copyright notice to current year Shannon Sterz
2026-11-16 16:01 ` Shannon Sterz [this message]
2026-11-16 16:01 ` [PATCH cluster 3/4] pmxcfs: dfsm: wake waiters in dfsm_send_message_sync() on error mode Shannon Sterz
2026-11-16 16:01 ` [PATCH cluster 4/4] pmxcfs: dfsm: make sure error results are processed correctly Shannon Sterz
2026-11-17  1:32 ` applied: [PATCH cluster 0/4] Fix 8325 and minor pmxcfs Clean-Up Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261116160131.344857-3-s.sterz@proxmox.com \
    --to=s.sterz@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal