all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: Shannon Sterz <s.sterz@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH cluster 2/4] fix #8325: pmxcfs: dfsm: abort send if dfsm is not joined via a handle
Date: Mon, 16 Nov 2026 17:01:29 +0100	[thread overview]
Message-ID: <20261116160131.344857-3-s.sterz@proxmox.com> (raw)
In-Reply-To: <20261116160131.344857-1-s.sterz@proxmox.com>

`cpg_mcast_joined()` seems to return `CS_OK` for unfragmented messages
even if `cpg_join()` has not joined a group yet. the message will be
dropped and an error message is logged by Corosync ("corosync[1028]:
[CPG   ] *** 0xXXXXXXXXXXXXX can't mcast to group state:0, error:12").

this leads to the following situation in which a thread can get stuck:

1. a new dfsm is initialized via `dfsm_initialize()`
2. a FUSE thread tries to issue a write
   (`dcdb_send_fuse_message_full()`) -> this triggers a
   `dfsm_send_message_sync()` which in turn triggers
   `dfsm_send_message_full()` and `cpg_mcast_joined()`. the message is
   dropped as `cpg_join()` did not successfully complete yet.
3. only now `dfsm_join()` successfully returns from `cpg_join()` (note
   that this step can take longer if the cluster is unstable as it
   will be indefinitely retried).
4. the thread handling `dfsm_send_message_sync()` is stuck waiting on
   `dfsm->sync_cond`

in theory a later message from the same node could wake the thread
again. however, if the FUSE thread is trying to acquire a lock, it is
highly likely that all other requests get stuck on trying to acquire
the same lock before ever sending a message out via dfsm. so threads
can get stuck.

Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=8325
Signed-off-by: Shannon Sterz <s.sterz@proxmox.com>
---

Notes:
    to reproduce run the following:
    
    - set up at least a quorate two node cluster.
    - install pve-cluster debug symbols via the debug repo [1] and gdb
      (`apt install pve-cluster-dbgsym gdb`)
    - save the following gdb script in `reproducer.gdb`:
      ```gdb
      tbreak dfsm_dispatch if $_streq(dfsm->log_domain, "dcdb")
      commands
        return CS_ERR_LIBRARY
        continue
      end
    
      break dfsm_join if $_streq(dfsm->log_domain, "dcdb")
      ```
        this sets two break points:
        1. a temporary breakpoint that will return an error on the next
           `dfsm_dispatch()` call to simulate a dropped corosync
           connection. this will trigger pmxcfs to re-initialize dfsm.
        2. a break point on `dfsm_join()` that will make the next `dcdb`
           join take as long as needed.
    - open three shells to the same node of the cluster
        1. run `journalctl -u pve-cluster.service -u corosync.service -f`
        2. run `gdb -iex 'set non-stop on' -iex 'set pagination off' -iex\
           'set confirm off' -x reproducer.gdb -p "$(pidof pmxcfs)"`
           this will attach gdb to pmxcfs in non-stop mode and execute the
           script from the previous step. type `continue -a` and hit
           enter. wait until gdb reports a stop at `dfsm_join`. note that
           the temporary breakpoint at `dfsm_dispatch()` may need an event
           on another node, however by default `pvestatd` is likely to do
           this for you anyway.
        3. run `mkdir /etc/pve/priv/lock/repro &`
    - run `del 2` to remove the second breakpoint (the first one should be
      removed automatically since its a temporary one) and then execute
      `continue -a`
    - the `mkdir` process in terminal 3. should be stuck in D state now
      and the journal in terminal 1. should contain log lines such as:
      ```
      corosync[1208]:   [CPG   ] *** 0xXXXXXXXXXXXXX can't mcast to group  state:0, error:12
      ```
    
    with this patch applied the reproducer will stop working, the fuse
    request will simply fail (it should log `cpg_send_message failed:
    CS_ERR_NOT_EXIST`). nothing about a failed mcast will be logged and
    processes should not be stuck in D state.
    
    [1]: https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_dbgsym_repo

 src/pmxcfs/dfsm.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/src/pmxcfs/dfsm.c b/src/pmxcfs/dfsm.c
index b9d07d0..741db54 100644
--- a/src/pmxcfs/dfsm.c
+++ b/src/pmxcfs/dfsm.c
@@ -187,7 +187,12 @@ dfsm_send_message_full(dfsm_t *dfsm, struct iovec *iov, unsigned int len, int re
     int retries = 0;
 loop:
     g_mutex_lock(&dfsm->cpg_mutex);
-    result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len);
+    /* only send when joined, Corosync will otherwise silently drop messages. */
+    if (!dfsm->joined) {
+        result = CS_ERR_NOT_EXIST;
+    } else {
+        result = cpg_mcast_joined(dfsm->cpg_handle, CPG_TYPE_AGREED, iov, len);
+    }
     g_mutex_unlock(&dfsm->cpg_mutex);
     if (retry && result == CS_ERR_TRY_AGAIN) {
         nanosleep(&tvreq, NULL);
@@ -1362,7 +1367,9 @@ cs_error_t dfsm_initialize(dfsm_t *dfsm, int *fd) {
     dfsm_free_message_queue(dfsm);
     dfsm_send_sync_message_abort(dfsm);
 
+    g_mutex_lock(&dfsm->cpg_mutex);
     dfsm->joined = FALSE;
+    g_mutex_unlock(&dfsm->cpg_mutex);
     dfsm->we_are_member = 0;
 
     dfsm_set_mode(dfsm, DFSM_MODE_START);
@@ -1425,6 +1432,9 @@ cs_error_t dfsm_join(dfsm_t *dfsm) {
 loop:
     g_mutex_lock(&dfsm->cpg_mutex);
     result = cpg_join(dfsm->cpg_handle, &dfsm->cpg_group_name);
+    if (result == CS_OK) {
+        dfsm->joined = TRUE;
+    }
     g_mutex_unlock(&dfsm->cpg_mutex);
     if (result == CS_ERR_TRY_AGAIN) {
         nanosleep(&tvreq, NULL);
@@ -1440,7 +1450,6 @@ loop:
         return result;
     }
 
-    dfsm->joined = TRUE;
     return TRUE;
 }
 
@@ -1455,6 +1464,9 @@ cs_error_t dfsm_leave(dfsm_t *dfsm) {
 loop:
     g_mutex_lock(&dfsm->cpg_mutex);
     result = cpg_leave(dfsm->cpg_handle, &dfsm->cpg_group_name);
+    if (result == CS_OK) {
+        dfsm->joined = FALSE;
+    }
     g_mutex_unlock(&dfsm->cpg_mutex);
     if (result == CS_ERR_TRY_AGAIN) {
         nanosleep(&tvreq, NULL);
@@ -1470,8 +1482,6 @@ loop:
         return result;
     }
 
-    dfsm->joined = FALSE;
-
     return TRUE;
 }
 
@@ -1485,9 +1495,11 @@ gboolean dfsm_finalize(dfsm_t *dfsm) {
     }
 
     if (dfsm->cpg_handle) {
+        g_mutex_lock(&dfsm->cpg_mutex);
         cpg_finalize(dfsm->cpg_handle);
         dfsm->cpg_handle = 0;
         dfsm->joined = FALSE;
+        g_mutex_unlock(&dfsm->cpg_mutex);
         dfsm->we_are_member = 0;
     }
 
-- 
2.47.3





  parent reply	other threads:[~2026-11-16 16:02 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16 16:01 [PATCH cluster 0/4] Fix 8325 and minor pmxcfs Clean-Up Shannon Sterz
2026-11-16 16:01 ` [PATCH cluster 1/4] tree wide: update copyright notice to current year Shannon Sterz
2026-11-16 16:01 ` Shannon Sterz [this message]
2026-11-16 16:01 ` [PATCH cluster 3/4] pmxcfs: dfsm: wake waiters in dfsm_send_message_sync() on error mode Shannon Sterz
2026-11-16 16:01 ` [PATCH cluster 4/4] pmxcfs: dfsm: make sure error results are processed correctly Shannon Sterz
2026-11-17  1:32 ` applied: [PATCH cluster 0/4] Fix 8325 and minor pmxcfs Clean-Up Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261116160131.344857-3-s.sterz@proxmox.com \
    --to=s.sterz@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal