public inbox for pve-devel@lists.proxmox.com
 help / color / mirror / Atom feed
From: Kefu Chai <k.chai@proxmox.com>
To: pve-devel@lists.proxmox.com
Subject: [PATCH pve-ceph squid-stable-9] patches: backport messenger reverts to fix peering after OSD restarts
Date: Sun, 15 Nov 2026 21:09:01 +0800	[thread overview]
Message-ID: <20261115130901.937954-1-k.chai@proxmox.com> (raw)

OSD connections racing after a restart can lose peering messages, leaving
PGs stuck in peering or activating until an OSDMap change or a manual
repeer restarts the handshake. Affected PGs block client I/O.

Carry all three messenger reverts from upstream Squid PR #71956, in their
original order. The equivalent reverts already ship in Tentacle 20.2.5;
for Squid they are merged for 19.2.8 but have not been released yet.

Link: https://tracker.ceph.com/issues/80327
Link: https://github.com/ceph/ceph/pull/71956
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
---
 ...c-ProtocolV2-send_server_ident-updat.patch | 81 ++++++++++++++++
 ...-async-Improve-logging-and-prefixes-.patch | 94 +++++++++++++++++++
 ...nc-ProtocolV2-Speed-up-connection-lo.patch | 86 +++++++++++++++++
 patches/series                                |  3 +
 4 files changed, 264 insertions(+)
 create mode 100644 patches/0058-Revert-msg-async-ProtocolV2-send_server_ident-updat.patch
 create mode 100644 patches/0059-Revert-src-msg-async-Improve-logging-and-prefixes-.patch
 create mode 100644 patches/0060-Revert-msg-async-ProtocolV2-Speed-up-connection-lo.patch

diff --git a/patches/0058-Revert-msg-async-ProtocolV2-send_server_ident-updat.patch b/patches/0058-Revert-msg-async-ProtocolV2-send_server_ident-updat.patch
new file mode 100644
index 0000000000..305b6ba309
--- /dev/null
+++ b/patches/0058-Revert-msg-async-ProtocolV2-send_server_ident-updat.patch
@@ -0,0 +1,81 @@
+From ee490d3cce8fd504301c535e721c3640764729ba Mon Sep 17 00:00:00 2001
+From: "Kamoltat (Junior) Sirivadhna" <ksirivad@redhat.com>
+Date: Mon, 21 Sep 2026 18:08:23 +0000
+Subject: [PATCH 1/3] Revert "msg/async: ProtocolV2::send_server_ident update
+ ProtocolV2::global_seq"
+
+This reverts commit ae5d336fa447f8d19262a26e929c48126de146c6.
+
+Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
+
+Conflicts:
+	src/msg/async/ProtocolV2.cc
+	Accept:
+	    auto& oq = out_queue[m->get_priority()];
+            oq.emplace_back(out_queue_entry_t{is_prepared, std::move(m)});
+(cherry picked from commit ee490d3cce8fd504301c535e721c3640764729ba)
+---
+ src/msg/async/AsyncMessenger.cc | 7 ++-----
+ src/msg/async/ProtocolV2.cc     | 6 +++---
+ src/msg/async/ProtocolV2.h      | 2 +-
+ 3 files changed, 6 insertions(+), 9 deletions(-)
+
+diff --git a/src/msg/async/AsyncMessenger.cc b/src/msg/async/AsyncMessenger.cc
+index 97568a0779e2..639b59937065 100644
+--- a/src/msg/async/AsyncMessenger.cc
++++ b/src/msg/async/AsyncMessenger.cc
+@@ -1082,12 +1082,9 @@ __u32 AsyncMessenger::get_global_seq(__u32 old_global_seq)
+   } // release lock
+ 
+   if (did_update_to_old) {
+-    ldout(cct, 10) 
+-      << __func__ << " old_global_seq=" << old_global_seq
++    ldout(cct, 10) << __func__ << " old_global_seq=" << old_global_seq
+       << " > global_seq=" << global_seq
+-      << "; new global_seq=" << updated_to
+-      << " (was " << prev_global << ")"
+-      << dendl;
++      << "; new global_seq=" << updated_to << dendl;
+   }
+   ldout(cct, 10) << __func__ << " increment to global_seq=" << global_seq << dendl;
+   return ret;
+diff --git a/src/msg/async/ProtocolV2.cc b/src/msg/async/ProtocolV2.cc
+index 25fcd46242ca..a0672507464a 100644
+--- a/src/msg/async/ProtocolV2.cc
++++ b/src/msg/async/ProtocolV2.cc
+@@ -2977,11 +2977,11 @@ CtPtr ProtocolV2::send_server_ident() {
+     flags = flags | CEPH_MSG_CONNECT_LOSSY;
+   }
+ 
+-  global_seq = messenger->get_global_seq();
++  uint64_t gs = messenger->get_global_seq();
+   auto server_ident = ServerIdentFrame::Encode(
+           messenger->get_myaddrs(),
+           messenger->get_myname().num(),
+-          global_seq,
++          gs,
+           connection->policy.features_supported,
+           connection->policy.features_required | msgr2_required,
+           flags,
+@@ -2990,7 +2990,7 @@ CtPtr ProtocolV2::send_server_ident() {
+   ldout(cct, 5) << __func__ << " sending identification:"
+                 << " addrs=" << messenger->get_myaddrs()
+                 << " gid=" << messenger->get_myname().num()
+-                << " global_seq=" << global_seq << " features_supported=" << std::hex
++                << " global_seq=" << gs << " features_supported=" << std::hex
+                 << connection->policy.features_supported
+                 << " features_required="
+ 		            << (connection->policy.features_required | msgr2_required)
+diff --git a/src/msg/async/ProtocolV2.h b/src/msg/async/ProtocolV2.h
+index a70be01c9a38..786fa4db6d5c 100644
+--- a/src/msg/async/ProtocolV2.h
++++ b/src/msg/async/ProtocolV2.h
+@@ -84,7 +84,7 @@ class ProtocolV2 : public Protocol {
+ 
+   uint64_t client_cookie;
+   uint64_t server_cookie;
+-  uint64_t global_seq; // Snapshot of AsyncMessenger::global_seq
++  uint64_t global_seq;
+   uint64_t connect_seq;
+   uint64_t peer_global_seq;
+   uint64_t message_seq;
diff --git a/patches/0059-Revert-src-msg-async-Improve-logging-and-prefixes-.patch b/patches/0059-Revert-src-msg-async-Improve-logging-and-prefixes-.patch
new file mode 100644
index 0000000000..0a3a466950
--- /dev/null
+++ b/patches/0059-Revert-src-msg-async-Improve-logging-and-prefixes-.patch
@@ -0,0 +1,94 @@
+From 38641deba3cc8d154215436530bb8912983ee8b9 Mon Sep 17 00:00:00 2001
+From: "Kamoltat (Junior) Sirivadhna" <ksirivad@redhat.com>
+Date: Mon, 21 Sep 2026 18:10:03 +0000
+Subject: [PATCH 2/3] Revert "src/msg/async: Improve logging and prefixes for
+ global_seq"
+
+This reverts commit 958406ef50160dd5c34f57730d9eaff8b2a0d94a.
+
+Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
+(cherry picked from commit 38641deba3cc8d154215436530bb8912983ee8b9)
+---
+ src/msg/async/AsyncMessenger.cc | 31 -------------------------------
+ src/msg/async/AsyncMessenger.h  |  9 ++++++++-
+ src/msg/async/ProtocolV1.cc     |  4 ++--
+ 3 files changed, 10 insertions(+), 34 deletions(-)
+
+diff --git a/src/msg/async/AsyncMessenger.cc b/src/msg/async/AsyncMessenger.cc
+index 639b59937065..3a8f9e0c1894 100644
+--- a/src/msg/async/AsyncMessenger.cc
++++ b/src/msg/async/AsyncMessenger.cc
+@@ -1059,37 +1059,6 @@ void AsyncMessenger::mark_down_addrs(const entity_addrvec_t& addrs)
+   }
+ }
+ 
+-__u32 AsyncMessenger::get_global_seq(__u32 old_global_seq)
+-{
+-  __u32 ret;
+-  // These are only used for logging.
+-  __u32 prev_global = 0;
+-  __u32 updated_to = 0;
+-  bool did_update_to_old = false;
+-
+-  { // acquire lock
+-    std::lock_guard<ceph::spinlock> lg(global_seq_lock);
+-
+-    if (old_global_seq > global_seq) {
+-      // These are all for logging purposes
+-      prev_global = global_seq;
+-      updated_to = old_global_seq;
+-      did_update_to_old = true;
+-      // global_seq and ret are the only that matters.
+-      global_seq = old_global_seq;
+-    }
+-    ret = ++global_seq;
+-  } // release lock
+-
+-  if (did_update_to_old) {
+-    ldout(cct, 10) << __func__ << " old_global_seq=" << old_global_seq
+-      << " > global_seq=" << global_seq
+-      << "; new global_seq=" << updated_to << dendl;
+-  }
+-  ldout(cct, 10) << __func__ << " increment to global_seq=" << global_seq << dendl;
+-  return ret;
+-}
+-
+ int AsyncMessenger::get_proto_version(int peer_type, bool connect) const
+ {
+   int my_type = my_name.type();
+diff --git a/src/msg/async/AsyncMessenger.h b/src/msg/async/AsyncMessenger.h
+index 8d5c8f709c5a..7003bf939f73 100644
+--- a/src/msg/async/AsyncMessenger.h
++++ b/src/msg/async/AsyncMessenger.h
+@@ -401,8 +401,15 @@ class AsyncMessenger : public SimplePolicyMessenger {
+    *
+    * @return a global sequence ID that nobody else has seen.
+    */
+-  __u32 get_global_seq(__u32 old=0);
++  __u32 get_global_seq(__u32 old=0) {
++    std::lock_guard<ceph::spinlock> lg(global_seq_lock);
+ 
++    if (old > global_seq)
++      global_seq = old;
++    __u32 ret = ++global_seq;
++
++    return ret;
++  }
+   /**
+    * Get the protocol version we support for the given peer type: either
+    * a peer protocol (if it matches our own), the protocol version for the
+diff --git a/src/msg/async/ProtocolV1.cc b/src/msg/async/ProtocolV1.cc
+index 9d6dbaae0a24..51687d7fca8e 100644
+--- a/src/msg/async/ProtocolV1.cc
++++ b/src/msg/async/ProtocolV1.cc
+@@ -21,8 +21,8 @@ std::ostream &ProtocolV1::_conn_prefix(std::ostream *_dout) {
+ 		<< " conn("
+                 << connection << " " << this
+                 << " :" << connection->port << " s=" << get_state_name(state)
+-                << " pgs=" << peer_global_seq << " gs=" << global_seq
+-                << " cs=" << connect_seq << " l=" << connection->policy.lossy << ").";
++                << " pgs=" << peer_global_seq << " cs=" << connect_seq
++                << " l=" << connection->policy.lossy << ").";
+ }
+ 
+ #define WRITE(B, C) write(CONTINUATION(C), B)
diff --git a/patches/0060-Revert-msg-async-ProtocolV2-Speed-up-connection-lo.patch b/patches/0060-Revert-msg-async-ProtocolV2-Speed-up-connection-lo.patch
new file mode 100644
index 0000000000..ec244804b9
--- /dev/null
+++ b/patches/0060-Revert-msg-async-ProtocolV2-Speed-up-connection-lo.patch
@@ -0,0 +1,86 @@
+From 412820b71fc9f21489e431c989add783c882dbf9 Mon Sep 17 00:00:00 2001
+From: "Kamoltat (Junior) Sirivadhna" <ksirivad@redhat.com>
+Date: Mon, 21 Sep 2026 18:11:55 +0000
+Subject: [PATCH 3/3] Revert "msg/async/ProtocolV2: Speed up connection logic
+ when peer restarts"
+
+This reverts commit d552063c39d57218a6cfea0eb69d479ab6e5215d.
+
+Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
+(cherry picked from commit 412820b71fc9f21489e431c989add783c882dbf9)
+---
+ src/msg/async/ProtocolV2.cc | 45 +++++--------------------------------
+ 1 file changed, 5 insertions(+), 40 deletions(-)
+
+diff --git a/src/msg/async/ProtocolV2.cc b/src/msg/async/ProtocolV2.cc
+index a0672507464a..d4f2c894437e 100644
+--- a/src/msg/async/ProtocolV2.cc
++++ b/src/msg/async/ProtocolV2.cc
+@@ -23,11 +23,7 @@ std::ostream &ProtocolV2::_conn_prefix(std::ostream *_dout) {
+ 		<< " " << ceph_con_mode_name(auth_meta->con_mode)
+ 		<< " :" << connection->port
+                 << " s=" << get_state_name(state) << " pgs=" << peer_global_seq
+-                << " gs=" << global_seq << " cs=" << connect_seq
+-                << " l=" << connection->policy.lossy
+-                << " c_cookie=" << client_cookie
+-                << " s_cookie=" << server_cookie
+-                << " reconnecting=" << reconnecting
++                << " cs=" << connect_seq << " l=" << connection->policy.lossy
+                 << " rev1=" << HAVE_MSGR2_FEATURE(peer_supported_features,
+                                                   REVISION_1)
+                 << " crypto rx=" << session_stream_handlers.rx.get()
+@@ -2700,40 +2696,11 @@ CtPtr ProtocolV2::handle_existing_connection(const AsyncConnectionRef& existing)
+     auto wait = WaitFrame::Encode();
+     return WRITE(wait, "wait", read_frame);
+   }
+-  // Detect a peer (client) restart and prefer the new connection.
+-  // Rationale:
+-  // - The client cookie is regenerated in send_client_ident() on client startup.
+-  // - If the existing connection (exproto) has a strictly higher peer_global_seq
+-  //   than the incoming one, the peer’s sequence appears to have reset on the new
+-  //   connection (exproto->peer_global_seq > peer_global_seq).
+-  // - Both sides present non-zero cookies and they differ, which strongly indicates
+-  //   a new client instance (exproto->client_cookie != client_cookie).
+-  // - We only do this when the existing connection is not in a benign terminal/idle
+-  //   state (READY or STANDBY); otherwise we might drop a healthy/idle connection.
+-  // Action:
+-  // Drop the existing connection in favor of the incoming one, then continue the
+-  // handshake on the new connection.
+-  if (exproto->peer_global_seq > peer_global_seq &&
+-    exproto->client_cookie && client_cookie &&
+-    exproto->client_cookie != client_cookie &&
+-    exproto->state != READY && exproto->state != STANDBY) {
+-    ldout(cct, 1) << __func__ << " client has clearly restarted (ex_peer_global_seq="
+-                  << exproto->peer_global_seq << " > peer_global_seq=" << peer_global_seq
+-                  << " && cookie changed: client_cookie=" << client_cookie << " != ex_client_cookie="
+-                  << exproto->client_cookie << "), "
+-                  << "existing connection state is " << get_state_name(exproto->state)
+-                  << " (not READY or STANDBY), dropping existing_connection=" << existing
+-                  << " in favor of new_connection=" << connection << dendl;
+-    existing->protocol->stop();
+-    existing->dispatch_queue->queue_reset(existing.get());
+-    l.unlock();
+-    return send_server_ident();
+-  }
+ 
+   if (exproto->peer_global_seq > peer_global_seq) {
+-    ldout(cct, 1) << __func__ << " this is a stale connection, ex_peer_global_seq="
+-                  << exproto->peer_global_seq
+-                  << " > peer_global_seq=" << peer_global_seq
++    ldout(cct, 1) << __func__ << " this is a stale connection, peer_global_seq="
++                  << peer_global_seq
++                  << " existing->peer_global_seq=" << exproto->peer_global_seq
+                   << ", stopping this connection." << dendl;
+     stop();
+     connection->dispatch_queue->queue_reset(connection);
+@@ -2758,9 +2725,7 @@ CtPtr ProtocolV2::handle_existing_connection(const AsyncConnectionRef& existing)
+     // peer has reseted and we're going to reuse the existing connection
+     // by replacing the communication socket
+     ldout(cct, 1) << __func__ << " found previous session existing=" << existing
+-                  << ", ex_client_cookie=" << exproto->client_cookie
+-                  << "!= client_cookie=" << client_cookie
+-                  << ", peer must have reseted!" << dendl;
++                  << ", peer must have reseted." << dendl;
+     if (connection->policy.resetcheck) {
+       exproto->reset_session();
+     }
diff --git a/patches/series b/patches/series
index f3567c88f1..ead683bb88 100644
--- a/patches/series
+++ b/patches/series
@@ -55,3 +55,6 @@
 0055-auth-cephx-track-pending-key-use-per-accepted-sessio.patch
 0056-auth-cephx-allow-manual-pending-key-promotion-for-cl.patch
 0057-mon-record-which-key-authenticated-a-session.patch
+0058-Revert-msg-async-ProtocolV2-send_server_ident-updat.patch
+0059-Revert-src-msg-async-Improve-logging-and-prefixes-.patch
+0060-Revert-msg-async-ProtocolV2-Speed-up-connection-lo.patch
-- 
2.47.3





             reply	other threads:[~2026-11-15 13:09 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-15 13:09 Kefu Chai [this message]
2026-11-18 14:51 ` applied: [PATCH pve-ceph squid-stable-9] patches: backport messenger reverts to fix peering after OSD restarts Thomas Lamprecht

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261115130901.937954-1-k.chai@proxmox.com \
    --to=k.chai@proxmox.com \
    --cc=pve-devel@lists.proxmox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal