From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 29AFB1FF0E5 for ; Wed, 12 Aug 2026 10:20:39 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 5BA9421571; Wed, 12 Aug 2026 10:20:38 +0200 (CEST) From: Kefu Chai To: pve-devel@lists.proxmox.com Subject: [PATCH] backport fix for EC truncate+write planning corrupting shards Date: Wed, 12 Aug 2026 16:20:22 +0800 Message-ID: <20260812082022.2910768-1-k.chai@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1786522815066 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.800 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: GJ2NWNKM32SD6DOFQRSFSRJ524YKFCJ3 X-Message-ID-Hash: GJ2NWNKM32SD6DOFQRSFSRJ524YKFCJ3 X-MailFrom: k.chai@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Kefu Chai X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Kefu Chai A truncate combined with a write in a single EC transaction can persist mis-sized shards on pools with allow_ec_optimizations. Reading such an object fails decode and asserts the OSD, and since recovery reads hit the same path, affected OSDs crash-loop on startup and the damage follows PG remapping. Plain CephFS in-place rewrites and librbd clones are enough to trigger it (https://tracker.ceph.com/issues/77276, https://tracker.ceph.com/issues/78400). The upstream tentacle backport (https://tracker.ceph.com/issues/77918) merged after v20.2.2 was tagged, so it only ships with the next point release. Pick the single commit up front; it drops out on the next rebase. Signed-off-by: Kefu Chai --- ...n-fix-truncate-write-planning-for-EC.patch | 128 ++++++++++++++++++ patches/series | 1 + 2 files changed, 129 insertions(+) create mode 100644 patches/0014-osd-ECTransaction-fix-truncate-write-planning-for-EC.patch diff --git a/patches/0014-osd-ECTransaction-fix-truncate-write-planning-for-EC.patch b/patches/0014-osd-ECTransaction-fix-truncate-write-planning-for-EC.patch new file mode 100644 index 0000000000..4112aaf299 --- /dev/null +++ b/patches/0014-osd-ECTransaction-fix-truncate-write-planning-for-EC.patch @@ -0,0 +1,128 @@ +From a1bb747293bbf97cb698cfa1280eecdb2ef8a3e0 Mon Sep 17 00:00:00 2001 +From: Alex Ainscow +Date: Tue, 9 Jun 2026 15:45:25 +0100 +Subject: [PATCH] osd/ECTransaction: fix truncate+write planning for EC shard + sizes + +There are multiple problems fixed here, which are caused by operations +which perform a truncate-then-write in a single transaction. + +NOTE: The only known scenario for these operations are CLS-sparsify operations. +I recommend backporting these changes to Tentacle, however, as they are +regressions in the RADOS API which can lead to data corruption. + +PROBLEM: Cache not invalidated correctly: +If projected_size >= orig_size, the invalidate_cache flag is not set in the plan +This means there is potentially data in the RMW extent cache, which +may cause data corruption in subsequent writes (although not the write +being made). No actual use case for this operation is known, so it may not be as serious as +it sounds. + +FIX: Invalidate cache on any truncate or "delete_first" + +PROBLEM: Parity writes permitted with truncates: +Currently parity delta writes do not support operations which may have +truncates. However, if the object ended up the same size as pre-truncate, a +parity delta write may have been attemtped. This may lead to data corruption. + +FIX: Block parity writes in this case. +NOTE: It is not worth the development effort to support PDW in this scenario, as +performance benefit would be minimal overall. + +PROBLEM: RMW Reads can occur beyond first truncate point. +Such reads reflect the pre-truncate data (which is incorrect) and be +preserved, even if the invalidate cache flag is set (since the cache is +invalidated before the reads). This can lead to similar corruption as +the earlier "cache not invalidated" + +FIX: trim the read to the lower truncate size. + +PROBLEM: +When performing a truncate, the parity shards may need to be updated to +reflect truncates on other shards. These truncate writes in the plan were +overriding, rather than adding to, other writes in the op. The result was +a potentially truncated coding shard. This leads to assertions on reads. + +FIX: Replace = with insert() + +PROBLEM: Some shards not set to correct size on truncate-then-write +If projected_size is smaller or equal to the original size, then the +code which attempts to correctly size a shard will not run. However if +an operation performs a truncate then a partial write and does not +write to a particular shard AND the shard ends up smaller, then this +can lead to an incorrectly sized shard. This leads to assertion on reads. + +FIX: Execute the shard-resize code on all truncates. + +AI assistance was mainly used to write unit test. However, I cannot rule out a +contribution to the simple fixes found in this commit, so out of caution, +I place the Assisted-by tag. + +Fixes: https://tracker.ceph.com/issues/77276 + +Signed-off-by: Alex Ainscow +Assisted-by: IBM-Bob:ClaudeSonnet/GPT +(cherry picked from commit 51d8c5c489ba3e664209fb3316f8d6e03e257e28) +--- + src/osd/ECTransaction.cc | 17 +++++++++++++---- + 1 file changed, 13 insertions(+), 4 deletions(-) + +diff --git a/src/osd/ECTransaction.cc b/src/osd/ECTransaction.cc +index efbc57a08dd..cd03849ed4d 100644 +--- a/src/osd/ECTransaction.cc ++++ b/src/osd/ECTransaction.cc +@@ -140,7 +140,7 @@ ECTransaction::WritePlanObj::WritePlanObj( + * 2. ALL delete operations (do NOT use is_delete() here!!!) + * 3. Truncates that reduce size. + */ +- invalidates_cache = op.has_source(&source) || op.delete_first || projected_size < orig_size; ++ invalidates_cache = op.has_source(&source) || op.delete_first || (op.truncate && op.truncate->first < orig_size); + + op.buffer_updates.to_interval_set(unaligned_ro_writes); + +@@ -206,7 +206,7 @@ ECTransaction::WritePlanObj::WritePlanObj( + + /* Here we decide if we want to do a conventional write or a parity delta write. */ + if (sinfo.supports_parity_delta_writes() && !object_in_cache && +- orig_size == projected_size && !reads.empty()) { ++ orig_size == projected_size && !reads.empty() && !op.truncate) { + + shard_id_set read_shards = reads.get_shard_id_set(); + shard_id_set pdw_read_shards = pdw_reads.get_shard_id_set(); +@@ -258,6 +258,15 @@ ECTransaction::WritePlanObj::WritePlanObj( + * read the existing data on the partial stripe. + */ + if (op.truncate && op.truncate->first < orig_size) { ++ if (to_read) { ++ ECUtil::shard_extent_set_t truncate_mask(sinfo.get_k_plus_m()); ++ sinfo.ro_range_to_shard_extent_set(0, op.truncate->first, truncate_mask); ++ ++ to_read->intersection_of(truncate_mask); ++ if (to_read->empty()) { ++ to_read = std::nullopt; ++ } ++ } + ECUtil::shard_extent_set_t truncate_read(sinfo.get_k_plus_m()); + uint64_t prev_stripe = sinfo.ro_offset_to_prev_stripe_ro_offset(op.truncate->first); + uint64_t next_align = ECUtil::align_next(op.truncate->first); +@@ -285,7 +294,7 @@ ECTransaction::WritePlanObj::WritePlanObj( + + // We only need to update the parity buffer for the write + for (auto && shard : sinfo.get_parity_shards()) { +- will_write[shard] = truncate_write; ++ will_write[shard].insert(truncate_write); + } + } + } +@@ -752,7 +761,7 @@ void ECTransaction::Generate::appends_and_clone_ranges() { + ECUtil::shard_extent_set_t cloneable_range(sinfo.get_k_plus_m()); + sinfo.ro_size_to_read_mask(clone_max, cloneable_range); + +- if (plan.orig_size < plan.projected_size) { ++ if (op.delete_first || op.truncate || plan.orig_size < plan.projected_size) { + ECUtil::shard_extent_set_t projected_cloneable_range(sinfo.get_k_plus_m()); + sinfo.ro_size_to_read_mask(plan.projected_size,projected_cloneable_range); + +-- +2.47.3 + diff --git a/patches/series b/patches/series index db38f90574..accabf1432 100644 --- a/patches/series +++ b/patches/series @@ -11,3 +11,4 @@ 0011-debian-do-not-ship-legacy-init.d-ceph-script-anymore.patch 0012-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch 0013-pybind-rbd-disable-on_progress-callbacks-to-prevent-.patch +0014-osd-ECTransaction-fix-truncate-write-planning-for-EC.patch -- 2.47.3