From: Kefu Chai <k.chai@proxmox.com>
To: pve-devel@lists.proxmox.com
Cc: Kefu Chai <tchaikov@gmail.com>
Subject: [PATCH] backport fix for EC truncate+write planning corrupting shards
Date: Wed, 12 Aug 2026 16:20:22 +0800 [thread overview]
Message-ID: <20260812082022.2910768-1-k.chai@proxmox.com> (raw)
From: Kefu Chai <tchaikov@gmail.com>
A truncate combined with a write in a single EC transaction can persist
mis-sized shards on pools with allow_ec_optimizations. Reading such an
object fails decode and asserts the OSD, and since recovery reads hit
the same path, affected OSDs crash-loop on startup and the damage
follows PG remapping. Plain CephFS in-place rewrites and librbd clones
are enough to trigger it (https://tracker.ceph.com/issues/77276,
https://tracker.ceph.com/issues/78400).
The upstream tentacle backport (https://tracker.ceph.com/issues/77918)
merged after v20.2.2 was tagged, so it only ships with the next point
release. Pick the single commit up front; it drops out on the next
rebase.
Signed-off-by: Kefu Chai <k.chai@proxmox.com>
---
...n-fix-truncate-write-planning-for-EC.patch | 128 ++++++++++++++++++
patches/series | 1 +
2 files changed, 129 insertions(+)
create mode 100644 patches/0014-osd-ECTransaction-fix-truncate-write-planning-for-EC.patch
diff --git a/patches/0014-osd-ECTransaction-fix-truncate-write-planning-for-EC.patch b/patches/0014-osd-ECTransaction-fix-truncate-write-planning-for-EC.patch
new file mode 100644
index 0000000000..4112aaf299
--- /dev/null
+++ b/patches/0014-osd-ECTransaction-fix-truncate-write-planning-for-EC.patch
@@ -0,0 +1,128 @@
+From a1bb747293bbf97cb698cfa1280eecdb2ef8a3e0 Mon Sep 17 00:00:00 2001
+From: Alex Ainscow <aainscow@uk.ibm.com>
+Date: Tue, 9 Jun 2026 15:45:25 +0100
+Subject: [PATCH] osd/ECTransaction: fix truncate+write planning for EC shard
+ sizes
+
+There are multiple problems fixed here, which are caused by operations
+which perform a truncate-then-write in a single transaction.
+
+NOTE: The only known scenario for these operations are CLS-sparsify operations.
+I recommend backporting these changes to Tentacle, however, as they are
+regressions in the RADOS API which can lead to data corruption.
+
+PROBLEM: Cache not invalidated correctly:
+If projected_size >= orig_size, the invalidate_cache flag is not set in the plan
+This means there is potentially data in the RMW extent cache, which
+may cause data corruption in subsequent writes (although not the write
+being made). No actual use case for this operation is known, so it may not be as serious as
+it sounds.
+
+FIX: Invalidate cache on any truncate or "delete_first"
+
+PROBLEM: Parity writes permitted with truncates:
+Currently parity delta writes do not support operations which may have
+truncates. However, if the object ended up the same size as pre-truncate, a
+parity delta write may have been attemtped. This may lead to data corruption.
+
+FIX: Block parity writes in this case.
+NOTE: It is not worth the development effort to support PDW in this scenario, as
+performance benefit would be minimal overall.
+
+PROBLEM: RMW Reads can occur beyond first truncate point.
+Such reads reflect the pre-truncate data (which is incorrect) and be
+preserved, even if the invalidate cache flag is set (since the cache is
+invalidated before the reads). This can lead to similar corruption as
+the earlier "cache not invalidated"
+
+FIX: trim the read to the lower truncate size.
+
+PROBLEM:
+When performing a truncate, the parity shards may need to be updated to
+reflect truncates on other shards. These truncate writes in the plan were
+overriding, rather than adding to, other writes in the op. The result was
+a potentially truncated coding shard. This leads to assertions on reads.
+
+FIX: Replace = with insert()
+
+PROBLEM: Some shards not set to correct size on truncate-then-write
+If projected_size is smaller or equal to the original size, then the
+code which attempts to correctly size a shard will not run. However if
+an operation performs a truncate then a partial write and does not
+write to a particular shard AND the shard ends up smaller, then this
+can lead to an incorrectly sized shard. This leads to assertion on reads.
+
+FIX: Execute the shard-resize code on all truncates.
+
+AI assistance was mainly used to write unit test. However, I cannot rule out a
+contribution to the simple fixes found in this commit, so out of caution,
+I place the Assisted-by tag.
+
+Fixes: https://tracker.ceph.com/issues/77276
+
+Signed-off-by: Alex Ainscow <aainscow@uk.ibm.com>
+Assisted-by: IBM-Bob:ClaudeSonnet/GPT
+(cherry picked from commit 51d8c5c489ba3e664209fb3316f8d6e03e257e28)
+---
+ src/osd/ECTransaction.cc | 17 +++++++++++++----
+ 1 file changed, 13 insertions(+), 4 deletions(-)
+
+diff --git a/src/osd/ECTransaction.cc b/src/osd/ECTransaction.cc
+index efbc57a08dd..cd03849ed4d 100644
+--- a/src/osd/ECTransaction.cc
++++ b/src/osd/ECTransaction.cc
+@@ -140,7 +140,7 @@ ECTransaction::WritePlanObj::WritePlanObj(
+ * 2. ALL delete operations (do NOT use is_delete() here!!!)
+ * 3. Truncates that reduce size.
+ */
+- invalidates_cache = op.has_source(&source) || op.delete_first || projected_size < orig_size;
++ invalidates_cache = op.has_source(&source) || op.delete_first || (op.truncate && op.truncate->first < orig_size);
+
+ op.buffer_updates.to_interval_set(unaligned_ro_writes);
+
+@@ -206,7 +206,7 @@ ECTransaction::WritePlanObj::WritePlanObj(
+
+ /* Here we decide if we want to do a conventional write or a parity delta write. */
+ if (sinfo.supports_parity_delta_writes() && !object_in_cache &&
+- orig_size == projected_size && !reads.empty()) {
++ orig_size == projected_size && !reads.empty() && !op.truncate) {
+
+ shard_id_set read_shards = reads.get_shard_id_set();
+ shard_id_set pdw_read_shards = pdw_reads.get_shard_id_set();
+@@ -258,6 +258,15 @@ ECTransaction::WritePlanObj::WritePlanObj(
+ * read the existing data on the partial stripe.
+ */
+ if (op.truncate && op.truncate->first < orig_size) {
++ if (to_read) {
++ ECUtil::shard_extent_set_t truncate_mask(sinfo.get_k_plus_m());
++ sinfo.ro_range_to_shard_extent_set(0, op.truncate->first, truncate_mask);
++
++ to_read->intersection_of(truncate_mask);
++ if (to_read->empty()) {
++ to_read = std::nullopt;
++ }
++ }
+ ECUtil::shard_extent_set_t truncate_read(sinfo.get_k_plus_m());
+ uint64_t prev_stripe = sinfo.ro_offset_to_prev_stripe_ro_offset(op.truncate->first);
+ uint64_t next_align = ECUtil::align_next(op.truncate->first);
+@@ -285,7 +294,7 @@ ECTransaction::WritePlanObj::WritePlanObj(
+
+ // We only need to update the parity buffer for the write
+ for (auto && shard : sinfo.get_parity_shards()) {
+- will_write[shard] = truncate_write;
++ will_write[shard].insert(truncate_write);
+ }
+ }
+ }
+@@ -752,7 +761,7 @@ void ECTransaction::Generate::appends_and_clone_ranges() {
+ ECUtil::shard_extent_set_t cloneable_range(sinfo.get_k_plus_m());
+ sinfo.ro_size_to_read_mask(clone_max, cloneable_range);
+
+- if (plan.orig_size < plan.projected_size) {
++ if (op.delete_first || op.truncate || plan.orig_size < plan.projected_size) {
+ ECUtil::shard_extent_set_t projected_cloneable_range(sinfo.get_k_plus_m());
+ sinfo.ro_size_to_read_mask(plan.projected_size,projected_cloneable_range);
+
+--
+2.47.3
+
diff --git a/patches/series b/patches/series
index db38f90574..accabf1432 100644
--- a/patches/series
+++ b/patches/series
@@ -11,3 +11,4 @@
0011-debian-do-not-ship-legacy-init.d-ceph-script-anymore.patch
0012-mgr-fix-module-import-by-making-NOTIFY_TYPES-in-py-m.patch
0013-pybind-rbd-disable-on_progress-callbacks-to-prevent-.patch
+0014-osd-ECTransaction-fix-truncate-write-planning-for-EC.patch
--
2.47.3
reply other threads:[~2026-08-12 8:20 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260812082022.2910768-1-k.chai@proxmox.com \
--to=k.chai@proxmox.com \
--cc=pve-devel@lists.proxmox.com \
--cc=tchaikov@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.