[PATCH pve-doc] pveceph: advise against using FastEC in production

[PATCH pve-doc] pveceph: advise against using FastEC in production

[Kefu Chai]

The FastEC section reads as an endorsement and describes enabling the
flag on a populated pool as a change to pool metadata only. That does
not match what users have seen. Since Tentacle was released, FastEC has
hit several bugs [1][2][3][4][5], mostly around truncate, discard and
object size handling, and mostly on pools that already held data when
the flag was set. The symptoms range from bogus scrub errors, over
'pg repair' making objects unfound, to OSDs crash-looping on their
first reconstruct read and taking the pool inactive. Some of the fixes
do not repair damage that was already done, and more fixes for
peering, recovery and the extent cache are still open upstream.

The text avoids version numbers, as they would be stale soon.

[1] https://tracker.ceph.com/issues/71642
[2] https://tracker.ceph.com/issues/77276
[3] https://tracker.ceph.com/issues/78400
[4] https://tracker.ceph.com/issues/81915
[5] https://bugzilla.proxmox.com/show_bug.cgi?id=7784

Signed-off-by: Kefu Chai <k.chai@proxmox.com>
---
 pveceph.adoc | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/pveceph.adoc b/pveceph.adoc
index 90ee35d..62405d4 100644
--- a/pveceph.adoc
+++ b/pveceph.adoc
@@ -773,11 +773,21 @@ FastEC (`allow_ec_optimizations`)
 Ceph Tentacle introduces a new erasure coding I/O path called FastEC,
 enabled per pool via the `allow_ec_optimizations` flag. FastEC adds partial
 reads and partial writes on EC pools, substantially improving small random
-I/O performance -- the dominant access pattern of virtual machine disk I/O.
-Upstream recommends it for RBD and CephFS backed storage
+I/O performance -- the dominant access pattern of virtual machine disk I/O
 footnote:[Ceph Erasure Code
 {cephdocs-url}/rados/operations/erasure-code/].
 
+WARNING: FastEC is a young feature and is still receiving fixes for data
+integrity and availability problems. Do not enable it on production pools,
+particularly not on pools that already hold data. If you want to try it, do
+so on a test cluster running the same Ceph version as production, with your
+typical workloads. Besides regular I/O, test at least upgrading an existing
+pool that holds data, discard and trim, resizing, scrubbing and repair, and
+recovery after an OSD failure. Pools without FastEC are not affected.
+
+{pve} never enables FastEC on its own, it is only active if you set the flag
+yourself.
+
 FastEC has four requirements that the Ceph monitor enforces:
 
 * the pool must be erasure coded;
@@ -833,14 +843,15 @@ If this reports a release older than `tentacle`, run
 IMPORTANT: `allow_ec_optimizations` is a one-way switch. Once enabled on a
 pool, the Ceph monitor refuses to clear the flag again, so rolling back
 requires draining and recreating the pool. Enable it only after validating
-that FastEC works for your workload on a test pool.
+that FastEC works for your workload on a test cluster.
 
 Enabling FastEC on a pool
 +++++++++++++++++++++++++
 
 FastEC is enabled per pool via the `ceph` CLI. It can be set on both new
-and existing (populated) pools. No data migration or re-encoding takes
-place; only pool metadata changes.
+and existing (populated) pools, but enabling it on a populated pool is
+discouraged. No data migration or re-encoding takes place; only pool metadata
+changes. Existing objects keep their old on-disk layout.
 
 To enable FastEC on a pool, run the following command against the data pool
 (typically named `<pool-name>-data`):
-- 
2.47.3

Re: [PATCH pve-doc] pveceph: advise against using FastEC in production

[Laurențiu Leahu-Vlăducu]

LGTM. I would be happy to have these warnings in our docs as soon as 
possible, since this came up as well in our enterprise support channel. 
The warnings are great in order to avoid issues on production systems - 
we can still remove them whenever the feature is more stable, in future 
Ceph versions.


Reviewed-by: Laurențiu Leahu-Vlăducu <l.leahu-vladucu@proxmox.com>


On 04.11.26 05:05, Kefu Chai wrote:
> The FastEC section reads as an endorsement and describes enabling the
> flag on a populated pool as a change to pool metadata only. That does
> not match what users have seen. Since Tentacle was released, FastEC has
> hit several bugs [1][2][3][4][5], mostly around truncate, discard and
> object size handling, and mostly on pools that already held data when
> the flag was set. The symptoms range from bogus scrub errors, over
> 'pg repair' making objects unfound, to OSDs crash-looping on their
> first reconstruct read and taking the pool inactive. Some of the fixes
> do not repair damage that was already done, and more fixes for
> peering, recovery and the extent cache are still open upstream.
> 
> The text avoids version numbers, as they would be stale soon.
> 
> [1] https://tracker.ceph.com/issues/71642
> [2] https://tracker.ceph.com/issues/77276
> [3] https://tracker.ceph.com/issues/78400
> [4] https://tracker.ceph.com/issues/81915
> [5] https://bugzilla.proxmox.com/show_bug.cgi?id=7784
> 
> Signed-off-by: Kefu Chai <k.chai@proxmox.com>
> ---
>   pveceph.adoc | 21 ++++++++++++++++-----
>   1 file changed, 16 insertions(+), 5 deletions(-)
> 
> diff --git a/pveceph.adoc b/pveceph.adoc
> index 90ee35d..62405d4 100644
> --- a/pveceph.adoc
> +++ b/pveceph.adoc
> @@ -773,11 +773,21 @@ FastEC (`allow_ec_optimizations`)
>   Ceph Tentacle introduces a new erasure coding I/O path called FastEC,
>   enabled per pool via the `allow_ec_optimizations` flag. FastEC adds partial
>   reads and partial writes on EC pools, substantially improving small random
> -I/O performance -- the dominant access pattern of virtual machine disk I/O.
> -Upstream recommends it for RBD and CephFS backed storage
> +I/O performance -- the dominant access pattern of virtual machine disk I/O
>   footnote:[Ceph Erasure Code
>   {cephdocs-url}/rados/operations/erasure-code/].
>   
> +WARNING: FastEC is a young feature and is still receiving fixes for data
> +integrity and availability problems. Do not enable it on production pools,
> +particularly not on pools that already hold data. If you want to try it, do
> +so on a test cluster running the same Ceph version as production, with your
> +typical workloads. Besides regular I/O, test at least upgrading an existing
> +pool that holds data, discard and trim, resizing, scrubbing and repair, and
> +recovery after an OSD failure. Pools without FastEC are not affected.
> +
> +{pve} never enables FastEC on its own, it is only active if you set the flag
> +yourself.
> +
>   FastEC has four requirements that the Ceph monitor enforces:
>   
>   * the pool must be erasure coded;
> @@ -833,14 +843,15 @@ If this reports a release older than `tentacle`, run
>   IMPORTANT: `allow_ec_optimizations` is a one-way switch. Once enabled on a
>   pool, the Ceph monitor refuses to clear the flag again, so rolling back
>   requires draining and recreating the pool. Enable it only after validating
> -that FastEC works for your workload on a test pool.
> +that FastEC works for your workload on a test cluster.
>   
>   Enabling FastEC on a pool
>   +++++++++++++++++++++++++
>   
>   FastEC is enabled per pool via the `ceph` CLI. It can be set on both new
> -and existing (populated) pools. No data migration or re-encoding takes
> -place; only pool metadata changes.
> +and existing (populated) pools, but enabling it on a populated pool is
> +discouraged. No data migration or re-encoding takes place; only pool metadata
> +changes. Existing objects keep their old on-disk layout.
>   
>   To enable FastEC on a pool, run the following command against the data pool
>   (typically named `<pool-name>-data`):