Re: [PATCH manager 0/1] ceph: add opt-in locality-aware replica reads (crush_location_hook)

["Kefu Chai" <k.chai@proxmox.com>]

Hi,

Putting a hold on this series for now.

Friedrich kindly pointed out that Tentacle v20.2.0 ships with a regression [1]
that affects rbd_read_from_replica_policy=localize. The issue is that commit
4b01c004b5d [2] ("PrimaryLogPG: don't accept ops with mixed balance_reads and
rwordered flags") causes the OSD to reject write ops that carry the LOCALIZE_READS
flag, returning -EINVAL. Since librbd sets this flag connection-wide when the
localize policy is active, this can lead to silent write failures.

The fix (a revert, PR #66611 [3]) has been merged to the tentacle branch and
should ship with v20.2.1, which is currently in QE validation [4]. Squid is not
affected — the problematic commit was only cherry-picked into tentacle.

I'll resend once v20.2.1 is released and picked up by our Tentacle packages.
The patch itself is opt-in, so there's no urgency.

Thanks,
Kefu

[1] https://tracker.ceph.com/issues/73997
[2] https://github.com/ceph/ceph/commit/4b01c004b5dc342cbdfb7cb26b47f6afe6245599
[3] https://github.com/ceph/ceph/pull/66611
[4] https://tracker.ceph.com/issues/74838

On Wed Mar 25, 2026 at 11:51 AM CST, Kefu Chai wrote:
> This patch was prompted by a forum thread [1] in which a user reported
> persistent high IO wait on PostgreSQL VMs running on a three-AZ Ceph
> cluster. The discussion surfaced a general optimization opportunity:
> librbd, by default, always reads from the primary OSD regardless of
> its location. In a multi-AZ deployment, that can mean every read pays
> a cross-AZ round-trip even when a same-AZ replica is available.
>
> rbd_read_from_replica_policy = localize addresses this by directing
> librbd to prefer the nearest replica, but it requires the client to
> declare its own position in the CRUSH hierarchy. This patch ships a
> hook script that supplies that position by querying the live CRUSH map
> (ceph osd crush find), and wires it up as an opt-in in pveceph init.
>
> The benefit scales with topology: in a multi-AZ cluster it keeps reads
> within the same AZ; in a hyperconverged setup, reads to a co-located
> OSD never leave the host at all. The feature is opt-in because it can
> degrade performance when replicas are equidistant or when the hook
> falls back to an incorrect CRUSH root — see the commit message for
> details.
>
> [1] https://forum.proxmox.com/threads/ceph-vm-with-high-io-wait.181751/
>   
>
> Kefu Chai (1):
>   ceph: add opt-in locality-aware replica reads (crush_location_hook)
>
>  PVE/API2/Ceph.pm                       | 17 ++++++++++
>  bin/Makefile                           |  3 +-
>  bin/ceph-crush-location                | 43 ++++++++++++++++++++++++++
>  www/manager6/ceph/CephInstallWizard.js |  8 ++++-
>  4 files changed, 69 insertions(+), 2 deletions(-)
>  create mode 100644 bin/ceph-crush-location