From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 770321FF0B2 for ; Tue, 08 Sep 2026 20:16:54 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 823E92158E; Tue, 08 Sep 2026 20:16:50 +0200 (CEST) Message-ID: Date: Tue, 8 Sep 2026 20:16:44 +0200 MIME-Version: 1.0 User-Agent: Thunderbird Daily Subject: Re: [PATCH docs v2 3/5] cephx-migration: Move all requirements into prelude To: Maximiliano Sandoval , pve-devel@lists.proxmox.com References: <20260908153932.577051-1-m.sandoval@proxmox.com> <20260908153932.577051-4-m.sandoval@proxmox.com> Content-Language: en-US From: Thomas Lamprecht In-Reply-To: <20260908153932.577051-4-m.sandoval@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1788891397040 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.887 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: 4LJYKKGJBFPPOEOGWK2ECUZN53RVIJIH X-Message-ID-Hash: 4LJYKKGJBFPPOEOGWK2ECUZN53RVIJIH X-MailFrom: t.lamprecht@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Am 08.09.26 um 17:39 schrieb Maximiliano Sandoval: > The migration should start after all requirements are meet. > > Note that the script will warn if it is not being run as root. There is no need > to mention that the script should be run as root. I mean, fine by me as that user is for such PVE operations rather commonly needed anyway, but I'd not sign off on that argument in general; especially for such migrations I > > Signed-off-by: Maximiliano Sandoval > --- > pveceph.adoc | 51 ++++++++++++++++++++++++++------------------------- > 1 file changed, 26 insertions(+), 25 deletions(-) > > diff --git a/pveceph.adoc b/pveceph.adoc > index 625768f..40ca16b 100644 > --- a/pveceph.adoc > +++ b/pveceph.adoc > @@ -1363,21 +1363,35 @@ Reruns do not rotate completed keys again. After each run, invoke the helper > without options to check the current state and follow the next steps it prints. > > .Before You Begin > -* Upgrade `pve-manager` to 9.2.17 or newer and install the latest Ceph > - packages on every node. Staged client-key rotation requires Ceph > - 19.2.6-pve3, 20.2.4-pve3, or newer on every monitor. Then complete the > - rolling restart of all Ceph services. > +* Upgrade all cluster nodes to the latest {pve} packages. In particular, > + ** `pve-manager` '9.2.17' or newer > + ** Ceph '20.2.4-pve3' or newer if using Tentacle > + ** Ceph '19.2.6-pve3' or newer if using Squid > + ** Kernel 7.0 or newer. note that this is not strictly true, you can migrate the most essential keys owned by the cluster just fine without upgrading to kernel 7.0; that's purely needed for CephFS and krbd, but neither the service daemons nor VMs running on librbd. Getting imprecise here is IMO not that ideal. btw. for the case it matters upgrading alone isn't enough, one actually needs to run it. And just migrating all ceph services over gets rid of the health errors and improves data at rest safety and some other relevant properties already, so IMO it can be a relevant for those pondering if they really must upgrade to 7.0 (e.g. due to some compat issue). > +* Ensure all external clients are running Ceph and Kernel versions supporting `aes256k`. > +* Complete a rolling restart of Ceph services. > * Resolve health warnings and errors not listed in > - xref:pveceph_cephx_health_check_reference[Health Check Reference]. Ensure > - every node is reachable over SSH. > -* Run the helper as `root` on one node only. Do not start another migration, > - a rolling restart, or change Ceph keys with other tools while it runs. The > - helper's lock does not block direct Ceph authentication commands. > + xref:pveceph_cephx_health_check_reference[Health Check Reference]. > +* Ensure every node is reachable over SSH. > + > +.Kernel Support Considerations > +A storage client requires support for `aes256k` in the `rbd` kernel module if: > + > +* A container is using a RBD storage. > +* or, A VM is using a RBD storage, if `krbd` is enabled on the storage configuration. > +* or, a CephFS is mounted, unless `fuse` is enabled in the storage configuration. > + > +Clients requiring kernel support should not downgrade to older kernel versions > +after the migration. this is a bit odd on its own given that it doesn't mentions the actual required kernel version (and above requirement doesn't links to it). and IMO the table was slightly clearer. > > 1. Migrate the Cluster-Owned Keys > +++++++++++++++++++++++++++++++++ > > -Review the cluster-owned migration in the default dry-run mode: > +WARNING: Do not start another migration, a rolling restart, or change Ceph keys > +with other tools during the migration. The helper's lock does not block direct > +Ceph authentication commands. > + > +Review the cluster-owned key migration in dry-run mode: > > [source,bash] > ---- > @@ -1405,21 +1419,8 @@ automatically. > 2. Migrate the Keys of Compatible Ceph Users > ++++++++++++++++++++++++++++++++++++++++++++ > > -[[pveceph_cipher_support]] > -.Check compatibility > -Several workloads can share one Ceph user. Migrate its key only when every > -client supports `aes256k`, including disconnected and external clients. > -Ceph programs from the updated {pve} packages support it; kernel clients > -require a *running* kernel version 7.0 or newer. > - > -[cols="2,3",options="header"] > -|=== > -|Virtual machine with RBD disks |Userspace, unless `krbd` is enabled. > -|Container on RBD |Always the kernel client. > -|CephFS mount |Kernel, unless `fuse` is enabled. > -|=== > - > -Check external systems too. If any affected client is incompatible or > +Migrate a user's key only once all its clients support `aes256k`, including > +disconnected and external clients. If any affected client is incompatible or > unknown, leave its user's key unchanged and postpone this step. > > .Stage the keys