[pve-devel] [PATCH manager] ui: ceph installer: Run ceph/init if joining a cluster

[pve-devel] [PATCH manager] ui: ceph installer: Run ceph/init if joining a cluster

[Maximiliano Sandoval]

Currently joining an already existing Ceph cluster won't create symlinks
from /etc/pve/ceph.conf to /etc/ceph/ceph.conf on the recently added
node. This can be remediated by running `pveceph init` but ideally
should be done by the web UI.

We remediate this by setting `configuration` to false if we reach the
branch where `operation.error.statusText` matches:

    binary not installed: /usr/bin/ceph-mon

This allow us to configure this node and run `init` in the API.

Suggested-by: Mira Limbeck <mira.limbeck@gmail.com>
Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
---
This was found on a ticker from the support portal.

 www/manager6/ceph/CephInstallWizard.js | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/www/manager6/ceph/CephInstallWizard.js b/www/manager6/ceph/CephInstallWizard.js
index f33ae868..d3c46506 100644
--- a/www/manager6/ceph/CephInstallWizard.js
+++ b/www/manager6/ceph/CephInstallWizard.js
@@ -369,6 +369,9 @@ Ext.define('PVE.ceph.CephInstallWizard', {
 				    me.down('textfield').setValue('success');
 				} else if (!operation.error.statusText.match("not installed", "i")) {
 				    Proxmox.Utils.setErrorMask(me, operation.error.statusText);
+				} else {
+				    // For when we join an already existing Ceph cluster.
+				    me.up('pveCephInstallWizard').getViewModel().set('configuration', false);
 				}
 			    },
 			},
-- 
2.39.2

Re: [pve-devel] [PATCH manager] ui: ceph installer: Run ceph/init if joining a cluster

[Aaron Lauterer]

The commit message doesn't explain the actual issue that it is trying to solve.
AFAICT we do not need the ceph.conf symlinked right away for normal PVE operations. If it is not present in /etc/ceph/ceph.conf, the RBD and CephFS connections will use the dedicated parameters to connect and authenticate against the Ceph cluster, which are all present in the /etc/pve/priv... directory.

And even in a newly set up HCI PVE + Ceph cluster, the symlink for the Ceph config will only be created once you set up a Ceph service (MON, MGR, MDS, OSD) on that node.

Even with a valid use case (see first line of my response), I am not sure if the UI is the right place, as a user can still run into the same situation if they use 'pveceph install' on the CLI without running 'pveceph init' afterwards.
Therefore, if there is a reason to always have the symlink, wouldn't it be better to add it to the install routine to create the symlink if the /etc/pve/ceph.conf file is detected?

On 12/18/23 15:25, Maximiliano Sandoval wrote:
> Currently joining an already existing Ceph cluster won't create symlinks
> from /etc/pve/ceph.conf to /etc/ceph/ceph.conf on the recently added
> node. This can be remediated by running `pveceph init` but ideally
> should be done by the web UI.
> 
> We remediate this by setting `configuration` to false if we reach the
> branch where `operation.error.statusText` matches:
> 
>      binary not installed: /usr/bin/ceph-mon
> 
> This allow us to configure this node and run `init` in the API.
> 
> Suggested-by: Mira Limbeck <mira.limbeck@gmail.com>
> Signed-off-by: Maximiliano Sandoval <m.sandoval@proxmox.com>
> ---
> This was found on a ticker from the support portal.
> 
>   www/manager6/ceph/CephInstallWizard.js | 3 +++
>   1 file changed, 3 insertions(+)
> 
> diff --git a/www/manager6/ceph/CephInstallWizard.js b/www/manager6/ceph/CephInstallWizard.js
> index f33ae868..d3c46506 100644
> --- a/www/manager6/ceph/CephInstallWizard.js
> +++ b/www/manager6/ceph/CephInstallWizard.js
> @@ -369,6 +369,9 @@ Ext.define('PVE.ceph.CephInstallWizard', {
>   				    me.down('textfield').setValue('success');
>   				} else if (!operation.error.statusText.match("not installed", "i")) {
>   				    Proxmox.Utils.setErrorMask(me, operation.error.statusText);
> +				} else {
> +				    // For when we join an already existing Ceph cluster.
> +				    me.up('pveCephInstallWizard').getViewModel().set('configuration', false);
>   				}
>   			    },
>   			},

Re: [pve-devel] [PATCH manager] ui: ceph installer: Run ceph/init if joining a cluster

[Mira Limbeck]

On 12/19/23 18:09, Aaron Lauterer wrote:
> The commit message doesn't explain the actual issue that it is trying to
> solve.
> AFAICT we do not need the ceph.conf symlinked right away for normal PVE
> operations. If it is not present in /etc/ceph/ceph.conf, the RBD and
> CephFS connections will use the dedicated parameters to connect and
> authenticate against the Ceph cluster, which are all present in the
> /etc/pve/priv... directory.
> 
> And even in a newly set up HCI PVE + Ceph cluster, the symlink for the
> Ceph config will only be created once you set up a Ceph service (MON,
> MGR, MDS, OSD) on that node.
> 
> Even with a valid use case (see first line of my response), I am not
> sure if the UI is the right place, as a user can still run into the same
> situation if they use 'pveceph install' on the CLI without running
> 'pveceph init' afterwards.
> Therefore, if there is a reason to always have the symlink, wouldn't it
> be better to add it to the install routine to create the symlink if the
> /etc/pve/ceph.conf file is detected?


Yes, you're right. Since we call `pveceph install` even when installing
from the GUI, that should be the place to add it.

We've had a few cases where Ceph commands could not be run on the newly
added host, even though Ceph was installed because of the missing symlink.
Usually on a hyper-converged I'd assume that Ceph commands can simply be
run without specifying a Ceph config, since that's how we set it up. But
for newly added nodes that's currently not the case.

This also affects the system reports, since we don't pass the Ceph
config there at all for any of the Ceph commands.


@Maximiliano: This info, the `why`, would have been great in the commit
message.

Re: [pve-devel] [PATCH manager] ui: ceph installer: Run ceph/init if joining a cluster

[Thomas Lamprecht]

On 20/12/2023 10:53, Mira Limbeck wrote:
> On 12/19/23 18:09, Aaron Lauterer wrote:
>> The commit message doesn't explain the actual issue that it is trying to
>> solve.
>> AFAICT we do not need the ceph.conf symlinked right away for normal PVE
>> operations. If it is not present in /etc/ceph/ceph.conf, the RBD and
>> CephFS connections will use the dedicated parameters to connect and
>> authenticate against the Ceph cluster, which are all present in the
>> /etc/pve/priv... directory.
>>
>> And even in a newly set up HCI PVE + Ceph cluster, the symlink for the
>> Ceph config will only be created once you set up a Ceph service (MON,
>> MGR, MDS, OSD) on that node.
>>
>> Even with a valid use case (see first line of my response), I am not
>> sure if the UI is the right place, as a user can still run into the same
>> situation if they use 'pveceph install' on the CLI without running
>> 'pveceph init' afterwards.
>> Therefore, if there is a reason to always have the symlink, wouldn't it
>> be better to add it to the install routine to create the symlink if the
>> /etc/pve/ceph.conf file is detected?
> 
> 
> Yes, you're right. Since we call `pveceph install` even when installing
> from the GUI, that should be the place to add it.
> 
> We've had a few cases where Ceph commands could not be run on the newly
> added host, even though Ceph was installed because of the missing symlink.
> Usually on a hyper-converged I'd assume that Ceph commands can simply be
> run without specifying a Ceph config, since that's how we set it up. But
> for newly added nodes that's currently not the case.
> 
> This also affects the system reports, since we don't pass the Ceph
> config there at all for any of the Ceph commands.
> 

FWIW, we could also create such links on cluster join if ceph.conf is
present in the pmxcfs, as then client-operations would already work
without running pveceph (server) install, e.g., on a cluster with both
ceph server nodes and others, e.g., compute/guest focused, nodes.

Not saying that this is the ultimate way to go, but should be IMO at
least evaluated w.r.t pros/cons.