[pve-devel] Allow dynamic pool name discovery on ZFSPoolPlugin

[pve-devel] Allow dynamic pool name discovery on ZFSPoolPlugin

[Pablo Ruiz]

Hi,

I've just made a little custom storage plugin which basically
overrides/extends ZFSPoolPluging so the pool name instead of being fixed,
can be 'dynamically' guessed on each cluster node.

While adding some new nodes to one of our current clusters, due to hardware
differences, we ended up with new nodes having a different zpool's layout
than the already existing nodes. Our existing nodes had a pool named
'rpool/data' (a dataset nested into system's main pool). While the new
nodes have a dedicated pool (data) independent of 'rpool'. So we needed a
way to have the same storage use different backing zfs pools on each server
(ie. on old servers it should use 'rpool/data', while on newer ones it
should be using 'data').

Currently proxmox does not allow overriding an storage.cfg's property
'per-node'. So we came up with a simple custom plugin which basically looks
up the pool with an attribute like 'pve:id=$storeid', and uses this pool
obtained dynamically.

We could have simply added a new store with a different id, but that would
break our orchestration and automatic deployment of machines, and add some
additional management issues by having to track the store of each machine,
specially if/when moving vms around.

That said, the plugin works, but I think this feature may be of use to
others. Would a patch against upstream ZFSPoolPlugin accepted, so we can
avoid this custom plugin in the future?

I've posted the custom plugin as a gist:
https://gist.github.com/pruiz/5d7fbd75efb413ac15d2d0e3ef54f32a

Regards
Pablo

Re: [pve-devel] Allow dynamic pool name discovery on ZFSPoolPlugin

[Thomas Lamprecht]

Hi,

On 21.11.20 14:08, Pablo Ruiz wrote:
> Hi,
> 
> I've just made a little custom storage plugin which basically
> overrides/extends ZFSPoolPluging so the pool name instead of being fixed,
> can be 'dynamically' guessed on each cluster node.
> 
> While adding some new nodes to one of our current clusters, due to hardware
> differences, we ended up with new nodes having a different zpool's layout
> than the already existing nodes. Our existing nodes had a pool named
> 'rpool/data' (a dataset nested into system's main pool). While the new
> nodes have a dedicated pool (data) independent of 'rpool'. So we needed a
> way to have the same storage use different backing zfs pools on each server
> (ie. on old servers it should use 'rpool/data', while on newer ones it
> should be using 'data').
> 
> Currently proxmox does not allow overriding an storage.cfg's property
> 'per-node'. So we came up with a simple custom plugin which basically looks
> up the pool with an attribute like 'pve:id=$storeid', and uses this pool
> obtained dynamically.
> 
> We could have simply added a new store with a different id, but that would
> break our orchestration and automatic deployment of machines, and add some
> additional management issues by having to track the store of each machine,
> specially if/when moving vms around.
> 
> That said, the plugin works, but I think this feature may be of use to
> others. Would a patch against upstream ZFSPoolPlugin accepted, so we can
> avoid this custom plugin in the future?
> 
> I've posted the custom plugin as a gist:
> https://gist.github.com/pruiz/5d7fbd75efb413ac15d2d0e3ef54f32a
> 

thanks for sharing your solution!

It's an interesting way to solve this, the thing I do not really like, is
that the storage config is now not the only single source of truth for
"which pool to use/import" anymore

We could actually add a nodename to pool mapping as a property to the
ZFS Pool storage config schema instead, that'd would keep this information
contained to the configuration and make lookup a bit cheaper.

cheers,
Thomas

Re: [pve-devel] Allow dynamic pool name discovery on ZFSPoolPlugin

[Carsten Härle]

Why not just rename the ZFS-dataset on the old server with "zfs rename"? Seems to be much easier than implementing a separate feature.

-----Ursprüngliche Nachricht-----
Von: pve-devel [mailto:pve-devel-bounces@lists.proxmox.com] Im Auftrag von Thomas Lamprecht
Gesendet: Montag, 23. November 2020 08:50
An: Proxmox VE development discussion; Pablo Ruiz; pve-devel@pve.proxmox.com; Fabian Ebner
Betreff: Re: [pve-devel] Allow dynamic pool name discovery on ZFSPoolPlugin

Hi,

On 21.11.20 14:08, Pablo Ruiz wrote:
> Hi,
> 
> I've just made a little custom storage plugin which basically
> overrides/extends ZFSPoolPluging so the pool name instead of being fixed,
> can be 'dynamically' guessed on each cluster node.
> 
> While adding some new nodes to one of our current clusters, due to hardware
> differences, we ended up with new nodes having a different zpool's layout
> than the already existing nodes. Our existing nodes had a pool named
> 'rpool/data' (a dataset nested into system's main pool). While the new
> nodes have a dedicated pool (data) independent of 'rpool'. So we needed a
> way to have the same storage use different backing zfs pools on each server
> (ie. on old servers it should use 'rpool/data', while on newer ones it
> should be using 'data').
> 
> Currently proxmox does not allow overriding an storage.cfg's property
> 'per-node'. So we came up with a simple custom plugin which basically looks
> up the pool with an attribute like 'pve:id=$storeid', and uses this pool
> obtained dynamically.
> 
> We could have simply added a new store with a different id, but that would
> break our orchestration and automatic deployment of machines, and add some
> additional management issues by having to track the store of each machine,
> specially if/when moving vms around.
> 
> That said, the plugin works, but I think this feature may be of use to
> others. Would a patch against upstream ZFSPoolPlugin accepted, so we can
> avoid this custom plugin in the future?
> 
> I've posted the custom plugin as a gist:
> https://gist.github.com/pruiz/5d7fbd75efb413ac15d2d0e3ef54f32a
> 

thanks for sharing your solution!

It's an interesting way to solve this, the thing I do not really like, is
that the storage config is now not the only single source of truth for
"which pool to use/import" anymore

We could actually add a nodename to pool mapping as a property to the
ZFS Pool storage config schema instead, that'd would keep this information
contained to the configuration and make lookup a bit cheaper.

cheers,
Thomas



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Allow dynamic pool name discovery on ZFSPoolPlugin

[Pablo Ruiz]

Cause the dataset lies at different pools on each server group:


   - old servers: rpool/data
   - new servers: newpool/data

You can only rename pools (or datasets) within their own.

Regards
Pablo


On Mon, Nov 23, 2020 at 8:55 AM Carsten Härle <Carsten.Haerle@straightec.de>
wrote:

> Why not just rename the ZFS-dataset on the old server with "zfs rename"?
> Seems to be much easier than implementing a separate feature.
>
> -----Ursprüngliche Nachricht-----
> Von: pve-devel [mailto:pve-devel-bounces@lists.proxmox.com] Im Auftrag
> von Thomas Lamprecht
> Gesendet: Montag, 23. November 2020 08:50
> An: Proxmox VE development discussion; Pablo Ruiz;
> pve-devel@pve.proxmox.com; Fabian Ebner
> Betreff: Re: [pve-devel] Allow dynamic pool name discovery on ZFSPoolPlugin
>
> Hi,
>
> On 21.11.20 14:08, Pablo Ruiz wrote:
> > Hi,
> >
> > I've just made a little custom storage plugin which basically
> > overrides/extends ZFSPoolPluging so the pool name instead of being fixed,
> > can be 'dynamically' guessed on each cluster node.
> >
> > While adding some new nodes to one of our current clusters, due to
> hardware
> > differences, we ended up with new nodes having a different zpool's layout
> > than the already existing nodes. Our existing nodes had a pool named
> > 'rpool/data' (a dataset nested into system's main pool). While the new
> > nodes have a dedicated pool (data) independent of 'rpool'. So we needed a
> > way to have the same storage use different backing zfs pools on each
> server
> > (ie. on old servers it should use 'rpool/data', while on newer ones it
> > should be using 'data').
> >
> > Currently proxmox does not allow overriding an storage.cfg's property
> > 'per-node'. So we came up with a simple custom plugin which basically
> looks
> > up the pool with an attribute like 'pve:id=$storeid', and uses this pool
> > obtained dynamically.
> >
> > We could have simply added a new store with a different id, but that
> would
> > break our orchestration and automatic deployment of machines, and add
> some
> > additional management issues by having to track the store of each
> machine,
> > specially if/when moving vms around.
> >
> > That said, the plugin works, but I think this feature may be of use to
> > others. Would a patch against upstream ZFSPoolPlugin accepted, so we can
> > avoid this custom plugin in the future?
> >
> > I've posted the custom plugin as a gist:
> > https://gist.github.com/pruiz/5d7fbd75efb413ac15d2d0e3ef54f32a
> >
>
> thanks for sharing your solution!
>
> It's an interesting way to solve this, the thing I do not really like, is
> that the storage config is now not the only single source of truth for
> "which pool to use/import" anymore
>
> We could actually add a nodename to pool mapping as a property to the
> ZFS Pool storage config schema instead, that'd would keep this information
> contained to the configuration and make lookup a bit cheaper.
>
> cheers,
> Thomas
>
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
>
>
>
>
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>

Re: [pve-devel] Allow dynamic pool name discovery on ZFSPoolPlugin

[Pablo Ruiz]

Hi Thomas,

I could try that one, but I am not sure I understood what you mean. Can you
provide an example of the syntax you envision for storage.cfg?

Regards

On Mon, Nov 23, 2020 at 8:49 AM Thomas Lamprecht <t.lamprecht@proxmox.com>
wrote:

> Hi,
>
> On 21.11.20 14:08, Pablo Ruiz wrote:
> > Hi,
> >
> > I've just made a little custom storage plugin which basically
> > overrides/extends ZFSPoolPluging so the pool name instead of being fixed,
> > can be 'dynamically' guessed on each cluster node.
> >
> > While adding some new nodes to one of our current clusters, due to
> hardware
> > differences, we ended up with new nodes having a different zpool's layout
> > than the already existing nodes. Our existing nodes had a pool named
> > 'rpool/data' (a dataset nested into system's main pool). While the new
> > nodes have a dedicated pool (data) independent of 'rpool'. So we needed a
> > way to have the same storage use different backing zfs pools on each
> server
> > (ie. on old servers it should use 'rpool/data', while on newer ones it
> > should be using 'data').
> >
> > Currently proxmox does not allow overriding an storage.cfg's property
> > 'per-node'. So we came up with a simple custom plugin which basically
> looks
> > up the pool with an attribute like 'pve:id=$storeid', and uses this pool
> > obtained dynamically.
> >
> > We could have simply added a new store with a different id, but that
> would
> > break our orchestration and automatic deployment of machines, and add
> some
> > additional management issues by having to track the store of each
> machine,
> > specially if/when moving vms around.
> >
> > That said, the plugin works, but I think this feature may be of use to
> > others. Would a patch against upstream ZFSPoolPlugin accepted, so we can
> > avoid this custom plugin in the future?
> >
> > I've posted the custom plugin as a gist:
> > https://gist.github.com/pruiz/5d7fbd75efb413ac15d2d0e3ef54f32a
> >
>
> thanks for sharing your solution!
>
> It's an interesting way to solve this, the thing I do not really like, is
> that the storage config is now not the only single source of truth for
> "which pool to use/import" anymore
>
> We could actually add a nodename to pool mapping as a property to the
> ZFS Pool storage config schema instead, that'd would keep this information
> contained to the configuration and make lookup a bit cheaper.
>
> cheers,
> Thomas
>
>
>