[PVE-User] Shared ZFS storage that supports LXC's

[PVE-User] Shared ZFS storage that supports LXC's

[Bryan Fields]

I've been working to migrate my servers to a shared storage from a local-zfs.

My backend is a Linux server with 24 16T SAS drives in ZFS as follows:
4x (6d raidz2) + 3x mirror 4T NVME + 375G Optane split 32G/312G ZIL/L2ARC
768G of ram, and 1/2 that is for zfs arc.  All connected via 2x10g lag with 
jumbo frames to the other servers.

I have been working with the ZFS over ISCSI backend for my VM's and it works 
very well.  The only issue is there's not any support for multipath, but with 
a 10g network, I'm not running into limits here for practical purposes.

I do  get a warning in proxmox when moving a VM into it or making a snapshot:
Warning: volblocksize (8192) is much less than the minimum allocation
unit (32768), which wastes at least 75% of space. To reduce wasted space,
use a larger volblocksize (32768 is recommended), fewer dRAID data disks
per group, or smaller sector size (ashift).

I can't find exactly what this is referring to or how to fix it.  Does anyone 
have insight into this message?

With the LXC's I've found they don't support this backend storage.  (and it's 
not mentioned in the docs) I assume this is do to them needing a filesystem, 
not a block device.  My option here would be to run NFS for shared storage, 
but this loses the ability to do snapshots (a must have).  LVM would work, but 
it's not able to be shared.

I was thinking it might make sense to do this as a per LXC NFS mount via ZFS, 
and then the PVE node makes a new dataset in zfs on the shared storage server 
via ssh for that lxc.  This is basically how the zfs over iscsi is handled 
today, as how I understand it.

Has anyone solved this?  I'd like an option here to migrate LXC's from local 
to shared storage on a linux/zfs server.

Thanks,
-- 
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

Re: [PVE-User] Shared ZFS storage that supports LXC's

[Fiona Ebner]

Hi,

Am 29.10.23 um 11:16 schrieb Bryan Fields:
> I do  get a warning in proxmox when moving a VM into it or making a
> snapshot:
> Warning: volblocksize (8192) is much less than the minimum allocation
> unit (32768), which wastes at least 75% of space. To reduce wasted space,
> use a larger volblocksize (32768 is recommended), fewer dRAID data disks
> per group, or smaller sector size (ashift).
> 
> I can't find exactly what this is referring to or how to fix it.  Does
> anyone have insight into this message?
> 

You can set the default blocksize in the storage configuration in
Proxmox VE, either in the UI (Block Size option) or via 'pvesm set <your
storage ID> --blocksize 32K'. Note that this will only apply to newly
allocated or moved volumes and not change existing ones.

> With the LXC's I've found they don't support this backend storage.  (and
> it's not mentioned in the docs) I assume this is do to them needing a
> filesystem, not a block device.  My option here would be to run NFS for
> shared storage, but this loses the ability to do snapshots (a must
> have).  LVM would work, but it's not able to be shared.
> 

It is documented via the content types (only 'images' for ZFS over
iSCSI, but not 'rootdir'):

https://pve.proxmox.com/pve-docs/chapter-pvesm.html#_common_storage_propertieshttps://pve.proxmox.com/pve-docs/chapter-pvesm.html#storage_zfs

From a quick glance, the only storage type that fits your needs, i.e.
snapshots, shared and support for 'rootdir' content type is Ceph/RBD.
Everything else requires workarounds and won't be fully integrated.

Best Regards,
Fiona