[PVE-User] Proxmox and ZFS on large JBOD ?

[PVE-User] Proxmox and ZFS on large JBOD ?

[Martin Dziobek]

Hi pve-users !

Does anybody has experiences if proxmox works
flawlessly to manage a large zfs volume consisting
of a SAS-connected  JBOD of 60 * 1TB-HDDs ?

Right now, management is done with a regular
Debian 11 installation, and rebooting the thing
always ends up in a timeout mess at network startup,
because it takes ages to enumerate all those member disks,
import the zpool and export it via NFS.

I am considering to install Proxmox on this server for the
sole purpose of smooth startup and management operation.
Might that be a stable solution ?

Best regards,
Martin
 

Re: [PVE-User] Proxmox and ZFS on large JBOD ?

[Adam Thompson]

IMHO, you want TrueNAS, not Proxmox, to solve this problem.

Ultimately, the problem is SystemD's notion of dependencies and timeouts, which Proxmox+OpenZFS still relies on.

Source: I have a Debian 10 system with 29 storage devices, 24 of which are multipathed, and have had to edit & override various systemd settings to get it to boot cleanly, reliably.
-Adam

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: pve-user <pve-user-bounces@lists.proxmox.com> on behalf of Martin Dziobek <dziobek@hlrs.de>
Sent: Monday, January 10, 2022 7:13:50 AM
To: pve-user@pve.proxmox.com <pve-user@pve.proxmox.com>
Subject: [PVE-User] Proxmox and ZFS on large JBOD ?

Hi pve-users !

Does anybody has experiences if proxmox works
flawlessly to manage a large zfs volume consisting
of a SAS-connected  JBOD of 60 * 1TB-HDDs ?

Right now, management is done with a regular
Debian 11 installation, and rebooting the thing
always ends up in a timeout mess at network startup,
because it takes ages to enumerate all those member disks,
import the zpool and export it via NFS.

I am considering to install Proxmox on this server for the
sole purpose of smooth startup and management operation.
Might that be a stable solution ?

Best regards,
Martin


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Proxmox and ZFS on large JBOD ?

[Adam Thompson]

For additional clarity, I meant TrueNAS CORE, which is still based on FreeBSD and is still free as in beer.
TrueNAS SCALE is Linux-based, as another commenter mentioned, but also optimized for large storage... and is also free?
SCALE is the future direction, I think, while CORE is the tried and true, mature product but is now sort of deprecated?  iX's messaging and branding isn't very clear on this.
Either should work for you, AFAICT, a lot better than trying to use Proxmox as a NAS.
FWIW, I have used TrueNAS CORE in the past (when it was still called FreeNAS) as the NFS server *for* a Proxmox install, and it was solid.
-Adam

-----Original Message-----
From: pve-user <pve-user-bounces@lists.proxmox.com> On Behalf Of Adam Thompson
Sent: Monday, January 10, 2022 8:32 AM
To: Proxmox VE user list <pve-user@lists.proxmox.com>; pve-user@pve.proxmox.com
Subject: Re: [PVE-User] Proxmox and ZFS on large JBOD ?

IMHO, you want TrueNAS, not Proxmox, to solve this problem.

Ultimately, the problem is SystemD's notion of dependencies and timeouts, which Proxmox+OpenZFS still relies on.

Source: I have a Debian 10 system with 29 storage devices, 24 of which are multipathed, and have had to edit & override various systemd settings to get it to boot cleanly, reliably.
-Adam

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: pve-user <pve-user-bounces@lists.proxmox.com> on behalf of Martin Dziobek <dziobek@hlrs.de>
Sent: Monday, January 10, 2022 7:13:50 AM
To: pve-user@pve.proxmox.com <pve-user@pve.proxmox.com>
Subject: [PVE-User] Proxmox and ZFS on large JBOD ?

Hi pve-users !

Does anybody has experiences if proxmox works
flawlessly to manage a large zfs volume consisting
of a SAS-connected  JBOD of 60 * 1TB-HDDs ?

Right now, management is done with a regular
Debian 11 installation, and rebooting the thing
always ends up in a timeout mess at network startup,
because it takes ages to enumerate all those member disks,
import the zpool and export it via NFS.

I am considering to install Proxmox on this server for the
sole purpose of smooth startup and management operation.
Might that be a stable solution ?

Best regards,
Martin


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Proxmox and ZFS on large JBOD ?

[Ralf Storm]

Hello Martin,

should be no problem and you can configure wait-time at startup for ZFS 
to avoid timeouts, this is also described in the proxmox doku in the ZFS 
chapter.

The doku for proxmox is very good, keep on it and you will be happy.

best regards

Ralf

Am 10/01/2022 um 14:13 schrieb Martin Dziobek:
> Hi pve-users !
>
> Does anybody has experiences if proxmox works
> flawlessly to manage a large zfs volume consisting
> of a SAS-connected  JBOD of 60 * 1TB-HDDs ?
>
> Right now, management is done with a regular
> Debian 11 installation, and rebooting the thing
> always ends up in a timeout mess at network startup,
> because it takes ages to enumerate all those member disks,
> import the zpool and export it via NFS.
>
> I am considering to install Proxmox on this server for the
> sole purpose of smooth startup and management operation.
> Might that be a stable solution ?
>
> Best regards,
> Martin
>   
>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
-- 
Ralf Storm

Systemadministrator

Re: [PVE-User] Proxmox and ZFS on large JBOD ?

[Kyle Schmitt]

I would separate it out.  I'm running a moderate sized ZFS array on
one box, with a small 10GB ethernet network dedicated to serving NFS
from that array to my proxmox nodes.  I had bonded 1G before for the
same setup, and usually it was a non-issue, but sometimes there was a
slowdown. With 10G no slowdown on my workloads.

I haven't explored serving ZFS from linux, so mine is on FreeBSD, and
it's rock solid.

--Kyle

On Mon, Jan 10, 2022 at 7:21 AM Martin Dziobek <dziobek@hlrs.de> wrote:
>
> Hi pve-users !
>
> Does anybody has experiences if proxmox works
> flawlessly to manage a large zfs volume consisting
> of a SAS-connected  JBOD of 60 * 1TB-HDDs ?
>
> Right now, management is done with a regular
> Debian 11 installation, and rebooting the thing
> always ends up in a timeout mess at network startup,
> because it takes ages to enumerate all those member disks,
> import the zpool and export it via NFS.
>
> I am considering to install Proxmox on this server for the
> sole purpose of smooth startup and management operation.
> Might that be a stable solution ?
>
> Best regards,
> Martin
>
>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>

Re: [PVE-User] Proxmox and ZFS on large JBOD ?

[Kyle Schmitt]

Oh, and since we're throwing around NAS flavors that support ZFS,
Xigma-NAS, which used to be NAS4Free, is free as in speech and beer,
FreeBSD based.  The UI looks a little dated, but it's very solid.  I
only moved away from it because I don't use any features but ZFS and
NFS.

--Kyle

On Mon, Jan 10, 2022 at 1:50 PM Kyle Schmitt <kyleaschmitt@gmail.com> wrote:
>
> I would separate it out.  I'm running a moderate sized ZFS array on
> one box, with a small 10GB ethernet network dedicated to serving NFS
> from that array to my proxmox nodes.  I had bonded 1G before for the
> same setup, and usually it was a non-issue, but sometimes there was a
> slowdown. With 10G no slowdown on my workloads.
>
> I haven't explored serving ZFS from linux, so mine is on FreeBSD, and
> it's rock solid.
>
> --Kyle
>
> On Mon, Jan 10, 2022 at 7:21 AM Martin Dziobek <dziobek@hlrs.de> wrote:
> >
> > Hi pve-users !
> >
> > Does anybody has experiences if proxmox works
> > flawlessly to manage a large zfs volume consisting
> > of a SAS-connected  JBOD of 60 * 1TB-HDDs ?
> >
> > Right now, management is done with a regular
> > Debian 11 installation, and rebooting the thing
> > always ends up in a timeout mess at network startup,
> > because it takes ages to enumerate all those member disks,
> > import the zpool and export it via NFS.
> >
> > I am considering to install Proxmox on this server for the
> > sole purpose of smooth startup and management operation.
> > Might that be a stable solution ?
> >
> > Best regards,
> > Martin
> >
> >
> > _______________________________________________
> > pve-user mailing list
> > pve-user@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >

Re: [PVE-User] Proxmox and ZFS on large JBOD ?

[Daniel Plominski]

Hello Martin,

what is the exact use case?

If it is "only" about providing a robust ZFS based storage server for 
(NFS, SMB, ISCSI), then I can only recommend TrueNAS CORE.

One of our servers with 128 GB RAM handles about 3 million snapshots 
without problems (with a complex replication structure).

After about 10 years of ZFS experience, I find the ZFS implementation in 
the FreeBSD kernel to be one of the most robust besides Solaris.

If you also want to virtualize on this machine, Proxmox itself or 
TrueNAS Scale would of course still be a possibility.

For larger data server setups, however, I think the interaction between 
the FreeBSD kernel and ZFS is better than ZFSonLinux.

Mit freundlichen Grüßen

DANIEL PLOMINSKI
Leitung IT | Head of IT

Telefon 09265 808-151 | Mobil 0151 58026316 | dpl@ass.de
PGP Key: https://pgp.ass.de/dpl@ass.de.asc
PGP Fingerprint: 74DBC06BD9F63187C4DF8934C96585A89CFC10B3

		 Company Logo

			ASS-Einrichtungssysteme GmbH 		
			ASS-Adam-Stegner-Straße 19 | D-96342 Stockheim

			Geschäftsführer: Matthias Stegner, Michael Stegner, Ralph Müller 		
			Amtsgericht Coburg HRB 3395 | Ust-ID: DE218715721
Bottom_Line

Re: [PVE-User] Proxmox and ZFS on large JBOD ?

[Martin Dziobek]

Thanks for all your input !

I tried TrueNAS Scale, but switched to TrueNAS Core, because Scale is still somewhat Beta,
and license conditions are unclear.

After discovering the possibility to run VMs in TrueNAS, it came to my mind to run
the Proxmox Backup Server in a VM on TrueNas, with an underlying TrueNAS Volume
as a Backup media. Lets see ...

Best regards,
Martin



On Mon, 10 Jan 2022 14:13:50 +0100
Martin Dziobek <dziobek@hlrs.de> wrote:

> Hi pve-users !
> 
> Does anybody has experiences if proxmox works
> flawlessly to manage a large zfs volume consisting
> of a SAS-connected  JBOD of 60 * 1TB-HDDs ?
> 
> Right now, management is done with a regular
> Debian 11 installation, and rebooting the thing
> always ends up in a timeout mess at network startup,
> because it takes ages to enumerate all those member disks,
> import the zpool and export it via NFS.
> 
> I am considering to install Proxmox on this server for the
> sole purpose of smooth startup and management operation.
> Might that be a stable solution ?
> 
> Best regards,
> Martin
>  
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>