[PVE-User] ZFZ questions - performance and caching

[PVE-User] ZFZ questions - performance and caching

[Gregor Burck]

Hi,

for calculating an backup server wich should be in worst case  
additionaly an virtualisation server we thougth about a setup with  
HDDs and for ZFZ chaching NVMes.

We plan all with enterprise products,...

I guess following Setup:

A 12 3,5 '' bay Server, 6 PVEe sockets, maybee with an 2,5'' Rear  
entension kit for the OS

Backup envirement:

4 (or6) HDDs, 6 (or 8) TB without chaching

For virtualisation enviroment:

4 (or6) HDDs, 6 (or 8) TB with NVMe chaching


I think, for the backup enviroment there shouldn't be a chaching  
neccessary, cause here is the network the limitation.

The VE part should be a little performer, but how big should the NVMe  
cache? As I read, it is for the ZFZ log.
And how much is the performance increasemnt?

Bye

Gregor

Re: [PVE-User] ZFZ questions - performance and caching

[Gregor Burck]

Hi,

here: https://pbs.proxmox.com/docs/sysadmin.html#zfs-administration

I read something about the log device:
The maximum size of a log device should be about half the size of  
physical memory, so this is usually quite small. The rest of the SSD  
can be used as cache.

So the main question is, how many RAM I should plan,...

When I should plan 5GB RAM (without desuplication) for every TB HDD I  
think I need :

minimum:

4 (hdds) x 6 (TB) x 5 (RAM) x 2 (PBS and PVE) = 120GB RAM

maximum:

6 (hdds) x 8 (TB) x 5 (RAM) x 2 (PBS and PVE) = 240GB RAM

only for the System without RAM for the VMs? That's quite a lot,....
Maybee it is better to forget ZFS and use a hardware raid controler?

The planed controler is an HP P840, wich could RAID or HBA.

Bye

Gregor

Re: [PVE-User] ZFZ questions - performance and caching

[Mikhail]

Hi Gregor,

How do you plan to setup backup on this server? Are you going to use
Proxmox Backup Server for this, or just plain vzdump files?

If you're considering PBS, which you should, and you have at least a
pair of NVME devices available, then I would set ZFS to use those NVME
as special device. We have similar setup deployed recently - 12x3,5 HGST
14TB SAS disks + 2 x Intel 1.6TB NVME (using rear bays, Supermicro 2U
platform). This server is purely used as PBS. ZFS is built on top of 11
x 14TB SAS drives in RAIDZ-3 config (I know not optimal for speed, but
we want to see all space available under single pool) and NVME disks are
used as ZFS special device (mirrored). Remaining 12th SAS disk is set to
be hot-spare under ZFS pool. This gives us ~100TB of space and we are
satisfied with RAIDZ-3 speeds we get - we're able to read and write at 1
Gigabyte/s, which is enough for our case as we're limited by 10gbit
networking. Using NVME as SLOG/L2ARC device does not seem to be optimal
in case when used with PBS. Also, if you're thinking about Supermicro
then I would add SATADOM devices for the OS - this will allow you to use
entire NVME disks for ZFS without partitioning it for OS/boot.

regards,
Mikhail.

On 3/29/21 4:28 PM, Gregor Burck wrote:
> Hi,
> 
> for calculating an backup server wich should be in worst case
> additionaly an virtualisation server we thougth about a setup with HDDs
> and for ZFZ chaching NVMes.
> 
> We plan all with enterprise products,...
> 
> I guess following Setup:
> 
> A 12 3,5 '' bay Server, 6 PVEe sockets, maybee with an 2,5'' Rear
> entension kit for the OS
> 
> Backup envirement:
> 
> 4 (or6) HDDs, 6 (or 8) TB without chaching
> 
> For virtualisation enviroment:
> 
> 4 (or6) HDDs, 6 (or 8) TB with NVMe chaching
> 
> 
> I think, for the backup enviroment there shouldn't be a chaching
> neccessary, cause here is the network the limitation.
> 
> The VE part should be a little performer, but how big should the NVMe
> cache? As I read, it is for the ZFZ log.
> And how much is the performance increasemnt?
> 
> Bye
> 
> Gregor
> 
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 

Re: [PVE-User] ZFZ questions - performance and caching

[Gregor Burck]

Hi Mikhail,


> How do you plan to setup backup on this server? Are you going to use
> Proxmox Backup Server for this, or just plain vzdump files?
sure I plan to use PBS for Backup, in our Testsystem it is brilliant ;-)

I think, our Setup is quite smaller than yours, I calculate that 16 TB  
space for Backup is enough, that's three times the volume that we 've  
used for all VMs complete, and not all VMs are produktive and should  
have to backuped.

The network will be a 10GB too.

But I plan to use the Backupserver as Backup Virtualisation System  
too, so I plan two Raid Pools.
For my opinion the Backup Pool could be 'slower'. The Virtualisation  
Pool should be 'quicker'

So I wonder if I should use the Rais Controller for Rais or as HBA and  
made the Raid over ZFS.
HW Raid is conservativ but with ZFS I've no experience.

So I've to decide between both models.

Bye

Gregor