[PVE-User] PBS and ZFS Pools Compression?

[PVE-User] PBS and ZFS Pools Compression?

[Lindsay Mathieson]

Setup a ZFS RAID1 pool on my PBS server and enabled lz4 compression on 
it (habit). But given the backups are already compressed, its not really 
going to gain anything is it? possibly even counter productive?


Thanks.

-- 
Lindsay

Re: [PVE-User] PBS and ZFS Pools Compression?

[Roland]

hi,

i think it's counter productive, as you are wasting cpu for
compress/uncompress data,
which should not be further compressible (except some smaller files like
img.fidx etc)

regards
roland

Am 08.04.21 um 15:06 schrieb Lindsay Mathieson:
> Setup a ZFS RAID1 pool on my PBS server and enabled lz4 compression on
> it (habit). But given the backups are already compressed, its not
> really going to gain anything is it? possibly even counter productive?
>
>
> Thanks.
>

Re: [PVE-User] PBS and ZFS Pools Compression?

[Lindsay Mathieson]

On 8/04/2021 11:12 pm, Roland wrote:
> i think it's counter productive, as you are wasting cpu for
> compress/uncompress data, 

Yah, I thought so, thanks.

-- 
Lindsay

Re: [PVE-User] PBS and ZFS Pools Compression?

[Mikhail]

On 4/8/21 4:13 PM, Lindsay Mathieson wrote:
> On 8/04/2021 11:12 pm, Roland wrote:
>> i think it's counter productive, as you are wasting cpu for
>> compress/uncompress data, 
> 
> Yah, I thought so, thanks.
> 

I may be wrong, but AFAIK ZFS detects compressed data and thus it is not
doing double-compression in such cases, so I guess there's no harm here
(we also have lz4 enabled on datastore where Proxmox sends backups).
Also lz4 is cheap, so I doubt it has any significant impact on modern CPUs.

regards,

Re: [PVE-User] PBS and ZFS Pools Compression?

[Dietmar Maurer]

> I may be wrong, but AFAIK ZFS detects compressed data and thus it is not
> doing double-compression in such cases,

AFAIK the only way to detect compressed data is to actually compress it, then
test the size. So this is double-compression ...

Re: [PVE-User] PBS and ZFS Pools Compression?

[Geoff Seeley]

>
> > I may be wrong, but AFAIK ZFS detects compressed data and thus it is not
> > doing double-compression in such cases,
>
> AFAIK the only way to detect compressed data is to actually compress it,
> then
> test the size. So this is double-compression ...
>

ZFS compression is a little more complex than this, but the good news is
that ZFS is also smart enough not to do this!

This is a good article on the subject:
https://klarasystems.com/articles/openzfs1-understanding-transparent-compression/

TL;DR: Enable compression at the pool level and forget about it.

-Geoff

Re: [PVE-User] PBS and ZFS Pools Compression?

[Dietmar Maurer]

> This is a good article on the subject:
> https://klarasystems.com/articles/openzfs1-understanding-transparent-compression/

Can't find where the explain it. ZFS magically detects if data is compressible?
Please can someone give me a hint how they do that?

Re: [PVE-User] PBS and ZFS Pools Compression?

[Roland]

i know that there was smartcompression feature in nexenta:

https://openzfs.org/w/images/4/4d/Compression-Saso_Kiselkov.pdf

afaik, it does not exist in zfsonlinux/openzfs.

on my system, i'm getting

- <250MB/s when writing uncompressible data to zfs pool with lz4 enabled

and

- >450MB/s when writing uncompressible data to zfs pool without compression

regards
roland

Am 08.04.21 um 20:06 schrieb Dietmar Maurer:
>> This is a good article on the subject:
>> https://klarasystems.com/articles/openzfs1-understanding-transparent-compression/
> Can't find where the explain it. ZFS magically detects if data is compressible?
> Please can someone give me a hint how they do that?
>
>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>

Re: [PVE-User] PBS and ZFS Pools Compression?

[Fabian Grünbichler]

On April 8, 2021 8:06 pm, Dietmar Maurer wrote:
>> This is a good article on the subject:
>> https://klarasystems.com/articles/openzfs1-understanding-transparent-compression/
> 
> Can't find where the explain it. ZFS magically detects if data is compressible?
> Please can someone give me a hint how they do that?

no. they compress, and if the result is over a certain threshold, they 
save the uncompressed data to avoid the decompress overhead for barely 
any/no gain.