[PVE-User] pbs - Nasty atime bug in Ubuntu's kernel-builtin zfs

[PVE-User] pbs - Nasty atime bug in Ubuntu's kernel-builtin zfs

[Cornelius Hoffmann]

Hi,

I ran in to a very nasty bug with the zfs version included in Ubuntu 
(NOT the pbs default kernel, I'm running PBS in a container), which is 
the base for Proxmox as well as far as I understand it and wanted to 
warn about a problem with access times that is encountered there.

Essentially, this is [1] and it causes the access time update of a 
`touch` command to fail and set it to epoch 0, so Jan 1, 1970. This will 
cause all chunks that should not be pruned to have an extremely old 
atime and thus be pruned, causing a corruption of all backups.

I'm writing here to make sure there is not maybe a kernel update waiting 
with these changes. The newest zfs-linux version in noble has the bug 
fixed as it includes [2], but that change is only visible in the dkms 
module, which I installed to fix this problem in the meantime.

Cc to the pve-user list, if anybody is also running pbs on the normal 
Ubunutu noble kernel on zfs storage.

Best,
Cornelius

[1] https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2077487

[2] https://github.com/openzfs/zfs/pull/15773

-- 
Cornelius Hoffmann (he/him er/ihm)
#gernperDu


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] pbs - Nasty atime bug in Ubuntu's kernel-builtin zfs

[Thomas Lamprecht]

Hello,

Am 16/10/2024 um 17:06 schrieb Cornelius Hoffmann:
> I ran in to a very nasty bug with the zfs version included in Ubuntu 
> (NOT the pbs default kernel, I'm running PBS in a container), which is 
> the base for Proxmox as well as far as I understand it and wanted to 
> warn about a problem with access times that is encountered there.
> 
> Essentially, this is [1] and it causes the access time update of a 
> `touch` command to fail and set it to epoch 0, so Jan 1, 1970. This will 
> cause all chunks that should not be pruned to have an extremely old 
> atime and thus be pruned, causing a corruption of all backups.
> 
> I'm writing here to make sure there is not maybe a kernel update waiting 
> with these changes. The newest zfs-linux version in noble has the bug 
> fixed as it includes [2], but that change is only visible in the dkms 
> module, which I installed to fix this problem in the meantime.

Thanks for reaching out to us, but please note that Proxmox maintains
its own ZFS downstream package for both user-space tools and kernel
module.

Both the bad commit and the fix were in ZFS 2.2.3 [0], and unlike Ubuntu
we never backported the problematic commit manually, so the Proxmox
kernel and its ZFS module were never affected, FWICT.

I also tried the reproducer, as it is simple enough and just to be sure,
and I could _not_ observe the broken behavior from ZFS.

[0]: https://github.com/openzfs/zfs/releases/tag/zfs-2.2.3

regards,
 Thomas


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] pbs - Nasty atime bug in Ubuntu's kernel-builtin zfs

[Cornelius Hoffmann]

Hi Thomas,

On 16.10.24 17:56, Thomas Lamprecht wrote:
> note that Proxmox maintains
> its own ZFS downstream package for both user-space tools and kernel
> module.
>
> Both the bad commit and the fix were in ZFS 2.2.3 [0], and unlike Ubuntu
> we never backported the problematic commit manually, so the Proxmox
> kernel and its ZFS module were never affected, FWICT.

Good to hear, mainly just wanted to make sure it stays that way :D
And maybe warn other people, since there are threads on the forum about 
running PBS as a container, mostly LXC but also some that are not on 
proxmox.

>
> I also tried the reproducer, as it is simple enough and just to be sure,
> and I could_not_ observe the broken behavior from ZFS.

I setup a PBS in a VM in the meantime for testing and also couldn't 
reproduce it with your kernel.

Best,
Cornelius


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user