[PVE-User] Proxmox VE 8.1 released!

[PVE-User] Proxmox VE 8.1 released!

[Martin Maurer]

We're very excited to announce the release 8.1 of Proxmox Virtual 
Environment! It's based on Debian 12.2 "Bookworm" but uses a newer Linux 
kernel 6.5, QEMU 8.1.2, and OpenZFS 2.2.0 ((with stable fixes backported)

Here is a selection of the highlights of Proxmox VE 8.1

- Debian 12.2 (“Bookworm”), but uses a newer Linux kernel 6.5 as stable 
default
- latest versions of QEMU 8.1.2 and ZFS 2.2.0 including the most 
important bugfixes from 2.2.1 already
- Software-defined Networking (SDN)
- Secure Boot
- New flexible notification system with matcher-based approach
- Ceph Server: Ceph Reef 18.2.0 is default, and Ceph Quincy 17.2.7 comes 
with continued support.
-Countless GUI and API improvements.

As always, we have included countless bugfixes and improvements on many 
places; see the release notes for all details.

Release notes
https://pve.proxmox.com/wiki/Roadmap

Press release
https://www.proxmox.com/en/news/press-releases/

Video tutorial
https://www.proxmox.com/en/training/video-tutorials/item/what-s-new-in-proxmox-ve-8-1

Download
https://www.proxmox.com/en/downloads
Alternate ISO download:
https://enterprise.proxmox.com/iso

Documentation
https://pve.proxmox.com/pve-docs

Community Forum
https://forum.proxmox.com

Bugtracker
https://bugzilla.proxmox.com

Source code
https://git.proxmox.com

There has been a lot of feedback from our community members and 
customers, and many of you reported bugs, submitted patches and were 
involved in testing - THANK YOU for your support!

FAQ

Q: Can I upgrade latest Proxmox VE 7 to 8 with apt?
A: Yes, please follow the upgrade instructions on 
https://pve.proxmox.com/wiki/Upgrade_from_7_to_8

Q: Can I upgrade an 8.0 installation to the stable 8.1 via apt?
A: Yes, upgrading from is possible via apt and GUI.

Q: Can I install Proxmox VE 8.1 on top of Debian 12 "Bookworm"?
A: Yes, see 
https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm

Q: Can I upgrade my Proxmox VE 7.4 cluster with Ceph Pacific to Proxmox 
VE 8.1 and to Ceph Reef?
A: This is a three-step process. First, you have to upgrade Ceph from 
Pacific to Quincy, and afterwards you can then upgrade Proxmox VE from 
7.4 to 8.1. As soon as you run Proxmox VE 8.1, you can upgrade Ceph to 
Reef. There are a lot of improvements and changes, so please follow 
exactly the upgrade documentation:
https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy
https://pve.proxmox.com/wiki/Upgrade_from_7_to_8
https://pve.proxmox.com/wiki/Ceph_Quincy_to_Reef

Q: Where can I get more information about feature updates?
A: Check the roadmap, forum, the mailing list, and/or subscribe to our 
newsletter.

-- 
Best Regards,

Martin Maurer
Proxmox VE project leader

Re: [PVE-User] Proxmox VE 8.1 released!

[Jan Vlach]

Hello Martin,

I'm sorry for stupid question, but I don't quite follow what the OpenZFS  2.2.0 with "the most important bugfixes from 2.2.1" means.

there is issue 15526 (https://github.com/openzfs/zfs/issues/15526 <https://github.com/openzfs/zfs/issues/15526>) in openzfs that enables block cloning feature by default and that on linux "eats" data.

OpenZFS 2.2.1 specifically switches the block cloning on by default to off by default and nothing else. ( https://github.com/openzfs/zfs/commit/479dca51c66a731e637bd2d4f9bba01a05f9ac9f <https://github.com/openzfs/zfs/commit/479dca51c66a731e637bd2d4f9bba01a05f9ac9f>  )

So is the code at 2.2.0 (toxic) eating data by default or at 2.2.1 where block cloning is disabled by default and therefore safe?

Thank you for clarification,
JV

> On 23. 11. 2023, at 15:27, Martin Maurer <martin@proxmox.com> wrote:
> 
> We're very excited to announce the release 8.1 of Proxmox Virtual Environment! It's based on Debian 12.2 "Bookworm" but uses a newer Linux kernel 6.5, QEMU 8.1.2, and OpenZFS 2.2.0 ((with stable fixes backported)
> 
> Here is a selection of the highlights of Proxmox VE 8.1
> 
> - Debian 12.2 (“Bookworm”), but uses a newer Linux kernel 6.5 as stable default
> - latest versions of QEMU 8.1.2 and ZFS 2.2.0 including the most important bugfixes from 2.2.1 already
> - Software-defined Networking (SDN)
> - Secure Boot
> - New flexible notification system with matcher-based approach
> - Ceph Server: Ceph Reef 18.2.0 is default, and Ceph Quincy 17.2.7 comes with continued support.
> -Countless GUI and API improvements.
> 
> As always, we have included countless bugfixes and improvements on many places; see the release notes for all details.
> 
> Release notes
> https://pve.proxmox.com/wiki/Roadmap
> 
> Press release
> https://www.proxmox.com/en/news/press-releases/
> 
> Video tutorial
> https://www.proxmox.com/en/training/video-tutorials/item/what-s-new-in-proxmox-ve-8-1
> 
> Download
> https://www.proxmox.com/en/downloads
> Alternate ISO download:
> https://enterprise.proxmox.com/iso
> 
> Documentation
> https://pve.proxmox.com/pve-docs
> 
> Community Forum
> https://forum.proxmox.com
> 
> Bugtracker
> https://bugzilla.proxmox.com
> 
> Source code
> https://git.proxmox.com
> 
> There has been a lot of feedback from our community members and customers, and many of you reported bugs, submitted patches and were involved in testing - THANK YOU for your support!
> 
> FAQ
> 
> Q: Can I upgrade latest Proxmox VE 7 to 8 with apt?
> A: Yes, please follow the upgrade instructions on https://pve.proxmox.com/wiki/Upgrade_from_7_to_8
> 
> Q: Can I upgrade an 8.0 installation to the stable 8.1 via apt?
> A: Yes, upgrading from is possible via apt and GUI.
> 
> Q: Can I install Proxmox VE 8.1 on top of Debian 12 "Bookworm"?
> A: Yes, see https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
> 
> Q: Can I upgrade my Proxmox VE 7.4 cluster with Ceph Pacific to Proxmox VE 8.1 and to Ceph Reef?
> A: This is a three-step process. First, you have to upgrade Ceph from Pacific to Quincy, and afterwards you can then upgrade Proxmox VE from 7.4 to 8.1. As soon as you run Proxmox VE 8.1, you can upgrade Ceph to Reef. There are a lot of improvements and changes, so please follow exactly the upgrade documentation:
> https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy
> https://pve.proxmox.com/wiki/Upgrade_from_7_to_8
> https://pve.proxmox.com/wiki/Ceph_Quincy_to_Reef
> 
> Q: Where can I get more information about feature updates?
> A: Check the roadmap, forum, the mailing list, and/or subscribe to our newsletter.
> 
> -- 
> Best Regards,
> 
> Martin Maurer
> Proxmox VE project leader
> 
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Proxmox VE 8.1 released!

[Jan Vlach]

https://www.reddit.com/r/zfs/comments/1826lgs/psa_its_not_block_cloning_its_a_data_corruption/ <https://www.reddit.com/r/zfs/comments/1826lgs/psa_its_not_block_cloning_its_a_data_corruption/>

found this one in meantime, no idea what's the current state of affairs in proxmox ...

JV

> On 23. 11. 2023, at 23:35, Jan Vlach <janus@volny.cz> wrote:
> 
> Hello Martin,
> 
> I'm sorry for stupid question, but I don't quite follow what the OpenZFS  2.2.0 with "the most important bugfixes from 2.2.1" means.
> 
> there is issue 15526 (https://github.com/openzfs/zfs/issues/15526 <https://github.com/openzfs/zfs/issues/15526>) in openzfs that enables block cloning feature by default and that on linux "eats" data.
> 
> OpenZFS 2.2.1 specifically switches the block cloning on by default to off by default and nothing else. ( https://github.com/openzfs/zfs/commit/479dca51c66a731e637bd2d4f9bba01a05f9ac9f <https://github.com/openzfs/zfs/commit/479dca51c66a731e637bd2d4f9bba01a05f9ac9f>  )
> 
> So is the code at 2.2.0 (toxic) eating data by default or at 2.2.1 where block cloning is disabled by default and therefore safe?
> 
> Thank you for clarification,
> JV
> 
>> On 23. 11. 2023, at 15:27, Martin Maurer <martin@proxmox.com <mailto:martin@proxmox.com>> wrote:
>> 
>> We're very excited to announce the release 8.1 of Proxmox Virtual Environment! It's based on Debian 12.2 "Bookworm" but uses a newer Linux kernel 6.5, QEMU 8.1.2, and OpenZFS 2.2.0 ((with stable fixes backported)
>> 
>> Here is a selection of the highlights of Proxmox VE 8.1
>> 
>> - Debian 12.2 (“Bookworm”), but uses a newer Linux kernel 6.5 as stable default
>> - latest versions of QEMU 8.1.2 and ZFS 2.2.0 including the most important bugfixes from 2.2.1 already
>> - Software-defined Networking (SDN)
>> - Secure Boot
>> - New flexible notification system with matcher-based approach
>> - Ceph Server: Ceph Reef 18.2.0 is default, and Ceph Quincy 17.2.7 comes with continued support.
>> -Countless GUI and API improvements.
>> 
>> As always, we have included countless bugfixes and improvements on many places; see the release notes for all details.
>> 
>> Release notes
>> https://pve.proxmox.com/wiki/Roadmap <https://pve.proxmox.com/wiki/Roadmap>
>> 
>> Press release
>> https://www.proxmox.com/en/news/press-releases/
>> 
>> Video tutorial
>> https://www.proxmox.com/en/training/video-tutorials/item/what-s-new-in-proxmox-ve-8-1
>> 
>> Download
>> https://www.proxmox.com/en/downloads
>> Alternate ISO download:
>> https://enterprise.proxmox.com/iso
>> 
>> Documentation
>> https://pve.proxmox.com/pve-docs
>> 
>> Community Forum
>> https://forum.proxmox.com
>> 
>> Bugtracker
>> https://bugzilla.proxmox.com
>> 
>> Source code
>> https://git.proxmox.com
>> 
>> There has been a lot of feedback from our community members and customers, and many of you reported bugs, submitted patches and were involved in testing - THANK YOU for your support!
>> 
>> FAQ
>> 
>> Q: Can I upgrade latest Proxmox VE 7 to 8 with apt?
>> A: Yes, please follow the upgrade instructions on https://pve.proxmox.com/wiki/Upgrade_from_7_to_8
>> 
>> Q: Can I upgrade an 8.0 installation to the stable 8.1 via apt?
>> A: Yes, upgrading from is possible via apt and GUI.
>> 
>> Q: Can I install Proxmox VE 8.1 on top of Debian 12 "Bookworm"?
>> A: Yes, see https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm
>> 
>> Q: Can I upgrade my Proxmox VE 7.4 cluster with Ceph Pacific to Proxmox VE 8.1 and to Ceph Reef?
>> A: This is a three-step process. First, you have to upgrade Ceph from Pacific to Quincy, and afterwards you can then upgrade Proxmox VE from 7.4 to 8.1. As soon as you run Proxmox VE 8.1, you can upgrade Ceph to Reef. There are a lot of improvements and changes, so please follow exactly the upgrade documentation:
>> https://pve.proxmox.com/wiki/Ceph_Pacific_to_Quincy
>> https://pve.proxmox.com/wiki/Upgrade_from_7_to_8
>> https://pve.proxmox.com/wiki/Ceph_Quincy_to_Reef
>> 
>> Q: Where can I get more information about feature updates?
>> A: Check the roadmap, forum, the mailing list, and/or subscribe to our newsletter.
>> 
>> -- 
>> Best Regards,
>> 
>> Martin Maurer
>> Proxmox VE project leader
>> 
>> 
>> _______________________________________________
>> pve-user mailing list
>> pve-user@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 

Re: [PVE-User] Proxmox VE 8.1 released!

[Fabian Grünbichler]

> Jan Vlach <janus@volny.cz> hat am 23.11.2023 23:35 CET geschrieben:
> 
>  
> Hello Martin,
> 
> I'm sorry for stupid question, but I don't quite follow what the OpenZFS  2.2.0 with "the most important bugfixes from 2.2.1" means.
> 
> there is issue 15526 (https://github.com/openzfs/zfs/issues/15526 <https://github.com/openzfs/zfs/issues/15526>) in openzfs that enables block cloning feature by default and that on linux "eats" data.
> 
> OpenZFS 2.2.1 specifically switches the block cloning on by default to off by default and nothing else. ( https://github.com/openzfs/zfs/commit/479dca51c66a731e637bd2d4f9bba01a05f9ac9f <https://github.com/openzfs/zfs/commit/479dca51c66a731e637bd2d4f9bba01a05f9ac9f>  )
> 
> So is the code at 2.2.0 (toxic) eating data by default or at 2.2.1 where block cloning is disabled by default and therefore safe?
> 
> Thank you for clarification,
> JV

we cherry-picked the relevant patches from 2.2.1, but it seems there is now new information that block cloning just made the bug easier to trigger but was not actually the cause. we'll include any follow-up fixes promptly after testing, as usual.

https://git.proxmox.com/?p=zfsonlinux.git;a=commit;h=96c807af63f70dc930328e5801659a5bd40e6d47

Re: [PVE-User] Proxmox VE 8.1 released!

[Fabian Grünbichler]

> Daniel Plominski via pve-user <pve-user@lists.proxmox.com> hat am 24.11.2023 08:57 CET geschrieben:
>
> We are currently using: Proxmox 8.0.4 (Enterprise Sub)
> 
> root@assg32:~# dpkg -l | grep zfs
> ii  libzfs4linux                         2.1.12-pve1                         amd64        OpenZFS filesystem library for Linux - general support
> ii  zfs-initramfs                        2.1.12-pve1                         all          OpenZFS root filesystem capabilities for Linux - initramfs
> ii  zfs-zed                              2.1.12-pve1                         amd64        OpenZFS Event Daemon
> ii  zfsutils-linux                       2.1.12-pve1                         amd64        command-line tools to manage OpenZFS filesystems
> root@assg32:~#
> 
> Are we affected by any ZFS "Silent Corruption Bugs"?

according to the upstream bug tracker, there is one (hard to trigger!) issue that probably dates back to 2013.
 
> On 19.11.2023 we had unexplained ZFS errors on a node:
> 
> root@assg26:~# zpool status -v
>   pool: rpool
> state: ONLINE
> status: One or more devices is currently being resilvered.  The pool will
>         continue to function, possibly in a degraded state.
> action: Wait for the resilver to complete.
>   scan: resilver in progress since Sun Nov 19 08:30:03 2023
>         309G scanned at 2.31G/s, 3.00M issued at 22.9K/s, 742G total
>         0B resilvered, 0.00% done, no estimated completion time
> config:
> 
>         NAME                                                   STATE     READ WRITE CKSUM
>         rpool                                                  ONLINE       0     0     0
>           mirror-0                                             ONLINE       0     0     0
>             spare-0                                            ONLINE       0     0     0
>               nvme-eui.35354830526004030025384700000003-part3  ONLINE       0     0     3
>               nvme0n1p3                                        ONLINE       0     0     0
>             nvme-eui.35354830526003280025384700000003-part3    ONLINE       0     0     3

both your vdevs had 3 checksum errors - that sound peculiar, but more like a hardware issue (memory?). can you try scrubbing the pool when the resilvering is completed?

Re: [PVE-User] Proxmox VE 8.1 released!

[Fabian Grünbichler]

> So if we patch to Proxmox 8.1 (zfs 2.2.0-pve3) and disable "feature@block_cloning" or don't do a pool upgrade, would we have a current reliable state?

PVE 8.1 comes with ZFS with block cloning disabled by default (this was cherry-picked from ZFS 2.2.1, even though PVE comes with 2.2.0). There is no other issue that is known at the moment that would be worse with 2.2 compared to 2.1 or 2.0.

Re: [PVE-User] Proxmox VE 8.1 released!

[Jan Vlach]

as per https://github.com/openzfs/zfs/issues/15526 <https://github.com/openzfs/zfs/issues/15526> discussion,

this is the workaround for the silent data corruption:\x10

echo 0 >  /sys/module/zfs/parameters/zfs_dmu_offset_next_sync

TL;DR version: this parameter (https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#zfs-dmu-offset-next-sync <https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#zfs-dmu-offset-next-sync>) has been switched to 1 in 2.1.5 to trade safety for performance. 


reproducer script is here:
https://github.com/openzfs/zfs/issues/15526#issuecomment-1824966856 <https://github.com/openzfs/zfs/issues/15526#issuecomment-1824966856>

I've managed to hit the data corruption with 8 simultaneous instances of the script. Allegedly, one needs a beefy computer to hit the bug, but I've managed to hit it on 2 core AMD fujitsu futro thin client running opnsense too. (after couple tries)

I didn't hit the behavior on debian 11 (PVE8), but managed to reproduce it in debian 12 (PVE8), FreeBSD 13 and 14.
Illumos' ZFS is fine.

It seems that the bug gets more exposed, because both freebsd's and coreutils's >8.32 cp uses 'file_copy_range'.

JV