[pve-devel] [PATCH installer 0/3] fix #6285: always set up zfs modprobe configuration

[pve-devel] [PATCH installer 0/3] fix #6285: always set up zfs modprobe configuration

[Christoph Heiss]

Fixes #6285 [0].

This also came up quite a few times already in the forum, most recently
[1] (german).

Since we do expose the option in the UI for all products nowadays (on
ZFS-on-root installations only tho), just always write to
/etc/modprobe.d/zfs.conf.

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6285
[1] https://forum.proxmox.com/threads/ram-auslastung-nimmt-%C3%BCber-die-zeit-drastisch-zu.164527/#post-760989

Testing
=======

Tested for each product PVE, PBS, PMG the following
  - install on ZFS, not touching the ARC size setting
  - install on ZFS, set some explicit value for ARC size
  - installing on non-ZFS such as ext4

.. and checking afterwards that /etc/modprobe.d/zfs.conf was written and
contained respectively
  - the default value for the product
  - the set value
  - again the default value for the product

Also tested each case with the auto-installer for PVE and PMG.

Diffstat
========

Christoph Heiss (3):
  run env: always return proper value for default zfs max arc size
  tui: bootdisk: always return proper value for default zfs max arc size
  fix #6285: install: always set up zfs modprobe configuration

 Proxmox/Install.pm                          | 11 +++---
 Proxmox/Install/RunEnv.pm                   |  9 +++--
 proxmox-tui-installer/src/views/bootdisk.rs | 32 +++++++++-------
 test/zfs-arc-max.pl                         | 42 +++++++++++++--------
 4 files changed, 57 insertions(+), 37 deletions(-)

-- 
2.48.1



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH installer 1/3] run env: always return proper value for default zfs max arc size

[Christoph Heiss]

In preparation for fixing #6285 [0].

`0` means to just skip writing the module parameter. But (especially)
with the upcoming change in ZFS 2.3 - which makes the size basically
that of the system memory minus 1 GiB - we want to always write some
value.

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6285

Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
---
 Proxmox/Install/RunEnv.pm |  9 +++++----
 test/zfs-arc-max.pl       | 42 +++++++++++++++++++++++++--------------
 2 files changed, 32 insertions(+), 19 deletions(-)

diff --git a/Proxmox/Install/RunEnv.pm b/Proxmox/Install/RunEnv.pm
index 407d05c..02c603d 100644
--- a/Proxmox/Install/RunEnv.pm
+++ b/Proxmox/Install/RunEnv.pm
@@ -337,14 +337,15 @@ sub default_zfs_arc_max {
     my $product = Proxmox::Install::ISOEnv::get('product');
     my $total_memory = query_total_memory();
 
-    # By default limit PVE and low-memory systems, for all others let ZFS decide on its own by
-    # returning `0`, which causes the installer to skip writing the `zfs_arc_max` module parameter.
+    # By default limit PVE and low-memory systems, for all just use the 50% of system memory
     if ($product ne 'pve') {
-	return 0 if $total_memory >= 2048 && $product ne 'pmg';
-	return 0 if $total_memory >= 4096; # PMG's base memory requirement is much higer
+	my $zfs_default_mib = int(sprintf('%.0f', $total_memory / 2));
+	return $zfs_default_mib if $total_memory >= 2048 && $product ne 'pmg';
+	return $zfs_default_mib if $total_memory >= 4096; # PMG's base memory requirement is much higher
     }
 
     my $default_mib = $total_memory * $ZFS_ARC_SYSMEM_PERCENTAGE;
+    # int(sprintf()) rounds up instead of truncating
     my $rounded_mib = int(sprintf('%.0f', $default_mib));
 
     if ($rounded_mib > $ZFS_ARC_MAX_SIZE_MIB) {
diff --git a/test/zfs-arc-max.pl b/test/zfs-arc-max.pl
index 8cf093f..a8c0a37 100755
--- a/test/zfs-arc-max.pl
+++ b/test/zfs-arc-max.pl
@@ -21,8 +21,10 @@ sub mock_product {
     );
 }
 
-my %default_tests = (
-    16 => 64, # at least 64 MiB
+use constant ZFS_ARC_MIN_MIB => 64;
+
+my %default_tests_pve = (
+    16 => ZFS_ARC_MIN_MIB, # at least 64 MiB
     1024 => 102,
     4 * 1024 => 410,
     8 * 1024 => 819,
@@ -31,26 +33,36 @@ my %default_tests = (
     1024 * 1024 => 16384, # maximum of 16 GiB
 );
 
-while (my ($total_mem, $expected) = each %default_tests) {
+my %default_tests_others = (
+    16 => ZFS_ARC_MIN_MIB, # at least 64 MiB
+    1024 => 102,
+    4 * 1024 => 2048,
+    8 * 1024 => 4096,
+    150 * 1024 => 76800,
+    160 * 1024 => 81920,
+    1024 * 1024 => 524288,
+);
+
+mock_product('pve');
+while (my ($total_mem, $expected) = each %default_tests_pve) {
     $proxmox_install_runenv->redefine(
 	query_total_memory => sub { return $total_mem; },
     );
 
-    mock_product('pve');
     is(Proxmox::Install::RunEnv::default_zfs_arc_max(), $expected,
-	"$expected MiB should be zfs_arc_max for PVE with $total_mem MiB system memory");
+	"zfs_arc_max should default to $expected for pve with $total_mem MiB system memory");
+}
 
-    mock_product('pbs');
-    is(Proxmox::Install::RunEnv::default_zfs_arc_max(), $total_mem < 2048 ? $expected : 0,
-	"zfs_arc_max should default to `0` for PBS with $total_mem MiB system memory");
+while (my ($total_mem, $expected) = each %default_tests_others) {
+    $proxmox_install_runenv->redefine(
+	query_total_memory => sub { return $total_mem; },
+    );
 
-    mock_product('pmg');
-    is(Proxmox::Install::RunEnv::default_zfs_arc_max(), $total_mem < 4096 ? $expected : 0,
-	"zfs_arc_max should default to `0` for PMG with $total_mem MiB system memory");
-
-    mock_product('pdm');
-    is(Proxmox::Install::RunEnv::default_zfs_arc_max(), $total_mem < 2048 ? $expected : 0,
-	"zfs_arc_max should default to `0` for PDM with $total_mem MiB system memory");
+    foreach my $product ('pbs', 'pmg', 'pdm') {
+	mock_product($product);
+	is(Proxmox::Install::RunEnv::default_zfs_arc_max(), $expected,
+	    "zfs_arc_max should default to $expected for $product with $total_mem MiB system memory");
+    }
 }
 
 my @clamp_tests = (
-- 
2.48.1



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH installer 2/3] tui: bootdisk: always return proper value for default zfs max arc size

[Christoph Heiss]

In preparation for fixing #6285 [0].

`0` means to just skip writing the module parameter. But (especially)
with the upcoming change in ZFS 2.3 - which makes the size basically
that of the system memory minus 1 GiB - we want to always write some
value.

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6285

Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
---
 proxmox-tui-installer/src/views/bootdisk.rs | 32 ++++++++++++---------
 1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/proxmox-tui-installer/src/views/bootdisk.rs b/proxmox-tui-installer/src/views/bootdisk.rs
index b0fc131..313a3c9 100644
--- a/proxmox-tui-installer/src/views/bootdisk.rs
+++ b/proxmox-tui-installer/src/views/bootdisk.rs
@@ -626,26 +626,27 @@ impl ViewWrapper for BtrfsBootdiskOptionsView {
 
 struct ZfsBootdiskOptionsView {
     view: MultiDiskOptionsView<FormView>,
+    zfs_arc_max_default: usize,
 }
 
 impl ZfsBootdiskOptionsView {
     // TODO: Re-apply previous disk selection from `options` correctly
     fn new(runinfo: &RuntimeInfo, options: &ZfsBootdiskOptions) -> Self {
+        let zfs_arc_max_default = runinfo.total_memory.div_ceil(2);
+
         let arc_max_view = {
             // Always leave a GiB of headroom for the OS.
             let view =
                 IntegerEditView::new_with_suffix("MiB").max_value(runinfo.total_memory - 1024);
 
-            // If the runtime environment provides a non-zero value, that is
-            // also not the built-in ZFS default of half the system memory, use
-            // that as default.
-            // Otherwise, just place the ZFS default into the placeholder.
-            if runinfo.default_zfs_arc_max > 0
-                && runinfo.default_zfs_arc_max != runinfo.total_memory / 2
-            {
+            // If the runtime environment provides a value that is not the (ZFS) default of
+            // half the system memory, use that as default.
+            // Otherwise, just place the (ZFS) default of 50% of available system memory into the
+            // placeholder.
+            if runinfo.default_zfs_arc_max != zfs_arc_max_default {
                 view.content(options.arc_max)
             } else {
-                view.placeholder(runinfo.total_memory / 2)
+                view.placeholder(zfs_arc_max_default)
             }
         };
 
@@ -687,7 +688,10 @@ impl ZfsBootdiskOptionsView {
                 "ZFS is not compatible with hardware RAID controllers, for details see the documentation."
             ).center());
 
-        Self { view }
+        Self {
+            view,
+            zfs_arc_max_default,
+        }
     }
 
     fn new_with_defaults(runinfo: &RuntimeInfo) -> Self {
@@ -706,13 +710,15 @@ impl ZfsBootdiskOptionsView {
 
         // If a value is set, return that and clamp it to at least [`ZFS_ARC_MIN_SIZE_MIB`].
         //
-        // Otherwise, if no value was set or an error occurred return `0`. The former simply means
-        // that the placeholder value is still there.
+        // Otherwise, if no value was set or an error occurred return the default value. The former
+        // simply means that the placeholder value is still there.
         let arc_max = view
             .get_child::<IntegerEditView>(4)?
             .get_content_maybe()
-            .map_or(Ok(0), |v| v.map(|v| v.max(ZFS_ARC_MIN_SIZE_MIB)))
-            .unwrap_or(0);
+            .map_or(Ok(self.zfs_arc_max_default), |v| {
+                v.map(|v| v.max(ZFS_ARC_MIN_SIZE_MIB))
+            })
+            .unwrap_or(self.zfs_arc_max_default);
 
         Some((
             disks,
-- 
2.48.1



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH installer 3/3] fix #6285: install: always set up zfs modprobe configuration

[Christoph Heiss]

Fixes #6285 [0].

Came up a few times now in the forum (most recently [0], german) and is
a potential source of confusion for users, if the file does not exist on
(new) installations.

It makes indeed sense to just unconditionally write to
/etc/modprobe.d/zfs.conf. Often users create a separate ZFS pool after
installation, on separate disks, where it still makes sense to have a
more sensible zfs_arc_max default, at least on PVE.

It has been also exposed for all products, with sensible defaults each,
for some time now [1][2].

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6285
[1] https://forum.proxmox.com/threads/ram-auslastung-nimmt-%C3%BCber-die-zeit-drastisch-zu.164527/#post-760989
[2] https://git.proxmox.com/?p=pve-installer.git;a=commitdiff;h=a42a9db20976fbc0abb35d53416cee926b6efafe
[3] https://git.proxmox.com/?p=pve-installer.git;a=commitdiff;h=8772ebc35a7f43b97f5433c4d328ea784eaf902c

Signed-off-by: Christoph Heiss <c.heiss@proxmox.com>
---
 Proxmox/Install.pm | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/Proxmox/Install.pm b/Proxmox/Install.pm
index 57fd899..2e9b131 100644
--- a/Proxmox/Install.pm
+++ b/Proxmox/Install.pm
@@ -340,9 +340,7 @@ my sub zfs_setup_module_conf {
     my $arc_max_mib = Proxmox::Install::Config::get_zfs_opt('arc_max');
     my $arc_max = Proxmox::Install::RunEnv::clamp_zfs_arc_max($arc_max_mib) * 1024 * 1024;
 
-    if ($arc_max > 0) {
-	file_write_all("$targetdir/etc/modprobe.d/zfs.conf", "options zfs zfs_arc_max=$arc_max\n")
-    }
+    file_write_all("$targetdir/etc/modprobe.d/zfs.conf", "options zfs zfs_arc_max=$arc_max\n");
 }
 
 sub get_btrfs_raid_setup {
@@ -1329,10 +1327,13 @@ _EOD
 	    file_write_all("$targetdir/etc/default/grub.d/zfs.cfg", $zfs_snippet);
 
 	    file_write_all("$targetdir/etc/kernel/cmdline", "root=ZFS=$zfs_pool_name/ROOT/$zfs_root_volume_name boot=zfs $target_cmdline\n");
-
-	    zfs_setup_module_conf($targetdir);
 	}
 
+	# Always write zfs module parameter - even if the user did not select ZFS-on-root.
+	# It still makes sense to provide a sensible default for zfs_arc_max, in case a
+	# separate zfs pool is created afterwards.
+	zfs_setup_module_conf($targetdir);
+
 	diversion_remove($targetdir, "/usr/sbin/update-grub");
 	diversion_remove($targetdir, "/usr/sbin/update-initramfs");
 
-- 
2.48.1



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied-series: [PATCH installer 0/3] fix #6285: always set up zfs modprobe configuration

[Thomas Lamprecht]

Am 04.04.25 um 14:46 schrieb Christoph Heiss:
> Fixes #6285 [0].
> 
> This also came up quite a few times already in the forum, most recently
> [1] (german).
> 
> Since we do expose the option in the UI for all products nowadays (on
> ZFS-on-root installations only tho), just always write to
> /etc/modprobe.d/zfs.conf.
> 
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=6285
> [1] https://forum.proxmox.com/threads/ram-auslastung-nimmt-%C3%BCber-die-zeit-drastisch-zu.164527/#post-760989
> 
> Testing
> =======
> 
> Tested for each product PVE, PBS, PMG the following
>   - install on ZFS, not touching the ARC size setting
>   - install on ZFS, set some explicit value for ARC size
>   - installing on non-ZFS such as ext4
> 
> .. and checking afterwards that /etc/modprobe.d/zfs.conf was written and
> contained respectively
>   - the default value for the product
>   - the set value
>   - again the default value for the product
> 
> Also tested each case with the auto-installer for PVE and PMG.
> 
> Diffstat
> ========
> 
> Christoph Heiss (3):
>   run env: always return proper value for default zfs max arc size
>   tui: bootdisk: always return proper value for default zfs max arc size
>   fix #6285: install: always set up zfs modprobe configuration
> 
>  Proxmox/Install.pm                          | 11 +++---
>  Proxmox/Install/RunEnv.pm                   |  9 +++--
>  proxmox-tui-installer/src/views/bootdisk.rs | 32 +++++++++-------
>  test/zfs-arc-max.pl                         | 42 +++++++++++++--------
>  4 files changed, 57 insertions(+), 37 deletions(-)
> 


applied series, thanks for the quick implementation!


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel