[pmg-devel] [PATCH pmg-api v3] config: adjust max_filters calculation to reflect current memory usage

[pmg-devel] [PATCH pmg-api v3] config: adjust max_filters calculation to reflect current memory usage

[Markus Frank]

Change max_filters calculation for systems with recommended memory
setup (>4GB).

The values of 2816 and 150 are based on testing with 4GB, 6GB & 8GB
memory configurations, large and small numbers of added objects and
sending multiple mails simultaneously.

On setups with less memory, it is difficult to completely prevent OOM kills.
So for these setups the calculation remains similar, but a warning is sent.

Related OOM killer problem found in forum:
https://forum.proxmox.com/threads/123531/

Signed-off-by: Markus Frank <m.frank@proxmox.com>
---
Tested edge cases with setting min_servers to max_servers
in src/bin/pmg-smtp-filter

 src/PMG/Config.pm | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm
index 7339e0d..061396e 100644
--- a/src/PMG/Config.pm
+++ b/src/PMG/Config.pm
@@ -459,8 +459,17 @@ sub get_max_filters {
 
     my $max_servers = 5;
     my $servermem = 120;
+    my $base;
     my $memory = physical_memory();
-    my $add_servers = int(($memory - 512)/$servermem);
+    if ($memory < 3840) {
+	warn "low amount of system memory installed, recommended is 4+ GB\n"
+	    ."to prevent OOM kills, it is better to set max_filters manually\n";
+	$base = $memory > 1536 ? 1024 : 512;
+    } else {
+	$base = 2816;
+	$servermem = 150;
+    }
+    my $add_servers = int(($memory - $base)/$servermem);
     $max_servers += $add_servers if $add_servers > 0;
     $max_servers = 40 if  $max_servers > 40;
 
-- 
2.39.2

Re: [pmg-devel] [PATCH pmg-api v3] config: adjust max_filters calculation to reflect current memory usage

[Alexander Zeidler]

On Thu, 2024-01-18 at 15:55 +0100, Markus Frank wrote:

>      my $max_servers = 5;
>      my $servermem = 120;
> +    my $base;
>      my $memory = physical_memory();
> -    my $add_servers = int(($memory - 512)/$servermem);
> +    if ($memory < 3840) {
> +	warn "low amount of system memory installed, recommended is 4+ GB\n"
> +	    ."to prevent OOM kills, it is better to set max_filters manually\n";
> +	$base = $memory > 1536 ? 1024 : 512;
> +    } else {
> +	$base = 2816;
> +	$servermem = 150;
> +    }
> +    my $add_servers = int(($memory - $base)/$servermem);
>      $max_servers += $add_servers if $add_servers > 0;
>      $max_servers = 40 if  $max_servers > 40;
1. Perhaps we also want to increase the current max_filters limit of 40
minus 2?


2. With the recommended 4GB setup, the patch now only returns
max_filters = 10 instead of the previous 32 as marked in the table
below.

(table with temporarily removed 38-limit for the patch columns)

[1]     [2]    [3]  [4]   [5]
RAM *0.975=    yet *120  left     patch       left
0.5     499      3        139         3 *120   139
  1     998      7        158         7 *120   158
  2    1996     15        196        11 *120   676
  3    2995     23        235        19 *120   715

  4    3993     32        153    ==> 10 *150  2493
  8    7987     38       3427        37 *150  2437
 16   15974     38      11414        90 *150  2474
 32   31948     38      27388       197 *150  2398
 64   63897     38      59337       410 *150  2397

[1] installed
[2] ca. MemTotal according to /proc/meminfo, hence used for calculation
[3] max_filters returned by current code
[4] expected servermem used in code and table for calculation
[5] (RAM * 0.975) - (max_filters * servermem)

Re: [pmg-devel] [PATCH pmg-api v3] config: adjust max_filters calculation to reflect current memory usage

[Markus Frank]

Hello,

thanks for looking into this.

On  2024-02-19 19:04, Alexander Zeidler wrote:
> On Thu, 2024-01-18 at 15:55 +0100, Markus Frank wrote:
> 
>>       my $max_servers = 5;
>>       my $servermem = 120;
>> +    my $base;
>>       my $memory = physical_memory();
>> -    my $add_servers = int(($memory - 512)/$servermem);
>> +    if ($memory < 3840) {
>> +	warn "low amount of system memory installed, recommended is 4+ GB\n"
>> +	    ."to prevent OOM kills, it is better to set max_filters manually\n";
>> +	$base = $memory > 1536 ? 1024 : 512;
>> +    } else {
>> +	$base = 2816;
>> +	$servermem = 150;
>> +    }
>> +    my $add_servers = int(($memory - $base)/$servermem);
>>       $max_servers += $add_servers if $add_servers > 0;
>>       $max_servers = 40 if  $max_servers > 40;
> 1. Perhaps we also want to increase the current max_filters limit of 40
> minus 2?
What would be the improvement? 38 processes seem enough to me.
Could you maybe trigger this much process at the same time?
> 
> 
> 2. With the recommended 4GB setup, the patch now only returns
> max_filters = 10 instead of the previous 32 as marked in the table
> below.
Since 4GB is lowest recommended memory setup, 10 processes seem fine to me.
Also I do not think a 2-4 Core 4GB setup could reach more 10 processes at the same time.
I could only trigger 3-6 processes running simultaneously in my testing with a flood of emails.
> 
> (table with temporarily removed 38-limit for the patch columns)
> 
> [1]     [2]    [3]  [4]   [5]
> RAM *0.975=    yet *120  left     patch       left
> 0.5     499      3        139         3 *120   139
>    1     998      7        158         7 *120   158
>    2    1996     15        196        11 *120   676
>    3    2995     23        235        19 *120   715
> 
>    4    3993     32        153    ==> 10 *150  2493
>    8    7987     38       3427        37 *150  2437
>   16   15974     38      11414        90 *150  2474
>   32   31948     38      27388       197 *150  2398
>   64   63897     38      59337       410 *150  2397
> 
Thanks for this calculation.
It shows that the remaining memory would be similar across
multiple memory configurations with this new calculation.

> [1] installed
> [2] ca. MemTotal according to /proc/meminfo, hence used for calculation
> [3] max_filters returned by current code
> [4] expected servermem used in code and table for calculation
> [5] (RAM * 0.975) - (max_filters * servermem)

[pmg-devel] applied: [PATCH pmg-api v3] config: adjust max_filters calculation to reflect current memory usage

[Thomas Lamprecht]

Am 18/01/2024 um 15:55 schrieb Markus Frank:
> Change max_filters calculation for systems with recommended memory
> setup (>4GB).
> 
> The values of 2816 and 150 are based on testing with 4GB, 6GB & 8GB
> memory configurations, large and small numbers of added objects and
> sending multiple mails simultaneously.
> 
> On setups with less memory, it is difficult to completely prevent OOM kills.
> So for these setups the calculation remains similar, but a warning is sent.
> 
> Related OOM killer problem found in forum:
> https://forum.proxmox.com/threads/123531/
> 
> Signed-off-by: Markus Frank <m.frank@proxmox.com>
> ---
> Tested edge cases with setting min_servers to max_servers
> in src/bin/pmg-smtp-filter
> 
>  src/PMG/Config.pm | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
>

applied, , thanks!

but I reworked the heuristic again a bit to avoid the downward glitch when
crossing the 3840 MiB boundary, old/new comparison of max_servers (the actual
count, i.e. get_max_filters() + 2):

 GiB     # max old        # max new
 0.5       5               5
 1.0       9               7
 1.5       13              8
 2.0       13              10
 2.5       17              11
 3.0       22              12
 3.5       26              13
 4.0       13              15
 4.5       16              18
 5.0       20              22
 5.5       23              25
 6.0       27              28
 6.5       30              32
 7.0       34              35
 7.5       37              39
 8.0       40              40

This should be a bit less confusing than the old one.