[pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[Markus Frank]

One pmg-smtp-filter process uses at least 220 MiB.
When having 100000 rules one process can take up to 330 MiB.
Therefore using 300 for the calculation more closely
reflects the current memory usage of one filter process.

Related OOM killer problem found in forum:
https://forum.proxmox.com/threads/123531/

Signed-off-by: Markus Frank <m.frank@proxmox.com>
---
 src/PMG/Config.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm
index 7339e0d..90e39bb 100644
--- a/src/PMG/Config.pm
+++ b/src/PMG/Config.pm
@@ -458,7 +458,7 @@ sub get_max_filters {
     # estimate optimal number of filter servers
 
     my $max_servers = 5;
-    my $servermem = 120;
+    my $servermem = 300;
     my $memory = physical_memory();
     my $add_servers = int(($memory - 512)/$servermem);
     $max_servers += $add_servers if $add_servers > 0;
-- 
2.39.2

Re: [pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[Dietmar Maurer]

> One pmg-smtp-filter process uses at least 220 MiB.
> When having 100000 rules one process can take up to 330 MiB.

I guess almost nobody has that many rules! So IMHO, with this patch we
are now too conservative for 99% of all installations.

Besides, the rule system is not meant to be used as large block-list. People should set-up a RBL server instead, which is far more effective.

- Dietmar

Re: [pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[Markus Frank]

On  2024-01-10 13:52, Dietmar Maurer wrote:
>> One pmg-smtp-filter process uses at least 220 MiB.
>> When having 100000 rules one process can take up to 330 MiB.
> 
> I guess almost nobody has that many rules! So IMHO, with this patch we
> are now too conservative for 99% of all installations.
The person in the forum did somehow use 500 MiB per process.

> 
> Besides, the rule system is not meant to be used as large block-list. People should set-up a RBL server instead, which is far more effective.
> 
> - Dietmar
I see, how about 230?

Re: [pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[Dominik Csapak]

On 1/10/24 13:52, Dietmar Maurer wrote:
>> One pmg-smtp-filter process uses at least 220 MiB.
>> When having 100000 rules one process can take up to 330 MiB.
> 
> I guess almost nobody has that many rules! So IMHO, with this patch we
> are now too conservative for 99% of all installations.

i think what markus meant is that he had 100000 entries in a who object,
not rules. That is (sadly) rather realistic and we encounter it often with users/customers.
(even if we don't recommend it)

i think it's easier than you think to blow up the filter memory usage, we're already
100 MiB up from the initial assumption without adding any new rules/objects by default.
(e.g. a new spamassassin update can increase that probably quite easily)

while 300 might be excessive, on our machines we easily observe 230 MiB per worker
so that could be a fine value (like Markus suggested)

> 
> Besides, the rule system is not meant to be used as large block-list. People should set-up a RBL server instead, which is far more effective.
> 
> - Dietmar
> 
> 
> _______________________________________________
> pmg-devel mailing list
> pmg-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel
> 
> 

Re: [pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[Dominik Csapak]

On 1/10/24 14:38, Dominik Csapak wrote:
> On 1/10/24 13:52, Dietmar Maurer wrote:
>>> One pmg-smtp-filter process uses at least 220 MiB.
>>> When having 100000 rules one process can take up to 330 MiB.
>>
>> I guess almost nobody has that many rules! So IMHO, with this patch we
>> are now too conservative for 99% of all installations.
> 
> i think what markus meant is that he had 100000 entries in a who object,
> not rules. That is (sadly) rather realistic and we encounter it often with users/customers.
> (even if we don't recommend it)

after thinking a bit, 100.000 might really be on the large side, but i saw
multiple thousand entries quite often

> 
> i think it's easier than you think to blow up the filter memory usage, we're already
> 100 MiB up from the initial assumption without adding any new rules/objects by default.
> (e.g. a new spamassassin update can increase that probably quite easily)
> 
> while 300 might be excessive, on our machines we easily observe 230 MiB per worker
> so that could be a fine value (like Markus suggested)
> 
>>
>> Besides, the rule system is not meant to be used as large block-list. People should set-up a RBL 
>> server instead, which is far more effective.
>>
>> - Dietmar
>>
>>
>> _______________________________________________
>> pmg-devel mailing list
>> pmg-devel@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel
>>
>>
> 
> 
> 
> _______________________________________________
> pmg-devel mailing list
> pmg-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel
> 
> 

Re: [pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[Dietmar Maurer]

> i think it's easier than you think to blow up the filter memory usage, we're already

I am sure it is easy. But it is also wrong, as there are more efficient ways to implement blocklists.

Anyway, the suggested patch limits the number of parallel filters, and can thus severely limit mail throughput.

Re: [pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[Dominik Csapak]

On 1/10/24 14:57, Dietmar Maurer wrote:
>> i think it's easier than you think to blow up the filter memory usage, we're already
> 
> I am sure it is easy. But it is also wrong, as there are more efficient ways to implement blocklists.
> 
> Anyway, the suggested patch limits the number of parallel filters, and can thus severely limit mail throughput.

sure, but getting the filter oom killed (like the user in the forum did) also limits mail throughput

e.g. if you have 8 GiB of physical memory, the resulting max_workers is currently

(8192-512)/120 => 64 which results in a maximum of 40 processes (we cap it there)

but 40 processes at 220 MiB are already over 8 GiB

realistically the server can only handle

8GiB - 2.5Gib (clamav+remaining pmg daemond + postfix, etc, probably more with avast)
  = 5.5Gib / 220 MiB ~ 25 workers before something gets oom killed

using an estimated value of 300MiB results in a max_worker of 25, which would fit right
in that scenario, and for systems with more memory it looks even better
(as we approach the max 40 workers)

one could argue that we could improve the memory calculation by subtracting more base memory
(e.g. 2GiB) before calculating, but that also is a rather moving target (and i guess more
stable than the pmg-smtp-filter daemons? though not so sure about that)

Re: [pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[Dietmar Maurer]

> sure, but getting the filter oom killed (like the user in the forum did) also limits mail throughput

Just keep in mind that low-memory systems now runs with less parallel filters. If users reports performance/throughput problem we have to suggest to add more RAM...

Note: The code was written at a time when RAM <= 2GB

Re: [pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[Thomas Lamprecht]

Am 10/01/2024 um 12:56 schrieb Markus Frank:
> One pmg-smtp-filter process uses at least 220 MiB.
> When having 100000 rules one process can take up to 330 MiB.

That's probably talking about RSS here, or? That would be rather useless as
it re-counts the memory used by shared libraries, which bloats the number,
as they're actually only loaded once in memory.

What is the newer PSS (Proportional Set Size) metric in that case?  As that
would be a better metric due to actually accounting for the proportional use
of shared libraries.

For example, use the following bash one-liner to get both, PSS and RSS of
each pmg-smtp-filter processes:

 for pid in $(pidof pmg-smtp-filter); do printf "PID %s: " $pid; awk '/Pss:/{ pss += $2 } /Rss:/{ rss += $2 } END { print "PSS =", pss, " RSS =", rss }' "/proc/$pid/smaps"; done

Here, on a pretty much idle setup with only a handful of rules, I get:

PID 405810: PSS = 84700  RSS = 225000
PID 405809: PSS = 84714  RSS = 225000

As PSS is what matters here, the 84.7 MB are is quite in line with the 120 MB
$servermem, at least for my (underused) setup.

In the get_max_filters calculation I'd rather look at the accounting for the
system baseline memory usage through subtracting 512 MB, as that is rather
way to low nowadays, a clamav alone takes up 1.2 - 1.5 GB.

Maybe turn that up first depending on physical_memory, i.e., for < 2 GB I'd
keep it as is, otherwise deduct something like 1.5 GB.

Then check the actual memory growth per added filter process via checking
the PSS sizes, if huge setups then use 200 MB we could increase that a bit,
but it probably won't need to be the 300 MB of your patch. And we always can
make $servermem dependent of available memory size too, e.g., assume bigger
rule sets due to bigger resources available, like > 4 GB (or 8 GB) memory.

i.e., having a three-branch if here to cover the cases for
- "low-memory but might work for small setups"
- "ok'ish memory but needs some special tuning"
- "more than the minimal recommended amount of memory"

if ($memory < 2000) {
    warn "low amount of system memory installed, minimum requirement is 2 GB, recommended is 4+ GB\n".
    $base = $memory > 1536 ? 1024 : 512;
    $servermem = 120;
} elsif ($memory < 4096) {
    $base = 1500;
    $servermem = 150;
} else {
    $base = 2500;
    $servermem = 200;
}

The $base and $servermem values are just guesstimated without to much
thought and can surely be better chosen (PSS size of huge setups would be
required for that).

A complete different alternative:
rewrite the filter in rust and decimate the memory usage making it actual
more performant and allowing really higher throughput.

Re: [pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[Stoiko Ivanov]

hi,

a bit late to the discussion on-list - had a few chats with Dominik
off-list and did some minimal testing:

On Fri, 12 Jan 2024 11:13:57 +0100
Thomas Lamprecht <t.lamprecht@proxmox.com> wrote:

> Am 10/01/2024 um 12:56 schrieb Markus Frank:
> > One pmg-smtp-filter process uses at least 220 MiB.
> > When having 100000 rules one process can take up to 330 MiB.  
> 
> That's probably talking about RSS here, or? That would be rather useless as
> it re-counts the memory used by shared libraries, which bloats the number,
> as they're actually only loaded once in memory.
> 
> What is the newer PSS (Proportional Set Size) metric in that case?  As that
> would be a better metric due to actually accounting for the proportional use
> of shared libraries.
> 
> For example, use the following bash one-liner to get both, PSS and RSS of
> each pmg-smtp-filter processes:
> 
>  for pid in $(pidof pmg-smtp-filter); do printf "PID %s: " $pid; awk '/Pss:/{ pss += $2 } /Rss:/{ rss += $2 } END { print "PSS =", pss, " RSS =", rss }' "/proc/$pid/smaps"; done

TIL: PSS/USS - thanks! (also for the one-liner :)

> 
> Here, on a pretty much idle setup with only a handful of rules, I get:
> 
> PID 405810: PSS = 84700  RSS = 225000
> PID 405809: PSS = 84714  RSS = 225000


for comparison - 2 more loaded productive instances (although also with a
quite small ruleset) I have access to:
PID 2908376: PSS = 114567  RSS = 227528
PID 2908355: PSS = 115023  RSS = 227776
and
PID 788678: PSS = 65242  RSS = 217564
PID 788600: PSS = 69126  RSS = 220656
PID 788507: PSS = 102765  RSS = 227596

> 
> As PSS is what matters here, the 84.7 MB are is quite in line with the 120 MB
> $servermem, at least for my (underused) setup.
I tried to get a few more filter processes to spawn (while `watch -n2` the
one-liner from above), by:
* stopping postfix on the sending system
* queueing 250 (also 1250) tiny mails to one recipient there
* starting postfix (so it starts to drain the queue at once)

to get more than 2-3 filter processes I had to either add a sleep to the
processing (e.g. at analyze_virus_clam), or send a larger mail (with a pdf
attachment that gets handed through pdf2text).

With that (and a small ruleset, without huge numbers of objects) the
PSS-size was between:
31967 and 62725 (for 20 parallel processes, although sometimes some pids did not yield
one or the other metric - probably while being started/torn-down)
numbers were similar for both types of mail (small text/plain only, and
with a ~156k pdf attachment).

> 
> In the get_max_filters calculation I'd rather look at the accounting for the
> system baseline memory usage through subtracting 512 MB, as that is rather
> way to low nowadays, a clamav alone takes up 1.2 - 1.5 GB.
Would also be my first go-to for adaptation (based on guess-work and
watching top/htop output of a few PMG instances in our support-channels)

I think  I can't remember any other (than the one that lead to this
patch) case of OOM-killed pmg-smtp-filter processes (for any system with >
2Gb memory) 
Neither do I recall recommending someone to manually tweak the
max_filter setting - so while adaptation might make sense - I think even
the current code works well in practice.

We do not need to cover all possible scenarios with it - as if someone
adds tons of signatures to clamAV (as described in the thread from the
commit-message), or avast, or many SpamAssassin plugins - they always have
the option to limit the filter-processes in the config).

> 
> Maybe turn that up first depending on physical_memory, i.e., for < 2 GB I'd
> keep it as is, otherwise deduct something like 1.5 GB.
> 
> Then check the actual memory growth per added filter process via checking
> the PSS sizes, if huge setups then use 200 MB we could increase that a bit,
> but it probably won't need to be the 300 MB of your patch. And we always can
> make $servermem dependent of available memory size too, e.g., assume bigger
> rule sets due to bigger resources available, like > 4 GB (or 8 GB) memory.
> 
> i.e., having a three-branch if here to cover the cases for
> - "low-memory but might work for small setups"
> - "ok'ish memory but needs some special tuning"
> - "more than the minimal recommended amount of memory"
> 
> if ($memory < 2000) {
>     warn "low amount of system memory installed, minimum requirement is 2 GB, recommended is 4+ GB\n".
>     $base = $memory > 1536 ? 1024 : 512;
>     $servermem = 120;
> } elsif ($memory < 4096) {
>     $base = 1500;
>     $servermem = 150;
> } else {
>     $base = 2500;
>     $servermem = 200;
> }
> 
> The $base and $servermem values are just guesstimated without to much
> thought and can surely be better chosen (PSS size of huge setups would be
> required for that).
> 
> A complete different alternative:
> rewrite the filter in rust and decimate the memory usage making it actual
> more performant and allowing really higher throughput.
For this I usually handwave around - pointing to SpamAssassin being the
largest memory-hog (and run-time and cpu) for the complete pmg-smtp-filter
- but I never actually checked if that's true - so I tried removing
SpamAssassin (very crudely by commenting the relevant imports and
replacing the actually checking with a no-op). The test with the small
mail described above yields:
PSS = 13829  RSS = 82420
so ~75-80% of memory of each pmg-smtp-filter seems used by SpamAssassin.

Not that there's not quite a bit of room for improvement in the code-base,
I'd probably first look at pmgpolicy for rewriting first (simply because of
SpamAssassin, and my doubt, that running SA in another way (spamd[0],
writing our own spamd in rust) will make it significantly faster/smaller).

summing up -  the warn for low memory Thomas suggested sounds like a good
idea, I'd probably only stick with 2 branches (<=2.5GB and >2.5GB) - but
fine either way

my tests with small ruleset, but many filter-processes put the
memory-requirement (PSS-based) at <70MiB/filter-process