Re: [pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[Dominik Csapak <d.csapak@proxmox.com>]

On 1/10/24 14:57, Dietmar Maurer wrote:
>> i think it's easier than you think to blow up the filter memory usage, we're already
> 
> I am sure it is easy. But it is also wrong, as there are more efficient ways to implement blocklists.
> 
> Anyway, the suggested patch limits the number of parallel filters, and can thus severely limit mail throughput.

sure, but getting the filter oom killed (like the user in the forum did) also limits mail throughput

e.g. if you have 8 GiB of physical memory, the resulting max_workers is currently

(8192-512)/120 => 64 which results in a maximum of 40 processes (we cap it there)

but 40 processes at 220 MiB are already over 8 GiB

realistically the server can only handle

8GiB - 2.5Gib (clamav+remaining pmg daemond + postfix, etc, probably more with avast)
  = 5.5Gib / 220 MiB ~ 25 workers before something gets oom killed

using an estimated value of 300MiB results in a max_worker of 25, which would fit right
in that scenario, and for systems with more memory it looks even better
(as we approach the max 40 workers)

one could argue that we could improve the memory calculation by subtracting more base memory
(e.g. 2GiB) before calculating, but that also is a rather moving target (and i guess more
stable than the pmg-smtp-filter daemons? though not so sure about that)