Re: [pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

Am 10/01/2024 um 12:56 schrieb Markus Frank:
> One pmg-smtp-filter process uses at least 220 MiB.
> When having 100000 rules one process can take up to 330 MiB.

That's probably talking about RSS here, or? That would be rather useless as
it re-counts the memory used by shared libraries, which bloats the number,
as they're actually only loaded once in memory.

What is the newer PSS (Proportional Set Size) metric in that case?  As that
would be a better metric due to actually accounting for the proportional use
of shared libraries.

For example, use the following bash one-liner to get both, PSS and RSS of
each pmg-smtp-filter processes:

 for pid in $(pidof pmg-smtp-filter); do printf "PID %s: " $pid; awk '/Pss:/{ pss += $2 } /Rss:/{ rss += $2 } END { print "PSS =", pss, " RSS =", rss }' "/proc/$pid/smaps"; done

Here, on a pretty much idle setup with only a handful of rules, I get:

PID 405810: PSS = 84700  RSS = 225000
PID 405809: PSS = 84714  RSS = 225000

As PSS is what matters here, the 84.7 MB are is quite in line with the 120 MB
$servermem, at least for my (underused) setup.

In the get_max_filters calculation I'd rather look at the accounting for the
system baseline memory usage through subtracting 512 MB, as that is rather
way to low nowadays, a clamav alone takes up 1.2 - 1.5 GB.

Maybe turn that up first depending on physical_memory, i.e., for < 2 GB I'd
keep it as is, otherwise deduct something like 1.5 GB.

Then check the actual memory growth per added filter process via checking
the PSS sizes, if huge setups then use 200 MB we could increase that a bit,
but it probably won't need to be the 300 MB of your patch. And we always can
make $servermem dependent of available memory size too, e.g., assume bigger
rule sets due to bigger resources available, like > 4 GB (or 8 GB) memory.

i.e., having a three-branch if here to cover the cases for
- "low-memory but might work for small setups"
- "ok'ish memory but needs some special tuning"
- "more than the minimal recommended amount of memory"

if ($memory < 2000) {
    warn "low amount of system memory installed, minimum requirement is 2 GB, recommended is 4+ GB\n".
    $base = $memory > 1536 ? 1024 : 512;
    $servermem = 120;
} elsif ($memory < 4096) {
    $base = 1500;
    $servermem = 150;
} else {
    $base = 2500;
    $servermem = 200;
}

The $base and $servermem values are just guesstimated without to much
thought and can surely be better chosen (PSS size of huge setups would be
required for that).

A complete different alternative:
rewrite the filter in rust and decimate the memory usage making it actual
more performant and allowing really higher throughput.