From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <d.csapak@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id F371EC050C for <pmg-devel@lists.proxmox.com>; Wed, 10 Jan 2024 15:13:53 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id CD9BB3548C for <pmg-devel@lists.proxmox.com>; Wed, 10 Jan 2024 15:13:23 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for <pmg-devel@lists.proxmox.com>; Wed, 10 Jan 2024 15:13:23 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 072A54909C; Wed, 10 Jan 2024 15:13:23 +0100 (CET) Message-ID: <e191921c-e6df-4929-96ba-00e8bd43fd6d@proxmox.com> Date: Wed, 10 Jan 2024 15:13:22 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta Content-Language: en-US To: Dietmar Maurer <dietmar@proxmox.com>, Markus Frank <m.frank@proxmox.com>, pmg-devel@lists.proxmox.com References: <20240110115654.763820-1-m.frank@proxmox.com> <477134155.2147.1704891166704@webmail.proxmox.com> <411ba218-cfea-4075-bfb0-bee51cbc6051@proxmox.com> <1362643070.2197.1704895037948@webmail.proxmox.com> From: Dominik Csapak <d.csapak@proxmox.com> In-Reply-To: <1362643070.2197.1704895037948@webmail.proxmox.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.020 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pmg-devel] [PATCH pmg-api] config: adjust max_filters calculation to reflect current memory usage X-BeenThere: pmg-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Mail Gateway development discussion <pmg-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pmg-devel>, <mailto:pmg-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pmg-devel/> List-Post: <mailto:pmg-devel@lists.proxmox.com> List-Help: <mailto:pmg-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel>, <mailto:pmg-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Wed, 10 Jan 2024 14:13:54 -0000 On 1/10/24 14:57, Dietmar Maurer wrote: >> i think it's easier than you think to blow up the filter memory usage, we're already > > I am sure it is easy. But it is also wrong, as there are more efficient ways to implement blocklists. > > Anyway, the suggested patch limits the number of parallel filters, and can thus severely limit mail throughput. sure, but getting the filter oom killed (like the user in the forum did) also limits mail throughput e.g. if you have 8 GiB of physical memory, the resulting max_workers is currently (8192-512)/120 => 64 which results in a maximum of 40 processes (we cap it there) but 40 processes at 220 MiB are already over 8 GiB realistically the server can only handle 8GiB - 2.5Gib (clamav+remaining pmg daemond + postfix, etc, probably more with avast) = 5.5Gib / 220 MiB ~ 25 workers before something gets oom killed using an estimated value of 300MiB results in a max_worker of 25, which would fit right in that scenario, and for systems with more memory it looks even better (as we approach the max 40 workers) one could argue that we could improve the memory calculation by subtracting more base memory (e.g. 2GiB) before calculating, but that also is a rather moving target (and i guess more stable than the pmg-smtp-filter daemons? though not so sure about that)