From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <s.ivanov@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 59C9DC0F47 for <pmg-devel@lists.proxmox.com>; Fri, 12 Jan 2024 21:00:13 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3AFB236372 for <pmg-devel@lists.proxmox.com>; Fri, 12 Jan 2024 20:59:43 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for <pmg-devel@lists.proxmox.com>; Fri, 12 Jan 2024 20:59:42 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id E48B6490C3 for <pmg-devel@lists.proxmox.com>; Fri, 12 Jan 2024 20:59:41 +0100 (CET) Date: Fri, 12 Jan 2024 20:59:40 +0100 From: Stoiko Ivanov <s.ivanov@proxmox.com> To: Dominik Csapak <d.csapak@proxmox.com> Cc: pmg-devel@lists.proxmox.com Message-ID: <20240112205940.71d776f5@rosa.proxmox.com> In-Reply-To: <eda519de-d269-4e94-9ba6-0ab4ca7567ed@proxmox.com> References: <20230911142317.19746-1-s.ivanov@proxmox.com> <eda519de-d269-4e94-9ba6-0ab4ca7567ed@proxmox.com> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.38; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.087 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pmg-devel] [PATCH pmg-api/docs] make filter timeout configurable X-BeenThere: pmg-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Mail Gateway development discussion <pmg-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pmg-devel>, <mailto:pmg-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pmg-devel/> List-Post: <mailto:pmg-devel@lists.proxmox.com> List-Help: <mailto:pmg-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel>, <mailto:pmg-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Fri, 12 Jan 2024 20:00:13 -0000 Thank you very much for the review, testing and the discussion! Just summing up some points from the talks: On Fri, 12 Jan 2024 09:35:09 +0100 Dominik Csapak <d.csapak@proxmox.com> wrote: > gave this a look & spin, and it works as intended > > writing here what we discussed off-list: > > there is still the same race when processing + the actual action > takes longer than the time out, but that is no trivially fixable One option to handle this would be to 'dry-run' the (final) actions for each target-group, assuming that they work, i.e.: that the mail is considered 'delivered' for accept and quarantine, and 'blocked' for block. with that information pmg-smtp-filter could answer to the sending postfix, and then start processing the actions. It would only need to generate a bounce if an accepted mail is not successfully sent to the postfix on 10025. But this entails refactoring PMG::SMTP and pmg-smtp-filter - pmg-smtp-filter would need to return after processing who,what,when and then SMTP would need to call a second function which deals with running the actions, writing statistics, ...) > > for "normal" timeouts (e.g. the default of 600s) this should not > matter much, except for pathological cases where e.g. writing > to the quarantine takes an absurd amount of time In case a quarantine insert takes so long (and the processing of a mail took 599s before) - the issue of getting a mail multiple times is probably not your largest problem. > > secondly, we probably should adapt the timeouts for virus+custom check scripts > to the configured one too (or e.g. half) but that can be done afterwards Thinking about it - I think min($filter_timeout, 5*60) (5 minutes is the current timeout for virus+custom-checks, might make for a robust change. splitting the timeout seems odd (just because clamAV needs more than half the timeout, does not mean that custom checks + avast (if configured at all) or spamassassin won't finish in a very short time. as an alternative (which I'd only consider if someone actually runs into such an issue) - we might introduce timeouts for each of the potentially long-running things. > and does not impact the actual issue here > (except that probably the command runs unnecessarily long) > > IMHO we should still mark the increased timeout for before queue filter > in the next release notes, since that can be a bit unexpected Good point - definitely worth mentioning > > so from my side, this series is: > > Reviewed-by: Dominik Csapak <d.csapak@proxmox.com> > Tested-by: Dominik Csapak <d.csapak@proxmox.com> >