From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 4AC67B928 for ; Mon, 3 Jul 2023 08:40:28 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 2DCFB186A4 for ; Mon, 3 Jul 2023 08:39:58 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 3 Jul 2023 08:39:57 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id A142E4361B for ; Mon, 3 Jul 2023 08:39:56 +0200 (CEST) From: Dominik Csapak To: pve-devel@lists.proxmox.com Date: Mon, 3 Jul 2023 08:39:55 +0200 Message-Id: <20230703063955.575427-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.016 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pve-devel] [PATCH http-server] make daemons compatible with installed AnyEvent::AIO X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jul 2023 06:40:28 -0000 when installing AnyEvent::AIO (by the package libanyevent-aio-perl), the worker forks of our daemons using AnyEvent would consume 100% cpu cycles while trying to do an epoll_wait which no one read from. It was not really clear which part of the code set that fd up. Reading the documentation of the related perl modules, it became clear that the issue was with AnyEvent::IO. By default this uses AnyEvent::AIO (if installed) which in turn uses IO::AIO which explicitly says it uses pthreads and is not really fork compatible (which we rely heavy upon). It seems that IO::AIO sets up some fds with epoll in the END handler of it's library (or earlier, but sends data to it in the END handler), so that when using 'exit' instead of 'POSIX::_exit' (which we do in PVE::Daemon) creates the observed behavior. Interestingly we did not use any of AnyEvent::IO's functionality, so we can safely remove it. Even if we would have used it in the past, without AnyEvent::AIO the IO would not have been async anyway (the pure perl impl doesn't do async IO). My best guess is that we wanted to use it, but noticed that we can't, and forgot to remove the use statement. (This is indicated by a comment that says aio_load is not async unless IO::AIO is used) This only occurs now, since bookworm is the first debian release to package the library. if we ever wanted to use AnyEvent::AIO, there are probably two other ways that could fix it: * replace our 'exit()' calls with 'POSIX::_exit()', which seems to fix it, but other side effects are currently unknown * use 'IO::AIO::reinit()' after forking, which also seems to fix it, but perldoc says it 'is not an operation supported by any standards, but happens to work on GNU/LINUX and some newer BSD systems' With this fix, one can safely install 'libanyevent-aio-perl' and 'libperl-languageserver-perl' (the only user of it AFAICS) on a PVE or PMG system. Signed-off-by: Dominik Csapak --- maybe we should leave the use statement in and only comment it out, with a note not to use this? src/PVE/APIServer/AnyEvent.pm | 1 - 1 file changed, 1 deletion(-) diff --git a/src/PVE/APIServer/AnyEvent.pm b/src/PVE/APIServer/AnyEvent.pm index 1fd7a74..8fb1a7a 100644 --- a/src/PVE/APIServer/AnyEvent.pm +++ b/src/PVE/APIServer/AnyEvent.pm @@ -12,7 +12,6 @@ use warnings; use AnyEvent::HTTP; use AnyEvent::Handle; -use AnyEvent::IO; use AnyEvent::Socket; # use AnyEvent::Strict; # only use this for debugging use AnyEvent::TLS; -- 2.30.2