From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id C2DD068E8B for ; Fri, 4 Dec 2020 18:56:50 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B275AD390 for ; Fri, 4 Dec 2020 18:56:50 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 4FF0FD297 for ; Fri, 4 Dec 2020 18:56:49 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 1BBB144D25 for ; Fri, 4 Dec 2020 18:56:49 +0100 (CET) From: Stoiko Ivanov To: pve-devel@lists.proxmox.com Date: Fri, 4 Dec 2020 18:56:26 +0100 Message-Id: <20201204175629.30116-3-s.ivanov@proxmox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20201204175629.30116-1-s.ivanov@proxmox.com> References: <20201204175629.30116-1-s.ivanov@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.080 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [anyevent.pm, proxmox.com] Subject: [pve-devel] [PATCH http-server v2 2/5] accept-phase: fix conn_count "leak" X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Dec 2020 17:56:50 -0000 When handling new connections in 'accept_connections' the number of active connections got increased before the AnyEvent::Handle registered the callback which would decrement it on error/eof. Any error/die beforehand would skip the decrement, and leave the process in an endless loop upon exiting in wait_end_loop. This can happen e.g. when the call to getpeername fails, or if the connection is denied by the ALLOW_FROM/DENY_FROM settings in '/etc/default/pveproxy' (which is also a simple reproducer for that). Additionally it can cause a denial of service, by attempting to connect from a denied ip until the connection count exeeds the maximum connections of all child-processes. Should the connection count become negative a warning is logged in both places where we decrement it, in case of a failed AnyEvent::Handle->new(), the count is not decremented if this would happen. Reported via our community-forum: https://forum.proxmox.com/threads/pveproxy-eats-available-ram.79617/ Co-Authored-by: Dominik Csapak Signed-off-by: Stoiko Ivanov --- PVE/APIServer/AnyEvent.pm | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/PVE/APIServer/AnyEvent.pm b/PVE/APIServer/AnyEvent.pm index 7916bdd..af2fde8 100644 --- a/PVE/APIServer/AnyEvent.pm +++ b/PVE/APIServer/AnyEvent.pm @@ -157,6 +157,11 @@ sub client_do_disconnect { &$shutdown_hdl($hdl); + if ($self->{conn_count} <= 0) { + my $msg = "connection count <= 0!\n"; + syslog('warn', $msg); + $self->dprint("warn: $msg"); + } $self->{conn_count}--; print "$$: CLOSE FH" . $hdl->{fh}->fileno() . " CONN$self->{conn_count}\n" if $self->{debug}; @@ -1489,8 +1494,6 @@ sub accept { fh_nonblocking $clientfh, 1; - $self->{conn_count}++; - return $clientfh; } @@ -1564,6 +1567,7 @@ sub check_host_access { sub accept_connections { my ($self) = @_; + my $hdl_err; eval { while (my $clientfh = $self->accept()) { @@ -1571,7 +1575,7 @@ sub accept_connections { my $reqstate = { keep_alive => $self->{keep_alive} }; # stop keep-alive when there are many open connections - if ($self->{conn_count} >= $self->{max_conn_soft_limit}) { + if ($self->{conn_count} + 1 >= $self->{max_conn_soft_limit}) { $reqstate->{keep_alive} = 0; } @@ -1587,6 +1591,8 @@ sub accept_connections { next; } + $hdl_err = 1; + $self->{conn_count}++; $reqstate->{hdl} = AnyEvent::Handle->new( fh => $clientfh, rbuf_max => 64*1024, @@ -1609,6 +1615,7 @@ sub accept_connections { if (my $err = $@) { syslog('err', "$err"); } }, ($self->{tls_ctx} ? (tls => "accept", tls_ctx => $self->{tls_ctx}) : ())); + $hdl_err = 0; print "$$: ACCEPT FH" . $clientfh->fileno() . " CONN$self->{conn_count}\n" if $self->{debug}; @@ -1618,6 +1625,15 @@ sub accept_connections { if (my $err = $@) { syslog('err', $err); + if ($hdl_err) { + if ($self->{conn_count} <= 0) { + my $msg = "connection count <= 0 not decrementing!\n"; + syslog('warn', $msg); + $self->dprint("warn: $msg"); + } else { + $self->{conn_count}--; + } + } $self->{end_loop} = 1; } -- 2.20.1