From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 7D013B149 for ; Wed, 6 Apr 2022 13:28:04 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 735ED29FB1 for ; Wed, 6 Apr 2022 13:27:34 +0200 (CEST) Received: from kerio.tuxis.nl (alrami.saas.tuxis.net [31.3.111.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id EB35929FA8 for ; Wed, 6 Apr 2022 13:27:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=tuxis.nl; s=mail; h=from:subject:date:message-id:to:mime-version:content-type: content-transfer-encoding; bh=V1ejFtThbPI4HCTVRJ1kCNKCPTk6InrF29VHbcKbP+c=; b=k3OzY3HNM3ub13q3gHeKWI9jbYKxHgcDsk/U4L3CEkmsK8gFcjIKPEE8gYglzTqiIzhSymOnVkhwx 6/HYj1+hQrEbZGXL98SOZLGoapFvZeerzaR1KEBimcWktOIhvNF4bRfLEhIADCbQLHQm0+NXh4D3DK YHl4SFKOQMJM5gnfEsLJggePyMPDu/jSOfmWLMwCqOxnII2LkVXGvlfjfrI9JEOpf40LyPXo4pRxuy OjyeHQW3P/NDp1TRKxW1hazaX7WSj0C77Nmw/xM9PotyT6B7KqqptMzqlBzM1KQQjoErq9S9lbYxpP NID0aPD+jqwdeiI+QQkrWNHCXx+X2dw== X-Footer: dHV4aXMubmw= Received: from smtpclient.apple ([2a03:7900:64::1001]) (authenticated user mark@tuxis.nl) by kerio.tuxis.nl (Kerio Connect 9.4.1) with ESMTPSA (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256 bits)) for pbs-devel@lists.proxmox.com; Wed, 6 Apr 2022 12:57:26 +0200 From: Mark Schouten Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.80.82.1.1\)) Message-Id: Date: Wed, 6 Apr 2022 12:57:14 +0200 To: pbs-devel X-Mailer: Apple Mail (2.3696.80.82.1.1) X-SPAM-LEVEL: Spam detection results: 0 AWL 0.042 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pbs-devel] Timeouts on Proxmox Backup Server X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Apr 2022 11:28:04 -0000 Hi, We are suffering from some slowness in Proxmox Backup Server, which = results in timeouts in Proxmox VE and monitoring-notifications thereof. = We=E2=80=99re looking into why this is happening, but I cannot really = find anything wrong with the setup. As you may know, we have a few PBS-machines running with more than a few = users on it. The machine that suffers the most of this issue is pbs001, = which is handling about 34 authentication requests per second. We have = over 250 users on that machine. What I notice is that even though there = are no tasks running, we see quite a lot of CPU usage: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ = COMMAND 3035778 root 20 0 857840 27900 14124 S 163.7 0.0 59:22.43 = /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-api 7658 backup 20 0 6603600 528644 14004 S 74.5 0.8 40706:05 = /usr/lib/x86_64-linux-gnu/proxmox-backup/proxmox-backup-proxy When stracing PID 3035778, most output I see is the reading of the file = '/etc/proxmox-backup/tfa.json=E2=80=99, one byte at a time. Could it be that this part of authentication could use some = optimisation? Happy to help, if we need to provide information about this machine or = the setup. Please let me know. =E2=80=94=20 Mark Schouten, CTO Tuxis B.V. mark@tuxis.nl