From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 1FF5D7B158 for ; Thu, 7 Jul 2022 17:49:26 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 15FF820E67 for ; Thu, 7 Jul 2022 17:49:26 +0200 (CEST) Received: from kerio.tuxis.nl (alrami.saas.tuxis.net [31.3.111.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 7 Jul 2022 17:49:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=tuxis.nl; s=mail; h=from:reply-to:subject:date:message-id:to:mime-version:content-type: content-transfer-encoding; bh=F6YEKu77DkUt/6iopLYtFa8i5wqYjc5Cu2EhdrPvjSQ=; b=XM603vGUT9cryM34y9G69mp3f24IHa5FCJYbsvi74NXWkD93dddiPTXGABWITdk8qTMKWADjQt3e4 Y3Az4Al0qccdlBZcyQ/M7atfuV0u6EOjVBHVY2M5wNNhgg11/gaR5wMpGF090e/VeAgi6bwUNM3vN0 OJv4JMsCcgLD/qgskn07APsGvenmKG/RYekMMDhhuYBGw/58axqm8D17D6Va3qJlufChjl/ALyHfam 0daHve7hGfVFqZs2f7cG6+dVAcrK09scnYAxI7E+gh7KVd5F+186F7HmZuepdCHrfiB6OFPUOAzAd1 LowYIqE03ASx7oTaYJ7+eAzl16QOjxg== X-Footer: dHV4aXMubmw= Received: from [IPv6:2a03:7900:64::1000] ([2a03:7900:64::1000]) (authenticated user mark@tuxis.nl) by kerio.tuxis.nl (Kerio Connect 9.4.1 patch 1) with ESMTPSA (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256 bits)) for pbs-devel@lists.proxmox.com; Thu, 7 Jul 2022 17:49:17 +0200 From: "Mark Schouten" To: pbs-devel Date: Thu, 07 Jul 2022 15:49:11 +0000 Message-Id: Reply-To: "Mark Schouten" User-Agent: eM_Client/9.0.1755.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL 0.031 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox-backup-proxy.rs, tuxis.nl] Subject: [pbs-devel] Scheduler causing connectivity issues? X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2022 15:49:26 -0000 Hi, We=E2=80=99re getting complaints that one of our PBS=E2=80=99es is periodic= ally=20 unreachable. After investigation if the network might be at fault (even=20 though it=E2=80=99s handling about 5.5Gbit at night), we found that PBS is= =20 piling up waiting connections every minute, on the minute, as you can=20 see below. You see the output of `date`, combined with `ss -np | grep -c=20 8007`, the number of active connections. At first I thought that pvestatd was ddossing PBS, but pvestatd seems to=20 run more often than once in a minute. So stracing the API process, I found that that process is also just=20 waiting for something; must be the proxy-process. grepping for =E2=80=98minute=E2=80=99 in the code, I stumbled upon the func= tion=20 `next_minute` in ./src/bin/proxmox-backup-proxy.rs. I=E2=80=99m not quite s= ure=20 if I understand it correctly, but it seems that every minute, the=20 scheduler is going to try and find out if it should be doing something. Drilling down on that in my strace-foo, I think I see quite some=20 read/write/rename actions on jobstate-files. Which leads me to conclude=20 that the proxy process is waiting for the scheduler.. This is just guess-work, but you guys can surely find out better what=E2=80= =99s=20 going on than me. This PBS is running with 45 users and 67 datastores. Hope you guys can find something.. If I need to debug anything, let me=20 know! =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Timestamp | `ss -np|grep -c 8007` Thu 07 Jul 2022 05:38:00 PM CEST | 13 Thu 07 Jul 2022 05:38:00 PM CEST | 24 Thu 07 Jul 2022 05:38:01 PM CEST | 32 Thu 07 Jul 2022 05:38:01 PM CEST | 45 Thu 07 Jul 2022 05:38:02 PM CEST | 58 Thu 07 Jul 2022 05:38:02 PM CEST | 65 Thu 07 Jul 2022 05:38:03 PM CEST | 68 Thu 07 Jul 2022 05:38:03 PM CEST | 76 Thu 07 Jul 2022 05:38:04 PM CEST | 79 Thu 07 Jul 2022 05:38:05 PM CEST | 82 Thu 07 Jul 2022 05:38:05 PM CEST | 88 Thu 07 Jul 2022 05:38:06 PM CEST | 96 Thu 07 Jul 2022 05:38:06 PM CEST | 102 Thu 07 Jul 2022 05:38:07 PM CEST | 104 Thu 07 Jul 2022 05:38:07 PM CEST | 111 Thu 07 Jul 2022 05:38:08 PM CEST | 124 Thu 07 Jul 2022 05:38:08 PM CEST | 130 Thu 07 Jul 2022 05:38:09 PM CEST | 133 Thu 07 Jul 2022 05:38:09 PM CEST | 137 Thu 07 Jul 2022 05:38:10 PM CEST | 22 Thu 07 Jul 2022 05:38:11 PM CEST | 23 Thu 07 Jul 2022 05:39:00 PM CEST | 20 Thu 07 Jul 2022 05:39:01 PM CEST | 36 Thu 07 Jul 2022 05:39:01 PM CEST | 48 Thu 07 Jul 2022 05:39:02 PM CEST | 57 Thu 07 Jul 2022 05:39:02 PM CEST | 64 Thu 07 Jul 2022 05:39:03 PM CEST | 69 Thu 07 Jul 2022 05:39:03 PM CEST | 76 Thu 07 Jul 2022 05:39:04 PM CEST | 78 Thu 07 Jul 2022 05:39:04 PM CEST | 84 Thu 07 Jul 2022 05:39:05 PM CEST | 88 Thu 07 Jul 2022 05:39:06 PM CEST | 96 Thu 07 Jul 2022 05:39:06 PM CEST | 102 Thu 07 Jul 2022 05:39:07 PM CEST | 104 Thu 07 Jul 2022 05:39:07 PM CEST | 111 Thu 07 Jul 2022 05:39:08 PM CEST | 120 Thu 07 Jul 2022 05:39:08 PM CEST | 127 Thu 07 Jul 2022 05:39:09 PM CEST | 131 Thu 07 Jul 2022 05:39:09 PM CEST | 133 Thu 07 Jul 2022 05:39:10 PM CEST | 29 Thu 07 Jul 2022 05:39:10 PM CEST | 24 Thu 07 Jul 2022 05:40:00 PM CEST | 21 Thu 07 Jul 2022 05:40:01 PM CEST | 33 Thu 07 Jul 2022 05:40:01 PM CEST | 45 Thu 07 Jul 2022 05:40:02 PM CEST | 58 Thu 07 Jul 2022 05:40:02 PM CEST | 64 Thu 07 Jul 2022 05:40:03 PM CEST | 70 Thu 07 Jul 2022 05:40:03 PM CEST | 75 Thu 07 Jul 2022 05:40:04 PM CEST | 79 Thu 07 Jul 2022 05:40:04 PM CEST | 83 Thu 07 Jul 2022 05:40:05 PM CEST | 88 Thu 07 Jul 2022 05:40:05 PM CEST | 96 Thu 07 Jul 2022 05:40:06 PM CEST | 102 Thu 07 Jul 2022 05:40:07 PM CEST | 105 Thu 07 Jul 2022 05:40:07 PM CEST | 113 Thu 07 Jul 2022 05:40:08 PM CEST | 122 Thu 07 Jul 2022 05:40:08 PM CEST | 129 Thu 07 Jul 2022 05:40:09 PM CEST | 134 Thu 07 Jul 2022 05:40:09 PM CEST | 135 Thu 07 Jul 2022 05:40:10 PM CEST | 27 =E2=80=94 Mark Schouten, CTO Tuxis B.V. mark@tuxis.nl