From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <jbonor@gmail.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 654F77B31A for <pbs-devel@lists.proxmox.com>; Fri, 8 Jul 2022 11:37:01 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 5A671270AD for <pbs-devel@lists.proxmox.com>; Fri, 8 Jul 2022 11:37:01 +0200 (CEST) Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for <pbs-devel@lists.proxmox.com>; Fri, 8 Jul 2022 11:37:00 +0200 (CEST) Received: by mail-wr1-x432.google.com with SMTP id q9so29745288wrd.8 for <pbs-devel@lists.proxmox.com>; Fri, 08 Jul 2022 02:37:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language:to :references:from:in-reply-to:content-transfer-encoding; bh=GXuM85nTw7nFnBFYperp3RRWqabAxwE/GE2Df/J9Ch8=; b=qPV43JZOnAPc1RyheRiZVCJk6Y1D/cwSGrsAFFq3z9pD5Hu9DBCnK787+CXo4lTYXU ob7omc+bBkBcVw6n8JFbLQlK1dZ7cM9/GuICaKejSz+ajqPDuLBNCZKWqzIkkpyTKK0s cJ0WnZXCjUPuScFqF0KrdrmyqZN0gYGHqn/S6luC0RAzgwl37gj03FnMUOJ300ct1gSj 3p7Cz7maDUqmTWicEevCF/WuvRs/ziBHQBLdRcUBWilgM4PqRDaQ8F1x7K0A9mOXd10Q F3966LcUO4FJ5rDfiBfQ9+KD4n5X6MNO0JrraWy0Pr708gdTXik3/wUto7hemjkcSOOk Id3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:references:from:in-reply-to :content-transfer-encoding; bh=GXuM85nTw7nFnBFYperp3RRWqabAxwE/GE2Df/J9Ch8=; b=6NrkvizwnzKa49KqpUrPaYvjosqzS/ncvGvcUSadPjLzmspkxqW26nr7Z+O3oPcgMF ef0agrTRHbcmJm8gRz4YjcMF7OrcaLX6NY2j09IwiFhu6cjvpkXMzgsdZoGjqBDpksVB X/wyZmHsgl+KbHaBj8j3AHtSH+DGf7AxMa9ru3KMBPu8SXynStgbENdmxfNKosmsYoSs adg4r+XN5EGITUccp1UGklC60w5kxMbqNPNFvx2ura+nbrHXp28nyZoOr8Xgh+8Q+5L8 EnjQY3uhOLsJwCWHEluMSjy3QCn4y6opia263zv4Mz4hsZpWIqVko6ZtVSdXhyC4U5vQ G03g== X-Gm-Message-State: AJIora/fTLLoRuiBKM6TL3kzIUYLwWflZ/3Ro/yWzvN7TK79e4t4EkrA ZMyeyUFRYrm4hEjC+3uZGoUobnsh2Qc= X-Google-Smtp-Source: AGRyM1tB1io8TZrd6SZ1cBTpltI2CE3j7Smr1zuAbScBY0LXmvShsrXbvshJocMbx9GPhiGo2tDg9g== X-Received: by 2002:a05:6000:1a8d:b0:21d:9055:96a6 with SMTP id f13-20020a0560001a8d00b0021d905596a6mr1927501wry.245.1657273013682; Fri, 08 Jul 2022 02:36:53 -0700 (PDT) Received: from [10.10.0.129] ([80.30.43.32]) by smtp.gmail.com with ESMTPSA id n35-20020a05600c502300b003a2d0f0ccaesm1567234wmr.34.2022.07.08.02.36.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 08 Jul 2022 02:36:53 -0700 (PDT) Message-ID: <94e2e388-2d57-dd62-1063-585e7af2ab38@gmail.com> Date: Fri, 8 Jul 2022 11:36:50 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 Content-Language: es-ES To: Proxmox Backup Server development discussion <pbs-devel@lists.proxmox.com>, Mark Schouten <mark@tuxis.nl> References: <emf3a204a3-c161-48aa-9337-914001134e66@e231eb23.com> <b7d46762-b3bb-93af-2f8a-ad3742b7b57e@proxmox.com> From: Jorge Boncompte <jbonor@gmail.com> In-Reply-To: <b7d46762-b3bb-93af-2f8a-ad3742b7b57e@proxmox.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider NICE_REPLY_A -0.001 Looks like a legit reply (A) RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pbs-devel] Scheduler causing connectivity issues? X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion <pbs-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/> List-Post: <mailto:pbs-devel@lists.proxmox.com> List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Fri, 08 Jul 2022 09:37:01 -0000 El 8/7/22 a las 9:09, Thomas Lamprecht escribió: > Hi, > > On 07/07/2022 17:49, Mark Schouten wrote: >> We’re getting complaints that one of our PBS’es is periodically unreachable. After investigation if the network might be at fault (even though it’s handling about 5.5Gbit at night), we found that PBS is piling up waiting connections every minute, on the minute, as you can see below. You see the output of `date`, combined with `ss -np | grep -c 8007`, the number of active connections. >> >> At first I thought that pvestatd was ddossing PBS, but pvestatd seems to run more often than once in a minute. >> >> So stracing the API process, I found that that process is also just waiting for something; must be the proxy-process. >> >> grepping for ‘minute’ in the code, I stumbled upon the function `next_minute` in ./src/bin/proxmox-backup-proxy.rs. I’m not quite sure if I understand it correctly, but it seems that every minute, the scheduler is going to try and find out if it should be doing something. >> >> Drilling down on that in my strace-foo, I think I see quite some read/write/rename actions on jobstate-files. Which leads me to conclude that the proxy process is waiting for the scheduler.. >> >> This is just guess-work, but you guys can surely find out better what’s going on than me. >> >> This PBS is running with 45 users and 67 datastores. >> >> Hope you guys can find something.. If I need to debug anything, let me know! > > Thanks for the info, this already helps quite a bit. We'll look into it and re-check > with you if we need more info. Hi, We've been having a problem that resembles this one with several proxmox-backup-server 2.2.x versions. Our PBS stopped accepting backups jobs sometimes, but if we retried manully they started fine. The only message I could find was: backup failed: could not activate storage 'XXXXXXX': XXXXXX: error fetching datastores - 500 Can't connect to XXXXXXXX:8007 Restarting the proxy seemed to help to get it working one or two days more. We have reverted proxmox-backup-server to 2.1.8-1 and every is fine again. Regards. > > cheers, > Thomas > > > _______________________________________________ > pbs-devel mailing list > pbs-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel