From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 52A4A1FF166 for ; Fri, 11 Oct 2024 13:19:45 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 16D2F3C229; Fri, 11 Oct 2024 13:20:12 +0200 (CEST) Message-ID: Date: Fri, 11 Oct 2024 13:20:06 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: pve-devel@lists.proxmox.com References: <20241011093744.25545-1-f.weber@proxmox.com> Content-Language: en-US From: Mira Limbeck In-Reply-To: <20241011093744.25545-1-f.weber@proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.338 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [iscsiplugin.pm, proxmox.com] Subject: Re: [pve-devel] [PATCH storage] iscsi: disable Open-iSCSI login retries to avoid blocking pvestatd X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" On 10/11/24 11:37, Friedrich Weber wrote: > Since 90c1b10 ("fix #254: iscsi: add support for multipath targets"), > iSCSI storage activation checks whether a session exists for each > discovered portal. If there is a discovered portal without a session, > it performs a discovery and login in the hope of establishing a > session to the portal. If the portal is unreachable when trying to log > in, Open-iSCSI's default behavior is to retry for up to 2 minutes, as > explained in /etc/iscsi/iscid.conf: > >> # The default node.session.initial_login_retry_max is 8 and >> # node.conn[0].timeo.login_timeout is 15 so we have: >> # >> # node.conn[0].timeo.login_timeout * \ >> node.session.initial_login_retry_max = 120s > > If pvestatd is activating the storage, it will be blocked during that > time, which is undesirable. This is particularly unfortunate if the > target announces portals that the host permanently cannot reach. In > that case, every pvestatd iteration will take 2 minutes. While it can > be argued that such setups are misconfigured, it is still desirable to > keep the fallout of that misconfiguration as low as possible. > > In order to reduce the time Open-iSCSI tries to log in, instruct > Open-ISCSI to not perform login retries for that target. For this, set > node.session.initial_login_retry_max for the target to 0. This setting > is stored in Open-iSCSI's records under /etc/iscsi/nodes. As these > records are overwritten with the defaults from /etc/iscsi/iscsid.conf > on discovery, the setting needs to be applied after discovery. > > With this setting, one login attempt should take at most 15 seconds. > This is still higher than pvestatd's iteration time of 10 seconds, but > more tolerable. Logins will still be continuously retried by pvestatd > in every iteration until there is a session to each discovered portal. > > Signed-off-by: Friedrich Weber > --- > src/PVE/Storage/ISCSIPlugin.pm | 8 ++++++++ > 1 file changed, 8 insertions(+) > > diff --git a/src/PVE/Storage/ISCSIPlugin.pm b/src/PVE/Storage/ISCSIPlugin.pm > index 2bdd9a2..efd9de4 100644 > --- a/src/PVE/Storage/ISCSIPlugin.pm > +++ b/src/PVE/Storage/ISCSIPlugin.pm > @@ -132,6 +132,14 @@ sub iscsi_login { > eval { iscsi_discovery($portals); }; > warn $@ if $@; > > + # Disable retries to avoid blocking pvestatd for too long, next iteration will retry anyway > + eval { > + my $cmd = [$ISCSIADM, '--mode', 'node', '--targetname', $target, '--op', 'update', > + '--name', 'node.session.initial_login_retry_max', '--value', '0']; As shortly discussed off-list, this should probably follow a similar style as the `Wrapping Arguments` section in the Perl Style Guide, but grouping option and value together in the same line? https://pve.proxmox.com/wiki/Perl_Style_Guide#Wrapping_Arguments > + run_command($cmd); > + }; > + warn $@ if $@; > + > run_command([$ISCSIADM, '--mode', 'node', '--targetname', $target, '--login']); > } > Tested this with 4 portals by disconnecting 2. With this patch the pvestatd update time was at ~30 seconds, matching 2 failed logins. Without the patch it was ~484 seconds. Since a login fails in 7 seconds, the old behavior actually did more than 8 retries, see the comment for `initial_login_retry_max`: # Note that if the login fails # quickly (before node.conn[0].timeo.login_timeout fires) because the network # layer or the target returns an error, iscsid may retry the login more than # node.session.initial_login_retry_max times. So especially for the cases of `no route to host` this should improve the update time significantly for multiple portals where some are never reachable. Consider this patch: Tested-by: Mira Limbeck Reviewed-by: Mira Limbeck _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel