Re: [pve-devel] [PATCH] iscsi: fix excessive connection test spam on storage monitoring

[Stelios Vailakakis <stelios@libvirt.dev>]

Hi Friedrich,

Thank you for the warm welcome!

> This seems like an unrelated change that should be in its own patch,
> could you please re-send it separately?

I submitted a separate patch in the format requested at https://lists.proxmox.com/pipermail/pve-devel/2025-June/071942.html

> This sounds like #957 [1]. A patch [2] with similar objective was merged
> [3] and included in Proxmox VE 8.4. Can you check whether your patch is
> still needed on Proxmox VE 8.4?

On 8.4.1 with the sole Proxmox node that was hosting the TrueNAS instance (proxmox5), I continued to get below errors as well as the errors on the TrueNAS syslog after doing a full pveupgrade.

- proxmox5 does list the hostname instead of the IP address under iscsiadm -m session, while 1-4 list the IP address. Forcing this behavior on proxmox1 did not cause errors to appear (reverted back to IP Address after testing).
- By utilizing my patch above that includes only hostname support, the error continued on proxmox5
- By reverting back to my patched version which included both fixes (excessive connection attempts + hostname support) the errors went away from automated checking
- pvesm status no longer reports the error code 15 when run manually
- This patch is working from an older version of ISCSIPlugin.pm and I haven't checked to see why it's only an issue on this proxmox5 node.

Jul 06 08:30:55 proxmox5 pvestatd[490256]: command '/usr/bin/iscsiadm --mode node --targetname iqn.aaaa-bb.com.redacted.ssd-vm-stor --login' failed: exit code 15
Jul 06 08:30:55 proxmox5 pvestatd[490256]: command '/usr/bin/iscsiadm --mode node --targetname iqn.xxxx-yy.com.redacted.vm-stor --login' failed: exit code 15


root@proxmox1:~# pdsh -g pve pveversion 2>/dev/null | dshbak -c
----------------
proxmox[1-2,4-5]
----------------
pve-manager/8.4.1/2a5fa54a8503f96d (running kernel: 6.8.12-11-pve)

root@proxmox1:~# pdsh -g pve 'pvesm status | grep vm-stor' 2>/dev/null | dshbak -c
----------------
proxmox[1-2,4-5]
----------------
lvm-nas-ssd-vm-stor         lvm     active      6442442752       723525632      5718917120   11.23%
lvm-nas-vm-stor             lvm     active      2684346368      1006669824      1677676544   37.50%
nas-ssd-vm-stor           iscsi     active               0               0               0    0.00%
nas-vm-stor               iscsi     active               0               0               0    0.00%

root@proxmox1:~# pdsh -g pve 'ls -l /usr/share/perl5/PVE/Storage/ISCSIPlugin.pm' 2>/dev/null | dshbak -c
----------------
proxmox[1-2,4]
----------------
-rw-r--r-- 1 root root 16849 Apr  7 15:18 /usr/share/perl5/PVE/Storage/ISCSIPlugin.pm
----------------
proxmox5
----------------
-rw-r--r-- 1 root root 19767 Jul  6 08:33 /usr/share/perl5/PVE/Storage/ISCSIPlugin.pm

Regards,
Stelios Vailakakis

________________________________
From: Friedrich Weber <f.weber@proxmox.com>
Sent: Monday, June 23, 2025 3:24 AM
To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>; Stelios Vailakakis <stelios@libvirt.dev>
Subject: Re: [pve-devel] [PATCH] iscsi: fix excessive connection test spam on storage monitoring

Hi,

Thanks for getting involved!

On 20/06/2025 02:44, Stelios Vailakakis wrote:
> Hi all,
>
> This patch addresses excessive "connection lost" and "connection reset" log
> spam on iSCSI targets caused by Proxmox storage monitoring performing TCP
> connection tests every 10 seconds, even when iSCSI sessions are active.
>
> The issue appears as continuous log entries on iSCSI targets:
> "ctld[PID]: 192.168.1.XX: read: connection lost" every 10 seconds on each node.
>
> Others have experienced this, as seen in forum posts:
> iSCSI Reconnecting every 10 seconds to FreeNAS solution | Proxmox Support Forum<https://forum.proxmox.com/threads/iscsi-reconnecting-every-10-seconds-to-freenas-solution.21205/>
> freenas ctld: read: connection lost | TrueNAS Community<https://www.truenas.com/community/threads/freenas-ctld-read-connection-lost.45513/>

This sounds like #957 [1]. A patch [2] with similar objective was merged
[3] and included in Proxmox VE 8.4. Can you check whether your patch is
still needed on Proxmox VE 8.4?

> Changes:
> - Support hostnames in portal addresses (not just IP addresses) by updating
>   ISCSI_TARGET_RE and session parsing regex

This seems like an unrelated change that should be in its own patch,
could you please re-send it separately? If you do, please review our
developer docs [4], especially with regards to the CLA requirement and
instructions how to use git format-patch/send-mail to submit patches
(AFAICT it wasn't used here).

> - Skip TCP connection tests in check_connection when active sessions exist
> - Skip storage activation when sessions are already established
>
> This reduces iSCSI target log spam from continuous entries to only
> legitimate connection issues, while maintaining monitoring functionality.

Best wishes,

Friedrich

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=957
[2]
https://lore.proxmox.com/pve-devel/mailman.14.1742481496.359.pve-devel@lists.proxmox.com/
[3]
https://git.proxmox.com/?p=pve-storage.git;a=commit;h=32e482a6170b3a882dc332263281b7f449066a8d
[4] https://pve.proxmox.com/wiki/Developer_Documentation#Preparing_Patches

>
> Tested with:
> - IPv4 and IPv6 portals
> - Hostname based portals (like nas.example.com)
> - Multiple Proxmox nodes in cluster 8.2.4 + 8.3.0 (Requires latest ISCSIPlugin.pm. taken from raw.githubusercontent.com/proxmox/pve-storage/823707a7ac99eafe345d9131d10837ebefbb9078/src/PVE/Storage/ISCSIPlugin.pm<https://raw.githubusercontent.com/proxmox/pve-storage/823707a7ac99eafe345d9131d10837ebefbb9078/src/PVE/Storage/ISCSIPlugin.pm>)
> - TrueNAS and Linux iSCSI targets
> - Can apply from /usr/share/perl5/PVE/Storage# patch -p0 < /path/to/iscsifix.patch
>
> ---
>  PVE/Storage/ISCSIPlugin.pm | 13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
>
> --- ISCSIPlugin.pm.new-upstream 2025-06-19 16:24:09.220675514 -0500
> +++ ISCSIPlugin.pm      2025-06-19 16:24:42.265694074 -0500
> @@ -33,7 +33,7 @@
>  }
>
>  # Example: 192.168.122.252:3260,1 iqn.2003-01.org.linux-iscsi.proxmox-nfs.x8664:sn.00567885ba8f
> -my $ISCSI_TARGET_RE = qr/^((?:$IPV4RE|\[$IPV6RE\]):\d+)\,\S+\s+(\S+)\s*$/;
> +my $ISCSI_TARGET_RE = qr/^(\S+:\d+)\,\S+\s+(\S+)\s*$/;
>
>  sub iscsi_session_list {
>      assert_iscsi_support();
> @@ -49,7 +49,7 @@
>                  my $line = shift;
>                  # example: tcp: [1] 192.168.122.252:3260,1 iqn.2003-01.org.linux-iscsi.proxmox-nfs.x8664:sn.00567885ba8f (non-flash)
>                  if ($line =~
> -                    m/^tcp:\s+\[(\S+)\]\s+((?:$IPV4RE|\[$IPV6RE\]):\d+)\,\S+\s+(\S+)\s+\S+?\s*$/
> +                    m/^tcp:\s+\[(\S+)\]\s+(\S+:\d+)\,\S+\s+(\S+)\s+\S+?\s*$/
>                  ) {
>                      my ($session_id, $portal, $target) = ($1, $2, $3);
>                      # there can be several sessions per target (multipath)
> @@ -480,6 +480,10 @@
>  sub activate_storage {
>      my ($class, $storeid, $scfg, $cache) = @_;
>
> +    # Skip if session already exists
> +    my $existing_sessions = iscsi_session($cache, $scfg->{target});
> +    return 1 if defined($existing_sessions) && scalar(@$existing_sessions) > 0;
> +
>      return if !assert_iscsi_support(1);
>
>      my $sessions = iscsi_session($cache, $scfg->{target});
> @@ -618,6 +622,11 @@
>  sub check_connection {
>      my ($class, $storeid, $scfg) = @_;
>      my $cache = {};
> +    # Check for active session first
> +    my $sessions = iscsi_session_list();
> +    return 1 if defined($sessions->{$scfg->{target}});
> +
> +    # Only check portals if no active session
>      my $portals = iscsi_portals($scfg->{target}, $scfg->{portal});
>
>      for my $portal (@$portals) {
>
> ---
>
>
> Regards,
> Stelios Vailakakis
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel