From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <gilberto.nunes32@gmail.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 28898905A6
for <pve-user@pve.proxmox.com>; Tue, 2 Apr 2024 13:15:23 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 109964780
for <pve-user@pve.proxmox.com>; Tue, 2 Apr 2024 13:15:23 +0200 (CEST)
Received: from mail-lj1-x235.google.com (mail-lj1-x235.google.com
[IPv6:2a00:1450:4864:20::235])
(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pve-user@pve.proxmox.com>; Tue, 2 Apr 2024 13:15:20 +0200 (CEST)
Received: by mail-lj1-x235.google.com with SMTP id
38308e7fff4ca-2d6ee81bc87so44879231fa.1
for <pve-user@pve.proxmox.com>; Tue, 02 Apr 2024 04:15:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20230601; t=1712056514; x=1712661314; darn=pve.proxmox.com;
h=cc:to:subject:message-id:date:from:in-reply-to:references
:mime-version:from:to:cc:subject:date:message-id:reply-to;
bh=i2K1ZjfaFJkMbFUXQidrUw/IoViXqlH3Escsci7etLU=;
b=G4zosy0rQXNwp14iK8jJL4mOrCKpLU2tkMlYouS5WjGFDq51/Q1qdV2GfrjZ8XYQkI
UVzy4RUhnLzsXk9KJWOxWfH22CG1cY11HFF0BdKhIg7z1t4P69Y4OoXA5+pJEcwjeESm
7dZSewwt7zHBHgYr6pcMmhxuuIdivmU4rjtCHLEJUT94g1nebNF9PRRFn/H1/AsHJ5a/
Oeg0FIgiKG+2vo874MARnS84lBRjtkRH9+Y4ALfTJVYKEVGxCQFzyvi5Z+euiLGNHPAH
rWrtbFKMIWjH7PsoP8bizQdawJTfb+WbL7jjo1EFWJ+QkJPpeMW2aPZOIyOtPmi2UrNq
JSow==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1712056514; x=1712661314;
h=cc:to:subject:message-id:date:from:in-reply-to:references
:mime-version:x-gm-message-state:from:to:cc:subject:date:message-id
:reply-to;
bh=i2K1ZjfaFJkMbFUXQidrUw/IoViXqlH3Escsci7etLU=;
b=lsxUt8GuHCN0pKVkjJMC3nNZQiKqqncLyNceYg9k+pEwN76SGDujMMKh9MsIjJKU32
0d0Vbe/e9/ICsl3M9sHCkCFsest3kXUwhvZsxczVaqpfc4Pf8npwgQxX6D4Q7kKTyW8m
fRp/cPU444rjVNZLNf1DQJYvLQdFXRhkz4nbyk1/tHwCL5SQVj5tONha1EVCAjqmgMLa
d3FP7SkVFeO8FvNYOrRetBLKxTc6Ac7Up9lglyJuuzAKJksW6Cb1km1rob/IZPbzFt5i
cE/OIcAndms5Ufqd43xmVQFsRJEyEX2Y7U5gdYppP5XmzuccbYSX38/Z9JIMC2EXsIaY
vtXA==
X-Gm-Message-State: AOJu0Yzk+Oh20vq5a5CFdOXaSz3EQzdhRleUgTtKt5Jnm/EiUfYLLt0j
6vsTiluQAJpUpbXjpIEPz49/BN84XDy3cc3RrukaR+EBDo5dRgftT6zfOLunuq1g/bMYuq4PJLQ
LemBO+A4LFLkMNr/Z+JfEuIrix26siPcAsug=
X-Google-Smtp-Source: AGHT+IFU6aBw0XvKFkXaO/btIbBgRuyG8Vnd9QICYocRewE15eAMHf78V7xXUmQGgOsLuDG167qGEDjW3jGiJs0bZyQ=
X-Received: by 2002:a2e:9cd2:0:b0:2d7:7c0:b077 with SMTP id
g18-20020a2e9cd2000000b002d707c0b077mr7827677ljj.43.1712056513243; Tue, 02
Apr 2024 04:15:13 -0700 (PDT)
MIME-Version: 1.0
References: <D8D305A2-D2B7-4A5D-821C-65DE75621457@kmi.com>
<93280CB8-7582-4456-9101-D594CE2C86A2@kmi.com>
<mailman.755.1711637904.434.pve-user@lists.proxmox.com>
<mailman.785.1712036605.434.pve-user@lists.proxmox.com>
<CAOKSTBukcW2gPax=xHB3jTZDUNNv6zRmHbiVAxfJDpqUA5TG2Q@mail.gmail.com>
<5D727A1E-902A-4CA4-BEF8-A0F1CBFA754E@me.com>
In-Reply-To: <5D727A1E-902A-4CA4-BEF8-A0F1CBFA754E@me.com>
From: Gilberto Ferreira <gilberto.nunes32@gmail.com>
Date: Tue, 2 Apr 2024 08:15:00 -0300
Message-ID: <CAOKSTButEg+8Xi-3tkyJQY9JgkG3ywq8x=9iX7Pn9vbrSUxj_w@mail.gmail.com>
To: Stefan Radman <stefan.radman@me.com>
Cc: PVE User List <pve-user@pve.proxmox.com>
X-SPAM-LEVEL: Spam detection results: 0
AWL -0.088 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DKIM_SIGNED 0.1 Message has a DKIM or DK signature,
not necessarily valid
DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature
DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's
domain
DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from
domain DMARC_PASS -0.1 DMARC pass policy
FREEMAIL_ENVFROM_END_DIGIT 0.25 Envelope-from freemail username ends in digit
FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider
HTML_MESSAGE 0.001 HTML included in message
KAM_LOTSOFHASH 0.25 Emails with lots of hash-like gibberish
POISEN_SPAM_PILL_4 0.1 random spam to be learned in bayes
RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/,
no trust
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [launchpad.net, linaro.org, proxmox.com]
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Content-Filtered-By: Mailman/MimeDel 2.1.29
Subject: Re: [PVE-User] 6.5.13-3-pve kernel panic on shutdown
X-BeenThere: pve-user@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE user list <pve-user.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>,
<mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-user/>
List-Post: <mailto:pve-user@lists.proxmox.com>
List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>,
<mailto:pve-user-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Tue, 02 Apr 2024 11:15:23 -0000
Good
Em ter., 2 de abr. de 2024, 08:09, Stefan Radman <stefan.radman@me.com>
escreveu:
> Workaround: No more kernel panics on reboot when pinning kernel
> 6.2.16-20-pve.
>
> Affected kernels:
> 6.5.13-1-pve
> 6.5.13-3-pve
>
> The original issue [1] was solved long ago [2] but apparently
> re-introduced recently [3].
>
> Regression [4] being discussed on kernel.org
>
> Looks like a back and forth in the tg3 driver.
>
> Note that the kernel panic is only triggered by =E2=80=9Creboot=E2=80=9D =
and not by
> =E2=80=9Cshutdown=E2=80=9D.
>
> Stefan
>
> root@per740:~# proxmox-boot-tool kernel list
> Manually selected kernels:
> None.
>
> Automatically selected kernels:
> 6.2.16-20-pve
> 6.5.13-1-pve
> 6.5.13-3-pve
>
> Pinned kernel:
> 6.2.16-20-pve
> root@per740:~# pveversion
> pve-manager/8.1.10/4b06efb5db453f29 (running kernel: 6.2.16-20-pve)
>
> [1] Use ACPI S5 for reboot #1904225: causes reboot crash on Dell T440
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1962730
>
> [2] tg3: Disable tg3 device on system reboot to avoid triggering AER
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit=
/?id=3D2ca1c94ce0b65a2ce7512b718f3d8a0fe6224bca
>
> [3] tg3: power down device only on SYSTEM_POWER_OFF
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit=
/?id=3D9fc3bc7643341dc5be7d269f3d3dbe441d8d7ac3
>
> [4] * [PATCH] tg3: add new module param to force device power down on
> reboot
>
> https://lore.kernel.org/lkml/d8ed4af1-5c83-4895-9fc3-9aea25724fd9@gmail.c=
om/T/
>
>
> On Apr 2, 2024, at 09:37, Gilberto Ferreira <gilberto.nunes32@gmail.com>
> wrote:
>
> Perhaps you should try another kernel besides 6.15 like 6.2 for instance.
>
> Em ter., 2 de abr. de 2024, 02:43, Stefan Radman via pve-user <
> pve-user@lists.proxmox.com> escreveu:
>
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Stefan Radman <stefan.radman@me.com>
>> To: Proxmox VE user list <pve-user@lists.proxmox.com>
>> Cc: PVE User List <pve-user@pve.proxmox.com>
>> Bcc:
>> Date: Tue, 2 Apr 2024 07:42:32 +0200
>> Subject: Re: [PVE-User] 6.5.13-3-pve kernel panic on shutdown
>> Yesterday I had the same thing happen when shutting down a Dell PowerEdg=
e
>> R740.
>>
>> Again, the kernel panic was triggered by a BCM5720 running Broadcom
>> firmware 22.71.3 and the tg3 driver from kernel 6.5.13-3-pve.
>>
>> R740 BIOS 2.21.2 (but also happened with 2.20.1)
>>
>> Stefan
>>
>> [1325586.715465] ACPI: PM: Preparing to enter system sleep state S5
>> [1325589.991219] {1}[Hardware Error]: Hardware error from APEI Generic
>> Hardware Error Source: 5
>> [1325589.991223] {1}[Hardware Error]: event severity: fatal
>> [1325589.991225] {1}[Hardware Error]: Error 0, type: fatal
>> [1325589.991227] {1}[Hardware Error]: section_type: PCIe error
>> [1325589.991228] {1}[Hardware Error]: port_type: 0, PCIe end point
>> [1325589.991231] {1}[Hardware Error]: version: 3.0
>> [1325589.991233] {1}[Hardware Error]: command: 0x0002, status: 0x0010
>> [1325589.991235] {1}[Hardware Error]: device_id: 0000:01:00.1
>> [1325589.991237] {1}[Hardware Error]: slot: 0
>> [1325589.991239] {1}[Hardware Error]: secondary_bus: 0x00
>> [1325589.991240] {1}[Hardware Error]: vendor_id: 0x14e4, device_id:
>> 0x165f
>> [1325589.991242] {1}[Hardware Error]: class_code: 020000
>> [1325589.991244] {1}[Hardware Error]: aer_uncor_status: 0x00100000,
>> aer_uncor_mask: 0x00010000
>> [1325589.991246] {1}[Hardware Error]: aer_uncor_severity: 0x000ef030
>> [1325589.991248] {1}[Hardware Error]: TLP Header: 40000001 0000010f
>> 90028090 00000000
>> [1325589.991252] Kernel panic - not syncing: Fatal hardware error!
>> [1325589.991254] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O
>> 6.5.13-1-pve #1
>> [1325589.991258] Hardware name: Dell Inc. PowerEdge R740/0WXD1Y, BIOS
>> 2.20.1 09/13/2023
>> [1325589.991259] Call Trace:
>> [1325589.991261] <NMI>
>>
>> root@per740:~# pveversion
>> pve-manager/8.1.10/4b06efb5db453f29 (running kernel: 6.5.13-3-pve)
>>
>> root@per740:~# ethtool -i eno4
>> driver: tg3
>> version: 6.5.13-3-pve
>> firmware-version: FFV22.71.3 bc 5720-v1.39
>> expansion-rom-version:
>> bus-info: 0000:01:00.1
>> supports-statistics: yes
>> supports-test: yes
>> supports-eeprom-access: yes
>> supports-register-dump: yes
>> supports-priv-flags: no
>>
>>
>> > On Mar 28, 2024, at 15:50, Stefan Radman via pve-user <
>> pve-user@lists.proxmox.com> wrote:
>> >
>> >
>> > From: Stefan Radman <stefan.radman@me.com>
>> > Subject: 6.5.13-3-pve kernel panic on shutdown
>> > Date: March 28, 2024 at 15:50:02 GMT+1
>> > To: PVE User List <pve-user@pve.proxmox.com>
>> >
>> >
>> > I recently noticed that a Dell Poweredge R540 currently running Proxmo=
x
>> VE 8.1.8 (kernel 6.5.13-3-pve) throws a kernel panic on shutdown.
>> >
>> > The kernel panic is triggered 3-4 seconds after the last network
>> interface goes down (onboard BCM5720 LOM), while the system enters S5
>> (sleep) state.
>> >
>> > [84459.970212] bond0: (slave eno1): link status definitely down,
>> disabling slave
>> > [84459.982170] bond0: (slave eno2): link status definitely down,
>> disabling slave
>> > [84459.990037] tg3 0000:04:00.0 eno1: left promiscuous mode
>> > [84459.995822] tg3 0000:04:00.0 eno1: left allmulticast mode
>> > [84460.001615] bond0: now running without any active interface!
>> > [84460.018133] vmbr0: port 1(bond0) entered disabled state
>> > [84460.291379] ACPI: PM: Preparing to enter system sleep state S5
>> > [84463.685113] {1}[Hardware Error]: Hardware error from APEI Generic
>> Hardware Error Source: 5
>> >
>> > This is reproducible on every reboot.
>> >
>> > R540 and BCM5720 are running the latest firmware available from the
>> Dell support website.
>> >
>> > Link [2] below seem to suggest that my problem is related to a
>> combination of ACPI S5, the tg3 driver and the BCM5720 on-board NIC.
>> >
>> > Has anyone else seen this lately (or ever) with Promox VE?
>> >
>> > Thank you
>> >
>> > Stefan
>> >
>> > [1] Use ACPI S5 for reboot #1904225: causes reboot crash on Dell T440
>> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1962730
>> >
>> > [2] [SRU][Regression] Revert "PM: ACPI: reboot: Use S5 for reboot"
>> which causes Bus Fatal Error when rebooting system with BCM5720 NIC
>> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1917471
>> >
>> > [3] tg3: Disable tg3 device on system reboot to avoid triggering AER
>> >
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commi=
t/?id=3D2ca1c94ce0b65a2ce7512b718f3d8a0fe6224bca
>> >
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/=
drivers/net/ethernet/broadcom/tg3.c?id=3D2ca1c94ce0b65a2ce7512b718f3d8a0fe6=
224bca#n18074
>> >
>> > [4] * [PATCH] tg3: Disable tg3 device on system reboot to avoid
>> triggering AER
>> >
>> https://lore.kernel.org/netdev/CAAd53p7PmEp+vWLz+fGdDntGQ2KqgL54fo86Bpy7=
oy9tKzXsAg@mail.gmail.com/T/
>> >
>> > [5] [v4,2/2] PM: ACPI: reboot: Reinstate S5 for reboot
>> >
>> https://patches.linaro.org/project/linux-acpi/patch/20220916043319.11971=
6-2-kai.heng.feng@canonical.com/
>> >
>> > [6] * [PATCH] tg3: add new module param to force device power down on
>> reboot
>> >
>> https://lore.kernel.org/lkml/d8ed4af1-5c83-4895-9fc3-9aea25724fd9@gmail.=
com/T/
>> >
>> >
>> > [84458.600189] systemd-shutdown[1]: Syncing filesystems and block
>> devices.
>> > [84458.607141] systemd-shutdown[1]: Rebooting.
>> > [84458.612283] spi-nor spi0.0: Software reset failed: -524
>> > [84459.777370] megaraid_sas 0000:17:00.0: megasas_disable_intr_fusion
>> is called outbound_intr_mask:0x40000009
>> > [84459.970212] bond0: (slave eno1): link status definitely down,
>> disabling slave
>> > [84459.982170] bond0: (slave eno2): link status definitely down,
>> disabling slave
>> > [84459.990037] tg3 0000:04:00.0 eno1: left promiscuous mode
>> > [84459.995822] tg3 0000:04:00.0 eno1: left allmulticast mode
>> > [84460.001615] bond0: now running without any active interface!
>> > [84460.018133] vmbr0: port 1(bond0) entered disabled state
>> > [84460.291379] ACPI: PM: Preparing to enter system sleep state S5
>> > [84463.685113] {1}[Hardware Error]: Hardware error from APEI Generic
>> Hardware Error Source: 5
>> > [84463.685116] {1}[Hardware Error]: event severity: fatal
>> > [84463.685117] {1}[Hardware Error]: Error 0, type: fatal
>> > [84463.685119] {1}[Hardware Error]: section_type: PCIe error
>> > [84463.685120] {1}[Hardware Error]: port_type: 0, PCIe end point
>> > [84463.685121] {1}[Hardware Error]: version: 3.0
>> > [84463.685122] {1}[Hardware Error]: command: 0x0002, status: 0x0010
>> > [84463.685123] {1}[Hardware Error]: device_id: 0000:04:00.1
>> > [84463.685125] {1}[Hardware Error]: slot: 0
>> > [84463.685126] {1}[Hardware Error]: secondary_bus: 0x00
>> > [84463.685127] {1}[Hardware Error]: vendor_id: 0x14e4, device_id:
>> 0x165f
>> > [84463.685128] {1}[Hardware Error]: class_code: 020000
>> > [84463.685129] {1}[Hardware Error]: aer_uncor_status: 0x00100000,
>> aer_uncor_mask: 0x00010000
>> > [84463.685130] {1}[Hardware Error]: aer_uncor_severity: 0x000ef030
>> > [84463.685131] {1}[Hardware Error]: TLP Header: 40000001 0000010f
>> 90028090 00000000
>> > [84463.685134] Kernel panic - not syncing: Fatal hardware error!
>> > [84463.685136] CPU: 0 PID: 1 Comm: systemd-shutdow Tainted: P
>> O 6.5.13-3-pve #1
>> > [84463.685139] Hardware name: Dell Inc. PowerEdge R540/0VC7DK, BIOS
>> 2.21.1 03/07/2024
>> > [84463.685140] Call Trace:
>> > [84463.685142] <NMI>
>> > =E2=80=A6
>> >
>> > root@pve:~# pveversion
>> > pve-manager/8.1.8/d29041d9f87575d0 (running kernel: 6.5.13-3-pve)
>> > root@pve:~# ethtool -i eno2
>> > driver: tg3
>> > version: 6.5.13-3-pve
>> > firmware-version: FFV22.71.3 bc 5720-v1.39
>> > expansion-rom-version:
>> > bus-info: 0000:04:00.1
>> > supports-statistics: yes
>> > supports-test: yes
>> > supports-eeprom-access: yes
>> > supports-register-dump: yes
>> > supports-priv-flags: no
>> > root@pve:~# lspci | fgrep 04:00.1
>> > 04:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme
>> BCM5720 Gigabit Ethernet PCIe
>> >
>> >
>> >
>> > _______________________________________________
>> > pve-user mailing list
>> > pve-user@lists.proxmox.com
>> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Stefan Radman via pve-user <pve-user@lists.proxmox.com>
>> To: Proxmox VE user list <pve-user@lists.proxmox.com>
>> Cc: Stefan Radman <stefan.radman@me.com>, PVE User List <
>> pve-user@pve.proxmox.com>
>> Bcc:
>> Date: Tue, 2 Apr 2024 07:42:32 +0200
>> Subject: Re: [PVE-User] 6.5.13-3-pve kernel panic on shutdown
>> _______________________________________________
>> pve-user mailing list
>> pve-user@lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
>
>