[PVE-User] Dell PowerEdge T140, megaraid, BIOS Boot...

[PVE-User] Dell PowerEdge T140, megaraid, BIOS Boot...

[Marco Gaiarin]

I've just upgraded to PVE8 a little cluster built with an PowerEdge T340
(boot in UEFI mode) and an T140 (boot in BIOS mode).

T340 works like a charme; T140 after rebooting panicked because cannot find
root filesystem; looking carefully at kernel log revealed that disks get not
recognized, and going deeper lead me to:

 gaio@leia:~$ grep megaraid T140_boot_log.txt
 [   31.952470] megaraid_sas 0000:01:00.0: FW now in Ready state
 [   31.952472] megaraid_sas 0000:01:00.0: 63 bit DMA mask and 32 bit consistent mask
 [   31.952624] megaraid_sas 0000:01:00.0: firmware supports msix        : (96)
 [   31.952750] megaraid_sas 0000:01:00.0: requested/available msix 5/5 poll_queue 0
 [   31.952752] megaraid_sas 0000:01:00.0: current msix/online cpus      : (5/4)
 [   31.952753] megaraid_sas 0000:01:00.0: RDPQ mode     : (disabled)
 [   31.952758] megaraid_sas 0000:01:00.0: Current firmware supports maximum commands: 928        LDIO threshold: 237
 [   31.954076] megaraid_sas 0000:01:00.0: Performance mode :Latency (latency index = 1)
 [   31.954078] megaraid_sas 0000:01:00.0: FW supports sync cache        : No
 [   31.954080] megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
 [   32.362294] megaraid_sas 0000:01:00.0: Ignore DCMD timeout: megasas_get_ctrl_info 5382
 [   32.665126] megaraid_sas 0000:01:00.0: Could not get controller info. Fail from megasas_init_adapter_fusion 1907
 [   32.908139] megaraid_sas 0000:01:00.0: Failed from megasas_init_fw 6539

googling a bit lead me to some forum and pages; after some test i've added
to kernel boot parameters:

	iommu=pt

and now server boot flawlessy.


The two servers have different hardware but share the same BIOS, so this
lead to me to the hypotesis that the new megaraid_sas kernel driver depend
on UEFI initialization to work, or something like this.

Also, i've understood that 'pt' mean 'PassThrought', but i've not clear what
overral consequences (performance, stability, ...) have this options.


I'm seeking feedback. Thanks.

-- 



_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Dell PowerEdge T140, megaraid, BIOS Boot...

[RolandK via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 10933 bytes --]

From: RolandK <devzero@web.de>
To: Proxmox VE user list <pve-user@lists.proxmox.com>
Subject: Re: [PVE-User] Dell PowerEdge T140, megaraid, BIOS Boot...
Date: Fri, 2 May 2025 19:03:01 +0200
Message-ID: <b7165214-1db9-46a8-8d9d-2e8cb5cd1c6c@web.de>

hello,

there is a note at https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_8.2


        Kernel: |intel_iommu| now defaults to on

The |intel_iommu| parameter defaults to |on| in the kernel 6.8 series. 
Enabling IOMMU can cause problems with older hardware, or systems with 
not up to date BIOS, due to bugs in the BIOS.

The issue can be fixed by explicitly disabling |intel_iommu| on the 
kernel commandline (|intel_iommu=off|) following the reference 
documentation 
<https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_edit_kernel_cmdline>. 


does intel_iommu=off also help ?

regards
Roland

Am 02.05.25 um 17:21 schrieb Marco Gaiarin:
> I've just upgraded to PVE8 a little cluster built with an PowerEdge T340
> (boot in UEFI mode) and an T140 (boot in BIOS mode).
>
> T340 works like a charme; T140 after rebooting panicked because cannot find
> root filesystem; looking carefully at kernel log revealed that disks get not
> recognized, and going deeper lead me to:
>
>   gaio@leia:~$ grep megaraid T140_boot_log.txt
>   [   31.952470] megaraid_sas 0000:01:00.0: FW now in Ready state
>   [   31.952472] megaraid_sas 0000:01:00.0: 63 bit DMA mask and 32 bit consistent mask
>   [   31.952624] megaraid_sas 0000:01:00.0: firmware supports msix        : (96)
>   [   31.952750] megaraid_sas 0000:01:00.0: requested/available msix 5/5 poll_queue 0
>   [   31.952752] megaraid_sas 0000:01:00.0: current msix/online cpus      : (5/4)
>   [   31.952753] megaraid_sas 0000:01:00.0: RDPQ mode     : (disabled)
>   [   31.952758] megaraid_sas 0000:01:00.0: Current firmware supports maximum commands: 928        LDIO threshold: 237
>   [   31.954076] megaraid_sas 0000:01:00.0: Performance mode :Latency (latency index = 1)
>   [   31.954078] megaraid_sas 0000:01:00.0: FW supports sync cache        : No
>   [   31.954080] megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
>   [   32.362294] megaraid_sas 0000:01:00.0: Ignore DCMD timeout: megasas_get_ctrl_info 5382
>   [   32.665126] megaraid_sas 0000:01:00.0: Could not get controller info. Fail from megasas_init_adapter_fusion 1907
>   [   32.908139] megaraid_sas 0000:01:00.0: Failed from megasas_init_fw 6539
>
> googling a bit lead me to some forum and pages; after some test i've added
> to kernel boot parameters:
>
> 	iommu=pt
>
> and now server boot flawlessy.
>
>
> The two servers have different hardware but share the same BIOS, so this
> lead to me to the hypotesis that the new megaraid_sas kernel driver depend
> on UEFI initialization to work, or something like this.
>
> Also, i've understood that 'pt' mean 'PassThrought', but i've not clear what
> overral consequences (performance, stability, ...) have this options.
>
>
> I'm seeking feedback. Thanks.
>

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Dell PowerEdge T140, megaraid, BIOS Boot...

[Marco Gaiarin]

Mandi! RolandK
  In chel di` si favelave...

> does intel_iommu=off also help ?

Yes, also this fix the trouble.

But is really strange... most of our T140 (ALL boot in BIOS mode) work
without 'intel_iommu=off', but three server need it to boot. All get upgaded
to the latest BIOS revision.

Also, our T340 start to suffer ZFS trouble, and 'intel_iommu=off' seems to
cure also that.


We have added 'intel_iommu=off' to all our server, as a safety measure...

-- 


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Dell PowerEdge T140, megaraid, BIOS Boot...

[Arjen via pve-user]

[-- Attachment #1: Type: message/rfc822, Size: 4575 bytes --]

From: Arjen <leesteken+proxmox@pm.me>
To: Proxmox VE user list <pve-user@lists.proxmox.com>
Subject: Re: [PVE-User] Dell PowerEdge T140, megaraid, BIOS Boot...
Date: Wed, 14 May 2025 20:46:36 +0000
Message-ID: <oj5gf2D_oUcP0yEhjWbdHZXjiz6ULD0wSK9pbeUseDy663MWkg2MIClfGGzZFoAfO4XahMOmIQ78iYDJoBKRMYb6Bf6LjAlk8ECZD1Q6xlk=@pm.me>

On Wednesday, 14 May 2025 at 22:40, Marco Gaiarin <gaio@lilliput.linux.it> wrote:

> But is really strange... most of our T140 (ALL boot in BIOS mode) work
> without 'intel_iommu=off', but three server need it to boot. All get upgaded
> to the latest BIOS revision.

Maybe those three have VT-d (or IOMMU) enabled in the motherboard BIOS? intel_iommu=on became the default in kernel version 6.8.
Unless VT-d is enabled in the motherboard BIOS settings, intel_iommu=off does nothing (as it is disabled in the motherboard BIOS).

kind regards



[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Dell PowerEdge T140, megaraid, BIOS Boot...

[Marco Gaiarin]

Mandi! Arjen via pve-user
  In chel di` si favelave...

> Maybe those three have VT-d (or IOMMU) enabled in the motherboard BIOS? intel_iommu=on became the default in kernel version 6.8.
> Unless VT-d is enabled in the motherboard BIOS settings, intel_iommu=off does nothing (as it is disabled in the motherboard BIOS).

I've downloaded BIOS settings in XML format from iDRAC, and:

 gaio@leia:~$ diff -ud Scaricati/sdpve2-bios.xml.xml Scaricati/tvpve2-bios.xml 
 --- Scaricati/sdpve2-bios.xml.xml	2025-05-16 15:33:39.309180290 +0200
 +++ Scaricati/tvpve2-bios.xml	2025-05-16 15:40:10.974747104 +0200
 @@ -1,4 +1,4 @@
 -<SystemConfiguration Model="PowerEdge T140" ServiceTag="1234567" TimeStamp="Fri May 16 13:33:25 2025">
 +<SystemConfiguration Model="PowerEdge T140" ServiceTag="9876543" TimeStamp="Fri May 16 13:40:01 2025">
  <!--Export type is Normal,XML,Selective-->
  <!--Exported configuration may contain commented attributes. Attributes may be commented due to dependency, destructive nature, preserving server identity or for security reasons.-->
  <Component FQDD="BIOS.Setup.1-1">

exactly the same...

-- 



_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user