Re: [pve-devel] FW: issues with Virtio-SCSI devicde on Proxmox...

["DERUMIER, Alexandre via pve-devel" <pve-devel@lists.proxmox.com>]

[-- Attachment #1: Type: message/rfc822, Size: 33368 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "cmos@maklee.com" <cmos@maklee.com>
Cc: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: FW: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
Date: Wed, 14 Aug 2024 13:05:35 +0000
Message-ID: <6ea8c90b469da48525ed743b6541bb3644be4a6c.camel@groupe-cyllene.com>

>>I'm talking about virtio-scsi. Our virtio-network device is working
>>fine 

Yes, sorry, I wanted to said virtio-scsi.   

All pci devices excluding passthrough devices (with pcie=on flag) are
actually plugged to pci bridge


sub get_pci_addr_map {
    $pci_addr_map = {
	piix3 => { bus => 0, addr => 1, conflict_ok => qw(ehci)  },
	ehci => { bus => 0, addr => 1, conflict_ok => qw(piix3) }, #
instead of piix3 on arm
	vga => { bus => 0, addr => 2, conflict_ok => qw(legacy-igd) },
	'legacy-igd' => { bus => 0, addr => 2, conflict_ok => qw(vga)
}, # legacy-igd requires vga=none
	balloon0 => { bus => 0, addr => 3 },
	watchdog => { bus => 0, addr => 4 },
	scsihw0 => { bus => 0, addr => 5, conflict_ok => qw(pci.3) },
	'pci.3' => { bus => 0, addr => 5, conflict_ok => qw(scsihw0)
}, # also used for virtio-scsi-single bridge
	scsihw1 => { bus => 0, addr => 6 },
	ahci0 => { bus => 0, addr => 7 },
	qga0 => { bus => 0, addr => 8 },
	spice => { bus => 0, addr => 9 },
	virtio0 => { bus => 0, addr => 10 },
	virtio1 => { bus => 0, addr => 11 },
	virtio2 => { bus => 0, addr => 12 },
	virtio3 => { bus => 0, addr => 13 },
	virtio4 => { bus => 0, addr => 14 },
	virtio5 => { bus => 0, addr => 15 },
	hostpci0 => { bus => 0, addr => 16 },
	hostpci1 => { bus => 0, addr => 17 },
	net0 => { bus => 0, addr => 18 },
	net1 => { bus => 0, addr => 19 },
	net2 => { bus => 0, addr => 20 },
	net3 => { bus => 0, addr => 21 },
	net4 => { bus => 0, addr => 22 },
	net5 => { bus => 0, addr => 23 },
	vga1 => { bus => 0, addr => 24 },
	vga2 => { bus => 0, addr => 25 },
	vga3 => { bus => 0, addr => 26 },
	hostpci2 => { bus => 0, addr => 27 },
	hostpci3 => { bus => 0, addr => 28 },
	#addr29 : usb-host (pve-usb.cfg)
	'pci.1' => { bus => 0, addr => 30 },
	'pci.2' => { bus => 0, addr => 31 },
	'net6' => { bus => 1, addr => 1 },
	'net7' => { bus => 1, addr => 2 },
	'net8' => { bus => 1, addr => 3 },
	'net9' => { bus => 1, addr => 4 },
	'net10' => { bus => 1, addr => 5 },
	'net11' => { bus => 1, addr => 6 },
	'net12' => { bus => 1, addr => 7 },
	'net13' => { bus => 1, addr => 8 },
	'net14' => { bus => 1, addr => 9 },
	'net15' => { bus => 1, addr => 10 },
	'net16' => { bus => 1, addr => 11 },
	'net17' => { bus => 1, addr => 12 },
	'net18' => { bus => 1, addr => 13 },
	'net19' => { bus => 1, addr => 14 },
	'net20' => { bus => 1, addr => 15 },
	'net21' => { bus => 1, addr => 16 },
	'net22' => { bus => 1, addr => 17 },
	'net23' => { bus => 1, addr => 18 },
	'net24' => { bus => 1, addr => 19 },
	'net25' => { bus => 1, addr => 20 },
	'net26' => { bus => 1, addr => 21 },
	'net27' => { bus => 1, addr => 22 },
	'net28' => { bus => 1, addr => 23 },
	'net29' => { bus => 1, addr => 24 },
	'net30' => { bus => 1, addr => 25 },
	'net31' => { bus => 1, addr => 26 },
	'xhci' => { bus => 1, addr => 27 },
	'pci.4' => { bus => 1, addr => 28 },
	'rng0' => { bus => 1, addr => 29 },
	'pci.2-igd' => { bus => 1, addr => 30 }, # replaces pci.2 in
case a legacy IGD device is passed through
	'virtio6' => { bus => 2, addr => 1 },
	'virtio7' => { bus => 2, addr => 2 },
	'virtio8' => { bus => 2, addr => 3 },
	'virtio9' => { bus => 2, addr => 4 },
	'virtio10' => { bus => 2, addr => 5 },
	'virtio11' => { bus => 2, addr => 6 },
	'virtio12' => { bus => 2, addr => 7 },
	'virtio13' => { bus => 2, addr => 8 },
	'virtio14' => { bus => 2, addr => 9 },
	'virtio15' => { bus => 2, addr => 10 },
	'ivshmem' => { bus => 2, addr => 11 },
	'audio0' => { bus => 2, addr => 12 },
	hostpci4 => { bus => 2, addr => 13 },
	hostpci5 => { bus => 2, addr => 14 },
	hostpci6 => { bus => 2, addr => 15 },
	hostpci7 => { bus => 2, addr => 16 },
	hostpci8 => { bus => 2, addr => 17 },
	hostpci9 => { bus => 2, addr => 18 },
	hostpci10 => { bus => 2, addr => 19 },
	hostpci11 => { bus => 2, addr => 20 },
	hostpci12 => { bus => 2, addr => 21 },
	hostpci13 => { bus => 2, addr => 22 },
	hostpci14 => { bus => 2, addr => 23 },
	hostpci15 => { bus => 2, addr => 24 },
	'virtioscsi0' => { bus => 3, addr => 1 },
	'virtioscsi1' => { bus => 3, addr => 2 },
	'virtioscsi2' => { bus => 3, addr => 3 },
	'virtioscsi3' => { bus => 3, addr => 4 },
	'virtioscsi4' => { bus => 3, addr => 5 },
	'virtioscsi5' => { bus => 3, addr => 6 },
	'virtioscsi6' => { bus => 3, addr => 7 },
	'virtioscsi7' => { bus => 3, addr => 8 },
	'virtioscsi8' => { bus => 3, addr => 9 },
	'virtioscsi9' => { bus => 3, addr => 10 },
	'virtioscsi10' => { bus => 3, addr => 11 },
	'virtioscsi11' => { bus => 3, addr => 12 },
	'virtioscsi12' => { bus => 3, addr => 13 },
	'virtioscsi13' => { bus => 3, addr => 14 },
	'virtioscsi14' => { bus => 3, addr => 15 },
	'virtioscsi15' => { bus => 3, addr => 16 },
	'virtioscsi16' => { bus => 3, addr => 17 },
	'virtioscsi17' => { bus => 3, addr => 18 },
	'virtioscsi18' => { bus => 3, addr => 19 },
	'virtioscsi19' => { bus => 3, addr => 20 },
	'virtioscsi20' => { bus => 3, addr => 21 },
	'virtioscsi21' => { bus => 3, addr => 22 },
	'virtioscsi22' => { bus => 3, addr => 23 },
	'virtioscsi23' => { bus => 3, addr => 24 },
	'virtioscsi24' => { bus => 3, addr => 25 },
	'virtioscsi25' => { bus => 3, addr => 26 },
	'virtioscsi26' => { bus => 3, addr => 27 },
	'virtioscsi27' => { bus => 3, addr => 28 },
	'virtioscsi28' => { bus => 3, addr => 29 },
	'virtioscsi29' => { bus => 3, addr => 30 },
	'virtioscsi30' => { bus => 3, addr => 31 },
	'scsihw2' => { bus => 4, addr => 1 },
	'scsihw3' => { bus => 4, addr => 2 },
	'scsihw4' => { bus => 4, addr => 3 },
    } if !defined($pci_addr_map);
    return $pci_addr_map;

> 
> > > Alexandre,
> > > 
> > > the statement below is not true for our case. The OpenVMS guest OS is
> > > using a PCIE bus, so the virtio-scsi device should be exposed as
> > > "modern", but is not. Not sure why not at this point
> > 
> 
> See Fiona response,
> 
> the pci express bridge is present, but the virtio-net is plugged on a
> simple pci bridge.
> ²
> pci express slots are mostly used for passthrough devices
> 
> 
> 
> my $pcie_addr_map;
> sub get_pcie_addr_map {
>     $pcie_addr_map = {
>         vga => { bus => 'pcie.0', addr => 1 },
>         hostpci0 => { bus => "ich9-pcie-port-1", addr => 0 },
>         hostpci1 => { bus => "ich9-pcie-port-2", addr => 0 },
>         hostpci2 => { bus => "ich9-pcie-port-3", addr => 0 },
>         hostpci3 => { bus => "ich9-pcie-port-4", addr => 0 },
>         hostpci4 => { bus => "ich9-pcie-port-5", addr => 0 },
>         hostpci5 => { bus => "ich9-pcie-port-6", addr => 0 },
>         hostpci6 => { bus => "ich9-pcie-port-7", addr => 0 },
>         hostpci7 => { bus => "ich9-pcie-port-8", addr => 0 },
>         hostpci8 => { bus => "ich9-pcie-port-9", addr => 0 },
>         hostpci9 => { bus => "ich9-pcie-port-10", addr => 0 },
>         hostpci10 => { bus => "ich9-pcie-port-11", addr => 0 },
>         hostpci11 => { bus => "ich9-pcie-port-12", addr => 0 },
>         hostpci12 => { bus => "ich9-pcie-port-13", addr => 0 },
>         hostpci13 => { bus => "ich9-pcie-port-14", addr => 0 },
>         hostpci14 => { bus => "ich9-pcie-port-15", addr => 0 },
>         hostpci15 => { bus => "ich9-pcie-port-16", addr => 0 },
>         # win7 is picky about pcie assignments
>         hostpci0bus0 => { bus => "pcie.0", addr => 16 },
>         hostpci1bus0 => { bus => "pcie.0", addr => 17 },
>         hostpci2bus0 => { bus => "pcie.0", addr => 18 },
>         hostpci3bus0 => { bus => "pcie.0", addr => 19 },
>         ivshmem => { bus => 'pcie.0', addr => 20 },
>         hostpci4bus0 => { bus => "pcie.0", addr => 9 },
>         hostpci5bus0 => { bus => "pcie.0", addr => 10 },
>         hostpci6bus0 => { bus => "pcie.0", addr => 11 },
>         hostpci7bus0 => { bus => "pcie.0", addr => 12 },
>         hostpci8bus0 => { bus => "pcie.0", addr => 13 },
>         hostpci9bus0 => { bus => "pcie.0", addr => 14 },
>         hostpci10bus0 => { bus => "pcie.0", addr => 15 },
>         hostpci11bus0 => { bus => "pcie.0", addr => 21 },
>         hostpci12bus0 => { bus => "pcie.0", addr => 22 },
>         hostpci13bus0 => { bus => "pcie.0", addr => 23 },
>         hostpci14bus0 => { bus => "pcie.0", addr => 24 },
>         hostpci15bus0 => { bus => "pcie.0", addr => 25 },
> }
> 
> 
> Christian Moser
> Mobile:    +358-40-5022105   
> Email:     cmos@maklee.com
> URL:       www.maklee.com
> -----Original Message-----
> From: DERUMIER, Alexandre <alexandre.derumier@groupe-cyllene.com> 
> Sent: Wednesday, August 14, 2024 09:45
> To: pve-devel@lists.proxmox.com; cmos@maklee.com
> Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
> 
> Hi,
> 
> I didn't see the responde of Fiona but indeed:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg01567.html
> 
> "virtio devices can be exposed in upto three ways
> 
>  - Legacy - follows virtio 0.9 specification. always uses PCI
>             ID range 0x1000-0x103F
>  - Transitional - follows virtio 0.9 specification by default, but
>                   can auto-negotiate with guest for 1.0 spce. Always
>                   uses PCI ID range 0x1000-0x103F
>  - Modern - follows virtio 1.0 specification. always uses PCI
>             ID range 0x1040-0x107F
> 
> With QEMU, historically devices placed on a PCI bus will always default
> to being in transitional mode, while devices placed on a PCI-E bus will
> always dfault to being in modern mode.
> "
> 
> 
> -------- Message initial --------
> De: Fiona Ebner <f.ebner@proxmox.com>
> Répondre à: Proxmox VE development discussion <pve-
> devel@lists.proxmox.com>
> À: Proxmox VE development discussion <pve-devel@lists.proxmox.com>,
> Christian Moser <cmos@maklee.com>
> Objet: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
> Date: 13/08/2024 10:55:47
> 
> Hi,
> 
> Am 12.08.24 um 12:40 schrieb Christian Moser:
> >  Hello,
> >  
> >  I work for VSI (VMS Software Inc) which is porting the OpenVMS 
> >  operating system to x86. At this point we successfully on various 
> >  hypervisors, but have some issues on the KVM running on Proxmox.
> >  
> >  The OpenVMS VM works just fine with SATA disks and it also works with
> >  for example virtio-network device etc., but trying to use virtio-scsi
> >  hangs  the driver. I have debugged this and I can successfully 
> >  configure the port/controller, send the IO request to the device. It 
> >  then gets processed by the device, which posts the results and sets 
> >  the interrupt bit in the ISR register, but it never asserts the 
> >  interrupt hence the driver never gets notified and the I/O hangs.
> >  
> >  I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no 
> >  luck. The emulated virtio-scsi device is a legacy device. But then 
> >  again, the virtio-network device is also a legacy device and here we 
> >  are getting interrupts. One thing which bothers me is the fact that 
> >  the “legacy interrupt disable” bit is set in the PCI config space of 
> >  the virtio-scsi device (i.e. bit 10 at offset 4)
> >  
> >  Questions:
> >  * is there a way to configure a modern virtio-scsi devcie (i.e.
> >  disable_legacy=on) ?
> > 
> 
> I've already answered this question when you asked in a mail addressed
> directly to me:
> 
> Am 12.08.24 um 11:58 schrieb Fiona Ebner:
> >  Hi,
> >  
> >  It seems that you either need to attach the "virtio-scsi-pci" device 
> >  to a pcie bus or explicitly set the "disable_legacy=on" option for
> >  the 
> >  device, neither of which Proxmox VE currently does or allows 
> >  configuring. The only way right now seems to be to attach the disk 
> >  yourself via custom arguments (the 'args' in the Proxmox VE VM 
> >  configuration), but then the disk will be invisible to Proxmox VE 
> >  operations which look at specific disk keys in the configuration!
> >  
> >  Feel free to open a feature request on our bug tracker to make this
> >  configurable:
> >  https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr
> >  aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj-
> >  1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob
> >  kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=1aab5a2ada7
> >  3beb46aa02df4e18ff6c5ba2db6d6ff2d1f302a3c4c83b13c8ef6&u=https%3A%2F%2
> >  Fbugzilla.proxmox.com%2F
> >  
> >  P.S. Please write to the developer list rather than individual 
> >  developers for such questions in the feature:
> >  https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr
> >  aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj-
> >  1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob
> >  kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c87
> >  16b5426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2
> >  Flists.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel
> >  
> >  Best Regards,
> >  Fiona
> > 
> 
> >  * why is the legacy interrupt bit set in the PCI config space ?
> > 
> 
> Most likely because the virtio-scsi-pci is configured without the
> "disable_legacy=on" option. If not explicitly set, the option will be
> "disable_legacy=auto" and when not attached to PCIe (which is the case
> for Proxmox VE currently), then legacy mode will be enabled.
> 
> >  * Are there any working driver for virtio-scsi on this KVM using Q35 
> >  machine? i.e. any other OS
> > 
> 
> The virtio drivers for Windows and the ones in Linux work just fine
> with our configuration.
> 
> 
> >  Any thoughts why these interrupts are not getting delivered on the 
> >  PCIE bus?
> > 
> 
> We do not configure the virtio-scsi-pci on a PCIe bus currently, see my
> initial answer.
> 
> Best Regards,
> Fiona
> 
> 
> 
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTraK
> 
> 7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj-
> 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1RobkV
> 
> xVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c8716b5
> 
> 426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2Flists
> 
> .proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel
> 
> 
> 


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel