all lists on lists.proxmox.com
 help / color / mirror / Atom feed
From: "DERUMIER, Alexandre via pve-devel" <pve-devel@lists.proxmox.com>
To: "cmos@maklee.com" <cmos@maklee.com>
Cc: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>,
	"pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: [pve-devel] FW:  issues with Virtio-SCSI devicde on Proxmox...
Date: Wed, 14 Aug 2024 13:05:35 +0000	[thread overview]
Message-ID: <mailman.261.1723640743.302.pve-devel@lists.proxmox.com> (raw)
In-Reply-To: <0ee0c7d5-e2e3-43ae-b77f-18f4ece29e0a@maklee.com>

[-- Attachment #1: Type: message/rfc822, Size: 33368 bytes --]

From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com>
To: "cmos@maklee.com" <cmos@maklee.com>
Cc: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>
Subject: Re: FW: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
Date: Wed, 14 Aug 2024 13:05:35 +0000
Message-ID: <6ea8c90b469da48525ed743b6541bb3644be4a6c.camel@groupe-cyllene.com>

>>I'm talking about virtio-scsi. Our virtio-network device is working
>>fine 

Yes, sorry, I wanted to said virtio-scsi.   

All pci devices excluding passthrough devices (with pcie=on flag) are
actually plugged to pci bridge


sub get_pci_addr_map {
    $pci_addr_map = {
	piix3 => { bus => 0, addr => 1, conflict_ok => qw(ehci)  },
	ehci => { bus => 0, addr => 1, conflict_ok => qw(piix3) }, #
instead of piix3 on arm
	vga => { bus => 0, addr => 2, conflict_ok => qw(legacy-igd) },
	'legacy-igd' => { bus => 0, addr => 2, conflict_ok => qw(vga)
}, # legacy-igd requires vga=none
	balloon0 => { bus => 0, addr => 3 },
	watchdog => { bus => 0, addr => 4 },
	scsihw0 => { bus => 0, addr => 5, conflict_ok => qw(pci.3) },
	'pci.3' => { bus => 0, addr => 5, conflict_ok => qw(scsihw0)
}, # also used for virtio-scsi-single bridge
	scsihw1 => { bus => 0, addr => 6 },
	ahci0 => { bus => 0, addr => 7 },
	qga0 => { bus => 0, addr => 8 },
	spice => { bus => 0, addr => 9 },
	virtio0 => { bus => 0, addr => 10 },
	virtio1 => { bus => 0, addr => 11 },
	virtio2 => { bus => 0, addr => 12 },
	virtio3 => { bus => 0, addr => 13 },
	virtio4 => { bus => 0, addr => 14 },
	virtio5 => { bus => 0, addr => 15 },
	hostpci0 => { bus => 0, addr => 16 },
	hostpci1 => { bus => 0, addr => 17 },
	net0 => { bus => 0, addr => 18 },
	net1 => { bus => 0, addr => 19 },
	net2 => { bus => 0, addr => 20 },
	net3 => { bus => 0, addr => 21 },
	net4 => { bus => 0, addr => 22 },
	net5 => { bus => 0, addr => 23 },
	vga1 => { bus => 0, addr => 24 },
	vga2 => { bus => 0, addr => 25 },
	vga3 => { bus => 0, addr => 26 },
	hostpci2 => { bus => 0, addr => 27 },
	hostpci3 => { bus => 0, addr => 28 },
	#addr29 : usb-host (pve-usb.cfg)
	'pci.1' => { bus => 0, addr => 30 },
	'pci.2' => { bus => 0, addr => 31 },
	'net6' => { bus => 1, addr => 1 },
	'net7' => { bus => 1, addr => 2 },
	'net8' => { bus => 1, addr => 3 },
	'net9' => { bus => 1, addr => 4 },
	'net10' => { bus => 1, addr => 5 },
	'net11' => { bus => 1, addr => 6 },
	'net12' => { bus => 1, addr => 7 },
	'net13' => { bus => 1, addr => 8 },
	'net14' => { bus => 1, addr => 9 },
	'net15' => { bus => 1, addr => 10 },
	'net16' => { bus => 1, addr => 11 },
	'net17' => { bus => 1, addr => 12 },
	'net18' => { bus => 1, addr => 13 },
	'net19' => { bus => 1, addr => 14 },
	'net20' => { bus => 1, addr => 15 },
	'net21' => { bus => 1, addr => 16 },
	'net22' => { bus => 1, addr => 17 },
	'net23' => { bus => 1, addr => 18 },
	'net24' => { bus => 1, addr => 19 },
	'net25' => { bus => 1, addr => 20 },
	'net26' => { bus => 1, addr => 21 },
	'net27' => { bus => 1, addr => 22 },
	'net28' => { bus => 1, addr => 23 },
	'net29' => { bus => 1, addr => 24 },
	'net30' => { bus => 1, addr => 25 },
	'net31' => { bus => 1, addr => 26 },
	'xhci' => { bus => 1, addr => 27 },
	'pci.4' => { bus => 1, addr => 28 },
	'rng0' => { bus => 1, addr => 29 },
	'pci.2-igd' => { bus => 1, addr => 30 }, # replaces pci.2 in
case a legacy IGD device is passed through
	'virtio6' => { bus => 2, addr => 1 },
	'virtio7' => { bus => 2, addr => 2 },
	'virtio8' => { bus => 2, addr => 3 },
	'virtio9' => { bus => 2, addr => 4 },
	'virtio10' => { bus => 2, addr => 5 },
	'virtio11' => { bus => 2, addr => 6 },
	'virtio12' => { bus => 2, addr => 7 },
	'virtio13' => { bus => 2, addr => 8 },
	'virtio14' => { bus => 2, addr => 9 },
	'virtio15' => { bus => 2, addr => 10 },
	'ivshmem' => { bus => 2, addr => 11 },
	'audio0' => { bus => 2, addr => 12 },
	hostpci4 => { bus => 2, addr => 13 },
	hostpci5 => { bus => 2, addr => 14 },
	hostpci6 => { bus => 2, addr => 15 },
	hostpci7 => { bus => 2, addr => 16 },
	hostpci8 => { bus => 2, addr => 17 },
	hostpci9 => { bus => 2, addr => 18 },
	hostpci10 => { bus => 2, addr => 19 },
	hostpci11 => { bus => 2, addr => 20 },
	hostpci12 => { bus => 2, addr => 21 },
	hostpci13 => { bus => 2, addr => 22 },
	hostpci14 => { bus => 2, addr => 23 },
	hostpci15 => { bus => 2, addr => 24 },
	'virtioscsi0' => { bus => 3, addr => 1 },
	'virtioscsi1' => { bus => 3, addr => 2 },
	'virtioscsi2' => { bus => 3, addr => 3 },
	'virtioscsi3' => { bus => 3, addr => 4 },
	'virtioscsi4' => { bus => 3, addr => 5 },
	'virtioscsi5' => { bus => 3, addr => 6 },
	'virtioscsi6' => { bus => 3, addr => 7 },
	'virtioscsi7' => { bus => 3, addr => 8 },
	'virtioscsi8' => { bus => 3, addr => 9 },
	'virtioscsi9' => { bus => 3, addr => 10 },
	'virtioscsi10' => { bus => 3, addr => 11 },
	'virtioscsi11' => { bus => 3, addr => 12 },
	'virtioscsi12' => { bus => 3, addr => 13 },
	'virtioscsi13' => { bus => 3, addr => 14 },
	'virtioscsi14' => { bus => 3, addr => 15 },
	'virtioscsi15' => { bus => 3, addr => 16 },
	'virtioscsi16' => { bus => 3, addr => 17 },
	'virtioscsi17' => { bus => 3, addr => 18 },
	'virtioscsi18' => { bus => 3, addr => 19 },
	'virtioscsi19' => { bus => 3, addr => 20 },
	'virtioscsi20' => { bus => 3, addr => 21 },
	'virtioscsi21' => { bus => 3, addr => 22 },
	'virtioscsi22' => { bus => 3, addr => 23 },
	'virtioscsi23' => { bus => 3, addr => 24 },
	'virtioscsi24' => { bus => 3, addr => 25 },
	'virtioscsi25' => { bus => 3, addr => 26 },
	'virtioscsi26' => { bus => 3, addr => 27 },
	'virtioscsi27' => { bus => 3, addr => 28 },
	'virtioscsi28' => { bus => 3, addr => 29 },
	'virtioscsi29' => { bus => 3, addr => 30 },
	'virtioscsi30' => { bus => 3, addr => 31 },
	'scsihw2' => { bus => 4, addr => 1 },
	'scsihw3' => { bus => 4, addr => 2 },
	'scsihw4' => { bus => 4, addr => 3 },
    } if !defined($pci_addr_map);
    return $pci_addr_map;

> 
> > > Alexandre,
> > > 
> > > the statement below is not true for our case. The OpenVMS guest OS is
> > > using a PCIE bus, so the virtio-scsi device should be exposed as
> > > "modern", but is not. Not sure why not at this point
> > 
> 
> See Fiona response,
> 
> the pci express bridge is present, but the virtio-net is plugged on a
> simple pci bridge.
> ²
> pci express slots are mostly used for passthrough devices
> 
> 
> 
> my $pcie_addr_map;
> sub get_pcie_addr_map {
>     $pcie_addr_map = {
>         vga => { bus => 'pcie.0', addr => 1 },
>         hostpci0 => { bus => "ich9-pcie-port-1", addr => 0 },
>         hostpci1 => { bus => "ich9-pcie-port-2", addr => 0 },
>         hostpci2 => { bus => "ich9-pcie-port-3", addr => 0 },
>         hostpci3 => { bus => "ich9-pcie-port-4", addr => 0 },
>         hostpci4 => { bus => "ich9-pcie-port-5", addr => 0 },
>         hostpci5 => { bus => "ich9-pcie-port-6", addr => 0 },
>         hostpci6 => { bus => "ich9-pcie-port-7", addr => 0 },
>         hostpci7 => { bus => "ich9-pcie-port-8", addr => 0 },
>         hostpci8 => { bus => "ich9-pcie-port-9", addr => 0 },
>         hostpci9 => { bus => "ich9-pcie-port-10", addr => 0 },
>         hostpci10 => { bus => "ich9-pcie-port-11", addr => 0 },
>         hostpci11 => { bus => "ich9-pcie-port-12", addr => 0 },
>         hostpci12 => { bus => "ich9-pcie-port-13", addr => 0 },
>         hostpci13 => { bus => "ich9-pcie-port-14", addr => 0 },
>         hostpci14 => { bus => "ich9-pcie-port-15", addr => 0 },
>         hostpci15 => { bus => "ich9-pcie-port-16", addr => 0 },
>         # win7 is picky about pcie assignments
>         hostpci0bus0 => { bus => "pcie.0", addr => 16 },
>         hostpci1bus0 => { bus => "pcie.0", addr => 17 },
>         hostpci2bus0 => { bus => "pcie.0", addr => 18 },
>         hostpci3bus0 => { bus => "pcie.0", addr => 19 },
>         ivshmem => { bus => 'pcie.0', addr => 20 },
>         hostpci4bus0 => { bus => "pcie.0", addr => 9 },
>         hostpci5bus0 => { bus => "pcie.0", addr => 10 },
>         hostpci6bus0 => { bus => "pcie.0", addr => 11 },
>         hostpci7bus0 => { bus => "pcie.0", addr => 12 },
>         hostpci8bus0 => { bus => "pcie.0", addr => 13 },
>         hostpci9bus0 => { bus => "pcie.0", addr => 14 },
>         hostpci10bus0 => { bus => "pcie.0", addr => 15 },
>         hostpci11bus0 => { bus => "pcie.0", addr => 21 },
>         hostpci12bus0 => { bus => "pcie.0", addr => 22 },
>         hostpci13bus0 => { bus => "pcie.0", addr => 23 },
>         hostpci14bus0 => { bus => "pcie.0", addr => 24 },
>         hostpci15bus0 => { bus => "pcie.0", addr => 25 },
> }
> 
> 
> Christian Moser
> Mobile:    +358-40-5022105   
> Email:     cmos@maklee.com
> URL:       www.maklee.com
> -----Original Message-----
> From: DERUMIER, Alexandre <alexandre.derumier@groupe-cyllene.com> 
> Sent: Wednesday, August 14, 2024 09:45
> To: pve-devel@lists.proxmox.com; cmos@maklee.com
> Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
> 
> Hi,
> 
> I didn't see the responde of Fiona but indeed:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg01567.html
> 
> "virtio devices can be exposed in upto three ways
> 
>  - Legacy - follows virtio 0.9 specification. always uses PCI
>             ID range 0x1000-0x103F
>  - Transitional - follows virtio 0.9 specification by default, but
>                   can auto-negotiate with guest for 1.0 spce. Always
>                   uses PCI ID range 0x1000-0x103F
>  - Modern - follows virtio 1.0 specification. always uses PCI
>             ID range 0x1040-0x107F
> 
> With QEMU, historically devices placed on a PCI bus will always default
> to being in transitional mode, while devices placed on a PCI-E bus will
> always dfault to being in modern mode.
> "
> 
> 
> -------- Message initial --------
> De: Fiona Ebner <f.ebner@proxmox.com>
> Répondre à: Proxmox VE development discussion <pve-
> devel@lists.proxmox.com>
> À: Proxmox VE development discussion <pve-devel@lists.proxmox.com>,
> Christian Moser <cmos@maklee.com>
> Objet: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
> Date: 13/08/2024 10:55:47
> 
> Hi,
> 
> Am 12.08.24 um 12:40 schrieb Christian Moser:
> >  Hello,
> >  
> >  I work for VSI (VMS Software Inc) which is porting the OpenVMS 
> >  operating system to x86. At this point we successfully on various 
> >  hypervisors, but have some issues on the KVM running on Proxmox.
> >  
> >  The OpenVMS VM works just fine with SATA disks and it also works with
> >  for example virtio-network device etc., but trying to use virtio-scsi
> >  hangs  the driver. I have debugged this and I can successfully 
> >  configure the port/controller, send the IO request to the device. It 
> >  then gets processed by the device, which posts the results and sets 
> >  the interrupt bit in the ISR register, but it never asserts the 
> >  interrupt hence the driver never gets notified and the I/O hangs.
> >  
> >  I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no 
> >  luck. The emulated virtio-scsi device is a legacy device. But then 
> >  again, the virtio-network device is also a legacy device and here we 
> >  are getting interrupts. One thing which bothers me is the fact that 
> >  the “legacy interrupt disable” bit is set in the PCI config space of 
> >  the virtio-scsi device (i.e. bit 10 at offset 4)
> >  
> >  Questions:
> >  * is there a way to configure a modern virtio-scsi devcie (i.e.
> >  disable_legacy=on) ?
> > 
> 
> I've already answered this question when you asked in a mail addressed
> directly to me:
> 
> Am 12.08.24 um 11:58 schrieb Fiona Ebner:
> >  Hi,
> >  
> >  It seems that you either need to attach the "virtio-scsi-pci" device 
> >  to a pcie bus or explicitly set the "disable_legacy=on" option for
> >  the 
> >  device, neither of which Proxmox VE currently does or allows 
> >  configuring. The only way right now seems to be to attach the disk 
> >  yourself via custom arguments (the 'args' in the Proxmox VE VM 
> >  configuration), but then the disk will be invisible to Proxmox VE 
> >  operations which look at specific disk keys in the configuration!
> >  
> >  Feel free to open a feature request on our bug tracker to make this
> >  configurable:
> >  https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr
> >  aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj-
> >  1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob
> >  kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=1aab5a2ada7
> >  3beb46aa02df4e18ff6c5ba2db6d6ff2d1f302a3c4c83b13c8ef6&u=https%3A%2F%2
> >  Fbugzilla.proxmox.com%2F
> >  
> >  P.S. Please write to the developer list rather than individual 
> >  developers for such questions in the feature:
> >  https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr
> >  aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj-
> >  1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob
> >  kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c87
> >  16b5426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2
> >  Flists.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel
> >  
> >  Best Regards,
> >  Fiona
> > 
> 
> >  * why is the legacy interrupt bit set in the PCI config space ?
> > 
> 
> Most likely because the virtio-scsi-pci is configured without the
> "disable_legacy=on" option. If not explicitly set, the option will be
> "disable_legacy=auto" and when not attached to PCIe (which is the case
> for Proxmox VE currently), then legacy mode will be enabled.
> 
> >  * Are there any working driver for virtio-scsi on this KVM using Q35 
> >  machine? i.e. any other OS
> > 
> 
> The virtio drivers for Windows and the ones in Linux work just fine
> with our configuration.
> 
> 
> >  Any thoughts why these interrupts are not getting delivered on the 
> >  PCIE bus?
> > 
> 
> We do not configure the virtio-scsi-pci on a PCIe bus currently, see my
> initial answer.
> 
> Best Regards,
> Fiona
> 
> 
> 
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTraK
> 
> 7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj-
> 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1RobkV
> 
> xVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c8716b5
> 
> 426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2Flists
> 
> .proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel
> 
> 
> 


[-- Attachment #2: Type: text/plain, Size: 160 bytes --]

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

  reply	other threads:[~2024-08-14 13:05 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-08-12 10:40 [pve-devel] " Christian Moser
2024-08-13  8:55 ` Fiona Ebner
2024-08-13  9:33   ` [pve-devel] FW: " Christian Moser
2024-08-14  6:44   ` [pve-devel] " DERUMIER, Alexandre via pve-devel
     [not found]   ` <1178cb3475d719ae31f0c375cd3930fc24d98401.camel@groupe-cyllene.com>
2024-08-14  7:20     ` [pve-devel] FW: " Christian Moser
2024-08-14 12:22       ` DERUMIER, Alexandre via pve-devel
     [not found]       ` <c851a27f4f5eca7c767fbffa48cb208e2d8fe1e6.camel@groupe-cyllene.com>
2024-08-14 12:48         ` Christian Moser
2024-08-14 13:05           ` DERUMIER, Alexandre via pve-devel [this message]
     [not found]           ` <6ea8c90b469da48525ed743b6541bb3644be4a6c.camel@groupe-cyllene.com>
2024-08-15  5:35             ` [pve-devel] FW: " Christian Moser
2024-08-13 16:12 ` [pve-devel] " DERUMIER, Alexandre via pve-devel
     [not found] ` <67b222322506a9eab0f7cf7da5a9fd715c8a91ff.camel@groupe-cyllene.com>
2024-08-14  6:42   ` [pve-devel] FW: " Christian Moser

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=mailman.261.1723640743.302.pve-devel@lists.proxmox.com \
    --to=pve-devel@lists.proxmox.com \
    --cc=alexandre.derumier@groupe-cyllene.com \
    --cc=cmos@maklee.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal