* [pve-devel] issues with Virtio-SCSI devicde on Proxmox...
@ 2024-08-12 10:40 Christian Moser
2024-08-13 8:55 ` Fiona Ebner
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Christian Moser @ 2024-08-12 10:40 UTC (permalink / raw)
To: pve-devel; +Cc: Christian Moser
Hello,
I work for VSI (VMS Software Inc) which is porting the OpenVMS operating system to x86. At this point we successfully on various
hypervisors, but have some issues on the KVM running on Proxmox.
The OpenVMS VM works just fine with SATA disks and it also works with for example virtio-network device etc., but trying to use
virtio-scsi hangs the driver. I have debugged this and I can successfully configure the port/controller, send the IO request to the
device. It then gets processed by the device, which posts the results and sets the interrupt bit in the ISR register, but it never
asserts the interrupt hence the driver never gets notified and the I/O hangs.
I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no luck. The emulated virtio-scsi device is a legacy device. But
then again, the virtio-network device is also a legacy device and here we are getting interrupts. One thing which bothers me is the
fact that the “legacy interrupt disable” bit is set in the PCI config space of the virtio-scsi device (i.e. bit 10 at offset 4)
Questions:
* is there a way to configure a modern virtio-scsi devcie (i.e. disable_legacy=on) ?
* why is the legacy interrupt bit set in the PCI config space ?
* Are there any working driver for virtio-scsi on this KVM using Q35 machine? i.e. any other OS
Any thoughts why these interrupts are not getting delivered on the PCIE bus?
thanks
/cmos
_______________________________________________________
Christian Moser
Mobile: +358-40-5022105
Email: cmos@maklee.com
URL: www.maklee.com
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... 2024-08-12 10:40 [pve-devel] issues with Virtio-SCSI devicde on Proxmox Christian Moser @ 2024-08-13 8:55 ` Fiona Ebner 2024-08-13 9:33 ` [pve-devel] FW: " Christian Moser ` (2 more replies) 2024-08-13 16:12 ` [pve-devel] " DERUMIER, Alexandre via pve-devel [not found] ` <67b222322506a9eab0f7cf7da5a9fd715c8a91ff.camel@groupe-cyllene.com> 2 siblings, 3 replies; 11+ messages in thread From: Fiona Ebner @ 2024-08-13 8:55 UTC (permalink / raw) To: Proxmox VE development discussion, Christian Moser Hi, Am 12.08.24 um 12:40 schrieb Christian Moser: > Hello, > > I work for VSI (VMS Software Inc) which is porting the OpenVMS operating system to x86. At this point we successfully on various > hypervisors, but have some issues on the KVM running on Proxmox. > > The OpenVMS VM works just fine with SATA disks and it also works with for example virtio-network device etc., but trying to use > virtio-scsi hangs the driver. I have debugged this and I can successfully configure the port/controller, send the IO request to the > device. It then gets processed by the device, which posts the results and sets the interrupt bit in the ISR register, but it never > asserts the interrupt hence the driver never gets notified and the I/O hangs. > > I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no luck. The emulated virtio-scsi device is a legacy device. But > then again, the virtio-network device is also a legacy device and here we are getting interrupts. One thing which bothers me is the > fact that the “legacy interrupt disable” bit is set in the PCI config space of the virtio-scsi device (i.e. bit 10 at offset 4) > > Questions: > * is there a way to configure a modern virtio-scsi devcie (i.e. disable_legacy=on) ? I've already answered this question when you asked in a mail addressed directly to me: Am 12.08.24 um 11:58 schrieb Fiona Ebner: > Hi, > > It seems that you either need to attach the "virtio-scsi-pci" device to > a pcie bus or explicitly set the "disable_legacy=on" option for the > device, neither of which Proxmox VE currently does or allows > configuring. The only way right now seems to be to attach the disk > yourself via custom arguments (the 'args' in the Proxmox VE VM > configuration), but then the disk will be invisible to Proxmox VE > operations which look at specific disk keys in the configuration! > > Feel free to open a feature request on our bug tracker to make this > configurable: https://bugzilla.proxmox.com/ > > P.S. Please write to the developer list rather than individual > developers for such questions in the feature: > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > Best Regards, > Fiona > * why is the legacy interrupt bit set in the PCI config space ? Most likely because the virtio-scsi-pci is configured without the "disable_legacy=on" option. If not explicitly set, the option will be "disable_legacy=auto" and when not attached to PCIe (which is the case for Proxmox VE currently), then legacy mode will be enabled. > * Are there any working driver for virtio-scsi on this KVM using Q35 machine? i.e. any other OS The virtio drivers for Windows and the ones in Linux work just fine with our configuration. > Any thoughts why these interrupts are not getting delivered on the PCIE bus? We do not configure the virtio-scsi-pci on a PCIe bus currently, see my initial answer. Best Regards, Fiona _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 11+ messages in thread
* [pve-devel] FW: issues with Virtio-SCSI devicde on Proxmox... 2024-08-13 8:55 ` Fiona Ebner @ 2024-08-13 9:33 ` Christian Moser 2024-08-14 6:44 ` [pve-devel] " DERUMIER, Alexandre via pve-devel [not found] ` <1178cb3475d719ae31f0c375cd3930fc24d98401.camel@groupe-cyllene.com> 2 siblings, 0 replies; 11+ messages in thread From: Christian Moser @ 2024-08-13 9:33 UTC (permalink / raw) To: 'Proxmox VE development discussion'; +Cc: Christian Moser Hello, what I have been trying to tell is that both virtio-scsi and virtio-network device on Proxmox are both getting configured on PCIE. I do not understand why you claim that virtio-scsi is not configured on PCIE. So both virtio-scsi and virtio-network devices on Proxmox are configured as legacy devices on PCIE and the virtio-network device works just fine and does get interrupts. I also believe that the virtio-scsi device is working fine and it does set the interrupt bit in the ISR register, but no interrupt gets delivered to the driver. I'm now trying to find out if the problem is on our side, i.e. that our PCIE "setup" is getting the interrupt, but not delivering it to the device. Or that we never got the interrupt and then the issue would most likely be in the Proxomox implementation. But your statement that other OS virtio-scsi devices are working fine might indicate the problem to be on our side. We do have a problem with interrupt delivery on Virtualbox (by Oracle) and they acknowledged an issue with bridges and interrupt routing, that's why I was contacting Proxmox to see if there could be an issue there too thanks /cmos _______________________________________________________ Christian Moser Mobile: +358-40-5022105 Email: cmos@maklee.com URL: www.maklee.com -----Original Message----- From: Fiona Ebner <f.ebner@proxmox.com> Sent: Tuesday, August 13, 2024 11:56 To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>; Christian Moser <cmos@maklee.com> Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Hi, Am 12.08.24 um 12:40 schrieb Christian Moser: > Hello, > > I work for VSI (VMS Software Inc) which is porting the OpenVMS > operating system to x86. At this point we successfully on various hypervisors, but have some issues on the KVM running on Proxmox. > > The OpenVMS VM works just fine with SATA disks and it also works with > for example virtio-network device etc., but trying to use virtio-scsi > hangs the driver. I have debugged this and I can successfully > configure the port/controller, send the IO request to the device. It then gets processed by the device, which posts the results and sets the interrupt bit in the ISR register, but it never asserts the interrupt hence the driver never gets notified and the I/O hangs. > > I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no > luck. The emulated virtio-scsi device is a legacy device. But then > again, the virtio-network device is also a legacy device and here we > are getting interrupts. One thing which bothers me is the fact that > the “legacy interrupt disable” bit is set in the PCI config space of > the virtio-scsi device (i.e. bit 10 at offset 4) > > Questions: > * is there a way to configure a modern virtio-scsi devcie (i.e. disable_legacy=on) ? I've already answered this question when you asked in a mail addressed directly to me: Am 12.08.24 um 11:58 schrieb Fiona Ebner: > Hi, > > It seems that you either need to attach the "virtio-scsi-pci" device > to a pcie bus or explicitly set the "disable_legacy=on" option for the > device, neither of which Proxmox VE currently does or allows > configuring. The only way right now seems to be to attach the disk > yourself via custom arguments (the 'args' in the Proxmox VE VM > configuration), but then the disk will be invisible to Proxmox VE > operations which look at specific disk keys in the configuration! > > Feel free to open a feature request on our bug tracker to make this > configurable: https://bugzilla.proxmox.com/ > > P.S. Please write to the developer list rather than individual > developers for such questions in the feature: > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > Best Regards, > Fiona > * why is the legacy interrupt bit set in the PCI config space ? Most likely because the virtio-scsi-pci is configured without the "disable_legacy=on" option. If not explicitly set, the option will be "disable_legacy=auto" and when not attached to PCIe (which is the case for Proxmox VE currently), then legacy mode will be enabled. > * Are there any working driver for virtio-scsi on this KVM using Q35 machine? i.e. any other OS The virtio drivers for Windows and the ones in Linux work just fine with our configuration. > Any thoughts why these interrupts are not getting delivered on the PCIE bus? We do not configure the virtio-scsi-pci on a PCIe bus currently, see my initial answer. Best Regards, Fiona _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... 2024-08-13 8:55 ` Fiona Ebner 2024-08-13 9:33 ` [pve-devel] FW: " Christian Moser @ 2024-08-14 6:44 ` DERUMIER, Alexandre via pve-devel [not found] ` <1178cb3475d719ae31f0c375cd3930fc24d98401.camel@groupe-cyllene.com> 2 siblings, 0 replies; 11+ messages in thread From: DERUMIER, Alexandre via pve-devel @ 2024-08-14 6:44 UTC (permalink / raw) To: pve-devel, cmos; +Cc: DERUMIER, Alexandre [-- Attachment #1: Type: message/rfc822, Size: 19942 bytes --] From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com> To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>, "cmos@maklee.com" <cmos@maklee.com> Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Date: Wed, 14 Aug 2024 06:44:36 +0000 Message-ID: <1178cb3475d719ae31f0c375cd3930fc24d98401.camel@groupe-cyllene.com> Hi, I didn't see the responde of Fiona but indeed: https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg01567.html "virtio devices can be exposed in upto three ways - Legacy - follows virtio 0.9 specification. always uses PCI ID range 0x1000-0x103F - Transitional - follows virtio 0.9 specification by default, but can auto-negotiate with guest for 1.0 spce. Always uses PCI ID range 0x1000-0x103F - Modern - follows virtio 1.0 specification. always uses PCI ID range 0x1040-0x107F With QEMU, historically devices placed on a PCI bus will always default to being in transitional mode, while devices placed on a PCI-E bus will always dfault to being in modern mode. " -------- Message initial -------- De: Fiona Ebner <f.ebner@proxmox.com> Répondre à: Proxmox VE development discussion <pve- devel@lists.proxmox.com> À: Proxmox VE development discussion <pve-devel@lists.proxmox.com>, Christian Moser <cmos@maklee.com> Objet: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Date: 13/08/2024 10:55:47 Hi, Am 12.08.24 um 12:40 schrieb Christian Moser: > Hello, > > I work for VSI (VMS Software Inc) which is porting the OpenVMS > operating system to x86. At this point we successfully on various > hypervisors, but have some issues on the KVM running on Proxmox. > > The OpenVMS VM works just fine with SATA disks and it also works with > for example virtio-network device etc., but trying to use > virtio-scsi hangs the driver. I have debugged this and I can > successfully configure the port/controller, send the IO request to > the > device. It then gets processed by the device, which posts the results > and sets the interrupt bit in the ISR register, but it never > asserts the interrupt hence the driver never gets notified and the > I/O hangs. > > I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no > luck. The emulated virtio-scsi device is a legacy device. But > then again, the virtio-network device is also a legacy device and > here we are getting interrupts. One thing which bothers me is the > fact that the “legacy interrupt disable” bit is set in the PCI config > space of the virtio-scsi device (i.e. bit 10 at offset 4) > > Questions: > * is there a way to configure a modern virtio-scsi devcie (i.e. > disable_legacy=on) ? I've already answered this question when you asked in a mail addressed directly to me: Am 12.08.24 um 11:58 schrieb Fiona Ebner: > Hi, > > It seems that you either need to attach the "virtio-scsi-pci" device > to > a pcie bus or explicitly set the "disable_legacy=on" option for the > device, neither of which Proxmox VE currently does or allows > configuring. The only way right now seems to be to attach the disk > yourself via custom arguments (the 'args' in the Proxmox VE VM > configuration), but then the disk will be invisible to Proxmox VE > operations which look at specific disk keys in the configuration! > > Feel free to open a feature request on our bug tracker to make this > configurable: > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=1aab5a2ada7 > 3beb46aa02df4e18ff6c5ba2db6d6ff2d1f302a3c4c83b13c8ef6&u=https%3A%2F%2 > Fbugzilla.proxmox.com%2F > > P.S. Please write to the developer list rather than individual > developers for such questions in the feature: > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c87 > 16b5426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2 > Flists.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel > > Best Regards, > Fiona > * why is the legacy interrupt bit set in the PCI config space ? Most likely because the virtio-scsi-pci is configured without the "disable_legacy=on" option. If not explicitly set, the option will be "disable_legacy=auto" and when not attached to PCIe (which is the case for Proxmox VE currently), then legacy mode will be enabled. > * Are there any working driver for virtio-scsi on this KVM using Q35 > machine? i.e. any other OS The virtio drivers for Windows and the ones in Linux work just fine with our configuration. > Any thoughts why these interrupts are not getting delivered on the > PCIE bus? We do not configure the virtio-scsi-pci on a PCIe bus currently, see my initial answer. Best Regards, Fiona _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTraK 7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1RobkV xVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c8716b5 426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2Flists .proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel [-- Attachment #2: Type: text/plain, Size: 160 bytes --] _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <1178cb3475d719ae31f0c375cd3930fc24d98401.camel@groupe-cyllene.com>]
* [pve-devel] FW: issues with Virtio-SCSI devicde on Proxmox... [not found] ` <1178cb3475d719ae31f0c375cd3930fc24d98401.camel@groupe-cyllene.com> @ 2024-08-14 7:20 ` Christian Moser 2024-08-14 12:22 ` DERUMIER, Alexandre via pve-devel [not found] ` <c851a27f4f5eca7c767fbffa48cb208e2d8fe1e6.camel@groupe-cyllene.com> 0 siblings, 2 replies; 11+ messages in thread From: Christian Moser @ 2024-08-14 7:20 UTC (permalink / raw) To: 'DERUMIER, Alexandre', pve-devel; +Cc: Christian Moser Alexandre, the statement below is not true for our case. The OpenVMS guest OS is using a PCIE bus, so the virtio-scsi device should be exposed as "modern", but is not. Not sure why not at this point /cmos _______________________________________________________ Christian Moser Mobile: +358-40-5022105 Email: cmos@maklee.com URL: www.maklee.com -----Original Message----- From: DERUMIER, Alexandre <alexandre.derumier@groupe-cyllene.com> Sent: Wednesday, August 14, 2024 09:45 To: pve-devel@lists.proxmox.com; cmos@maklee.com Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Hi, I didn't see the responde of Fiona but indeed: https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg01567.html "virtio devices can be exposed in upto three ways - Legacy - follows virtio 0.9 specification. always uses PCI ID range 0x1000-0x103F - Transitional - follows virtio 0.9 specification by default, but can auto-negotiate with guest for 1.0 spce. Always uses PCI ID range 0x1000-0x103F - Modern - follows virtio 1.0 specification. always uses PCI ID range 0x1040-0x107F With QEMU, historically devices placed on a PCI bus will always default to being in transitional mode, while devices placed on a PCI-E bus will always dfault to being in modern mode. " -------- Message initial -------- De: Fiona Ebner <f.ebner@proxmox.com> Répondre à: Proxmox VE development discussion <pve- devel@lists.proxmox.com> À: Proxmox VE development discussion <pve-devel@lists.proxmox.com>, Christian Moser <cmos@maklee.com> Objet: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Date: 13/08/2024 10:55:47 Hi, Am 12.08.24 um 12:40 schrieb Christian Moser: > Hello, > > I work for VSI (VMS Software Inc) which is porting the OpenVMS > operating system to x86. At this point we successfully on various > hypervisors, but have some issues on the KVM running on Proxmox. > > The OpenVMS VM works just fine with SATA disks and it also works with > for example virtio-network device etc., but trying to use virtio-scsi > hangs the driver. I have debugged this and I can successfully > configure the port/controller, send the IO request to the device. It > then gets processed by the device, which posts the results and sets > the interrupt bit in the ISR register, but it never asserts the > interrupt hence the driver never gets notified and the I/O hangs. > > I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no > luck. The emulated virtio-scsi device is a legacy device. But then > again, the virtio-network device is also a legacy device and here we > are getting interrupts. One thing which bothers me is the fact that > the “legacy interrupt disable” bit is set in the PCI config space of > the virtio-scsi device (i.e. bit 10 at offset 4) > > Questions: > * is there a way to configure a modern virtio-scsi devcie (i.e. > disable_legacy=on) ? I've already answered this question when you asked in a mail addressed directly to me: Am 12.08.24 um 11:58 schrieb Fiona Ebner: > Hi, > > It seems that you either need to attach the "virtio-scsi-pci" device > to a pcie bus or explicitly set the "disable_legacy=on" option for the > device, neither of which Proxmox VE currently does or allows > configuring. The only way right now seems to be to attach the disk > yourself via custom arguments (the 'args' in the Proxmox VE VM > configuration), but then the disk will be invisible to Proxmox VE > operations which look at specific disk keys in the configuration! > > Feel free to open a feature request on our bug tracker to make this > configurable: > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=1aab5a2ada7 > 3beb46aa02df4e18ff6c5ba2db6d6ff2d1f302a3c4c83b13c8ef6&u=https%3A%2F%2 > Fbugzilla.proxmox.com%2F > > P.S. Please write to the developer list rather than individual > developers for such questions in the feature: > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c87 > 16b5426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2 > Flists.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel > > Best Regards, > Fiona > * why is the legacy interrupt bit set in the PCI config space ? Most likely because the virtio-scsi-pci is configured without the "disable_legacy=on" option. If not explicitly set, the option will be "disable_legacy=auto" and when not attached to PCIe (which is the case for Proxmox VE currently), then legacy mode will be enabled. > * Are there any working driver for virtio-scsi on this KVM using Q35 > machine? i.e. any other OS The virtio drivers for Windows and the ones in Linux work just fine with our configuration. > Any thoughts why these interrupts are not getting delivered on the > PCIE bus? We do not configure the virtio-scsi-pci on a PCIe bus currently, see my initial answer. Best Regards, Fiona _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTraK 7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1RobkV xVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c8716b5 426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2Flists .proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [pve-devel] FW: issues with Virtio-SCSI devicde on Proxmox... 2024-08-14 7:20 ` [pve-devel] FW: " Christian Moser @ 2024-08-14 12:22 ` DERUMIER, Alexandre via pve-devel [not found] ` <c851a27f4f5eca7c767fbffa48cb208e2d8fe1e6.camel@groupe-cyllene.com> 1 sibling, 0 replies; 11+ messages in thread From: DERUMIER, Alexandre via pve-devel @ 2024-08-14 12:22 UTC (permalink / raw) To: pve-devel, cmos; +Cc: DERUMIER, Alexandre [-- Attachment #1: Type: message/rfc822, Size: 25233 bytes --] From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com> To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com>, "cmos@maklee.com" <cmos@maklee.com> Subject: Re: FW: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Date: Wed, 14 Aug 2024 12:22:12 +0000 Message-ID: <c851a27f4f5eca7c767fbffa48cb208e2d8fe1e6.camel@groupe-cyllene.com> >>Alexandre, >> >>the statement below is not true for our case. The OpenVMS guest OS is >>using a PCIE bus, so the virtio-scsi device should be exposed as >>"modern", but is not. Not sure why not at this point See Fiona response, the pci express bridge is present, but the virtio-net is plugged on a simple pci bridge. pci express slots are mostly used for passthrough devices /usr/share/perl5/PVE/QemuServer/PCI.pm my $pcie_addr_map; sub get_pcie_addr_map { $pcie_addr_map = { vga => { bus => 'pcie.0', addr => 1 }, hostpci0 => { bus => "ich9-pcie-port-1", addr => 0 }, hostpci1 => { bus => "ich9-pcie-port-2", addr => 0 }, hostpci2 => { bus => "ich9-pcie-port-3", addr => 0 }, hostpci3 => { bus => "ich9-pcie-port-4", addr => 0 }, hostpci4 => { bus => "ich9-pcie-port-5", addr => 0 }, hostpci5 => { bus => "ich9-pcie-port-6", addr => 0 }, hostpci6 => { bus => "ich9-pcie-port-7", addr => 0 }, hostpci7 => { bus => "ich9-pcie-port-8", addr => 0 }, hostpci8 => { bus => "ich9-pcie-port-9", addr => 0 }, hostpci9 => { bus => "ich9-pcie-port-10", addr => 0 }, hostpci10 => { bus => "ich9-pcie-port-11", addr => 0 }, hostpci11 => { bus => "ich9-pcie-port-12", addr => 0 }, hostpci12 => { bus => "ich9-pcie-port-13", addr => 0 }, hostpci13 => { bus => "ich9-pcie-port-14", addr => 0 }, hostpci14 => { bus => "ich9-pcie-port-15", addr => 0 }, hostpci15 => { bus => "ich9-pcie-port-16", addr => 0 }, # win7 is picky about pcie assignments hostpci0bus0 => { bus => "pcie.0", addr => 16 }, hostpci1bus0 => { bus => "pcie.0", addr => 17 }, hostpci2bus0 => { bus => "pcie.0", addr => 18 }, hostpci3bus0 => { bus => "pcie.0", addr => 19 }, ivshmem => { bus => 'pcie.0', addr => 20 }, hostpci4bus0 => { bus => "pcie.0", addr => 9 }, hostpci5bus0 => { bus => "pcie.0", addr => 10 }, hostpci6bus0 => { bus => "pcie.0", addr => 11 }, hostpci7bus0 => { bus => "pcie.0", addr => 12 }, hostpci8bus0 => { bus => "pcie.0", addr => 13 }, hostpci9bus0 => { bus => "pcie.0", addr => 14 }, hostpci10bus0 => { bus => "pcie.0", addr => 15 }, hostpci11bus0 => { bus => "pcie.0", addr => 21 }, hostpci12bus0 => { bus => "pcie.0", addr => 22 }, hostpci13bus0 => { bus => "pcie.0", addr => 23 }, hostpci14bus0 => { bus => "pcie.0", addr => 24 }, hostpci15bus0 => { bus => "pcie.0", addr => 25 }, } _______________________________________________________ Christian Moser Mobile: +358-40-5022105 Email: cmos@maklee.com URL: www.maklee.com -----Original Message----- From: DERUMIER, Alexandre <alexandre.derumier@groupe-cyllene.com> Sent: Wednesday, August 14, 2024 09:45 To: pve-devel@lists.proxmox.com; cmos@maklee.com Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Hi, I didn't see the responde of Fiona but indeed: https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg01567.html "virtio devices can be exposed in upto three ways - Legacy - follows virtio 0.9 specification. always uses PCI ID range 0x1000-0x103F - Transitional - follows virtio 0.9 specification by default, but can auto-negotiate with guest for 1.0 spce. Always uses PCI ID range 0x1000-0x103F - Modern - follows virtio 1.0 specification. always uses PCI ID range 0x1040-0x107F With QEMU, historically devices placed on a PCI bus will always default to being in transitional mode, while devices placed on a PCI-E bus will always dfault to being in modern mode. " -------- Message initial -------- De: Fiona Ebner <f.ebner@proxmox.com> Répondre à: Proxmox VE development discussion <pve- devel@lists.proxmox.com> À: Proxmox VE development discussion <pve-devel@lists.proxmox.com>, Christian Moser <cmos@maklee.com> Objet: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Date: 13/08/2024 10:55:47 Hi, Am 12.08.24 um 12:40 schrieb Christian Moser: > Hello, > > I work for VSI (VMS Software Inc) which is porting the OpenVMS > operating system to x86. At this point we successfully on various > hypervisors, but have some issues on the KVM running on Proxmox. > > The OpenVMS VM works just fine with SATA disks and it also works with > for example virtio-network device etc., but trying to use virtio-scsi > hangs the driver. I have debugged this and I can successfully > configure the port/controller, send the IO request to the device. It > then gets processed by the device, which posts the results and sets > the interrupt bit in the ISR register, but it never asserts the > interrupt hence the driver never gets notified and the I/O hangs. > > I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no > luck. The emulated virtio-scsi device is a legacy device. But then > again, the virtio-network device is also a legacy device and here we > are getting interrupts. One thing which bothers me is the fact that > the “legacy interrupt disable” bit is set in the PCI config space of > the virtio-scsi device (i.e. bit 10 at offset 4) > > Questions: > * is there a way to configure a modern virtio-scsi devcie (i.e. > disable_legacy=on) ? I've already answered this question when you asked in a mail addressed directly to me: Am 12.08.24 um 11:58 schrieb Fiona Ebner: > Hi, > > It seems that you either need to attach the "virtio-scsi-pci" device > to a pcie bus or explicitly set the "disable_legacy=on" option for > the > device, neither of which Proxmox VE currently does or allows > configuring. The only way right now seems to be to attach the disk > yourself via custom arguments (the 'args' in the Proxmox VE VM > configuration), but then the disk will be invisible to Proxmox VE > operations which look at specific disk keys in the configuration! > > Feel free to open a feature request on our bug tracker to make this > configurable: > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=1aab5a2ada7 > 3beb46aa02df4e18ff6c5ba2db6d6ff2d1f302a3c4c83b13c8ef6&u=https%3A%2F%2 > Fbugzilla.proxmox.com%2F > > P.S. Please write to the developer list rather than individual > developers for such questions in the feature: > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c87 > 16b5426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2 > Flists.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel > > Best Regards, > Fiona > * why is the legacy interrupt bit set in the PCI config space ? Most likely because the virtio-scsi-pci is configured without the "disable_legacy=on" option. If not explicitly set, the option will be "disable_legacy=auto" and when not attached to PCIe (which is the case for Proxmox VE currently), then legacy mode will be enabled. > * Are there any working driver for virtio-scsi on this KVM using Q35 > machine? i.e. any other OS The virtio drivers for Windows and the ones in Linux work just fine with our configuration. > Any thoughts why these interrupts are not getting delivered on the > PCIE bus? We do not configure the virtio-scsi-pci on a PCIe bus currently, see my initial answer. Best Regards, Fiona _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTraK 7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1RobkV xVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c8716b5 426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2Flists .proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel [-- Attachment #2: Type: text/plain, Size: 160 bytes --] _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <c851a27f4f5eca7c767fbffa48cb208e2d8fe1e6.camel@groupe-cyllene.com>]
* Re: [pve-devel] FW: issues with Virtio-SCSI devicde on Proxmox... [not found] ` <c851a27f4f5eca7c767fbffa48cb208e2d8fe1e6.camel@groupe-cyllene.com> @ 2024-08-14 12:48 ` Christian Moser 2024-08-14 13:05 ` DERUMIER, Alexandre via pve-devel [not found] ` <6ea8c90b469da48525ed743b6541bb3644be4a6c.camel@groupe-cyllene.com> 0 siblings, 2 replies; 11+ messages in thread From: Christian Moser @ 2024-08-14 12:48 UTC (permalink / raw) To: 'DERUMIER, Alexandre'; +Cc: Christian Moser, pve-devel I'm talking about virtio-scsi. Our virtio-network device is working fine Sent from BlueMail Get BlueMail for Android On Aug 14, 2024, 15:22, at 15:22, "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com> wrote: >>>Alexandre, >>> >>>the statement below is not true for our case. The OpenVMS guest OS is >>>using a PCIE bus, so the virtio-scsi device should be exposed as >>>"modern", but is not. Not sure why not at this point > >See Fiona response, > >the pci express bridge is present, but the virtio-net is plugged on a >simple pci bridge. > >pci express slots are mostly used for passthrough devices > >/usr/share/perl5/PVE/QemuServer/PCI.pm > >my $pcie_addr_map; >sub get_pcie_addr_map { > $pcie_addr_map = { > vga => { bus => 'pcie.0', addr => 1 }, > hostpci0 => { bus => "ich9-pcie-port-1", addr => 0 }, > hostpci1 => { bus => "ich9-pcie-port-2", addr => 0 }, > hostpci2 => { bus => "ich9-pcie-port-3", addr => 0 }, > hostpci3 => { bus => "ich9-pcie-port-4", addr => 0 }, > hostpci4 => { bus => "ich9-pcie-port-5", addr => 0 }, > hostpci5 => { bus => "ich9-pcie-port-6", addr => 0 }, > hostpci6 => { bus => "ich9-pcie-port-7", addr => 0 }, > hostpci7 => { bus => "ich9-pcie-port-8", addr => 0 }, > hostpci8 => { bus => "ich9-pcie-port-9", addr => 0 }, > hostpci9 => { bus => "ich9-pcie-port-10", addr => 0 }, > hostpci10 => { bus => "ich9-pcie-port-11", addr => 0 }, > hostpci11 => { bus => "ich9-pcie-port-12", addr => 0 }, > hostpci12 => { bus => "ich9-pcie-port-13", addr => 0 }, > hostpci13 => { bus => "ich9-pcie-port-14", addr => 0 }, > hostpci14 => { bus => "ich9-pcie-port-15", addr => 0 }, > hostpci15 => { bus => "ich9-pcie-port-16", addr => 0 }, > # win7 is picky about pcie assignments > hostpci0bus0 => { bus => "pcie.0", addr => 16 }, > hostpci1bus0 => { bus => "pcie.0", addr => 17 }, > hostpci2bus0 => { bus => "pcie.0", addr => 18 }, > hostpci3bus0 => { bus => "pcie.0", addr => 19 }, > ivshmem => { bus => 'pcie.0', addr => 20 }, > hostpci4bus0 => { bus => "pcie.0", addr => 9 }, > hostpci5bus0 => { bus => "pcie.0", addr => 10 }, > hostpci6bus0 => { bus => "pcie.0", addr => 11 }, > hostpci7bus0 => { bus => "pcie.0", addr => 12 }, > hostpci8bus0 => { bus => "pcie.0", addr => 13 }, > hostpci9bus0 => { bus => "pcie.0", addr => 14 }, > hostpci10bus0 => { bus => "pcie.0", addr => 15 }, > hostpci11bus0 => { bus => "pcie.0", addr => 21 }, > hostpci12bus0 => { bus => "pcie.0", addr => 22 }, > hostpci13bus0 => { bus => "pcie.0", addr => 23 }, > hostpci14bus0 => { bus => "pcie.0", addr => 24 }, > hostpci15bus0 => { bus => "pcie.0", addr => 25 }, >} > >_______________________________________________________ >Christian Moser >Mobile: +358-40-5022105 >Email: cmos@maklee.com >URL: www.maklee.com >-----Original Message----- >From: DERUMIER, Alexandre <alexandre.derumier@groupe-cyllene.com> >Sent: Wednesday, August 14, 2024 09:45 >To: pve-devel@lists.proxmox.com; cmos@maklee.com >Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... > >Hi, > >I didn't see the responde of Fiona but indeed: > >https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg01567.html > >"virtio devices can be exposed in upto three ways > > - Legacy - follows virtio 0.9 specification. always uses PCI > ID range 0x1000-0x103F > - Transitional - follows virtio 0.9 specification by default, but > can auto-negotiate with guest for 1.0 spce. Always > uses PCI ID range 0x1000-0x103F > - Modern - follows virtio 1.0 specification. always uses PCI > ID range 0x1040-0x107F > >With QEMU, historically devices placed on a PCI bus will always default >to being in transitional mode, while devices placed on a PCI-E bus will >always dfault to being in modern mode. >" > > >-------- Message initial -------- >De: Fiona Ebner <f.ebner@proxmox.com> >Répondre à: Proxmox VE development discussion <pve- >devel@lists.proxmox.com> >À: Proxmox VE development discussion <pve-devel@lists.proxmox.com>, >Christian Moser <cmos@maklee.com> >Objet: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... >Date: 13/08/2024 10:55:47 > >Hi, > >Am 12.08.24 um 12:40 schrieb Christian Moser: >> Hello, >> >> I work for VSI (VMS Software Inc) which is porting the OpenVMS >> operating system to x86. At this point we successfully on various >> hypervisors, but have some issues on the KVM running on Proxmox. >> >> The OpenVMS VM works just fine with SATA disks and it also works with >> for example virtio-network device etc., but trying to use virtio-scsi >> hangs the driver. I have debugged this and I can successfully >> configure the port/controller, send the IO request to the device. It >> then gets processed by the device, which posts the results and sets >> the interrupt bit in the ISR register, but it never asserts the >> interrupt hence the driver never gets notified and the I/O hangs. >> >> I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no >> luck. The emulated virtio-scsi device is a legacy device. But then >> again, the virtio-network device is also a legacy device and here we >> are getting interrupts. One thing which bothers me is the fact that >> the “legacy interrupt disable” bit is set in the PCI config space of >> the virtio-scsi device (i.e. bit 10 at offset 4) >> >> Questions: >> * is there a way to configure a modern virtio-scsi devcie (i.e. >> disable_legacy=on) ? > >I've already answered this question when you asked in a mail addressed >directly to me: > >Am 12.08.24 um 11:58 schrieb Fiona Ebner: >> Hi, >> >> It seems that you either need to attach the "virtio-scsi-pci" device >> to a pcie bus or explicitly set the "disable_legacy=on" option for >> the >> device, neither of which Proxmox VE currently does or allows >> configuring. The only way right now seems to be to attach the disk >> yourself via custom arguments (the 'args' in the Proxmox VE VM >> configuration), but then the disk will be invisible to Proxmox VE >> operations which look at specific disk keys in the configuration! >> >> Feel free to open a feature request on our bug tracker to make this >> configurable: >> https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr >> aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- >> 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob >> kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=1aab5a2ada7 >> 3beb46aa02df4e18ff6c5ba2db6d6ff2d1f302a3c4c83b13c8ef6&u=https%3A%2F%2 >> Fbugzilla.proxmox.com%2F >> >> P.S. Please write to the developer list rather than individual >> developers for such questions in the feature: >> https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr >> aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- >> 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob >> kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c87 >> 16b5426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2 >> Flists.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel >> >> Best Regards, >> Fiona > >> * why is the legacy interrupt bit set in the PCI config space ? > >Most likely because the virtio-scsi-pci is configured without the >"disable_legacy=on" option. If not explicitly set, the option will be >"disable_legacy=auto" and when not attached to PCIe (which is the case >for Proxmox VE currently), then legacy mode will be enabled. > >> * Are there any working driver for virtio-scsi on this KVM using Q35 >> machine? i.e. any other OS > >The virtio drivers for Windows and the ones in Linux work just fine >with our configuration. > > >> Any thoughts why these interrupts are not getting delivered on the >> PCIE bus? > >We do not configure the virtio-scsi-pci on a PCIe bus currently, see my >initial answer. > >Best Regards, >Fiona > > >_______________________________________________ >pve-devel mailing list >pve-devel@lists.proxmox.com >https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTraK > >7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- >1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1RobkV > >xVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c8716b5 > >426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2Flists > >.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [pve-devel] FW: issues with Virtio-SCSI devicde on Proxmox... 2024-08-14 12:48 ` Christian Moser @ 2024-08-14 13:05 ` DERUMIER, Alexandre via pve-devel [not found] ` <6ea8c90b469da48525ed743b6541bb3644be4a6c.camel@groupe-cyllene.com> 1 sibling, 0 replies; 11+ messages in thread From: DERUMIER, Alexandre via pve-devel @ 2024-08-14 13:05 UTC (permalink / raw) To: cmos; +Cc: DERUMIER, Alexandre, pve-devel [-- Attachment #1: Type: message/rfc822, Size: 33368 bytes --] From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com> To: "cmos@maklee.com" <cmos@maklee.com> Cc: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com> Subject: Re: FW: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Date: Wed, 14 Aug 2024 13:05:35 +0000 Message-ID: <6ea8c90b469da48525ed743b6541bb3644be4a6c.camel@groupe-cyllene.com> >>I'm talking about virtio-scsi. Our virtio-network device is working >>fine Yes, sorry, I wanted to said virtio-scsi. All pci devices excluding passthrough devices (with pcie=on flag) are actually plugged to pci bridge sub get_pci_addr_map { $pci_addr_map = { piix3 => { bus => 0, addr => 1, conflict_ok => qw(ehci) }, ehci => { bus => 0, addr => 1, conflict_ok => qw(piix3) }, # instead of piix3 on arm vga => { bus => 0, addr => 2, conflict_ok => qw(legacy-igd) }, 'legacy-igd' => { bus => 0, addr => 2, conflict_ok => qw(vga) }, # legacy-igd requires vga=none balloon0 => { bus => 0, addr => 3 }, watchdog => { bus => 0, addr => 4 }, scsihw0 => { bus => 0, addr => 5, conflict_ok => qw(pci.3) }, 'pci.3' => { bus => 0, addr => 5, conflict_ok => qw(scsihw0) }, # also used for virtio-scsi-single bridge scsihw1 => { bus => 0, addr => 6 }, ahci0 => { bus => 0, addr => 7 }, qga0 => { bus => 0, addr => 8 }, spice => { bus => 0, addr => 9 }, virtio0 => { bus => 0, addr => 10 }, virtio1 => { bus => 0, addr => 11 }, virtio2 => { bus => 0, addr => 12 }, virtio3 => { bus => 0, addr => 13 }, virtio4 => { bus => 0, addr => 14 }, virtio5 => { bus => 0, addr => 15 }, hostpci0 => { bus => 0, addr => 16 }, hostpci1 => { bus => 0, addr => 17 }, net0 => { bus => 0, addr => 18 }, net1 => { bus => 0, addr => 19 }, net2 => { bus => 0, addr => 20 }, net3 => { bus => 0, addr => 21 }, net4 => { bus => 0, addr => 22 }, net5 => { bus => 0, addr => 23 }, vga1 => { bus => 0, addr => 24 }, vga2 => { bus => 0, addr => 25 }, vga3 => { bus => 0, addr => 26 }, hostpci2 => { bus => 0, addr => 27 }, hostpci3 => { bus => 0, addr => 28 }, #addr29 : usb-host (pve-usb.cfg) 'pci.1' => { bus => 0, addr => 30 }, 'pci.2' => { bus => 0, addr => 31 }, 'net6' => { bus => 1, addr => 1 }, 'net7' => { bus => 1, addr => 2 }, 'net8' => { bus => 1, addr => 3 }, 'net9' => { bus => 1, addr => 4 }, 'net10' => { bus => 1, addr => 5 }, 'net11' => { bus => 1, addr => 6 }, 'net12' => { bus => 1, addr => 7 }, 'net13' => { bus => 1, addr => 8 }, 'net14' => { bus => 1, addr => 9 }, 'net15' => { bus => 1, addr => 10 }, 'net16' => { bus => 1, addr => 11 }, 'net17' => { bus => 1, addr => 12 }, 'net18' => { bus => 1, addr => 13 }, 'net19' => { bus => 1, addr => 14 }, 'net20' => { bus => 1, addr => 15 }, 'net21' => { bus => 1, addr => 16 }, 'net22' => { bus => 1, addr => 17 }, 'net23' => { bus => 1, addr => 18 }, 'net24' => { bus => 1, addr => 19 }, 'net25' => { bus => 1, addr => 20 }, 'net26' => { bus => 1, addr => 21 }, 'net27' => { bus => 1, addr => 22 }, 'net28' => { bus => 1, addr => 23 }, 'net29' => { bus => 1, addr => 24 }, 'net30' => { bus => 1, addr => 25 }, 'net31' => { bus => 1, addr => 26 }, 'xhci' => { bus => 1, addr => 27 }, 'pci.4' => { bus => 1, addr => 28 }, 'rng0' => { bus => 1, addr => 29 }, 'pci.2-igd' => { bus => 1, addr => 30 }, # replaces pci.2 in case a legacy IGD device is passed through 'virtio6' => { bus => 2, addr => 1 }, 'virtio7' => { bus => 2, addr => 2 }, 'virtio8' => { bus => 2, addr => 3 }, 'virtio9' => { bus => 2, addr => 4 }, 'virtio10' => { bus => 2, addr => 5 }, 'virtio11' => { bus => 2, addr => 6 }, 'virtio12' => { bus => 2, addr => 7 }, 'virtio13' => { bus => 2, addr => 8 }, 'virtio14' => { bus => 2, addr => 9 }, 'virtio15' => { bus => 2, addr => 10 }, 'ivshmem' => { bus => 2, addr => 11 }, 'audio0' => { bus => 2, addr => 12 }, hostpci4 => { bus => 2, addr => 13 }, hostpci5 => { bus => 2, addr => 14 }, hostpci6 => { bus => 2, addr => 15 }, hostpci7 => { bus => 2, addr => 16 }, hostpci8 => { bus => 2, addr => 17 }, hostpci9 => { bus => 2, addr => 18 }, hostpci10 => { bus => 2, addr => 19 }, hostpci11 => { bus => 2, addr => 20 }, hostpci12 => { bus => 2, addr => 21 }, hostpci13 => { bus => 2, addr => 22 }, hostpci14 => { bus => 2, addr => 23 }, hostpci15 => { bus => 2, addr => 24 }, 'virtioscsi0' => { bus => 3, addr => 1 }, 'virtioscsi1' => { bus => 3, addr => 2 }, 'virtioscsi2' => { bus => 3, addr => 3 }, 'virtioscsi3' => { bus => 3, addr => 4 }, 'virtioscsi4' => { bus => 3, addr => 5 }, 'virtioscsi5' => { bus => 3, addr => 6 }, 'virtioscsi6' => { bus => 3, addr => 7 }, 'virtioscsi7' => { bus => 3, addr => 8 }, 'virtioscsi8' => { bus => 3, addr => 9 }, 'virtioscsi9' => { bus => 3, addr => 10 }, 'virtioscsi10' => { bus => 3, addr => 11 }, 'virtioscsi11' => { bus => 3, addr => 12 }, 'virtioscsi12' => { bus => 3, addr => 13 }, 'virtioscsi13' => { bus => 3, addr => 14 }, 'virtioscsi14' => { bus => 3, addr => 15 }, 'virtioscsi15' => { bus => 3, addr => 16 }, 'virtioscsi16' => { bus => 3, addr => 17 }, 'virtioscsi17' => { bus => 3, addr => 18 }, 'virtioscsi18' => { bus => 3, addr => 19 }, 'virtioscsi19' => { bus => 3, addr => 20 }, 'virtioscsi20' => { bus => 3, addr => 21 }, 'virtioscsi21' => { bus => 3, addr => 22 }, 'virtioscsi22' => { bus => 3, addr => 23 }, 'virtioscsi23' => { bus => 3, addr => 24 }, 'virtioscsi24' => { bus => 3, addr => 25 }, 'virtioscsi25' => { bus => 3, addr => 26 }, 'virtioscsi26' => { bus => 3, addr => 27 }, 'virtioscsi27' => { bus => 3, addr => 28 }, 'virtioscsi28' => { bus => 3, addr => 29 }, 'virtioscsi29' => { bus => 3, addr => 30 }, 'virtioscsi30' => { bus => 3, addr => 31 }, 'scsihw2' => { bus => 4, addr => 1 }, 'scsihw3' => { bus => 4, addr => 2 }, 'scsihw4' => { bus => 4, addr => 3 }, } if !defined($pci_addr_map); return $pci_addr_map; > > > > Alexandre, > > > > > > the statement below is not true for our case. The OpenVMS guest OS is > > > using a PCIE bus, so the virtio-scsi device should be exposed as > > > "modern", but is not. Not sure why not at this point > > > > See Fiona response, > > the pci express bridge is present, but the virtio-net is plugged on a > simple pci bridge. > ² > pci express slots are mostly used for passthrough devices > > > > my $pcie_addr_map; > sub get_pcie_addr_map { > $pcie_addr_map = { > vga => { bus => 'pcie.0', addr => 1 }, > hostpci0 => { bus => "ich9-pcie-port-1", addr => 0 }, > hostpci1 => { bus => "ich9-pcie-port-2", addr => 0 }, > hostpci2 => { bus => "ich9-pcie-port-3", addr => 0 }, > hostpci3 => { bus => "ich9-pcie-port-4", addr => 0 }, > hostpci4 => { bus => "ich9-pcie-port-5", addr => 0 }, > hostpci5 => { bus => "ich9-pcie-port-6", addr => 0 }, > hostpci6 => { bus => "ich9-pcie-port-7", addr => 0 }, > hostpci7 => { bus => "ich9-pcie-port-8", addr => 0 }, > hostpci8 => { bus => "ich9-pcie-port-9", addr => 0 }, > hostpci9 => { bus => "ich9-pcie-port-10", addr => 0 }, > hostpci10 => { bus => "ich9-pcie-port-11", addr => 0 }, > hostpci11 => { bus => "ich9-pcie-port-12", addr => 0 }, > hostpci12 => { bus => "ich9-pcie-port-13", addr => 0 }, > hostpci13 => { bus => "ich9-pcie-port-14", addr => 0 }, > hostpci14 => { bus => "ich9-pcie-port-15", addr => 0 }, > hostpci15 => { bus => "ich9-pcie-port-16", addr => 0 }, > # win7 is picky about pcie assignments > hostpci0bus0 => { bus => "pcie.0", addr => 16 }, > hostpci1bus0 => { bus => "pcie.0", addr => 17 }, > hostpci2bus0 => { bus => "pcie.0", addr => 18 }, > hostpci3bus0 => { bus => "pcie.0", addr => 19 }, > ivshmem => { bus => 'pcie.0', addr => 20 }, > hostpci4bus0 => { bus => "pcie.0", addr => 9 }, > hostpci5bus0 => { bus => "pcie.0", addr => 10 }, > hostpci6bus0 => { bus => "pcie.0", addr => 11 }, > hostpci7bus0 => { bus => "pcie.0", addr => 12 }, > hostpci8bus0 => { bus => "pcie.0", addr => 13 }, > hostpci9bus0 => { bus => "pcie.0", addr => 14 }, > hostpci10bus0 => { bus => "pcie.0", addr => 15 }, > hostpci11bus0 => { bus => "pcie.0", addr => 21 }, > hostpci12bus0 => { bus => "pcie.0", addr => 22 }, > hostpci13bus0 => { bus => "pcie.0", addr => 23 }, > hostpci14bus0 => { bus => "pcie.0", addr => 24 }, > hostpci15bus0 => { bus => "pcie.0", addr => 25 }, > } > > > Christian Moser > Mobile: +358-40-5022105 > Email: cmos@maklee.com > URL: www.maklee.com > -----Original Message----- > From: DERUMIER, Alexandre <alexandre.derumier@groupe-cyllene.com> > Sent: Wednesday, August 14, 2024 09:45 > To: pve-devel@lists.proxmox.com; cmos@maklee.com > Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... > > Hi, > > I didn't see the responde of Fiona but indeed: > > https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg01567.html > > "virtio devices can be exposed in upto three ways > > - Legacy - follows virtio 0.9 specification. always uses PCI > ID range 0x1000-0x103F > - Transitional - follows virtio 0.9 specification by default, but > can auto-negotiate with guest for 1.0 spce. Always > uses PCI ID range 0x1000-0x103F > - Modern - follows virtio 1.0 specification. always uses PCI > ID range 0x1040-0x107F > > With QEMU, historically devices placed on a PCI bus will always default > to being in transitional mode, while devices placed on a PCI-E bus will > always dfault to being in modern mode. > " > > > -------- Message initial -------- > De: Fiona Ebner <f.ebner@proxmox.com> > Répondre à: Proxmox VE development discussion <pve- > devel@lists.proxmox.com> > À: Proxmox VE development discussion <pve-devel@lists.proxmox.com>, > Christian Moser <cmos@maklee.com> > Objet: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... > Date: 13/08/2024 10:55:47 > > Hi, > > Am 12.08.24 um 12:40 schrieb Christian Moser: > > Hello, > > > > I work for VSI (VMS Software Inc) which is porting the OpenVMS > > operating system to x86. At this point we successfully on various > > hypervisors, but have some issues on the KVM running on Proxmox. > > > > The OpenVMS VM works just fine with SATA disks and it also works with > > for example virtio-network device etc., but trying to use virtio-scsi > > hangs the driver. I have debugged this and I can successfully > > configure the port/controller, send the IO request to the device. It > > then gets processed by the device, which posts the results and sets > > the interrupt bit in the ISR register, but it never asserts the > > interrupt hence the driver never gets notified and the I/O hangs. > > > > I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no > > luck. The emulated virtio-scsi device is a legacy device. But then > > again, the virtio-network device is also a legacy device and here we > > are getting interrupts. One thing which bothers me is the fact that > > the “legacy interrupt disable” bit is set in the PCI config space of > > the virtio-scsi device (i.e. bit 10 at offset 4) > > > > Questions: > > * is there a way to configure a modern virtio-scsi devcie (i.e. > > disable_legacy=on) ? > > > > I've already answered this question when you asked in a mail addressed > directly to me: > > Am 12.08.24 um 11:58 schrieb Fiona Ebner: > > Hi, > > > > It seems that you either need to attach the "virtio-scsi-pci" device > > to a pcie bus or explicitly set the "disable_legacy=on" option for > > the > > device, neither of which Proxmox VE currently does or allows > > configuring. The only way right now seems to be to attach the disk > > yourself via custom arguments (the 'args' in the Proxmox VE VM > > configuration), but then the disk will be invisible to Proxmox VE > > operations which look at specific disk keys in the configuration! > > > > Feel free to open a feature request on our bug tracker to make this > > configurable: > > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr > > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- > > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob > > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=1aab5a2ada7 > > 3beb46aa02df4e18ff6c5ba2db6d6ff2d1f302a3c4c83b13c8ef6&u=https%3A%2F%2 > > Fbugzilla.proxmox.com%2F > > > > P.S. Please write to the developer list rather than individual > > developers for such questions in the feature: > > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTr > > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- > > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Rob > > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c87 > > 16b5426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2 > > Flists.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel > > > > Best Regards, > > Fiona > > > > > * why is the legacy interrupt bit set in the PCI config space ? > > > > Most likely because the virtio-scsi-pci is configured without the > "disable_legacy=on" option. If not explicitly set, the option will be > "disable_legacy=auto" and when not attached to PCIe (which is the case > for Proxmox VE currently), then legacy mode will be enabled. > > > * Are there any working driver for virtio-scsi on this KVM using Q35 > > machine? i.e. any other OS > > > > The virtio drivers for Windows and the ones in Linux work just fine > with our configuration. > > > > Any thoughts why these interrupts are not getting delivered on the > > PCIE bus? > > > > We do not configure the virtio-scsi-pci on a PCIe bus currently, see my > initial answer. > > Best Regards, > Fiona > > > > pve-devel mailing list > pve-devel@lists.proxmox.com > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTraK > > 7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1RobkV > > xVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c8716b5 > > 426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2Flists > > .proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel > > > [-- Attachment #2: Type: text/plain, Size: 160 bytes --] _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <6ea8c90b469da48525ed743b6541bb3644be4a6c.camel@groupe-cyllene.com>]
* [pve-devel] FW: FW: issues with Virtio-SCSI devicde on Proxmox... [not found] ` <6ea8c90b469da48525ed743b6541bb3644be4a6c.camel@groupe-cyllene.com> @ 2024-08-15 5:35 ` Christian Moser 0 siblings, 0 replies; 11+ messages in thread From: Christian Moser @ 2024-08-15 5:35 UTC (permalink / raw) To: 'DERUMIER, Alexandre', pve-devel; +Cc: Christian Moser I finally got it to work. The virtio-scsi device has 2 interrupt models, either the legacy "wire" interrupt or MSIX interrupts. During boot, the OpenVMS bus support code is probing the bus and configures the devices. It will default to the traditional IOAPIC interrupt and if the capability bit is set in the PCI config status register it will parse the device capabilities. Virtio-SCSI reports that it has MSIX capability, so now the bus support code is setting the interrupt mechanism to MSIX. But our OpenVMS driver for the virtio-scsi device does not yet support MSIX, hence the disconnect and hang, because we do not get any interrupts delivered. I can special case the virtio-scsi device in the PCI config routine to ignore the MSIX capability and then everything is working fine. I checked on other hypervisors, like for example Virtualbox and here the virtio-scsi device is reporting capability as "vendor specific" which are ignored by our bus support code. I guess there is no way of disabling the MSIX capability on Proxmox with some flags forvirtio-scsi? In the long run, we should modify our driver to support MSIX interrupts. But I had issues enabling them correctly on virtio-scsi, I need to find some driver for Linux or Windows which supports MSIX and verify what is incorrect on our side. thanks for your help and information /cmos _______________________________________________________ Christian Moser Mobile: +358-40-5022105 Email: cmos@maklee.com URL: www.maklee.com -----Original Message----- From: DERUMIER, Alexandre <alexandre.derumier@groupe-cyllene.com> Sent: Wednesday, August 14, 2024 16:06 To: cmos@maklee.com Cc: pve-devel@lists.proxmox.com Subject: Re: FW: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... >>I'm talking about virtio-scsi. Our virtio-network device is working >>fine Yes, sorry, I wanted to said virtio-scsi. All pci devices excluding passthrough devices (with pcie=on flag) are actually plugged to pci bridge sub get_pci_addr_map { $pci_addr_map = { piix3 => { bus => 0, addr => 1, conflict_ok => qw(ehci) }, ehci => { bus => 0, addr => 1, conflict_ok => qw(piix3) }, # instead of piix3 on arm vga => { bus => 0, addr => 2, conflict_ok => qw(legacy-igd) }, 'legacy-igd' => { bus => 0, addr => 2, conflict_ok => qw(vga) }, # legacy-igd requires vga=none balloon0 => { bus => 0, addr => 3 }, watchdog => { bus => 0, addr => 4 }, scsihw0 => { bus => 0, addr => 5, conflict_ok => qw(pci.3) }, 'pci.3' => { bus => 0, addr => 5, conflict_ok => qw(scsihw0) }, # also used for virtio-scsi-single bridge scsihw1 => { bus => 0, addr => 6 }, ahci0 => { bus => 0, addr => 7 }, qga0 => { bus => 0, addr => 8 }, spice => { bus => 0, addr => 9 }, virtio0 => { bus => 0, addr => 10 }, virtio1 => { bus => 0, addr => 11 }, virtio2 => { bus => 0, addr => 12 }, virtio3 => { bus => 0, addr => 13 }, virtio4 => { bus => 0, addr => 14 }, virtio5 => { bus => 0, addr => 15 }, hostpci0 => { bus => 0, addr => 16 }, hostpci1 => { bus => 0, addr => 17 }, net0 => { bus => 0, addr => 18 }, net1 => { bus => 0, addr => 19 }, net2 => { bus => 0, addr => 20 }, net3 => { bus => 0, addr => 21 }, net4 => { bus => 0, addr => 22 }, net5 => { bus => 0, addr => 23 }, vga1 => { bus => 0, addr => 24 }, vga2 => { bus => 0, addr => 25 }, vga3 => { bus => 0, addr => 26 }, hostpci2 => { bus => 0, addr => 27 }, hostpci3 => { bus => 0, addr => 28 }, #addr29 : usb-host (pve-usb.cfg) 'pci.1' => { bus => 0, addr => 30 }, 'pci.2' => { bus => 0, addr => 31 }, 'net6' => { bus => 1, addr => 1 }, 'net7' => { bus => 1, addr => 2 }, 'net8' => { bus => 1, addr => 3 }, 'net9' => { bus => 1, addr => 4 }, 'net10' => { bus => 1, addr => 5 }, 'net11' => { bus => 1, addr => 6 }, 'net12' => { bus => 1, addr => 7 }, 'net13' => { bus => 1, addr => 8 }, 'net14' => { bus => 1, addr => 9 }, 'net15' => { bus => 1, addr => 10 }, 'net16' => { bus => 1, addr => 11 }, 'net17' => { bus => 1, addr => 12 }, 'net18' => { bus => 1, addr => 13 }, 'net19' => { bus => 1, addr => 14 }, 'net20' => { bus => 1, addr => 15 }, 'net21' => { bus => 1, addr => 16 }, 'net22' => { bus => 1, addr => 17 }, 'net23' => { bus => 1, addr => 18 }, 'net24' => { bus => 1, addr => 19 }, 'net25' => { bus => 1, addr => 20 }, 'net26' => { bus => 1, addr => 21 }, 'net27' => { bus => 1, addr => 22 }, 'net28' => { bus => 1, addr => 23 }, 'net29' => { bus => 1, addr => 24 }, 'net30' => { bus => 1, addr => 25 }, 'net31' => { bus => 1, addr => 26 }, 'xhci' => { bus => 1, addr => 27 }, 'pci.4' => { bus => 1, addr => 28 }, 'rng0' => { bus => 1, addr => 29 }, 'pci.2-igd' => { bus => 1, addr => 30 }, # replaces pci.2 in case a legacy IGD device is passed through 'virtio6' => { bus => 2, addr => 1 }, 'virtio7' => { bus => 2, addr => 2 }, 'virtio8' => { bus => 2, addr => 3 }, 'virtio9' => { bus => 2, addr => 4 }, 'virtio10' => { bus => 2, addr => 5 }, 'virtio11' => { bus => 2, addr => 6 }, 'virtio12' => { bus => 2, addr => 7 }, 'virtio13' => { bus => 2, addr => 8 }, 'virtio14' => { bus => 2, addr => 9 }, 'virtio15' => { bus => 2, addr => 10 }, 'ivshmem' => { bus => 2, addr => 11 }, 'audio0' => { bus => 2, addr => 12 }, hostpci4 => { bus => 2, addr => 13 }, hostpci5 => { bus => 2, addr => 14 }, hostpci6 => { bus => 2, addr => 15 }, hostpci7 => { bus => 2, addr => 16 }, hostpci8 => { bus => 2, addr => 17 }, hostpci9 => { bus => 2, addr => 18 }, hostpci10 => { bus => 2, addr => 19 }, hostpci11 => { bus => 2, addr => 20 }, hostpci12 => { bus => 2, addr => 21 }, hostpci13 => { bus => 2, addr => 22 }, hostpci14 => { bus => 2, addr => 23 }, hostpci15 => { bus => 2, addr => 24 }, 'virtioscsi0' => { bus => 3, addr => 1 }, 'virtioscsi1' => { bus => 3, addr => 2 }, 'virtioscsi2' => { bus => 3, addr => 3 }, 'virtioscsi3' => { bus => 3, addr => 4 }, 'virtioscsi4' => { bus => 3, addr => 5 }, 'virtioscsi5' => { bus => 3, addr => 6 }, 'virtioscsi6' => { bus => 3, addr => 7 }, 'virtioscsi7' => { bus => 3, addr => 8 }, 'virtioscsi8' => { bus => 3, addr => 9 }, 'virtioscsi9' => { bus => 3, addr => 10 }, 'virtioscsi10' => { bus => 3, addr => 11 }, 'virtioscsi11' => { bus => 3, addr => 12 }, 'virtioscsi12' => { bus => 3, addr => 13 }, 'virtioscsi13' => { bus => 3, addr => 14 }, 'virtioscsi14' => { bus => 3, addr => 15 }, 'virtioscsi15' => { bus => 3, addr => 16 }, 'virtioscsi16' => { bus => 3, addr => 17 }, 'virtioscsi17' => { bus => 3, addr => 18 }, 'virtioscsi18' => { bus => 3, addr => 19 }, 'virtioscsi19' => { bus => 3, addr => 20 }, 'virtioscsi20' => { bus => 3, addr => 21 }, 'virtioscsi21' => { bus => 3, addr => 22 }, 'virtioscsi22' => { bus => 3, addr => 23 }, 'virtioscsi23' => { bus => 3, addr => 24 }, 'virtioscsi24' => { bus => 3, addr => 25 }, 'virtioscsi25' => { bus => 3, addr => 26 }, 'virtioscsi26' => { bus => 3, addr => 27 }, 'virtioscsi27' => { bus => 3, addr => 28 }, 'virtioscsi28' => { bus => 3, addr => 29 }, 'virtioscsi29' => { bus => 3, addr => 30 }, 'virtioscsi30' => { bus => 3, addr => 31 }, 'scsihw2' => { bus => 4, addr => 1 }, 'scsihw3' => { bus => 4, addr => 2 }, 'scsihw4' => { bus => 4, addr => 3 }, } if !defined($pci_addr_map); return $pci_addr_map; > > > > Alexandre, > > > > > > the statement below is not true for our case. The OpenVMS guest OS > > > is using a PCIE bus, so the virtio-scsi device should be exposed > > > as "modern", but is not. Not sure why not at this point > > > > See Fiona response, > > the pci express bridge is present, but the virtio-net is plugged on a > simple pci bridge. > ² > pci express slots are mostly used for passthrough devices > > > > my $pcie_addr_map; > sub get_pcie_addr_map { > $pcie_addr_map = { > vga => { bus => 'pcie.0', addr => 1 }, > hostpci0 => { bus => "ich9-pcie-port-1", addr => 0 }, > hostpci1 => { bus => "ich9-pcie-port-2", addr => 0 }, > hostpci2 => { bus => "ich9-pcie-port-3", addr => 0 }, > hostpci3 => { bus => "ich9-pcie-port-4", addr => 0 }, > hostpci4 => { bus => "ich9-pcie-port-5", addr => 0 }, > hostpci5 => { bus => "ich9-pcie-port-6", addr => 0 }, > hostpci6 => { bus => "ich9-pcie-port-7", addr => 0 }, > hostpci7 => { bus => "ich9-pcie-port-8", addr => 0 }, > hostpci8 => { bus => "ich9-pcie-port-9", addr => 0 }, > hostpci9 => { bus => "ich9-pcie-port-10", addr => 0 }, > hostpci10 => { bus => "ich9-pcie-port-11", addr => 0 }, > hostpci11 => { bus => "ich9-pcie-port-12", addr => 0 }, > hostpci12 => { bus => "ich9-pcie-port-13", addr => 0 }, > hostpci13 => { bus => "ich9-pcie-port-14", addr => 0 }, > hostpci14 => { bus => "ich9-pcie-port-15", addr => 0 }, > hostpci15 => { bus => "ich9-pcie-port-16", addr => 0 }, > # win7 is picky about pcie assignments > hostpci0bus0 => { bus => "pcie.0", addr => 16 }, > hostpci1bus0 => { bus => "pcie.0", addr => 17 }, > hostpci2bus0 => { bus => "pcie.0", addr => 18 }, > hostpci3bus0 => { bus => "pcie.0", addr => 19 }, > ivshmem => { bus => 'pcie.0', addr => 20 }, > hostpci4bus0 => { bus => "pcie.0", addr => 9 }, > hostpci5bus0 => { bus => "pcie.0", addr => 10 }, > hostpci6bus0 => { bus => "pcie.0", addr => 11 }, > hostpci7bus0 => { bus => "pcie.0", addr => 12 }, > hostpci8bus0 => { bus => "pcie.0", addr => 13 }, > hostpci9bus0 => { bus => "pcie.0", addr => 14 }, > hostpci10bus0 => { bus => "pcie.0", addr => 15 }, > hostpci11bus0 => { bus => "pcie.0", addr => 21 }, > hostpci12bus0 => { bus => "pcie.0", addr => 22 }, > hostpci13bus0 => { bus => "pcie.0", addr => 23 }, > hostpci14bus0 => { bus => "pcie.0", addr => 24 }, > hostpci15bus0 => { bus => "pcie.0", addr => 25 }, } > > > Christian Moser > Mobile: +358-40-5022105 > Email: cmos@maklee.com > URL: www.maklee.com > -----Original Message----- > From: DERUMIER, Alexandre <alexandre.derumier@groupe-cyllene.com> > Sent: Wednesday, August 14, 2024 09:45 > To: pve-devel@lists.proxmox.com; cmos@maklee.com > Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... > > Hi, > > I didn't see the responde of Fiona but indeed: > > https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg01567.html > > "virtio devices can be exposed in upto three ways > > - Legacy - follows virtio 0.9 specification. always uses PCI > ID range 0x1000-0x103F > - Transitional - follows virtio 0.9 specification by default, but > can auto-negotiate with guest for 1.0 spce. Always > uses PCI ID range 0x1000-0x103F > - Modern - follows virtio 1.0 specification. always uses PCI > ID range 0x1040-0x107F > > With QEMU, historically devices placed on a PCI bus will always > default to being in transitional mode, while devices placed on a PCI-E > bus will always dfault to being in modern mode. > " > > > -------- Message initial -------- > De: Fiona Ebner <f.ebner@proxmox.com> > Répondre à: Proxmox VE development discussion <pve- > devel@lists.proxmox.com> > À: Proxmox VE development discussion <pve-devel@lists.proxmox.com>, > Christian Moser <cmos@maklee.com> > Objet: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... > Date: 13/08/2024 10:55:47 > > Hi, > > Am 12.08.24 um 12:40 schrieb Christian Moser: > > Hello, > > > > I work for VSI (VMS Software Inc) which is porting the OpenVMS > > operating system to x86. At this point we successfully on various > > hypervisors, but have some issues on the KVM running on Proxmox. > > > > The OpenVMS VM works just fine with SATA disks and it also works > > with for example virtio-network device etc., but trying to use > > virtio-scsi hangs the driver. I have debugged this and I can > > successfully configure the port/controller, send the IO request to > > the device. It then gets processed by the device, which posts the > > results and sets the interrupt bit in the ISR register, but it > > never asserts the interrupt hence the driver never gets notified and the I/O hangs. > > > > I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but > > no luck. The emulated virtio-scsi device is a legacy device. But > > then again, the virtio-network device is also a legacy device and > > here we are getting interrupts. One thing which bothers me is the > > fact that the “legacy interrupt disable” bit is set in the PCI > > config space of the virtio-scsi device (i.e. bit 10 at offset 4) > > > > Questions: > > * is there a way to configure a modern virtio-scsi devcie (i.e. > > disable_legacy=on) ? > > > > I've already answered this question when you asked in a mail addressed > directly to me: > > Am 12.08.24 um 11:58 schrieb Fiona Ebner: > > Hi, > > > > It seems that you either need to attach the "virtio-scsi-pci" > > device to a pcie bus or explicitly set the "disable_legacy=on" > > option for the device, neither of which Proxmox VE currently does > > or allows configuring. The only way right now seems to be to attach > > the disk yourself via custom arguments (the 'args' in the Proxmox > > VE VM configuration), but then the disk will be invisible to > > Proxmox VE operations which look at specific disk keys in the > > configuration! > > > > Feel free to open a feature request on our bug tracker to make this > > configurable: > > > > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbT > > r > > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- > > > > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Ro > > b > > > > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=1aab5a2ada > > 7 > > > > 3beb46aa02df4e18ff6c5ba2db6d6ff2d1f302a3c4c83b13c8ef6&u=https%3A%2F% > > 2 > > Fbugzilla.proxmox.com%2F > > > > P.S. Please write to the developer list rather than individual > > developers for such questions in the feature: > > > > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbT > > r > > aK7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- > > > > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Ro > > b > > > > kVxVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c8 > > 7 > > > > 16b5426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F% > > 2 Flists.proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel > > > > Best Regards, > > Fiona > > > > > * why is the legacy interrupt bit set in the PCI config space ? > > > > Most likely because the virtio-scsi-pci is configured without the > "disable_legacy=on" option. If not explicitly set, the option will be > "disable_legacy=auto" and when not attached to PCIe (which is the case > for Proxmox VE currently), then legacy mode will be enabled. > > > * Are there any working driver for virtio-scsi on this KVM using > > Q35 machine? i.e. any other OS > > > > The virtio drivers for Windows and the ones in Linux work just fine > with our configuration. > > > > Any thoughts why these interrupts are not getting delivered on the > > PCIE bus? > > > > We do not configure the virtio-scsi-pci on a PCIe bus currently, see > my initial answer. > > Best Regards, > Fiona > > > > pve-devel mailing list > pve-devel@lists.proxmox.com > https://antiphishing.vadesecure.com/v4?f=MDk0SW9xRkhTVGYydkJlTIW3tbTra > K > > 7neiUoWcvis0-pokd-Q2cwuWCZhgBcTw_yw4KqJS-oP6jCsk-zj- > 1YMQ&i=ZURHSDhnY0huQ2tPS3VZahJdhRaQu1ItpJrYkl8wXrA&k=q1N6&r=RjIyR1Robk > V > > xVWlHTXhKT3I72JjP2S9ryFg3B_csBogfeb2oROpP8B8yUJUd6awk&s=50133960c8716b > 5 > > 426bc084f398f7760f04af8739fd68cad36d17b1dcd887778&u=https%3A%2F%2Flist > s > > .proxmox.com%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpve-devel > > > _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... 2024-08-12 10:40 [pve-devel] issues with Virtio-SCSI devicde on Proxmox Christian Moser 2024-08-13 8:55 ` Fiona Ebner @ 2024-08-13 16:12 ` DERUMIER, Alexandre via pve-devel [not found] ` <67b222322506a9eab0f7cf7da5a9fd715c8a91ff.camel@groupe-cyllene.com> 2 siblings, 0 replies; 11+ messages in thread From: DERUMIER, Alexandre via pve-devel @ 2024-08-13 16:12 UTC (permalink / raw) To: pve-devel; +Cc: DERUMIER, Alexandre, cmos [-- Attachment #1: Type: message/rfc822, Size: 19027 bytes --] From: "DERUMIER, Alexandre" <alexandre.derumier@groupe-cyllene.com> To: "pve-devel@lists.proxmox.com" <pve-devel@lists.proxmox.com> Cc: "cmos@maklee.com" <cmos@maklee.com> Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Date: Tue, 13 Aug 2024 16:12:42 +0000 Message-ID: <67b222322506a9eab0f7cf7da5a9fd715c8a91ff.camel@groupe-cyllene.com> Hi, Currently they are no way to add custom options for virtio-devices command line from vm config, so it should be patched to add support for openvms os and add special tuning. for example:https://git.proxmox.com/?p=qemu-server.git;a=blob_plain;f=PVE/QemuServer.pm;hb=HEAD sub print_netdevice_full { .... if ($conf->{ostype} && $conf->{ostype} eq 'openvms') { tmpstr .= ",disable_legacy=on"; } >>I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no >>luck. The emulated virtio-scsi device is a legacy device >> >> >>Questions: >>* is there a way to configure a modern virtio-scsi devcie (i.e. >>disable_legacy=on) ? >> Do you mean than currently on proxmox, the default is legacy mode and and not modern ? I was pretty sure than qemu was defaulting to modern, and that you need to use ",disable-legacy=off,disable-modern=true" to switch to legacy. (and AFAIK, proxmox qemu version don't have any patch forcing legacy) but maybe I'm wrong maybe it's only enabled by default with pci-express (using q35 machine model) ? https://github.com/qemu/qemu/blob/9eb51530c12ae645b91e308d16196c68563ea883/docs/pcie.txt "Virtio devices plugged into PCI Express ports are PCI Express devices and have "1.0" behavior by default without IO support. In both cases disable-legacy and disable-modern properties can be used to override the behaviour." in qemu code I see: https://github.com/qemu/qemu/blob/9eb51530c12ae645b91e308d16196c68563ea883/hw/core/machine.c#L258 GlobalProperty hw_compat_2_6[] = { { "virtio-mmio", "format_transport_address", "off" }, /* Optional because not all virtio-pci devices support legacy mode */ { "virtio-pci", "disable-modern", "on", .optional = true }, { "virtio-pci", "disable-legacy", "off", .optional = true }, }; const size_t hw_compat_2_6_len = G_N_ELEMENTS(hw_compat_2_6); so I'm really not sure what is the real default do you have qemu command line of another kvm hypervisor (ovirt,rhev,...) where it's working correctly ? Maybe they are enabling modern bits manually , when a virtio device is supporting it ? -------- Message initial -------- De: Christian Moser <cmos@maklee.com> Répondre à: Proxmox VE development discussion <pve- devel@lists.proxmox.com> À: pve-devel@lists.proxmox.com Cc: Christian Moser <cmos@maklee.com> Objet: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Date: 12/08/2024 12:40:10 Hello, I work for VSI (VMS Software Inc) which is porting the OpenVMS operating system to x86. At this point we successfully on various hypervisors, but have some issues on the KVM running on Proxmox. The OpenVMS VM works just fine with SATA disks and it also works with for example virtio-network device etc., but trying to use virtio-scsi hangs the driver. I have debugged this and I can successfully configure the port/controller, send the IO request to the device. It then gets processed by the device, which posts the results and sets the interrupt bit in the ISR register, but it never asserts the interrupt hence the driver never gets notified and the I/O hangs. I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no luck. The emulated virtio-scsi device is a legacy device. But then again, the virtio-network device is also a legacy device and here we are getting interrupts. One thing which bothers me is the fact that the “legacy interrupt disable” bit is set in the PCI config space of the virtio-scsi device (i.e. bit 10 at offset 4) Questions: * is there a way to configure a modern virtio-scsi devcie (i.e. disable_legacy=on) ? * why is the legacy interrupt bit set in the PCI config space ? * Are there any working driver for virtio-scsi on this KVM using Q35 machine? i.e. any other OS Any thoughts why these interrupts are not getting delivered on the PCIE bus? thanks /cmos _______________________________________________________ Christian Moser Mobile: +358-40-5022105 Email: cmos@maklee.com URL: www.maklee.com _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://antiphishing.vadesecure.com/v4?f=MXNCQ25TYXBra2RFV0VXZHojMIIFPS zCvkhJApFQpU82zfAhP9K7STA7y- wx1jAbgWgLhPFHm0qw3LR0qKok0w&i=cVVpR014Sk93Z0c1QzFSMs3n08s7G9NqllHFFNJ9 1W0&k=ThnE&r=VWoxMWNDTTl5UmRwenhPYicvAeXd4xlreb0WEHQKfL1pVhoIeFvTknKKfK g5BYLZ&s=c0303a1ebcdeef9c5d38eaa27d25b873b9eed7bddb72bc270d5944f93d63bb ec&u=https%3A%2F%2Flists.proxmox.com%2Fcgi- bin%2Fmailman%2Flistinfo%2Fpve-devel [-- Attachment #2: Type: text/plain, Size: 160 bytes --] _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 11+ messages in thread
[parent not found: <67b222322506a9eab0f7cf7da5a9fd715c8a91ff.camel@groupe-cyllene.com>]
* [pve-devel] FW: issues with Virtio-SCSI devicde on Proxmox... [not found] ` <67b222322506a9eab0f7cf7da5a9fd715c8a91ff.camel@groupe-cyllene.com> @ 2024-08-14 6:42 ` Christian Moser 0 siblings, 0 replies; 11+ messages in thread From: Christian Moser @ 2024-08-14 6:42 UTC (permalink / raw) To: 'DERUMIER, Alexandre', pve-devel; +Cc: Christian Moser Alexandre, thank you for your information. On another KVM, the virtio-scsi device has the hardware id of 0x10481AF4, whereas on Proxmox is has 0x10041AF4, which is a legacy device. But after more testing, I do not think that the issue is related to legacy vs modern device, but rather interrupt related, because our driver on Proxmox does not receive the device interrupt. I'm doing now more homework on our side to see if the system (i.e. OpenVMS) does get the interrupt, but is not delivering it to the device. So I'm looking at out bus support code to check out things related to interrupt delivery. If the system does get the interrupt (since the virtio-scsi device did set the interrupt bit in the ISR register), then the issue is on our side. If we do not get the interrupt at all, then there is still something going with Proxmox and OpenVMS guest OS. Thanks for your answers and pointers, but for now, I do some more homework /cmos _______________________________________________________ Christian Moser Mobile: +358-40-5022105 Email: cmos@maklee.com URL: www.maklee.com -----Original Message----- From: DERUMIER, Alexandre <alexandre.derumier@groupe-cyllene.com> Sent: Tuesday, August 13, 2024 19:13 To: pve-devel@lists.proxmox.com Cc: cmos@maklee.com Subject: Re: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Hi, Currently they are no way to add custom options for virtio-devices command line from vm config, so it should be patched to add support for openvms os and add special tuning. for example:https://git.proxmox.com/?p=qemu-server.git;a=blob_plain;f=PVE/QemuServer.pm;hb=HEAD sub print_netdevice_full { .... if ($conf->{ostype} && $conf->{ostype} eq 'openvms') { tmpstr .= ",disable_legacy=on"; } >>I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no >>luck. The emulated virtio-scsi device is a legacy device >> >> >>Questions: >>* is there a way to configure a modern virtio-scsi devcie (i.e. >>disable_legacy=on) ? >> Do you mean than currently on proxmox, the default is legacy mode and and not modern ? I was pretty sure than qemu was defaulting to modern, and that you need to use ",disable-legacy=off,disable-modern=true" to switch to legacy. (and AFAIK, proxmox qemu version don't have any patch forcing legacy) but maybe I'm wrong maybe it's only enabled by default with pci-express (using q35 machine model) ? https://github.com/qemu/qemu/blob/9eb51530c12ae645b91e308d16196c68563ea883/docs/pcie.txt "Virtio devices plugged into PCI Express ports are PCI Express devices and have "1.0" behavior by default without IO support. In both cases disable-legacy and disable-modern properties can be used to override the behaviour." in qemu code I see: https://github.com/qemu/qemu/blob/9eb51530c12ae645b91e308d16196c68563ea883/hw/core/machine.c#L258 GlobalProperty hw_compat_2_6[] = { { "virtio-mmio", "format_transport_address", "off" }, /* Optional because not all virtio-pci devices support legacy mode */ { "virtio-pci", "disable-modern", "on", .optional = true }, { "virtio-pci", "disable-legacy", "off", .optional = true }, }; const size_t hw_compat_2_6_len = G_N_ELEMENTS(hw_compat_2_6); so I'm really not sure what is the real default do you have qemu command line of another kvm hypervisor (ovirt,rhev,...) where it's working correctly ? Maybe they are enabling modern bits manually , when a virtio device is supporting it ? -------- Message initial -------- De: Christian Moser <cmos@maklee.com> Répondre à: Proxmox VE development discussion <pve- devel@lists.proxmox.com> À: pve-devel@lists.proxmox.com Cc: Christian Moser <cmos@maklee.com> Objet: [pve-devel] issues with Virtio-SCSI devicde on Proxmox... Date: 12/08/2024 12:40:10 Hello, I work for VSI (VMS Software Inc) which is porting the OpenVMS operating system to x86. At this point we successfully on various hypervisors, but have some issues on the KVM running on Proxmox. The OpenVMS VM works just fine with SATA disks and it also works with for example virtio-network device etc., but trying to use virtio-scsi hangs the driver. I have debugged this and I can successfully configure the port/controller, send the IO request to the device. It then gets processed by the device, which posts the results and sets the interrupt bit in the ISR register, but it never asserts the interrupt hence the driver never gets notified and the I/O hangs. I have tried both “virtio-scsi-pci” and “virtio-scsi-single”, but no luck. The emulated virtio-scsi device is a legacy device. But then again, the virtio-network device is also a legacy device and here we are getting interrupts. One thing which bothers me is the fact that the “legacy interrupt disable” bit is set in the PCI config space of the virtio-scsi device (i.e. bit 10 at offset 4) Questions: * is there a way to configure a modern virtio-scsi devcie (i.e. disable_legacy=on) ? * why is the legacy interrupt bit set in the PCI config space ? * Are there any working driver for virtio-scsi on this KVM using Q35 machine? i.e. any other OS Any thoughts why these interrupts are not getting delivered on the PCIE bus? thanks /cmos _______________________________________________________ Christian Moser Mobile: +358-40-5022105 Email: cmos@maklee.com URL: www.maklee.com _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://antiphishing.vadesecure.com/v4?f=MXNCQ25TYXBra2RFV0VXZHojMIIFPS zCvkhJApFQpU82zfAhP9K7STA7y- wx1jAbgWgLhPFHm0qw3LR0qKok0w&i=cVVpR014Sk93Z0c1QzFSMs3n08s7G9NqllHFFNJ9 1W0&k=ThnE&r=VWoxMWNDTTl5UmRwenhPYicvAeXd4xlreb0WEHQKfL1pVhoIeFvTknKKfK g5BYLZ&s=c0303a1ebcdeef9c5d38eaa27d25b873b9eed7bddb72bc270d5944f93d63bb ec&u=https%3A%2F%2Flists.proxmox.com%2Fcgi- bin%2Fmailman%2Flistinfo%2Fpve-devel _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-08-19 8:37 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-08-12 10:40 [pve-devel] issues with Virtio-SCSI devicde on Proxmox Christian Moser
2024-08-13 8:55 ` Fiona Ebner
2024-08-13 9:33 ` [pve-devel] FW: " Christian Moser
2024-08-14 6:44 ` [pve-devel] " DERUMIER, Alexandre via pve-devel
[not found] ` <1178cb3475d719ae31f0c375cd3930fc24d98401.camel@groupe-cyllene.com>
2024-08-14 7:20 ` [pve-devel] FW: " Christian Moser
2024-08-14 12:22 ` DERUMIER, Alexandre via pve-devel
[not found] ` <c851a27f4f5eca7c767fbffa48cb208e2d8fe1e6.camel@groupe-cyllene.com>
2024-08-14 12:48 ` Christian Moser
2024-08-14 13:05 ` DERUMIER, Alexandre via pve-devel
[not found] ` <6ea8c90b469da48525ed743b6541bb3644be4a6c.camel@groupe-cyllene.com>
2024-08-15 5:35 ` [pve-devel] FW: " Christian Moser
2024-08-13 16:12 ` [pve-devel] " DERUMIER, Alexandre via pve-devel
[not found] ` <67b222322506a9eab0f7cf7da5a9fd715c8a91ff.camel@groupe-cyllene.com>
2024-08-14 6:42 ` [pve-devel] FW: " Christian Moser
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox