From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id B895E1FF167 for ; Wed, 31 Jul 2024 15:41:06 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id A5603EEE5; Wed, 31 Jul 2024 15:41:09 +0200 (CEST) From: Aaron Lauterer To: pve-devel@lists.proxmox.com Date: Wed, 31 Jul 2024 15:40:32 +0200 Message-Id: <20240731134033.1733932-2-a.lauterer@proxmox.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240731134033.1733932-1-a.lauterer@proxmox.com> References: <20240731131658.1719688-1-a.lauterer@proxmox.com> <20240731134033.1733932-1-a.lauterer@proxmox.com> MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.039 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [drive.pm, gnu.org, qemuserver.pm] Subject: [pve-devel] [WIP qemu-server 2/3] nvme: make it somewhat work with current version X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox VE development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" this patch is not meant to be applied but captures my efforts to make the old series work (somewhat). This way I was able to achieve hot-plugging on an i440 VM. Hot plugging on a q35 VM did not work in my tests, but I think it is related to how the NVME device was attached to the PCI tree of the VM. By controlling this more directly, we should be able to attach it to a PCI port/bridge that allows hot-plugging. For some reason that I need to investigate further, giving the nvme device the id `nvmeX` will cause issues when unplugging. It seems our code will still see the `drive-nvmeX` drive and will add it to the list of devices present. Therefore, the unplug code will think it did not remove successfully. I worked around that for now by calling the nvme device `id=nvmecontX` instead of `id=nvmeX`. Live migration is still a blocker as the NVME QEMU device does not allow it. It fails with the following errors: 2024-07-31 09:31:14 migrate uri => unix:/run/qemu-server/102.migrate failed: VM 102 qmp command 'migrate' failed - State blocked by non-migratable device '0000:00:04.0/nvme' 2024-07-31 09:31:15 ERROR: online migrate failure - VM 102 qmp command 'migrate' failed - State blocked by non-migratable device '0000:00:04.0/nvme' We need to check (with upstream) what is missing. Also if there are plans upstream to implement the missing pieces, or if we could do it ourselves. I found the following older thread on the qemu mailing list: https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg05788.html Signed-off-by: Aaron Lauterer --- PVE/QemuServer.pm | 15 +++++++++------ PVE/QemuServer/Drive.pm | 1 + 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm index 7d8d75b..6dc3eb2 100644 --- a/PVE/QemuServer.pm +++ b/PVE/QemuServer.pm @@ -1436,7 +1436,7 @@ sub print_drivedevice_full { } elsif ($drive->{interface} eq 'nvme') { my $path = $drive->{file}; $drive->{serial} //= "$drive->{interface}$drive->{index}"; # serial is mandatory for nvme - $device = "nvme,drive=drive-$drive->{interface}$drive->{index},id=$drive->{interface}$drive->{index}"; + $device = "nvme,drive=drive-$drive->{interface}$drive->{index},id=$drive->{interface}cont$drive->{index}"; } elsif ($drive->{interface} eq 'ide' || $drive->{interface} eq 'sata') { my $maxdev = ($drive->{interface} eq 'sata') ? $PVE::QemuServer::Drive::MAX_SATA_DISKS : 2; my $controller = int($drive->{index} / $maxdev); @@ -4292,11 +4292,10 @@ sub vm_deviceplug { warn $@ if $@; die $err; } - } elsif ($deviceid =~ m/^(nvme)(\d+)$/) { - + } elsif ($deviceid =~ m/^nvme(\d+)$/) { qemu_driveadd($storecfg, $vmid, $device); - my $devicefull = print_drivedevice_full($storecfg, $conf, $vmid, $device, $arch, $machine_type); + my $devicefull = print_drivedevice_full($storecfg, $conf, $vmid, $device, undef, $arch, $machine_type); eval { qemu_deviceadd($vmid, $devicefull); }; if (my $err = $@) { eval { qemu_drivedel($vmid, $deviceid); }; @@ -4375,8 +4374,12 @@ sub vm_deviceunplug { qemu_iothread_del($vmid, "virtioscsi$device->{index}", $device) if $conf->{scsihw} && ($conf->{scsihw} eq 'virtio-scsi-single'); - } elsif ($deviceid =~ m/^(nvme)(\d+)$/) { - qemu_devicedel($vmid, $deviceid); + } elsif ($deviceid =~ m/^nvme(\d+)$/) { + my $device = parse_drive($deviceid, $conf->{$deviceid}); + + my $nvmecont = "nvmecont$1"; + qemu_devicedel($vmid, $nvmecont); + qemu_devicedelverify($vmid, $nvmecont); qemu_drivedel($vmid, $deviceid); } elsif ($deviceid =~ m/^(net)(\d+)$/) { qemu_devicedel($vmid, $deviceid); diff --git a/PVE/QemuServer/Drive.pm b/PVE/QemuServer/Drive.pm index f05ad26..b055674 100644 --- a/PVE/QemuServer/Drive.pm +++ b/PVE/QemuServer/Drive.pm @@ -532,6 +532,7 @@ for (my $i = 0; $i < $MAX_SCSI_DISKS; $i++) { for (my $i = 0; $i < $MAX_NVME_DISKS; $i++) { $drivedesc_hash->{"nvme$i"} = $nvmedesc; + $drivedesc_hash_with_alloc->{"nvme$i"} = $desc_with_alloc->('nvme', $nvmedesc); } -- 2.39.2 _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel