From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 15E5C7375C for ; Thu, 7 Oct 2021 15:46:04 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 1153C1AC5F for ; Thu, 7 Oct 2021 15:45:34 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id C593E1AC49 for ; Thu, 7 Oct 2021 15:45:32 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 883074588F for ; Thu, 7 Oct 2021 15:45:32 +0200 (CEST) From: Dominik Csapak To: pve-devel@lists.proxmox.com Date: Thu, 7 Oct 2021 15:45:31 +0200 Message-Id: <20211007134531.1693674-4-d.csapak@proxmox.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20211007134531.1693674-1-d.csapak@proxmox.com> References: <20211007134531.1693674-1-d.csapak@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.302 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [qemuserver.pm] Subject: [pve-devel] [PATCH qemu-server v3 3/3] fix #3258: block vm start when pci device is already in use X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Oct 2021 13:46:04 -0000 on vm start, we reserve all pciids that we use, and remove the reservation again in vm_stop_cleanup first with only a time-based reservation but after the vm is started, we reserve again but with the pid. for this, we have to move the start_timeout calculation above the hostpci handling. also moved the pci initialization out of the conf parsing loop so that we can reserve all ids before we actually touch any of them while touching the lines, fix the indentation this way, when a vm starts with a pci device that is already configured for a different running vm, will not be started and the user gets the error that the device is already in use Signed-off-by: Dominik Csapak --- PVE/QemuServer.pm | 50 +++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 8 deletions(-) diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm index f78b2cc..e504e9a 100644 --- a/PVE/QemuServer.pm +++ b/PVE/QemuServer.pm @@ -5381,16 +5381,40 @@ sub vm_start_nolock { push @$cmd, '-S'; } + my $start_timeout = $params->{timeout} // config_aware_timeout($conf, $resume); + my $pciids = []; + my $pci_devices = {}; + # host pci devices for (my $i = 0; $i < $PVE::QemuServer::PCI::MAX_HOSTPCI_DEVICES; $i++) { - my $d = parse_hostpci($conf->{"hostpci$i"}); - next if !$d; - my $pcidevices = $d->{pciid}; - foreach my $pcidevice (@$pcidevices) { - my $pciid = $pcidevice->{id}; + my $d = parse_hostpci($conf->{"hostpci$i"}); + next if !$d; + $pci_devices->{$i} = $d; - PVE::QemuServer::PCI::prepare_pci_device($vmid, $pciid, $i, $d->{mdev}); - } + my $pcidevices = $d->{pciid}; + + my $ids = [map { $_->{id} } @$pcidevices]; + push @$pciids, @$ids; + } + + # reserve all pci ids before actually doing anything with them + PVE::QemuServer::PCI::reserve_pci_usage($pciids, $vmid, $start_timeout); + + eval { + for my $i (sort keys %$pci_devices) { + my $d = $pci_devices->{$i}; + my $pcidevices = $d->{pciid}; + foreach my $pcidevice (@$pcidevices) { + my $pciid = $pcidevice->{id}; + PVE::QemuServer::PCI::prepare_pci_device($vmid, $pciid, $i, $d->{mdev}); + } + } + }; + + if (my $err = $@) { + eval { PVE::QemuServer::PCI::remove_pci_reservation($pciids) }; + warn $@ if $@; + die $err; } PVE::Storage::activate_volumes($storecfg, $vollist); @@ -5405,7 +5429,6 @@ sub vm_start_nolock { my $cpuunits = get_cpuunits($conf); - my $start_timeout = $params->{timeout} // config_aware_timeout($conf, $resume); my %run_params = ( timeout => $statefile ? undef : $start_timeout, umask => 0077, @@ -5485,9 +5508,17 @@ sub vm_start_nolock { if (my $err = $@) { # deactivate volumes if start fails eval { PVE::Storage::deactivate_volumes($storecfg, $vollist); }; + eval { PVE::QemuServer::PCI::remove_pci_reservation($pciids) }; + die "start failed: $err"; } + # reserve all pciids again with the pid + # the vm is already started, we can only warn on error here + my $pid = PVE::QemuServer::Helpers::vm_running_locally($vmid); + eval { PVE::QemuServer::PCI::reserve_pci_usage($pciids, $vmid, undef, $pid) }; + warn $@ if $@; + print "migration listens on $migrate_uri\n" if $migrate_uri; $res->{migrate_uri} = $migrate_uri; @@ -5676,6 +5707,7 @@ sub vm_stop_cleanup { unlink '/dev/shm/pve-shm-' . ($ivshmem->{name} // $vmid); } + my $ids = []; foreach my $key (keys %$conf) { next if $key !~ m/^hostpci(\d+)$/; my $hostpciindex = $1; @@ -5684,9 +5716,11 @@ sub vm_stop_cleanup { foreach my $pci (@{$d->{pciid}}) { my $pciid = $pci->{id}; + push @$ids, $pci->{id}; PVE::SysFSTools::pci_cleanup_mdev_device($pciid, $uuid); } } + PVE::QemuServer::PCI::remove_pci_reservation($ids); vmconfig_apply_pending($vmid, $conf, $storecfg) if $apply_pending_changes; }; -- 2.30.2