From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 0E7BA98588 for ; Wed, 15 Nov 2023 11:12:48 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id E605E313D for ; Wed, 15 Nov 2023 11:12:47 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 15 Nov 2023 11:12:47 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id E03D842EFA for ; Wed, 15 Nov 2023 11:12:46 +0100 (CET) Date: Wed, 15 Nov 2023 11:12:45 +0100 From: Wolfgang Bumiller To: Fabian =?utf-8?Q?Gr=C3=BCnbichler?= Cc: Proxmox VE development discussion Message-ID: References: <20231114140204.27679-1-f.ebner@proxmox.com> <20231114140204.27679-2-f.ebner@proxmox.com> <1700038325.l2cibe7y7o.astroid@yuna.none> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1700038325.l2cibe7y7o.astroid@yuna.none> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.101 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [qemuserver.pm] Subject: Re: [pve-devel] [RFC qemu-server 1/1] partially fix #4501: migration: start vm: move port reservation and usage closer together X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Nov 2023 10:12:48 -0000 On Wed, Nov 15, 2023 at 09:55:22AM +0100, Fabian Grünbichler wrote: > On November 14, 2023 3:02 pm, Fiona Ebner wrote: > > Currently, volume activation, PCI reservation and resetting systemd > > scope happen in between and the 5 second expiretime used for port > > reservation might not be enough. > > > > Still not ideal, because entering systemd scope and maybe starting > > swtpm still happen after reservation before the QEMU binary can be > > invoked and actually use the port, but the reservation needs to happen > > outside of the fork, because the result is used there too. > > > > Signed-off-by: Fiona Ebner > > Acked-by: Fabian Grünbichler > > we could move the whole statefile handling further down, but then some > additional side-effects need to be taken care of/refactored, this seems > like a minimal-invasive version for the uncommon (insecure) case. > > > --- > > PVE/QemuServer.pm | 20 ++++++++++++++------ > > 1 file changed, 14 insertions(+), 6 deletions(-) > > > > diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm > > index c465fb6f..aeaea8eb 100644 > > --- a/PVE/QemuServer.pm > > +++ b/PVE/QemuServer.pm > > @@ -5697,6 +5697,9 @@ sub vm_start_nolock { > > return $migration_ip; > > }; > > > > + # helper to move port reservation and usage closer together to avoid expiry (bug #4501) > > + my $append_tcp_migration_cmdline; > > + > > if ($statefile) { > > if ($statefile eq 'tcp') { > > my $migrate = $res->{migrate} = { proto => 'tcp' }; > > @@ -5717,12 +5720,13 @@ sub vm_start_nolock { > > $migrate->{addr} = "[$migrate->{addr}]" if Net::IP::ip_is_ipv6($migrate->{addr}); > > } > > > > - my $pfamily = PVE::Tools::get_host_address_family($nodename); > > - $migrate->{port} = PVE::Tools::next_migrate_port($pfamily); > > - $migrate->{uri} = "tcp:$migrate->{addr}:$migrate->{port}"; > > - push @$cmd, '-incoming', $migrate->{uri}; > > - push @$cmd, '-S'; > > - > > nit: I'd maybe add another comment here, maybe something like > > # delay migration port reservation to prevent expiry before binding What about adding an option to `next_migrate_port()` to actually return the open socket to keep the reservation? Also, did we consider passing the file descriptor through to qemu via `-incoming fd:$number`? > > > + $append_tcp_migration_cmdline = sub { > > + my $pfamily = PVE::Tools::get_host_address_family($nodename); > > + $migrate->{port} = PVE::Tools::next_migrate_port($pfamily); > > + $migrate->{uri} = "tcp:$migrate->{addr}:$migrate->{port}"; > > + push @$cmd, '-incoming', $migrate->{uri}; > > + push @$cmd, '-S'; > > + }; > > } elsif ($statefile eq 'unix') { > > # should be default for secure migrations as a ssh TCP forward > > # tunnel is not deterministic reliable ready and fails regurarly > > @@ -5840,6 +5844,10 @@ sub vm_start_nolock { > > $systemd_properties{timeout} = 10 if $statefile; # setting up the scope shoul be quick > > > > my $run_qemu = sub { > > + # sets the port+uri for $res->{migrate} which is printed below and part of the result, so > > + # needs to happen outside of the fork. > > + $append_tcp_migration_cmdline->() if $append_tcp_migration_cmdline; > > + > > PVE::Tools::run_fork sub { > > PVE::Systemd::enter_systemd_scope($vmid, "Proxmox VE VM $vmid", %systemd_properties); > > > > -- > > 2.39.2