From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 2F265752F1 for ; Wed, 23 Jun 2021 19:57:45 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 24A1FECEA for ; Wed, 23 Jun 2021 19:57:15 +0200 (CEST) Received: from mx1.tetaneutral.net (mx1.tetaneutral.net [IPv6:2a03:7220:8081:5300::1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 371E0ECDF for ; Wed, 23 Jun 2021 19:57:13 +0200 (CEST) Received: by mx1.tetaneutral.net (Postfix, from userid 109) id 5B9178C0DE; Wed, 23 Jun 2021 19:56:57 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mx1.tetaneutral.net X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, FSL_HELO_NON_FQDN_1 autolearn=unavailable autolearn_force=no version=3.4.2 Received: from lgnuc (ip165.tetaneutral.net [IPv6:2a03:7220:8080:a500::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.tetaneutral.net (Postfix) with ESMTPS id 5CCA58C063; Wed, 23 Jun 2021 19:56:54 +0200 (CEST) Message-ID: <1624471013.17569.11.camel@guerby.net> From: Laurent GUERBY To: Proxmox VE development discussion , Fabian Ebner Date: Wed, 23 Jun 2021 19:56:53 +0200 In-Reply-To: References: <20210408103316.7619-1-f.ebner@proxmox.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.22.6-1+deb9u2 Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [POC qemu-server] fix 3303: allow "live" upgrade of qemu version X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jun 2021 17:57:45 -0000 On Thu, 2021-04-08 at 18:44 +0200, Thomas Lamprecht wrote: > On 08.04.21 12:33, Fabian Ebner wrote: > > The code is in a very early state, I'm just sending this to discuss > > the idea. > > I didn't do a whole lot of testing yet, but it does seem to work. > > > > The idea is rather simple: > > 1. save the state to ramfs > > 2. stop the VM > > 3. start the VM loading the state > > For the record, as we (Dietmar, you and I) discussed this a bit off- > list: > > The issue we see here is that one temporarily requires a potential > big chunk of > free memory, i.e., another time the amount the guest is assigned. So > tens to > hundreds of GiB, which (educated guess) > 90 % of our users just do > not have > available, at least for the bigger VMs of theirs. > > So, it would be nicer if we could makes this more QEMU internal, > e.g., just save > the state out (as that one may not be compatible 1:1 for reuse with > the new QEMU > version) and re-use the guest memory directly, e.g., start new QEMU > process > migrate state and map over the guest-memory, then pause old one, cont > new one and > be done (very condensed). > That may have it's own difficulties/edge-cases, but it would not > require having > so much extra memory freely available... Hi, I'm wondering how much ksm would help reduce the extra memory requirement during same host migration. May be there's a sweet spot by changing ksm to be more aggressive just before starting the migration and slowing down the migration using bandwidth control parameter so all new pages created by the migration process end up shared quickly? And returning ksmtuned to default after it's done. Or may be only lowering migration bandwidth will be enough with ksm settings unchanged (still has to be faster than mutation rate though so can't be too low). I assume for most users even if the migration to same host is slow it's fine since it will not consume network ressources, just a bit more cpu. Sincerely, Laurent PS: thanks Stefan_R for pointing this thread https://forum.proxmox.com/threads/upgrade-of-pve-qemu-kvm-and-running-v m.91236/ > > > > This approach solves the problem that our stack is (currently) not > > designed to > > have multiple instances with the same VM ID running. To do so, we'd > > need to > > handle config locking, sockets, pid file, passthrough resources?, > > etc. > > > > Another nice feature of this approach is that it doesn't require > > touching the > > vm_start or migration code at all, avoiding further bloating. > > > > > > Thanks to Fabian G. and Stefan for inspiring this idea: > > > > Fabian G. suggested using the suspend to disk + start route if the > > required > > changes to our stack would turn out to be infeasable. > > > > Stefan suggested migrating to a dummy VM (outside our stack) which > > just holds > > the state and migrating back right away. It seems that dummy VM is > > in fact not > > even needed ;) If we really really care about smallest possible > > downtime, this > > approach might still be the best, and we'd need to start the dummy > > VM while the > > backwards migration runs (resulting in two times the migration > > downtime). But > > it does have more moving parts and requires some migration/startup > > changes. > > > > > > Fabian Ebner (6): > >   create vmstate_size helper > >   create savevm_monitor helper > >   draft of upgrade_qemu function > >   draft of qemuupgrade API call > >   add timing for testing > >   add usleep parameter to savevm_monitor > > > >  PVE/API2/Qemu.pm  |  60 ++++++++++++++++++++++ > >  PVE/QemuConfig.pm |  10 +--- > >  PVE/QemuServer.pm | 125 +++++++++++++++++++++++++++++++++++++++--- > > ---- > >  3 files changed, 170 insertions(+), 25 deletions(-) > > > > > > _______________________________________________ > pve-devel mailing list > pve-devel@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel >