From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 2E2428D3D9 for ; Mon, 7 Nov 2022 22:59:49 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 115E92FC83 for ; Mon, 7 Nov 2022 22:59:19 +0100 (CET) Received: from gmmr-3.centrum.cz (gmmr-3.centrum.cz [IPv6:2a00:da80:0:502::8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 7 Nov 2022 22:59:17 +0100 (CET) Received: from gmmr-3.centrum.cz (localhost [127.0.0.1]) by gmmr-3.centrum.cz (Postfix) with ESMTP id AB98A204C2BE for ; Mon, 7 Nov 2022 22:59:04 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=volny.cz; s=mail; t=1667858344; bh=TnRhxGK75m7N1ThmXTK8M89/jdNsRKquY4a/Ej9kwtc=; h=From:Subject:Date:References:To:In-Reply-To:From; b=aFpYu967yV43TlcUoxZUaPSqpAtrmcQp+88t0d3OuNAeiU5lrdvWAPKRSzSFGKGrX VgI6rBbR8suL119bs8Y626YYRhSvAWtiiGaSAhb+8cc345NmNkWKV3eOIeCl09907b TZHY51JkNwx8ZMNtlgiPeBd7HIg0zPWkda5Fvidk= Received: from vm1.excello.cz (vm1.excello.cz [IPv6:2001:67c:1591::3]) by gmmr-3.centrum.cz (Postfix) with QMQP id A90762022461 for ; Mon, 7 Nov 2022 22:59:04 +0100 (CET) Received: from vm1.excello.cz by vm1.excello.cz (VF-Scanner: Clear:RC:0(2a00:da80:1:502::7):SC:0(-4.4/5.0):CC:0:; processed in 1.1 s); 07 Nov 2022 21:59:04 +0000 X-VF-Scanner-ID: 20221107215903.559205.2916.vm1.excello.cz.0 X-Spam-Status: No, hits=-4.4, required=5.0 Received: from gmmr-2.centrum.cz (2a00:da80:1:502::7) by out2.virusfree.cz with ESMTPS (TLSv1.3, TLS_AES_256_GCM_SHA384); 7 Nov 2022 22:59:03 +0100 Received: from gm-smtp10.centrum.cz (envoy-stl.cent [10.32.56.18]) by gmmr-2.centrum.cz (Postfix) with ESMTP id 7516C20388B7 for ; Mon, 7 Nov 2022 22:59:03 +0100 (CET) Received: from smtpclient.apple (unknown [10.128.64.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by gm-smtp10.centrum.cz (Postfix) with ESMTPSA id 707BC1683C7 for ; Mon, 7 Nov 2022 22:59:03 +0100 (CET) From: Jan Vlach Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Date: Mon, 7 Nov 2022 22:59:01 +0100 References: <1378480941-319@kerio.tuxis.nl> To: Proxmox VE user list In-Reply-To: Message-Id: <7827641E-40A1-4E5D-8EDF-4E37BA2BD5AB@volny.cz> X-Mailer: Apple Mail (2.3696.120.41.1.1) X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain HTML_MESSAGE 0.001 HTML included in message RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] VMs hung after live migration - Intel CPU X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Nov 2022 21:59:49 -0000 Hi, For what=E2=80=99s it worth, live VM migration with Linux VMs with = various debian versions work here just fine. I=E2=80=99m using virtio = for networking and virtio scsi for disks. (The only version where I had = problems was debian6 where the kernel does not support virtio scsi and = megaraid sas 8708EM2 needs to be used. I get kernel panic in mpt_sas on = thaw after migration.) We're running 5.15.60-1-pve on three node cluster with AMD EPYC 7551P = 32-Core Processor. These are supermicros with latest bios (latest = microcode?) and BMC=20 Storage is local ZFS pool, backed by SSDS in striped mirrors (4 devices = on each node). Migration has dedicated 2x 10GigE LACP and dedicated VLAN = on switch stack.=20 I have more nodes with EPYC3/Milan on the way, so I=E2=80=99ll test = those later as well. What does your cluster look hardware-wise? What are the problems you = experienced with VM migratio on 5.13->5.19?=20 Thanks, JV > On 7. 11. 2022, at 14:40, Eneko Lacunza via pve-user = wrote: >=20 >=20 > From: Eneko Lacunza > Subject: Re: [PVE-User] VMs hung after live migration - Intel CPU > Date: 7 November 2022 14:40:07 CET > To: Mark Schouten , Proxmox VE user list = >=20 >=20 > Hi, >=20 > Sadly I'm not sure what is best. For most of the clusters we admin, I = have decided to stay in 5.13 (pinning that version with = proxmox-boot-tool) because 5.19 seems will receive much more changes and = it will be more unstable... >=20 > Cheers >=20 > El 7/11/22 a las 13:56, Mark Schouten escribi=C3=B3: >> Hi, >>=20 >>=20 >> Thanks. What would you suggest? Downgrading to 5.13 ? >>=20 >> --=20 >> Mark Schouten >> CTO, Tuxis B.V. | https://www.tuxis.nl/ >> | +31 318 200208 >>=20 >>=20 >> *From: * Eneko Lacunza >> *To: * Mark Schouten , Proxmox VE user list = >> *Sent: * 2022-11-07 9:23 >> *Subject: * Re: [PVE-User] VMs hung after live migration - Intel CPU >>=20 >> Hi, >>=20 >> 5.15 has been a disaster for us, issues seem to have no end. >> Frankly, I don't understand how can it be the official supported >> kernel in PVE 7.2 right now. >>=20 >> Our tests with 5.19 in a pair of nodes (in another cluster) seem >> good, but I don't think 5.13 -> 5.19 migration is working well >> either. Both kernels not being the "official" one, I'm unable to >> decide what to do with our clusters... >>=20 >> This has been ongoing for some months... :-( >>=20 >> I see 5.15.64 has been promoted to enterprise repo this weekend, >> no idea if any attempt to fix live migration issues is included... >>=20 >> Thanks >>=20 >> El 6/11/22 a las 9:04, Mark Schouten escribi=C3=B3: >>> Hi, >>>=20 >>> I=E2=80=99ve seen the same behavior between two AMD cpu=E2=80=99s = with the -60 kernel. One of the vm=E2=80=99s the =E2=80=98crashed=E2=80=99= even started working after migrating back again.. >>>=20 >>> I=E2=80=99m probably going to 5.19, I=E2=80=99ve heard other = issues with 5.15 as well (CephFS client issues). >>>=20 >>> Mark Schouten >>>=20 >>>> Op 3 nov. 2022 om 17:55 heeft Eneko Lacunza via = pve-user = het volgende geschreven: >>>>=20 >>>> =EF=BB=BF >>>>=20 >>>>> _______________________________________________ >>>>> pve-user mailing list >>>>> pve-user@lists.proxmox.com >>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user = >>=20 >> Eneko Lacunza >> Zuzendari teknikoa | Director t=C3=A9cnico >> Binovo IT Human Project >>=20 >> Tel. +34 943 569 206 |https://www.binovo.es = >> Astigarragako Bidea, 2 - 2=C2=BA izda. Oficina 10-11, 20180 = Oiartzun >>=20 >> https://www.youtube.com/user/CANALBINOVO = >> https://www.linkedin.com/company/37269706/ = >>=20 >=20 > Eneko Lacunza > Zuzendari teknikoa | Director t=C3=A9cnico > Binovo IT Human Project >=20 > Tel. +34 943 569 206 |https://www.binovo.es > Astigarragako Bidea, 2 - 2=C2=BA izda. Oficina 10-11, 20180 Oiartzun >=20 > https://www.youtube.com/user/CANALBINOVO > https://www.linkedin.com/company/37269706/ >=20 >=20 > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user