From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 80F8D7147B for ; Sat, 18 Jun 2022 13:38:05 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 7498D1D65E for ; Sat, 18 Jun 2022 13:38:05 +0200 (CEST) Received: from amorong.geoffray-levasseur.org (unknown [91.224.149.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id DC2791D64E for ; Sat, 18 Jun 2022 13:38:03 +0200 (CEST) Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id 6F91D67CD9 for ; Sat, 18 Jun 2022 13:29:13 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=geoffray-levasseur.org; s=dkim; t=1655551754; h=from:subject:date:message-id:to:mime-version:content-type; bh=RWJlOoELFO/EFIJXuurUGcAiqYZSyzxgwxkOBAjbjBQ=; b=ntJx4S6VI0tSjsjh5LBz25/cx/zZ29mR+TJnmAcyh5AidPMkia8hjkhbulL9jaF7f1qu4L S9nSFFdmDU6bALU+hrU99nSVn4/FeklSDxCx0ObapHHilJ97FCzluLknzBCNPJ0K88B6x0 5z648tcPYWihU22cuqtd10tqOcwrZds2yqX6RYcBRFehC4u5pk/1EAfyaGe0aI6WbgL+MY yFuTIdB7kC+RyeXU/YX+4vtI8KQSB+pO7bNcoKnyn7dXfX4+9dB7nWZUQ2zkZ+mJcRyxAq U0IDvWbTHvKm65h3OLWMsaDNmDGQFTYPW5aJSjHF+DKsz/9/xHYZOgTJ8/7DTw== From: Geoffray Levasseur To: pve-user@lists.proxmox.com Date: Sat, 18 Jun 2022 13:29:05 +0200 Message-ID: <3555362.dWV9SEqChM@isarog> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart5502887.rdbgypaU67"; micalg="pgp-sha256"; protocol="application/pgp-signature" X-Last-TLS-Session-Version: TLSv1.3 X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_INVALID 0.1 DKIM or DK signature exists, but is not valid DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RDNS_NONE 0.793 Delivered to internal network by a host with no rDNS SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [PVE-User] Live migration fail from old nodes to freshly installed one X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jun 2022 11:38:05 -0000 --nextPart5502887.rdbgypaU67 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="iso-8859-1"; protected-headers="v1" From: Geoffray Levasseur To: pve-user@lists.proxmox.com Subject: Live migration fail from old nodes to freshly installed one Date: Sat, 18 Jun 2022 13:29:05 +0200 Message-ID: <3555362.dWV9SEqChM@isarog> Hi everyone, =46or a couple of foundation, I run and manage 4 Proxmox VE nodes with Ceph= as=20 main storage for VMs. After changing the hardware of one of the nodes, I ha= d=20 to reinstall it since the new motherboard don't support classic boot and ha= d=20 to change for UEFI. I was able to do that as cleanly as possible: * Successful restoration of SSH keys in /root and node configuration in /va= r/ lib/pve-cluster * Successful cleanup of former system keys of the reinstalled node in older= =20 nodes configuration * Successful restoration of Ceph (yet not simple at all, the documentation= =20 IMHO lacks a clean procedure, unless I couldn't find it) =3D> As far as I know, everything works as expected, except the online migr= ation=20 (with or without HA) to the reinstalled machine. Online migration from the= =20 reinstalled to older nodes works. =2D------ The fail happens with the following output: task started by HA resource agent 2022-06-18 13:13:20 starting migration of VM 119 to node 'taal'=20 (192.168.1.251) 2022-06-18 13:13:20 starting VM 119 on remote node 'taal' 2022-06-18 13:13:21 start remote tunnel 2022-06-18 13:13:22 ssh tunnel ver 1 2022-06-18 13:13:22 starting online/live migration on unix:/run/qemu-server/ 119.migrate 2022-06-18 13:13:22 set migration capabilities 2022-06-18 13:13:22 migration downtime limit: 100 ms 2022-06-18 13:13:22 migration cachesize: 64.0 MiB 2022-06-18 13:13:22 set migration parameters 2022-06-18 13:13:22 start migrate command to unix:/run/qemu-server/119.migr= ate channel 2: open failed: connect failed: open failed 2022-06-18 13:13:23 migration status error: failed 2022-06-18 13:13:23 ERROR: online migrate failure - aborting 2022-06-18 13:13:23 aborting phase 2 - cleanup resources 2022-06-18 13:13:23 migrate_cancel 2022-06-18 13:13:25 ERROR: migration finished with problems (duration=20 00:00:06) TASK ERROR: migration problems =2D----- =2D----- On syslog server I see this: Jun 18 13:13:20 taal qm[4036852]: starting task UPID:taal: 003D98F5:00FA02E3:62ADB350:qmstart:119:root@pam : Jun 18 13:13:20 taal qm[4036853]: start VM 119: UPID:taal: 003D98F5:00FA02E3:62ADB350:qmstart:119:root@pam: Jun 18 13:13:20 pinatubo pmxcfs[1551671]: [status] notice: received log Jun 18 13:13:20 mayon pmxcfs[1039713]: [status] notice: received log Jun 18 13:13:20 ragang pmxcfs[1659329]: [status] notice: received log Jun 18 13:13:20 taal systemd[1]: Started 119.scope. Jun 18 13:13:21 taal systemd-udevd[4036885]: Using default interface naming= =20 scheme 'v247'. Jun 18 13:13:21 taal systemd-udevd[4036885]: ethtool: autonegotiation is un= set=20 or enabled, the speed and duplex are=20 not writable. Jun 18 13:13:21 taal kernel: [163848.311597] device tap119i0 entered=20 promiscuous mode Jun 18 13:13:21 taal kernel: [163848.318750] vmbr0: port 6(tap119i0) entere= d=20 blocking state Jun 18 13:13:21 taal kernel: [163848.319250] vmbr0: port 6(tap119i0) entere= d=20 disabled state Jun 18 13:13:21 taal kernel: [163848.319797] vmbr0: port 6(tap119i0) entere= d=20 blocking state Jun 18 13:13:21 taal kernel: [163848.320320] vmbr0: port 6(tap119i0) entere= d=20 forwarding state Jun 18 13:13:21 taal systemd-udevd[4036884]: Using default interface naming= =20 scheme 'v247'. Jun 18 13:13:21 taal systemd-udevd[4036884]: ethtool: autonegotiation is un= set=20 or enabled, the speed and duplex are=20 not writable. Jun 18 13:13:21 taal kernel: [163848.711643] device tap119i1 entered=20 promiscuous mode Jun 18 13:13:21 taal kernel: [163848.718476] vmbr0: port 7(tap119i1) entere= d=20 blocking state Jun 18 13:13:21 taal kernel: [163848.718951] vmbr0: port 7(tap119i1) entere= d=20 disabled state Jun 18 13:13:21 taal kernel: [163848.719477] vmbr0: port 7(tap119i1) entere= d=20 blocking state Jun 18 13:13:21 taal kernel: [163848.719982] vmbr0: port 7(tap119i1) entere= d=20 forwarding state Jun 18 13:13:21 taal qm[4036852]: end task UPID:taal: 003D98F5:00FA02E3:62ADB350:qmstart:119:root@pam: OK Jun 18 13:13:21 ragang pmxcfs[1659329]: [status] notice: received log Jun 18 13:13:21 pinatubo pmxcfs[1551671]: [status] notice: received log Jun 18 13:13:21 mayon pmxcfs[1039713]: [status] notice: received log Jun 18 13:13:21 taal systemd[1]: session-343.scope: Succeeded. Jun 18 13:13:22 taal systemd[1]: Started Session 344 of user root. Jun 18 13:13:22 ragang QEMU[4364]: kvm: Unable to write to socket: Broken p= ipe Jun 18 13:13:23 taal systemd[1]: Started Session 345 of user root. Jun 18 13:13:24 taal qm[4036985]: starting task UPID:taal:003D99= 7A: 00FA042C:62ADB354:qmstop:119:root@pam: Jun 18 13:13:24 taal qm[4036986]: stop VM 119: UPID:taal:003D997A:00FA042C: 62ADB354:qmstop:119:root@pam: Jun 18 13:13:24 ragang pmxcfs[1659329]: [status] notice: received log Jun 18 13:13:24 pinatubo pmxcfs[1551671]: [status] notice: received log Jun 18 13:13:24 mayon pmxcfs[1039713]: [status] notice: received log Jun 18 13:13:24 taal QEMU[4036862]: kvm: terminating on signal 15 from pid= =20 4036986 (task UPID:taal:003D997A:00FA042C :62ADB354:qmstop:119:root@pam:) Jun 18 13:13:24 taal qm[4036985]: end task UPID:taal:003D997A: 00FA042C:62ADB354:qmstop:119:root@pam: OK Jun 18 13:13:24 pinatubo pmxcfs[1551671]: [status] notice: received log Jun 18 13:13:24 mayon pmxcfs[1039713]: [status] notice: received log Jun 18 13:13:24 ragang pmxcfs[1659329]: [status] notice: received log Jun 18 13:13:24 taal systemd[1]: session-345.scope: Succeeded. Jun 18 13:13:24 taal systemd[1]: session-344.scope: Succeeded. Jun 18 13:13:24 taal kernel: [163851.231170] vmbr0: port 6(tap119i0) entere= d=20 disabled state Jun 18 13:13:24 taal kernel: [163851.459489] vmbr0: port 7(tap119i1) entere= d=20 disabled state Jun 18 13:13:24 taal qmeventd[1993]: read: Connection reset by peer Jun 18 13:13:24 taal systemd[1]: 119.scope: Succeeded. Jun 18 13:13:24 taal systemd[1]: 119.scope: Consumed 1.099s CPU time. Jun 18 13:13:25 ragang pve-ha-lrm[2866146]: Task 'UPID:ragang: 002BBBE4:2519FE90:62ADB34F:qmigrate:119:root@pam:' sti ll active, waiting Jun 18 13:13:25 taal systemd[1]: Started Session 346 of user root. Jun 18 13:13:25 taal systemd[1]: session-346.scope: Succeeded. Jun 18 13:13:25 ragang pve-ha-lrm[2866148]: migration problems Jun 18 13:13:25 ragang pve-ha-lrm[2866146]: end task UPID:ragang: 002BBBE4:2519FE90:62ADB34F:qmigrate:119:root@pam: migration problems Jun 18 13:13:25 ragang pve-ha-lrm[2866146]: service vm:119 not moved=20 (migration error) =2D----- mayon, pinatubo and ragang are the old nodes that didn't change. The=20 reinstalled node is named taal. On those logs, ragang is the origin node=20 initiating the migration. I suspect lrm to be in trouble since on all the node status is shown active= ,=20 except on taal where it's idle. Restarting pve-ha-lrm service is not fixing= =20 the issue. Thank you for any your help. =2D-=20 Geoffray Levasseur Ing=E9nieur syst=E8me E-3S http://www.geoffray-levasseur.org GNU/PG public key : 2B3E 4116 769C 609F 0D17 07FD 5BA9 4CC9 E9D5 AC1B Tu patere legem quam ipse fecisti. --nextPart5502887.rdbgypaU67 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part. Content-Transfer-Encoding: 7Bit -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEEKz5BFnacYJ8NFwf9W6lMyenVrBsFAmKttwEACgkQW6lMyenV rBtpsgf7BmAkJtBsv/s0F323v8jhlkKULK/tpcBhzVaCqyBX/fzs4mGDl/v0c+wV MKrGdhRWm8E/vgbjl33X6WhxLLMiJXV/HjZp/HV78puycbQzph4TJxQaSNm5b36+ tTltU/1dKsonUAqO0o1HfZKPmPFX8eu44DpBmt03bLgzxx130yOiqGCM2gYnA+CG YglGx2P63UJA+qX6TQMvyqcaqUbqqgIa75udXOwHUPt/RC0n4uNqb6b8IiK7ksh3 9NRSxbuhigGBxQfoReYpVu+I0/DLxE9ZzyJMyeVIGUUMG/MfRTLgCrIMsl3nUZRT 2T/x4rx6Vh2gJrHZpwfWXHk9bKrKtQ== =PDRT -----END PGP SIGNATURE----- --nextPart5502887.rdbgypaU67--