* [PVE-User] VM Migration Not Happening :-( @ 2024-09-01 8:10 duluxoz 2024-09-01 17:14 ` Lund Svein-Erik 2024-09-01 22:02 ` Gilberto Ferreira 0 siblings, 2 replies; 6+ messages in thread From: duluxoz @ 2024-09-01 8:10 UTC (permalink / raw) To: Proxmox VE user list Hi All, I need help with figuring out why I can't migrate a VM from one Proxmox Node to another (in the same cluster, of course). These are the details provided by the Proxmox Task Log: ``` task started by HA resource agent 2024-09-01 18:02:30 use dedicated network address for sending migration traffic (192.168.200.103) 2024-09-01 18:02:30 starting migration of VM 100 to node 'pven3' (192.168.200.103) 2024-09-01 18:02:30 starting VM 100 on remote node 'pven3' 2024-09-01 18:02:30 [pven3] 2024-09-01 18:02:32 start remote tunnel 2024-09-01 18:02:33 ssh tunnel ver 1 2024-09-01 18:02:33 starting online/live migration on unix:/run/qemu-server/100.migrate 2024-09-01 18:02:33 set migration capabilities 2024-09-01 18:02:33 migration downtime limit: 100 ms 2024-09-01 18:02:33 migration cachesize: 256.0 MiB 2024-09-01 18:02:33 set migration parameters 2024-09-01 18:02:33 start migrate command to unix:/run/qemu-server/100.migrate channel 2: open failed: connect failed: open failed 2024-09-01 18:02:34 migration status error: failed - Unable to write to socket: Broken pipe 2024-09-01 18:02:34 ERROR: online migrate failure - aborting 2024-09-01 18:02:34 aborting phase 2 - cleanup resources 2024-09-01 18:02:34 migrate_cancel 2024-09-01 18:02:36 ERROR: migration finished with problems (duration 00:00:07) TASK ERROR: migration problems ``` If someone could point me in the correct direction to resolve this issue I'd be very grateful - thanks Cheer Dulux-Oz _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PVE-User] VM Migration Not Happening :-( 2024-09-01 8:10 [PVE-User] VM Migration Not Happening :-( duluxoz @ 2024-09-01 17:14 ` Lund Svein-Erik 2024-09-01 22:02 ` Gilberto Ferreira 1 sibling, 0 replies; 6+ messages in thread From: Lund Svein-Erik @ 2024-09-01 17:14 UTC (permalink / raw) To: matthew, Proxmox VE user list I Can't give you any exact cause, but the message "channel 2: open failed....." comes from an issue with ssh between the nodes. I can't really tell anything else from the log you've posted. --- --------------------------- From: "duluxoz" To: "Proxmox VE user list" Sent: Sunday, September 1, 2024 10:12 AM Subject: [PVE-User] VM Migration Not Happening :-( Hi All, I need help with figuring out why I can't migrate a VM from one Proxmox Node to another (in the same cluster, of course). These are the details provided by the Proxmox Task Log: ``` task started by HA resource agent 2024-09-01 18:02:30 use dedicated network address for sending migration traffic (192.168.200.103) 2024-09-01 18:02:30 starting migration of VM 100 to node 'pven3' (192.168.200.103) 2024-09-01 18:02:30 starting VM 100 on remote node 'pven3' 2024-09-01 18:02:30 [pven3] 2024-09-01 18:02:32 start remote tunnel 2024-09-01 18:02:33 ssh tunnel ver 1 2024-09-01 18:02:33 starting online/live migration on unix:/run/qemu-server/100.migrate 2024-09-01 18:02:33 set migration capabilities 2024-09-01 18:02:33 migration downtime limit: 100 ms 2024-09-01 18:02:33 migration cachesize: 256.0 MiB 2024-09-01 18:02:33 set migration parameters 2024-09-01 18:02:33 start migrate command to unix:/run/qemu-server/100.migrate channel 2: open failed: connect failed: open failed 2024-09-01 18:02:34 migration status error: failed - Unable to write to socket: Broken pipe 2024-09-01 18:02:34 ERROR: online migrate failure - aborting 2024-09-01 18:02:34 aborting phase 2 - cleanup resources 2024-09-01 18:02:34 migrate_cancel 2024-09-01 18:02:36 ERROR: migration finished with problems (duration 00:00:07) TASK ERROR: migration problems ``` If someone could point me in the correct direction to resolve this issue I'd be very grateful - thanks Cheer Dulux-Oz _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PVE-User] VM Migration Not Happening :-( 2024-09-01 8:10 [PVE-User] VM Migration Not Happening :-( duluxoz 2024-09-01 17:14 ` Lund Svein-Erik @ 2024-09-01 22:02 ` Gilberto Ferreira 2024-09-02 5:50 ` duluxoz 1 sibling, 1 reply; 6+ messages in thread From: Gilberto Ferreira @ 2024-09-01 22:02 UTC (permalink / raw) To: matthew, Proxmox VE user list Hi We need more details. Send us the following: cat /etc/pve/storage cat /etc/pvedatacenter cat /etc/pve/corosync.conf cat /etc/hosts cat /etc/network/interfaces Can you ssh between the nodes? --- Gilberto Nunes Ferreira (47) 99676-7530 - Whatsapp / Telegram Em dom., 1 de set. de 2024 às 05:11, duluxoz <duluxoz@gmail.com> escreveu: > Hi All, > > I need help with figuring out why I can't migrate a VM from one Proxmox > Node to another (in the same cluster, of course). > > These are the details provided by the Proxmox Task Log: > > ``` > > task started by HA resource agent > 2024-09-01 18:02:30 use dedicated network address for sending migration > traffic (192.168.200.103) > 2024-09-01 18:02:30 starting migration of VM 100 to node 'pven3' > (192.168.200.103) > 2024-09-01 18:02:30 starting VM 100 on remote node 'pven3' > 2024-09-01 18:02:30 [pven3] > 2024-09-01 18:02:32 start remote tunnel > 2024-09-01 18:02:33 ssh tunnel ver 1 > 2024-09-01 18:02:33 starting online/live migration on > unix:/run/qemu-server/100.migrate > 2024-09-01 18:02:33 set migration capabilities > 2024-09-01 18:02:33 migration downtime limit: 100 ms > 2024-09-01 18:02:33 migration cachesize: 256.0 MiB > 2024-09-01 18:02:33 set migration parameters > 2024-09-01 18:02:33 start migrate command to > unix:/run/qemu-server/100.migrate > channel 2: open failed: connect failed: open failed > 2024-09-01 18:02:34 migration status error: failed - Unable to write to > socket: Broken pipe > 2024-09-01 18:02:34 ERROR: online migrate failure - aborting > 2024-09-01 18:02:34 aborting phase 2 - cleanup resources > 2024-09-01 18:02:34 migrate_cancel > 2024-09-01 18:02:36 ERROR: migration finished with problems (duration > 00:00:07) > TASK ERROR: migration problems > ``` > > If someone could point me in the correct direction to resolve this issue > I'd be very grateful - thanks > > Cheer > > Dulux-Oz > > > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PVE-User] VM Migration Not Happening :-( 2024-09-01 22:02 ` Gilberto Ferreira @ 2024-09-02 5:50 ` duluxoz 2024-09-02 7:51 ` Fabian Grünbichler 0 siblings, 1 reply; 6+ messages in thread From: duluxoz @ 2024-09-02 5:50 UTC (permalink / raw) To: Gilberto Ferreira, Proxmox VE user list Hi Gilberto, and Thank you for getting back to me. Just to be 100% clear: the Proxmox (with Hyper-Converged Ceph) Cluster is working AOK, except for the face I can't migrate *any* of the VMs (live or shutdown). Yes, I can SSH into each node from every other node using: * the hostname of the "management" NIC * the hostname of the "migration traffic" NIC * the ip address of the "managment" NIC * the IP address of the "migration traffic" NIC The VM's HDD is on the rbd storage (see below). As requested: /etc/pve/storage.cfg ``` dir: local path /var/lib/vz content vztmpl,iso,backup lvmthin: local-lvm thinpool data vgname pve content images,rootdir rbd: rbd content images,rootdir krbd 0 pool rbd cephfs: cephfs path /data/cephfs content backup,vztmpl,iso fs-name cephfs ``` /etc/pve/datacenter.cfg ``` console: html5 crs: ha-rebalance-on-start=1 ha: shutdown_policy=migrate keyboard: en-us migration: secure,network=192.168.200.0/24 next-id: lower=1000 ``` /etc/pve/corosync.conf ``` logging { debug: off to_syslog: yes } nodelist { node { name: pven1 nodeid: 1 quorum_votes: 1 ring0_addr: 192.168.100.101 ring1_addr: 192.168.200.101 } node { name: pven2 nodeid: 2 quorum_votes: 1 ring0_addr: 192.168.100.102 ring1_addr: 192.168.200.102 } node { name: pven3 nodeid: 3 quorum_votes: 1 ring0_addr: 192.168.100.103 ring1_addr: 192.168.200.103 } } quorum { provider: corosync_votequorum } totem { cluster_name: cluster1 config_version: 4 interface { knet_link_priority: 10 linknumber: 0 } interface { knet_link_priority: 20 linknumber: 1 } ip_version: ipv4-6 link_mode: passive secauth: on version: 2 } ``` /etc/hosts ``` 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 ip6-localhost ip6-loopback 192.168.100.101 pven1.mydomain.local pven1 192.168.100.102 pven2.mydomain.local pven2 192.168.100.103 pven3.mydomain.local pven3 192.168.200.101 pvent1.mydomain.local pvent1 192.168.200.102 pvent2.mydomain.local pvent2 192.168.200.103 pvent3.mydomain.local pvent3 ``` /etc/network/interfaces (for pven1; pven2 & pven3 are the same, except for the IP Address (see above)) ``` auto lo iface lo inet loopback iface eno1 inet manual iface eno2 inet manual auto enp2s0 iface enp2s0 inet static address 192.168.200.20/24 mtu 9000 auto bond0 iface bond0 inet manual bond-slaves eno1 eno2 bond-mode 802.3ad bond-xmit-hash-policy layer2+3 bond-miimon 100 bond-downdelay 200 bond-updelay 200 auto vmbr0 iface vmbr0 inet static bridge-ports bond0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 100,200 auto vmbr0.200 iface vmbr0.200 inet static address 192.168.100.101/24 gateway 192.168.100.1 ``` Note: iface enp2s0 (on all 3 Nodes) sits on an isolated VLAN which (obviously) has no gateway ie the only hosts on that VLAN are pven1, pven2, & pven3, and all are "pingable" from each other. Thanks for the help Dulux-Oz On 2/9/24 08:02, Gilberto Ferreira wrote: > Hi > We need more details. > Send us the following: > > cat /etc/pve/storage > cat /etc/pvedatacenter > cat /etc/pve/corosync.conf > cat /etc/hosts > cat /etc/network/interfaces > > Can you ssh between the nodes? > > > --- > > > Gilberto Nunes Ferreira > (47) 99676-7530 - Whatsapp / Telegram > > > > > > > Em dom., 1 de set. de 2024 às 05:11, duluxoz <duluxoz@gmail.com> escreveu: > > Hi All, > > I need help with figuring out why I can't migrate a VM from one > Proxmox > Node to another (in the same cluster, of course). > > These are the details provided by the Proxmox Task Log: > > ``` > > task started by HA resource agent > 2024-09-01 18:02:30 use dedicated network address for sending > migration > traffic (192.168.200.103) > 2024-09-01 18:02:30 starting migration of VM 100 to node 'pven3' > (192.168.200.103) > 2024-09-01 18:02:30 starting VM 100 on remote node 'pven3' > 2024-09-01 18:02:30 [pven3] > 2024-09-01 18:02:32 start remote tunnel > 2024-09-01 18:02:33 ssh tunnel ver 1 > 2024-09-01 18:02:33 starting online/live migration on > unix:/run/qemu-server/100.migrate > 2024-09-01 18:02:33 set migration capabilities > 2024-09-01 18:02:33 migration downtime limit: 100 ms > 2024-09-01 18:02:33 migration cachesize: 256.0 MiB > 2024-09-01 18:02:33 set migration parameters > 2024-09-01 18:02:33 start migrate command to > unix:/run/qemu-server/100.migrate > channel 2: open failed: connect failed: open failed > 2024-09-01 18:02:34 migration status error: failed - Unable to > write to > socket: Broken pipe > 2024-09-01 18:02:34 ERROR: online migrate failure - aborting > 2024-09-01 18:02:34 aborting phase 2 - cleanup resources > 2024-09-01 18:02:34 migrate_cancel > 2024-09-01 18:02:36 ERROR: migration finished with problems (duration > 00:00:07) > TASK ERROR: migration problems > ``` > > If someone could point me in the correct direction to resolve this > issue > I'd be very grateful - thanks > > Cheer > > Dulux-Oz > > > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PVE-User] VM Migration Not Happening :-( 2024-09-02 5:50 ` duluxoz @ 2024-09-02 7:51 ` Fabian Grünbichler 2024-09-02 9:54 ` duluxoz 0 siblings, 1 reply; 6+ messages in thread From: Fabian Grünbichler @ 2024-09-02 7:51 UTC (permalink / raw) To: matthew, Proxmox VE user list, duluxoz, Gilberto Ferreira > duluxoz <duluxoz@gmail.com> hat am 02.09.2024 07:50 CEST geschrieben: > > > Hi Gilberto, and Thank you for getting back to me. > > Just to be 100% clear: the Proxmox (with Hyper-Converged Ceph) Cluster > is working AOK, except for the face I can't migrate *any* of the VMs > (live or shutdown). > > Yes, I can SSH into each node from every other node using: > > * the hostname of the "management" NIC > * the hostname of the "migration traffic" NIC > * the ip address of the "managment" NIC > * the IP address of the "migration traffic" NIC you probably configured SSH to restrict port/socket forwarding. see `man sshd_config` for why this is usually not really a meaningful config change. _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PVE-User] VM Migration Not Happening :-( 2024-09-02 7:51 ` Fabian Grünbichler @ 2024-09-02 9:54 ` duluxoz 0 siblings, 0 replies; 6+ messages in thread From: duluxoz @ 2024-09-02 9:54 UTC (permalink / raw) To: Fabian Grünbichler, matthew, Proxmox VE user list, Gilberto Ferreira Yeap - that was it Thanks for helping me out 😬 Cheers Dulux-Oz On 2/9/24 17:51, Fabian Grünbichler wrote: >> duluxoz <duluxoz@gmail.com> hat am 02.09.2024 07:50 CEST geschrieben: >> >> >> Hi Gilberto, and Thank you for getting back to me. >> >> Just to be 100% clear: the Proxmox (with Hyper-Converged Ceph) Cluster >> is working AOK, except for the face I can't migrate *any* of the VMs >> (live or shutdown). >> >> Yes, I can SSH into each node from every other node using: >> >> * the hostname of the "management" NIC >> * the hostname of the "migration traffic" NIC >> * the ip address of the "managment" NIC >> * the IP address of the "migration traffic" NIC > you probably configured SSH to restrict port/socket forwarding. > > see `man sshd_config` for why this is usually not really a meaningful config change. > _______________________________________________ pve-user mailing list pve-user@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-09-02 9:54 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-09-01 8:10 [PVE-User] VM Migration Not Happening :-( duluxoz 2024-09-01 17:14 ` Lund Svein-Erik 2024-09-01 22:02 ` Gilberto Ferreira 2024-09-02 5:50 ` duluxoz 2024-09-02 7:51 ` Fabian Grünbichler 2024-09-02 9:54 ` duluxoz
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox