[PVE-User] VM Migration Not Happening :-(

[PVE-User] VM Migration Not Happening :-(

[duluxoz]

Hi All,

I need help with figuring out why I can't migrate a VM from one Proxmox 
Node to another (in the same cluster, of course).

These are the details provided by the Proxmox Task Log:

```

task started by HA resource agent
2024-09-01 18:02:30 use dedicated network address for sending migration 
traffic (192.168.200.103)
2024-09-01 18:02:30 starting migration of VM 100 to node 'pven3' 
(192.168.200.103)
2024-09-01 18:02:30 starting VM 100 on remote node 'pven3'
2024-09-01 18:02:30 [pven3]
2024-09-01 18:02:32 start remote tunnel
2024-09-01 18:02:33 ssh tunnel ver 1
2024-09-01 18:02:33 starting online/live migration on 
unix:/run/qemu-server/100.migrate
2024-09-01 18:02:33 set migration capabilities
2024-09-01 18:02:33 migration downtime limit: 100 ms
2024-09-01 18:02:33 migration cachesize: 256.0 MiB
2024-09-01 18:02:33 set migration parameters
2024-09-01 18:02:33 start migrate command to 
unix:/run/qemu-server/100.migrate
channel 2: open failed: connect failed: open failed
2024-09-01 18:02:34 migration status error: failed - Unable to write to 
socket: Broken pipe
2024-09-01 18:02:34 ERROR: online migrate failure - aborting
2024-09-01 18:02:34 aborting phase 2 - cleanup resources
2024-09-01 18:02:34 migrate_cancel
2024-09-01 18:02:36 ERROR: migration finished with problems (duration 
00:00:07)
TASK ERROR: migration problems
```

If someone could point me in the correct direction to resolve this issue 
I'd be very grateful - thanks

Cheer

Dulux-Oz


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] VM Migration Not Happening :-(

[Lund Svein-Erik]

I Can't give you any exact cause, but the message "channel 2: open failed....." comes from an issue with ssh between the nodes. I can't really tell anything else from the log you've posted.

---
 ---------------------------
From: "duluxoz" 
To: "Proxmox VE user list" 


Sent: Sunday, September 1, 2024 10:12 AM
Subject: [PVE-User] VM Migration Not Happening :-(

Hi All,

I need help with figuring out why I can't migrate a VM from one Proxmox 
Node to another (in the same cluster, of course).

These are the details provided by the Proxmox Task Log:

```

task started by HA resource agent
2024-09-01 18:02:30 use dedicated network address for sending migration 
traffic (192.168.200.103)
2024-09-01 18:02:30 starting migration of VM 100 to node 'pven3' 
(192.168.200.103)
2024-09-01 18:02:30 starting VM 100 on remote node 'pven3'
2024-09-01 18:02:30 [pven3]
2024-09-01 18:02:32 start remote tunnel
2024-09-01 18:02:33 ssh tunnel ver 1
2024-09-01 18:02:33 starting online/live migration on 
unix:/run/qemu-server/100.migrate
2024-09-01 18:02:33 set migration capabilities
2024-09-01 18:02:33 migration downtime limit: 100 ms
2024-09-01 18:02:33 migration cachesize: 256.0 MiB
2024-09-01 18:02:33 set migration parameters
2024-09-01 18:02:33 start migrate command to 
unix:/run/qemu-server/100.migrate
channel 2: open failed: connect failed: open failed
2024-09-01 18:02:34 migration status error: failed - Unable to write to 
socket: Broken pipe
2024-09-01 18:02:34 ERROR: online migrate failure - aborting
2024-09-01 18:02:34 aborting phase 2 - cleanup resources
2024-09-01 18:02:34 migrate_cancel
2024-09-01 18:02:36 ERROR: migration finished with problems (duration 
00:00:07)
TASK ERROR: migration problems
```

If someone could point me in the correct direction to resolve this issue 
I'd be very grateful - thanks

Cheer

Dulux-Oz


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] VM Migration Not Happening :-(

[Gilberto Ferreira]

Hi
We need more details.
Send us the following:

cat /etc/pve/storage
cat /etc/pvedatacenter
cat /etc/pve/corosync.conf
cat /etc/hosts
cat /etc/network/interfaces

Can you ssh between the nodes?


---


Gilberto Nunes Ferreira
(47) 99676-7530 - Whatsapp / Telegram






Em dom., 1 de set. de 2024 às 05:11, duluxoz <duluxoz@gmail.com> escreveu:

> Hi All,
>
> I need help with figuring out why I can't migrate a VM from one Proxmox
> Node to another (in the same cluster, of course).
>
> These are the details provided by the Proxmox Task Log:
>
> ```
>
> task started by HA resource agent
> 2024-09-01 18:02:30 use dedicated network address for sending migration
> traffic (192.168.200.103)
> 2024-09-01 18:02:30 starting migration of VM 100 to node 'pven3'
> (192.168.200.103)
> 2024-09-01 18:02:30 starting VM 100 on remote node 'pven3'
> 2024-09-01 18:02:30 [pven3]
> 2024-09-01 18:02:32 start remote tunnel
> 2024-09-01 18:02:33 ssh tunnel ver 1
> 2024-09-01 18:02:33 starting online/live migration on
> unix:/run/qemu-server/100.migrate
> 2024-09-01 18:02:33 set migration capabilities
> 2024-09-01 18:02:33 migration downtime limit: 100 ms
> 2024-09-01 18:02:33 migration cachesize: 256.0 MiB
> 2024-09-01 18:02:33 set migration parameters
> 2024-09-01 18:02:33 start migrate command to
> unix:/run/qemu-server/100.migrate
> channel 2: open failed: connect failed: open failed
> 2024-09-01 18:02:34 migration status error: failed - Unable to write to
> socket: Broken pipe
> 2024-09-01 18:02:34 ERROR: online migrate failure - aborting
> 2024-09-01 18:02:34 aborting phase 2 - cleanup resources
> 2024-09-01 18:02:34 migrate_cancel
> 2024-09-01 18:02:36 ERROR: migration finished with problems (duration
> 00:00:07)
> TASK ERROR: migration problems
> ```
>
> If someone could point me in the correct direction to resolve this issue
> I'd be very grateful - thanks
>
> Cheer
>
> Dulux-Oz
>
>
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
>
_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] VM Migration Not Happening :-(

[duluxoz]

Hi Gilberto, and Thank you for getting back to me.

Just to be 100% clear: the Proxmox (with Hyper-Converged Ceph) Cluster 
is working AOK, except for the face I can't migrate *any* of the VMs 
(live or shutdown).

Yes, I can SSH into each node from every other node using:

  * the hostname of the "management" NIC
  * the hostname of the "migration traffic" NIC
  * the ip address of the "managment" NIC
  * the IP address of the "migration traffic" NIC

The VM's HDD is on the rbd storage (see below).

As requested:

/etc/pve/storage.cfg

```

dir: local
        path /var/lib/vz
        content vztmpl,iso,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

rbd: rbd
        content images,rootdir
        krbd 0
        pool rbd

cephfs: cephfs
        path /data/cephfs
        content backup,vztmpl,iso
        fs-name cephfs
```

/etc/pve/datacenter.cfg

```

console: html5
crs: ha-rebalance-on-start=1
ha: shutdown_policy=migrate
keyboard: en-us
migration: secure,network=192.168.200.0/24
next-id: lower=1000
```

/etc/pve/corosync.conf

```

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: pven1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: 192.168.100.101
    ring1_addr: 192.168.200.101
  }
  node {
    name: pven2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.100.102
    ring1_addr: 192.168.200.102
}
  node {
    name: pven3
    nodeid: 3
    quorum_votes: 1
    ring0_addr: 192.168.100.103
    ring1_addr: 192.168.200.103
}
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: cluster1
  config_version: 4
  interface {
    knet_link_priority: 10
    linknumber: 0
  }
  interface {
    knet_link_priority: 20
    linknumber: 1
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

```

/etc/hosts

```

127.0.0.1   localhost localhost.localdomain localhost4 
localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 
localhost6.localdomain6 ip6-localhost ip6-loopback
192.168.100.101  pven1.mydomain.local pven1
192.168.100.102 pven2.mydomain.local pven2
192.168.100.103 pven3.mydomain.local pven3
192.168.200.101 pvent1.mydomain.local pvent1
192.168.200.102 pvent2.mydomain.local pvent2
192.168.200.103 pvent3.mydomain.local pvent3
```

/etc/network/interfaces (for pven1; pven2 & pven3 are the same, except 
for the IP Address (see above))

```

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto enp2s0
iface enp2s0 inet static
  address 192.168.200.20/24
  mtu 9000

auto bond0
iface bond0 inet manual
  bond-slaves eno1 eno2
  bond-mode 802.3ad
  bond-xmit-hash-policy layer2+3
  bond-miimon 100
  bond-downdelay 200
  bond-updelay 200

auto vmbr0
iface vmbr0 inet static
  bridge-ports bond0
  bridge-stp off
  bridge-fd 0
  bridge-vlan-aware yes
  bridge-vids 100,200

auto vmbr0.200
iface vmbr0.200 inet static
  address 192.168.100.101/24
  gateway 192.168.100.1
```

Note: iface enp2s0 (on all 3 Nodes) sits on an isolated VLAN which 
(obviously) has no gateway ie the only hosts on that VLAN are pven1, 
pven2, & pven3, and all are "pingable" from each other.

Thanks for the help

Dulux-Oz


On 2/9/24 08:02, Gilberto Ferreira wrote:
> Hi
> We need more details.
> Send us the following:
>
> cat /etc/pve/storage
> cat /etc/pvedatacenter
> cat /etc/pve/corosync.conf
> cat /etc/hosts
> cat /etc/network/interfaces
>
> Can you ssh between the nodes?
>
>
> ---
>
>
> Gilberto Nunes Ferreira
> (47) 99676-7530 - Whatsapp / Telegram
>
>
>
>
>
>
> Em dom., 1 de set. de 2024 às 05:11, duluxoz <duluxoz@gmail.com> escreveu:
>
>     Hi All,
>
>     I need help with figuring out why I can't migrate a VM from one
>     Proxmox
>     Node to another (in the same cluster, of course).
>
>     These are the details provided by the Proxmox Task Log:
>
>     ```
>
>     task started by HA resource agent
>     2024-09-01 18:02:30 use dedicated network address for sending
>     migration
>     traffic (192.168.200.103)
>     2024-09-01 18:02:30 starting migration of VM 100 to node 'pven3'
>     (192.168.200.103)
>     2024-09-01 18:02:30 starting VM 100 on remote node 'pven3'
>     2024-09-01 18:02:30 [pven3]
>     2024-09-01 18:02:32 start remote tunnel
>     2024-09-01 18:02:33 ssh tunnel ver 1
>     2024-09-01 18:02:33 starting online/live migration on
>     unix:/run/qemu-server/100.migrate
>     2024-09-01 18:02:33 set migration capabilities
>     2024-09-01 18:02:33 migration downtime limit: 100 ms
>     2024-09-01 18:02:33 migration cachesize: 256.0 MiB
>     2024-09-01 18:02:33 set migration parameters
>     2024-09-01 18:02:33 start migrate command to
>     unix:/run/qemu-server/100.migrate
>     channel 2: open failed: connect failed: open failed
>     2024-09-01 18:02:34 migration status error: failed - Unable to
>     write to
>     socket: Broken pipe
>     2024-09-01 18:02:34 ERROR: online migrate failure - aborting
>     2024-09-01 18:02:34 aborting phase 2 - cleanup resources
>     2024-09-01 18:02:34 migrate_cancel
>     2024-09-01 18:02:36 ERROR: migration finished with problems (duration
>     00:00:07)
>     TASK ERROR: migration problems
>     ```
>
>     If someone could point me in the correct direction to resolve this
>     issue
>     I'd be very grateful - thanks
>
>     Cheer
>
>     Dulux-Oz
>
>
>     _______________________________________________
>     pve-user mailing list
>     pve-user@lists.proxmox.com
>     https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] VM Migration Not Happening :-(

[Fabian Grünbichler]

> duluxoz <duluxoz@gmail.com> hat am 02.09.2024 07:50 CEST geschrieben:
> 
>  
> Hi Gilberto, and Thank you for getting back to me.
> 
> Just to be 100% clear: the Proxmox (with Hyper-Converged Ceph) Cluster 
> is working AOK, except for the face I can't migrate *any* of the VMs 
> (live or shutdown).
> 
> Yes, I can SSH into each node from every other node using:
> 
>   * the hostname of the "management" NIC
>   * the hostname of the "migration traffic" NIC
>   * the ip address of the "managment" NIC
>   * the IP address of the "migration traffic" NIC

you probably configured SSH to restrict port/socket forwarding.

see `man sshd_config` for why this is usually not really a meaningful config change.


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] VM Migration Not Happening :-(

[duluxoz]

Yeap - that was it

Thanks for helping me out  😬

Cheers

Dulux-Oz

On 2/9/24 17:51, Fabian Grünbichler wrote:
>> duluxoz <duluxoz@gmail.com> hat am 02.09.2024 07:50 CEST geschrieben:
>>
>>   
>> Hi Gilberto, and Thank you for getting back to me.
>>
>> Just to be 100% clear: the Proxmox (with Hyper-Converged Ceph) Cluster
>> is working AOK, except for the face I can't migrate *any* of the VMs
>> (live or shutdown).
>>
>> Yes, I can SSH into each node from every other node using:
>>
>>    * the hostname of the "management" NIC
>>    * the hostname of the "migration traffic" NIC
>>    * the ip address of the "managment" NIC
>>    * the IP address of the "migration traffic" NIC
> you probably configured SSH to restrict port/socket forwarding.
>
> see `man sshd_config` for why this is usually not really a meaningful config change.
>


_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user