Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown

[Alexandre DERUMIER <aderumier@odiso.com>]

>>Indeed, this should not happen. Do you use a spearate network for corosync? 

No, I use 2x40GB lacp link. 

>>was there high traffic on the network? 

but I'm far from saturated them. (in pps or througput),  (I'm around 3-4gbps)


The cluster is 14 nodes, with around 1000vms (with ha enabled on all vms)


From my understanding, watchdog-mux was still runing as the watchdog have reset only after 1min and not 10s,
 so it's like the lrm was blocked and not sending watchdog timer reset to watchdog-mux.


I'll do tests with softdog + soft_noboot=1, so if that happen again,I'll able to debug.



>>What kind of maintenance was the reason for the shutdown?

ram upgrade. (the server was running ok before shutdown, no hardware problem)  
(I just shutdown the server, and don't have started it yet when problem occur)



>>Do you use the default corosync timeout values, or do you have a special setup?


no special tuning, default values. (I don't have any retransmit since months in the logs)

>>Can you please post the full corosync config?

(I have verified, the running version was corosync was 3.0.3 with libknet 1.15)


here the config:

"
logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: m6kvm1
    nodeid: 1
    quorum_votes: 1
    ring0_addr: m6kvm1
  }
  node {
    name: m6kvm10
    nodeid: 10
    quorum_votes: 1
    ring0_addr: m6kvm10
  }
  node {
    name: m6kvm11
    nodeid: 11
    quorum_votes: 1
    ring0_addr: m6kvm11
  }
  node {
    name: m6kvm12
    nodeid: 12
    quorum_votes: 1
    ring0_addr: m6kvm12
  }
  node {
    name: m6kvm13
    nodeid: 13
    quorum_votes: 1
    ring0_addr: m6kvm13
  }
  node {
    name: m6kvm14
    nodeid: 14
    quorum_votes: 1
    ring0_addr: m6kvm14
  }
  node {
    name: m6kvm2
    nodeid: 2
    quorum_votes: 1
    ring0_addr: m6kvm2
  }
  node {
    name: m6kvm3
    nodeid: 3
    quorum_votes: 1
    ring0_addr: m6kvm3
  }
  node {
    name: m6kvm4
    nodeid: 4
    quorum_votes: 1
    ring0_addr: m6kvm4
  }
  node {
    name: m6kvm5
    nodeid: 5
    quorum_votes: 1
    ring0_addr: m6kvm5
  }
  node {
    name: m6kvm6
    nodeid: 6
    quorum_votes: 1
    ring0_addr: m6kvm6
  }
  node {
    name: m6kvm7
    nodeid: 7
    quorum_votes: 1
    ring0_addr: m6kvm7
  }

  node {
    name: m6kvm8
    nodeid: 8
    quorum_votes: 1
    ring0_addr: m6kvm8
  }
  node {
    name: m6kvm9
    nodeid: 9
    quorum_votes: 1
    ring0_addr: m6kvm9
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: m6kvm
  config_version: 19
  interface {
    bindnetaddr: 10.3.94.89
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  transport: knet
  version: 2
}



----- Mail original -----
De: "dietmar" <dietmar@proxmox.com>
À: "aderumier" <aderumier@odiso.com>, "Proxmox VE development discussion" <pve-devel@lists.proxmox.com>
Cc: "pve-devel" <pve-devel@pve.proxmox.com>
Envoyé: Dimanche 6 Septembre 2020 14:14:06
Objet: Re: [pve-devel] corosync bug: cluster break after 1 node clean shutdown

> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds) 
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds) 

Indeed, this should not happen. Do you use a spearate network for corosync? Or 
was there high traffic on the network? What kind of maintenance was the reason 
for the shutdown?