Re: [PVE-User] Locking HA during UPS shutdown

[admins@telehouse.solutions]

Hi, 

here are two ideas: shutdown sequence -and- command sequence
1: shutdown sequence you may achieve when you set NUT’s on each node to only monitor the UPS power, then configure each node to shutdown itself on a different ups power levels, ex: node1 on 15% battery, node2 on 10% battery and so on
2: you can set a cmd sequence to firstly execute  pve node maintenance mode , and then execute shutdown -> this way HA will not try to migrate vm to node in maintenance, and the chance all nodes to goes into maintenance in exactly same second seems to be not a risk at all.

hope thats helpful.

Regards,
Sto.

> On Mar 10, 2022, at 1:10 PM, Stefan Radman via pve-user <pve-user@lists.proxmox.com <mailto:pve-user@lists.proxmox.com>> wrote:
> 
> 
> From: Stefan Radman <stefan.radman@me.com <mailto:stefan.radman@me.com>>
> Subject: Locking HA during UPS shutdown
> Date: March 10, 2022 at 1:10:09 PM GMT+2
> To: PVE User List <pve-user@pve.proxmox.com <mailto:pve-user@pve.proxmox.com>>
> 
> 
> Hi 
> 
> I am configuring a 3 node PVE cluster with integrated Ceph storage.
> 
> It is powered by 2 UPS that are monitored by NUT (Network UPS Tools).
> 
> HA is configured with 3 groups:
> group pve1 nodes pve1:1,pve2,pve3
> group pve2 nodes pve1,pve2:1,pve3
> group pve3 nodes pve1,pve2,pve3:1
> 
> That will normally place the VMs in each group on the corresponding node, unless that node fails.
> 
> The cluster is configured to migrate VMs away from a node before shutting it down (Cluster=>Options=>HA Settings: shutdown_policy=migrate).
> 
> NUT is configured to shut down the serves once the last of the two UPS is running low on battery.
> 
> My problem:
> When NUT starts shutting down the 3 nodes, HA will first try to live-migrate them to another node.
> That live migration process gets stuck because all the nodes are shutting down simultaneously.
> It seems that the whole process runs into a timeout, finally “powers off” all the VMs and shuts down the nodes.
> 
> My question:
> Is there a way to “lock” or temporarily de-activate HA before shutting down a node to avoid that deadlock?
> 
> Thank you
> 
> Stefan
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com <mailto:pve-user@lists.proxmox.com>
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user


Best Regards,

Stoyan Stoyanov Sto | Solutions Manager
| Telehouse.Solutions | ICT Department
| phone/viber:  +359 894774934 <tel:+359 894774934>
| telegram:  @prostoSto <https://mysignature.io/redirect/skype:prosto.sto?chat>
| skype:  prosto.sto <https://mysignature.io/redirect/skype:prosto.sto?chat>
| email:  sto@telehouse.solutions <mailto:sto@telehouse.solutions>
| website: www.telehouse.solutions <https://mysig.io/MTRmMTg>
| address: Telepoint #2, Sofia, Bulgaria
 <https://mysignature.io/editor/?utm_source=freepixel>

 <https://mysig.io/ZDNkNWY>
Save paper. Don’t print







Best Regards,

Stoyan Stoyanov Sto | Solutions Manager
| Telehouse.Solutions | ICT Department
| phone/viber:  +359 894774934 <tel:+359 894774934>
| telegram:  @prostoSto <https://mysignature.io/redirect/skype:prosto.sto?chat>
| skype:  prosto.sto <https://mysignature.io/redirect/skype:prosto.sto?chat>
| email:  sto@telehouse.solutions <mailto:sto@telehouse.solutions>
| website: www.telehouse.solutions <https://mysig.io/MTRmMTg>
| address: Telepoint #2, Sofia, Bulgaria
 <https://mysignature.io/editor/?utm_source=freepixel>

 <https://mysig.io/ZDNkNWY>
Save paper. Don’t print