From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id CED4468D72 for ; Thu, 10 Mar 2022 15:30:33 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id BC9021B6BC for ; Thu, 10 Mar 2022 15:30:03 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 349931B6B3 for ; Thu, 10 Mar 2022 15:30:02 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 1BC7346413 for ; Thu, 10 Mar 2022 15:24:39 +0100 (CET) Date: Thu, 10 Mar 2022 15:24:28 +0100 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= To: Proxmox VE user list References: <9032C6EF-53C1-4804-AF68-198C8DB2426C@telehouse.solutions> <0F233260-1E25-4379-9EE2-FF64FFC62278@telehouse.solutions> In-Reply-To: MIME-Version: 1.0 User-Agent: astroid/0.15.0 (https://github.com/astroidmail/astroid) Message-Id: <1646920824.w5mef4abey.astroid@nora.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL -0.364 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_ASCII_DIVIDERS 0.8 Spam that uses ascii formatting tricks KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [telehouse.solutions, mysignature.io, proxmox.com, mysig.io] Subject: Re: [PVE-User] Locking HA during UPS shutdown X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Mar 2022 14:30:33 -0000 On March 10, 2022 2:48 pm, admins@telehouse.solutions wrote: > That was actually a really BAD ADVICE=E2=80=A6. as when node initiate mai= ntenance mode it will try to migrate hosted vms =E2=80=A6 and eventually en= ds up in the same Lock loop.. > what you really need is to remove started vms from ha-manager, so when th= e node initiate shutdown it will do firstly do regular shutdown vm per vm. >=20 > So, do something like below as first command in your NUT command sequence= : >=20 > for a in `ha-manager status | grep started|awk '{print $2}'|sed 's/vm://g= '`; do ha-manager remove $a;done what you should do is just change the policy to freeze or fail-over=20 before triggering the shutdown. and once power comes back up and your=20 cluster has booted, switch it back to migrate. that way, the shutdown will just stop and freeze the resources, similar=20 to what happens when rebooting using the default conditional policy. note that editing datacenter.cfg (where the shutdown_policy is=20 configured) is currently not exposed in any CLI tool, but you can update=20 it using pvesh or the API. there is still one issue though - if the whole cluster is shutdown at=20 the same time, at some point during the shutdown a non-quorate partition=20 will be all that's left, and at that point certain actions won't work=20 anymore and the node probably will get fenced. fixing this effectively=20 would require some sort of conditional delay at the right point in the=20 shutdown sequence that waits for all guests on all nodes(!) to stop=20 before proceeding with stopping the PVE services and corosync (nodes=20 still might get fenced if they take too long shutting down after the=20 last guest has exited, but that shouldn't cause much issues other than=20 noise). one way to do this would be for your NUT script to set a flag=20 file in /etc/pve, and some systemd service with the right Wants/After=20 settings that blocks the shutdown if the flag file exists and any guests=20 are still running. probably requires some tinkering, but can be safely=20 tested in a virtual cluster before moving to production ;) this last problem is not related to HA though (other than HA introducing=20 another source of trouble courtesy of fencing being active) - you will=20 also potentially hit it with your approach. the 'stop all guests on=20 node' logic that PVE has on shutdown is for shutting down one node=20 without affecting quorum, it doesn't work reliably for full-cluster=20 shutdowns (you might not see problems if timing works out, but it's=20 based on chance). an alternative approach would be to request all HA resources to be stopped=20 or disabled (`ha-manager set .. --state ..`), wait for that to be done=20 cluster-wide (e.g. by polling /cluster/resources API path), and then=20 trigger the shutdown. disadvantage of that is you have to remember the=20 pre-shutdown state and restore that afterwards for each resource.. https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_node_maintenance >> On Mar 10, 2022, at 2:48 PM, admins@telehouse.solutions wrote: >>=20 >> I don=E2=80=99t remember, search into pvecm and pve[tab][tab] related co= mmands man pages=20 >>=20 >>> On Mar 10, 2022, at 2:19 PM, Stefan Radman wrote= : >>>=20 >>> Hi Sto >>>=20 >>> Thanks for the suggestions. >>>=20 >>> The second option is what I was looking for. >>>=20 >>> How do I initiate =E2=80=9Cpve node maintenance mode=E2=80=9D? >>>=20 >>> The =E2=80=9CNode Maintenance=E2=80=9D paragraph in the HA documentatio= n is quite brief and does not refer to any command or GUI component. >>>=20 >>> Thank you >>>=20 >>> Stefan >>>=20 >>>=20 >>>> On Mar 10, 2022, at 14:50, admins@telehouse.solutions wrote: >>>>=20 >>>> Hi,=20 >>>>=20 >>>> here are two ideas: shutdown sequence -and- command sequence >>>> 1: shutdown sequence you may achieve when you set NUT=E2=80=99s on eac= h node to only monitor the UPS power, then configure each node to shutdown = itself on a different ups power levels, ex: node1 on 15% battery, node2 on = 10% battery and so on >>>> 2: you can set a cmd sequence to firstly execute pve node maintenance= mode , and then execute shutdown -> this way HA will not try to migrate vm= to node in maintenance, and the chance all nodes to goes into maintenance = in exactly same second seems to be not a risk at all. >>>>=20 >>>> hope thats helpful. >>>>=20 >>>> Regards, >>>> Sto. >>>>=20 >>>>> On Mar 10, 2022, at 1:10 PM, Stefan Radman via pve-user > wrote: >>>>>=20 >>>>>=20 >>>>> From: Stefan Radman > >>>>> Subject: Locking HA during UPS shutdown >>>>> Date: March 10, 2022 at 1:10:09 PM GMT+2 >>>>> To: PVE User List > >>>>>=20 >>>>>=20 >>>>> Hi=20 >>>>>=20 >>>>> I am configuring a 3 node PVE cluster with integrated Ceph storage. >>>>>=20 >>>>> It is powered by 2 UPS that are monitored by NUT (Network UPS Tools). >>>>>=20 >>>>> HA is configured with 3 groups: >>>>> group pve1 nodes pve1:1,pve2,pve3 >>>>> group pve2 nodes pve1,pve2:1,pve3 >>>>> group pve3 nodes pve1,pve2,pve3:1 >>>>>=20 >>>>> That will normally place the VMs in each group on the corresponding n= ode, unless that node fails. >>>>>=20 >>>>> The cluster is configured to migrate VMs away from a node before shut= ting it down (Cluster=3D>Options=3D>HA Settings: shutdown_policy=3Dmigrate)= . >>>>>=20 >>>>> NUT is configured to shut down the serves once the last of the two UP= S is running low on battery. >>>>>=20 >>>>> My problem: >>>>> When NUT starts shutting down the 3 nodes, HA will first try to live-= migrate them to another node. >>>>> That live migration process gets stuck because all the nodes are shut= ting down simultaneously. >>>>> It seems that the whole process runs into a timeout, finally =E2=80= =9Cpowers off=E2=80=9D all the VMs and shuts down the nodes. >>>>>=20 >>>>> My question: >>>>> Is there a way to =E2=80=9Clock=E2=80=9D or temporarily de-activate H= A before shutting down a node to avoid that deadlock? >>>>>=20 >>>>> Thank you >>>>>=20 >>>>> Stefan >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> _______________________________________________ >>>>> pve-user mailing list >>>>> pve-user@lists.proxmox.com >>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>>>=20 >>>>=20 >>>> Best Regards, >>>>=20 >>>> Stoyan Stoyanov Sto | Solutions Manager >>>> | Telehouse.Solutions | ICT Department >>>> | phone/viber: +359 894774934 >>>> | telegram: @prostoSto >>>> | skype: prosto.sto >>>> | email: sto@telehouse.solutions >>>> | website: www.telehouse.solutions >>>> | address: Telepoint #2, Sofia, Bulgaria >>>> <356841.png> >>>>=20 >>>> >>>> Save paper. Don=E2=80=99t print >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> Best Regards, >>>>=20 >>>> Stoyan Stoyanov Sto | Solutions Manager >>>> | Telehouse.Solutions | ICT Department >>>> | phone/viber: +359 894774934 >>>> | telegram: @prostoSto >>>> | skype: prosto.sto >>>> | email: sto@telehouse.solutions >>>> | website: www.telehouse.solutions >>>> | address: Telepoint #2, Sofia, Bulgaria >>>> <356841.png> >>>>=20 >>>> >>>> Save paper. Don=E2=80=99t print >>>=20 >>=20 >>=20 >> Best Regards, >>=20 >> Stoyan Stoyanov Sto | Solutions Manager >> | Telehouse.Solutions | ICT Department >> | phone/viber: +359 894774934 >> | telegram: @prostoSto >> | skype: prosto.sto >> | email: sto@telehouse.solutions >> | website: www.telehouse.solutions >> | address: Telepoint #2, Sofia, Bulgaria >> >>=20 >> >> Save paper. Don=E2=80=99t print >>=20 >>=20 >>=20 >>=20 >> _______________________________________________ >> pve-user mailing list >> pve-user@lists.proxmox.com >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >=20 >=20 > Best Regards, >=20 > Stoyan Stoyanov Sto | Solutions Manager > | Telehouse.Solutions | ICT Department > | phone/viber: +359 894774934 > | telegram: @prostoSto > | skype: prosto.sto > | email: sto@telehouse.solutions > | website: www.telehouse.solutions > | address: Telepoint #2, Sofia, Bulgaria > >=20 > > Save paper. Don=E2=80=99t print >=20 >=20 >=20 >=20 > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >=20