From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 81A6968E47 for ; Thu, 10 Mar 2022 17:08:38 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 7225D1C671 for ; Thu, 10 Mar 2022 17:08:08 +0100 (CET) Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [IPv6:2a00:1450:4864:20::12a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 145A51C65D for ; Thu, 10 Mar 2022 17:08:05 +0100 (CET) Received: by mail-lf1-x12a.google.com with SMTP id r7so10275489lfc.4 for ; Thu, 10 Mar 2022 08:08:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=QnHef03PoYRJtFAT0M8Js1jya/+O2xgFFfy39aZ5H+E=; b=SI4YSu70/rjPmyftMfYoagdCUvRD5EMmP4YUDnx+sQR6ItZcp6cFgKybUr/6EpUHKQ Y9QUvHuuORCT252dkWJXuV6yCj4+aKbz51htyczLwnbWEdMSFAE7S6BMJfn1juzjHl/W fP8WOmayZfwYBSBnD562931lRGOwxOMYnF+EvEpNP9zld+dSNo8aGk78v97rpa+koWkF 1Un9jVhSegPLpFHiNNnp+Achc0F9ilXGPaqV7oUYOvqHEM702woOvL7YjCjrfCMjIueB rNqxfQryUW7Co7SaM+w3ynfRHGX08q0dWU0opJPAhQksUTGeWNrhFQgQ+cgKBMnC5MCB oSpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=QnHef03PoYRJtFAT0M8Js1jya/+O2xgFFfy39aZ5H+E=; b=YZE3ACLJi3g1z0kHCmpkGYZG1Yota+tY+9dpazgdxGzKGkCOEU23J2YbLqvbV0Yrkh C7ehCZyNy4LZ5D6Fl8fIcVw+WQhOv7aLMIldWcQbeC4uCdMKZSSq7i0AUuxujVq8W0Eo GFWCEHVWJ89VOowLVd++U+0Zq3a8ZaA3fdBRID1CvJ9mXyobtMQLq7/Vk+7A7g6EIdT/ lnXUQxa4NAyhxW91rQr74z2Tpe7Y1S6yqrvXTienNfRizVp4Z1PwDqengR35qe3yb40l aLwyynZ3oOHOk/7cg+DxgNfYAf/o/mdCqsKuE0CwxUUjfm20Sw6jTekrIIdmf6hCLx7/ du5Q== X-Gm-Message-State: AOAM533VUpHTmJuiAoPUPlTjXHVj8P/3b3kgnmChclLp6qsiMqHGPlfE uGZt99YOofsDiNoWYyU/nSjYJiEaZarB2rD5folKHwMAftw= X-Google-Smtp-Source: ABdhPJyNam5pnv/0PTyXVTObRJBGCo3l7GmgZ2/leq0b2aDDzIJugZ37KRiYeJANsPKLZz8uydhq3PwtmRQwdhUB9x0= X-Received: by 2002:a19:4941:0:b0:445:8fd5:5cea with SMTP id l1-20020a194941000000b004458fd55ceamr3345321lfj.179.1646928477872; Thu, 10 Mar 2022 08:07:57 -0800 (PST) MIME-Version: 1.0 References: <9032C6EF-53C1-4804-AF68-198C8DB2426C@telehouse.solutions> <0F233260-1E25-4379-9EE2-FF64FFC62278@telehouse.solutions> <1646920824.w5mef4abey.astroid@nora.none> In-Reply-To: <1646920824.w5mef4abey.astroid@nora.none> From: "M. Lyakhovsky" Date: Thu, 10 Mar 2022 11:07:46 -0500 Message-ID: To: Proxmox VE user list X-SPAM-LEVEL: Spam detection results: 0 AWL -0.125 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain FREEMAIL_ENVFROM_END_DIGIT 0.25 Envelope-from freemail username ends in digit FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider HTML_MESSAGE 0.001 HTML included in message POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_4 0.1 random spam to be learned in bayes RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [mysignature.io, telehouse.solutions, mysig.io, proxmox.com] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] Locking HA during UPS shutdown X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Mar 2022 16:08:38 -0000 Hi I asked a. Question earlier and no one answered about not being able load lv2 library because I need lvremove And please can someone tell me how anable WiFi by putting a sting in. /etc/network/devices On Thu, Mar 10, 2022 at 9:30 AM Fabian Gr=C3=BCnbichler < f.gruenbichler@proxmox.com> wrote: > On March 10, 2022 2:48 pm, admins@telehouse.solutions wrote: > > That was actually a really BAD ADVICE=E2=80=A6. as when node initiate > maintenance mode it will try to migrate hosted vms =E2=80=A6 and eventual= ly ends up > in the same Lock loop.. > > what you really need is to remove started vms from ha-manager, so when > the node initiate shutdown it will do firstly do regular shutdown vm per = vm. > > > > So, do something like below as first command in your NUT command > sequence: > > > > for a in `ha-manager status | grep started|awk '{print $2}'|sed > 's/vm://g'`; do ha-manager remove $a;done > > what you should do is just change the policy to freeze or fail-over > before triggering the shutdown. and once power comes back up and your > cluster has booted, switch it back to migrate. > > that way, the shutdown will just stop and freeze the resources, similar > to what happens when rebooting using the default conditional policy. > > note that editing datacenter.cfg (where the shutdown_policy is > configured) is currently not exposed in any CLI tool, but you can update > it using pvesh or the API. > > there is still one issue though - if the whole cluster is shutdown at > the same time, at some point during the shutdown a non-quorate partition > will be all that's left, and at that point certain actions won't work > anymore and the node probably will get fenced. fixing this effectively > would require some sort of conditional delay at the right point in the > shutdown sequence that waits for all guests on all nodes(!) to stop > before proceeding with stopping the PVE services and corosync (nodes > still might get fenced if they take too long shutting down after the > last guest has exited, but that shouldn't cause much issues other than > noise). one way to do this would be for your NUT script to set a flag > file in /etc/pve, and some systemd service with the right Wants/After > settings that blocks the shutdown if the flag file exists and any guests > are still running. probably requires some tinkering, but can be safely > tested in a virtual cluster before moving to production ;) > > this last problem is not related to HA though (other than HA introducing > another source of trouble courtesy of fencing being active) - you will > also potentially hit it with your approach. the 'stop all guests on > node' logic that PVE has on shutdown is for shutting down one node > without affecting quorum, it doesn't work reliably for full-cluster > shutdowns (you might not see problems if timing works out, but it's > based on chance). > > an alternative approach would be to request all HA resources to be stoppe= d > or disabled (`ha-manager set .. --state ..`), wait for that to be done > cluster-wide (e.g. by polling /cluster/resources API path), and then > trigger the shutdown. disadvantage of that is you have to remember the > pre-shutdown state and restore that afterwards for each resource.. > > https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_node_maintenance > > >> On Mar 10, 2022, at 2:48 PM, admins@telehouse.solutions wrote: > >> > >> I don=E2=80=99t remember, search into pvecm and pve[tab][tab] related = commands > man pages > >> > >>> On Mar 10, 2022, at 2:19 PM, Stefan Radman > wrote: > >>> > >>> Hi Sto > >>> > >>> Thanks for the suggestions. > >>> > >>> The second option is what I was looking for. > >>> > >>> How do I initiate =E2=80=9Cpve node maintenance mode=E2=80=9D? > >>> > >>> The =E2=80=9CNode Maintenance=E2=80=9D paragraph in the HA documentat= ion is quite > brief and does not refer to any command or GUI component. > >>> > >>> Thank you > >>> > >>> Stefan > >>> > >>> > >>>> On Mar 10, 2022, at 14:50, admins@telehouse.solutions admins@telehouse.solutions> wrote: > >>>> > >>>> Hi, > >>>> > >>>> here are two ideas: shutdown sequence -and- command sequence > >>>> 1: shutdown sequence you may achieve when you set NUT=E2=80=99s on e= ach node > to only monitor the UPS power, then configure each node to shutdown itsel= f > on a different ups power levels, ex: node1 on 15% battery, node2 on 10% > battery and so on > >>>> 2: you can set a cmd sequence to firstly execute pve node > maintenance mode , and then execute shutdown -> this way HA will not try = to > migrate vm to node in maintenance, and the chance all nodes to goes into > maintenance in exactly same second seems to be not a risk at all. > >>>> > >>>> hope thats helpful. > >>>> > >>>> Regards, > >>>> Sto. > >>>> > >>>>> On Mar 10, 2022, at 1:10 PM, Stefan Radman via pve-user < > pve-user@lists.proxmox.com > wrote: > >>>>> > >>>>> > >>>>> From: Stefan Radman stefan.radman@me.com>> > >>>>> Subject: Locking HA during UPS shutdown > >>>>> Date: March 10, 2022 at 1:10:09 PM GMT+2 > >>>>> To: PVE User List pve-user@pve.proxmox.com>> > >>>>> > >>>>> > >>>>> Hi > >>>>> > >>>>> I am configuring a 3 node PVE cluster with integrated Ceph storage. > >>>>> > >>>>> It is powered by 2 UPS that are monitored by NUT (Network UPS Tools= ). > >>>>> > >>>>> HA is configured with 3 groups: > >>>>> group pve1 nodes pve1:1,pve2,pve3 > >>>>> group pve2 nodes pve1,pve2:1,pve3 > >>>>> group pve3 nodes pve1,pve2,pve3:1 > >>>>> > >>>>> That will normally place the VMs in each group on the corresponding > node, unless that node fails. > >>>>> > >>>>> The cluster is configured to migrate VMs away from a node before > shutting it down (Cluster=3D>Options=3D>HA Settings: shutdown_policy=3Dmi= grate). > >>>>> > >>>>> NUT is configured to shut down the serves once the last of the two > UPS is running low on battery. > >>>>> > >>>>> My problem: > >>>>> When NUT starts shutting down the 3 nodes, HA will first try to > live-migrate them to another node. > >>>>> That live migration process gets stuck because all the nodes are > shutting down simultaneously. > >>>>> It seems that the whole process runs into a timeout, finally =E2=80= =9Cpowers > off=E2=80=9D all the VMs and shuts down the nodes. > >>>>> > >>>>> My question: > >>>>> Is there a way to =E2=80=9Clock=E2=80=9D or temporarily de-activate= HA before > shutting down a node to avoid that deadlock? > >>>>> > >>>>> Thank you > >>>>> > >>>>> Stefan > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> _______________________________________________ > >>>>> pve-user mailing list > >>>>> pve-user@lists.proxmox.com > >>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user < > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user> > >>>> > >>>> > >>>> Best Regards, > >>>> > >>>> Stoyan Stoyanov Sto | Solutions Manager > >>>> | Telehouse.Solutions | ICT Department > >>>> | phone/viber: +359 894774934 > >>>> | telegram: @prostoSto < > https://mysignature.io/redirect/skype:prosto.sto?chat> > >>>> | skype: prosto.sto < > https://mysignature.io/redirect/skype:prosto.sto?chat> > >>>> | email: sto@telehouse.solutions > >>>> | website: www.telehouse.solutions > >>>> | address: Telepoint #2, Sofia, Bulgaria > >>>> <356841.png> > >>>> > >>>> > >>>> Save paper. Don=E2=80=99t print > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> Best Regards, > >>>> > >>>> Stoyan Stoyanov Sto | Solutions Manager > >>>> | Telehouse.Solutions | ICT Department > >>>> | phone/viber: +359 894774934 > >>>> | telegram: @prostoSto < > https://mysignature.io/redirect/skype:prosto.sto?chat> > >>>> | skype: prosto.sto < > https://mysignature.io/redirect/skype:prosto.sto?chat> > >>>> | email: sto@telehouse.solutions > >>>> | website: www.telehouse.solutions > >>>> | address: Telepoint #2, Sofia, Bulgaria > >>>> <356841.png> > >>>> > >>>> > >>>> Save paper. Don=E2=80=99t print > >>> > >> > >> > >> Best Regards, > >> > >> Stoyan Stoyanov Sto | Solutions Manager > >> | Telehouse.Solutions | ICT Department > >> | phone/viber: +359 894774934 > >> | telegram: @prostoSto < > https://mysignature.io/redirect/skype:prosto.sto?chat> > >> | skype: prosto.sto < > https://mysignature.io/redirect/skype:prosto.sto?chat> > >> | email: sto@telehouse.solutions > >> | website: www.telehouse.solutions > >> | address: Telepoint #2, Sofia, Bulgaria > >> > >> > >> > >> Save paper. Don=E2=80=99t print > >> > >> > >> > >> > >> _______________________________________________ > >> pve-user mailing list > >> pve-user@lists.proxmox.com > >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > Best Regards, > > > > Stoyan Stoyanov Sto | Solutions Manager > > | Telehouse.Solutions | ICT Department > > | phone/viber: +359 894774934 > > | telegram: @prostoSto < > https://mysignature.io/redirect/skype:prosto.sto?chat> > > | skype: prosto.sto < > https://mysignature.io/redirect/skype:prosto.sto?chat> > > | email: sto@telehouse.solutions > > | website: www.telehouse.solutions > > | address: Telepoint #2, Sofia, Bulgaria > > > > > > > > Save paper. Don=E2=80=99t print > > > > > > > > > > _______________________________________________ > > pve-user mailing list > > pve-user@lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > _______________________________________________ > pve-user mailing list > pve-user@lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > --=20 Do have a Blessed Day