[PVE-User] Proxmox Fencing

[PVE-User] Proxmox Fencing

[Alex K]

Hi all,

I'm new to proxmox and trying to setup a 2 + 1 node active/active HA
cluster on top glusterfs using latest community pve-manager/6.4-4/337d6701
(running kernel: 5.4.106-1-pve). The third node is  used for gluster
arbitration and perhaps I have to configure in it a quorum disk also to
keep quorum in case of a node failure (not clear yet at my mind, still
reading the docs).

I am stuck at the moment at the fencing part of the setup. Reading through
the docs it seems that I have only the option to setup hardware watchdog
fencing. I would expect to be able to use external media such as IPMI,
iDrac, HP iLO or UPS based power management (APC) though I can't find any
info how these are configured at current version of Proxmox.

In case of a network partition and not a node hardware issue, how is the
watchdog going to behave? Is a healthy but disconnected node going to be
power cycled? I will soon proceed with testing as soon as I manage to setup
fencing though I wanted to better understand this part of fencing.

Appreciate any feedback from your experience and use cases,
Alex

Re: [PVE-User] Proxmox Fencing

[Alex K]

Hi Eneko,

On Mon, Jul 5, 2021 at 2:30 PM Eneko Lacunza via pve-user <
pve-user@lists.proxmox.com> wrote:

>
>
>
> ---------- Forwarded message ----------
> From: Eneko Lacunza <elacunza@binovo.es>
> To: pve-user@lists.proxmox.com
> Cc:
> Bcc:
> Date: Mon, 5 Jul 2021 13:30:41 +0200
> Subject: Re: [PVE-User] Proxmox Fencing
> Hi Alex,
>
> El 5/7/21 a las 13:18, Alex K escribió:
> > Hi all,
> >
> > I'm new to proxmox and trying to setup a 2 + 1 node active/active HA
> > cluster on top glusterfs using latest community
> pve-manager/6.4-4/337d6701
> > (running kernel: 5.4.106-1-pve). The third node is  used for gluster
> > arbitration and perhaps I have to configure in it a quorum disk also to
> > keep quorum in case of a node failure (not clear yet at my mind, still
> > reading the docs).
> If you have 3 nodes, you want all them in Proxmox cluster for proper
> quorum majority. No need for quorum disk that way. (note that I don't
> know how gluster works).
>
Gluser has a similar concept for quorum so as to keep writes on the
storage. Hence I am placing a third node in the setup. Due to cost
limitations, the third node has minimal specs and is not meant to host VMs.
It is a mini-PC thats why I did not add it as a proxmox host. I am
wondering if it is possible to add it as a proxmox host and put a
constraint to avoid VMs migrating into it. In this way I will achieve the
required quorum levels without adding a full spec host.

> I am stuck at the moment at the fencing part of the setup. Reading through
> > the docs it seems that I have only the option to setup hardware watchdog
> > fencing. I would expect to be able to use external media such as IPMI,
> > iDrac, HP iLO or UPS based power management (APC) though I can't find any
> > info how these are configured at current version of Proxmox.
> Currently by default Proxmox uses a software watchdog. I'm not sure if
> hardware watchdog support was introduced, others may help with this.
>
According to the docs it seems there is hardware watchdog option:
https://pve.proxmox.com/pve-docs/chapter-ha-manager.html
Q+++
hardware watchdog - if not available we fall back to the linux kernel
software watchdog (softdog)
+++Q



> > In case of a network partition and not a node hardware issue, how is the
> > watchdog going to behave? Is a healthy but disconnected node going to be
> > power cycled? I will soon proceed with testing as soon as I manage to
> setup
> > fencing though I wanted to better understand this part of fencing.
>
> The node that drops out of quorum will be rebooted. If there where
> CM/CTs configured for HA in that node, Proxmox will attempt to restart
> them in another node.
>
So soft-fencing is done from ha-manager? How are the other nodes notified
that the rebooted host is indeed rebooted so as to start the HA VMs?

Thanx for the feedback


> Cheers
>
> Eneko Lacunza
> Zuzendari teknikoa | Director técnico
> Binovo IT Human Project
>
> Tel. +34 943 569 206 | https://www.binovo.es
> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
>
> https://www.youtube.com/user/CANALBINOVO
> https://www.linkedin.com/company/37269706/
>
>
>
>
>
> ---------- Forwarded message ----------
> From: Eneko Lacunza via pve-user <pve-user@lists.proxmox.com>
> To: pve-user@lists.proxmox.com
> Cc: Eneko Lacunza <elacunza@binovo.es>
> Bcc:
> Date: Mon, 5 Jul 2021 13:30:41 +0200
> Subject: Re: [PVE-User] Proxmox Fencing
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>

Re: [PVE-User] Proxmox Fencing

[Alex K]

On Mon, Jul 5, 2021 at 3:06 PM Eneko Lacunza via pve-user <
pve-user@lists.proxmox.com> wrote:

>
>
>
> ---------- Forwarded message ----------
> From: Eneko Lacunza <elacunza@binovo.es>
> To: pve-user@lists.proxmox.com
> Cc:
> Bcc:
> Date: Mon, 5 Jul 2021 14:06:31 +0200
> Subject: Re: [PVE-User] Proxmox Fencing
> Hi Alex,
>
> >> El 5/7/21 a las 13:18, Alex K escribió:
> >>> Hi all,
> >>>
> >>> I'm new to proxmox and trying to setup a 2 + 1 node active/active HA
> >>> cluster on top glusterfs using latest community
> >> pve-manager/6.4-4/337d6701
> >>> (running kernel: 5.4.106-1-pve). The third node is  used for gluster
> >>> arbitration and perhaps I have to configure in it a quorum disk also to
> >>> keep quorum in case of a node failure (not clear yet at my mind, still
> >>> reading the docs).
> >> If you have 3 nodes, you want all them in Proxmox cluster for proper
> >> quorum majority. No need for quorum disk that way. (note that I don't
> >> know how gluster works).
> >>
> > Gluser has a similar concept for quorum so as to keep writes on the
> > storage. Hence I am placing a third node in the setup. Due to cost
> > limitations, the third node has minimal specs and is not meant to host
> VMs.
> > It is a mini-PC thats why I did not add it as a proxmox host. I am
> > wondering if it is possible to add it as a proxmox host and put a
> > constraint to avoid VMs migrating into it. In this way I will achieve the
> > required quorum levels without adding a full spec host.
>
> Yes, you can create node-groups in HA groups, and add the desired nodes
> to the group. Then when adding a VM/CT to HA, configure the group there
> too.
>
I see. Thanx for the pointer.

>> I am stuck at the moment at the fencing part of the setup. Reading
> through
> >>> the docs it seems that I have only the option to setup hardware
> watchdog
> >>> fencing. I would expect to be able to use external media such as IPMI,
> >>> iDrac, HP iLO or UPS based power management (APC) though I can't find
> any
> >>> info how these are configured at current version of Proxmox.
> >> Currently by default Proxmox uses a software watchdog. I'm not sure if
> >> hardware watchdog support was introduced, others may help with this.
> >>
> > According to the docs it seems there is hardware watchdog option:
> > https://pve.proxmox.com/pve-docs/chapter-ha-manager.html
> > Q+++
> > hardware watchdog - if not available we fall back to the linux kernel
> > software watchdog (softdog)
> > +++Q
> Never used that, sorry.
> >>> In case of a network partition and not a node hardware issue, how is
> the
> >>> watchdog going to behave? Is a healthy but disconnected node going to
> be
> >>> power cycled? I will soon proceed with testing as soon as I manage to
> >> setup
> >>> fencing though I wanted to better understand this part of fencing.
> >> The node that drops out of quorum will be rebooted. If there where
> >> CM/CTs configured for HA in that node, Proxmox will attempt to restart
> >> them in another node.
> >>
> > So soft-fencing is done from ha-manager? How are the other nodes notified
> > that the rebooted host is indeed rebooted so as to start the HA VMs?
> There is a time delay that allows the fended node time to reboot before
> other nodes take over the HA VMs. It's like 1-2 minutes. The fenced node
> (the one out of the quorum) will reboot in max 60s.
>
OK. I understand that there is a locking mechanism which takes place and
determines the node states.


> Cheers
>
> Eneko Lacunza
> Zuzendari teknikoa | Director técnico
> Binovo IT Human Project
>
> Tel. +34 943 569 206 | https://www.binovo.es
> Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
>
> https://www.youtube.com/user/CANALBINOVO
> https://www.linkedin.com/company/37269706/
>
>
>
>
>
> ---------- Forwarded message ----------
> From: Eneko Lacunza via pve-user <pve-user@lists.proxmox.com>
> To: pve-user@lists.proxmox.com
> Cc: Eneko Lacunza <elacunza@binovo.es>
> Bcc:
> Date: Mon, 5 Jul 2021 14:06:31 +0200
> Subject: Re: [PVE-User] Proxmox Fencing
> _______________________________________________
> pve-user mailing list
> pve-user@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>