[PATCH docs v2 14/14] ha-manager: document the auto-rebalance margin mode, cooldown and failed migrations

[Michael Ryom <Michael@RyomHerold.dk>]

The relative imbalance improvement provides almost no hysteresis when a
single HA resource dominates the cluster load (see bug #8059), which the
new ha-auto-rebalance-margin-mode and ha-auto-rebalance-cooldown options
guard against. Also describe how the load balancer handles failed
rebalancing migrations now.

While at it, add an overview table of the load balancer options, since
the surrounding text refers to them by their user interface labels only.

Signed-off-by: Michael Ryom <Michael@RyomHerold.dk>
---
 ha-manager.adoc | 62 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/ha-manager.adoc b/ha-manager.adoc
index 95ab799..bbf1ce1 100644
--- a/ha-manager.adoc
+++ b/ha-manager.adoc
@@ -1614,6 +1614,68 @@ the expected imbalance does not change at all. The value can never be
 negative, so the balancer will never commit to a migration with an expected
 imbalance worse than the current one.
 
+By default, the _imbalance improvement_ is relative to the current imbalance.
+That provides very little hysteresis if a single HA resource dominates the
+cluster load: moving that resource in either direction can look like a large
+relative improvement, even though the cluster ends up about as imbalanced as
+it was before, so small fluctuations of the node loads can make the load
+balancer move the resource back and forth. With the _improvement mode_ set to
+`absolute`, the _imbalance improvement_ is the number of percentage points by
+which the expected imbalance has to drop instead, which ignores such marginal
+migrations. Additionally, the _cooldown_ option keeps HA resources, which were
+moved by the load balancer, from being moved again for the given number of
+seconds. It defaults to `0`, which disables the cooldown.
+
+If a rebalancing migration fails, for example because the guest uses a local
+resource that is not available on the target node, the load balancer does not
+try the same migration again until the imbalance has exceeded the threshold
+for the hold duration anew and no other rebalancing migration succeeded in
+the meantime. Instead, it tries the next best rebalancing migration. If the
+same migration keeps failing, it is skipped for exponentially more of these
+rebalancing attempts.
+
+[[ha_manager_crs_load_balancer_options]]
+Load Balancer Options
+^^^^^^^^^^^^^^^^^^^^^
+
+The automatic load balancer is configured with the following options of the
+`crs` setting in `/etc/pve/datacenter.cfg`, which can also be set in the web
+interface under `Datacenter` -> `Options` -> `Cluster Resource Scheduling`:
+
+[width="100%",cols="3m,1d,2d,5d",options="header"]
+|==========================================================================
+|Option |Default |Values |Description
+
+|ha-auto-rebalance |0 |0, 1
+|Enable the automatic load balancer. The options below have no effect unless
+it is enabled.
+
+|ha-auto-rebalance-threshold |30 |0-100%
+|Cluster imbalance above which the load balancer becomes active. With `0` it
+is always active.
+
+|ha-auto-rebalance-hold-duration |3 |>= 0 rounds
+|Number of consecutive HA Manager rounds, each around 10 seconds, for which
+the threshold has to be exceeded before a migration is selected.
+
+|ha-auto-rebalance-method |bruteforce |bruteforce, topsis
+|Method used to score the possible rebalancing migrations.
+
+|ha-auto-rebalance-margin |10 |0-100
+|_Imbalance improvement_: the reduction of the cluster imbalance a migration
+has to achieve to be committed, in percent of the current imbalance or in
+percentage points, depending on the improvement mode.
+
+|ha-auto-rebalance-margin-mode |relative |relative, absolute
+|_Improvement mode_: whether the imbalance improvement is relative to the
+current imbalance or absolute.
+
+|ha-auto-rebalance-cooldown |0 |>= 0 seconds
+|_Cooldown_: the time after a successful rebalancing migration in which the
+moved HA resources are not moved again by the load balancer. With `0` the
+cooldown is disabled.
+|==========================================================================
+
 NOTE: It is planned that load balancing will also include non-HA resources in
 the future.
 
-- 
2.47.3