[PATCH datacenter-manager v2 0/4] implement back-off mechanism for connection errors for remotes

[Dominik Csapak <d.csapak@proxmox.com>]

When a remote is not reachable (e.g. network outage, crashes, etc), PDM
tries to connect on every attempt with a timeout. This leads to heavily
delayed api calls in the PDM UI. To counter that, this series implements
a basic back-off mechanism that increases the time between actual api
calls in an exponential way (up to a maximum).

For details on how the back-off mechanism works see patch 1/4

Possible Improvements/Future Work:
* We could expose the back-off values via a config (either global or per
  remote) to give the admin some fine grained control over this behavior
* There is still quite a bit of logs after this, but this can be cleaned
  up/improved upon later too.

changes from v1:
* rebased on master (dropped equivalent patches)
* rework most of the code

Dominik Csapak (4):
  server: remote cache: prepare for back-off mechanism
  server: remote cache: introduce canary remote when none is reachable
  server: connection: multi-client: use back-off state from remote cache
  tasks: remote node mapping: use host cache for PBS too

 .../tasks/remote_node_mapping.rs              |  34 ++-
 server/src/connection.rs                      | 118 +++++++---
 server/src/remote_cache/back_off.rs           | 128 ++++++++++
 server/src/remote_cache/mod.rs                | 222 +++++++++++++++++-
 4 files changed, 439 insertions(+), 63 deletions(-)
 create mode 100644 server/src/remote_cache/back_off.rs

-- 
2.47.3