[PATCH manager] fix #4911: external metric: better handle failed connections

[Lukas Sichert <l.sichert@proxmox.com>]

When an external metric server configured to use TCP is unreachable, the
storage and VM indicators of the cluster nodes in the UI turn gray. This
is because, currently, a failed connection attempt raises an unhandled
exception, which aborts the status update flow.  As the connection
attempts happen at the beginning of the update process, status
information is then not broadcasted within the system or across the
cluster. After five minutes without updates, the frontend marks the
indicators as gray.

To catch connection errors, wrap connection establishment in an eval
block. The implementation ensures that other connections to external
metric servers are still established, even if one fails.

Signed-off-by: Lukas Sichert <l.sichert@proxmox.com>
---

Notes:
    Although this commit correctly catches the errors, it produces a lot of
    warnings in the system log. Every 10 seconds, 4 connection attempts
    (LXC, QEMU, Node and Storage) are made and therefore also 4 warnings are
    printed to the system log.
    
    Better implementations might be:
    	1. Establish connection once and then reuse it for all 4 data package
    	   types.
    	   This might be the most elegant solution, but if the connection is
    	   lost in between the messages, it could lead to other errors.
    	2. Use a Perl-hash that is propagated through the function call
    	   layers.  Check for uniqueness of the error, and if unique,
    	   add error message to dictionary. Then at the end print the messages
    	   in the dictionary to the system log.
    	  This solution is less elegant, but it is robust and does not
    	  require bigger changes in the code.
    	3. Use a timer and only print a warning if sufficient time has
    	   elapsed from the last print.
    	   This option might be slightly more elegant than option 2, however
    	   it could  lead to nondeterministic behavior, because it depends on
    	   which function the timer runs out in.
    
    If anyone has any suggestions regarding this, please let me know.
    Thanks in advance!

 PVE/ExtMetric.pm | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

diff --git a/PVE/ExtMetric.pm b/PVE/ExtMetric.pm
index ebc2817b..bb36930e 100644
--- a/PVE/ExtMetric.pm
+++ b/PVE/ExtMetric.pm
@@ -52,17 +52,21 @@ sub transactions_start {
         $cfg,
         sub {
             my ($plugin, $id, $plugin_config) = @_;
-
-            my $connection = $plugin->_connect($plugin_config, $id);
-
-            push @$transactions,
-                {
-                    connection => $connection,
-                    cfg => $plugin_config,
-                    id => $id,
-                    data => '',
-                };
-        },
+            eval {
+              my $connection = $plugin->_connect($plugin_config, $id);
+
+              push @$transactions,
+                  {
+                      connection => $connection,
+                      cfg => $plugin_config,
+                      id => $id,
+                      data => '',
+                  };
+            };
+            if (my $err = $@) {
+                syslog( "warning", "connection for plugin '$id' failed: $err");
+            }    
+          },
     );
 
     return $transactions;
-- 
2.47.3