From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id A2DC671E5B for ; Fri, 10 Jun 2022 13:18:02 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 82DF6269ED for ; Fri, 10 Jun 2022 13:18:02 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 34CFE269BA for ; Fri, 10 Jun 2022 13:17:59 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 06C9643AC9 for ; Fri, 10 Jun 2022 13:17:59 +0200 (CEST) From: Dominik Csapak To: pbs-devel@lists.proxmox.com Date: Fri, 10 Jun 2022 13:17:49 +0200 Message-Id: <20220610111757.2788694-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.047 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment POISEN_SPAM_PILL 0.1 Meta: its spam POISEN_SPAM_PILL_1 0.1 random spam to be learned in bayes POISEN_SPAM_PILL_3 0.1 random spam to be learned in bayes SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [influxdbhttp.rs, mod.rs, influxdbudp.rs, metrics.rs, lib.rs, proxmox-backup-proxy.rs] Subject: [pbs-devel] [PATCH proxmox/proxmox-backup v9] add metrics server capability X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Jun 2022 11:18:02 -0000 this series adds support for exporting metrics data to external metric servers. for now this includes only data we gather for RRD, though it should not be hard to extend that functionality also only influxdb (udp/http(s)) is currently supported, but it should also not be too hard to include more options here i changed the interface of send_data_to_channels, so we'd need to bump proxmox-metrics once and add the updated dependency changes from v8: * changed the send_data_to_channels interface slightly * changed the interface in get_metric_server_connections to Vec<(Metrics, String), instead of (Vec, Vec) * used updater in api type for enable, so that we have a bool instead of an Option * do 'const *_SCHEMA: &ObjectSchema' in config * fixed a small bug in the ui (not noticable without the dev console open), where we'd try to load and set data into the edit window when adding a new metric server changes from v7: * rebase on master * changes api paths from 'metricserver' to 'metrics' * moved gui option to configuration (instead of top-level) * moved metricserver type/xtype definition to 'PBS.Schema' * incorporated wolfgangs feedback for the config (unwrap_object_schema; unwrap_or_default) * incorporated matthias feedback (dropped unnecessary borrows) changes from v5/6: * rebased on master changes from v4: * rebase on master * move connect_to_udp to udp::connect(), and let it really try every address * adds 'test_influxdb_http/udp' functions that try to connect for some sanity checks. (needed a little refactor in the http/udp parts but mostly code move) * checks the server connection on create/edit via api when the enable flag is set * adds a generic 'list' api call in /admin/metricserver since we need a place where we return *all* metrics servers regardless of type (and this is the way we do it e.g. for realms) * adds the gui to view/add/edit/delete the metric servers (i put it under configuration, but that gets crowded... maybe there is a better place?) changes from v3: * rebase on master * introduced helper functions instead of InfluxDBHttp::new * start tokio task directly in the helper * combine channel close + join * fix api description * combine host/port/protocol in the api types * introduce a connect_to_udp helper * use NixPath in the fs_info helper changes from v2: * rebase on master * rustfmt * clippy (fixed not everything) * renamed DiskUsage in proxmox-sys and added some more fields * added 'enable' property for the config (like we have in pve) * subtracted 50bytes from mtu in the udp variant (for ip header) changes from v1: * fixed ipv6 support for udp (tested it this time ;) ) * dropped the 'flush' functionality of the MetricsChannel, but kept the wrapper struct: it did not do what i intended, and after rethinking it, turns out it's not necessary (as we autoflush when the data gets to large, or when we close the channel). kept the struct so that the interface can stay the same even if we want to implement a manual flush in the future * improved the influxdb line formatter * removed variables like 'names2' by reorganizing the code * used Arc::clone(&foo) instead of foo.clone() (better visibilty) * used CamelCase for the DeletableProperties proxmox: Dominik Csapak (1): proxmox-metrics: send_data_to_channels: change from slice to IntoIterator proxmox-metrics/src/lib.rs | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) proxmox-backup: Dominik Csapak (7): pbs-api-types: add metrics api types pbs-config: add metrics config class backup-proxy: decouple stats gathering from rrd update proxmox-backup-proxy: send metrics to configured metrics server api: add metricserver endpoints ui: add window/InfluxDbEdit ui: add MetricServerView and use it Cargo.toml | 1 + pbs-api-types/src/lib.rs | 17 ++ pbs-api-types/src/metrics.rs | 148 ++++++++++ pbs-config/Cargo.toml | 1 + pbs-config/src/lib.rs | 1 + pbs-config/src/metrics.rs | 105 ++++++++ src/api2/admin/metrics.rs | 91 +++++++ src/api2/admin/mod.rs | 2 + src/api2/config/metrics/influxdbhttp.rs | 315 ++++++++++++++++++++++ src/api2/config/metrics/influxdbudp.rs | 270 +++++++++++++++++++ src/api2/config/metrics/mod.rs | 16 ++ src/api2/config/mod.rs | 2 + src/bin/proxmox-backup-proxy.rs | 342 +++++++++++++++++++----- www/Makefile | 3 + www/Schema.js | 15 ++ www/SystemConfiguration.js | 6 + www/config/MetricServerView.js | 128 +++++++++ www/window/InfluxDbEdit.js | 218 +++++++++++++++ 18 files changed, 1610 insertions(+), 71 deletions(-) create mode 100644 pbs-api-types/src/metrics.rs create mode 100644 pbs-config/src/metrics.rs create mode 100644 src/api2/admin/metrics.rs create mode 100644 src/api2/config/metrics/influxdbhttp.rs create mode 100644 src/api2/config/metrics/influxdbudp.rs create mode 100644 src/api2/config/metrics/mod.rs create mode 100644 www/Schema.js create mode 100644 www/config/MetricServerView.js create mode 100644 www/window/InfluxDbEdit.js -- 2.30.2