From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <d.csapak@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 9C6E2615FE for <pbs-devel@lists.proxmox.com>; Mon, 17 Jan 2022 11:48:28 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 911EE18B89 for <pbs-devel@lists.proxmox.com>; Mon, 17 Jan 2022 11:48:28 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 55C5E18B72 for <pbs-devel@lists.proxmox.com>; Mon, 17 Jan 2022 11:48:27 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 2D64C460C3 for <pbs-devel@lists.proxmox.com>; Mon, 17 Jan 2022 11:48:27 +0100 (CET) From: Dominik Csapak <d.csapak@proxmox.com> To: pbs-devel@lists.proxmox.com Date: Mon, 17 Jan 2022 11:48:15 +0100 Message-Id: <20220117104825.2409598-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.163 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH proxmox/proxmox-backup v4] add metrics server capability X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion <pbs-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/> List-Post: <mailto:pbs-devel@lists.proxmox.com> List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>, <mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Mon, 17 Jan 2022 10:48:28 -0000 this series adds support for exporting metrics data to external metric servers. for now this includes only data we gather for RRD, though it should not be hard to extend that functionality also only influxdb (udp/http(s)) is currently supported, but it should also not be too hard to include more options here i did not include gui/cli patches yet, as i find the proxmox-backup-manager options are already too much and i waited for the gui for some feedback. for testing, the metric servers can be added either by calling 'proxmox-backup debug api ...' or by manually editing the file ofc, proxmox-backup depends on bumped versions of the proxmox-* crates changes from v3: * rebase on master * introduced helper functions instead of InfluxDBHttp::new * start tokio task directly in the helper * combine channel close + join * fix api description * combine host/port/protocol in the api types * introduce a connect_to_udp helper * use NixPath in the fs_info helper changes from v2: * rebase on master * rustfmt * clippy (fixed not everything) * renamed DiskUsage in proxmox-sys and added some more fields * added 'enable' property for the config (like we have in pve) * subtracted 50bytes from mtu in the udp variant (for ip header) changes from v1: * fixed ipv6 support for udp (tested it this time ;) ) * dropped the 'flush' functionality of the MetricsChannel, but kept the wrapper struct: it did not do what i intended, and after rethinking it, turns out it's not necessary (as we autoflush when the data gets to large, or when we close the channel). kept the struct so that the interface can stay the same even if we want to implement a manual flush in the future * improved the influxdb line formatter * removed variables like 'names2' by reorganizing the code * used Arc::clone(&foo) instead of foo.clone() (better visibilty) * used CamelCase for the DeletableProperties proxmox: Dominik Csapak (4): proxmox-sys: make some structs serializable proxmox-sys: add FileSystemInformation struct and helper proxmox-async: add connect_to_udp helper proxmox-metrics: implement metrics server client code Cargo.toml | 1 + proxmox-async/Cargo.toml | 2 +- proxmox-async/src/io/mod.rs | 3 + proxmox-async/src/io/udp_connect.rs | 18 ++++ proxmox-metrics/Cargo.toml | 21 ++++ proxmox-metrics/debian/changelog | 5 + proxmox-metrics/debian/copyright | 16 ++++ proxmox-metrics/debian/debcargo.toml | 7 ++ proxmox-metrics/src/influxdb/http.rs | 132 ++++++++++++++++++++++++++ proxmox-metrics/src/influxdb/mod.rs | 7 ++ proxmox-metrics/src/influxdb/udp.rs | 80 ++++++++++++++++ proxmox-metrics/src/influxdb/utils.rs | 50 ++++++++++ proxmox-metrics/src/lib.rs | 117 +++++++++++++++++++++++ proxmox-sys/Cargo.toml | 1 + proxmox-sys/src/fs/mod.rs | 37 ++++++++ proxmox-sys/src/linux/procfs/mod.rs | 7 +- 16 files changed, 500 insertions(+), 4 deletions(-) create mode 100644 proxmox-async/src/io/udp_connect.rs create mode 100644 proxmox-metrics/Cargo.toml create mode 100644 proxmox-metrics/debian/changelog create mode 100644 proxmox-metrics/debian/copyright create mode 100644 proxmox-metrics/debian/debcargo.toml create mode 100644 proxmox-metrics/src/influxdb/http.rs create mode 100644 proxmox-metrics/src/influxdb/mod.rs create mode 100644 proxmox-metrics/src/influxdb/udp.rs create mode 100644 proxmox-metrics/src/influxdb/utils.rs create mode 100644 proxmox-metrics/src/lib.rs proxmox-backup: Dominik Csapak (6): use 'fs_info' from proxmox-sys pbs-api-types: add metrics api types pbs-config: add metrics config class backup-proxy: decouple stats gathering from rrd update proxmox-backup-proxy: send metrics to configured metrics server api: add metricserver endpoints Cargo.toml | 1 + pbs-api-types/src/lib.rs | 15 + pbs-api-types/src/metrics.rs | 138 ++++++++ pbs-config/Cargo.toml | 1 + pbs-config/src/lib.rs | 1 + pbs-config/src/metrics.rs | 115 +++++++ src/api2/admin/datastore.rs | 4 +- src/api2/config/metricserver/influxdbhttp.rs | 266 +++++++++++++++ src/api2/config/metricserver/influxdbudp.rs | 243 +++++++++++++ src/api2/config/metricserver/mod.rs | 16 + src/api2/config/mod.rs | 2 + src/api2/node/status.rs | 11 +- src/api2/status.rs | 4 +- src/bin/proxmox-backup-proxy.rs | 342 +++++++++++++++---- src/tools/disks/mod.rs | 21 +- 15 files changed, 1081 insertions(+), 99 deletions(-) create mode 100644 pbs-api-types/src/metrics.rs create mode 100644 pbs-config/src/metrics.rs create mode 100644 src/api2/config/metricserver/influxdbhttp.rs create mode 100644 src/api2/config/metricserver/influxdbudp.rs create mode 100644 src/api2/config/metricserver/mod.rs -- 2.30.2