From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 8589F847BF for ; Tue, 14 Dec 2021 13:24:45 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 783A3242C8 for ; Tue, 14 Dec 2021 13:24:15 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 002042429E for ; Tue, 14 Dec 2021 13:24:13 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id CACAE4519F for ; Tue, 14 Dec 2021 13:24:13 +0100 (CET) From: Dominik Csapak To: pbs-devel@lists.proxmox.com Date: Tue, 14 Dec 2021 13:24:03 +0100 Message-Id: <20211214122412.4077902-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.177 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox-backup-proxy.rs, datastore.rs, lib.rs, status.rs, metrics.rs, udp.rs, mod.rs, influxdbhttp.rs, http.rs, influxdbudp.rs, utils.rs] Subject: [pbs-devel] [PATCH proxmox/proxmox-backup] add metrics server capability X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Dec 2021 12:24:45 -0000 this series adds support for exporting metrics data to external metric servers. for now this includes only data we gather for RRD, though it should not be hard to extend that functionality also only influxdb (udp/http(s)) is currently supported, but it should also not be too hard to include more options here i did not include gui/cli patches yet, as i find the proxmox-backup-manager options are already too much and i waited for the gui for some feedback. for testing, the metric servers can be added either by calling 'proxmox-backup debug api ...' or by manually editing the file a bit unrelated: i moved the 'disk_usage' method to proxmox-sys (fits better there, and we want to use that from other things too) ofc, proxmox-backup depends on bumped versions of the proxmox-* crates proxmox: Dominik Csapak (3): proxmox-sys: make some structs serializable proxmox-sys: add DiskUsage struct and helper proxmox-metrics: implement metrics server client code Cargo.toml | 1 + proxmox-metrics/Cargo.toml | 20 ++++ proxmox-metrics/debian/changelog | 5 + proxmox-metrics/debian/copyright | 16 +++ proxmox-metrics/debian/debcargo.toml | 7 ++ proxmox-metrics/src/influxdb/http.rs | 143 ++++++++++++++++++++++++++ proxmox-metrics/src/influxdb/mod.rs | 7 ++ proxmox-metrics/src/influxdb/udp.rs | 107 +++++++++++++++++++ proxmox-metrics/src/influxdb/utils.rs | 51 +++++++++ proxmox-metrics/src/lib.rs | 92 +++++++++++++++++ proxmox-sys/Cargo.toml | 1 + proxmox-sys/src/fs/mod.rs | 26 +++++ proxmox-sys/src/linux/procfs/mod.rs | 7 +- 13 files changed, 480 insertions(+), 3 deletions(-) create mode 100644 proxmox-metrics/Cargo.toml create mode 100644 proxmox-metrics/debian/changelog create mode 100644 proxmox-metrics/debian/copyright create mode 100644 proxmox-metrics/debian/debcargo.toml create mode 100644 proxmox-metrics/src/influxdb/http.rs create mode 100644 proxmox-metrics/src/influxdb/mod.rs create mode 100644 proxmox-metrics/src/influxdb/udp.rs create mode 100644 proxmox-metrics/src/influxdb/utils.rs create mode 100644 proxmox-metrics/src/lib.rs proxmox-backup: Dominik Csapak (6): use 'disk_usage' from proxmox-sys pbs-api-types: add metrics api types pbs-config: add metrics config class backup-proxy: decouple stats gathering from rrd update proxmox-backup-proxy: send metrics to configured metrics server api: add metricserver endpoints Cargo.toml | 1 + pbs-api-types/src/lib.rs | 2 + pbs-api-types/src/metrics.rs | 134 +++++++ pbs-config/Cargo.toml | 1 + pbs-config/src/lib.rs | 1 + pbs-config/src/metrics.rs | 122 +++++++ src/api2/admin/datastore.rs | 4 +- src/api2/config/metricserver/influxdbhttp.rs | 272 +++++++++++++++ src/api2/config/metricserver/influxdbudp.rs | 242 +++++++++++++ src/api2/config/metricserver/mod.rs | 16 + src/api2/config/mod.rs | 2 + src/api2/node/status.rs | 10 +- src/api2/status.rs | 4 +- src/bin/proxmox-backup-proxy.rs | 348 +++++++++++++++---- src/tools/disks/mod.rs | 21 +- 15 files changed, 1083 insertions(+), 97 deletions(-) create mode 100644 pbs-api-types/src/metrics.rs create mode 100644 pbs-config/src/metrics.rs create mode 100644 src/api2/config/metricserver/influxdbhttp.rs create mode 100644 src/api2/config/metricserver/influxdbudp.rs create mode 100644 src/api2/config/metricserver/mod.rs -- 2.30.2