From: Dominik Csapak <d.csapak@proxmox.com>
To: pbs-devel@lists.proxmox.com
Subject: [pbs-devel] [PATCH proxmox/proxmox-backup v3] add metrics server capability
Date: Fri, 17 Dec 2021 09:09:51 +0100 [thread overview]
Message-ID: <20211217081000.1061796-1-d.csapak@proxmox.com> (raw)
this series adds support for exporting metrics data to external
metric servers.
for now this includes only data we gather for RRD, though it should
not be hard to extend that functionality
also only influxdb (udp/http(s)) is currently supported, but it should
also not be too hard to include more options here
i did not include gui/cli patches yet, as i find the
proxmox-backup-manager options are already too much and i waited for
the gui for some feedback.
for testing, the metric servers can be added either by
calling 'proxmox-backup debug api ...' or by manually editing the
file
ofc, proxmox-backup depends on bumped versions of the proxmox-* crates
changes from v2:
* rebase on master
* rustfmt
* clippy (fixed not everything)
* renamed DiskUsage in proxmox-sys and added some more fields
* added 'enable' property for the config (like we have in pve)
* subtracted 50bytes from mtu in the udp variant (for ip header)
changes from v1:
* fixed ipv6 support for udp (tested it this time ;) )
* dropped the 'flush' functionality of the MetricsChannel, but kept the
wrapper struct: it did not do what i intended, and after rethinking it,
turns out it's not necessary (as we autoflush when the data gets to large,
or when we close the channel). kept the struct so that the interface
can stay the same even if we want to implement a manual flush in the future
* improved the influxdb line formatter
* removed variables like 'names2' by reorganizing the code
* used Arc::clone(&foo) instead of foo.clone() (better visibilty)
* used CamelCase for the DeletableProperties
proxmox:
Dominik Csapak (3):
proxmox-sys: make some structs serializable
proxmox-sys: add FileSystemInformation struct and helper
proxmox-metrics: implement metrics server client code
Cargo.toml | 1 +
proxmox-metrics/Cargo.toml | 20 +++++
proxmox-metrics/debian/changelog | 5 ++
proxmox-metrics/debian/copyright | 16 ++++
proxmox-metrics/debian/debcargo.toml | 7 ++
proxmox-metrics/src/influxdb/http.rs | 122 ++++++++++++++++++++++++++
proxmox-metrics/src/influxdb/mod.rs | 7 ++
proxmox-metrics/src/influxdb/udp.rs | 94 ++++++++++++++++++++
proxmox-metrics/src/influxdb/utils.rs | 50 +++++++++++
proxmox-metrics/src/lib.rs | 89 +++++++++++++++++++
proxmox-sys/Cargo.toml | 1 +
proxmox-sys/src/fs/mod.rs | 39 ++++++++
proxmox-sys/src/linux/procfs/mod.rs | 7 +-
13 files changed, 455 insertions(+), 3 deletions(-)
create mode 100644 proxmox-metrics/Cargo.toml
create mode 100644 proxmox-metrics/debian/changelog
create mode 100644 proxmox-metrics/debian/copyright
create mode 100644 proxmox-metrics/debian/debcargo.toml
create mode 100644 proxmox-metrics/src/influxdb/http.rs
create mode 100644 proxmox-metrics/src/influxdb/mod.rs
create mode 100644 proxmox-metrics/src/influxdb/udp.rs
create mode 100644 proxmox-metrics/src/influxdb/utils.rs
create mode 100644 proxmox-metrics/src/lib.rs
proxmox-backup:
Dominik Csapak (6):
use 'fs_info' from proxmox-sys
pbs-api-types: add metrics api types
pbs-config: add metrics config class
backup-proxy: decouple stats gathering from rrd update
proxmox-backup-proxy: send metrics to configured metrics server
api: add metricserver endpoints
Cargo.toml | 1 +
pbs-api-types/src/lib.rs | 2 +
pbs-api-types/src/metrics.rs | 147 ++++++++
pbs-config/Cargo.toml | 1 +
pbs-config/src/lib.rs | 1 +
pbs-config/src/metrics.rs | 128 +++++++
src/api2/admin/datastore.rs | 4 +-
src/api2/config/metricserver/influxdbhttp.rs | 275 +++++++++++++++
src/api2/config/metricserver/influxdbudp.rs | 245 +++++++++++++
src/api2/config/metricserver/mod.rs | 16 +
src/api2/config/mod.rs | 2 +
src/api2/node/status.rs | 10 +-
src/api2/status.rs | 4 +-
src/bin/proxmox-backup-proxy.rs | 346 +++++++++++++++----
src/tools/disks/mod.rs | 21 +-
15 files changed, 1105 insertions(+), 98 deletions(-)
create mode 100644 pbs-api-types/src/metrics.rs
create mode 100644 pbs-config/src/metrics.rs
create mode 100644 src/api2/config/metricserver/influxdbhttp.rs
create mode 100644 src/api2/config/metricserver/influxdbudp.rs
create mode 100644 src/api2/config/metricserver/mod.rs
--
2.30.2
next reply other threads:[~2021-12-17 8:10 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-17 8:09 Dominik Csapak [this message]
2021-12-17 8:09 ` [pbs-devel] [PATCH proxmox v3 1/3] proxmox-sys: make some structs serializable Dominik Csapak
2021-12-17 8:09 ` [pbs-devel] [PATCH proxmox v3 2/3] proxmox-sys: add FileSystemInformation struct and helper Dominik Csapak
2022-01-12 14:10 ` Wolfgang Bumiller
2021-12-17 8:09 ` [pbs-devel] [PATCH proxmox v3 3/3] proxmox-metrics: implement metrics server client code Dominik Csapak
2022-01-12 14:36 ` Wolfgang Bumiller
2021-12-17 8:09 ` [pbs-devel] [PATCH proxmox-backup v3 1/6] use 'fs_info' from proxmox-sys Dominik Csapak
2021-12-17 8:09 ` [pbs-devel] [PATCH proxmox-backup v3 2/6] pbs-api-types: add metrics api types Dominik Csapak
2022-01-12 14:40 ` Wolfgang Bumiller
2021-12-17 8:09 ` [pbs-devel] [PATCH proxmox-backup v3 3/6] pbs-config: add metrics config class Dominik Csapak
2021-12-17 8:09 ` [pbs-devel] [PATCH proxmox-backup v3 4/6] backup-proxy: decouple stats gathering from rrd update Dominik Csapak
2021-12-17 8:09 ` [pbs-devel] [PATCH proxmox-backup v3 5/6] proxmox-backup-proxy: send metrics to configured metrics server Dominik Csapak
2021-12-17 8:10 ` [pbs-devel] [PATCH proxmox-backup v3 6/6] api: add metricserver endpoints Dominik Csapak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211217081000.1061796-1-d.csapak@proxmox.com \
--to=d.csapak@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox