public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pbs-devel] [PATCH proxmox/proxmox-backup v4] add metrics server capability
@ 2022-01-17 10:48 Dominik Csapak
  2022-01-17 10:48 ` [pbs-devel] [PATCH proxmox v4 1/4] proxmox-sys: make some structs serializable Dominik Csapak
                   ` (10 more replies)
  0 siblings, 11 replies; 23+ messages in thread
From: Dominik Csapak @ 2022-01-17 10:48 UTC (permalink / raw)
  To: pbs-devel

this series adds support for exporting metrics data to external
metric servers.

for now this includes only data we gather for RRD, though it should
not be hard to extend that functionality

also only influxdb (udp/http(s)) is currently supported, but it should
also not be too hard to include more options here

i did not include gui/cli patches yet, as i find the
proxmox-backup-manager options are already too much and i waited for
the gui for some feedback.

for testing, the metric servers can be added either by
calling 'proxmox-backup debug api ...' or by manually editing the
file

ofc, proxmox-backup depends on bumped versions of the proxmox-* crates

changes from v3:
* rebase on master
* introduced helper functions instead of InfluxDBHttp::new
* start tokio task directly in the helper
* combine channel close + join
* fix api description
* combine host/port/protocol in the api types
* introduce a connect_to_udp helper
* use NixPath in the fs_info helper

changes from v2:
* rebase on master
* rustfmt
* clippy (fixed not everything)
* renamed DiskUsage in proxmox-sys and added some more fields
* added 'enable' property for the config (like we have in pve)
* subtracted 50bytes from mtu in the udp variant (for ip header)

changes from v1:
* fixed ipv6 support for udp (tested it this time ;) )
* dropped the 'flush' functionality of the MetricsChannel, but kept the
  wrapper struct: it did not do what i intended, and after rethinking it,
  turns out it's not necessary (as we autoflush when the data gets to large,
  or when we close the channel). kept the struct so that the interface
  can stay the same even if we want to implement a manual flush in the future
* improved the influxdb line formatter
* removed variables like 'names2' by reorganizing the code
* used Arc::clone(&foo) instead of foo.clone() (better visibilty)
* used CamelCase for the DeletableProperties

proxmox:

Dominik Csapak (4):
  proxmox-sys: make some structs serializable
  proxmox-sys: add FileSystemInformation struct and helper
  proxmox-async: add connect_to_udp helper
  proxmox-metrics: implement metrics server client code

 Cargo.toml                            |   1 +
 proxmox-async/Cargo.toml              |   2 +-
 proxmox-async/src/io/mod.rs           |   3 +
 proxmox-async/src/io/udp_connect.rs   |  18 ++++
 proxmox-metrics/Cargo.toml            |  21 ++++
 proxmox-metrics/debian/changelog      |   5 +
 proxmox-metrics/debian/copyright      |  16 ++++
 proxmox-metrics/debian/debcargo.toml  |   7 ++
 proxmox-metrics/src/influxdb/http.rs  | 132 ++++++++++++++++++++++++++
 proxmox-metrics/src/influxdb/mod.rs   |   7 ++
 proxmox-metrics/src/influxdb/udp.rs   |  80 ++++++++++++++++
 proxmox-metrics/src/influxdb/utils.rs |  50 ++++++++++
 proxmox-metrics/src/lib.rs            | 117 +++++++++++++++++++++++
 proxmox-sys/Cargo.toml                |   1 +
 proxmox-sys/src/fs/mod.rs             |  37 ++++++++
 proxmox-sys/src/linux/procfs/mod.rs   |   7 +-
 16 files changed, 500 insertions(+), 4 deletions(-)
 create mode 100644 proxmox-async/src/io/udp_connect.rs
 create mode 100644 proxmox-metrics/Cargo.toml
 create mode 100644 proxmox-metrics/debian/changelog
 create mode 100644 proxmox-metrics/debian/copyright
 create mode 100644 proxmox-metrics/debian/debcargo.toml
 create mode 100644 proxmox-metrics/src/influxdb/http.rs
 create mode 100644 proxmox-metrics/src/influxdb/mod.rs
 create mode 100644 proxmox-metrics/src/influxdb/udp.rs
 create mode 100644 proxmox-metrics/src/influxdb/utils.rs
 create mode 100644 proxmox-metrics/src/lib.rs

proxmox-backup:

Dominik Csapak (6):
  use 'fs_info' from proxmox-sys
  pbs-api-types: add metrics api types
  pbs-config: add metrics config class
  backup-proxy: decouple stats gathering from rrd update
  proxmox-backup-proxy: send metrics to configured metrics server
  api: add metricserver endpoints

 Cargo.toml                                   |   1 +
 pbs-api-types/src/lib.rs                     |  15 +
 pbs-api-types/src/metrics.rs                 | 138 ++++++++
 pbs-config/Cargo.toml                        |   1 +
 pbs-config/src/lib.rs                        |   1 +
 pbs-config/src/metrics.rs                    | 115 +++++++
 src/api2/admin/datastore.rs                  |   4 +-
 src/api2/config/metricserver/influxdbhttp.rs | 266 +++++++++++++++
 src/api2/config/metricserver/influxdbudp.rs  | 243 +++++++++++++
 src/api2/config/metricserver/mod.rs          |  16 +
 src/api2/config/mod.rs                       |   2 +
 src/api2/node/status.rs                      |  11 +-
 src/api2/status.rs                           |   4 +-
 src/bin/proxmox-backup-proxy.rs              | 342 +++++++++++++++----
 src/tools/disks/mod.rs                       |  21 +-
 15 files changed, 1081 insertions(+), 99 deletions(-)
 create mode 100644 pbs-api-types/src/metrics.rs
 create mode 100644 pbs-config/src/metrics.rs
 create mode 100644 src/api2/config/metricserver/influxdbhttp.rs
 create mode 100644 src/api2/config/metricserver/influxdbudp.rs
 create mode 100644 src/api2/config/metricserver/mod.rs

-- 
2.30.2





^ permalink raw reply	[flat|nested] 23+ messages in thread

end of thread, other threads:[~2022-02-01 13:26 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-17 10:48 [pbs-devel] [PATCH proxmox/proxmox-backup v4] add metrics server capability Dominik Csapak
2022-01-17 10:48 ` [pbs-devel] [PATCH proxmox v4 1/4] proxmox-sys: make some structs serializable Dominik Csapak
2022-02-01 11:39   ` [pbs-devel] applied: " Thomas Lamprecht
2022-01-17 10:48 ` [pbs-devel] [PATCH proxmox v4 2/4] proxmox-sys: add FileSystemInformation struct and helper Dominik Csapak
2022-02-01 11:40   ` [pbs-devel] applied: " Thomas Lamprecht
2022-01-17 10:48 ` [pbs-devel] [PATCH proxmox v4 3/4] proxmox-async: add connect_to_udp helper Dominik Csapak
2022-02-01 12:02   ` Thomas Lamprecht
2022-02-01 12:13     ` Dominik Csapak
2022-02-01 12:39       ` Thomas Lamprecht
2022-02-01 13:17         ` Wolfgang Bumiller
2022-01-17 10:48 ` [pbs-devel] [PATCH proxmox v4 4/4] proxmox-metrics: implement metrics server client code Dominik Csapak
2022-01-17 10:48 ` [pbs-devel] [PATCH proxmox-backup v4 1/6] use 'fs_info' from proxmox-sys Dominik Csapak
2022-01-17 10:48 ` [pbs-devel] [PATCH proxmox-backup v4 2/6] pbs-api-types: add metrics api types Dominik Csapak
2022-02-01  9:55   ` Matthias Heiserer
2022-02-01 10:11     ` Dominik Csapak
2022-01-17 10:48 ` [pbs-devel] [PATCH proxmox-backup v4 3/6] pbs-config: add metrics config class Dominik Csapak
2022-01-17 10:48 ` [pbs-devel] [PATCH proxmox-backup v4 4/6] backup-proxy: decouple stats gathering from rrd update Dominik Csapak
2022-01-17 10:48 ` [pbs-devel] [PATCH proxmox-backup v4 5/6] proxmox-backup-proxy: send metrics to configured metrics server Dominik Csapak
2022-01-17 10:48 ` [pbs-devel] [PATCH proxmox-backup v4 6/6] api: add metricserver endpoints Dominik Csapak
2022-02-01 11:01 ` [pbs-devel] [PATCH proxmox/proxmox-backup v4] add metrics server capability Matthias Heiserer
2022-02-01 11:39   ` Thomas Lamprecht
2022-02-01 13:22     ` Matthias Heiserer
2022-02-01 13:26       ` Thomas Lamprecht

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal