[pbs-devel] [PATCH proxmox{, -backup} v3 0/9] shared rate limiter for s3 client instances

[Christian Ebner <c.ebner@proxmox.com>]

This patches implement a shared bandwidth rate limiter for the s3
client instances, with the goal to allow users to avoid network
congestion by traffic to the s3 backend.

The limiter is shared accorss all the clients using the same s3
endpoint id, setting the limits on client instantiation. To utilize
the pre-existing shared rate limiter implementation from PBS, factor
it out into a dedicated crate, together with the non-shared rate limiter
implementation from proxmox-http.

Expose the rate limit configuration in the s3 endpoint configuration
and provide it to the client's https connector. Expose the settings
also in the advanced options of the s3 endpoint edit/crate window.

Changes since version 2 (thanks @Hannes for review and testing):
- Fix possible division by zero if rate is set to 0, consider as unlimited
  instead
- Rebased onto current master

Changes since version 1 (thanks @Thomas for feedback):
- Move shared/non-shared rate limiter into dedicated crate
- Use a more generic name for the magic number constant of the mmapped
  state file

proxmox:

Christian Ebner (5):
  rate-limiter: add crate for traffic rate limiter implementations
  rate-limiter: avoid division by zero in traffic updater
  http: drop factored out rate limiter implementation
  rest-server: optionally depend on factored out shared rate limiter
  s3-client: add shared rate limiter via https connector

 Cargo.toml                                    |   2 +
 proxmox-http/Cargo.toml                       |   4 +-
 proxmox-http/debian/control                   |  19 +--
 proxmox-http/src/client/connector.rs          |   3 +-
 proxmox-http/src/lib.rs                       |   5 -
 proxmox-http/src/rate_limited_stream.rs       |   2 +-
 proxmox-rate-limiter/Cargo.toml               |  29 ++++
 proxmox-rate-limiter/debian/changelog         |   5 +
 proxmox-rate-limiter/debian/control           |  69 ++++++++++
 proxmox-rate-limiter/debian/copyright         |  18 +++
 proxmox-rate-limiter/debian/debcargo.toml     |   7 +
 proxmox-rate-limiter/src/lib.rs               |  13 ++
 .../src/rate_limiter.rs                       |   3 +
 .../src/shared_rate_limiter.rs                | 130 ++++++++++++++++++
 proxmox-rest-server/Cargo.toml                |   2 +
 proxmox-rest-server/debian/control            |   4 +-
 proxmox-rest-server/src/connection.rs         |   4 +-
 proxmox-s3-client/Cargo.toml                  |   7 +-
 proxmox-s3-client/debian/control              |   7 +-
 proxmox-s3-client/examples/s3_client.rs       |   1 +
 proxmox-s3-client/src/api_types.rs            |  29 ++++
 proxmox-s3-client/src/client.rs               |  63 ++++++++-
 proxmox-s3-client/src/lib.rs                  |   4 +-
 23 files changed, 396 insertions(+), 34 deletions(-)
 create mode 100644 proxmox-rate-limiter/Cargo.toml
 create mode 100644 proxmox-rate-limiter/debian/changelog
 create mode 100644 proxmox-rate-limiter/debian/control
 create mode 100644 proxmox-rate-limiter/debian/copyright
 create mode 100644 proxmox-rate-limiter/debian/debcargo.toml
 create mode 100644 proxmox-rate-limiter/src/lib.rs
 rename {proxmox-http => proxmox-rate-limiter}/src/rate_limiter.rs (98%)
 create mode 100644 proxmox-rate-limiter/src/shared_rate_limiter.rs


proxmox-backup:

Christian Ebner (4):
  traffic control: use factored out shared rate limiter
  api: config: update s3 endpoint rate limits in config
  datastore: s3: set rate limiter options for s3 client
  ui: expose rate and burst limits for s3 endpoints

 Cargo.toml                       |   3 +
 debian/control                   |   1 -
 pbs-client/Cargo.toml            |   3 +-
 pbs-client/src/http_client.rs    |   3 +-
 pbs-datastore/src/datastore.rs   |  18 ++++-
 src/api2/admin/s3.rs             |   9 ++-
 src/api2/config/s3.rs            |  34 ++++++++-
 src/tools/mod.rs                 |   4 -
 src/tools/shared_rate_limiter.rs | 122 -------------------------------
 src/traffic_control_cache.rs     |   8 +-
 www/window/S3ClientEdit.js       |  34 +++++++++
 11 files changed, 102 insertions(+), 137 deletions(-)
 delete mode 100644 src/tools/shared_rate_limiter.rs


Summary over all repositories:
  34 files changed, 498 insertions(+), 171 deletions(-)

-- 
Generated by git-murpp 0.8.1


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel