[RFC proxmox{,-backup} 0/4] fix #6841: implement request rate limits for all methods

[Christian Ebner <c.ebner@proxmox.com>]

Currently request performed by the s3 client to the backend can only
be limited by bandwidth limits or on a per-instance based PUT rate
limit. Providers might however enforce limits on the number of requests
per second, non of which can currently be enforced.

Therefore this patch series implements request limits, using a memory
mapped shared request limiter, similar to what is used to enforce
bandwidth limits. Since providers might have different limits as e.g.
the soft limits by AWS S3 mentioned in [0], allow to set different limits
between `active` and `passive` requests methods.

Sending as RFC as a bit unsure if we should make this configurable
as currently the case or rather trying to dynamically adapt the request
rate in case of requests failing with the respective HTTP status codes.
The latter has the downside that different providers may not clearly
signal when a rate limit is being enforced and/or have different status
codes to do so. Therefore the current approach allows to hard code the
limits.

Currently the rate limit is not yet updated on config changes via API,
only on s3 client instantiation.

[1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html

Reference to the issue in bugzilla: https://bugzilla.proxmox.com/show_bug.cgi?id=6841


proxmox:

Christian Ebner (2):
  fix #6841: allow to set active/passive request rate limits
  s3-client: add 429 HTTP status code as retryable

 proxmox-s3-client/src/api_types.rs |  8 +++-
 proxmox-s3-client/src/client.rs    | 66 ++++++++++++++++++++++++++----
 2 files changed, 66 insertions(+), 8 deletions(-)


proxmox-backup:

Christian Ebner (2):
  s3: config: update or delete active/passive request limit options
  ui: expose request rate limit config options for S3 endpoints

 src/api2/config/s3.rs      | 16 ++++++++++++++++
 www/window/S3ClientEdit.js | 14 ++++++++++++++
 2 files changed, 30 insertions(+)


Summary over all repositories:
  4 files changed, 96 insertions(+), 8 deletions(-)

-- 
Generated by murpp 0.11.0