From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 969F91FF16F for ; Tue, 16 Sep 2025 14:42:31 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 496FC14230; Tue, 16 Sep 2025 14:42:45 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Tue, 16 Sep 2025 14:41:39 +0200 Message-ID: <20250916124147.513342-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.3 MIME-Version: 1.0 X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1758026524088 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.043 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH proxmox{, -backup} v2 0/8] shared rate limiter for s3 client instances X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" This patches implement a shared bandwidth rate limiter for the s3 client instances, with the goal to allow users to avoid network congestion by traffic to the s3 backend. The limiter is shared accorss all the clients using the same s3 endpoint id, setting the limits on client instantiation. To utilize the pre-existing shared rate limiter implementation from PBS, factor it out into a dedicated crate, together with the non-shared rate limiter implementation from proxmox-http. Expose the rate limit configuration in the s3 endpoint configuration and provide it to the client's https connector. Expose the settings also in the advanced options of the s3 endpoint edit/crate window. Changes since version 1 (thanks @Thomas for feedback): - Move shared/non-shared rate limiter into dedicated crate - Use a more generic name for the magic number constant of the mmapped state file proxmox: Christian Ebner (4): rate-limiter: add crate for traffic rate limiter implementations http: drop factored out rate limiter implementation rest-server: optionally depend on factored out shared rate limiter s3-client: add shared rate limiter via https connector Cargo.toml | 2 + proxmox-http/Cargo.toml | 4 +- proxmox-http/debian/control | 19 +-- proxmox-http/src/client/connector.rs | 3 +- proxmox-http/src/lib.rs | 5 - proxmox-http/src/rate_limited_stream.rs | 2 +- proxmox-rate-limiter/Cargo.toml | 29 ++++ proxmox-rate-limiter/debian/changelog | 5 + proxmox-rate-limiter/debian/control | 70 ++++++++++ proxmox-rate-limiter/debian/copyright | 18 +++ proxmox-rate-limiter/debian/debcargo.toml | 7 + proxmox-rate-limiter/src/lib.rs | 13 ++ .../src/rate_limiter.rs | 0 .../src/shared_rate_limiter.rs | 130 ++++++++++++++++++ proxmox-rest-server/Cargo.toml | 2 + proxmox-rest-server/debian/control | 14 +- proxmox-rest-server/src/connection.rs | 4 +- proxmox-s3-client/Cargo.toml | 7 +- proxmox-s3-client/debian/control | 7 +- proxmox-s3-client/examples/s3_client.rs | 1 + proxmox-s3-client/src/api_types.rs | 29 ++++ proxmox-s3-client/src/client.rs | 63 ++++++++- proxmox-s3-client/src/lib.rs | 4 +- 23 files changed, 399 insertions(+), 39 deletions(-) create mode 100644 proxmox-rate-limiter/Cargo.toml create mode 100644 proxmox-rate-limiter/debian/changelog create mode 100644 proxmox-rate-limiter/debian/control create mode 100644 proxmox-rate-limiter/debian/copyright create mode 100644 proxmox-rate-limiter/debian/debcargo.toml create mode 100644 proxmox-rate-limiter/src/lib.rs rename {proxmox-http => proxmox-rate-limiter}/src/rate_limiter.rs (100%) create mode 100644 proxmox-rate-limiter/src/shared_rate_limiter.rs proxmox-backup: Christian Ebner (4): traffic control: use factored out shared rate limiter api: config: update s3 endpoint rate limits in config datastore: s3: set rate limiter options for s3 client ui: expose rate and burst limits for s3 endpoints Cargo.toml | 3 + debian/control | 1 - pbs-client/Cargo.toml | 3 +- pbs-client/src/http_client.rs | 3 +- pbs-datastore/src/datastore.rs | 18 ++++- src/api2/admin/s3.rs | 9 ++- src/api2/config/s3.rs | 34 ++++++++- src/tools/mod.rs | 4 - src/tools/shared_rate_limiter.rs | 122 ------------------------------- src/traffic_control_cache.rs | 8 +- www/window/S3ClientEdit.js | 34 +++++++++ 11 files changed, 102 insertions(+), 137 deletions(-) delete mode 100644 src/tools/shared_rate_limiter.rs Summary over all repositories: 34 files changed, 501 insertions(+), 176 deletions(-) -- Generated by git-murpp 0.8.1 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel