From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 555A11FF165 for ; Thu, 28 Aug 2025 12:26:28 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C32AFCC6D; Thu, 28 Aug 2025 12:26:35 +0200 (CEST) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Thu, 28 Aug 2025 12:25:58 +0200 Message-ID: <20250828102604.463662-1-c.ebner@proxmox.com> X-Mailer: git-send-email 2.47.2 MIME-Version: 1.0 X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1756376785336 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.042 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_VALIDITY_CERTIFIED_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_RPBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. RCVD_IN_VALIDITY_SAFE_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to Validity was blocked. See https://knowledge.validity.com/hc/en-us/articles/20961730681243 for more information. SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH proxmox{, -backup} 0/6] shared rate limiter for s3 client instances X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" This patches implement a shared bandwidth rate limiter for the s3 client instances, with the goal to allow users to avoid network congestion by traffic to the s3 backend. The limiter is shared accorss all the clients using the same s3 endpoint id, setting the limits on client instantiation. To utilize the pre-existing shared rate limiter implementation from PBS, factor it out into proxmox-http as this already contains the non-shared rate limiter implementation. Expose the rate limit configuration in the s3 endpoint configuration and provide it to the client's https connector. Expose the settings also in the advanced options of the s3 endpoint edit/crate window. proxmox: Christian Ebner (2): http: factor out PBS shared rate limiter implementation s3-client: add shared rate limiter via https connector proxmox-http/Cargo.toml | 7 ++ proxmox-http/debian/control | 18 ++++ proxmox-http/src/lib.rs | 5 + proxmox-http/src/shared_rate_limiter.rs | 129 ++++++++++++++++++++++++ proxmox-s3-client/Cargo.toml | 5 +- proxmox-s3-client/debian/control | 4 + proxmox-s3-client/examples/s3_client.rs | 1 + proxmox-s3-client/src/api_types.rs | 29 ++++++ proxmox-s3-client/src/client.rs | 62 +++++++++++- proxmox-s3-client/src/lib.rs | 4 +- 10 files changed, 259 insertions(+), 5 deletions(-) create mode 100644 proxmox-http/src/shared_rate_limiter.rs proxmox-backup: Christian Ebner (4): traffic control: use factored out shared rate limiter api: config: update s3 endpoint rate limits in config datastore: s3: set rate limiter options for s3 client ui: expose rate and burst limits for s3 endpoints Cargo.toml | 2 +- pbs-datastore/src/datastore.rs | 18 ++++- src/api2/admin/s3.rs | 9 ++- src/api2/config/s3.rs | 34 ++++++++- src/tools/mod.rs | 4 - src/tools/shared_rate_limiter.rs | 122 ------------------------------- src/traffic_control_cache.rs | 8 +- www/window/S3ClientEdit.js | 34 +++++++++ 8 files changed, 96 insertions(+), 135 deletions(-) delete mode 100644 src/tools/shared_rate_limiter.rs Summary over all repositories: 18 files changed, 355 insertions(+), 140 deletions(-) -- Generated by git-murpp 0.8.1 _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel