From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id A17411FF17A for ; Tue, 11 Nov 2025 11:49:12 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 4C7C5629A; Tue, 11 Nov 2025 11:49:58 +0100 (CET) Message-ID: <1734ecfa-8386-4240-9af5-ecb2d136fd10@proxmox.com> Date: Tue, 11 Nov 2025 11:49:54 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Proxmox Backup Server development discussion , Christian Ebner References: <20250916124147.513342-1-c.ebner@proxmox.com> Content-Language: en-US From: Hannes Laimer In-Reply-To: <20250916124147.513342-1-c.ebner@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1762858171159 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.046 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] [PATCH proxmox{, -backup} v2 0/8] shared rate limiter for s3 client instances X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Proxmox Backup Server development discussion Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Errors-To: pbs-devel-bounces@lists.proxmox.com Sender: "pbs-devel" I did run into some conflicts when applying, mostly control files, other than that and a small div by zero problem this L(very)GTM! I did test this with a local S3 storage, worked as advertised. Reviewed-by: Hannes Laimer Tested-by: Hannes Laimer On 9/16/25 14:42, Christian Ebner wrote: > This patches implement a shared bandwidth rate limiter for the s3 > client instances, with the goal to allow users to avoid network > congestion by traffic to the s3 backend. > > The limiter is shared accorss all the clients using the same s3 > endpoint id, setting the limits on client instantiation. To utilize > the pre-existing shared rate limiter implementation from PBS, factor > it out into a dedicated crate, together with the non-shared rate limiter > implementation from proxmox-http. > > Expose the rate limit configuration in the s3 endpoint configuration > and provide it to the client's https connector. Expose the settings > also in the advanced options of the s3 endpoint edit/crate window. > > Changes since version 1 (thanks @Thomas for feedback): > - Move shared/non-shared rate limiter into dedicated crate > - Use a more generic name for the magic number constant of the mmapped > state file > > proxmox: > > Christian Ebner (4): > rate-limiter: add crate for traffic rate limiter implementations > http: drop factored out rate limiter implementation > rest-server: optionally depend on factored out shared rate limiter > s3-client: add shared rate limiter via https connector > > Cargo.toml | 2 + > proxmox-http/Cargo.toml | 4 +- > proxmox-http/debian/control | 19 +-- > proxmox-http/src/client/connector.rs | 3 +- > proxmox-http/src/lib.rs | 5 - > proxmox-http/src/rate_limited_stream.rs | 2 +- > proxmox-rate-limiter/Cargo.toml | 29 ++++ > proxmox-rate-limiter/debian/changelog | 5 + > proxmox-rate-limiter/debian/control | 70 ++++++++++ > proxmox-rate-limiter/debian/copyright | 18 +++ > proxmox-rate-limiter/debian/debcargo.toml | 7 + > proxmox-rate-limiter/src/lib.rs | 13 ++ > .../src/rate_limiter.rs | 0 > .../src/shared_rate_limiter.rs | 130 ++++++++++++++++++ > proxmox-rest-server/Cargo.toml | 2 + > proxmox-rest-server/debian/control | 14 +- > proxmox-rest-server/src/connection.rs | 4 +- > proxmox-s3-client/Cargo.toml | 7 +- > proxmox-s3-client/debian/control | 7 +- > proxmox-s3-client/examples/s3_client.rs | 1 + > proxmox-s3-client/src/api_types.rs | 29 ++++ > proxmox-s3-client/src/client.rs | 63 ++++++++- > proxmox-s3-client/src/lib.rs | 4 +- > 23 files changed, 399 insertions(+), 39 deletions(-) > create mode 100644 proxmox-rate-limiter/Cargo.toml > create mode 100644 proxmox-rate-limiter/debian/changelog > create mode 100644 proxmox-rate-limiter/debian/control > create mode 100644 proxmox-rate-limiter/debian/copyright > create mode 100644 proxmox-rate-limiter/debian/debcargo.toml > create mode 100644 proxmox-rate-limiter/src/lib.rs > rename {proxmox-http => proxmox-rate-limiter}/src/rate_limiter.rs (100%) > create mode 100644 proxmox-rate-limiter/src/shared_rate_limiter.rs > > > proxmox-backup: > > Christian Ebner (4): > traffic control: use factored out shared rate limiter > api: config: update s3 endpoint rate limits in config > datastore: s3: set rate limiter options for s3 client > ui: expose rate and burst limits for s3 endpoints > > Cargo.toml | 3 + > debian/control | 1 - > pbs-client/Cargo.toml | 3 +- > pbs-client/src/http_client.rs | 3 +- > pbs-datastore/src/datastore.rs | 18 ++++- > src/api2/admin/s3.rs | 9 ++- > src/api2/config/s3.rs | 34 ++++++++- > src/tools/mod.rs | 4 - > src/tools/shared_rate_limiter.rs | 122 ------------------------------- > src/traffic_control_cache.rs | 8 +- > www/window/S3ClientEdit.js | 34 +++++++++ > 11 files changed, 102 insertions(+), 137 deletions(-) > delete mode 100644 src/tools/shared_rate_limiter.rs > > > Summary over all repositories: > 34 files changed, 501 insertions(+), 176 deletions(-) > _______________________________________________ pbs-devel mailing list pbs-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel