From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 95D3F7D5D2 for ; Tue, 9 Nov 2021 07:53:40 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id C884B2E159 for ; Tue, 9 Nov 2021 07:53:09 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id A920E2DF71 for ; Tue, 9 Nov 2021 07:53:01 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id DE896427CE; Tue, 9 Nov 2021 07:52:56 +0100 (CET) From: Dietmar Maurer To: pbs-devel@lists.proxmox.com Date: Tue, 9 Nov 2021 07:52:37 +0100 Message-Id: <20211109065253.980304-1-dietmar@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.506 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [rest.rs, proxmox-backup-proxy.rs, mod.rs, memcom.rs, main.rs, time.rs, connector.rs, lib.rs, proxmox-backup-manager.rs] Subject: [pbs-devel] [PATCH proxmox/proxmox-backup] Rate Limiter Implementation X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Nov 2021 06:53:40 -0000 This implements a simple Token Bucket based rate limiter. That filter can be used to: - limit http client connection - limit speed at server side (proxmox-backup-proxy) There is also a new configuration file to configure the server side rate limits: /etc/proxmox-backup/traffic-control.cfg The server side filter dynamically updates when the user changes configuration (even for existing connections). Dietmar Maurer (proxmox/7): Implement a rate limiting stream (AsyncRead, AsyncWrite) RateLimitedStream: implement poll_write_vectored HttpsConnector: use RateLimitedStream RateLimitedStream: allow periodic limiter updates RateLimiter: avoid panic in time computations RateLimitedStream: implement peer_addr RateLimiter: add update_rate method proxmox-http/src/client/connector.rs | 51 +++- proxmox-http/src/client/mod.rs | 6 + .../src/client/rate_limited_stream.rs | 233 ++++++++++++++++++ proxmox-http/src/client/rate_limiter.rs | 84 +++++++ 4 files changed, 366 insertions(+), 8 deletions(-) create mode 100644 proxmox-http/src/client/rate_limited_stream.rs create mode 100644 proxmox-http/src/client/rate_limiter.rs Dietmar Maurer (proxmox-backup/9): pbs-client: add option to use the new RateLimiter proxmox-backup-client: add rate/burst parameter to backup CLI implement Servive for RateLimitedStream New DailyDuration type with nom parser DailyDuration: implement time_match() Add traffic control configuration config with API traffic_control: use Memcom to track. config versions implement a traffic control cache for fast rate control limiter lockups proxmox-backup-proxy: implement traffic control Cargo.toml | 1 + pbs-api-types/src/lib.rs | 7 + pbs-api-types/src/traffic_control.rs | 81 +++++ pbs-client/src/http_client.rs | 24 +- pbs-client/src/tools/mod.rs | 23 +- pbs-config/src/lib.rs | 3 +- pbs-config/src/memcom.rs | 14 + pbs-config/src/traffic_control.rs | 98 ++++++ proxmox-backup-client/src/main.rs | 19 +- proxmox-rest-server/Cargo.toml | 1 + proxmox-rest-server/src/rest.rs | 28 ++ proxmox-systemd/src/daily_duration.rs | 152 ++++++++++ proxmox-systemd/src/lib.rs | 1 + proxmox-systemd/src/parse_time.rs | 56 ++++ proxmox-systemd/src/time.rs | 2 +- src/api2/config/mod.rs | 2 + src/api2/config/traffic_control.rs | 283 ++++++++++++++++++ src/bin/proxmox-backup-manager.rs | 1 + src/bin/proxmox-backup-proxy.rs | 25 +- src/bin/proxmox_backup_manager/mod.rs | 2 + .../proxmox_backup_manager/traffic_control.rs | 105 +++++++ src/cached_traffic_control.rs | 240 +++++++++++++++ src/lib.rs | 3 + 23 files changed, 1161 insertions(+), 10 deletions(-) create mode 100644 pbs-api-types/src/traffic_control.rs create mode 100644 pbs-config/src/traffic_control.rs create mode 100644 proxmox-systemd/src/daily_duration.rs create mode 100644 src/api2/config/traffic_control.rs create mode 100644 src/bin/proxmox_backup_manager/traffic_control.rs create mode 100644 src/cached_traffic_control.rs -- 2.30.2