From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id B9B9BBEB9 for ; Fri, 8 Apr 2022 11:56:13 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B7FCBD2D5 for ; Fri, 8 Apr 2022 11:56:13 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 040E6D210 for ; Fri, 8 Apr 2022 11:56:08 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id C992846D46 for ; Fri, 8 Apr 2022 11:56:07 +0200 (CEST) From: Dominik Csapak To: pbs-devel@lists.proxmox.com Date: Fri, 8 Apr 2022 11:55:59 +0200 Message-Id: <20220408095606.2767234-1-d.csapak@proxmox.com> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.140 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: [pbs-devel] [PATCH proxmox/proxmox-backup] implement streaming serialization for api calls X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Apr 2022 09:56:13 -0000 this series aims to improve memory behaviour for api calls where the result is large. We currently convert that result into a serde `Value` which, depending on the actual structure of the data, can use much more memory than the original rust structure (e.g. i saw factor 5 to 10 for the backup snapshot list) which in addition seems to trigger bad behaviour in the memory allocator (this again....). by streaming the serialization, we don't need any in memory copy of the result, and we could probably even return an Arc<> or Rc<> so we maybe don't have to copy the data at all (we would have to implement Serialize on that ourselfs, or enable the 'rc' feature for serde to use that) to overcome that serde serialization is not async, use tokio::spawn_blocking for each api call. this will spawn a thread if none is idle, and only up to a max 'blocking thread pool' maximum. dependencies (if it's not clear already): proxmox-async needs a bump, router and api-macro need to depend on that proxmox-router/api macro would then also need a bump on which proxmox-rest-server/proxmox-backup needs to depend changes from rfc: * use tokio::spawn_blocking instead of a new thread * dependencies from proxmox-async are greatly reduced since we split it up recently proxmox: Dominik Csapak (4): proxmox-async: add SenderWriter helper promxox-router: add SerializableReturn Trait proxmox-router: add new ApiHandler variants for streaming serialization proxmox-api-macro: add 'streaming' option proxmox-api-macro/src/api/method.rs | 127 ++++++++++++++------ proxmox-api-macro/tests/api1.rs | 16 +++ proxmox-async/src/blocking/mod.rs | 3 + proxmox-async/src/blocking/sender_writer.rs | 47 ++++++++ proxmox-router/Cargo.toml | 2 + proxmox-router/src/cli/command.rs | 45 +++++++ proxmox-router/src/lib.rs | 2 + proxmox-router/src/router.rs | 78 ++++++++++++ proxmox-router/src/serializable_return.rs | 62 ++++++++++ 9 files changed, 343 insertions(+), 39 deletions(-) create mode 100644 proxmox-async/src/blocking/sender_writer.rs create mode 100644 proxmox-router/src/serializable_return.rs proxmox-backup: Dominik Csapak (3): proxmox-rest-server: OutputFormatter: add new format_data_streaming method adapt to the new ApiHandler variants api: admin/datastore: enable streaming for some api calls proxmox-rest-server/Cargo.toml | 1 + proxmox-rest-server/src/formatter.rs | 67 +++++++++++++++++++++++++++- proxmox-rest-server/src/rest.rs | 13 ++++++ src/api2/admin/datastore.rs | 1 + src/api2/node/tasks.rs | 1 + src/bin/proxmox_backup_debug/api.rs | 8 ++++ 6 files changed, 90 insertions(+), 1 deletion(-) -- 2.30.2