From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <d.csapak@proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 3F50260B21
for <pbs-devel@lists.proxmox.com>; Thu, 17 Feb 2022 10:40:44 +0100 (CET)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 3B4611D59D
for <pbs-devel@lists.proxmox.com>; Thu, 17 Feb 2022 10:40:44 +0100 (CET)
Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com
[94.136.29.106])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS id A2AF01D567
for <pbs-devel@lists.proxmox.com>; Thu, 17 Feb 2022 10:40:42 +0100 (CET)
Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1])
by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 7DB0746DCA
for <pbs-devel@lists.proxmox.com>; Thu, 17 Feb 2022 10:40:42 +0100 (CET)
From: Dominik Csapak <d.csapak@proxmox.com>
To: pbs-devel@lists.proxmox.com
Date: Thu, 17 Feb 2022 10:40:34 +0100
Message-Id: <20220217094041.1632033-1-d.csapak@proxmox.com>
X-Mailer: git-send-email 2.30.2
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-SPAM-LEVEL: Spam detection results: 0
AWL 0.158 Adjusted score from AWL reputation of From: address
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
T_SCC_BODY_TEXT_LINE -0.01 -
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more
information. [formatter.rs, tasks.rs, lib.rs, rest.rs, api1.rs, mod.rs,
method.rs, router.rs, command.rs, api.rs, datastore.rs]
Subject: [pbs-devel] [RFC PATCH proxmox/proxmox-backup] implement streaming
serialization for api calls
X-BeenThere: pbs-devel@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox Backup Server development discussion
<pbs-devel.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pbs-devel/>
List-Post: <mailto:pbs-devel@lists.proxmox.com>
List-Help: <mailto:pbs-devel-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel>,
<mailto:pbs-devel-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Thu, 17 Feb 2022 09:40:44 -0000
this series aims to improve memory behaviour for api calls where the
result is large. We currently convert that result into a serde `Value`
which, depending on the actual structure of the data, can use much more
memory than the original rust structure (e.g. i saw factor 5 to 10 for
the backup snapshot list) which in addition seems to trigger bad
behaviour in the memory allocator (this again....).
by streaming the serialization, we don't need any in memory copy of the
result, and we could probably even return an Arc<> or Rc<> so we maybe
don't have to copy the data at all (we would have to implement Serialize
on that ourselfs, or enable the 'rc' feature for serde to use that)
the only downside to that is that we need a thread per api call in which
we serialize into our tokio mpsc channel, since the serde serialization
has no async functionality.
sending as rfc because:
* not sure about the naming of things
* could probably use improved tests
* not really documented (i looked, but found no obvious place
where to add that to the api macro docs?)
proxmox:
Dominik Csapak (4):
proxmox-async: add SenderWriter helper
promxox-router: add SerializableReturn Trait
proxmox-router: add new ApiHandler variants for streaming
serialization
proxmox-api-macro: add 'streaming' option
proxmox-api-macro/src/api/method.rs | 127 ++++++++++++++------
proxmox-api-macro/tests/api1.rs | 16 +++
proxmox-async/src/blocking/mod.rs | 3 +
proxmox-async/src/blocking/sender_writer.rs | 47 ++++++++
proxmox-router/Cargo.toml | 2 +
proxmox-router/src/cli/command.rs | 45 +++++++
proxmox-router/src/lib.rs | 2 +
proxmox-router/src/router.rs | 78 ++++++++++++
proxmox-router/src/serializable_return.rs | 62 ++++++++++
9 files changed, 343 insertions(+), 39 deletions(-)
create mode 100644 proxmox-async/src/blocking/sender_writer.rs
create mode 100644 proxmox-router/src/serializable_return.rs
proxmox-backup:
Dominik Csapak (3):
proxmox-rest-server: OutputFormatter: add new format_data_streaming
method
adapt to the new ApiHandler variants
api: admin/datastore: enable streaming for some api calls
proxmox-rest-server/Cargo.toml | 1 +
proxmox-rest-server/src/formatter.rs | 52 +++++++++++++++++++++++++++-
proxmox-rest-server/src/rest.rs | 10 ++++++
src/api2/admin/datastore.rs | 1 +
src/api2/node/tasks.rs | 1 +
src/bin/proxmox_backup_debug/api.rs | 8 +++++
6 files changed, 72 insertions(+), 1 deletion(-)
--
2.30.2