public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* [pbs-devel] [RFC PATCH proxmox/proxmox-backup] implement streaming serialization for api calls
@ 2022-02-17  9:40 Dominik Csapak
  2022-02-17  9:40 ` [pbs-devel] [RFC PATCH proxmox 1/4] proxmox-async: add SenderWriter helper Dominik Csapak
                   ` (6 more replies)
  0 siblings, 7 replies; 10+ messages in thread
From: Dominik Csapak @ 2022-02-17  9:40 UTC (permalink / raw)
  To: pbs-devel

this series aims to improve memory behaviour for api calls where the
result is large. We currently convert that result into a serde `Value`
which, depending on the actual structure of the data, can use much more
memory than the original rust structure (e.g. i saw factor 5 to 10 for
the backup snapshot list) which in addition seems to trigger bad
behaviour in the memory allocator (this again....).

by streaming the serialization, we don't need any in memory copy of the
result, and we could probably even return an Arc<> or Rc<> so we maybe
don't have to copy the data at all (we would have to implement Serialize
on that ourselfs, or enable the 'rc' feature for serde to use that)

the only downside to that is that we need a thread per api call in which
we serialize into our tokio mpsc channel, since the serde serialization
has no async functionality.

sending as rfc because:
* not sure about the naming of things
* could probably use improved tests
* not really documented (i looked, but found no obvious place
  where to add that to the api macro docs?)

proxmox:

Dominik Csapak (4):
  proxmox-async: add SenderWriter helper
  promxox-router: add SerializableReturn Trait
  proxmox-router: add new ApiHandler variants for streaming
    serialization
  proxmox-api-macro: add 'streaming' option

 proxmox-api-macro/src/api/method.rs         | 127 ++++++++++++++------
 proxmox-api-macro/tests/api1.rs             |  16 +++
 proxmox-async/src/blocking/mod.rs           |   3 +
 proxmox-async/src/blocking/sender_writer.rs |  47 ++++++++
 proxmox-router/Cargo.toml                   |   2 +
 proxmox-router/src/cli/command.rs           |  45 +++++++
 proxmox-router/src/lib.rs                   |   2 +
 proxmox-router/src/router.rs                |  78 ++++++++++++
 proxmox-router/src/serializable_return.rs   |  62 ++++++++++
 9 files changed, 343 insertions(+), 39 deletions(-)
 create mode 100644 proxmox-async/src/blocking/sender_writer.rs
 create mode 100644 proxmox-router/src/serializable_return.rs

proxmox-backup:

Dominik Csapak (3):
  proxmox-rest-server: OutputFormatter: add new format_data_streaming
    method
  adapt to the new ApiHandler variants
  api: admin/datastore: enable streaming for some api calls

 proxmox-rest-server/Cargo.toml       |  1 +
 proxmox-rest-server/src/formatter.rs | 52 +++++++++++++++++++++++++++-
 proxmox-rest-server/src/rest.rs      | 10 ++++++
 src/api2/admin/datastore.rs          |  1 +
 src/api2/node/tasks.rs               |  1 +
 src/bin/proxmox_backup_debug/api.rs  |  8 +++++
 6 files changed, 72 insertions(+), 1 deletion(-)

-- 
2.30.2





^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2022-02-23  9:21 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-17  9:40 [pbs-devel] [RFC PATCH proxmox/proxmox-backup] implement streaming serialization for api calls Dominik Csapak
2022-02-17  9:40 ` [pbs-devel] [RFC PATCH proxmox 1/4] proxmox-async: add SenderWriter helper Dominik Csapak
2022-02-17  9:40 ` [pbs-devel] [RFC PATCH proxmox 2/4] promxox-router: add SerializableReturn Trait Dominik Csapak
2022-02-18  8:21   ` Dietmar Maurer
2022-02-17  9:40 ` [pbs-devel] [RFC PATCH proxmox 3/4] proxmox-router: add new ApiHandler variants for streaming serialization Dominik Csapak
2022-02-17  9:40 ` [pbs-devel] [RFC PATCH proxmox 4/4] proxmox-api-macro: add 'streaming' option Dominik Csapak
2022-02-17  9:40 ` [pbs-devel] [RFC PATCH proxmox-backup 1/3] proxmox-rest-server: OutputFormatter: add new format_data_streaming method Dominik Csapak
2022-02-23  9:20   ` Thomas Lamprecht
2022-02-17  9:40 ` [pbs-devel] [RFC PATCH proxmox-backup 2/3] adapt to the new ApiHandler variants Dominik Csapak
2022-02-17  9:40 ` [pbs-devel] [RFC PATCH proxmox-backup 3/3] api: admin/datastore: enable streaming for some api calls Dominik Csapak

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal