[RFC PATCH proxmox-backup 0/2] fix #7756: avoid qmp 'backup' timeout

[Christian Ebner <c.ebner@proxmox.com>]

During high load due to parallel backups or slow storage backend on
the PBS, tokio blocking thread pool can become an issue. This is
because fetching of the previous snapshot manifest and index files
may be blocked in the blocking queue for a long time, leading to
exceeded QMP command timeout, especially notable for snapshots with
large index files.

These patches optimize the async reader for content fetching and
draft a possible workaround by fully offloading the file download
handling to a dedicated thread, not offloading it to tokio's blocking
thread pool.

Open question is whether it would make sense to distinguish between
API endpoints with higher priority and ones with lower priority in
general, and offload them to high priority and low priority thread
pools instead of spawning new threads on demand.

Issue was raised again recently in the enterprise support and several
times in the community forum [0-2].

[0] https://forum.proxmox.com/threads/179304/
[1] https://forum.proxmox.com/threads/183380/
[2] https://forum.proxmox.com/threads/185679/

Christian Ebner (2):
  api: optimize content streaming reader for file download
  fix #7756: api: stream file contents via dedicated thread

 src/api2/helpers.rs | 169 ++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 155 insertions(+), 14 deletions(-)

-- 
2.47.3