[PATCH v1 proxmox-backup] api: do not block tokio worker threads during chunk inserts

[Robert Obkircher <r.obkircher@proxmox.com>]

Move synchronous operations off the worker threads to prevent blocking
the I/O and timer drivers of the entire runtime. This is especially
important for S3 uploads, which wait up to 3 hours for the chunk lock.

Also prevent worker starvation, which could happen because S3 uploads
are wrapped in proxmox_async::runtime::block_on, which prevents other
futures from running in the current thread.

In the backtrace from the linked forum post, two workers were waiting
for chunk locks (presumably due to duplicates) while the remaining 19
were stuck because block_on called std::thread::park.

Fixes: https://forum.proxmox.com/threads/183705
Signed-off-by: Robert Obkircher <r.obkircher@proxmox.com>
---
 src/api2/backup/upload_chunk.rs | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/src/api2/backup/upload_chunk.rs b/src/api2/backup/upload_chunk.rs
index 59e4caee2..eec24add2 100644
--- a/src/api2/backup/upload_chunk.rs
+++ b/src/api2/backup/upload_chunk.rs
@@ -1,4 +1,5 @@
 use std::pin::Pin;
+use std::sync::Arc;
 use std::task::{Context, Poll};
 
 use anyhow::{Error, bail, format_err};
@@ -8,6 +9,7 @@ use http_body_util::{BodyDataStream, BodyExt};
 use hyper::body::Incoming;
 use hyper::http::request::Parts;
 use serde_json::{Value, json};
+use tokio::task::spawn_blocking;
 
 use proxmox_router::{ApiHandler, ApiMethod, ApiResponseFuture, RpcEnvironment};
 use proxmox_schema::*;
@@ -232,14 +234,18 @@ async fn upload_to_backend(
     let (digest, size, chunk) =
         UploadChunk::new(BodyDataStream::new(req_body), digest, size, encoded_size).await?;
 
+    let datastore = Arc::clone(&env.datastore);
+    let backend = env.backend.clone();
+
     if env.no_cache {
         let (is_duplicate, chunk_size) =
-            env.datastore
-                .insert_chunk_no_cache(&chunk, &digest, &env.backend)?;
+            spawn_blocking(move || datastore.insert_chunk_no_cache(&chunk, &digest, &backend))
+                .await??;
         return Ok((digest, size, chunk_size as u32, is_duplicate));
     }
 
-    let (is_duplicate, chunk_size) = env.datastore.insert_chunk(&chunk, &digest, &env.backend)?;
+    let (is_duplicate, chunk_size) =
+        spawn_blocking(move || datastore.insert_chunk(&chunk, &digest, &backend)).await??;
     Ok((digest, size, chunk_size as u32, is_duplicate))
 }
 
-- 
2.47.3