[pbs-devel] [PATCH proxmox-backup] sync job: pull: avoid blocking future during thread pool completion

[pbs-devel] [PATCH proxmox-backup] sync job: pull: avoid blocking future during thread pool completion

[Christian Ebner]

Chunk insertion during a pull sync job is performed via a thread pool
to increase performance. The caller waits for all threads to be
completed at the end, relying on the respective associated threads to
return from `std::thread::JoinHandle::join`.

Insertion into the chunk store can however take time, especially
for chunks being uploaded to the object store for datastores with
S3 backend.

To avoid blocking the async runtime, spawn a dedicated task to await
the thread pool completion.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
Encountered while looking for possible causes of proxy hangs as reported
in https://forum.proxmox.com/threads/171422/post-807098

 src/server/pull.rs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/server/pull.rs b/src/server/pull.rs
index 08e6e6b64..53f1a570d 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -232,7 +232,7 @@ async fn pull_index_chunks<I: IndexFile>(
 
     drop(verify_and_write_channel);
 
-    verify_pool.complete()?;
+    let _ = tokio::task::spawn_blocking(|| verify_pool.complete()).await?;
 
     let elapsed = start_time.elapsed()?;
 
-- 
2.47.3



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup] sync job: pull: avoid blocking future during thread pool completion

[Fabian Grünbichler]

On October 9, 2025 11:20 am, Christian Ebner wrote:
> Chunk insertion during a pull sync job is performed via a thread pool
> to increase performance. The caller waits for all threads to be
> completed at the end, relying on the respective associated threads to
> return from `std::thread::JoinHandle::join`.
> 
> Insertion into the chunk store can however take time, especially
> for chunks being uploaded to the object store for datastores with
> S3 backend.
> 
> To avoid blocking the async runtime, spawn a dedicated task to await
> the thread pool completion.
> 
> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
> ---
> Encountered while looking for possible causes of proxy hangs as reported
> in https://forum.proxmox.com/threads/171422/post-807098
> 
>  src/server/pull.rs | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/src/server/pull.rs b/src/server/pull.rs
> index 08e6e6b64..53f1a570d 100644
> --- a/src/server/pull.rs
> +++ b/src/server/pull.rs
> @@ -232,7 +232,7 @@ async fn pull_index_chunks<I: IndexFile>(
>  
>      drop(verify_and_write_channel);
>  
> -    verify_pool.complete()?;
> +    let _ = tokio::task::spawn_blocking(|| verify_pool.complete()).await?;

doesn't this change error handling semantics? the inner Result is no
longer handled, AFAICT..

>  
>      let elapsed = start_time.elapsed()?;
>  
> -- 
> 2.47.3
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> 


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

Re: [pbs-devel] [PATCH proxmox-backup] sync job: pull: avoid blocking future during thread pool completion

[Christian Ebner]

On 10/14/25 1:25 PM, Fabian Grünbichler wrote:
> On October 9, 2025 11:20 am, Christian Ebner wrote:
>> Chunk insertion during a pull sync job is performed via a thread pool
>> to increase performance. The caller waits for all threads to be
>> completed at the end, relying on the respective associated threads to
>> return from `std::thread::JoinHandle::join`.
>>
>> Insertion into the chunk store can however take time, especially
>> for chunks being uploaded to the object store for datastores with
>> S3 backend.
>>
>> To avoid blocking the async runtime, spawn a dedicated task to await
>> the thread pool completion.
>>
>> Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
>> ---
>> Encountered while looking for possible causes of proxy hangs as reported
>> in https://forum.proxmox.com/threads/171422/post-807098
>>
>>   src/server/pull.rs | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/src/server/pull.rs b/src/server/pull.rs
>> index 08e6e6b64..53f1a570d 100644
>> --- a/src/server/pull.rs
>> +++ b/src/server/pull.rs
>> @@ -232,7 +232,7 @@ async fn pull_index_chunks<I: IndexFile>(
>>   
>>       drop(verify_and_write_channel);
>>   
>> -    verify_pool.complete()?;
>> +    let _ = tokio::task::spawn_blocking(|| verify_pool.complete()).await?;
> 
> doesn't this change error handling semantics? the inner Result is no
> longer handled, AFAICT..

Oh, you are right! This now only covers the JoinError... Will send a v2 
fixing this shortly.


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel

[pbs-devel] superseded: [PATCH proxmox-backup] sync job: pull: avoid blocking future during thread pool completion

[Christian Ebner]

superseded-by version 2:
https://lore.proxmox.com/pbs-devel/20251014115429.495501-1-c.ebner@proxmox.com/T/


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel