* [pbs-devel] [PATCH proxmox 1/3] s3-client: allow setting custom timeout when sending requests
2025-08-20 9:58 [pbs-devel] [PATCH proxmox{, -backup} 0/5] dynamically calculate request timeout for put object calls Christian Ebner
@ 2025-08-20 9:58 ` Christian Ebner
2025-08-20 9:58 ` [pbs-devel] [PATCH proxmox 2/3] s3-client: dynamically calculate put request timeout based on payload Christian Ebner
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Christian Ebner @ 2025-08-20 9:58 UTC (permalink / raw)
To: pbs-devel
Currently, all types of requests to the S3 API have the same default
timeout of 60 seconds, which was determined empirically.
Different types of requests to the S3 API will however take quite
different time, especially the uploading of objects via a put call
depends on the payload size and bandwidth available.
Therefore, in preparation for being able to set this on a per-request
basis, allow to optionally set the timeout when sending the request.
This now also allows to send the request without timeout.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
proxmox-s3-client/src/client.rs | 34 ++++++++++++++++++++-------------
1 file changed, 21 insertions(+), 13 deletions(-)
diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 3bc2a672..2a48240e 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -283,7 +283,11 @@ impl S3Client {
}
/// Send API request to the configured endpoint using the inner https client.
- async fn send(&self, request: Request<Body>) -> Result<Response<Incoming>, Error> {
+ async fn send(
+ &self,
+ request: Request<Body>,
+ timeout: Option<Duration>,
+ ) -> Result<Response<Incoming>, Error> {
let request = self.prepare(request).await?;
if request.method() == Method::PUT {
if let Some(limiter) = &self.put_rate_limiter {
@@ -294,9 +298,13 @@ impl S3Client {
tokio::time::sleep(sleep).await;
}
}
- let response = tokio::time::timeout(S3_HTTP_REQUEST_TIMEOUT, self.client.request(request))
- .await
- .context("request timeout")??;
+ let response = if let Some(timeout) = timeout {
+ tokio::time::timeout(timeout, self.client.request(request))
+ .await
+ .context("request timeout")??
+ } else {
+ self.client.request(request).await?
+ };
Ok(response)
}
@@ -307,7 +315,7 @@ impl S3Client {
.method(Method::HEAD)
.uri(self.build_uri("/", &[])?)
.body(Body::empty())?;
- let response = self.send(request).await?;
+ let response = self.send(request, Some(S3_HTTP_REQUEST_TIMEOUT)).await?;
let (parts, _body) = response.into_parts();
match parts.status {
@@ -328,7 +336,7 @@ impl S3Client {
.method(Method::GET)
.uri(self.build_uri("/", &[])?)
.body(Body::empty())?;
- let response = self.send(request).await?;
+ let response = self.send(request, Some(S3_HTTP_REQUEST_TIMEOUT)).await?;
let response_reader = ResponseReader::new(response);
response_reader.list_buckets_response().await
}
@@ -344,7 +352,7 @@ impl S3Client {
.method(Method::HEAD)
.uri(self.build_uri(&object_key, &[])?)
.body(Body::empty())?;
- let response = self.send(request).await?;
+ let response = self.send(request, Some(S3_HTTP_REQUEST_TIMEOUT)).await?;
let response_reader = ResponseReader::new(response);
response_reader.head_object_response().await
}
@@ -361,7 +369,7 @@ impl S3Client {
.uri(self.build_uri(&object_key, &[])?)
.body(Body::empty())?;
- let response = self.send(request).await?;
+ let response = self.send(request, Some(S3_HTTP_REQUEST_TIMEOUT)).await?;
let response_reader = ResponseReader::new(response);
response_reader.get_object_response().await
}
@@ -391,7 +399,7 @@ impl S3Client {
.uri(self.build_uri("/", &query)?)
.body(Body::empty())?;
- let response = self.send(request).await?;
+ let response = self.send(request, Some(S3_HTTP_REQUEST_TIMEOUT)).await?;
let response_reader = ResponseReader::new(response);
response_reader.list_objects_v2_response().await
}
@@ -427,7 +435,7 @@ impl S3Client {
let request = request.body(object_data)?;
- let response = self.send(request).await?;
+ let response = self.send(request, Some(S3_HTTP_REQUEST_TIMEOUT)).await?;
let response_reader = ResponseReader::new(response);
response_reader.put_object_response().await
}
@@ -441,7 +449,7 @@ impl S3Client {
.uri(self.build_uri(&object_key, &[])?)
.body(Body::empty())?;
- let response = self.send(request).await?;
+ let response = self.send(request, None).await?;
let response_reader = ResponseReader::new(response);
response_reader.delete_object_response().await
}
@@ -468,7 +476,7 @@ impl S3Client {
.uri(self.build_uri("/", &[("delete", "")])?)
.body(Body::from(body))?;
- let response = self.send(request).await?;
+ let response = self.send(request, Some(S3_HTTP_REQUEST_TIMEOUT)).await?;
let response_reader = ResponseReader::new(response);
response_reader.delete_objects_response().await
}
@@ -499,7 +507,7 @@ impl S3Client {
)
.body(Body::empty())?;
- let response = self.send(request).await?;
+ let response = self.send(request, Some(S3_HTTP_REQUEST_TIMEOUT)).await?;
let response_reader = ResponseReader::new(response);
response_reader.copy_object_response().await
}
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 7+ messages in thread* [pbs-devel] [PATCH proxmox 2/3] s3-client: dynamically calculate put request timeout based on payload
2025-08-20 9:58 [pbs-devel] [PATCH proxmox{, -backup} 0/5] dynamically calculate request timeout for put object calls Christian Ebner
2025-08-20 9:58 ` [pbs-devel] [PATCH proxmox 1/3] s3-client: allow setting custom timeout when sending requests Christian Ebner
@ 2025-08-20 9:58 ` Christian Ebner
2025-08-20 9:58 ` [pbs-devel] [PATCH proxmox 3/3] s3-client: expose default timeout for s3 api requests Christian Ebner
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Christian Ebner @ 2025-08-20 9:58 UTC (permalink / raw)
To: pbs-devel
The request-response time for uploading objects to the S3 API via
put_object call depends on the size of the payload to be transferred.
The current default of 60 seconds might not be enough to upload
chunks or blobs in case of low available upload bandwidth.
Blobs are currently the largest objects which can be uploaded
to the PBS API, reaching up to 128 MiB [0]. Other files such as
notes and index files are typically smaller.
To allow uploads over low upload bandwidth connections to S3 as well,
expose the optional timeout for the put object method and calculate
it dynamically, based on the size of the to be uploaded request
payload assuming a minimum average upload rate of 1KiB/s. However,
keep the default value as minimum timeout.
[0] https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=pbs-datastore/src/data_blob.rs;h=0c05c5a40ae67d4a0d7847817102f30de1df3933;hb=HEAD#l13
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
proxmox-s3-client/examples/s3_client.rs | 3 ++-
proxmox-s3-client/src/client.rs | 15 +++++++++++++--
2 files changed, 15 insertions(+), 3 deletions(-)
diff --git a/proxmox-s3-client/examples/s3_client.rs b/proxmox-s3-client/examples/s3_client.rs
index 61b88077..67baf467 100644
--- a/proxmox-s3-client/examples/s3_client.rs
+++ b/proxmox-s3-client/examples/s3_client.rs
@@ -50,8 +50,9 @@ async fn run() -> Result<(), anyhow::Error> {
let rel_object_key = S3ObjectKey::try_from("object.txt")?;
let body = proxmox_http::Body::empty();
let replace_existing_key = true;
+ let request_timeout = Some(std::time::Duration::from_secs(60));
let _response = s3_client
- .put_object(rel_object_key, body, replace_existing_key)
+ .put_object(rel_object_key, body, request_timeout, replace_existing_key)
.await?;
// List object, limiting to ones matching the given prefix. Since the api limits the response
diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 2a48240e..8a551f3a 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -35,6 +35,8 @@ const S3_HTTP_CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
const S3_HTTP_REQUEST_TIMEOUT: Duration = Duration::from_secs(60);
const S3_TCP_KEEPALIVE_TIME: u32 = 120;
const MAX_S3_UPLOAD_RETRY: usize = 3;
+// Assumed minimum upload rate of 1 KiB/s for dynamic put object request timeout calculation.
+const S3_MIN_ASSUMED_UPLOAD_RATE: u64 = 1024;
/// S3 object key path prefix without the context prefix as defined by the client options.
///
@@ -413,6 +415,7 @@ impl S3Client {
&self,
object_key: S3ObjectKey,
object_data: Body,
+ timeout: Option<Duration>,
replace: bool,
) -> Result<PutObjectResponse, Error> {
let object_key = object_key.to_full_key(&self.options.common_prefix);
@@ -435,7 +438,7 @@ impl S3Client {
let request = request.body(object_data)?;
- let response = self.send(request, Some(S3_HTTP_REQUEST_TIMEOUT)).await?;
+ let response = self.send(request, timeout).await?;
let response_reader = ResponseReader::new(response);
response_reader.put_object_response().await
}
@@ -664,9 +667,17 @@ impl S3Client {
object_data: Bytes,
replace: bool,
) -> Result<bool, Error> {
+ let content_size = object_data.len() as u64;
+ let timeout_secs = content_size
+ .div_ceil(S3_MIN_ASSUMED_UPLOAD_RATE)
+ .max(S3_HTTP_REQUEST_TIMEOUT.as_secs());
+ let timeout = Some(Duration::from_secs(timeout_secs));
for retry in 0..MAX_S3_UPLOAD_RETRY {
let body = Body::from(object_data.clone());
- match self.put_object(object_key.clone(), body, replace).await {
+ match self
+ .put_object(object_key.clone(), body, timeout, replace)
+ .await
+ {
Ok(PutObjectResponse::Success(_response_body)) => return Ok(false),
Ok(PutObjectResponse::PreconditionFailed) => return Ok(true),
Ok(PutObjectResponse::NeedsRetry) => {
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 7+ messages in thread* [pbs-devel] [PATCH proxmox 3/3] s3-client: expose default timeout for s3 api requests
2025-08-20 9:58 [pbs-devel] [PATCH proxmox{, -backup} 0/5] dynamically calculate request timeout for put object calls Christian Ebner
2025-08-20 9:58 ` [pbs-devel] [PATCH proxmox 1/3] s3-client: allow setting custom timeout when sending requests Christian Ebner
2025-08-20 9:58 ` [pbs-devel] [PATCH proxmox 2/3] s3-client: dynamically calculate put request timeout based on payload Christian Ebner
@ 2025-08-20 9:58 ` Christian Ebner
2025-08-20 9:58 ` [pbs-devel] [PATCH proxmox-backup 1/2] s3: pass now optional default request timeout for s3 api calls Christian Ebner
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: Christian Ebner @ 2025-08-20 9:58 UTC (permalink / raw)
To: pbs-devel
Make the default timeout public, so it can be reused for put object
requests in Proxmox Backup Server.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
proxmox-s3-client/src/client.rs | 4 +++-
proxmox-s3-client/src/lib.rs | 2 +-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/proxmox-s3-client/src/client.rs b/proxmox-s3-client/src/client.rs
index 8a551f3a..871cd3fa 100644
--- a/proxmox-s3-client/src/client.rs
+++ b/proxmox-s3-client/src/client.rs
@@ -31,8 +31,10 @@ use crate::response_reader::{
ListBucketsResponse, ListObjectsV2Response, PutObjectResponse, ResponseReader,
};
+/// Default timeout for s3 api requests.
+pub const S3_HTTP_REQUEST_TIMEOUT: Duration = Duration::from_secs(60);
+
const S3_HTTP_CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
-const S3_HTTP_REQUEST_TIMEOUT: Duration = Duration::from_secs(60);
const S3_TCP_KEEPALIVE_TIME: u32 = 120;
const MAX_S3_UPLOAD_RETRY: usize = 3;
// Assumed minimum upload rate of 1 KiB/s for dynamic put object request timeout calculation.
diff --git a/proxmox-s3-client/src/lib.rs b/proxmox-s3-client/src/lib.rs
index f485ac46..26e7032b 100644
--- a/proxmox-s3-client/src/lib.rs
+++ b/proxmox-s3-client/src/lib.rs
@@ -20,7 +20,7 @@ pub use aws_sign_v4::uri_decode;
#[cfg(feature = "impl")]
mod client;
#[cfg(feature = "impl")]
-pub use client::{S3Client, S3ClientOptions, S3PathPrefix};
+pub use client::{S3Client, S3ClientOptions, S3PathPrefix, S3_HTTP_REQUEST_TIMEOUT};
#[cfg(feature = "impl")]
mod timestamps;
#[cfg(feature = "impl")]
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 7+ messages in thread* [pbs-devel] [PATCH proxmox-backup 1/2] s3: pass now optional default request timeout for s3 api calls
2025-08-20 9:58 [pbs-devel] [PATCH proxmox{, -backup} 0/5] dynamically calculate request timeout for put object calls Christian Ebner
` (2 preceding siblings ...)
2025-08-20 9:58 ` [pbs-devel] [PATCH proxmox 3/3] s3-client: expose default timeout for s3 api requests Christian Ebner
@ 2025-08-20 9:58 ` Christian Ebner
2025-08-20 9:58 ` [pbs-devel] [PATCH proxmox-backup 2/2] api: format error to show full context for failed chunk uploads Christian Ebner
2025-08-25 9:35 ` [pbs-devel] applied-series: [PATCH proxmox{, -backup} 0/5] dynamically calculate request timeout for put object calls Wolfgang Bumiller
5 siblings, 0 replies; 7+ messages in thread
From: Christian Ebner @ 2025-08-20 9:58 UTC (permalink / raw)
To: pbs-devel
The request timeout is now exposed as optional duration to the put
object s3 api calls, so it can be dynamically adapted based on the
payload size when uploading objects via the respective helper
methods. Adapt the remaining direct call sites to use the previous
default, given their limited payload size.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
src/api2/admin/s3.rs | 9 +++++++--
src/api2/config/datastore.rs | 3 ++-
2 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/src/api2/admin/s3.rs b/src/api2/admin/s3.rs
index 209feaebc..73f3779a5 100644
--- a/src/api2/admin/s3.rs
+++ b/src/api2/admin/s3.rs
@@ -7,7 +7,7 @@ use proxmox_http::Body;
use proxmox_router::{list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap};
use proxmox_s3_client::{
S3Client, S3ClientConf, S3ClientOptions, S3ObjectKey, S3_BUCKET_NAME_SCHEMA,
- S3_CLIENT_ID_SCHEMA,
+ S3_CLIENT_ID_SCHEMA, S3_HTTP_REQUEST_TIMEOUT,
};
use proxmox_schema::*;
use proxmox_sortable_macro::sortable;
@@ -57,7 +57,12 @@ pub async fn check(
let client = S3Client::new(options).context("client creation failed")?;
client.head_bucket().await.context("head object failed")?;
client
- .put_object(test_object_key.clone(), Body::empty(), true)
+ .put_object(
+ test_object_key.clone(),
+ Body::empty(),
+ Some(S3_HTTP_REQUEST_TIMEOUT),
+ true,
+ )
.await
.context("put object failed")?;
client
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index 5b9fcd9da..e86ded0db 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -36,7 +36,7 @@ use pbs_datastore::{
get_datastore_mount_status, DataStore, DatastoreBackend, S3_DATASTORE_IN_USE_MARKER,
};
use proxmox_rest_server::WorkerTask;
-use proxmox_s3_client::S3ObjectKey;
+use proxmox_s3_client::{S3ObjectKey, S3_HTTP_REQUEST_TIMEOUT};
use crate::server::jobstate;
use crate::tools::disks::unmount_by_mountpoint;
@@ -217,6 +217,7 @@ pub(crate) fn do_create_datastore(
proxmox_async::runtime::block_on(s3_client.put_object(
object_key,
hyper::body::Bytes::from(content).into(),
+ Some(S3_HTTP_REQUEST_TIMEOUT),
true,
))
.context("failed to upload in-use marker for datastore")?;
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 7+ messages in thread* [pbs-devel] [PATCH proxmox-backup 2/2] api: format error to show full context for failed chunk uploads
2025-08-20 9:58 [pbs-devel] [PATCH proxmox{, -backup} 0/5] dynamically calculate request timeout for put object calls Christian Ebner
` (3 preceding siblings ...)
2025-08-20 9:58 ` [pbs-devel] [PATCH proxmox-backup 1/2] s3: pass now optional default request timeout for s3 api calls Christian Ebner
@ 2025-08-20 9:58 ` Christian Ebner
2025-08-25 9:35 ` [pbs-devel] applied-series: [PATCH proxmox{, -backup} 0/5] dynamically calculate request timeout for put object calls Wolfgang Bumiller
5 siblings, 0 replies; 7+ messages in thread
From: Christian Ebner @ 2025-08-20 9:58 UTC (permalink / raw)
To: pbs-devel
Currently, only the outer error context is shown, which can hide a
lot of useful information. By reformatting the error, the whole
error context is returned by the api call.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
---
src/api2/backup/upload_chunk.rs | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/api2/backup/upload_chunk.rs b/src/api2/backup/upload_chunk.rs
index 35378377f..cfbf4aca7 100644
--- a/src/api2/backup/upload_chunk.rs
+++ b/src/api2/backup/upload_chunk.rs
@@ -267,7 +267,7 @@ async fn upload_to_backend(
let is_duplicate = s3_client
.upload_no_replace_with_retry(object_key, data)
.await
- .context("failed to upload chunk to s3 backend")?;
+ .map_err(|err| format_err!("failed to upload chunk to s3 backend - {err:#}"))?;
return Ok((digest, size, encoded_size, is_duplicate));
}
@@ -291,7 +291,7 @@ async fn upload_to_backend(
let is_duplicate = s3_client
.upload_no_replace_with_retry(object_key, data.clone())
.await
- .context("failed to upload chunk to s3 backend")?;
+ .map_err(|err| format_err!("failed to upload chunk to s3 backend - {err:#}"))?;
// Only insert the chunk into the cache after it has been successufuly uploaded.
// Although less performant than doing this in parallel, it is required for consisency
--
2.47.2
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 7+ messages in thread* [pbs-devel] applied-series: [PATCH proxmox{, -backup} 0/5] dynamically calculate request timeout for put object calls
2025-08-20 9:58 [pbs-devel] [PATCH proxmox{, -backup} 0/5] dynamically calculate request timeout for put object calls Christian Ebner
` (4 preceding siblings ...)
2025-08-20 9:58 ` [pbs-devel] [PATCH proxmox-backup 2/2] api: format error to show full context for failed chunk uploads Christian Ebner
@ 2025-08-25 9:35 ` Wolfgang Bumiller
5 siblings, 0 replies; 7+ messages in thread
From: Wolfgang Bumiller @ 2025-08-25 9:35 UTC (permalink / raw)
To: Christian Ebner; +Cc: pbs-devel
applied, thanks
_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
^ permalink raw reply [flat|nested] 7+ messages in thread