[pbs-devel] [PATCH proxmox{, -backup} 0/5] dynamically calculate request timeout for put object calls

[Christian Ebner <c.ebner@proxmox.com>]

Currently, the s3 client sets a default timeout value of 60 seconds
for each request send to the s3 api, independent of the request type.

Put request uploading data can however take considerably more time,
depending on the payload size and available upload bandwidth. This
can lead to upload requests failing as reported in the community
forum [0,1].

To partially fix this, calculate the request timeout for put object
api calls based on the content size to be uploaded. Other issues due
to network congestion might still arise, but might be circumvented
via traffic shaping.

To check the behaviour, the following `tc` filtering has been performed
to limit the bandwidth to 10Mbit on the outgoing network interface ens18:
```
tc qdisc add dev ens18 root handle 1: htb
tc class add dev ens18 parent 1: classid 1:1 htb rate 10Mbit
tc filter add dev ens18 protocol ip parent 1: prio 1 u32 match ip dst 0/0 flowid 1:1
```

The outgoing traffic has been monitored via `atop`.

This is intended as stop gap until a more advance shared request and
rate limiter is implemented for the s3 client.

[0] https://forum.proxmox.com/threads/169620/
[1] https://forum.proxmox.com/threads/169432/

proxmox:

Christian Ebner (3):
  s3-client: allow setting custom timeout when sending requests
  s3-client: dynamically calculate put request timeout based on payload
  s3-client: expose default timeout for s3 api requests

 proxmox-s3-client/examples/s3_client.rs |  3 +-
 proxmox-s3-client/src/client.rs         | 51 +++++++++++++++++--------
 proxmox-s3-client/src/lib.rs            |  2 +-
 3 files changed, 39 insertions(+), 17 deletions(-)


proxmox-backup:

Christian Ebner (2):
  s3: pass now optional default request timeout for s3 api calls
  api: format error to show full context for failed chunk uploads

 src/api2/admin/s3.rs            | 9 +++++++--
 src/api2/backup/upload_chunk.rs | 4 ++--
 src/api2/config/datastore.rs    | 3 ++-
 3 files changed, 11 insertions(+), 5 deletions(-)


Summary over all repositories:
  6 files changed, 50 insertions(+), 22 deletions(-)

-- 
Generated by git-murpp 0.8.1


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel