[pbs-devel] applied: [PATCH proxmox-backup v3 1/1] tape: introduce a tape backup job worker thread option

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

Am 21.02.25 um 16:06 schrieb Dominik Csapak:
> Using a single thread for reading is not optimal in some cases, e.g.
> when the underlying storage can handle reads from multiple threads in
> parallel.
> 
> We use the ParallelHandler to handle the actual reads. Make the
> sync_channel buffer size depend on the number of threads so we have
> space for two chunks per thread. (But keep the minimum to 3 like
> before).
> 
> How this impacts the backup speed largely depends on the underlying
> storage and how the backup is laid out on it.
> 
> I benchmarked the following setups:
> 
> * Setup A: relatively spread out backup on a virtualized pbs on single HDDs
> * Setup B: mostly sequential chunks on a virtualized pbs on single HDDs
> * Setup C: backup on virtualized pbs on a fast NVME
> * Setup D: backup on bare metal pbs with ZFS in a RAID10 with 6 HDDs
>   and 2 fast special devices in a mirror
> 
> (values are reported in MB/s as seen in the task log, caches were
> cleared between runs, backups were bigger than the memory available)
> 
> setup  1 thread  2 threads  4 threads  8 threads
> A      55        70         80         95
> B      110       89         100        108
> C      294       294        294        294
> D      118       180        300        300
> 
> So there are cases where multiple read threads speed up the tape backup
> (dramatically). On the other hand there are situations where reading
> from a single thread is actually faster, probably because we can read
> from the HDD sequentially.
> 
> I left the default value of '1' to not change the default behavior.
> 
> Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
> ---
> changes from v2:
> * move the ui config to the tape backup jobs and tape backup window
> * use pool writer to store the thread count, not datastore
> * keep minimum of channel size 3
> * keep default of one thread
> 
> i left the benchmark data intact, since the actual code that does
> the multithreading is the same as before, and i could not find a
> virtual setup to replicate the performance of a hdd very well
> (limiting virtual disks iops does not really do the trick due to
> disk caching, etc.)
> 
> If wanted i can of course setup a physical testbed with multiple hdds
> again.
> 
>  src/api2/config/tape_backup_job.rs          |  8 ++++
>  src/api2/tape/backup.rs                     |  4 ++
>  src/tape/pool_writer/mod.rs                 | 14 ++++++-
>  src/tape/pool_writer/new_chunks_iterator.rs | 44 +++++++++++++--------
>  www/tape/window/TapeBackup.js               | 12 ++++++
>  www/tape/window/TapeBackupJob.js            | 14 +++++++
>  6 files changed, 79 insertions(+), 17 deletions(-)
> 
>

applied, thanks! bumped the dependency for pbs-api-types upfront and adapted
the comment as discussed in this thread. I also dropped the commented-out
printf statement, feels always rather odd to read them to me, especially for
compiled code where one cannot uncomment them ad-hoc. debug/trace-logs might
be better suited if that can be useful.


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel