[PVE-User] Timeouts while creating backups on PBS

[PVE-User] Timeouts while creating backups on PBS

[Sven]

Hello

we have several clusters (~250 VMs) attached to one PBS and run daily 
backups. Each day, there are random timeouts causing some backups to 
fail.

I've disabled the instant verification process to reduce IO-load on the 
PBS during that time. The timeouts are gone since then.

Because we do want to have verified backups, we tried out scheduled 
verification tasks. Then we have the problem, that it takes to long 
(>1d) to verify all ~250 new snapshots sequentially. 50 parallel verify 
jobs works well.

My current idea is to write a custom scheduler to make sure that there 
is a custom amount of verify jobs simultaneously when no backups are 
running.

Is there a better way I don't see? If I may place a feature request: It 
would be nice to be able to specify how many verify jobs can work 
simultaneously.

Thanks!
Sven

Re: [PVE-User] Timeouts while creating backups on PBS

[Victor Rodriguez]

> Because we do want to have verified backups, we tried out scheduled 
> verification tasks. Then we have the problem, that it takes to long 
> (>1d) to verify all ~250 new snapshots sequentially. 50 parallel 
> verify jobs works well.

AFAIK, a verification job will not re-verify a chunk that has been 
verified already by that same job, meaning that a verify job for 2+ 
snapshots should be more efficient that a verify job for each snapshot 
or doing an inmediate verify after each backup.

On the other hand, verifies are processed sequentially, so even if your 
host has enough resources the job will take some time to finish. The 
approach I currently use is to get a few verify jobs for each namespace 
and start them at the same time if possible.


--