[pbs-devel] Verify, Prune & GC sequence

[pbs-devel] Verify, Prune & GC sequence

[Graeme Seaton]

Hi,

If I have verify, prune & GC all set to run daily do the jobs run 
concurrently or in sequence?

Regrds,

Graeme

Re: [pbs-devel] Verify, Prune & GC sequence

[Thomas Lamprecht]

Hi,

On 30.10.20 10:31, Graeme Seaton wrote:
> If I have verify, prune & GC all set to run daily do the jobs run concurrently or in sequence?

Well, you could just try out ;-)

But yes, they can run in parallel. 

Since PBS allows multiple potentially interfering operations at the same time (e.g., garbage collect,
prune, multiple backup creations (only in separate groups), forget, ...), these need to lock against
each other in certain scenarios. There is no overarching global lock though, instead always the finest
grained lock possible is used, because running these operations concurrently is treated as a feature
on its own.

Still, while we handle locking fine grained an admin could try to optimize the order of scheduling
such things.

1. Prune, to increase the chance that GC can sweep more data and verify has less to look at.
   This is rather quick, so one could schedule the next probably half an hour afterwards and
   be confident it finishes before 2.

2. GC, as chunks are deleted, which may make verify also faster as it can make the underlying
   FS faster - but that's rather theory.

3. Verify

I think the order of 2. and 3. is not to important, but personally, I'd always schedule pruning
first, can only help.

cheers,
Thomas

Re: [pbs-devel] Verify, Prune & GC sequence

[Graeme Seaton]

Hi Thomas,

Thanks for the reply - didn't really want stay up to midnight to check 
;-)  At the moment, scheduling only allows daily (which is invoked at 
00:00).  If (when?) I can specify custom times then I'll probably use 
the sequence you described.

Regards,

Graeme

On 30/10/2020 11:09, Thomas Lamprecht wrote:
> Hi,
>
> On 30.10.20 10:31, Graeme Seaton wrote:
>> If I have verify, prune & GC all set to run daily do the jobs run concurrently or in sequence?
> Well, you could just try out ;-)
>
> But yes, they can run in parallel.
>
> Since PBS allows multiple potentially interfering operations at the same time (e.g., garbage collect,
> prune, multiple backup creations (only in separate groups), forget, ...), these need to lock against
> each other in certain scenarios. There is no overarching global lock though, instead always the finest
> grained lock possible is used, because running these operations concurrently is treated as a feature
> on its own.
>
> Still, while we handle locking fine grained an admin could try to optimize the order of scheduling
> such things.
>
> 1. Prune, to increase the chance that GC can sweep more data and verify has less to look at.
>     This is rather quick, so one could schedule the next probably half an hour afterwards and
>     be confident it finishes before 2.
>
> 2. GC, as chunks are deleted, which may make verify also faster as it can make the underlying
>     FS faster - but that's rather theory.
>
> 3. Verify
>
> I think the order of 2. and 3. is not to important, but personally, I'd always schedule pruning
> first, can only help.
>
> cheers,
> Thomas
>
>

Re: [pbs-devel] Verify, Prune & GC sequence

[Thomas Lamprecht]

Hi Graeme,

On 30.10.20 17:31, Graeme Seaton wrote:
> Hi Thomas,
> 
> Thanks for the reply - didn't really want stay up to midnight to check ;-)  At the moment, scheduling only allows daily (which is invoked at 00:00).  If (when?) I can specify custom times then I'll probably use the sequence you described.
> 

You can already specify custom times, just edit it, "daily" is just set as default
when creating a new job.

cheers,
Thomas

Re: [pbs-devel] Verify, Prune & GC sequence

[Dietmar Maurer]

> > Thanks for the reply - didn't really want stay up to midnight to check ;-)  At the moment, scheduling only allows daily (which is invoked at 00:00).  If (when?) I can specify custom times then I'll probably use the sequence you described.
> > 
> 
> You can already specify custom times, just edit it, "daily" is just set as default
> when creating a new job.
> 

To make it more clear, we support systemd Calendar events. For format details see:

https://www.freedesktop.org/software/systemd/man/systemd.time.html#Calendar%20Events

Re: [pbs-devel] Verify, Prune & GC sequence

[Thomas Lamprecht]

On 31.10.20 07:43, Dietmar Maurer wrote:
>>> Thanks for the reply - didn't really want stay up to midnight to check ;-)  At the moment, scheduling only allows daily (which is invoked at 00:00).  If (when?) I can specify custom times then I'll probably use the sequence you described.
>>>
>>
>> You can already specify custom times, just edit it, "daily" is just set as default
>> when creating a new job.
>>
> 
> To make it more clear, we support systemd Calendar events. For format details see:
> 
> https://www.freedesktop.org/software/systemd/man/systemd.time.html#Calendar%20Events
> 

Here the link to our docs, as we support only the relevant features of Calendar Events:

https://pbs.proxmox.com/docs/calendarevents.html

Re: [pbs-devel] Verify, Prune & GC sequence

[Graeme Seaton]

Thank you both for the links - tested last night and worked great.  My 
bad on manually editing the schedule - didn't find it immediately 
obvious that I could edit the value in the dropbox.

Would also congratulate you all on the project - it is looking really 
nice and the UI has come along in leaps and bounds :-D

On 31/10/2020 06:53, Thomas Lamprecht wrote:
> On 31.10.20 07:43, Dietmar Maurer wrote:
>>>> Thanks for the reply - didn't really want stay up to midnight to check ;-)  At the moment, scheduling only allows daily (which is invoked at 00:00).  If (when?) I can specify custom times then I'll probably use the sequence you described.
>>>>
>>> You can already specify custom times, just edit it, "daily" is just set as default
>>> when creating a new job.
>>>
>> To make it more clear, we support systemd Calendar events. For format details see:
>>
>> https://www.freedesktop.org/software/systemd/man/systemd.time.html#Calendar%20Events
>>
> Here the link to our docs, as we support only the relevant features of Calendar Events:
>
> https://pbs.proxmox.com/docs/calendarevents.html
>
>