Re: [pbs-devel] [PATCH proxmox-backup v2 0/7] improve catalog handling

[Dominik Csapak <d.csapak@proxmox.com>]

On 7/26/21 10:43 AM, Dietmar Maurer wrote:
> 
>> On 07/26/2021 10:37 AM Dominik Csapak <d.csapak@proxmox.com> wrote:
>>
>>   
>> On 7/26/21 10:26 AM, Dietmar Maurer wrote:
>>>
>>>> On 07/22/2021 3:40 PM Dominik Csapak <d.csapak@proxmox.com> wrote:
>>>>    
>>>> this series combines my previous catalog related patch-series[0][1][2]
>>>>
>>>> changes the catalog interface to be more concise, optimizes catalog
>>>> commit calls during restore, and implements a fast catalog for the
>>>> gui which only contains the snapshot lists
>>>>
>>>> changes from v1:
>>>> * only write snapshot list in new 'finish' method of the catalog
>>>> * add 'finish' also to pool writer
>>>> * replace pending offset counter with reducing the chunk_archive
>>>>     interface of the catalog
>>>
>>> Now, during tape backup, users do not see any progress on the GUI. This
>>> can be particularly confusing on long running tape backups.
>>>
>>> A simpler approach would be to only generate cache files for "finished" tapes (content
>>> will never change), while using the original catalog for tapes still writable. This should
>>> be much easier to implement?
>>>
>>
>> yes it would be simpler, but this does not completely solve the issue of
>> slow reads on large slow catalogs? (the last tape of the media-set can
>> still be so big that the reads take too long?)
> 
> I thought the performance problem is on media-sets with many tapes.
> A single catalog should not cause a large delay?
> 

many tapes makes the problem worse/more likely, but in general
if my disk is too slow to read a catalog in reasonable time, the
ux suffers. AFAIU a single catalog can be larger than 250MiB, which,
when on a storage that is slow to begin with and maybe under load,
can take quite some time to read completely.

e.g. 300MiB / 10MiB/s (slow spinner + big load) = 30s

not great ux when i have to wait more than half a minute to show the
content of my tape backup

in contrast, having a tape with 100000 snapshots with a 'fast catalog'

100000 * 50 bytes ~ 5 MiB should load in under a second