From: Dominik Csapak <d.csapak@proxmox.com>
To: Dietmar Maurer <dietmar@proxmox.com>,
Proxmox Backup Server development discussion
<pbs-devel@lists.proxmox.com>
Subject: Re: [pbs-devel] [PATCH proxmox-backup v2 0/7] improve catalog handling
Date: Mon, 26 Jul 2021 10:54:10 +0200 [thread overview]
Message-ID: <132e7fa3-018b-4492-c1ab-2ffc136ad965@proxmox.com> (raw)
In-Reply-To: <1781641437.1181.1627289020670@webmail.proxmox.com>
On 7/26/21 10:43 AM, Dietmar Maurer wrote:
>
>> On 07/26/2021 10:37 AM Dominik Csapak <d.csapak@proxmox.com> wrote:
>>
>>
>> On 7/26/21 10:26 AM, Dietmar Maurer wrote:
>>>
>>>> On 07/22/2021 3:40 PM Dominik Csapak <d.csapak@proxmox.com> wrote:
>>>>
>>>> this series combines my previous catalog related patch-series[0][1][2]
>>>>
>>>> changes the catalog interface to be more concise, optimizes catalog
>>>> commit calls during restore, and implements a fast catalog for the
>>>> gui which only contains the snapshot lists
>>>>
>>>> changes from v1:
>>>> * only write snapshot list in new 'finish' method of the catalog
>>>> * add 'finish' also to pool writer
>>>> * replace pending offset counter with reducing the chunk_archive
>>>> interface of the catalog
>>>
>>> Now, during tape backup, users do not see any progress on the GUI. This
>>> can be particularly confusing on long running tape backups.
>>>
>>> A simpler approach would be to only generate cache files for "finished" tapes (content
>>> will never change), while using the original catalog for tapes still writable. This should
>>> be much easier to implement?
>>>
>>
>> yes it would be simpler, but this does not completely solve the issue of
>> slow reads on large slow catalogs? (the last tape of the media-set can
>> still be so big that the reads take too long?)
>
> I thought the performance problem is on media-sets with many tapes.
> A single catalog should not cause a large delay?
>
many tapes makes the problem worse/more likely, but in general
if my disk is too slow to read a catalog in reasonable time, the
ux suffers. AFAIU a single catalog can be larger than 250MiB, which,
when on a storage that is slow to begin with and maybe under load,
can take quite some time to read completely.
e.g. 300MiB / 10MiB/s (slow spinner + big load) = 30s
not great ux when i have to wait more than half a minute to show the
content of my tape backup
in contrast, having a tape with 100000 snapshots with a 'fast catalog'
100000 * 50 bytes ~ 5 MiB should load in under a second
next prev parent reply other threads:[~2021-07-26 8:54 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-26 8:43 Dietmar Maurer
2021-07-26 8:54 ` Dominik Csapak [this message]
-- strict thread matches above, loose matches on Subject: below --
2021-07-26 8:26 Dietmar Maurer
2021-07-26 8:37 ` Dominik Csapak
2021-07-22 13:40 Dominik Csapak
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=132e7fa3-018b-4492-c1ab-2ffc136ad965@proxmox.com \
--to=d.csapak@proxmox.com \
--cc=dietmar@proxmox.com \
--cc=pbs-devel@lists.proxmox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox