public inbox for pbs-devel@lists.proxmox.com
 help / color / mirror / Atom feed
* Re: [pbs-devel] [PATCH proxmox-backup v2 0/7] improve catalog handling
@ 2021-07-26  8:26 Dietmar Maurer
  2021-07-26  8:37 ` Dominik Csapak
  0 siblings, 1 reply; 5+ messages in thread
From: Dietmar Maurer @ 2021-07-26  8:26 UTC (permalink / raw)
  To: Proxmox Backup Server development discussion, Dominik Csapak


> On 07/22/2021 3:40 PM Dominik Csapak <d.csapak@proxmox.com> wrote:
>  
> this series combines my previous catalog related patch-series[0][1][2]
> 
> changes the catalog interface to be more concise, optimizes catalog
> commit calls during restore, and implements a fast catalog for the
> gui which only contains the snapshot lists
> 
> changes from v1:
> * only write snapshot list in new 'finish' method of the catalog
> * add 'finish' also to pool writer
> * replace pending offset counter with reducing the chunk_archive
>   interface of the catalog

Now, during tape backup, users do not see any progress on the GUI. This 
can be particularly confusing on long running tape backups.

A simpler approach would be to only generate cache files for "finished" tapes (content 
will never change), while using the original catalog for tapes still writable. This should 
be much easier to implement?




^ permalink raw reply	[flat|nested] 5+ messages in thread
* Re: [pbs-devel] [PATCH proxmox-backup v2 0/7] improve catalog handling
@ 2021-07-26  8:43 Dietmar Maurer
  2021-07-26  8:54 ` Dominik Csapak
  0 siblings, 1 reply; 5+ messages in thread
From: Dietmar Maurer @ 2021-07-26  8:43 UTC (permalink / raw)
  To: Dominik Csapak, Proxmox Backup Server development discussion


> On 07/26/2021 10:37 AM Dominik Csapak <d.csapak@proxmox.com> wrote:
> 
>  
> On 7/26/21 10:26 AM, Dietmar Maurer wrote:
> > 
> >> On 07/22/2021 3:40 PM Dominik Csapak <d.csapak@proxmox.com> wrote:
> >>   
> >> this series combines my previous catalog related patch-series[0][1][2]
> >>
> >> changes the catalog interface to be more concise, optimizes catalog
> >> commit calls during restore, and implements a fast catalog for the
> >> gui which only contains the snapshot lists
> >>
> >> changes from v1:
> >> * only write snapshot list in new 'finish' method of the catalog
> >> * add 'finish' also to pool writer
> >> * replace pending offset counter with reducing the chunk_archive
> >>    interface of the catalog
> > 
> > Now, during tape backup, users do not see any progress on the GUI. This
> > can be particularly confusing on long running tape backups.
> > 
> > A simpler approach would be to only generate cache files for "finished" tapes (content
> > will never change), while using the original catalog for tapes still writable. This should
> > be much easier to implement?
> > 
> 
> yes it would be simpler, but this does not completely solve the issue of
> slow reads on large slow catalogs? (the last tape of the media-set can
> still be so big that the reads take too long?)

I thought the performance problem is on media-sets with many tapes. 
A single catalog should not cause a large delay?




^ permalink raw reply	[flat|nested] 5+ messages in thread
* [pbs-devel] [PATCH proxmox-backup v2 0/7] improve catalog handling
@ 2021-07-22 13:40 Dominik Csapak
  0 siblings, 0 replies; 5+ messages in thread
From: Dominik Csapak @ 2021-07-22 13:40 UTC (permalink / raw)
  To: pbs-devel

this series combines my previous catalog related patch-series[0][1][2]

changes the catalog interface to be more concise, optimizes catalog
commit calls during restore, and implements a fast catalog for the
gui which only contains the snapshot lists

changes from v1:
* only write snapshot list in new 'finish' method of the catalog
* add 'finish' also to pool writer
* replace pending offset counter with reducing the chunk_archive
  interface of the catalog

0: https://lists.proxmox.com/pipermail/pbs-devel/2021-July/003711.html
1: https://lists.proxmox.com/pipermail/pbs-devel/2021-July/003715.html
2: https://lists.proxmox.com/pipermail/pbs-devel/2021-July/003714.html

Dominik Csapak (7):
  tape: media_catalog: improve chunk_archive interface
  tape: media_catalog: add fast_catalog beside normal catalog
  tape: pool_writer: finish the catalog when its done
  tape: media_catalog: add local type aliases to make code more clear
  api2: tape/backup: commit pool_writer even on error
  api2: tape/restore: finish temporary catalog at the end
  api2: tape: media: use MediaCatalog::snapshot_list for content listing

 src/api2/tape/backup.rs             | 117 +++++++++----------
 src/api2/tape/media.rs              |  44 ++++----
 src/api2/tape/restore.rs            |  11 +-
 src/tape/media_catalog.rs           | 167 +++++++++++++++++++++++++---
 src/tape/pool_writer/catalog_set.rs |   9 +-
 src/tape/pool_writer/mod.rs         |  15 ++-
 6 files changed, 250 insertions(+), 113 deletions(-)

-- 
2.30.2





^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-07-26  8:54 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-26  8:26 [pbs-devel] [PATCH proxmox-backup v2 0/7] improve catalog handling Dietmar Maurer
2021-07-26  8:37 ` Dominik Csapak
  -- strict thread matches above, loose matches on Subject: below --
2021-07-26  8:43 Dietmar Maurer
2021-07-26  8:54 ` Dominik Csapak
2021-07-22 13:40 Dominik Csapak

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Service provided by Proxmox Server Solutions GmbH | Privacy | Legal