[pbs-devel] [PATCH proxmox-backup 0/5] GC: avoid multiple atime updates

[Christian Ebner <c.ebner@proxmox.com>]

This patches implement the logic to greatly improve the performance
of phase 1 garbage collection by avoiding multiple atime updates on
the same chunk.

Currently, phase 1 GC iterates over all folders in the datastore
looking and collecting all image index files without taking any
logical assumptions (e.g. namespaces, groups, snapshots, ...). This
is to avoid accidentally missing image index files located in
unexpected paths and therefore not marking their chunks as in use,
leading to potential data losses.

This patches improve phase 1 by inserting encountered index image
paths into a data structure which allows to iterate the index files
in a more logical manner, following the same principle as for
incremental backup snapshots. The index files for the same namespace
and group as well as image filename can therefore be consecutevly
inspected.

Further, by keeping track of already seen and therefore updated chunk
atimes, it is now avoided to update the atime over and over again on the
chunks shared by consecutive backup snaphshots.

To give some ballpark figures, this reduced phase 1 garbage collection
on a real world datastore containing some of my backups from around
2 minutes to about 16 seconds.

Christian Ebner (5):
  datastore: restrict datastores list_images method scope to module
  garbage collection: refactor archive type based chunk marking logic
  garbage collection: add structure for optimized image iteration
  garbage collection: allow to keep track of already touched chunks
  fix #5331: garbage collection: avoid multiple chunk atime updates

 pbs-datastore/src/datastore.rs | 204 ++++++++++++++++++++++++++-------
 1 file changed, 160 insertions(+), 44 deletions(-)

-- 
2.39.5



_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel