Re: [pbs-devel] applied-series: [PATCH v5 proxmox-backup 0/5] fix #5331: GC: avoid multiple atime updates

From: Thomas Lamprecht <t.lamprecht@proxmox.com>
To: Christian Ebner <c.ebner@proxmox.com>,
	Proxmox Backup Server development discussion
	<pbs-devel@lists.proxmox.com>
Subject: Re: [pbs-devel] applied-series: [PATCH v5 proxmox-backup 0/5] fix #5331: GC: avoid multiple atime updates
Date: Thu, 3 Apr 2025 12:17:58 +0200	[thread overview]
Message-ID: <6ccf474a-3137-4e44-88dc-319da5281841@proxmox.com> (raw)
In-Reply-To: <fa3800dd-e812-4c9a-9d3d-2d8673e05355@proxmox.com>

Am 03.04.25 um 11:55 schrieb Christian Ebner:
> Here the results of the suggested additional tests with focus
> on the memory usage dependence on the LRU cache capacity
> (average of 3 runs):
> 
> cache capacity  runtime [s]  peak RSS [MiB]  peak PSS [MiB]
>      128 * 1024   45.3 ± 6.3      92.7 ± 9.3     88.5 ± 12.7
>      256 * 1024   41.2 ± 5.3     117.6 ± 5.3    110.8 ±  4.2
>      512 * 1024   31.2 ± 5.6     162.7 ± 8.1    165.0 ±  0.1
>     1024 * 1024   27.8 ± 3.0     227.3 ± 0.1    232.5 ±  9.1
>     2048 * 1024   26.1 ± 3.0     339.3 ± 6.0    332.5 ±  4.3
> 
> More of interest is however the difference between GC phase1 and
> phase2, observed by a distinct drop in memory usage (not observed if
> cache capacity is 0, so I would speculate this can mostly be
> attributed to the cached items being dropped). 128k is a bit of
> an outlier, not sure about that datapoint.
> 
> cache capacity  runtime [s]  delta RSS [MiB]  delta PSS [MiB]
>      128 * 1024   45.3 ± 6.3       26.2 ± 9.5       24.0 ± 6.2
>      256 * 1024   41.2 ± 5.3       17.5 ± 5.4       17.5 ± 5.5
>      512 * 1024   31.2 ± 5.6       41.2 ± 0.1       41.2 ± 0.1
>     1024 * 1024   27.8 ± 3.0       82.3 ± 0.1       82.5 ± 0.5
>     2048 * 1024   26.1 ± 3.0      163.7 ± 0.6      163.6 ± 0.6

Thanks for the new metrics, that RSS and PSS is the same makes even sense
after rethinking this as those pages do not really hold data that is
likely to be shared with other processes, especially due to the nature of
hash methods having often uniform bit distribution.

Anyway, so half of VmPeak, that already looking much nicer to me.

> 
> Given these values, halving the currently default capacity of 1024k
> might be a reasonable tradeoff and I can send the patch to implement
> the datastore tuning option to allow configuring larger capacities if
> desired by system administrators.
> 
> Rough datastore statistics for these runs were:
> 
> Index files: 6837
> Original data usage: 120.447 TiB
> On-Disk usage: 2.753 TiB (2.29%)
> On-Disk chunks: 1525362
> Deduplication factor: 43.74
> Average chunk size: 1.893 MiB
> 
> What do you think?

Hmm, having the tuning option now already might be indeed worth it.
Halving the default to 512 would be fine to me, OTOH ~ 80 MiB extra
memory for such a demanding operation like GC is not _that_ much for
modern HW, so maybe it would be best to keep that default.

_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel