Re: [pbs-devel] applied-series: [PATCH v5 proxmox-backup 0/5] fix #5331: GC: avoid multiple atime updates

[Christian Ebner <c.ebner@proxmox.com>]

On 4/3/25 12:17, Thomas Lamprecht wrote:
> Am 03.04.25 um 11:55 schrieb Christian Ebner:
>> Here the results of the suggested additional tests with focus
>> on the memory usage dependence on the LRU cache capacity
>> (average of 3 runs):
>>
>> cache capacity  runtime [s]  peak RSS [MiB]  peak PSS [MiB]
>>       128 * 1024   45.3 ± 6.3      92.7 ± 9.3     88.5 ± 12.7
>>       256 * 1024   41.2 ± 5.3     117.6 ± 5.3    110.8 ±  4.2
>>       512 * 1024   31.2 ± 5.6     162.7 ± 8.1    165.0 ±  0.1
>>      1024 * 1024   27.8 ± 3.0     227.3 ± 0.1    232.5 ±  9.1
>>      2048 * 1024   26.1 ± 3.0     339.3 ± 6.0    332.5 ±  4.3
>>
>> More of interest is however the difference between GC phase1 and
>> phase2, observed by a distinct drop in memory usage (not observed if
>> cache capacity is 0, so I would speculate this can mostly be
>> attributed to the cached items being dropped). 128k is a bit of
>> an outlier, not sure about that datapoint.
>>
>> cache capacity  runtime [s]  delta RSS [MiB]  delta PSS [MiB]
>>       128 * 1024   45.3 ± 6.3       26.2 ± 9.5       24.0 ± 6.2
>>       256 * 1024   41.2 ± 5.3       17.5 ± 5.4       17.5 ± 5.5
>>       512 * 1024   31.2 ± 5.6       41.2 ± 0.1       41.2 ± 0.1
>>      1024 * 1024   27.8 ± 3.0       82.3 ± 0.1       82.5 ± 0.5
>>      2048 * 1024   26.1 ± 3.0      163.7 ± 0.6      163.6 ± 0.6
> 
> Thanks for the new metrics, that RSS and PSS is the same makes even sense
> after rethinking this as those pages do not really hold data that is
> likely to be shared with other processes, especially due to the nature of
> hash methods having often uniform bit distribution.
> 
> Anyway, so half of VmPeak, that already looking much nicer to me.
> 
>>
>> Given these values, halving the currently default capacity of 1024k
>> might be a reasonable tradeoff and I can send the patch to implement
>> the datastore tuning option to allow configuring larger capacities if
>> desired by system administrators.
>>
>> Rough datastore statistics for these runs were:
>>
>> Index files: 6837
>> Original data usage: 120.447 TiB
>> On-Disk usage: 2.753 TiB (2.29%)
>> On-Disk chunks: 1525362
>> Deduplication factor: 43.74
>> Average chunk size: 1.893 MiB
>>
>> What do you think?
> 
> Hmm, having the tuning option now already might be indeed worth it.
> Halving the default to 512 would be fine to me, OTOH ~ 80 MiB extra
> memory for such a demanding operation like GC is not _that_ much for
> modern HW, so maybe it would be best to keep that default.

Okay, so let's keep the default then and I will work on the patches to 
add the datastore tuning option and expose this via the UI.

Not sure if we should opt for using the size in MiB as you suggested in 
the other reply, given that the actual memory requirements are higher? 
On the other hand that would give a more intuitive parameter/value to 
work with...


_______________________________________________
pbs-devel mailing list
pbs-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel