[pve-devel] Further improvement to GC

[pve-devel] Further improvement to GC

[Kamil Trzciński]

Thanks for introducing LRU Cache.

I have another suggestion that we further the sweep phase could be improved:

1. The sweep_unused_chunks could use LRU cache to avoid lstatat if
object is found in cache.

The lstatat would only be required for potential objects that might
have been added while running GC cycle, or that might have been
evicted from LRU cache.
For big enough LRU caches and the fact that only small amount of
objects are usually removed this should make the GC almost in-memory.

2. The process could add a warning that LRU cache is too small, and
propose the value big enough to avoid cache eviction based on number
of chunks observed.

Kamil

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Further improvement to GC

[Christian Ebner]

Hi,

thanks for sharing your ideas!

On 6/8/25 16:25, Kamil Trzciński wrote:
> Thanks for introducing LRU Cache.
> 
> I have another suggestion that we further the sweep phase could be improved:
> 
> 1. The sweep_unused_chunks could use LRU cache to avoid lstatat if
> object is found in cache.

That is indeed true and a good further potential optimization, given 
that cached chunk digest can be considered to have an atime newer than 
the cutoff time. Cached items must and could be accounted differently in 
the GC stats then.

> The lstatat would only be required for potential objects that might
> have been added while running GC cycle, or that might have been
> evicted from LRU cache.
> For big enough LRU caches and the fact that only small amount of
> objects are usually removed this should make the GC almost in-memory.
> 
> 2. The process could add a warning that LRU cache is too small, and
> propose the value big enough to avoid cache eviction based on number
> of chunks observed.

The number of in use chunks can be rather volatile, so just defining a 
suggested cache capacity based on that does not seem to be that ideal to 
me. Also, memory usage by the cache must be taken into account.

But I agree with your points for further improvements and kindly ask you 
to open a new issue for this in our bugtracker [0].

As side note: please use the PBS developer mailing list [1] for 
discussions regarding Proxmox Backup Server.

Best regards,
Chris

[0] https://bugzilla.proxmox.com/
[1] https://lore.proxmox.com/pbs-devel/


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] Further improvement to GC

[Kamil Trzciński]

Thanks. I created in https://bugzilla.proxmox.com/show_bug.cgi?id=6452.

Kamil

On Tue, 10 Jun 2025 at 09:20, Christian Ebner <c.ebner@proxmox.com> wrote:
>
> Hi,
>
> thanks for sharing your ideas!
>
> On 6/8/25 16:25, Kamil Trzciński wrote:
> > Thanks for introducing LRU Cache.
> >
> > I have another suggestion that we further the sweep phase could be improved:
> >
> > 1. The sweep_unused_chunks could use LRU cache to avoid lstatat if
> > object is found in cache.
>
> That is indeed true and a good further potential optimization, given
> that cached chunk digest can be considered to have an atime newer than
> the cutoff time. Cached items must and could be accounted differently in
> the GC stats then.
>
> > The lstatat would only be required for potential objects that might
> > have been added while running GC cycle, or that might have been
> > evicted from LRU cache.
> > For big enough LRU caches and the fact that only small amount of
> > objects are usually removed this should make the GC almost in-memory.
> >
> > 2. The process could add a warning that LRU cache is too small, and
> > propose the value big enough to avoid cache eviction based on number
> > of chunks observed.
>
> The number of in use chunks can be rather volatile, so just defining a
> suggested cache capacity based on that does not seem to be that ideal to
> me. Also, memory usage by the cache must be taken into account.
>
> But I agree with your points for further improvements and kindly ask you
> to open a new issue for this in our bugtracker [0].
>
> As side note: please use the PBS developer mailing list [1] for
> discussions regarding Proxmox Backup Server.
>
> Best regards,
> Chris
>
> [0] https://bugzilla.proxmox.com/
> [1] https://lore.proxmox.com/pbs-devel/
>

_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel