Re: [PATCH v2 proxmox proxmox-backup 0/3] datastore: gc: defer and batch chunk atime updates

[plantulli@gmail.com]

Hi,

follow-up with the real-world numbers I promised for datastore A, now that
its first full GC cycle with the v1 (whole-store prefetch) series has
completed:

  store: ~82M chunks on disk before the run (81,778,438 entries counted by
         the initial whole-store prefetch below), 1.269 PiB logical on a
         7x raidz1 (28 HDDs) ZFS pool
  start: 2026-08-10 09:01:09
  end:   2026-08-17 16:51:06  (TASK OK)
  total: 7 days 7h50m

  prefetch (whole-store):  81,778,438 entries in 59m33s
  phase 1 (mark):          2d 5h28m (53.5h)
  phase 2 (sweep):         5d 1h23m (121.4h)

  removed garbage:  33.788 TiB
  removed chunks:   18,186,013
  on-disk after:    124.64 TiB / 65,997,624 chunks
  dedup factor:     10.43
  chunk cache hit ratio: 81.08%

Historical baseline for this store, six completed cycles before the patch:
21.25 / 23.50 / 39.99 / 43.64 / 47.24 / 65.81 days. This run beat the best
of those by close to 3x, and the worst by close to 9x.

Two things worth reporting honestly, since they bear directly on the v2
design you proposed:

* Phase 1 was fast (53.5h) but showed the survival problem in practice.
  On the dense stretches (multi-TB VM index groups) the GC worker was
  repeatedly observed blocked in zio_wait <- dbuf_read <- dnode_hold_impl,
  re-reading metadata the whole-store prefetch had already loaded hours
  earlier - the ARC was sitting at its adaptive target, not at c_max.
  I ran an external re-warm (same directory walk, done again mid-run) as
  a workaround, and it recovered the rate by only about +30%, not fully.
  This is exactly the failure mode your deferred-batch design (v2) is
  built to avoid, and I think this run is a fairly clean empirical
  argument for it: one upfront pass does not survive a phase 1 that runs
  for two days on a store this size.

* Phase 2 (121.4h) turned out to be the larger share of the total time,
  more than I expected. Its rate varied widely - from ~50-80 chunks/s
  during dense stretches and heavy overnight contention (up to 10
  concurrent backup jobs observed at once, including one writing directly
  into this same datastore) to ~300 chunks/s when the pool was otherwise
  idle. One ~4h stretch stayed pinned in dmu_tx_hold_free the whole time
  (batch chunk deletion), then resolved on its own. dmu_tx_dirty_delay
  only moved once in the entire week-long run, so the ZFS write throttle
  was not the dominant limit here - contention for disk time between GC,
  ingest and other datastores' GCs was.

Since sending the v1 numbers above, I've started exercising the v2
(deferred, batched atime update) design you proposed, on this same store.
First finding: at this store's chunk density (~1000 chunks per chunk
directory on average), a single flush at the default batch size
(1,048,576) statistically covers close to all 65,536 chunk directories -
so without further care, the whole-store readdir cost effectively repeats
on every flush instead of being paid once. I've added a per-run tracker
that records which directories were already warmed and skips the
redundant readdir on later flushes revisiting them; that's in place now,
but I don't yet have a full end-to-end v2 cycle completed on this store to
report a headline number for it.

None of this changes the headline number: from a worst case of 66 days to
7.3 is already a result I'm happy to ship as-is. But it does suggest v2's
warm-right-before-use approach should help more on this store's phase 1
than the numbers from datastore B alone implied, and that phase 2 might
benefit from a similar treatment if you'd ever consider it - happy to
discuss if useful, otherwise treat that as a side note, not a request to
scope-creep this series.

CLA: signed and sent to office@proxmox.com on Aug 10; I understand it's
being processed.

I'll follow up separately once v2 has run end-to-end on this store.

Thanks,
Enrico