From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 6CF3E94A24 for ; Thu, 11 Apr 2024 09:22:27 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 508541B39C for ; Thu, 11 Apr 2024 09:22:27 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 11 Apr 2024 09:22:26 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 138AB44597 for ; Thu, 11 Apr 2024 09:22:26 +0200 (CEST) Date: Thu, 11 Apr 2024 09:22:17 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= To: Hannes =?iso-8859-1?b?RPxycg==?= , Proxmox Backup Server development discussion References: <20240405130543.259220-1-h.duerr@proxmox.com> <20240405130543.259220-2-h.duerr@proxmox.com> <1712567154.6c6yxorn2q.astroid@yuna.none> In-Reply-To: MIME-Version: 1.0 User-Agent: astroid/0.16.0 (https://github.com/astroidmail/astroid) Message-Id: <1712819712.b6xapom32d.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL 0.057 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] [PATCH proxmox-backup v2 1/3] docs: centralise and update garbage collection description X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2024 07:22:27 -0000 On April 10, 2024 3:38 pm, Hannes D=C3=BCrr wrote: >=20 > On 4/8/24 11:20, Fabian Gr=C3=BCnbichler wrote: >> On April 5, 2024 3:05 pm, Hannes Duerr wrote: >>> +Chunks accessed after the cut-off time are marked as *Pending removals= * >>> +by the GC as it cannot be certain whether they are still needed. >> this is rather incomplete and a bit hard to parse as well. I'd replace >> "accessed after" with "with an atime after". >> >> pending is actually: >> - chunks with atime between the cut-off and the oldest writer (if one >> exists) > At this point i am slightly confused as we defined earlier: > the cut-off is the start of oldest backup writer* (if one exists) the cut off is the minimum of (oldest worker start, now-24h) (minus 5 minutes). there can be an oldest worker that was started after the cut off timestamp, in which case GC might find pending chunks (see below). >=20 > Which would lead to the following: >=20 > - chunks with atime between the cut-off (which is the start of the=20 > oldest existing writer) and the oldest writer (if one exists) >=20 > which does not make any sense, where is my mistake ? if the cut off is determined by a worker started more than 24h before the start of the GC, then there cannot be any pending chunks - because all the chunks which might be considered pending (otherwise) could have been written by that worker, we can't tell. pending chunks can only happen if there is - no backup writer - the olders writer was started less than 24h before the GC then any chunks written in the time frame between cut-off and writer start (or GC start, if no writer exists), which are not yet referenced by any snapshot/index, are considered pending. because those chunks are neither referenced nor can they have been written by any still going writer, so they are most likely "garbage".