From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id 1B59B1FF0E7 for ; Fri, 10 Jul 2026 14:37:31 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 6A263212C1; Fri, 10 Jul 2026 14:37:30 +0200 (CEST) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Fri, 10 Jul 2026 14:36:56 +0200 Message-Id: Subject: Re: [PATCH datacenter-manager 08/15] task cache: archive iterator: don't yield more items if reading from file failed From: "Lukas Wagner" To: "Dominik Csapak" , "Lukas Wagner" , X-Mailer: aerc 0.21.0-0-g5549850facc2-dirty References: <20260702092258.174740-1-l.wagner@proxmox.com> <20260702092258.174740-9-l.wagner@proxmox.com> <3210199f-485c-4e99-b0ec-67c28fd7e716@proxmox.com> In-Reply-To: <3210199f-485c-4e99-b0ec-67c28fd7e716@proxmox.com> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1783687005263 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.000 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: TECDB6FYNTOC6OY7FOPQEB4XYDZWNRQ4 X-Message-ID-Hash: TECDB6FYNTOC6OY7FOPQEB4XYDZWNRQ4 X-MailFrom: l.wagner@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox Datacenter Manager development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Fri Jul 10, 2026 at 1:36 PM CEST, Dominik Csapak wrote: > sorry maybe i'm not deep enough into the code here but > how would a single err result produce an endless loop? > > the underlying lines iterator should not infinitely return the error? > > maybe a bit more context for that issue in the commit message would > be nice.... (or maybe it's obvious enough for other, just not me ;) ) Fair enough, I've rewritten the commit message to include a more detail: task cache: archive iterator: don't yield more items if reading from file= failed When iterating over lines in a file with Lines> and the BufRead is based on a zstd Decoder, a malformed zstd file will produce an endless stream of Errors during iteration; apparently the read position inside the decoder is never advanced and thus never reaches the end of the file. To avoid running into an endless loop, don't yield any more items from ArchiveIterator after the first read has failed. Also add a test case that verifies the absence of the issue. > > also should we handle partial corruption, e.g. when only a single line > can't be read? (this skips the rest IIUC) I think it's fine to skip the rest of the file if the read_line fails, since that is probably caused by an IO error or by a malformed zstd file - so we cannot really read anything anyways. If only a single line could not be decoded (e.g. malformed JSON), the later introduced recovery code will skip it and mark the file for recovery.