From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 589AE9E8E4 for ; Tue, 28 Nov 2023 11:07:17 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 3AD291433F for ; Tue, 28 Nov 2023 11:07:17 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Tue, 28 Nov 2023 11:07:16 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 63BE144D0B for ; Tue, 28 Nov 2023 11:07:16 +0100 (CET) Date: Tue, 28 Nov 2023 11:07:15 +0100 From: Wolfgang Bumiller To: Fabian =?utf-8?Q?Gr=C3=BCnbichler?= Cc: pbs-devel@lists.proxmox.com Message-ID: References: <20230629103213.1041236-1-f.gruenbichler@proxmox.com> <1541339416.519.1701106073073@webmail.proxmox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1541339416.519.1701106073073@webmail.proxmox.com> X-SPAM-LEVEL: Spam detection results: 0 AWL 0.097 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record T_SCC_BODY_TEXT_LINE -0.01 - Subject: Re: [pbs-devel] [PATCH proxmox-backup] mapped loop device: use read loop instead of read_exact X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2023 10:07:17 -0000 On Mon, Nov 27, 2023 at 06:27:53PM +0100, Fabian Grünbichler wrote: > > Wolfgang Bumiller hat am 27.11.2023 14:22 CET geschrieben: > > > > On Thu, Jun 29, 2023 at 12:32:13PM +0200, Fabian Grünbichler wrote: > > > since read_exact does not support short reads, which can easily happen if the > > > mapped image's EOF is not aligned with the request size. > > > > > > Signed-off-by: Fabian Grünbichler > > > --- > > > > > > Notes: > > > reported on the forum: > > > > > > https://forum.proxmox.com/threads/problem-backing-up-using-backup-client.129347 > > > > > > did a quick test reading from a mapped image full of random data, observed > > > no performance difference.. > > > > Do you get one if we just drop the loop logic and *actually* just > > `read()` once? IMO this is more in line with what a read syscall > > *should* be doing. > > Further, we use a `CachedChunkReader` under it which actually does a > > read loop anyway, so AFAICT this *can't* make a difference. > > with a plain read (+ optional truncate of the reply buf) performance is still the same. but (and I am unfortunately not sure if this is a regression in the meantime, or was also broken back when I originally wrote this patch) access via the loop device actually truncates the resulting data: > > - my test input image is 1701838801 bytes long (arbitrary misaligned size, straight from /dev/urandom) it's a loop device -> these are block devices defaulting to 512 blocks 1701838801 % 512 = 465 if you call `losetup` manually on it you'll get a warning like: losetup: /your/file: Warning: file does not fit into a 512-byte sector; the end of the file will be ignored. > - the fuse session correctly gets this passed in as size > - a regular restore restores as many (correct) bytes > - reading via the loop device with bs=1024 or bs=512 or bs=32 only returns 1701838336 bytes (465 are missing) > -- the fuse requests quickly ramp up to 128k request size (no matter the block size used to read from the loop device) > -- the last fuse read request is for 16384 bytes, but the read from PBS (correctly!) only returns 16337 > -- 16337 - 31*512 = 465 > -- so it seems the short read result is lost somewhere? > -- reading with O_DIRECT doesn't help (in fact, it tanks performance while still reproducing the issue) > > anyhow, this requires further analysis and fixing before being applied in whichever fashion..