From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id AEC9092999 for ; Mon, 8 Apr 2024 10:29:02 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 7FE706265 for ; Mon, 8 Apr 2024 10:28:32 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 8 Apr 2024 10:28:31 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 555134450C for ; Mon, 8 Apr 2024 10:28:30 +0200 (CEST) Date: Mon, 08 Apr 2024 10:28:26 +0200 From: Fabian =?iso-8859-1?q?Gr=FCnbichler?= To: Christian Ebner , Dietmar Maurer , Proxmox Backup Server development discussion References: <20240328123707.336951-1-c.ebner@proxmox.com> <20240328123707.336951-50-c.ebner@proxmox.com> <171231015999.1926770.4997530295571319940@yuna.proxmox.com> <1971461654.4558.1712314150119@webmail.proxmox.com> In-Reply-To: <1971461654.4558.1712314150119@webmail.proxmox.com> MIME-Version: 1.0 User-Agent: astroid/0.16.0 (https://github.com/astroidmail/astroid) Message-Id: <1712564671.kzyltpg23p.astroid@yuna.none> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-SPAM-LEVEL: Spam detection results: 0 AWL 0.058 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pbs-devel] [PATCH v3 proxmox-backup 49/58] client: backup: increase average chunk size for metadata X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2024 08:29:02 -0000 On April 5, 2024 12:49 pm, Dietmar Maurer wrote: >> for the payload stream simple accumulating 1..N files (or rather, their >> contents) in a chunk until a certain size threshold is reached might per= form >> better (as in, both be faster than the current chunker, and give us more= /better >> re-usable chunks). >=20 > Sorry, but that way you would never reuse any chunks! How is > that supposed to work? the chunk re-usage would be moved to the metadata-based caching, basically: - big files get a sequence of chunks according to some splitting rules, those chunks are completely just for that file (so if you just modify a bit at the front, only the first chunk would be new, the rest still re-used, but with read penalty) - smaller files are aggregated into a single chunk, those would not be re-used if too many of them changed (payload threshold) it might just trade on set of issues with another (higher padding vs less deduplication), not sure.