From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id E9316BC2ED for ; Thu, 28 Mar 2024 13:38:37 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 55407A055 for ; Thu, 28 Mar 2024 13:37:49 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [94.136.29.106]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Thu, 28 Mar 2024 13:37:48 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 537AF42A19 for ; Thu, 28 Mar 2024 13:37:48 +0100 (CET) From: Christian Ebner To: pbs-devel@lists.proxmox.com Date: Thu, 28 Mar 2024 13:36:58 +0100 Message-Id: <20240328123707.336951-50-c.ebner@proxmox.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240328123707.336951-1-c.ebner@proxmox.com> References: <20240328123707.336951-1-c.ebner@proxmox.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.029 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pbs-devel] [PATCH v3 proxmox-backup 49/58] client: backup: increase average chunk size for metadata X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Mar 2024 12:38:38 -0000 Use double the average chunk size for the metadata archive as compared to the payload stream. This does not only reduce the number of unique chunks produced by the metadata archive, not well chunkable because mainly many localized small changes, but further has the positive side effect of producing well compressable larger chunks. The reduced number of chunks further increases the performance for access because of reduced number of download requests and increased cachability. Signed-off-by: Christian Ebner --- changes since version 2: - not present in previous version proxmox-backup-client/src/main.rs | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs index 66dcaa63e..4aad0ff8c 100644 --- a/proxmox-backup-client/src/main.rs +++ b/proxmox-backup-client/src/main.rs @@ -78,6 +78,8 @@ pub(crate) use helper::*; pub mod key; pub mod namespace; +const AVG_METADATA_CHUNK_SIZE: usize = 8 * 1024 * 1024; + fn record_repository(repo: &BackupRepository) { let base = match BaseDirectories::with_prefix("proxmox-backup") { Ok(v) => v, @@ -209,7 +211,15 @@ async fn backup_directory>( payload_target.is_some(), )?; - let mut chunk_stream = ChunkStream::new(pxar_stream, chunk_size, None); + let avg_chunk_size = if payload_stream.is_none() { + chunk_size + } else { + chunk_size + .map(|size| 2 * size) + .or_else(|| Some(AVG_METADATA_CHUNK_SIZE)) + }; + + let mut chunk_stream = ChunkStream::new(pxar_stream, avg_chunk_size, None); let (tx, rx) = mpsc::channel(10); // allow to buffer 10 chunks let stream = ReceiverStream::new(rx).map_err(Error::from); -- 2.39.2