From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 41E9361329 for ; Fri, 20 Nov 2020 06:00:03 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 2EB6BCD85 for ; Fri, 20 Nov 2020 06:00:03 +0100 (CET) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 5721ECD74 for ; Fri, 20 Nov 2020 06:00:01 +0100 (CET) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 1D3A143CF4; Fri, 20 Nov 2020 06:00:01 +0100 (CET) Date: Fri, 20 Nov 2020 05:59:53 +0100 (CET) From: Dietmar Maurer To: Proxmox VE development discussion , =?UTF-8?Q?Carsten_H=C3=A4rle?= Message-ID: <901006600.10.1605848394705@webmail.proxmox.com> In-Reply-To: <57EF5F8B433A6742AD548ABECE78FE48539302@hal9001.straightec.lokal> References: <57EF5F8B433A6742AD548ABECE78FE48539302@hal9001.straightec.lokal> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Priority: 3 Importance: Normal X-Mailer: Open-Xchange Mailer v7.10.4-Rev13 X-Originating-Client: open-xchange-appsuite X-SPAM-LEVEL: Spam detection results: 0 AWL 0.117 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Subject: Re: [pve-devel] Improve container backup speed dramatically (factor 100-1000) X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 05:00:03 -0000 > Container backup is very slow compared to VM backup. I have a 500 GB container (sftp server) with minimal changing files, but even the incremental bakcups take 2 hours with heavy disk activity. Almost nothing is transfered to the backup server. It seems that it it reads the whole container everytime, without any optimization. Before I did backup with zfs send it there it took only a couple of seconds or minutes for every didfferencal backup. Yes, that is how the current variable sized chunking algorithm works. > See discussion here: https://forum.proxmox.com/threads/no-differantial-container-backup-with-big-containers.75676/#post-338868 > > PBS is not storage agnostic but uses underlying snapshot feature according to the documentation: For container, the underlying snapshot feature of the file system ARE used, it already uses ZFS feature. > https://pve.proxmox.com/wiki/Backup_and_Restore Yes, we use the snapshot feature. But the backup code is totally storage agnostic. > For zfs file systems the set of changed file between snapshots can easy be displayed with "zfs diff", so PBS should use this feature to speed up large container backups dramatically. "zfs diff" does not provide the information needed for our deduplication algorithm, so we cannot use that. But if you have ideas how to make that work, please shared them here.