From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 7AFEE611C9 for ; Thu, 19 Nov 2020 20:35:23 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 6DE93A8D9 for ; Thu, 19 Nov 2020 20:35:23 +0100 (CET) Received: from server12.webgo24.de (server12.webgo24.de [185.30.32.12]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id BA2E9A8C3 for ; Thu, 19 Nov 2020 20:35:21 +0100 (CET) Received: from straightec.de (i6DFAF5F3.versanet.de [109.250.245.243]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by server12.webgo24.de (Postfix) with ESMTPSA id 38B02A54164E for ; Thu, 19 Nov 2020 20:29:32 +0100 (CET) Content-class: urn:content-classes:message MIME-Version: 1.0 X-MimeOLE: Produced By Microsoft Exchange V6.5 Date: Thu, 19 Nov 2020 20:29:26 +0100 Message-ID: <57EF5F8B433A6742AD548ABECE78FE48539302@hal9001.straightec.lokal> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Improve container backup speed dramatically (factor 100-1000) Thread-Index: Ada+qksZniFgiT/QQ9it0xQql2sJoQ== From: =?iso-8859-1?Q?Carsten_H=E4rle?= To: X-SPAM-LEVEL: Spam detection results: 1 HTML_MESSAGE 0.001 HTML included in message KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment KAM_LAZY_DOMAIN_SECURITY 1 Sending domain does not have any anti-forgery methods SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_NONE 0.001 SPF: sender does not publish an SPF Record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: [pve-devel] Improve container backup speed dramatically (factor 100-1000) X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Nov 2020 19:35:23 -0000 Container backup is very slow compared to VM backup. I have a 500 GB contai= ner (sftp server) with minimal changing files, but even the incremental bak= cups take 2 hours with heavy disk activity. Almost nothing is transfered to= the backup server. It seems that it it reads the whole container everytime= , without any optimization. Before I did backup with zfs send it there it t= ook only a couple of seconds or minutes for every didfferencal backup.=20 =20 See discussion here: https://forum.proxmox.com/threads/no-differantial-cont= ainer-backup-with-big-containers.75676/#post-338868 =20 =20 PBS is not storage agnostic but uses underlying snapshot feature according = to the documentation: For container, the underlying snapshot feature of the= file system ARE used, it already uses ZFS feature.=20 https://pve.proxmox.com/wiki/Backup_and_Restore =20 =20 For zfs file systems the set of changed file between snapshots can easy be = displayed with "zfs diff", so PBS should use this feature to speed up large= container backups dramatically. In my case it would be faktor about factor= 1000! Alternatively the snapshots data can be accessed via the hidden .zfs= directories, so PBS has knows which files changed and has access to the ol= d and the new data. =20 https://bugzilla.proxmox.com/show_bug.cgi?id=3D3138 =20