Re: [pve-devel] Improve container backup speed dramatically (factor 100-1000)

[Dietmar Maurer <dietmar@proxmox.com>]

> Container backup is very slow compared to VM backup. I have a 500 GB container (sftp server) with minimal changing files, but even the incremental bakcups take 2 hours with heavy disk activity. Almost nothing is transfered to the backup server. It seems that it it reads the whole container everytime, without any optimization. Before I did backup with zfs send it there it took only a couple of seconds or minutes for every didfferencal backup. 

Yes, that is how the current variable sized chunking algorithm works.
  
> See discussion here: https://forum.proxmox.com/threads/no-differantial-container-backup-with-big-containers.75676/#post-338868 <https://forum.proxmox.com/threads/no-differantial-container-backup-with-big-containers.75676/#post-338868> 
>  
> PBS is not storage agnostic but uses underlying snapshot feature according to the documentation: For container, the underlying snapshot feature of the file system ARE used, it already uses ZFS feature. 
> https://pve.proxmox.com/wiki/Backup_and_Restore <https://pve.proxmox.com/wiki/Backup_and_Restore> 

Yes, we use the snapshot feature. But the backup code is totally storage agnostic.

> For zfs file systems the set of changed file between snapshots can easy be displayed with "zfs diff", so PBS should use this feature to speed up large container backups dramatically.

"zfs diff" does not provide the information needed for our deduplication 
algorithm, so we cannot use that. But if you have ideas how to make that work, 
please shared them here.