From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id C96E364CEE for ; Sun, 1 Nov 2020 14:14:56 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id BFED61C8E7 for ; Sun, 1 Nov 2020 14:14:56 +0100 (CET) Received: from mail.bdick.de (he3.bdick.de [136.243.64.144]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 717071C8D7 for ; Sun, 1 Nov 2020 14:14:55 +0100 (CET) Received: from ip-88-153-16-148.hsi04.unitymediagroup.de ([88.153.16.148] helo=[192.168.1.101]) by mail.bdick.de with esmtpsa (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1kZDCB-0005Ad-59 for pve-user@lists.proxmox.com; Sun, 01 Nov 2020 14:14:55 +0100 To: Proxmox VE user list References: <1bea39e7-d656-d809-0774-baa935479f53@bdick.de> <7d0ea9da-3b9e-d9da-a2f6-1c7f5eca11f8@proxmox.com> From: Bernhard Dick Message-ID: <746ccce3-578f-c75d-8958-f3776b7f1166@bdick.de> Date: Sun, 1 Nov 2020 14:14:48 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.4.0 MIME-Version: 1.0 In-Reply-To: <7d0ea9da-3b9e-d9da-a2f6-1c7f5eca11f8@proxmox.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: de-DE Content-Transfer-Encoding: 7bit X-SPAM-LEVEL: Spam detection results: 0 AWL 0.129 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment NICE_REPLY_A -0.257 Looks like a legit reply (A) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [bdick.de, your-storagebox.de] Subject: Re: [PVE-User] Backup broken? X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Nov 2020 13:14:56 -0000 Hi, Am 01.11.2020 um 13:48 schrieb Thomas Lamprecht: > Hi, > > On 31.10.20 15:11, Bernhard Dick wrote: >> I've a weekly backup job on my proxmox (community version 6.2) infrastructure. Yesterday I've updated to the latest package versions available and in the night the backup job ran. But he decided to first create a backup and then delete _all_ backups being on the backup location which is now empty besides some log files. >> Here is an excerpt of one of the mailed logs: >> >> [..] > > It'd be great if you could provided the full task log, not only an excerpt. Here is a full log of two Systems (one container, one VM) of from the job: vzdump --quiet 1 --mailnotification always --mailto bernhard@bdick.de --compress lzo --mode snapshot --storage backup-fsn0 --node he2 --all 1 101: 2020-10-31 20:43:26 INFO: Starting Backup of VM 101 (lxc) 101: 2020-10-31 20:43:26 INFO: status = running 101: 2020-10-31 20:43:26 INFO: CT Name: ns0.bdick.de 101: 2020-10-31 20:43:26 INFO: including mount point rootfs ('/') in backup 101: 2020-10-31 20:43:26 INFO: backup mode: snapshot 101: 2020-10-31 20:43:26 INFO: ionice priority: 7 101: 2020-10-31 20:43:26 INFO: create storage snapshot 'vzdump' 101: 2020-10-31 20:43:27 INFO: creating vzdump archive '/mnt/pve/backup-fsn0/dump/vzdump-lxc-101-2020_10_31-20_43_26.tar.lzo' 101: 2020-10-31 20:44:05 INFO: Total bytes written: 838082560 (800MiB, 21MiB/s) 101: 2020-10-31 20:44:06 INFO: archive file size: 383MB 101: 2020-10-31 20:44:06 INFO: removing backup 'backup-fsn0:backup/vzdump-lxc-101-2020_10_31-20_43_26.tar.lzo' 101: 2020-10-31 20:44:06 INFO: remove vzdump snapshot 101: 2020-10-31 20:44:06 INFO: Finished Backup of VM 101 (00:00:40) 102: 2020-10-31 20:44:06 INFO: Starting Backup of VM 102 (qemu) 102: 2020-10-31 20:44:06 INFO: status = running 102: 2020-10-31 20:44:06 INFO: VM Name: db.bdick.de 102: 2020-10-31 20:44:06 INFO: include disk 'sata0' 'local-zfs:vm-102-disk-0' 8G 102: 2020-10-31 20:44:06 INFO: backup mode: snapshot 102: 2020-10-31 20:44:06 INFO: ionice priority: 7 102: 2020-10-31 20:44:07 INFO: creating vzdump archive '/mnt/pve/backup-fsn0/dump/vzdump-qemu-102-2020_10_31-20_44_06.vma.lzo' 102: 2020-10-31 20:44:07 INFO: started backup task 'edde0ee1-047d-49a4-8b2d-22da79c5b540' 102: 2020-10-31 20:44:07 INFO: resuming VM again 102: 2020-10-31 20:44:10 INFO: 9% (738.9 MiB of 8.0 GiB) in 3s, read: 246.3 MiB/s, write: 45.4 MiB/s 102: 2020-10-31 20:44:13 INFO: 12% (1.0 GiB of 8.0 GiB) in 6s, read: 102.4 MiB/s, write: 89.2 MiB/s 102: 2020-10-31 20:44:16 INFO: 16% (1.3 GiB of 8.0 GiB) in 9s, read: 96.8 MiB/s, write: 82.3 MiB/s 102: 2020-10-31 20:44:19 INFO: 17% (1.4 GiB of 8.0 GiB) in 12s, read: 35.8 MiB/s, write: 26.7 MiB/s 102: 2020-10-31 20:44:22 INFO: 21% (1.7 GiB of 8.0 GiB) in 15s, read: 115.6 MiB/s, write: 77.3 MiB/s 102: 2020-10-31 20:44:25 INFO: 24% (2.0 GiB of 8.0 GiB) in 18s, read: 72.1 MiB/s, write: 71.7 MiB/s 102: 2020-10-31 20:44:28 INFO: 26% (2.1 GiB of 8.0 GiB) in 21s, read: 53.6 MiB/s, write: 53.6 MiB/s 102: 2020-10-31 20:44:31 INFO: 27% (2.2 GiB of 8.0 GiB) in 24s, read: 16.1 MiB/s, write: 16.1 MiB/s 102: 2020-10-31 20:44:34 INFO: 29% (2.4 GiB of 8.0 GiB) in 27s, read: 78.6 MiB/s, write: 73.4 MiB/s 102: 2020-10-31 20:44:37 INFO: 33% (2.7 GiB of 8.0 GiB) in 30s, read: 102.6 MiB/s, write: 71.5 MiB/s 102: 2020-10-31 20:44:40 INFO: 36% (2.9 GiB of 8.0 GiB) in 33s, read: 80.7 MiB/s, write: 80.6 MiB/s 102: 2020-10-31 20:44:43 INFO: 39% (3.1 GiB of 8.0 GiB) in 36s, read: 71.3 MiB/s, write: 71.3 MiB/s 102: 2020-10-31 20:44:46 INFO: 41% (3.3 GiB of 8.0 GiB) in 39s, read: 66.5 MiB/s, write: 66.5 MiB/s 102: 2020-10-31 20:44:49 INFO: 44% (3.5 GiB of 8.0 GiB) in 42s, read: 72.2 MiB/s, write: 72.2 MiB/s 102: 2020-10-31 20:44:52 INFO: 47% (3.8 GiB of 8.0 GiB) in 45s, read: 97.6 MiB/s, write: 97.3 MiB/s 102: 2020-10-31 20:44:55 INFO: 52% (4.2 GiB of 8.0 GiB) in 48s, read: 116.8 MiB/s, write: 55.6 MiB/s 102: 2020-10-31 20:44:58 INFO: 89% (7.2 GiB of 8.0 GiB) in 51s, read: 1023.2 MiB/s, write: 0 B/s 102: 2020-10-31 20:44:59 INFO: 100% (8.0 GiB of 8.0 GiB) in 52s, read: 846.9 MiB/s, write: 0 B/s 102: 2020-10-31 20:44:59 INFO: backup is sparse: 4.92 GiB (61%) total zero data 102: 2020-10-31 20:44:59 INFO: transferred 8.00 GiB in 52 seconds (157.5 MiB/s) 102: 2020-10-31 20:44:59 INFO: archive file size: 1.40GB 102: 2020-10-31 20:44:59 INFO: removing backup 'backup-fsn0:backup/vzdump-qemu-102-2020_10_31-20_44_06.vma.lzo' 102: 2020-10-31 20:44:59 INFO: Finished Backup of VM 102 (00:00:53) > >> Is it possible that I've triggert a very evil bug here? >> > > Can you please post the storage config and the backup job one? I.e., > > # cat /etc/pve/storage.cfg dir: local path /var/lib/vz content backup,iso,vztmpl zfspool: local-zfs pool rpool/data content rootdir,images sparse 1 cifs: backup-fsn0 path /mnt/pve/backup-fsn0 server uXXXXXX-sub1.your-storagebox.de share uXXXXXX-sub1 content backup maxfiles 0 username uXXXXXX-sub1 > > # cat /etc/pve/vzdump.cron # cluster wide vzdump cron schedule # Automatically generated file - do not edit PATH="/usr/sbin:/usr/bin:/sbin:/bin" 0 2 * * 6 root vzdump --mailto bernhard@bdick.de --all 1 --quiet 1 --compress lzo --mode snapshot --mailnotification always --storage backup-fsn0 I can even reproduce this behaviour by triggering the global Backup job from the web console. If I backup single VMs/Containers from the Host part of the web console it runs fine, however the global job removes also those backups when it is running. Regards, Bernhard > > thanks! > - Thomas > >