From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id D3A206710A for ; Tue, 25 Aug 2020 15:16:16 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id AFC5627640 for ; Tue, 25 Aug 2020 15:15:46 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 2A87B2761F for ; Tue, 25 Aug 2020 15:15:45 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id E405744878 for ; Tue, 25 Aug 2020 15:15:44 +0200 (CEST) From: Stefan Reiter To: pve-devel@lists.proxmox.com Date: Tue, 25 Aug 2020 15:15:35 +0200 Message-Id: <20200825131538.26359-1-s.reiter@proxmox.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.053 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [RFC v2 0/3] Add job transaction support for backup X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Aug 2020 13:16:16 -0000 Still RFC and experimental, though with more (successful) testing since v1. Our backup starts one QEMU block job per drive that is included in the final archive. Currently, we start them all in 'paused' state, manually calling job_start for the next one we whenever one calls its completion callback. By using a transaction, we can automate that part, meaning we need a lot less code in pve-backup.c. Additionally, we gain the benefit of having transactional backups, which is important for dirty-bitmaps, since it allows us to keep all bitmaps even when a backup fails or is aborted. QEMU does the heavy lifting here, by already having support for transactions in backup jobs and handling the bitmaps accordingly. A small change to the job transaction API of QEMU is necessary (patch 2) to allow the jobs to run in sequence, instead of all at once. This property is good to have for several reasons, like not overloading a server, writing VMA files in sequential order and supporting bandwidth limiting (which only works per job in QEMU). v2: * revert/remove existing 'PVE-Backup: modify job api' qemu: Stefan Reiter (3): Revert "PVE-Backup: modify job api" PVE: Add sequential job transaction support PVE-Backup: Use a transaction to synchronize job states block/backup.c | 3 - block/replication.c | 2 +- blockdev.c | 3 +- include/block/block_int.h | 1 - include/qemu/job.h | 12 +++ job.c | 26 +++++- pve-backup.c | 169 +++++++++++--------------------------- 7 files changed, 89 insertions(+), 127 deletions(-) -- 2.20.1