From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <s.reiter@proxmox.com> Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 73F5D62414 for <pve-devel@lists.proxmox.com>; Thu, 20 Aug 2020 15:49:00 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 63D9A2D95E for <pve-devel@lists.proxmox.com>; Thu, 20 Aug 2020 15:49:00 +0200 (CEST) Received: from proxmox-new.maurer-it.com (proxmox-new.maurer-it.com [212.186.127.180]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 9AEFA2D946 for <pve-devel@lists.proxmox.com>; Thu, 20 Aug 2020 15:48:59 +0200 (CEST) Received: from proxmox-new.maurer-it.com (localhost.localdomain [127.0.0.1]) by proxmox-new.maurer-it.com (Proxmox) with ESMTP id 5C9FA4499C for <pve-devel@lists.proxmox.com>; Thu, 20 Aug 2020 15:48:59 +0200 (CEST) From: Stefan Reiter <s.reiter@proxmox.com> To: pve-devel@lists.proxmox.com Date: Thu, 20 Aug 2020 15:48:35 +0200 Message-Id: <20200820134837.27407-1-s.reiter@proxmox.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.056 Adjusted score from AWL reputation of From: address KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment RCVD_IN_DNSWL_MED -2.3 Sender listed at https://www.dnswl.org/, medium trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: [pve-devel] [RFC 0/2] Add job transaction support for backup X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com> List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe> List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/> List-Post: <mailto:pve-devel@lists.proxmox.com> List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help> List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>, <mailto:pve-devel-request@lists.proxmox.com?subject=subscribe> X-List-Received-Date: Thu, 20 Aug 2020 13:49:00 -0000 RFC since very experimental and only lightly tested. Our backup starts one QEMU block job per drive that is included in the final archive. Currently, we start them all in 'paused' state, manually calling job_start for the next one we whenever one calls its completion callback. By using a transaction, we can automate that part, meaning we need a lot less code in pve-backup.c. Additionally, we gain the benefit of having transactional backups, which is important for dirty-bitmaps, since it allows us to keep all bitmaps even when a backup fails or is aborted. QEMU does the heavy lifting here, by already having support for transactions in backup jobs and handling the bitmaps accordingly. A small change to the job transaction API of QEMU is necessary (patch 1) to allow the jobs to run in sequence, instead of all at once. This property is good to have for several reasons, like not overloading a server, writing VMA files in sequential order and supporting bandwidth limiting (which only works per job in QEMU). qemu: Stefan Reiter (2): PVE: Add sequential job transaction support PVE-Backup: Use a transaction to synchronize job states include/qemu/job.h | 12 ++++ job.c | 24 +++++++ pve-backup.c | 169 ++++++++++++++------------------------------- 3 files changed, 86 insertions(+), 119 deletions(-) -- 2.20.1