From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 3BF776712E for ; Mon, 11 Jan 2021 16:50:25 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 2FBAB1D034 for ; Mon, 11 Jan 2021 16:50:25 +0100 (CET) Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 95C6E1D022 for ; Mon, 11 Jan 2021 16:50:23 +0100 (CET) Received: by mail-wr1-x42f.google.com with SMTP id y17so230906wrr.10 for ; Mon, 11 Jan 2021 07:50:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=odiso-com.20150623.gappssmtp.com; s=20150623; h=message-id:subject:from:to:date:in-reply-to:references:user-agent :mime-version:content-transfer-encoding; bh=x84+6f3WMY6kZtQlA63VBSPLVIpsrHIex5CaNH7neOE=; b=Dg6+40aTvcoAlAhLv+KUxVbJyTpQmJnYg8YZoe4Svbt4QDL9DR4kWnw43cB0d3LVFd SZfNpLjYlSt5+9zc2wbk1RthmEC1+iJm2rW3vTq8h2rBI7Mtq6yVPJm1BjplNHO+aTel /KVPVwhtPaDDGxTUEPCGhcMs+sZLd+6jprVtbRpKtHilymq6zoi2tSHJZOnk0NBpIDBY iep5AziD8p0umJl07W2lhyA6jnFjrg54W3Sh20dQAh+Vv3F3iXUX4JZCGHuYRGKIGz+R 2MHT777syMXE9EPOxSKPwA/TIu9uSlXkGmrJILstNdprDisGbNhNt/dAtEPV+uEy9Wel t08Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=x84+6f3WMY6kZtQlA63VBSPLVIpsrHIex5CaNH7neOE=; b=UMoHKhjFc/M0n5uq9YQ1g46XSEPlNJl/LYK6pLYGF2sp2GEydRyeNhrsbZ/0lkGEIA YGpWaIRtaM9Zj0n1Xq3fr8d7LPGW1rF52KXDfbfu8xfe5mT7dbWpeOfkc3G/JGogrI++ BLMo8YTIw4jFXWIATQK0olHx09/MBzGjVOFI1f8990Y0cJKCNI/IDJJmFzJnzxImqL5J 8YD2pEbU7w2DyccWf5cPQQ3PodabD7G5MRvhwIMbMlN4Yqo3l75CbZjExIxkxNIdL1GD EpLUfNHD7t2NbeRKMBJV6iZ0O+gfxe5Wn2JLcNZgZ1juh3JPViqu6Pls1nXkOieANY0r KqXg== X-Gm-Message-State: AOAM533DpC1GZidnlxZ/O+FpVduCJuUzuKsOzIqNAB5pgXKHSl7zMDT7 g9kUUD3Zq3V1E9hf5OTwbhsl8kgxT6c8Hqm1x2g= X-Google-Smtp-Source: ABdhPJwnG6M+n2Gr247PuDlmIBYHUAbAUfed418VNZEiXDdctVwgoMWHfrEGO0RRGJxGeXDi2uWItQ== X-Received: by 2002:adf:dc8d:: with SMTP id r13mr16892963wrj.325.1610380217069; Mon, 11 Jan 2021 07:50:17 -0800 (PST) Received: from ?IPv6:2a0a:1580:0:1::100c? (ovpn1.odiso.net. [2a0a:1580:2000::3f]) by smtp.gmail.com with ESMTPSA id r15sm46510wrq.1.2021.01.11.07.50.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Jan 2021 07:50:16 -0800 (PST) Message-ID: From: aderumier@odiso.com To: Proxmox VE development discussion , pbs-devel@lists.proxmox.com Date: Mon, 11 Jan 2021 16:50:14 +0100 In-Reply-To: <20210111111409.32385-1-s.reiter@proxmox.com> References: <20210111111409.32385-1-s.reiter@proxmox.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.38.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.412 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Subject: Re: [pve-devel] [PATCH 00/11] live-restore for PBS snapshots X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Jan 2021 15:50:25 -0000 Hi, that's great ! I'm not sure it's related, but do you have any plan to add support to restore/extract files from the vm filesystem directly ?  if we could make some kind of qemu-nbd + mount loop of the backup volume for example. This is the main blocking feature for me, instead using my rbd import/export for backups. Regards, Alexandre Le lundi 11 janvier 2021 à 12:13 +0100, Stefan Reiter a écrit : > "live-restore" allows starting a VM immediately from a backup > snapshot, no > waiting for a long restore process. This is made possible with QEMU > backing > images, i.e. data is read from the backup which is attached to the VM > as a > drive, but new data is written to the destination, while a background > process > ('block-stream') copies over data in a linear fashion as well. > > QEMU backing images are normally only supported for qcow2 images, but > since the > destination always starts out empty, we can use a dirty bitmap to > achieve the > same effect - this is implemented as the 'alloc-track' driver in the > 'qemu' part > of the series. > > The Rust part of the equation is adjusted to provide (quiet a lot) > more caching, > as mixing random read/write from the guest with the linear reads from > the > background process (both of which may use read sizes smaller or > bigger than a > single chunk) would thrash performance without large buffers. > > I've marked the feature as 'experimental' in the GUI for now, as I'm > sure there > are a lot of edge cases I've missed to test, and there's also the > possibility of > data loss, since anything the VM writes during the restore is removed > if the > restore fails. > > > qemu: Stefan Reiter (3): >   PVE: explicitly add libuuid as linking dependency >   PVE: block/pbs: fast-path reads without allocation if possible >   block: add alloc-track driver > >  Makefile.objs       |   2 + >  block/Makefile.objs |   1 + >  block/alloc-track.c | 319 > ++++++++++++++++++++++++++++++++++++++++++++ >  block/pbs.c         |  17 ++- >  4 files changed, 336 insertions(+), 3 deletions(-) >  create mode 100644 block/alloc-track.c > > proxmox-backup: Stefan Reiter (1): >   RemoteChunkReader: add LRU cached variant > >  src/bin/proxmox_backup_client/mount.rs |  4 +- >  src/client/remote_chunk_reader.rs      | 77 ++++++++++++++++++++---- > -- >  2 files changed, 62 insertions(+), 19 deletions(-) > > proxmox-backup-qemu: Stefan Reiter (1): >   access: use bigger cache and LRU chunk reader > >  src/restore.rs | 5 +++-- >  1 file changed, 3 insertions(+), 2 deletions(-) > > qemu-server: Stefan Reiter (5): >   make qemu_drive_mirror_monitor more generic >   cfg2cmd: allow PBS snapshots as backing files for drives >   enable live-restore for PBS >   extract register_qmeventd_handle to QemuServer.pm >   live-restore: register qmeventd handle > >  PVE/API2/Qemu.pm         |  15 ++- >  PVE/QemuServer.pm        | 263 ++++++++++++++++++++++++++++++++----- > -- >  PVE/VZDump/QemuServer.pm |  32 +---- >  3 files changed, 233 insertions(+), 77 deletions(-) > > manager: Stefan Reiter (1): >   ui: restore: add live-restore checkbox > >  www/manager6/grid/BackupView.js    |  8 +++++-- >  www/manager6/storage/BackupView.js |  7 ++++-- >  www/manager6/window/Restore.js     | 38 > +++++++++++++++++++++++++++++- >  3 files changed, 48 insertions(+), 5 deletions(-) >