From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 56E8B67240 for ; Tue, 12 Jan 2021 10:10:23 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 48B37220D6 for ; Tue, 12 Jan 2021 10:10:23 +0100 (CET) Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 7723A220CD for ; Tue, 12 Jan 2021 10:10:21 +0100 (CET) Received: by mail-wr1-x434.google.com with SMTP id w5so1602716wrm.11 for ; Tue, 12 Jan 2021 01:10:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=odiso-com.20150623.gappssmtp.com; s=20150623; h=message-id:subject:from:to:cc:date:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=/pzxRSHDXcS4ypGvE7uhHUJm2YLa63t3epUW8mblF+Q=; b=m0Io0848Ix2QySbheS5fpUBxxX+K8lP9V0vIJ8ADjQodKvA9nzPgSScxSOqtEvSr7J NIaprj90Qw7o453kHsRPlGr6jn+h9zoOgA9NojevSOdfHomebvw/MuLDtTaymfymMJ75 PNOUCNx/EbKbD7RrtcHlMUF7Co4dcrmvFO87CDvUtZ9u8SWWR3IVpEGzPf4rP3K8crl7 wa+RBruN5zQ3+LXKM3QG7S30l+I2iFOxUfkh5eWF/jdEFncd1C7OYBcE57/ToeqbIL1j WOPeXT3z8k4DpxheYrc5WCQOGTyVmbvdRjsRQI8MXxOJBWL/ZABwS0+zq6jSRuwIyp7K yCdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=/pzxRSHDXcS4ypGvE7uhHUJm2YLa63t3epUW8mblF+Q=; b=Vh3pHeQ/wk7c1qCo5SSQdJpOWjVbNYIgjhmH25OpGbkEczu4cC1kTbfzBylHtmig5L O/8DPKMtO2PAO2aXEhpHNLS+aFIDKbOrUyLU4hEdima4YStfsFEWYybiVWHRpVZHlhEU B6bw8oe4ZZtIfuJ8/sL+8NaeKUQWs20pT2n3Bjetta6n+KjSQ91bApf5vHzNQp9zl+Hp EgCo1bGHwyXoc6F9RDr1YEjRfvMHWRMyCZftJIxN5/jVKEnHlYqA0bzPJfJsAwoeWOTz xUyT2PQ3NjHdg25JNXjiPyiu++7WlFxoqTZA945i1DxcFAn2pK6fKK0hbXUcIRjoDzZT SY6A== X-Gm-Message-State: AOAM531D0t4+Lq/tORVKYYFhsy8oZDPuYVPHv23iDp4KCDdX/uaEReQ7 WL8wKAsNHNmr/iqMiy9SaTfJEw== X-Google-Smtp-Source: ABdhPJwf/Kg/lgbL65JwABAwV34PrzzUye/uOit69+8pG2aj+oOQRjgpVC6RWzC9PxMbxCMNjNTf8A== X-Received: by 2002:a5d:69d1:: with SMTP id s17mr2842014wrw.161.1610442615100; Tue, 12 Jan 2021 01:10:15 -0800 (PST) Received: from ?IPv6:2a0a:1580:0:1::100c? (ovpn1.odiso.net. [2a0a:1580:2000::3f]) by smtp.gmail.com with ESMTPSA id k18sm4146749wrd.45.2021.01.12.01.10.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Jan 2021 01:10:14 -0800 (PST) Message-ID: <4d98fee42a16da9ddee8670a633bc40f572446ae.camel@odiso.com> From: aderumier@odiso.com To: Stefan Reiter , pve-devel@lists.proxmox.com Cc: Proxmox VE development discussion Date: Tue, 12 Jan 2021 10:10:12 +0100 In-Reply-To: References: <20210111111409.32385-1-s.reiter@proxmox.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.38.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SPAM-LEVEL: Spam detection results: 0 AWL -0.330 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [restore.rs, qemuserver.pm, proxmox.com, mount.rs] Subject: Re: [pve-devel] [PATCH 00/11] live-restore for PBS snapshots X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2021 09:10:23 -0000 >>What's planned is a fully featured file-restore, including a GUI in PVE  >>(similar to what's already available for CTs in the PBS interface). Wonderful !  Thanks about the "proxmox-backup-client map", I'll try it. Le lundi 11 janvier 2021 à 17:42 +0100, Stefan Reiter a écrit : > On 1/11/21 4:50 PM, aderumier@odiso.com wrote: > > Hi, > > that's great ! > > > > I'm not sure it's related, but do you have any plan to add support > > to > > restore/extract files from the vm filesystem directly ? > > if we could make some kind of qemu-nbd + mount loop of the backup > > volume for example. > > > > Unrelated, but yes, we're working on it. > > At the moment you can already map snapshots via loop devices using > the > map command, e.g.: > proxmox-backup-client map vm/115/2020-10-13T15:03:12Z drive-scsi0.img > # mapped as /dev/loopN > proxmox-backup-client unmap ... > > What's planned is a fully featured file-restore, including a GUI in > PVE > (similar to what's already available for CTs in the PBS interface). > > ~ Stefan > > > > > This is the main blocking feature for me, instead using my rbd > > import/export for backups. > > > > Regards, > > > > Alexandre > >  > Le lundi 11 janvier 2021 à 12:13 +0100, Stefan Reiter a écrit : > > > "live-restore" allows starting a VM immediately from a backup > > > snapshot, no > > > waiting for a long restore process. This is made possible with > > > QEMU > > > backing > > > images, i.e. data is read from the backup which is attached to > > > the VM > > > as a > > > drive, but new data is written to the destination, while a > > > background > > > process > > > ('block-stream') copies over data in a linear fashion as well. > > > > > > QEMU backing images are normally only supported for qcow2 images, > > > but > > > since the > > > destination always starts out empty, we can use a dirty bitmap to > > > achieve the > > > same effect - this is implemented as the 'alloc-track' driver in > > > the > > > 'qemu' part > > > of the series. > > > > > > The Rust part of the equation is adjusted to provide (quiet a > > > lot) > > > more caching, > > > as mixing random read/write from the guest with the linear reads > > > from > > > the > > > background process (both of which may use read sizes smaller or > > > bigger than a > > > single chunk) would thrash performance without large buffers. > > > > > > I've marked the feature as 'experimental' in the GUI for now, as > > > I'm > > > sure there > > > are a lot of edge cases I've missed to test, and there's also the > > > possibility of > > > data loss, since anything the VM writes during the restore is > > > removed > > > if the > > > restore fails. > > > > > > > > > qemu: Stefan Reiter (3): > > >    PVE: explicitly add libuuid as linking dependency > > >    PVE: block/pbs: fast-path reads without allocation if possible > > >    block: add alloc-track driver > > > > > >   Makefile.objs       |   2 + > > >   block/Makefile.objs |   1 + > > >   block/alloc-track.c | 319 > > > ++++++++++++++++++++++++++++++++++++++++++++ > > >   block/pbs.c         |  17 ++- > > >   4 files changed, 336 insertions(+), 3 deletions(-) > > >   create mode 100644 block/alloc-track.c > > > > > > proxmox-backup: Stefan Reiter (1): > > >    RemoteChunkReader: add LRU cached variant > > > > > >   src/bin/proxmox_backup_client/mount.rs |  4 +- > > >   src/client/remote_chunk_reader.rs      | 77 > > > ++++++++++++++++++++---- > > > -- > > >   2 files changed, 62 insertions(+), 19 deletions(-) > > > > > > proxmox-backup-qemu: Stefan Reiter (1): > > >    access: use bigger cache and LRU chunk reader > > > > > >   src/restore.rs | 5 +++-- > > >   1 file changed, 3 insertions(+), 2 deletions(-) > > > > > > qemu-server: Stefan Reiter (5): > > >    make qemu_drive_mirror_monitor more generic > > >    cfg2cmd: allow PBS snapshots as backing files for drives > > >    enable live-restore for PBS > > >    extract register_qmeventd_handle to QemuServer.pm > > >    live-restore: register qmeventd handle > > > > > >   PVE/API2/Qemu.pm         |  15 ++- > > >   PVE/QemuServer.pm        | 263 > > > ++++++++++++++++++++++++++++++++----- > > > -- > > >   PVE/VZDump/QemuServer.pm |  32 +---- > > >   3 files changed, 233 insertions(+), 77 deletions(-) > > > > > > manager: Stefan Reiter (1): > > >    ui: restore: add live-restore checkbox > > > > > >   www/manager6/grid/BackupView.js    |  8 +++++-- > > >   www/manager6/storage/BackupView.js |  7 ++++-- > > >   www/manager6/window/Restore.js     | 38 > > > +++++++++++++++++++++++++++++- > > >   3 files changed, 48 insertions(+), 5 deletions(-) > > > > > > > > > > > _______________________________________________ > > pve-devel mailing list > > pve-devel@lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > > >