From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <pve-devel-bounces@lists.proxmox.com>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
by lore.proxmox.com (Postfix) with ESMTPS id EA2C51FF170
for <inbox@lore.proxmox.com>; Sat, 27 Jul 2024 17:09:00 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 49D5F3752;
Sat, 27 Jul 2024 17:08:59 +0200 (CEST)
Date: Fri, 26 Jul 2024 15:47:12 -0400
To: pve-devel@lists.proxmox.com
X-Mailman-Approved-At: Sat, 27 Jul 2024 17:08:57 +0200
MIME-Version: 1.0
Message-ID: <mailman.0.1722092937.302.pve-devel@lists.proxmox.com>
List-Id: Proxmox VE development discussion <pve-devel.lists.proxmox.com>
List-Post: <mailto:pve-devel@lists.proxmox.com>
From: Jonathan Nicklin via pve-devel <pve-devel@lists.proxmox.com>
Precedence: list
Cc: Jonathan Nicklin <jnicklin@blockbridge.com>
X-Mailman-Version: 2.1.29
X-BeenThere: pve-devel@lists.proxmox.com
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=subscribe>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-devel>,
<mailto:pve-devel-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-devel/>
Reply-To: Proxmox VE development discussion <pve-devel@lists.proxmox.com>
List-Help: <mailto:pve-devel-request@lists.proxmox.com?subject=help>
Subject: Re: [pve-devel] [RFC qemu/storage/qemu-server/container/manager
00/23] backup provider API
Content-Type: multipart/mixed; boundary="===============3965623324821893881=="
Errors-To: pve-devel-bounces@lists.proxmox.com
Sender: "pve-devel" <pve-devel-bounces@lists.proxmox.com>
--===============3965623324821893881==
Content-Type: message/rfc822
Content-Disposition: inline
Return-Path: <jnicklin@blockbridge.com>
X-Original-To: pve-devel@lists.proxmox.com
Delivered-To: pve-devel@lists.proxmox.com
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits))
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 5505DC3AFA
for <pve-devel@lists.proxmox.com>; Fri, 26 Jul 2024 21:47:23 +0200 (CEST)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 3170A3345F
for <pve-devel@lists.proxmox.com>; Fri, 26 Jul 2024 21:47:23 +0200 (CEST)
Received: from mail-qk1-x732.google.com (mail-qk1-x732.google.com [IPv6:2607:f8b0:4864:20::732])
(using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pve-devel@lists.proxmox.com>; Fri, 26 Jul 2024 21:47:22 +0200 (CEST)
Received: by mail-qk1-x732.google.com with SMTP id af79cd13be357-7a1d6f4714bso83345785a.1
for <pve-devel@lists.proxmox.com>; Fri, 26 Jul 2024 12:47:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=blockbridge.com; s=google; t=1722023234; x=1722628034; darn=lists.proxmox.com;
h=to:date:message-id:subject:mime-version:content-transfer-encoding
:from:from:to:cc:subject:date:message-id:reply-to;
bh=sHjy2b6XOiNziIjL70i59JTAzjGFvfYq8SrapqQoVLU=;
b=IGZ13fKCFdvm7iPTzXqk8AKJjd8eN8SVtG5roCtW1w1e4YkJBbaFPYZLf63/VCAYBQ
SODbLPnv9vaMd1Nd8VGl4/o7mUvJtJnG3WXpBpMXV/hxzDyLVLbNwJFK9CKhD8zvuX/G
BmtyqlLpqbWW72S09qXBgb1wN7DRtiTPFQyrh8mf+huq43f8A6fp2hHq6YPfEMVmGl5W
/DvTylIz0C/GCcQkerZbuMSDzyPBy+3MXt9vaLUKauqrSzU3jL1nw7QGYSUckDwPxAwg
WLWoxeFOsG/+Ifk1i+TSj1uv/TX4/iDUP+qbfbXZ9YHT3b2+UMyV+nFVGlTSVi+IZk9q
3XeQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1722023234; x=1722628034;
h=to:date:message-id:subject:mime-version:content-transfer-encoding
:from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
bh=sHjy2b6XOiNziIjL70i59JTAzjGFvfYq8SrapqQoVLU=;
b=tz794wBf20KeQBg+5rTE7BTdAfES7qwWLnpR3cW90oSRGAKrpIWTfk5tXxj762d1VP
YFYACIqWtRI14CLseQ706RIRldox0Nr6fFnoS9SGESck0TiLtQ5dr2nMG7Y4BckVzBtt
YcpZe7jcoEehNOXvOw+bWAXbNkaAZBnntogO0/gMk5CcGCtjgJzQCPlsmfWeY/amZGFT
SHt3x1I8qwCpF9JPMh0Dop1mapBuoUaBqsnBCo210eTw+hvWaZjk/8DJT0j+EYPwVEsV
TGd7hMibJ2eIYCpZH5oryrRduWzdC7VdZ+85kiF1p0v0Rlxwp1ZX/M7ywfuOgu0vyiX1
MRwQ==
X-Gm-Message-State: AOJu0Yx1MB6BF7943rz8iH1w7k8at5G3Zf2llKNXWK3CHwoA1NQ0DV5s
uaGFeWE/yMoOKFxt3PeNrh2g2XtURohyckei2pR4WfJYBaSeeGTtK4j9fAhjTxXGHWfJcEw++NM
c
X-Google-Smtp-Source: AGHT+IG6t8or/PNzxpkKrfgG8CVqb0DUdOuAFuwTxMFzhJNyHHq5AN+u/stxdOZ3Eve3ex7lYpIP4A==
X-Received: by 2002:a05:620a:c50:b0:79c:1345:a7f0 with SMTP id af79cd13be357-7a1e56a0298mr113305885a.27.1722023234000;
Fri, 26 Jul 2024 12:47:14 -0700 (PDT)
Received: from smtpclient.apple (pool-108-7-52-138.bstnma.fios.verizon.net. [108.7.52.138])
by smtp.gmail.com with ESMTPSA id af79cd13be357-7a1d73b500fsm208046185a.41.2024.07.26.12.47.13
for <pve-devel@lists.proxmox.com>
(version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
Fri, 26 Jul 2024 12:47:13 -0700 (PDT)
From: Jonathan Nicklin <jnicklin@blockbridge.com>
Content-Type: text/plain;
charset=us-ascii
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\))
Subject: Re: [pve-devel] [RFC qemu/storage/qemu-server/container/manager
00/23] backup provider API
Message-Id: <6B38485B-5520-4E28-824D-E50C699E96A1@blockbridge.com>
Date: Fri, 26 Jul 2024 15:47:12 -0400
To: pve-devel@lists.proxmox.com
X-Mailer: Apple Mail (2.3693.20.0.1.32)
X-SPAM-LEVEL: Spam detection results: 0
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid
DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature
DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain
DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain
DMARC_PASS -0.1 DMARC pass policy
FAKE_REPLY_B 0.918 -
RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust
SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record
SPF_PASS -0.001 SPF: sender matches SPF record
URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [blockbridge.com]
X-Mailman-Approved-At: Sat, 27 Jul 2024 17:08:57 +0200
Hi Fiona,
Would adding support for offloading incremental difference detection
to the underlying storage be feasible with the API updates? The QEMU
bitmap strategy works for all storage devices but is far from
optimal. If backup coordinated a storage snapshot, the underlying
storage could enumerate the differences (or generate a bitmap).
This would allow PBS to connect directly to storage and retrieve
incremental differences, which could remove the PVE hosts from the
equation. This "storage-direct" approach for backup would improve
performance, reduce resources, and support incremental backups in all
cases (i.e., power failues, shutdowns, etc.). It would also eliminate
the dependency on QEMU bitmaps and the overhead of fleecing.
Theoretically, this should be possible with any shared storage that
can enumerate incremental differences between snapshots: Ceph,
Blockbridge, iSCSi/ZFS?
Thoughts?
--===============3965623324821893881==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
--===============3965623324821893881==--