From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 144C61FF187 for ; Mon, 28 Jul 2025 15:14:05 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 6E5C33346F; Mon, 28 Jul 2025 15:15:28 +0200 (CEST) To: pbs-devel@lists.proxmox.com Date: Mon, 28 Jul 2025 15:14:47 +0200 In-Reply-To: <1753447301.win4nv1cia.astroid@yuna.none> References: <2ac072513fdb726fd0e22fbc79425537dcf63a41.camel@notnullmakers.com> <1753447301.win4nv1cia.astroid@yuna.none> MIME-Version: 1.0 Message-ID: List-Id: Proxmox VE development discussion List-Post: From: Adam Kalisz via pve-devel Precedence: list Cc: Adam Kalisz , Proxmox VE development discussion , Thomas Lamprecht X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: , List-Unsubscribe: , List-Archive: Reply-To: Proxmox VE development discussion List-Help: Subject: Re: [pve-devel] [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere Content-Type: multipart/mixed; boundary="===============4591639256251760304==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" --===============4591639256251760304== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 73755CB2C1 for ; Mon, 28 Jul 2025 15:15:26 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 592C933431 for ; Mon, 28 Jul 2025 15:14:56 +0200 (CEST) Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 28 Jul 2025 15:14:54 +0200 (CEST) Received: by mail-ej1-x62b.google.com with SMTP id a640c23a62f3a-ae6f8d3bcd4so860476066b.1 for ; Mon, 28 Jul 2025 06:14:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=notnullmakers.com; s=google; t=1753708488; x=1754313288; darn=lists.proxmox.com; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:from:to:cc:subject :date:message-id:reply-to; bh=WfpataUVI4dSypFSJWdTka3TOX/4Uw8F9e8jxg7lOVM=; b=1EPvgDPi0zxMidyKiPnhQyma7OtfrlvWh2efxFD3RvWtxy1BNcZoppqIOxB5EGbAzp Ag2s7D+4wdsDT733zU5e6soCxa02X2WlJoItZ46LJoe+N++7UTKRzIVgEiw2TUueNWZy yfq0mya+xT/c7+ygm0A169o1wACm932QOyb0rygl6DEsLTOHFCPRpRzW2EhvRQYOuMLr latpDO9wjFa/ELAK6UtXmWBxWvbQUIh180ApK5BAmRc0aXS6+1fVO1CpLeTeV48hfXP8 3aDMCLPsUwsXWGBJ20s4sMuWIROJW8fuUL4GfGIVOOXLdK1u87dZHvdgbS4+dg62txeG 8xJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753708488; x=1754313288; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=WfpataUVI4dSypFSJWdTka3TOX/4Uw8F9e8jxg7lOVM=; b=mUn6i6bFQ6Jng28SGARcthDqrSN4Z69dUD3HomGkb76UfQQsXqKoK9owB+YEAuIRSG 7Msdf9RD7wLADFW+CpItnzsq7eKOKHguPiOgTlxzYWgHbopStE/+bCJDBiKF7h5Ka2/q WjFf4BwcA3cUXtk9gw9BiO8CP9n2uljV18Qawyquta1YBMcZ2IdC5pSWeM3WXYDBwEjO f1Bm/NYOD7XRFtCW7NHjdXq8lDRQ6qEjNGtCzYZgrts5ICPdcKUvgBTIHiagNrGLjkel +3QvdomITJ0bNVyZrduX9JLdjYi8u8LPZqHw8fQbv9F9G3JXVZooAdoZg+9B/YnTomPd Rygg== X-Gm-Message-State: AOJu0YytiMWR8TPehWzYoZz7atQFChdfiAf1iZo9PtWSpeGGBLCwTymd 1BAra5ojCpl5ahI4fLe0tLpYyBsbqo2QWiM2csyaLmFGeXnNWBlRgsuFYQm/OJip/tY= X-Gm-Gg: ASbGnctLfGh43DowWcP7ZaTOdzM1UaLXE99vZwnk+rAmx23iwTOzCGCV97e0LeM5Xtc ew16ssiPBLsIGx9gMmA1Tm0UIB1zhFzKw98WLFEZ97rss87Mu9Ze8W961V5f9iOnkxqMMj35Ghx wTCysj7hEUXLfAaGtpIUbg710pnyGZ07QaPgRHB8jchVJoP3ze+gPfWvjdfZblHKrGgrMkeJ9Hp xcTiq+Mh5I/KbmZdob7vd8oNKrlrYBZVgClM+XmylH7riH9GiBpyXJGMShmHTLY8/qKQWwZisYU SD2K/ttN4JzuzPkMFgFudLZKiA/RiuR8zWkwHc4fV1SHdkPeMGNqa4gb9laUca+LAnEqgY3RsYA k2ol5IglpoSggL4QSQbsqsXa0ZZ1dF6hrtDA5yKexO03S X-Google-Smtp-Source: AGHT+IEnsMMmXYDcxN/oZRJdab44E/Rhsm6QwMG5Ccurpdm/iJVx7tem8IrAUNeesoEGIUNJrsFX9w== X-Received: by 2002:a17:906:fe0c:b0:ad8:96d2:f3a with SMTP id a640c23a62f3a-af616efc784mr1119857166b.8.1753708487938; Mon, 28 Jul 2025 06:14:47 -0700 (PDT) Received: from ?IPv6:2001:1ae9:104a:f400::ce0? ([2001:1ae9:104a:f400::ce0]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-af6358600e5sm424121166b.26.2025.07.28.06.14.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jul 2025 06:14:47 -0700 (PDT) Message-ID: <874cbfc3d659edb7c8ebdda71e77530e437065a2.camel@notnullmakers.com> Subject: Re: [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere From: Adam Kalisz To: pbs-devel@lists.proxmox.com Cc: Proxmox VE development discussion , Fabian =?ISO-8859-1?Q?Gr=FCnbichler?= , Thomas Lamprecht , Dominik Csapak Date: Mon, 28 Jul 2025 15:14:47 +0200 In-Reply-To: <1753447301.win4nv1cia.astroid@yuna.none> References: <2ac072513fdb726fd0e22fbc79425537dcf63a41.camel@notnullmakers.com> <1753447301.win4nv1cia.astroid@yuna.none> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.56.1-1 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record On Fri, 2025-07-25 at 14:46 +0200, Fabian Gr=C3=BCnbichler wrote: > On July 25, 2025 1:23 pm, Adam Kalisz wrote: > > Hi list, > >=20 > > I missed whether the chunk verification speedup when loading chunks > > got > > applied or whether it was somehow included in the S3-like storage > > option change set. >=20 > https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@prox= mox.com/ >=20 > hasn't been applied (and probably needs a rebase post-S3 ;)) Yes and the opt-out knob that Thomas mentioned on Friday should be added: "So the mentioned series should IMO only be applied if it can be opted out, or, better in the long term, a pacing/IO bandwidth limit can be enforced. That is naturally a big amount of work, mostly in evaluating different options and deciding for what can make sense, that's why I mentioned that being able to opt-out could be enough, then admins could at least keep the status quo." > > In > > https://forum.proxmox.com/threads/abysmally-slow-restore-from-backup.13= 3602/page-7 > > we have discussed some other opportunities for speedup using > > similar > > patterns. People mentioned LXC container restore speed and=C2=A0 host- > > based > > backup restore, which looking at the code for the latter seems like > > a > > similar async loop pattern would bring some improvement without too > > much trouble: > >=20 > > https://github.com/proxmox/proxmox-backup/blob/4940514b0f05d6cd6a5f711e= dfdd47c1fa41b537/proxmox-backup-client/src/main.rs#L1109 >=20 > that's not the code path for container/host backups, that's the code > path for fixed-size indices like vm backups, but just dumping to a > file instead of via qmrestore.. i.e., nothing that is used by PVE or > users usually, but can still be improved of course ;) Ah ok, I just skimmed over it to find patterns that look like the things Dominik and I improved before elsewhere. > > Similarly the sync performance between two Proxmox Backup Servers > > and > > live-migration got mentioned in various places. >=20 > do you mean live-restore here? live migration has nothing to do with > PBS.. I threw more things into a single basked as the e-mail was sent to both lists and meant live migration that seems like it could go faster in some cases. (But that might be code that originates in the QEMU project, right?) We could of course have a look at live-restore or even some things like migration from VMware/ ESXi too (again, unrelated to PBS). I just want to confirm if any of this makes sense and where to look basically so we can have a list of areas for improvement and what code path that is. The speed improvements already caused qualitative improvements that e.g. allow some companies to choose PVE+PBS instead of other options even in more demanding cases. Thank you for bringing me up to speed. Have a successful week! --===============4591639256251760304== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============4591639256251760304==--