From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) by lore.proxmox.com (Postfix) with ESMTPS id 183881FF15C for ; Fri, 8 Aug 2025 15:07:04 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 7B6EA18160; Fri, 8 Aug 2025 15:08:36 +0200 (CEST) To: Fabian =?ISO-8859-1?Q?Gr=FCnbichler?= , pbs-devel@lists.proxmox.com Date: Fri, 08 Aug 2025 15:08:26 +0200 In-Reply-To: <1753708847.wueupzjnbf.astroid@yuna.none> References: <2ac072513fdb726fd0e22fbc79425537dcf63a41.camel@notnullmakers.com> <1753447301.win4nv1cia.astroid@yuna.none> <874cbfc3d659edb7c8ebdda71e77530e437065a2.camel@notnullmakers.com> <1753708847.wueupzjnbf.astroid@yuna.none> MIME-Version: 1.0 Message-ID: List-Id: Proxmox VE development discussion List-Post: From: Adam Kalisz via pve-devel Precedence: list Cc: Adam Kalisz , Proxmox VE development discussion , Thomas Lamprecht X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: , List-Unsubscribe: , List-Archive: Reply-To: Proxmox VE development discussion List-Help: Subject: Re: [pve-devel] [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere Content-Type: multipart/mixed; boundary="===============5385482997169945603==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" --===============5385482997169945603== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id EE4F2D3904 for ; Fri, 8 Aug 2025 15:08:34 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id CDAC618141 for ; Fri, 8 Aug 2025 15:08:34 +0200 (CEST) Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Fri, 8 Aug 2025 15:08:34 +0200 (CEST) Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-acb5ec407b1so311730166b.1 for ; Fri, 08 Aug 2025 06:08:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=notnullmakers.com; s=google; t=1754658507; x=1755263307; darn=lists.proxmox.com; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:from:to:cc:subject :date:message-id:reply-to; bh=/pG91/M+FBPGW4nwESlKK2i8oBmkNcon0IeY18TuWHk=; b=wsD6qZizmWyf16FMxxqqSXQImCjfZgi4ePN+9SH/I65fp47Uyk6ae9PhvJa/BRwJCI D59BgeJiDO/9y1m/w25+dVjg3RIZkwknprNCLMakumBHBUXjDHNp8p6t3tTYqAxXnPHu VfXJOo8QIJ8MWOd2WsOQskFxbDvTUQwHyaXN5ZkdWaqEAUH8O58IUFAKhCBPB/SKogp3 LP3XuYYExvy5qprsf5Oc00Wb8WmisToQ1HA40Ghn2v/5Jj4PYg4zi2xwgbt8Wy9PgDKB mLkO+EU2VX5JcLKJp4IYHKmvbalXtfmsBViBNqFLGQyQQniOY41lNsawhk0TGc3w/mx/ O2nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754658507; x=1755263307; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=/pG91/M+FBPGW4nwESlKK2i8oBmkNcon0IeY18TuWHk=; b=E+/etEMnaoKy2GaHLVxmO0bSO8NqnFNKK0PgpYI77nvzc0vqwYn0dGO6WMRAIa8km2 m28lwenaFetHg9sVyZlRABtyBKlm2I4hYPLWqP25kRn94Q9I//ebfp2Hk/oB0fAMjTlz wXMKzeKMhA+qPpNfnr5xaJBRZSo15dq6+xLPdT92jpQLTaWJ2a0YM8oiUztxDzUDG4fD BgK+PTMOHDIpRNyMOwjv1KGPisBEzIyDNC0o1aTKcZ1q5WZBJCIWAKRKN+2qZNjuryB0 WMi/u26bhTikqYrM+XegpB14Jnyxbkb+xh+vSld+Ff+gnIzXyuXgtH8mrsEO9FAU3xrL mbcA== X-Forwarded-Encrypted: i=1; AJvYcCVVk4BfJ2Yb8lGBAxoDeR2vLK8Cb/cowFYMbMuGpyuyKZyu4c5NXOIT2+cohvFALeZNzDAAnXaYYnU=@lists.proxmox.com X-Gm-Message-State: AOJu0YyRtCwdW4WbOz/ARX7ieAMsPbPhaQCu+g3o2U0X9P5DWJN93wvy C/Goce9aHYBM2m1CVuSKBABTIjD4qGu6ejRk0lMG3A2LJyLPsIF9C0CRyOy1OlyDboI= X-Gm-Gg: ASbGncs6hsiEWO+jY+uCeykONJbtf+OJCJzmBeGgGTjWyqZV/KQlt0VuNhNwhzVu0tS lidqg+TaCcOcVOon47RFz8fV+LVzIufBnpslFY4EaPcsqL79HhueFaTYrhAUmZpFiDnh98r/lHG SWxktj79dRb0zOceFB3aA0KTouQZvBi4UwKnu488ot6jGb+yhoRPx/FMN3l3eHZSBsQhKKEkucV f1RIJrkRNQygQAVWYjWnYQeT7Sf5RtbTFaOkFjJnSYUZf8mkB5TjWUzHTaU3StQshbuguEmNxc9 Q42gMGJws05lPfKgEH1Xp0oYCdV6w1yKncLoiiYMEo/fe+4+lDuNWbBnLVxOFhM2/Xoxn+P4LRZ 63id+WRbD2ilUsqa2F1KCKjs+D2OcNifc2URP49w4/7iAcg== X-Google-Smtp-Source: AGHT+IFLsVC/1c5yGdImb6nwbHBNIhv4pgdT3RN5NHG6Keec6xQvn7Y8S9i7lRGATGKAZEsspvPxPQ== X-Received: by 2002:a17:907:2dac:b0:af9:61b1:e180 with SMTP id a640c23a62f3a-af9c65b4127mr262495366b.41.1754658507303; Fri, 08 Aug 2025 06:08:27 -0700 (PDT) Received: from ?IPv6:2a02:8308:299:4600::9185? ([2a02:8308:299:4600::9185]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-af91a22057esm1484887566b.115.2025.08.08.06.08.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Aug 2025 06:08:26 -0700 (PDT) Message-ID: <010164753bbda5705109d2e192f9d2a361205049.camel@notnullmakers.com> Subject: Re: [pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere From: Adam Kalisz To: Fabian =?ISO-8859-1?Q?Gr=FCnbichler?= , pbs-devel@lists.proxmox.com Cc: Dominik Csapak , Proxmox VE development discussion , Thomas Lamprecht Date: Fri, 08 Aug 2025 15:08:26 +0200 In-Reply-To: <1753708847.wueupzjnbf.astroid@yuna.none> References: <2ac072513fdb726fd0e22fbc79425537dcf63a41.camel@notnullmakers.com> <1753447301.win4nv1cia.astroid@yuna.none> <874cbfc3d659edb7c8ebdda71e77530e437065a2.camel@notnullmakers.com> <1753708847.wueupzjnbf.astroid@yuna.none> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.56.1-1 MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.013 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com,notnullmakers.com] Hi, any news about the chunk verification speed-up? On Mon, 2025-07-28 at 15:22 +0200, Fabian Gr=C3=BCnbichler wrote: > On July 28, 2025 3:14 pm, Adam Kalisz wrote: > > On Fri, 2025-07-25 at 14:46 +0200, Fabian Gr=C3=BCnbichler wrote: > > > On July 25, 2025 1:23 pm, Adam Kalisz wrote: > > > > Similarly the sync performance between two Proxmox Backup > > > > Servers and live-migration got mentioned in various places. > > >=20 > > > do you mean live-restore here? live migration has nothing to do > > > with PBS.. > >=20 > > I threw more things into a single basked as the e-mail was sent to > > both lists and meant live migration that seems like it could go > > faster in some cases. (But that might be code that originates in > > the QEMU project, right?) We could of course have a look at live- > > restore or even some things like migration from VMware/ ESXi too > > (again, unrelated to PBS). >=20 > yes, live migration also has some performance optimizations that we > want to check out, in particular the multi-FD feature: >=20 > https://bugzilla.proxmox.com/show_bug.cgi?id=3D5766 >=20 > live restore uses an entirely different mechanism, and migration from > ESXi uses yet another mechanism that is bottle-necked by the API on > the other side at the moment. The guys over at XCP-ng seem to have switched to VDDK. I just thought I would point to it for reference: "Our test VM used to take 25 minutes to migrate, now it's done in under 2 =F0=9F=A4=AF . Same VM, same setup, just a much better approach. What changed? We stopped using the native ESXi API to pull disks and switched to VDDK (VMware=E2=80=99s own library for accessing VM data), whic= h is way more efficient. Speed is one thing, but it=E2=80=99s not the only win. We now transfer only= the used blocks, without reading the whole disk first, which cuts down migration time significantly. VDDK also brings back warm migration support (even on recent ESXi versions), and keeps the load on the ESXi side barely noticeable (unlike before, where it could freeze the API entirely)." https://www.linkedin.com/feed/update/urn:li:activity:7359195462538604547/ --===============5385482997169945603== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============5385482997169945603==--