From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id EB84E6A122 for ; Wed, 9 Dec 2020 03:07:49 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id D99C1ACB8 for ; Wed, 9 Dec 2020 03:07:19 +0100 (CET) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-db3eur04on0601.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe0c::601]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 06675ACAF for ; Wed, 9 Dec 2020 03:07:18 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JUlQM7LMI4mES/yoqjOsDlxOfMcQISKVD7N6FwkIKMGE59qQDXnSpFxiD9rBLMRxYnQPIMma4LTOGi/WQd1njtsJ3d/qAwbExgMDlDRS3ym/kAeRGn6JwY8iX6uoD6CrX+IAzusvmSsEnUG2BkwwjhhU/uyej3V+N59svNpwMFjJ73BR1gzH6DnrUL/+Mz6y1vWQCjBBjzx4g8d3X9odGrA9Zm3J1l6vCXEs4MLNRAmcYGiErEx9SnuFFbszJav9NOwaqBHwpLI2fm0XCpHwfSBAjX2aF4Bc+my0iMqlvEoaLsLmi0CcclY3EALnrNem8lzDiMwhh6GnRIia9yXckg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rWd54irv5YmWO+sApE75ER613McphxNDoO5lcsfTQNQ=; b=b90vugwowcpNiucMvlcxNL7PY1EBAud2Db5d8TmEJCH+NLNnXEJ6Z4frvt96V/lNFdIJYkqvVLEHD04VDS4C9mED62VMpI4IWi+KjgQwet0tPTKD+F4fLHzhrQiUFaSPggRN57FugSw8maMv66YV28GC6I0bhq3Q4aWxl4KXnVQzxGoSNCLcKyYH1ofEGvakDdmFYoOQPaxG9H6C667SBoYJCeBRGu27bnCssrdkI8NpgzzxrCK1IjR8cuOogKteXyDO8a6GVlYCrP3vW8PKua/Nzi63NqeChI7Kg93qaZjU1lPTomfMaBXnfECw+Bt3acrgNdezv+djbmhHw+nRuA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=logics.de; dmarc=pass action=none header.from=logics.de; dkim=pass header.d=logics.de; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=logics99.onmicrosoft.com; s=selector2-logics99-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rWd54irv5YmWO+sApE75ER613McphxNDoO5lcsfTQNQ=; b=cSQqPFsUthKri/XDAu147Mzu+Lw3q0tE5UqomBQnv6J851jjgcgf0wh0K9FTfOuq0dd0tKqvGo+ElZ2YVnRsccbd3x3dnGJPcbZzZTFGkApe4+DohHOVJqPK3mL8oyYUNwAHzZHj3FuOLK3TVHridU3Rnw/s644criHuvdGmZO0= Received: from AM0PR09MB2754.eurprd09.prod.outlook.com (2603:10a6:208:12e::18) by AM0PR09MB4145.eurprd09.prod.outlook.com (2603:10a6:208:191::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.18; Wed, 9 Dec 2020 02:07:11 +0000 Received: from AM0PR09MB2754.eurprd09.prod.outlook.com ([fe80::68c5:8966:920c:8d24]) by AM0PR09MB2754.eurprd09.prod.outlook.com ([fe80::68c5:8966:920c:8d24%3]) with mapi id 15.20.3632.023; Wed, 9 Dec 2020 02:07:11 +0000 From: Niko Fellner To: "pbs-devel@lists.proxmox.com" Thread-Topic: Re: [pbs-devel] parallelize restore.rs fn restore_image: problems in Thread-Index: AdbNyi1MmemiS292SjySQUTVrL7ZIg== Date: Wed, 9 Dec 2020 02:07:11 +0000 Message-ID: Accept-Language: de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: lists.proxmox.com; dkim=none (message not signed) header.d=none;lists.proxmox.com; dmarc=none action=none header.from=logics.de; x-originating-ip: [93.104.29.163] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 7db9193c-954a-4f37-8849-08d89be72428 x-ms-traffictypediagnostic: AM0PR09MB4145: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3513; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: ZFlsLuS+zI/7lvGQirHm99EtdcLIs0r9FbKm4fK7OYOAFpXczaKdM2cyJzAAZ4Pm+TlHl6RHq9c98MvNiMaiHh5uy1C7kjb5byUd9UQBHbw4BcH3ecsp4f3o1g5uOjjZ1o39HRxPrQQ4ng6Woei2VQdJrkpjS7RhM+S48Mtvn1GnjFlpSHk18Fbi19XwGOVevXqLRicGpRAG+k4tLqsvj4PtpYCT48wZa/0D+x24gP3qrOeEWLFZpNI5IqQDdaT3hhu2z6foJUmewVvSZBVKrEWXb8zIDhlzcgBU1umgoWHfsBoU/G/8hBIzN9m7T08ClcfjjPQiM4qsCtza2eRwvfRLMB3T3vDZW3eDkjsGMOh7tAUuPzjTDbcDeBrYjPMd x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR09MB2754.eurprd09.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(376002)(136003)(346002)(6916009)(508600001)(8676002)(33656002)(55016002)(26005)(71200400001)(186003)(64756008)(9686003)(66446008)(66476007)(66556008)(7696005)(52536014)(83380400001)(86362001)(6506007)(2906002)(76116006)(66946007)(5660300002)(8936002)(14943795004); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?wlv4IeAtnlk3cGTjatvNd3sMNDhw9RusdN8fFksSwImY3f5SAh9ahW1cPI8T?= =?us-ascii?Q?+WJ9uJWd7Hn9H8TntS3V46/lTpshhPe5EyE3sZwr4cYPIw+mdP1A49rUJQ6/?= =?us-ascii?Q?NImkfKaQX/7636wpDEFHMsYW2XIoDjep2Y0Ltyy2+CSEv+ZY7nvP3HiW9VDs?= =?us-ascii?Q?XOS/ClRDK9HspOlTcHd61jjYhV23Sr3dna/iy2Va287VDkmr+K4Cm2G75DHp?= =?us-ascii?Q?bhKkVn9Z7toSv+memyOqI7wHQcItGuu6v7wQY8QLjMFQmB7vHCRG9aGxUj/e?= =?us-ascii?Q?kBEuyTXq9fDBAGSO9NZZKQxhejP4uEAEZIZThRbH0lnpg3XG34DH8jFBhd6C?= =?us-ascii?Q?hl6YBIhQCrBkV6F/YX+eexw2LXHqM1GprcJRTH6VGhPiWFuP3RuzYdi6cDu8?= =?us-ascii?Q?xqIQd+qHSTdI0SOxVE+ERVvDpr2gIrsiY8m2wkfyqWo/p7xCIloBaG9tSKBi?= =?us-ascii?Q?EZDhvjZf6JLo0pYWpGbZvXxqQryzViMahyQT9cJ71Figj7tmuDHAMMG9aKW0?= =?us-ascii?Q?uczdUCcD5+Jimeka6OjLdM+joL0a6H5ydp+D+h+uG3oDkHw2J3EEVRTlmK8A?= =?us-ascii?Q?OQmI9k+uAFzv0lxdq3RDEbYKjuh58w4cXdLwgUtmTq9lYvvkG2DrGG2Sygyb?= =?us-ascii?Q?0iBG3ifl+ahEo/nEaaTNjJMjmL5NiPGxwiHaEEn9cXME9oBAXIdCBohdIYUP?= =?us-ascii?Q?IMQiUISfAP7nZ5Qmm1756Am8J9xZQw4FQVPGIspeWdyomECXl/mNJBF8lkiN?= =?us-ascii?Q?AAmq+7mvBRfvok0CTWsnuCHFnUBCPUsAbzVSAlk+J9kl6EGl4sqAFY1JJgB1?= =?us-ascii?Q?Yxob2I89ASnC0I4k5T2PUDujGWAByr3DpC8kEgHy98em9+wcAQGArJqUffzZ?= =?us-ascii?Q?ZuxvROMl5Daz5Z2yK/5nbIEAS3j/x5jYRYqxvwUo2ZTsCEY+Reb5nxysVsZc?= =?us-ascii?Q?z7LRVMZrX3Kxfy2br/CSjtCO+7SmrXMdfp1h/xFNFFU=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: logics.de X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AM0PR09MB2754.eurprd09.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7db9193c-954a-4f37-8849-08d89be72428 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Dec 2020 02:07:11.8582 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: fdfa9215-653f-430f-b8ab-a8728140f97a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: KhCeSEPNoQBENGBs5mW+gAEKnG+GzySbslvHv1xcx+ca/N6Du6xCAym0rfwf5S6MJm7Bhco+LVWyPb4K0NTQM02uJkr59MZmIfN6dLf6910= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR09MB4145 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.033 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [restore.rs, logics99.onmicrosoft.com] Subject: Re: [pbs-devel] parallelize restore.rs fn restore_image: problems in X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Dec 2020 02:07:50 -0000 I did some benchmarks now at an Azure L32s_v2 (32 vCPUs, 256 GB memory, 4x = 2 TB NVMe; on tests with ZFS I reduced the zfs-arc-max to 8 GiB, because I = didn't want to test the memory only) Overall the speedup of the parallel pbs-restore I encountered was at least = 2.1x here, often about 2.6x, but in some tests even up to 6.8x; On big VMs = a speedup of 3.6x. The CPU usage of the most busy pbs-restore process during the 750 GiB resto= re was often about 300-450% (not great for 32 threads, but we only do paral= lel reads, so...); the original sync pbs-restore only showed about 98% CPU = usage in htop. First, some tests with my 32 GiB VM (zeroes =3D 64%): > Results (relative to 32 GiB): > Host; Method; source; target; = Start; End; seconds(End - Start); speed (End-Start); = speedup; seconds(PVE log); speed(PVE log); speedup > Azure L32s_v2; orig. pbs-restore; nvme disk 1 ZFS pool; nvme disk 2 Z= FS dir; 21:10:38; 21:13:14; 156; 210.05 MiB/s; = 1.0; 58.69; 558.31 MiB/s; 1.0 > Azure L32s_v2; paral.pbs-restore; nvme disk 1 ZFS pool; nvme disk 2 Z= FS dir; 21:34:25; 21:34:49; 24; 1365.33 MiB/s; = 6.5; 22.85; 1433.99 MiB/s; 2.6 > Azure L32s_v2; orig. pbs-restore; nvme disk 1 ZFS pool; nvme disk 2 Z= FS dir; 21:19:51; 21:22:27; 156; 210.05 MiB/s; = 1.0; 54.46; 601.66 MiB/s; 1.0 > Azure L32s_v2; paral.pbs-restore; nvme disk 1 ZFS pool; nvme disk 2 Z= FS dir; 21:41:53; 21:42:16; 23; 1424.70 MiB/s; = 6.8; 22.63; 1447.70 MiB/s; 2.4 > Azure L32s_v2; orig. pbs-restore; nvme disk 1 ext4 dir; nvme disk 2 e= xt4 dir; 02:10:04; 02:10:49; 45; 728.18 MiB/s; = 1.0; 43.97; 745.16 MiB/s; 1.0 > Azure L32s_v2; paral.pbs-restore; nvme disk 1 ext4 dir; nvme disk 2 e= xt4 dir; 02:12:35; 02:12:56; 21; 1560.38 MiB/s; = 2.1; 21.06; 1556.12 MiB/s; 2.1 > Azure L32s_v2; orig. pbs-restore; 2x nvme RAID0 ext4 dir; 2x nvme RAID0= ext4 dir; 01:56:35; 01:57:24; 49; 668.73 MiB/s; = 1.0; 48.94; 669.52 MiB/s; 1.0 > Azure L32s_v2; paral.pbs-restore; 2x nvme RAID0 ext4 dir; 2x nvme RAID0= ext4 dir; 22:24:45; 22:25:04; 19; 1724.63 MiB/s; = 2.6; 18.67; 1754.79 MiB/s; 2.6 Now some tests with my 750 GiB VM (zeroes =3D 5%): > Results (relative to 750 GiB): > Host; Method; source; target; = Start; End; seconds(End - Start); speed (End-Start); = speedup; seconds(PVE log); speed(PVE log); speedup > Azure L32s_v2; orig. pbs-restore; 2x nvme RAID0 ext4 dir; 2x nvme RAID0= ext4 dir; 00:45:19; 01:47:38; 3739; 205,40 MiB/s; = 1.0; 3736.73; 205.53 MiB/s; 1.0 > Azure L32s_v2; paral.pbs-restore; 2x nvme RAID0 ext4 dir; 2x nvme RAID0= ext4 dir; 00:23:44; 00:41:00; 1036; 741,31 MiB/s; = 3.6; 1034.96; 742.06 MiB/s; 3.6 PVE Logs: > orig. pbs-restore: restore image complete (bytes=3D34359738368, duration= =3D58.69s, speed=3D558.31MB/s) > paral.pbs-restore: restore image complete (bytes=3D34359738368, duration= =3D22.85s, speed=3D1433.99MB/s) > orig. pbs-restore: restore image complete (bytes=3D34359738368, duration= =3D54.46s, speed=3D601.66MB/s) > paral.pbs-restore: restore image complete (bytes=3D34359738368, duration= =3D22.63s, speed=3D1447.70MB/s) > orig. pbs-restore: restore image complete (bytes=3D34359738368, duration= =3D43.97s, speed=3D745.16MB/s) > paral.pbs-restore: restore image complete (bytes=3D34359738368, duration= =3D21.06s, speed=3D1556.12MB/s) > orig. pbs-restore: restore image complete (bytes=3D34359738368, duration= =3D48.94s, speed=3D669.52MB/s) > paral.pbs-restore: restore image complete (bytes=3D34359738368, duration= =3D18.67s, speed=3D1754.79MB/s) > orig. pbs-restore: restore image complete (bytes=3D805306368000, duration= =3D3736.73s, speed=3D205.53MB/s) > paral.pbs-restore: restore image complete (bytes=3D805306368000, duration= =3D1034.96s, speed=3D742.06MB/s) Maybe anyone is interested in "proxmox-backup-client benchmark --repository= chunks2tbnvme": Host | Datastore
(tls bench only) | aes256_gcm
MB/s | compress
MB/= s | decompress
MB/s | sha256
MB/s | tls
MB/s | verify
MB/s -- | -- | --: | --: | --: | --: | --: | --: Azure L32s_v2 | chunks2tbnvme | 2552.57 | 549.95 | 856.72 | 1479.59 | 588.7= 8 | 545.02