From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id C719068F59 for ; Sat, 5 Dec 2020 02:12:01 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id BA278FAE5 for ; Sat, 5 Dec 2020 02:12:01 +0100 (CET) Received: from EUR02-VE1-obe.outbound.protection.outlook.com (mail-eopbgr20085.outbound.protection.outlook.com [40.107.2.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 79A76FADA for ; Sat, 5 Dec 2020 02:12:00 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hCkaizBb9+3yQ+wq5aTwrDYdMsMEsAk7sR3IWMjQ+J/523ttam0xZm8j3RXviHckoUEw1jLfM1pgDYWtUJzRDzINeOWMia50OfL4npFDgfqhyAC+fRjPlmlkK/3Vzqt/RjZLZJPLSjFYcT8HH+f0aI0r4MZRXO4bAumnPApBWEWfA/6/M6ntoAXCh1oL6oxy7AwWTl1Umo88g9o8vmvq2QPbsdiOC+YmPSNgKxYc8enHOhzP5ljxloSFhmMc+mTT5EPejzIMD1aKtvWwrRRaaI5DnWtXb3H9eNKadODwwluYT0x75cRqjLuaTHeBEfTSvgsT8M/jGbwF9f2FfZgUxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=815C2ju4Dl1r+cjX7V0RuAH9kKwu/7cQEDigljO5HO4=; b=GzR4txtl0AzOH4l6XoMkf+oE6Bs7v1TftdTXghXPSlkt7Q3zPuJ5RUXDPgSTXH06YkUZeggcWSefx8c9ULR/JBgKDaNkk/VFP+j31aU/XRl0XoOUFpfrug4GbBdhrWifFbnzwRlaiBKglO12rn1re6XvSYxgQAo9Dc1vJre//M/hdyD8aZ55Z9+7B+VXD/oFd/+yxuOgyWubnI1gHM+XmoStniVwh/bvk3+rWh53upJhgoX7iYsJnpc7ZXDvB/jBfZ8xVvzR4q2+pexqfrSIBysEc7zqvMvukv2yY0nKiNkcBkdqK0Wyis7xwXZ9swKLTJFsGu3oasaPXRvnmm4CxA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=logics.de; dmarc=pass action=none header.from=logics.de; dkim=pass header.d=logics.de; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=logics99.onmicrosoft.com; s=selector2-logics99-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=815C2ju4Dl1r+cjX7V0RuAH9kKwu/7cQEDigljO5HO4=; b=WyWAVvRvUIkaax2gnU+Q8J5EeZZBx4e++EJSttJ2OlpDH6AaX53aoyR+KVuknyMKYzbokBFkbXdmg+M7kU1uIl7TBqAZOpHGvmTmeckSJesb/rqaguho/OcBtI8TiQrTv/KppVyzye1D3STr4+6NjCBC4ZVZdKhXxXk9mrOAuWE= Received: from AM0PR09MB2754.eurprd09.prod.outlook.com (2603:10a6:208:12e::18) by AM0PR09MB2305.eurprd09.prod.outlook.com (2603:10a6:208:e0::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.21; Sat, 5 Dec 2020 01:11:52 +0000 Received: from AM0PR09MB2754.eurprd09.prod.outlook.com ([fe80::68c5:8966:920c:8d24]) by AM0PR09MB2754.eurprd09.prod.outlook.com ([fe80::68c5:8966:920c:8d24%3]) with mapi id 15.20.3632.021; Sat, 5 Dec 2020 01:11:52 +0000 From: Niko Fellner To: Proxmox Backup Server development discussion Thread-Topic: [pbs-devel] parallelize restore.rs fn restore_image: problems in async move Thread-Index: AdbKR5NQuUK4YTZbRJ+mjP9LatDx2QABnFyAABG2veA= Date: Sat, 5 Dec 2020 01:11:52 +0000 Message-ID: References: In-Reply-To: Accept-Language: de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: lists.proxmox.com; dkim=none (message not signed) header.d=none;lists.proxmox.com; dmarc=none action=none header.from=logics.de; x-originating-ip: [93.104.29.163] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 98d6de28-6909-4bd0-4647-08d898bac013 x-ms-traffictypediagnostic: AM0PR09MB2305: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: IZ8XRgkHJ9PlGJqj7qV4RU9yKwz+cfcBNfGpPs+yCjR5pDrmSoctJPEdgx7f3Ogu4zgE15IwPoTILKJ45kq82TqsmL5FTzqI2TG+YSUQ+3ZFQwcouZidE2UeKS66+wsACP69ESyiE1ncSe1nb/D+6RZ8ym5CqJYfaqpaefY68tizi0mOvxxrscT03/pQw4ven/eCv2AT1LcW85/NPCoXsoEexgJ58SqEy67d81y7pNvquE+ES2LStHALJjnmn8Fy8Ta1+fjgCXgRN5gWe2uKHE2XCf7lExiMQ/puf/32qNGpjijaUDq5k2RIuHlb0MoteYHjcJb59jeAttX8AT34vA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM0PR09MB2754.eurprd09.prod.outlook.com; PTR:; CAT:NONE; SFS:(376002)(366004)(346002)(396003)(39830400003)(136003)(52536014)(2906002)(5660300002)(316002)(71200400001)(9686003)(7696005)(55016002)(8676002)(83380400001)(478600001)(76116006)(86362001)(8936002)(33656002)(6916009)(6506007)(64756008)(66446008)(66556008)(186003)(66476007)(66946007)(26005); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?Ug8TT9QtkQ1EHsKDj8qcmippGH0DDzYIfO9+2MdaapXXrFnvJoOmXFNlkfFX?= =?us-ascii?Q?NWTx9Gttsiz1KqJC+AQpjXMIhLxXDS94ifcaNoHwgZuGY7zyUmrGgYbGk7dS?= =?us-ascii?Q?HORqZ6XrIa6EcxPqfLOF6TvtatlquZI7MK8y8eKfqb811oo5au8MFFccDKY4?= =?us-ascii?Q?QmTF1JRMMI86TxXC9opqcI/e251iSLQsGWLCDentX2X92+5vkJv6HpfPXvKV?= =?us-ascii?Q?5AgSgfLjmzmrGk6/rK3umVk0KiQlEDiCSoWd+SiC03uG2vraHFP98HCWiUdO?= =?us-ascii?Q?6+v0byvE7Q9oCp4hqBQzgZwgJK7MDtf0G8Ndg1yjoJRoRPA2JBxYliQWo263?= =?us-ascii?Q?DcN5XAQe452+Mjau6HB84ReTUEN1R4eZemVfCU2lf0wwXcoOCrzqv4u8nUys?= =?us-ascii?Q?olw5P7geCmETsxvsp674crdYmyV06NdIDLubG4KJGfoZG/Zc+WBqcskGzXMp?= =?us-ascii?Q?MbH9wN3xfeoIHdOTMt0x6nKtGEegWwQlUpw4lGAPziIQnbuhmQSM+2YSlcex?= =?us-ascii?Q?6UKTcS0vcvnT6kZON2iXUqK/R/ecXGqj3itlFPNrSlqYwoC5SafFKdxmRJtb?= =?us-ascii?Q?rAhMuX1YcpjkZyrtQ8jYR9TXnl0mKfTgyZvJJtz/yuRY+2hK1SqLFnZx1tM3?= =?us-ascii?Q?WuPklWdT7H8XHMg0iDA8MoKZYgEdqsbuB+U3UY4Bi+QfAPq/bYV69xhxrsBj?= =?us-ascii?Q?grgzirPIOm+cTsbjsQ5D7puHmY6exNiKv6WysM5bj7AHWLaUekMmbo5Hod0X?= =?us-ascii?Q?oXYp51fvp9pB06zrtGwb//snQMMESg+e04el6/xGGDWNl6jrlRZkaKkNdnKh?= =?us-ascii?Q?I9u1DSPlCXYt4tZCM6MKbAoWLXy2/4B1hYkuk3uVfM8oZCFUIAxoD1L5PlJ2?= =?us-ascii?Q?1yYtvWSRWEdehNkgp3O8huNEsMnXWb+SmQQRs5GxN+IevE+dOEnLIpcLiJvW?= =?us-ascii?Q?SIdd8WbrGCK7Eey4yO7YoOFTiKqCyJi7cYNmkgWsTqQ=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: logics.de X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: AM0PR09MB2754.eurprd09.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 98d6de28-6909-4bd0-4647-08d898bac013 X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Dec 2020 01:11:52.5303 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: fdfa9215-653f-430f-b8ab-a8728140f97a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: CudqjdSEdaLSXsoKVcbMtkuEgakqVuZK9I7kSKfP558TfhjQNn+Z40cqN878NqkaD2h3Jl/0i4+S8Kxg7x5Z7AckbTSIKsujS+AUrwOUhrw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR09MB2305 X-SPAM-LEVEL: Spam detection results: 0 AWL -0.039 Adjusted score from AWL reputation of From: address DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust RCVD_IN_MSPIKE_H2 -0.001 Average reputation (+2) SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [pull.rs, logics99.onmicrosoft.com, restore.rs] Subject: Re: [pbs-devel] parallelize restore.rs fn restore_image: problems in async move X-BeenThere: pbs-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox Backup Server development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Dec 2020 01:12:01 -0000 > first thing: the error you get is simply a return type mismatch, the 'asy= nc move' block expects to return a Result<(), Error> because there is a bai= l!() and multiple '?' calls. > the simplest way to fix this is to write > Ok(()) > at the end of the async move block > the next problem you will run into is that there are multiple uses of var= iables that get moved in a loop (which does not work in rust) so you have t= o pull out some things, and clone some others @Dominik: Thanks a lot for your help! Yes, and after following your instruc= tions I've discovered more and more similarities with "pull.rs"...=20 > a much bigger problem is the following: > the callbacks that get called (write_data_callback/write_zero_callback) > are not 'Send' meaning they cannot send between threads safely (needed by= tokio::spawn) because they both are closures that use a '*mut T' > in that code this would not actually be a problem, since the restored blo= cks never overlap, but rust does not know that > without digging further, i think this is not easily solvable in rust with= out either writing unsafe code or changing our c <-> rust api (idk if that = is possible) So that's not only a restriction of tokio::spawn, but also anything that Ra= yon provides (including its ParallelBridge?), and the code in pull.rs, too?= (ParallelHandler, futures::stream, ...) ? > as pointed out in the bug report, we already have such code in pull.rs > Why not use that instead of something new? @Dietmar: You're right, that looks like a gold mine - but unfortunately one= that I cannot mine due to my limited knowledge of Rust... Maybe you can h= elp out to apply its magic on restore.rs? Most importantly, will it work wi= th that write_data_callback and write_zero_callback which are not Send? > this is not easily solvable in rust without either writing unsafe code or= changing our c <-> rust api (idk if that is possible) "Unsafe" seems to be the right keyword here. In proxmox-backup-qemu the Cal= lbackPointers (capi_types.rs) do this: "unsafe impl std::marker::Send". Tho= se CallbackPointers are used by lots of "_async" functions within the c <->= rust api. So I guess a hypothetic proxmox_restore_image_async needs to use= CallbackPointers, too?