From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id C6B2A1FF16B for ; Fri, 26 Sep 2025 18:56:56 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 4B0B015649; Fri, 26 Sep 2025 18:57:27 +0200 (CEST) To: "Max R. Carrara" , Proxmox VE development discussion Date: Fri, 26 Sep 2025 16:41:36 +0000 References: In-Reply-To: MIME-Version: 1.0 Message-ID: List-Id: Proxmox VE development discussion List-Post: From: Lorne Guse via pve-devel Precedence: list Cc: Lorne Guse X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: , List-Unsubscribe: , List-Archive: Reply-To: Proxmox VE development discussion List-Help: Subject: Re: [pve-devel] How does proxmox handle loss of connection / reboot of iSCSI storage Content-Type: multipart/mixed; boundary="===============5301243123368276901==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" --===============5301243123368276901== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 26DBCD6388 for ; Fri, 26 Sep 2025 18:57:25 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 0B0831562C for ; Fri, 26 Sep 2025 18:57:25 +0200 (CEST) Received: from CH5PR02CU005.outbound.protection.outlook.com (mail-northcentralusazolkn190120002.outbound.protection.outlook.com [IPv6:2a01:111:f403:d111::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (secp384r1) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Fri, 26 Sep 2025 18:57:23 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=A9mgfi/Pwr28I6pm2eHDz7j3LQZf0X3AwMU8qc941DEHrNrR4VB+kGyaH0lPO7Vel4dAytgWGt13aR8x++KxI64GUmcI2+wgpPg42ILKjtQUCv69BsJacquj+IP2kFi3bX1UzMib+oDWSbs5TI4OEFx4mXZvU3f3oR/yYkWg2rmCR6VyetKMMwEZeFM6rJHRAzd4f6tV1IBgFjSV1lyGqxu4laPgLeBU+xbfNwBg8Vrc1S31eMKIxijPDqwdyFUyY3t65iFNAYnMlW8mpmOFJf1cNIu6GOAuC35sgwPYp+ZrPayzNWZplavQcrN8xsPcNTgDbpU5j/IX7RPzlRXz7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VWhKE/edQRWiHxiXe72c/wa/Hv+bFR60B220ClhIG18=; b=yIToUd+fAf6zKIYQIlB9DVG8+Ny0G4+gMjOWg9kwoD4HFqU4+gkRz06xEgiMqPhoEI7BmmtdnKmAPw6LnFbQbN+2YuWyWe3/WyMiLuLjhTBIlW7tMg6f6xsRS7nUznbGSFKWcJuT74YZ6+71PTuf6Tj59WzODORuwDb6Ox0CqUQq54MsxInS1H8yJWwho58+K4NZS3X3Nb3DAk5sKTuQGiG8eRWD4GKnjdwAu9CtptNCpkuew9294mCfKBolt6q1VybO26uKnw0ItSqFza8OnCT/6vXLiD8ePO3sapNA/JHFZGp8RG0V7IBErSCfQV916WHtqnCmiLc6lK/RYjla3A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VWhKE/edQRWiHxiXe72c/wa/Hv+bFR60B220ClhIG18=; b=Qa8fC+I6KlBwOdip2YurZVVqkJFtSSViec/RluTw7gM6zUgIQSkSxdmufkAHfHq55Q5sj+z/CrrrmSN7vztNI+FIabLI++EO5Vx1j7WsXHbjwyuo8wlZYDQPSZpx70u+9IkMSRae+/jMi9KNV6ShBYsk3+VyEQom7SUt8wZ1MyLXFt14b2Tsa1jyH9m3nyfWbgz/TJpdlwnCviAApqUMg6Gn4Y2txIm6jjh73hrgt9eqIDDtNZu5N31Ez1AzFGqqE+j8WypPvTMuq86fKmtGGBdbIT87TfgBnaQLExhAW5Gw3/fdJ9FWzd4j202x+IGUPT71kkiVLKJX5t95PMIh8Q== Received: from DM6PR17MB3466.namprd17.prod.outlook.com (2603:10b6:5:1df::23) by IA0PR17MB6736.namprd17.prod.outlook.com (2603:10b6:208:408::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9160.10; Fri, 26 Sep 2025 16:41:36 +0000 Received: from DM6PR17MB3466.namprd17.prod.outlook.com ([fe80::e9a8:2d84:b202:de5c]) by DM6PR17MB3466.namprd17.prod.outlook.com ([fe80::e9a8:2d84:b202:de5c%6]) with mapi id 15.20.9137.018; Fri, 26 Sep 2025 16:41:36 +0000 From: Lorne Guse To: "Max R. Carrara" , Proxmox VE development discussion Subject: Re: How does proxmox handle loss of connection / reboot of iSCSI storage Thread-Topic: How does proxmox handle loss of connection / reboot of iSCSI storage Thread-Index: AQHcLojdtVnzKE2JGkixO6vzBOw54bSlduaAgAAzsXY= Date: Fri, 26 Sep 2025 16:41:36 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US, en-CA Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DM6PR17MB3466:EE_|IA0PR17MB6736:EE_ x-ms-office365-filtering-correlation-id: 4a2942bd-ba69-40d8-fd40-08ddfd1b8f24 x-microsoft-antispam: BCL:0;ARA:14566002|9400799040|19110799012|31061999003|8062599012|8060799015|461199028|15030799006|12050799012|15080799012|440099028|40105399003|3412199025|10035399007|102099032; x-microsoft-antispam-message-info: =?Windows-1252?Q?YUuDfA62LU1nQ0VtkxoHqORJiSUuzYOQd1j/+hFTqUJNQmSNYdajgvRH?= =?Windows-1252?Q?LQB0ZO0dIGud4n5FPudvS8D8VvU7Kff3TLBQCIt303P9Waw2ESLFzdZs?= =?Windows-1252?Q?fcATC6mF5+Jhd0mVbANRZbS79lHBfKiioObgqK5qPINgPwSqUyOj9xLS?= =?Windows-1252?Q?f6eQHO0GHW46tGg0aReFrDJTrc3i0VqFXxwxillPha5xupVY8vVLb7ra?= =?Windows-1252?Q?i+LK6UCPzKMvFH3cIkrRa6vV9BKPZG0hC56csHXbkUG6L2M+8LhBAXY1?= =?Windows-1252?Q?VnOsbcOC68b9fvifbhZ0bfvWgogHa9I7aZUiSxeUR2W/FBVuS7pq8KrU?= =?Windows-1252?Q?5dL+5OUiw017pKm3B1q5SZGw2yS7ub4UQCAC+x/o3eSSURiE1YEOWLwh?= =?Windows-1252?Q?dyMWM+thlV5TdotZariWhQaO7uBJfCiee8I6QJea4Ym0utOL8KCAifns?= =?Windows-1252?Q?22yVFBKKl5b1PKtC9B74mdow2SBO01xkXRQNDq/LgdmpYYDFsGqlzRof?= =?Windows-1252?Q?QWJUnODM/yJeewgmJs845B6w4q9DSfazr6BGKyvbdvDK6ZxZSZAuzPqJ?= =?Windows-1252?Q?iPUiqK1+vRx2HEysaPVBGR5UCsKcKcHcZZPdPzIkMkzXQXbhZwJ9YmtA?= =?Windows-1252?Q?+AU4J6A16iFMdOKwAdYYEZ6LdojlHnWcVlndYCwSRiV+JpTY/piTgIf9?= =?Windows-1252?Q?w8aqLD17E7/jUjjtu3O46SqSAbrZj67JQDYhtf97feXQUe3nmxq6d1Z8?= =?Windows-1252?Q?3KitHr1a+xb8C+sKhL2gNMtwPw+1RqJ52iqt+lgK+IPxmqTX5jFfAyza?= =?Windows-1252?Q?nDuWH4haZXSS8xlc+GHk0DRNZYzdvfD2R6JIftEbIrAsmCuOiZ/tDI6I?= =?Windows-1252?Q?KD8/K0QCwB4N/Hq1vPr0CazoyKiQPgri0Xg+kd15I5W1nZG553aOX1Z5?= =?Windows-1252?Q?gimXkqWsPqb0ETOPseM/YWYJIQHUE/mzqGiLKQuqdenyybghc7drGs/5?= =?Windows-1252?Q?3Th+ELuOpQqMQTj0fonqMtA18tkx7M+/TPuTpDBuZqALPKodC8TTiWLs?= =?Windows-1252?Q?EU2LXN8ExmBf2nTCYmhE44DHlrRBcIESSOyZ12OClGkLOEig/Thits24?= =?Windows-1252?Q?yzLjPN229PiY+ygUVDI1BO6fkGuZHwe59cwmPgpSj5KySlGCk5szhMrp?= =?Windows-1252?Q?kvQ0E96yvYaCH82SD4eatjBu7jjJo989AnFTVk2o41s+zra3xEsOPSU8?= =?Windows-1252?Q?2BoGsX/GuUtQxJlpgmfohBZ3vOVmKGfEtj7t4/53bTiXYzYLIxfDqnqB?= =?Windows-1252?Q?vwgjqwoON2DU37e4zjGgBMAXJJUQOykbYLErncQVuzgh6dMW?= x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?DzLjctGgUSKv34xUxp6TsYRu5PmVnF25351FIChIWI6eBk3fLMMKDK+D?= =?Windows-1252?Q?n2WDqkJvqriu/zxXTJgDYsSLoWXBY3dMzoS9AdA3s9bX6O8NGR5IP+yZ?= =?Windows-1252?Q?mOOY0hLm1OdEgoD7VMyqclmkik70XaSNjn/nVDbWbmpe9cG4Z4KVsjvp?= =?Windows-1252?Q?viY2Kmt+xKE+vl9tOsiogDcfNvZ1wGl7E8KPxxPOICNRlcreYEGjLm4s?= =?Windows-1252?Q?JEw5bVZdLAsw6r3zomq7ABJZRTUvsSY7hdeXOSicnN5TZNcbPUg5sZ+p?= =?Windows-1252?Q?Rj0QB7jcrmbfPTX3BSgzRyUyjJYRM5/p90OZz8S+ZrQEWkITh5kS4VUM?= =?Windows-1252?Q?EcKJbSHWTQpohDBZRd2fNWztad29WuOnGHmJATSawZoQEWxxrlicclPT?= =?Windows-1252?Q?D+y5YwWa2FZb/dH5mjjq/Ig6I7yneXZBRSVAQqD7bcvUlEHgarU7Wozw?= =?Windows-1252?Q?xQDTo7Av3U7a934Lye0nKUKBNCYWNILbiIvzEBS2jhGnXKXR3MFiFWoo?= =?Windows-1252?Q?+qoaN/AnJZ1XpI6UBqIP6BbheZdIq8u9uS9VC26q/pl+3FtrCEZcJ27q?= =?Windows-1252?Q?o0KawLyXqxm61IBh7OdnXvePpKW2YP7iVcVhq5GoWLH0uUVQi1YfYvXf?= =?Windows-1252?Q?nmH0TwJTV3qORWPQLAh2YE1caTF4jVdVBLLF1oSuwsJaNog9ZnYXXWvb?= =?Windows-1252?Q?zPewBfrkG9gVzbecObCy8m8bHjt3XXiDWXgXBfqhp0snIJJlxD4kSheu?= =?Windows-1252?Q?77jUMLu449NFfhCuwFE1MTzPaBPuEKmSLeUg03k1GjuzUfSSXdmZDMA5?= =?Windows-1252?Q?w8/0LwHPupU8v4ALRG9PNqFGn6RBt7y3miC4gP2DrYUsK3jtosWaqgsk?= =?Windows-1252?Q?lD5aNx1FxORKm8wpLO4/Lvy/YJijG4u4SKlX7kTspYWvT91QFoMYx1xF?= =?Windows-1252?Q?cQTkWwd2ISQ4pMjm0aoP0UeLFc/yWX0gfZha1iXE2U9OGFF/L83mYSTB?= =?Windows-1252?Q?fAP/HHqrRURUrxx9OKuQp6Yy9AaD6Hsa+Q88IYBxCDWHMNpRlmccytzu?= =?Windows-1252?Q?x+TxK0QqfoIrrsIol/q/OPrxVVIn3TiLKEZXDcLxWjHu5NfqlzANrB59?= =?Windows-1252?Q?6ZJUgtUylU/NIeQhIGYWKLwhGNNUY7zfLb7e/DVj1RjGLcWFtSGExvI+?= =?Windows-1252?Q?+PZuXoSuzZaA9ujgdzYHEgwgKodQVKfCTMxFV448UFEd9dG8v4QojQ+a?= =?Windows-1252?Q?RMLvg+nwjNuSDU6k4BZG8JL+iGqmaocT8vXf7/ZzQ8gwo2+n6HeZHaMQ?= =?Windows-1252?Q?NLSkyYrOrcYfbxx7Rx4c8MLQg6Y=3D?= MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-8534-20-msonline-outlook-31e5f.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR17MB3466.namprd17.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 4a2942bd-ba69-40d8-fd40-08ddfd1b8f24 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Sep 2025 16:41:36.3883 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR17MB6736 X-SPAM-LEVEL: Spam detection results: 0 ARC_SIGNED 0.001 Message has a ARC signature ARC_VALID 0.001 Message has a valid ARC signature AWL -0.191 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider GB_GEN_REDIR_URL 0.5 Redirector found in href link HTML_MESSAGE 0.001 HTML included in message HTTPS_HTTP_MISMATCH 0.1 - KAM_LOTSOFHASH 0.25 Emails with lots of hash-like gibberish SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [proxmox.com] Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 TIL what nerd-sniping is. I was worried that I broke some kind of rule a fi= rst. LOL Thank you for your response. I will do some more extensive testing to see i= f there is a limit. Some TrueNAS updates can take longer than 3 min. I imagine it might be guest-dependent. I always assumed that I had to shut down my VMs before updating TrueNAS. On= the next update I'll run some backups and update while my proxmox cluster = is online. ________________________________ From: Max R. Carrara Sent: Friday, September 26, 2025 7:32 AM To: Lorne Guse ; Proxmox VE development discussio= n Subject: Re: How does proxmox handle loss of connection / reboot of iSCSI s= torage On Fri Sep 26, 2025 at 4:06 AM CEST, Lorne Guse wrote: > RE: TrueNAS over iSCSI Custom Storage Plugin > > TrueNAS has asked me to investigate how Proxmox reacts to reboot of the s= torage server while VMs and cluster are active. This is especially relevant= for updates to TrueNAS. > > >The one test we'd like to see work is reboot of TrueNAS node while VMs a= nd cluster are operational=85 does it it "resume" cleanly? A TrueNAS softwa= re update will be similar. > > I don't think the storage plugin is responsible for this level of interac= tion with the storage server. Is there anything that can be done at the sto= rage plugin level to facilitate graceful recovery when the storage server g= oes down? > > > -- > Lorne Guse >From what I have experienced, it depends entirely on the underlying storage implementation. Since you nerd-sniped me a little here, I decided to do some testing. On ZFS over iSCSI (using LIO), the downtime does not affect the VM at all, except that I/O is stalled while the remote storage is rebooting. So while I/O operations might take a little while to go through from the VMs perspective, nothing broke here (in my Debian VM at least). Note that with "broke" I mean that the VM kept on running, the OS and its parts didn't throw any errors, no systemd units failed, etc. Of course, if an application running inside the VM for example sets a timeout on some disk operation and throws an error because of that, that's an "issue" with the application. I even shut down the ZFS-over-iSCSI-via-LIO remote for a couple minutes to see if it would throw any errors eventually, but nope, it doesn't; things just take a while: Starting: Fri Sep 26 02:32:52 PM CEST 2025 d5ae75665497b917c70216497a480104b0395e0b53c6256b1f1e3de96c29eb87 foo Done: Fri Sep 26 02:32:58 PM CEST 2025 Starting: Fri Sep 26 02:32:59 PM CEST 2025 d5ae75665497b917c70216497a480104b0395e0b53c6256b1f1e3de96c29eb87 foo Done: Fri Sep 26 02:33:04 PM CEST 2025 Starting: Fri Sep 26 02:33:05 PM CEST 2025 d5ae75665497b917c70216497a480104b0395e0b53c6256b1f1e3de96c29eb87 foo Done: Fri Sep 26 02:36:16 PM CEST 2025 Starting: Fri Sep 26 02:36:17 PM CEST 2025 d5ae75665497b917c70216497a480104b0395e0b53c6256b1f1e3de96c29eb87 foo Done: Fri Sep 26 02:36:23 PM CEST 2025 Starting: Fri Sep 26 02:36:24 PM CEST 2025 d5ae75665497b917c70216497a480104b0395e0b53c6256b1f1e3de96c29eb87 foo Done: Fri Sep 26 02:36:29 PM CEST 2025 The timestamps there show that the storage was down for ~3 minutes, which is a *lot*, but nevertheless everything kept on running. The above is the output of the following: while sleep 1; do echo "Starting: $(date)"; sha256sum foo; echo "Done: = $(date)"; done ... where "foo" is a 4 GiB large file I had created with: dd if=3D/dev/urandom of=3D./foo bs=3D1M count=3D4000 With the TrueNAS legacy plugin (also ZFS over iSCSI, as you know), reboots of TrueNAS are also handled "graciously" in this way; I was able to observe the same behavior as with the LIO iSCSI provider. So if you keep using iSCSI for the new plugin (which I think you do, IIRC), everything should be fine. But as I said, it's up to the applications inside the guest whether long disk I/O latencies are a problem or not. On a side note, I'm not too familiar with how QEMU handles iSCSI sessions in particular, but from what it seems it just waits until the iSCSI session resumes; at least that's what I'm assuming here. For curiosity's sake I also tested this with my SSHFS plugin [0], and in that case the VM remained online, but threw I/O errors immediately and remained in an unusable state even once the storage was up again. (I'll actually see if I can prevent that from happening; IIRC there's an option for reconnecting, unless I'm mistaken.) Regarding your question what the plugin can do to facilitate graceful recovery: In your case, things should be fine "out of the box" because of the magic intricacies of iSCSI + QEMU, with other plugins & storage implementations it really depends. Hope that helps clearing some things up! [0]: https://na01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgit= .proxmox.com%2F%3Fp%3Dpve-storage-plugin-examples.git%3Ba%3Dblob%3Bf%3Dplug= in-sshfs%2Fsrc%2FPVE%2FStorage%2FCustom%2FSSHFSPlugin.pm%3Bh%3D2d1612b139a3= 342e7a91b9d2809c2cf209ed9b05%3Bhb%3Drefs%2Fheads%2Fmaster&data=3D05%7C02%7C= %7C2383b70d0bbe497f5e8f08ddfd01203d%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%= 7C0%7C638944903454301887%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIl= YiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7= C%7C&sdata=3D3zfBzsMeJwbVPTbH5YERI8xEc%2BLsRtlOUSVd3r%2BZBoI%3D&reserved=3D= 0 --===============5301243123368276901== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============5301243123368276901==--