From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [45.144.208.40]) by lore.proxmox.com (Postfix) with ESMTPS id B69FA1FF0A7 for ; Wed, 02 Sep 2026 10:32:30 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id 2B6F3212F0; Wed, 02 Sep 2026 10:32:30 +0200 (CEST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787957478; x=1788562278; h=mime-version:content-transfer-encoding:content-type:message-id:date :subject:to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=8zEOa+UyTRCxJUpLY3PDG5B7j0Wrfwn+w9wbGSF4Is0=; b=SmNAvVyeeRKbqMfyTqC0v5TfTAhFBX4T1+r+Id/AJrA4gjIsbknjLFVzbAzqLEd9/O CEjf54SRAUTbG0FriWvz7JYxbyJ1L4xbPQjlZjF5ZQN775BWmtNXoTdxRSxyWy46DMSM DdG7B+JUrtjIRuyU+5xoZyILHDZcuvv8tJxQtPGDXqkuyYBR+3O3qurTtsnu6MQS2413 wfX4fwIw5auaLctZKemIyF2yaikl7yCmMm2/g1Oe002gGXYkJTjb5kENvN7Ho1DRV8pN K5lBrJczMfeBjsuC7fM94bw47A/Q7EMSDx6+HUDM66HssgacMPNi9wM9XhbuIO58wj/w ci9Q== X-Gm-Message-State: AFuF++lOgW5PxEHRVPmidx8V5S3A/M5ck2hsOI6K+55YxeoV0eV2fhau nCHMWoiwaW8pbWZxkPDGFJkkCN3FPKQQiUtG+TIVL/xUDEHG8fZZvOtlyRM93AmoHm8WuTCWFBq uzyQf X-Gm-Gg: AR+sD13fidqoPk8gN2v+lMX1AzBYS84dXe74DsF/NAM9FQjE1bRp2ZYTQYT/MdQU3G5 eO0xE1Id5Rn0vBQyyeKJO8NNCe7hZDVsgnzlSXRDpBM4BvKkPgNdw7pv0+uGvsYkMx6TkTZB7Cw 0cZsZfAhWTy+Y0CbWN1T4v/nn1ewOEYqtllD/BVQBn7cEXgrATUhFUJng86PzWQGA4L3OABP+Gs u8FZb5GTnz+mKinHlwjBLUk+p1uSkCIVhtzewLzRSTEsMFHbu6zsRWDnqYkCE44QRz9RJOMmgvU zzINFN7EdJiqVffG2DVK8/VQLQXgkt42mMxAbQqj5PLDnBHJgDyhByr9WVDqsQ3DBcVAagBXj37 jjyM98x1ypdwWJjDx5P09zduGs2xdJQMbCWiMvR2RRmgbzWBiFHdraamNwZb6fc7i5N3T9rVdjY LuA5mUosnR814Plw4GG9eWIK1m2ByzIstPCUaV17Sugxgs4sOqHDOfuk8mylFLdM0vkIPPBJKzz BR95lehPE2beTdKu3jWKaLPfFsi5fgTNSEAJFQRbPOAGQ== X-Received: by 2002:a05:6102:a4b:b0:777:4e99:6f84 with SMTP id ada2fe7eead31-78596dd6eb3mr3034581137.1.1787955747367; Fri, 28 Aug 2026 15:22:27 -0700 (PDT) From: Ing. Alfonso Kuen Arroyo To: pve-devel@lists.proxmox.com Subject: qemu-img convert runs with cache=unsafe for every block storage except zfspool Date: Fri, 28 Aug 2026 17:22:26 -0500 Message-ID: <178795574681.28052.11392785472014160013@idkmanager.com> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-SPAM-LEVEL: Spam detection results: 0 DMARC_PASS -0.1 DMARC pass policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust RCVD_IN_MSPIKE_H3 0.001 Good reputation (+3) RCVD_IN_MSPIKE_WL 0.001 Mailspike good senders SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record X-MailFrom: gerencia@idkmanager.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation Message-ID-Hash: UFMJZ2ML6DITOR7YN2CWL7HZACBCHKTA X-Message-ID-Hash: UFMJZ2ML6DITOR7YN2CWL7HZACBCHKTA X-Mailman-Approved-At: Wed, 02 Sep 2026 10:32:21 +0200 X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hello, Reporting a data-loss path in disk move / migration that we hit in production= on PVE 9.2.4 (qemu-server 9.1.18). Happy to move this to Bugzilla if that is the preferred channel -- I do not have an account there yet. THE ISSUE PVE/QemuServer/QemuImage.pm protects the cache mode only when the storage is a ZFS pool: 122: $cachemode =3D 'none' if $src_scfg->{type} eq 'zfspool'; # sour= ce (-T) 149: push @$cmd, '-t', 'none' if $dst_scfg->{type} eq 'zfspool'; # target= (-t) For every other destination, qemu-img convert falls back to its own default f= or -t, which is 'unsafe', i.e. BDRV_O_NO_FLUSH. If a write fails while the data = is still a dirty page, the page is dropped, the errseq is never consumed by an fsync that would report it, and qemu-img exits 0. PVE then reports TASK OK fo= r a copy that silently lost data. This affects any block destination: iSCSI, FC, LVM over SAN, NVMe-oF, and third-party storage plugins. It is not specific to any one of them. WHY THE EXISTING SAFETY ARGUMENT DOES NOT HOLD The pve-devel thread that introduced this (Aug 2016) asked exactly the right question -- "is this really safe?" -- and the answer was that bdrv_close() se= nds a flush. But QEMU's bdrv_co_flush() contains: /* But don't actually force it to the disk with cache=3Dunsafe */ if (bs->open_flags & BDRV_O_NO_FLUSH) { goto flush_children; } The flush is issued, and cache=3Dunsafe is precisely the mode in which it does nothing. The safety argument invalidates itself, and it has been in production for ten years. WHAT IT COST US Three PVE 9 nodes against a TrueNAS SCALE 25.10.4 NVMe/TCP target. The target was failing large commands under memory pressure -- a separate bug, ours to f= ix, written up at: https://github.com/truenas/truenas-proxmox-plugin/issues/96 Those failures should have surfaced as a failed migration. Instead: - A 2 TiB image copy lost roughly 98 GiB, and qemu-img exited 0. - The guest then ran 41 hours on that copy, logging 2346 XFS metadata corruption events, before we noticed. - All 520 "lost async page write" events were on the destination. With -t none the copy would have failed loudly and no data would have been lo= st. The storage bug was ours; the silence was not. SUGGESTED FIX Pass -t none for any destination that is not a file-backed storage where the performance tradeoff was deliberately accepted. More conservatively: invert t= he condition so 'unsafe' is opt-in per storage type, rather than the default for everything except one. REPRODUCING WITHOUT A FAULTY ARRAY dmsetup a 'flakey' or 'error' target under a test LV, qemu-img convert onto i= t, and check the exit code. The point of the report is that the exit code is 0. Regards, Alfonso Kuen