From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from gate001.proxmox.com (gate001.proxmox.com [IPv6:2a0f:8001:1:32::40]) by lore.proxmox.com (Postfix) with ESMTPS id 290F01FF0E1 for ; Mon, 27 Jul 2026 14:50:25 +0200 (CEST) Received: from gate001.proxmox.com (localhost.localdomain [127.0.0.1]) by gate001.proxmox.com (Proxmox) with ESMTP id AE750213ED; Mon, 27 Jul 2026 14:50:24 +0200 (CEST) Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=UTF-8 Date: Mon, 27 Jul 2026 14:50:20 +0200 Message-Id: Subject: Re: [PATCH storage] fix #7598: qemu-img resize: tolerate timeout if resize succeeded From: "Jakob Klocker" To: =?utf-8?q?Fabian_Gr=C3=BCnbichler?= , X-Mailer: aerc 0.20.0 References: <20260603082557.25359-1-j.klocker@proxmox.com> <1784886368.5u9kt3tdeb.astroid@yuna.none> In-Reply-To: <1784886368.5u9kt3tdeb.astroid@yuna.none> X-Bm-Milter-Handled: 55990f41-d878-4baa-be0a-ee34c49e34d2 X-Bm-Transport-Timestamp: 1785156585408 X-SPAM-LEVEL: Spam detection results: 0 AWL 0.845 Adjusted score from AWL reputation of From: address DMARC_MISSING 0.1 Missing DMARC policy KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment (newer systems) RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Message-ID-Hash: NC3B7UPC4ZHCEL2YCNH3UAERTVDBB4MW X-Message-ID-Hash: NC3B7UPC4ZHCEL2YCNH3UAERTVDBB4MW X-MailFrom: j.klocker@proxmox.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.10 Precedence: list List-Id: Proxmox VE development discussion List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Thanks for the feedback. Replies inline On Fri Jul 24, 2026 at 11:57 AM CEST, Fabian Gr=C3=BCnbichler wrote: > On June 3, 2026 10:25 am, Jakob Klocker wrote: >> On slow storages there is a chance the 10 second timeout is triggered >> when resizing a volume. If the timeout fires while the resize is in a >> certain state near the end, the operation can still complete >> successfully even though a timeout error is thrown. In that case the >> config is never updated and keeps the old, wrong size. >>=20 >> Because the config is out of sync, the volume is then displayed with >> the wrong size in the web interface. >>=20 >>[SNIP] >> =20 >> $timeout =3D 10 if !$timeout; > > I think making this worker-task aware and bumping the timeout in that > case would make more sense - storage performance varies wildy, and like > I noted in the bug, this only seems to be called from a worker context > where the additional time hurts way less than risking running into the > timeout just because something is slow. > > e.g., in the RBD plugin we have a default connect timeout of 60s in > workers, in the ZFS plugin we bump ZFS requests to 300s if a smaller > timeout is set, and default to a timeout of one hour if none is set. > I'll mirror the ZFS pluign in a v2 and default the timeout to an hour in worker context. >> - run_command($cmd, timeout =3D> $timeout); >> + eval { run_command($cmd, timeout =3D> $timeout); }; >> + if (my $err =3D $@) { >> + >> + die $err if $err !~ /got timeout/; >> + >> + my $info =3D JSON::decode_json(qemu_img_info($path, $format)); >> + die $err if !$info; >> + >> + my $actual_size =3D $info->{'virtual-size'}; >> + die $err if !defined($actual_size) || $actual_size < $size; >> + } > > IMHO this doesn't fix the actual bug mentioned.. I guess what you see > here if running into this behaviour is that the resize was done, but > syncing then takes long and hits the timeout? > > because what `qemu-img resize` does for raw images is basically just > > ftruncate(..) > fdatasync(..) > > and just because you read back the updated size in the timeout case > (from local, cached metadata), does not necessarily mean it got > persisted to the storage (which might be on a different system) - that's > what the sync is for after all.. > What concerned me most in the bug report was the state after a failed=20 resize, where a subsequent relative resize would grow the disk further=20 than intended - the reporter's 2G disk ending up at 32G after two +16G requests, even though the first one appeared to fail [0]. I reproduced this and what happened is: The first resize runs into the timeout. The resize did take effect on the storage, but because the command was reported as failed, the size property in the VM config was never updated - it still reads the old value. The second resize reads the actual on-disk size (not the stale config value), so it computes the correct target and succeeds. In doing so it also refreshes the config's size property to the storage's current size - which by now reflects both resizes, hence the jump to 32G. Since I missed that the read-back size isn't necessarily the persisted size, there's no real way to deterministically tell whether the resize was actually persisted. Therefore I'll bump the timeout as you suggested.=20 That doesn't make the bug strictly impossible - if a resize still hits the (much larger) worker timeout, the same stale-config situation could occur. But with the timeout raised that should barely be the case. Thanks for pointing out the sync issue! [0] https://bugzilla.proxmox.com/show_bug.cgi?id=3D7598#c3 >> } >> =20 >> 1; >> --=20 >> 2.47.3 >>=20 >>=20 >>=20 >>=20 >>=20