From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [IPv6:2a01:7e0:0:424::9]) by lore.proxmox.com (Postfix) with ESMTPS id 54F711FF183 for ; Wed, 22 Oct 2025 16:38:40 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id B91371B1FE; Wed, 22 Oct 2025 16:39:06 +0200 (CEST) Date: Wed, 22 Oct 2025 10:38:45 -0400 To: =?UTF-8?Q?Fabian_Gr=C3=BCnbichler?= , Proxmox VE development discussion References: <1761126324.5glolx6b39.astroid@yuna.none> In-Reply-To: <1761126324.5glolx6b39.astroid@yuna.none> MIME-Version: 1.0 Message-ID: List-Id: Proxmox VE development discussion List-Post: From: Andrei Perepiolkin via pve-devel Precedence: list Cc: Andrei Perepiolkin X-Mailman-Version: 2.1.29 X-BeenThere: pve-devel@lists.proxmox.com List-Subscribe: , List-Unsubscribe: , List-Archive: Reply-To: Proxmox VE development discussion List-Help: Subject: Re: [pve-devel] Consistency in volume deletion in process of concurrent VM deletion Content-Type: multipart/mixed; boundary="===============8590879885619942561==" Errors-To: pve-devel-bounces@lists.proxmox.com Sender: "pve-devel" --===============8590879885619942561== Content-Type: message/rfc822 Content-Disposition: inline Return-Path: X-Original-To: pve-devel@lists.proxmox.com Delivered-To: pve-devel@lists.proxmox.com Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 25D43D3A88 for ; Wed, 22 Oct 2025 16:39:05 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 103E21B240 for ; Wed, 22 Oct 2025 16:39:05 +0200 (CEST) Received: from mout.kundenserver.de (mout.kundenserver.de [212.227.17.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Wed, 22 Oct 2025 16:39:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=open-e.com; s=s1-ionos; t=1761143928; x=1761748728; i=andrei.perepiolkin@open-e.com; bh=X/ONh/rd4c41JfmHgb+y6z6F92Ti74Fn95/Zry+eJEY=; h=X-UI-Sender-Class:Message-ID:Date:MIME-Version:Subject:To: References:From:In-Reply-To:Content-Type: Content-Transfer-Encoding:cc:content-transfer-encoding: content-type:date:from:message-id:mime-version:reply-to:subject: to; b=WMkIw+PF4LoOQUcSuYjm4cnM/uctxwngTJuXpG/kqjDhOdwzHHst78L250JolyOU UzRSzpk4T90DBXeKABGB1+mCOW8HTIlnUyLc+49Tf+oNc9Mt4OZFzBzh7GHG05UFH 3BNlQZeUgR4Vg1c7zEO5HhCcckLXTrdfwE7jV0Fvql+Rc7ec+vLm4iPSrQiA9/yb2 fNGUwHDMDOm6CCdN1h0geZHuX7cyacTDhgaFebawmet+W8OZjoRDvIhbg6CLXp3B5 RFtkuwLIZ9GMZKLByLGR3JBOvZX1XDL5DhI1FBEhkneB+7qLLMmnuGxHXh/eqBLKo Ktjw6nyPUEGehCpK9A== X-UI-Sender-Class: 55c96926-9e95-11ee-ae09-1f7a4046a0f6 Received: from [10.137.0.75] ([149.102.246.34]) by mrelayeu.kundenserver.de (mreue109 [212.227.15.183]) with ESMTPSA (Nemesis) id 1Mbj3Y-1ubgTJ2qaM-00lKNQ; Wed, 22 Oct 2025 16:38:47 +0200 Message-ID: Date: Wed, 22 Oct 2025 10:38:45 -0400 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [pve-devel] Consistency in volume deletion in process of concurrent VM deletion To: =?UTF-8?Q?Fabian_Gr=C3=BCnbichler?= , Proxmox VE development discussion References: <1761126324.5glolx6b39.astroid@yuna.none> Content-Language: en-US From: Andrei Perepiolkin In-Reply-To: <1761126324.5glolx6b39.astroid@yuna.none> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:egUgZWJNM04AAxWd4g3wqYskqTh0qh2wKaExK/x9HdrNfMoX1rn JTnKHwNcTC/4noUhBBeKu1q8eMgsj16YCZ5nldnvl8SviodIA8pgxPsUuGkLW4urnnmfKbg ZYzRVpWPQqxTC6kFmHjbzDdYumAER/mkEenEdJUofGwzsS8avADLLf3CKK8BOoUUEoaWfH2 hDmlRiyiNtRTAzEwb18fA== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:VlfD0UlVymE=;aTOL1MGreM8uErb627sCV2Gp+yH 8dS/bUMuzJ373MxQSRTBJjtF7jwzK9TItIjd8a53D6jxp7r1pyeDsc3RbtP59iTkCJXG1Jrss Wm7oRBXnFzvexEVCQY1OHI2r5rR4hA/1J+LZ9uO3cjCk7kf75YomaLBwEbimrd4uAJNfXMQxG +KDqGYktDbat6OxQLUE/C+pnp76nWSGk/kYhLk6HR247cZr8E2np7B3VqRFCTSPySVTSv+rbk 9tHbWlrdOvaIL32QML8mhjFapch8J7RX+S8rYmmv6EzXQT6x/H87u7fpyiBF0FBB2yc44jtVv OTqS+RxaNoqiXNqdYz9K6y9cr8M0FXsoYvlSPKZvv0iCaN18/zhWAI0wDp/r5zYzSN6oyF9T1 Ee5PNI2rk8Xki7WFoO7cCEA15yvA9fEu/Jm6e8YWNH2EudmAldgy86WndVkrOSkw3//LetKkt Q360eOT6TkEPw7eMbECQKru2vuoh+amX/pWC6Ldpdb3svlmxFHCN9/dd4XbHhU3iomC/Us7QF lMJb+GjZPHzzQrAUKukjWS7Ang+VP/EYMcltG9AxeNDDD0fMy/KzUlJckyMwDCtZhFqr2cPu1 7caOs6eU9yZiTlHDR05knVLD8Dp2r84S+A4sOrkoP0/bRo3IlX+59nIVzgVMVQvydTuDzsVNT mgqfhpQ0PeAR+qsR0+FOhqfrzuM6UoA+yz5P7kO0X8AnyFgEc+uCWVR7DMgrFnErxrVY71ML4 QWDDQ5BVXfUYquL8UKBJqXdL6aJyrRiV2EUwDWdyYo81/opONdDCLImETQtDhhCBJXHxKErJh Z5TODbMqfpnpHztNIKiQiYQJqovDrQTMCOXwopg3OkBhPKd8n7wozz2+LQa46HTEi4cJeUcIL 1CPED3Lvr/wZZkzLxBSxR6E/7dPAbqTk40rD9di4AUdPxJLXCNnzalFkDvE84WVn+RAp6Du8y 1Bi/E996Nc5qx3tIPAwftkoBew7q4I0e3Pq2pMwpeoQDB4rFSKKm89VhFJxmOgqGRHbN6u2iz jgJ1ySWnJ7UUa1KHsHW5mQSBtBMcuKx+vXS0iGoJgLpUJ72Pu4cNgCDzMRfVM/DPFtKESMr5p E8WbJwaH59ny5oDrN4NMNTZUFFrPfq3YEk3IX2L/2+5G7XOE6TISCZEm6gW9SuKPVkFHx3ur7 msspmivaG0H900WAxTD9Ai2hVMjnUWY0WjycehQNKlJv4zAgr8uE2oH4HrRYPEfeIQzGFTqhU v1C7fsZs0nd1dr5SlfUKSDIRlWCAD2Gs1MmHOjU2Ulaoeq4J4odJqxla9JFStuFw5N9CfJsfK Grzm1l+lJXxatJitIuiXCMvVFeznDlKvghu6ErxjloZGFDrMW7vTLjEE3BDB+6he1WZNl5nuj W4CZ6w+cWsFyJIqW0LT2Pfnuq4xqrrcBjM7EMTRFGQSzfnJA4ECoXApdDPE7aLKYVEIFRHTaD 6zsCIuswBEzEfJYst+hQUm+gix6wYpzIz5pKUsQWqQy+m2FF3xwzOLD+1DHUYHu8rd/YlAtOV dW7GpH5w4Q1qGGy9ibNirKCGgyhigecQ1Av99iKIvsUIYk4nelvBWHLPx1BRd2aAqbguE2obl bx1VXXosd/LhX9U80Y+8um5SsHWofpz1r2WvalE6Mb7TmzrqEpZsFwfLjQhRn2vbwn5AcGLhI n6jytrX956wh+STRih6vaK8uCorJFul40o3oDuG5u7A6nwacTpB6qHSXloopoklkcFY+Kwa3Y Zx3smWVjh1AD7n8aP87TUE/DcMhYTFABZZg== X-SPAM-LEVEL: Spam detection results: 0 AWL -0.004 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain DMARC_PASS -0.1 DMARC pass policy RCVD_IN_MSPIKE_H2 0.001 Average reputation (+2) SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record Hi Fabian, I can try to prototype some proof-of-concept solution for 'lock=20 granularity'. Once it is done, the issue of ssh session termination should become clear. Im new to mail-based contribution and Proxmox code itself. So I will probably have questions on various topics. Should I send this questions via email, as messages in bugzila or via=20 other tool? Best regards, Andrei Perepiolkin On 10/22/25 05:49, Fabian Gr=C3=BCnbichler wrote: > On October 21, 2025 5:33 pm, Andrei Perepiolkin via pve-devel wrote: >> Hi Proxmox Community, >> >> There might be a potential consistency problem with Proxmox vm deletion= . >> >> If Proxmox receives multiple concurrent VM deletion requests, where eac= h >> VM has multiple disks located on shared storage. >> >> The deletion process may fail or hang when attempting to acquire the >> storage >> lock(https://github.com/proxmox/pve-storage/blob/master/src/PVE/Storage= .pm#L1196C1-L1209C7). >> >> ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> cfs-lock 'storage-jdss-Pool-2' error: got lock request timeout >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> cfs-lock 'storage-jdss-Pool-2' error: got lock request timeout >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> trying to acquire cfs lock 'storage-jdss-Pool-2' ... >> cfs-lock 'storage-jdss-Pool-2' error: got lock request timeout >> ... >> >> Eventually, the VM configuration files in /etc/pve are removed, but som= e >> VM disks may remain. >> >> Additionally, the Web UI shows all deletions as successful, even though >> some disks were not deleted. >> >> In my opinion, a VM should either be deleted completely=E2=80=94includi= ng all >> dependent resources=E2=80=94or the deletion should fail, leaving the VM >> configuration file with an updated state. > the underlying issue is that the scope of the lock taken for certain > storage operations is very big for shared storages. we could probably > reduce it to a more meaningful level for most such storages: > > https://bugzilla.proxmox.com/show_bug.cgi?id=3D1962 > > but the the error handling might also be lacking in this case, would > have to double-check. > >> Im reproducing this by: >> >> for i in `seq 401 420` ; do qm clone 104 $i --name "win-$i" --fu= ll >> --storage jdss-Pool-2 ; done; >> >> for i in `seq 401 410` ; do qm destroy $i >> --destroy-unreferenced-disks 1 --purge 1 & done ; >> >> >> Have to notice that ssh session that I use to conduct 'qm destroy' >> command get terminated by Proxmox. > that seems unexpected, are you sure this is caused by PVE? > >> Ive duplicated as a bug at: >> https://bugzilla.proxmox.com/show_bug.cgi?id=3D6957 > it would be better to either send a mail or file a bug, to not risk > splitting the discussion.. > >> Is this a bug and will it be addressed in near future? > nobody picked up the work regarding the lock granularity, but it would > be a nice improvement IMHO! > > Fabian > --===============8590879885619942561== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel --===============8590879885619942561==--