From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 2941E92D8C for ; Mon, 2 Jan 2023 10:57:30 +0100 (CET) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 033532DC0D for ; Mon, 2 Jan 2023 10:57:00 +0100 (CET) Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS for ; Mon, 2 Jan 2023 10:56:58 +0100 (CET) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 8D218320091B; Mon, 2 Jan 2023 04:56:49 -0500 (EST) Received: from imap50 ([10.202.2.100]) by compute5.internal (MEProxy); Mon, 02 Jan 2023 04:56:49 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dearriba.es; h= cc:content-type:date:date:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to; s=fm1; t=1672653409; x=1672739809; bh=ERKGUQEuF2 tuFTYPtbcU530v/ivQ5aAWmxXx76xZXE8=; b=DJeVJmkI5ID0JnmxMLGGFPebIT vKsbK+B9TkDAB0Vx+RkG+tTRF+7LRVlQ5QnIYxS5DmpgweVWTa7+Ay5OBs8iru5Z HXERb1E75t8USGLLBgqr9gHzwFDZny40FxBVi3CHOMzGXcARAGWgSb3q1IA/gdtv 3yj5nSWbtwPJZ7/CEUa5oTXw/uRfsDJXm0LYeheQ/51MVXxpNhU0iYCBDw3b7Aej Gli2HAWu5oZ1V3jzPztCVZnnDi8G/tyyFQtVdyO/IboGQG7JJQe0jlabmiaGpfKo Juq893e8EDHjwjvwMT5WrsYZHawo6j6aR417Z8jK3fCa4eO8/bDDFLlyPI9g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:in-reply-to:message-id :mime-version:references:reply-to:sender:subject:subject:to:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1672653409; x=1672739809; bh=ERKGUQEuF2tuFTYPtbcU530v/ivQ 5aAWmxXx76xZXE8=; b=XwTGMYvBqKubgsBGG4otfaEkGKyldcQzN04MHUZ+g4al IKzA5zPNAXhZu4g/5UD7fpt/iXUQ+F9OWkLtNGWq8OMc27XuEH5pXusgvui/vtHt 8LTbcddTPm6yS68+nSMt3OSddgJJ2FQSzHuNQ9AismyCytq83pfRlz8YvnlJrWbq kuMOJxJ+LUhTNjt3v/IWpmyy5V5nJrz4zJieqtwzFZKyRm49YZ+W3opk190lBDKb BDp9/MH6uePZgGKSpuALFNcxmCBqsQZNLoa6LPp+zAZXdgzt3pHdcfRYOIKP+yuJ FsOQpZGbzh3Coqx+Hz1EBiGjJHkev/eLANk0tbg9CA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrjedvgddtlecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefofgggkfgjfhffhffvufgtsegrtderreerreejnecuhfhrohhmpenmshgtrghr ucguvgcutehrrhhisggruceoohhstggrrhesuggvrghrrhhisggrrdgvsheqnecuggftrf grthhtvghrnhephedtteeukeefhfeukeelieefteegfeekkefhkeekgfeijeelkeehtdej leduueeknecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh epohhstggrrhesuggvrghrrhhisggrrdgvsh X-ME-Proxy: Feedback-ID: ib1694623:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id 924721700090; Mon, 2 Jan 2023 04:56:48 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.7.0-alpha0-1185-g841157300a-fm-20221208.002-g84115730 Mime-Version: 1.0 Message-Id: <18fed6c7-94cd-4cfd-b8b5-9c7f6a92b186@app.fastmail.com> In-Reply-To: References: <64119642-e2a3-428a-a053-371f9fc6bda0@app.fastmail.com> Date: Mon, 02 Jan 2023 10:56:27 +0100 From: =?UTF-8?Q?=C3=93scar_de_Arriba?= To: "Martin Holub" , pve-user@lists.proxmox.com X-SPAM-LEVEL: Spam detection results: 0 BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain HTML_MESSAGE 0.001 HTML included in message JMQ_SPF_NEUTRAL 0.5 SPF set to ?all RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/, low trust RCVD_IN_MSPIKE_H2 -0.001 Average reputation (+2) SPF_HELO_PASS -0.001 SPF: HELO matches SPF record SPF_PASS -0.001 SPF: sender matches SPF record Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [PVE-User] Thin LVM showing more used space than expected X-BeenThere: pve-user@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE user list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jan 2023 09:57:30 -0000 I ended up reinstalling th server and using ZFS this time. It solved all= the issues - the ghost used space is back and I could restore even a VM= 3x times bigger without any slowdown.=20 Not sure what happened but with all that space occupied my best guess is= that during the restoration of VMs LVM was getting out of blocks to use= and needed to do some cleanups, which led to that slow performance at ~= 5Mbps. However, it is strange that I could not find anyone else facing t= his issue. On Thu, Dec 29, 2022, at 17:58, =C3=93scar de Arriba wrote: > Update: I have enabled `Discard` option in all disks of the VMs on tha= t server and then `fstim` did the work and freed some space. >=20 > However, even removing all VMs except one (which is hard to remove wit= hout disruption) I can see that: >=20 > root@venom:~# lvs > LV VG Attr LSize Pool Origin Data% Meta% Move = Log Cpy%Sync Convert > data pve twi-aotz-- 377.55g 60.65 0.67 = =20 > root pve -wi-ao---- 60.00g = =20 > swap pve -wi-ao---- 4.00g = =20 > vm-201-disk-0 pve Vwi-aotz-- 4.00m data 14.06 = =20 > vm-201-disk-1 pve Vwi-aotz-- 40.00g data 56.58 >=20 > Which means that I have about 200 GB used out of nowhere :( At least i= t is no longer under pressure of being almost 100% full. >=20 > On Thu, Dec 29, 2022, at 11:48, =C3=93scar de Arriba wrote: >> Any idea why it still has 96.23% of space used but the VMs are using = way less? I'm starting to worry a lot about it (I don't kant tobe really= full) and my current only hope is backup + reinstall PVE. >>=20 >> Thanks, >> Oscar >>=20 >> On Thu, Dec 29, 2022, at 11:01, Martin Holub wrote: >>>=20 >>>=20 >>> Am 28.12.2022 um 12:44 schrieb =C3=93scar de Arriba: >>>> Hi Martin, >>>>=20 >>>> > Did you try to run a fstrim on the VMs to regain the allocated sp= ace? At least on linux something like "fstrim -av" should do the trick. >>>>=20 >>>> I did it now and it freed ~55GiB of a running isntance (the one wit= h 128 GiB allocated). However that should only free blocks of the LV use= d to store that VM disk, right? And the issue itself is that the sum of = maximum allocations of those disks is much lower than the space occupied. >>>>=20 >>>> I also have the feeling that those blocks remain used by a no longe= r existant LVs, but I don't know how to fix it. >>>>=20 >>>> Should I also enable trim/execute trim on Proxmox itself? >>>>=20 >>>> Oscar >>>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>> Hi, >>>=20 >>> TRIM only works on a filesystem level, so you can't trim a VG or sim= ilar. On the pve host i doubt it will help, but it wouldn't harm either.=20 >>>=20 >>> hth >>> Martin >>>=20 >>>=20 >>=20 >=20