From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <oscar@dearriba.es>
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by lists.proxmox.com (Postfix) with ESMTPS id 2941E92D8C
for <pve-user@lists.proxmox.com>; Mon, 2 Jan 2023 10:57:30 +0100 (CET)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
by firstgate.proxmox.com (Proxmox) with ESMTP id 033532DC0D
for <pve-user@lists.proxmox.com>; Mon, 2 Jan 2023 10:57:00 +0100 (CET)
Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com
[64.147.123.24])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by firstgate.proxmox.com (Proxmox) with ESMTPS
for <pve-user@lists.proxmox.com>; Mon, 2 Jan 2023 10:56:58 +0100 (CET)
Received: from compute5.internal (compute5.nyi.internal [10.202.2.45])
by mailout.west.internal (Postfix) with ESMTP id 8D218320091B;
Mon, 2 Jan 2023 04:56:49 -0500 (EST)
Received: from imap50 ([10.202.2.100])
by compute5.internal (MEProxy); Mon, 02 Jan 2023 04:56:49 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dearriba.es; h=
cc:content-type:date:date:from:from:in-reply-to:in-reply-to
:message-id:mime-version:references:reply-to:sender:subject
:subject:to:to; s=fm1; t=1672653409; x=1672739809; bh=ERKGUQEuF2
tuFTYPtbcU530v/ivQ5aAWmxXx76xZXE8=; b=DJeVJmkI5ID0JnmxMLGGFPebIT
vKsbK+B9TkDAB0Vx+RkG+tTRF+7LRVlQ5QnIYxS5DmpgweVWTa7+Ay5OBs8iru5Z
HXERb1E75t8USGLLBgqr9gHzwFDZny40FxBVi3CHOMzGXcARAGWgSb3q1IA/gdtv
3yj5nSWbtwPJZ7/CEUa5oTXw/uRfsDJXm0LYeheQ/51MVXxpNhU0iYCBDw3b7Aej
Gli2HAWu5oZ1V3jzPztCVZnnDi8G/tyyFQtVdyO/IboGQG7JJQe0jlabmiaGpfKo
Juq893e8EDHjwjvwMT5WrsYZHawo6j6aR417Z8jK3fCa4eO8/bDDFLlyPI9g==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=
messagingengine.com; h=cc:content-type:date:date:feedback-id
:feedback-id:from:from:in-reply-to:in-reply-to:message-id
:mime-version:references:reply-to:sender:subject:subject:to:to
:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=
fm2; t=1672653409; x=1672739809; bh=ERKGUQEuF2tuFTYPtbcU530v/ivQ
5aAWmxXx76xZXE8=; b=XwTGMYvBqKubgsBGG4otfaEkGKyldcQzN04MHUZ+g4al
IKzA5zPNAXhZu4g/5UD7fpt/iXUQ+F9OWkLtNGWq8OMc27XuEH5pXusgvui/vtHt
8LTbcddTPm6yS68+nSMt3OSddgJJ2FQSzHuNQ9AismyCytq83pfRlz8YvnlJrWbq
kuMOJxJ+LUhTNjt3v/IWpmyy5V5nJrz4zJieqtwzFZKyRm49YZ+W3opk190lBDKb
BDp9/MH6uePZgGKSpuALFNcxmCBqsQZNLoa6LPp+zAZXdgzt3pHdcfRYOIKP+yuJ
FsOQpZGbzh3Coqx+Hz1EBiGjJHkev/eLANk0tbg9CA==
X-ME-Sender: <xms:YKqyY0BRvFESIdykBtdaWdTv01vYb-Z7A6FnkapDBx7ES_2uWJ_D9w>
<xme:YKqyY2g0JK3TG16lxGhTOKj25MVCH0Q1q-2GXAXPlxmtTD5UqdDzJIoDLrQs3UND7
kXNDFN61uwpq864QsA>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrjedvgddtlecutefuodetggdotefrodftvf
curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu
uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc
fjughrpefofgggkfgjfhffhffvufgtsegrtderreerreejnecuhfhrohhmpenmshgtrghr
ucguvgcutehrrhhisggruceoohhstggrrhesuggvrghrrhhisggrrdgvsheqnecuggftrf
grthhtvghrnhephedtteeukeefhfeukeelieefteegfeekkefhkeekgfeijeelkeehtdej
leduueeknecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh
epohhstggrrhesuggvrghrrhhisggrrdgvsh
X-ME-Proxy: <xmx:YKqyY3kYwLYrvJCv3mP95COW3d77t7hNFgUmgPrJBPdrdgc5iKudAg>
<xmx:YKqyY6yNNihNQqOoqjqL94C9Audv6p4ubec3huWQ7xRDMgG6csCDrQ>
<xmx:YKqyY5SeViFVi77QfVM2AXg9yFCXWDFpaHJRL2ErEwvtZMZd7BPd2w>
<xmx:YaqyY0OMA2kHezu56QrBJcBb4IugopApsOuBRB_k3T_LRQt-miUnbA>
Feedback-ID: ib1694623:Fastmail
Received: by mailuser.nyi.internal (Postfix, from userid 501)
id 924721700090; Mon, 2 Jan 2023 04:56:48 -0500 (EST)
X-Mailer: MessagingEngine.com Webmail Interface
User-Agent: Cyrus-JMAP/3.7.0-alpha0-1185-g841157300a-fm-20221208.002-g84115730
Mime-Version: 1.0
Message-Id: <18fed6c7-94cd-4cfd-b8b5-9c7f6a92b186@app.fastmail.com>
In-Reply-To: <cae87493-98a5-4256-b9a3-8af41931afea@app.fastmail.com>
References: <mailman.3.1672225201.17323.pve-user@lists.proxmox.com>
<e207771d-7f93-415a-82fd-c3c8488d1e80@app.fastmail.com>
<b462e244-86a1-eada-c50b-4361f037dc1e@holub.co.at>
<64119642-e2a3-428a-a053-371f9fc6bda0@app.fastmail.com>
<cae87493-98a5-4256-b9a3-8af41931afea@app.fastmail.com>
Date: Mon, 02 Jan 2023 10:56:27 +0100
From: =?UTF-8?Q?=C3=93scar_de_Arriba?= <oscar@dearriba.es>
To: "Martin Holub" <martin@holub.co.at>, pve-user@lists.proxmox.com
X-SPAM-LEVEL: Spam detection results: 0
BAYES_00 -1.9 Bayes spam probability is 0 to 1%
DKIM_SIGNED 0.1 Message has a DKIM or DK signature,
not necessarily valid
DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature
DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's
domain
DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from
domain HTML_MESSAGE 0.001 HTML included in message
JMQ_SPF_NEUTRAL 0.5 SPF set to ?all
RCVD_IN_DNSWL_LOW -0.7 Sender listed at https://www.dnswl.org/,
low trust RCVD_IN_MSPIKE_H2 -0.001 Average reputation (+2)
SPF_HELO_PASS -0.001 SPF: HELO matches SPF record
SPF_PASS -0.001 SPF: sender matches SPF record
Content-Type: text/plain;charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Content-Filtered-By: Mailman/MimeDel 2.1.29
Subject: Re: [PVE-User] Thin LVM showing more used space than expected
X-BeenThere: pve-user@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE user list <pve-user.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>,
<mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-user/>
List-Post: <mailto:pve-user@lists.proxmox.com>
List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>,
<mailto:pve-user-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Mon, 02 Jan 2023 09:57:30 -0000
I ended up reinstalling th server and using ZFS this time. It solved all=
the issues - the ghost used space is back and I could restore even a VM=
3x times bigger without any slowdown.=20
Not sure what happened but with all that space occupied my best guess is=
that during the restoration of VMs LVM was getting out of blocks to use=
and needed to do some cleanups, which led to that slow performance at ~=
5Mbps. However, it is strange that I could not find anyone else facing t=
his issue.
On Thu, Dec 29, 2022, at 17:58, =C3=93scar de Arriba wrote:
> Update: I have enabled `Discard` option in all disks of the VMs on tha=
t server and then `fstim` did the work and freed some space.
>=20
> However, even removing all VMs except one (which is hard to remove wit=
hout disruption) I can see that:
>=20
> root@venom:~# lvs
> LV VG Attr LSize Pool Origin Data% Meta% Move =
Log Cpy%Sync Convert
> data pve twi-aotz-- 377.55g 60.65 0.67 =
=20
> root pve -wi-ao---- 60.00g =
=20
> swap pve -wi-ao---- 4.00g =
=20
> vm-201-disk-0 pve Vwi-aotz-- 4.00m data 14.06 =
=20
> vm-201-disk-1 pve Vwi-aotz-- 40.00g data 56.58
>=20
> Which means that I have about 200 GB used out of nowhere :( At least i=
t is no longer under pressure of being almost 100% full.
>=20
> On Thu, Dec 29, 2022, at 11:48, =C3=93scar de Arriba wrote:
>> Any idea why it still has 96.23% of space used but the VMs are using =
way less? I'm starting to worry a lot about it (I don't kant tobe really=
full) and my current only hope is backup + reinstall PVE.
>>=20
>> Thanks,
>> Oscar
>>=20
>> On Thu, Dec 29, 2022, at 11:01, Martin Holub wrote:
>>>=20
>>>=20
>>> Am 28.12.2022 um 12:44 schrieb =C3=93scar de Arriba:
>>>> Hi Martin,
>>>>=20
>>>> > Did you try to run a fstrim on the VMs to regain the allocated sp=
ace? At least on linux something like "fstrim -av" should do the trick.
>>>>=20
>>>> I did it now and it freed ~55GiB of a running isntance (the one wit=
h 128 GiB allocated). However that should only free blocks of the LV use=
d to store that VM disk, right? And the issue itself is that the sum of =
maximum allocations of those disks is much lower than the space occupied.
>>>>=20
>>>> I also have the feeling that those blocks remain used by a no longe=
r existant LVs, but I don't know how to fix it.
>>>>=20
>>>> Should I also enable trim/execute trim on Proxmox itself?
>>>>=20
>>>> Oscar
>>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>> Hi,
>>>=20
>>> TRIM only works on a filesystem level, so you can't trim a VG or sim=
ilar. On the pve host i doubt it will help, but it wouldn't harm either.=20
>>>=20
>>> hth
>>> Martin
>>>=20
>>>=20
>>=20
>=20