[PVE-User] Force reclaiming space on a vdisk...

[PVE-User] Force reclaiming space on a vdisk...

[Marco Gaiarin]

Situation: VM with a disk on a ZFS storage, 2TB disk; the disk have some
'system' partitions, and two big partition /dev/sda7 and /dev/sda8 for /home
and /srv.

I've added two more disk (/dev/sdb and /dev/sdc), created one partition per
disk and moved data from old partition to the ones. After that, i've deleted
/dev/sda7 and /dev/sda8, so now /dev/sda is a 2TB disk with roughly 100GB of
data in.

Space get not reclaimed. All disks have 'discard=1'.

I've tried to move /dev/sda to another ZFS storage (that have 500GB of
free space), and move fail.


How can i reclaim the free space on /dev/sda?! I need to create one
partition on the free space, format it, fstrim it?


Thanks.

-- 
  Siamo circondati da troppa gente piena di sé. E a quelli pieni di sé,
  io preferisco le persone piene di se, di ma, di forse. (Tonio Dell'Olio)

Re: [PVE-User] Force reclaiming space on a vdisk...

[Daniel Plominski]

Hello Marco,

enable ZFS set compression=lz4 on the zvolume and perform a memory limit 
dd run (inside the vm).

dd if=/dev/zero of=/mountpoint_sda7/CLEANUP bs=99M count=xxx

(this will release the storage space)

Mit freundlichen Grüßen

DANIEL PLOMINSKI
Leitung IT | Head of IT

Telefon 09265 808-151 | Mobil 0151 58026316 | dpl@ass.de
PGP Key: https://pgp.ass.de/dpl@ass.de.asc
PGP Fingerprint: 74DBC06BD9F63187C4DF8934C96585A89CFC10B3

		 Company Logo

			ASS-Einrichtungssysteme GmbH 		
			ASS-Adam-Stegner-Straße 19 | D-96342 Stockheim

			Geschäftsführer: Matthias Stegner, Michael Stegner, Ralph Müller 		
			Amtsgericht Coburg HRB 3395 | Ust-ID: DE218715721
Bottom_Line
From tsabolov@t8.ru  Wed Jan 19 12:22:38 2022
Return-Path: <tsabolov@t8.ru>
X-Original-To: pve-user@lists.proxmox.com
Delivered-To: pve-user@lists.proxmox.com
Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68])
 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange X25519 server-signature RSA-PSS (2048 bits))
 (No client certificate requested)
 by lists.proxmox.com (Postfix) with ESMTPS id 2F80E6202A
 for <pve-user@lists.proxmox.com>; Wed, 19 Jan 2022 12:22:38 +0100 (CET)
Received: from firstgate.proxmox.com (localhost [127.0.0.1])
 by firstgate.proxmox.com (Proxmox) with ESMTP id 24D7F7B21
 for <pve-user@lists.proxmox.com>; Wed, 19 Jan 2022 12:22:38 +0100 (CET)
Received: from relay162.nicmail.ru (relay162.nicmail.ru [91.189.117.6])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (No client certificate requested)
 by firstgate.proxmox.com (Proxmox) with ESMTPS id 216C77B0D
 for <pve-user@lists.proxmox.com>; Wed, 19 Jan 2022 12:22:37 +0100 (CET)
Received: from [10.28.138.149] (port"082 helo=[192.168.8.155])
 by relay.hosting.mail.nic.ru with esmtp (Exim 5.55)
 (envelope-from <tsabolov@t8.ru>) id 1nA92r-0007Xi-9f
 for pve-user@lists.proxmox.com; Wed, 19 Jan 2022 14:22:30 +0300
Received: from [62.105.41.93] (account tsabolov@t8.ru HELO [192.168.8.155])
 by incarp1102.int.hosting.nic.ru (Exim 5.55)
 with id 1nA92r-0004wk-S3 for pve-user@lists.proxmox.com;
 Wed, 19 Jan 2022 14:22:29 +0300
To: Proxmox VE user list <pve-user@lists.proxmox.com>
From: =?UTF-8?B?0KHQtdGA0LPQtdC5INCm0LDQsdC+0LvQvtCy?= <tsabolov@t8.ru>
Message-ID: <58d43866-6451-c266-7985-07ff89c0e05a@t8.ru>
Date: Wed, 19 Jan 2022 14:22:29 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.14.0
MIME-Version: 1.0
Content-Language: en-US
X-KLMS-AntiSpam-Auth: dkim=none
X-MS-Exchange-Organization-SCL: -1
X-SPAM-LEVEL: Spam detection results:  0
 AWL -0.665 Adjusted score from AWL reputation of From: address
 BAYES_00                 -1.9 Bayes spam probability is 0 to 1%
 HTML_MESSAGE            0.001 HTML included in message
 KAM_DMARC_STATUS 0.01 Test Rule for DKIM or SPF Failure with Strict Alignment
 RCVD_IN_DNSWL_NONE     -0.0001 Sender listed at https://www.dnswl.org/,
 no trust RCVD_IN_MSPIKE_H2      -0.001 Average reputation (+2)
 SPF_HELO_NONE           0.001 SPF: HELO does not publish an SPF Record
 SPF_PASS               -0.001 SPF: sender matches SPF record
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Content-Filtered-By: Mailman/MimeDel 2.1.29
Subject: [PVE-User] Unexpected reboot on of 6 node
X-BeenThere: pve-user@lists.proxmox.com
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Proxmox VE user list <pve-user.lists.proxmox.com>
List-Unsubscribe: <https://lists.proxmox.com/cgi-bin/mailman/options/pve-user>,
 <mailto:pve-user-request@lists.proxmox.com?subject=unsubscribe>
List-Archive: <http://lists.proxmox.com/pipermail/pve-user/>
List-Post: <mailto:pve-user@lists.proxmox.com>
List-Help: <mailto:pve-user-request@lists.proxmox.com?subject=help>
List-Subscribe: <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>,
 <mailto:pve-user-request@lists.proxmox.com?subject=subscribe>
X-List-Received-Date: Wed, 19 Jan 2022 11:22:38 -0000

Hi,

Like in this old thread
https://forum.proxmox.com/threads/unexpected-reboots-help-need.34310/ I
have similar problem.

In cluster I have 7 node.

root@pve-1: pveversion -v

proxmox-ve: 6.4-1 (running kernel: 5.4.143-1-pve)
pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
pve-kernel-helper: 6.4-8
pve-kernel-5.4: 6.4-7
pve-kernel-5.4.143-1-pve: 5.4.143-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
ceph: 15.2.15-pve1~bpo10
ceph-fuse: 15.2.15-pve1~bpo10
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve4~bpo10
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve1~bpo10+1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-4
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-3
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.13-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-1
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.6-pve1~bpo10+1

I need to disable complete  the  IPMI Watchdog or  Dell IDrac (module
"ipmi_watchdog") For Dell IDrac, please desactivate the Automated System
Recovery Agent in IDrac configuration.

https://pve.proxmox.com/wiki/High_Availability_Cluster_4.x#IPMI_Watchdog_.28module_.22ipmi_watchdog.22.29

Sergey TS
The best Regard

_______________________________________________
pve-user mailing list
pve-user@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Force reclaiming space on a vdisk...

[Marco Gaiarin]

Mandi! Daniel Plominski
  In chel di` si favelave...

> enable ZFS set compression=lz4 on the zvolume

Seems just enabled:

 root@ctpve1:~# zpool get all rpool | grep lz4
 rpool  feature@lz4_compress           active                         local
 root@ctpve1:~# zpool get all rpool-data | grep lz4
 rpool-data  feature@lz4_compress           active                         local


> and perform a memory limit dd run (inside the vm).
> dd if=/dev/zero of=/mountpoint_sda7/CLEANUP bs=99M count=xxx
> (this will release the storage space)

OK. But i've just deleted the partitions. I have to create a new partition,
format them, create a 'dd-zero' file in them and then the space will be
released?

Really?!


I'm asking because i supposed that was the 'trim/thin' feature of ZFS to
permit to shrink a disk (eg, 'don't save the unallocated space'), not the
compressione feature (eg, 'don't save a bunch of consecutive zero, compress
it').


Speaking more clearly, i hope: i've perfectly clear that 'zeroing' a portion
of a disk permit the compression feature of zfs to compress it, but i
supposed that was the management of allocated spaces that make their
business here... i'm only a bit puzzled.


I hope someone can clarify, thanks.


PS: this server have a 'Proxmox VE Community Subscription 1 CPU/year'
 currently active on, but i prefere if possible to use mailing list for this
 support question. FYI.

-- 
  Ognuno vada dove vuole andare, ognuno invecchi come gli pare
  ma non raccontate a me che cos'e` la LIBERTA`.	(F. Guccini)

Re: [PVE-User] Force reclaiming space on a vdisk...

[Daniel Plominski]

Hello,

we have had very bad experience with trim, due to a virtio driver error, 
a special release, the trim resulted in a corrupt Windows NTFS file 
system.

The bug is fixed in the latest stable VirtIO drivers, but we only use 
the legacy method using targeted "dd" (zero) cleanup on Linux / BSD VMs 
and on Windows using "fsutil".

From the perspective of the HOST system, a zvolume formatted with 
filesystem and stored data is simply a "data container" with 
unstructured data (depending on the structure, however, this can be 
compressed well).
If now within the VM an area with zeros is written, this zvolume area is 
stimulated to overwrite the contained data, the process is compressed 
and the occupied blocks are released.

The block free up should also work with the ZFS algorithm "zle".

How often the cells of an SSD / NVME memory are overwritten depends on 
the internal intelligence of the firmware / controller of the respective 
memory.

TRIM "Discard" (if supported by the hardware, the hypervisor, the 
hypervisor driver and the guest OS) can be used to better control this 
remapping.

ASS - Der Bildungseinrichter GmbH
Mit freundlichen Grüßen

DANIEL PLOMINSKI
Leitung IT | Head of IT

Telefon 09265 808-151 | Mobil 0151 58026316 | dpl@ass.de
PGP Key: https://pgp.ass.de/dpl@ass.de.asc
PGP Fingerprint: 74DBC06BD9F63187C4DF8934C96585A89CFC10B3

		 Company Logo

			ASS-Einrichtungssysteme GmbH 		
			ASS-Adam-Stegner-Straße 19 | D-96342 Stockheim

			Geschäftsführer: Matthias Stegner, Michael Stegner, Ralph Müller 		
			Amtsgericht Coburg HRB 3395 | Ust-ID: DE218715721
Bottom_Line

Re: [PVE-User] Force reclaiming space on a vdisk...

[Marco Gaiarin]

Mandi! Arjen via pve-user
  In chel di` si favelave...

> When discard is enabled on the virtual disk and also the OS inside the VM executes trim commands for deleted data, then it should inform ZFS that the blocks are free. Marking blocks as usused will not work without the help of the OS inside the VM.

OK. So, speking practically:

a) i've do the wrong things, deleting the partitions, because the guest OS
 had not the opportunity to reclaim free space; because trim/discard need
the cooperation of all the 'chain', probably it was needed to delete all the
file in the partition, the trim them, then delete them.

b) if the guest OS does not support trim, or support wrongly/buggy, the same
 result will be achieved (creating and) zeroing the partition, if
conpression are enabled. This does not involve trim/discard, but only
compression, acheving the same result indeed.


> Was trimming enabled for the OS inside the VM? Otherwise, you need to create a partition and trim it inside the VM.

Guest os support fstrim, so i'll try a).


Thanks.

-- 
  Con Windows sei in vacanza: ti diverti senza pensare a ciò che fai,
  ma dopo un po' finisce. In Linux entri nella vita reale:
  Devi tirar fuori le palle!				(Alain Modolo)

Re: [PVE-User] Force reclaiming space on a vdisk...

[Marco Gaiarin]

> Guest os support fstrim, so i'll try a).

For the archive: a) worked.

Also, formatting (i've used mke2fs) trigger trim, so i was not forced to
format and thet trim the disk, only format.


Thanks.

-- 
  Utopia aveva una sorella maggiore,
  che si chiamava Verita` senza errore			(Nomadi)