[pve-devel] [pve-manager] Adding real disk usage information (discussion)

[pve-devel] [pve-manager] Adding real disk usage information (discussion)

[Bruce Wainer]

Hello,

I am interested in seeing real disk usage information for VM Disks and CT
Volumes, on storage types that have thin provisioning and/or snapshots.
Specifically I would like to see "Current Disk Usage (Thin)" and either
"Snapshot Usage" or "Total Disk Usage". I only use local ZFS on servers at
this time, but I'm sure the GUI side would be best made flexible.

Is someone interested in helping with this? Where would I start, especially
on the GUI part, if I were to develop this myself?

Thank you,
Bruce Wainer

Re: [pve-devel] [pve-manager] Adding real disk usage information (discussion)

[Dominik Csapak]

On 4/16/21 22:18, Bruce Wainer wrote:
> Hello,
> 

Hi,

> I am interested in seeing real disk usage information for VM Disks and CT
> Volumes, on storage types that have thin provisioning and/or snapshots.
> Specifically I would like to see "Current Disk Usage (Thin)" and either
> "Snapshot Usage" or "Total Disk Usage". I only use local ZFS on servers at
> this time, but I'm sure the GUI side would be best made flexible.

while this sounds sensible, this will get hard very fast.
For example, take a LVM-Thin storage.

I have a template which has an LV which uses some space.
This can have X linked clones, where each clone can have Y snapshots.

since lvmthin lvs/snapshots/etc. are only very loosely coupled.
It is very hard to attribute the correct number to any
of those vms/templates. (e.g. do you want to calculate the
template storage again for each vm? only once? what if
you cloned a vm from a snapshot?)

It gets even harder on storage that can deduplicate (e.g. ZFS) or
where the 'real' usage is dynamically inflated by some form of replica 
(e.g. Ceph).

So, while this sounds nice, and we would probably not oppose a clean
solution, this is not a trivial problem to solve.

> 
> Is someone interested in helping with this? Where would I start, especially
> on the GUI part, if I were to develop this myself?

anyway, to answer this question, the storage plugins in the backend can
be found in the pve-storage git repo[0]

the point where the status api calls of the vms/cts are called live
in qemu-server[1] and pve-container[2] respectively
(the api part is in PVE/API2/)

you can find the gui part in pve-manger[3] in www/manager6

also if you want to send patches, please read the developer 
documentation [4] especially the bit about the CLA

if you have any more question, please ask :)

hope this helps
kind regards

0: 
https://git.proxmox.com/?p=pve-storage.git;a=tree;f=PVE/Storage;h=fd53af5e74407deda65785b164fb61a4f644a6e0;hb=refs/heads/master
1: https://git.proxmox.com/?p=qemu-server.git;a=summary
2: https://git.proxmox.com/?p=pve-container.git;a=summary
3: 
https://git.proxmox.com/?p=pve-manager.git;a=tree;f=www/manager6;hb=refs/heads/master
4: https://pve.proxmox.com/wiki/Developer_Documentation

> 
> Thank you,
> Bruce Wainer
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
> 

Re: [pve-devel] [pve-manager] Adding real disk usage information (discussion)

[Bruce Wainer]

Dominik,
Thank you for the insight. There is certainly complexity I did not
consider, even if I were to look at the narrow case of local ZFS storage.
Regardless, this would be helpful to me and if I make anything then I will
submit it. I already have signed the CLA and have code accepted in
pve-zsync.
Thank you,
Bruce

On Mon, Apr 19, 2021 at 2:55 AM Dominik Csapak <d.csapak@proxmox.com> wrote:

> On 4/16/21 22:18, Bruce Wainer wrote:
> > Hello,
> >
>
> Hi,
>
> > I am interested in seeing real disk usage information for VM Disks and CT
> > Volumes, on storage types that have thin provisioning and/or snapshots.
> > Specifically I would like to see "Current Disk Usage (Thin)" and either
> > "Snapshot Usage" or "Total Disk Usage". I only use local ZFS on servers
> at
> > this time, but I'm sure the GUI side would be best made flexible.
>
> while this sounds sensible, this will get hard very fast.
> For example, take a LVM-Thin storage.
>
> I have a template which has an LV which uses some space.
> This can have X linked clones, where each clone can have Y snapshots.
>
> since lvmthin lvs/snapshots/etc. are only very loosely coupled.
> It is very hard to attribute the correct number to any
> of those vms/templates. (e.g. do you want to calculate the
> template storage again for each vm? only once? what if
> you cloned a vm from a snapshot?)
>
> It gets even harder on storage that can deduplicate (e.g. ZFS) or
> where the 'real' usage is dynamically inflated by some form of replica
> (e.g. Ceph).
>
> So, while this sounds nice, and we would probably not oppose a clean
> solution, this is not a trivial problem to solve.
>
> >
> > Is someone interested in helping with this? Where would I start,
> especially
> > on the GUI part, if I were to develop this myself?
>
> anyway, to answer this question, the storage plugins in the backend can
> be found in the pve-storage git repo[0]
>
> the point where the status api calls of the vms/cts are called live
> in qemu-server[1] and pve-container[2] respectively
> (the api part is in PVE/API2/)
>
> you can find the gui part in pve-manger[3] in www/manager6
>
> also if you want to send patches, please read the developer
> documentation [4] especially the bit about the CLA
>
> if you have any more question, please ask :)
>
> hope this helps
> kind regards
>
> 0:
>
> https://git.proxmox.com/?p=pve-storage.git;a=tree;f=PVE/Storage;h=fd53af5e74407deda65785b164fb61a4f644a6e0;hb=refs/heads/master
> 1: https://git.proxmox.com/?p=qemu-server.git;a=summary
> 2: https://git.proxmox.com/?p=pve-container.git;a=summary
> 3:
>
> https://git.proxmox.com/?p=pve-manager.git;a=tree;f=www/manager6;hb=refs/heads/master
> 4: https://pve.proxmox.com/wiki/Developer_Documentation
>
> >
> > Thank you,
> > Bruce Wainer
> > _______________________________________________
> > pve-devel mailing list
> > pve-devel@lists.proxmox.com
> > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> >
> >
>
>
>