[pve-devel] [PATCH pve-manager] pvestatd: stream host pressure counters

[pve-devel] [PATCH pve-manager] pvestatd: stream host pressure counters

[Alexandre Derumier]

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/Service/pvestatd.pm | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
index 5e533ca3..238c36ae 100755
--- a/PVE/Service/pvestatd.pm
+++ b/PVE/Service/pvestatd.pm
@@ -161,12 +161,16 @@ sub update_node_status {
     );
     PVE::Cluster::broadcast_rrd("pve2-node/$nodename", $data);
 
+    my $pressure = PVE::ProcFSTools::read_pressure();
+    PVE::Cluster::broadcast_node_kv("pressure", encode_json($pressure));
+
     my $node_metric = {
 	uptime => $uptime,
 	cpustat => $stat,
 	memory => $meminfo,
 	blockstat => $dinfo,
 	nics => $netdev,
+	pressure => $pressure,
     };
     $node_metric->{cpustat}->@{qw(avg1 avg5 avg15)} = ($avg1, $avg5, $avg15);
     $node_metric->{cpustat}->{cpus} = $maxcpu;
-- 
2.20.1

Re: [pve-devel] [PATCH pve-manager] pvestatd: stream host pressure counters

[Dietmar Maurer]

I wonder if we want to store that information with RRD instead?

> On 10/06/2020 1:58 PM Alexandre Derumier <aderumier@odiso.com> wrote:
> 
>  
> Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
> ---
>  PVE/Service/pvestatd.pm | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
> index 5e533ca3..238c36ae 100755
> --- a/PVE/Service/pvestatd.pm
> +++ b/PVE/Service/pvestatd.pm
> @@ -161,12 +161,16 @@ sub update_node_status {
>      );
>      PVE::Cluster::broadcast_rrd("pve2-node/$nodename", $data);
>  
> +    my $pressure = PVE::ProcFSTools::read_pressure();
> +    PVE::Cluster::broadcast_node_kv("pressure", encode_json($pressure));
> +

Re: [pve-devel] [PATCH pve-manager] pvestatd: stream host pressure counters

[Alexandre Derumier]

Hi Dietmar,

I would like also to add some improvement for vm memory/cpu stats.

for cpu, currently, we only monitor the qemu process cpu usage, but
with virtio-net + vhost-net,
we are missing vhost-* process cpu usage.  (For vms will a lot of
traffic, this is really signifiant).

I would like to add something like "hostcpu" counter, will real metric
(reading the vm cgroup cpu.stat)

Also, for memory, when ballooning is enabled, we don't have anymore
the real memory usage on the host.
For example, a windows vm with 16g mem is filling memory with zeroes
at boot, so from the host all the 16G memory is used, but in windows
guest we only see something like 2G.

so, I would like to add something like "hostmemory" counter


what do you think about this ?


Le mar. 13 oct. 2020 à 07:57, Dietmar Maurer <dietmar@proxmox.com> a écrit :
>
> I wonder if we want to store that information with RRD instead?
>
> > On 10/06/2020 1:58 PM Alexandre Derumier <aderumier@odiso.com> wrote:
> >
> >
> > Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
> > ---
> >  PVE/Service/pvestatd.pm | 4 ++++
> >  1 file changed, 4 insertions(+)
> >
> > diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
> > index 5e533ca3..238c36ae 100755
> > --- a/PVE/Service/pvestatd.pm
> > +++ b/PVE/Service/pvestatd.pm
> > @@ -161,12 +161,16 @@ sub update_node_status {
> >      );
> >      PVE::Cluster::broadcast_rrd("pve2-node/$nodename", $data);
> >
> > +    my $pressure = PVE::ProcFSTools::read_pressure();
> > +    PVE::Cluster::broadcast_node_kv("pressure", encode_json($pressure));
> > +
>

Re: [pve-devel] [PATCH pve-manager] pvestatd: stream host pressure counters

[Dietmar Maurer]

> I would like also to add some improvement for vm memory/cpu stats.
> 
> for cpu, currently, we only monitor the qemu process cpu usage, but
> with virtio-net + vhost-net,
> we are missing vhost-* process cpu usage.  (For vms will a lot of
> traffic, this is really signifiant).
> 
> I would like to add something like "hostcpu" counter, will real metric
> (reading the vm cgroup cpu.stat)

make sense.

> Also, for memory, when ballooning is enabled, we don't have anymore
> the real memory usage on the host.
> For example, a windows vm with 16g mem is filling memory with zeroes
> at boot, so from the host all the 16G memory is used, but in windows
> guest we only see something like 2G.
> 
> so, I would like to add something like "hostmemory" counter
> 
> 
> what do you think about this ?

Sounds reasonable. We just need to find a way to add new values
in a compatible way.

Re: [pve-devel] [PATCH pve-manager] pvestatd: stream host pressure counters

[Alexandre Derumier]

> Sounds reasonable. We just need to find a way to add new values
> in a compatible way.

Last time in 2013, we had added a new rrd "pve2.3-vm/"
https://lists.proxmox.com/pipermail/pve-devel/2013-January/005612.html

Le ven. 23 oct. 2020 à 09:50, Dietmar Maurer <dietmar@proxmox.com> a écrit :
>
> > I would like also to add some improvement for vm memory/cpu stats.
> >
> > for cpu, currently, we only monitor the qemu process cpu usage, but
> > with virtio-net + vhost-net,
> > we are missing vhost-* process cpu usage.  (For vms will a lot of
> > traffic, this is really signifiant).
> >
> > I would like to add something like "hostcpu" counter, will real metric
> > (reading the vm cgroup cpu.stat)
>
> make sense.
>
> > Also, for memory, when ballooning is enabled, we don't have anymore
> > the real memory usage on the host.
> > For example, a windows vm with 16g mem is filling memory with zeroes
> > at boot, so from the host all the 16G memory is used, but in windows
> > guest we only see something like 2G.
> >
> > so, I would like to add something like "hostmemory" counter
> >
> >
> > what do you think about this ?
>
> Sounds reasonable. We just need to find a way to add new values
> in a compatible way.
>

Re: [pve-devel] [PATCH pve-manager] pvestatd: stream host pressure counters

[Thomas Lamprecht]

On 24.10.20 15:46, Alexandre Derumier wrote:
>> Sounds reasonable. We just need to find a way to add new values
>> in a compatible way.
> 
> Last time in 2013, we had added a new rrd "pve2.3-vm/"
> https://lists.proxmox.com/pipermail/pve-devel/2013-January/005612.html

that'd drop all old statistics, and be a bit of a nuisance to upgrade inside
a cluster (old nodes get stuff they do not comprehend).

We may rather want to have a new, separate, file for this.
We could add general support for and error handling now, and to the final add
with 7.0 next year or so. Maybe there's even a way to add it now without other
nodes producing errors, but from top of my head I am not sure..

In Proxmox Backup Server we have each metric in a separate file, makes it a bit
easier to extend on.

# ls -l /var/lib/proxmox-backup/rrdb/host/
total 160
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 cpu
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 io_ticks
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 iowait
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 loadavg
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 memtotal
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 memused
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 netin
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 netout
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 read_bytes
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 read_ios
-rw-r--r-- 1 backup backup 6008 Jun  4 18:03 read_ticks
-rw-r--r-- 1 backup backup 6008 May 28 13:00 roottotal
-rw-r--r-- 1 backup backup 6008 May 28 13:00 rootused
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 swaptotal
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 swapused
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 total
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 used
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 write_bytes
-rw-r--r-- 1 backup backup 6008 Oct 23 07:45 write_ios
-rw-r--r-- 1 backup backup 6008 Jun  4 18:03 write_ticks


We could say we keep the base stuff as is in a single file and add
the new one with a "<vmid>.<metric-name>" scheme.

Re: [pve-devel] [PATCH pve-manager] pvestatd: stream host pressure counters

[Alexandre Derumier]

>>that'd drop all old statistics, and be a bit of a nuisance to upgrade inside
>>a cluster (old nodes get stuff they do not comprehend).

oh ok, I thinked that rrd was able to add new value. (don't have used
rrd since a long time since I'm using influxdb or other metrics
databases)

I'll look to add them in separated rrd files

Le sam. 24 oct. 2020 à 17:43, Thomas Lamprecht
<t.lamprecht@proxmox.com> a écrit :
>
> On 24.10.20 15:46, Alexandre Derumier wrote:
> >> Sounds reasonable. We just need to find a way to add new values
> >> in a compatible way.
> >
> > Last time in 2013, we had added a new rrd "pve2.3-vm/"
> > https://lists.proxmox.com/pipermail/pve-devel/2013-January/005612.html
>
> that'd drop all old statistics, and be a bit of a nuisance to upgrade inside
> a cluster (old nodes get stuff they do not comprehend).
>
> We may rather want to have a new, separate, file for this.
> We could add general support for and error handling now, and to the final add
> with 7.0 next year or so. Maybe there's even a way to add it now without other
> nodes producing errors, but from top of my head I am not sure..
>
> In Proxmox Backup Server we have each metric in a separate file, makes it a bit
> easier to extend on.
>
> # ls -l /var/lib/proxmox-backup/rrdb/host/
> total 160
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 cpu
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 io_ticks
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 iowait
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 loadavg
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 memtotal
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 memused
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 netin
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 netout
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 read_bytes
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 read_ios
> -rw-r--r-- 1 backup backup 6008 Jun  4 18:03 read_ticks
> -rw-r--r-- 1 backup backup 6008 May 28 13:00 roottotal
> -rw-r--r-- 1 backup backup 6008 May 28 13:00 rootused
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 swaptotal
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 swapused
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 total
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 used
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 write_bytes
> -rw-r--r-- 1 backup backup 6008 Oct 23 07:45 write_ios
> -rw-r--r-- 1 backup backup 6008 Jun  4 18:03 write_ticks
>
>
> We could say we keep the base stuff as is in a single file and add
> the new one with a "<vmid>.<metric-name>" scheme.
>
>