[pve-devel] [PATCH manager] fix #3815: influxdb vmname should always be a string

[pve-devel] [PATCH manager] fix #3815: influxdb vmname should always be a string

[markus frank]

InfluxDB interprets the vmname 66601 as a number and the vmname vm42 as a String. This leads to problematic metrics, that will be dropped by influxdb.
To change that I added a $quoted hashmap (simular to $excluded) to quote a value. In this case the value of name.

Signed-off-by: markus frank <m.frank@proxmox.com>
---
 PVE/Status/InfluxDB.pm | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/PVE/Status/InfluxDB.pm b/PVE/Status/InfluxDB.pm
index def7e2fd..f49feac4 100644
--- a/PVE/Status/InfluxDB.pm
+++ b/PVE/Status/InfluxDB.pm
@@ -116,7 +116,7 @@ sub update_qemu_status {
     $object =~ s/\s/\\ /g;
 
     # VMID is already added in base $object above, so exclude it from being re-added
-    build_influxdb_payload($class, $txn, $data, $ctime, $object, { 'vmid' => 1 });
+    build_influxdb_payload($class, $txn, $data, $ctime, $object, { 'vmid' => 1 }, { 'name' => 1 });
 }
 
 sub update_lxc_status {
@@ -131,7 +131,7 @@ sub update_lxc_status {
     $object =~ s/\s/\\ /g;
 
     # VMID is already added in base $object above, so exclude it from being re-added
-    build_influxdb_payload($class, $txn, $data, $ctime, $object, { 'vmid' => 1 });
+    build_influxdb_payload($class, $txn, $data, $ctime, $object, { 'vmid' => 1 }, { 'name' => 1 });
 }
 
 sub update_storage_status {
@@ -274,7 +274,7 @@ sub test_connection {
 }
 
 sub build_influxdb_payload {
-    my ($class, $txn, $data, $ctime, $tags, $excluded, $measurement, $instance) = @_;
+    my ($class, $txn, $data, $ctime, $tags, $excluded, $quoted, $measurement, $instance) = @_;
 
     my @values = ();
 
@@ -283,6 +283,10 @@ sub build_influxdb_payload {
 	my $value = $data->{$key};
 	next if !defined($value);
 
+	if (defined($quoted) && $quoted->{$key}){
+	    $value =~ s/\"/\\\"/g;
+            $value = "\"$value\"";
+        }
 	if (!ref($value) && $value ne '') {
 	    # value is scalar
 
@@ -293,9 +297,9 @@ sub build_influxdb_payload {
 	    # value is a hash
 
 	    if (!defined($measurement)) {
-		build_influxdb_payload($class, $txn, $value, $ctime, $tags, $excluded, $key);
+		build_influxdb_payload($class, $txn, $value, $ctime, $tags, $excluded, $quoted, $key);
 	    } elsif(!defined($instance)) {
-		build_influxdb_payload($class, $txn, $value, $ctime, $tags, $excluded, $measurement, $key);
+		build_influxdb_payload($class, $txn, $value, $ctime, $tags, $excluded, $quoted, $measurement, $key);
 	    } else {
 		push @values, get_recursive_values($value);
 	    }
-- 
2.30.2

Re: [pve-devel] [PATCH manager] fix #3815: influxdb vmname should always be a string

[Thomas Lamprecht]

On 27.01.22 12:13, markus frank wrote:
> InfluxDB interprets the vmname 66601 as a number and the vmname vm42 as a String. This leads to problematic metrics, that will be dropped by influxdb.

which one will? I'd guess whatever comes first matters in what "schema" is
decided on?

> To change that I added a $quoted hashmap (simular to $excluded) to quote a value. In this case the value of name.

Breaks our commit line style guide:

> Make sure the line length of the commit's message is not longer than 70 characters,
> HTTPS links are an exception and should not be splitted.
https://pve.proxmox.com/wiki/Developer_Documentation#Commits_and_Commit_Messages

> 
> Signed-off-by: markus frank <m.frank@proxmox.com>

Nit, can you start your name with a captibal letter?

Besides those style remarks:

* what about nodenames? as in the `host` and `nodename` parameters can also have the
  same issue in a cluster I'd figure, as those can be fully numeric too

* I'd avoid chaining through this as param if we can avoid it for now, handling
  that more centrally in build_influxdb_payload seems feasible to me at the first
  look

> ---
>  PVE/Status/InfluxDB.pm | 14 +++++++++-----
>  1 file changed, 9 insertions(+), 5 deletions(-)
> 
> diff --git a/PVE/Status/InfluxDB.pm b/PVE/Status/InfluxDB.pm
> index def7e2fd..f49feac4 100644
> --- a/PVE/Status/InfluxDB.pm
> +++ b/PVE/Status/InfluxDB.pm
> @@ -116,7 +116,7 @@ sub update_qemu_status {
>      $object =~ s/\s/\\ /g;
>  
>      # VMID is already added in base $object above, so exclude it from being re-added
> -    build_influxdb_payload($class, $txn, $data, $ctime, $object, { 'vmid' => 1 });
> +    build_influxdb_payload($class, $txn, $data, $ctime, $object, { 'vmid' => 1 }, { 'name' => 1 });
>  }
>  
>  sub update_lxc_status {
> @@ -131,7 +131,7 @@ sub update_lxc_status {
>      $object =~ s/\s/\\ /g;
>  
>      # VMID is already added in base $object above, so exclude it from being re-added
> -    build_influxdb_payload($class, $txn, $data, $ctime, $object, { 'vmid' => 1 });
> +    build_influxdb_payload($class, $txn, $data, $ctime, $object, { 'vmid' => 1 }, { 'name' => 1 });
>  }
>  
>  sub update_storage_status {
> @@ -274,7 +274,7 @@ sub test_connection {
>  }
>  
>  sub build_influxdb_payload {
> -    my ($class, $txn, $data, $ctime, $tags, $excluded, $measurement, $instance) = @_;
> +    my ($class, $txn, $data, $ctime, $tags, $excluded, $quoted, $measurement, $instance) = @_;
>  
>      my @values = ();
>  
> @@ -283,6 +283,10 @@ sub build_influxdb_payload {
>  	my $value = $data->{$key};
>  	next if !defined($value);
>  
> +	if (defined($quoted) && $quoted->{$key}){
> +	    $value =~ s/\"/\\\"/g;
> +            $value = "\"$value\"";
> +        }

indentation for above looks off:
https://pve.proxmox.com/wiki/Perl_Style_Guide

>  	if (!ref($value) && $value ne '') {
>  	    # value is scalar
>  
> @@ -293,9 +297,9 @@ sub build_influxdb_payload {
>  	    # value is a hash
>  
>  	    if (!defined($measurement)) {
> -		build_influxdb_payload($class, $txn, $value, $ctime, $tags, $excluded, $key);
> +		build_influxdb_payload($class, $txn, $value, $ctime, $tags, $excluded, $quoted, $key);
>  	    } elsif(!defined($instance)) {
> -		build_influxdb_payload($class, $txn, $value, $ctime, $tags, $excluded, $measurement, $key);
> +		build_influxdb_payload($class, $txn, $value, $ctime, $tags, $excluded, $quoted, $measurement, $key);
>  	    } else {
>  		push @values, get_recursive_values($value);
>  	    }