[pve-devel] [RFC many 0/3] combine and simplify RRD handling

[pve-devel] [RFC many 0/3] combine and simplify RRD handling

[Aaron Lauterer]

this series does two things:
* switch the RRD API backend to use the old RRD files is available
* always use RRD files with the new 9.0 schema

The motivation for the combination of old and new RRD files is that during the
migration of old RRD files to the new ones, spikes get flattened. By combining
the old and new RRD files, we can keep the old coarse data with all it spikes
and only show the new much mode finer stepped data where available.

This will also enable us to just write any new data into an RRD file that is
created with the new 9.0 schema. As a result, no migration step is needed
anymore and we can simplify the logic around the whole "does the only the old
file exist or is the new one also present" alot in quite a few places.

Some examples which we can simplify or revert:
pve-manager: API2Tools::get_rrd_key can most likely be dropped
qemu-server, pve-storage, pve-container,…: rrd API endpoints which check which files exist
drop the whole migration step on upgrade

other TODOs:

RRD::create_rrd_graph -> see if we can combine both files into one graph. But we
are relying on RRD itself for this. So that might be interesting. On the other
hand, do we want to keep that API endpoint around? People are using it though.
So we need to weigh that decision accoringly.


This is currently marked as RFC. Tested by installing it on an existing 9.0.6
cluster. Added a 8.4 node with latest updates. Renamed {rrd}.old to {rrd}.

cluster:

Aaron Lauterer (3):
  rrd: fix rrd time frames
  RRD: fetch data from old rrd file if present and needed
  pmxcfs: status.c: always use 9.0 rrd files

 src/PVE/RRD.pm      | 113 ++++++++++++++-------
 src/pmxcfs/status.c | 238 ++++++--------------------------------------
 2 files changed, 108 insertions(+), 243 deletions(-)


Summary over all repositories:
  2 files changed, 108 insertions(+), 243 deletions(-)

-- 
Generated by git-murpp 0.8.1


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH cluster 1/3] rrd: fix rrd time frames

[Aaron Lauterer]

month with the new rrd files provides a resoltion of 30 minutes

year had a typo 1140 which should be 1440

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
---
 src/PVE/RRD.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/PVE/RRD.pm b/src/PVE/RRD.pm
index 8fe1927..34dafef 100644
--- a/src/PVE/RRD.pm
+++ b/src/PVE/RRD.pm
@@ -28,8 +28,8 @@ sub create_rrd_data {
         hour => [60, 60], # 1 min resolution
         day => [60, 1440], # 1 min resolution, full day
         week => [60 * 30, 336], # 30 min resolution, 7 days
-        month => [3600 * 6, 121], # 6 hour resolution, 30 days, need one more count. Otherwise RRD gets wrong $step
-        year => [3600 * 6, 1140], # 6 hour resolution, 360 days
+        month => [60 * 30, 1440], # 30 min resolution  30 days
+        year => [3600 * 6, 1440], # 6 hour resolution, 360 days
         decade => [86400 * 7, 570], # 1 week resolution, 10 years
     };
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH cluster 2/3] RRD: fetch data from old rrd file if present and needed

[Aaron Lauterer]

One side effect of the RRD migration is that MAX spikes are flattened
due to the new more fine grained resolution.

If we request RRD data that was still covered in the old rrd file, we
can fetch those data from there and then append newer data from the new
RRD files. This way, we still have the older more coarse resolution for
the old data and keep spikes present.

Old RRD files can be either in {rrd}.old or just {rrd} without the .old
suffix. This depends on if the rrd migration tool has been run on the
host or not.

One side effect of this more dynamic approach is that the step size
determined by RRDs::fetch will not match up in many situations.
Therefore we drop the check for the step size!

For example, if we choose Year as the time frame, but the new file only
has a much shorter amound of data present, RRDs::fetch will use a
smaller step size, if it can still cover the requested data with it.

Visually this will result in very small step sizes, that should get
wider once we get more or all data from the new RRD files, where it
needs to then switch to the longer step sizes to cover the requested
time frame.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
---
 src/PVE/RRD.pm | 109 ++++++++++++++++++++++++++++++++++---------------
 1 file changed, 76 insertions(+), 33 deletions(-)

diff --git a/src/PVE/RRD.pm b/src/PVE/RRD.pm
index 34dafef..a6ce34b 100644
--- a/src/PVE/RRD.pm
+++ b/src/PVE/RRD.pm
@@ -7,6 +7,32 @@ use RRDs;
 
 use PVE::Tools;
 
+my $get_rrd_data = sub {
+    my ($rrd, $cf, $is_node, $reso, $args, $res) = @_;
+    my ($start, $step, $names, $data) = RRDs::fetch($rrd, $cf, @$args);
+
+    my $err = RRDs::error;
+    die "RRD error: $err\n" if $err;
+
+    my $fields = scalar(@$names);
+    for my $line (@$data) {
+        my $entry = { 'time' => $start };
+        $start += $step;
+        for (my $i = 0; $i < $fields; $i++) {
+            my $name = $names->[$i];
+            if (defined(my $val = $line->[$i])) {
+                $entry->{$name} = $val;
+                $entry->{memavailable} = $val
+                    if $is_node && $name eq 'memfree' && !exists($entry->{memavailable});
+            } else {
+                # leave empty fields undefined
+                # maybe make this configurable?
+            }
+        }
+        push @$res, $entry;
+    }
+};
+
 sub create_rrd_data {
     my ($rrdname, $timeframe, $cf) = @_;
 
@@ -33,54 +59,71 @@ sub create_rrd_data {
         decade => [86400 * 7, 570], # 1 week resolution, 10 years
     };
 
+    my $is_node = !!($rrdname =~ /^pve-node/);
+    $cf = "AVERAGE" if !$cf;
+    my $res = [];
+
     if ($rrdname =~ /^pve2/) {
         $setup = $setup_pve2;
         $timeframe = "year" if $timeframe eq "decade"; # we only store up to one year in the old format
     }
-    my $is_node = !!($rrdname =~ /^pve-node/);
 
     my ($reso, $count) = @{ $setup->{$timeframe} };
     my $ctime = $reso * int(time() / $reso);
     my $req_start = $ctime - $reso * $count;
 
-    $cf = "AVERAGE" if !$cf;
-
-    my @args = (
-        "-s" => $req_start,
-        "-e" => $ctime - 1,
-        "-r" => $reso,
-    );
-
-    my $socket = "/var/run/rrdcached.sock";
-    push @args, "--daemon" => "unix:$socket" if -S $socket;
-
-    my ($start, $step, $names, $data) = RRDs::fetch($rrd, $cf, @args);
-
-    my $err = RRDs::error;
-    die "RRD error: $err\n" if $err;
-
-    die "got wrong time resolution ($step != $reso)\n"
-        if $step != $reso;
+    my $last_old;
+    # check if we have old rrd file and if the start point is still covered by
+    # it, fetch that data from it for any data not available in the old file we
+    # will fetch it from the new file.
+    if ($rrdname =~ /pve-(?<type>node|vm|storage)-[0-9]*\.[0-9]*\/(?<resource>.*)/) {
+        my $old_rrd = "${rrddir}/pve2-$+{type}/$+{resource}";
+        my $old_exists = 0;
+
+        # we can have already migrated rrd files that have the .old suffix too
+        if (-e $old_rrd) {
+            $old_exists = 1;
+        } elsif (-e "${old_rrd}.old") {
+            $old_exists = 1;
+            $old_rrd = "${old_rrd}.old";
+        }
 
-    my $res = [];
-    my $fields = scalar(@$names);
-    for my $line (@$data) {
-        my $entry = { 'time' => $start };
-        $start += $step;
-        for (my $i = 0; $i < $fields; $i++) {
-            my $name = $names->[$i];
-            if (defined(my $val = $line->[$i])) {
-                $entry->{$name} = $val;
-                $entry->{memavailable} = $val
-                    if $is_node && $name eq 'memfree' && !exists($entry->{memavailable});
+        if ($old_exists) {
+            $last_old = RRDs::last($old_rrd);
+            if ($req_start < $last_old) {
+                my ($reso_old, $count_old) = @{ $setup_pve2->{$timeframe} };
+                my $ctime_old = $reso_old * int(time() / $reso_old);
+                my $req_start_old = $ctime_old - $reso_old * $count_old;
+                my $args = [];
+                push(@$args, "-s" => $req_start_old);
+                push(@$args, "-e" => $last_old);
+                push(@$args, "-r" => $reso_old);
+
+                my $socket = "/var/run/rrdcached.sock";
+                push @$args, "--daemon" => "unix:$socket" if -S $socket;
+
+                $get_rrd_data->($old_rrd, $cf, $is_node, $reso_old, $args, $res);
             } else {
-                # leave empty fields undefined
-                # maybe make this configurable?
+                $last_old = undef;
             }
         }
-        push @$res, $entry;
     }
 
+    my $args = [];
+    if ($last_old) {
+        push(@$args, "-s" => $last_old);
+    } else {
+        push(@$args, "-s" => $req_start);
+    }
+
+    push(@$args, "-e" => $ctime - 1);
+    push(@$args, "-r" => $reso);
+
+    my $socket = "/var/run/rrdcached.sock";
+    push @$args, "--daemon" => "unix:$socket" if -S $socket;
+
+    $get_rrd_data->($rrd, $cf, $is_node, $reso, $args, $res);
+
     return $res;
 }
 
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH cluster 3/3] pmxcfs: status.c: always use 9.0 rrd files

[Aaron Lauterer]

Assuming that we can keep the old RRD files around and fetch data
directly there if needed, we can skip the migration step and just write
to new 9.0 RRD files.

This simplifies the whole approach introduced in
9b00ac9 status: introduce new pve-{type}- rrd and metric format

We will always create a new 9.0 file if not present and then need to
handle the following cases:

* old pve2-{type} format -> pad missing columns
* pve-{type}-9.0 matches -> no changes needed
* pve-{type}- received, but version doesn't match -> new format, cut at
  expected columns

Since we don't need the old RRD definitions anymore, they are removed.
Otherwise the compile will throw an unused variable warning.
The `rrd_def_{type}_columns` variables get a `_pve2` suffix to make it
clear what they are for.

The `checked_mkdir` pattern for VMs is aligned with how we do it for
nodes and storages, to call it once more, to be safe.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
---

Notes:
    if I misunderstood the reason for the doulbe checked_mkdir, then it
    might only be needed for the storage due to the additional subdirs?

 src/pmxcfs/status.c | 238 ++++++--------------------------------------
 1 file changed, 30 insertions(+), 208 deletions(-)

diff --git a/src/pmxcfs/status.c b/src/pmxcfs/status.c
index 130f08f..cb03e4e 100644
--- a/src/pmxcfs/status.c
+++ b/src/pmxcfs/status.c
@@ -1097,36 +1097,9 @@ kventry_hash_set(GHashTable *kvhash, const char *key, gconstpointer data, size_t
 // We create the RRD files with a 60 second stepsize, therefore, RRA timesteps
 // are alwys per 60 seconds. These 60 seconds are usually showing up in other
 // code paths where we interact with RRD data!
-static const char *rrd_def_node[] = {
-    "DS:loadavg:GAUGE:120:0:U",
-    "DS:maxcpu:GAUGE:120:0:U",
-    "DS:cpu:GAUGE:120:0:U",
-    "DS:iowait:GAUGE:120:0:U",
-    "DS:memtotal:GAUGE:120:0:U",
-    "DS:memused:GAUGE:120:0:U",
-    "DS:swaptotal:GAUGE:120:0:U",
-    "DS:swapused:GAUGE:120:0:U",
-    "DS:roottotal:GAUGE:120:0:U",
-    "DS:rootused:GAUGE:120:0:U",
-    "DS:netin:DERIVE:120:0:U",
-    "DS:netout:DERIVE:120:0:U",
-
-    "RRA:AVERAGE:0.5:1:70",     // 1 min avg - one hour
-    "RRA:AVERAGE:0.5:30:70",    // 30 min avg - one day
-    "RRA:AVERAGE:0.5:180:70",   // 3 hour avg - one week
-    "RRA:AVERAGE:0.5:720:70",   // 12 hour avg - one month
-    "RRA:AVERAGE:0.5:10080:70", // 7 day avg - ony year
-
-    "RRA:MAX:0.5:1:70",     // 1 min max - one hour
-    "RRA:MAX:0.5:30:70",    // 30 min max - one day
-    "RRA:MAX:0.5:180:70",   // 3 hour max - one week
-    "RRA:MAX:0.5:720:70",   // 12 hour max - one month
-    "RRA:MAX:0.5:10080:70", // 7 day max - ony year
-    NULL,
-};
 
 // This *must* be the number of columns as defined above.
-static const int rrd_def_node_columns = 12;
+static const int rrd_def_node_columns_pve2 = 12;
 
 static const char *rrd_def_node_pve9_0[] = {
     "DS:loadavg:GAUGE:120:0:U",
@@ -1165,34 +1138,8 @@ static const char *rrd_def_node_pve9_0[] = {
 // This *must* be the number of columns as defined above.
 static const int rrd_def_node_pve9_0_columns = 19;
 
-static const char *rrd_def_vm[] = {
-    "DS:maxcpu:GAUGE:120:0:U",
-    "DS:cpu:GAUGE:120:0:U",
-    "DS:maxmem:GAUGE:120:0:U",
-    "DS:mem:GAUGE:120:0:U",
-    "DS:maxdisk:GAUGE:120:0:U",
-    "DS:disk:GAUGE:120:0:U",
-    "DS:netin:DERIVE:120:0:U",
-    "DS:netout:DERIVE:120:0:U",
-    "DS:diskread:DERIVE:120:0:U",
-    "DS:diskwrite:DERIVE:120:0:U",
-
-    "RRA:AVERAGE:0.5:1:70",     // 1 min avg - one hour
-    "RRA:AVERAGE:0.5:30:70",    // 30 min avg - one day
-    "RRA:AVERAGE:0.5:180:70",   // 3 hour avg - one week
-    "RRA:AVERAGE:0.5:720:70",   // 12 hour avg - one month
-    "RRA:AVERAGE:0.5:10080:70", // 7 day avg - ony year
-
-    "RRA:MAX:0.5:1:70",     // 1 min max - one hour
-    "RRA:MAX:0.5:30:70",    // 30 min max - one day
-    "RRA:MAX:0.5:180:70",   // 3 hour max - one week
-    "RRA:MAX:0.5:720:70",   // 12 hour max - one month
-    "RRA:MAX:0.5:10080:70", // 7 day max - ony year
-    NULL,
-};
-
 // This *must* be the number of columns as defined above.
-static const int rrd_def_vm_columns = 10;
+static const int rrd_def_vm_columns_pve2 = 10;
 
 static const char *rrd_def_vm_pve9_0[] = {
     "DS:maxcpu:GAUGE:120:0:U",
@@ -1229,24 +1176,6 @@ static const char *rrd_def_vm_pve9_0[] = {
 // This *must* be the number of columns as defined above.
 static const int rrd_def_vm_pve9_0_columns = 17;
 
-static const char *rrd_def_storage[] = {
-    "DS:total:GAUGE:120:0:U",
-    "DS:used:GAUGE:120:0:U",
-
-    "RRA:AVERAGE:0.5:1:70",     // 1 min avg - one hour
-    "RRA:AVERAGE:0.5:30:70",    // 30 min avg - one day
-    "RRA:AVERAGE:0.5:180:70",   // 3 hour avg - one week
-    "RRA:AVERAGE:0.5:720:70",   // 12 hour avg - one month
-    "RRA:AVERAGE:0.5:10080:70", // 7 day avg - ony year
-
-    "RRA:MAX:0.5:1:70",     // 1 min max - one hour
-    "RRA:MAX:0.5:30:70",    // 30 min max - one day
-    "RRA:MAX:0.5:180:70",   // 3 hour max - one week
-    "RRA:MAX:0.5:720:70",   // 12 hour max - one month
-    "RRA:MAX:0.5:10080:70", // 7 day max - ony year
-    NULL,
-};
-
 static const char *rrd_def_storage_pve9_0[] = {
     "DS:total:GAUGE:120:0:U",
     "DS:used:GAUGE:120:0:U",
@@ -1356,68 +1285,27 @@ static void update_rrd_data(const char *key, gconstpointer data, size_t len) {
         }
 
         filename = g_strdup_printf(RRDDIR "/pve-node-9.0/%s", node);
-        char *filename_pve2 = g_strdup_printf(RRDDIR "/pve2-node/%s", node);
-
-        int use_pve2_file = 0;
 
-        // check existing rrd files and directories
-        if (g_file_test(filename, G_FILE_TEST_EXISTS)) {
-            // pve-node-9.0 file exists, we use that
-            // TODO: get conditions so, that we do not have this empty branch
-        } else if (g_file_test(filename_pve2, G_FILE_TEST_EXISTS)) {
-            // old file exists, use it
-            use_pve2_file = 1;
-
-            g_free(filename);
-            filename = filename_pve2;
-            filename_pve2 = NULL;
-        } else {
-            // neither file exists, check for directories to decide and create file
+        if (!g_file_test(filename, G_FILE_TEST_EXISTS)) {
+            checked_mkdir(RRDDIR "/pve-node-9.0", 0755);
 
-            if (g_file_test(RRDDIR "/pve-node-9.0", G_FILE_TEST_IS_DIR)) {
+            char *dir = g_path_get_dirname(filename);
+            checked_mkdir(dir, 0755);
+            g_free(dir);
 
-                int argcount = sizeof(rrd_def_node_pve9_0) / sizeof(void *) - 1;
-                create_rrd_file(filename, argcount, rrd_def_node_pve9_0);
-            } else if (g_file_test(RRDDIR "/pve2-node", G_FILE_TEST_IS_DIR)) {
-                use_pve2_file = 1;
-
-                g_free(filename);
-                filename = filename_pve2;
-                filename_pve2 = NULL;
-
-                char *dir = g_path_get_dirname(filename);
-                checked_mkdir(dir, 0755);
-                g_free(dir);
-
-                int argcount = sizeof(rrd_def_node) / sizeof(void *) - 1;
-                create_rrd_file(filename, argcount, rrd_def_node);
-            } else {
-                // no dir exists yet, use new pve-node-9.0
-                checked_mkdir(RRDDIR "/pve-node-9.0", 0755);
-
-                char *dir = g_path_get_dirname(filename);
-                checked_mkdir(dir, 0755);
-                g_free(dir);
-
-                int argcount = sizeof(rrd_def_node_pve9_0) / sizeof(void *) - 1;
-                create_rrd_file(filename, argcount, rrd_def_node_pve9_0);
-            }
+            int argcount = sizeof(rrd_def_node_pve9_0) / sizeof(void *) - 1;
+            create_rrd_file(filename, argcount, rrd_def_node_pve9_0);
         }
 
         skip = 2; // first two columns are live data that isn't archived
 
-        if (strncmp(key, "pve2-node/", 10) == 0 && !use_pve2_file) {
-            padding = rrd_def_node_pve9_0_columns - rrd_def_node_columns;
-        } else if (strncmp(key, "pve-node-", 9) == 0 && use_pve2_file) {
-            keep_columns = rrd_def_node_columns;
+        if (strncmp(key, "pve2-node/", 10) == 0) {
+            padding = rrd_def_node_pve9_0_columns - rrd_def_node_columns_pve2;
         } else if (strncmp(key, "pve-node-9.0/", 13) != 0) {
             // we received an unknown format, expectation is it is newer and has more columns
             // than we can currently handle
             keep_columns = rrd_def_node_pve9_0_columns;
         }
-
-        g_free(filename_pve2);
-
     } else if (strncmp(key, "pve2.3-vm/", 10) == 0 || strncmp(key, "pve-vm-", 7) == 0) {
 
         const char *vmid = rrd_skip_data(key, 1, '/');
@@ -1431,59 +1319,28 @@ static void update_rrd_data(const char *key, gconstpointer data, size_t len) {
         }
 
         filename = g_strdup_printf(RRDDIR "/pve-vm-9.0/%s", vmid);
-        char *filename_pve2 = g_strdup_printf(RRDDIR "/pve2-vm/%s", vmid);
-
-        int use_pve2_file = 0;
-
-        // check existing rrd files and directories
-        if (g_file_test(filename, G_FILE_TEST_EXISTS)) {
-            // pve-vm-9.0 file exists, we use that
-            // TODO: get conditions so, that we do not have this empty branch
-        } else if (g_file_test(filename_pve2, G_FILE_TEST_EXISTS)) {
-            // old file exists, use it
-            use_pve2_file = 1;
-            g_free(filename);
-            filename = filename_pve2;
-            filename_pve2 = NULL;
-        } else {
-            // neither file exists, check for directories to decide and create file
-
-            if (g_file_test(RRDDIR "/pve-vm-9.0", G_FILE_TEST_IS_DIR)) {
 
-                int argcount = sizeof(rrd_def_vm_pve9_0) / sizeof(void *) - 1;
-                create_rrd_file(filename, argcount, rrd_def_vm_pve9_0);
-            } else if (g_file_test(RRDDIR "/pve2-vm", G_FILE_TEST_IS_DIR)) {
-                use_pve2_file = 1;
+        if (!g_file_test(filename, G_FILE_TEST_EXISTS)) {
+            // no dir exists yet, use new pve-vm-9.0
+            checked_mkdir(RRDDIR "/pve-vm-9.0", 0755);
 
-                g_free(filename);
-                filename = filename_pve2;
-                filename_pve2 = NULL;
+            char *dir = g_path_get_dirname(filename);
+            checked_mkdir(dir, 0755);
+            g_free(dir);
 
-                int argcount = sizeof(rrd_def_vm) / sizeof(void *) - 1;
-                create_rrd_file(filename, argcount, rrd_def_vm);
-            } else {
-                // no dir exists yet, use new pve-vm-9.0
-                checked_mkdir(RRDDIR "/pve-vm-9.0", 0755);
-
-                int argcount = sizeof(rrd_def_vm_pve9_0) / sizeof(void *) - 1;
-                create_rrd_file(filename, argcount, rrd_def_vm_pve9_0);
-            }
+            int argcount = sizeof(rrd_def_vm_pve9_0) / sizeof(void *) - 1;
+            create_rrd_file(filename, argcount, rrd_def_vm_pve9_0);
         }
 
         skip = 4; // first 4 columns are live data that isn't archived
 
-        if (strncmp(key, "pve2.3-vm/", 10) == 0 && !use_pve2_file) {
-            padding = rrd_def_vm_pve9_0_columns - rrd_def_vm_columns;
-        } else if (strncmp(key, "pve-vm-", 7) == 0 && use_pve2_file) {
-            keep_columns = rrd_def_vm_columns;
+        if (strncmp(key, "pve2.3-vm/", 10) == 0) {
+            padding = rrd_def_vm_pve9_0_columns - rrd_def_vm_columns_pve2;
         } else if (strncmp(key, "pve-vm-9.0/", 11) != 0) {
             // we received an unknown format, expectation is it is newer and has more columns
             // than we can currently handle
             keep_columns = rrd_def_vm_pve9_0_columns;
         }
-
-        g_free(filename_pve2);
-
     } else if (strncmp(key, "pve2-storage/", 13) == 0 || strncmp(key, "pve-storage-", 12) == 0) {
         const char *node = rrd_skip_data(key, 1, '/'); // will contain {node}/{storage}
 
@@ -1502,49 +1359,17 @@ static void update_rrd_data(const char *key, gconstpointer data, size_t len) {
         }
 
         filename = g_strdup_printf(RRDDIR "/pve-storage-9.0/%s", node);
-        char *filename_pve2 = g_strdup_printf(RRDDIR "/pve2-storage/%s", node);
-
-        // check existing rrd files and directories
-        if (g_file_test(filename, G_FILE_TEST_EXISTS)) {
-            // pve-storage-9.0 file exists, we use that
-            // TODO: get conditions so, that we do not have this empty branch
-        } else if (g_file_test(filename_pve2, G_FILE_TEST_EXISTS)) {
-            // old file exists, use it
-            g_free(filename);
-            filename = filename_pve2;
-            filename_pve2 = NULL;
-        } else {
-            // neither file exists, check for directories to decide and create file
-
-            if (g_file_test(RRDDIR "/pve-storage-9.0", G_FILE_TEST_IS_DIR)) {
-                char *dir = g_path_get_dirname(filename);
-                checked_mkdir(dir, 0755);
-                g_free(dir);
-
-                int argcount = sizeof(rrd_def_storage_pve9_0) / sizeof(void *) - 1;
-                create_rrd_file(filename, argcount, rrd_def_storage_pve9_0);
-            } else if (g_file_test(RRDDIR "/pve2-storage", G_FILE_TEST_IS_DIR)) {
-                g_free(filename);
-                filename = filename_pve2;
-                filename_pve2 = NULL;
-
-                char *dir = g_path_get_dirname(filename);
-                checked_mkdir(dir, 0755);
-                g_free(dir);
-
-                int argcount = sizeof(rrd_def_storage) / sizeof(void *) - 1;
-                create_rrd_file(filename, argcount, rrd_def_storage);
-            } else {
-                // no dir exists yet, use new pve-storage-9.0
-                checked_mkdir(RRDDIR "/pve-storage-9.0", 0755);
 
-                char *dir = g_path_get_dirname(filename);
-                checked_mkdir(dir, 0755);
-                g_free(dir);
+        if (!g_file_test(filename, G_FILE_TEST_EXISTS)) {
+            // no dir exists yet, use new pve-storage-9.0
+            checked_mkdir(RRDDIR "/pve-storage-9.0", 0755);
 
-                int argcount = sizeof(rrd_def_storage_pve9_0) / sizeof(void *) - 1;
-                create_rrd_file(filename, argcount, rrd_def_storage_pve9_0);
-            }
+            char *dir = g_path_get_dirname(filename);
+            checked_mkdir(dir, 0755);
+            g_free(dir);
+
+            int argcount = sizeof(rrd_def_storage_pve9_0) / sizeof(void *) - 1;
+            create_rrd_file(filename, argcount, rrd_def_storage_pve9_0);
         }
 
         // actual data columns didn't change between pve2-storage and pve-storage-9.0
@@ -1553,9 +1378,6 @@ static void update_rrd_data(const char *key, gconstpointer data, size_t len) {
             // than we can currently handle
             keep_columns = rrd_def_storage_pve9_0_columns;
         }
-
-        g_free(filename_pve2);
-
     } else {
         goto keyerror;
     }
-- 
2.47.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [RFC many 0/3] combine and simplify RRD handling

[Thomas Lamprecht]

Am 04.09.25 um 16:09 schrieb Aaron Lauterer:
> this series does two things:
> * switch the RRD API backend to use the old RRD files is available
> * always use RRD files with the new 9.0 schema
> 
> The motivation for the combination of old and new RRD files is that during the
> migration of old RRD files to the new ones, spikes get flattened. By combining
> the old and new RRD files, we can keep the old coarse data with all it spikes
> and only show the new much mode finer stepped data where available.
> 
> This will also enable us to just write any new data into an RRD file that is
> created with the new 9.0 schema. As a result, no migration step is needed
> anymore and we can simplify the logic around the whole "does the only the old
> file exist or is the new one also present" alot in quite a few places.
> 
> Some examples which we can simplify or revert:
> pve-manager: API2Tools::get_rrd_key can most likely be dropped
> qemu-server, pve-storage, pve-container,…: rrd API endpoints which check which files exist
> drop the whole migration step on upgrade
> 
> other TODOs:
> 
> RRD::create_rrd_graph -> see if we can combine both files into one graph. But we
> are relying on RRD itself for this. So that might be interesting. On the other
> hand, do we want to keep that API endpoint around? People are using it though.
> So we need to weigh that decision accoringly.

Keeping it around for the time being is definitively required now that
PVE 9.0 is out, we might sunset it for a future major release.

Another option might be to generate both images and merge them manually,
i.e. literally make the right part the old and the left part the new
image with some imagemagick command. Or does rrd indeed supports merging
two files together?

> 
> 
> This is currently marked as RFC. Tested by installing it on an existing 9.0.6
> cluster. Added a 8.4 node with latest updates. Renamed {rrd}.old to {rrd}.
> 
> cluster:
> 
> Aaron Lauterer (3):
>   rrd: fix rrd time frames
>   RRD: fetch data from old rrd file if present and needed
>   pmxcfs: status.c: always use 9.0 rrd files
> 
>  src/PVE/RRD.pm      | 113 ++++++++++++++-------
>  src/pmxcfs/status.c | 238 ++++++--------------------------------------
>  2 files changed, 108 insertions(+), 243 deletions(-)
> 
> 
> Summary over all repositories:
>   2 files changed, 108 insertions(+), 243 deletions(-)
> 



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [RFC many 0/3] combine and simplify RRD handling

[Aaron Lauterer]

On  2025-09-04  20:20, Thomas Lamprecht wrote:
> Am 04.09.25 um 16:09 schrieb Aaron Lauterer:
>> this series does two things:
>> * switch the RRD API backend to use the old RRD files is available
>> * always use RRD files with the new 9.0 schema
>>
>> The motivation for the combination of old and new RRD files is that during the
>> migration of old RRD files to the new ones, spikes get flattened. By combining
>> the old and new RRD files, we can keep the old coarse data with all it spikes
>> and only show the new much mode finer stepped data where available.
>>
>> This will also enable us to just write any new data into an RRD file that is
>> created with the new 9.0 schema. As a result, no migration step is needed
>> anymore and we can simplify the logic around the whole "does the only the old
>> file exist or is the new one also present" alot in quite a few places.
>>
>> Some examples which we can simplify or revert:
>> pve-manager: API2Tools::get_rrd_key can most likely be dropped
>> qemu-server, pve-storage, pve-container,…: rrd API endpoints which check which files exist
>> drop the whole migration step on upgrade
>>
>> other TODOs:
>>
>> RRD::create_rrd_graph -> see if we can combine both files into one graph. But we
>> are relying on RRD itself for this. So that might be interesting. On the other
>> hand, do we want to keep that API endpoint around? People are using it though.
>> So we need to weigh that decision accoringly.
> 
> Keeping it around for the time being is definitively required now that
> PVE 9.0 is out, we might sunset it for a future major release.
> 
> Another option might be to generate both images and merge them manually,
> i.e. literally make the right part the old and the left part the new
> image with some imagemagick command. Or does rrd indeed supports merging
> two files together?

One thing that should be possible is to give it two datasets from 
different files. That would be a middle ground as in, all data is there, 
but it would show up as separata data lines.

I am looking into it to see if we could get it nicer. Manually stitching 
images together will most likely not work because I don't expected the 
scales to match up perfectly.

> 
>>
>>
>> This is currently marked as RFC. Tested by installing it on an existing 9.0.6
>> cluster. Added a 8.4 node with latest updates. Renamed {rrd}.old to {rrd}.
>>
>> cluster:
>>
>> Aaron Lauterer (3):
>>    rrd: fix rrd time frames
>>    RRD: fetch data from old rrd file if present and needed
>>    pmxcfs: status.c: always use 9.0 rrd files
>>
>>   src/PVE/RRD.pm      | 113 ++++++++++++++-------
>>   src/pmxcfs/status.c | 238 ++++++--------------------------------------
>>   2 files changed, 108 insertions(+), 243 deletions(-)
>>
>>
>> Summary over all repositories:
>>    2 files changed, 108 insertions(+), 243 deletions(-)
>>
> 



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [RFC many 0/3] combine and simplify RRD handling

[Thomas Lamprecht]

Am 05.09.25 um 10:03 schrieb Aaron Lauterer:
> 
> 
> On  2025-09-04  20:20, Thomas Lamprecht wrote:
>> Am 04.09.25 um 16:09 schrieb Aaron Lauterer:
>>> this series does two things:
>>> * switch the RRD API backend to use the old RRD files is available
>>> * always use RRD files with the new 9.0 schema
>>>
>>> The motivation for the combination of old and new RRD files is that during the
>>> migration of old RRD files to the new ones, spikes get flattened. By combining
>>> the old and new RRD files, we can keep the old coarse data with all it spikes
>>> and only show the new much mode finer stepped data where available.
>>>
>>> This will also enable us to just write any new data into an RRD file that is
>>> created with the new 9.0 schema. As a result, no migration step is needed
>>> anymore and we can simplify the logic around the whole "does the only the old
>>> file exist or is the new one also present" alot in quite a few places.
>>>
>>> Some examples which we can simplify or revert:
>>> pve-manager: API2Tools::get_rrd_key can most likely be dropped
>>> qemu-server, pve-storage, pve-container,…: rrd API endpoints which check which files exist
>>> drop the whole migration step on upgrade
>>>
>>> other TODOs:
>>>
>>> RRD::create_rrd_graph -> see if we can combine both files into one graph. But we
>>> are relying on RRD itself for this. So that might be interesting. On the other
>>> hand, do we want to keep that API endpoint around? People are using it though.
>>> So we need to weigh that decision accoringly.
>>
>> Keeping it around for the time being is definitively required now that
>> PVE 9.0 is out, we might sunset it for a future major release.
>>
>> Another option might be to generate both images and merge them manually,
>> i.e. literally make the right part the old and the left part the new
>> image with some imagemagick command. Or does rrd indeed supports merging
>> two files together?
> 
> One thing that should be possible is to give it two datasets from different files. That would be a middle ground as in, all data is there, but it would show up as separata data lines.
> 
> I am looking into it to see if we could get it nicer. Manually stitching images together will most likely not work because I don't expected the scales to match up perfectly.

True w.r.t. not matching scales. But as one can override the line colors we could probably just do that and use the same ones for both data series?


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [RFC many 0/3] combine and simplify RRD handling

[Aaron Lauterer]

On  2025-09-05  10:12, Thomas Lamprecht wrote:
> Am 05.09.25 um 10:03 schrieb Aaron Lauterer:
>>
>>
>> On  2025-09-04  20:20, Thomas Lamprecht wrote:
>>> Am 04.09.25 um 16:09 schrieb Aaron Lauterer:
>>>> this series does two things:
>>>> * switch the RRD API backend to use the old RRD files is available
>>>> * always use RRD files with the new 9.0 schema
>>>>
>>>> The motivation for the combination of old and new RRD files is that during the
>>>> migration of old RRD files to the new ones, spikes get flattened. By combining
>>>> the old and new RRD files, we can keep the old coarse data with all it spikes
>>>> and only show the new much mode finer stepped data where available.
>>>>
>>>> This will also enable us to just write any new data into an RRD file that is
>>>> created with the new 9.0 schema. As a result, no migration step is needed
>>>> anymore and we can simplify the logic around the whole "does the only the old
>>>> file exist or is the new one also present" alot in quite a few places.
>>>>
>>>> Some examples which we can simplify or revert:
>>>> pve-manager: API2Tools::get_rrd_key can most likely be dropped
>>>> qemu-server, pve-storage, pve-container,…: rrd API endpoints which check which files exist
>>>> drop the whole migration step on upgrade
>>>>
>>>> other TODOs:
>>>>
>>>> RRD::create_rrd_graph -> see if we can combine both files into one graph. But we
>>>> are relying on RRD itself for this. So that might be interesting. On the other
>>>> hand, do we want to keep that API endpoint around? People are using it though.
>>>> So we need to weigh that decision accoringly.
>>>
>>> Keeping it around for the time being is definitively required now that
>>> PVE 9.0 is out, we might sunset it for a future major release.
>>>
>>> Another option might be to generate both images and merge them manually,
>>> i.e. literally make the right part the old and the left part the new
>>> image with some imagemagick command. Or does rrd indeed supports merging
>>> two files together?
>>
>> One thing that should be possible is to give it two datasets from different files. That would be a middle ground as in, all data is there, but it would show up as separata data lines.
>>
>> I am looking into it to see if we could get it nicer. Manually stitching images together will most likely not work because I don't expected the scales to match up perfectly.
> 
> True w.r.t. not matching scales. But as one can override the line colors we could probably just do that and use the same ones for both data series?

I did some prototyping and we can get both lines looking the same in the 
final graph directly from rrdtool graph.

I am now preparing a larger patch series, that also will include 
simplified/reverted changes now that we can count on the new 
pve-TYPE-9.0 rrd file to be present.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [RFC many 0/3] combine and simplify RRD handling

[Aaron Lauterer]

sent a v1 
https://lore.proxmox.com/pve-devel/20250905135517.4005478-1-a.lauterer@proxmox.com/T/#t

On  2025-09-04  16:09, Aaron Lauterer wrote:
> this series does two things:
> * switch the RRD API backend to use the old RRD files is available
> * always use RRD files with the new 9.0 schema
> 
> The motivation for the combination of old and new RRD files is that during the
> migration of old RRD files to the new ones, spikes get flattened. By combining
> the old and new RRD files, we can keep the old coarse data with all it spikes
> and only show the new much mode finer stepped data where available.
> 
> This will also enable us to just write any new data into an RRD file that is
> created with the new 9.0 schema. As a result, no migration step is needed
> anymore and we can simplify the logic around the whole "does the only the old
> file exist or is the new one also present" alot in quite a few places.
> 
> Some examples which we can simplify or revert:
> pve-manager: API2Tools::get_rrd_key can most likely be dropped
> qemu-server, pve-storage, pve-container,…: rrd API endpoints which check which files exist
> drop the whole migration step on upgrade
> 
> other TODOs:
> 
> RRD::create_rrd_graph -> see if we can combine both files into one graph. But we
> are relying on RRD itself for this. So that might be interesting. On the other
> hand, do we want to keep that API endpoint around? People are using it though.
> So we need to weigh that decision accoringly.
> 
> 
> This is currently marked as RFC. Tested by installing it on an existing 9.0.6
> cluster. Added a 8.4 node with latest updates. Renamed {rrd}.old to {rrd}.
> 
> cluster:
> 
> Aaron Lauterer (3):
>    rrd: fix rrd time frames
>    RRD: fetch data from old rrd file if present and needed
>    pmxcfs: status.c: always use 9.0 rrd files
> 
>   src/PVE/RRD.pm      | 113 ++++++++++++++-------
>   src/pmxcfs/status.c | 238 ++++++--------------------------------------
>   2 files changed, 108 insertions(+), 243 deletions(-)
> 
> 
> Summary over all repositories:
>    2 files changed, 108 insertions(+), 243 deletions(-)
> 



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel