Re: [pve-devel] [PATCH qemu-server] fix #6207: vm status: cache last disk read/write values

[Thomas Lamprecht <t.lamprecht@proxmox.com>]

Am 25.09.25 um 10:27 schrieb Fiona Ebner:
> Am 22.09.25 um 7:26 PM schrieb Thomas Lamprecht:
>> Am 22.09.25 um 12:18 schrieb Fiona Ebner:
>>> If disk read/write cannot be queried because of QMP timeout, they
>>> should not be reported as 0, but the last value should be re-used.
>>> Otherwise, the difference between that reported 0 and the next value,
>>> when the stats are queried successfully, will show up as a huge spike
>>> in the RRD graphs.
>>
>> Fine with the idea in general, but this is effectively relevant for
>> the pvestatd only though?
>>
>> As of now we would also cache in the API daemon, without every using
>> this. Might not be _that_ much, so not really a problem of the amount,
>> but feels a bit wrong to me w.r.t. "code place".
>>
>> Has pvestatd the necessary info, directly on indirectly through the
>> existence of some other vmstatus properties, to derive when it can
>> safely reuse the previous value?
> 
> It's safe (and sensible/required) if and only if there is no new value.
> We could have the cache be only inside pvestatd, initialize the cache
> with a value of 0 and properly report diskread/write values as undef if
> we cannot get an actual value, and have that mean "re-use previous
> value". (Aside: we cannot use 0 instead of undef to mean "re-use
> previous value", because there are edge cases where a later 0 actually
> means 0 again, for example, all disk unplugged).

Yeah, it would have to be a invalid value like -1, but even that is
naturally not ideal, an explicit undefined or null value would
naturally be better to signal what's happening.

>> Or maybe we could make this caching opt-in through some module flag
>> that only pvestatd sets? But not really thought that through, so
>> please take this with a grain of salt.
>>
>> btw. what about QMP being "stuck" for a prolonged time, should we
>> stop using the previous value after a few times (or duration)? 
> 
> What other value could we use? Since the graph looks at the differences
> of reported values, the only reasonable value we can use if we cannot
> get a new one is the previous one. No matter how long it takes to get a
> new one, or there will be that completely wrong spike again. Or is there
> a N/A kind of value that we could use, where RRD/graph would be smart
> enough to know "I cannot calculate a difference now, will have to wait
> for multiple good values"? Then I'd go for that instead of the current
> approach.

That should never be the problem of the metric collecting entity, but of
the one interpreting or displaying the data, as else this is creating a
false impression of reality.

So the more I think of this, the more I'm sure that we won't do anybody
a favor in the mid/long term here with "faking it" in the backend.

I'd need to look into RRD, but even if there wasn't a way there to
submit null-ish values, I'd rather see that as further argument for
switching out RRD with the rust based proxmox-rrd crate, where we have
control over these things, compared to recording measurements that did
not happen.

That does not mean that doing this correctly in proxmox-rrd will be
trivial to do once we migrated–which is non-trivial on it's own–though.
There are also some ideas to switching to a rather different way to
encode metrics, using a more flexible format and stuff like delta
encoding, i.e. closer to modern time series DBs like influxdb do it,
Lukas signaled some interest in this work here.
But that is vaporware as of now, so no need to wait on that to happen
now, just wanted to mention it to not have those ideas isolated to much.


But taking a step back, why is QMP even timing out here? Is this not
just reading some in-memory counters that QEMU has ready to go?


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel