[pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Aaron Lauterer]

Until now, the pvestatd did broadcast the pve-manager version only once
after startup of the service. But there are some situations, where the
local pmxcfs (pve-cluster) restarts and loses that information.
Basically everytime we restart the pmxcfs without restarting pvestatd
too.

For example, on a cluster join, or if the pmxcfs has been restarted
manually.

By additionally checking if the local kv-store of the pmxcfs has any
version info for the node, we can decide if another broadcast is
necessary.
Therefore after the next run of pvestatd, we should have the full
version info available again.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
---
This patch is preparation to get reliable version infos as I am picking
of the patch series of Folke to include more metrics into the RRD data
and summary graphs. [0]
This was a big blocker and now with the major version change coming up,
we at least can assume the latest 8.x installed as part of the update to
PVE 9.
Therefore, we should get this in with PVE 8. Additional patches for PVE
8 will follow to make the transition smoother. But as mentioned, this
here is one of the things that needs to work reliably, which is why I
submit the patch already now.

[0] https://lore.proxmox.com/pve-devel/20231211144721.212071-1-f.gleumes@proxmox.com/

 PVE/Service/pvestatd.pm | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
index 7fa003fe..03c578e1 100755
--- a/PVE/Service/pvestatd.pm
+++ b/PVE/Service/pvestatd.pm
@@ -527,7 +527,10 @@ sub update_sdn_status {
 
 my $broadcast_version_info_done = 0;
 my sub broadcast_version_info : prototype() {
-    if (!$broadcast_version_info_done) {
+    if (
+	!$broadcast_version_info_done
+	|| !keys PVE::Cluster::get_node_kv('version-info', $nodename)->%*
+    ) {
 	PVE::Cluster::broadcast_node_kv(
 	    'version-info',
 	    encode_json(PVE::pvecfg::version_info()),
-- 
2.39.5



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Christian Ebner]

On 1/16/25 17:30, Aaron Lauterer wrote:
> Until now, the pvestatd did broadcast the pve-manager version only once
> after startup of the service. But there are some situations, where the
> local pmxcfs (pve-cluster) restarts and loses that information.
> Basically everytime we restart the pmxcfs without restarting pvestatd
> too.
> 
> For example, on a cluster join, or if the pmxcfs has been restarted
> manually.
> 
> By additionally checking if the local kv-store of the pmxcfs has any
> version info for the node, we can decide if another broadcast is
> necessary.
> Therefore after the next run of pvestatd, we should have the full
> version info available again.
> 
> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
> ---
> This patch is preparation to get reliable version infos as I am picking
> of the patch series of Folke to include more metrics into the RRD data
> and summary graphs. [0]
> This was a big blocker and now with the major version change coming up,
> we at least can assume the latest 8.x installed as part of the update to
> PVE 9.
> Therefore, we should get this in with PVE 8. Additional patches for PVE
> 8 will follow to make the transition smoother. But as mentioned, this
> here is one of the things that needs to work reliably, which is why I
> submit the patch already now.
> 
> [0] https://lore.proxmox.com/pve-devel/20231211144721.212071-1-f.gleumes@proxmox.com/
> 
>   PVE/Service/pvestatd.pm | 5 ++++-
>   1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
> index 7fa003fe..03c578e1 100755
> --- a/PVE/Service/pvestatd.pm
> +++ b/PVE/Service/pvestatd.pm
> @@ -527,7 +527,10 @@ sub update_sdn_status {
>   
>   my $broadcast_version_info_done = 0;
>   my sub broadcast_version_info : prototype() {
> -    if (!$broadcast_version_info_done) {
> +    if (
> +	!$broadcast_version_info_done
> +	|| !keys PVE::Cluster::get_node_kv('version-info', $nodename)->%*
> +    ) {
>   	PVE::Cluster::broadcast_node_kv(
>   	    'version-info',
>   	    encode_json(PVE::pvecfg::version_info()),

This will close issue 5894 I guess [0]?

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=5894


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Aaron Lauterer]

On  2025-01-16  17:35, Christian Ebner wrote:
> On 1/16/25 17:30, Aaron Lauterer wrote:
[…]
> This will close issue 5894 I guess [0]?
> 
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5894

Specifically, the 'version-info', yes. Are there other properties too?



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Christian Ebner]

On 1/16/25 17:38, Aaron Lauterer wrote:
> 
> 
> On  2025-01-16  17:35, Christian Ebner wrote:
>> On 1/16/25 17:30, Aaron Lauterer wrote:
> […]
>> This will close issue 5894 I guess [0]?
>>
>> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5894
> 
> Specifically, the 'version-info', yes. Are there other properties too?

If I remember correctly all are cleared. But only the `version-info` is 
not re-broadcasted as far as I can see from a quick glance at the code.

But might be worth to double check!


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Fiona Ebner]

Am 16.01.25 um 17:30 schrieb Aaron Lauterer:
> Until now, the pvestatd did broadcast the pve-manager version only once
> after startup of the service. But there are some situations, where the
> local pmxcfs (pve-cluster) restarts and loses that information.
> Basically everytime we restart the pmxcfs without restarting pvestatd
> too.
> 
> For example, on a cluster join, or if the pmxcfs has been restarted
> manually.
> 
> By additionally checking if the local kv-store of the pmxcfs has any
> version info for the node, we can decide if another broadcast is
> necessary.
> Therefore after the next run of pvestatd, we should have the full
> version info available again.
> 
> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
> ---
> This patch is preparation to get reliable version infos as I am picking
> of the patch series of Folke to include more metrics into the RRD data
> and summary graphs. [0]
> This was a big blocker and now with the major version change coming up,
> we at least can assume the latest 8.x installed as part of the update to
> PVE 9.
> Therefore, we should get this in with PVE 8. Additional patches for PVE
> 8 will follow to make the transition smoother. But as mentioned, this
> here is one of the things that needs to work reliably, which is why I
> submit the patch already now.

If we start relying more on this, we likely also want:
https://lore.proxmox.com/pve-devel/20221006125414.58279-1-f.ebner@proxmox.com/

> 
> [0] https://lore.proxmox.com/pve-devel/20231211144721.212071-1-f.gleumes@proxmox.com/
> 
>  PVE/Service/pvestatd.pm | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
> index 7fa003fe..03c578e1 100755
> --- a/PVE/Service/pvestatd.pm
> +++ b/PVE/Service/pvestatd.pm
> @@ -527,7 +527,10 @@ sub update_sdn_status {
>  
>  my $broadcast_version_info_done = 0;
>  my sub broadcast_version_info : prototype() {
> -    if (!$broadcast_version_info_done) {
> +    if (
> +	!$broadcast_version_info_done
> +	|| !keys PVE::Cluster::get_node_kv('version-info', $nodename)->%*

Style nit: IMHO, it would be easier to read if surrounded by an explicit
scalar()

> +    ) {
>  	PVE::Cluster::broadcast_node_kv(
>  	    'version-info',
>  	    encode_json(PVE::pvecfg::version_info()),



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Aaron Lauterer]

On  2025-01-17  13:18, Fiona Ebner wrote:
> Am 16.01.25 um 17:30 schrieb Aaron Lauterer:
>> Until now, the pvestatd did broadcast the pve-manager version only once
>> after startup of the service. But there are some situations, where the
>> local pmxcfs (pve-cluster) restarts and loses that information.
>> Basically everytime we restart the pmxcfs without restarting pvestatd
>> too.
>>
>> For example, on a cluster join, or if the pmxcfs has been restarted
>> manually.
>>
>> By additionally checking if the local kv-store of the pmxcfs has any
>> version info for the node, we can decide if another broadcast is
>> necessary.
>> Therefore after the next run of pvestatd, we should have the full
>> version info available again.
>>
>> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
>> ---
>> This patch is preparation to get reliable version infos as I am picking
>> of the patch series of Folke to include more metrics into the RRD data
>> and summary graphs. [0]
>> This was a big blocker and now with the major version change coming up,
>> we at least can assume the latest 8.x installed as part of the update to
>> PVE 9.
>> Therefore, we should get this in with PVE 8. Additional patches for PVE
>> 8 will follow to make the transition smoother. But as mentioned, this
>> here is one of the things that needs to work reliably, which is why I
>> submit the patch already now.
> 
> If we start relying more on this, we likely also want:
> https://lore.proxmox.com/pve-devel/20221006125414.58279-1-f.ebner@proxmox.com/

Hmm, honestly, I might prefer having the last known version info still 
present. That would make it easier to determine if all cluster nodes are 
on at least a required version ;).

But I think it would be better, with RRD data migration in mind, to make 
it mandatory that all cluster nodes are online before one can proceed 
instead of relying on stale version infos.

> 
>>
>> [0] https://lore.proxmox.com/pve-devel/20231211144721.212071-1-f.gleumes@proxmox.com/
>>
>>   PVE/Service/pvestatd.pm | 5 ++++-
>>   1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
>> index 7fa003fe..03c578e1 100755
>> --- a/PVE/Service/pvestatd.pm
>> +++ b/PVE/Service/pvestatd.pm
>> @@ -527,7 +527,10 @@ sub update_sdn_status {
>>   
>>   my $broadcast_version_info_done = 0;
>>   my sub broadcast_version_info : prototype() {
>> -    if (!$broadcast_version_info_done) {
>> +    if (
>> +	!$broadcast_version_info_done
>> +	|| !keys PVE::Cluster::get_node_kv('version-info', $nodename)->%*
> 
> Style nit: IMHO, it would be easier to read if surrounded by an explicit
> scalar()

You mean to have it like this?
| !scaler(keys PVE::Cluster::get_node_kv('version-info', $nodename)->%*)
> 
>> +    ) {
>>   	PVE::Cluster::broadcast_node_kv(
>>   	    'version-info',
>>   	    encode_json(PVE::pvecfg::version_info()),
> 



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Fiona Ebner]

Am 26.02.25 um 17:02 schrieb Aaron Lauterer:
> 
> 
> On  2025-01-17  13:18, Fiona Ebner wrote:
>> Am 16.01.25 um 17:30 schrieb Aaron Lauterer:
>>> Until now, the pvestatd did broadcast the pve-manager version only once
>>> after startup of the service. But there are some situations, where the
>>> local pmxcfs (pve-cluster) restarts and loses that information.
>>> Basically everytime we restart the pmxcfs without restarting pvestatd
>>> too.
>>>
>>> For example, on a cluster join, or if the pmxcfs has been restarted
>>> manually.
>>>
>>> By additionally checking if the local kv-store of the pmxcfs has any
>>> version info for the node, we can decide if another broadcast is
>>> necessary.
>>> Therefore after the next run of pvestatd, we should have the full
>>> version info available again.
>>>
>>> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
>>> ---
>>> This patch is preparation to get reliable version infos as I am picking
>>> of the patch series of Folke to include more metrics into the RRD data
>>> and summary graphs. [0]
>>> This was a big blocker and now with the major version change coming up,
>>> we at least can assume the latest 8.x installed as part of the update to
>>> PVE 9.
>>> Therefore, we should get this in with PVE 8. Additional patches for PVE
>>> 8 will follow to make the transition smoother. But as mentioned, this
>>> here is one of the things that needs to work reliably, which is why I
>>> submit the patch already now.
>>
>> If we start relying more on this, we likely also want:
>> https://lore.proxmox.com/pve-devel/20221006125414.58279-1-
>> f.ebner@proxmox.com/
> 
> Hmm, honestly, I might prefer having the last known version info still
> present. That would make it easier to determine if all cluster nodes are
> on at least a required version ;).

That is an edge case where it might be useful, but I'd argue that in
general, it can be problematic to rely on stale information, especially
if you can't detect if it's stale or not. And IMHO, it's worth doing
properly here too, i.e. wait for the node to send its current version.
You already need to wait for nodes that were not online before.

> 
> But I think it would be better, with RRD data migration in mind, to make
> it mandatory that all cluster nodes are online before one can proceed
> instead of relying on stale version infos.
> 
>>
>>>
>>> [0] https://lore.proxmox.com/pve-devel/20231211144721.212071-1-
>>> f.gleumes@proxmox.com/
>>>
>>>   PVE/Service/pvestatd.pm | 5 ++++-
>>>   1 file changed, 4 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
>>> index 7fa003fe..03c578e1 100755
>>> --- a/PVE/Service/pvestatd.pm
>>> +++ b/PVE/Service/pvestatd.pm
>>> @@ -527,7 +527,10 @@ sub update_sdn_status {
>>>     my $broadcast_version_info_done = 0;
>>>   my sub broadcast_version_info : prototype() {
>>> -    if (!$broadcast_version_info_done) {
>>> +    if (
>>> +    !$broadcast_version_info_done
>>> +    || !keys PVE::Cluster::get_node_kv('version-info', $nodename)->%*
>>
>> Style nit: IMHO, it would be easier to read if surrounded by an explicit
>> scalar()
> 
> You mean to have it like this?
> | !scaler(keys PVE::Cluster::get_node_kv('version-info', $nodename)->%*)

Yes (except for the typo ;P)


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Aaron Lauterer]

On  2025-01-16  17:50, Christian Ebner wrote:
> On 1/16/25 17:38, Aaron Lauterer wrote:
>>
>>
>> On  2025-01-16  17:35, Christian Ebner wrote:
>>> On 1/16/25 17:30, Aaron Lauterer wrote:
>> […]
>>> This will close issue 5894 I guess [0]?
>>>
>>> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5894
>>
>> Specifically, the 'version-info', yes. Are there other properties too?
> 
> If I remember correctly all are cleared. But only the `version-info` is 
> not re-broadcasted as far as I can see from a quick glance at the code.
> 
> But might be worth to double check!

At least in pvestatd I didn't find any other broadcast_node_kv call 
behind a similar condition that would only run it once per start of the 
service.



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Aaron Lauterer]

sent a v2 
https://lore.proxmox.com/pve-devel/20250227143356.1089350-1-a.lauterer@proxmox.com/T/#u

On  2025-01-16  17:30, Aaron Lauterer wrote:
> Until now, the pvestatd did broadcast the pve-manager version only once
> after startup of the service. But there are some situations, where the
> local pmxcfs (pve-cluster) restarts and loses that information.
> Basically everytime we restart the pmxcfs without restarting pvestatd
> too.
> 
> For example, on a cluster join, or if the pmxcfs has been restarted
> manually.
> 
> By additionally checking if the local kv-store of the pmxcfs has any
> version info for the node, we can decide if another broadcast is
> necessary.
> Therefore after the next run of pvestatd, we should have the full
> version info available again.
> 
> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
> ---
> This patch is preparation to get reliable version infos as I am picking
> of the patch series of Folke to include more metrics into the RRD data
> and summary graphs. [0]
> This was a big blocker and now with the major version change coming up,
> we at least can assume the latest 8.x installed as part of the update to
> PVE 9.
> Therefore, we should get this in with PVE 8. Additional patches for PVE
> 8 will follow to make the transition smoother. But as mentioned, this
> here is one of the things that needs to work reliably, which is why I
> submit the patch already now.
> 
> [0] https://lore.proxmox.com/pve-devel/20231211144721.212071-1-f.gleumes@proxmox.com/
> 
>   PVE/Service/pvestatd.pm | 5 ++++-
>   1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/PVE/Service/pvestatd.pm b/PVE/Service/pvestatd.pm
> index 7fa003fe..03c578e1 100755
> --- a/PVE/Service/pvestatd.pm
> +++ b/PVE/Service/pvestatd.pm
> @@ -527,7 +527,10 @@ sub update_sdn_status {
>   
>   my $broadcast_version_info_done = 0;
>   my sub broadcast_version_info : prototype() {
> -    if (!$broadcast_version_info_done) {
> +    if (
> +	!$broadcast_version_info_done
> +	|| !keys PVE::Cluster::get_node_kv('version-info', $nodename)->%*
> +    ) {
>   	PVE::Cluster::broadcast_node_kv(
>   	    'version-info',
>   	    encode_json(PVE::pvecfg::version_info()),



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Fabian Grünbichler]

On February 27, 2025 9:59 am, Fiona Ebner wrote:
> Am 26.02.25 um 17:02 schrieb Aaron Lauterer:
>> 
>> 
>> On  2025-01-17  13:18, Fiona Ebner wrote:
>>> Am 16.01.25 um 17:30 schrieb Aaron Lauterer:
>>>> Until now, the pvestatd did broadcast the pve-manager version only once
>>>> after startup of the service. But there are some situations, where the
>>>> local pmxcfs (pve-cluster) restarts and loses that information.
>>>> Basically everytime we restart the pmxcfs without restarting pvestatd
>>>> too.
>>>>
>>>> For example, on a cluster join, or if the pmxcfs has been restarted
>>>> manually.
>>>>
>>>> By additionally checking if the local kv-store of the pmxcfs has any
>>>> version info for the node, we can decide if another broadcast is
>>>> necessary.
>>>> Therefore after the next run of pvestatd, we should have the full
>>>> version info available again.
>>>>
>>>> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
>>>> ---
>>>> This patch is preparation to get reliable version infos as I am picking
>>>> of the patch series of Folke to include more metrics into the RRD data
>>>> and summary graphs. [0]
>>>> This was a big blocker and now with the major version change coming up,
>>>> we at least can assume the latest 8.x installed as part of the update to
>>>> PVE 9.
>>>> Therefore, we should get this in with PVE 8. Additional patches for PVE
>>>> 8 will follow to make the transition smoother. But as mentioned, this
>>>> here is one of the things that needs to work reliably, which is why I
>>>> submit the patch already now.
>>>
>>> If we start relying more on this, we likely also want:
>>> https://lore.proxmox.com/pve-devel/20221006125414.58279-1-
>>> f.ebner@proxmox.com/
>> 
>> Hmm, honestly, I might prefer having the last known version info still
>> present. That would make it easier to determine if all cluster nodes are
>> on at least a required version ;).
> 
> That is an edge case where it might be useful, but I'd argue that in
> general, it can be problematic to rely on stale information, especially
> if you can't detect if it's stale or not. And IMHO, it's worth doing
> properly here too, i.e. wait for the node to send its current version.
> You already need to wait for nodes that were not online before.

we could make it detectable by including a timestamp? that way, if using
stale information is (not) okay, that decision can be made by the
consumer of the information, instead of only allowing either variant?


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Fiona Ebner]

Am 27.02.25 um 15:52 schrieb Fabian Grünbichler:
> On February 27, 2025 9:59 am, Fiona Ebner wrote:
>> Am 26.02.25 um 17:02 schrieb Aaron Lauterer:
>>>
>>>
>>> On  2025-01-17  13:18, Fiona Ebner wrote:
>>>> Am 16.01.25 um 17:30 schrieb Aaron Lauterer:
>>>>> Until now, the pvestatd did broadcast the pve-manager version only once
>>>>> after startup of the service. But there are some situations, where the
>>>>> local pmxcfs (pve-cluster) restarts and loses that information.
>>>>> Basically everytime we restart the pmxcfs without restarting pvestatd
>>>>> too.
>>>>>
>>>>> For example, on a cluster join, or if the pmxcfs has been restarted
>>>>> manually.
>>>>>
>>>>> By additionally checking if the local kv-store of the pmxcfs has any
>>>>> version info for the node, we can decide if another broadcast is
>>>>> necessary.
>>>>> Therefore after the next run of pvestatd, we should have the full
>>>>> version info available again.
>>>>>
>>>>> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
>>>>> ---
>>>>> This patch is preparation to get reliable version infos as I am picking
>>>>> of the patch series of Folke to include more metrics into the RRD data
>>>>> and summary graphs. [0]
>>>>> This was a big blocker and now with the major version change coming up,
>>>>> we at least can assume the latest 8.x installed as part of the update to
>>>>> PVE 9.
>>>>> Therefore, we should get this in with PVE 8. Additional patches for PVE
>>>>> 8 will follow to make the transition smoother. But as mentioned, this
>>>>> here is one of the things that needs to work reliably, which is why I
>>>>> submit the patch already now.
>>>>
>>>> If we start relying more on this, we likely also want:
>>>> https://lore.proxmox.com/pve-devel/20221006125414.58279-1-
>>>> f.ebner@proxmox.com/
>>>
>>> Hmm, honestly, I might prefer having the last known version info still
>>> present. That would make it easier to determine if all cluster nodes are
>>> on at least a required version ;).
>>
>> That is an edge case where it might be useful, but I'd argue that in
>> general, it can be problematic to rely on stale information, especially
>> if you can't detect if it's stale or not. And IMHO, it's worth doing
>> properly here too, i.e. wait for the node to send its current version.
>> You already need to wait for nodes that were not online before.
> 
> we could make it detectable by including a timestamp? that way, if using
> stale information is (not) okay, that decision can be made by the
> consumer of the information, instead of only allowing either variant?

If it's broadcast only once then the timestamp doesn't help much? Or do
you mean also keeping track/checking when the node last joined the
quorum to decide?


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Fabian Grünbichler]

> Fiona Ebner <f.ebner@proxmox.com> hat am 27.02.2025 16:00 CET geschrieben:
> 
>  
> Am 27.02.25 um 15:52 schrieb Fabian Grünbichler:
> > On February 27, 2025 9:59 am, Fiona Ebner wrote:
> >> Am 26.02.25 um 17:02 schrieb Aaron Lauterer:
> >>>
> >>>
> >>> On  2025-01-17  13:18, Fiona Ebner wrote:
> >>>> Am 16.01.25 um 17:30 schrieb Aaron Lauterer:
> >>>>> Until now, the pvestatd did broadcast the pve-manager version only once
> >>>>> after startup of the service. But there are some situations, where the
> >>>>> local pmxcfs (pve-cluster) restarts and loses that information.
> >>>>> Basically everytime we restart the pmxcfs without restarting pvestatd
> >>>>> too.
> >>>>>
> >>>>> For example, on a cluster join, or if the pmxcfs has been restarted
> >>>>> manually.
> >>>>>
> >>>>> By additionally checking if the local kv-store of the pmxcfs has any
> >>>>> version info for the node, we can decide if another broadcast is
> >>>>> necessary.
> >>>>> Therefore after the next run of pvestatd, we should have the full
> >>>>> version info available again.
> >>>>>
> >>>>> Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
> >>>>> ---
> >>>>> This patch is preparation to get reliable version infos as I am picking
> >>>>> of the patch series of Folke to include more metrics into the RRD data
> >>>>> and summary graphs. [0]
> >>>>> This was a big blocker and now with the major version change coming up,
> >>>>> we at least can assume the latest 8.x installed as part of the update to
> >>>>> PVE 9.
> >>>>> Therefore, we should get this in with PVE 8. Additional patches for PVE
> >>>>> 8 will follow to make the transition smoother. But as mentioned, this
> >>>>> here is one of the things that needs to work reliably, which is why I
> >>>>> submit the patch already now.
> >>>>
> >>>> If we start relying more on this, we likely also want:
> >>>> https://lore.proxmox.com/pve-devel/20221006125414.58279-1-
> >>>> f.ebner@proxmox.com/
> >>>
> >>> Hmm, honestly, I might prefer having the last known version info still
> >>> present. That would make it easier to determine if all cluster nodes are
> >>> on at least a required version ;).
> >>
> >> That is an edge case where it might be useful, but I'd argue that in
> >> general, it can be problematic to rely on stale information, especially
> >> if you can't detect if it's stale or not. And IMHO, it's worth doing
> >> properly here too, i.e. wait for the node to send its current version.
> >> You already need to wait for nodes that were not online before.
> > 
> > we could make it detectable by including a timestamp? that way, if using
> > stale information is (not) okay, that decision can be made by the
> > consumer of the information, instead of only allowing either variant?
> 
> If it's broadcast only once then the timestamp doesn't help much? Or do
> you mean also keeping track/checking when the node last joined the
> quorum to decide?

no, I meant broadcast it regularly (e.g., one could refresh and rebroadcast
the information based on some file being changed that is always touched by
dpkg/apt on package operations? or just on a schedule that is less frequent
than "every pvestatd cycle") *and* include the timestamp of the last update
so the other side can act on that..


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager] pvestatd: improve broadcast of node version-info

[Thomas Lamprecht]

Am 27.02.25 um 16:11 schrieb Fabian Grünbichler:
>> Fiona Ebner <f.ebner@proxmox.com> hat am 27.02.2025 16:00 CET geschrieben:
>> Am 27.02.25 um 15:52 schrieb Fabian Grünbichler:
>>> we could make it detectable by including a timestamp? that way, if using
>>> stale information is (not) okay, that decision can be made by the
>>> consumer of the information, instead of only allowing either variant?
>>
>> If it's broadcast only once then the timestamp doesn't help much? Or do
>> you mean also keeping track/checking when the node last joined the
>> quorum to decide?
> 
> no, I meant broadcast it regularly (e.g., one could refresh and rebroadcast
> the information based on some file being changed that is always touched by
> dpkg/apt on package operations? or just on a schedule that is less frequent
> than "every pvestatd cycle") *and* include the timestamp of the last update
> so the other side can act on that..

For version specific information we could also hook into apt/dpkg to trigger
such an update on-demand plus with a low frequency (say hourly) periodically,
and that could be optimized to just write a newer timestamp if the info stayed
the same, the simplest way to do that would probably be having to keys, one
for the timestamp and one for the version info.

FWIW, I'd also slightly favor in having the information but with a timestamp,
with periodic updates and a node-last-online timestamp one could determine
if the information is stale for sure if the node is offline or if the info is
older than $now minus the period length. That said, I did not evaluate that
with all potential use cases in mind, so not a hard recommendation to go that
way.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel