[pve-devel] [PATCH manager 1/7] report: add kernel command line from current boot

[pve-devel] [PATCH manager 1/7] report: add kernel command line from current boot

[Alexander Zeidler]

to get a first clue for debugging passthrough and similar issues, when
no dmesg output has been provided yet.

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
v2:
* move away from dmesg base
* only print kernel command line (boot times can be added by another patch)

v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062342.html


 PVE/Report.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index 53ffdcbb..a8c1cab9 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -31,6 +31,7 @@ my $init_report_cmds = sub {
 	    cmds => [
 		'hostname',
 		'date -R',
+		'cat /proc/cmdline',
 		'pveversion --verbose',
 		'cat /etc/hosts',
 		'pvesubscription get',
-- 
2.39.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager 2/7] report: create "jobs" section, add `jobs.cfg`

[Alexander Zeidler]

to recognize temporal correlations with network/load/backup/etc issues

Suggested-by: Friedrich Weber <f.weber@proxmox.com>
Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
v2:
* move away from "general" section

v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062346.html


 PVE/Report.pm | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index a8c1cab9..4588b2da 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -99,6 +99,12 @@ my $init_report_cmds = sub {
 		'cat /etc/pve/datacenter.cfg',
 	    ],
 	},
+	jobs => {
+	    order => 65,
+	    cmds => [
+		'cat /etc/pve/jobs.cfg',
+	    ],
+	},
 	hardware => {
 	    order => 70,
 	    cmds => [
-- 
2.39.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager 3/7] report: add `apt-cache policy` to list recognized APT sources

[Alexander Zeidler]

with their details as well as pinned packages. Omit the "origin"
lines, as their value is already visible in the URLs.

 # apt-cache policy ...
 Package files:
  100 /var/lib/dpkg/status
      release a=now
  500 https://enterprise.proxmox.com/debian/pve bookworm/pve-enterprise amd64 Packages
      release o=Proxmox,a=stable,n=bookworm,l=Proxmox VE Enterprise Debian Repository,c=pve-enterprise,b=amd64
 ...
 Pinned packages:
      intel-microcode -> 3.20231114.1~deb12u1 with priority 1234

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
Expects applied:
report: fix regex of config filenames
https://lists.proxmox.com/pipermail/pve-devel/2024-April/063254.html


v2:
* no changes

v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062344.html


 PVE/Report.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index 4588b2da..9b6cd95c 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -38,6 +38,7 @@ my $init_report_cmds = sub {
 		'cat /etc/apt/sources.list',
 		sub { dir2text('/etc/apt/sources.list.d/', '.+\.list') },
 		sub { dir2text('/etc/apt/sources.list.d/', '.+\.sources') },
+		'apt-cache policy | grep -vP "^ +origin "',
 		'lscpu',
 		'pvesh get /cluster/resources --type node --output-format=yaml',
 	    ],
-- 
2.39.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager 4/7] report: list held back packages

[Alexander Zeidler]

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
v2:
* newly added


 PVE/Report.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index 9b6cd95c..1ed91c8e 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -39,6 +39,7 @@ my $init_report_cmds = sub {
 		sub { dir2text('/etc/apt/sources.list.d/', '.+\.list') },
 		sub { dir2text('/etc/apt/sources.list.d/', '.+\.sources') },
 		'apt-cache policy | grep -vP "^ +origin "',
+		'apt-mark showhold',
 		'lscpu',
 		'pvesh get /cluster/resources --type node --output-format=yaml',
 	    ],
-- 
2.39.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager 5/7] report: overhaul `dmidecode` related output

[Alexander Zeidler]

While using keywords (-t bios,...) would be possible, depending on the
server it also bloats the report with uninteresting information,
hiding the relevant.

Therefore the non-grouped, specific number types are used. Where we
only need specific information, not serial numbers etc., we print the
information from /sys/... which is the same source that dmidecode uses
per default.

New output includes:

sys_vendor:	HP
product_name:	ProLiant DL380p Gen8
product_version:	Not specified

and

board_vendor:	ASUSTeK COMPUTER INC.
board_name:	Z13PP-D32 Series
board_version:	60SB09M0-SB0G11

also because there are not always both data blocks available.

`-t 0`:
(like the previous "BIOS" output, but without "BIOS Language" block)

`-t 3`:
Chassis Information
	Manufacturer: HP
	Type: Rack Mount Chassis
	Boot-up State: Critical
	Power Supply State: Critical
	Thermal State: Safe
	Number Of Power Cords: 2
	(...)

and

`-t 32`:
System Boot Information
	Status: Firmware-detected hardware failure

which can hint to Proxmox-independant issues, debug-able via IPMI.

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
v2:
* reformat and extend board output
* add product output
* adapt dmidecode output

v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062350.html


 PVE/Report.pm | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index 1ed91c8e..9d1b9b27 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -110,7 +110,9 @@ my $init_report_cmds = sub {
 	hardware => {
 	    order => 70,
 	    cmds => [
-		'dmidecode -t bios',
+		'cd /sys/devices/virtual/dmi/id; grep -HT "" sys_vendor product_name product_version',
+		'cd /sys/devices/virtual/dmi/id; grep -HT "" board_vendor board_name board_version',
+		'dmidecode -t 0,3,32',
 		'lspci -nnk',
 	    ],
 	},
-- 
2.39.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager 6/7] report: add info about (un)used memory slots

[Alexander Zeidler]

* to see if a RAM upgrade is slot/capacity-wise possible
* to spot added/replaced RAM that may now be causing issues

	Maximum Capacity: 2 TB
	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
	Size: No Module Installed
	Size: No Module Installed
	Size: No Module Installed
	Size: No Module Installed

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
v2:
* make regex stricter
* reduce possible dmidecode output which is not needed here
* reduce and clarify the printed information

v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062348.html


 PVE/Report.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index 9d1b9b27..d9f81a0f 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -113,6 +113,7 @@ my $init_report_cmds = sub {
 		'cd /sys/devices/virtual/dmi/id; grep -HT "" sys_vendor product_name product_version',
 		'cd /sys/devices/virtual/dmi/id; grep -HT "" board_vendor board_name board_version',
 		'dmidecode -t 0,3,32',
+		'dmidecode -t16,17 | grep -P "^\t(Max[^:]*city|Size|Part)" | sed -Ez "s/\n(\tP[^\n]*)/\1/g" | sort',
 		'lspci -nnk',
 	    ],
 	},
-- 
2.39.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] [PATCH manager 7/7] report: add recent boot timestamps which may show fencing/crash events

[Alexander Zeidler]

Successful boots which crashed somehow and sometime afterwards, will
show the same "until" value ("still running" or timestamp) as the next
following boot(s). The most recent boot from such a sequence of
duplicated "until" lines, has not been crashed or not yet.

Example output where only the boot from 16:25:41 crashed:
 reboot system boot 6.5.11-7-pve Thu Apr 11 16:31:24 2024 still running
 reboot system boot 6.5.11-7-pve Thu Apr 11 16:29:17 2024 - Thu Apr 11 16:31:12 2024 (00:01)
 reboot system boot 6.5.11-7-pve Thu Apr 11 16:25:41 2024 - Thu Apr 11 16:31:12 2024 (00:05)
 ...

Furthermore, it shows the booted/crashed/problematic kernel version.

`last` is also used since currently `journalctl --list-boots` can take
10 seconds or even longer on some systems, with no option to limit the
amount of reported boot lines.

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
v2:
* move away from dmesg base
* list also recent (5) boot timestamps with additional information

v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062342.html


 PVE/Report.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index d9f81a0f..c3abb776 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -32,6 +32,7 @@ my $init_report_cmds = sub {
 		'hostname',
 		'date -R',
 		'cat /proc/cmdline',
+		'last reboot -F -n5',
 		'pveversion --verbose',
 		'cat /etc/hosts',
 		'pvesubscription get',
-- 
2.39.2



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 1/7] report: add kernel command line from current boot

[Alexander Zeidler]

Sorry, this is of course v2.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 6/7] report: add info about (un)used memory slots

[Aaron Lauterer]

I am not sure how often we actually need that information as it can add 
quite a bit of additional lines in the report in larger machines with 
many memory slots.

It might be better to keep that command in a cheatsheet to ask for it if 
actually needed instead of polluting the report :)

On  2024-04-18  11:16, Alexander Zeidler wrote:
> * to see if a RAM upgrade is slot/capacity-wise possible
> * to spot added/replaced RAM that may now be causing issues
> 
> 	Maximum Capacity: 2 TB
> 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> 	Size: No Module Installed
> 	Size: No Module Installed
> 	Size: No Module Installed
> 	Size: No Module Installed
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> v2:
> * make regex stricter
> * reduce possible dmidecode output which is not needed here
> * reduce and clarify the printed information
> 
> v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062348.html
> 
> 
>   PVE/Report.pm | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 9d1b9b27..d9f81a0f 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -113,6 +113,7 @@ my $init_report_cmds = sub {
>   		'cd /sys/devices/virtual/dmi/id; grep -HT "" sys_vendor product_name product_version',
>   		'cd /sys/devices/virtual/dmi/id; grep -HT "" board_vendor board_name board_version',
>   		'dmidecode -t 0,3,32',
> +		'dmidecode -t16,17 | grep -P "^\t(Max[^:]*city|Size|Part)" | sed -Ez "s/\n(\tP[^\n]*)/\1/g" | sort',
>   		'lspci -nnk',
>   	    ],
>   	},


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 1/7] report: add kernel command line from current boot

[Aaron Lauterer]

gave the series a test by creating a system report on my local machine.

looks good and contains usefull additional information.

The only thing I am not so sure about, is the memory dimm info (patch 6).

Reviewed-By: Aaron Lauterer <a.lauterer@proxmox.com>
Tested-By: Aaron Lauterer <a.lauterer@proxmox.com>

On  2024-04-18  11:16, Alexander Zeidler wrote:
> to get a first clue for debugging passthrough and similar issues, when
> no dmesg output has been provided yet.
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> v2:
> * move away from dmesg base
> * only print kernel command line (boot times can be added by another patch)
> 
> v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062342.html
> 
> 
>   PVE/Report.pm | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 53ffdcbb..a8c1cab9 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -31,6 +31,7 @@ my $init_report_cmds = sub {
>   	    cmds => [
>   		'hostname',
>   		'date -R',
> +		'cat /proc/cmdline',
>   		'pveversion --verbose',
>   		'cat /etc/hosts',
>   		'pvesubscription get',


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 6/7] report: add info about (un)used memory slots

[Mira Limbeck]

On 4/18/24 11:16, Alexander Zeidler wrote:
> * to see if a RAM upgrade is slot/capacity-wise possible
> * to spot added/replaced RAM that may now be causing issues
> 
> 	Maximum Capacity: 2 TB
> 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> 	Size: No Module Installed
> 	Size: No Module Installed
> 	Size: No Module Installed
> 	Size: No Module Installed
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> v2:
> * make regex stricter
> * reduce possible dmidecode output which is not needed here
> * reduce and clarify the printed information
> 
> v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062348.html
> 
> 
>  PVE/Report.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 9d1b9b27..d9f81a0f 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -113,6 +113,7 @@ my $init_report_cmds = sub {
>  		'cd /sys/devices/virtual/dmi/id; grep -HT "" sys_vendor product_name product_version',
>  		'cd /sys/devices/virtual/dmi/id; grep -HT "" board_vendor board_name board_version',
>  		'dmidecode -t 0,3,32',
> +		'dmidecode -t16,17 | grep -P "^\t(Max[^:]*city|Size|Part)" | sed -Ez "s/\n(\tP[^\n]*)/\1/g" | sort',
Please add a space between `-t` and `16,17`.

>  		'lspci -nnk',
>  	    ],
>  	},

To be honest, I'm not a fan of this command. In those few cases it was
interesting to see, we often wanted to see all the information. And the
full dmidecode -t 17 output is just too much for the report.

Not to mention the quite complicated `grep | sed | sort` part that might
break on some systems that report things differently.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 5/7] report: overhaul `dmidecode` related output

[Mira Limbeck]

On 4/18/24 11:16, Alexander Zeidler wrote:
> While using keywords (-t bios,...) would be possible, depending on the
> server it also bloats the report with uninteresting information,
> hiding the relevant.
> 
> Therefore the non-grouped, specific number types are used. Where we
> only need specific information, not serial numbers etc., we print the
> information from /sys/... which is the same source that dmidecode uses
> per default.
> 
> New output includes:
> 
> sys_vendor:	HP
> product_name:	ProLiant DL380p Gen8
> product_version:	Not specified
Would it be possible to align these correctly, or just use a single
space between type and value?

>> and
> 
> board_vendor:	ASUSTeK COMPUTER INC.
> board_name:	Z13PP-D32 Series
> board_version:	60SB09M0-SB0G11
> 
> also because there are not always both data blocks available.
> 
> `-t 0`:
> (like the previous "BIOS" output, but without "BIOS Language" block)
> 
> `-t 3`:
> Chassis Information
> 	Manufacturer: HP
> 	Type: Rack Mount Chassis
> 	Boot-up State: Critical
> 	Power Supply State: Critical
> 	Thermal State: Safe
> 	Number Of Power Cords: 2
> 	(...)
> 
> and
> 
> `-t 32`:
> System Boot Information
> 	Status: Firmware-detected hardware failure
> 
> which can hint to Proxmox-independant issues, debug-able via IPMI.
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> v2:
> * reformat and extend board output
> * add product output
> * adapt dmidecode output
> 
> v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062350.html
> 
> 
>  PVE/Report.pm | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 1ed91c8e..9d1b9b27 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -110,7 +110,9 @@ my $init_report_cmds = sub {
>  	hardware => {
>  	    order => 70,
>  	    cmds => [
> -		'dmidecode -t bios',
> +		'cd /sys/devices/virtual/dmi/id; grep -HT "" sys_vendor product_name product_version',
> +		'cd /sys/devices/virtual/dmi/id; grep -HT "" board_vendor board_name board_version',
I'd prefer explicit commands like:
cat /sys/devices/virtual/dmi/id/sys_vendor
cat /sys/devices/virtual/dmi/id/product_name
cat /sys/devices/virtual/dmi/id/product_version

This results in a few more lines in the report, since we have the
following structure:
# <command>
<output of command>
<newline>
# <command>
...

Adding at least one additional newline per file read, but there would be
no need for cd-ing, printing the filename with grep and trying to align
the output.

> +		'dmidecode -t 0,3,32',
>  		'lspci -nnk',
>  	    ],
>  	},


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 7/7] report: add recent boot timestamps which may show fencing/crash events

[Mira Limbeck]

On 4/18/24 11:16, Alexander Zeidler wrote:
> Successful boots which crashed somehow and sometime afterwards, will
> show the same "until" value ("still running" or timestamp) as the next
> following boot(s). The most recent boot from such a sequence of
> duplicated "until" lines, has not been crashed or not yet.
> 
> Example output where only the boot from 16:25:41 crashed:
>  reboot system boot 6.5.11-7-pve Thu Apr 11 16:31:24 2024 still running
>  reboot system boot 6.5.11-7-pve Thu Apr 11 16:29:17 2024 - Thu Apr 11 16:31:12 2024 (00:01)
>  reboot system boot 6.5.11-7-pve Thu Apr 11 16:25:41 2024 - Thu Apr 11 16:31:12 2024 (00:05)
>  ...
> 
> Furthermore, it shows the booted/crashed/problematic kernel version.
> 
> `last` is also used since currently `journalctl --list-boots` can take
> 10 seconds or even longer on some systems, with no option to limit the
> amount of reported boot lines.
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> v2:
> * move away from dmesg base
> * list also recent (5) boot timestamps with additional information
> 
> v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062342.html
> 
> 
>  PVE/Report.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index d9f81a0f..c3abb776 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -32,6 +32,7 @@ my $init_report_cmds = sub {
>  		'hostname',
>  		'date -R',
>  		'cat /proc/cmdline',
> +		'last reboot -F -n5',
>  		'pveversion --verbose',
>  		'cat /etc/hosts',
>  		'pvesubscription get',

Do we want the reboot info that far up, even above the version output?
I'd say it's less interesting most of the time than the `pveversion` output.

And for uptime, we do have /cluster/resources and `top` which both show it.
Maybe it could be moved a bit further down? After /cluster/resources
could perhaps be a nice spot since it is (currently) followed by `top`?


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 1/7] report: add kernel command line from current boot

[Mira Limbeck]

On 4/18/24 11:16, Alexander Zeidler wrote:
> to get a first clue for debugging passthrough and similar issues, when
> no dmesg output has been provided yet.
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> v2:
> * move away from dmesg base
> * only print kernel command line (boot times can be added by another patch)
> 
> v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062342.html
> 
> 
>  PVE/Report.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 53ffdcbb..a8c1cab9 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -31,6 +31,7 @@ my $init_report_cmds = sub {
>  	    cmds => [
>  		'hostname',
>  		'date -R',
> +		'cat /proc/cmdline',
>  		'pveversion --verbose',
>  		'cat /etc/hosts',
>  		'pvesubscription get',
Tested the whole patch series.
Looks good so far, especially the cmdline and APT information is nice to
have.

With the added APT information between subscription info, lscpu and
/cluster/resources, would it maybe make sense to move `pvesubscription
get` and `pvesh get /cluster/resources` next to each other, and `lscpu`
to the hardware section where we have all the `dmidecode` output, and
`lspci`?


Except for the `dmidecode -t16,17` command needing a space for
consistency, I'm fine with any of the additions.
So consider the whole series:

Reviewed-by: Mira Limbeck <m.limbeck@proxmox.com>
Tested-by: Mira Limbeck <m.limbeck@proxmox.com>


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 5/7] report: overhaul `dmidecode` related output

[Alexander Zeidler]

On Thu, 2024-04-18 at 12:24 +0200, Mira Limbeck wrote:
> Would it be possible to align these correctly, or just use a single
> space between type and value?

> I'd prefer explicit commands like:

Better formatting can of course be achieved, see the adapted example below, followed by your mentioned `cat` structure.


# cd /sys/devices/virtual/dmi/id; grep -H "" sys_vendor product_name product_version | column -t -s:
sys_vendor       HP
product_name     ProLiant DL380p Gen8
product_version  Not specified

# cd /sys/devices/virtual/dmi/id; grep -H "" board_vendor board_name board_version | column -t -s:
board_vendor   ASUSTeK COMPUTER INC.
board_name     Z13PP-D32 Series
board_version  60SB09M0-SB0G11


# cat /sys/devices/virtual/dmi/id/sys_vendor
HP

# cat /sys/devices/virtual/dmi/id/product_name
ProLiant DL380p Gen8

# cat /sys/devices/virtual/dmi/id/product_version
Not specified

# cat /sys/devices/virtual/dmi/id/board_vendor
ASUSTeK COMPUTER INC.

# cat /sys/devices/virtual/dmi/id/board_name
Z13PP-D32 Series

# cat /sys/devices/virtual/dmi/id/board_version
60SB09M0-SB0G11


> Adding at least one additional newline per file read, but there would be
> no need for cd-ing, printing the filename with grep and trying to align
> the output.

While so far I see no problem with using `cd` or pipes in general, maybe someone else has a strong preference.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 6/7] report: add info about (un)used memory slots

[Alexander Zeidler]

On Thu, 2024-04-18 at 12:16 +0200, Aaron Lauterer wrote:
> I am not sure how often we actually need that information as it can add 
> quite a bit of additional lines in the report in larger machines with 
> many memory slots.

Good point. So in this case we could simply compact the output by counting, without loosing any information:

# dmidecode -t 16,17 | grep -P "^\t(Max[^:]*city|Size|Part)" | sed -Ez "s/\n(\tP[^\n]*)/\1/g" | sort | uniq -c
      1 	Maximum Capacity: 2 TB
      4 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
      4 	Size: No Module Installed


> It might be better to keep that command in a cheatsheet to ask for it if 
> actually needed instead of polluting the report :)
> 
> On  2024-04-18  11:16, Alexander Zeidler wrote:
> > * to see if a RAM upgrade is slot/capacity-wise possible
> > * to spot added/replaced RAM that may now be causing issues
> > 
> > 	Maximum Capacity: 2 TB
> > 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> > 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> > 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> > 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> > 	Size: No Module Installed
> > 	Size: No Module Installed
> > 	Size: No Module Installed
> > 	Size: No Module Installed



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 6/7] report: add info about (un)used memory slots

[Alexander Zeidler]

On Thu, 2024-04-18 at 12:20 +0200, Mira Limbeck wrote:
> On 4/18/24 11:16, Alexander Zeidler wrote:
> > * to see if a RAM upgrade is slot/capacity-wise possible
> > * to spot added/replaced RAM that may now be causing issues
> > 
> > 	Maximum Capacity: 2 TB
> > 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> > 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> > 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> > 	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1
> > 	Size: No Module Installed
> > 	Size: No Module Installed
> > 	Size: No Module Installed
> > 	Size: No Module Installed
> > 
> > Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>

> >  		'dmidecode -t 0,3,32',
> > +		'dmidecode -t16,17 | grep -P "^\t(Max[^:]*city|Size|Part)" | sed -Ez "s/\n(\tP[^\n]*)/\1/g" | sort',
> Please add a space between `-t` and `16,17`.
Ok

> >  		'lspci -nnk',
> >  	    ],
> >  	},
> 
> To be honest, I'm not a fan of this command.
I see.


> In those few cases it was
> interesting to see, we often wanted to see all the information. And the
> full dmidecode -t 17 output is just too much for the report.
I agree that the full output is neither often needed nor suitable for the
system report. Therefore printing only some information, which can be even
more compact as shown here:
https://lists.proxmox.com/pipermail/pve-devel/2024-April/063313.html


> Not to mention the quite complicated `grep | sed | sort` part that might
> break on some systems that report things differently.
Hm, from a technical point of view this should still be okay? At least
grep and sed is also useful for such (trivial) cases.

If it breaks in the future, we should be able to fix it. If we have never
used it by then, we can also remove it.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 7/7] report: add recent boot timestamps which may show fencing/crash events

[Alexander Zeidler]

On Thu, 2024-04-18 at 12:43 +0200, Mira Limbeck wrote:
> On 4/18/24 11:16, Alexander Zeidler wrote:
> > Successful boots which crashed somehow and sometime afterwards, will
> > show the same "until" value ("still running" or timestamp) as the next
> > following boot(s). The most recent boot from such a sequence of
> > duplicated "until" lines, has not been crashed or not yet.
> > 
> > Example output where only the boot from 16:25:41 crashed:
> >  reboot system boot 6.5.11-7-pve Thu Apr 11 16:31:24 2024 still running
> >  reboot system boot 6.5.11-7-pve Thu Apr 11 16:29:17 2024 - Thu Apr 11 16:31:12 2024 (00:01)
> >  reboot system boot 6.5.11-7-pve Thu Apr 11 16:25:41 2024 - Thu Apr 11 16:31:12 2024 (00:05)
> >  ...
> > 
> > Furthermore, it shows the booted/crashed/problematic kernel version.
> > 
> > `last` is also used since currently `journalctl --list-boots` can take
> > 10 seconds or even longer on some systems, with no option to limit the
> > amount of reported boot lines.
> > 
> > Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> > ---
> > v2:
> > * move away from dmesg base
> > * list also recent (5) boot timestamps with additional information
> > 
> > v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062342.html
> > 
> > 
> >  PVE/Report.pm | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/PVE/Report.pm b/PVE/Report.pm
> > index d9f81a0f..c3abb776 100644
> > --- a/PVE/Report.pm
> > +++ b/PVE/Report.pm
> > @@ -32,6 +32,7 @@ my $init_report_cmds = sub {
> >  		'hostname',
> >  		'date -R',
> >  		'cat /proc/cmdline',
> > +		'last reboot -F -n5',
> >  		'pveversion --verbose',
> >  		'cat /etc/hosts',
> >  		'pvesubscription get',
> 
> Do we want the reboot info that far up, even above the version output?
> I'd say it's less interesting most of the time than the `pveversion` output.

I'm not sure if it really fits better with your suggestion. Because, while
the pveversion output can be considered as often more relevant, I have placed
it as it is because it fits well with the surrounding information:

* You can see/compare the booted kernel versions to the kernel command line
  and pveversion output.

* For the kernel command line it makes rather sense to have it at the
  beginning of the report.

* Also it may be interesting how frequent the host is rebooted (e.g. after
  kernel updates)


Btw. the "wtmp begins ..." output does not have to be the installation date.
In case we do not store this information somewhere, currently something like

stat / | grep Birth

could be used if needed.

> 
> And for uptime, we do have /cluster/resources and `top` which both show it.
> Maybe it could be moved a bit further down? After /cluster/resources
> could perhaps be a nice spot since it is (currently) followed by `top`?



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 1/7] report: add kernel command line from current boot

[Alexander Zeidler]

On Thu, 2024-04-18 at 13:05 +0200, Mira Limbeck wrote:
> On 4/18/24 11:16, Alexander Zeidler wrote:
> > to get a first clue for debugging passthrough and similar issues, when
> > no dmesg output has been provided yet.
> > 
> > Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> > ---
> > v2:
> > * move away from dmesg base
> > * only print kernel command line (boot times can be added by another patch)
> > 
> > v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062342.html
> > 
> > 
> >  PVE/Report.pm | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/PVE/Report.pm b/PVE/Report.pm
> > index 53ffdcbb..a8c1cab9 100644
> > --- a/PVE/Report.pm
> > +++ b/PVE/Report.pm
> > @@ -31,6 +31,7 @@ my $init_report_cmds = sub {
> >  	    cmds => [
> >  		'hostname',
> >  		'date -R',
> > +		'cat /proc/cmdline',
> >  		'pveversion --verbose',
> >  		'cat /etc/hosts',
> >  		'pvesubscription get',
> Tested the whole patch series.
> Looks good so far, especially the cmdline and APT information is nice to
> have.
> 
> With the added APT information between subscription info, lscpu and
> /cluster/resources, would it maybe make sense to move `pvesubscription
> get` and `pvesh get /cluster/resources` next to each other,

Good idea.

>  and `lscpu`
> to the hardware section where we have all the `dmidecode` output, and
> `lspci`?

Maybe placing `lscpu` above or below the potentially accepted (compact
version) of the new memory output? I think it makes sense to start with the
mainboard, followed by cpu/memory and ending with the extensive `lspci`.

The move of `lscpu` was already discussed in v1 here:

[PATCH manager 5/9] report: move `lscpu` & cluster info to more appropriate sections
https://lists.proxmox.com/pipermail/pve-devel/2024-April/062871.html

but I decided to drop it since it was related to another now dropped change.


> Except for the `dmidecode -t16,17` command needing a space for
> consistency, I'm fine with any of the additions.
> So consider the whole series:
> 
> Reviewed-by: Mira Limbeck <m.limbeck@proxmox.com>
> Tested-by: Mira Limbeck <m.limbeck@proxmox.com>



_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH manager 1/7] report: add kernel command line from current boot

[Thomas Lamprecht]

Am 18/04/2024 um 11:16 schrieb Alexander Zeidler:
> to get a first clue for debugging passthrough and similar issues, when
> no dmesg output has been provided yet.
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> v2:
> * move away from dmesg base
> * only print kernel command line (boot times can be added by another patch)
> 
> v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062342.html
> 
> 
>  PVE/Report.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
>

applied, thanks!


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH manager 2/7] report: create "jobs" section, add `jobs.cfg`

[Thomas Lamprecht]

Am 18/04/2024 um 11:16 schrieb Alexander Zeidler:
> to recognize temporal correlations with network/load/backup/etc issues
> 
> Suggested-by: Friedrich Weber <f.weber@proxmox.com>
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> v2:
> * move away from "general" section
> 
> v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062346.html
> 
> 
>  PVE/Report.pm | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
>

applied, thanks!


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH manager 4/7] report: list held back packages

[Thomas Lamprecht]

Am 18/04/2024 um 11:16 schrieb Alexander Zeidler:
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> v2:
> * newly added
> 
> 
>  PVE/Report.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
>

applied, thanks!


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

[pve-devel] applied: [PATCH manager 3/7] report: add `apt-cache policy` to list recognized APT sources

[Thomas Lamprecht]

Am 18/04/2024 um 11:16 schrieb Alexander Zeidler:
> with their details as well as pinned packages. Omit the "origin"
> lines, as their value is already visible in the URLs.
> 
>  # apt-cache policy ...
>  Package files:
>   100 /var/lib/dpkg/status
>       release a=now
>   500 https://enterprise.proxmox.com/debian/pve bookworm/pve-enterprise amd64 Packages
>       release o=Proxmox,a=stable,n=bookworm,l=Proxmox VE Enterprise Debian Repository,c=pve-enterprise,b=amd64
>  ...
>  Pinned packages:
>       intel-microcode -> 3.20231114.1~deb12u1 with priority 1234
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
> Expects applied:
> report: fix regex of config filenames
> https://lists.proxmox.com/pipermail/pve-devel/2024-April/063254.html
> 
> 
> v2:
> * no changes
> 
> v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062344.html
> 
> 
>  PVE/Report.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
>

applied, thanks!


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 7/7] report: add recent boot timestamps which may show fencing/crash events

[Mira Limbeck]

On 4/18/24 17:45, Alexander Zeidler wrote:
> On Thu, 2024-04-18 at 12:43 +0200, Mira Limbeck wrote:
>> On 4/18/24 11:16, Alexander Zeidler wrote:
>>> Successful boots which crashed somehow and sometime afterwards, will
>>> show the same "until" value ("still running" or timestamp) as the next
>>> following boot(s). The most recent boot from such a sequence of
>>> duplicated "until" lines, has not been crashed or not yet.
>>>
>>> Example output where only the boot from 16:25:41 crashed:
>>>  reboot system boot 6.5.11-7-pve Thu Apr 11 16:31:24 2024 still running
>>>  reboot system boot 6.5.11-7-pve Thu Apr 11 16:29:17 2024 - Thu Apr 11 16:31:12 2024 (00:01)
>>>  reboot system boot 6.5.11-7-pve Thu Apr 11 16:25:41 2024 - Thu Apr 11 16:31:12 2024 (00:05)
>>>  ...
>>>
>>> Furthermore, it shows the booted/crashed/problematic kernel version.
>>>
>>> `last` is also used since currently `journalctl --list-boots` can take
>>> 10 seconds or even longer on some systems, with no option to limit the
>>> amount of reported boot lines.
>>>
>>> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
>>> ---
>>> v2:
>>> * move away from dmesg base
>>> * list also recent (5) boot timestamps with additional information
>>>
>>> v1: https://lists.proxmox.com/pipermail/pve-devel/2024-March/062342.html
>>>
>>>
>>>  PVE/Report.pm | 1 +
>>>  1 file changed, 1 insertion(+)
>>>
>>> diff --git a/PVE/Report.pm b/PVE/Report.pm
>>> index d9f81a0f..c3abb776 100644
>>> --- a/PVE/Report.pm
>>> +++ b/PVE/Report.pm
>>> @@ -32,6 +32,7 @@ my $init_report_cmds = sub {
>>>  		'hostname',
>>>  		'date -R',
>>>  		'cat /proc/cmdline',
>>> +		'last reboot -F -n5',
>>>  		'pveversion --verbose',
>>>  		'cat /etc/hosts',
>>>  		'pvesubscription get',
>>
>> Do we want the reboot info that far up, even above the version output?
>> I'd say it's less interesting most of the time than the `pveversion` output.
> 
> I'm not sure if it really fits better with your suggestion. Because, while
> the pveversion output can be considered as often more relevant, I have placed
> it as it is because it fits well with the surrounding information:
> 
> * You can see/compare the booted kernel versions to the kernel command line
>   and pveversion output.
> 
> * For the kernel command line it makes rather sense to have it at the
>   beginning of the report.
The kernel command line makes sense up there. I agree.
But the reboots are often less interesting/important than the pveversion
output.
So I'd prefer the pveversion output to stay as far up as possible (after
hostname, date and cmdline).

>> * Also it may be interesting how frequent the host is rebooted (e.g. after
>   kernel updates)
> 
> 
> Btw. the "wtmp begins ..." output does not have to be the installation date.
> In case we do not store this information somewhere, currently something like
> 
> stat / | grep Birth
> 
> could be used if needed>
>>
>> And for uptime, we do have /cluster/resources and `top` which both show it.
>> Maybe it could be moved a bit further down? After /cluster/resources
>> could perhaps be a nice spot since it is (currently) followed by `top`?
> 
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel@lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
> 


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

Re: [pve-devel] [PATCH manager 7/7] report: add recent boot timestamps which may show fencing/crash events

[Friedrich Weber]

I think having recent boot timestamps and kernel versions in the report
would be nice, I can think of some situations where having this info
available upfront would have sped things up.

I just checked, the patch still applies cleanly.

On 19/04/2024 10:56, Mira Limbeck wrote:
> [...]
> The kernel command line makes sense up there. I agree.
> But the reboots are often less interesting/important than the pveversion
> output.
> So I'd prefer the pveversion output to stay as far up as possible (after
> hostname, date and cmdline).

Regarding the placement of the `last` output, I agree with Mira though
-- I'd also prefer to keep pveversion as far up as possible.


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel