[pve-devel] [PATCH manager 1/9] report: add kernel command line including boot time

[pve-devel] [PATCH manager 1/9] report: add kernel command line including boot time

[Alexander Zeidler]

While using `/proc/cmdline` would already provide an initial info for
debugging passthrough and similar, the use of `dmesg` is an easy way
to get the boot date as an absolute value for free (additional to the
relative value in `uptime` from `top`).

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
 PVE/Report.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index 53ffdcbb..d4191769 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -31,6 +31,7 @@ my $init_report_cmds = sub {
 	    cmds => [
 		'hostname',
 		'date -R',
+		'dmesg -T | head | grep Command',
 		'pveversion --verbose',
 		'cat /etc/hosts',
 		'pvesubscription get',
-- 
2.39.2

[pve-devel] [PATCH manager 2/9] report: add `jobs.cfg` to debug related network/load/backup/etc issues

[Alexander Zeidler]

Suggested-by: Friedrich Weber <f.weber@proxmox.com>
Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
 PVE/Report.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index d4191769..6014f13e 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -35,6 +35,7 @@ my $init_report_cmds = sub {
 		'pveversion --verbose',
 		'cat /etc/hosts',
 		'pvesubscription get',
+		'cat /etc/pve/jobs.cfg',
 		'cat /etc/apt/sources.list',
 		sub { dir2text('/etc/apt/sources.list.d/', '.+\.list') },
 		sub { dir2text('/etc/apt/sources.list.d/', '.+\.sources') },
-- 
2.39.2

[pve-devel] [PATCH manager 3/9] report: add list of upgradable packages

[Alexander Zeidler]

* to easily see if APT already knows about old packages in use and
  their exact version
* to reconsider asking for applying updates as a first recommendation
  if the list is empty and no updates have been released very recently

 # apt list --upgradable ...
 Listing...
 pve-manager  testing  8.1.6  amd64  [upgradable  from:  8.1.4]

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
 PVE/Report.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index 6014f13e..ad5c2aa0 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -36,6 +36,7 @@ my $init_report_cmds = sub {
 		'cat /etc/hosts',
 		'pvesubscription get',
 		'cat /etc/pve/jobs.cfg',
+		'apt list --upgradable 2>/dev/null | sed "s/\//\t/g" | column -tL',
 		'cat /etc/apt/sources.list',
 		sub { dir2text('/etc/apt/sources.list.d/', '.+\.list') },
 		sub { dir2text('/etc/apt/sources.list.d/', '.+\.sources') },
-- 
2.39.2

[pve-devel] [PATCH manager 4/9] report: add `apt-cache policy` to list recognized APT sources

[Alexander Zeidler]

with their details as well as pinned packages. Omit the "origin"
lines, as their value is already visible in the URLs.

 # apt-cache policy ...
 Package files:
  100 /var/lib/dpkg/status
      release a=now
  500 https://enterprise.proxmox.com/debian/pve bookworm/pve-enterprise amd64 Packages
      release o=Proxmox,a=stable,n=bookworm,l=Proxmox VE Enterprise Debian Repository,c=pve-enterprise,b=amd64
 ...
 Pinned packages:
      intel-microcode -> 3.20231114.1~deb12u1 with priority 1234

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
 PVE/Report.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index ad5c2aa0..2c2a5e12 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -37,6 +37,7 @@ my $init_report_cmds = sub {
 		'pvesubscription get',
 		'cat /etc/pve/jobs.cfg',
 		'apt list --upgradable 2>/dev/null | sed "s/\//\t/g" | column -tL',
+		'apt-cache policy | grep -vP "^ +origin "',
 		'cat /etc/apt/sources.list',
 		sub { dir2text('/etc/apt/sources.list.d/', '.+\.list') },
 		sub { dir2text('/etc/apt/sources.list.d/', '.+\.sources') },
-- 
2.39.2

[pve-devel] [PATCH manager 5/9] report: move `lscpu` & cluster info to more appropriate sections

[Alexander Zeidler]

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
 PVE/Report.pm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index 2c2a5e12..505629c7 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -41,8 +41,6 @@ my $init_report_cmds = sub {
 		'cat /etc/apt/sources.list',
 		sub { dir2text('/etc/apt/sources.list.d/', '.+\.list') },
 		sub { dir2text('/etc/apt/sources.list.d/', '.+\.sources') },
-		'lscpu',
-		'pvesh get /cluster/resources --type node --output-format=yaml',
 	    ],
 	},
 	'system-load' => {
@@ -96,6 +94,7 @@ my $init_report_cmds = sub {
 	    order => 60,
 	    cmds => [
 		'pvecm nodes',
+		'pvesh get /cluster/resources --type node --output-format=yaml',
 		'pvecm status',
 		'cat /etc/pve/corosync.conf 2>/dev/null',
 		'ha-manager status',
@@ -106,6 +105,7 @@ my $init_report_cmds = sub {
 	    order => 70,
 	    cmds => [
 		'dmidecode -t bios',
+		'lscpu',
 		'lspci -nnk',
 	    ],
 	},
-- 
2.39.2

[pve-devel] [PATCH manager 6/9] report: switch `dmidecode` to quiet to omit almost never needed info

[Alexander Zeidler]

like on this system:

 # dmidecode -t bios
 # dmidecode 3.4
 Getting SMBIOS data from sysfs.
 SMBIOS 3.0.0 present.

 Handle 0x0000, DMI type 0, 24 bytes

 Handle 0x005C, DMI type 13, 22 bytes

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
 PVE/Report.pm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index 505629c7..3a81bdb2 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -104,7 +104,7 @@ my $init_report_cmds = sub {
 	hardware => {
 	    order => 70,
 	    cmds => [
-		'dmidecode -t bios',
+		'dmidecode -t bios -q',
 		'lscpu',
 		'lspci -nnk',
 	    ],
-- 
2.39.2

[pve-devel] [PATCH manager 7/9] report: add info of the mainboard in use

[Alexander Zeidler]

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
 PVE/Report.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index 3a81bdb2..f28d7b38 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -104,6 +104,7 @@ my $init_report_cmds = sub {
 	hardware => {
 	    order => 70,
 	    cmds => [
+		'cd /sys/devices/virtual/dmi/id; cat board_vendor board_name',
 		'dmidecode -t bios -q',
 		'lscpu',
 		'lspci -nnk',
-- 
2.39.2

[pve-devel] [PATCH manager 8/9] report: add info about (un)used memory slots

[Alexander Zeidler]

* to see if a RAM upgrade is slot/capacity-wise possible
* to spot added/replaced RAM that may now be causing issues

 # dmidecode -t memory ...
 	Maximum Capacity: 2 TB
 	Number Of Devices: 8
 	Size: 16 GB	Micron Technology	18ASF2G72PZ-2G6D1
 	Size: 16 GB	Micron Technology	18ASF2G72PZ-2G6D1
 	Size: 16 GB	Micron Technology	18ASF2G72PZ-2G6D1
 	Size: 16 GB	Micron Technology	18ASF2G72PZ-2G6D1
 	Size: No Module Installed
 	Size: No Module Installed
 	Size: No Module Installed
 	Size: No Module Installed

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
 PVE/Report.pm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index f28d7b38..fe497b43 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -106,6 +106,7 @@ my $init_report_cmds = sub {
 	    cmds => [
 		'cd /sys/devices/virtual/dmi/id; cat board_vendor board_name',
 		'dmidecode -t bios -q',
+		'dmidecode -t memory | grep -E "Capacity|Devices|Size|Manu|Part" | sed -Ez "s/\n\t(M|P)[^:]*: (\S*)/\t\2/g" | sort',
 		'lscpu',
 		'lspci -nnk',
 	    ],
-- 
2.39.2

[pve-devel] [PATCH manager 9/9] report: add microcode info to better assess possible system impacts

[Alexander Zeidler]

* list availability and installation status of `*microcode` packages
* grep for applied "Early OS Microcode Updates"
* grep for (un)patched CPU vulnerability messages

Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
---
 PVE/Report.pm | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/PVE/Report.pm b/PVE/Report.pm
index fe497b43..18c554ec 100644
--- a/PVE/Report.pm
+++ b/PVE/Report.pm
@@ -108,6 +108,8 @@ my $init_report_cmds = sub {
 		'dmidecode -t bios -q',
 		'dmidecode -t memory | grep -E "Capacity|Devices|Size|Manu|Part" | sed -Ez "s/\n\t(M|P)[^:]*: (\S*)/\t\2/g" | sort',
 		'lscpu',
+		'apt list *microcode 2>/dev/null | column -tL',
+		'dmesg | grep -i "microcode\|vuln"',
 		'lspci -nnk',
 	    ],
 	},
-- 
2.39.2

Re: [pve-devel] [PATCH manager 7/9] report: add info of the mainboard in use

[Aaron Lauterer]

If we add more info, then dmidecode -t {1,2,3} might be interesting as 
well as those deliver motherboard, system and chassis infos.

If you want to see what you get there, run them on some decent servers 
as most consumer boards and systems will not show useful information here

On  2024-03-22  14:59, Alexander Zeidler wrote:
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
>   PVE/Report.pm | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 3a81bdb2..f28d7b38 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -104,6 +104,7 @@ my $init_report_cmds = sub {
>   	hardware => {
>   	    order => 70,
>   	    cmds => [
> +		'cd /sys/devices/virtual/dmi/id; cat board_vendor board_name',
>   		'dmidecode -t bios -q',
>   		'lscpu',
>   		'lspci -nnk',

Re: [pve-devel] [PATCH manager 9/9] report: add microcode info to better assess possible system impacts

[Stoiko Ivanov]

On Fri, 22 Mar 2024 14:59:33 +0100
Alexander Zeidler <a.zeidler@proxmox.com> wrote:

> * list availability and installation status of `*microcode` packages
> * grep for applied "Early OS Microcode Updates"
> * grep for (un)patched CPU vulnerability messages
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
>  PVE/Report.pm | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index fe497b43..18c554ec 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -108,6 +108,8 @@ my $init_report_cmds = sub {
>  		'dmidecode -t bios -q',
>  		'dmidecode -t memory | grep -E "Capacity|Devices|Size|Manu|Part" | sed -Ez "s/\n\t(M|P)[^:]*: (\S*)/\t\2/g" | sort',
>  		'lscpu',
> +		'apt list *microcode 2>/dev/null | column -tL',
While `apt` works really well and its output hasn't changed since I
started using it (wheezy or jessie) - I still want to mention it's output
when piping:
```
WARNING: apt does not have a stable CLI interface. Use with caution in
scripts. ```
potentially consider either using our code directly or switching to 
`dpkg -l`?
(but as said `apt` has been pretty stable, and we simply dump the output -
so probably the warning is not too relevant here)

> +		'dmesg | grep -i "microcode\|vuln"',
>  		'lspci -nnk',
>  	    ],
>  	},

Re: [pve-devel] [PATCH manager 8/9] report: add info about (un)used memory slots

[Alexander Zeidler]

On Fri, 2024-03-22 at 14:59 +0100, Alexander Zeidler wrote:
> * to see if a RAM upgrade is slot/capacity-wise possible
> * to spot added/replaced RAM that may now be causing issues
> 
>  # dmidecode -t memory ...
>  	Maximum Capacity: 2 TB
>  	Number Of Devices: 8
>  	Size: 16 GB	Micron Technology	18ASF2G72PZ-2G6D1
>  	Size: 16 GB	Micron Technology	18ASF2G72PZ-2G6D1
>  	Size: 16 GB	Micron Technology	18ASF2G72PZ-2G6D1
>  	Size: 16 GB	Micron Technology	18ASF2G72PZ-2G6D1
>  	Size: No Module Installed
>  	Size: No Module Installed
>  	Size: No Module Installed
>  	Size: No Module Installed
> 
> +		'dmidecode -t memory | grep -E "Capacity|Devices|Size|Manu|Part" | sed -Ez "s/\n\t(M|P)[^:]*: (\S*)/\t\2/g" | sort',
Thank you Mira for pointed out, that the regex is not strict enough on your
system. Originally, I limited the strictness as a trade-off to the command
length. However, by slightly modifying the output, the command line can be
shortened and a higher level of strictness can be achieved:

dmidecode -t16,17 | grep -P "^\t(Max[^:]*city|Size|Part)" | sed -Ez "s/\n(\tP[^\n]*)/\1/g" | sort

	Maximum Capacity: 2 TB
	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1   
	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1   
	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1   
	Size: 16 GB	Part Number: 18ASF2G72PZ-2G6D1   
	Size: No Module Installed
	Size: No Module Installed
	Size: No Module Installed
	Size: No Module Installed

Re: [pve-devel] [PATCH manager 9/9] report: add microcode info to better assess possible system impacts

[Alexander Zeidler]

On Fri, 2024-03-22 at 17:44 +0100, Stoiko Ivanov wrote:
> On Fri, 22 Mar 2024 14:59:33 +0100
> Alexander Zeidler <a.zeidler@proxmox.com> wrote:
> 
> > * list availability and installation status of `*microcode` packages
> > * grep for applied "Early OS Microcode Updates"
> > * grep for (un)patched CPU vulnerability messages
> > 
> > Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> > ---
> >  PVE/Report.pm | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/PVE/Report.pm b/PVE/Report.pm
> > index fe497b43..18c554ec 100644
> > --- a/PVE/Report.pm
> > +++ b/PVE/Report.pm
> > @@ -108,6 +108,8 @@ my $init_report_cmds = sub {
> >  		'dmidecode -t bios -q',
> >  		'dmidecode -t memory | grep -E "Capacity|Devices|Size|Manu|Part" | sed -Ez "s/\n\t(M|P)[^:]*: (\S*)/\t\2/g" | sort',
> >  		'lscpu',
> > +		'apt list *microcode 2>/dev/null | column -tL',
> While `apt` works really well and its output hasn't changed since I
> started using it (wheezy or jessie) - I still want to mention it's output
> when piping:
> ```
> WARNING: apt does not have a stable CLI interface. Use with caution in
> scripts. ```
> potentially consider either using our code directly or switching to 
> `dpkg -l`?
> (but as said `apt` has been pretty stable, and we simply dump the output -
> so probably the warning is not too relevant here)
Thank you! I have noticed the missing -a to list possible further package
versions for downgrading if needed. So `dpkg` and its verbose output would
not be an equal solution. However, since previous package versions can be
looked up in the Debian repo, the whole command may not be needed in the
first place.

Instead it may be better to include the current installed microcode version
in `pveversion` and use the

> > +		'dmesg | grep -i "microcode\|vuln"',

to see if microcode was loaded during this boot.

> >  		'lspci -nnk',
> >  	    ],
> >  	},
> 

Re: [pve-devel] [PATCH manager 1/9] report: add kernel command line including boot time

[Thomas Lamprecht]

On 22/03/2024 14:59, Alexander Zeidler wrote:
> While using `/proc/cmdline` would already provide an initial info for
> debugging passthrough and similar, the use of `dmesg` is an easy way
> to get the boot date as an absolute value for free (additional to the
> relative value in `uptime` from `top`).
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
>  PVE/Report.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 53ffdcbb..d4191769 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -31,6 +31,7 @@ my $init_report_cmds = sub {
>  	    cmds => [
>  		'hostname',
>  		'date -R',
> +		'dmesg -T | head | grep Command',

Meh, I'd rather get the command line explicitly and the last boots
using:

journalctl --list-boots


As here you get much more info, e.g. about (recent) frequent reboots.

Please remember to add this also to the PMG and PBS reports. 

>  		'pveversion --verbose',
>  		'cat /etc/hosts',
>  		'pvesubscription get',

Re: [pve-devel] [PATCH manager 3/9] report: add list of upgradable packages

[Thomas Lamprecht]

On 22/03/2024 14:59, Alexander Zeidler wrote:
> * to easily see if APT already knows about old packages in use and
>   their exact version
> * to reconsider asking for applying updates as a first recommendation
>   if the list is empty and no updates have been released very recently
> 
>  # apt list --upgradable ...
>  Listing...
>  pve-manager  testing  8.1.6  amd64  [upgradable  from:  8.1.4]
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
>  PVE/Report.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 6014f13e..ad5c2aa0 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -36,6 +36,7 @@ my $init_report_cmds = sub {
>  		'cat /etc/hosts',
>  		'pvesubscription get',
>  		'cat /etc/pve/jobs.cfg',
> +		'apt list --upgradable 2>/dev/null | sed "s/\//\t/g" | column -tL',

this can easily add a few 100s KiB of text to the report on major upgrades,
this is just to verbose for an initial report.
I'd rather add just the count of available upgrades, i.e. lie `apt update`
does, which allows you to draw similar conclusions without bloating the
report to unwieldy sizes.

Maybe we could even add that info to the pveversion --verbose output,
as then it'd be already included here and also help on other channels
like the forum. 

>  		'cat /etc/apt/sources.list',
>  		sub { dir2text('/etc/apt/sources.list.d/', '.+\.list') },
>  		sub { dir2text('/etc/apt/sources.list.d/', '.+\.sources') },

Re: [pve-devel] [PATCH manager 2/9] report: add `jobs.cfg` to debug related network/load/backup/etc issues

[Thomas Lamprecht]

On 22/03/2024 14:59, Alexander Zeidler wrote:
> Suggested-by: Friedrich Weber <f.weber@proxmox.com>
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
>  PVE/Report.pm | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index d4191769..6014f13e 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -35,6 +35,7 @@ my $init_report_cmds = sub {
>  		'pveversion --verbose',
>  		'cat /etc/hosts',
>  		'pvesubscription get',
> +		'cat /etc/pve/jobs.cfg',

This does not really fits the general section, and makes it harder to
see the core subscription and repository state. I'd either add it to the
cluster one, as cluster-wide jobs config, or to a new "Jobs" section,
with a slight preference to the latter. 

>  		'cat /etc/apt/sources.list',
>  		sub { dir2text('/etc/apt/sources.list.d/', '.+\.list') },
>  		sub { dir2text('/etc/apt/sources.list.d/', '.+\.sources') },

Re: [pve-devel] [PATCH manager 5/9] report: move `lscpu` & cluster info to more appropriate sections

[Thomas Lamprecht]

And why are those more appropriate? Both fit's the general "always important"
section, so even though they fit the section you moved them too, they also
fit the general one, so some actual reasoning here would be good..

On 22/03/2024 14:59, Alexander Zeidler wrote:
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
>  PVE/Report.pm | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 2c2a5e12..505629c7 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -41,8 +41,6 @@ my $init_report_cmds = sub {
>  		'cat /etc/apt/sources.list',
>  		sub { dir2text('/etc/apt/sources.list.d/', '.+\.list') },
>  		sub { dir2text('/etc/apt/sources.list.d/', '.+\.sources') },
> -		'lscpu',
> -		'pvesh get /cluster/resources --type node --output-format=yaml',
>  	    ],
>  	},
>  	'system-load' => {
> @@ -96,6 +94,7 @@ my $init_report_cmds = sub {
>  	    order => 60,
>  	    cmds => [
>  		'pvecm nodes',
> +		'pvesh get /cluster/resources --type node --output-format=yaml',
>  		'pvecm status',
>  		'cat /etc/pve/corosync.conf 2>/dev/null',
>  		'ha-manager status',
> @@ -106,6 +105,7 @@ my $init_report_cmds = sub {
>  	    order => 70,
>  	    cmds => [
>  		'dmidecode -t bios',
> +		'lscpu',
>  		'lspci -nnk',
>  	    ],
>  	},

Re: [pve-devel] [PATCH manager 6/9] report: switch `dmidecode` to quiet to omit almost never needed info

[Thomas Lamprecht]

On 22/03/2024 14:59, Alexander Zeidler wrote:
> like on this system:
> 
>  # dmidecode -t bios
>  # dmidecode 3.4
>  Getting SMBIOS data from sysfs.
>  SMBIOS 3.0.0 present.
> 
>  Handle 0x0000, DMI type 0, 24 bytes
> 
>  Handle 0x005C, DMI type 13, 22 bytes


The manual page here states that this option also hides other entries:

> Unknown, inactive and OEM-specific entries are not displayed. Meta-data and handle references are hidden.

So especially on newer HW this might contain some actual info beyond
the SMBIOS version/metadata that gets hidden.

I mean, I have no hard feelings here, but I'm wondering if it's really
worth hiding that info if it can include potential relevant stuff.

> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
>  PVE/Report.pm | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index 505629c7..3a81bdb2 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -104,7 +104,7 @@ my $init_report_cmds = sub {
>  	hardware => {
>  	    order => 70,
>  	    cmds => [
> -		'dmidecode -t bios',
> +		'dmidecode -t bios -q',
>  		'lscpu',
>  		'lspci -nnk',
>  	    ],

Re: [pve-devel] [PATCH manager 9/9] report: add microcode info to better assess possible system impacts

[Thomas Lamprecht]

On 22/03/2024 14:59, Alexander Zeidler wrote:
> * list availability and installation status of `*microcode` packages
> * grep for applied "Early OS Microcode Updates"
> * grep for (un)patched CPU vulnerability messages
> 
> Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> ---
>  PVE/Report.pm | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/PVE/Report.pm b/PVE/Report.pm
> index fe497b43..18c554ec 100644
> --- a/PVE/Report.pm
> +++ b/PVE/Report.pm
> @@ -108,6 +108,8 @@ my $init_report_cmds = sub {
>  		'dmidecode -t bios -q',
>  		'dmidecode -t memory | grep -E "Capacity|Devices|Size|Manu|Part" | sed -Ez "s/\n\t(M|P)[^:]*: (\S*)/\t\2/g" | sort',
>  		'lscpu',
> +		'apt list *microcode 2>/dev/null | column -tL',
> +		'dmesg | grep -i "microcode\|vuln"',

I'm wondering if instead of having a handful of dmesg + grep instances
it makes more sense to just add the whole dmesg output as separate
file.

I.e., I would like to have a cluster-wide report collection API that
spawns a task, calls to all nodes to generate a report, saves all of
those reports, including commands or files with very long output as
separate files, and then assembles an archive with all that.

On the long run that would provide nicer UX and also avoid that some
to strict filter hides information that might be relevant for a
specific setup.

I'd much more prefer work time spent on something like that than on
adding the same command a few times with each having some different,
rather a bit brittle looking, pipe chains..

>  		'lspci -nnk',
>  	    ],
>  	},

Re: [pve-devel] [PATCH manager 1/9] report: add kernel command line including boot time

[Alexander Zeidler]

On Mon, 2024-03-25 at 08:52 +0100, Thomas Lamprecht wrote:
> > > > On 22/03/2024 14:59, Alexander Zeidler wrote:
> > > > > > > > While using `/proc/cmdline` would already provide an initial info for
> > > > > > > > debugging passthrough and similar, the use of `dmesg` is an easy way
> > > > > > > > to get the boot date as an absolute value for free (additional to the
> > > > > > > > relative value in `uptime` from `top`).
> > > > > > > > 
> > > > > > > > Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> > > > > > > > ---
> > > > > > > >  PVE/Report.pm | 1 +
> > > > > > > >  1 file changed, 1 insertion(+)
> > > > > > > > 
> > > > > > > > diff --git a/PVE/Report.pm b/PVE/Report.pm
> > > > > > > > index 53ffdcbb..d4191769 100644
> > > > > > > > --- a/PVE/Report.pm
> > > > > > > > +++ b/PVE/Report.pm
> > > > > > > > @@ -31,6 +31,7 @@ my $init_report_cmds = sub {
> > > > > > > >  	    cmds => [
> > > > > > > >  		'hostname',
> > > > > > > >  		'date -R',
> > > > > > > > +		'dmesg -T | head | grep Command',
> > > > 
> > > > Meh, I'd rather get the command line explicitly and the last boots

Alright!

> > > > using:
> > > > 
> > > > journalctl --list-boots

On slow servers / storage drives this execution can easily take 10 seconds
or longer. But there is an alternative that is quite fast and even shows
the booted kernel version:

# last reboot -F -n10
reboot system boot 6.5.11-7-pve Thu Apr 11 16:31:24 2024 still running
reboot system boot 6.5.11-7-pve Thu Apr 11 16:29:17 2024 - Thu Apr 11 16:31:12 2024 (00:01)
reboot system boot 6.5.11-7-pve Thu Apr 11 16:25:41 2024 - Thu Apr 11 16:31:12 2024 (00:05)
...

Furthermore, as simulated and visible above, a duplicated "until" field
hints that the earlier system boot (16:25:41) crashed unexpectedly (sometime
before the next boot at 16:29:17).

Limiting the output to 10 lines seems reasonable for most cases.

> > > > 
> > > > 
> > > > As here you get much more info, e.g. about (recent) frequent reboots.
> > > > 
> > > > Please remember to add this also to the PMG and PBS reports. 
> > > > 
> > > > > > > >  		'pveversion --verbose',
> > > > > > > >  		'cat /etc/hosts',
> > > > > > > >  		'pvesubscription get',
> > > > 

Re: [pve-devel] [PATCH manager 2/9] report: add `jobs.cfg` to debug related network/load/backup/etc issues

[Alexander Zeidler]

On Mon, 2024-03-25 at 09:08 +0100, Thomas Lamprecht wrote:
> > > > On 22/03/2024 14:59, Alexander Zeidler wrote:
> > > > > > > > Suggested-by: Friedrich Weber <f.weber@proxmox.com>
> > > > > > > > Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> > > > > > > > ---
> > > > > > > >  PVE/Report.pm | 1 +
> > > > > > > >  1 file changed, 1 insertion(+)
> > > > > > > > 
> > > > > > > > diff --git a/PVE/Report.pm b/PVE/Report.pm
> > > > > > > > index d4191769..6014f13e 100644
> > > > > > > > --- a/PVE/Report.pm
> > > > > > > > +++ b/PVE/Report.pm
> > > > > > > > @@ -35,6 +35,7 @@ my $init_report_cmds = sub {
> > > > > > > >  		'pveversion --verbose',
> > > > > > > >  		'cat /etc/hosts',
> > > > > > > >  		'pvesubscription get',
> > > > > > > > +		'cat /etc/pve/jobs.cfg',
> > > > 
> > > > This does not really fits the general section, and makes it harder to
> > > > see the core subscription and repository state. I'd either add it to the
> > > > cluster one, as cluster-wide jobs config, or to a new "Jobs" section,
> > > > with a slight preference to the latter. 

A new "jobs" section would indeed be more appropriate.

> > > > 
> > > > > > > >  		'cat /etc/apt/sources.list',
> > > > > > > >  		sub { dir2text('/etc/apt/sources.list.d/', '.+\.list') },
> > > > > > > >  		sub { dir2text('/etc/apt/sources.list.d/', '.+\.sources') },
> > > > 

Re: [pve-devel] [PATCH manager 3/9] report: add list of upgradable packages

[Alexander Zeidler]

On Mon, 2024-03-25 at 09:02 +0100, Thomas Lamprecht wrote:
> On 22/03/2024 14:59, Alexander Zeidler wrote:
> > * to easily see if APT already knows about old packages in use and
> >   their exact version
> > * to reconsider asking for applying updates as a first recommendation
> >   if the list is empty and no updates have been released very recently
> > 
> >  # apt list --upgradable ...
> >  Listing...
> >  pve-manager  testing  8.1.6  amd64  [upgradable  from:  8.1.4]
> > 
> > Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> > ---
> >  PVE/Report.pm | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/PVE/Report.pm b/PVE/Report.pm
> > index 6014f13e..ad5c2aa0 100644
> > --- a/PVE/Report.pm
> > +++ b/PVE/Report.pm
> > @@ -36,6 +36,7 @@ my $init_report_cmds = sub {
> >  		'cat /etc/hosts',
> >  		'pvesubscription get',
> >  		'cat /etc/pve/jobs.cfg',
> > +		'apt list --upgradable 2>/dev/null | sed "s/\//\t/g" | column -tL',
> 
> this can easily add a few 100s KiB of text to the report on major upgrades,
> this is just to verbose for an initial report.
> I'd rather add just the count of available upgrades, i.e. lie `apt update`
> does, which allows you to draw similar conclusions without bloating the
> report to unwieldy sizes.
> 
> Maybe we could even add that info to the pveversion --verbose output,
> as then it'd be already included here and also help on other channels
> like the forum. 

Good points. I plan to include it into pveversion --verbose at the end like this:

```
# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.5.13-5-pve)
pve-manager: 8.1.6 (running version: 8.1.6/b7e8e914a1db70cc) [available: 8.1.10]
...
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve1 [available: 2.2.3-pve2]

Number of upgradable packages: 25
```

Separated by a new line since it is not an expected <package>: <version> line.

Rather placed at the end because, for example, there are sometimes users
who unfortunately only attach a screenshot or copy the last visible lines.

Furthermore, it hints the user to update the system before posting this
information. But if doing so without instruction and if thereby the issue
gets not solved, it might also be bad since it changes the system setup
and previously reported observations/behavior might not be correct any more.


As seen in the example output above, it would be helpful if we append
upgradable package lines with their newest available version.

> 
> >  		'cat /etc/apt/sources.list',
> >  		sub { dir2text('/etc/apt/sources.list.d/', '.+\.list') },
> >  		sub { dir2text('/etc/apt/sources.list.d/', '.+\.sources') },
> 

Re: [pve-devel] [PATCH manager 5/9] report: move `lscpu` & cluster info to more appropriate sections

[Alexander Zeidler]

On Mon, 2024-03-25 at 09:11 +0100, Thomas Lamprecht wrote:
> And why are those more appropriate? Both fit's the general "always important"
> section, so even though they fit the section you moved them too, they also
> fit the general one, so some actual reasoning here would be good..
> 
> On 22/03/2024 14:59, Alexander Zeidler wrote:
> > Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> > ---
> >  PVE/Report.pm | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/PVE/Report.pm b/PVE/Report.pm
> > index 2c2a5e12..505629c7 100644
> > --- a/PVE/Report.pm
> > +++ b/PVE/Report.pm
> > @@ -41,8 +41,6 @@ my $init_report_cmds = sub {
> >  		'cat /etc/apt/sources.list',
> >  		sub { dir2text('/etc/apt/sources.list.d/', '.+\.list') },
> >  		sub { dir2text('/etc/apt/sources.list.d/', '.+\.sources') },
> > -		'lscpu',

Because of the commands from [PATCH manager 9/9]:
+  'apt list *microcode 2>/dev/null | column -tL',
+  'dmesg | grep -i "microcode\|vuln"',
which are associated to the lscpu output. But due to your suggestions in
[PATCH manager 9/9], like:

> I'm wondering if instead of having a handful of dmesg + grep instances
> it makes more sense to just add the whole dmesg output as separate
> file.

both patches can be dropped.

> > -		'pvesh get /cluster/resources --type node --output-format=yaml',

This move just felt more appropriate since we have a dedicated cluster section.
Beside that we often jump to or filter for keywords, so the actual position might
not be that important. I'm fine with dropping this change and stick with the
position we are used to.

> >  	    ],
> >  	},
> >  	'system-load' => {
> > @@ -96,6 +94,7 @@ my $init_report_cmds = sub {
> >  	    order => 60,
> >  	    cmds => [
> >  		'pvecm nodes',
> > +		'pvesh get /cluster/resources --type node --output-format=yaml',
> >  		'pvecm status',
> >  		'cat /etc/pve/corosync.conf 2>/dev/null',
> >  		'ha-manager status',
> > @@ -106,6 +105,7 @@ my $init_report_cmds = sub {
> >  	    order => 70,
> >  	    cmds => [
> >  		'dmidecode -t bios',
> > +		'lscpu',
> >  		'lspci -nnk',
> >  	    ],
> >  	},
> 

Re: [pve-devel] [PATCH manager 6/9] report: switch `dmidecode` to quiet to omit almost never needed info

[Alexander Zeidler]

On Mon, 2024-03-25 at 09:20 +0100, Thomas Lamprecht wrote:
> On 22/03/2024 14:59, Alexander Zeidler wrote:
> > like on this system:
> > 
> >  # dmidecode -t bios
> >  # dmidecode 3.4
> >  Getting SMBIOS data from sysfs.
> >  SMBIOS 3.0.0 present.
> > 
> >  Handle 0x0000, DMI type 0, 24 bytes
> > 
> >  Handle 0x005C, DMI type 13, 22 bytes
> 
> 
> The manual page here states that this option also hides other entries:
> 
> > Unknown, inactive and OEM-specific entries are not displayed. Meta-data and handle references are hidden.
> 
> So especially on newer HW this might contain some actual info beyond
> the SMBIOS version/metadata that gets hidden.
> 
> I mean, I have no hard feelings here, but I'm wondering if it's really
> worth hiding that info if it can include potential relevant stuff.

Sounds reasonable, thank you! Will drop this patch.

> 
> > 
> > Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> > ---
> >  PVE/Report.pm | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/PVE/Report.pm b/PVE/Report.pm
> > index 505629c7..3a81bdb2 100644
> > --- a/PVE/Report.pm
> > +++ b/PVE/Report.pm
> > @@ -104,7 +104,7 @@ my $init_report_cmds = sub {
> >  	hardware => {
> >  	    order => 70,
> >  	    cmds => [
> > -		'dmidecode -t bios',
> > +		'dmidecode -t bios -q',
> >  		'lscpu',
> >  		'lspci -nnk',
> >  	    ],
> 

Re: [pve-devel] [PATCH manager 7/9] report: add info of the mainboard in use

[Alexander Zeidler]

On Fri, 2024-03-22 at 15:26 +0100, Aaron Lauterer wrote:
> If we add more info, then dmidecode -t {1,2,3} might be interesting as 
> well as those deliver motherboard, system and chassis infos.
> 
> If you want to see what you get there, run them on some decent servers 
> as most consumer boards and systems will not show useful information here

Thanks! I've now looked at the output from different servers/motherboards
from the past years (ASUS, Dell, GIGABYTE, HP, 2x Supermicro).


In the system and base board output there was (only) the following
relevant information:

System Information
	Manufacturer: Thomas-Krenn.AG
	Product Name: RS700-E11-RS12U 1HE Intel Dual-CPU RI2112-ASXSN Server CHECKLISTE
	Version: 00
Base Board Information
	Manufacturer: ASUSTeK COMPUTER INC.
	Product Name: Z13PP-D32 Series
	Version: 60SB09M0-SB0G11

But to avoid blindness, and since the unfiltered output is not that long,
we could include it as a whole.


The chassis output contains several potentially interesting lines. So we
could include that as a whole too:

Handle 0x0300, DMI type 3, 21 bytes
Chassis Information
->	Manufacturer: HP
->	Type: Rack Mount Chassis
	Lock: Not Present
	Version: Not Specified
	Serial Number: (...)
	Asset Tag:
->	Boot-up State: Critical
->	Power Supply State: Critical
->	Thermal State: Safe
	Security Status: Unknown
	OEM Information: 0x00000000
	Height: 2 U
->	Number Of Power Cords: 2
	Contained Elements: 0

> 
> On  2024-03-22  14:59, Alexander Zeidler wrote:
> > Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> > ---
> >   PVE/Report.pm | 1 +
> >   1 file changed, 1 insertion(+)
> > 
> > diff --git a/PVE/Report.pm b/PVE/Report.pm
> > index 3a81bdb2..f28d7b38 100644
> > --- a/PVE/Report.pm
> > +++ b/PVE/Report.pm
> > @@ -104,6 +104,7 @@ my $init_report_cmds = sub {
> >   	hardware => {
> >   	    order => 70,
> >   	    cmds => [
> > +		'cd /sys/devices/virtual/dmi/id; cat board_vendor board_name',
> >   		'dmidecode -t bios -q',
> >   		'lscpu',
> >   		'lspci -nnk',

Re: [pve-devel] [PATCH manager 9/9] report: add microcode info to better assess possible system impacts

[Alexander Zeidler]

On Mon, 2024-03-25 at 10:00 +0100, Thomas Lamprecht wrote:
> On 22/03/2024 14:59, Alexander Zeidler wrote:
> > * list availability and installation status of `*microcode` packages
> > * grep for applied "Early OS Microcode Updates"
> > * grep for (un)patched CPU vulnerability messages
> > 
> > Signed-off-by: Alexander Zeidler <a.zeidler@proxmox.com>
> > ---
> >  PVE/Report.pm | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/PVE/Report.pm b/PVE/Report.pm
> > index fe497b43..18c554ec 100644
> > --- a/PVE/Report.pm
> > +++ b/PVE/Report.pm
> > @@ -108,6 +108,8 @@ my $init_report_cmds = sub {
> >  		'dmidecode -t bios -q',
> >  		'dmidecode -t memory | grep -E "Capacity|Devices|Size|Manu|Part" | sed -Ez "s/\n\t(M|P)[^:]*: (\S*)/\t\2/g" | sort',
> >  		'lscpu',
> > +		'apt list *microcode 2>/dev/null | column -tL',
> > +		'dmesg | grep -i "microcode\|vuln"',
> 
> I'm wondering if instead of having a handful of dmesg + grep instances
> it makes more sense to just add the whole dmesg output as separate
> file.
> 
> I.e., I would like to have a cluster-wide report collection API that
> spawns a task, calls to all nodes to generate a report, saves all of
> those reports, including commands or files with very long output as
> separate files, and then assembles an archive with all that.

Great idea. The implementation of this seems not to be that trivial with
our current code base, so I plan to work on this in the near future.

And for a perhaps currently installed microcode package version, this
can be well included into the pveversion -v output. Whether it has been
applied at the current boot, will only later be visible in the dmesg file.

So this patch can be dropped.

> 
> On the long run that would provide nicer UX and also avoid that some
> to strict filter hides information that might be relevant for a
> specific setup.
> 
> I'd much more prefer work time spent on something like that than on
> adding the same command a few times with each having some different,
> rather a bit brittle looking, pipe chains..
> 
> >  		'lspci -nnk',
> >  	    ],
> >  	},
> 

Re: [pve-devel] [PATCH manager 1/9] report: add kernel command line including boot time

[Thomas Lamprecht]

On 11/04/2024 19:07, Alexander Zeidler wrote:
> On Mon, 2024-03-25 at 08:52 +0100, Thomas Lamprecht wrote:
>> journalctl --list-boots
> 
> On slow servers / storage drives this execution can easily take 10 seconds
> or longer. But there is an alternative that is quite fast and even shows
> the booted kernel version:
> 
> # last reboot -F -n10

ack, `last` is using /var/log/wtmp as source which is maintained by login, and/or
init and/or getty, i.e., not through any traditional log mechanism like the now
optional rsyslog, so this would be fine by me.

Re: [pve-devel] [PATCH manager 1/9] report: add kernel command line including boot time

[Alexander Zeidler]

v2 of this series:

[PATCH manager 1/7] report: add kernel command line from current boot
https://lists.proxmox.com/pipermail/pve-devel/2024-April/063282.html


_______________________________________________
pve-devel mailing list
pve-devel@lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel