[pve-devel] How to compile pve-kernel with debug info?

[pve-devel] How to compile pve-kernel with debug info?

[Nathan L]

Hello,

I'm trying to open a kdump crash dump using the 'crash' tool but I think I
need a version of vmlinux with debug info included in order to do this. It
sounds like this isn't published so I believe I need to recompile the same
kernel version to get a version that includes the debug info.

I'm currently running kernel 5.4.114-1-pve so I've cloned the pve-kernel
git repo and checked out the
commit ee60bb7ce033f8d44e29ac1f1ec50dee836c6742 for "update ABI file for
5.4.114-1-pve" under the pve-kernel-5.4 branch. With this, I'm able to run
'make' and successfully recompile the kernel and produce
'pve-kernel-5.4.114-1-pve_5.4.114-1_amd64.deb'.

root@hostname:~/pve-kernel# file
build/debian/pve-kernel-5.4.114-1-pve/boot/vmlinuz-5.4.114-1-pve
build/debian/pve-kernel-5.4.114-1-pve/boot/vmlinuz-5.4.114-1-pve: Linux
kernel x86 boot executable bzImage, version 5.4.114-1-pve (root@hostname)
#1 SMP PVE 5.4.114-1 (Sun, 09 May 2021 17:13:05 +0200), RO-rootFS, swap_dev
0xB, Normal VGA


root@hostname:~/pve-kernel# uname -a
Linux hostname 5.4.114-1-pve #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021
17:13:05 +0200) x86_64 GNU/Linux


My issue is, when I attempt to run crash using the compiled vmlinux file, I
receive the following:

root@hostname:~/pve-kernel# crash ~/pve-kernel/build/ubuntu-focal/vmlinux
/var/crash/202106021922/dump.202106021922

crash 7.2.5
Copyright (C) 2002-2019  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [840MB]: patching 114251 gdb minimal_symbol values

WARNING: kernel version inconsistency between vmlinux and dumpfile

crash: seek error: kernel virtual address: ffff905a5f606024  type:
"tss_struct ist array"

Lots of questions that probably all have the same answer:

   - What am I missing here?
   - Should I not use the compiled vmlinux file at
   ./build/ubuntu-focal/vmlinux?
   - Is there a flag I need to change or add somewhere to properly compile
   a version of the kernel I can use with crash?
   - Is there a difference between the vmlinux file in the
   build/ubuntu-focal directory vs what I could extract from the vmlinuz file
   in the build/debian directory? The latter had the debug info stripped when
   I extracted it.

Thanks in advance,

Nathan

Re: [pve-devel] How to compile pve-kernel with debug info?

[Nathan L]

Following up to answer my own questions and to ask a follow up question.

On Fri, Jun 4, 2021 at 8:18 PM Nathan L <nathanlangford.4@gmail.com> wrote:

> What am I missing here?
1. The correct git commit to use if you want to match the published
package is specified in /usr/share/doc/insert-package-name/SOURCE. For
example:
# cat /usr/share/doc/pve-kernel-5.4.114-1-pve/SOURCE
git clone git://git.proxmox.com/git/pve-kernel.git
git checkout cd81f880a481ec8e3366cd493bb9d42821ebf967

2. As best I can tell, by using the git clone and git checkout
commands included in the SOURCE file with the published package, and
then running the make command, in theory you should get the same
output as that used for the published package. I say "in theory"
because there is some info like the hostname embedded in the vmlinuz
file that will cause checksums not to match between the published
version and your compiled version. I think there also may be some more
substantial differences that are causing the crash tool errors
"WARNING: kernel version inconsistency between vmlinux and dumpfile".
This leads to my follow up question at the end.

3. The above means that, in theory, all you need to do to get a
vmlinux file to debug your current kernel version is git clone ...;
git checkout ...; make; and then you should be able to use the vmlinux
file located at build/ubuntu-focal/vmlinux. This doesn't work in
practice though.

4. Point 3 seems to be true because if you compile the kernel as
mentioned previously, actually install your compiled
pve-kernel-5.4.114-1-pve_5.4.114-1_amd64.deb file, boot into that
kernel, and run 'echo c > /proc/sysrq-trigger' to create a crash dump,
that crash dump can be successfully opened in 'crash' using the
vmlinux file compiled and located at build/ubuntu-focal/vmlinux.
However, if you try to use the published pve-kernel .deb package and
your compiled vmlinux file, it won't work.

> Should I not use the compiled vmlinux file at ./build/ubuntu-focal/vmlinux?
No. That is the correct vmlinux but for some reason it doesn't work
with the published kernel package.

> Is there a flag I need to change or add somewhere to properly compile a version of the kernel I can use with crash?
No. The kernel is compiled into a vmlinux file with all debug info
included by default.

> Is there a difference between the vmlinux file in the build/ubuntu-focal directory vs what I could extract from the vmlinuz file in the build/debian directory? The latter had the debug info stripped when I extracted it.
Yes. A big difference. The kernel is compiled into a vmlinux file with
all debug info included by default. It's also 743MB. Which is why it's
further processed, added to, stripped of lots of data, etc to become
the final vmlinuz file that gets packaged. That vmlinuz file is 12MB.
There is no way to extract the full vmlinux from vmlinuz because so
much data, like the debug info, was stripped along the way.

Okay, follow up question:
Why doesn't my compiled vmlinux for pve-kernel-5.4.114-1-pve work for
debugging crash dumps from a machine running the same kernel version
but from the published package? (example below)

I should also make another thread or file a bug report on this but the
included makedumpfile version (1.6.5-1 I think) with Proxmox didn't
work to capture crash dumps at all. I had to upgrade it to the 1.6.8-4
version from the debian testing repo.



# cat /var/crash/202106081333/dmesg.202106081333 | grep "Linux version"
[    0.000000] Linux version 5.4.114-1-pve (build@proxmox) (gcc
version 8.3.0 (Debian 8.3.0-6)) #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021
17:13:05 +0200) ()
###The published version of the kernel was installed and loaded when
this crash dump occurred.

# file /root/pve-kernel/build/ubuntu-focal/arch/x86/boot/bzImage
/root/pve-kernel/build/ubuntu-focal/arch/x86/boot/bzImage: Linux
kernel x86 boot executable bzImage, version 5.4.114-1-pve
(root@hostname) #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021 17:13:05
+0200), RO-rootFS, swap_dev 0xB, Normal VGA
###The version I compiled is exactly the same. This bzImage is derived
from vmlinux used in the below crash command.

# crash ~/pve-kernel/build/ubuntu-focal/vmlinux
/var/crash/202106081333/dump.202106081333

crash 7.2.5
Copyright (C) 2002-2019  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

WARNING: kernel relocated [710MB]: patching 114251 gdb minimal_symbol values

WARNING: kernel version inconsistency between vmlinux and dumpfile

crash: seek error: kernel virtual address: ffff99711f606024  type:
"tss_struct ist array"

##Crash doesn't like this.

Best regards,

Nathan