From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from firstgate.proxmox.com (firstgate.proxmox.com [212.224.123.68]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.proxmox.com (Postfix) with ESMTPS id 558FD70C7C for ; Tue, 8 Jun 2021 22:43:43 +0200 (CEST) Received: from firstgate.proxmox.com (localhost [127.0.0.1]) by firstgate.proxmox.com (Proxmox) with ESMTP id 444901E5DD for ; Tue, 8 Jun 2021 22:43:13 +0200 (CEST) Received: from mail-lj1-x231.google.com (mail-lj1-x231.google.com [IPv6:2a00:1450:4864:20::231]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by firstgate.proxmox.com (Proxmox) with ESMTPS id 201C51E5D0 for ; Tue, 8 Jun 2021 22:43:09 +0200 (CEST) Received: by mail-lj1-x231.google.com with SMTP id u18so8604882lju.12 for ; Tue, 08 Jun 2021 13:43:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=MC3/Y/IBPHzJpqld205oX3O8pl30P08DZN3uSJoR7bA=; b=Z/kts3QgSSREZQaLU3s39KIKWXsfY6oqW7j+MOMYgpdCkviFA+VPVQFblrzjFEd3BZ ZfS7Yqn6NtelAKwVBRslhVZlniSF+yobN4KHme7fRCoWAO+5Pfv5x3MSqCVxfpq9vXuU zaRUtkuX1jSfSBAXAGhDGMhEctfrUAagJpzKfeILcHi43+hZnuCNMJQthJmE2X0Up+r/ rfYy5mIL1nBVG4/96oYVohPu6Z9IfRNaTn0BkLcv4+Hz3X61SEVZTWYG8rN6FpvELsZv y6gCf8xQUC2QZejjaz3KtMs5/BJMLBxHR2c7N0a19lbsMT4smxLPfsisogmcP1D7EXqT AacQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=MC3/Y/IBPHzJpqld205oX3O8pl30P08DZN3uSJoR7bA=; b=E5wIqEA9r2WBi8ffgnkXFKjfGsFnoD+qpBVdErSCYDygLNjEmTcO9qkf2ZqCb8Ve/c 2LM8wuDLdMnI+g1cC5IsJ3bvyMVrWlTxJdWC29Ux1Z7KjWRXnwmMrOU4MFS/lRo6liqv /EEEsqf9ZzH5MNQ+B8EAtKocFY3d4OlVe3d3hG9/ACUdVxcbxGElXiB27P4IUN09dfsL Bb0ufA38lFt0zJQXtqL/FbUWxKCvylvno4zNChkSTCMFUMXPnyIbozEqvbZNa67xPV+S qcl+d5BCOMRWFWZ82wgNp/XCsjzXP+PRxRINyAQzOYwPrHa2RHSOH6E5mhHBwBSopyF8 NIjg== X-Gm-Message-State: AOAM5310YvJZd/zUoZj1toVmWHZmPPRH0/fnAsDH8MfXa6kaI7Sb7ycj 08iIFG98tXQ8q0iP/cYe9DDUfS0s2mCv3ZiDqZez3dciSN8= X-Google-Smtp-Source: ABdhPJzlzQ9X59Y67gAyU0mT4uH+XKHIqAabdqh9gVDmbPa0aJqTxGnSEYS6A6OUBgu6LBGLuwjdD7zu2jCYe+T2pTs= X-Received: by 2002:a2e:a7c9:: with SMTP id x9mr19874019ljp.216.1623184981924; Tue, 08 Jun 2021 13:43:01 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Nathan L Date: Tue, 8 Jun 2021 14:42:49 -0600 Message-ID: To: pve-devel@lists.proxmox.com Content-Type: text/plain; charset="UTF-8" X-SPAM-LEVEL: Spam detection results: 0 AWL 0.950 Adjusted score from AWL reputation of From: address BAYES_00 -1.9 Bayes spam probability is 0 to 1% DKIM_SIGNED 0.1 Message has a DKIM or DK signature, not necessarily valid DKIM_VALID -0.1 Message has at least one valid DKIM or DK signature DKIM_VALID_AU -0.1 Message has a valid DKIM or DK signature from author's domain DKIM_VALID_EF -0.1 Message has a valid DKIM or DK signature from envelope-from domain FREEMAIL_ENVFROM_END_DIGIT 0.25 Envelope-from freemail username ends in digit FREEMAIL_FROM 0.001 Sender email is commonly abused enduser mail provider KAM_SHORT 0.001 Use of a URL Shortener for very short URL RCVD_IN_DNSWL_NONE -0.0001 Sender listed at https://www.dnswl.org/, no trust SPF_HELO_NONE 0.001 SPF: HELO does not publish an SPF Record SPF_PASS -0.001 SPF: sender matches SPF record URIBL_BLOCKED 0.001 ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [gnu.org, proxmox.com] Subject: Re: [pve-devel] How to compile pve-kernel with debug info? X-BeenThere: pve-devel@lists.proxmox.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Proxmox VE development discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jun 2021 20:43:43 -0000 Following up to answer my own questions and to ask a follow up question. On Fri, Jun 4, 2021 at 8:18 PM Nathan L wrote: > What am I missing here? 1. The correct git commit to use if you want to match the published package is specified in /usr/share/doc/insert-package-name/SOURCE. For example: # cat /usr/share/doc/pve-kernel-5.4.114-1-pve/SOURCE git clone git://git.proxmox.com/git/pve-kernel.git git checkout cd81f880a481ec8e3366cd493bb9d42821ebf967 2. As best I can tell, by using the git clone and git checkout commands included in the SOURCE file with the published package, and then running the make command, in theory you should get the same output as that used for the published package. I say "in theory" because there is some info like the hostname embedded in the vmlinuz file that will cause checksums not to match between the published version and your compiled version. I think there also may be some more substantial differences that are causing the crash tool errors "WARNING: kernel version inconsistency between vmlinux and dumpfile". This leads to my follow up question at the end. 3. The above means that, in theory, all you need to do to get a vmlinux file to debug your current kernel version is git clone ...; git checkout ...; make; and then you should be able to use the vmlinux file located at build/ubuntu-focal/vmlinux. This doesn't work in practice though. 4. Point 3 seems to be true because if you compile the kernel as mentioned previously, actually install your compiled pve-kernel-5.4.114-1-pve_5.4.114-1_amd64.deb file, boot into that kernel, and run 'echo c > /proc/sysrq-trigger' to create a crash dump, that crash dump can be successfully opened in 'crash' using the vmlinux file compiled and located at build/ubuntu-focal/vmlinux. However, if you try to use the published pve-kernel .deb package and your compiled vmlinux file, it won't work. > Should I not use the compiled vmlinux file at ./build/ubuntu-focal/vmlinux? No. That is the correct vmlinux but for some reason it doesn't work with the published kernel package. > Is there a flag I need to change or add somewhere to properly compile a version of the kernel I can use with crash? No. The kernel is compiled into a vmlinux file with all debug info included by default. > Is there a difference between the vmlinux file in the build/ubuntu-focal directory vs what I could extract from the vmlinuz file in the build/debian directory? The latter had the debug info stripped when I extracted it. Yes. A big difference. The kernel is compiled into a vmlinux file with all debug info included by default. It's also 743MB. Which is why it's further processed, added to, stripped of lots of data, etc to become the final vmlinuz file that gets packaged. That vmlinuz file is 12MB. There is no way to extract the full vmlinux from vmlinuz because so much data, like the debug info, was stripped along the way. Okay, follow up question: Why doesn't my compiled vmlinux for pve-kernel-5.4.114-1-pve work for debugging crash dumps from a machine running the same kernel version but from the published package? (example below) I should also make another thread or file a bug report on this but the included makedumpfile version (1.6.5-1 I think) with Proxmox didn't work to capture crash dumps at all. I had to upgrade it to the 1.6.8-4 version from the debian testing repo. # cat /var/crash/202106081333/dmesg.202106081333 | grep "Linux version" [ 0.000000] Linux version 5.4.114-1-pve (build@proxmox) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021 17:13:05 +0200) () ###The published version of the kernel was installed and loaded when this crash dump occurred. # file /root/pve-kernel/build/ubuntu-focal/arch/x86/boot/bzImage /root/pve-kernel/build/ubuntu-focal/arch/x86/boot/bzImage: Linux kernel x86 boot executable bzImage, version 5.4.114-1-pve (root@hostname) #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021 17:13:05 +0200), RO-rootFS, swap_dev 0xB, Normal VGA ###The version I compiled is exactly the same. This bzImage is derived from vmlinux used in the below crash command. # crash ~/pve-kernel/build/ubuntu-focal/vmlinux /var/crash/202106081333/dump.202106081333 crash 7.2.5 Copyright (C) 2002-2019 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: kernel relocated [710MB]: patching 114251 gdb minimal_symbol values WARNING: kernel version inconsistency between vmlinux and dumpfile crash: seek error: kernel virtual address: ffff99711f606024 type: "tss_struct ist array" ##Crash doesn't like this. Best regards, Nathan