Hi , Sorry about re posting this as i did not find solution ... I m facing a issue where on #crash /data/linux-2.6.30.8/vmlinux /proc/vmcore

crash 6.1.6 Copyright (C) 2002-2013 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.3.1 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"... crash: read error: kernel virtual address: c127a9c8 type: "cpu_possible_mask"

I did follow few threads around , I have a case similar to https://www.redhat.com/archives/crash-utility/2010-August/msg00029.html Both the System Kernel and the Debug Kernel are the same.

My current config for CONFIG_PHYSICAL_START=0x1000000 CONFIG_RELOCATABLE=y CONFIG_PHYSICAL_ALIGN=0x400000 The /proc/kallsyms is diffrent between the System Kernel and Debug Kernel. On the System Kernel it starts from 0x400000 .On the Debug kernel it starts from 0x1000000.

if i change the CONFIG_PHYSICAL_START=0x400000 . It does NOT core dump. my crashkernel=128M is the setting on grub. Let me know if you need more info

> Hi , > Sorry about re posting this as i did not find solution ... > > I m facing a issue where on > > #crash /data/linux-2.6.30.8/vmlinux /proc/vmcore Do yourself a favor and copy /proc/vmcore to somewhere on disk. Then reboot the system into the original kernel, and run crash on the saved vmcore. I've never seen crash run on a /proc/vmcore file from the secondary kernel. Also, after rebooting the the original kernel, please confirm that crash runs OK on the live system. > crash 6.1.6 > Copyright (C) 2002-2013 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > Copyright (C) 1999-2006 Hewlett-Packard Co > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > Copyright (C) 2005, 2011 NEC Corporation > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public License, > and you are welcome to change it and/or distribute copies of it under > certain conditions. Enter "help copying" to see the conditions. > This program has absolutely no warranty. Enter "help warranty" for details. > > GNU gdb (GDB) 7.3.1 > Copyright (C) 2011 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "i686-pc-linux-gnu"... > > crash: read error: kernel virtual address: c127a9c8 type: "cpu_possible_mask" Please show the full output of "crash -d8 vmlinux vmcore".

> > I did follow few threads around , I have a case similar to > https://www.redhat.com/archives/crash-utility/2010-August/msg00029.html > > Both the System Kernel and the Debug Kernel are the same. I don't know what you mean by "System Kernel" and "Debug Kernel". With respect to kdump and the crash utility, the only kernel that is of interest is the vmlinux that was running when the primary system crashed. > > My current config for > > CONFIG_PHYSICAL_START=0x1000000 > CONFIG_RELOCATABLE=y > CONFIG_PHYSICAL_ALIGN=0x400000 > > The /proc/kallsyms is diffrent between the System Kernel and Debug Kernel. > On the System Kernel it starts from 0x400000 .On the Debug kernel it > starts from 0x1000000. Again, the "Debug" kernel is irrelevant -- presuming by "Debug" you mean the relocated "secondary" kernel that was kexec'd after the primary kernel crashed.

> > if i change the CONFIG_PHYSICAL_START=0x400000 . It does NOT core dump. > > my crashkernel=128M is the setting on grub. > Let me know if you need more info > What "core dumps", the crash utility?

Again, trying to run the crash utility while running on a 128MB secondary kernel is definitely not advisable.

Dave

----- Original Message ----- > On Wed, May 8, 2013 at 11:23 AM, Anand Raj Manickam <anandrm(a)gmail.com&gt; > wrote: > >> Hi , > >> Sorry about re posting this as i did not find solution ... > >> > >> I m facing a issue where on > >> > >> #crash /data/linux-2.6.30.8/vmlinux /proc/vmcore > > > > Do yourself a favor and copy /proc/vmcore to somewhere on disk. Then > > reboot the system into the original kernel, and run crash on the saved > > vmcore. I've never seen crash run on a /proc/vmcore file from the > > secondary kernel. > > > > Also, after rebooting the the original kernel, please confirm that > > crash runs OK on the live system. > > > > > >> crash 6.1.6 > >> Copyright (C) 2002-2013 Red Hat, Inc. > >> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > >> Copyright (C) 1999-2006 Hewlett-Packard Co > >> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > >> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > >> Copyright (C) 2005, 2011 NEC Corporation > >> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > >> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > >> This program is free software, covered by the GNU General Public License, > >> and you are welcome to change it and/or distribute copies of it under > >> certain conditions. Enter "help copying" to see the conditions. > >> This program has absolutely no warranty. Enter "help warranty" for > >> details. > >> > >> GNU gdb (GDB) 7.3.1 > >> Copyright (C) 2011 Free Software Foundation, Inc. > >> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > >> This is free software: you are free to change and redistribute it. > >> There is NO WARRANTY, to the extent permitted by law. Type "show copying" > >> and "show warranty" for details. > >> This GDB was configured as "i686-pc-linux-gnu"... > >> > >> crash: read error: kernel virtual address: c127a9c8 type: "cpu_possible_mask" > > > > Please show the full output of "crash -d8 vmlinux vmcore". > > After rebooting to the primary / live kernel > > I have attached the output for "crash -d8 vmlinux /data/vmcore" in d8.txt Looking at the ELF header data in the debug output, your configuration does require special handling. X86 kernels are typically not configured with CONFIG_PHYSICAL_START larger than CONFIG_PHYSICAL_ALIGN, because when that is done, the vmlinux unity-mapped symbol values do not match their relocated values when the kernel is loaded. This entry in the crash.changelog describes the situation: http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5 4.0-4.5 - Addresses FC7/upstream x86 kernels that have been configured such that the vmlinux symbol values do not match their relocated values when loaded. If CONFIG_PHYSICAL_START contains a value that is greater then CONFIG_PHYSICAL_ALIGN, then this mismatch occurs. Since the crash utility and its embedded gdb have always expected that the compiled-in kernel symbol addresses are "real", the virtual to physical translation fails, leading to an initialization-time failure with the message: "crash: vmlinux and /dev/crash do not match!" (/dev/mem or the dumpfile name may replace /dev/crash). To deal with this issue, there are several alternatives: 1) Configure the kernel with CONFIG_PHYSICAL_START less than or equal to CONFIG_PHYSICAL_ALIGN. Having done that, there is no problem; the resultant vmlinux file will be loaded at the address for which it was compiled, which has always been the case. 2) Since /proc/kallsyms uses the same format as a System.map file, and since it reflects the relocated symbol addresses, it can be placed on the crash command line as if it were a System.map file. (Note that the System.map file created by these relocated kernels contains the same "wrong" symbol values as the vmlinux file from which it was created.) 3) On a live system that has /proc/kallsyms (i.e., the kernel was configured with CONFIG_KALLSYMS), this version of the crash utility will replace/patch the vmlinux symbol values with those seen in /proc/kallsyms. The relocation value will be displayed as a WARNING message during initialization. 4) On a dumpfile, the relocation will not be performed automatically as on a live system. It will require the addition of the /proc/kallsyms on the command line, or if run on a different host, a copy of the crashed system's /proc/kallsyms may be used. 5) Alternatively on a dumpfile, a new command line option has been created to specify the relocation amount. For example, if a kernel was configured with a CONFIG_PHYSICAL_START value of 16MB and a CONFIG_PHYSICAL_ALIGN of 4MB, that results in a relocation of 12MB. To specify that, enter "crash --reloc=12m ..." on the command line. (Recall that if crash is run on the live system, a WARNING message will specify the relocation amount.) Using /proc/kallsyms or a --reloc=[size] as a command line argument is similar to using a System.map file, in that it results in the loss of the use of line number debug data. (anderson(a)redhat.com) ... >> My current config for >> >> CONFIG_PHYSICAL_START=0x1000000 >> CONFIG_RELOCATABLE=y >> CONFIG_PHYSICAL_ALIGN=0x400000 >> >> The /proc/kallsyms is diffrent between the System Kernel and Debug Kernel. >> On the System Kernel it starts from 0x400000 .On the Debug kernel it >> starts from 0x1000000. So your configuration exactly matches paragraph 5) above, you should be able to come up OK by entering either:

$ crash vmlinux vmcore --reloc=12m

or $ crash vmlinux vmcore /proc/kallsyms where the /proc/kallsyms is from the *primary* kernel, so option #2 will *not* work when running from the secondary kernel. You didn't show any output of a live crash session (while running on the primary kernel), but I believe it should work without using the /proc/kallsyms as an argument or using --reloc=12m option. There was a further fix for these configurations for running on live systems: http://people.redhat.com/anderson/crash.changelog.html#5_1_4 5.1.4 - Fix for RT kernels in which the schedule() function has become a ... [ cut] ... - Fix for running against live x86 kernels that were configured with CONFIG_PHYSICAL_START containing a value that is greater than its CONFIG_PHYSICAL_ALIGN value, and where the first symbol listed by /proc/kallsyms is not "_text". Without the patch, the crash session fails during invocation with the error message "crash: vmlinux and /dev/mem do not match!" (or "/dev/crash" if applicable). As a work- around, "/proc/kallsyms" can be entered on the command line, or the "--reloc=<size>" option could be used, but the fix obviates that requirement for live systems. It should be noted that dumpfiles of kernels configured that way still do require that "/proc/kallsyms", or a copy of it, or alternatively the "--reloc=<size>" option, to be entered on the command line, as detailed in this changelog entry: http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5 (anderson(a)redhat.com) So on the live system on the primary kernel, I believe this should just work: $ crash vmlinux Normally you don't even need to put the vmlinux file on the command line for running on the live system because it will be "found" automatically unless you've put it in some non-standard location. Dave