5:03 a.m.
Hi List,
I am new to crash / kdump tool and failing some problem as mentioned below.
I am referring the Linux Kernel Crash Book (http://www.dedoimedo.com/)
and URL http://www.dedoimedo.com/computers/crash.html.
I am building modified kernel source (2.6.32 based) and added my
modules (for study purpose). Building, installing and booting kernel
is successful. I have enabled the options for kdump as mentioned in
the book:
Enable Kexec system call:
CONFIG_KEXEC=y
Enable kernel crash dumps:
CONFIG_CRASH_DUMP=y
Optional: Disable Symmetric Multi-Processing (SMP) support
CONFIG_SMP=y
Enable sysfs file system support:
CONFIG_SYSFS=y
Enable /proc/vmcore support:
CONFIG_PROC_VMCORE=y
Configure the kernel with debug info:
CONFIG_DEBUG_INFO=y
Configure the start section for reserved RAM for the crash kernel:
CONFIG_PHYSICAL_START=0x200000 (2MB)
Configure kdump kernel so it can be identified:
CONFIG_LOCALVERSION="-crash"
Kdump configuration /etc/sysconfig/kdump:
KDUMP_KERNELVER=""
KDUMP_COMMANDLINE=""
KDUMP_COMMANDLINE_APPEND="maxcpus=1 "
KEXEC_OPTIONS=""
KDUMP_IMMEDIATE_REBOOT="yes"
KDUMP_TRANSFER=""
KDUMP_SAVEDIR="file:///var/crash"
KDUMP_KEEP_OLD_DUMPS="5"
KDUMP_FREE_DISK_SIZE="64"
KDUMP_VERBOSE="3"
KDUMP_DUMPLEVEL="0"
KDUMP_DUMPFORMAT="compressed"
There is no option KDUMP_DUMPDEV option
There is no option KDUMP_RUNLEVEL
I booted successfully with this kernel and tried to crash it by
module. After rebooting, I found that vmcore is generated under
/var/crash/ But I am not able to analyze it with crash command.
linux:/home/adil # cat /proc/cmdline
root=/dev/disk/by-id/ata-WDC_WD800BD-22LRA1_WD-WMAM9ZS19445-part1
resume=/dev/disk/by-id/ata-WDC_WD800BD-22LRA1_WD-WMAM9ZS19445-part2
splash=silent crashkernel=256M-:128M vga=0x31a
linux:/home/adil #
linux:/home/adil # crash /boot/System.map-2.6.32.12-crash-crash
/boot/vmlinuz-2.6.32.12-crash-crash
crash 5.0.1
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
crash: /boot/vmlinuz-2.6.32.12-crash-crash: not a supported file format
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
Enter "crash -h" for details.
linux:/home/adil #
The URL http://www.dedoimedo.com/computers/crash.html mentioned that
"The newer versions of Kdump can work with compressed kernel images.
Furthermore, they copy the System map file and the kernel image into
the crash directory, making the use of crash utility somewhat
simpler."
linux:/home/adil # ls -al /var/crash/2012-01-30-18\:08/
total 1335536
drwxr-xr-x 2 root root 4096 2012-01-30 18:13 .
drwxr-xr-x 8 root root 4096 2012-01-31 12:17 ..
-rw-r--r-- 1 root root 187 2012-01-30 18:13 README.txt
-rw-r--r-- 1 root root 1716605 2012-01-30 18:13
System.map-2.6.32.12-0.7-default
-rw------- 1 root root 1360732590 2012-01-30 18:13 vmcore
-rw-r--r-- 1 root root 3774506 2012-01-30 18:13
vmlinux-2.6.32.12-0.7-default.gz
linux:/home/adil # ls -al /var/crash/2012-01-31-12\:17/
total 1343860
drwxr-xr-x 2 root root 4096 2012-01-31 12:24 .
drwxr-xr-x 8 root root 4096 2012-01-31 12:17 ..
-rw-r--r-- 1 root root 187 2012-01-31 12:24 README.txt
-rw------- 1 root root 1374748735 2012-01-31 12:24 vmcore
linux:/home/adil #
linux:/home/adil # ls /boot/
backup_mbr
boot
boot.readme
config-2.6.32.12-0.7-default
config-2.6.32.12-0.7-xen
grub
initrd
initrd-2.6.32.12-0.7-default
initrd-2.6.32.12-0.7-default-kdump
initrd-2.6.32.12-0.7-xen
initrd-2.6.32.12-crash-crash
initrd-2.6.32.12-crash-crash-kdump
initrd-xen
message
symsets-2.6.32.12-0.7-default.tar.gz
symtypes-2.6.32.12-0.7-default.gz
symvers-2.6.32.12-0.7-default.gz
symvers-2.6.32.12-0.7-xen.gz
System.map-2.6.32.12-0.7-default
System.map-2.6.32.12-0.7-xen
System.map-2.6.32.12-crash-crash
vmlinux-2.6.32.12-0.7-xen.gz
vmlinuz
vmlinuz-2.6.32.12-0.7-default
vmlinuz-2.6.32.12-0.7-xen
vmlinuz-2.6.32.12-crash-crash
vmlinuz-xen
vmlinux-2.6.32.12-0.7-default.gz
linux:/home/adil #
Another observation is boot.kdump seems to on but manually start
giving me error:
linux:/home/adil # chkconfig boot.kdump
boot.kdump on
linux:/home/adil #
linux:/home/adil # /etc/init.d/boot.kdump start
Loading kdump
Regenerating kdump initrd ...
Can't find kernel text map area from kcore
Cannot load /boot/vmlinuz-2.6.32.12-crash-crash
failed
linux:/home/adil #
Other Query: Following is not clear mentioned in the book under
"section 11.2 Crash (capture) kernel":
--------------
This means that while your production kernels will most likely be
named vmlinuz, the Kdump crash kernels need to be uncompressed,
hence named vmlinux, or rather vmlinux-kdump.
---------------
Please help how to correctly setup and use crash on my machine.
Thank you,
Adil
8:58 a.m.
linux:/home/adil # crash /boot/System.map-2.6.32.12-crash-crash
/boot/vmlinuz-2.6.32.12-crash-crash
crash 5.0.1
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
[...]
This program has absolutely no warranty. Enter "help
warranty" for details.
crash: /boot/vmlinuz-2.6.32.12-crash-crash: not a supported file format
First and formeost you should pass the vmcore file.
Thanks
Suzuki
12:29 a.m.
Hi,
Well, that's your first mistake...
Thanks for pointing it out :)
..
[snip]
..
Like the error message indicates, the vmlinuz-2.6.32.12-crash-crash
file is not a supported file format. Never has been, never will be...
I tried with vmlinux and it seems ok. But got another error
hltncra110731:/home/adil # crash linux-2.6.32.12-0.7/vmlinux
/var/crash/2012-02-08-14\:13/vmcore
--------------------
crash 5.0.1
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
crash: seek error: kernel virtual address: ffffffff81827aa0 type:
"kernel_config_data"
WARNING: cannot read kernel_config_data
crash: seek error: kernel virtual address: ffffffff8181a660 type:
"cpu_possible_mask"
hltncra110731:/home/adil #
----------------------------------
Also, do yourself a favor, and upgrade your crash utility version
from crash-5.0.1, which is two years old.
I am using for learning purpose. So i guess this version should be Ok
to use for learning purpose.
And lastly, although you may get some responses, this is not the list
for
kdump-related questions. That is found here:
http://lists.infradead.org/mailman/listinfo/kexec
Thank. I will ask the list for my kexec/kdump queries.
Thanks,
Adil
Dave Anderson
8:46 a.m.
----- Original Message -----
Hi,
>
> Well, that's your first mistake...
>
Thanks for pointing it out :)
..
[snip]
..
> Like the error message indicates, the vmlinuz-2.6.32.12-crash-crash
> file is not a supported file format. Never has been, never will
> be...
>
I tried with vmlinux and it seems ok. But got another error
hltncra110731:/home/adil # crash linux-2.6.32.12-0.7/vmlinux
/var/crash/2012-02-08-14\:13/vmcore
--------------------
crash 5.0.1
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
crash: seek error: kernel virtual address: ffffffff81827aa0 type:
"kernel_config_data"
WARNING: cannot read kernel_config_data
crash: seek error: kernel virtual address: ffffffff8181a660 type:
"cpu_possible_mask"
hltncra110731:/home/adil #
----------------------------------
> Also, do yourself a favor, and upgrade your crash utility version
> from crash-5.0.1, which is two years old.
I am using for learning purpose. So i guess this version should be Ok
to use for learning purpose.
No, that's really not true. crash-5.0.1 is two years old, which is ancient
by crash standards. And it's very easy to upgrade:
$ wget http://people.redhat.com/anderson/crash-6.0.3.tar.gz
...
$ cd crash-6.0.3
$ make
...
$
As far as the "seek errors" you're seeing, I don't know what's
happening,
other than crash can't find the physical address page associated with those
unity-mapped kernel virtual addresses in the vmcore. But I'm sorry -- I don't
spend time debugging issues encountered with obsolete crash versions. If you
can upgrade, and then post the output of:
$ crash -d8 linux-2.6.32.12-0.7/vmlinux /var/crash/2012-02-08-14\:13/vmcore
there will be a plethora of debug output that can help determine the
problem.
Dave
10:06 a.m.
Hi Dave,
No, that's really not true. crash-5.0.1 is two years old, which
is ancient
by crash standards. And it's very easy to upgrade:
$ wget http://people.redhat.com/anderson/crash-6.0.3.tar.gz
...
$ cd crash-6.0.3
$ make
...
$
I tried but at the end build failed:
-----------------------
cc -c -g -DX86_64 -DGDB_7_3_1 -g -O2 main.c
In file included from main.c:18:
defs.h:43:18: error: zlib.h: No such file or directory
-----------------------
Any idea which package it needed to get the zlib.h
Thanks for your help.
Regards,
Adil
On Thu, Feb 9, 2012 at 8:16 PM, Dave Anderson <anderson(a)redhat.com> wrote:
>
>
> ----- Original Message -----
>> Hi,
>>
>> >
>> > Well, that's your first mistake...
>> >
>>
>> Thanks for pointing it out :)
>> ..
>> [snip]
>> ..
>> > Like the error message indicates, the vmlinuz-2.6.32.12-crash-crash
>> > file is not a supported file format. Never has been, never will
>> > be...
>> >
>>
>> I tried with vmlinux and it seems ok. But got another error
>> hltncra110731:/home/adil # crash linux-2.6.32.12-0.7/vmlinux
>> /var/crash/2012-02-08-14\:13/vmcore
>>
>> --------------------
>> crash 5.0.1
>> Copyright (C) 2002-2010 Red Hat, Inc.
>> Copyright (C) 2004, 2005, 2006 IBM Corporation
>> Copyright (C) 1999-2006 Hewlett-Packard Co
>> Copyright (C) 2005, 2006 Fujitsu Limited
>> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
>> Copyright (C) 2005 NEC Corporation
>> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
>> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
>> This program is free software, covered by the GNU General Public License,
>> and you are welcome to change it and/or distribute copies of it under
>> certain conditions. Enter "help copying" to see the conditions.
>> This program has absolutely no warranty. Enter "help warranty" for
>> details.
>>
>> GNU gdb (GDB) 7.0
>> Copyright (C) 2009 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-unknown-linux-gnu"...
>>
>> crash: seek error: kernel virtual address: ffffffff81827aa0 type:
"kernel_config_data"
>> WARNING: cannot read kernel_config_data
>> crash: seek error: kernel virtual address: ffffffff8181a660 type:
"cpu_possible_mask"
>> hltncra110731:/home/adil #
>> ----------------------------------
>>
>>
>> > Also, do yourself a favor, and upgrade your crash utility version
>> > from crash-5.0.1, which is two years old.
>>
>> I am using for learning purpose. So i guess this version should be Ok
>> to use for learning purpose.
>
No, that's really not true. crash-5.0.1 is two years old, which
is ancient
by crash standards. And it's very easy to upgrade:
$ wget http://people.redhat.com/anderson/crash-6.0.3.tar.gz
...
$ cd crash-6.0.3
$ make
...
$
>
> As far as the "seek errors" you're seeing, I don't know what's
happening,
> other than crash can't find the physical address page associated with those
> unity-mapped kernel virtual addresses in the vmcore. But I'm sorry -- I
don't
> spend time debugging issues encountered with obsolete crash versions. If you
> can upgrade, and then post the output of:
>
> $ crash -d8 linux-2.6.32.12-0.7/vmlinux /var/crash/2012-02-08-14\:13/vmcore
>
> there will be a plethora of debug output that can help determine the
> problem.
>
> Dave
10:12 a.m.
----- Original Message -----
Hi Dave,
> No, that's really not true. crash-5.0.1 is two years old, which is
> ancient
> by crash standards. And it's very easy to upgrade:
>
> $ wget http://people.redhat.com/anderson/crash-6.0.3.tar.gz
> ...
> $ cd crash-6.0.3
> $ make
> ...
> $
I tried but at the end build failed:
-----------------------
cc -c -g -DX86_64 -DGDB_7_3_1 -g -O2 main.c
In file included from main.c:18:
defs.h:43:18: error: zlib.h: No such file or directory
-----------------------
Any idea which package it needed to get the zlib.h
Thanks for your help.
Regards,
Adil
Right -- sorry about that....
If you had downloaded crash-6.0.3-0.src.rpm instead of the tar.gz
file, it would have prevented the build because the crash.spec file
requires these two packages:
BuildRequires: ncurses-devel zlib-devel
Anyway, those two packages are required.
Dave
12:28 a.m.
Thanks Dave.
If you had downloaded crash-6.0.3-0.src.rpm instead of the tar.gz
file, it would have prevented the build because the crash.spec file
requires these two packages:
BuildRequires: ncurses-devel zlib-devel
Anyway, those two packages are required.
I installed zlib-devel and build was successful.
If you can upgrade, and then post the output of:
$ crash -d8 linux-2.6.32.12-0.7/vmlinux /var/crash/2012-02-08-14\:13/vmcore
there will be a plethora of debug output that can help determine the problem.
Please find the output below:
---------------------------
hltncra110731:/home/adil/crash-6.0.3 # ./crash -d8
../linux-2.6.32.12-0.7/vmlinux /var/crash/2012-02-08-14\:13/vmcore
crash 6.0.3
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
compressed kdump: header->utsname.machine:
compressed kdump: memory bitmap offset: 2000
diskdump_data:
filename: /var/crash/2012-02-08-14:13/vmcore
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
dfd: 3
ofp: 0
machine_type: 62 (EM_X86_64)
header: dedfe0
signature: "KDUMP "
header_version: 1
utsname:
sysname:
nodename:
release:
version:
machine:
domainname:
timestamp:
tv_sec: 0
tv_usec: 0
status: 0 ()
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 76
max_mapnr: 1245184
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 1
tasks[nr_cpus]: 0
sub_header: 0 (n/a)
sub_header_kdump: deeff0
phys_base: 0
dump_level: 0 (0x0)
data_offset: 4e000
block_size: 4096
block_shift: 12
bitmap: 7f99c0b4e010
bitmap_len: 311296
dumpable_bitmap: 7f99c0b01010
byte: 0
bit: 0
compressed_page: e009a0
curbufptr: 0
page_cache_hdr[0]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df0990
pg_hit_count: 0
page_cache_hdr[1]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df1990
pg_hit_count: 0
page_cache_hdr[2]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df2990
pg_hit_count: 0
page_cache_hdr[3]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df3990
pg_hit_count: 0
page_cache_hdr[4]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df4990
pg_hit_count: 0
page_cache_hdr[5]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df5990
pg_hit_count: 0
page_cache_hdr[6]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df6990
pg_hit_count: 0
page_cache_hdr[7]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df7990
pg_hit_count: 0
page_cache_hdr[8]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df8990
pg_hit_count: 0
page_cache_hdr[9]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df9990
pg_hit_count: 0
page_cache_hdr[10]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfa990
pg_hit_count: 0
page_cache_hdr[11]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfb990
pg_hit_count: 0
page_cache_hdr[12]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfc990
pg_hit_count: 0
page_cache_hdr[13]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfd990
pg_hit_count: 0
page_cache_hdr[14]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfe990
pg_hit_count: 0
page_cache_hdr[15]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dff990
pg_hit_count: 0
page_cache_buf: df0990
evict_index: 0
evictions: 0
accesses: 0
cached_reads: 0
valid_pages: df0000
readmem: read_diskdump()
crash: pv_init_ops exists: ARCH_PVOPS
compressed kdump: phys_base: 0
gdb ../linux-2.6.32.12-0.7/vmlinux
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
GETBUF(248 -> 0)
GETBUF(1500 -> 1)
FREEBUF(1)
FREEBUF(0)
<readmem: ffffffff82827aa0, KVADDR, "kernel_config_data", 32768, (ROE),
1a1bab0>
<read_diskdump: addr: ffffffff82827aa0 paddr: 2827aa0 cnt: 1376>
read_diskdump: SEEK_ERROR: paddr/pfn: 2827aa0/2827 !page_is_ram
crash: seek error: kernel virtual address: ffffffff82827aa0 type:
"kernel_config_data"
WARNING: cannot read kernel_config_data
GETBUF(248 -> 0)
FREEBUF(0)
GETBUF(512 -> 0)
<readmem: ffffffff8281a660, KVADDR, "cpu_possible_mask", 8, (FOE),
7fff1b538678>
<read_diskdump: addr: ffffffff8281a660 paddr: 281a660 cnt: 8>
read_diskdump: SEEK_ERROR: paddr/pfn: 281a660/281a !page_is_ram
crash: seek error: kernel virtual address: ffffffff8281a660 type:
"cpu_possible_mask"
hltncra110731:/home/adil/crash-6.0.3 #
---------------------------
BTW i tried to setup a virtual machine (using VirtualBox) on another
machine and on that Virtual machine following the same setup, crash
tool was working. But not on this :(
Thanks,
Adil
On Thu, Feb 9, 2012 at 9:42 PM, Dave Anderson <anderson(a)redhat.com> wrote:
>
>
> ----- Original Message -----
>> Hi Dave,
>>
>> > No, that's really not true. crash-5.0.1 is two years old, which is
>> > ancient
>> > by crash standards. And it's very easy to upgrade:
>> >
>> > $ wget http://people.redhat.com/anderson/crash-6.0.3.tar.gz
>> > ...
>> > $ cd crash-6.0.3
>> > $ make
>> > ...
>> > $
>>
>> I tried but at the end build failed:
>> -----------------------
>> cc -c -g -DX86_64 -DGDB_7_3_1 -g -O2 main.c
>> In file included from main.c:18:
>> defs.h:43:18: error: zlib.h: No such file or directory
>> -----------------------
>>
>> Any idea which package it needed to get the zlib.h
>>
>> Thanks for your help.
>>
>> Regards,
>> Adil
>
> Right -- sorry about that....
>
If you had downloaded crash-6.0.3-0.src.rpm instead of the tar.gz
file, it would have prevented the build because the crash.spec file
requires these two packages:
BuildRequires: ncurses-devel zlib-devel
Anyway, those two packages are required.
>
> Dave
9:55 a.m.
----- Original Message -----
Thanks Dave.
> If you had downloaded crash-6.0.3-0.src.rpm instead of the tar.gz
> file, it would have prevented the build because the crash.spec file
> requires these two packages:
>
> BuildRequires: ncurses-devel zlib-devel
>
> Anyway, those two packages are required.
I installed zlib-devel and build was successful.
> If you can upgrade, and then post the output of:
>
> $ crash -d8 linux-2.6.32.12-0.7/vmlinux
> /var/crash/2012-02-08-14\:13/vmcore
> there will be a plethora of debug output that can help determine
> the problem.
Please find the output below:
---------------------------
hltncra110731:/home/adil/crash-6.0.3 # ./crash -d8
../linux-2.6.32.12-0.7/vmlinux /var/crash/2012-02-08-14\:13/vmcore
crash 6.0.3
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
compressed kdump: header->utsname.machine:
compressed kdump: memory bitmap offset: 2000
diskdump_data:
filename: /var/crash/2012-02-08-14:13/vmcore
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
dfd: 3
ofp: 0
machine_type: 62 (EM_X86_64)
header: dedfe0
signature: "KDUMP "
header_version: 1
utsname:
sysname:
nodename:
release:
version:
machine:
domainname:
timestamp:
tv_sec: 0
tv_usec: 0
status: 0 ()
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 76
max_mapnr: 1245184
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 1
tasks[nr_cpus]: 0
sub_header: 0 (n/a)
sub_header_kdump: deeff0
phys_base: 0
dump_level: 0 (0x0)
data_offset: 4e000
block_size: 4096
block_shift: 12
bitmap: 7f99c0b4e010
bitmap_len: 311296
dumpable_bitmap: 7f99c0b01010
byte: 0
bit: 0
compressed_page: e009a0
curbufptr: 0
page_cache_hdr[0]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df0990
pg_hit_count: 0
page_cache_hdr[1]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df1990
pg_hit_count: 0
page_cache_hdr[2]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df2990
pg_hit_count: 0
page_cache_hdr[3]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df3990
pg_hit_count: 0
page_cache_hdr[4]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df4990
pg_hit_count: 0
page_cache_hdr[5]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df5990
pg_hit_count: 0
page_cache_hdr[6]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df6990
pg_hit_count: 0
page_cache_hdr[7]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df7990
pg_hit_count: 0
page_cache_hdr[8]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df8990
pg_hit_count: 0
page_cache_hdr[9]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df9990
pg_hit_count: 0
page_cache_hdr[10]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfa990
pg_hit_count: 0
page_cache_hdr[11]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfb990
pg_hit_count: 0
page_cache_hdr[12]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfc990
pg_hit_count: 0
page_cache_hdr[13]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfd990
pg_hit_count: 0
page_cache_hdr[14]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfe990
pg_hit_count: 0
page_cache_hdr[15]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dff990
pg_hit_count: 0
page_cache_buf: df0990
evict_index: 0
evictions: 0
accesses: 0
cached_reads: 0
valid_pages: df0000
readmem: read_diskdump()
crash: pv_init_ops exists: ARCH_PVOPS
compressed kdump: phys_base: 0
gdb ../linux-2.6.32.12-0.7/vmlinux
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
GETBUF(248 -> 0)
GETBUF(1500 -> 1)
FREEBUF(1)
FREEBUF(0)
<readmem: ffffffff82827aa0, KVADDR, "kernel_config_data", 32768,
(ROE), 1a1bab0>
<read_diskdump: addr: ffffffff82827aa0 paddr: 2827aa0 cnt: 1376>
read_diskdump: SEEK_ERROR: paddr/pfn: 2827aa0/2827 !page_is_ram
crash: seek error: kernel virtual address: ffffffff82827aa0 type:
"kernel_config_data"
WARNING: cannot read kernel_config_data
GETBUF(248 -> 0)
FREEBUF(0)
GETBUF(512 -> 0)
<readmem: ffffffff8281a660, KVADDR, "cpu_possible_mask", 8, (FOE),
7fff1b538678>
<read_diskdump: addr: ffffffff8281a660 paddr: 281a660 cnt: 8>
read_diskdump: SEEK_ERROR: paddr/pfn: 281a660/281a !page_is_ram
crash: seek error: kernel virtual address: ffffffff8281a660 type:
"cpu_possible_mask"
hltncra110731:/home/adil/crash-6.0.3 #
The debug output looks reasonable, and it doesn't seem to
be a relocation issue because the dumpfile indicates this:
sub_header_kdump: deeff0
phys_base: 0
dump_level: 0 (0x0)
and
compressed kdump: phys_base: 0
So for the first two readmem() attempts, at ffffffff82827aa0
(kernel_config_data) and ffffffff8281a660 (cpu_possible_mask),
the kernel start map identifier of ffffffff80000000 is stripped,
leaving the physical addresses, which are then passed to the
compressed-kdump function read_diskdump(). But those physical
addresses cannot be found in the dumpfile.
I am presuming that the machine that generated the vmcore is still
running your ../linux-2.6.32.12-0.7/vmlinux kernel, and that you
can log onto it as root. If that's not the case, I can't help
much more.
If you are running that kernel on the crashed machine, for sanity's sake,
can you verify that your kernel has not relocated itself by doing this:
# nm -Bn ../linux-2.6.32.12-0.7/vmlinux | grep _stext
and comparing it to:
# cat /proc/kallsyms | grep _stext
The symbol values should be the same. For example, on this 2.6.32-based
system I see this:
# nm -Bn /usr/lib/debug/lib/modules/2.6.32-70.el6.x86_64/vmlinux | grep _stext
ffffffff81009000 T _stext
# cat /proc/kallsyms | grep _stext
ffffffff81009000 T _stext
#
Also do this:
# cat /proc/kallsyms | grep -e kernel_config_data -e cpu_possible_mask
and confirm that they are ffffffff82827aa0 and ffffffff82827aa0.
And lastly, in all cases where a dumpfile cannot be read correctly,
first verify that crash works with the live system. Presuming
t, and it was configured CONFIG_STRICT_DEVMEM, try this:
# crash ../linux-2.6.32.12-0.7/vmlinux
If that comes up OK, then we can rule out more fundamental
failure causes.
Dave
10:03 a.m.
----- Original Message -----
And lastly, in all cases where a dumpfile cannot be read correctly,
first verify that crash works with the live system. Presuming
t, and it was configured CONFIG_STRICT_DEVMEM, try this:
Sorry -- cut-and-paste error -- I meant to state "Presuming that
the kernel was *NOT* configured with CONFIG_STRICT_DEVMEM".
# crash ../linux-2.6.32.12-0.7/vmlinux
If that comes up OK, then we can rule out more fundamental
failure causes.
If you did configure the kernel with CONFIG_STRICT_DEVMEM,
then you can try the supplied crash utility memory driver
as a replacement for /dev/mem:
# cd <path-to>/crash-6.0.3/memory_driver
# make
...
# insmod crash.ko
and then try running on the live system.
Dave
1:29 a.m.
Thanks Dave for looking into the problem i am facing.
as suggested, i tried the things. Please see the output below:
--------------------------------------
hltncra110731:/home/adil # nm -Bn linux-2.6.32.12-0.7/vmlinux | grep _stext
ffffffff82000198 T _stext
hltncra110731:/home/adil # cat /proc/kallsyms | grep _stext
ffffffff82000198 T _stext
hltncra110731:/home/adil # cat /proc/kallsyms | grep -e
kernel_config_data -e cpu_possible_mask
ffffffff8281a660 R cpu_possible_mask
ffffffff82827aa0 r kernel_config_data
ffffffff82a16ec0 r __ksymtab_cpu_possible_mask
ffffffff82a28730 r __kcrctab_cpu_possible_mask
ffffffff82a322b6 r __kstrtab_cpu_possible_mask
hltncra110731:/home/adil #
--------------------------------------
--------------------------------------
hltncra110731:/home/adil # crash linux-2.6.32.12-0.7/vmlinux
crash 5.0.1
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
--------------------------------------
After the above output the machine got freezed. After waiting for
sometime, i have to hard reboot the machine. There is no crash log
generated.
I checked the CONFIG_STRICT_DEVMEM:
--------------------------------------
hltncra110731:/home/adil # zcat /proc/config.gz | grep CONFIG_STRICT_DEVMEM
# CONFIG_STRICT_DEVMEM is not set
hltncra110731:/home/adil #
--------------------------------------
My running kernel is the one which i have build:
--------------------------------------
adil@hltncra110731:~> uname -a
Linux hltncra110731 2.6.32.12-crash-crash #66 SMP Thu Feb 9 19:58:42
IST 2012 x86_64 x86_64 x86_64 GNU/Linux
adil@hltncra110731:~>
--------------------------------------
Does this problem something related to hardware / bios configuration?
I am providing the details, if by any chance it helps you:
--------------------------------------
adil@hltncra110731:~> cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU 4300 @ 1.80GHz
stepping : 2
cpu MHz : 1203.000
cache size : 2048 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64
monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm lahf_lm
bogomips : 3591.03
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU 4300 @ 1.80GHz
stepping : 2
cpu MHz : 1803.000
cache size : 2048 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
lm constant_tsc arch_perfmon pebs bts rep_good aperfmperf pni dtes64
monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm lahf_lm
bogomips : 3591.07
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
--------------------------------------
Thanks.
Regards,
Adil
On Fri, Feb 10, 2012 at 9:33 PM, Dave Anderson <anderson(a)redhat.com> wrote:
----- Original Message -----
>
> And lastly, in all cases where a dumpfile cannot be read correctly,
> first verify that crash works with the live system. Presuming
> t, and it was configured CONFIG_STRICT_DEVMEM, try this:
Sorry -- cut-and-paste error -- I meant to state "Presuming that
the kernel was *NOT* configured with CONFIG_STRICT_DEVMEM".
>
> # crash ../linux-2.6.32.12-0.7/vmlinux
>
> If that comes up OK, then we can rule out more fundamental
> failure causes.
If you did configure the kernel with CONFIG_STRICT_DEVMEM,
then you can try the supplied crash utility memory driver
as a replacement for /dev/mem:
# cd <path-to>/crash-6.0.3/memory_driver
# make
...
# insmod crash.ko
and then try running on the live system.
Dave
2:46 p.m.
----- Original Message -----
Thanks Dave for looking into the problem i am facing.
as suggested, i tried the things. Please see the output below:
--------------------------------------
hltncra110731:/home/adil # nm -Bn linux-2.6.32.12-0.7/vmlinux | grep
_stext
ffffffff82000198 T _stext
hltncra110731:/home/adil # cat /proc/kallsyms | grep _stext
ffffffff82000198 T _stext
hltncra110731:/home/adil # cat /proc/kallsyms | grep -e
kernel_config_data -e cpu_possible_mask
ffffffff8281a660 R cpu_possible_mask
ffffffff82827aa0 r kernel_config_data
ffffffff82a16ec0 r __ksymtab_cpu_possible_mask
ffffffff82a28730 r __kcrctab_cpu_possible_mask
ffffffff82a322b6 r __kstrtab_cpu_possible_mask
hltncra110731:/home/adil #
--------------------------------------
--------------------------------------
hltncra110731:/home/adil # crash linux-2.6.32.12-0.7/vmlinux
crash 5.0.1
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
--------------------------------------
After the above output the machine got freezed. After waiting for
sometime, i have to hard reboot the machine. There is no crash log
generated.
I checked the CONFIG_STRICT_DEVMEM:
--------------------------------------
hltncra110731:/home/adil # zcat /proc/config.gz | grep
CONFIG_STRICT_DEVMEM
# CONFIG_STRICT_DEVMEM is not set
hltncra110731:/home/adil #
--------------------------------------
My running kernel is the one which i have build:
--------------------------------------
adil@hltncra110731:~> uname -a
Linux hltncra110731 2.6.32.12-crash-crash #66 SMP Thu Feb 9 19:58:42
IST 2012 x86_64 x86_64 x86_64 GNU/Linux
adil@hltncra110731:~>
--------------------------------------
Your system is running a kernel with a utsname version "2.6.32.12-crash-crash",
but you used a file named "linux-2.6.32.12-0.7/vmlinux"? Are they exactly
the same kernel?
Also, even though you're back to using crash-5.0.1 again, I afraid I can't even
speculate as to what would cause the machine to freeze, nor have I ever seen
such a thing happen before. Perhaps the /dev/mem driver read something it
should not have based upon a faulty virtual address? I don't know...
Does this problem something related to hardware / bios configuration?
I am providing the details, if by any chance it helps you:
Sorry no, that doesn't help. If you can't even get crash running
on a live system, then I'm running out of suggestions.
Dave
12:41 a.m.
Hi Dave,
On Tue, Feb 14, 2012 at 2:16 AM, Dave Anderson <anderson(a)redhat.com> wrote:
----- Original Message -----
> Thanks Dave for looking into the problem i am facing.
>
> as suggested, i tried the things. Please see the output below:
>
> --------------------------------------
> hltncra110731:/home/adil # nm -Bn linux-2.6.32.12-0.7/vmlinux | grep
> _stext
> ffffffff82000198 T _stext
> hltncra110731:/home/adil # cat /proc/kallsyms | grep _stext
> ffffffff82000198 T _stext
> hltncra110731:/home/adil # cat /proc/kallsyms | grep -e
> kernel_config_data -e cpu_possible_mask
> ffffffff8281a660 R cpu_possible_mask
> ffffffff82827aa0 r kernel_config_data
> ffffffff82a16ec0 r __ksymtab_cpu_possible_mask
> ffffffff82a28730 r __kcrctab_cpu_possible_mask
> ffffffff82a322b6 r __kstrtab_cpu_possible_mask
> hltncra110731:/home/adil #
> --------------------------------------
>
> --------------------------------------
> hltncra110731:/home/adil # crash linux-2.6.32.12-0.7/vmlinux
>
> crash 5.0.1
> Copyright (C) 2002-2010 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.0
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> --------------------------------------
> After the above output the machine got freezed. After waiting for
> sometime, i have to hard reboot the machine. There is no crash log
> generated.
>
> I checked the CONFIG_STRICT_DEVMEM:
> --------------------------------------
> hltncra110731:/home/adil # zcat /proc/config.gz | grep
> CONFIG_STRICT_DEVMEM
> # CONFIG_STRICT_DEVMEM is not set
> hltncra110731:/home/adil #
> --------------------------------------
>
> My running kernel is the one which i have build:
> --------------------------------------
> adil@hltncra110731:~> uname -a
> Linux hltncra110731 2.6.32.12-crash-crash #66 SMP Thu Feb 9 19:58:42
> IST 2012 x86_64 x86_64 x86_64 GNU/Linux
> adil@hltncra110731:~>
> --------------------------------------
Your system is running a kernel with a utsname version
"2.6.32.12-crash-crash",
but you used a file named "linux-2.6.32.12-0.7/vmlinux"? Are they exactly
the same kernel?
Yes they are same.
Also, even though you're back to using crash-5.0.1 again, I
afraid I can't even
speculate as to what would cause the machine to freeze, nor have I ever seen
such a thing happen before. Perhaps the /dev/mem driver read something it
should not have based upon a faulty virtual address? I don't know...
Sorry that was my mistake.
i tried with the new crash binaries and result is same. machine is
freezed and i have to hard reboot.
----------------
hltncra110731:/home/adil # ./crash-6.0.3/crash linux-2.6.32.12-0.7/vmlinux
crash 6.0.3
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
----------------
I checked the system, no display on screen and keyboard lights are
blinking and no dump generated. No option but to hard reboot.
>
> Does this problem something related to hardware / bios configuration?
> I am providing the details, if by any chance it helps you:
Sorry no, that doesn't help. If you can't even get crash running
on a live system, then I'm running out of suggestions.
Even i am giving up now :(
i should do my work on that machine where crash / kdump is working.
Thanks a lot for all your inputs and showing your patience on my queries :)
Thanks,
Adil
Dave
9 a.m.
----- Original Message -----
Hi List,
I am new to crash / kdump tool and failing some problem as mentioned
below.
I am referring the Linux Kernel Crash Book (http://www.dedoimedo.com/)
and URL http://www.dedoimedo.com/computers/crash.html.
Well, that's your first mistake...
The section in that book that has caused your confusion is this part:
| You can use the old way. Here's an example on CentOS 5.4:
|
| crash /boot/System.map-2.6.18-164.10.1.el5 /boot/vmlinuz-2.6.18-164.10.1.el5 vmcore
|
| Notice the use of vmlinuz kernel image, as opposed to vmlinux previously required.
Given CentOS 5.4 is a copy of RHEL5.4, I cannot even imagine where the
author got the impression that the /boot/vmlinuz-<release> file
could *ever* have been used by the crash utility. The vmlinuz file is
a stripped and compressed output from the vmlinux file, and is completely
unsuitable for use by the crash utility. His statement above is just
complete fanatasy. The crash utility needs an unstripped vmlinux file
containing the debuginfo sections for the base kernel. He does subsequently
state that "alternatively" you can use vmlinux file:
| Alternatively, you can use only the debug information under /usr/lib/debug. The
| information is extracted during the installation of kernel-debuginfo packages
| matching the kernel that was running at the time of the kernel crash.
But in reality -- that is the only file that can be used with the
crash utility, at least from a RHEL5/CentOS perspective. I can't
speak for any other distribution.
I am building modified kernel source (2.6.32 based) and added my
modules (for study purpose). Building, installing and booting kernel
is successful. I have enabled the options for kdump as mentioned in
the book:
Enable Kexec system call:
CONFIG_KEXEC=y
Enable kernel crash dumps:
CONFIG_CRASH_DUMP=y
Optional: Disable Symmetric Multi-Processing (SMP) support
CONFIG_SMP=y
Enable sysfs file system support:
CONFIG_SYSFS=y
Enable /proc/vmcore support:
CONFIG_PROC_VMCORE=y
Configure the kernel with debug info:
CONFIG_DEBUG_INFO=y
Configure the start section for reserved RAM for the crash kernel:
CONFIG_PHYSICAL_START=0x200000 (2MB)
Configure kdump kernel so it can be identified:
CONFIG_LOCALVERSION="-crash"
Kdump configuration /etc/sysconfig/kdump:
KDUMP_KERNELVER=""
KDUMP_COMMANDLINE=""
KDUMP_COMMANDLINE_APPEND="maxcpus=1 "
KEXEC_OPTIONS=""
KDUMP_IMMEDIATE_REBOOT="yes"
KDUMP_TRANSFER=""
KDUMP_SAVEDIR="file:///var/crash"
KDUMP_KEEP_OLD_DUMPS="5"
KDUMP_FREE_DISK_SIZE="64"
KDUMP_VERBOSE="3"
KDUMP_DUMPLEVEL="0"
KDUMP_DUMPFORMAT="compressed"
There is no option KDUMP_DUMPDEV option
There is no option KDUMP_RUNLEVEL
I booted successfully with this kernel and tried to crash it by
module. After rebooting, I found that vmcore is generated under
/var/crash/ But I am not able to analyze it with crash command.
linux:/home/adil # cat /proc/cmdline
root=/dev/disk/by-id/ata-WDC_WD800BD-22LRA1_WD-WMAM9ZS19445-part1
resume=/dev/disk/by-id/ata-WDC_WD800BD-22LRA1_WD-WMAM9ZS19445-part2
splash=silent crashkernel=256M-:128M vga=0x31a
linux:/home/adil #
linux:/home/adil # crash /boot/System.map-2.6.32.12-crash-crash
/boot/vmlinuz-2.6.32.12-crash-crash
crash 5.0.1
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
crash: /boot/vmlinuz-2.6.32.12-crash-crash: not a supported file format
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist]
[dumpfile]
Enter "crash -h" for details.
linux:/home/adil #
Like the error message indicates, the vmlinuz-2.6.32.12-crash-crash
file is not a supported file format. Never has been, never will be...
The URL http://www.dedoimedo.com/computers/crash.html mentioned that
"The newer versions of Kdump can work with compressed kernel images.
Furthermore, they copy the System map file and the kernel image into
the crash directory, making the use of crash utility somewhat
simpler."
What has been supported since crash-5.1.3 is the usage of
vmlinux files that have been subsequently compressed with either
gzip or bzip2. But again, the /boot/vmlinuz-<release> file is
a completely different animal.
linux:/home/adil # ls -al /var/crash/2012-01-30-18\:08/
total 1335536
drwxr-xr-x 2 root root 4096 2012-01-30 18:13 .
drwxr-xr-x 8 root root 4096 2012-01-31 12:17 ..
-rw-r--r-- 1 root root 187 2012-01-30 18:13 README.txt
-rw-r--r-- 1 root root 1716605 2012-01-30 18:13 System.map-2.6.32.12-0.7-default
-rw------- 1 root root 1360732590 2012-01-30 18:13 vmcore
-rw-r--r-- 1 root root 3774506 2012-01-30 18:13 vmlinux-2.6.32.12-0.7-default.gz
linux:/home/adil # ls -al /var/crash/2012-01-31-12\:17/
total 1343860
drwxr-xr-x 2 root root 4096 2012-01-31 12:24 .
drwxr-xr-x 8 root root 4096 2012-01-31 12:17 ..
-rw-r--r-- 1 root root 187 2012-01-31 12:24 README.txt
-rw------- 1 root root 1374748735 2012-01-31 12:24 vmcore
linux:/home/adil #
linux:/home/adil # ls /boot/
backup_mbr
boot
boot.readme
config-2.6.32.12-0.7-default
config-2.6.32.12-0.7-xen
grub
initrd
initrd-2.6.32.12-0.7-default
initrd-2.6.32.12-0.7-default-kdump
initrd-2.6.32.12-0.7-xen
initrd-2.6.32.12-crash-crash
initrd-2.6.32.12-crash-crash-kdump
initrd-xen
message
symsets-2.6.32.12-0.7-default.tar.gz
symtypes-2.6.32.12-0.7-default.gz
symvers-2.6.32.12-0.7-default.gz
symvers-2.6.32.12-0.7-xen.gz
System.map-2.6.32.12-0.7-default
System.map-2.6.32.12-0.7-xen
System.map-2.6.32.12-crash-crash
vmlinux-2.6.32.12-0.7-xen.gz
vmlinuz
vmlinuz-2.6.32.12-0.7-default
vmlinuz-2.6.32.12-0.7-xen
vmlinuz-2.6.32.12-crash-crash
vmlinuz-xen
vmlinux-2.6.32.12-0.7-default.gz
linux:/home/adil #
Another observation is boot.kdump seems to on but manually start
giving me error:
linux:/home/adil # chkconfig boot.kdump
boot.kdump on
linux:/home/adil #
linux:/home/adil # /etc/init.d/boot.kdump start
Loading kdump
Regenerating kdump initrd ...
Can't find kernel text map area from kcore
Cannot load /boot/vmlinuz-2.6.32.12-crash-crash
failed
linux:/home/adil #
Other Query: Following is not clear mentioned in the book under
"section 11.2 Crash (capture) kernel":
--------------
This means that while your production kernels will most likely be
named vmlinuz, the Kdump crash kernels need to be uncompressed,
hence named vmlinux, or rather vmlinux-kdump.
---------------
Please help how to correctly setup and use crash on my machine.
With respect to the crash utility, if your kernel build procedure
has created a /usr/lib/debug/lib/modules/<kernel-version>/vmlinux,
then you don't need a System.map file, but rather:
$ crash /usr/lib/debug/lib/modules/<kernel-version>/vmlinux vmcore
If you don't have the file above, and if whatever non-RHEL system you're
running actually puts the vmlinux-<version>.gz file in the /var/crash
subdirectory with the vmcore, then use that file:
$ crash vmlinux-<kernel-version>.gz vmcore
You may also need the System.map file found there as well. I believe
Suse post-builds their debuginfo kernels instead of saving the vmlinux
file from when it was originally built. Could be that their version
of kexec-tools does the copy of the extra files into /var/crash?
I really don't know -- in a RHEL system, only the vmcore files are
copied there.
Anyway, if you have neither of the above, then check your modified kernel
build tree. In the top-level directory, there should be a "vmlinux" file,
which is the file that the crash utility requires. This also presumes that
you built the kernel with -g, which would get turned on when the kernel
is configured with CONFIG_DEBUG_INFO. However, it should be noted
that if your kernel build procedure built more than one kernel "flavor",
then the vmlinux file found there will be that of the last kernel that
was built -- which may not be the one you need.
For example, RHEL5 x86_64 builds three x86_64 kernel rpms:
kernel-2.6.18-232.el5.x86_64.rpm
kernel-debug-2.6.18-232.el5.x86_64.rpm
kernel-xen-2.6.18-232.el5.x86_64.rpm
The rpm files above contain the /boot/vmlinuz-<release> files.
Then, each kernel above has a debuginfo package associated
with it, which contains its original unstripped vmlinux file,
which gets put in /usr/lib/debug/lib/modules/<kernel-version>/vmlinux:
kernel-debuginfo-2.6.18-232.el5.x86_64.rpm
kernel-debug-debuginfo-2.6.18-232.el5.x86_64.rpm
kernel-xen-debuginfo-2.6.18-232.el5.x86_64.rpm
And there is a "common" debuginfo file that is typically installed
with the kernel-specific file above:
kernel-debuginfo-common-2.6.18-232.el5.x86_64.rpm
Again, I can't speak for the procedures used by any other distribution.
Also, do yourself a favor, and upgrade your crash utility version
from crash-5.0.1, which is two years old.
And lastly, although you may get some responses, this is not the list for
kdump-related questions. That is found here:
http://lists.infradead.org/mailman/listinfo/kexec
Dave Anderson
4700
days inactive
4708
days old
devel@lists.crash-utility.osci.io
12 comments
3 participants
participants (3)
-
Adil Mujeeb -
Dave Anderson -
Suzuki K. Poulose