[RFC] Crash extension for SystemTap
by Satoru MORIYA
Hi,
Here is an extension(shared object) of the crash to retrieve the trace
data of systemtap scripts.
I'd like to analyze what caused the kernel panic by using the systemtap.
However, currently the systemtap's trace data can't be retrieved from a
dumped image easily. So, I developed a crash's extension which retrieves
the data recorded by systemtap from the dumped image.
Here is a brief document of this extension. This extension supports the new
utt-based buffer as well as the bulk-mode buffer of old systemtap module.
I have tested this extention on the following system.
* FC6, i386, kernel-2.6.21, systemtap-0.5.14, crash-4.0-1.1
* FC6, i386, kernel-2.6.20, systemtap-0.5.13/14, crash-4.0-1.1
* RHEL5, i386, kernel-2.6.18-8.el5, systemtap-0.5.12, crash-4.0-3.14
Preparation
==============
(A) Build the shared-object(stplog.so).
1. Put Makefile and stplog.c into a directory ($DIR)
$ cd $DIR
2. Make the symbolic link to the crash source code directory
$ ln -s $WHERE_CRASH_PLACED crash
3. Build
$ make
(B) Make the crash dump which includes SystemTap trace data.
(*)If you analyze the live system memory, ignore this section.
1. Install kdump
If you use FC6, see following URL.
http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29
2. Use SystemTap
$ stap foo.stp
3. Panic
$ echo c > /proc/sysrq-trigger
How to use
==============
1. start crash
$ crash vmlinux vmcore
(*) If you analyze the live system memory, you don't need "vmcore".
$ crash vmlinux
2. load the shared-object
crash> extend $(WHERE_OBJ_PLACED)/stplog.so
3. retrieve the data
crash> stplog -m <mod_name>
(*) <mod_name> is the name of trace module from which you retrieve data.
4. You can get output files under the directory whose name is <mod_name>.
Output
==============
stplog command makes a file per channel buffer of relayfs(equivalent to per cpu).
And it also removes padding bytes.
I believe this command is very useful for system administrators
if they monitor their systems with SystemTap.
Best Regards,
---
Satoru MORIYA
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: satoru.moriya.br(a)hitachi.com
16 years, 11 months
- 4
- 8
crash and sles 9 dumps
by Daniel Li
Hey There,
Does anyone know if the latest crash is supposed to work with kernel
dumps produced by SLES 9 LKCD facilities? If yes, HOW? I was told it
should be working at one point, but with the latest crash (4.0-4.5) it
doesn't seem to work. Here is what I did:
1) With the map and kerntypes files created by lkcd along with the dump:
# /dumps/new/sles/native$ /home/dli/bin/crash map.0 kerntypes.0 dump.0
Then crash complains the kerntypes file is not SMP while dump is (see
below), even they were produced by lkcd at the same time and worked with
'lcrash' with no problem what-so-ever.
2) Then I built a debug vmlinux using kernel-source package on the same
host that produced the dump, and tried to feed it to crash:
# /dumps/new/sles/native$ /home/dli/bin/crash vmlinux dump.0
And crash complains the debug vmlinux file and the dump created by the
non-debug kernel do not match.
Here is output of 'uname -a' from the host:
tst037:/usr/src/linux-2.6.5-7.97 # uname -a
Linux tst037 2.6.5-7.97-smp #1 SMP Fri Jul 2 14:21:59 UTC 2004 x86_64
x86_64 x86_64 GNU/Linux
Any thoughts? (Outputs from crash with '-d 4' flag are attached below... )
Thanks,
Daniel
tst037:/usr/src/linux-2.6.5-7.97 # gcc -v
Reading specs from /usr/lib64/gcc-lib/x86_64-suse-linux/3.3.3/specs
Configured with: ../configure --enable-threads=posix --prefix=/usr
--with-local-prefix=/usr/local --infodir=/usr/share/info
--mandir=/usr/share/man --enable-languages=c,c++,f77,objc,java,ada
--disable-checking --libdir=/usr/lib64 --enable-libgcj
--with-gxx-include-dir=/usr/include/g++ --with-slibdir=/lib64
--with-system-zlib --enable-shared --enable-__cxa_atexit x86_64-suse-linux
Thread model: posix
gcc version 3.3.3 (SuSE Linux)
# /home/dli/bin/crash -v
crash 4.0-4.5
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
# /home/dli/bin/crash map.0 kerntypes.0 dump.0 -d 4
crash 4.0-4.5
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
crash: xc_core_elf_verify: not a xen ELF core file
crash: diskdump: dump does not have panic dump header
crash: xc_core_elf_verify: not a xen ELF core file
crash: diskdump: dump does not have panic dump header
dump_header:
dh_magic_number: a8190173618f23ed (DUMP_MAGIC_NUMBER)
dh_version: 8 (LKCD_DUMP_V8)
dh_header_size: 734
dh_dump_level: 3 (DUMP_LEVEL_HEADER|DUMP_LEVEL_KERN)
dh_page_size: 4096
dh_memory_size: 1012523
dh_memory_start: 10000000000
dh_memory_end: a8190173618f23ed
dh_num_pages: 57696
dh_panic_string: sysrq
dh_time: Thu Aug 2 10:55:35 2007
dh_utsname_sysname: Linux
dh_utsname_nodename: tst037
dh_utsname_release: 2.6.5-7.97-smp
dh_utsname_version: #1 SMP Fri Jul 2 14:21:59 UTC 2004
dh_utsname_machine: x86_64
dh_utsname_domainname:
dh_current_task: 100096e6080
dh_dump_compress: 2 (DUMP_COMPRESS_GZIP)
dh_dump_flags: 80000000 ()
dh_dump_device: 0
found DUMP_DH_END
gdb kerntypes.0
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
<readmem: ffffffff80348440, KVADDR, "kernel_config_data", 32768, (ROE),
ae6860>
WARNING: could not find MAGIC_START!
<readmem: ffffffff804ee1d0, KVADDR, "xtime", 16, (FOE), 905c70>
<readmem: ffffffff803a8540, KVADDR, "system_utsname", 390, (ROE), 90625c>
<readmem: ffffffff803a8528, KVADDR, "linux_banner", 8, (FOE), 7fff31c22538>
<readmem: ffffffff8035b2c8, KVADDR, "accessible check", 8, (ROE|Q),
7fff31c221c8>
<readmem: ffffffff8035b2c8, KVADDR, "readstring characters", 1499,
(ROE|Q), 7fff31c211b0>
WARNING: kernels compiled by different gcc versions:
kerntypes.0: (unknown)
dump.0 kernel: 3.3.3
verify_namelist:
/proc/version:
Linux version 2.6.5-7.97-smp (geeko@buildhost) (gcc version 3.3.3 (SuSE
Linux)) #1 SMP Fri Jul 2 14:21:59 UTC 2004
utsname version: #1 SMP Fri Jul 2 14:21:59 UTC 2004
kerntypes.0:
p�
crash: incompatible arguments: kerntypes.0 is not SMP -- dump.0 is SMP
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist]
[dumpfile]
Enter "crash -h" for details.
# /home/dli/bin/crash vmlinux dump.0 -d 4
crash 4.0-4.5
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
crash: xc_core_elf_verify: not a xen ELF core file
crash: diskdump: dump does not have panic dump header
dump_header:
dh_magic_number: a8190173618f23ed (DUMP_MAGIC_NUMBER)
dh_version: 8 (LKCD_DUMP_V8)
dh_header_size: 734
dh_dump_level: 3 (DUMP_LEVEL_HEADER|DUMP_LEVEL_KERN)
dh_page_size: 4096
dh_memory_size: 1012523
dh_memory_start: 10000000000
dh_memory_end: a8190173618f23ed
dh_num_pages: 57696
dh_panic_string: sysrq
dh_time: Thu Aug 2 10:55:35 2007
dh_utsname_sysname: Linux
dh_utsname_nodename: tst037
dh_utsname_release: 2.6.5-7.97-smp
dh_utsname_version: #1 SMP Fri Jul 2 14:21:59 UTC 2004
dh_utsname_machine: x86_64
dh_utsname_domainname:
dh_current_task: 100096e6080
dh_dump_compress: 2 (DUMP_COMPRESS_GZIP)
dh_dump_flags: 80000000 ()
dh_dump_device: 0
found DUMP_DH_END
gdb vmlinux
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
<readmem: ffffffff80323d80, KVADDR, "kernel_config_data", 32768, (ROE),
12c88b0>
WARNING: could not find MAGIC_START!
<readmem: ffffffff804c21d0, KVADDR, "xtime", 16, (FOE), 905c70>
<readmem: ffffffff80383460, KVADDR, "system_utsname", 390, (ROE), 90625c>
<readmem: ffffffff80383448, KVADDR, "linux_banner", 8, (FOE), 7fff21a5e388>
<readmem: 6f69637072206f74, KVADDR, "accessible check", 8, (ROE|Q),
7fff21a5e018>
crash: seek error: kernel virtual address: 6f69637072206f74 type:
"accessible check"
crash: vmlinux and dump.0 do not match!
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist]
[dumpfile]
Enter "crash -h" for details.
>/dumps/new/sles/native$ /home/dli/bin/crash map.0 kerntypes.0 dump.0 -d 4
crash 4.0-4.5
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
crash: xc_core_elf_verify: not a xen ELF core file
crash: diskdump: dump does not have panic dump header
crash: xc_core_elf_verify: not a xen ELF core file
crash: diskdump: dump does not have panic dump header
dump_header:
dh_magic_number: a8190173618f23ed (DUMP_MAGIC_NUMBER)
dh_version: 8 (LKCD_DUMP_V8)
dh_header_size: 734
dh_dump_level: 3 (DUMP_LEVEL_HEADER|DUMP_LEVEL_KERN)
dh_page_size: 4096
dh_memory_size: 1012523
dh_memory_start: 10000000000
dh_memory_end: a8190173618f23ed
dh_num_pages: 57696
dh_panic_string: sysrq
dh_time: Thu Aug 2 10:55:35 2007
dh_utsname_sysname: Linux
dh_utsname_nodename: tst037
dh_utsname_release: 2.6.5-7.97-smp
dh_utsname_version: #1 SMP Fri Jul 2 14:21:59 UTC 2004
dh_utsname_machine: x86_64
dh_utsname_domainname:
dh_current_task: 100096e6080
dh_dump_compress: 2 (DUMP_COMPRESS_GZIP)
dh_dump_flags: 80000000 ()
dh_dump_device: 0
found DUMP_DH_END
gdb kerntypes.0
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
<readmem: ffffffff80348440, KVADDR, "kernel_config_data", 32768, (ROE), ae6860>
WARNING: could not find MAGIC_START!
<readmem: ffffffff804ee1d0, KVADDR, "xtime", 16, (FOE), 905c70>
<readmem: ffffffff803a8540, KVADDR, "system_utsname", 390, (ROE), 90625c>
<readmem: ffffffff803a8528, KVADDR, "linux_banner", 8, (FOE), 7fff31c22538>
<readmem: ffffffff8035b2c8, KVADDR, "accessible check", 8, (ROE|Q), 7fff31c221c8>
<readmem: ffffffff8035b2c8, KVADDR, "readstring characters", 1499, (ROE|Q), 7fff31c211b0>
WARNING: kernels compiled by different gcc versions:
kerntypes.0: (unknown)
dump.0 kernel: 3.3.3
verify_namelist:
/proc/version:
Linux version 2.6.5-7.97-smp (geeko@buildhost) (gcc version 3.3.3 (SuSE Linux)) #1 SMP Fri Jul 2 14:21:59 UTC 2004
utsname version: #1 SMP Fri Jul 2 14:21:59 UTC 2004
kerntypes.0:
p�
crash: incompatible arguments: kerntypes.0 is not SMP -- dump.0 is SMP
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
Enter "crash -h" for details.
/dumps/new/sles/native$ /home/dli/bin/crash vmlinux dump.0 -d 4
crash 4.0-4.5
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
crash: xc_core_elf_verify: not a xen ELF core file
crash: diskdump: dump does not have panic dump header
dump_header:
dh_magic_number: a8190173618f23ed (DUMP_MAGIC_NUMBER)
dh_version: 8 (LKCD_DUMP_V8)
dh_header_size: 734
dh_dump_level: 3 (DUMP_LEVEL_HEADER|DUMP_LEVEL_KERN)
dh_page_size: 4096
dh_memory_size: 1012523
dh_memory_start: 10000000000
dh_memory_end: a8190173618f23ed
dh_num_pages: 57696
dh_panic_string: sysrq
dh_time: Thu Aug 2 10:55:35 2007
dh_utsname_sysname: Linux
dh_utsname_nodename: tst037
dh_utsname_release: 2.6.5-7.97-smp
dh_utsname_version: #1 SMP Fri Jul 2 14:21:59 UTC 2004
dh_utsname_machine: x86_64
dh_utsname_domainname:
dh_current_task: 100096e6080
dh_dump_compress: 2 (DUMP_COMPRESS_GZIP)
dh_dump_flags: 80000000 ()
dh_dump_device: 0
found DUMP_DH_END
gdb vmlinux
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
<readmem: ffffffff80323d80, KVADDR, "kernel_config_data", 32768, (ROE), 12c88b0>
WARNING: could not find MAGIC_START!
<readmem: ffffffff804c21d0, KVADDR, "xtime", 16, (FOE), 905c70>
<readmem: ffffffff80383460, KVADDR, "system_utsname", 390, (ROE), 90625c>
<readmem: ffffffff80383448, KVADDR, "linux_banner", 8, (FOE), 7fff21a5e388>
<readmem: 6f69637072206f74, KVADDR, "accessible check", 8, (ROE|Q), 7fff21a5e018>
crash: seek error: kernel virtual address: 6f69637072206f74 type: "accessible check"
crash: vmlinux and dump.0 do not match!
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
Enter "crash -h" for details.
16 years, 11 months
- 1
- 0
Debugging PPC crash-dumps on x86_64
by Vivek P
Hi , I have a crash-dump generated on PPC using diskdump .Can this crash-dump be analyzed using crash-utility running on a x86_64machine ?Thanks and regardsVivek
16 years, 11 months
- 4
- 3
Moderator is on vacation
by Dave Anderson
That would be me, from now through August 17th.
New subscription requests, approval of non-member posts,
and any answers requested of me will be deferred until
then.
Otherwise, just talk quietly amongst yourselves... ;-)
Later,
Dave
16 years, 12 months
- 1
- 0