Re: [Crash-utility] Cell/B.E. SPU commands extension
by Lucio Correia
> Lucio Correia wrote:
> > Hi,
> >
> > I've developed this crash extension to analyze SPU specific data for
> > Cell/B.E. processor. This extension makes use of some important data
> > saved by this kernel patch (that is not mainline yet)
> > http://ozlabs.org/pipermail/cbe-oss-dev/2007-May/001848.html during the
> > crash dump.
> >
> > I would like to check if there is any issue with the code.
> >
>
> Functionally it looks fine.
>
> I changed the _init() function to just do an error(INFO, ...)
> so I could load the extensions, and the only suggestion
> I can make is purely aesthetic, which would be to make
> the "help" messages 80 characters or less like the regular
> commands are. In other words, the "DESCRIPTION" section
> outputs, and the sentences in in the "spuctx" EXAMPLE
> section are kind of ugly the way that they run on with
> no linefeeds.
>
> But like I said before, I don't see any issues/problems
> with the code -- pretty nifty extension...
>
> Dave
>
Thanks for the comments, Dave. I'm correcting these issues.
Regards,
--
Lucio Correia
Software Engineer
IBM LTC Brazil
17 years, 2 months
[RFC] Crash extension for SystemTap
by Satoru MORIYA
Hi,
Here is an extension(shared object) of the crash to retrieve the trace
data of systemtap scripts.
I'd like to analyze what caused the kernel panic by using the systemtap.
However, currently the systemtap's trace data can't be retrieved from a
dumped image easily. So, I developed a crash's extension which retrieves
the data recorded by systemtap from the dumped image.
Here is a brief document of this extension. This extension supports the new
utt-based buffer as well as the bulk-mode buffer of old systemtap module.
I have tested this extention on the following system.
* FC6, i386, kernel-2.6.21, systemtap-0.5.14, crash-4.0-1.1
* FC6, i386, kernel-2.6.20, systemtap-0.5.13/14, crash-4.0-1.1
* RHEL5, i386, kernel-2.6.18-8.el5, systemtap-0.5.12, crash-4.0-3.14
Preparation
==============
(A) Build the shared-object(stplog.so).
1. Put Makefile and stplog.c into a directory ($DIR)
$ cd $DIR
2. Make the symbolic link to the crash source code directory
$ ln -s $WHERE_CRASH_PLACED crash
3. Build
$ make
(B) Make the crash dump which includes SystemTap trace data.
(*)If you analyze the live system memory, ignore this section.
1. Install kdump
If you use FC6, see following URL.
http://fedoraproject.org/wiki/FC6KdumpKexecHowTo?highlight=%28kdump%29
2. Use SystemTap
$ stap foo.stp
3. Panic
$ echo c > /proc/sysrq-trigger
How to use
==============
1. start crash
$ crash vmlinux vmcore
(*) If you analyze the live system memory, you don't need "vmcore".
$ crash vmlinux
2. load the shared-object
crash> extend $(WHERE_OBJ_PLACED)/stplog.so
3. retrieve the data
crash> stplog -m <mod_name>
(*) <mod_name> is the name of trace module from which you retrieve data.
4. You can get output files under the directory whose name is <mod_name>.
Output
==============
stplog command makes a file per channel buffer of relayfs(equivalent to per cpu).
And it also removes padding bytes.
I believe this command is very useful for system administrators
if they monitor their systems with SystemTap.
Best Regards,
---
Satoru MORIYA
Linux Technology Center
Hitachi, Ltd., Systems Development Laboratory
E-mail: satoru.moriya.br(a)hitachi.com
17 years, 3 months
Re: [Crash-utility] Scripting infrastructure in crash
by Luc Chouinard
I'd would like very much to get the sial interpreter into the crash tool.
It's a matter of mating the apiops (defined in sial_api.h) to the gdb framework.
Things are a bit hectic right now but I should be able to look at in the next few weeks.
It's definitively something I'll look at in the short term.
Now, if anyone wants to start looking, please do. keep me in the loop.
Luc
----- Original Message ----
From: Dave Anderson <anderson(a)redhat.com>
To: sachinp(a)in.ibm.com; "Discussion list for crash utility usage, maintenance and development" <crash-utility(a)redhat.com>
Sent: Wednesday, May 2, 2007 9:44:20 AM
Subject: Re: [Crash-utility] Scripting infrastructure in crash
"Sachin P. Sant" wrote:
> Dave, i came across one of the Crash TODO list items about having
> a scripting infrastructure in crash.
>
> I was trying to evaluate the Alicia utility [mentioned in todo list].
> Here are some of my observations about Alicia.
>
> 1] It is a wrapper on top of crash.
> 2] One can write perl based scripts to extract infromation from dumps.
> 3] It has a nice report generation functionality which presents
> the data in text as well as html format.
> 4] Provides functions which could be used to read data from crash
> dumps.
> 5] It is easy to use and effective too.
>
> Attached here is a sample script which i tried using Alicia to
> display block and character devices. [ I know dev command already
> does this stuff .. but for the sake of trying out the Alicia i chose
> to write such a script ].
>
> Also encountered few problems while trying out Alicia.
> 1] On PPC64 arch came across data type overflow problem while
> executing the attached script.
> 2] On s390/s390x architecture class function provided by Alicia
> seems broken.
> 3] Alicia is a wrapper on top of crash.
>
> Other dump solutions [ lkcd ] has sial scripting which is c like
> and very effective. Not sure how difficult it will be to implement
> something like sial in crash.
>
> Do you have any plans of having scripting infrastructure in
> crash ? If yes your thoughts on Alicia / sial / < any other stuff >
>
I personally do not.
But, a few days ago, Luc Chouinard started looking into what it
would take to support sial scripting.
Dave
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
17 years, 4 months
crash version 4.0-4.3 is available
by Dave Anderson
- Tentatively scheduled as the baseline version for RHEL4.6 and RHEL5.1
crash utility errata releases.
- Fix for "kmem -f" command on 2.6.17 and later CONFIG_DISCONTIGMEM
kernels. Without the patch, the command would fail with the error
message "kmem: cannot determine zone mem_map: TBD".
(troy.heber(a)hp.com)
- Fix for segmentation violation when using the wrong vmlinux file
command line argument on a live system on either the x86_64 or
ia64 architectures. If attempted with this version, the normal
"WARNING: vmlinux and /proc/version do not match!" message will
be followed by an additional warning message that displays the
Linux version number from /proc/version, and then the final message:
"crash: please use the vmlinux file for that kernel version, or
try using the System.map for that kernel version as an additional
argument." (anderson(a)redhat.com)
- For all 4 types of input-file processing:
1) $HOME/.crashrc
2) ./.crashrc
3) "crash -i input"
4) session-time "< input"
If a command in the input file encounters a FATAL error, the
remainder of the commands will be executed. Until now, if any
command in the input file caused a FATAL error, the processing
of the remainder of the commands would be aborted.
(anderson(a)redhat.com)
Download from: http://people.redhat.com/anderson
17 years, 5 months
Re: Crash segv
by Dave Anderson
> Not sure how to fix this one yet. Given the live-kernel /proc/version
> mismatch, it's looking at the /boot/System.map of the actual (booted) kernel
> for the address of the system utsname structure. And it that matches OK,
> it will use that System.map file for symbol values, and automatically
> do the bait-and-switch operation for you. (i.e. the "please wait...
> (patching ### gdb minimal_symbol values)" operation, as if you had
> manually entered the System.map file on the command line)
>
> But, as it's trying to do a kernel vtop operation on the system utsname
> structure address -- which is no longer a simple mask operation on x86_64
> because of the potentially varying physical base address -- it's crapping
> out in x86_64_VTOP() using an uninitialized machdep structure.
> May have to disallow that feature on x86_64 (and ia64 as well I think...)
Actually, it only segv's on x86_64 and ia64, but the search for the
booted System.map file fails quietly for all but s390 and s390x.
VTOP() used to be a simple masking of the kernel virtual address,
but over time, the base kernel virtual addresses can vary, and the
base address determination has not happened this early in the crash
session.
So I'm going to change it such that a more useful error message
gets printed, and then a graceful exit, like this:
# crash vmlinux-2.6.18-8.el5
crash 4.0-4.3rc1
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
WARNING: vmlinux-2.6.18-8.el5 and /proc/version do not match!
WARNING: /proc/version indicates kernel version: 2.6.18-8.1.3.el5
crash: please use the vmlinux file for that kernel version, or try using
the System.map for that kernel version as an additional argument.
#
Now obviously the best remedy is to use the correct vmlinux file.
But if it's not available, and the non-matching vmlinux file is
"close enough", then the System.map suggestion may work. Taking the
example above, it works like so:
# crash vmlinux-2.6.18-8.el5 /boot/System.map-2.6.18-8.1.3.el5
crash 4.0-4.3rc1
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "ia64-unknown-linux-gnu"...
please wait... (patching 27218 gdb minimal_symbol values)
SYSTEM MAP: /boot/System.map-2.6.18-8.1.3.el5
DEBUG KERNEL: vmlinux-2.6.18-8.el5 (2.6.18-8.el5)
DUMPFILE: /dev/mem
CPUS: 2
DATE: Wed Jun 13 15:42:55 2007
UPTIME: 5 days, 15:12:08
LOAD AVERAGE: 1.10, 1.04, 1.05
TASKS: 123
NODENAME: hp-rx2660-01.lab.boston.redhat.com
RELEASE: 2.6.18-8.1.3.el5
VERSION: #1 SMP Mon Apr 16 15:55:02 EDT 2007
MACHINE: ia64 (1594 Mhz)
MEMORY: 16 GB
PID: 29467
COMMAND: "crash"
TASK: e00000012f2e8000 [THREAD_INFO: e00000012f2e9040]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
crash>
Dave
17 years, 5 months
[PATCH] Use dha_kernel_addr from LKCD header
by Bernhard Walle
Currently, ia64_calc_phys_start() in ia64.c silently assumes DEFAULT_PHYS_START
as start address with LKCD dumps. There are machines where this is not true.
This patch implements querying from the dump header.
Signed-off-by: Bernhard Walle <bwalle(a)suse.de>
---
ia64.c | 11 ++++++++++-
lkcd_common.c | 18 ++++++++++++++++++
lkcd_fix_mem.c | 10 ++++++++++
3 files changed, 38 insertions(+), 1 deletion(-)
--- a/ia64.c
+++ b/ia64.c
@@ -3810,7 +3810,16 @@ ia64_calc_phys_start(void)
phys_start);
}
return;
- }
+ } else if (LKCD_DUMPFILE()) {
+
+ if (lkcd_get_kernel_start(&phys_start)) {
+ machdep->machspec->phys_start = phys_start;
+ if (CRASHDEBUG(1))
+ fprintf(fp,
+ "LKCD dump: phys_start: %lx\n",
+ phys_start);
+ }
+ }
if ((vd = get_kdump_vmcore_data())) {
/*
--- a/lkcd_common.c
+++ b/lkcd_common.c
@@ -787,6 +787,24 @@ get_offset(uint64_t paddr)
}
+int
+lkcd_get_kernel_start(ulong *addr)
+{
+ if (!addr)
+ return 0;
+
+ switch (lkcd->version)
+ {
+ case LKCD_DUMP_V8:
+ case LKCD_DUMP_V9:
+ return lkcd_get_kernel_start_v8(addr);
+
+ default:
+ return 0;
+ }
+}
+
+
int
lkcd_lseek(physaddr_t paddr)
--- a/lkcd_fix_mem.c
+++ b/lkcd_fix_mem.c
@@ -113,4 +113,14 @@ get_lkcd_switch_stack(ulong task)
return 0;
}
+int lkcd_get_kernel_start_v8(ulong *addr)
+{
+ if (!addr)
+ return 0;
+
+ *addr = ((dump_header_asm_t *)lkcd->dump_header_asm)->dha_kernel_addr;
+
+ return 1;
+}
+
#endif // IA64
17 years, 5 months
Crash segv
by Ben Woodard
Dave here is a segv for you to look at. I have the core file if you want
it. It looks to me like it doesn't like it when the booted kernel is not
the mismatch between the kernel and the vmlinux file. I don't expect it
crash to run successfully in this situation, trying to do this was a
mistake on my part. However, I do think that crash shouldn't segv.
BTW IT#123806
4 [ben@wopri ~]$sudo crash /boot/vmlinux-2.6.9-70chaos
crash 4.0-4.1
<snip>
WARNING: /boot/vmlinux-2.6.9-70chaos and /proc/version do not match!
Segmentation fault (core dumped)
5 [ben@wopri ~]$uname -a
Linux wopri 2.6.9-69chaos #1 SMP Wed May 23 07:41:13 PDT 2007 x86_64
x86_64 x86_64 GNU/Linux
--
-ben
-=-
17 years, 5 months
crash version 4.0-4.2 is available
by Dave Anderson
- Fix for support of 2.6.22 kernels, which have changed the name
of the task_struct's "thread_info" member to the "stack" member.
This would cause the crash session to fail during initialization.
(troy.heber(a)hp.com, anderson(a)redhat.com)
- Fix to account for the number of pgdata nodes being less than the
number of cpus. Without the patch, the crash session would fail
during initialization with the error message: "crash: numnodes out
of sync with pgdat_list?" (sharyath(a)in.ibm.com)
- Implemented support for ia64 dom0/HV kdump dumpfile support, taken
either via the traditional kdump process, or simulated via the
Fujitsu "sadump" facility. (oda(a)valinux.co.jp)
- Created a "--no_panic" command line option to avoid the panic-task
search during initialization. (anderson(a)redhat.com)
- Implmented a new "ps -r" option to display resource limits (ulimits):
crash> ps -r 20618
PID: 20618 TASK: 1003cb82030 CPU: 1 COMMAND: "bash"
RLIMIT CURRENT MAXIMUM
CPU (unlimited) (unlimited)
FSIZE (unlimited) (unlimited)
DATA (unlimited) (unlimited)
STACK 10485760 (unlimited)
CORE 0 (unlimited)
RSS (unlimited) (unlimited)
NPROC 8180 8180
NOFILE 1024 1024
MEMLOCK 32768 32768
AS (unlimited) (unlimited)
LOCKS (unlimited) (unlimited)
SIGPENDING 1024 1024
MSGQUEUE 819200 819200
(anderson(a)redhat.com)
- Implement support for the registration of CLEANUP extension commands
that do not show up in help menu, but get called by restore_sanity().
Extension modules may also register HIDDEN_COMMAND functions; and the
"help -e" debug output has been enhanced. (anderson(a)redhat.com)
- Implemented a new symbol_value_module() primitive, primarily for use
by extension modules to quickly access the address of a module symbol
in cases where a name-clash may exist between the base kernel and/or
other modules. (anderson(a)redhat.com)
- The crash-4.0-4.2.src.rpm package will create an additional package
named crash-devel-4.0-4.2.i386.rpm, which is for use by extension
modules. The -devel package installs the top-level "defs.h" file in
"/usr/include/crash/defs.h". (anderson(a)redhat.com)
Download from: http://people.redhat.com/anderson
17 years, 5 months
