Re: [Crash-utility] crash can't read vmlinux on live system?
by Kris Corwin
I wasn't aware of the new driver, crash. I have now
built CONFIG_CRASH into my kernel. crash still has
trouble reading though.
kris
[root@f14 linux-2.6.9-22.0.2.EL]# crash -d7 ./vmlinux
crash 4.0-2
Copyright (C) 2002, 2003, 2004, 2005 Red Hat, Inc.
Copyright (C) 2004, 2005 IBM Corporation
Copyright (C) 1999-2005 Hewlett-Packard Co
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
get_live_memory_source: /dev/mem
/proc/version:
Linux version 2.6.9-prep.qp2.2.5.11.3qsnet (root@f16) (gcc version 3.2.3
20030502 (Red Hat Linux 3.2.3-20)) #10 SMP Fri Feb 24 15:22:41 EST 2006
./vmlinux:
Linux version 2.6.9-prep.qp2.2.5.11.3qsnet (root@f16) (gcc version 3.2.3
20030502 (Red Hat Linux 3.2.3-20)) #10 SMP Fri Feb 24 15:22:41 EST 2006
<readmem: c0153373, KVADDR, "x86_omit_frame_pointer", 4, (ROE), bff651e0>
/dev/mem: Operation not permitted
crash: read(/dev/mem, 153373, 0): 4 (ffffffff)
crash: read error: kernel virtual address: c0153373 type:
"x86_omit_frame_pointer"
<readmem: c0439d10, KVADDR, "xtime", 8, (FOE), 8349634>
/dev/mem: Operation not permitted
crash: read(/dev/mem, 439d10, 0): 8 (ffffffff)
crash: read error: kernel virtual address: c0439d10 type: "xtime"
[root@f14 linux-2.6.9-22.0.2.EL]# ls -l /dev/crash
crw------- 1 root root 10, 63 Feb 24 10:35 /dev/crash
18 years, 10 months
Re: [Crash-utility] crash can't read vmlinux on live system?
by Kris Corwin
> First, is the vmlinuz that the kernel is running created from the ./vmlinux
> file that you're using as an argument?
yes.
> I'm guessing that it continued on and complained
> about not being able to read the the linux_banner string, and then died?
crash exited right after what I posted.
I can run crash on 2.6.9-11 kernels, but not 2.6.9-22.
Here's the -d run. It looks like I have a memory read issue.
[root@f14 linux-2.6.9-22.0.2.EL]# crash -d7 ./vmlinux
crash 4.0-2
Copyright (C) 2002, 2003, 2004, 2005 Red Hat, Inc.
Copyright (C) 2004, 2005 IBM Corporation
Copyright (C) 1999-2005 Hewlett-Packard Co
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
get_live_memory_source: /dev/mem
/proc/version:
Linux version 2.6.9-prep.qp2.2.5.11.3qsnet (root@f15) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-20)) #3 SMP Tue Feb 21 15:12:39 EST 2006
./vmlinux:
Linux version 2.6.9-prep.qp2.2.5.11.3qsnet (root@f15) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-20)) #3 SMP Tue Feb 21 15:12:39 EST 2006
<readmem: c0153373, KVADDR, "x86_omit_frame_pointer", 4, (ROE), bff71420>
/dev/mem: Operation not permitted
crash: read(/dev/mem, 153373, 0): 4 (ffffffff)
crash: read error: kernel virtual address: c0153373 type: "x86_omit_frame_pointer"
<readmem: c0439d10, KVADDR, "xtime", 8, (FOE), 8349634>
/dev/mem: Operation not permitted
crash: read(/dev/mem, 439d10, 0): 8 (ffffffff)
crash: read error: kernel virtual address: c0439d10 type: "xtime"
[root@f14 linux-2.6.9-22.0.2.EL]# whoami
root
[root@f14 linux-2.6.9-22.0.2.EL]# ls -l /dev/mem
crw-r----- 1 root kmem 1, 1 Feb 22 06:31 /dev/mem
18 years, 10 months
crash can't read vmlinux on live system?
by Kris Corwin
I've had problems running crash on a live system. I've
used various versions of crash and different 2.6 kernels.
Any help would be appreciated.
Kris
[root@f14 linux-2.6.9]# /usr/users/corwin/crash/crash-4.0-2.21/crash
./System.map ./vmlinux
crash 4.0-2.21
Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005 Fujitsu Limited
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
crash: read error: kernel virtual address: c0152223 type:
"x86_omit_frame_pointer"
crash: read error: kernel virtual address: c0454c90 type: "xtime"
[root@f14 linux-2.6.9]# grep FRAME .config
CONFIG_FRAME_POINTER=y
[root@f14 linux-2.6.9]# grep DEBUG_INFO .config
CONFIG_DEBUG_INFO=y
18 years, 10 months
crash for 2.6.15 kernel ?
by xb
Hello Dave and all,
I tried to use crash 4.0-2.20 on 2.6.15 kernel, but it takes a SIGSEGV.
I think this is due to kmem_cache_s structure being renamed to
kmem_cache, as explained in:
http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=co...
Could you tell me if there exists a crash version that supports 2.6.15,
and how to get it ?
Thanks in advance.
Xavier.
traces
---------------------------------------------------------------------------------------------------------------------
Program received signal SIGSEGV, Segmentation fault.
kmem_cache_init () at memory.c:6264
6264 cache = ULONG(cache_buf + next_offset);
(gdb) bt
#0 kmem_cache_init () at memory.c:6264
#1 0x40000000000892f0 in vm_init () at memory.c:673
#2 0x40000000000585b0 in main_loop () at main.c:381
#3 0x400000000047f120 in current_interp_command_loop () at interps.c:274
#4 0x40000000002baf00 in captured_command_loop
(data=0x4000000000346800) at ./main.c:99
#5 0x4000000000346800 in do_catch_errors (uiout=0x600000000015b040,
data=0x600fffffffdaf0a0) at top.c:523
#6 0x40000000003464b0 in catcher (func=@0x4000000000729d00:
0x40000000003467b0 <do_catch_errors>,
func_uiout=0x600000000015b040, func_args=0x600fffffffdaf0a0,
func_val=0x600fffffffdaf0b0,
func_caught=0x600fffffffdaf0b4, errstring=0x40000000006f9a28 "",
gdberrmsg=0x0, mask=6) at top.c:430
#7 0x40000000003468a0 in catch_errors (func=@0x400000000072cf20:
0x40000000002baee0 <captured_command_loop>,
func_args=0x0, errstring=0x40000000006f9a28 "", mask=6) at top.c:535
#8 0x40000000002bbeb0 in captured_main (data=0x600fffffffdaf5a8) at
./main.c:813
#9 0x4000000000346800 in do_catch_errors (uiout=0x600000000001c8a0,
data=0x600fffffffdaf570) at top.c:523
#10 0x40000000003464b0 in catcher (func=@0x4000000000729d00:
0x40000000003467b0 <do_catch_errors>,
func_uiout=0x600000000001c8a0, func_args=0x600fffffffdaf570,
func_val=0x600fffffffdaf580,
func_caught=0x600fffffffdaf584, errstring=0x40000000006f9a28 "",
gdberrmsg=0x0, mask=6) at top.c:430
#11 0x40000000003468a0 in catch_errors (func=@0x4000000000729350:
0x40000000002baf80 <captured_main>,
func_args=0x600fffffffdaf590, errstring=0x40000000006f9a28 "",
mask=6) at top.c:535
#12 0x40000000002bd0e0 in gdb_main (args=0x600fffffffdaf590) at ./main.c:831
#13 0x40000000002bd180 in gdb_main_entry (argc=2,
argv=0x600fffffffdaf8b8) at ./main.c:851
#14 0x4000000000160d90 in gdb_main_loop (argc=2,
argv=0x600fffffffdaf8b8) at gdb_interface.c:66
#15 0x4000000000058490 in main (argc=2, argv=0x600fffffffdaf8b8) at
main.c:362
18 years, 10 months
Re: [Crash-utility] Re: Problem with using crash 4.0-2.21 on ppc
by Dave Anderson
> > Rachita, please let us know if still an issue. Badari, is there any
> > issue with this patch?
>
> Haren,
>
> Yes. My first version of the patch did exactly this. Later, while
> discussing with you we decided that checking for "-1" for CPU ID
> is the right thing to identify the presence of the CPU.
>
> Yep. you are right on kdump boot, you do set cpuid to -1 when
> they are stopped.
>
> So, I guess I am okay with your fix.
>
> Thanks,
> Badari
>
Ok, then I guess I'll take that as a thumbs-up.
Waiting on Rachita's go-ahead...
Thanks,
Dave
18 years, 10 months
Fix for kmem -p
by David Wilder
The struct page has been changed to use an unnamed struct inside a
union. Gdb can't determine the offset of members of this unnamed
structure. This broke the kmem -p command in crash. I am pursuing a
fix from the gdb folks, but I am not convinced they will accept this as
a bug:) In this patch, if the offset of page->mapping can't be found
directly I calculate it. If the order of the elements in struct page
should change this will break again, but I cant figure out what else to do.
-------------
gdb has a problem with unnamed structures in a union. The offset for
page->mapping can't be found breaking the "kmem -p" command. I use a
alternate way to find the offset. Ugly, but works.
Signed-off-by: David Wilder <dwilder(a)us.ibm.com>
--- memory.c.orig 2006-02-16 16:03:17.000000000 -0800
+++ memory.c 2006-02-16 17:30:01.000000000 -0800
@@ -236,6 +236,14 @@
MEMBER_OFFSET_INIT(page_count, "page", "_count");
MEMBER_OFFSET_INIT(page_flags, "page", "flags");
MEMBER_OFFSET_INIT(page_mapping, "page", "mapping");
+ /* gdb is having a problem with unnamed structures in a union
+ * The following "hack" works around this problem. */
+ if ( !VALID_MEMBER(page_mapping) ){
+ MEMBER_OFFSET_INIT(page_mapping, "page", "_mapcount");
+ ASSIGN_OFFSET(page_mapping) =
+ MEMBER_OFFSET("page","_mapcount") +
+ MEMBER_SIZE("struct page","_mapcount");
+ }
MEMBER_OFFSET_INIT(page_index, "page", "index");
MEMBER_OFFSET_INIT(page_buffers, "page", "buffers");
MEMBER_OFFSET_INIT(page_lru, "page", "lru");
-------------------------
--
David Wilder
IBM Linux Technology Center
Beaverton, Oregon, USA
dwilder(a)us.ibm.com
(503)578-3789
18 years, 10 months
crash version 4.0-2.21 is available
by Dave Anderson
Fix to recognize post-2.6.15 ppc64 kernels moving the per_cpu_offsets
to the "paca" structure. Without this patch, crash fails with the
following error messages: "crash: cannot determine idle task addresses
from init_tasks[] or runqueues[]" and "crash: cannot resolve
init_task_union". (pbadari(a)us.ibm.com)
Incorporated a patch containing ppc64 specific changes when reading
kdump vmcores. Kdump vmcores contain pt_regs for all cpus in the ELF
header, so they are read from there rather than from the active tasks'
kernel stacks; also, the registers contents are printed before any
active task backtrace. (haren(a)us.ibm.com)
If pglist_data.node_mem_map structure member does not exist, as in a
ppc64 kernel built with CONFIG_SPARSEMEM, print an init-time warning
message instead of failing with "crash: invalid structure member
offset: pglist_data.node_mem_map" message. (haren(a)us.ibm.com,
anderson(a)redhat.com)
Download from: http://people.redhat.com/anderson
18 years, 10 months
ppc64 specific changes to support kdump vmcore
by Haren Myneni
Reposting this patch as it did not made to crash mailing list. Dave,
Thanks for adding my different e-mail address to subscriber list.
Dave,
Attaching a patch which contains ppc64 specific changes to read kdump
vmcore. As we are saving pt_regs for all cpus, it reads them from vmcore
instead of looking for specific symbols, used for PPC64 netdump/diskdump
vmcore. Also, prints regs before any active backtrace. Whereas for
netdump vmcore, regs will be displayed anyway as part of exception frame
since the 'bt' command displays from the top frame.
Verified with "Make Warn"
Please let me know if you have any comments.
As Badari mentioned, when the sparsemem is enabled, pg_dat->node_mem_map
member does not exists. mem_map for each node is scattered across
multiple sections and is not contiguous. Therefore, some detailed
changes are needed. Until we fix this issue, can we include some hack
so that users can use other commands for vmcore analysis.
The fix will be:
if (MEMBER_EXISTS("pglist_data", "node_mem_map"))
readmem(pgdat+OFFSET(pglist_data_node_mem_map), KVADDR,
&node_mem_map, sizeof(ulong),
"node_mem_map", FAULT_ON_ERROR);
[I noticed your response to Badari's posting. Since the sparsemem is
effected only for powerpc at this point, we thought, user can use the
limited functionality with the above change. (Ex: bt) until we have the
complete fix. At least we will be having crash tool available for kdump
vmore on ppc64. The sparsemem issue will be fixed soon. Ok, please
ignore this if you prefer to wait for the complete fix]
Thanks
Haren
18 years, 10 months
crash ppc64 fixes to deal with per_cpu_offset changes
by Badari Pulavarty
Hi,
After 2.6.15, ppc64 per_cpu_offsets are moved to "paca"
structure. Here is the patch to make crash read per-cpu-offsets
from paca structure.
Without this patch, you get following error while running
crash. Please apply.
crash: cannot determine idle task addresses from init_tasks[] or
runqueues[]
crash: cannot resolve "init_task_union"
Thanks,
Badari
18 years, 10 months
