cannot find stack info on ppc64le
by Han Pingtian
Hello,
I just noticed that on ppc64le, sometimes "bt" cannot find the stack
info of current process. For example, there is a vmcore captured by
kdump on a ppc64le system, which running with a kernel version 3.10. The
vmcore was captured when kernel oopsed. There is no stack info found by
bt:
crash 7.0.9-2.ael7b
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu"...
KERNEL: /usr/lib/debug/lib/modules/3.10.0-221.ael7b.ppc64le/vmlinux
DUMPFILE: /var/crash/127.0.0.1-2015.01.15-22:19:14/vmcore [PARTIAL DUMP]
CPUS: 16
DATE: Thu Jan 15 21:18:16 2015
UPTIME: 17:53:43
LOAD AVERAGE: 213.58, 213.23, 212.70
TASKS: 1383
NODENAME: thymelp2.isst.aus.stglabs.ibm.com
RELEASE: 3.10.0-221.ael7b.ppc64le
VERSION: #1 SMP Wed Jan 7 09:27:09 EST 2015
MACHINE: ppc64le (3425 Mhz)
MEMORY: 15 GB
PANIC: "Oops: Kernel access of bad area, sig: 11 [#1]" (check log for details)
PID: 1970
COMMAND: "cat"
TASK: c0000003130874a0 [THREAD_INFO: c00000005069c000]
CPU: 5
STATE: TASK_RUNNING (PANIC)
crash> set debug 99
debug: 99
crash> bt
PID: 1970 TASK: c0000003130874a0 CPU: 5 COMMAND: "cat"
GETBUF(16384 -> 0)
<readmem: c00000005069c000, KVADDR, "stack contents", 16384, (ROE), 10a81570>
<read_diskdump: addr: c00000005069c000 paddr: 5069c000 cnt: 16384>
read_diskdump: paddr/pfn: 5069c000/5069 -> cache physical page: 50690000
c00000005069c018: do_no_restart_syscall
c00000005069e870: blk_throtl_bio+240
c00000005069e990: clone_endio
c00000005069ea00: generic_make_request_checks+836
c00000005069eab8: hardware_interrupt_common+128
c00000005069eac0: generic_make_request+36
c00000005069eb10: mempool_alloc_slab+36
c00000005069eb30: mempool_alloc+256
c00000005069eb50: mempool_alloc_slab+36
c00000005069ebc0: get_request+948
c00000005069ec00: __split_and_process_bio+1408
c00000005069ec20: autoremove_wake_function
c00000005069ec80: find_busiest_group+544
c00000005069edf0: load_balance+684
c00000005069ee10: blk_throtl_bio+240
c00000005069ee70: find_busiest_group+544
c00000005069eee0: dequeue_task_fair+968
c00000005069ef30: clone_endio
c00000005069ef50: get_page_from_freelist+1436
c00000005069f0a0: pSeries_cause_ipi_mux+112
c00000005069f0c0: smp_send_reschedule+164
c00000005069f0e0: default_wake_function+708
c00000005069f160: __wake_up_locked+116
c00000005069f1b0: ep_poll_callback+444
c00000005069f250: run_posix_cpu_timers+104
c00000005069f2c0: hvterm_raw_put_chars+64
c00000005069f2e0: hvc_console_print+336
c00000005069f3a8: initial_stab+2048
c00000005069f3b0: crash_save_cpu+252
c00000005069f488: cik_cp_resume+13476
c00000005069f490: dev_get_drvdata
c00000005069f580: default_machine_kexec+332
c00000005069f610: pSeries_machine_kexec+60
c00000005069f680: machine_kexec+56
c00000005069f6a0: crash_kexec+312
c00000005069f6f0: dev_attr_show+64
c00000005069f748: cik_cp_resume+13476
c00000005069f750: dev_get_drvdata
c00000005069f7f0: radeon_hwmon_show_temp+72
c00000005069f800: slb_miss_realmode+80
c00000005069f808: dev_get_drvdata
c00000005069f810: radeon_hwmon_show_temp+32
c00000005069f890: die+840
c00000005069f930: bad_page_fault+224
c00000005069f948: radeon_hwmon_show_temp+72
c00000005069f9a0: handle_page_fault+44
c00000005069fa00: dev_attr_show+64
c00000005069fa58: cik_cp_resume+13476
c00000005069fa60: dev_get_drvdata
c00000005069fb00: radeon_hwmon_show_temp+72
c00000005069fb10: slb_miss_realmode+80
c00000005069fb18: dev_get_drvdata
c00000005069fb20: radeon_hwmon_show_temp+32
c00000005069fb60: handle_mm_fault+1724
c00000005069fb80: sysfs_open_file
c00000005069fbd0: handle_page_fault+16
c00000005069fc90: alloc_pages_current+416
c00000005069fd00: dev_attr_show+64
c00000005069fd30: sysfs_read_file+220
c00000005069fde0: sys_read+304
c00000005069fe40: syscall_exit
[3fffd0d6fe88] back_trace:
task: c0000003130874a0
flags: 0
instptr: 0
stkptr: 0
bptr: 0
stackbase: c00000005069c000
stacktop: c0000000506a0000
tc: 1003c7b9fa8 (1970, c0000003130874a0)
hp: 0
ref: 0
stackbuf: 10a81570
textlist: 0
frameptr: 0
call_target: none
eframe_ip: 0
debug: 0
radix: 0
cpumask: 0
=> PC: 0 () FP: 0
GETBUF(248 -> 1)
GETBUF(1500 -> 2)
cannot find the stack info.
FREEBUF(2)
FREEBUF(1)
crash>
Is this a problem?
Thanks in advance!
11 years, 1 month
Small fix for pages_scanned value
by Alexandr Terekhov
Hi Dave,
there seems to be typo while printing pages_scanned value - signed output for unsigned variable.
Patch:
--- crash-7.0.9.orig/memory.c 2014-11-13 15:33:40.000000000 -0500
+++ crash-7.0.9/memory.c 2015-01-20 05:19:07.000000000 -0500
@@ -15776,7 +15776,7 @@ dump_zone_stats(void)
if (VALID_MEMBER(zone_pages_scanned)) {
value1 = ULONG(zonebuf +
OFFSET(zone_pages_scanned));
- fprintf(fp, "PAGES_SCANNED: %ld ", value1);
+ fprintf(fp, "PAGES_SCANNED: %lu ", value1);
}
fprintf(fp, "\n");
regards,
Alexandr
11 years, 1 month
[PATCH 0/4] MIPS support
by Rabin Vincent
This adds support for analyzing dumps from 32-bit MIPS.
I've uploaded some sample dumps generated from the QEMU MIPS Malta machine
(little-endian) at the following location:
https://drive.google.com/folderview?id=0B4tMLbMvJ-l6R3J4LWJFc1k0eFU
Rabin Vincent (4):
netdump: Make a helper out of the PPC getregs function
Add gdb patch to fix sim/igen build
Rename convert() to converts() to avoid conflit with gdb
Add support for MIPS
Makefile | 7 +-
configure.c | 48 +++
defs.h | 113 ++++++-
gdb-7.6.patch | 13 +
kernel.c | 6 +-
lkcd_vmdump_v2_v3.h | 3 +-
mips.c | 874 ++++++++++++++++++++++++++++++++++++++++++++++++++++
netdump.c | 27 +-
symbols.c | 5 +
tools.c | 2 +-
10 files changed, 1084 insertions(+), 14 deletions(-)
create mode 100644 mips.c
--
2.1.4
11 years, 2 months
Make note information human readable when help -D
by "Zhou, Wenjian/周文剑"
Hello all,
When analysing elf format core, if "help -D" is specified, crash will
show dumpfile contents/statistics. But the output is hardly to read.
So the patches is to make it human readable.
When analysing kdump format core, if "help -D" is specified, crash won't
show the note information. The patches add note information to the output.
When analysing elf format core, if "help -D" is specified,
the original output like this:
:
:
Elf64_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 336
n_type: 1 (NT_PRSTATUS)
0000000000000000 0000000000000000
0000000000000000 0000000000000000
000000000000141d 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000004 0000000000000286
ffffffff81b01a40 0000000000000000
ffff88003d86be18 0000000000000063
0000000000000000 0000000000000001
ffffffff81645da0 0000000000000000
0000000000000010 0000000000000000
0000000000000000 0000000000000000
0000000000000063 ffffffffffffffff
ffffffff8134b6c6 0000000000000010
0000000000010096 ffff88003d86be18
0000000000000018 00007fc165c0b700
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
:
:
It is hardly to get some useful information. The patch is to make it
human readable. And the output changes to:
:
:
Elf64_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 336
n_type: 1 (NT_PRSTATUS)
si.signo: 0, si.code: 0, si.errno: 0, cursig:0
sigpend: 0
sighold: 0
pid: 5149, ppid: 0, pgrp: 0, sid:0
utime: 0.000000, stime: 0.000000, cutime: 0.000000, cstime: 0.000000
ORIG_RAX: -1, fpvalid: 0
R15 : 0x0000000000000004 R14 : 0x0000000000000286
R13 : 0xffffffff81b01a40 R12 : 0x0000000000000000
RBP : 0xffff88003d86be18 RBX : 0x0000000000000063
R11 : 0x0000000000000000 R10 : 0x0000000000000001
R9 : 0xffffffff81645da0 R8 : 0x0000000000000000
RAX : 0x0000000000000010 RCX : 0x0000000000000000
RDX : 0x0000000000000000 RSI : 0x0000000000000000
RDI : 0x0000000000000063 RIP : 0xffffffff8134b6c6
RFLAGS : 0x0000000000010096 RSP : 0xffff88003d86be18
FS_BASE: 0x00007fc165c0b700 GS_BASE : 0x0000000000000000
CS: 0x0010 SS: 0x0018 DS: 0x0000 ES: 0x0000 FS: 0x0000 GS: 0x0000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
000000000000141d 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000004 0000000000000286
ffffffff81b01a40 0000000000000000
ffff88003d86be18 0000000000000063
0000000000000000 0000000000000001
ffffffff81645da0 0000000000000000
0000000000000010 0000000000000000
0000000000000000 0000000000000000
0000000000000063 ffffffffffffffff
ffffffff8134b6c6 0000000000000010
0000000000010096 ffff88003d86be18
0000000000000018 00007fc165c0b700
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
:
:
The patches support elf64, elf32, qemu64, qemu32 core format.
The kdump format didn't show the note information originally when help -D.
The patches add it.
the original output:
:
:
sub_header_kdump: 222aff0
phys_base: 0
dump_level: 0 (0x0)
split: 0
start_pfn: (unused)
end_pfn: (unused)
offset_vmcoreinfo: 4556 (0x11cc)
size_vmcoreinfo: 1440 (0x5a0)
OSRELEASE=2.6.32-431.el6.x86_64
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffffffff81a8e940
SYMBOL(node_online_map)=ffffffff81c08f40
:
:
LENGTH(node_data)=512
CRASHTIME=1417488164
offset_note: 4176 (0x1050)
size_note: 1820 (0x71c)
num_prstatus_notes: 1
notes_buf: 224c020
notes[0]: 224c020
NT_PRSTATUS_offset: 1050
offset_eraseinfo: 0 (0x0)
size_eraseinfo: 0 (0x0)
:
:
the changed output:
:
:
sub_header_kdump: 2c75ff0
phys_base: 0
dump_level: 0 (0x0)
split: 0
start_pfn: (unused)
end_pfn: (unused)
offset_vmcoreinfo: 4556 (0x11cc)
size_vmcoreinfo: 1440 (0x5a0)
OSRELEASE=2.6.32-431.el6.x86_64
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffffffff81a8e940
SYMBOL(node_online_map)=ffffffff81c08f40
:
:
LENGTH(node_data)=512
CRASHTIME=1417488164
offset_note: 4176 (0x1050)
size_note: 1820 (0x71c)
num_prstatus_notes: 1
notes_buf: 2c97020
notes[0]: 2c97020
si.signo: 0, si.code: 0, si.errno: 0, cursig:0
sigpend: 0
sighold: 0
pid: 5149, ppid: 0, pgrp: 0, sid:0
utime: 0.000000, stime: 0.000000, cutime: 0.000000, cstime: 0.000000
ORIG_RAX: -1, fpvalid: 0
R15 : 0x0000000000000004 R14 : 0x0000000000000286
R13 : 0xffffffff81b01a40 R12 : 0x0000000000000000
RBP : 0xffff88003d86be18 RBX : 0x0000000000000063
R11 : 0x0000000000000000 R10 : 0x0000000000000001
R9 : 0xffffffff81645da0 R8 : 0x0000000000000000
RAX : 0x0000000000000010 RCX : 0x0000000000000000
RDX : 0x0000000000000000 RSI : 0x0000000000000000
RDI : 0x0000000000000063 RIP : 0xffffffff8134b6c6
RFLAGS : 0x0000000000010096 RSP : 0xffff88003d86be18
FS_BASE: 0x00007fc165c0b700 GS_BASE : 0x0000000000000000
CS: 0x0010 SS: 0x0018 DS: 0x0000 ES: 0x0000 FS: 0x0000 GS: 0x0000
NT_PRSTATUS_offset: 1050
offset_eraseinfo: 0 (0x0)
size_eraseinfo: 0 (0x0)
:
:
--
Thanks
Zhou Wenjian
11 years, 2 months
crash: invalid kernel virtual address: 0 type: "memory section"
by Ilya Dryomov
Hello,
I have a couple dumps generated on Ubuntu Trusty LTS (3.13.0-39-generic
kernel) which crash fails on.
$ ./crash ../ddeb/usr/lib/debug/boot/vmlinux-3.13.0-39-generic
../dump.201412280256
crash 7.0.9
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
crash: cannot determine thread return address
please wait... (gathering kmem slab cache data)
crash: invalid kernel virtual address: 1c type: "kmem_cache
objsize/object_size"
crash: failed to read pageflag_names entry
please wait... (gathering module symbol data)
WARNING: invalid kernel module size: 0
crash: cannot determine idle task addresses from init_tasks[] or runqueues[]
crash: cannot resolve "init_task_union"
vmlinux-3.13.0-39-generic was extracted from Ubuntu ddeb:
$ file ../ddeb/usr/lib/debug/boot/vmlinux-3.13.0-39-generic
../ddeb/usr/lib/debug/boot/vmlinux-3.13.0-39-generic: ELF 64-bit LSB
executable, x86-64, version 1 (SYSV), statically linked,
BuildID[sha1]=c4fa631d2cc34a0b2628a5de01a04e81a0667555, not stripped
With -d8 I get:
...
<read_diskdump: addr: ffffffffffffffff paddr: 7fffffff cnt: 1>
read_diskdump: paddr/pfn: 7fffffff/7ffff -> cache physical page: 7ffff000
crash: invalid kernel virtual address: 0 type: "memory section"
The entire -d8 output is attached.
Bogus "base kernel version" stands out immediately and I'm pretty sure
I've seen "0.0.0" in there a couple times with exactly the same dump.
>From a quick look, the base kernel version code in kernel.c is not safe
against kt->utsname.release being all zeroes.
Eddy Gonzalo (CC'ed) can probably provide access to the dumps if
needed.
Thanks,
Ilya
11 years, 2 months
