crash version 4.0-3.17 is available
by Dave Anderson
- Two fixes for "dev -p" command option:
1) The head entry of the PCI device list was being skipped.
2) For systems with no PCI devices, exit gracefully rather than
failing the command due to the use of an invalid virtual
address.
(rachita(a)in.ibm.com, anderson(a)redhat.com)
- Fix to recognize "linux_banner" symbol type change from 'R'
to 'r' in 2.6.20-rc2 kernels. Without the patch, the session
fails during initialization with the error message " WARNING:
invalid linux_banner pointer: 756e694c", and then "crash: vmlinux
and vmcore do not match! (vgoyal(a)in.ibm.com)
- Fix to recognize "__per_cpu_start" and "__per_cpu_end" symbol
type change from 'A' to 'D' in relocatable kernels. Without
the patch, SMP kernels running on uniprocessor systems may fail
during initialization with the message "crash: cannot resolve
init_task_union". (sachinp(a)in.ibm.com)
- Fix for the xencrash "dumpinfo -t" command to properly cycle
through the ELF_timeval structures for each cpu.
(anderson(a)redhat.com)
- Fix for x86_64 backtraces that may end prematurely at either a
stale "schedule" or "schedule_timeout" reference when doing a
"bt" on an active task in a dumpfile. (anderson(a)redhat.com)
- Fix for a possible empty panic message in 2.6 kernels both during
initialization and when running the "sys" command, because of
the change of the kernel panic() string from "Kernel panic: " to
"Kernel panic -- not syncing: ". If the panic message was not
recognized in another manner, such as by an oops message, by a
kernel BUG message, or sysrq-generated crash, the "PANIC:" status
would be empty. (anderson(a)redhat.com)
Download from: http://people.redhat.com/anderson
17 years, 5 months
warning when compiling xen_hyper_command.c with "make warn" and gcc 4.1.1
by Dave Anderson
Hello Oda-san,
I just noticed this warning message when compiling xen_hyper_command.c
with gcc 4.1.1:
$ make warn
...
cc -c -g -DX86 -D_FILE_OFFSET_BITS=64 xen_hyper_command.c -Wall -Wstrict-prototypes -Wmissing-prototypes
xen_hyper_command.c: In function show_xen_hyper_dumpinfo:
xen_hyper_command.c:426: warning: value computed is not used
...
$
Here is the code it is referring to:
424 addr = (ulong)note_buf +
425 XEN_HYPER_OFFSET(ELF_Prstatus_pr_utime);
426 for (i = 0; i < 4; i++, addr+XEN_HYPER_SIZE(ELF_Timeval)) {
I'm presuming that it should be: "addr+=XEN_HYPER_SIZE(ELF_Timeval)?
Dave
17 years, 5 months
Analyzing panic dumps
by ram ba
hi all,
I am analyzing the kernel panic dumps using crash utility. Can u people suggest some of the important structures that may be helpful for analysis.
Thanks,
Ramya
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
17 years, 5 months
Re: [Crash-utility] crash: cannot resolve "init_task_union"
by Dave Anderson
Sachin,
This may require help from the IBM ppc64 people out there,
but it appears that the issue at hand has something to do
with the uniprocessor aspect.
Your vmlinux is an SMP kernel, but I'm guessing that the wrong
choice of "runq" addresses is being made below:
if (symbol_exists("per_cpu__runqueues") &&
VALID_MEMBER(runqueue_idle)) {
runqbuf = GETBUF(SIZE(runqueue));
for (i = 0; i < nr_cpus; i++) {
if ((kt->flags & SMP) && (kt->flags & PER_CPU_OFF)) {
runq = symbol_value("per_cpu__runqueues") +
kt->__per_cpu_offset[i];
} else
runq = symbol_value("per_cpu__runqueues");
readmem(runq, KVADDR, runqbuf,
SIZE(runqueue), "runqueues entry (per_cpu)",
FAULT_ON_ERROR);
tasklist[i] = ULONG(runqbuf + OFFSET(runqueue_idle));
if (IS_KVADDR(tasklist[i]))
cnt++;
I don't know what your "kt->flags" is showing at the
decision point above, but if you hack the code and force
it to select the "other" runq value, does it work OK?
When CONFIG_SMP is not configured into the kernel, then
the direct value of "per_cpu__runqueues" is used, whereas
with SMP kernels, the appropriate offset needs to be
applied. At least that's how it works (has worked?) with
the other architectures.
Dave
17 years, 6 months
RE: [Crash-utility] test results of latest 4.0-3.16.sym.patch (ia64)
by Castor Fu
Ouch. I don't really want to track down and fix bugs inside gdb. I
wonder if we can prune down the
list of symbols to something more 'useful'.
I probably won't get to this for another day or so, but was planning on
building a module
with a lot of sections to see if that's the problem, and if I can
produce it on x86...
________________________________
From: crash-utility-bounces(a)redhat.com
[mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson
Sent: Thursday, January 04, 2007 8:23 AM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] test results of latest 4.0-3.16.sym.patch
(ia64)
Hi Castor,
Another FYI re: the xrealloc() crash. The problem appears
to be specific to gdb.
I captured the "add-symbol-file" command string and saved
it in an input file. Then I brought crash up and executed
the input file, which simply passes the suspect command line
directly to gdb, and it crashes on its own:
crash> < /tmp/junk
crash> add-symbol-file
/lib/modules/2.6.18-1.2767.el5/kernel/net/ipv6/ipv6.ko
0xa00000021ed605b0 -s .exit.text 0xa00000021edb49a0 -s .rodata
0xa00000021edbd4c8 -s __ksymtab_strings 0xa00000021edbdc08 -s __versions
0xa00000021edbdf98 -s .data 0xa00000021edd6a20 -s .data.rel.ro
0xa00000021edd6c00 -s __ksymtab_gpl 0xa00000021edd6df8 -s __kcrctab_gpl
0xa00000021edd6ed8 -s .data.rel 0xa00000021edd6f48 -s .data.rel.local
0xa00000021ee39940 -s .data.rel.ro.local 0xa00000021ee3a9c0 -s
.data.read_mostly 0xa00000021ee3a9e0 -s __ksymtab 0xa00000021ee3aa60 -s
__kcrctab 0xa00000021ee3ac30 -s .gnu.linkonce.this_module
0xa00000021ee3ad80 -s .sdata 0xa00000021ee5d730 -s .bss
0xa00000021ee5b000 -s .sbss 0xa00000021ee5e8b8
add_symbol_file_command: calling xrealloc w/argcnt: 49 arg:
[0xa00000021ee5d730]...
*** glibc detected *** ./crash: realloc(): invalid next size:
0x6000000001921fe0 ***
======= Backtrace: =========
/lib/libc.so.6.1[0x20000000002f2a70]
/lib/libc.so.6.1(realloc-0x1cb0b0)[0x20000000002f5e20]
./crash(xmrealloc+0x1fffffffffee6e20)[0x40000000003a7d00]
./crash[0x40000000002ff500]
./crash[0x40000000004221e0]
./crash(cmd_func+0x1ffffffffff61610)[0x4000000000422500]
./crash(execute_command+0x1fffffffffee25f0)[0x40000000003a34f0]
./crash(gdb_command_funnel+0x1fffffffffe2feb0)[0x40000000002f0dc0]
./crash(gdb_interface+0x1fffffffffcd7590)[0x40000000001984b0]
./crash(gdb_pass_through+0x1fffffffffcd6cb0)[0x4000000000197be0]
./crash(cmd_gdb+0x2000000000151068)[0x400000000019bbc0]
./crash(exec_command+0x1fffffffffb99db0)[0x400000000005acf0]
./crash(exec_input_file+0x1fffffffffd86d40)[0x4000000000247c90]
./crash[0x400000000005b420]
./crash(exec_command+0x1fffffffffb99e50)[0x400000000005ad90]
./crash(main_loop+0x1fffffffffb9a2e0)[0x400000000005a8e0]
./crash(current_interp_command_loop+0x200000000001fd60)[0x40000000004e0c
c0]
./crash[0x40000000003199c0]
./crash[0x400000000039f370]
./crash[0x40000000003a4260]
./crash(catch_errors+0x1fffffffffee33b0)[0x40000000003a4320]
./crash[0x400000000031a930]
./crash[0x400000000039f370]
./crash[0x40000000003a4260]
./crash(catch_errors+0x1fffffffffee33b0)[0x40000000003a4320]
./crash(gdb_main+0x1fffffffffe58960)[0x40000000003198e0]
./crash(gdb_main_entry+0x1fffffffffe589f0)[0x4000000000319980]
./crash(gdb_main_loop+0x1fffffffffcd54d0)[0x4000000000196470]
./crash(main+0x1fffffffffb99820)[0x400000000005a330]
/lib/libc.so.6.1(__libc_start_main-0x2818f0)[0x200000000023f6c0]
./crash(_start+0x1fffffffffb95240)[0x4000000000056200]
======= Memory map: ========
00000000-00004000 r--p 00000000 00:00 0
2000000000000000-2000000000038000 r-xp 00000000 fd:00 10256390
/lib/ld-2.5.so
2000000000044000-2000000000050000 rw-p 00034000 fd:00 10256390
/lib/ld-2.5.so
2000000000050000-2000000000114000 r-xp 00000000 fd:00 10256405
/lib/libm-2.5.so
2000000000114000-2000000000120000 ---p 000c4000 fd:00 10256405
/lib/libm-2.5.so
2000000000120000-2000000000124000 rw-p 000c0000 fd:00 10256405
/lib/libm-2.5.so
2000000000124000-20000000001b0000 r-xp 00000000 fd:00 10883077
/usr/lib/libncurses.so.5.5
20000000001b0000-20000000001bc000 ---p 0008c000 fd:00 10883077
/usr/lib/libncurses.so.5.5
20000000001bc000-20000000001cc000 rw-p 00088000 fd:00 10883077
/usr/lib/libncurses.so.5.5
20000000001cc000-20000000001d0000 rw-p 20000000001cc000 00:00 0
20000000001d0000-20000000001d8000 r-xp 00000000 fd:00 10256403
/lib/libdl-2.5.so
20000000001d8000-20000000001e4000 ---p 00008000 fd:00 10256403
/lib/libdl-2.5.so
20000000001e4000-20000000001e8000 rw-p 00004000 fd:00 10256403
/lib/libdl-2.5.so
20000000001e8000-200000000020c000 r-xp 00000000 fd:00 10882711
/usr/lib/libz.so.1.2.3
200000000020c000-2000000000218000 ---p 00024000 fd:00 10882711
/usr/lib/libz.so.1.2.3
2000000000218000-200000000021c000 rw-p 00020000 fd:00 10882711
/usr/lib/libz.so.1.2.3
200000000021c000-2000000000480000 r-xp 00000000 fd:00 10256397
/lib/libc-2.5.so
2000000000480000-200000000048c000 ---p 00264000 fd:00 10256397
/lib/libc-2.5.so
200000000048c000-2000000000498000 rw-p 00260000 fd:00 10256397
/lib/libc-2.5.so
2000000000498000-20000000004d8000 rw-p 2000000000498000 00:00 0
20000000004d8000-2000000003c1c000 r--p 00000000 fd:00 10882710
/usr/lib/locale/locale-archive
2000000003c1c000-2000000003c2c000 rw-p 2000000003c1c000 00:00 0
2000000003c38000-2000000003c44000 r-xp 00000000 fd:00 10256427
/lib/libthread_db-1.0.so
2000000003c44000-2000000003c50000 ---p 0000c000 fd:00 10256427
/lib/libthread_db-1.0.so
2000000003c50000-2000000003c54000 rw-p 00008000 fd:00 10256427
/lib/libthread_db-1.0.so
2000000003c54000-2000000003c58000 rw-p 2000000003c54000 00:00 0
2000000003c6c000-2000000003da0000 rw-p 2000000003c6c000 00:00 0
2000000003da0000-2000000003dbc000 r-xp 00000000 fd:00 10884674
/usr/lib/libunwind.so.7.0.0
2000000003dbc000-2000000003dc8000 ---p 0001c000 fd:00 10884674
/usr/lib/libunwind.so.7.0.0
2000000003dc8000-2000000003dcc000 rw-p 00018000 fd:00 10884674
/usr/lib/libunwind.so.7.0.0
2000000003dcc000-2000000003df0000 rw-p 2000000003dcc000 00:00 0
2000000003e00000-2000000003e08000 r--s 00000000 fd:00 10977539
/usr/lib/gconv/gconv-modules.cache
2000000003e08000-2000000003e18000 rw-p 2000000003e08000 00:00 0
2000000003e1c000-2000000006edc000 rw-p 2000000003e1c000 00:00 0
2000000006ee8000-2000000006f04000 r-xp 00000000 fd:00 10256386
/lib/libgcc_s-4.1.1-20061130.so.1
2000000006f04000-2000000006f10000 ---p 0001c000 fd:00 10256386
/lib/libgcc_s-4.1.1-20061130.so.1
2000000006f10000-2000000006f14000 rw-p 00018000 fd:00 10256386
/lib/libgcc_s-4.1.1-20061130.so.1
2000000006f14000-2000000006f24000 rw-p 2000000006f14000 00:00 0
2000000008000000-2000000008024000 rw-p 2000000008000000 00:00 0
2000000008024000-200000000c000000 ---p 2000000008024000 00:00 0
4000000000000000-40000000007e0000 r-xp 00000000 fd:00 9633909
/var/tmp/crash-4.0-3.16/crash
600000000000c000-600000000006c000 rw-p 007dc000 fd:00 9633909
/var/tmp/crash-4.0-3.16/crash
600000000006c000-6000000001fc0000 rw-p 600000000006c000 00:00 0
[heap]
60000fff7fffc000-60000fff80004000 rw-p 60000fff7fffc000 00:00 0
60000ffffe068000-60000ffffe0bc000 rw-p 60000ffffe068000 00:00 0
[stack]
a000000000000000-a000000000020000 ---p 00000000 00:00 0
[vdso]
Aborted
17 years, 6 months
Can't open i386 vmcore with crash on 2.6.20-rc2 kernels
by Vivek Goyal
Hi Dave,
I can't open vmcore for 2.6.20-rc2 vanilla kernel with crash. I am using
latest crash version 4.0-3.16.
******************************************************************************
crash 4.0-3.16
Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
WARNING: invalid linux_banner pointer: 756e694c
crash: vmlinux and vmcore do not match!
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
Enter "crash -h" for details.
******************************************************************************
Looks like address of linux_banner is being interpreted wrongly.
Following are the vmcore program headers.
Program Headers:
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
NOTE 0x0000f4 0x00000000 0x00000000 0x00290 0x00290 0
LOAD 0x000384 0xc0000000 0x00000000 0xa0000 0xa0000 RWE 0
LOAD 0x0a0384 0xc0100000 0x00100000 0xf00000 0xf00000 RWE 0
LOAD 0xfa0384 0xc9000000 0x09000000 0x2f000000 0x2f000000 RWE 0
LOAD 0x2ffa0384 0xffffffff 0x38000000 0x9ffb0580 0x9ffb0580 RWE 0
LOAD 0xcff50904 0xffffffff 0x00000000 0x28000000 0x28000000 RWE 0
I am also pasting below "crash -d7 vmlinux vmcore" output.
crash 4.0-3.16
Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
vmcore_data:
flags: a0 (KDUMP_LOCAL|KDUMP_ELF32)
ndfd: 3
ofp: 456764c0
header_size: 900
num_pt_load_segments: 5
pt_load_segment[0]:
file_offset: 384
phys_start: 0
phys_end: a0000
zero_fill: 0
pt_load_segment[1]:
file_offset: a0384
phys_start: 100000
phys_end: 1000000
zero_fill: 0
pt_load_segment[2]:
file_offset: fa0384
phys_start: 9000000
phys_end: 38000000
zero_fill: 0
pt_load_segment[3]:
file_offset: 2ffa0384
phys_start: 38000000
phys_end: d7fb0580
zero_fill: 0
pt_load_segment[4]:
file_offset: cff50904
phys_start: 0
phys_end: 28000000
zero_fill: 0
elf_header: 84056f8
elf32: 84056f8
notes32: 840572c
load32: 840574c
elf64: 0
notes64: 0
load64: 0
nt_prstatus: 84057ec
nt_prpsinfo: 0
nt_taskstruct: 0
task_struct: 0
page_size: 0
switch_stack: 0
xen_kdump_data: (unused)
num_prstatus_notes: 4
nt_prstatus_percpu:
084057ec 08405890 08405934 084059d8
Elf32_Ehdr:
e_ident: \177ELF
e_ident[EI_CLASS]: 1 (ELFCLASS32)
e_ident[EI_DATA]: 1 (ELFDATA2LSB)
e_ident[EI_VERSION]: 1 (EV_CURRENT)
e_ident[EI_OSABI]: 0 (ELFOSABI_SYSV)
e_ident[EI_ABIVERSION]: 0
e_type: 4 (ET_CORE)
e_machine: 3 (EM_386)
e_version: 1 (EV_CURRENT)
e_entry: 0
e_phoff: 34
e_shoff: 0
e_flags: 0
e_ehsize: 34
e_phentsize: 20
e_phnum: 6
e_shentsize: 0
e_shnum: 0
e_shstrndx: 0
Elf32_Phdr:
p_type: 4 (PT_NOTE)
p_offset: 244 (f4)
p_vaddr: 0
p_paddr: 0
p_filesz: 656 (290)
p_memsz: 656 (290)
p_flags: 0 ()
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 900 (384)
p_vaddr: c0000000
p_paddr: 0
p_filesz: 655360 (a0000)
p_memsz: 655360 (a0000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 656260 (a0384)
p_vaddr: c0100000
p_paddr: 100000
p_filesz: 15728640 (f00000)
p_memsz: 15728640 (f00000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 16384900 (fa0384)
p_vaddr: c9000000
p_paddr: 9000000
p_filesz: 788529152 (2f000000)
p_memsz: 788529152 (2f000000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 804914052 (2ffa0384)
p_vaddr: ffffffff
p_paddr: 38000000
p_filesz: 2684028288 (9ffb0580)
p_memsz: 2684028288 (9ffb0580)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: -806024956 (cff50904)
p_vaddr: ffffffff
p_paddr: 0
p_filesz: 671088640 (28000000)
p_memsz: 671088640 (28000000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 144
n_type: 1 (NT_PRSTATUS)
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 c058e008 00000020
00000000 f70b007b 0000007b 00000000
000000d8 00000000 c0103240 00000060
00000246 c058ffc0 00000068 00000000
Elf32_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 144
n_type: 1 (NT_PRSTATUS)
00000000 00000000 00000000 00000000
00000000 00000000 00000da5 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 c053c240 c0527930
00000000 00000000 00000063 00000000
00000000 0000007b 0000007b 00000000
f618df30 c053c240 c013e8cf 00000060
00000046 f618df08 00000068 00000000
Elf32_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 144
n_type: 1 (NT_PRSTATUS)
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 cb124008 00000000
00000000 f77c007b 0000007b 00000000
000000d8 00000000 c0103240 00000060
00000246 cb125fa4 00000068 00000000
Elf32_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 144
n_type: 1 (NT_PRSTATUS)
00000000 00000000 00000000 00000000
00000000 00000000 00000ade 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 000000fa 003d0900
00000000 00000998 f75bbf6c f75bbf40
cb02aa00 0000007b f70b007b 00000000
f7e800d8 cb02aa00 c040dac7 00000060
00000046 f75bbed4 00000068 00000000
gdb vmlinux
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
<readmem: c0413a20, KVADDR, "kernel_config_data", 32768, (ROE), 8fa1020>
crash: CONFIG_NR_CPUS: 32
crash: CONFIG_HZ: 250
<readmem: c0623d00, KVADDR, "xtime", 8, (FOE), 83be098>
<readmem: c0524344, KVADDR, "init_uts_ns", 390, (ROE), 83be67c>
<readmem: c0412000, KVADDR, "linux_banner", 4, (FOE), bfee229c>
WARNING: invalid linux_banner pointer: 756e694c
<readmem: 756e694c, KVADDR, "accessible check", 4, (ROE|Q), bfee20d4>
crash: invalid kernel virtual address: 756e694c type: "accessible check"
crash: vmlinux and vmcore do not match!
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
Enter "crash -h" for details.
Thanks
Vivek
17 years, 6 months
crash: cannot resolve "init_task_union"
by Sachin P. Sant
Hi Dave,
I am facing some problem while running crash on a live system. The machine
is a uniprocessor machine [ Arch is powerpc ]
Crash fails with error crash: cannot resolve "init_task_union"
Some debugging shows that crash fails to read idle tasks related
information from per cpu runqueues data structure. The per cpu
runqueue dosen't seems to be having proper values.
Breakpoint 2, get_idle_threads (tasklist=0x106bd3f0, nr_cpus=1) at task.c:5189
5189 BZERO(tasklist, sizeof(ulong) * NR_CPUS);
(gdb) n
....
....
(gdb) n
5206 tasklist[i] = ULONG(runqbuf + OFFSET(runqueue_idle));
(gdb) n
5208 if (IS_KVADDR(tasklist[i]))
(gdb) print /x tasklist[i]
$1 = 0xd00000000
(gdb)
I am attaching crash -d4 output here.
Thanks
-Sachin
17 years, 6 months
RE: [Crash-utility] crash-4.0-3.14-sym.3.patch (was: modules and data/bss initialization)
by Castor Fu
Oops, I sent the wrong patch...
I had found the second problem when reading
through the patch, but somehow grabbed the wrong file. The 'mod -l'
thing
seems like it'll take more to fix, but there's another problem which I
found and
had fixed.
Sections which are not '.text' '.data' or '.bss' will get loaded at the
wrong
addresses, which is unfortunate. The following patch addresses this
by providing addreses for each section.
Happy New Year!
-castor
________________________________
From: crash-utility-bounces(a)redhat.com
[mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson
Sent: Tuesday, January 02, 2007 1:04 PM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] crash-4.0-3.14-sym.3.patch (was: modules
and data/bss initialization)
Castor Fu wrote:
Hi Dave!
Sorry about the regression.
I looked further through the code, and have modified it to only
add the symbols
for which it has actually identified a matching section, and to
do it in a way
which supports whatever sections are defined.
This doesn't explain why a match isn't being found in the ia64
case, but shouldat least avoid the regression in the s390x case.I've
also attached a test which one could run which dumps output which
mighthelp me fix things by tracing through loading the 'md5' module. By
default the new code is off, as requested, and can be forced witha '-l'
option to 'mod', e.g.crash> mod -l -s md5Thanks again for the testing,
and hopefully this will work for most people.It would be great if
someone could send me output from the rarerplatforms like ppc64 or 390
if it doesn't work.Happy New Year! -castor
Somebody forgot to QA this patch... ;-)
Unfortunately, "mod -l -s module-name" won't work, nor will
"mod -l -S", because of the unique way that cmd_mod() handles
-s and -S -- which can optionally take additional "objfile" or
"directory"
arguments respectively:
crash> mod -l -S
mod: -S is not a directory
Usage: mod [ -s module [objfile] | -d module | -S [directory] | -D | -r
]
Enter "help mod" for details.
crash> mod -l -s nfs
mod: /usr/lib/debug/lib/modules/2.4.21-37.ELsmp/kernel/fs/nfs: not an
ELF format object file
Usage: mod [ -s module [objfile] | -d module | -S [directory] | -D | -r
]
Enter "help mod" for details.
crash>
So to get around that, you can just enter "mod -l" alone
prior to any -s or -S commands, which will just set the
flag.
But having done that, it actually does just the opposite,
because of backwards logic here:
@@ -7288,7 +7410,10 @@
strcpy(lm->mod_namelist, namelist);
else
strncpy(lm->mod_namelist, namelist,
MAX_MOD_NAMELIST-1);
- goto add_symbols;
+ if (USE_V2_MOD_SYM()) {
+ goto add_symbols;
+ }
+
}
if ((mbfd = bfd_openr(namelist, NULL)) == NULL)
It should be: "if (!USE_V2_MOD_SYM())".
Anyway, don't bother updating the patch for the above,
because I can keep testing with the two work-arounds.
I'd also appreciate any ppc64, s390 and s390x testers
out there...
Thanks,
Dave
17 years, 6 months
RE: [Crash-utility] crash-4.0-3.14-sym.3.patch (was: modules and data /bss initialization)
by Castor Fu
Hi Dave!
Sorry about the regression.
I looked further through the code, and have modified it to only add the
symbols
for which it has actually identified a matching section, and to do it in
a way
which supports whatever sections are defined.
This doesn't explain why a match isn't being found in the ia64 case, but
should
at least avoid the regression in the s390x case.
I've also attached a test which one could run which dumps output which
might
help me fix things by tracing through loading the 'md5' module.
By default the new code is off, as requested, and can be forced with
a '-l' option to 'mod', e.g.
crash> mod -l -s md5
Thanks again for the testing, and hopefully this will work for most
people.
It would be great if someone could send me output from the rarer
platforms like ppc64 or 390 if it doesn't work.
Happy New Year!
-castor
________________________________
From: crash-utility-bounces(a)redhat.com
[mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson
Sent: Tuesday, December 19, 2006 7:44 AM
To: crash-utility(a)redhat.com; Castor Fu
Subject: [Crash-utility] crash-4.0-3.14-sym.3.patch (was: modules and
data /bss initialization)
Hi Castor,
While I have had some success with this patch on x86 and
x86_64 machines, I haven't been so fortunate with other
architectures.
On ia64 machines for example, it quite often is unable
to determine the .data segment address of a module, leaving
it as 0, so there is no improvement over the current way it works.
On the s390x, it actually causes a pretty serious regression.
For example without the patch, here are the data addresses
of the ext3 module before and after it gets loaded:
$ /usr/bin/crash -s vmlinux-2.6.18-1.2767.el5
crash> sym -m ext3 | grep "([dD])"
208bfea0 (d) ext3_dir_operations
208bff88 (d) ext3_file_operations
208c0070 (d) ext3_ordered_aops
208c00e0 (d) ext3_writeback_aops
208c0150 (d) ext3_journalled_aops
208c01c0 (d) ext3_xattr_handler_map
208c01f8 (d) ext3_file_inode_operations
208c02a0 (d) ext3_dir_inode_operations
208c0348 (d) ext3_special_inode_operations
208c03f0 (d) ext3_fs_type
208c0430 (d) ext3_sops
208c04d8 (d) ext3_export_ops
208c0508 (d) ext3_qctl_operations
208c0560 (d) ext3_quota_operations
208c05c0 (d) ext3_symlink_inode_operations
208c0668 (d) ext3_fast_symlink_inode_operations
208c0710 (d) ext3_xattr_handlers
208c0740 (d) tokens
208c0a40 (d) ext3_xattr_user_handler
208c0a60 (d) ext3_xattr_trusted_handler
208c0a80 (d) ext3_xattr_acl_access_handler
208c0aa0 (d) ext3_xattr_acl_default_handler
208c0ac0 (d) ext3_xattr_security_handler
208c0b00 (d) __this_module
crash> mod -s ext3
MODULE NAME SIZE OBJECT FILE
208c0b00 ext3 200208
/lib/modules/2.6.18-1.2767.el5/kernel/fs/ext3/ext3.ko
crash> sym -m ext3 | grep "([dD])"
208bfea0 (d) ext3_dir_operations
208bff88 (d) ext3_file_operations
208c0070 (d) ext3_ordered_aops
208c00e0 (d) ext3_writeback_aops
208c0150 (d) ext3_journalled_aops
208c01c0 (d) ext3_xattr_handler_map
208c01f8 (d) ext3_file_inode_operations
208c02a0 (d) ext3_dir_inode_operations
208c0348 (d) ext3_special_inode_operations
208c03f0 (d) ext3_fs_type
208c0430 (d) ext3_sops
208c04d8 (d) ext3_export_ops
208c0508 (d) ext3_qctl_operations
208c0560 (d) ext3_quota_operations
208c05c0 (d) ext3_symlink_inode_operations
208c0668 (d) ext3_fast_symlink_inode_operations
208c0710 (d) ext3_xattr_handlers
208c0740 (d) tokens
208c0a40 (d) ext3_xattr_user_handler
208c0a60 (d) ext3_xattr_trusted_handler
208c0a80 (d) ext3_xattr_acl_access_handler
208c0aa0 (d) ext3_xattr_acl_default_handler
208c0ac0 (d) ext3_xattr_security_handler
208c0b00 (d) __this_module
crash>
That looks fine...
However, with the patch applied, I get the following behavior:
$ crash*14/crash -s vmlinux-2.6.18-1.2767.el5
crash> sym -m ext3 | grep "([dD])"
208bfea0 (d) ext3_dir_operations
208bff88 (d) ext3_file_operations
208c0070 (d) ext3_ordered_aops
208c00e0 (d) ext3_writeback_aops
208c0150 (d) ext3_journalled_aops
208c01c0 (d) ext3_xattr_handler_map
208c01f8 (d) ext3_file_inode_operations
208c02a0 (d) ext3_dir_inode_operations
208c0348 (d) ext3_special_inode_operations
208c03f0 (d) ext3_fs_type
208c0430 (d) ext3_sops
208c04d8 (d) ext3_export_ops
208c0508 (d) ext3_qctl_operations
208c0560 (d) ext3_quota_operations
208c05c0 (d) ext3_symlink_inode_operations
208c0668 (d) ext3_fast_symlink_inode_operations
208c0710 (d) ext3_xattr_handlers
208c0740 (d) tokens
208c0a40 (d) ext3_xattr_user_handler
208c0a60 (d) ext3_xattr_trusted_handler
208c0a80 (d) ext3_xattr_acl_access_handler
208c0aa0 (d) ext3_xattr_acl_default_handler
208c0ac0 (d) ext3_xattr_security_handler
208c0b00 (d) __this_module
crash> mod -s ext3
MODULE NAME SIZE OBJECT FILE
208c0b00 ext3 200208
/lib/modules/2.6.18-1.2767.el5/kernel/fs/ext3/ext3.ko
crash> sym -m ext3 | grep "([dD])"
0 (D) __this_module
0 (D) ext3_dir_operations
0 (D) ext3_file_inode_operations
0 (d) tokens
a8 (D) ext3_dir_inode_operations
e8 (D) ext3_file_operations
150 (D) ext3_special_inode_operations
1d0 (d) ext3_ordered_aops
1f8 (d) ext3_fs_type
238 (d) ext3_sops
240 (d) ext3_writeback_aops
2b0 (d) ext3_journalled_aops
2e0 (d) ext3_export_ops
300 (D) ext3_xattr_user_handler
310 (d) ext3_qctl_operations
320 (d) ext3_xattr_handler_map
320 (D) ext3_xattr_trusted_handler
340 (D) ext3_xattr_acl_access_handler
360 (D) ext3_xattr_acl_default_handler
368 (d) ext3_quota_operations
380 (D) ext3_xattr_security_handler
3c8 (D) ext3_symlink_inode_operations
470 (D) ext3_fast_symlink_inode_operations
518 (D) ext3_xattr_handlers
crash>
I haven't been able to test it on a ppc64 or s390. (That's
why I asked for help from others on the mailing list to test
it out, but I didn't get any takers...)
In any case, as much as I like what the patch wants to do,
I can't accept it given such a serious regression. Perhaps
it could be made an experimental run-time option that could
be turned on prior to doing any "mod" commands, leaving the
current behaviour in place by default?
Thanks,
Dave
17 years, 6 months
