2:57 p.m.
I forgot that I had recompiled the kernel between my first mail and your
answer, so here's an up to date dump with the current kernel and last
dump done with it:
/var/crash/201008232109$ crash -d7 /usr/src/linux-2.6.35.3/vmlinux
vmcore.201008232109
crash 5.0.6
Copyright (C) 2002-2010 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
vmcore_data:
flags: a0 (KDUMP_LOCAL|KDUMP_ELF32)
ndfd: 3
ofp: b782a4c0
header_size: 1844
num_pt_load_segments: 9
pt_load_segment[0]:
file_offset: 734
phys_start: 0
phys_end: a0000
zero_fill: 0
pt_load_segment[1]:
file_offset: a0734
phys_start: 100000
phys_end: 1000000
zero_fill: 0
pt_load_segment[2]:
file_offset: fa0734
phys_start: 5000000
phys_end: 38000000
zero_fill: 0
pt_load_segment[3]:
file_offset: 33fa0734
phys_start: 38000000
phys_end: 3e628000
zero_fill: 0
pt_load_segment[4]:
file_offset: 3a5c8734
phys_start: 3e6db000
phys_end: 3f594000
zero_fill: 0
pt_load_segment[5]:
file_offset: 3b481734
phys_start: 3f59c000
phys_end: 3f62a000
zero_fill: 0
pt_load_segment[6]:
file_offset: 3b50f734
phys_start: 3f62e000
phys_end: 3f6a9000
zero_fill: 0
pt_load_segment[7]:
file_offset: 3b58a734
phys_start: 3f6e9000
phys_end: 3f6ed000
zero_fill: 0
pt_load_segment[8]:
file_offset: 3b58e734
phys_start: 3f6ff000
phys_end: 3f700000
zero_fill: 0
elf_header: 85368c0
elf32: 85368c0
notes32: 85368f4
load32: 8536914
elf64: 0
notes64: 0
load64: 0
nt_prstatus: 8536a34
nt_prpsinfo: 0
nt_taskstruct: 0
task_struct: 0
page_size: 0
switch_stack: 0
xen_kdump_data: (unused)
num_prstatus_notes: 2
vmcoreinfo: 0
size_vmcoreinfo: 0
nt_prstatus_percpu:
08536a34 08536ad8
Elf32_Ehdr:
e_ident: \177ELF
e_ident[EI_CLASS]: 1 (ELFCLASS32)
e_ident[EI_DATA]: 1 (ELFDATA2LSB)
e_ident[EI_VERSION]: 1 (EV_CURRENT)
e_ident[EI_OSABI]: 0 (ELFOSABI_SYSV)
e_ident[EI_ABIVERSION]: 0
e_type: 4 (ET_CORE)
e_machine: 3 (EM_386)
e_version: 1 (EV_CURRENT)
e_entry: 0
e_phoff: 34
e_shoff: 0
e_flags: 0
e_ehsize: 34
e_phentsize: 20
e_phnum: a
e_shentsize: 0
e_shnum: 0
e_shstrndx: 0
Elf32_Phdr:
p_type: 4 (PT_NOTE)
p_offset: 372 (174)
p_vaddr: 0
p_paddr: 0
p_filesz: 1472 (5c0)
p_memsz: 1472 (5c0)
p_flags: 0 ()
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 1844 (734)
p_vaddr: c0000000
p_paddr: 0
p_filesz: 655360 (a0000)
p_memsz: 655360 (a0000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 657204 (a0734)
p_vaddr: c0100000
p_paddr: 100000
p_filesz: 15728640 (f00000)
p_memsz: 15728640 (f00000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 16385844 (fa0734)
p_vaddr: c5000000
p_paddr: 5000000
p_filesz: 855638016 (33000000)
p_memsz: 855638016 (33000000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 872023860 (33fa0734)
p_vaddr: ffffffff
p_paddr: 38000000
p_filesz: 107118592 (6628000)
p_memsz: 107118592 (6628000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 979142452 (3a5c8734)
p_vaddr: ffffffff
p_paddr: 3e6db000
p_filesz: 15437824 (eb9000)
p_memsz: 15437824 (eb9000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 994580276 (3b481734)
p_vaddr: ffffffff
p_paddr: 3f59c000
p_filesz: 581632 (8e000)
p_memsz: 581632 (8e000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 995161908 (3b50f734)
p_vaddr: ffffffff
p_paddr: 3f62e000
p_filesz: 503808 (7b000)
p_memsz: 503808 (7b000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 995665716 (3b58a734)
p_vaddr: ffffffff
p_paddr: 3f6e9000
p_filesz: 16384 (4000)
p_memsz: 16384 (4000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 995682100 (3b58e734)
p_vaddr: ffffffff
p_paddr: 3f6ff000
p_filesz: 4096 (1000)
p_memsz: 4096 (1000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf32_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 144
n_type: 1 (NT_PRSTATUS)
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 369c6c02 00000040
f70a650c f70a6000 f8098000 00000000
00000000 0000245c 00000000 00000000
00000000 00000040 f70a650c 00000000
c074dee4 c074dfa4 00000068 00000000
Elf32_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 144
n_type: 1 (NT_PRSTATUS)
00000000 00000000 00000000 00000000
00000000 00000000 00000a9e 00000000
00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000
00000000 00000000 c077b6c0 00000303
00000000 00000063 00000000 f5a0ff0c
00000063 0000007b 0000007b 000000d8
00000033 ffffffff c0377692 00000060
00010086 f5a0ff0c 00000068 00000000
Elf32_Nhdr:
n_namesz: 11 ("VMCOREINFO")
n_descsz: 1117
n_type: 0 (unused)
OSRELEASE=2.6.35.3-saber
PAGESIZE=4096
SYMBOL(init_uts_ns)=c0762120
SYMBOL(node_online_map)=c0799944
SYMBOL(swapper_pg_dir)=c0816000
SYMBOL(_stext)=c0101000
SYMBOL(vmlist)=c083ed3c
SYMBOL(mem_map)=c083ed00
SYMBOL(contig_page_data)=c0796180
SIZE(page)=32
SIZE(pglist_data)=4224
SIZE(zone)=1024
SIZE(free_area)=44
SIZE(list_head)=8
SIZE(nodemask_t)=4
OFFSET(page.flags)=0
OFFSET(page._count)=4
OFFSET(page.mapping)=16
OFFSET(page.lru)=24
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=4140
OFFSET(pglist_data.node_mem_map)=4144
OFFSET(pglist_data.node_start_pfn)=4148
OFFSET(pglist_data.node_spanned_pages)=4156
OFFSET(pglist_data.node_id)=4160
OFFSET(zone.free_area)=40
OFFSET(zone.vm_stat)=728
OFFSET(zone.spanned_pages)=916
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=4
OFFSET(vm_struct.addr)=4
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=c07656bc
SYMBOL(log_end)=c08247ec
SYMBOL(log_buf_len)=c07656b8
SYMBOL(logged_chars)=c082c8a0
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=11
NUMBER(PG_swapcache)=16
CRASHTIME=1282590563
cannot determine relocation value: not a live system
gdb /usr/src/linux-2.6.35.3/vmlinux
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
<readmem: c14ea900, KVADDR, "kernel_config_data", 32768, (ROE), 8bdda28>
crash: read error: kernel virtual address: c14ea900 type:
"kernel_config_data"
WARNING: cannot read kernel_config_data
<readmem: c14e7d88, KVADDR, "cpu_possible_mask", 4, (FOE), bf8cf28c>
crash: read error: kernel virtual address: c14e7d88 type:
"cpu_possible_mask"
And here's what you asked with --minimal:
crash> log
<readmem: c16656b8, KVADDR, "log_buf_len", 4, (FOE), bfa8f594>
log: read error: kernel virtual address: c16656b8 type: "log_buf_len"
crash> sym cpu_possible_mask
c14e7d88 (R) cpu_possible_mask
crash> sym kernel_config_data
c14ea900 (r) kernel_config_data
crash> sym -l
[...]
c175d4d0 (B) rpc_debug
c175d4d4 (B) nfs_debug
c175d4d8 (B) nfsd_debug
c175d4dc (B) nlm_debug
c175d4e0 (b) sunrpc_table_header
c175d4e4 (B) sctp_assocs_id_lock
c175d4e8 (B) proc_net_sctp
c175d4ec (B) sctp_assocs_id
c175d500 (B) sysctl_sctp_mem
c175d50c (B) sysctl_sctp_rmem
c175d518 (B) sysctl_sctp_wmem
c175d524 (b) __key.46606
c175d524 (b) sctp_ctl_sock
c175d528 (b) sctp_pf_inet6_specific
c175d52c (b) sctp_pf_inet_specific
c175d530 (b) sctp_af_v4_specific
c175d534 (b) sctp_af_v6_specific
c175d538 (b) __key.44408
c175d538 (b) sctp_rand.42824
c175d53c (B) sctp_sockets_allocated
c175d54c (b) sctp_memory_pressure
c175d550 (b) sctp_memory_allocated
c175d554 (b) sctp_sysctl_header
c175d558 (b) zero
c175d55c (b) klist_remove_lock
c175d560 (B) __bss_stop
c175e000 (b) .brk
c175e000 (B) __brk_base
c176e000 (b) .brk.pagetables
c17ee000 (b) .brk.dmi_alloc
c17fe000 (B) __brk_limit
c17fe000 (A) _end