10:52 p.m.
On Mon, Sep 10, 2007 at 11:35:21AM -0700, Randy Dunlap wrote:
On Fri, 7 Sep 2007 17:57:46 +0900 Ken'ichi Ohmichi wrote:
> Hi,
> I released a new makedumpfile (version 1.2.0) with vmcoreinfo support.
> I updated the patches for linux and kexec-tools.
>
> PATCH SET:
> [1/2] [linux-2.6.22] Add vmcoreinfo
> The patch is for linux-2.6.22.
> The patch adds the vmcoreinfo data. Its address and size are output
> to /sys/kernel/vmcoreinfo.
>
> [2/2] [kexec-tools] Pass vmcoreinfo's address and size
> The patch is for kexec-tools-testing-20070330.
> (http://www.kernel.org/pub/linux/kernel/people/horms/kexec-tools/)
> kexec command gets the address and size of the vmcoreinfo data from
> /sys/kernel/vmcoreinfo, and passes them to the second kernel through
> ELF header of /proc/vmcore. When the second kernel is booting, the
> kernel gets them from the ELF header and creates vmcoreinfo's PT_NOTE
> segment into /proc/vmcore.
Hi,
When using the vmcoreinfo patches, what tool(s) are available for
analyzing the vmcore (dump) file? E.g., lkcd or crash or just gdb?
gdb works for me, but I tried to use crash (4.0-4.6 from
http://people.redhat.com/anderson/) and crash complained:
crash: invalid kernel virtual address: 0 type: "cpu_pda entry"
Should crash work, or does it need to be modified?
Hi Randy,
Crash should just work. It might broken on latest kernel. Copying it
to crash-utility mailing list. Dave will be able to tell us better.
This is on a 2.6.23-rc3 kernel with vmcoreinfo patches and a dump
file
with -l 31 (dump level 31, omitting all possible pages).
Thanks
Vivek
11:45 p.m.
New subject: [PATCH 0/2] vmcoreinfo support for dump filtering #2
Vivek Goyal wrote:
How do I subscribe to the kexec mailing list on lists.infradead.org?
I tried majordomo, as on vger, but it didn't work.
-piet
On Mon, Sep 10, 2007 at 11:35:21AM -0700, Randy Dunlap wrote:
> On Fri, 7 Sep 2007 17:57:46 +0900 Ken'ichi Ohmichi wrote:
>
>
>> Hi,
>>
>> I released a new makedumpfile (version 1.2.0) with vmcoreinfo support.
>> I updated the patches for linux and kexec-tools.
>>
>> PATCH SET:
>> [1/2] [linux-2.6.22] Add vmcoreinfo
>> The patch is for linux-2.6.22.
>> The patch adds the vmcoreinfo data. Its address and size are output
>> to /sys/kernel/vmcoreinfo.
>>
>> [2/2] [kexec-tools] Pass vmcoreinfo's address and size
>> The patch is for kexec-tools-testing-20070330.
>> (http://www.kernel.org/pub/linux/kernel/people/horms/kexec-tools/)
>> kexec command gets the address and size of the vmcoreinfo data from
>> /sys/kernel/vmcoreinfo, and passes them to the second kernel through
>> ELF header of /proc/vmcore. When the second kernel is booting, the
>> kernel gets them from the ELF header and creates vmcoreinfo's PT_NOTE
>> segment into /proc/vmcore.
>>
> Hi,
> When using the vmcoreinfo patches, what tool(s) are available for
> analyzing the vmcore (dump) file? E.g., lkcd or crash or just gdb?
>
> gdb works for me, but I tried to use crash (4.0-4.6 from
> http://people.redhat.com/anderson/) and crash complained:
>
> crash: invalid kernel virtual address: 0 type: "cpu_pda entry"
>
> Should crash work, or does it need to be modified?
>
>
Hi Randy,
Crash should just work. It might broken on latest kernel. Copying it
to crash-utility mailing list. Dave will be able to tell us better.
> This is on a 2.6.23-rc3 kernel with vmcoreinfo patches and a dump file
> with -l 31 (dump level 31, omitting all possible pages).
>
>
Thanks
Vivek
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
12:23 a.m.
New subject: [PATCH 0/2] vmcoreinfo support for dump filtering #2
On Mon, 10 Sep 2007 21:45:43 -0700 Piet Delaney wrote:
Vivek Goyal wrote:
How do I subscribe to the kexec mailing list on lists.infradead.org?
I tried majordomo, as on vger, but it didn't work.
See http://lists.infradead.org/mailman/listinfo/kexec
---
~Randy
12:58 a.m.
New subject: [PATCH 0/2] vmcoreinfo support for dump filtering #2
On Mon, Sep 10, 2007 at 09:45:43PM -0700, Piet Delaney wrote:
Vivek Goyal wrote:
How do I subscribe to the kexec mailing list on lists.infradead.org?
I tried majordomo, as on vger, but it didn't work.
To unsubscribe, please go here:
http://lists.infradead.org/mailman/admindb/kexec
If you have any problems please feel free to email me.
--
Horms
H: http://www.vergenet.net/~horms/
W: http://www.valinux.co.jp/en/
9:03 a.m.
New subject: [PATCH 0/2] vmcoreinfo support for dump filtering #2
Vivek Goyal wrote:
On Mon, Sep 10, 2007 at 11:35:21AM -0700, Randy Dunlap wrote:
>On Fri, 7 Sep 2007 17:57:46 +0900 Ken'ichi Ohmichi wrote:
>
>
>>Hi,
>
>>I released a new makedumpfile (version 1.2.0) with vmcoreinfo support.
>>I updated the patches for linux and kexec-tools.
>>
>>PATCH SET:
>>[1/2] [linux-2.6.22] Add vmcoreinfo
>> The patch is for linux-2.6.22.
>> The patch adds the vmcoreinfo data. Its address and size are output
>> to /sys/kernel/vmcoreinfo.
>>
>>[2/2] [kexec-tools] Pass vmcoreinfo's address and size
>> The patch is for kexec-tools-testing-20070330.
>> (http://www.kernel.org/pub/linux/kernel/people/horms/kexec-tools/)
>> kexec command gets the address and size of the vmcoreinfo data from
>> /sys/kernel/vmcoreinfo, and passes them to the second kernel through
>> ELF header of /proc/vmcore. When the second kernel is booting, the
>> kernel gets them from the ELF header and creates vmcoreinfo's PT_NOTE
>> segment into /proc/vmcore.
>
>Hi,
>When using the vmcoreinfo patches, what tool(s) are available for
>analyzing the vmcore (dump) file? E.g., lkcd or crash or just gdb?
>
>gdb works for me, but I tried to use crash (4.0-4.6 from
>http://people.redhat.com/anderson/) and crash complained:
>
>crash: invalid kernel virtual address: 0 type: "cpu_pda entry"
>
>Should crash work, or does it need to be modified?
>
Hi Randy,
Crash should just work. It might broken on latest kernel. Copying it
to crash-utility mailing list. Dave will be able to tell us better.
>This is on a 2.6.23-rc3 kernel with vmcoreinfo patches and a dump file
>with -l 31 (dump level 31, omitting all possible pages).
There's always the possibility that something crucial (to the crash
utility) has changed in the upstream kernel; that's just the nature
of the beast.
In this case, crash is reading this set of per-cpu pointers:
struct x8664_pda *_cpu_pda[NR_CPUS] __read_mostly;
and for each one, it then reads the x8664_pda data structure
that it points to -- but finds a NULL. It's possible that it
has incorrectly determined the number of x8664_pda structures
(cpus) that exist. Or less likely, the array contents were read
as zeroes from the dumpfile.
Anyway, with any initialization-time failure, it's usually helpful
to invoke crash with the "-d7" (debug level) argument, as in:
$ crash -d7 vmlinux vmcore
That will display information re: every read made to the dumpfile.
In this case, normally you would see, for each cpu, a read of the
individual 8-byte address from the array, and then based upon what
it read, the subsequent read of the whole 128-byte data structure:
<readmem: ffffffff8042d9c0, KVADDR, "_cpu_pda addr", 8, (FOE),
7fbffff210>
<readmem: ffffffff80406000, KVADDR, "cpu_pda entry", 128, (FOE), 937680>
CPU0: level4_pgt: 200000010 data_offset: ffff8100899c1000
<readmem: ffffffff8042d9c8, KVADDR, "_cpu_pda addr", 8, (FOE),
7fbffff210>
<readmem: ffff81003ff027c0, KVADDR, "cpu_pda entry", 128, (FOE), 937680>
CPU1: level4_pgt: 200000010 data_offset: ffff8100899c9000
<readmem: ffffffff8042d9d0, KVADDR, "_cpu_pda addr", 8, (FOE),
7fbffff210>
<readmem: ffff81003ff19e40, KVADDR, "cpu_pda entry", 128, (FOE), 937680>
CPU2: level4_pgt: 200000010 data_offset: ffff8100899d1000
<readmem: ffffffff8042d9d8, KVADDR, "_cpu_pda addr", 8, (FOE),
7fbffff210>
<readmem: ffff81003ff19640, KVADDR, "cpu_pda entry", 128, (FOE), 937680>
CPU3: level4_pgt: 200000010 data_offset: ffff8100899d9000
<readmem: ffffffff8042d9e0, KVADDR, "_cpu_pda addr", 8, (FOE),
7fbffff210>
<readmem: ffffffff80406200, KVADDR, "cpu_pda entry", 128, (FOE), 937680>
From that data structure it grabs the level4_pgt and data_offset
fields for subsequent use. So in your case, it should show how
many (if any) of the x8664_pda structures it read before encountering
a NULL pointer in one of the array entries.
Dave
11:25 a.m.
New subject: [PATCH 0/2] vmcoreinfo support for dump filtering #2
On Tue, 11 Sep 2007 10:03:43 -0400 Dave Anderson wrote:
Vivek Goyal wrote:
> On Mon, Sep 10, 2007 at 11:35:21AM -0700, Randy Dunlap wrote:
>
>>On Fri, 7 Sep 2007 17:57:46 +0900 Ken'ichi Ohmichi wrote:
>>
>>
>>>Hi,
>>
>>>I released a new makedumpfile (version 1.2.0) with vmcoreinfo support.
>>>I updated the patches for linux and kexec-tools.
>>>
>>>PATCH SET:
>>>[1/2] [linux-2.6.22] Add vmcoreinfo
>>> The patch is for linux-2.6.22.
>>> The patch adds the vmcoreinfo data. Its address and size are output
>>> to /sys/kernel/vmcoreinfo.
>>>
>>>[2/2] [kexec-tools] Pass vmcoreinfo's address and size
>>> The patch is for kexec-tools-testing-20070330.
>>> (http://www.kernel.org/pub/linux/kernel/people/horms/kexec-tools/)
>>> kexec command gets the address and size of the vmcoreinfo data from
>>> /sys/kernel/vmcoreinfo, and passes them to the second kernel through
>>> ELF header of /proc/vmcore. When the second kernel is booting, the
>>> kernel gets them from the ELF header and creates vmcoreinfo's PT_NOTE
>>> segment into /proc/vmcore.
>>
>>Hi,
>>When using the vmcoreinfo patches, what tool(s) are available for
>>analyzing the vmcore (dump) file? E.g., lkcd or crash or just gdb?
>>
>>gdb works for me, but I tried to use crash (4.0-4.6 from
>>http://people.redhat.com/anderson/) and crash complained:
>>
>>crash: invalid kernel virtual address: 0 type: "cpu_pda entry"
>>
>>Should crash work, or does it need to be modified?
>>
>
>
> Hi Randy,
>
> Crash should just work. It might broken on latest kernel. Copying it
> to crash-utility mailing list. Dave will be able to tell us better.
>
>
>>This is on a 2.6.23-rc3 kernel with vmcoreinfo patches and a dump file
>>with -l 31 (dump level 31, omitting all possible pages).
There's always the possibility that something crucial (to the crash
utility) has changed in the upstream kernel; that's just the nature
of the beast.
In this case, crash is reading this set of per-cpu pointers:
struct x8664_pda *_cpu_pda[NR_CPUS] __read_mostly;
and for each one, it then reads the x8664_pda data structure
that it points to -- but finds a NULL. It's possible that it
has incorrectly determined the number of x8664_pda structures
(cpus) that exist. Or less likely, the array contents were read
as zeroes from the dumpfile.
Anyway, with any initialization-time failure, it's usually helpful
to invoke crash with the "-d7" (debug level) argument, as in:
$ crash -d7 vmlinux vmcore
That will display information re: every read made to the dumpfile.
In this case, normally you would see, for each cpu, a read of the
individual 8-byte address from the array, and then based upon what
it read, the subsequent read of the whole 128-byte data structure:
<readmem: ffffffff8042d9c0, KVADDR, "_cpu_pda addr", 8, (FOE),
7fbffff210>
<readmem: ffffffff80406000, KVADDR, "cpu_pda entry", 128, (FOE), 937680>
CPU0: level4_pgt: 200000010 data_offset: ffff8100899c1000
<readmem: ffffffff8042d9c8, KVADDR, "_cpu_pda addr", 8, (FOE),
7fbffff210>
<readmem: ffff81003ff027c0, KVADDR, "cpu_pda entry", 128, (FOE), 937680>
CPU1: level4_pgt: 200000010 data_offset: ffff8100899c9000
<readmem: ffffffff8042d9d0, KVADDR, "_cpu_pda addr", 8, (FOE),
7fbffff210>
<readmem: ffff81003ff19e40, KVADDR, "cpu_pda entry", 128, (FOE), 937680>
CPU2: level4_pgt: 200000010 data_offset: ffff8100899d1000
<readmem: ffffffff8042d9d8, KVADDR, "_cpu_pda addr", 8, (FOE),
7fbffff210>
<readmem: ffff81003ff19640, KVADDR, "cpu_pda entry", 128, (FOE), 937680>
CPU3: level4_pgt: 200000010 data_offset: ffff8100899d9000
<readmem: ffffffff8042d9e0, KVADDR, "_cpu_pda addr", 8, (FOE),
7fbffff210>
<readmem: ffffffff80406200, KVADDR, "cpu_pda entry", 128, (FOE), 937680>
From that data structure it grabs the level4_pgt and data_offset
fields for subsequent use. So in your case, it should show how
many (if any) of the x8664_pda structures it read before encountering
a NULL pointer in one of the array entries.
Thanks, Dave.
I have the vmcoreinfo patch applied.
Kernel is 2.6.23-rc3.
The crash debug output is below. Please let me know if you'd like
me to test without the vmcoreinfo patch or anything else.
---
crash 4.0-4.6
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
vmcore_data:
flags: c0 (KDUMP_LOCAL|KDUMP_ELF64)
ndfd: 3
ofp: 322af48760
header_size: 1580
num_pt_load_segments: 4
pt_load_segment[0]:
file_offset: 62c
phys_start: 200000
phys_end: bda000
zero_fill: 0
pt_load_segment[1]:
file_offset: 9da62c
phys_start: 0
phys_end: a0000
zero_fill: 0
pt_load_segment[2]:
file_offset: a7a62c
phys_start: 100000
phys_end: 1000000
zero_fill: 0
pt_load_segment[3]:
file_offset: 197a62c
phys_start: 5000000
phys_end: 3ffc0000
zero_fill: 0
elf_header: 21b2c70
elf32: 0
notes32: 0
load32: 0
elf64: 21b2c70
notes64: 21b2cb0
load64: 21b2ce8
nt_prstatus: 21b2dc8
nt_prpsinfo: 0
nt_taskstruct: 0
task_struct: 0
page_size: 0
switch_stack: 0
xen_kdump_data: (unused)
num_prstatus_notes: 1
nt_prstatus_percpu: 00000000021b2dc8
Elf64_Ehdr:
e_ident: \177ELF
e_ident[EI_CLASS]: 2 (ELFCLASS64)
e_ident[EI_DATA]: 1 (ELFDATA2LSB)
e_ident[EI_VERSION]: 1 (EV_CURRENT)
e_ident[EI_OSABI]: 0 (ELFOSABI_SYSV)
e_ident[EI_ABIVERSION]: 0
e_type: 4 (ET_CORE)
e_machine: 62 (EM_X86_64)
e_version: 1 (EV_CURRENT)
e_entry: 0
e_phoff: 40
e_shoff: 0
e_flags: 0
e_ehsize: 40
e_phentsize: 38
e_phnum: 5
e_shentsize: 0
e_shnum: 0
e_shstrndx: 0
Elf64_Phdr:
p_type: 4 (PT_NOTE)
p_offset: 344 (158)
p_vaddr: 0
p_paddr: 0
p_filesz: 1236 (4d4)
p_memsz: 1236 (4d4)
p_flags: 0 ()
p_align: 0
Elf64_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 1580 (62c)
p_vaddr: ffffffff80200000
p_paddr: 200000
p_filesz: 10330112 (9da000)
p_memsz: 10330112 (9da000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf64_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 10331692 (9da62c)
p_vaddr: ffff810000000000
p_paddr: 0
p_filesz: 655360 (a0000)
p_memsz: 655360 (a0000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf64_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 10987052 (a7a62c)
p_vaddr: ffff810000100000
p_paddr: 100000
p_filesz: 15728640 (f00000)
p_memsz: 15728640 (f00000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf64_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 26715692 (197a62c)
p_vaddr: ffff810005000000
p_paddr: 5000000
p_filesz: 989593600 (3afc0000)
p_memsz: 989593600 (3afc0000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf64_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 336
n_type: 1 (NT_PRSTATUS)
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000002b1a 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000006 0000000000000000
0000000000000063 0000000000000000
ffff810019769e48 ffffffff806aeda0
ffff81003d64eac0 ffffffff8023b1df
ffffffff8023b1df ffff810019769ca8
0000000000000000 0000000000000000
ffff8100848a9000 0000000000000000
0000000000000000 0000000000000292
ffffffff80260532 0000000000000010
0000000000000046 ffff810019769d98
0000000000000018 00002b7fb481df10
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
Elf64_Nhdr:
n_namesz: 11 ("VMCOREINFO")
n_descsz: 856
n_type: 0 (?)
41454c4552534f00 322e362e323d4553
41500a3363722d33 343d455a49534547
424d59530a363930 5f74696e69284c4f
3d29736e5f737475 6666666666666666
3036373538363038 284c4f424d59530a
6c6e6f5f65646f6e 2970616d5f656e69
666666666666663d 3432633037303866
4c4f424d59530a30 7265707061777328
297269645f67705f 666666666666663d
3030313032303866 4c4f424d59530a30
2974786574735f28 666666666666663d
3030393032303866 7028455a49530a30
0a36393d29656761 6c677028455a4953
617461645f747369 0a30323734313d29
6e6f7a28455a4953 0a343230313d2965
65726628455a4953 3d29616572615f65
28455a49530a3432 6165685f7473696c
464f0a36313d2964 6761702854455346
297367616c662e65 455346464f0a303d
5f2e656761702854 383d29746e756f63
2854455346464f0a 70616d2e65676170
34323d29676e6970 2854455346464f0a
75726c2e65676170 46464f0a30383d29
696c677028544553 2e617461645f7473
6e6f7a5f65646f6e 464f0a303d297365
6c67702854455346 617461645f747369
656e6f7a5f726e2e 30363534313d2973
2854455346464f0a 645f7473696c6770
65646f6e2e617461 70616d5f6d656d5f
0a38363534313d29 702854455346464f
61645f7473696c67 5f65646f6e2e6174
66705f7472617473 34383534313d296e
2854455346464f0a 645f7473696c6770
65646f6e2e617461 64656e6e6170735f
3d2973656761705f 464f0a3030363431
6c67702854455346 617461645f747369
64695f65646f6e2e 0a38303634313d29
7a2854455346464f 656572662e656e6f
323d29616572615f 455346464f0a3030
762e656e6f7a2854 3d29746174735f6d
5346464f0a323336 2e656e6f7a285445
5f64656e6e617073 393d297365676170
455346464f0a3633 615f656572662854
656572662e616572 303d297473696c5f
2854455346464f0a 6165685f7473696c
3d297478656e2e64 54455346464f0a30
65685f7473696c28 29766572702e6461
54474e454c0a383d 662e656e6f7a2848
616572615f656572 4d59530a31313d29
65646f6e284c4f42 663d29617461645f
3866666666666666 0a30346561303730
6e284854474e454c 617461645f65646f
4152430a34363d29 313d454d49544853
3938333339383831
p_vaddr: ffffffff80200000 p_paddr: 200000 -> phys_base: 0
gdb /boot/vmlinux-2.6.23-rc3
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
<readmem: ffffffff8053bc40, KVADDR, "kernel_config_data", 32768, (ROE),
3723960>
crash: CONFIG_NR_CPUS: 8
crash: CONFIG_HZ: 250
WARNING: Because this kernel was compiled with gcc version 4.1.1, certain
commands or command options may fail unless crash is invoked with
the "--readnow" command line option.
GNU_GET_DATATYPE[runqueue]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[runqueue]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[prio_array]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[prio_array]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[prio_array]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[irq_desc_t]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[hw_interrupt_type]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[irq_cpustat_t]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[irq_cpustat_t]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[irq_cpustat_t]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[timer_vec_root]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[timer_vec]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[softirq_state]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[kallsyms_header]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[user_regs_struct]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[user_regs_struct]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[user_regs_struct]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[user_regs_struct]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[user_regs_struct]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[user_regs_struct]: returned via gdb_error_hook (1 buffer in use)
<readmem: ffffffff80837580, KVADDR, "xtime", 16, (FOE), a09c90>
<readmem: ffffffff80685764, KVADDR, "init_uts_ns", 390, (ROE), a0a27c>
<readmem: ffffffff80537000, KVADDR, "accessible check", 8, (ROE|Q),
7fff10bb8dc8>
<readmem: ffffffff80537000, KVADDR, "readstring characters", 1499, (ROE|Q),
7fff10bb7db0>
verify_namelist:
/proc/version:
Linux version 2.6.23-rc3 (rddunlap(a)unicorn.site) (gcc version 4.1.1 20070105 (Red Hat
4.1.1-52)) #19 SMP Tue Sep 4 09:52:06 PDT 2007
utsname version: #19 SMP Tue Sep 4 09:52:06 PDT 2007
/boot/vmlinux-2.6.23-rc3:
Linux version 2.6.23-rc3 (rddunlap(a)unicorn.site) (gcc version 4.1.1 20070105 (Red Hat
4.1.1-52)) #22 SMP Thu Sep 6 21:24:54 PDT 2007
<readmem: ffffffff80707940, KVADDR, "_cpu_pda addr", 8, (FOE),
7fff10bba538>
<readmem: 0, KVADDR, "cpu_pda entry", 128, (FOE), a3a820>
crash: invalid kernel virtual address: 0 type: "cpu_pda entry"
1:12 p.m.
New subject: [PATCH 0/2] vmcoreinfo support for dump filtering #2
Randy Dunlap wrote:
I have the vmcoreinfo patch applied.
Kernel is 2.6.23-rc3.
The crash debug output is below. Please let me know if you'd like
me to test without the vmcoreinfo patch or anything else.
---
crash 4.0-4.6
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
vmcore_data:
flags: c0 (KDUMP_LOCAL|KDUMP_ELF64)
ndfd: 3
ofp: 322af48760
header_size: 1580
num_pt_load_segments: 4
pt_load_segment[0]:
file_offset: 62c
phys_start: 200000
phys_end: bda000
zero_fill: 0
pt_load_segment[1]:
file_offset: 9da62c
phys_start: 0
phys_end: a0000
zero_fill: 0
pt_load_segment[2]:
file_offset: a7a62c
phys_start: 100000
phys_end: 1000000
zero_fill: 0
pt_load_segment[3]:
file_offset: 197a62c
phys_start: 5000000
phys_end: 3ffc0000
zero_fill: 0
elf_header: 21b2c70
elf32: 0
notes32: 0
load32: 0
elf64: 21b2c70
notes64: 21b2cb0
load64: 21b2ce8
nt_prstatus: 21b2dc8
nt_prpsinfo: 0
nt_taskstruct: 0
task_struct: 0
page_size: 0
switch_stack: 0
xen_kdump_data: (unused)
num_prstatus_notes: 1
nt_prstatus_percpu: 00000000021b2dc8
Elf64_Ehdr:
e_ident: \177ELF
e_ident[EI_CLASS]: 2 (ELFCLASS64)
e_ident[EI_DATA]: 1 (ELFDATA2LSB)
e_ident[EI_VERSION]: 1 (EV_CURRENT)
e_ident[EI_OSABI]: 0 (ELFOSABI_SYSV)
e_ident[EI_ABIVERSION]: 0
e_type: 4 (ET_CORE)
e_machine: 62 (EM_X86_64)
e_version: 1 (EV_CURRENT)
e_entry: 0
e_phoff: 40
e_shoff: 0
e_flags: 0
e_ehsize: 40
e_phentsize: 38
e_phnum: 5
e_shentsize: 0
e_shnum: 0
e_shstrndx: 0
Elf64_Phdr:
p_type: 4 (PT_NOTE)
p_offset: 344 (158)
p_vaddr: 0
p_paddr: 0
p_filesz: 1236 (4d4)
p_memsz: 1236 (4d4)
p_flags: 0 ()
p_align: 0
Elf64_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 1580 (62c)
p_vaddr: ffffffff80200000
p_paddr: 200000
p_filesz: 10330112 (9da000)
p_memsz: 10330112 (9da000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf64_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 10331692 (9da62c)
p_vaddr: ffff810000000000
p_paddr: 0
p_filesz: 655360 (a0000)
p_memsz: 655360 (a0000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf64_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 10987052 (a7a62c)
p_vaddr: ffff810000100000
p_paddr: 100000
p_filesz: 15728640 (f00000)
p_memsz: 15728640 (f00000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf64_Phdr:
p_type: 1 (PT_LOAD)
p_offset: 26715692 (197a62c)
p_vaddr: ffff810005000000
p_paddr: 5000000
p_filesz: 989593600 (3afc0000)
p_memsz: 989593600 (3afc0000)
p_flags: 7 (PF_X|PF_W|PF_R)
p_align: 0
Elf64_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 336
n_type: 1 (NT_PRSTATUS)
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000002b1a 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000006 0000000000000000
0000000000000063 0000000000000000
ffff810019769e48 ffffffff806aeda0
ffff81003d64eac0 ffffffff8023b1df
ffffffff8023b1df ffff810019769ca8
0000000000000000 0000000000000000
ffff8100848a9000 0000000000000000
0000000000000000 0000000000000292
ffffffff80260532 0000000000000010
0000000000000046 ffff810019769d98
0000000000000018 00002b7fb481df10
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
Elf64_Nhdr:
n_namesz: 11 ("VMCOREINFO")
n_descsz: 856
n_type: 0 (?)
41454c4552534f00 322e362e323d4553
41500a3363722d33 343d455a49534547
424d59530a363930 5f74696e69284c4f
3d29736e5f737475 6666666666666666
3036373538363038 284c4f424d59530a
6c6e6f5f65646f6e 2970616d5f656e69
666666666666663d 3432633037303866
4c4f424d59530a30 7265707061777328
297269645f67705f 666666666666663d
3030313032303866 4c4f424d59530a30
2974786574735f28 666666666666663d
3030393032303866 7028455a49530a30
0a36393d29656761 6c677028455a4953
617461645f747369 0a30323734313d29
6e6f7a28455a4953 0a343230313d2965
65726628455a4953 3d29616572615f65
28455a49530a3432 6165685f7473696c
464f0a36313d2964 6761702854455346
297367616c662e65 455346464f0a303d
5f2e656761702854 383d29746e756f63
2854455346464f0a 70616d2e65676170
34323d29676e6970 2854455346464f0a
75726c2e65676170 46464f0a30383d29
696c677028544553 2e617461645f7473
6e6f7a5f65646f6e 464f0a303d297365
6c67702854455346 617461645f747369
656e6f7a5f726e2e 30363534313d2973
2854455346464f0a 645f7473696c6770
65646f6e2e617461 70616d5f6d656d5f
0a38363534313d29 702854455346464f
61645f7473696c67 5f65646f6e2e6174
66705f7472617473 34383534313d296e
2854455346464f0a 645f7473696c6770
65646f6e2e617461 64656e6e6170735f
3d2973656761705f 464f0a3030363431
6c67702854455346 617461645f747369
64695f65646f6e2e 0a38303634313d29
7a2854455346464f 656572662e656e6f
323d29616572615f 455346464f0a3030
762e656e6f7a2854 3d29746174735f6d
5346464f0a323336 2e656e6f7a285445
5f64656e6e617073 393d297365676170
455346464f0a3633 615f656572662854
656572662e616572 303d297473696c5f
2854455346464f0a 6165685f7473696c
3d297478656e2e64 54455346464f0a30
65685f7473696c28 29766572702e6461
54474e454c0a383d 662e656e6f7a2848
616572615f656572 4d59530a31313d29
65646f6e284c4f42 663d29617461645f
3866666666666666 0a30346561303730
6e284854474e454c 617461645f65646f
4152430a34363d29 313d454d49544853
3938333339383831
p_vaddr: ffffffff80200000 p_paddr: 200000 -> phys_base: 0
gdb /boot/vmlinux-2.6.23-rc3
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
<readmem: ffffffff8053bc40, KVADDR, "kernel_config_data", 32768, (ROE),
3723960>
crash: CONFIG_NR_CPUS: 8
crash: CONFIG_HZ: 250
WARNING: Because this kernel was compiled with gcc version 4.1.1, certain
commands or command options may fail unless crash is invoked with
the "--readnow" command line option.
GNU_GET_DATATYPE[runqueue]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[runqueue]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[prio_array]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[prio_array]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[prio_array]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[irq_desc_t]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[hw_interrupt_type]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[irq_cpustat_t]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[irq_cpustat_t]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[irq_cpustat_t]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[timer_vec_root]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[timer_vec]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[softirq_state]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[kallsyms_header]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[user_regs_struct]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[user_regs_struct]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[user_regs_struct]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[user_regs_struct]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[user_regs_struct]: returned via gdb_error_hook (1 buffer in use)
GNU_GET_DATATYPE[user_regs_struct]: returned via gdb_error_hook (1 buffer in use)
<readmem: ffffffff80837580, KVADDR, "xtime", 16, (FOE), a09c90>
<readmem: ffffffff80685764, KVADDR, "init_uts_ns", 390, (ROE), a0a27c>
<readmem: ffffffff80537000, KVADDR, "accessible check", 8, (ROE|Q),
7fff10bb8dc8>
<readmem: ffffffff80537000, KVADDR, "readstring characters", 1499, (ROE|Q),
7fff10bb7db0>
verify_namelist:
/proc/version:
Linux version 2.6.23-rc3 (rddunlap(a)unicorn.site) (gcc version 4.1.1 20070105 (Red Hat
4.1.1-52)) #19 SMP Tue Sep 4 09:52:06 PDT 2007
utsname version: #19 SMP Tue Sep 4 09:52:06 PDT 2007
/boot/vmlinux-2.6.23-rc3:
Linux version 2.6.23-rc3 (rddunlap(a)unicorn.site) (gcc version 4.1.1 20070105 (Red Hat
4.1.1-52)) #22 SMP Thu Sep 6 21:24:54 PDT 2007
<readmem: ffffffff80707940, KVADDR, "_cpu_pda addr", 8, (FOE),
7fff10bba538>
<readmem: 0, KVADDR, "cpu_pda entry", 128, (FOE), a3a820>
crash: invalid kernel virtual address: 0 type: "cpu_pda entry"
A few things come to mind. Walking through the debug data above...
The very first readmem() from the dumpfile is from the kernel symbol
"kernel_config_data", where you can see that it found the CONFIG_HZ and
CONFIG_NR_CPUS values. The next readmem()'s are of "xtime" and then
"init_uts_ns". We don't know what was read from the "xtime"
location,
but the utsname data from "init_uts_ns" gets displayed later on here:
utsname version: #19 SMP Tue Sep 4 09:52:06 PDT 2007
And then the "linux_banner" address of ffffffff80537000 is first
checked for accessibility (OK), and then it is read successfully,
and its contents are displayed here:
/proc/version:
Linux version 2.6.23-rc3 (rddunlap(a)unicorn.site) (gcc version 4.1.1 20070105
(Red
Hat 4.1.1-52)) #19 SMP Tue Sep 4 09:52:06 PDT 2007
The string above from the dumpfile is correlated against the
linux_banner string in the vmlinux file, which is subsequently
displayed here:
/boot/vmlinux-2.6.23-rc3:
Linux version 2.6.23-rc3 (rddunlap(a)unicorn.site) (gcc version 4.1.1 20070105
(Red
Hat 4.1.1-52)) #22 SMP Thu Sep 6 21:24:54 PDT 2007
The utsname data and the linux_banner string from the dumpfile
are from "Tue Sep 4 09:52:06 PDT 2007", whereas the vmlinux file
was built 2 days later at "Thu Sep 6 21:24:54 PDT 2007". I don't
know whether that's the issue or not. Is there a reason that
you are *not* using the same vmlinux that the dumpfile was created
from?
So the first thing to verify is that you use the same vmlinux
that was booted and dumped. If you cannot dig up the original
vmlinux file, get the System.map file from the dumped kernel,
and throw that on the command line, and see if that helps:
$ crash vmlinux vmcore System.map
Anyway, next it reads the _cpu_pda[0] at address ffffffff80707940 to
find the address of cpu 0's x8664_pda structure:
<readmem: ffffffff80707940, KVADDR, "_cpu_pda addr", 8,
(FOE), 7fff10bba538>
But it finds a zero there:
<readmem: 0, KVADDR, "cpu_pda entry", 128, (FOE),
a3a820>
crash: invalid kernel virtual address: 0 type: "cpu_pda entry"
At this point crash is done, the readmem() is "FOE" (fault-on-error),
because there's no sense in continuing.
If the vmlinux and dumpfile are different, it's possible that the
_cpu_pda[] array, which is the highest address read so far, (the
xtime data which is even higher may be garbage as well), may
have been "pushed up" by some other changes in the kernel?
Or, if they do "line up", something may have changed with respect
to the kernel's _cpu_pda[] handling or its data declaration
Or, it actually read zeroes from the dumpfile.
But, for now let's suppose that the two kernels are identical except
for the date in the linux_banner strings. I don't have a 2.6.23
kernel source tree handy, but at least as of 2.6.22-5, it was still
declared statically like so:
struct x8664_pda *_cpu_pda[NR_CPUS] __read_mostly;
Has that changed?
If not, it would be worth checking a dumpfile with no pages
excluded with makedumpfile. I wouldn't think the in-kernel
part of the vmcoreinfo patches would make a difference, but
I suppose anything's possible.
You also mentioned that gdb worked OK. What happens when
you enter this:
(gdb) p _cpu_pda[0]
And if you enter:
(gdb) p &_cpu_pda[0]
does it show 0xffffffff80707940? Which is what crash thinks is
the correct address:
<readmem: ffffffff80707940, KVADDR, "_cpu_pda addr", 8,
(FOE), 7fff10bba538>
But again -- the very first thing to do is make sure that you
are using the exact same vmlinux as was booted/dumped.
Dave
6:01 p.m.
New subject: [PATCH 0/2] vmcoreinfo support for dump filtering #2
On Tue, 11 Sep 2007 14:12:00 -0400 Dave Anderson wrote:
Randy Dunlap wrote:
>
> I have the vmcoreinfo patch applied.
> Kernel is 2.6.23-rc3.
>
> The crash debug output is below. Please let me know if you'd like
> me to test without the vmcoreinfo patch or anything else.
>
> ---
[snipped]
A few things come to mind. Walking through the debug data above...
The very first readmem() from the dumpfile is from the kernel symbol
"kernel_config_data", where you can see that it found the CONFIG_HZ and
CONFIG_NR_CPUS values. The next readmem()'s are of "xtime" and then
"init_uts_ns". We don't know what was read from the "xtime"
location,
but the utsname data from "init_uts_ns" gets displayed later on here:
> utsname version: #19 SMP Tue Sep 4 09:52:06 PDT 2007
And then the "linux_banner" address of ffffffff80537000 is first
checked for accessibility (OK), and then it is read successfully,
and its contents are displayed here:
> /proc/version:
> Linux version 2.6.23-rc3 (rddunlap(a)unicorn.site) (gcc version 4.1.1 20070105
(Red Hat 4.1.1-52)) #19 SMP Tue Sep 4 09:52:06 PDT 2007
The string above from the dumpfile is correlated against the
linux_banner string in the vmlinux file, which is subsequently
displayed here:
> /boot/vmlinux-2.6.23-rc3:
> Linux version 2.6.23-rc3 (rddunlap(a)unicorn.site) (gcc version 4.1.1 20070105
(Red Hat 4.1.1-52)) #22 SMP Thu Sep 6 21:24:54 PDT 2007
The utsname data and the linux_banner string from the dumpfile
are from "Tue Sep 4 09:52:06 PDT 2007", whereas the vmlinux file
was built 2 days later at "Thu Sep 6 21:24:54 PDT 2007". I don't
know whether that's the issue or not. Is there a reason that
you are *not* using the same vmlinux that the dumpfile was created
from?
Just sorry user error. Sorry to use your time like that
and thanks for the intro-to-crash lesson.
It's working now as expected. Thanks.
But, for now let's suppose that the two kernels are identical
except
for the date in the linux_banner strings. I don't have a 2.6.23
kernel source tree handy, but at least as of 2.6.22-5, it was still
declared statically like so:
struct x8664_pda *_cpu_pda[NR_CPUS] __read_mostly;
Has that changed?
Nope.
If not, it would be worth checking a dumpfile with no pages
excluded with makedumpfile. I wouldn't think the in-kernel
part of the vmcoreinfo patches would make a difference, but
I suppose anything's possible.
crash works (loads without error) with the vmcore file and one that
has all possible pages removed from it using 'makedumpfile'.
But again -- the very first thing to do is make sure that you
are using the exact same vmlinux as was booted/dumped.
Very true.
Thanks again.
---
~Randy
6138
days inactive
6138
days old
devel@lists.crash-utility.osci.io
7 comments
5 participants
participants (5)
-
Dave Anderson -
Horms -
Piet Delaney -
Randy Dunlap -
Vivek Goyal