4:11 p.m.
Hi Andy,
I've got a ELF kdump vmcore that was created in-house from a kernel configured
with CONFIG_RANDOMIZE_BASE. I thought I might be able to analyze it by applying
your earlier patch that introduced the --kaslr option. The kernel does not
have the offset registered in the vmcoreinfo, and so I'm trying to determine
the offset, but with no luck.
Earlier, Kees had mentioned this:
> FWIW, the offset reported during a panic to dmesg is:
> (unsigned long)&_text - __START_KERNEL
Where does it get reported during a panic exactly? Here's the oops trace, gotten
by running "strings" on the vmcore:
$ strings vmcore
... [ cut ] ...
SysRq : Trigger a crash
"BUG: unable to handle kernel NULL pointer dereference at (null)
"IP: [<ffffffff992bf6cf>] sysrq_handle_crash+0x11/0x1b
PGD 3a067 PUD 2e067 PMD 0
Oops: 0002 [#1] PREEMPT SMP 0"
Modules linked in:
CPU: 0 PID: 1720 Comm: bash Not tainted 3.14.0-rc1+ #1130"
task: ffff88001d028000 ti: ffff88001c986000 task.ti: ffff88001c986000
RIP: 0010:[<ffffffff992bf6cf>] [<ffffffff992bf6cf>]
sysrq_handle_crash+0x11/0x1b
RSP: 0018:ffff88001c987e90 EFLAGS: 000100920"
RAX: 000000000000000f RBX: ffffffff9975ed50 RCX: 0000000000000000
RDX: ffff88001d028000 RSI: ffff88001cc0e338 RDI: 0000000000000063
RBP: ffff88001c987e90 R08: 0000000000000002 R09: 0000000000000000
R10: ffffffff994e9630 R11: 0000000000000000 R12: 0000000000000007
R13: 0000000000000246 R14: 0000000000000063 R15: 0000000000000000
FS: 00007f0ec2181740(0000) GS:ffff88001cc00000(0000) knlGS:00000000000000000"
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000001d36f000 CR4: 00000000000006f0
Stack:
ffff88001c987ec8 ffffffff992bfc88 0000000000000002 00007f0ec21870000"
0000000000000002 ffff88001c987f58 0000000000000000 ffff88001c987ee80"
ffffffff992c000c ffff88001c92acc0 00007f0ec2187000 ffff88001c987f080"
Call Trace:
[<ffffffff992bfc88>] __handle_sysrq+0x9b/0x133
[<ffffffff992c000c>] write_sysrq_trigger+0x2d/0x3e
[<ffffffff991681cb>] proc_reg_write+0x45/0x65
[<ffffffff9911897c>] vfs_write+0xbf/0x17c
[<ffffffff9911918f>] SyS_write+0x44/0x7a
[<ffffffff9949ad7d>] system_call_fastpath+0x1a/0x1f0"
Code: 4f 00 00 55 b8 01 00 00 00 48 89 e5 75 07 0f b6 05 b3 20 4f 00 83 e0 01 5d c3 55
c7 05 03 18 61 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55
31 c0 c7 05 ba dc 46 00 07 00 0"
"RIP [<ffffffff992bf6cf>] sysrq_handle_crash+0x11/0x1b
RSP <ffff88001c987e90>
CR2: 0000000000000000
ttySffffffff99000000 T _text
UUUU
UUUU
VMCOREINFO
OSRELEASE=3.14.0-rc1+
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffffffff99713250
SYMBOL(node_online_map)=ffffffff997b0c68
SYMBOL(swapper_pg_dir)=ffffffff9970e000
SYMBOL(_stext)=ffffffff990001c8
SYMBOL(vmap_area_list)=ffffffff99745c20
SYMBOL(mem_map)=ffffffff9a1253a8
SYMBOL(contig_page_data)=ffffffff99790000
SYMBOL(mem_section)=ffffffff9a126000
LENGTH(mem_section)=2048
SIZE(mem_section)=16
OFFSET(mem_section.section_mem_map)=0
SIZE(page)=64
SIZE(pglist_data)=53248
SIZE(zone)=12288
SIZE(free_area)=88
SIZE(list_head)=16
SIZE(nodemask_t)=8
OFFSET(page.flags)=0
OFFSET(page._count)=28
OFFSET(page.mapping)=8
OFFSET(page.lru)=32
OFFSET(page._mapcount)=24
OFFSET(page.private)=48
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=49240
OFFSET(pglist_data.node_start_pfn)=49304
OFFSET(pglist_data.node_spanned_pages)=49320
OFFSET(pglist_data.node_id)=49328
OFFSET(zone.free_area)=256
OFFSET(zone.vm_stat)=4280
OFFSET(zone.spanned_pages)=8232
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=8
OFFSET(vmap_area.va_start)=0
OFFSET(vmap_area.list)=48
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=ffffffff9972d290
SYMBOL(log_buf_len)=ffffffff9972d288
SYMBOL(log_first_idx)=ffffffff9a11eb48
SYMBOL(log_next_idx)=ffffffff9a11eb38
SIZE(printk_log)=16
OFFSET(printk_log.ts_nsec)=0
OFFSET(printk_log.len)=8
OFFSET(printk_log.text_len)=10
OFFSET(printk_log.dict_len)=12
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=11
NUMBER(PG_swapcache)=16
NUMBER(PG_slab)=7
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
SYMBOL(phys_base)=ffffffff99713010
SYMBOL(init_level4_pgt)=ffffffff9970e000
CRASHTIME=1391826079
OSRELEASE=3.14.0-rc1+
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffffffff99713250
SYMBOL(node_online_map)=ffffffff997b0c68
SYMBOL(swapper_pg_dir)=ffffffff9970e000
SYMBOL(_stext)=ffffffff990001c8
SYMBOL(vmap_area_list)=ffffffff99745c20
SYMBOL(mem_map)=ffffffff9a1253a8
SYMBOL(contig_page_data)=ffffffff99790000
SYMBOL(mem_section)=ffffffff9a126000
LENGTH(mem_section)=2048
SIZE(mem_section)=16
OFFSET(mem_section.section_mem_map)=0
SIZE(page)=64
SIZE(pglist_data)=53248
SIZE(zone)=12288
SIZE(free_area)=88
SIZE(list_head)=16
SIZE(nodemask_t)=8
OFFSET(page.flags)=0
OFFSET(page._count)=28
OFFSET(page.mapping)=8
OFFSET(page.lru)=32
OFFSET(page._mapcount)=24
OFFSET(page.private)=48
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=49240
OFFSET(pglist_data.node_start_pfn)=49304
OFFSET(pglist_data.node_spanned_pages)=49320
OFFSET(pglist_data.node_id)=49328
OFFSET(zone.free_area)=256
OFFSET(zone.vm_stat)=4280
OFFSET(zone.spanned_pages)=8232
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=8
OFFSET(vmap_area.va_start)=0
OFFSET(vmap_area.list)=48
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=ffffffff9972d290
SYMBOL(log_buf_len)=ffffffff9972d288
SYMBOL(log_first_idx)=ffffffff9a11eb48
SYMBOL(log_next_idx)=ffffffff9a11eb38
SIZE(printk_log)=16
OFFSET(printk_log.ts_nsec)=0
OFFSET(printk_log.len)=8
OFFSET(printk_log.text_len)=10
OFFSET(printk_log.dict_len)=12
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=11
NUMBER(PG_swapcache)=16
NUMBER(PG_slab)=7
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
SYMBOL(phys_base)=ffffffff99713010
SYMBOL(init_level4_pgt)=ffffffff9970e000
CRASHTIME=1391826079
...
Anyway, the /proc/kallsyms file of the crashing system was saved,
and it shows this:
ffffffff99000000 T _text
and if I subtract __START_KERNEL (ffffffff80000000) from that, I get
what I presume is the kaslr offset of 0x19000000. The vmcore core
header would seeminlgy confirm that:
$ readelf -a vmcore
... [ cut ] ...
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
NOTE 0x0000000000001000 0x0000000000000000 0x0000000000000000
0x00000000000007f8 0x00000000000007f8 0
LOAD 0x0000000000002000 0xffffffff99000000 0x0000000019000000 <===
0x0000000001183000 0x0000000001183000 RWE 0
LOAD 0x0000000001185000 0xffff880000001000 0x0000000000001000
0x000000000009f000 0x000000000009f000 RWE 0
LOAD 0x0000000001224000 0xffff880000100000 0x0000000000100000
0x0000000010f00000 0x0000000010f00000 RWE 0
LOAD 0x0000000012124000 0xffff880019000000 0x0000000019000000
0x0000000005194000 0x0000000005194000 RWE 0
LOAD 0x00000000172b8000 0xffff88001e1c1000 0x000000001e1c1000
0x00000000017c0000 0x00000000017c0000 RWE 0
LOAD 0x0000000018a78000 0xffff88001f9e5000 0x000000001f9e5000
0x00000000005fb000 0x00000000005fb000 RWE 0
But if I try that value with your patch applied, it fails in the same manner
as if I don't use the --kaslr option at all:
$ crash --kaslr 0x19000000 vmlinux vmcore
crash 7.0.5rc12
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: could not find MAGIC_START!
WARNING: cannot read linux_banner string
crash: vmlinux and vmcore do not match!
Usage:
crash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
Enter "crash -h" for details.
$
Any ideas? I can give you the vmlinux/vmcore/kallsyms triplet if you'd like.
Thanks,
Dave
6:13 p.m.
On Mon, Feb 10, 2014 at 2:11 PM, Dave Anderson <anderson(a)redhat.com> wrote:
Hi Andy,
I've got a ELF kdump vmcore that was created in-house from a kernel configured
with CONFIG_RANDOMIZE_BASE. I thought I might be able to analyze it by applying
your earlier patch that introduced the --kaslr option. The kernel does not
have the offset registered in the vmcoreinfo, and so I'm trying to determine
the offset, but with no luck.
Earlier, Kees had mentioned this:
>> FWIW, the offset reported during a panic to dmesg is:
>> (unsigned long)&_text - __START_KERNEL
Where does it get reported during a panic exactly? Here's the oops trace, gotten
by running "strings" on the vmcore:
The commit that reports it is here: f32360ef6608434a032dc7ad262d45e9693c27f3
And maybe related, but 6723734cdff15211bb78aeea76ca847374bd93ae makes
sure panic handlers are called before kmsg_dump.
And "strings" doesn't show anything starting with "Kernel Offset:"
?
Maybe it's not being saved in the kcore?
-Kees
$ strings vmcore
... [ cut ] ...
SysRq : Trigger a crash
"BUG: unable to handle kernel NULL pointer dereference at (null)
"IP: [<ffffffff992bf6cf>] sysrq_handle_crash+0x11/0x1b
PGD 3a067 PUD 2e067 PMD 0
Oops: 0002 [#1] PREEMPT SMP 0"
Modules linked in:
CPU: 0 PID: 1720 Comm: bash Not tainted 3.14.0-rc1+ #1130"
task: ffff88001d028000 ti: ffff88001c986000 task.ti: ffff88001c986000
RIP: 0010:[<ffffffff992bf6cf>] [<ffffffff992bf6cf>]
sysrq_handle_crash+0x11/0x1b
RSP: 0018:ffff88001c987e90 EFLAGS: 000100920"
RAX: 000000000000000f RBX: ffffffff9975ed50 RCX: 0000000000000000
RDX: ffff88001d028000 RSI: ffff88001cc0e338 RDI: 0000000000000063
RBP: ffff88001c987e90 R08: 0000000000000002 R09: 0000000000000000
R10: ffffffff994e9630 R11: 0000000000000000 R12: 0000000000000007
R13: 0000000000000246 R14: 0000000000000063 R15: 0000000000000000
FS: 00007f0ec2181740(0000) GS:ffff88001cc00000(0000) knlGS:00000000000000000"
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000001d36f000 CR4: 00000000000006f0
Stack:
ffff88001c987ec8 ffffffff992bfc88 0000000000000002 00007f0ec21870000"
0000000000000002 ffff88001c987f58 0000000000000000 ffff88001c987ee80"
ffffffff992c000c ffff88001c92acc0 00007f0ec2187000 ffff88001c987f080"
Call Trace:
[<ffffffff992bfc88>] __handle_sysrq+0x9b/0x133
[<ffffffff992c000c>] write_sysrq_trigger+0x2d/0x3e
[<ffffffff991681cb>] proc_reg_write+0x45/0x65
[<ffffffff9911897c>] vfs_write+0xbf/0x17c
[<ffffffff9911918f>] SyS_write+0x44/0x7a
[<ffffffff9949ad7d>] system_call_fastpath+0x1a/0x1f0"
Code: 4f 00 00 55 b8 01 00 00 00 48 89 e5 75 07 0f b6 05 b3 20 4f 00 83 e0 01 5d c3 55
c7 05 03 18 61 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55
31 c0 c7 05 ba dc 46 00 07 00 0"
"RIP [<ffffffff992bf6cf>] sysrq_handle_crash+0x11/0x1b
RSP <ffff88001c987e90>
CR2: 0000000000000000
ttySffffffff99000000 T _text
UUUU
UUUU
VMCOREINFO
OSRELEASE=3.14.0-rc1+
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffffffff99713250
SYMBOL(node_online_map)=ffffffff997b0c68
SYMBOL(swapper_pg_dir)=ffffffff9970e000
SYMBOL(_stext)=ffffffff990001c8
SYMBOL(vmap_area_list)=ffffffff99745c20
SYMBOL(mem_map)=ffffffff9a1253a8
SYMBOL(contig_page_data)=ffffffff99790000
SYMBOL(mem_section)=ffffffff9a126000
LENGTH(mem_section)=2048
SIZE(mem_section)=16
OFFSET(mem_section.section_mem_map)=0
SIZE(page)=64
SIZE(pglist_data)=53248
SIZE(zone)=12288
SIZE(free_area)=88
SIZE(list_head)=16
SIZE(nodemask_t)=8
OFFSET(page.flags)=0
OFFSET(page._count)=28
OFFSET(page.mapping)=8
OFFSET(page.lru)=32
OFFSET(page._mapcount)=24
OFFSET(page.private)=48
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=49240
OFFSET(pglist_data.node_start_pfn)=49304
OFFSET(pglist_data.node_spanned_pages)=49320
OFFSET(pglist_data.node_id)=49328
OFFSET(zone.free_area)=256
OFFSET(zone.vm_stat)=4280
OFFSET(zone.spanned_pages)=8232
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=8
OFFSET(vmap_area.va_start)=0
OFFSET(vmap_area.list)=48
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=ffffffff9972d290
SYMBOL(log_buf_len)=ffffffff9972d288
SYMBOL(log_first_idx)=ffffffff9a11eb48
SYMBOL(log_next_idx)=ffffffff9a11eb38
SIZE(printk_log)=16
OFFSET(printk_log.ts_nsec)=0
OFFSET(printk_log.len)=8
OFFSET(printk_log.text_len)=10
OFFSET(printk_log.dict_len)=12
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=11
NUMBER(PG_swapcache)=16
NUMBER(PG_slab)=7
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
SYMBOL(phys_base)=ffffffff99713010
SYMBOL(init_level4_pgt)=ffffffff9970e000
CRASHTIME=1391826079
OSRELEASE=3.14.0-rc1+
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffffffff99713250
SYMBOL(node_online_map)=ffffffff997b0c68
SYMBOL(swapper_pg_dir)=ffffffff9970e000
SYMBOL(_stext)=ffffffff990001c8
SYMBOL(vmap_area_list)=ffffffff99745c20
SYMBOL(mem_map)=ffffffff9a1253a8
SYMBOL(contig_page_data)=ffffffff99790000
SYMBOL(mem_section)=ffffffff9a126000
LENGTH(mem_section)=2048
SIZE(mem_section)=16
OFFSET(mem_section.section_mem_map)=0
SIZE(page)=64
SIZE(pglist_data)=53248
SIZE(zone)=12288
SIZE(free_area)=88
SIZE(list_head)=16
SIZE(nodemask_t)=8
OFFSET(page.flags)=0
OFFSET(page._count)=28
OFFSET(page.mapping)=8
OFFSET(page.lru)=32
OFFSET(page._mapcount)=24
OFFSET(page.private)=48
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=49240
OFFSET(pglist_data.node_start_pfn)=49304
OFFSET(pglist_data.node_spanned_pages)=49320
OFFSET(pglist_data.node_id)=49328
OFFSET(zone.free_area)=256
OFFSET(zone.vm_stat)=4280
OFFSET(zone.spanned_pages)=8232
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=8
OFFSET(vmap_area.va_start)=0
OFFSET(vmap_area.list)=48
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=ffffffff9972d290
SYMBOL(log_buf_len)=ffffffff9972d288
SYMBOL(log_first_idx)=ffffffff9a11eb48
SYMBOL(log_next_idx)=ffffffff9a11eb38
SIZE(printk_log)=16
OFFSET(printk_log.ts_nsec)=0
OFFSET(printk_log.len)=8
OFFSET(printk_log.text_len)=10
OFFSET(printk_log.dict_len)=12
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=11
NUMBER(PG_swapcache)=16
NUMBER(PG_slab)=7
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
SYMBOL(phys_base)=ffffffff99713010
SYMBOL(init_level4_pgt)=ffffffff9970e000
CRASHTIME=1391826079
...
Anyway, the /proc/kallsyms file of the crashing system was saved,
and it shows this:
ffffffff99000000 T _text
and if I subtract __START_KERNEL (ffffffff80000000) from that, I get
what I presume is the kaslr offset of 0x19000000. The vmcore core
header would seeminlgy confirm that:
$ readelf -a vmcore
... [ cut ] ...
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
NOTE 0x0000000000001000 0x0000000000000000 0x0000000000000000
0x00000000000007f8 0x00000000000007f8 0
LOAD 0x0000000000002000 0xffffffff99000000 0x0000000019000000 <===
0x0000000001183000 0x0000000001183000 RWE 0
LOAD 0x0000000001185000 0xffff880000001000 0x0000000000001000
0x000000000009f000 0x000000000009f000 RWE 0
LOAD 0x0000000001224000 0xffff880000100000 0x0000000000100000
0x0000000010f00000 0x0000000010f00000 RWE 0
LOAD 0x0000000012124000 0xffff880019000000 0x0000000019000000
0x0000000005194000 0x0000000005194000 RWE 0
LOAD 0x00000000172b8000 0xffff88001e1c1000 0x000000001e1c1000
0x00000000017c0000 0x00000000017c0000 RWE 0
LOAD 0x0000000018a78000 0xffff88001f9e5000 0x000000001f9e5000
0x00000000005fb000 0x00000000005fb000 RWE 0
But if I try that value with your patch applied, it fails in the same manner
as if I don't use the --kaslr option at all:
$ crash --kaslr 0x19000000 vmlinux vmcore
crash 7.0.5rc12
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: could not find MAGIC_START!
WARNING: cannot read linux_banner string
crash: vmlinux and vmcore do not match!
Usage:
crash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
Enter "crash -h" for details.
$
Any ideas? I can give you the vmlinux/vmcore/kallsyms triplet if you'd like.
Thanks,
Dave
--
Kees Cook
Chrome OS Security
8:25 a.m.
----- Original Message -----
On Mon, Feb 10, 2014 at 2:11 PM, Dave Anderson
<anderson(a)redhat.com> wrote:
>
> Hi Andy,
>
> I've got a ELF kdump vmcore that was created in-house from a kernel configured
> with CONFIG_RANDOMIZE_BASE. I thought I might be able to analyze it by applying
> your earlier patch that introduced the --kaslr option. The kernel does not
> have the offset registered in the vmcoreinfo, and so I'm trying to determine
> the offset, but with no luck.
>
> Earlier, Kees had mentioned this:
>
>>> FWIW, the offset reported during a panic to dmesg is:
>>> (unsigned long)&_text - __START_KERNEL
>
> Where does it get reported during a panic exactly? Here's the oops trace,
gotten
> by running "strings" on the vmcore:
The commit that reports it is here: f32360ef6608434a032dc7ad262d45e9693c27f3
And maybe related, but 6723734cdff15211bb78aeea76ca847374bd93ae makes
sure panic handlers are called before kmsg_dump.
And "strings" doesn't show anything starting with "Kernel
Offset:" ?
Maybe it's not being saved in the kcore?
Correct:
$ strings vmcore | grep 'Kernel Offset'
0Kernel Offset: 0x%lx from 0x%lx (relocation range: 0x%lx-0x%lx)
0Kernel Offset: 0x%lx from 0x%lx (relocation range: 0x%lx-0x%lx)
$
And that's because when kdump is enabled, crash_kexec() does not return:
void panic(const char *fmt, ...)
{
... [ cut ] ...
/*
* If we have crashed and we have a crash kernel loaded let it handle
* everything else.
* Do we want to call this before we try to display a message?
*/
crash_kexec(NULL);
/*
* Note smp_send_stop is the usual smp shutdown function, which
* unfortunately means it may not be hardened to work in a panic
* situation.
*/
smp_send_stop();
/*
* Run any panic handlers, including those that might need to
* add information to the kmsg dump output.
*/
atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
kmsg_dump(KMSG_DUMP_PANIC);
...
So for kdump, the panic_notifier_list is useless. That's why the vmcoreinfo
addition is necessary.
Dave
12:19 p.m.
But if I try that value with your patch applied, it fails in the same manner
as if I don't use the --kaslr option at all:
$ crash --kaslr 0x19000000 vmlinux vmcore
crash 7.0.5rc12
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: could not find MAGIC_START!
WARNING: cannot read linux_banner string
crash: vmlinux and vmcore do not match!
Usage:
crash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
Enter "crash -h" for details.
$
Any ideas? I can give you the vmlinux/vmcore/kallsyms triplet if you'd like.
As far as why the offset wasn't present, sorry about that. I guess
our tool chain is a bit more different than upstream, I've had a bit
of trouble generating kdump files for upstream.
Your manual calculations look correct to me though and it should work.
If you can send me the vmlinux/vmcore triplet (I don't think I need
the kallsyms) then I will look at this today. Can you post it
somewhere I can download it?
Sorry for the delay I was on vacation.
thanks,
Andy
Thanks,
Dave
2:27 p.m.
----- Original Message -----
>
> But if I try that value with your patch applied, it fails in the same manner
> as if I don't use the --kaslr option at all:
>
> $ crash --kaslr 0x19000000 vmlinux vmcore
>
> crash 7.0.5rc12
> Copyright (C) 2002-2014 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> WARNING: could not find MAGIC_START!
> WARNING: cannot read linux_banner string
> crash: vmlinux and vmcore do not match!
>
> Usage:
>
> crash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form)
> crash [OPTION]... [NAMELIST] (live system form)
>
> Enter "crash -h" for details.
> $
>
> Any ideas? I can give you the vmlinux/vmcore/kallsyms triplet if you'd
> like.
As far as why the offset wasn't present, sorry about that. I guess
our tool chain is a bit more different than upstream, I've had a bit
of trouble generating kdump files for upstream.
Your manual calculations look correct to me though and it should work.
If you can send me the vmlinux/vmcore triplet (I don't think I need
the kallsyms) then I will look at this today. Can you post it
somewhere I can download it?
Sorry for the delay I was on vacation.
thanks,
Andy
Actually you do need the kallsyms...
As I mentioned to Kees in an earlier message, the display of
the offset in the kernel log will not be done in the case of
kdump-enabled kernels, because crash_kexec() is called before
the panic_notifier list is executed, and crash_kexec() does
not return.
I've put the triplet here:
http://people.redhat.com/anderson/kaslr
Note that we can only create ELF vmcores with kASLR because the makedumpfile
facility also needs to be fixed to handle kASLR kernels.
Anyway, as it turns out, the offset is not 0x19000000 (&_text - __START_KERNEL),
but rather it is 0x18000000, which is the difference between the
kallsyms symbol values and those compiled into the vmlinux file.
With that --kaslr offset value, the crash session comes up OK.
So it seems that the (incorrect) advertised offset may not be needed,
because you can take any of the symbol values exported in the vmcoreinfo data
and compare them to their symbol value from the vmlinux file. For live systems,
the same sort of thing could be done by comparing the vmlinux symbols to the
/proc/kallsyms values. But we still should have a command line option like
your original patch as a "back-up".
Thanks,
Dave
2:29 p.m.
On Tue, Feb 18, 2014 at 12:27 PM, Dave Anderson <anderson(a)redhat.com> wrote:
----- Original Message -----
> >
> > But if I try that value with your patch applied, it fails in the same manner
> > as if I don't use the --kaslr option at all:
> >
> > $ crash --kaslr 0x19000000 vmlinux vmcore
> >
> > crash 7.0.5rc12
> > Copyright (C) 2002-2014 Red Hat, Inc.
> > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> > Copyright (C) 1999-2006 Hewlett-Packard Co
> > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> > Copyright (C) 2005, 2011 NEC Corporation
> > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > This program is free software, covered by the GNU General Public License,
> > and you are welcome to change it and/or distribute copies of it under
> > certain conditions. Enter "help copying" to see the conditions.
> > This program has absolutely no warranty. Enter "help warranty" for
> > details.
> >
> > GNU gdb (GDB) 7.6
> > Copyright (C) 2013 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later
> > <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-unknown-linux-gnu"...
> >
> > WARNING: could not find MAGIC_START!
> > WARNING: cannot read linux_banner string
> > crash: vmlinux and vmcore do not match!
> >
> > Usage:
> >
> > crash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form)
> > crash [OPTION]... [NAMELIST] (live system form)
> >
> > Enter "crash -h" for details.
> > $
> >
> > Any ideas? I can give you the vmlinux/vmcore/kallsyms triplet if you'd
> > like.
>
> As far as why the offset wasn't present, sorry about that. I guess
> our tool chain is a bit more different than upstream, I've had a bit
> of trouble generating kdump files for upstream.
>
> Your manual calculations look correct to me though and it should work.
> If you can send me the vmlinux/vmcore triplet (I don't think I need
> the kallsyms) then I will look at this today. Can you post it
> somewhere I can download it?
>
> Sorry for the delay I was on vacation.
>
> thanks,
> Andy
Actually you do need the kallsyms...
As I mentioned to Kees in an earlier message, the display of
the offset in the kernel log will not be done in the case of
kdump-enabled kernels, because crash_kexec() is called before
the panic_notifier list is executed, and crash_kexec() does
not return.
I've put the triplet here:
http://people.redhat.com/anderson/kaslr
Note that we can only create ELF vmcores with kASLR because the makedumpfile
facility also needs to be fixed to handle kASLR kernels.
Anyway, as it turns out, the offset is not 0x19000000 (&_text - __START_KERNEL),
but rather it is 0x18000000, which is the difference between the
kallsyms symbol values and those compiled into the vmlinux file.
With that --kaslr offset value, the crash session comes up OK.
Oh, hm. Is the difference maybe due to CONFIG_PHYSICAL_START some how?
-Kees
So it seems that the (incorrect) advertised offset may not be
needed,
because you can take any of the symbol values exported in the vmcoreinfo data
and compare them to their symbol value from the vmlinux file. For live systems,
the same sort of thing could be done by comparing the vmlinux symbols to the
/proc/kallsyms values. But we still should have a command line option like
your original patch as a "back-up".
Thanks,
Dave
--
Kees Cook
Chrome OS Security
2:39 p.m.
----- Original Message -----
On Tue, Feb 18, 2014 at 12:27 PM, Dave Anderson
<anderson(a)redhat.com> wrote:
>
>
> ----- Original Message -----
>> >
>> > But if I try that value with your patch applied, it fails in the same
>> > manner
>> > as if I don't use the --kaslr option at all:
>> >
>> > $ crash --kaslr 0x19000000 vmlinux vmcore
>> >
>> > crash 7.0.5rc12
>> > Copyright (C) 2002-2014 Red Hat, Inc.
>> > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
>> > Copyright (C) 1999-2006 Hewlett-Packard Co
>> > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
>> > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
>> > Copyright (C) 2005, 2011 NEC Corporation
>> > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
>> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
>> > This program is free software, covered by the GNU General Public
>> > License,
>> > and you are welcome to change it and/or distribute copies of it under
>> > certain conditions. Enter "help copying" to see the
conditions.
>> > This program has absolutely no warranty. Enter "help warranty"
for
>> > details.
>> >
>> > GNU gdb (GDB) 7.6
>> > Copyright (C) 2013 Free Software Foundation, Inc.
>> > License GPLv3+: GNU GPL version 3 or later
>> > <http://gnu.org/licenses/gpl.html>
>> > This is free software: you are free to change and redistribute it.
>> > There is NO WARRANTY, to the extent permitted by law. Type "show
>> > copying"
>> > and "show warranty" for details.
>> > This GDB was configured as "x86_64-unknown-linux-gnu"...
>> >
>> > WARNING: could not find MAGIC_START!
>> > WARNING: cannot read linux_banner string
>> > crash: vmlinux and vmcore do not match!
>> >
>> > Usage:
>> >
>> > crash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form)
>> > crash [OPTION]... [NAMELIST] (live system form)
>> >
>> > Enter "crash -h" for details.
>> > $
>> >
>> > Any ideas? I can give you the vmlinux/vmcore/kallsyms triplet if
you'd
>> > like.
>>
>> As far as why the offset wasn't present, sorry about that. I guess
>> our tool chain is a bit more different than upstream, I've had a bit
>> of trouble generating kdump files for upstream.
>>
>> Your manual calculations look correct to me though and it should work.
>> If you can send me the vmlinux/vmcore triplet (I don't think I need
>> the kallsyms) then I will look at this today. Can you post it
>> somewhere I can download it?
>>
>> Sorry for the delay I was on vacation.
>>
>> thanks,
>> Andy
>
> Actually you do need the kallsyms...
>
> As I mentioned to Kees in an earlier message, the display of
> the offset in the kernel log will not be done in the case of
> kdump-enabled kernels, because crash_kexec() is called before
> the panic_notifier list is executed, and crash_kexec() does
> not return.
>
> I've put the triplet here:
>
> http://people.redhat.com/anderson/kaslr
>
> Note that we can only create ELF vmcores with kASLR because the
> makedumpfile
> facility also needs to be fixed to handle kASLR kernels.
>
> Anyway, as it turns out, the offset is not 0x19000000 (&_text -
> __START_KERNEL),
> but rather it is 0x18000000, which is the difference between the
> kallsyms symbol values and those compiled into the vmlinux file.
> With that --kaslr offset value, the crash session comes up OK.
Oh, hm. Is the difference maybe due to CONFIG_PHYSICAL_START some how?
-Kees
Right -- I didn't create the kernel, but I presume both CONFIG_PHYSICAL_START
and CONFIG_PHYSICAL_ALIGN are both 16MB, resulting in:
$ grep _text kallsyms | head -1
ffffffff99000000 T _text
$ nm -Bn vmlinux | grep _text | head -1
ffffffff81000000 T _text
$
Dave
3863
days inactive
3871
days old
devel@lists.crash-utility.osci.io
6 comments
3 participants
participants (3)
-
Andrew Honig -
Dave Anderson -
Kees Cook