[Crash-utility] earlier patch introducing the --kaslr option

Monday, 10 February 2014

Hi Andy,

I've got a ELF kdump vmcore that was created in-house from a kernel configured
with CONFIG_RANDOMIZE_BASE.  I thought I might be able to analyze it by applying
your earlier patch that introduced the --kaslr option.  The kernel does not
have the offset registered in the vmcoreinfo, and so I'm trying to determine
the offset, but with no luck.

Earlier, Kees had mentioned this:

...
> FWIW, the offset reported during a panic to dmesg is:
>     (unsigned long)&_text - __START_KERNEL 
Where does it get reported during a panic exactly?  Here's the oops trace, gotten
by running "strings" on the vmcore:

  $ strings vmcore
  ... [ cut ] ...    
  SysRq : Trigger a crash
  "BUG: unable to handle kernel NULL pointer dereference at           (null)
  "IP: [<ffffffff992bf6cf>] sysrq_handle_crash+0x11/0x1b
  PGD 3a067 PUD 2e067 PMD 0 
  Oops: 0002 [#1] PREEMPT SMP 0"
  Modules linked in:
  CPU: 0 PID: 1720 Comm: bash Not tainted 3.14.0-rc1+ #1130"
  task: ffff88001d028000 ti: ffff88001c986000 task.ti: ffff88001c986000
  RIP: 0010:[<ffffffff992bf6cf>]  [<ffffffff992bf6cf>]
sysrq_handle_crash+0x11/0x1b
  RSP: 0018:ffff88001c987e90  EFLAGS: 000100920"
  RAX: 000000000000000f RBX: ffffffff9975ed50 RCX: 0000000000000000
  RDX: ffff88001d028000 RSI: ffff88001cc0e338 RDI: 0000000000000063
  RBP: ffff88001c987e90 R08: 0000000000000002 R09: 0000000000000000
  R10: ffffffff994e9630 R11: 0000000000000000 R12: 0000000000000007
  R13: 0000000000000246 R14: 0000000000000063 R15: 0000000000000000
  FS:  00007f0ec2181740(0000) GS:ffff88001cc00000(0000) knlGS:00000000000000000"
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 0000000000000000 CR3: 000000001d36f000 CR4: 00000000000006f0
  Stack:
   ffff88001c987ec8 ffffffff992bfc88 0000000000000002 00007f0ec21870000"
   0000000000000002 ffff88001c987f58 0000000000000000 ffff88001c987ee80"
   ffffffff992c000c ffff88001c92acc0 00007f0ec2187000 ffff88001c987f080"
  Call Trace:
   [<ffffffff992bfc88>] __handle_sysrq+0x9b/0x133
   [<ffffffff992c000c>] write_sysrq_trigger+0x2d/0x3e
   [<ffffffff991681cb>] proc_reg_write+0x45/0x65
   [<ffffffff9911897c>] vfs_write+0xbf/0x17c
   [<ffffffff9911918f>] SyS_write+0x44/0x7a
   [<ffffffff9949ad7d>] system_call_fastpath+0x1a/0x1f0"
  Code: 4f 00 00 55 b8 01 00 00 00 48 89 e5 75 07 0f b6 05 b3 20 4f 00 83 e0 01 5d c3 55
c7 05 03 18 61 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55
31 c0 c7 05 ba dc 46 00 07 00 0"
  "RIP  [<ffffffff992bf6cf>] sysrq_handle_crash+0x11/0x1b
   RSP <ffff88001c987e90>
  CR2: 0000000000000000
  ttySffffffff99000000 T _text

  UUUU
  UUUU
  VMCOREINFO
  OSRELEASE=3.14.0-rc1+
  PAGESIZE=4096
  SYMBOL(init_uts_ns)=ffffffff99713250
  SYMBOL(node_online_map)=ffffffff997b0c68
  SYMBOL(swapper_pg_dir)=ffffffff9970e000
  SYMBOL(_stext)=ffffffff990001c8
  SYMBOL(vmap_area_list)=ffffffff99745c20
  SYMBOL(mem_map)=ffffffff9a1253a8
  SYMBOL(contig_page_data)=ffffffff99790000
  SYMBOL(mem_section)=ffffffff9a126000
  LENGTH(mem_section)=2048
  SIZE(mem_section)=16
  OFFSET(mem_section.section_mem_map)=0
  SIZE(page)=64
  SIZE(pglist_data)=53248
  SIZE(zone)=12288
  SIZE(free_area)=88
  SIZE(list_head)=16
  SIZE(nodemask_t)=8
  OFFSET(page.flags)=0
  OFFSET(page._count)=28
  OFFSET(page.mapping)=8
  OFFSET(page.lru)=32
  OFFSET(page._mapcount)=24
  OFFSET(page.private)=48
  OFFSET(pglist_data.node_zones)=0
  OFFSET(pglist_data.nr_zones)=49240
  OFFSET(pglist_data.node_start_pfn)=49304
  OFFSET(pglist_data.node_spanned_pages)=49320
  OFFSET(pglist_data.node_id)=49328
  OFFSET(zone.free_area)=256
  OFFSET(zone.vm_stat)=4280
  OFFSET(zone.spanned_pages)=8232
  OFFSET(free_area.free_list)=0
  OFFSET(list_head.next)=0
  OFFSET(list_head.prev)=8
  OFFSET(vmap_area.va_start)=0
  OFFSET(vmap_area.list)=48
  LENGTH(zone.free_area)=11
  SYMBOL(log_buf)=ffffffff9972d290
  SYMBOL(log_buf_len)=ffffffff9972d288
  SYMBOL(log_first_idx)=ffffffff9a11eb48
  SYMBOL(log_next_idx)=ffffffff9a11eb38
  SIZE(printk_log)=16
  OFFSET(printk_log.ts_nsec)=0
  OFFSET(printk_log.len)=8
  OFFSET(printk_log.text_len)=10
  OFFSET(printk_log.dict_len)=12
  LENGTH(free_area.free_list)=5
  NUMBER(NR_FREE_PAGES)=0
  NUMBER(PG_lru)=5
  NUMBER(PG_private)=11
  NUMBER(PG_swapcache)=16
  NUMBER(PG_slab)=7
  NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
  SYMBOL(phys_base)=ffffffff99713010
  SYMBOL(init_level4_pgt)=ffffffff9970e000
  CRASHTIME=1391826079
  OSRELEASE=3.14.0-rc1+
  PAGESIZE=4096
  SYMBOL(init_uts_ns)=ffffffff99713250
  SYMBOL(node_online_map)=ffffffff997b0c68
  SYMBOL(swapper_pg_dir)=ffffffff9970e000
  SYMBOL(_stext)=ffffffff990001c8
  SYMBOL(vmap_area_list)=ffffffff99745c20
  SYMBOL(mem_map)=ffffffff9a1253a8
  SYMBOL(contig_page_data)=ffffffff99790000
  SYMBOL(mem_section)=ffffffff9a126000
  LENGTH(mem_section)=2048
  SIZE(mem_section)=16
  OFFSET(mem_section.section_mem_map)=0
  SIZE(page)=64
  SIZE(pglist_data)=53248
  SIZE(zone)=12288
  SIZE(free_area)=88
  SIZE(list_head)=16
  SIZE(nodemask_t)=8
  OFFSET(page.flags)=0
  OFFSET(page._count)=28
  OFFSET(page.mapping)=8
  OFFSET(page.lru)=32
  OFFSET(page._mapcount)=24
  OFFSET(page.private)=48
  OFFSET(pglist_data.node_zones)=0
  OFFSET(pglist_data.nr_zones)=49240
  OFFSET(pglist_data.node_start_pfn)=49304
  OFFSET(pglist_data.node_spanned_pages)=49320
  OFFSET(pglist_data.node_id)=49328
  OFFSET(zone.free_area)=256
  OFFSET(zone.vm_stat)=4280
  OFFSET(zone.spanned_pages)=8232
  OFFSET(free_area.free_list)=0
  OFFSET(list_head.next)=0
  OFFSET(list_head.prev)=8
  OFFSET(vmap_area.va_start)=0
  OFFSET(vmap_area.list)=48
  LENGTH(zone.free_area)=11
  SYMBOL(log_buf)=ffffffff9972d290
  SYMBOL(log_buf_len)=ffffffff9972d288
  SYMBOL(log_first_idx)=ffffffff9a11eb48
  SYMBOL(log_next_idx)=ffffffff9a11eb38
  SIZE(printk_log)=16
  OFFSET(printk_log.ts_nsec)=0
  OFFSET(printk_log.len)=8
  OFFSET(printk_log.text_len)=10
  OFFSET(printk_log.dict_len)=12
  LENGTH(free_area.free_list)=5
  NUMBER(NR_FREE_PAGES)=0
  NUMBER(PG_lru)=5
  NUMBER(PG_private)=11
  NUMBER(PG_swapcache)=16
  NUMBER(PG_slab)=7
  NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
  SYMBOL(phys_base)=ffffffff99713010
  SYMBOL(init_level4_pgt)=ffffffff9970e000
  CRASHTIME=1391826079
  ...

Anyway, the /proc/kallsyms file of the crashing system was saved,
and it shows this:

  ffffffff99000000 T _text

and if I subtract __START_KERNEL (ffffffff80000000) from that, I get
what I presume is the kaslr offset of 0x19000000.  The vmcore core
header would seeminlgy confirm that:

 $ readelf -a vmcore
 ... [ cut ] ...
  Program Headers:
   Type           Offset             VirtAddr           PhysAddr
                  FileSiz            MemSiz              Flags  Align
   NOTE           0x0000000000001000 0x0000000000000000 0x0000000000000000
                  0x00000000000007f8 0x00000000000007f8         0
   LOAD           0x0000000000002000 0xffffffff99000000 0x0000000019000000   <===
                  0x0000000001183000 0x0000000001183000  RWE    0
   LOAD           0x0000000001185000 0xffff880000001000 0x0000000000001000
                  0x000000000009f000 0x000000000009f000  RWE    0
   LOAD           0x0000000001224000 0xffff880000100000 0x0000000000100000
                  0x0000000010f00000 0x0000000010f00000  RWE    0
   LOAD           0x0000000012124000 0xffff880019000000 0x0000000019000000
                  0x0000000005194000 0x0000000005194000  RWE    0
   LOAD           0x00000000172b8000 0xffff88001e1c1000 0x000000001e1c1000
                  0x00000000017c0000 0x00000000017c0000  RWE    0
   LOAD           0x0000000018a78000 0xffff88001f9e5000 0x000000001f9e5000
                  0x00000000005fb000 0x00000000005fb000  RWE    0

But if I try that value with your patch applied, it fails in the same manner
as if I don't use the --kaslr option at all:

 $ crash --kaslr 0x19000000 vmlinux vmcore

 crash 7.0.5rc12
 Copyright (C) 2002-2014  Red Hat, Inc.
 Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
 Copyright (C) 1999-2006  Hewlett-Packard Co
 Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
 Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
 Copyright (C) 2005, 2011  NEC Corporation
 Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
 Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
 This program is free software, covered by the GNU General Public License,
 and you are welcome to change it and/or distribute copies of it under
 certain conditions.  Enter "help copying" to see the conditions.
 This program has absolutely no warranty.  Enter "help warranty" for details.

 GNU gdb (GDB) 7.6
 Copyright (C) 2013 Free Software Foundation, Inc.
 License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 This is free software: you are free to change and redistribute it.
 There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
 and "show warranty" for details.
 This GDB was configured as "x86_64-unknown-linux-gnu"...

 WARNING: could not find MAGIC_START!                                   
 WARNING: cannot read linux_banner string
 crash: vmlinux and vmcore do not match!

 Usage:

  crash [OPTION]... NAMELIST MEMORY-IMAGE  (dumpfile form)
  crash [OPTION]... [NAMELIST]             (live system form)

 Enter "crash -h" for details.
 $ 

Any ideas?  I can give you the vmlinux/vmcore/kallsyms triplet if you'd like.

Thanks,
  Dave

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

[Crash-utility] earlier patch introducing the --kaslr option