12:41 a.m.
Good point, enough the –kaslr=auto option worked well. Same when I passed
--kaslr=0x8000000
root@instance-2:~# crash --kaslr=auto vmlinux-17162.336.25 /proc/kcore
crash 8.0.4
Copyright (C) 2002-2022 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2022 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
Copyright (C) 2015, 2021 VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
KERNEL: vmlinux-17162.336.25 [TAINTED]
DUMPFILE: /proc/kcore
CPUS: 2
DATE: Wed Nov 22 06:37:56 UTC 2023
UPTIME: 19:15:54
LOAD AVERAGE: 0.15, 0.03, 0.01
TASKS: 132
NODENAME: instance-2
RELEASE: 5.15.133+
VERSION: #1 SMP Sat Nov 11 11:15:28 UTC 2023
MACHINE: x86_64 (2249 Mhz)
MEMORY: 4 GB
PID: 160180
COMMAND: "crash"
TASK: ffff8ec242ec53c0 [THREAD_INFO: ffff8ec242ec53c0]
CPU: 1
STATE: TASK_RUNNING (ACTIVE)
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 ffffffff8a616540 RU 0.0 0 0 [swapper/0]
0 0 1 ffff8ec240276480 RU 0.0 0 0
[swapper/1]
1 0 1 ffff8ec24025c300 IN 0.2 96020 9660 systemd
2 0 0 ffff8ec240258000 IN 0.0 0 0 [kthreadd]
3 2 0 ffff8ec24025e480 ID 0.0 0 0 [rcu_gp]
4 2 0 ffff8ec24025a180 ID 0.0 0 0 [rcu_par_gp]
5 2 0 ffff8ec24025b240 ID 0.0 0 0 [slub_flushwq]
From: HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab(a)nec.com>
Date: Wednesday, November 22, 2023 at 8:36 AM
To: Matt Suiche <matt.suiche(a)magnetforensics.com>, devel(a)lists.crash-utility.osci.io
<devel(a)lists.crash-utility.osci.io>
Subject: EXTERNAL SENDER Re: [Crash-utility] Google Container OS and crash 8.0.4
Hi Matt,
Thank you for trying the latest.
SYMBOL(_stext)=ffffffff89000000
KERNELOFFSET=8000000
<readmem: ffffffff82239750, KVADDR, "page_offset_base",
8, (FOE|Q),
5642aae35c08>
$ curl -O
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstorage...
$ nm vmlinux | grep -e ' _stext' -e ' page_offset_base'
ffffffff81000000 T _stext
ffffffff82239750 R page_offset_base
To me, it looks like KASLR detection doesn't work. The randomized
offset of the page_offset_base should be 0xffffffff82239750 + 0x8000000
= 0xffffffff8a239750, but crash is trying to read 0xffffffff82239750.
We need to look into why it doesn't work, firstly does this option work?
If this works, I think it will be a clue.
# crash --kaslr=auto vmlinux /proc/kcore
or
# crash --kaslr=<KERNELOFFSET value> vmlinux /proc/kcore
i.e. --kaslr=8000000 during that system session.
(this will vary after system reboot)
Thanks,
Kazu
On 2023/11/21 23:21, Matt Suiche wrote:
> Dear,
>
> I tried to use crash 8.0.4 on Google Container OS (17162.336.25) but for some reason
there is resistance.
>
> Step to reproduce:
>
> 1. Create a Virtual Machine in Google Cloud using Google Container OS as a base
image
> 2. Run “toolkit”
> 3. Download the vmlinux symbols for the current base image
> * curl
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstorage...
> symbols/vmlinux-$container_host_build_id
> 4. Run crash on /proc/kcore
>
> Thanks,
>
> Logs:
>
> root@instance-2:~# crash /proc/kcore vmlinux-17162.336.25 -d 99
>
>
>
> crash 8.0.4
>
> Copyright (C) 2002-2022 Red Hat, Inc.
>
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
>
> Copyright (C) 1999-2006 Hewlett-Packard Co
>
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
>
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
>
> Copyright (C) 2005, 2011, 2020-2022 NEC Corporation
>
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
>
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
>
> Copyright (C) 2015, 2021 VMware, Inc.
>
> This program is free software, covered by the GNU General Public License,
>
> and you are welcome to change it and/or distribute copies of it under
>
> certain conditions. Enter "help copying" to see the conditions.
>
> This program has absolutely no warranty. Enter "help warranty" for
details.
>
>
>
> get_live_memory_source: /proc/kcore
>
> proc_kcore_data:
>
> flags: 500 (KCORE_LOCAL|KCORE_ELF64)
>
> segments: 12
>
> elf_header: 5642ab6d3f40
>
> header_size: 8636
>
> notes64: 5642ab6d3f80
>
> load64: 5642ab6d3fb8
>
> notes32: 0
>
> load32: 0
>
> vmcoreinfo: 0
>
> size_vmcoreinfo: 0
>
>
>
> Elf64_Phdr:
>
> p_type: 4 (PT_NOTE)
>
> p_flags: 0
>
> p_offset: 318
>
> p_vaddr: 0
>
> p_paddr: 0
>
> p_filesz: 7844
>
> p_memsz: 0
>
> p_align: 0
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 7fff89003000
>
> p_vaddr: ffffffff89000000
>
> p_paddr: 13a000000
>
> p_filesz: 35831808
>
> p_memsz: 35831808
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 130900003000
>
> p_vaddr: ffff930900000000
>
> p_paddr: ffffffffffffffff
>
> p_filesz: 35184372088831
>
> p_memsz: 35184372088831
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 7fffc0003000
>
> p_vaddr: ffffffffc0000000
>
> p_paddr: ffffffffffffffff
>
> p_filesz: 1056964608
>
> p_memsz: 1056964608
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: ec140004000
>
> p_vaddr: ffff8ec140001000
>
> p_paddr: 1000
>
> p_filesz: 344064
>
> p_memsz: 344064
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 7be8c0003000
>
> p_vaddr: fffffbe8c0000000
>
> p_paddr: ffffffffffffffff
>
> p_filesz: 8192
>
> p_memsz: 8192
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: ec140063000
>
> p_vaddr: ffff8ec140060000
>
> p_paddr: 60000
>
> p_filesz: 229376
>
> p_memsz: 229376
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: ec140103000
>
> p_vaddr: ffff8ec140100000
>
> p_paddr: 100000
>
> p_filesz: 3212759040
>
> p_memsz: 3212759040
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 7be8c0007000
>
> p_vaddr: fffffbe8c0004000
>
> p_paddr: ffffffffffffffff
>
> p_filesz: 50200576
>
> p_memsz: 50200576
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: ec1ffc02000
>
> p_vaddr: ffff8ec1ffbff000
>
> p_paddr: bfbff000
>
> p_filesz: 4067328
>
> p_memsz: 4067328
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 7be8c2ff2000
>
> p_vaddr: fffffbe8c2fef000
>
> p_paddr: ffffffffffffffff
>
> p_filesz: 69632
>
> p_memsz: 69632
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: ec240003000
>
> p_vaddr: ffff8ec240000000
>
> p_paddr: 100000000
>
> p_filesz: 1073741824
>
> p_memsz: 1073741824
>
> p_align: 4096
>
>
>
> Elf64_Phdr:
>
> p_type: 1 (PT_LOAD)
>
> p_flags: 7
>
> p_offset: 7be8c4003000
>
> p_vaddr: fffffbe8c4000000
>
> p_paddr: ffffffffffffffff
>
> p_filesz: 16777216
>
> p_memsz: 16777216
>
> p_align: 4096
>
>
>
> Elf64_Nhdr:
>
> n_namesz: 5 ("CORE")
>
> n_descsz: 336
>
> n_type: 1 (NT_PRSTATUS)
>
>
>
> Elf64_Nhdr:
>
> n_namesz: 5 ("CORE")
>
> n_descsz: 136
>
> n_type: 3 (NT_PRPSINFO)
>
>
>
> Elf64_Nhdr:
>
> n_namesz: 5 ("CORE")
>
> n_descsz: 4288
>
> n_type: 4 (NT_TASKSTRUCT)
>
>
>
> Elf64_Nhdr:
>
> n_namesz: 11 ("VMCOREINFO")
>
> n_descsz: 3000
>
> n_type: 0 (unknown)
>
>
>
> OSRELEASE=5.15.133+
>
> BUILD-ID=f16c9f1b53617d7b151c4d18d79c6ccbb44ea6d6
>
> PAGESIZE=4096
>
> SYMBOL(init_uts_ns)=ffffffff8a615698
>
> OFFSET(uts_namespace.name)=0
>
> SYMBOL(node_online_map)=ffffffff8a85d638
>
> SYMBOL(swapper_pg_dir)=ffffffff8a60c000
>
SYMBOL(_stext)=ffffffff89000000
>
> SYMBOL(vmap_area_list)=ffffffff8a774208
>
> SYMBOL(mem_section)=ffff8ec27fff8000
>
> LENGTH(mem_section)=2048
>
> SIZE(mem_section)=16
>
> OFFSET(mem_section.section_mem_map)=0
>
> NUMBER(SECTION_SIZE_BITS)=27
>
> NUMBER(MAX_PHYSMEM_BITS)=46
>
> SIZE(page)=64
>
> SIZE(pglist_data)=15616
>
> SIZE(zone)=1664
>
> SIZE(free_area)=104
>
> SIZE(list_head)=16
>
> SIZE(nodemask_t)=8
>
> OFFSET(page.flags)=0
>
> OFFSET(page._refcount)=52
>
> OFFSET(page.mapping)=24
>
> OFFSET(page.lru)=8
>
> OFFSET(page._mapcount)=48
>
> OFFSET(page.private)=40
>
> OFFSET(page.compound_dtor)=16
>
> OFFSET(page.compound_order)=17
>
> OFFSET(page.compound_head)=8
>
> OFFSET(pglist_data.node_zones)=0
>
> OFFSET(pglist_data.nr_zones)=14880
>
> OFFSET(pglist_data.node_start_pfn)=14888
>
> OFFSET(pglist_data.node_spanned_pages)=14904
>
> OFFSET(pglist_data.node_id)=14912
>
> OFFSET(zone.free_area)=192
>
> OFFSET(zone.vm_stat)=1472
>
> OFFSET(zone.spanned_pages)=128
>
> OFFSET(free_area.free_list)=0
>
> OFFSET(list_head.next)=0
>
> OFFSET(list_head.prev)=8
>
> OFFSET(vmap_area.va_start)=0
>
> OFFSET(vmap_area.list)=40
>
> LENGTH(zone.free_area)=11
>
> SYMBOL(prb)=ffffffff8a662318
>
> SYMBOL(printk_rb_static)=ffffffff8a662320
>
> SYMBOL(clear_seq)=ffffffff8ad8c0d8
>
> SIZE(printk_ringbuffer)=80
>
> OFFSET(printk_ringbuffer.desc_ring)=0
>
> OFFSET(printk_ringbuffer.text_data_ring)=40
>
> OFFSET(printk_ringbuffer.fail)=72
>
> SIZE(prb_desc_ring)=40
>
> OFFSET(prb_desc_ring.count_bits)=0
>
> OFFSET(prb_desc_ring.descs)=8
>
> OFFSET(prb_desc_ring.infos)=16
>
> OFFSET(prb_desc_ring.head_id)=24
>
> OFFSET(prb_desc_ring.tail_id)=32
>
> SIZE(prb_desc)=24
>
> OFFSET(prb_desc.state_var)=0
>
> OFFSET(prb_desc.text_blk_lpos)=8
>
> SIZE(prb_data_blk_lpos)=16
>
> OFFSET(prb_data_blk_lpos.begin)=0
>
> OFFSET(prb_data_blk_lpos.next)=8
>
> SIZE(printk_info)=88
>
> OFFSET(printk_info.seq)=0
>
> OFFSET(printk_info.ts_nsec)=8
>
> OFFSET(printk_info.text_len)=16
>
> OFFSET(printk_info.caller_id)=20
>
> OFFSET(printk_info.dev_info)=24
>
> SIZE(dev_printk_info)=64
>
> OFFSET(dev_printk_info.subsystem)=0
>
> LENGTH(printk_info_subsystem)=16
>
> OFFSET(dev_printk_info.device)=16
>
> LENGTH(printk_info_device)=48
>
> SIZE(prb_data_ring)=32
>
> OFFSET(prb_data_ring.size_bits)=0
>
> OFFSET(prb_data_ring.data)=8
>
> OFFSET(prb_data_ring.head_lpos)=16
>
> OFFSET(prb_data_ring.tail_lpos)=24
>
> SIZE(atomic_long_t)=8
>
> OFFSET(atomic_long_t.counter)=0
>
> SIZE(latched_seq)=24
>
> OFFSET(latched_seq.val)=8
>
> LENGTH(free_area.free_list)=6
>
> NUMBER(NR_FREE_PAGES)=0
>
> NUMBER(PG_lru)=4
>
> NUMBER(PG_private)=13
>
> NUMBER(PG_swapcache)=10
>
> NUMBER(PG_swapbacked)=19
>
> NUMBER(PG_slab)=9
>
> NUMBER(PG_hwpoison)=23
>
> NUMBER(PG_head_mask)=65536
>
> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
>
> NUMBER(HUGETLB_PAGE_DTOR)=2
>
> NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
>
> NUMBER(phys_base)=5117050880
>
> SYMBOL(init_top_pgt)=ffffffff8a60c000
>
> NUMBER(pgtable_l5_enabled)=0
>
> SYMBOL(node_data)=ffffffff8a85c5d0
>
> LENGTH(node_data)=64
>
KERNELOFFSET=8000000
>
> NUMBER(KERNEL_IMAGE_SIZE)=1073741824
>
> NUMBER(sme_mask)=0
>
>
>
> /proc/version:
>
> Linux version 5.15.133+ (builder@localhost) (Chromium OS 14.0_pre445002_p20220217-r3
clang version 14.0.0
(/var/tmp/portage/sys-devel/llvm-14.0_pre445002_p20220217-r3/work/llvm-14.0_pre445002_p20220217/clang
18308e171b5b1dd99627a4d88c7d6c5ff21b8c96), LLD 14.0.0) #1 SMP Sat Nov 11 11:15:28 UTC
2023
>
> vmlinux-17162.336.25:
>
> Linux version 5.15.133+ (builder@localhost) (Chromium OS 14.0_pre445002_p20220217-r3
clang version 14.0.0
(/var/tmp/portage/sys-devel/llvm-14.0_pre445002_p20220217-r3/work/llvm-14.0_pre445002_p20220217/clang
18308e171b5b1dd99627a4d88c7d6c5ff21b8c96), LLD 14.0.0) #1 SMP Sat Nov 11 11:15:28 UTC
2023
>
> readmem: read_proc_kcore() -> /proc/kcore
>
> crash: pv_ops exists: ARCH_PVOPS
>
> VMCOREINFO: NUMBER(phys_base): 5117050880 -> 131000000
>
> gdb vmlinux-17162.336.25
>
> GNU gdb (GDB) 10.2
>
> Copyright (C) 2021 Free Software Foundation, Inc.
>
> License GPLv3+: GNU GPL version 3 or later
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgnu.org%...
>
> This is free software: you are free to change and redistribute it.
>
> There is NO WARRANTY, to the extent permitted by law.
>
> Type "show copying" and "show warranty" for details.
>
> This GDB was configured as "x86_64-pc-linux-gnu".
>
> Type "show configuration" for configuration details.
>
> Find the GDB manual and other documentation resources online at:
>
>
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gnu....;.
>
>
>
> For help, type "help".
>
> Type "apropos word" to search for commands related to "word"...
>
> GETBUF(344 -> 0)
>
> GETBUF(1500 -> 1)
>
>
>
> FREEBUF(1)
>
> FREEBUF(0)
>
> <readmem: ffffffff82239750, KVADDR, "page_offset_base", 8, (FOE|Q),
5642aae35c08>
>
> <read_proc_kcore: addr: ffffffff82239750 paddr: 133239750 cnt: 8>
>
> crash: seek error: kernel virtual address: ffffffff82239750 type:
"page_offset_base"
>
>
>
> root@instance-2:~# env
>
> container_host_version_id=101
>
> PWD=/root
>
> LOGNAME=root
>
> container=systemd-nspawn
>
> HOME=/root
>
> TERM=xterm-256color
>
> USER=root
>
> NOTIFY_SOCKET=/run/host/notify
>
> SHLVL=1
>
> container_host_id=cos
>
> container_host_build_id=17162.336.25
>
> PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>
> container_uuid=d8282d15-c11a-416b-9371-94db01a7ca15
>
> _=/usr/bin/env
>
> OLDPWD=/
>
>
> This email including any attachments may contain confidential material for the sole
use of the intended recipient. If you are not the intended recipient please immediately
notify the sender by reply email, permanently delete this message and do not forward it or
any part of it to anyone else.
>
This email including any attachments may contain confidential material for the sole use of
the intended recipient. If you are not the intended recipient please immediately notify
the sender by reply email, permanently delete this message and do not forward it or any
part of it to anyone else.