From: HAGIO KAZUHITO(萩尾　一仁) <k-hagio-ab@nec.com>
Date: Wednesday, November 22, 2023 at 8:36 AM
To: Matt Suiche <matt.suiche@magnetforensics.com>, devel@lists.crash-utility.osci.io <devel@lists.crash-utility.osci.io>
Subject: EXTERNAL SENDER Re: [Crash-utility] Google Container OS and crash 8.0.4

Hi Matt,

Thank you for trying the latest.

>        SYMBOL(_stext)=ffffffff89000000

>        KERNELOFFSET=8000000

> <readmem: ffffffff82239750, KVADDR, "page_offset_base", 8, (FOE|Q),
5642aae35c08>

$ curl -O https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstorage.googleapis.com%2Fcos-tools%2F17162.336.25%2Fvmlinux&data=05%7C01%7Cmatt.suiche%40magnetforensics.com%7C0bd964f454dc4c6f88f808dbeb14902f%7C631f5ed9a7994296b4149aee47fce3dd%7C1%7C0%7C638362245856890013%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vA2UZsSrOk5soDnqJPbWBcj4zxmgHlFMAc5Qzt9jW9A%3D&reserved=0
$ nm vmlinux | grep -e ' _stext' -e ' page_offset_base'
ffffffff81000000 T _stext
ffffffff82239750 R page_offset_base

To me, it looks like KASLR detection doesn't work.  The randomized
offset of the page_offset_base should be 0xffffffff82239750 + 0x8000000
= 0xffffffff8a239750, but crash is trying to read 0xffffffff82239750.

We need to look into why it doesn't work, firstly does this option work?
  If this works, I think it will be a clue.

# crash --kaslr=auto vmlinux /proc/kcore
   or
# crash --kaslr=<KERNELOFFSET value> vmlinux /proc/kcore

   i.e. --kaslr=8000000 during that system session.
   (this will vary after system reboot)

Thanks,
Kazu


On 2023/11/21 23:21, Matt Suiche wrote:
> Dear,
>
> I tried to use crash 8.0.4 on Google Container OS (17162.336.25) but for some reason there is resistance.
>
> Step to reproduce:
>
>    1.  Create a Virtual Machine in Google Cloud using Google Container OS as a base image
>    2.  Run “toolkit”
>    3.  Download the vmlinux symbols for the current base image
>       *   curl https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstorage.googleapis.com%2Fcos-tools%2F%24container_host_build_id%2Fvmlinux&data=05%7C01%7Cmatt.suiche%40magnetforensics.com%7C0bd964f454dc4c6f88f808dbeb14902f%7C631f5ed9a7994296b4149aee47fce3dd%7C1%7C0%7C638362245856890013%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=D0p5i7vh4r71U4jPcbJudkhKgTnTlLiBOkhdyBf4v8o%3D&reserved=0 > symbols/vmlinux-$container_host_build_id
>    4.  Run crash on /proc/kcore
>
> Thanks,
>
> Logs:
>
> root@instance-2:~# crash /proc/kcore vmlinux-17162.336.25 -d 99
>
>
>
> crash 8.0.4
>
> Copyright (C) 2002-2022  Red Hat, Inc.
>
> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
>
> Copyright (C) 1999-2006  Hewlett-Packard Co
>
> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
>
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
>
> Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
>
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
>
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
>
> Copyright (C) 2015, 2021  VMware, Inc.
>
> This program is free software, covered by the GNU General Public License,
>
> and you are welcome to change it and/or distribute copies of it under
>
> certain conditions.  Enter "help copying" to see the conditions.
>
> This program has absolutely no warranty.  Enter "help warranty" for details.
>
>
>
> get_live_memory_source: /proc/kcore
>
> proc_kcore_data:
>
>             flags: 500 (KCORE_LOCAL|KCORE_ELF64)
>
>          segments: 12
>
>        elf_header: 5642ab6d3f40
>
>       header_size: 8636
>
>           notes64: 5642ab6d3f80
>
>            load64: 5642ab6d3fb8
>
>           notes32: 0
>
>            load32: 0
>
>        vmcoreinfo: 0
>
>   size_vmcoreinfo: 0
>
>
>
>    Elf64_Phdr:
>
>          p_type: 4 (PT_NOTE)
>
>         p_flags: 0
>
>        p_offset: 318
>
>         p_vaddr: 0
>
>         p_paddr: 0
>
>        p_filesz: 7844
>
>         p_memsz: 0
>
>         p_align: 0
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 7fff89003000
>
>         p_vaddr: ffffffff89000000
>
>         p_paddr: 13a000000
>
>        p_filesz: 35831808
>
>         p_memsz: 35831808
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 130900003000
>
>         p_vaddr: ffff930900000000
>
>         p_paddr: ffffffffffffffff
>
>        p_filesz: 35184372088831
>
>         p_memsz: 35184372088831
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 7fffc0003000
>
>         p_vaddr: ffffffffc0000000
>
>         p_paddr: ffffffffffffffff
>
>        p_filesz: 1056964608
>
>         p_memsz: 1056964608
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: ec140004000
>
>         p_vaddr: ffff8ec140001000
>
>         p_paddr: 1000
>
>        p_filesz: 344064
>
>         p_memsz: 344064
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 7be8c0003000
>
>         p_vaddr: fffffbe8c0000000
>
>         p_paddr: ffffffffffffffff
>
>        p_filesz: 8192
>
>         p_memsz: 8192
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: ec140063000
>
>         p_vaddr: ffff8ec140060000
>
>         p_paddr: 60000
>
>        p_filesz: 229376
>
>         p_memsz: 229376
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: ec140103000
>
>         p_vaddr: ffff8ec140100000
>
>         p_paddr: 100000
>
>        p_filesz: 3212759040
>
>         p_memsz: 3212759040
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 7be8c0007000
>
>         p_vaddr: fffffbe8c0004000
>
>         p_paddr: ffffffffffffffff
>
>        p_filesz: 50200576
>
>         p_memsz: 50200576
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: ec1ffc02000
>
>         p_vaddr: ffff8ec1ffbff000
>
>         p_paddr: bfbff000
>
>        p_filesz: 4067328
>
>         p_memsz: 4067328
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 7be8c2ff2000
>
>         p_vaddr: fffffbe8c2fef000
>
>         p_paddr: ffffffffffffffff
>
>        p_filesz: 69632
>
>         p_memsz: 69632
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: ec240003000
>
>         p_vaddr: ffff8ec240000000
>
>         p_paddr: 100000000
>
>        p_filesz: 1073741824
>
>         p_memsz: 1073741824
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 7be8c4003000
>
>         p_vaddr: fffffbe8c4000000
>
>         p_paddr: ffffffffffffffff
>
>        p_filesz: 16777216
>
>         p_memsz: 16777216
>
>         p_align: 4096
>
>
>
>    Elf64_Nhdr:
>
>        n_namesz: 5 ("CORE")
>
>        n_descsz: 336
>
>          n_type: 1 (NT_PRSTATUS)
>
>
>
>    Elf64_Nhdr:
>
>        n_namesz: 5 ("CORE")
>
>        n_descsz: 136
>
>          n_type: 3 (NT_PRPSINFO)
>
>
>
>    Elf64_Nhdr:
>
>        n_namesz: 5 ("CORE")
>
>        n_descsz: 4288
>
>          n_type: 4 (NT_TASKSTRUCT)
>
>
>
>    Elf64_Nhdr:
>
>        n_namesz: 11 ("VMCOREINFO")
>
>        n_descsz: 3000
>
>          n_type: 0 (unknown)
>
>
>
>        OSRELEASE=5.15.133+
>
>        BUILD-ID=f16c9f1b53617d7b151c4d18d79c6ccbb44ea6d6
>
>        PAGESIZE=4096
>
>        SYMBOL(init_uts_ns)=ffffffff8a615698
>
>        OFFSET(uts_namespace.name)=0
>
>        SYMBOL(node_online_map)=ffffffff8a85d638
>
>        SYMBOL(swapper_pg_dir)=ffffffff8a60c000
>
>        SYMBOL(_stext)=ffffffff89000000
>
>        SYMBOL(vmap_area_list)=ffffffff8a774208
>
>        SYMBOL(mem_section)=ffff8ec27fff8000
>
>        LENGTH(mem_section)=2048
>
>        SIZE(mem_section)=16
>
>        OFFSET(mem_section.section_mem_map)=0
>
>        NUMBER(SECTION_SIZE_BITS)=27
>
>        NUMBER(MAX_PHYSMEM_BITS)=46
>
>        SIZE(page)=64
>
>        SIZE(pglist_data)=15616
>
>        SIZE(zone)=1664
>
>        SIZE(free_area)=104
>
>        SIZE(list_head)=16
>
>        SIZE(nodemask_t)=8
>
>        OFFSET(page.flags)=0
>
>        OFFSET(page._refcount)=52
>
>        OFFSET(page.mapping)=24
>
>        OFFSET(page.lru)=8
>
>        OFFSET(page._mapcount)=48
>
>        OFFSET(page.private)=40
>
>        OFFSET(page.compound_dtor)=16
>
>        OFFSET(page.compound_order)=17
>
>        OFFSET(page.compound_head)=8
>
>        OFFSET(pglist_data.node_zones)=0
>
>        OFFSET(pglist_data.nr_zones)=14880
>
>        OFFSET(pglist_data.node_start_pfn)=14888
>
>        OFFSET(pglist_data.node_spanned_pages)=14904
>
>        OFFSET(pglist_data.node_id)=14912
>
>        OFFSET(zone.free_area)=192
>
>        OFFSET(zone.vm_stat)=1472
>
>        OFFSET(zone.spanned_pages)=128
>
>        OFFSET(free_area.free_list)=0
>
>        OFFSET(list_head.next)=0
>
>        OFFSET(list_head.prev)=8
>
>        OFFSET(vmap_area.va_start)=0
>
>        OFFSET(vmap_area.list)=40
>
>        LENGTH(zone.free_area)=11
>
>        SYMBOL(prb)=ffffffff8a662318
>
>        SYMBOL(printk_rb_static)=ffffffff8a662320
>
>        SYMBOL(clear_seq)=ffffffff8ad8c0d8
>
>        SIZE(printk_ringbuffer)=80
>
>        OFFSET(printk_ringbuffer.desc_ring)=0
>
>        OFFSET(printk_ringbuffer.text_data_ring)=40
>
>        OFFSET(printk_ringbuffer.fail)=72
>
>        SIZE(prb_desc_ring)=40
>
>        OFFSET(prb_desc_ring.count_bits)=0
>
>        OFFSET(prb_desc_ring.descs)=8
>
>        OFFSET(prb_desc_ring.infos)=16
>
>        OFFSET(prb_desc_ring.head_id)=24
>
>        OFFSET(prb_desc_ring.tail_id)=32
>
>        SIZE(prb_desc)=24
>
>        OFFSET(prb_desc.state_var)=0
>
>        OFFSET(prb_desc.text_blk_lpos)=8
>
>        SIZE(prb_data_blk_lpos)=16
>
>        OFFSET(prb_data_blk_lpos.begin)=0
>
>        OFFSET(prb_data_blk_lpos.next)=8
>
>        SIZE(printk_info)=88
>
>        OFFSET(printk_info.seq)=0
>
>        OFFSET(printk_info.ts_nsec)=8
>
>        OFFSET(printk_info.text_len)=16
>
>        OFFSET(printk_info.caller_id)=20
>
>        OFFSET(printk_info.dev_info)=24
>
>        SIZE(dev_printk_info)=64
>
>        OFFSET(dev_printk_info.subsystem)=0
>
>        LENGTH(printk_info_subsystem)=16
>
>        OFFSET(dev_printk_info.device)=16
>
>        LENGTH(printk_info_device)=48
>
>        SIZE(prb_data_ring)=32
>
>        OFFSET(prb_data_ring.size_bits)=0
>
>        OFFSET(prb_data_ring.data)=8
>
>        OFFSET(prb_data_ring.head_lpos)=16
>
>        OFFSET(prb_data_ring.tail_lpos)=24
>
>        SIZE(atomic_long_t)=8
>
>        OFFSET(atomic_long_t.counter)=0
>
>        SIZE(latched_seq)=24
>
>        OFFSET(latched_seq.val)=8
>
>        LENGTH(free_area.free_list)=6
>
>        NUMBER(NR_FREE_PAGES)=0
>
>        NUMBER(PG_lru)=4
>
>        NUMBER(PG_private)=13
>
>        NUMBER(PG_swapcache)=10
>
>        NUMBER(PG_swapbacked)=19
>
>        NUMBER(PG_slab)=9
>
>        NUMBER(PG_hwpoison)=23
>
>        NUMBER(PG_head_mask)=65536
>
>        NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
>
>        NUMBER(HUGETLB_PAGE_DTOR)=2
>
>        NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
>
>        NUMBER(phys_base)=5117050880
>
>        SYMBOL(init_top_pgt)=ffffffff8a60c000
>
>        NUMBER(pgtable_l5_enabled)=0
>
>        SYMBOL(node_data)=ffffffff8a85c5d0
>
>        LENGTH(node_data)=64
>
>        KERNELOFFSET=8000000
>
>        NUMBER(KERNEL_IMAGE_SIZE)=1073741824
>
>        NUMBER(sme_mask)=0
>
>
>
> /proc/version:
>
> Linux version 5.15.133+ (builder@localhost) (Chromium OS 14.0_pre445002_p20220217-r3 clang version 14.0.0 (/var/tmp/portage/sys-devel/llvm-14.0_pre445002_p20220217-r3/work/llvm-14.0_pre445002_p20220217/clang 18308e171b5b1dd99627a4d88c7d6c5ff21b8c96), LLD 14.0.0) #1 SMP Sat Nov 11 11:15:28 UTC 2023
>
> vmlinux-17162.336.25:
>
> Linux version 5.15.133+ (builder@localhost) (Chromium OS 14.0_pre445002_p20220217-r3 clang version 14.0.0 (/var/tmp/portage/sys-devel/llvm-14.0_pre445002_p20220217-r3/work/llvm-14.0_pre445002_p20220217/clang 18308e171b5b1dd99627a4d88c7d6c5ff21b8c96), LLD 14.0.0) #1 SMP Sat Nov 11 11:15:28 UTC 2023
>
> readmem: read_proc_kcore() -> /proc/kcore
>
> crash: pv_ops exists: ARCH_PVOPS
>
> VMCOREINFO: NUMBER(phys_base): 5117050880 -> 131000000
>
> gdb vmlinux-17162.336.25
>
> GNU gdb (GDB) 10.2
>
> Copyright (C) 2021 Free Software Foundation, Inc.
>
> License GPLv3+: GNU GPL version 3 or later https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgnu.org%2Flicenses%2Fgpl.html&data=05%7C01%7Cmatt.suiche%40magnetforensics.com%7C0bd964f454dc4c6f88f808dbeb14902f%7C631f5ed9a7994296b4149aee47fce3dd%7C1%7C0%7C638362245857046312%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FbWjtW1ptyhQp92FQdaESU%2Fu%2Fy89TzJ0jeWrCIp2i6g%3D&reserved=0
>
> This is free software: you are free to change and redistribute it.
>
> There is NO WARRANTY, to the extent permitted by law.
>
> Type "show copying" and "show warranty" for details.
>
> This GDB was configured as "x86_64-pc-linux-gnu".
>
> Type "show configuration" for configuration details.
>
> Find the GDB manual and other documentation resources online at:
>
>      https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gnu.org%2Fsoftware%2Fgdb%2Fdocumentation%2F&data=05%7C01%7Cmatt.suiche%40magnetforensics.com%7C0bd964f454dc4c6f88f808dbeb14902f%7C631f5ed9a7994296b4149aee47fce3dd%7C1%7C0%7C638362245857046312%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NLGcPFC1hwRUWLUGGbgiygIaKOKVZCGeGZsGLFvkiXk%3D&reserved=0.
>
>
>
> For help, type "help".
>
> Type "apropos word" to search for commands related to "word"...
>
> GETBUF(344 -> 0)
>
>    GETBUF(1500 -> 1)
>
>
>
>    FREEBUF(1)
>
> FREEBUF(0)
>
> <readmem: ffffffff82239750, KVADDR, "page_offset_base", 8, (FOE|Q), 5642aae35c08>
>
> <read_proc_kcore: addr: ffffffff82239750 paddr: 133239750 cnt: 8>
>
> crash: seek error: kernel virtual address: ffffffff82239750  type: "page_offset_base"
>
>
>
> root@instance-2:~# env
>
> container_host_version_id=101
>
> PWD=/root
>
> LOGNAME=root
>
> container=systemd-nspawn
>
> HOME=/root
>
> TERM=xterm-256color
>
> USER=root
>
> NOTIFY_SOCKET=/run/host/notify
>
> SHLVL=1
>
> container_host_id=cos
>
> container_host_build_id=17162.336.25
>
> PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>
> container_uuid=d8282d15-c11a-416b-9371-94db01a7ca15
>
> _=/usr/bin/env
>
> OLDPWD=/
>
>
> This email including any attachments may contain confidential material for the sole use of the intended recipient. If you are not the intended recipient please immediately notify the sender by reply email, permanently delete this message and do not forward it or any part of it to anyone else.
>