Good point, enough the ¡Vkaslr=auto option worked well. Same when I passed --kaslr=0x8000000

 

root@instance-2:~# crash --kaslr=auto vmlinux-17162.336.25 /proc/kcore

 

crash 8.0.4

Copyright (C) 2002-2022  Red Hat, Inc.

Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation

Copyright (C) 1999-2006  Hewlett-Packard Co

Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited

Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.

Copyright (C) 2005, 2011, 2020-2022  NEC Corporation

Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.

Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.

Copyright (C) 2015, 2021  VMware, Inc.

This program is free software, covered by the GNU General Public License,

and you are welcome to change it and/or distribute copies of it under

certain conditions.  Enter "help copying" to see the conditions.

This program has absolutely no warranty.  Enter "help warranty" for details.

 

GNU gdb (GDB) 10.2

Copyright (C) 2021 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

Type "show copying" and "show warranty" for details.

This GDB was configured as "x86_64-pc-linux-gnu".

Type "show configuration" for configuration details.

Find the GDB manual and other documentation resources online at:

    http://www.gnu.org/software/gdb/documentation/.

 

For help, type "help".

Type "apropos word" to search for commands related to "word"...

 

      KERNEL: vmlinux-17162.336.25  [TAINTED]

    DUMPFILE: /proc/kcore

        CPUS: 2

        DATE: Wed Nov 22 06:37:56 UTC 2023

      UPTIME: 19:15:54

LOAD AVERAGE: 0.15, 0.03, 0.01

       TASKS: 132

    NODENAME: instance-2

     RELEASE: 5.15.133+

     VERSION: #1 SMP Sat Nov 11 11:15:28 UTC 2023

     MACHINE: x86_64  (2249 Mhz)

      MEMORY: 4 GB

         PID: 160180

     COMMAND: "crash"

        TASK: ffff8ec242ec53c0  [THREAD_INFO: ffff8ec242ec53c0]

         CPU: 1

       STATE: TASK_RUNNING (ACTIVE)

 

crash> ps

      PID    PPID  CPU       TASK        ST  %MEM      VSZ      RSS  COMM

        0       0   0  ffffffff8a616540  RU   0.0        0        0  [swapper/0]

>       0       0   1  ffff8ec240276480  RU   0.0        0        0  [swapper/1]

        1       0   1  ffff8ec24025c300  IN   0.2    96020     9660  systemd

        2       0   0  ffff8ec240258000  IN   0.0        0        0  [kthreadd]

        3       2   0  ffff8ec24025e480  ID   0.0        0        0  [rcu_gp]

        4       2   0  ffff8ec24025a180  ID   0.0        0        0  [rcu_par_gp]

        5       2   0  ffff8ec24025b240  ID   0.0        0        0  [slub_flushwq]

 

 

From: HAGIO KAZUHITO(ßǧÀ¡@¤@¤¯) <k-hagio-ab@nec.com>
Date: Wednesday, November 22, 2023 at 8:36 AM
To: Matt Suiche <matt.suiche@magnetforensics.com>, devel@lists.crash-utility.osci.io <devel@lists.crash-utility.osci.io>
Subject: EXTERNAL SENDER Re: [Crash-utility] Google Container OS and crash 8.0.4

Hi Matt,

Thank you for trying the latest.

>        SYMBOL(_stext)=ffffffff89000000

>        KERNELOFFSET=8000000

> <readmem: ffffffff82239750, KVADDR, "page_offset_base", 8, (FOE|Q),
5642aae35c08>

$ curl -O https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstorage.googleapis.com%2Fcos-tools%2F17162.336.25%2Fvmlinux&data=05%7C01%7Cmatt.suiche%40magnetforensics.com%7C0bd964f454dc4c6f88f808dbeb14902f%7C631f5ed9a7994296b4149aee47fce3dd%7C1%7C0%7C638362245856890013%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vA2UZsSrOk5soDnqJPbWBcj4zxmgHlFMAc5Qzt9jW9A%3D&reserved=0
$ nm vmlinux | grep -e ' _stext' -e ' page_offset_base'
ffffffff81000000 T _stext
ffffffff82239750 R page_offset_base

To me, it looks like KASLR detection doesn't work.  The randomized
offset of the page_offset_base should be 0xffffffff82239750 + 0x8000000
= 0xffffffff8a239750, but crash is trying to read 0xffffffff82239750.

We need to look into why it doesn't work, firstly does this option work?
  If this works, I think it will be a clue.

# crash --kaslr=auto vmlinux /proc/kcore
   or
# crash --kaslr=<KERNELOFFSET value> vmlinux /proc/kcore

   i.e. --kaslr=8000000 during that system session.
   (this will vary after system reboot)

Thanks,
Kazu


On 2023/11/21 23:21, Matt Suiche wrote:
> Dear,
>
> I tried to use crash 8.0.4 on Google Container OS (17162.336.25) but for some reason there is resistance.
>
> Step to reproduce:
>
>    1.  Create a Virtual Machine in Google Cloud using Google Container OS as a base image
>    2.  Run ¡§toolkit¡¨
>    3.  Download the vmlinux symbols for the current base image
>       *   curl https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstorage.googleapis.com%2Fcos-tools%2F%24container_host_build_id%2Fvmlinux&data=05%7C01%7Cmatt.suiche%40magnetforensics.com%7C0bd964f454dc4c6f88f808dbeb14902f%7C631f5ed9a7994296b4149aee47fce3dd%7C1%7C0%7C638362245856890013%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=D0p5i7vh4r71U4jPcbJudkhKgTnTlLiBOkhdyBf4v8o%3D&reserved=0 > symbols/vmlinux-$container_host_build_id
>    4.  Run crash on /proc/kcore
>
> Thanks,
>
> Logs:
>
> root@instance-2:~# crash /proc/kcore vmlinux-17162.336.25 -d 99
>
>
>
> crash 8.0.4
>
> Copyright (C) 2002-2022  Red Hat, Inc.
>
> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
>
> Copyright (C) 1999-2006  Hewlett-Packard Co
>
> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
>
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
>
> Copyright (C) 2005, 2011, 2020-2022  NEC Corporation
>
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
>
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
>
> Copyright (C) 2015, 2021  VMware, Inc.
>
> This program is free software, covered by the GNU General Public License,
>
> and you are welcome to change it and/or distribute copies of it under
>
> certain conditions.  Enter "help copying" to see the conditions.
>
> This program has absolutely no warranty.  Enter "help warranty" for details.
>
>
>
> get_live_memory_source: /proc/kcore
>
> proc_kcore_data:
>
>             flags: 500 (KCORE_LOCAL|KCORE_ELF64)
>
>          segments: 12
>
>        elf_header: 5642ab6d3f40
>
>       header_size: 8636
>
>           notes64: 5642ab6d3f80
>
>            load64: 5642ab6d3fb8
>
>           notes32: 0
>
>            load32: 0
>
>        vmcoreinfo: 0
>
>   size_vmcoreinfo: 0
>
>
>
>    Elf64_Phdr:
>
>          p_type: 4 (PT_NOTE)
>
>         p_flags: 0
>
>        p_offset: 318
>
>         p_vaddr: 0
>
>         p_paddr: 0
>
>        p_filesz: 7844
>
>         p_memsz: 0
>
>         p_align: 0
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 7fff89003000
>
>         p_vaddr: ffffffff89000000
>
>         p_paddr: 13a000000
>
>        p_filesz: 35831808
>
>         p_memsz: 35831808
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 130900003000
>
>         p_vaddr: ffff930900000000
>
>         p_paddr: ffffffffffffffff
>
>        p_filesz: 35184372088831
>
>         p_memsz: 35184372088831
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 7fffc0003000
>
>         p_vaddr: ffffffffc0000000
>
>         p_paddr: ffffffffffffffff
>
>        p_filesz: 1056964608
>
>         p_memsz: 1056964608
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: ec140004000
>
>         p_vaddr: ffff8ec140001000
>
>         p_paddr: 1000
>
>        p_filesz: 344064
>
>         p_memsz: 344064
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 7be8c0003000
>
>         p_vaddr: fffffbe8c0000000
>
>         p_paddr: ffffffffffffffff
>
>        p_filesz: 8192
>
>         p_memsz: 8192
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: ec140063000
>
>         p_vaddr: ffff8ec140060000
>
>         p_paddr: 60000
>
>        p_filesz: 229376
>
>         p_memsz: 229376
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: ec140103000
>
>         p_vaddr: ffff8ec140100000
>
>         p_paddr: 100000
>
>        p_filesz: 3212759040
>
>         p_memsz: 3212759040
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 7be8c0007000
>
>         p_vaddr: fffffbe8c0004000
>
>         p_paddr: ffffffffffffffff
>
>        p_filesz: 50200576
>
>         p_memsz: 50200576
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: ec1ffc02000
>
>         p_vaddr: ffff8ec1ffbff000
>
>         p_paddr: bfbff000
>
>        p_filesz: 4067328
>
>         p_memsz: 4067328
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 7be8c2ff2000
>
>         p_vaddr: fffffbe8c2fef000
>
>         p_paddr: ffffffffffffffff
>
>        p_filesz: 69632
>
>         p_memsz: 69632
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: ec240003000
>
>         p_vaddr: ffff8ec240000000
>
>         p_paddr: 100000000
>
>        p_filesz: 1073741824
>
>         p_memsz: 1073741824
>
>         p_align: 4096
>
>
>
>    Elf64_Phdr:
>
>          p_type: 1 (PT_LOAD)
>
>         p_flags: 7
>
>        p_offset: 7be8c4003000
>
>         p_vaddr: fffffbe8c4000000
>
>         p_paddr: ffffffffffffffff
>
>        p_filesz: 16777216
>
>         p_memsz: 16777216
>
>         p_align: 4096
>
>
>
>    Elf64_Nhdr:
>
>        n_namesz: 5 ("CORE")
>
>        n_descsz: 336
>
>          n_type: 1 (NT_PRSTATUS)
>
>
>
>    Elf64_Nhdr:
>
>        n_namesz: 5 ("CORE")
>
>        n_descsz: 136
>
>          n_type: 3 (NT_PRPSINFO)
>
>
>
>    Elf64_Nhdr:
>
>        n_namesz: 5 ("CORE")
>
>        n_descsz: 4288
>
>          n_type: 4 (NT_TASKSTRUCT)
>
>
>
>    Elf64_Nhdr:
>
>        n_namesz: 11 ("VMCOREINFO")
>
>        n_descsz: 3000
>
>          n_type: 0 (unknown)
>
>
>
>        OSRELEASE=5.15.133+
>
>        BUILD-ID=f16c9f1b53617d7b151c4d18d79c6ccbb44ea6d6
>
>        PAGESIZE=4096
>
>        SYMBOL(init_uts_ns)=ffffffff8a615698
>
>        OFFSET(uts_namespace.name)=0
>
>        SYMBOL(node_online_map)=ffffffff8a85d638
>
>        SYMBOL(swapper_pg_dir)=ffffffff8a60c000
>
>        SYMBOL(_stext)=ffffffff89000000
>
>        SYMBOL(vmap_area_list)=ffffffff8a774208
>
>        SYMBOL(mem_section)=ffff8ec27fff8000
>
>        LENGTH(mem_section)=2048
>
>        SIZE(mem_section)=16
>
>        OFFSET(mem_section.section_mem_map)=0
>
>        NUMBER(SECTION_SIZE_BITS)=27
>
>        NUMBER(MAX_PHYSMEM_BITS)=46
>
>        SIZE(page)=64
>
>        SIZE(pglist_data)=15616
>
>        SIZE(zone)=1664
>
>        SIZE(free_area)=104
>
>        SIZE(list_head)=16
>
>        SIZE(nodemask_t)=8
>
>        OFFSET(page.flags)=0
>
>        OFFSET(page._refcount)=52
>
>        OFFSET(page.mapping)=24
>
>        OFFSET(page.lru)=8
>
>        OFFSET(page._mapcount)=48
>
>        OFFSET(page.private)=40
>
>        OFFSET(page.compound_dtor)=16
>
>        OFFSET(page.compound_order)=17
>
>        OFFSET(page.compound_head)=8
>
>        OFFSET(pglist_data.node_zones)=0
>
>        OFFSET(pglist_data.nr_zones)=14880
>
>        OFFSET(pglist_data.node_start_pfn)=14888
>
>        OFFSET(pglist_data.node_spanned_pages)=14904
>
>        OFFSET(pglist_data.node_id)=14912
>
>        OFFSET(zone.free_area)=192
>
>        OFFSET(zone.vm_stat)=1472
>
>        OFFSET(zone.spanned_pages)=128
>
>        OFFSET(free_area.free_list)=0
>
>        OFFSET(list_head.next)=0
>
>        OFFSET(list_head.prev)=8
>
>        OFFSET(vmap_area.va_start)=0
>
>        OFFSET(vmap_area.list)=40
>
>        LENGTH(zone.free_area)=11
>
>        SYMBOL(prb)=ffffffff8a662318
>
>        SYMBOL(printk_rb_static)=ffffffff8a662320
>
>        SYMBOL(clear_seq)=ffffffff8ad8c0d8
>
>        SIZE(printk_ringbuffer)=80
>
>        OFFSET(printk_ringbuffer.desc_ring)=0
>
>        OFFSET(printk_ringbuffer.text_data_ring)=40
>
>        OFFSET(printk_ringbuffer.fail)=72
>
>        SIZE(prb_desc_ring)=40
>
>        OFFSET(prb_desc_ring.count_bits)=0
>
>        OFFSET(prb_desc_ring.descs)=8
>
>        OFFSET(prb_desc_ring.infos)=16
>
>        OFFSET(prb_desc_ring.head_id)=24
>
>        OFFSET(prb_desc_ring.tail_id)=32
>
>        SIZE(prb_desc)=24
>
>        OFFSET(prb_desc.state_var)=0
>
>        OFFSET(prb_desc.text_blk_lpos)=8
>
>        SIZE(prb_data_blk_lpos)=16
>
>        OFFSET(prb_data_blk_lpos.begin)=0
>
>        OFFSET(prb_data_blk_lpos.next)=8
>
>        SIZE(printk_info)=88
>
>        OFFSET(printk_info.seq)=0
>
>        OFFSET(printk_info.ts_nsec)=8
>
>        OFFSET(printk_info.text_len)=16
>
>        OFFSET(printk_info.caller_id)=20
>
>        OFFSET(printk_info.dev_info)=24
>
>        SIZE(dev_printk_info)=64
>
>        OFFSET(dev_printk_info.subsystem)=0
>
>        LENGTH(printk_info_subsystem)=16
>
>        OFFSET(dev_printk_info.device)=16
>
>        LENGTH(printk_info_device)=48
>
>        SIZE(prb_data_ring)=32
>
>        OFFSET(prb_data_ring.size_bits)=0
>
>        OFFSET(prb_data_ring.data)=8
>
>        OFFSET(prb_data_ring.head_lpos)=16
>
>        OFFSET(prb_data_ring.tail_lpos)=24
>
>        SIZE(atomic_long_t)=8
>
>        OFFSET(atomic_long_t.counter)=0
>
>        SIZE(latched_seq)=24
>
>        OFFSET(latched_seq.val)=8
>
>        LENGTH(free_area.free_list)=6
>
>        NUMBER(NR_FREE_PAGES)=0
>
>        NUMBER(PG_lru)=4
>
>        NUMBER(PG_private)=13
>
>        NUMBER(PG_swapcache)=10
>
>        NUMBER(PG_swapbacked)=19
>
>        NUMBER(PG_slab)=9
>
>        NUMBER(PG_hwpoison)=23
>
>        NUMBER(PG_head_mask)=65536
>
>        NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
>
>        NUMBER(HUGETLB_PAGE_DTOR)=2
>
>        NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
>
>        NUMBER(phys_base)=5117050880
>
>        SYMBOL(init_top_pgt)=ffffffff8a60c000
>
>        NUMBER(pgtable_l5_enabled)=0
>
>        SYMBOL(node_data)=ffffffff8a85c5d0
>
>        LENGTH(node_data)=64
>
>        KERNELOFFSET=8000000
>
>        NUMBER(KERNEL_IMAGE_SIZE)=1073741824
>
>        NUMBER(sme_mask)=0
>
>
>
> /proc/version:
>
> Linux version 5.15.133+ (builder@localhost) (Chromium OS 14.0_pre445002_p20220217-r3 clang version 14.0.0 (/var/tmp/portage/sys-devel/llvm-14.0_pre445002_p20220217-r3/work/llvm-14.0_pre445002_p20220217/clang 18308e171b5b1dd99627a4d88c7d6c5ff21b8c96), LLD 14.0.0) #1 SMP Sat Nov 11 11:15:28 UTC 2023
>
> vmlinux-17162.336.25:
>
> Linux version 5.15.133+ (builder@localhost) (Chromium OS 14.0_pre445002_p20220217-r3 clang version 14.0.0 (/var/tmp/portage/sys-devel/llvm-14.0_pre445002_p20220217-r3/work/llvm-14.0_pre445002_p20220217/clang 18308e171b5b1dd99627a4d88c7d6c5ff21b8c96), LLD 14.0.0) #1 SMP Sat Nov 11 11:15:28 UTC 2023
>
> readmem: read_proc_kcore() -> /proc/kcore
>
> crash: pv_ops exists: ARCH_PVOPS
>
> VMCOREINFO: NUMBER(phys_base): 5117050880 -> 131000000
>
> gdb vmlinux-17162.336.25
>
> GNU gdb (GDB) 10.2
>
> Copyright (C) 2021 Free Software Foundation, Inc.
>
> License GPLv3+: GNU GPL version 3 or later https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgnu.org%2Flicenses%2Fgpl.html&data=05%7C01%7Cmatt.suiche%40magnetforensics.com%7C0bd964f454dc4c6f88f808dbeb14902f%7C631f5ed9a7994296b4149aee47fce3dd%7C1%7C0%7C638362245857046312%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=FbWjtW1ptyhQp92FQdaESU%2Fu%2Fy89TzJ0jeWrCIp2i6g%3D&reserved=0
>
> This is free software: you are free to change and redistribute it.
>
> There is NO WARRANTY, to the extent permitted by law.
>
> Type "show copying" and "show warranty" for details.
>
> This GDB was configured as "x86_64-pc-linux-gnu".
>
> Type "show configuration" for configuration details.
>
> Find the GDB manual and other documentation resources online at:
>
>      https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gnu.org%2Fsoftware%2Fgdb%2Fdocumentation%2F&data=05%7C01%7Cmatt.suiche%40magnetforensics.com%7C0bd964f454dc4c6f88f808dbeb14902f%7C631f5ed9a7994296b4149aee47fce3dd%7C1%7C0%7C638362245857046312%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NLGcPFC1hwRUWLUGGbgiygIaKOKVZCGeGZsGLFvkiXk%3D&reserved=0.
>
>
>
> For help, type "help".
>
> Type "apropos word" to search for commands related to "word"...
>
> GETBUF(344 -> 0)
>
>    GETBUF(1500 -> 1)
>
>
>
>    FREEBUF(1)
>
> FREEBUF(0)
>
> <readmem: ffffffff82239750, KVADDR, "page_offset_base", 8, (FOE|Q), 5642aae35c08>
>
> <read_proc_kcore: addr: ffffffff82239750 paddr: 133239750 cnt: 8>
>
> crash: seek error: kernel virtual address: ffffffff82239750  type: "page_offset_base"
>
>
>
> root@instance-2:~# env
>
> container_host_version_id=101
>
> PWD=/root
>
> LOGNAME=root
>
> container=systemd-nspawn
>
> HOME=/root
>
> TERM=xterm-256color
>
> USER=root
>
> NOTIFY_SOCKET=/run/host/notify
>
> SHLVL=1
>
> container_host_id=cos
>
> container_host_build_id=17162.336.25
>
> PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>
> container_uuid=d8282d15-c11a-416b-9371-94db01a7ca15
>
> _=/usr/bin/env
>
> OLDPWD=/
>
>
> This email including any attachments may contain confidential material for the sole use of the intended recipient. If you are not the intended recipient please immediately notify the sender by reply email, permanently delete this message and do not forward it or any part of it to anyone else.
>

This email including any attachments may contain confidential material for the sole use of the intended recipient. If you are not the intended recipient please immediately notify the sender by reply email, permanently delete this message and do not forward it or any part of it to anyone else.