[Crash-utility] Re: Google Container OS and crash 8.0.4

Hi Matt, Thank you for trying the latest. 5642aae35c08> $ curl -O https://storage.googleapis.com/cos-tools/17162.336.25/vmlinux $ nm vmlinux | grep -e ' _stext' -e ' page_offset_base' ffffffff81000000 T _stext ffffffff82239750 R page_offset_base To me, it looks like KASLR detection doesn't work. The randomized offset of the page_offset_base should be 0xffffffff82239750 + 0x8000000 = 0xffffffff8a239750, but crash is trying to read 0xffffffff82239750. We need to look into why it doesn't work, firstly does this option work? If this works, I think it will be a clue. # crash --kaslr=auto vmlinux /proc/kcore or # crash --kaslr=<KERNELOFFSET value> vmlinux /proc/kcore i.e. --kaslr=8000000 during that system session. (this will vary after system reboot) Thanks, Kazu On 2023/11/21 23:21, Matt Suiche wrote: > Dear, > > I tried to use crash 8.0.4 on Google Container OS (17162.336.25) but for some reason there is resistance. > > Step to reproduce: > > 1. Create a Virtual Machine in Google Cloud using Google Container OS as a base image > 2. Run “toolkit” > 3. Download the vmlinux symbols for the current base image > * curl https://storage.googleapis.com/cos-tools/$container_host_build_id/vmlinux > symbols/vmlinux-$container_host_build_id > 4. Run crash on /proc/kcore > > Thanks, > > Logs: > > root@instance-2:~# crash /proc/kcore vmlinux-17162.336.25 -d 99 > > > > crash 8.0.4 > > Copyright (C) 2002-2022 Red Hat, Inc. > > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > > Copyright (C) 1999-2006 Hewlett-Packard Co > > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > > Copyright (C) 2005, 2011, 2020-2022 NEC Corporation > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > Copyright (C) 2015, 2021 VMware, Inc. > > This program is free software, covered by the GNU General Public License, > > and you are welcome to change it and/or distribute copies of it under > > certain conditions. Enter "help copying" to see the conditions. > > This program has absolutely no warranty. Enter "help warranty" for details. > > > > get_live_memory_source: /proc/kcore > > proc_kcore_data: > > flags: 500 (KCORE_LOCAL|KCORE_ELF64) > > segments: 12 > > elf_header: 5642ab6d3f40 > > header_size: 8636 > > notes64: 5642ab6d3f80 > > load64: 5642ab6d3fb8 > > notes32: 0 > > load32: 0 > > vmcoreinfo: 0 > > size_vmcoreinfo: 0 > > > > Elf64_Phdr: > > p_type: 4 (PT_NOTE) > > p_flags: 0 > > p_offset: 318 > > p_vaddr: 0 > > p_paddr: 0 > > p_filesz: 7844 > > p_memsz: 0 > > p_align: 0 > > > > Elf64_Phdr: > > p_type: 1 (PT_LOAD) > > p_flags: 7 > > p_offset: 7fff89003000 > > p_vaddr: ffffffff89000000 > > p_paddr: 13a000000 > > p_filesz: 35831808 > > p_memsz: 35831808 > > p_align: 4096 > > > > Elf64_Phdr: > > p_type: 1 (PT_LOAD) > > p_flags: 7 > > p_offset: 130900003000 > > p_vaddr: ffff930900000000 > > p_paddr: ffffffffffffffff > > p_filesz: 35184372088831 > > p_memsz: 35184372088831 > > p_align: 4096 > > > > Elf64_Phdr: > > p_type: 1 (PT_LOAD) > > p_flags: 7 > > p_offset: 7fffc0003000 > > p_vaddr: ffffffffc0000000 > > p_paddr: ffffffffffffffff > > p_filesz: 1056964608 > > p_memsz: 1056964608 > > p_align: 4096 > > > > Elf64_Phdr: > > p_type: 1 (PT_LOAD) > > p_flags: 7 > > p_offset: ec140004000 > > p_vaddr: ffff8ec140001000 > > p_paddr: 1000 > > p_filesz: 344064 > > p_memsz: 344064 > > p_align: 4096 > > > > Elf64_Phdr: > > p_type: 1 (PT_LOAD) > > p_flags: 7 > > p_offset: 7be8c0003000 > > p_vaddr: fffffbe8c0000000 > > p_paddr: ffffffffffffffff > > p_filesz: 8192 > > p_memsz: 8192 > > p_align: 4096 > > > > Elf64_Phdr: > > p_type: 1 (PT_LOAD) > > p_flags: 7 > > p_offset: ec140063000 > > p_vaddr: ffff8ec140060000 > > p_paddr: 60000 > > p_filesz: 229376 > > p_memsz: 229376 > > p_align: 4096 > > > > Elf64_Phdr: > > p_type: 1 (PT_LOAD) > > p_flags: 7 > > p_offset: ec140103000 > > p_vaddr: ffff8ec140100000 > > p_paddr: 100000 > > p_filesz: 3212759040 > > p_memsz: 3212759040 > > p_align: 4096 > > > > Elf64_Phdr: > > p_type: 1 (PT_LOAD) > > p_flags: 7 > > p_offset: 7be8c0007000 > > p_vaddr: fffffbe8c0004000 > > p_paddr: ffffffffffffffff > > p_filesz: 50200576 > > p_memsz: 50200576 > > p_align: 4096 > > > > Elf64_Phdr: > > p_type: 1 (PT_LOAD) > > p_flags: 7 > > p_offset: ec1ffc02000 > > p_vaddr: ffff8ec1ffbff000 > > p_paddr: bfbff000 > > p_filesz: 4067328 > > p_memsz: 4067328 > > p_align: 4096 > > > > Elf64_Phdr: > > p_type: 1 (PT_LOAD) > > p_flags: 7 > > p_offset: 7be8c2ff2000 > > p_vaddr: fffffbe8c2fef000 > > p_paddr: ffffffffffffffff > > p_filesz: 69632 > > p_memsz: 69632 > > p_align: 4096 > > > > Elf64_Phdr: > > p_type: 1 (PT_LOAD) > > p_flags: 7 > > p_offset: ec240003000 > > p_vaddr: ffff8ec240000000 > > p_paddr: 100000000 > > p_filesz: 1073741824 > > p_memsz: 1073741824 > > p_align: 4096 > > > > Elf64_Phdr: > > p_type: 1 (PT_LOAD) > > p_flags: 7 > > p_offset: 7be8c4003000 > > p_vaddr: fffffbe8c4000000 > > p_paddr: ffffffffffffffff > > p_filesz: 16777216 > > p_memsz: 16777216 > > p_align: 4096 > > > > Elf64_Nhdr: > > n_namesz: 5 ("CORE") > > n_descsz: 336 > > n_type: 1 (NT_PRSTATUS) > > > > Elf64_Nhdr: > > n_namesz: 5 ("CORE") > > n_descsz: 136 > > n_type: 3 (NT_PRPSINFO) > > > > Elf64_Nhdr: > > n_namesz: 5 ("CORE") > > n_descsz: 4288 > > n_type: 4 (NT_TASKSTRUCT) > > > > Elf64_Nhdr: > > n_namesz: 11 ("VMCOREINFO") > > n_descsz: 3000 > > n_type: 0 (unknown) > > > > OSRELEASE=5.15.133+ > > BUILD-ID=f16c9f1b53617d7b151c4d18d79c6ccbb44ea6d6 > > PAGESIZE=4096 > > SYMBOL(init_uts_ns)=ffffffff8a615698 > > OFFSET(uts_namespace.name)=0 > > SYMBOL(node_online_map)=ffffffff8a85d638 > > SYMBOL(swapper_pg_dir)=ffffffff8a60c000 > > > SYMBOL(vmap_area_list)=ffffffff8a774208 > > SYMBOL(mem_section)=ffff8ec27fff8000 > > LENGTH(mem_section)=2048 > > SIZE(mem_section)=16 > > OFFSET(mem_section.section_mem_map)=0 > > NUMBER(SECTION_SIZE_BITS)=27 > > NUMBER(MAX_PHYSMEM_BITS)=46 > > SIZE(page)=64 > > SIZE(pglist_data)=15616 > > SIZE(zone)=1664 > > SIZE(free_area)=104 > > SIZE(list_head)=16 > > SIZE(nodemask_t)=8 > > OFFSET(page.flags)=0 > > OFFSET(page._refcount)=52 > > OFFSET(page.mapping)=24 > > OFFSET(page.lru)=8 > > OFFSET(page._mapcount)=48 > > OFFSET(page.private)=40 > > OFFSET(page.compound_dtor)=16 > > OFFSET(page.compound_order)=17 > > OFFSET(page.compound_head)=8 > > OFFSET(pglist_data.node_zones)=0 > > OFFSET(pglist_data.nr_zones)=14880 > > OFFSET(pglist_data.node_start_pfn)=14888 > > OFFSET(pglist_data.node_spanned_pages)=14904 > > OFFSET(pglist_data.node_id)=14912 > > OFFSET(zone.free_area)=192 > > OFFSET(zone.vm_stat)=1472 > > OFFSET(zone.spanned_pages)=128 > > OFFSET(free_area.free_list)=0 > > OFFSET(list_head.next)=0 > > OFFSET(list_head.prev)=8 > > OFFSET(vmap_area.va_start)=0 > > OFFSET(vmap_area.list)=40 > > LENGTH(zone.free_area)=11 > > SYMBOL(prb)=ffffffff8a662318 > > SYMBOL(printk_rb_static)=ffffffff8a662320 > > SYMBOL(clear_seq)=ffffffff8ad8c0d8 > > SIZE(printk_ringbuffer)=80 > > OFFSET(printk_ringbuffer.desc_ring)=0 > > OFFSET(printk_ringbuffer.text_data_ring)=40 > > OFFSET(printk_ringbuffer.fail)=72 > > SIZE(prb_desc_ring)=40 > > OFFSET(prb_desc_ring.count_bits)=0 > > OFFSET(prb_desc_ring.descs)=8 > > OFFSET(prb_desc_ring.infos)=16 > > OFFSET(prb_desc_ring.head_id)=24 > > OFFSET(prb_desc_ring.tail_id)=32 > > SIZE(prb_desc)=24 > > OFFSET(prb_desc.state_var)=0 > > OFFSET(prb_desc.text_blk_lpos)=8 > > SIZE(prb_data_blk_lpos)=16 > > OFFSET(prb_data_blk_lpos.begin)=0 > > OFFSET(prb_data_blk_lpos.next)=8 > > SIZE(printk_info)=88 > > OFFSET(printk_info.seq)=0 > > OFFSET(printk_info.ts_nsec)=8 > > OFFSET(printk_info.text_len)=16 > > OFFSET(printk_info.caller_id)=20 > > OFFSET(printk_info.dev_info)=24 > > SIZE(dev_printk_info)=64 > > OFFSET(dev_printk_info.subsystem)=0 > > LENGTH(printk_info_subsystem)=16 > > OFFSET(dev_printk_info.device)=16 > > LENGTH(printk_info_device)=48 > > SIZE(prb_data_ring)=32 > > OFFSET(prb_data_ring.size_bits)=0 > > OFFSET(prb_data_ring.data)=8 > > OFFSET(prb_data_ring.head_lpos)=16 > > OFFSET(prb_data_ring.tail_lpos)=24 > > SIZE(atomic_long_t)=8 > > OFFSET(atomic_long_t.counter)=0 > > SIZE(latched_seq)=24 > > OFFSET(latched_seq.val)=8 > > LENGTH(free_area.free_list)=6 > > NUMBER(NR_FREE_PAGES)=0 > > NUMBER(PG_lru)=4 > > NUMBER(PG_private)=13 > > NUMBER(PG_swapcache)=10 > > NUMBER(PG_swapbacked)=19 > > NUMBER(PG_slab)=9 > > NUMBER(PG_hwpoison)=23 > > NUMBER(PG_head_mask)=65536 > > NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129 > > NUMBER(HUGETLB_PAGE_DTOR)=2 > > NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257 > > NUMBER(phys_base)=5117050880 > > SYMBOL(init_top_pgt)=ffffffff8a60c000 > > NUMBER(pgtable_l5_enabled)=0 > > SYMBOL(node_data)=ffffffff8a85c5d0 > > LENGTH(node_data)=64 > > > NUMBER(KERNEL_IMAGE_SIZE)=1073741824 > > NUMBER(sme_mask)=0 > > > > /proc/version: > > Linux version 5.15.133+ (builder@localhost) (Chromium OS 14.0_pre445002_p20220217-r3 clang version 14.0.0 (/var/tmp/portage/sys-devel/llvm-14.0_pre445002_p20220217-r3/work/llvm-14.0_pre445002_p20220217/clang 18308e171b5b1dd99627a4d88c7d6c5ff21b8c96), LLD 14.0.0) #1 SMP Sat Nov 11 11:15:28 UTC 2023 > > vmlinux-17162.336.25: > > Linux version 5.15.133+ (builder@localhost) (Chromium OS 14.0_pre445002_p20220217-r3 clang version 14.0.0 (/var/tmp/portage/sys-devel/llvm-14.0_pre445002_p20220217-r3/work/llvm-14.0_pre445002_p20220217/clang 18308e171b5b1dd99627a4d88c7d6c5ff21b8c96), LLD 14.0.0) #1 SMP Sat Nov 11 11:15:28 UTC 2023 > > readmem: read_proc_kcore() -> /proc/kcore > > crash: pv_ops exists: ARCH_PVOPS > > VMCOREINFO: NUMBER(phys_base): 5117050880 -> 131000000 > > gdb vmlinux-17162.336.25 > > GNU gdb (GDB) 10.2 > > Copyright (C) 2021 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html > > This is free software: you are free to change and redistribute it. > > There is NO WARRANTY, to the extent permitted by law. > > Type "show copying" and "show warranty" for details. > > This GDB was configured as "x86_64-pc-linux-gnu". > > Type "show configuration" for configuration details. > > Find the GDB manual and other documentation resources online at: > > http://www.gnu.org/software/gdb/documentation/. > > > > For help, type "help". > > Type "apropos word" to search for commands related to "word"... > > GETBUF(344 -> 0) > > GETBUF(1500 -> 1) > > > > FREEBUF(1) > > FREEBUF(0) > > <readmem: ffffffff82239750, KVADDR, "page_offset_base", 8, (FOE|Q), 5642aae35c08> > > <read_proc_kcore: addr: ffffffff82239750 paddr: 133239750 cnt: 8> > > crash: seek error: kernel virtual address: ffffffff82239750 type: "page_offset_base" > > > > root@instance-2:~# env > > container_host_version_id=101 > > PWD=/root > > LOGNAME=root > > container=systemd-nspawn > > HOME=/root > > TERM=xterm-256color > > USER=root > > NOTIFY_SOCKET=/run/host/notify > > SHLVL=1 > > container_host_id=cos > > container_host_build_id=17162.336.25 > > PATH=/root/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin > > container_uuid=d8282d15-c11a-416b-9371-94db01a7ca15 > > _=/usr/bin/env > > OLDPWD=/ > > > This email including any attachments may contain confidential material for the sole use of the intended recipient. If you are not the intended recipient please immediately notify the sender by reply email, permanently delete this message and do not forward it or any part of it to anyone else. >