Re: [Crash-utility] EXT: Re: crash start in CentOS 8

Hello all, Back on this topic: I compiled a kernel with debuginfo on the Virtual Machine with CentOS8. I made a crash on this same VM. I transferred the vmcore and vmlinux.debuginfo files on another CentOS8 baremetal machine. The command '/usr/bin/crash -d 15 vmlinux.debuginfo vmcore' produces following logs, but still failing to get the crash> prompt. Still KASLR ? Thanks in advance for any information. Best regards, Patrick Agrain -- Logs: crash 7.2.6-2.el8

Copyright (C) 2002-2019 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. compressed kdump: header->utsname.machine: x86_64 compressed kdump: memory bitmap offset: 2000 diskdump_data: filename: vmcore flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED) dfd: 3 ofp: 0 machine_type: 62 (EM_X86_64) header: 212b5d0 signature: "KDUMP " header_version: 6 utsname: sysname: Linux nodename: localhost.localdomain release: 4.18.0-147.el8.x86_64 version: #1 SMP Wed Dec 4 21:51:45 UTC 2019 machine: x86_64 domainname: (none) timestamp: tv_sec: 5ee72d23 tv_usec: 0 status: 2 (DUMP_DH_COMPRESSED_LZO) block_size: 4096 sub_hdr_size: 1 bitmap_blocks: 32 max_mapnr: 524288 total_ram_blocks: 0 device_blocks: 0 written_blocks: 0 current_cpu: 0 nr_cpus: 1 tasks[nr_cpus]: 0 sub_header: 0 (n/a) sub_header_kdump: 212c5e0 phys_base: 7c00000 dump_level: 31 (0x1f) (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE) split: 0 start_pfn: (unused) end_pfn: (unused) offset_vmcoreinfo: 4580 (0x11e4) size_vmcoreinfo: 2025 (0x7e9) OSRELEASE=4.18.0-147.el8.x86_64 PAGESIZE=4096 SYMBOL(init_uts_ns)=ffffffffb5a12540 SYMBOL(node_online_map)=ffffffffb5c06d60 SYMBOL(swapper_pg_dir)=ffffffffb5a0a000 SYMBOL(_stext)=ffffffffb4800000 SYMBOL(vmap_area_list)=ffffffffb5ad2a90 SYMBOL(mem_section)=ffff89df7ffd2000 LENGTH(mem_section)=2048 SIZE(mem_section)=16 OFFSET(mem_section.section_mem_map)=0 SIZE(page)=64 SIZE(pglist_data)=171968 SIZE(zone)=1472 SIZE(free_area)=88 SIZE(list_head)=16 SIZE(nodemask_t)=128 OFFSET(page.flags)=0 OFFSET(page._refcount)=52 OFFSET(page.mapping)=24 OFFSET(page.lru)=8 OFFSET(page._mapcount)=48 OFFSET(page.private)=40 OFFSET(page.compound_dtor)=16 OFFSET(page.compound_order)=17 OFFSET(page.compound_head)=8 OFFSET(pglist_data.node_zones)=0 OFFSET(pglist_data.nr_zones)=171232 OFFSET(pglist_data.node_start_pfn)=171240 OFFSET(pglist_data.node_spanned_pages)=171256 OFFSET(pglist_data.node_id)=171264 OFFSET(zone.free_area)=192 OFFSET(zone.vm_stat)=1296 OFFSET(zone.spanned_pages)=112 OFFSET(free_area.free_list)=0 OFFSET(list_head.next)=0 OFFSET(list_head.prev)=8 OFFSET(vmap_area.va_start)=0 OFFSET(vmap_area.list)=48 LENGTH(zone.free_area)=11 SYMBOL(log_buf)=ffffffffb5a5a280 SYMBOL(log_buf_len)=ffffffffb5a5a27c SYMBOL(log_first_idx)=ffffffffb633a770 SYMBOL(clear_idx)=ffffffffb633a744 SYMBOL(log_next_idx)=ffffffffb633a760 SIZE(printk_log)=16 OFFSET(printk_log.ts_nsec)=0 OFFSET(printk_log.len)=8 OFFSET(printk_log.text_len)=10 OFFSET(printk_log.dict_len)=12 LENGTH(free_area.free_list)=5 NUMBER(NR_FREE_PAGES)=0 NUMBER(PG_lru)=5 NUMBER(PG_private)=12 NUMBER(PG_swapcache)=9 NUMBER(PG_swapbacked)=18 NUMBER(PG_slab)=8 NUMBER(PG_hwpoison)=22 NUMBER(PG_head_mask)=32768 NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129 NUMBER(HUGETLB_PAGE_DTOR)=2 NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257 NUMBER(phys_base)=130023424 SYMBOL(init_top_pgt)=ffffffffb5a0a000 NUMBER(pgtable_l5_enabled)=0 SYMBOL(node_data)=ffffffffb5c02580 LENGTH(node_data)=1024 KERNELOFFSET=33800000 NUMBER(KERNEL_IMAGE_SIZE)=1073741824 NUMBER(sme_mask)=0 CRASHTIME=1592208675 offset_note: 4200 (0x1068) size_note: 2408 (0x968) notes_buf: 2129460 num_vmcoredd_notes: 0 num_prstatus_notes: 1 notes[0]: 2129460 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 2846 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: 0000000000000000 R14: 00005616bb1208f0 R13: ffffffffb5b36fc0 R12: 0000000000000000 RBP: 0000000000000004 RBX: 0000000000000063 R11: 6873617263206120 R10: 7265676769725420 R9: 54203a2071527379 R8: 000000000000056a RAX: ffffffffb4d12d10 RCX: 0000000000000006 RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0000000000000063 RIP: ffffffffb4d12d22 RFLAGS: 0000000000010246 RSP: ffffb03903147e80 FS_BASE: 00007f5d6e06f740 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 snapshot_task: 0 num_qemu_notes: 0 NOTE offsets: 1068 (NT_PRSTATUS) offset_eraseinfo: 0 (0x0) size_eraseinfo: 0 (0x0) start_pfn_64: (unused) end_pfn_64: (unused) max_mapnr_64: 524288 (0x80000) data_offset: 22000 block_size: 4096 block_shift: 12 bitmap: 212f190 bitmap_len: 131072 max_mapnr: 524288 (0x80000) dumpable_bitmap: 7f937ad93010 byte: 0 bit: 0 compressed_page: 212d5f0 curbufptr: 0 page_cache_hdr[0]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 217f1d0 pg_hit_count: 0 page_cache_hdr[1]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 21801d0 pg_hit_count: 0 page_cache_hdr[2]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 21811d0 pg_hit_count: 0 page_cache_hdr[3]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 21821d0 pg_hit_count: 0 page_cache_hdr[4]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 21831d0 pg_hit_count: 0 page_cache_hdr[5]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 21841d0 pg_hit_count: 0 page_cache_hdr[6]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 21851d0 pg_hit_count: 0 page_cache_hdr[7]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 21861d0 pg_hit_count: 0 page_cache_hdr[8]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 21871d0 pg_hit_count: 0 page_cache_hdr[9]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 21881d0 pg_hit_count: 0 page_cache_hdr[10]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 21891d0 pg_hit_count: 0 page_cache_hdr[11]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 218a1d0 pg_hit_count: 0 page_cache_hdr[12]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 218b1d0 pg_hit_count: 0 page_cache_hdr[13]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 218c1d0 pg_hit_count: 0 page_cache_hdr[14]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 218d1d0 pg_hit_count: 0 page_cache_hdr[15]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 218e1d0 pg_hit_count: 0 page_cache_buf: 217f1d0 evict_index: 0 evictions: 0 accesses: 0 cached_reads: 0 valid_pages: 212a3b0 readmem: read_diskdump() KASLR: _stext from vmlinux.debuginfo: ffffffff81000000 _stext from vmcoreinfo: ffffffffb4800000 relocate: 33800000 (824MB) crash: pv_init_ops exists: ARCH_PVOPS VMCOREINFO: NUMBER(phys_base): 130023424 -> 7c00000 gdb vmlinux.debuginfo GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... GETBUF(328 -> 0) GETBUF(1500 -> 1) WARNING: kernel relocated [824MB]: patching 109676 gdb minimal_symbol values FREEBUF(1) FREEBUF(0) <readmem: ffffffffb70df958, KVADDR, "page_offset_base", 8, (FOE|Q), d8b948> <read_diskdump: addr: ffffffffb70df958 paddr: 3ecdf958 cnt: 8> read_diskdump: paddr/pfn: 3ecdf958/3ecdf -> cache physical page: 3ecdf000 GETBUF(328 -> 0) FREEBUF(0) GETBUF(1024 -> 0) <readmem: ffffffffb7f0c5e0, KVADDR, "possible", 1024, (ROE), fcd120> <read_diskdump: addr: ffffffffb7f0c5e0 paddr: 3fb0c5e0 cnt: 1024> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c5e0/3fb0c crash: page excluded: kernel virtual address: ffffffffb7f0c5e0 type: "possible" WARNING: cannot read cpu_possible_map <readmem: ffffffffb7f0bde0, KVADDR, "present", 1024, (ROE), fcd120> <read_diskdump: addr: ffffffffb7f0bde0 paddr: 3fb0bde0 cnt: 544> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0bde0/3fb0b crash: page excluded: kernel virtual address: ffffffffb7f0bde0 type: "present" WARNING: cannot read cpu_present_map <readmem: ffffffffb7f0c1e0, KVADDR, "online", 1024, (ROE), fcd120> <read_diskdump: addr: ffffffffb7f0c1e0 paddr: 3fb0c1e0 cnt: 1024> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c1e0/3fb0c crash: page excluded: kernel virtual address: ffffffffb7f0c1e0 type: "online" WARNING: cannot read cpu_online_map <readmem: ffffffffb7f0b9e0, KVADDR, "active", 1024, (ROE), fcd120> <read_diskdump: addr: ffffffffb7f0b9e0 paddr: 3fb0b9e0 cnt: 1024> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0b9e0/3fb0b crash: page excluded: kernel virtual address: ffffffffb7f0b9e0 type: "active" WARNING: cannot read cpu_active_map FREEBUF(0) <readmem: ffffffffb731a5b0, KVADDR, "pv_init_ops", 8, (ROE), 7ffd3655d640> <read_diskdump: addr: ffffffffb731a5b0 paddr: 3ef1a5b0 cnt: 8> read_diskdump: paddr/pfn: 3ef1a5b0/3ef1a -> cache physical page: 3ef1a000 GETBUF(328 -> 0) FREEBUF(0) GETBUF(328 -> 0) FREEBUF(0) <readmem: ffffffffb9bad1d0, KVADDR, "shadow_timekeeper xtime_sec", 8, (ROE), 7ffd3655d5f0> <read_diskdump: addr: ffffffffb9bad1d0 paddr: 417ad1d0 cnt: 8> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 417ad1d0/417ad crash: page excluded: kernel virtual address: ffffffffb9bad1d0 type: "shadow_timekeeper xtime_sec" xtime timespec.tv_sec: 900f27: Mon Apr 20 07:31:03 1970 <readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE), d71b7c> <read_diskdump: addr: ffffffffb7221d64 paddr: 3ee21d64 cnt: 390> read_diskdump: paddr/pfn: 3ee21d64/3ee21 -> cache physical page: 3ee21000 utsname: sysname: nodename: release: version: machine: domainname: base kernel version: 0.0.0 <readmem: ffffffffb6a016c0, KVADDR, "accessible check", 8, (ROE|Q), 7ffd3655af30> <read_diskdump: addr: ffffffffb6a016c0 paddr: 3e6016c0 cnt: 8> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3e6016c0/3e601 crash: page excluded: kernel virtual address: ffffffffb6a016c0 type: "accessible check" crash: vmlinux.debuginfo and vmcore do not match! Usage: crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form) crash [OPTION]... [NAMELIST] (live system form) Enter "crash -h" for details. -----Message d'origine----- De : crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] De la part de HAGIO KAZUHITO(?? ??) Envoyé : mercredi 17 juin 2020 17:17 À : Discussion list for crash utility usage, maintenance and development <crash-utility(a)redhat.com&gt; Objet : EXT: Re: [Crash-utility] crash start in CentOS 8 ** External email - Please consider with caution ** Hi Bhupesh, -----Original Message----- > > Just FYI, with respect to the System.map file, the following Dave's > > post would be helpful: > > https://www.redhat.com/archives/crash-utility/2018-June/msg00002.htm > > l > > > > If you have the vmlinux corresponding to a running kernel, you don't > > need the System.map file. > > Indeed, I remember discussing this with Dave A. some months back. > Infact with newer KASLR enabled kernels it makes little sense to use > System.map file as it contains the non-relocated symbol values that > were compiled into the vmlinux file - that's why I suggested just to > use to the vmlinux and vmcore file as an arguments while invoking the > crash tool to Patrick. > > However, going back to the discussion I had with Dave A., I think the > main point of confusion was the 'crash_whitepaper' which is still > located at [1], and which provides the following example to invoke the > crash utility with the System.map file, vmlinux and vmcore > combination: > > Examples when running on a dumpfile: > > $ crash /boot/System.map vmlinux.dbg vmcore > > I think while this was ok for older kernels which did not support > KASLR, for newer kernels this needs an update. > I am trying to see if I can modify the 'crash_whitepaper' and push the > same to github, so that users are not confused with the System.map > option. > > What's your views on the same? Please let me know. > > [1]. https://crash-utility.github.io/crash_whitepaper.html There are several command lines with System.map in "4. Invocation" chapter and I think that the ones in "RHEL2.1 Kernels (or kernels built without -g flag)" section are no problem because the reason why it's needed is explained well. The others in the "crash -h" output in "Invocation output" section might be a little confusing as you say if they see only the -S option's explanation.. I think it would be fine to omit the part of the output below the [mapfile] argument and itself because the output looks to be intended almost only to introduce the "crash -h" option. Then users see the current crash -h output which doesn't have command lines with System.map. So, for instance: diff --git a/crash_whitepaper.html b/crash_whitepaper.html index 9b2e88c8acb4..7cb2608e6062 100644 --- a/crash_whitepaper.html +++ b/crash_whitepaper.html @@ -1166,7 +1166,8 @@ Usage: file. If the [dumpfile] argument is not entered, the session will be invoked on the live system using /dev/mem, which usually requires root privileges. - +... +<!-- Omit this part so that users don't use System.map needlessly. [mapfile] If the live system kernel, or the kernel from which the [dumpfile] was derived, was not compiled with the -g switch, then the additional @@ -1215,7 +1216,7 @@ Usage: [-d num] Set crash debug level [num]. The higher the number, the more debug data will be printed during crash runtime. - +--> </pre> </td> </tr> How about this? Thanks, Kazu -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility