Hello, I tried to test the crash utility from CentOS 8 with following requirements: First machine (VMware) on which CentOS 8 is installed and on which I performed a fake crash using SysRq. After reboot, The /var/crash was populated with the vmcore. Second machine (Server) on which CentOS 8 is also installed and on which I wish to analyze the crash. Both machine have the same CentOS build installed. I imported System.map-4.18.0-147.el8.x86_64, vmcore and vmlinux.debuginfo from first machine to a directory on the second machine. When I run ‘crash’, I have following messages (see below), but no ‘crash>’ prompt. What have I missed ? What Should I check ? Thanks. Best regards, Patrick Agrain [root@lxkmg-pag-ale test64]# /usr/bin/crash System.map-4.18.0-147.el8.x86_64 vmlinux.debuginfo vmcore crash 7.2.6-2.el8 Copyright (C) 2002-2019 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... please wait... (patching 92149 gdb minimal_symbol values) [Vanished after a few seconds] [root@lxkmg-pag-ale test64]#

From CRASH(8) man page:

-----Message d'origine----- De : crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] De la part de Bhupesh Sharma Envoyé : lundi 15 juin 2020 09:13 À : Discussion list for crash utility usage, maintenance and development <crash-utility(a)redhat.com&gt; Objet : EXT: Re: [Crash-utility] crash start in CentOS 8 ** External email - Please consider with caution ** Hello Patrick, On Mon, Jun 15, 2020 at 12:28 PM Agrain Patrick <patrick.agrain(a)al-enterprise.com&gt; wrote: > > Hello, > > > > I tried to test the crash utility from CentOS 8 with following requirements: > > First machine (VMware) on which CentOS 8 is installed and on which I performed a fake crash using SysRq. After reboot, The /var/crash was populated with the vmcore. > Second machine (Server) on which CentOS 8 is also installed and on which I wish to analyze the crash. > > > > Both machine have the same CentOS build installed. > > I imported System.map-4.18.0-147.el8.x86_64, vmcore and vmlinux.debuginfo from first machine to a directory on the second machine. > > When I run ‘crash’, I have following messages (see below), but no ‘crash>’ prompt. > > > > What have I missed ? What Should I check ? > > > > Thanks. > > Best regards, > > Patrick Agrain > > > > [root@lxkmg-pag-ale test64]# /usr/bin/crash > System.map-4.18.0-147.el8.x86_64 vmlinux.debuginfo vmcore > > > > crash 7.2.6-2.el8 > > Copyright (C) 2002-2019 Red Hat, Inc. > > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > > Copyright (C) 1999-2006 Hewlett-Packard Co > > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > > Copyright (C) 2005, 2011 NEC Corporation > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > This program is free software, covered by the GNU General Public > License, > > and you are welcome to change it and/or distribute copies of it > under > > certain conditions. Enter "help copying" to see the conditions. > > This program has absolutely no warranty. Enter "help warranty" for details. > > > > GNU gdb (GDB) 7.6 > > Copyright (C) 2013 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later > <http://gnu.org/licenses/gpl.html> > > This is free software: you are free to change and redistribute it. > > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > > and "show warranty" for details. > > This GDB was configured as "x86_64-unknown-linux-gnu"... > > > > please wait... (patching 92149 gdb minimal_symbol values) [Vanished > after a few seconds] > > > > [root@lxkmg-pag-ale test64]# Can you please increase the verbosity level by specifying the -d parameter while invoking the crash command? >From CRASH(8) man page: -d num: Set the internal debug level. The higher the number, the more debugging data will be printed when crash initializes and runs. I normally would run crash with -d 15 to get verbose logs (but not too verbose to overwhelm the console :)). Can you please share the output with the same? Thanks, Bhupesh Thanks Bhupesh for taking some time. Please find the logs with -d 15 verbosity. Thanks. Best regards, Patrick Agrain [root@lxkmg-pag-ale test64]# /usr/bin/crash -d 15 System.map-4.18.0-147.el8.x86_64 vmlinux.debuginfo vmcore crash 7.2.6-2.el8 Copyright (C) 2002-2019 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. compressed kdump: header->utsname.machine: x86_64 compressed kdump: memory bitmap offset: 2000 diskdump_data: filename: vmcore flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED) dfd: 3 ofp: 0 machine_type: 62 (EM_X86_64) header: 2d006a0 signature: "KDUMP " header_version: 6 utsname: sysname: Linux nodename: localhost.localdomain release: 4.18.0-147.el8.x86_64 version: #1 SMP Wed Dec 4 21:51:45 UTC 2019 machine: x86_64 domainname: (none) timestamp: tv_sec: 5ea68afe tv_usec: 0 status: 2 (DUMP_DH_COMPRESSED_LZO) block_size: 4096 sub_hdr_size: 1 bitmap_blocks: 32 max_mapnr: 524288 total_ram_blocks: 0 device_blocks: 0 written_blocks: 0 current_cpu: 0 nr_cpus: 1 tasks[nr_cpus]: 0 sub_header: 0 (n/a) sub_header_kdump: 2d016b0 phys_base: 35a00000 dump_level: 31 (0x1f) (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE) split: 0 start_pfn: (unused) end_pfn: (unused) offset_vmcoreinfo: 4580 (0x11e4) size_vmcoreinfo: 2024 (0x7e8) OSRELEASE=4.18.0-147.el8.x86_64 PAGESIZE=4096 SYMBOL(init_uts_ns)=ffffffff86812540 SYMBOL(node_online_map)=ffffffff86a06d60 SYMBOL(swapper_pg_dir)=ffffffff8680a000 SYMBOL(_stext)=ffffffff85600000 SYMBOL(vmap_area_list)=ffffffff868d2a90 SYMBOL(mem_section)=ffff95a67ffd2000 LENGTH(mem_section)=2048 SIZE(mem_section)=16 OFFSET(mem_section.section_mem_map)=0 SIZE(page)=64 SIZE(pglist_data)=171968 SIZE(zone)=1472 SIZE(free_area)=88 SIZE(list_head)=16 SIZE(nodemask_t)=128 OFFSET(page.flags)=0 OFFSET(page._refcount)=52 OFFSET(page.mapping)=24 OFFSET(page.lru)=8 OFFSET(page._mapcount)=48 OFFSET(page.private)=40 OFFSET(page.compound_dtor)=16 OFFSET(page.compound_order)=17 OFFSET(page.compound_head)=8 OFFSET(pglist_data.node_zones)=0 OFFSET(pglist_data.nr_zones)=171232 OFFSET(pglist_data.node_start_pfn)=171240 OFFSET(pglist_data.node_spanned_pages)=171256 OFFSET(pglist_data.node_id)=171264 OFFSET(zone.free_area)=192 OFFSET(zone.vm_stat)=1296 OFFSET(zone.spanned_pages)=112 OFFSET(free_area.free_list)=0 OFFSET(list_head.next)=0 OFFSET(list_head.prev)=8 OFFSET(vmap_area.va_start)=0 OFFSET(vmap_area.list)=48 LENGTH(zone.free_area)=11 SYMBOL(log_buf)=ffffffff8685a280 SYMBOL(log_buf_len)=ffffffff8685a27c SYMBOL(log_first_idx)=ffffffff8713a770 SYMBOL(clear_idx)=ffffffff8713a744 SYMBOL(log_next_idx)=ffffffff8713a760 SIZE(printk_log)=16 OFFSET(printk_log.ts_nsec)=0 OFFSET(printk_log.len)=8 OFFSET(printk_log.text_len)=10 OFFSET(printk_log.dict_len)=12 LENGTH(free_area.free_list)=5 NUMBER(NR_FREE_PAGES)=0 NUMBER(PG_lru)=5 NUMBER(PG_private)=12 NUMBER(PG_swapcache)=9 NUMBER(PG_swapbacked)=18 NUMBER(PG_slab)=8 NUMBER(PG_hwpoison)=22 NUMBER(PG_head_mask)=32768 NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129 NUMBER(HUGETLB_PAGE_DTOR)=2 NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257 NUMBER(phys_base)=899678208 SYMBOL(init_top_pgt)=ffffffff8680a000 NUMBER(pgtable_l5_enabled)=0 SYMBOL(node_data)=ffffffff86a02580 LENGTH(node_data)=1024 KERNELOFFSET=4600000 NUMBER(KERNEL_IMAGE_SIZE)=1073741824 NUMBER(sme_mask)=0 CRASHTIME=1587972862 offset_note: 4200 (0x1068) size_note: 2404 (0x964) notes_buf: 2cff690 num_vmcoredd_notes: 0 num_prstatus_notes: 1 notes[0]: 2cff690 (NT_PRSTATUS) si.signo: 0 si.code: 0 si.errno: 0 cursig: 0 sigpend: 0 sighold: 0 pid: 7020 ppid: 0 pgrp: 0 sid:0 utime: 0.000000 stime: 0.000000 cutime: 0.000000 cstime: 0.000000 ORIG_RAX: ffffffffffffffff fpvalid: 0 R15: 0000000000000000 R14: 000056500b98d6a0 R13: ffffffff86936fc0 R12: 0000000000000000 RBP: 0000000000000004 RBX: 0000000000000063 R11: 6873617263206120 R10: 7265676769725420 R9: 54203a2071527379 R8: 0000000000000556 RAX: ffffffff85b12d10 RCX: 0000000000000006 RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0000000000000063 RIP: ffffffff85b12d22 RFLAGS: 0000000000010246 RSP: ffffbd9943093e80 FS_BASE: 00007f2d9a1f6740 GS_BASE: 0000000000000000 CS: 0010 SS: 0018 DS: 0000 ES: 0000 FS: 0000 GS: 0000 snapshot_task: 0 num_qemu_notes: 0 NOTE offsets: 1068 (NT_PRSTATUS) offset_eraseinfo: 0 (0x0) size_eraseinfo: 0 (0x0) start_pfn_64: (unused) end_pfn_64: (unused) max_mapnr_64: 524288 (0x80000) data_offset: 22000 block_size: 4096 block_shift: 12 bitmap: 2d05250 bitmap_len: 131072 max_mapnr: 524288 (0x80000) dumpable_bitmap: 7f454b94e010 byte: 0 bit: 0 compressed_page: 2d026c0 curbufptr: 0 page_cache_hdr[0]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d55290 pg_hit_count: 0 page_cache_hdr[1]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d56290 pg_hit_count: 0 page_cache_hdr[2]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d57290 pg_hit_count: 0 page_cache_hdr[3]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d58290 pg_hit_count: 0 page_cache_hdr[4]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d59290 pg_hit_count: 0 page_cache_hdr[5]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d5a290 pg_hit_count: 0 page_cache_hdr[6]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d5b290 pg_hit_count: 0 page_cache_hdr[7]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d5c290 pg_hit_count: 0 page_cache_hdr[8]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d5d290 pg_hit_count: 0 page_cache_hdr[9]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d5e290 pg_hit_count: 0 page_cache_hdr[10]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d5f290 pg_hit_count: 0 page_cache_hdr[11]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d60290 pg_hit_count: 0 page_cache_hdr[12]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d61290 pg_hit_count: 0 page_cache_hdr[13]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d62290 pg_hit_count: 0 page_cache_hdr[14]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d63290 pg_hit_count: 0 page_cache_hdr[15]: pg_flags: 0 () pg_addr: 0 pg_bufptr: 2d64290 pg_hit_count: 0 page_cache_buf: 2d55290 evict_index: 0 evictions: 0 accesses: 0 cached_reads: 0 valid_pages: 2d00210 readmem: read_diskdump() crash: pv_init_ops exists: ARCH_PVOPS VMCOREINFO: NUMBER(phys_base): 899678208 -> 35a00000 gdb vmlinux.debuginfo GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... GETBUF(328 -> 0) GETBUF(1500 -> 1) FREEBUF(1) FREEBUF(0) <readmem: ffffffff82131200, KVADDR, "page_offset_base", 8, (FOE|Q), d8b948> <read_diskdump: addr: ffffffff82131200 paddr: 37b31200 cnt: 8> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 37b31200/37b31 crash: page excluded: kernel virtual address: ffffffff82131200 type: "page_offset_base" [root@lxkmg-pag-ale test64]#

-----Original Message----- > > [root@lxkmg-pag-ale test64]# /usr/bin/crash -d 15 > > System.map-4.18.0-147.el8.x86_64 vmlinux.debuginfo vmcore ... > > <read_diskdump: addr: ffffffff82131200 paddr: 37b31200 cnt: 8> > > read_diskdump: PAGE_EXCLUDED: paddr/pfn: 37b31200/37b31 > > crash: page excluded: kernel virtual address: ffffffff82131200 type: "page_offset_base" > > [root@lxkmg-pag-ale test64]# > > Hmmm, I think you should find better results with: > > $ crash vmlinux vmcore (On the machine used for post-processing) > > and > > $ crash vmlinux /proc/kcore (On the original crashing system) > > I think if that helps you reach the crash > prompt, we have a potential issue which needs to be looked-at. > Can you please try to run the same and let me know the outcome? > > Thanks, > Bhupesh > > > -- > Crash-utility mailing list > Crash-utility(a)redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility > > Hello, > > Running the above mentioned command on the server used to post process the crash shows a message indicating > a mismatch between the vmcore and kernel.debug binary. > > I will now first try to clarify this and come back later on if the issue is still there. > > Thanks for helping. > Best regards, > Patrick Agrain > Just FYI, with respect to the System.map file, the following Dave's post would be helpful: https://www.redhat.com/archives/crash-utility/2018-June/msg00002.html If you have the vmlinux corresponding to a running kernel, you don't need the System.map file.