10:16 a.m.
Hi Bhupesh,
-----Original Message-----
> Just FYI, with respect to the System.map file, the following
Dave's post
> would be helpful:
> https://www.redhat.com/archives/crash-utility/2018-June/msg00002.html
>
> If you have the vmlinux corresponding to a running kernel, you don't need
> the System.map file.
Indeed, I remember discussing this with Dave A. some months back.
Infact with newer KASLR enabled kernels it makes little sense to use
System.map file as it contains the non-relocated symbol values that
were compiled into the vmlinux file - that's why I suggested just to
use to the vmlinux and vmcore file as an arguments while invoking the
crash tool to Patrick.
However, going back to the discussion I had with Dave A., I think the
main point of confusion was the 'crash_whitepaper' which is still
located at [1], and which provides the following example to invoke the
crash utility with the System.map file, vmlinux and vmcore
combination:
Examples when running on a dumpfile:
$ crash /boot/System.map vmlinux.dbg vmcore
I think while this was ok for older kernels which did not support
KASLR, for newer kernels this needs an update.
I am trying to see if I can modify the 'crash_whitepaper' and push the
same to github, so that users are not confused with the System.map
option.
What's your views on the same? Please let me know.
[1]. https://crash-utility.github.io/crash_whitepaper.html
There are several command lines with System.map in "4. Invocation" chapter and
I think that the ones in "RHEL2.1 Kernels (or kernels built without -g flag)"
section are no problem because the reason why it's needed is explained well.
The others in the "crash -h" output in "Invocation output" section
might be
a little confusing as you say if they see only the -S option's explanation..
I think it would be fine to omit the part of the output below the [mapfile]
argument and itself because the output looks to be intended almost only to
introduce the "crash -h" option. Then users see the current crash -h output
which doesn't have command lines with System.map. So, for instance:
diff --git a/crash_whitepaper.html b/crash_whitepaper.html
index 9b2e88c8acb4..7cb2608e6062 100644
--- a/crash_whitepaper.html
+++ b/crash_whitepaper.html
@@ -1166,7 +1166,8 @@ Usage:
file. If the [dumpfile] argument is not entered, the session will be
invoked on the live system using /dev/mem, which usually requires root
privileges.
-
+...
+<!-- Omit this part so that users don't use System.map needlessly.
[mapfile]
If the live system kernel, or the kernel from which the [dumpfile]
was derived, was not compiled with the -g switch, then the additional
@@ -1215,7 +1216,7 @@ Usage:
[-d num]
Set crash debug level [num]. The higher the number, the more debug data
will be printed during crash runtime.
-
+-->
</pre>
</td>
</tr>
How about this?
Thanks,
Kazu
8:03 a.m.
New subject: EXT: Re: crash start in CentOS 8
Hello all,
Back on this topic:
I compiled a kernel with debuginfo on the Virtual Machine with CentOS8.
I made a crash on this same VM.
I transferred the vmcore and vmlinux.debuginfo files on another CentOS8 baremetal
machine.
The command '/usr/bin/crash -d 15 vmlinux.debuginfo vmcore' produces following
logs, but still failing to get the crash> prompt.
Still KASLR ?
Thanks in advance for any information.
Best regards,
Patrick Agrain
-- Logs:
crash 7.2.6-2.el8
Copyright (C) 2002-2019 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
compressed kdump: header->utsname.machine: x86_64
compressed kdump: memory bitmap offset: 2000
diskdump_data:
filename: vmcore
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
dfd: 3
ofp: 0
machine_type: 62 (EM_X86_64)
header: 212b5d0
signature: "KDUMP "
header_version: 6
utsname:
sysname: Linux
nodename: localhost.localdomain
release: 4.18.0-147.el8.x86_64
version: #1 SMP Wed Dec 4 21:51:45 UTC 2019
machine: x86_64
domainname: (none)
timestamp:
tv_sec: 5ee72d23
tv_usec: 0
status: 2 (DUMP_DH_COMPRESSED_LZO)
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 32
max_mapnr: 524288
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 1
tasks[nr_cpus]: 0
sub_header: 0 (n/a)
sub_header_kdump: 212c5e0
phys_base: 7c00000
dump_level: 31 (0x1f)
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
split: 0
start_pfn: (unused)
end_pfn: (unused)
offset_vmcoreinfo: 4580 (0x11e4)
size_vmcoreinfo: 2025 (0x7e9)
OSRELEASE=4.18.0-147.el8.x86_64
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffffffffb5a12540
SYMBOL(node_online_map)=ffffffffb5c06d60
SYMBOL(swapper_pg_dir)=ffffffffb5a0a000
SYMBOL(_stext)=ffffffffb4800000
SYMBOL(vmap_area_list)=ffffffffb5ad2a90
SYMBOL(mem_section)=ffff89df7ffd2000
LENGTH(mem_section)=2048
SIZE(mem_section)=16
OFFSET(mem_section.section_mem_map)=0
SIZE(page)=64
SIZE(pglist_data)=171968
SIZE(zone)=1472
SIZE(free_area)=88
SIZE(list_head)=16
SIZE(nodemask_t)=128
OFFSET(page.flags)=0
OFFSET(page._refcount)=52
OFFSET(page.mapping)=24
OFFSET(page.lru)=8
OFFSET(page._mapcount)=48
OFFSET(page.private)=40
OFFSET(page.compound_dtor)=16
OFFSET(page.compound_order)=17
OFFSET(page.compound_head)=8
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=171232
OFFSET(pglist_data.node_start_pfn)=171240
OFFSET(pglist_data.node_spanned_pages)=171256
OFFSET(pglist_data.node_id)=171264
OFFSET(zone.free_area)=192
OFFSET(zone.vm_stat)=1296
OFFSET(zone.spanned_pages)=112
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=8
OFFSET(vmap_area.va_start)=0
OFFSET(vmap_area.list)=48
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=ffffffffb5a5a280
SYMBOL(log_buf_len)=ffffffffb5a5a27c
SYMBOL(log_first_idx)=ffffffffb633a770
SYMBOL(clear_idx)=ffffffffb633a744
SYMBOL(log_next_idx)=ffffffffb633a760
SIZE(printk_log)=16
OFFSET(printk_log.ts_nsec)=0
OFFSET(printk_log.len)=8
OFFSET(printk_log.text_len)=10
OFFSET(printk_log.dict_len)=12
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=12
NUMBER(PG_swapcache)=9
NUMBER(PG_swapbacked)=18
NUMBER(PG_slab)=8
NUMBER(PG_hwpoison)=22
NUMBER(PG_head_mask)=32768
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
NUMBER(HUGETLB_PAGE_DTOR)=2
NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
NUMBER(phys_base)=130023424
SYMBOL(init_top_pgt)=ffffffffb5a0a000
NUMBER(pgtable_l5_enabled)=0
SYMBOL(node_data)=ffffffffb5c02580
LENGTH(node_data)=1024
KERNELOFFSET=33800000
NUMBER(KERNEL_IMAGE_SIZE)=1073741824
NUMBER(sme_mask)=0
CRASHTIME=1592208675
offset_note: 4200 (0x1068)
size_note: 2408 (0x968)
notes_buf: 2129460
num_vmcoredd_notes: 0
num_prstatus_notes: 1
notes[0]: 2129460 (NT_PRSTATUS)
si.signo: 0 si.code: 0 si.errno: 0
cursig: 0 sigpend: 0 sighold: 0
pid: 2846 ppid: 0 pgrp: 0 sid:0
utime: 0.000000 stime: 0.000000
cutime: 0.000000 cstime: 0.000000
ORIG_RAX: ffffffffffffffff fpvalid: 0
R15: 0000000000000000 R14: 00005616bb1208f0
R13: ffffffffb5b36fc0 R12: 0000000000000000
RBP: 0000000000000004 RBX: 0000000000000063
R11: 6873617263206120 R10: 7265676769725420
R9: 54203a2071527379 R8: 000000000000056a
RAX: ffffffffb4d12d10 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000086
RDI: 0000000000000063 RIP: ffffffffb4d12d22
RFLAGS: 0000000000010246 RSP: ffffb03903147e80
FS_BASE: 00007f5d6e06f740
GS_BASE: 0000000000000000
CS: 0010 SS: 0018 DS: 0000
ES: 0000 FS: 0000 GS: 0000
snapshot_task: 0
num_qemu_notes: 0
NOTE offsets: 1068 (NT_PRSTATUS)
offset_eraseinfo: 0 (0x0)
size_eraseinfo: 0 (0x0)
start_pfn_64: (unused)
end_pfn_64: (unused)
max_mapnr_64: 524288 (0x80000)
data_offset: 22000
block_size: 4096
block_shift: 12
bitmap: 212f190
bitmap_len: 131072
max_mapnr: 524288 (0x80000)
dumpable_bitmap: 7f937ad93010
byte: 0
bit: 0
compressed_page: 212d5f0
curbufptr: 0
page_cache_hdr[0]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 217f1d0
pg_hit_count: 0
page_cache_hdr[1]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21801d0
pg_hit_count: 0
page_cache_hdr[2]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21811d0
pg_hit_count: 0
page_cache_hdr[3]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21821d0
pg_hit_count: 0
page_cache_hdr[4]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21831d0
pg_hit_count: 0
page_cache_hdr[5]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21841d0
pg_hit_count: 0
page_cache_hdr[6]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21851d0
pg_hit_count: 0
page_cache_hdr[7]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21861d0
pg_hit_count: 0
page_cache_hdr[8]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21871d0
pg_hit_count: 0
page_cache_hdr[9]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21881d0
pg_hit_count: 0
page_cache_hdr[10]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21891d0
pg_hit_count: 0
page_cache_hdr[11]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218a1d0
pg_hit_count: 0
page_cache_hdr[12]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218b1d0
pg_hit_count: 0
page_cache_hdr[13]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218c1d0
pg_hit_count: 0
page_cache_hdr[14]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218d1d0
pg_hit_count: 0
page_cache_hdr[15]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218e1d0
pg_hit_count: 0
page_cache_buf: 217f1d0
evict_index: 0
evictions: 0
accesses: 0
cached_reads: 0
valid_pages: 212a3b0
readmem: read_diskdump()
KASLR:
_stext from vmlinux.debuginfo: ffffffff81000000
_stext from vmcoreinfo: ffffffffb4800000
relocate: 33800000 (824MB)
crash: pv_init_ops exists: ARCH_PVOPS
VMCOREINFO: NUMBER(phys_base): 130023424 -> 7c00000
gdb vmlinux.debuginfo
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
GETBUF(328 -> 0)
GETBUF(1500 -> 1)
WARNING: kernel relocated [824MB]: patching 109676 gdb minimal_symbol values
FREEBUF(1)
FREEBUF(0)
<readmem: ffffffffb70df958, KVADDR, "page_offset_base", 8, (FOE|Q),
d8b948>
<read_diskdump: addr: ffffffffb70df958 paddr: 3ecdf958 cnt: 8>
read_diskdump: paddr/pfn: 3ecdf958/3ecdf -> cache physical page: 3ecdf000
GETBUF(328 -> 0)
FREEBUF(0)
GETBUF(1024 -> 0)
<readmem: ffffffffb7f0c5e0, KVADDR, "possible", 1024, (ROE), fcd120>
<read_diskdump: addr: ffffffffb7f0c5e0 paddr: 3fb0c5e0 cnt: 1024>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c5e0/3fb0c
crash: page excluded: kernel virtual address: ffffffffb7f0c5e0 type:
"possible"
WARNING: cannot read cpu_possible_map
<readmem: ffffffffb7f0bde0, KVADDR, "present", 1024, (ROE), fcd120>
<read_diskdump: addr: ffffffffb7f0bde0 paddr: 3fb0bde0 cnt: 544>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0bde0/3fb0b
crash: page excluded: kernel virtual address: ffffffffb7f0bde0 type: "present"
WARNING: cannot read cpu_present_map
<readmem: ffffffffb7f0c1e0, KVADDR, "online", 1024, (ROE), fcd120>
<read_diskdump: addr: ffffffffb7f0c1e0 paddr: 3fb0c1e0 cnt: 1024>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c1e0/3fb0c
crash: page excluded: kernel virtual address: ffffffffb7f0c1e0 type: "online"
WARNING: cannot read cpu_online_map
<readmem: ffffffffb7f0b9e0, KVADDR, "active", 1024, (ROE), fcd120>
<read_diskdump: addr: ffffffffb7f0b9e0 paddr: 3fb0b9e0 cnt: 1024>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0b9e0/3fb0b
crash: page excluded: kernel virtual address: ffffffffb7f0b9e0 type: "active"
WARNING: cannot read cpu_active_map
FREEBUF(0)
<readmem: ffffffffb731a5b0, KVADDR, "pv_init_ops", 8, (ROE),
7ffd3655d640>
<read_diskdump: addr: ffffffffb731a5b0 paddr: 3ef1a5b0 cnt: 8>
read_diskdump: paddr/pfn: 3ef1a5b0/3ef1a -> cache physical page: 3ef1a000
GETBUF(328 -> 0)
FREEBUF(0)
GETBUF(328 -> 0)
FREEBUF(0)
<readmem: ffffffffb9bad1d0, KVADDR, "shadow_timekeeper xtime_sec", 8, (ROE),
7ffd3655d5f0>
<read_diskdump: addr: ffffffffb9bad1d0 paddr: 417ad1d0 cnt: 8>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 417ad1d0/417ad
crash: page excluded: kernel virtual address: ffffffffb9bad1d0 type:
"shadow_timekeeper xtime_sec"
xtime timespec.tv_sec: 900f27: Mon Apr 20 07:31:03 1970
<readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE), d71b7c>
<read_diskdump: addr: ffffffffb7221d64 paddr: 3ee21d64 cnt: 390>
read_diskdump: paddr/pfn: 3ee21d64/3ee21 -> cache physical page: 3ee21000
utsname:
sysname:
nodename:
release:
version:
machine:
domainname:
base kernel version: 0.0.0
<readmem: ffffffffb6a016c0, KVADDR, "accessible check", 8, (ROE|Q),
7ffd3655af30>
<read_diskdump: addr: ffffffffb6a016c0 paddr: 3e6016c0 cnt: 8>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3e6016c0/3e601
crash: page excluded: kernel virtual address: ffffffffb6a016c0 type: "accessible
check"
crash: vmlinux.debuginfo and vmcore do not match!
Usage:
crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
Enter "crash -h" for details.
-----Message d'origine-----
De : crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] De la part
de HAGIO KAZUHITO(?? ??)
Envoyé : mercredi 17 juin 2020 17:17
À : Discussion list for crash utility usage, maintenance and development
<crash-utility(a)redhat.com>
Objet : EXT: Re: [Crash-utility] crash start in CentOS 8
** External email - Please consider with caution **
Hi Bhupesh,
-----Original Message-----
> Just FYI, with respect to the System.map file, the following
Dave's
> post would be helpful:
> https://www.redhat.com/archives/crash-utility/2018-June/msg00002.htm
> l
>
> If you have the vmlinux corresponding to a running kernel, you don't
> need the System.map file.
Indeed, I remember discussing this with Dave A. some months back.
Infact with newer KASLR enabled kernels it makes little sense to use
System.map file as it contains the non-relocated symbol values that
were compiled into the vmlinux file - that's why I suggested just to
use to the vmlinux and vmcore file as an arguments while invoking the
crash tool to Patrick.
However, going back to the discussion I had with Dave A., I think the
main point of confusion was the 'crash_whitepaper' which is still
located at [1], and which provides the following example to invoke the
crash utility with the System.map file, vmlinux and vmcore
combination:
Examples when running on a dumpfile:
$ crash /boot/System.map vmlinux.dbg vmcore
I think while this was ok for older kernels which did not support
KASLR, for newer kernels this needs an update.
I am trying to see if I can modify the 'crash_whitepaper' and push the
same to github, so that users are not confused with the System.map
option.
What's your views on the same? Please let me know.
[1]. https://crash-utility.github.io/crash_whitepaper.html
There are several command lines with System.map in "4. Invocation" chapter and I
think that the ones in "RHEL2.1 Kernels (or kernels built without -g flag)"
section are no problem because the reason why it's needed is explained well.
The others in the "crash -h" output in "Invocation output" section
might be a little confusing as you say if they see only the -S option's explanation..
I think it would be fine to omit the part of the output below the [mapfile] argument and
itself because the output looks to be intended almost only to introduce the "crash
-h" option. Then users see the current crash -h output which doesn't have
command lines with System.map. So, for instance:
diff --git a/crash_whitepaper.html b/crash_whitepaper.html index
9b2e88c8acb4..7cb2608e6062 100644
--- a/crash_whitepaper.html
+++ b/crash_whitepaper.html
@@ -1166,7 +1166,8 @@ Usage:
file. If the [dumpfile] argument is not entered, the session will be
invoked on the live system using /dev/mem, which usually requires root
privileges.
-
+...
+<!-- Omit this part so that users don't use System.map needlessly.
[mapfile]
If the live system kernel, or the kernel from which the [dumpfile]
was derived, was not compiled with the -g switch, then the additional @@ -1215,7
+1216,7 @@ Usage:
[-d num]
Set crash debug level [num]. The higher the number, the more debug data
will be printed during crash runtime.
-
+-->
</pre>
</td>
</tr>
How about this?
Thanks,
Kazu
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
4:53 p.m.
New subject: EXT: Re: crash start in CentOS 8
On Mon, Jul 6, 2020 at 6:03 AM Agrain Patrick
<patrick.agrain(a)al-enterprise.com> wrote:
Hello all,
Back on this topic:
I compiled a kernel with debuginfo on the Virtual Machine with CentOS8.
I made a crash on this same VM.
I transferred the vmcore and vmlinux.debuginfo files on another CentOS8 baremetal
machine.
The command '/usr/bin/crash -d 15 vmlinux.debuginfo vmcore' produces following
logs, but still failing to get the crash> prompt.
Still KASLR ?
Thanks in advance for any information.
Best regards,
Patrick Agrain
-- Logs:
crash 7.2.6-2.el8
You may try the latest release 7.2.8 or even better the tip repo
https://github.com/crash-utility/crash ?
Thanks,
Santosh
Copyright (C) 2002-2019 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
compressed kdump: header->utsname.machine: x86_64
compressed kdump: memory bitmap offset: 2000
diskdump_data:
filename: vmcore
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
dfd: 3
ofp: 0
machine_type: 62 (EM_X86_64)
header: 212b5d0
signature: "KDUMP "
header_version: 6
utsname:
sysname: Linux
nodename: localhost.localdomain
release: 4.18.0-147.el8.x86_64
version: #1 SMP Wed Dec 4 21:51:45 UTC 2019
machine: x86_64
domainname: (none)
timestamp:
tv_sec: 5ee72d23
tv_usec: 0
status: 2 (DUMP_DH_COMPRESSED_LZO)
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 32
max_mapnr: 524288
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 1
tasks[nr_cpus]: 0
sub_header: 0 (n/a)
sub_header_kdump: 212c5e0
phys_base: 7c00000
dump_level: 31 (0x1f)
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
split: 0
start_pfn: (unused)
end_pfn: (unused)
offset_vmcoreinfo: 4580 (0x11e4)
size_vmcoreinfo: 2025 (0x7e9)
OSRELEASE=4.18.0-147.el8.x86_64
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffffffffb5a12540
SYMBOL(node_online_map)=ffffffffb5c06d60
SYMBOL(swapper_pg_dir)=ffffffffb5a0a000
SYMBOL(_stext)=ffffffffb4800000
SYMBOL(vmap_area_list)=ffffffffb5ad2a90
SYMBOL(mem_section)=ffff89df7ffd2000
LENGTH(mem_section)=2048
SIZE(mem_section)=16
OFFSET(mem_section.section_mem_map)=0
SIZE(page)=64
SIZE(pglist_data)=171968
SIZE(zone)=1472
SIZE(free_area)=88
SIZE(list_head)=16
SIZE(nodemask_t)=128
OFFSET(page.flags)=0
OFFSET(page._refcount)=52
OFFSET(page.mapping)=24
OFFSET(page.lru)=8
OFFSET(page._mapcount)=48
OFFSET(page.private)=40
OFFSET(page.compound_dtor)=16
OFFSET(page.compound_order)=17
OFFSET(page.compound_head)=8
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=171232
OFFSET(pglist_data.node_start_pfn)=171240
OFFSET(pglist_data.node_spanned_pages)=171256
OFFSET(pglist_data.node_id)=171264
OFFSET(zone.free_area)=192
OFFSET(zone.vm_stat)=1296
OFFSET(zone.spanned_pages)=112
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=8
OFFSET(vmap_area.va_start)=0
OFFSET(vmap_area.list)=48
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=ffffffffb5a5a280
SYMBOL(log_buf_len)=ffffffffb5a5a27c
SYMBOL(log_first_idx)=ffffffffb633a770
SYMBOL(clear_idx)=ffffffffb633a744
SYMBOL(log_next_idx)=ffffffffb633a760
SIZE(printk_log)=16
OFFSET(printk_log.ts_nsec)=0
OFFSET(printk_log.len)=8
OFFSET(printk_log.text_len)=10
OFFSET(printk_log.dict_len)=12
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=12
NUMBER(PG_swapcache)=9
NUMBER(PG_swapbacked)=18
NUMBER(PG_slab)=8
NUMBER(PG_hwpoison)=22
NUMBER(PG_head_mask)=32768
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
NUMBER(HUGETLB_PAGE_DTOR)=2
NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
NUMBER(phys_base)=130023424
SYMBOL(init_top_pgt)=ffffffffb5a0a000
NUMBER(pgtable_l5_enabled)=0
SYMBOL(node_data)=ffffffffb5c02580
LENGTH(node_data)=1024
KERNELOFFSET=33800000
NUMBER(KERNEL_IMAGE_SIZE)=1073741824
NUMBER(sme_mask)=0
CRASHTIME=1592208675
offset_note: 4200 (0x1068)
size_note: 2408 (0x968)
notes_buf: 2129460
num_vmcoredd_notes: 0
num_prstatus_notes: 1
notes[0]: 2129460 (NT_PRSTATUS)
si.signo: 0 si.code: 0 si.errno: 0
cursig: 0 sigpend: 0 sighold: 0
pid: 2846 ppid: 0 pgrp: 0 sid:0
utime: 0.000000 stime: 0.000000
cutime: 0.000000 cstime: 0.000000
ORIG_RAX: ffffffffffffffff fpvalid: 0
R15: 0000000000000000 R14: 00005616bb1208f0
R13: ffffffffb5b36fc0 R12: 0000000000000000
RBP: 0000000000000004 RBX: 0000000000000063
R11: 6873617263206120 R10: 7265676769725420
R9: 54203a2071527379 R8: 000000000000056a
RAX: ffffffffb4d12d10 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000086
RDI: 0000000000000063 RIP: ffffffffb4d12d22
RFLAGS: 0000000000010246 RSP: ffffb03903147e80
FS_BASE: 00007f5d6e06f740
GS_BASE: 0000000000000000
CS: 0010 SS: 0018 DS: 0000
ES: 0000 FS: 0000 GS: 0000
snapshot_task: 0
num_qemu_notes: 0
NOTE offsets: 1068 (NT_PRSTATUS)
offset_eraseinfo: 0 (0x0)
size_eraseinfo: 0 (0x0)
start_pfn_64: (unused)
end_pfn_64: (unused)
max_mapnr_64: 524288 (0x80000)
data_offset: 22000
block_size: 4096
block_shift: 12
bitmap: 212f190
bitmap_len: 131072
max_mapnr: 524288 (0x80000)
dumpable_bitmap: 7f937ad93010
byte: 0
bit: 0
compressed_page: 212d5f0
curbufptr: 0
page_cache_hdr[0]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 217f1d0
pg_hit_count: 0
page_cache_hdr[1]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21801d0
pg_hit_count: 0
page_cache_hdr[2]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21811d0
pg_hit_count: 0
page_cache_hdr[3]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21821d0
pg_hit_count: 0
page_cache_hdr[4]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21831d0
pg_hit_count: 0
page_cache_hdr[5]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21841d0
pg_hit_count: 0
page_cache_hdr[6]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21851d0
pg_hit_count: 0
page_cache_hdr[7]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21861d0
pg_hit_count: 0
page_cache_hdr[8]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21871d0
pg_hit_count: 0
page_cache_hdr[9]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21881d0
pg_hit_count: 0
page_cache_hdr[10]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21891d0
pg_hit_count: 0
page_cache_hdr[11]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218a1d0
pg_hit_count: 0
page_cache_hdr[12]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218b1d0
pg_hit_count: 0
page_cache_hdr[13]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218c1d0
pg_hit_count: 0
page_cache_hdr[14]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218d1d0
pg_hit_count: 0
page_cache_hdr[15]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218e1d0
pg_hit_count: 0
page_cache_buf: 217f1d0
evict_index: 0
evictions: 0
accesses: 0
cached_reads: 0
valid_pages: 212a3b0
readmem: read_diskdump()
KASLR:
_stext from vmlinux.debuginfo: ffffffff81000000
_stext from vmcoreinfo: ffffffffb4800000
relocate: 33800000 (824MB)
crash: pv_init_ops exists: ARCH_PVOPS
VMCOREINFO: NUMBER(phys_base): 130023424 -> 7c00000
gdb vmlinux.debuginfo
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
GETBUF(328 -> 0)
GETBUF(1500 -> 1)
WARNING: kernel relocated [824MB]: patching 109676 gdb minimal_symbol values
FREEBUF(1)
FREEBUF(0)
<readmem: ffffffffb70df958, KVADDR, "page_offset_base", 8, (FOE|Q),
d8b948>
<read_diskdump: addr: ffffffffb70df958 paddr: 3ecdf958 cnt: 8>
read_diskdump: paddr/pfn: 3ecdf958/3ecdf -> cache physical page: 3ecdf000
GETBUF(328 -> 0)
FREEBUF(0)
GETBUF(1024 -> 0)
<readmem: ffffffffb7f0c5e0, KVADDR, "possible", 1024, (ROE), fcd120>
<read_diskdump: addr: ffffffffb7f0c5e0 paddr: 3fb0c5e0 cnt: 1024>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c5e0/3fb0c
crash: page excluded: kernel virtual address: ffffffffb7f0c5e0 type:
"possible"
WARNING: cannot read cpu_possible_map
<readmem: ffffffffb7f0bde0, KVADDR, "present", 1024, (ROE), fcd120>
<read_diskdump: addr: ffffffffb7f0bde0 paddr: 3fb0bde0 cnt: 544>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0bde0/3fb0b
crash: page excluded: kernel virtual address: ffffffffb7f0bde0 type:
"present"
WARNING: cannot read cpu_present_map
<readmem: ffffffffb7f0c1e0, KVADDR, "online", 1024, (ROE), fcd120>
<read_diskdump: addr: ffffffffb7f0c1e0 paddr: 3fb0c1e0 cnt: 1024>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c1e0/3fb0c
crash: page excluded: kernel virtual address: ffffffffb7f0c1e0 type: "online"
WARNING: cannot read cpu_online_map
<readmem: ffffffffb7f0b9e0, KVADDR, "active", 1024, (ROE), fcd120>
<read_diskdump: addr: ffffffffb7f0b9e0 paddr: 3fb0b9e0 cnt: 1024>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0b9e0/3fb0b
crash: page excluded: kernel virtual address: ffffffffb7f0b9e0 type: "active"
WARNING: cannot read cpu_active_map
FREEBUF(0)
<readmem: ffffffffb731a5b0, KVADDR, "pv_init_ops", 8, (ROE),
7ffd3655d640>
<read_diskdump: addr: ffffffffb731a5b0 paddr: 3ef1a5b0 cnt: 8>
read_diskdump: paddr/pfn: 3ef1a5b0/3ef1a -> cache physical page: 3ef1a000
GETBUF(328 -> 0)
FREEBUF(0)
GETBUF(328 -> 0)
FREEBUF(0)
<readmem: ffffffffb9bad1d0, KVADDR, "shadow_timekeeper xtime_sec", 8, (ROE),
7ffd3655d5f0>
<read_diskdump: addr: ffffffffb9bad1d0 paddr: 417ad1d0 cnt: 8>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 417ad1d0/417ad
crash: page excluded: kernel virtual address: ffffffffb9bad1d0 type:
"shadow_timekeeper xtime_sec"
xtime timespec.tv_sec: 900f27: Mon Apr 20 07:31:03 1970
<readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE), d71b7c>
<read_diskdump: addr: ffffffffb7221d64 paddr: 3ee21d64 cnt: 390>
read_diskdump: paddr/pfn: 3ee21d64/3ee21 -> cache physical page: 3ee21000
utsname:
sysname:
nodename:
release:
version:
machine:
domainname:
base kernel version: 0.0.0
<readmem: ffffffffb6a016c0, KVADDR, "accessible check", 8, (ROE|Q),
7ffd3655af30>
<read_diskdump: addr: ffffffffb6a016c0 paddr: 3e6016c0 cnt: 8>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3e6016c0/3e601
crash: page excluded: kernel virtual address: ffffffffb6a016c0 type: "accessible
check"
crash: vmlinux.debuginfo and vmcore do not match!
Usage:
crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
Enter "crash -h" for details.
-----Message d'origine-----
De : crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] De la
part de HAGIO KAZUHITO(?? ??)
Envoyé : mercredi 17 juin 2020 17:17
À : Discussion list for crash utility usage, maintenance and development
<crash-utility(a)redhat.com>
Objet : EXT: Re: [Crash-utility] crash start in CentOS 8
** External email - Please consider with caution **
Hi Bhupesh,
-----Original Message-----
> > Just FYI, with respect to the System.map file, the following Dave's
> > post would be helpful:
> > https://www.redhat.com/archives/crash-utility/2018-June/msg00002.htm
> > l
> >
> > If you have the vmlinux corresponding to a running kernel, you don't
> > need the System.map file.
>
> Indeed, I remember discussing this with Dave A. some months back.
> Infact with newer KASLR enabled kernels it makes little sense to use
> System.map file as it contains the non-relocated symbol values that
> were compiled into the vmlinux file - that's why I suggested just to
> use to the vmlinux and vmcore file as an arguments while invoking the
> crash tool to Patrick.
>
> However, going back to the discussion I had with Dave A., I think the
> main point of confusion was the 'crash_whitepaper' which is still
> located at [1], and which provides the following example to invoke the
> crash utility with the System.map file, vmlinux and vmcore
> combination:
>
> Examples when running on a dumpfile:
>
> $ crash /boot/System.map vmlinux.dbg vmcore
>
> I think while this was ok for older kernels which did not support
> KASLR, for newer kernels this needs an update.
> I am trying to see if I can modify the 'crash_whitepaper' and push the
> same to github, so that users are not confused with the System.map
> option.
>
> What's your views on the same? Please let me know.
>
> [1]. https://crash-utility.github.io/crash_whitepaper.html
There are several command lines with System.map in "4. Invocation" chapter and
I think that the ones in "RHEL2.1 Kernels (or kernels built without -g flag)"
section are no problem because the reason why it's needed is explained well.
The others in the "crash -h" output in "Invocation output" section
might be a little confusing as you say if they see only the -S option's explanation..
I think it would be fine to omit the part of the output below the [mapfile] argument and
itself because the output looks to be intended almost only to introduce the "crash
-h" option. Then users see the current crash -h output which doesn't have
command lines with System.map. So, for instance:
diff --git a/crash_whitepaper.html b/crash_whitepaper.html index
9b2e88c8acb4..7cb2608e6062 100644
--- a/crash_whitepaper.html
+++ b/crash_whitepaper.html
@@ -1166,7 +1166,8 @@ Usage:
file. If the [dumpfile] argument is not entered, the session will be
invoked on the live system using /dev/mem, which usually requires root
privileges.
-
+...
+<!-- Omit this part so that users don't use System.map needlessly.
[mapfile]
If the live system kernel, or the kernel from which the [dumpfile]
was derived, was not compiled with the -g switch, then the additional @@ -1215,7
+1216,7 @@ Usage:
[-d num]
Set crash debug level [num]. The higher the number, the more debug data
will be printed during crash runtime.
-
+-->
</pre>
</td>
</tr>
How about this?
Thanks,
Kazu
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
11:59 p.m.
New subject: EXT: Re: crash start in CentOS 8
Hi Patrick,
OSRELEASE=4.18.0-147.el8.x86_64
Hmm, I think you might boot the virtual machine with the original (unmodified)
CentOS 8 kernel, because the address of init_uts_ns looks to be its one:
SYMBOL(init_uts_ns)=ffffffffb5a12540
KERNELOFFSET=33800000
I happen to have a CentOS 8 machine..
# grep 'D init_uts_ns' /boot/System.map-4.18.0-147.el8.x86_64
ffffffff82212540 D init_uts_ns
# printf "%16llx\n" $(( 0xffffffff82212540 + 0x33800000 ))
ffffffffb5a12540
while your vmcore.debuginfo has the different address:
<readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390,
(ROE), d71b7c>
So, which kernel do you actually want to use on the virtual machine?
(1) The original CentOS 8 kernel
(2) The kernel you built
If it's (1), you can just install the kernel-debuginfo package corresponding to
the original CentOS 8 kernel into the machine on which you want to analyze
vmcores and execute:
# crash /usr/lib/debug/lib/modules/4.18.0-147.el8.x86_64/vmlinux vmcore
If it's (2), you need to boot the VM with the kernel and make a crash.
Could you check this first?
Thanks,
Kazu
On Mon, Jul 6, 2020 at 10:03 PM Agrain Patrick
<patrick.agrain(a)al-enterprise.com> wrote:
>
> Hello all,
>
> Back on this topic:
> I compiled a kernel with debuginfo on the Virtual Machine with CentOS8.
> I made a crash on this same VM.
> I transferred the vmcore and vmlinux.debuginfo files on another CentOS8 baremetal
machine.
>
> The command '/usr/bin/crash -d 15 vmlinux.debuginfo vmcore' produces
following logs, but still failing to get the crash> prompt.
> Still KASLR ?
>
> Thanks in advance for any information.
> Best regards,
> Patrick Agrain
>
> -- Logs:
> crash 7.2.6-2.el8
> Copyright (C) 2002-2019 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
details.
>
> compressed kdump: header->utsname.machine: x86_64
> compressed kdump: memory bitmap offset: 2000
> diskdump_data:
> filename: vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 3
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: 212b5d0
> signature: "KDUMP "
> header_version: 6
> utsname:
> sysname: Linux
> nodename: localhost.localdomain
> release: 4.18.0-147.el8.x86_64
> version: #1 SMP Wed Dec 4 21:51:45 UTC 2019
> machine: x86_64
> domainname: (none)
> timestamp:
> tv_sec: 5ee72d23
> tv_usec: 0
> status: 2 (DUMP_DH_COMPRESSED_LZO)
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 32
> max_mapnr: 524288
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1
> tasks[nr_cpus]: 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 212c5e0
> phys_base: 7c00000
> dump_level: 31 (0x1f)
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
> split: 0
> start_pfn: (unused)
> end_pfn: (unused)
> offset_vmcoreinfo: 4580 (0x11e4)
> size_vmcoreinfo: 2025 (0x7e9)
OSRELEASE=4.18.0-147.el8.x86_64
>
PAGESIZE=4096
> SYMBOL(init_uts_ns)=ffffffffb5a12540
> SYMBOL(node_online_map)=ffffffffb5c06d60
> SYMBOL(swapper_pg_dir)=ffffffffb5a0a000
> SYMBOL(_stext)=ffffffffb4800000
> SYMBOL(vmap_area_list)=ffffffffb5ad2a90
> SYMBOL(mem_section)=ffff89df7ffd2000
> LENGTH(mem_section)=2048
> SIZE(mem_section)=16
> OFFSET(mem_section.section_mem_map)=0
> SIZE(page)=64
> SIZE(pglist_data)=171968
> SIZE(zone)=1472
> SIZE(free_area)=88
> SIZE(list_head)=16
> SIZE(nodemask_t)=128
> OFFSET(page.flags)=0
> OFFSET(page._refcount)=52
> OFFSET(page.mapping)=24
> OFFSET(page.lru)=8
> OFFSET(page._mapcount)=48
> OFFSET(page.private)=40
> OFFSET(page.compound_dtor)=16
> OFFSET(page.compound_order)=17
> OFFSET(page.compound_head)=8
> OFFSET(pglist_data.node_zones)=0
> OFFSET(pglist_data.nr_zones)=171232
> OFFSET(pglist_data.node_start_pfn)=171240
> OFFSET(pglist_data.node_spanned_pages)=171256
> OFFSET(pglist_data.node_id)=171264
> OFFSET(zone.free_area)=192
> OFFSET(zone.vm_stat)=1296
> OFFSET(zone.spanned_pages)=112
> OFFSET(free_area.free_list)=0
> OFFSET(list_head.next)=0
> OFFSET(list_head.prev)=8
> OFFSET(vmap_area.va_start)=0
> OFFSET(vmap_area.list)=48
> LENGTH(zone.free_area)=11
> SYMBOL(log_buf)=ffffffffb5a5a280
> SYMBOL(log_buf_len)=ffffffffb5a5a27c
> SYMBOL(log_first_idx)=ffffffffb633a770
> SYMBOL(clear_idx)=ffffffffb633a744
> SYMBOL(log_next_idx)=ffffffffb633a760
> SIZE(printk_log)=16
> OFFSET(printk_log.ts_nsec)=0
> OFFSET(printk_log.len)=8
> OFFSET(printk_log.text_len)=10
> OFFSET(printk_log.dict_len)=12
> LENGTH(free_area.free_list)=5
> NUMBER(NR_FREE_PAGES)=0
> NUMBER(PG_lru)=5
> NUMBER(PG_private)=12
> NUMBER(PG_swapcache)=9
> NUMBER(PG_swapbacked)=18
> NUMBER(PG_slab)=8
> NUMBER(PG_hwpoison)=22
> NUMBER(PG_head_mask)=32768
> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
> NUMBER(HUGETLB_PAGE_DTOR)=2
> NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
> NUMBER(phys_base)=130023424
> SYMBOL(init_top_pgt)=ffffffffb5a0a000
> NUMBER(pgtable_l5_enabled)=0
> SYMBOL(node_data)=ffffffffb5c02580
> LENGTH(node_data)=1024
> KERNELOFFSET=33800000
> NUMBER(KERNEL_IMAGE_SIZE)=1073741824
> NUMBER(sme_mask)=0
> CRASHTIME=1592208675
> offset_note: 4200 (0x1068)
> size_note: 2408 (0x968)
> notes_buf: 2129460
> num_vmcoredd_notes: 0
> num_prstatus_notes: 1
> notes[0]: 2129460 (NT_PRSTATUS)
> si.signo: 0 si.code: 0 si.errno: 0
> cursig: 0 sigpend: 0 sighold: 0
> pid: 2846 ppid: 0 pgrp: 0 sid:0
> utime: 0.000000 stime: 0.000000
> cutime: 0.000000 cstime: 0.000000
> ORIG_RAX: ffffffffffffffff fpvalid: 0
> R15: 0000000000000000 R14: 00005616bb1208f0
> R13: ffffffffb5b36fc0 R12: 0000000000000000
> RBP: 0000000000000004 RBX: 0000000000000063
> R11: 6873617263206120 R10: 7265676769725420
> R9: 54203a2071527379 R8: 000000000000056a
> RAX: ffffffffb4d12d10 RCX: 0000000000000006
> RDX: 0000000000000000 RSI: 0000000000000086
> RDI: 0000000000000063 RIP: ffffffffb4d12d22
> RFLAGS: 0000000000010246 RSP: ffffb03903147e80
> FS_BASE: 00007f5d6e06f740
> GS_BASE: 0000000000000000
> CS: 0010 SS: 0018 DS: 0000
> ES: 0000 FS: 0000 GS: 0000
> snapshot_task: 0
> num_qemu_notes: 0
> NOTE offsets: 1068 (NT_PRSTATUS)
> offset_eraseinfo: 0 (0x0)
> size_eraseinfo: 0 (0x0)
> start_pfn_64: (unused)
> end_pfn_64: (unused)
> max_mapnr_64: 524288 (0x80000)
>
> data_offset: 22000
> block_size: 4096
> block_shift: 12
> bitmap: 212f190
> bitmap_len: 131072
> max_mapnr: 524288 (0x80000)
> dumpable_bitmap: 7f937ad93010
> byte: 0
> bit: 0
> compressed_page: 212d5f0
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 217f1d0
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21801d0
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21811d0
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21821d0
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21831d0
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21841d0
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21851d0
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21861d0
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21871d0
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21881d0
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21891d0
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218a1d0
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218b1d0
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218c1d0
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218d1d0
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218e1d0
> pg_hit_count: 0
>
> page_cache_buf: 217f1d0
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: 212a3b0
> readmem: read_diskdump()
> KASLR:
> _stext from vmlinux.debuginfo: ffffffff81000000
> _stext from vmcoreinfo: ffffffffb4800000
> relocate: 33800000 (824MB)
> crash: pv_init_ops exists: ARCH_PVOPS
> VMCOREINFO: NUMBER(phys_base): 130023424 -> 7c00000
> gdb vmlinux.debuginfo
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> GETBUF(328 -> 0)
> GETBUF(1500 -> 1)
>
> WARNING: kernel relocated [824MB]: patching 109676 gdb minimal_symbol values
>
> FREEBUF(1)
> FREEBUF(0)
> <readmem: ffffffffb70df958, KVADDR, "page_offset_base", 8, (FOE|Q),
d8b948>
> <read_diskdump: addr: ffffffffb70df958 paddr: 3ecdf958 cnt: 8>
> read_diskdump: paddr/pfn: 3ecdf958/3ecdf -> cache physical page: 3ecdf000
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(1024 -> 0)
> <readmem: ffffffffb7f0c5e0, KVADDR, "possible", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0c5e0 paddr: 3fb0c5e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c5e0/3fb0c
> crash: page excluded: kernel virtual address: ffffffffb7f0c5e0 type:
"possible"
> WARNING: cannot read cpu_possible_map
> <readmem: ffffffffb7f0bde0, KVADDR, "present", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0bde0 paddr: 3fb0bde0 cnt: 544>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0bde0/3fb0b
> crash: page excluded: kernel virtual address: ffffffffb7f0bde0 type:
"present"
> WARNING: cannot read cpu_present_map
> <readmem: ffffffffb7f0c1e0, KVADDR, "online", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0c1e0 paddr: 3fb0c1e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c1e0/3fb0c
> crash: page excluded: kernel virtual address: ffffffffb7f0c1e0 type:
"online"
> WARNING: cannot read cpu_online_map
> <readmem: ffffffffb7f0b9e0, KVADDR, "active", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0b9e0 paddr: 3fb0b9e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0b9e0/3fb0b
> crash: page excluded: kernel virtual address: ffffffffb7f0b9e0 type:
"active"
> WARNING: cannot read cpu_active_map
> FREEBUF(0)
> <readmem: ffffffffb731a5b0, KVADDR, "pv_init_ops", 8, (ROE),
7ffd3655d640>
> <read_diskdump: addr: ffffffffb731a5b0 paddr: 3ef1a5b0 cnt: 8>
> read_diskdump: paddr/pfn: 3ef1a5b0/3ef1a -> cache physical page: 3ef1a000
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(328 -> 0)
> FREEBUF(0)
> <readmem: ffffffffb9bad1d0, KVADDR, "shadow_timekeeper xtime_sec", 8,
(ROE), 7ffd3655d5f0>
> <read_diskdump: addr: ffffffffb9bad1d0 paddr: 417ad1d0 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 417ad1d0/417ad
> crash: page excluded: kernel virtual address: ffffffffb9bad1d0 type:
"shadow_timekeeper xtime_sec"
> xtime timespec.tv_sec: 900f27: Mon Apr 20 07:31:03 1970
<readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390,
(ROE), d71b7c>
> <read_diskdump: addr: ffffffffb7221d64 paddr: 3ee21d64
cnt: 390>
> read_diskdump: paddr/pfn: 3ee21d64/3ee21 -> cache physical page: 3ee21000
> utsname:
> sysname:
> nodename:
> release:
> version:
> machine:
> domainname:
> base kernel version: 0.0.0
> <readmem: ffffffffb6a016c0, KVADDR, "accessible check", 8, (ROE|Q),
7ffd3655af30>
> <read_diskdump: addr: ffffffffb6a016c0 paddr: 3e6016c0 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3e6016c0/3e601
> crash: page excluded: kernel virtual address: ffffffffb6a016c0 type:
"accessible check"
> crash: vmlinux.debuginfo and vmcore do not match!
>
> Usage:
>
> crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
> crash [OPTION]... [NAMELIST] (live system form)
>
> Enter "crash -h" for details.
>
12:20 a.m.
New subject: EXT: Re: EXT: Re: crash start in CentOS 8
Hi Kazu,
Thanks for pointing that.
I will check this part (I tought I have booted the VM on its original kernel, but I'm
maybe wrong).
Will come back.
Best regards,
Patrick Agrain
-----Message d'origine-----
De : crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] De la part
de Kazuhito Hagio
Envoyé : jeudi 9 juillet 2020 07:00
À : Discussion list for crash utility usage, maintenance and development
<crash-utility(a)redhat.com>
Objet : EXT: Re: [Crash-utility] EXT: Re: crash start in CentOS 8
** External email - Please consider with caution **
Hi Patrick,
OSRELEASE=4.18.0-147.el8.x86_64
Hmm, I think you might boot the virtual machine with the original (unmodified) CentOS 8
kernel, because the address of init_uts_ns looks to be its one:
SYMBOL(init_uts_ns)=ffffffffb5a12540
KERNELOFFSET=33800000
I happen to have a CentOS 8 machine..
# grep 'D init_uts_ns' /boot/System.map-4.18.0-147.el8.x86_64
ffffffff82212540 D init_uts_ns
# printf "%16llx\n" $(( 0xffffffff82212540 + 0x33800000 ))
ffffffffb5a12540
while your vmcore.debuginfo has the different address:
<readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390,
(ROE), d71b7c>
So, which kernel do you actually want to use on the virtual machine?
(1) The original CentOS 8 kernel
(2) The kernel you built
If it's (1), you can just install the kernel-debuginfo package corresponding to the
original CentOS 8 kernel into the machine on which you want to analyze vmcores and
execute:
# crash /usr/lib/debug/lib/modules/4.18.0-147.el8.x86_64/vmlinux vmcore
If it's (2), you need to boot the VM with the kernel and make a crash.
Could you check this first?
Thanks,
Kazu
On Mon, Jul 6, 2020 at 10:03 PM Agrain Patrick <patrick.agrain(a)al-enterprise.com>
wrote:
>
> Hello all,
>
> Back on this topic:
> I compiled a kernel with debuginfo on the Virtual Machine with CentOS8.
> I made a crash on this same VM.
> I transferred the vmcore and vmlinux.debuginfo files on another CentOS8 baremetal
machine.
>
> The command '/usr/bin/crash -d 15 vmlinux.debuginfo vmcore' produces
following logs, but still failing to get the crash> prompt.
> Still KASLR ?
>
> Thanks in advance for any information.
> Best regards,
> Patrick Agrain
>
> -- Logs:
> crash 7.2.6-2.el8
> Copyright (C) 2002-2019 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C)
> 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012
> Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002,
> 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License, and you are welcome to change it and/or distribute copies of
> it under certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
details.
>
> compressed kdump: header->utsname.machine: x86_64 compressed kdump:
> memory bitmap offset: 2000
> diskdump_data:
> filename: vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 3
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: 212b5d0
> signature: "KDUMP "
> header_version: 6
> utsname:
> sysname: Linux
> nodename: localhost.localdomain
> release: 4.18.0-147.el8.x86_64
> version: #1 SMP Wed Dec 4 21:51:45 UTC 2019
> machine: x86_64
> domainname: (none)
> timestamp:
> tv_sec: 5ee72d23
> tv_usec: 0
> status: 2 (DUMP_DH_COMPRESSED_LZO)
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 32
> max_mapnr: 524288
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1
> tasks[nr_cpus]: 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 212c5e0
> phys_base: 7c00000
> dump_level: 31 (0x1f)
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
> split: 0
> start_pfn: (unused)
> end_pfn: (unused)
> offset_vmcoreinfo: 4580 (0x11e4)
> size_vmcoreinfo: 2025 (0x7e9)
OSRELEASE=4.18.0-147.el8.x86_64
>
PAGESIZE=4096
> SYMBOL(init_uts_ns)=ffffffffb5a12540
> SYMBOL(node_online_map)=ffffffffb5c06d60
> SYMBOL(swapper_pg_dir)=ffffffffb5a0a000
> SYMBOL(_stext)=ffffffffb4800000
> SYMBOL(vmap_area_list)=ffffffffb5ad2a90
> SYMBOL(mem_section)=ffff89df7ffd2000
> LENGTH(mem_section)=2048
> SIZE(mem_section)=16
> OFFSET(mem_section.section_mem_map)=0
> SIZE(page)=64
> SIZE(pglist_data)=171968
> SIZE(zone)=1472
> SIZE(free_area)=88
> SIZE(list_head)=16
> SIZE(nodemask_t)=128
> OFFSET(page.flags)=0
> OFFSET(page._refcount)=52
> OFFSET(page.mapping)=24
> OFFSET(page.lru)=8
> OFFSET(page._mapcount)=48
> OFFSET(page.private)=40
> OFFSET(page.compound_dtor)=16
> OFFSET(page.compound_order)=17
> OFFSET(page.compound_head)=8
> OFFSET(pglist_data.node_zones)=0
> OFFSET(pglist_data.nr_zones)=171232
> OFFSET(pglist_data.node_start_pfn)=171240
> OFFSET(pglist_data.node_spanned_pages)=171256
> OFFSET(pglist_data.node_id)=171264
> OFFSET(zone.free_area)=192
> OFFSET(zone.vm_stat)=1296
> OFFSET(zone.spanned_pages)=112
> OFFSET(free_area.free_list)=0
> OFFSET(list_head.next)=0
> OFFSET(list_head.prev)=8
> OFFSET(vmap_area.va_start)=0
> OFFSET(vmap_area.list)=48
> LENGTH(zone.free_area)=11
> SYMBOL(log_buf)=ffffffffb5a5a280
> SYMBOL(log_buf_len)=ffffffffb5a5a27c
> SYMBOL(log_first_idx)=ffffffffb633a770
> SYMBOL(clear_idx)=ffffffffb633a744
> SYMBOL(log_next_idx)=ffffffffb633a760
> SIZE(printk_log)=16
> OFFSET(printk_log.ts_nsec)=0
> OFFSET(printk_log.len)=8
> OFFSET(printk_log.text_len)=10
> OFFSET(printk_log.dict_len)=12
> LENGTH(free_area.free_list)=5
> NUMBER(NR_FREE_PAGES)=0
> NUMBER(PG_lru)=5
> NUMBER(PG_private)=12
> NUMBER(PG_swapcache)=9
> NUMBER(PG_swapbacked)=18
> NUMBER(PG_slab)=8
> NUMBER(PG_hwpoison)=22
> NUMBER(PG_head_mask)=32768
> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
> NUMBER(HUGETLB_PAGE_DTOR)=2
> NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
> NUMBER(phys_base)=130023424
> SYMBOL(init_top_pgt)=ffffffffb5a0a000
> NUMBER(pgtable_l5_enabled)=0
> SYMBOL(node_data)=ffffffffb5c02580
> LENGTH(node_data)=1024
> KERNELOFFSET=33800000
> NUMBER(KERNEL_IMAGE_SIZE)=1073741824
> NUMBER(sme_mask)=0
> CRASHTIME=1592208675
> offset_note: 4200 (0x1068)
> size_note: 2408 (0x968)
> notes_buf: 2129460
> num_vmcoredd_notes: 0
> num_prstatus_notes: 1
> notes[0]: 2129460 (NT_PRSTATUS)
> si.signo: 0 si.code: 0 si.errno: 0
> cursig: 0 sigpend: 0 sighold: 0
> pid: 2846 ppid: 0 pgrp: 0 sid:0
> utime: 0.000000 stime: 0.000000
> cutime: 0.000000 cstime: 0.000000
> ORIG_RAX: ffffffffffffffff fpvalid: 0
> R15: 0000000000000000 R14: 00005616bb1208f0
> R13: ffffffffb5b36fc0 R12: 0000000000000000
> RBP: 0000000000000004 RBX: 0000000000000063
> R11: 6873617263206120 R10: 7265676769725420
> R9: 54203a2071527379 R8: 000000000000056a
> RAX: ffffffffb4d12d10 RCX: 0000000000000006
> RDX: 0000000000000000 RSI: 0000000000000086
> RDI: 0000000000000063 RIP: ffffffffb4d12d22
> RFLAGS: 0000000000010246 RSP: ffffb03903147e80
> FS_BASE: 00007f5d6e06f740
> GS_BASE: 0000000000000000
> CS: 0010 SS: 0018 DS: 0000
> ES: 0000 FS: 0000 GS: 0000
> snapshot_task: 0
> num_qemu_notes: 0
> NOTE offsets: 1068 (NT_PRSTATUS)
> offset_eraseinfo: 0 (0x0)
> size_eraseinfo: 0 (0x0)
> start_pfn_64: (unused)
> end_pfn_64: (unused)
> max_mapnr_64: 524288 (0x80000)
>
> data_offset: 22000
> block_size: 4096
> block_shift: 12
> bitmap: 212f190
> bitmap_len: 131072
> max_mapnr: 524288 (0x80000)
> dumpable_bitmap: 7f937ad93010
> byte: 0
> bit: 0
> compressed_page: 212d5f0
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 217f1d0
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21801d0
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21811d0
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21821d0
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21831d0
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21841d0
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21851d0
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21861d0
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21871d0
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21881d0
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21891d0
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218a1d0
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218b1d0
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218c1d0
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218d1d0
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218e1d0
> pg_hit_count: 0
>
> page_cache_buf: 217f1d0
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: 212a3b0
> readmem: read_diskdump()
> KASLR:
> _stext from vmlinux.debuginfo: ffffffff81000000
> _stext from vmcoreinfo: ffffffffb4800000
> relocate: 33800000 (824MB)
> crash: pv_init_ops exists: ARCH_PVOPS
> VMCOREINFO: NUMBER(phys_base): 130023424 -> 7c00000 gdb
> vmlinux.debuginfo GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software
> Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> GETBUF(328 -> 0)
> GETBUF(1500 -> 1)
>
> WARNING: kernel relocated [824MB]: patching 109676 gdb minimal_symbol
> values
>
> FREEBUF(1)
> FREEBUF(0)
> <readmem: ffffffffb70df958, KVADDR, "page_offset_base", 8, (FOE|Q),
> d8b948>
> <read_diskdump: addr: ffffffffb70df958 paddr: 3ecdf958 cnt: 8>
> read_diskdump: paddr/pfn: 3ecdf958/3ecdf -> cache physical page:
> 3ecdf000
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(1024 -> 0)
> <readmem: ffffffffb7f0c5e0, KVADDR, "possible", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0c5e0 paddr: 3fb0c5e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c5e0/3fb0c
> crash: page excluded: kernel virtual address: ffffffffb7f0c5e0 type:
"possible"
> WARNING: cannot read cpu_possible_map
> <readmem: ffffffffb7f0bde0, KVADDR, "present", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0bde0 paddr: 3fb0bde0 cnt: 544>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0bde0/3fb0b
> crash: page excluded: kernel virtual address: ffffffffb7f0bde0 type:
"present"
> WARNING: cannot read cpu_present_map
> <readmem: ffffffffb7f0c1e0, KVADDR, "online", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0c1e0 paddr: 3fb0c1e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c1e0/3fb0c
> crash: page excluded: kernel virtual address: ffffffffb7f0c1e0 type:
"online"
> WARNING: cannot read cpu_online_map
> <readmem: ffffffffb7f0b9e0, KVADDR, "active", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0b9e0 paddr: 3fb0b9e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0b9e0/3fb0b
> crash: page excluded: kernel virtual address: ffffffffb7f0b9e0 type:
"active"
> WARNING: cannot read cpu_active_map
> FREEBUF(0)
> <readmem: ffffffffb731a5b0, KVADDR, "pv_init_ops", 8, (ROE),
> 7ffd3655d640>
> <read_diskdump: addr: ffffffffb731a5b0 paddr: 3ef1a5b0 cnt: 8>
> read_diskdump: paddr/pfn: 3ef1a5b0/3ef1a -> cache physical page:
> 3ef1a000
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(328 -> 0)
> FREEBUF(0)
> <readmem: ffffffffb9bad1d0, KVADDR, "shadow_timekeeper xtime_sec", 8,
> (ROE), 7ffd3655d5f0>
> <read_diskdump: addr: ffffffffb9bad1d0 paddr: 417ad1d0 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 417ad1d0/417ad
> crash: page excluded: kernel virtual address: ffffffffb9bad1d0 type:
"shadow_timekeeper xtime_sec"
> xtime timespec.tv_sec: 900f27: Mon Apr 20 07:31:03 1970
<readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390,
(ROE), d71b7c>
> <read_diskdump: addr: ffffffffb7221d64 paddr: 3ee21d64
cnt: 390>
> read_diskdump: paddr/pfn: 3ee21d64/3ee21 -> cache physical page:
> 3ee21000
> utsname:
> sysname:
> nodename:
> release:
> version:
> machine:
> domainname:
> base kernel version: 0.0.0
> <readmem: ffffffffb6a016c0, KVADDR, "accessible check", 8, (ROE|Q),
> 7ffd3655af30>
> <read_diskdump: addr: ffffffffb6a016c0 paddr: 3e6016c0 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3e6016c0/3e601
> crash: page excluded: kernel virtual address: ffffffffb6a016c0 type:
"accessible check"
> crash: vmlinux.debuginfo and vmcore do not match!
>
> Usage:
>
> crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
> crash [OPTION]... [NAMELIST] (live system form)
>
> Enter "crash -h" for details.
>
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
7:04 a.m.
New subject: EXT: Re: EXT: Re: crash start in CentOS 8
Hello all,
I've upgraded my Centos 8 VM to the latest kernel (4.18.0-193.6.3.el8_2.x86_64).
I have then installed the kernel-debug packages (and dependencies), as suggested below.
Now, I have following kernels in /boot:
- vmlinuz-4.18.0-193.6.3.el8_2.x86_64
- vmlinuz-4.18.0-193.6.3.el8_2.x86_64+debug
And a vmlinuz file in /lib/modules/4.18.0-193.6.3.el8_2.x86_64+debug
Then, I reboot the VM on this latest kernel and extracted the vmlinux files with the
extract-vmlinux script from a CentOS 4.18.0 kernel source tree.
A last, the 'echo c > /proc/sysrq-trigger' creates the /var/crash/vmcore file.
Exporting the vmcore, vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug and vmlinux (from
/lib/modules) on another CentOS machine and executing:
/usr/bin/crash -d 15 vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug vmcore
returns:
crash: vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug: no .gnu_debuglink section
Note: same result when using the exported vmlinux from /lib/modules directory.
Still not driving me nut... but not far away...
To be continued...
Thanks.
Best regards,
Patrick
-----Message d'origine-----
De : crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] De la part
de Kazuhito Hagio
Envoyé : jeudi 9 juillet 2020 07:00
À : Discussion list for crash utility usage, maintenance and development
<crash-utility(a)redhat.com>
Objet : EXT: Re: [Crash-utility] EXT: Re: crash start in CentOS 8
** External email - Please consider with caution **
Hi Patrick,
OSRELEASE=4.18.0-147.el8.x86_64
Hmm, I think you might boot the virtual machine with the original (unmodified) CentOS 8
kernel, because the address of init_uts_ns looks to be its one:
SYMBOL(init_uts_ns)=ffffffffb5a12540
KERNELOFFSET=33800000
I happen to have a CentOS 8 machine..
# grep 'D init_uts_ns' /boot/System.map-4.18.0-147.el8.x86_64
ffffffff82212540 D init_uts_ns
# printf "%16llx\n" $(( 0xffffffff82212540 + 0x33800000 ))
ffffffffb5a12540
while your vmcore.debuginfo has the different address:
<readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390,
(ROE), d71b7c>
So, which kernel do you actually want to use on the virtual machine?
(1) The original CentOS 8 kernel
(2) The kernel you built
If it's (1), you can just install the kernel-debuginfo package corresponding to the
original CentOS 8 kernel into the machine on which you want to analyze vmcores and
execute:
# crash /usr/lib/debug/lib/modules/4.18.0-147.el8.x86_64/vmlinux vmcore
If it's (2), you need to boot the VM with the kernel and make a crash.
Could you check this first?
Thanks,
Kazu
On Mon, Jul 6, 2020 at 10:03 PM Agrain Patrick <patrick.agrain(a)al-enterprise.com>
wrote:
>
> Hello all,
>
> Back on this topic:
> I compiled a kernel with debuginfo on the Virtual Machine with CentOS8.
> I made a crash on this same VM.
> I transferred the vmcore and vmlinux.debuginfo files on another CentOS8 baremetal
machine.
>
> The command '/usr/bin/crash -d 15 vmlinux.debuginfo vmcore' produces
following logs, but still failing to get the crash> prompt.
> Still KASLR ?
>
> Thanks in advance for any information.
> Best regards,
> Patrick Agrain
>
> -- Logs:
> crash 7.2.6-2.el8
> Copyright (C) 2002-2019 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C)
> 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012
> Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002,
> 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License, and you are welcome to change it and/or distribute copies of
> it under certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
details.
>
> compressed kdump: header->utsname.machine: x86_64 compressed kdump:
> memory bitmap offset: 2000
> diskdump_data:
> filename: vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 3
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: 212b5d0
> signature: "KDUMP "
> header_version: 6
> utsname:
> sysname: Linux
> nodename: localhost.localdomain
> release: 4.18.0-147.el8.x86_64
> version: #1 SMP Wed Dec 4 21:51:45 UTC 2019
> machine: x86_64
> domainname: (none)
> timestamp:
> tv_sec: 5ee72d23
> tv_usec: 0
> status: 2 (DUMP_DH_COMPRESSED_LZO)
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 32
> max_mapnr: 524288
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1
> tasks[nr_cpus]: 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 212c5e0
> phys_base: 7c00000
> dump_level: 31 (0x1f)
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
> split: 0
> start_pfn: (unused)
> end_pfn: (unused)
> offset_vmcoreinfo: 4580 (0x11e4)
> size_vmcoreinfo: 2025 (0x7e9)
OSRELEASE=4.18.0-147.el8.x86_64
>
PAGESIZE=4096
> SYMBOL(init_uts_ns)=ffffffffb5a12540
> SYMBOL(node_online_map)=ffffffffb5c06d60
> SYMBOL(swapper_pg_dir)=ffffffffb5a0a000
> SYMBOL(_stext)=ffffffffb4800000
> SYMBOL(vmap_area_list)=ffffffffb5ad2a90
> SYMBOL(mem_section)=ffff89df7ffd2000
> LENGTH(mem_section)=2048
> SIZE(mem_section)=16
> OFFSET(mem_section.section_mem_map)=0
> SIZE(page)=64
> SIZE(pglist_data)=171968
> SIZE(zone)=1472
> SIZE(free_area)=88
> SIZE(list_head)=16
> SIZE(nodemask_t)=128
> OFFSET(page.flags)=0
> OFFSET(page._refcount)=52
> OFFSET(page.mapping)=24
> OFFSET(page.lru)=8
> OFFSET(page._mapcount)=48
> OFFSET(page.private)=40
> OFFSET(page.compound_dtor)=16
> OFFSET(page.compound_order)=17
> OFFSET(page.compound_head)=8
> OFFSET(pglist_data.node_zones)=0
> OFFSET(pglist_data.nr_zones)=171232
> OFFSET(pglist_data.node_start_pfn)=171240
> OFFSET(pglist_data.node_spanned_pages)=171256
> OFFSET(pglist_data.node_id)=171264
> OFFSET(zone.free_area)=192
> OFFSET(zone.vm_stat)=1296
> OFFSET(zone.spanned_pages)=112
> OFFSET(free_area.free_list)=0
> OFFSET(list_head.next)=0
> OFFSET(list_head.prev)=8
> OFFSET(vmap_area.va_start)=0
> OFFSET(vmap_area.list)=48
> LENGTH(zone.free_area)=11
> SYMBOL(log_buf)=ffffffffb5a5a280
> SYMBOL(log_buf_len)=ffffffffb5a5a27c
> SYMBOL(log_first_idx)=ffffffffb633a770
> SYMBOL(clear_idx)=ffffffffb633a744
> SYMBOL(log_next_idx)=ffffffffb633a760
> SIZE(printk_log)=16
> OFFSET(printk_log.ts_nsec)=0
> OFFSET(printk_log.len)=8
> OFFSET(printk_log.text_len)=10
> OFFSET(printk_log.dict_len)=12
> LENGTH(free_area.free_list)=5
> NUMBER(NR_FREE_PAGES)=0
> NUMBER(PG_lru)=5
> NUMBER(PG_private)=12
> NUMBER(PG_swapcache)=9
> NUMBER(PG_swapbacked)=18
> NUMBER(PG_slab)=8
> NUMBER(PG_hwpoison)=22
> NUMBER(PG_head_mask)=32768
> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
> NUMBER(HUGETLB_PAGE_DTOR)=2
> NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
> NUMBER(phys_base)=130023424
> SYMBOL(init_top_pgt)=ffffffffb5a0a000
> NUMBER(pgtable_l5_enabled)=0
> SYMBOL(node_data)=ffffffffb5c02580
> LENGTH(node_data)=1024
> KERNELOFFSET=33800000
> NUMBER(KERNEL_IMAGE_SIZE)=1073741824
> NUMBER(sme_mask)=0
> CRASHTIME=1592208675
> offset_note: 4200 (0x1068)
> size_note: 2408 (0x968)
> notes_buf: 2129460
> num_vmcoredd_notes: 0
> num_prstatus_notes: 1
> notes[0]: 2129460 (NT_PRSTATUS)
> si.signo: 0 si.code: 0 si.errno: 0
> cursig: 0 sigpend: 0 sighold: 0
> pid: 2846 ppid: 0 pgrp: 0 sid:0
> utime: 0.000000 stime: 0.000000
> cutime: 0.000000 cstime: 0.000000
> ORIG_RAX: ffffffffffffffff fpvalid: 0
> R15: 0000000000000000 R14: 00005616bb1208f0
> R13: ffffffffb5b36fc0 R12: 0000000000000000
> RBP: 0000000000000004 RBX: 0000000000000063
> R11: 6873617263206120 R10: 7265676769725420
> R9: 54203a2071527379 R8: 000000000000056a
> RAX: ffffffffb4d12d10 RCX: 0000000000000006
> RDX: 0000000000000000 RSI: 0000000000000086
> RDI: 0000000000000063 RIP: ffffffffb4d12d22
> RFLAGS: 0000000000010246 RSP: ffffb03903147e80
> FS_BASE: 00007f5d6e06f740
> GS_BASE: 0000000000000000
> CS: 0010 SS: 0018 DS: 0000
> ES: 0000 FS: 0000 GS: 0000
> snapshot_task: 0
> num_qemu_notes: 0
> NOTE offsets: 1068 (NT_PRSTATUS)
> offset_eraseinfo: 0 (0x0)
> size_eraseinfo: 0 (0x0)
> start_pfn_64: (unused)
> end_pfn_64: (unused)
> max_mapnr_64: 524288 (0x80000)
>
> data_offset: 22000
> block_size: 4096
> block_shift: 12
> bitmap: 212f190
> bitmap_len: 131072
> max_mapnr: 524288 (0x80000)
> dumpable_bitmap: 7f937ad93010
> byte: 0
> bit: 0
> compressed_page: 212d5f0
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 217f1d0
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21801d0
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21811d0
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21821d0
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21831d0
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21841d0
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21851d0
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21861d0
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21871d0
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21881d0
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21891d0
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218a1d0
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218b1d0
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218c1d0
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218d1d0
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218e1d0
> pg_hit_count: 0
>
> page_cache_buf: 217f1d0
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: 212a3b0
> readmem: read_diskdump()
> KASLR:
> _stext from vmlinux.debuginfo: ffffffff81000000
> _stext from vmcoreinfo: ffffffffb4800000
> relocate: 33800000 (824MB)
> crash: pv_init_ops exists: ARCH_PVOPS
> VMCOREINFO: NUMBER(phys_base): 130023424 -> 7c00000 gdb
> vmlinux.debuginfo GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software
> Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> GETBUF(328 -> 0)
> GETBUF(1500 -> 1)
>
> WARNING: kernel relocated [824MB]: patching 109676 gdb minimal_symbol
> values
>
> FREEBUF(1)
> FREEBUF(0)
> <readmem: ffffffffb70df958, KVADDR, "page_offset_base", 8, (FOE|Q),
> d8b948>
> <read_diskdump: addr: ffffffffb70df958 paddr: 3ecdf958 cnt: 8>
> read_diskdump: paddr/pfn: 3ecdf958/3ecdf -> cache physical page:
> 3ecdf000
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(1024 -> 0)
> <readmem: ffffffffb7f0c5e0, KVADDR, "possible", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0c5e0 paddr: 3fb0c5e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c5e0/3fb0c
> crash: page excluded: kernel virtual address: ffffffffb7f0c5e0 type:
"possible"
> WARNING: cannot read cpu_possible_map
> <readmem: ffffffffb7f0bde0, KVADDR, "present", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0bde0 paddr: 3fb0bde0 cnt: 544>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0bde0/3fb0b
> crash: page excluded: kernel virtual address: ffffffffb7f0bde0 type:
"present"
> WARNING: cannot read cpu_present_map
> <readmem: ffffffffb7f0c1e0, KVADDR, "online", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0c1e0 paddr: 3fb0c1e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c1e0/3fb0c
> crash: page excluded: kernel virtual address: ffffffffb7f0c1e0 type:
"online"
> WARNING: cannot read cpu_online_map
> <readmem: ffffffffb7f0b9e0, KVADDR, "active", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0b9e0 paddr: 3fb0b9e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0b9e0/3fb0b
> crash: page excluded: kernel virtual address: ffffffffb7f0b9e0 type:
"active"
> WARNING: cannot read cpu_active_map
> FREEBUF(0)
> <readmem: ffffffffb731a5b0, KVADDR, "pv_init_ops", 8, (ROE),
> 7ffd3655d640>
> <read_diskdump: addr: ffffffffb731a5b0 paddr: 3ef1a5b0 cnt: 8>
> read_diskdump: paddr/pfn: 3ef1a5b0/3ef1a -> cache physical page:
> 3ef1a000
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(328 -> 0)
> FREEBUF(0)
> <readmem: ffffffffb9bad1d0, KVADDR, "shadow_timekeeper xtime_sec", 8,
> (ROE), 7ffd3655d5f0>
> <read_diskdump: addr: ffffffffb9bad1d0 paddr: 417ad1d0 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 417ad1d0/417ad
> crash: page excluded: kernel virtual address: ffffffffb9bad1d0 type:
"shadow_timekeeper xtime_sec"
> xtime timespec.tv_sec: 900f27: Mon Apr 20 07:31:03 1970
<readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390,
(ROE), d71b7c>
> <read_diskdump: addr: ffffffffb7221d64 paddr: 3ee21d64
cnt: 390>
> read_diskdump: paddr/pfn: 3ee21d64/3ee21 -> cache physical page:
> 3ee21000
> utsname:
> sysname:
> nodename:
> release:
> version:
> machine:
> domainname:
> base kernel version: 0.0.0
> <readmem: ffffffffb6a016c0, KVADDR, "accessible check", 8, (ROE|Q),
> 7ffd3655af30>
> <read_diskdump: addr: ffffffffb6a016c0 paddr: 3e6016c0 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3e6016c0/3e601
> crash: page excluded: kernel virtual address: ffffffffb6a016c0 type:
"accessible check"
> crash: vmlinux.debuginfo and vmcore do not match!
>
> Usage:
>
> crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
> crash [OPTION]... [NAMELIST] (live system form)
>
> Enter "crash -h" for details.
>
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
7:49 a.m.
New subject: EXT: Re: EXT: Re: crash start in CentOS 8
Hello,
On 7/20/20 8:04 AM, Agrain Patrick wrote:
Hello all,
I've upgraded my Centos 8 VM to the latest kernel (4.18.0-193.6.3.el8_2.x86_64).
I have then installed the kernel-debug packages (and dependencies), as suggested below.
Now, I have following kernels in /boot:
- vmlinuz-4.18.0-193.6.3.el8_2.x86_64
- vmlinuz-4.18.0-193.6.3.el8_2.x86_64+debug
This is a kernel-debug pacakge, not kernel-debuginfo package.
I think you can get the kernel-debuginfo here:
http://debuginfo.centos.org/8/x86_64/Packages/
Probably you need to install kernel-debuginfo-common-x86_64 package as well.
After the debuginfo package is installed, you'll see the vmlinux as follows:
/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux
Then:
crash /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux vmcore
Could you try it?
Thanks,
Masa
And a vmlinuz file in /lib/modules/4.18.0-193.6.3.el8_2.x86_64+debug
Then, I reboot the VM on this latest kernel and extracted the vmlinux files with the
extract-vmlinux script from a CentOS 4.18.0 kernel source tree.
A last, the 'echo c > /proc/sysrq-trigger' creates the /var/crash/vmcore
file.
Exporting the vmcore, vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug and vmlinux (from
/lib/modules) on another CentOS machine and executing:
/usr/bin/crash -d 15 vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug vmcore
returns:
crash: vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug: no .gnu_debuglink section
Note: same result when using the exported vmlinux from /lib/modules directory.
Still not driving me nut... but not far away...
To be continued...
Thanks.
Best regards,
Patrick
-----Message d'origine-----
De : crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] De la
part de Kazuhito Hagio
Envoyé : jeudi 9 juillet 2020 07:00
À : Discussion list for crash utility usage, maintenance and development
<crash-utility(a)redhat.com>
Objet : EXT: Re: [Crash-utility] EXT: Re: crash start in CentOS 8
** External email - Please consider with caution **
Hi Patrick,
> OSRELEASE=4.18.0-147.el8.x86_64
Hmm, I think you might boot the virtual machine with the original (unmodified) CentOS 8
kernel, because the address of init_uts_ns looks to be its one:
> SYMBOL(init_uts_ns)=ffffffffb5a12540
> KERNELOFFSET=33800000
I happen to have a CentOS 8 machine..
# grep 'D init_uts_ns' /boot/System.map-4.18.0-147.el8.x86_64
ffffffff82212540 D init_uts_ns
# printf "%16llx\n" $(( 0xffffffff82212540 + 0x33800000 ))
ffffffffb5a12540
while your vmcore.debuginfo has the different address:
> <readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE),
d71b7c>
So, which kernel do you actually want to use on the virtual machine?
(1) The original CentOS 8 kernel
(2) The kernel you built
If it's (1), you can just install the kernel-debuginfo package corresponding to the
original CentOS 8 kernel into the machine on which you want to analyze vmcores and
execute:
# crash /usr/lib/debug/lib/modules/4.18.0-147.el8.x86_64/vmlinux vmcore
If it's (2), you need to boot the VM with the kernel and make a crash.
Could you check this first?
Thanks,
Kazu
On Mon, Jul 6, 2020 at 10:03 PM Agrain Patrick <patrick.agrain(a)al-enterprise.com>
wrote:
>
> Hello all,
>
> Back on this topic:
> I compiled a kernel with debuginfo on the Virtual Machine with CentOS8.
> I made a crash on this same VM.
> I transferred the vmcore and vmlinux.debuginfo files on another CentOS8 baremetal
machine.
>
> The command '/usr/bin/crash -d 15 vmlinux.debuginfo vmcore' produces
following logs, but still failing to get the crash> prompt.
> Still KASLR ?
>
> Thanks in advance for any information.
> Best regards,
> Patrick Agrain
>
> -- Logs:
> crash 7.2.6-2.el8
> Copyright (C) 2002-2019 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C)
> 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012
> Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002,
> 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License, and you are welcome to change it and/or distribute copies of
> it under certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
details.
>
> compressed kdump: header->utsname.machine: x86_64 compressed kdump:
> memory bitmap offset: 2000
> diskdump_data:
> filename: vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 3
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: 212b5d0
> signature: "KDUMP "
> header_version: 6
> utsname:
> sysname: Linux
> nodename: localhost.localdomain
> release: 4.18.0-147.el8.x86_64
> version: #1 SMP Wed Dec 4 21:51:45 UTC 2019
> machine: x86_64
> domainname: (none)
> timestamp:
> tv_sec: 5ee72d23
> tv_usec: 0
> status: 2 (DUMP_DH_COMPRESSED_LZO)
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 32
> max_mapnr: 524288
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1
> tasks[nr_cpus]: 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 212c5e0
> phys_base: 7c00000
> dump_level: 31 (0x1f)
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
> split: 0
> start_pfn: (unused)
> end_pfn: (unused)
> offset_vmcoreinfo: 4580 (0x11e4)
> size_vmcoreinfo: 2025 (0x7e9)
> OSRELEASE=4.18.0-147.el8.x86_64
> PAGESIZE=4096
> SYMBOL(init_uts_ns)=ffffffffb5a12540
> SYMBOL(node_online_map)=ffffffffb5c06d60
> SYMBOL(swapper_pg_dir)=ffffffffb5a0a000
> SYMBOL(_stext)=ffffffffb4800000
> SYMBOL(vmap_area_list)=ffffffffb5ad2a90
> SYMBOL(mem_section)=ffff89df7ffd2000
> LENGTH(mem_section)=2048
> SIZE(mem_section)=16
> OFFSET(mem_section.section_mem_map)=0
> SIZE(page)=64
> SIZE(pglist_data)=171968
> SIZE(zone)=1472
> SIZE(free_area)=88
> SIZE(list_head)=16
> SIZE(nodemask_t)=128
> OFFSET(page.flags)=0
> OFFSET(page._refcount)=52
> OFFSET(page.mapping)=24
> OFFSET(page.lru)=8
> OFFSET(page._mapcount)=48
> OFFSET(page.private)=40
> OFFSET(page.compound_dtor)=16
> OFFSET(page.compound_order)=17
> OFFSET(page.compound_head)=8
> OFFSET(pglist_data.node_zones)=0
> OFFSET(pglist_data.nr_zones)=171232
> OFFSET(pglist_data.node_start_pfn)=171240
> OFFSET(pglist_data.node_spanned_pages)=171256
> OFFSET(pglist_data.node_id)=171264
> OFFSET(zone.free_area)=192
> OFFSET(zone.vm_stat)=1296
> OFFSET(zone.spanned_pages)=112
> OFFSET(free_area.free_list)=0
> OFFSET(list_head.next)=0
> OFFSET(list_head.prev)=8
> OFFSET(vmap_area.va_start)=0
> OFFSET(vmap_area.list)=48
> LENGTH(zone.free_area)=11
> SYMBOL(log_buf)=ffffffffb5a5a280
> SYMBOL(log_buf_len)=ffffffffb5a5a27c
> SYMBOL(log_first_idx)=ffffffffb633a770
> SYMBOL(clear_idx)=ffffffffb633a744
> SYMBOL(log_next_idx)=ffffffffb633a760
> SIZE(printk_log)=16
> OFFSET(printk_log.ts_nsec)=0
> OFFSET(printk_log.len)=8
> OFFSET(printk_log.text_len)=10
> OFFSET(printk_log.dict_len)=12
> LENGTH(free_area.free_list)=5
> NUMBER(NR_FREE_PAGES)=0
> NUMBER(PG_lru)=5
> NUMBER(PG_private)=12
> NUMBER(PG_swapcache)=9
> NUMBER(PG_swapbacked)=18
> NUMBER(PG_slab)=8
> NUMBER(PG_hwpoison)=22
> NUMBER(PG_head_mask)=32768
> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
> NUMBER(HUGETLB_PAGE_DTOR)=2
> NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
> NUMBER(phys_base)=130023424
> SYMBOL(init_top_pgt)=ffffffffb5a0a000
> NUMBER(pgtable_l5_enabled)=0
> SYMBOL(node_data)=ffffffffb5c02580
> LENGTH(node_data)=1024
> KERNELOFFSET=33800000
> NUMBER(KERNEL_IMAGE_SIZE)=1073741824
> NUMBER(sme_mask)=0
> CRASHTIME=1592208675
> offset_note: 4200 (0x1068)
> size_note: 2408 (0x968)
> notes_buf: 2129460
> num_vmcoredd_notes: 0
> num_prstatus_notes: 1
> notes[0]: 2129460 (NT_PRSTATUS)
> si.signo: 0 si.code: 0 si.errno: 0
> cursig: 0 sigpend: 0 sighold: 0
> pid: 2846 ppid: 0 pgrp: 0 sid:0
> utime: 0.000000 stime: 0.000000
> cutime: 0.000000 cstime: 0.000000
> ORIG_RAX: ffffffffffffffff fpvalid: 0
> R15: 0000000000000000 R14: 00005616bb1208f0
> R13: ffffffffb5b36fc0 R12: 0000000000000000
> RBP: 0000000000000004 RBX: 0000000000000063
> R11: 6873617263206120 R10: 7265676769725420
> R9: 54203a2071527379 R8: 000000000000056a
> RAX: ffffffffb4d12d10 RCX: 0000000000000006
> RDX: 0000000000000000 RSI: 0000000000000086
> RDI: 0000000000000063 RIP: ffffffffb4d12d22
> RFLAGS: 0000000000010246 RSP: ffffb03903147e80
> FS_BASE: 00007f5d6e06f740
> GS_BASE: 0000000000000000
> CS: 0010 SS: 0018 DS: 0000
> ES: 0000 FS: 0000 GS: 0000
> snapshot_task: 0
> num_qemu_notes: 0
> NOTE offsets: 1068 (NT_PRSTATUS)
> offset_eraseinfo: 0 (0x0)
> size_eraseinfo: 0 (0x0)
> start_pfn_64: (unused)
> end_pfn_64: (unused)
> max_mapnr_64: 524288 (0x80000)
>
> data_offset: 22000
> block_size: 4096
> block_shift: 12
> bitmap: 212f190
> bitmap_len: 131072
> max_mapnr: 524288 (0x80000)
> dumpable_bitmap: 7f937ad93010
> byte: 0
> bit: 0
> compressed_page: 212d5f0
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 217f1d0
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21801d0
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21811d0
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21821d0
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21831d0
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21841d0
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21851d0
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21861d0
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21871d0
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21881d0
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21891d0
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218a1d0
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218b1d0
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218c1d0
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218d1d0
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218e1d0
> pg_hit_count: 0
>
> page_cache_buf: 217f1d0
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: 212a3b0
> readmem: read_diskdump()
> KASLR:
> _stext from vmlinux.debuginfo: ffffffff81000000
> _stext from vmcoreinfo: ffffffffb4800000
> relocate: 33800000 (824MB)
> crash: pv_init_ops exists: ARCH_PVOPS
> VMCOREINFO: NUMBER(phys_base): 130023424 -> 7c00000 gdb
> vmlinux.debuginfo GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software
> Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> GETBUF(328 -> 0)
> GETBUF(1500 -> 1)
>
> WARNING: kernel relocated [824MB]: patching 109676 gdb minimal_symbol
> values
>
> FREEBUF(1)
> FREEBUF(0)
> <readmem: ffffffffb70df958, KVADDR, "page_offset_base", 8, (FOE|Q),
> d8b948>
> <read_diskdump: addr: ffffffffb70df958 paddr: 3ecdf958 cnt: 8>
> read_diskdump: paddr/pfn: 3ecdf958/3ecdf -> cache physical page:
> 3ecdf000
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(1024 -> 0)
> <readmem: ffffffffb7f0c5e0, KVADDR, "possible", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0c5e0 paddr: 3fb0c5e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c5e0/3fb0c
> crash: page excluded: kernel virtual address: ffffffffb7f0c5e0 type:
"possible"
> WARNING: cannot read cpu_possible_map
> <readmem: ffffffffb7f0bde0, KVADDR, "present", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0bde0 paddr: 3fb0bde0 cnt: 544>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0bde0/3fb0b
> crash: page excluded: kernel virtual address: ffffffffb7f0bde0 type:
"present"
> WARNING: cannot read cpu_present_map
> <readmem: ffffffffb7f0c1e0, KVADDR, "online", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0c1e0 paddr: 3fb0c1e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c1e0/3fb0c
> crash: page excluded: kernel virtual address: ffffffffb7f0c1e0 type:
"online"
> WARNING: cannot read cpu_online_map
> <readmem: ffffffffb7f0b9e0, KVADDR, "active", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0b9e0 paddr: 3fb0b9e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0b9e0/3fb0b
> crash: page excluded: kernel virtual address: ffffffffb7f0b9e0 type:
"active"
> WARNING: cannot read cpu_active_map
> FREEBUF(0)
> <readmem: ffffffffb731a5b0, KVADDR, "pv_init_ops", 8, (ROE),
> 7ffd3655d640>
> <read_diskdump: addr: ffffffffb731a5b0 paddr: 3ef1a5b0 cnt: 8>
> read_diskdump: paddr/pfn: 3ef1a5b0/3ef1a -> cache physical page:
> 3ef1a000
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(328 -> 0)
> FREEBUF(0)
> <readmem: ffffffffb9bad1d0, KVADDR, "shadow_timekeeper xtime_sec", 8,
> (ROE), 7ffd3655d5f0>
> <read_diskdump: addr: ffffffffb9bad1d0 paddr: 417ad1d0 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 417ad1d0/417ad
> crash: page excluded: kernel virtual address: ffffffffb9bad1d0 type:
"shadow_timekeeper xtime_sec"
> xtime timespec.tv_sec: 900f27: Mon Apr 20 07:31:03 1970
> <readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE),
d71b7c>
> <read_diskdump: addr: ffffffffb7221d64 paddr: 3ee21d64 cnt: 390>
> read_diskdump: paddr/pfn: 3ee21d64/3ee21 -> cache physical page:
> 3ee21000
> utsname:
> sysname:
> nodename:
> release:
> version:
> machine:
> domainname:
> base kernel version: 0.0.0
> <readmem: ffffffffb6a016c0, KVADDR, "accessible check", 8, (ROE|Q),
> 7ffd3655af30>
> <read_diskdump: addr: ffffffffb6a016c0 paddr: 3e6016c0 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3e6016c0/3e601
> crash: page excluded: kernel virtual address: ffffffffb6a016c0 type:
"accessible check"
> crash: vmlinux.debuginfo and vmcore do not match!
>
> Usage:
>
> crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
> crash [OPTION]... [NAMELIST] (live system form)
>
> Enter "crash -h" for details.
>
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
7:52 a.m.
New subject: EXT: Re: EXT: Re: EXT: Re: crash start in CentOS 8
Hello,
Yes, I just saw that.
By default, the debuginfo repo is not enabled on CentOS 8, therefore the kernel-debuginfo
packages were not available.
Installation is in progress.
Will keep you in touch.
Thanks for pinting that.
Best regards,
Patrick Agrain
-----Message d'origine-----
De : Masayoshi Mizuma [mailto:msys.mizuma@gmail.com]
Envoyé : lundi 20 juillet 2020 14:50
À : Agrain Patrick <patrick.agrain(a)al-enterprise.com>
Cc : Discussion list for crash utility usage, maintenance and development
<crash-utility(a)redhat.com>
Objet : EXT: Re: [Crash-utility] EXT: Re: EXT: Re: crash start in CentOS 8
** External email - Please consider with caution **
Hello,
On 7/20/20 8:04 AM, Agrain Patrick wrote:
Hello all,
I've upgraded my Centos 8 VM to the latest kernel (4.18.0-193.6.3.el8_2.x86_64).
I have then installed the kernel-debug packages (and dependencies), as suggested below.
Now, I have following kernels in /boot:
- vmlinuz-4.18.0-193.6.3.el8_2.x86_64
- vmlinuz-4.18.0-193.6.3.el8_2.x86_64+debug
This is a kernel-debug pacakge, not kernel-debuginfo package.
I think you can get the kernel-debuginfo here:
http://debuginfo.centos.org/8/x86_64/Packages/
Probably you need to install kernel-debuginfo-common-x86_64 package as well.
After the debuginfo package is installed, you'll see the vmlinux as follows:
/usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux
Then:
crash /usr/lib/debug/lib/modules/4.18.0-193.6.3.el8_2.x86_64/vmlinux vmcore
Could you try it?
Thanks,
Masa
And a vmlinuz file in /lib/modules/4.18.0-193.6.3.el8_2.x86_64+debug
Then, I reboot the VM on this latest kernel and extracted the vmlinux files with the
extract-vmlinux script from a CentOS 4.18.0 kernel source tree.
A last, the 'echo c > /proc/sysrq-trigger' creates the /var/crash/vmcore
file.
Exporting the vmcore, vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug and vmlinux (from
/lib/modules) on another CentOS machine and executing:
/usr/bin/crash -d 15 vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug vmcore
returns:
crash: vmlinux-4.18.0-193.6.3.el8_2.x86_64+debug: no .gnu_debuglink
section
Note: same result when using the exported vmlinux from /lib/modules directory.
Still not driving me nut... but not far away...
To be continued...
Thanks.
Best regards,
Patrick
-----Message d'origine-----
De : crash-utility-bounces(a)redhat.com
[mailto:crash-utility-bounces@redhat.com] De la part de Kazuhito Hagio
Envoyé : jeudi 9 juillet 2020 07:00 À : Discussion list for crash
utility usage, maintenance and development <crash-utility(a)redhat.com>
Objet : EXT: Re: [Crash-utility] EXT: Re: crash start in CentOS 8
** External email - Please consider with caution **
Hi Patrick,
> OSRELEASE=4.18.0-147.el8.x86_64
Hmm, I think you might boot the virtual machine with the original (unmodified) CentOS 8
kernel, because the address of init_uts_ns looks to be its one:
> SYMBOL(init_uts_ns)=ffffffffb5a12540
> KERNELOFFSET=33800000
I happen to have a CentOS 8 machine..
# grep 'D init_uts_ns' /boot/System.map-4.18.0-147.el8.x86_64
ffffffff82212540 D init_uts_ns
# printf "%16llx\n" $(( 0xffffffff82212540 + 0x33800000 ))
ffffffffb5a12540
while your vmcore.debuginfo has the different address:
> <readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE),
> d71b7c>
So, which kernel do you actually want to use on the virtual machine?
(1) The original CentOS 8 kernel
(2) The kernel you built
If it's (1), you can just install the kernel-debuginfo package corresponding to the
original CentOS 8 kernel into the machine on which you want to analyze vmcores and
execute:
# crash /usr/lib/debug/lib/modules/4.18.0-147.el8.x86_64/vmlinux
vmcore
If it's (2), you need to boot the VM with the kernel and make a crash.
Could you check this first?
Thanks,
Kazu
On Mon, Jul 6, 2020 at 10:03 PM Agrain Patrick <patrick.agrain(a)al-enterprise.com>
wrote:
>
> Hello all,
>
> Back on this topic:
> I compiled a kernel with debuginfo on the Virtual Machine with CentOS8.
> I made a crash on this same VM.
> I transferred the vmcore and vmlinux.debuginfo files on another CentOS8 baremetal
machine.
>
> The command '/usr/bin/crash -d 15 vmlinux.debuginfo vmcore' produces
following logs, but still failing to get the crash> prompt.
> Still KASLR ?
>
> Thanks in advance for any information.
> Best regards,
> Patrick Agrain
>
> -- Logs:
> crash 7.2.6-2.el8
> Copyright (C) 2002-2019 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C)
> 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012
> Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002,
> 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License, and you are welcome to change it and/or distribute copies of
> it under certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
details.
>
> compressed kdump: header->utsname.machine: x86_64 compressed kdump:
> memory bitmap offset: 2000
> diskdump_data:
> filename: vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 3
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: 212b5d0
> signature: "KDUMP "
> header_version: 6
> utsname:
> sysname: Linux
> nodename: localhost.localdomain
> release: 4.18.0-147.el8.x86_64
> version: #1 SMP Wed Dec 4 21:51:45 UTC 2019
> machine: x86_64
> domainname: (none)
> timestamp:
> tv_sec: 5ee72d23
> tv_usec: 0
> status: 2 (DUMP_DH_COMPRESSED_LZO)
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 32
> max_mapnr: 524288
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1
> tasks[nr_cpus]: 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 212c5e0
> phys_base: 7c00000
> dump_level: 31 (0x1f)
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
> split: 0
> start_pfn: (unused)
> end_pfn: (unused)
> offset_vmcoreinfo: 4580 (0x11e4)
> size_vmcoreinfo: 2025 (0x7e9)
> OSRELEASE=4.18.0-147.el8.x86_64
> PAGESIZE=4096
> SYMBOL(init_uts_ns)=ffffffffb5a12540
> SYMBOL(node_online_map)=ffffffffb5c06d60
> SYMBOL(swapper_pg_dir)=ffffffffb5a0a000
> SYMBOL(_stext)=ffffffffb4800000
> SYMBOL(vmap_area_list)=ffffffffb5ad2a90
> SYMBOL(mem_section)=ffff89df7ffd2000
> LENGTH(mem_section)=2048
> SIZE(mem_section)=16
> OFFSET(mem_section.section_mem_map)=0
> SIZE(page)=64
> SIZE(pglist_data)=171968
> SIZE(zone)=1472
> SIZE(free_area)=88
> SIZE(list_head)=16
> SIZE(nodemask_t)=128
> OFFSET(page.flags)=0
> OFFSET(page._refcount)=52
> OFFSET(page.mapping)=24
> OFFSET(page.lru)=8
> OFFSET(page._mapcount)=48
> OFFSET(page.private)=40
> OFFSET(page.compound_dtor)=16
> OFFSET(page.compound_order)=17
> OFFSET(page.compound_head)=8
> OFFSET(pglist_data.node_zones)=0
> OFFSET(pglist_data.nr_zones)=171232
> OFFSET(pglist_data.node_start_pfn)=171240
> OFFSET(pglist_data.node_spanned_pages)=171256
> OFFSET(pglist_data.node_id)=171264
> OFFSET(zone.free_area)=192
> OFFSET(zone.vm_stat)=1296
> OFFSET(zone.spanned_pages)=112
> OFFSET(free_area.free_list)=0
> OFFSET(list_head.next)=0
> OFFSET(list_head.prev)=8
> OFFSET(vmap_area.va_start)=0
> OFFSET(vmap_area.list)=48
> LENGTH(zone.free_area)=11
> SYMBOL(log_buf)=ffffffffb5a5a280
> SYMBOL(log_buf_len)=ffffffffb5a5a27c
> SYMBOL(log_first_idx)=ffffffffb633a770
> SYMBOL(clear_idx)=ffffffffb633a744
> SYMBOL(log_next_idx)=ffffffffb633a760
> SIZE(printk_log)=16
> OFFSET(printk_log.ts_nsec)=0
> OFFSET(printk_log.len)=8
> OFFSET(printk_log.text_len)=10
> OFFSET(printk_log.dict_len)=12
> LENGTH(free_area.free_list)=5
> NUMBER(NR_FREE_PAGES)=0
> NUMBER(PG_lru)=5
> NUMBER(PG_private)=12
> NUMBER(PG_swapcache)=9
> NUMBER(PG_swapbacked)=18
> NUMBER(PG_slab)=8
> NUMBER(PG_hwpoison)=22
> NUMBER(PG_head_mask)=32768
> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
> NUMBER(HUGETLB_PAGE_DTOR)=2
> NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
> NUMBER(phys_base)=130023424
> SYMBOL(init_top_pgt)=ffffffffb5a0a000
> NUMBER(pgtable_l5_enabled)=0
> SYMBOL(node_data)=ffffffffb5c02580
> LENGTH(node_data)=1024
> KERNELOFFSET=33800000
> NUMBER(KERNEL_IMAGE_SIZE)=1073741824
> NUMBER(sme_mask)=0
> CRASHTIME=1592208675
> offset_note: 4200 (0x1068)
> size_note: 2408 (0x968)
> notes_buf: 2129460
> num_vmcoredd_notes: 0
> num_prstatus_notes: 1
> notes[0]: 2129460 (NT_PRSTATUS)
> si.signo: 0 si.code: 0 si.errno: 0
> cursig: 0 sigpend: 0 sighold: 0
> pid: 2846 ppid: 0 pgrp: 0 sid:0
> utime: 0.000000 stime: 0.000000
> cutime: 0.000000 cstime: 0.000000
> ORIG_RAX: ffffffffffffffff fpvalid: 0
> R15: 0000000000000000 R14: 00005616bb1208f0
> R13: ffffffffb5b36fc0 R12: 0000000000000000
> RBP: 0000000000000004 RBX: 0000000000000063
> R11: 6873617263206120 R10: 7265676769725420
> R9: 54203a2071527379 R8: 000000000000056a
> RAX: ffffffffb4d12d10 RCX: 0000000000000006
> RDX: 0000000000000000 RSI: 0000000000000086
> RDI: 0000000000000063 RIP: ffffffffb4d12d22
> RFLAGS: 0000000000010246 RSP: ffffb03903147e80
> FS_BASE: 00007f5d6e06f740
> GS_BASE: 0000000000000000
> CS: 0010 SS: 0018 DS: 0000
> ES: 0000 FS: 0000 GS: 0000
> snapshot_task: 0
> num_qemu_notes: 0
> NOTE offsets: 1068 (NT_PRSTATUS)
> offset_eraseinfo: 0 (0x0)
> size_eraseinfo: 0 (0x0)
> start_pfn_64: (unused)
> end_pfn_64: (unused)
> max_mapnr_64: 524288 (0x80000)
>
> data_offset: 22000
> block_size: 4096
> block_shift: 12
> bitmap: 212f190
> bitmap_len: 131072
> max_mapnr: 524288 (0x80000)
> dumpable_bitmap: 7f937ad93010
> byte: 0
> bit: 0
> compressed_page: 212d5f0
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 217f1d0
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21801d0
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21811d0
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21821d0
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21831d0
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21841d0
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21851d0
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21861d0
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21871d0
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21881d0
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21891d0
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218a1d0
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218b1d0
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218c1d0
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218d1d0
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218e1d0
> pg_hit_count: 0
>
> page_cache_buf: 217f1d0
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: 212a3b0
> readmem: read_diskdump()
> KASLR:
> _stext from vmlinux.debuginfo: ffffffff81000000
> _stext from vmcoreinfo: ffffffffb4800000
> relocate: 33800000 (824MB)
> crash: pv_init_ops exists: ARCH_PVOPS
> VMCOREINFO: NUMBER(phys_base): 130023424 -> 7c00000 gdb
> vmlinux.debuginfo GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software
> Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> GETBUF(328 -> 0)
> GETBUF(1500 -> 1)
>
> WARNING: kernel relocated [824MB]: patching 109676 gdb minimal_symbol
> values
>
> FREEBUF(1)
> FREEBUF(0)
> <readmem: ffffffffb70df958, KVADDR, "page_offset_base", 8, (FOE|Q),
> d8b948>
> <read_diskdump: addr: ffffffffb70df958 paddr: 3ecdf958 cnt: 8>
> read_diskdump: paddr/pfn: 3ecdf958/3ecdf -> cache physical page:
> 3ecdf000
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(1024 -> 0)
> <readmem: ffffffffb7f0c5e0, KVADDR, "possible", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0c5e0 paddr: 3fb0c5e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c5e0/3fb0c
> crash: page excluded: kernel virtual address: ffffffffb7f0c5e0 type:
"possible"
> WARNING: cannot read cpu_possible_map
> <readmem: ffffffffb7f0bde0, KVADDR, "present", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0bde0 paddr: 3fb0bde0 cnt: 544>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0bde0/3fb0b
> crash: page excluded: kernel virtual address: ffffffffb7f0bde0 type:
"present"
> WARNING: cannot read cpu_present_map
> <readmem: ffffffffb7f0c1e0, KVADDR, "online", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0c1e0 paddr: 3fb0c1e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c1e0/3fb0c
> crash: page excluded: kernel virtual address: ffffffffb7f0c1e0 type:
"online"
> WARNING: cannot read cpu_online_map
> <readmem: ffffffffb7f0b9e0, KVADDR, "active", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0b9e0 paddr: 3fb0b9e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0b9e0/3fb0b
> crash: page excluded: kernel virtual address: ffffffffb7f0b9e0 type:
"active"
> WARNING: cannot read cpu_active_map
> FREEBUF(0)
> <readmem: ffffffffb731a5b0, KVADDR, "pv_init_ops", 8, (ROE),
> 7ffd3655d640>
> <read_diskdump: addr: ffffffffb731a5b0 paddr: 3ef1a5b0 cnt: 8>
> read_diskdump: paddr/pfn: 3ef1a5b0/3ef1a -> cache physical page:
> 3ef1a000
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(328 -> 0)
> FREEBUF(0)
> <readmem: ffffffffb9bad1d0, KVADDR, "shadow_timekeeper xtime_sec", 8,
> (ROE), 7ffd3655d5f0>
> <read_diskdump: addr: ffffffffb9bad1d0 paddr: 417ad1d0 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 417ad1d0/417ad
> crash: page excluded: kernel virtual address: ffffffffb9bad1d0 type:
"shadow_timekeeper xtime_sec"
> xtime timespec.tv_sec: 900f27: Mon Apr 20 07:31:03 1970
> <readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE),
> d71b7c>
> <read_diskdump: addr: ffffffffb7221d64 paddr: 3ee21d64 cnt: 390>
> read_diskdump: paddr/pfn: 3ee21d64/3ee21 -> cache physical page:
> 3ee21000
> utsname:
> sysname:
> nodename:
> release:
> version:
> machine:
> domainname:
> base kernel version: 0.0.0
> <readmem: ffffffffb6a016c0, KVADDR, "accessible check", 8, (ROE|Q),
> 7ffd3655af30>
> <read_diskdump: addr: ffffffffb6a016c0 paddr: 3e6016c0 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3e6016c0/3e601
> crash: page excluded: kernel virtual address: ffffffffb6a016c0 type:
"accessible check"
> crash: vmlinux.debuginfo and vmcore do not match!
>
> Usage:
>
> crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
> crash [OPTION]... [NAMELIST] (live system form)
>
> Enter "crash -h" for details.
>
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
1440
days inactive
1473
days old
devel@lists.crash-utility.osci.io
7 comments
5 participants
participants (5)
-
Agrain Patrick -
HAGIO KAZUHITO(萩尾 一仁) -
Kazuhito Hagio -
Masayoshi Mizuma -
Santosh