10:03 a.m.
----- Original Message -----
----- Original Message -----
> Hi,
>
> I'm running vanilla Linux 4.14.91 and I cannot seem to obtain a
> working crash dump file from any kernel panic. We've experienced
> several recently, and each has similar output to this below when
> attempting to use the 'crash' utility:
>
> --snip--
> crash 7.2.3-2.fc29
Have you tried the latest version of crash? Preferably a freshly-built version
from github, or at least the latest Fedora build (crash-7.2.5-2.fc30)?
And does crash run on the live system?
Dave
Another question -- I see that your kernel is not KASLR, and there was a recent
change to the PAGE_OFFSET value. Look at your 4.14.91 kernel source tree and
find the values for __PAGE_OFFSET_BASE_L4 and __PAGE_OFFSET_BASE_L5 in
"arch/x86/include/asm/page_64_types.h".
If the 4.14.91 kernel has a backport of commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15
"x86/mm: Move LDT remap out of KASLR region on 5-level paging", then you'll
need
the github version of crash, which has this patch:
commit c63d678798c21a5379f132f124181715bb1d63d4
Author: Dave Anderson <anderson(a)redhat.com>
Date: Fri Jan 18 14:19:18 2019 -0500
Fix for Linux 4.20 and later x86_64 kernels which are NOT
configured with CONFIG_RANDOMIZE_BASE. Linux 4.20 introduced
kernel commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15, titled
"x86/mm: Move LDT remap out of KASLR region on 5-level paging",
which modified the 4-level and 5-level paging PAGE_OFFSET values.
Without this patch, the crash session fails during initialization
with the error message "crash: read error: kernel virtual address:
<address> type: tss_struct ist array". For kernels prior to
Linux 4.20.0 which have backports of the kernel commit, the kernel's
PAGE_OFFSET value must be manually specified via the command line
option "--machdep page_offset=ffff888000000000" for kernels with
4-level page tables, or "--machdep page_offset=ff11000000000000"
for kernels with 5-level paging. (or alternatively the shorter
version "-m page_offset=<address>" may be used). The command
line option requirement may be revisited in the future.
(anderson(a)redhat.com)
And then you'll have to use the "--machdep page_offset=" option.
Dave
> Copyright (C) 2002-2017 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> crash: read error: kernel virtual address: ffff88903f814d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903f854d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903f894d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903f8d4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903f914d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903f954d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903f994d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903f9d4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fa14d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fa54d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fa94d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fad4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fb14d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fb54d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fb94d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fbd4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fc14d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fc54d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ec14d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ec54d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ec94d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ecd4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ed14d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ed54d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ed94d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03edd4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ee14d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ee54d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ee94d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03eed4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ef14d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ef54d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03ef94d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03efd4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f014d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f054d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fc94d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fcd4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fd14d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fd54d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fd94d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fdd4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fe14d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fe54d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fe94d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903fed4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903ff14d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903ff54d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903ff94d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903ffd4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff889040014d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff889040054d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff889040094d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff8890400d4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f094d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f0d4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f114d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f154d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f194d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f1d4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f214d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f254d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f294d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f2d4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f314d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f354d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f394d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f3d4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f414d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f454d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f494d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88a03f4d4d40 type:
> "current_task (per_cpu)"
> crash: read error: kernel virtual address: ffff88903f805024 type:
> "tss_struct ist array"
> --snip--
>
> I've read a bit about similar errors to this on the mailing list, and
> some seem to indicate a physical memory offset issue (and/or use of
> high memory). I see the start and align values are the same in our
> kernel config so I don't believe that is the culprit (as others had
> seem to indicate with similar issues):
> CONFIG_PHYSICAL_START=0x1000000
> CONFIG_PHYSICAL_ALIGN=0x1000000
>
> When running crash with '-d8' I see the READ_ERROR messages:
> <readmem: ffff88903f814d40, KVADDR, "current_task (per_cpu)", 8,
> (ROE), 5593c9eb7c60>
> <read_kdump: addr: ffff88903f814d40 paddr: 903f814d40 cnt: 8>
> read_netdump: READ_ERROR: offset not found for paddr: 903f814d40
> crash: read error: kernel virtual address: ffff88903f814d40 type:
> "current_task (per_cpu)"
> <readmem: ffff88903f854d40, KVADDR, "current_task (per_cpu)", 8,
> (ROE), 5593c9eb7c68>
> <read_kdump: addr: ffff88903f854d40 paddr: 903f854d40 cnt: 8>
> read_netdump: READ_ERROR: offset not found for paddr: 903f854d40
> crash: read error: kernel virtual address: ffff88903f854d40 type:
> "current_task (per_cpu)"
> <readmem: ffff88903f894d40, KVADDR, "current_task (per_cpu)", 8,
> (ROE), 5593c9eb7c70>
> <read_kdump: addr: ffff88903f894d40 paddr: 903f894d40 cnt: 8>
> read_netdump: READ_ERROR: offset not found for paddr: 903f894d40
> crash: read error: kernel virtual address: ffff88903f894d40 type:
> "current_task (per_cpu)"
> <readmem: ffff88903f8d4d40, KVADDR, "current_task (per_cpu)", 8,
> (ROE), 5593c9eb7c78>
> <read_kdump: addr: ffff88903f8d4d40 paddr: 903f8d4d40 cnt: 8>
> read_netdump: READ_ERROR: offset not found for paddr: 903f8d4d40
> crash: read error: kernel virtual address: ffff88903f8d4d40 type:
> "current_task (per_cpu)"
> <readmem: ffff88903f914d40, KVADDR, "current_task (per_cpu)", 8,
> (ROE), 5593c9eb7c80>
> <read_kdump: addr: ffff88903f914d40 paddr: 903f914d40 cnt: 8>
> read_netdump: READ_ERROR: offset not found for paddr: 903f914d40
> crash: read error: kernel virtual address: ffff88903f914d40 type:
> "current_task (per_cpu)"
> <readmem: ffff88903f954d40, KVADDR, "current_task (per_cpu)", 8,
> (ROE), 5593c9eb7c88>
> <read_kdump: addr: ffff88903f954d40 paddr: 903f954d40 cnt: 8>
> read_netdump: READ_ERROR: offset not found for paddr: 903f954d40
> crash: read error: kernel virtual address: ffff88903f954d40 type:
> "current_task (per_cpu)"
> <readmem: ffff88903f994d40, KVADDR, "current_task (per_cpu)", 8,
> (ROE), 5593c9eb7c90>
> <read_kdump: addr: ffff88903f994d40 paddr: 903f994d40 cnt: 8>
> read_netdump: READ_ERROR: offset not found for paddr: 903f994d40
> ...
>
> I'm not sure where to go from here or what to look at next. These
> unusable dump files are produced by 'makedumpfile' version 1.6.4; this
> is the command used to capture the dump file in the crash dump kernel:
> makedumpfile -E -d 31 /proc/vmcore /mnt/tmp/dumpfile-`date +"%s"`
>
> Any help would be greatly appreciated... this previously worked with
> the same kernel config, the only thing that has changed that I can
> think of is patch release increments in the Linux kernel version
> (we've always stayed on Linux 4.14.x). But I'm not sure which one or
> when (or if that's it at all).
>
> I've attached our kernel configuration file as well.
>
>
> Thanks,
>
> Marc
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility