Re: [Crash-utility] crash: read error: kernel virtual address: ...

Hi, I'm running vanilla Linux 4.14.91 and I cannot seem to obtain a working crash dump file from any kernel panic. We've experienced several recently, and each has similar output to this below when attempting to use the 'crash' utility: --snip-- crash 7.2.3-2.fc29

Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... crash: read error: kernel virtual address: ffff88903f814d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903f854d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903f894d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903f8d4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903f914d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903f954d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903f994d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903f9d4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fa14d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fa54d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fa94d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fad4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fb14d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fb54d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fb94d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fbd4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fc14d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fc54d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ec14d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ec54d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ec94d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ecd4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ed14d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ed54d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ed94d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03edd4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ee14d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ee54d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ee94d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03eed4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ef14d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ef54d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03ef94d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03efd4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f014d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f054d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fc94d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fcd4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fd14d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fd54d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fd94d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fdd4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fe14d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fe54d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fe94d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903fed4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903ff14d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903ff54d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903ff94d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903ffd4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff889040014d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff889040054d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff889040094d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff8890400d4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f094d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f0d4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f114d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f154d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f194d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f1d4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f214d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f254d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f294d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f2d4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f314d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f354d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f394d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f3d4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f414d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f454d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f494d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88a03f4d4d40 type: "current_task (per_cpu)" crash: read error: kernel virtual address: ffff88903f805024 type: "tss_struct ist array" --snip-- I've read a bit about similar errors to this on the mailing list, and some seem to indicate a physical memory offset issue (and/or use of high memory). I see the start and align values are the same in our kernel config so I don't believe that is the culprit (as others had seem to indicate with similar issues): CONFIG_PHYSICAL_START=0x1000000 CONFIG_PHYSICAL_ALIGN=0x1000000 When running crash with '-d8' I see the READ_ERROR messages: <readmem: ffff88903f814d40, KVADDR, "current_task (per_cpu)", 8, (ROE), 5593c9eb7c60> <read_kdump: addr: ffff88903f814d40 paddr: 903f814d40 cnt: 8> read_netdump: READ_ERROR: offset not found for paddr: 903f814d40 crash: read error: kernel virtual address: ffff88903f814d40 type: "current_task (per_cpu)" <readmem: ffff88903f854d40, KVADDR, "current_task (per_cpu)", 8, (ROE), 5593c9eb7c68> <read_kdump: addr: ffff88903f854d40 paddr: 903f854d40 cnt: 8> read_netdump: READ_ERROR: offset not found for paddr: 903f854d40 crash: read error: kernel virtual address: ffff88903f854d40 type: "current_task (per_cpu)" <readmem: ffff88903f894d40, KVADDR, "current_task (per_cpu)", 8, (ROE), 5593c9eb7c70> <read_kdump: addr: ffff88903f894d40 paddr: 903f894d40 cnt: 8> read_netdump: READ_ERROR: offset not found for paddr: 903f894d40 crash: read error: kernel virtual address: ffff88903f894d40 type: "current_task (per_cpu)" <readmem: ffff88903f8d4d40, KVADDR, "current_task (per_cpu)", 8, (ROE), 5593c9eb7c78> <read_kdump: addr: ffff88903f8d4d40 paddr: 903f8d4d40 cnt: 8> read_netdump: READ_ERROR: offset not found for paddr: 903f8d4d40 crash: read error: kernel virtual address: ffff88903f8d4d40 type: "current_task (per_cpu)" <readmem: ffff88903f914d40, KVADDR, "current_task (per_cpu)", 8, (ROE), 5593c9eb7c80> <read_kdump: addr: ffff88903f914d40 paddr: 903f914d40 cnt: 8> read_netdump: READ_ERROR: offset not found for paddr: 903f914d40 crash: read error: kernel virtual address: ffff88903f914d40 type: "current_task (per_cpu)" <readmem: ffff88903f954d40, KVADDR, "current_task (per_cpu)", 8, (ROE), 5593c9eb7c88> <read_kdump: addr: ffff88903f954d40 paddr: 903f954d40 cnt: 8> read_netdump: READ_ERROR: offset not found for paddr: 903f954d40 crash: read error: kernel virtual address: ffff88903f954d40 type: "current_task (per_cpu)" <readmem: ffff88903f994d40, KVADDR, "current_task (per_cpu)", 8, (ROE), 5593c9eb7c90> <read_kdump: addr: ffff88903f994d40 paddr: 903f994d40 cnt: 8> read_netdump: READ_ERROR: offset not found for paddr: 903f994d40 ... I'm not sure where to go from here or what to look at next. These unusable dump files are produced by 'makedumpfile' version 1.6.4; this is the command used to capture the dump file in the crash dump kernel: makedumpfile -E -d 31 /proc/vmcore /mnt/tmp/dumpfile-`date +"%s"` Any help would be greatly appreciated... this previously worked with the same kernel config, the only thing that has changed that I can think of is patch release increments in the Linux kernel version (we've always stayed on Linux 4.14.x). But I'm not sure which one or when (or if that's it at all). I've attached our kernel configuration file as well. Thanks, Marc -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility