10:49 p.m.
----- Original Message -----
On 11/29/15 at 10:52pm, Dave Young wrote:
[snip]
With the new build it fails at later point, but I noticed I enabled
CONFIG_DEBUG_INFO_REDUCED=y
A retest without CONFIG_DEBUG_INFO_REDUCED succeed with latest latest
crash build, but there's some read errors. I'm not sure if they matters
but "bt" works well in crash.
[snip]
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "armv7l-unknown-linux-gnueabihf"...
please wait... (gathering kmem slab cache data)
crash: read error: kernel virtual address: ff7fb414 type: "array cache
limit"
crash: kmem_cache: df1e34c0: invalid array_cache pointer: ff7fb410
crash: read error: kernel virtual address: ff7fac08 type: "array cache
limit"
crash: kmem_cache: df1e3540: invalid array_cache pointer: ff7fac04
crash: read error: kernel virtual address: ff7f7edc type: "array cache
limit"
crash: kmem_cache: df1e35c0: invalid array_cache pointer: ff7f7ed8
crash: read error: kernel virtual address: ff7f7cec type: "array cache
limit"
crash: kmem_cache: df1e3640: invalid array_cache pointer: ff7f7ce8
crash: read error: kernel virtual address: ff7f7c04 type: "array cache
limit"
crash: kmem_cache: df1e36c0: invalid array_cache pointer: ff7f7c00
crash: read error: kernel virtual address: ff7f7894 type: "array cache
limit"
crash: kmem_cache: df1e3740: invalid array_cache pointer: ff7f7890
crash: read error: kernel virtual address: ff7f77ac type: "array cache
limit"
crash: kmem_cache: df1e37c0: invalid array_cache pointer: ff7f77a8
crash: read error: kernel virtual address: ff7f76c4 type: "array cache
limit"
crash: kmem_cache: df1e3840: invalid array_cache pointer: ff7f76c0
crash: read error: kernel virtual address: ff7f74d4 type: "array cache
limit"
crash: kmem_cache: df1e38c0: invalid array_cache pointer: ff7f74d0
crash: read error: kernel virtual address: ff7f73ec type: "array cache
limit"
crash: kmem_cache: df1e3940: invalid array_cache pointer: ff7f73e8
crash: read error: kernel virtual address: ff7f7304 type: "array cache
limit"
crash: kmem_cache: df1e39c0: invalid array_cache pointer: ff7f7300
crash: read error: kernel virtual address: ff7f707c type: "array cache
limit"
crash: kmem_cache: df1e3a40: invalid array_cache pointer: ff7f7078
crash: read error: kernel virtual address: ff7f6e8c type: "array cache
limit"
crash: kmem_cache: df1e3ac0: invalid array_cache pointer: ff7f6e88
crash: read error: kernel virtual address: ff7f6c9c type: "array cache
limit"
crash: kmem_cache: df1e3b40: invalid array_cache pointer: ff7f6c98
crash: read error: kernel virtual address: ff7f6aac type: "array cache
limit"
crash: kmem_cache: df1e3bc0: invalid array_cache pointer: ff7f6aa8
crash: read error: kernel virtual address: ff7f68bc type: "array cache
limit"
crash: kmem_cache: df1e3c40: invalid array_cache pointer: ff7f68b8
crash: read error: kernel virtual address: ff7f67d4 type: "array cache
limit"
crash: kmem_cache: df1e3cc0: invalid array_cache pointer: ff7f67d0
crash: read error: kernel virtual address: ff7f65e4 type: "array cache
limit"
crash: kmem_cache: df1e3d40: invalid array_cache pointer: ff7f65e0
crash: read error: kernel virtual address: ff7f63f4 type: "array cache
limit"
crash: kmem_cache: df1e3dc0: invalid array_cache pointer: ff7f63f0
crash: read error: kernel virtual address: ff7f6204 type: "array cache
limit"
crash: kmem_cache: df1e3e40: invalid array_cache pointer: ff7f6200
crash: read error: kernel virtual address: ff7f6014 type: "array cache
limit"
crash: kmem_cache: df1e3ec0: invalid array_cache pointer: ff7f6010
crash: read error: kernel virtual address: ff7f5e0c type: "array cache
limit"
crash: kmem_cache: df1e3f40: invalid array_cache pointer: ff7f5e08
crash: read error: kernel virtual address: ff7f5b04 type: "array cache
limit"
crash: kmem_cache: df017040: invalid array_cache pointer: ff7f5b00
crash: read error: kernel virtual address: ff7f57e0 type: "array cache
limit"
crash: kmem_cache: df0170c0: invalid array_cache pointer: ff7f57dc
crash: read error: kernel virtual address: ff7f5300 type: "array cache
limit"
crash: kmem_cache: df017140: invalid array_cache pointer: ff7f52fc
crash: read error: kernel virtual address: ff7f50ec type: "array cache
limit"
crash: kmem_cache: df0171c0: invalid array_cache pointer: ff7f50e8
crash: read error: kernel virtual address: ff7f4004 type: "array cache
limit"
crash: kmem_cache: df017240: invalid array_cache pointer: ff7f4000
WARNING: SLAB: cannot determine how compound pages are linked
KERNEL: vmlinux
DUMPFILE: /var/crash/127.0.0.1-2015-11-30-13:10:23/vmcore
CPUS: 1
DATE: Mon Nov 30 13:10:14 2015
UPTIME: 00:00:38
LOAD AVERAGE: 0.27, 0.08, 0.03
TASKS: 157
NODENAME: localhost
RELEASE: 4.4.0-rc2+
VERSION: #46 SMP Mon Nov 30 12:58:47 CST 2015
MACHINE: armv7l (unknown Mhz)
MEMORY: 510 MB
PANIC: "sysrq: SysRq : Trigger a crash"
PID: 1368
COMMAND: "bash"
TASK: ddbbeb40 [THREAD_INFO: ddbc4000]
CPU: 0
STATE: TASK_RUNNING (SYSRQ)
crash> bt
PID: 1368 TASK: ddbbeb40 CPU: 0 COMMAND: "bash"
#0 [<c021e23c>] (sysrq_handle_crash) from [<c021e5a5>]
#1 [<c021e5a5>] (__handle_sysrq) from [<c021e969>]
#2 [<c021e969>] (write_sysrq_trigger) from [<c01067df>]
#3 [<c01067df>] (proc_reg_write) from [<c00ca5db>]
#4 [<c00ca5db>] (__vfs_write) from [<c00cac0f>]
#5 [<c00cac0f>] (vfs_write) from [<c00cb24b>]
#6 [<c00cb24b>] (sys_write) from [<c000ddc1>]
crash>
> Can you send me the location of the vmlinux and vmcore files offline? I can
> try working with them tomorrow with on an x86_64 host with a crash utility
> built with "make target=ARM".
Will save the files somewhere and let you know..
Thanks a lot
Dave
Dave,
OK, so the problem is that in CONFIG_SLAB kernels, the array_cache
structure that is pointed to by each kmem_cache.cpu_cache can now
be a vmalloc'd memory location. That may be a new innovation for
32-bit kernels? I don't know, but in any case, all of these error
types:
crash: kmem_cache: df1e34c0: invalid array_cache pointer: ff7fb410
show array_cache pointers in the vmalloc region. The kmem_cache.cpu_cache
pointer is actually a per_cpu pointer, but on some of the kmem slab caches,
the translation of the per_cpu pointer result in a vmalloc() address instead
of a unity-mapped address. Perhaps there's a point at which the percpu
memory allocations switch from being unity-mapped structures to being
vmalloc()'d addresses?
In any case, at the point in time when the kmem caches are being
initialized (when the error messages get generated), the 32-bit ARM
arch has not yet set the base address of the vmalloc region in one
if its machine-dependent variables. And so the addresses are
mistakenly translated as if they are unity-mapped. The ARM arch
sets the internal vmalloc base address variable at a later point
in time, but it can be done earlier, before the kmem cache initialization
is started. So with this one-line patch, the dumpfile comes up cleanly:
--- arm.c.orig 2015-12-01 16:36:11.815775008 -0500
+++ arm.c 2015-12-01 16:25:44.580800293 -0500
@@ -1539,6 +1539,7 @@
static ulong
arm_vmalloc_start(void)
{
+ machdep->machspec->vmalloc_start_addr = vt->high_memory;
return vt->high_memory;
}
$ crash vmlinux vmcore-cp
crash 7.1.4rc21
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu
--target=arm-elf-linux"...
KERNEL: vmlinux
DUMPFILE: vmcore-cp
CPUS: 1
DATE: Mon Nov 30 00:10:14 2015
UPTIME: 00:00:38
LOAD AVERAGE: 0.27, 0.08, 0.03
TASKS: 157
NODENAME: localhost
RELEASE: 4.4.0-rc2+
VERSION: #46 SMP Mon Nov 30 12:58:47 CST 2015
MACHINE: armv7l (unknown Mhz)
MEMORY: 510 MB
PANIC: "sysrq: SysRq : Trigger a crash"
PID: 1368
COMMAND: "bash"
TASK: ddbbeb40 [THREAD_INFO: ddbc4000]
CPU: 0
STATE: TASK_RUNNING (SYSRQ)
crash>
I can only test the ELF vmcore because I don't have the 32-bit lzo library
installed for my "make target=ARM" x86 binary. But it won't be any
different from the one above.
I'll queue the patch for crash-7.1.4.
Thanks for the report,
Dave