Re: [Crash-utility] crash and upstream arm kernel

----- Original Message ----- > On 11/29/15 at 10:52pm, Dave Young wrote: > [snip] > > With the new build it fails at later point, but I noticed I enabled > CONFIG_DEBUG_INFO_REDUCED=y > > A retest without CONFIG_DEBUG_INFO_REDUCED succeed with latest latest > crash build, but there's some read errors. I'm not sure if they matters > but "bt" works well in crash. > > [snip] > GNU gdb (GDB) 7.6 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "armv7l-unknown-linux-gnueabihf"... > > please wait... (gathering kmem slab cache data) > crash: read error: kernel virtual address: ff7fb414 type: "array cache > limit" > > crash: kmem_cache: df1e34c0: invalid array_cache pointer: ff7fb410 > > crash: read error: kernel virtual address: ff7fac08 type: "array cache > limit" > > crash: kmem_cache: df1e3540: invalid array_cache pointer: ff7fac04 > > crash: read error: kernel virtual address: ff7f7edc type: "array cache > limit" > > crash: kmem_cache: df1e35c0: invalid array_cache pointer: ff7f7ed8 > > crash: read error: kernel virtual address: ff7f7cec type: "array cache > limit" > > crash: kmem_cache: df1e3640: invalid array_cache pointer: ff7f7ce8 > > crash: read error: kernel virtual address: ff7f7c04 type: "array cache > limit" > > crash: kmem_cache: df1e36c0: invalid array_cache pointer: ff7f7c00 > > crash: read error: kernel virtual address: ff7f7894 type: "array cache > limit" > > crash: kmem_cache: df1e3740: invalid array_cache pointer: ff7f7890 > > crash: read error: kernel virtual address: ff7f77ac type: "array cache > limit" > > crash: kmem_cache: df1e37c0: invalid array_cache pointer: ff7f77a8 > > crash: read error: kernel virtual address: ff7f76c4 type: "array cache > limit" > > crash: kmem_cache: df1e3840: invalid array_cache pointer: ff7f76c0 > > crash: read error: kernel virtual address: ff7f74d4 type: "array cache > limit" > > crash: kmem_cache: df1e38c0: invalid array_cache pointer: ff7f74d0 > > crash: read error: kernel virtual address: ff7f73ec type: "array cache > limit" > > crash: kmem_cache: df1e3940: invalid array_cache pointer: ff7f73e8 > > crash: read error: kernel virtual address: ff7f7304 type: "array cache > limit" > > crash: kmem_cache: df1e39c0: invalid array_cache pointer: ff7f7300 > > crash: read error: kernel virtual address: ff7f707c type: "array cache > limit" > > crash: kmem_cache: df1e3a40: invalid array_cache pointer: ff7f7078 > > crash: read error: kernel virtual address: ff7f6e8c type: "array cache > limit" > > crash: kmem_cache: df1e3ac0: invalid array_cache pointer: ff7f6e88 > > crash: read error: kernel virtual address: ff7f6c9c type: "array cache > limit" > > crash: kmem_cache: df1e3b40: invalid array_cache pointer: ff7f6c98 > > crash: read error: kernel virtual address: ff7f6aac type: "array cache > limit" > > crash: kmem_cache: df1e3bc0: invalid array_cache pointer: ff7f6aa8 > > crash: read error: kernel virtual address: ff7f68bc type: "array cache > limit" > > crash: kmem_cache: df1e3c40: invalid array_cache pointer: ff7f68b8 > > crash: read error: kernel virtual address: ff7f67d4 type: "array cache > limit" > > crash: kmem_cache: df1e3cc0: invalid array_cache pointer: ff7f67d0 > > crash: read error: kernel virtual address: ff7f65e4 type: "array cache > limit" > > crash: kmem_cache: df1e3d40: invalid array_cache pointer: ff7f65e0 > > crash: read error: kernel virtual address: ff7f63f4 type: "array cache > limit" > > crash: kmem_cache: df1e3dc0: invalid array_cache pointer: ff7f63f0 > > crash: read error: kernel virtual address: ff7f6204 type: "array cache > limit" > > crash: kmem_cache: df1e3e40: invalid array_cache pointer: ff7f6200 > > crash: read error: kernel virtual address: ff7f6014 type: "array cache > limit" > > crash: kmem_cache: df1e3ec0: invalid array_cache pointer: ff7f6010 > > crash: read error: kernel virtual address: ff7f5e0c type: "array cache > limit" > > crash: kmem_cache: df1e3f40: invalid array_cache pointer: ff7f5e08 > > crash: read error: kernel virtual address: ff7f5b04 type: "array cache > limit" > > crash: kmem_cache: df017040: invalid array_cache pointer: ff7f5b00 > > crash: read error: kernel virtual address: ff7f57e0 type: "array cache > limit" > > crash: kmem_cache: df0170c0: invalid array_cache pointer: ff7f57dc > > crash: read error: kernel virtual address: ff7f5300 type: "array cache > limit" > > crash: kmem_cache: df017140: invalid array_cache pointer: ff7f52fc > > crash: read error: kernel virtual address: ff7f50ec type: "array cache > limit" > > crash: kmem_cache: df0171c0: invalid array_cache pointer: ff7f50e8 > > crash: read error: kernel virtual address: ff7f4004 type: "array cache > limit" > > crash: kmem_cache: df017240: invalid array_cache pointer: ff7f4000 > WARNING: SLAB: cannot determine how compound pages are linked > > KERNEL: vmlinux > DUMPFILE: /var/crash/127.0.0.1-2015-11-30-13:10:23/vmcore > CPUS: 1 > DATE: Mon Nov 30 13:10:14 2015 > UPTIME: 00:00:38 > LOAD AVERAGE: 0.27, 0.08, 0.03 > TASKS: 157 > NODENAME: localhost > RELEASE: 4.4.0-rc2+ > VERSION: #46 SMP Mon Nov 30 12:58:47 CST 2015 > MACHINE: armv7l (unknown Mhz) > MEMORY: 510 MB > PANIC: "sysrq: SysRq : Trigger a crash" > PID: 1368 > COMMAND: "bash" > TASK: ddbbeb40 [THREAD_INFO: ddbc4000] > CPU: 0 > STATE: TASK_RUNNING (SYSRQ) > > crash> bt > PID: 1368 TASK: ddbbeb40 CPU: 0 COMMAND: "bash" > #0 [<c021e23c>] (sysrq_handle_crash) from [<c021e5a5>] > #1 [<c021e5a5>] (__handle_sysrq) from [<c021e969>] > #2 [<c021e969>] (write_sysrq_trigger) from [<c01067df>] > #3 [<c01067df>] (proc_reg_write) from [<c00ca5db>] > #4 [<c00ca5db>] (__vfs_write) from [<c00cac0f>] > #5 [<c00cac0f>] (vfs_write) from [<c00cb24b>] > #6 [<c00cb24b>] (sys_write) from [<c000ddc1>] > crash> > > > Can you send me the location of the vmlinux and vmcore files offline? I can > > try working with them tomorrow with on an x86_64 host with a crash utility > > built with "make target=ARM". > > Will save the files somewhere and let you know.. > > Thanks a lot > Dave > Dave, OK, so the problem is that in CONFIG_SLAB kernels, the array_cache structure that is pointed to by each kmem_cache.cpu_cache can now be a vmalloc'd memory location. That may be a new innovation for 32-bit kernels? I don't know, but in any case, all of these error

types: crash: kmem_cache: df1e34c0: invalid array_cache pointer: ff7fb410 show array_cache pointers in the vmalloc region. The kmem_cache.cpu_cache pointer is actually a per_cpu pointer, but on some of the kmem slab caches, the translation of the per_cpu pointer result in a vmalloc() address instead of a unity-mapped address. Perhaps there's a point at which the percpu memory allocations switch from being unity-mapped structures to being vmalloc()'d addresses?

In any case, at the point in time when the kmem caches are being initialized (when the error messages get generated), the 32-bit ARM arch has not yet set the base address of the vmalloc region in one if its machine-dependent variables. And so the addresses are mistakenly translated as if they are unity-mapped. The ARM arch sets the internal vmalloc base address variable at a later point in time, but it can be done earlier, before the kmem cache initialization is started. So with this one-line patch, the dumpfile comes up cleanly: --- arm.c.orig 2015-12-01 16:36:11.815775008 -0500 +++ arm.c 2015-12-01 16:25:44.580800293 -0500 @@ -1539,6 +1539,7 @@ static ulong arm_vmalloc_start(void) { + machdep->machspec->vmalloc_start_addr = vt->high_memory; return vt->high_memory; } $ crash vmlinux vmcore-cp crash 7.1.4rc21 Copyright (C) 2002-2014 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=arm-elf-linux"... KERNEL: vmlinux DUMPFILE: vmcore-cp CPUS: 1 DATE: Mon Nov 30 00:10:14 2015 UPTIME: 00:00:38 LOAD AVERAGE: 0.27, 0.08, 0.03 TASKS: 157 NODENAME: localhost RELEASE: 4.4.0-rc2+ VERSION: #46 SMP Mon Nov 30 12:58:47 CST 2015 MACHINE: armv7l (unknown Mhz) MEMORY: 510 MB PANIC: "sysrq: SysRq : Trigger a crash" PID: 1368 COMMAND: "bash" TASK: ddbbeb40 [THREAD_INFO: ddbc4000] CPU: 0 STATE: TASK_RUNNING (SYSRQ) crash> I can only test the ELF vmcore because I don't have the 32-bit lzo library installed for my "make target=ARM" x86 binary. But it won't be any different from the one above. I'll queue the patch for crash-7.1.4.