Hi Mike, On Tue, Mar 5, 2013 at 3:49 AM, Mika Westerberg <mika.westerberg(a)iki.fi&gt;wrote: > On Mon, Mar 04, 2013 at 10:20:42AM +0800, Lei Wen wrote: > > I met "dis" command not correct issue when use the crash, any idea? > > For built-in "dis" command in crash: > > crash> dis task_rq_lock > > 0xc015a2d8 <task_rq_lock>: rscsgt r0, sp, r3, lsl #14 > > 0xc015a2dc <task_rq_lock+4>: mrcgt 8, 7, r0, cr2, cr13, {5} > > 0xc015a2e0 <task_rq_lock+8>: mcrvc 8, 4, r3, cr13, cr3, {6} > > 0xc015a2e4 <task_rq_lock+12>: lslsvc r3, r10, r8 > > 0xc015a2e8 <task_rq_lock+16>: bl 0xc049fe34 > > <__ip_route_output_key+220> > > Looks weird. > > What is the kernel version? Does the 'dis' command work for other > functions? > Seems other functions is not working either... Is the page table being corrupted? But I still could parse out the dump image and check the kernel dmesg for it, which is also weird.

crash> dis sched_rt_handler 0xc0160ed8 <sched_rt_handler>: cdppl 13, 5, cr2, cr6, cr13, {4} 0xc0160edc <sched_rt_handler+4>: stcle 9, cr7, [r7], #132 ; 0x84 0xc0160ee0 <sched_rt_handler+8>: ; <UNDEFINED> instruction: 0x167256bb (gdb) disas sched_rt_handler Dump of assembler code for function sched_rt_handler: 0xc0160ed8 <+0>: push {r4, r5, r6, r7, r8, r9, r10, r11, lr} 0xc0160edc <+4>: add r11, sp, #32 0xc0160ee0 <+8>: ldr r4, [pc, #384] ; 0xc0161068 <sched_rt_handler+400>

> > What does 'nm vmlinux|grep sched_rt_handler' say? Here is the result: crash> !nm vmlinux|grep sched_rt_handler c0160ed8 T sched_rt_handler

> Since the dis command does nothing ARM specific, I wonder if we have GDB > version in crash that misses some newer GCC features? What GDB version are you > using? GDB version inside crash is 7.3.1, and crash version is 6.1.4. The standalone gdb version could be seen below: GNU gdb (Sourcery CodeBench Lite 2012.03-57) 7.2.50.20100908-cvs Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-none-linux-gnueabi". For bug reporting instructions, please see: <https://support.codesourcery.com/GNUToolchain/>;.

> Are you able to test with the same GDB version that is embedded in crash? > > I'm running out of ideas (I sure hope Per figures out what is going on :-)). Thanks for that vote of confidence =o) Alas, I must confess, all I have is a guess that "dot text" is partly a mess ...no less /Per > Are you able to make the vmlinux/vmcore pair available to us? >

> -- > Crash-utility mailing list > Crash-utility(a)redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility

On Thu, Mar 07, 2013 at 08:46:52PM +0100, Per Fransson wrote: > Hi, > > >> Are you able to test with the same GDB version that is embedded in crash? > >> > >> I'm running out of ideas (I sure hope Per figures out what is going on :-)). > > > > Thanks for that vote of confidence =o) > > > > Alas, I must confess, > > all I have is a guess > > that "dot text" is partly a mess > > ...no less > > > > /Per > > > >> Are you able to make the vmlinux/vmcore pair available to us? > >> > > Short version: > ############## > It's the compressed kernel, including decompression stub, occupying > the first 0x3ffe00 bytes of .text > > Long version: > ############# > I extracted the .text parts of the vmlinux and dump and diffed: > > vmlinux > ------- > .text: > start: 0x0081c0 = 33216 > size: 0x4a85d4 = 4883924 > dd bs=1 count=4883924 skip=33216 if=vmlinux of=vmlinux_text > > dump > ---- > .text: > start: 0xc01081c0 - 0xc0000000 + 0x94 = 1081940 > size: 0x4a85d4 = 4883924 > dd bs=1 count=4883924 skip=1081940 if=233_0128.dump of=dump_text > > First I thought I had made a mistake, because, to begin with, it's all > different, but then, after 0x3ffe00 bytes (that'd correspond to > virtual address 0xc01081c0+0x3ffe00=0xc0507fc0 in crash), it's all > identical (except for __v7_setup_stack, which for some reason is > placed in .text.) This made me run 'string' on the data in those first > 0x3ffe00 bytes of .text and they looked kind of familiar, for example: > > 3baf -- System halted > 3bc1 Attempting division by 0! > 3bdb Uncompressing Linux... > 3bf2 decompressor returned an error > 3c11 done, booting the kernel. > > and that's when I realized it was the image from > arch/arm/boot/compressed. I think you've loaded a new (capture?) > kernel at the location where the original one was loaded before > pulling out the dump. Nice finding! If you look the command line, it says crashkernel=4k@0x8140000 which might explain why the dump capture kernel has overridden the memory. However, there should be some check in the kexec code which prevents this.

Hi, >> Are you able to test with the same GDB version that is embedded in crash? >> >> I'm running out of ideas (I sure hope Per figures out what is going on :-)). > > Thanks for that vote of confidence =o) > > Alas, I must confess, > all I have is a guess > that "dot text" is partly a mess > ...no less > > /Per > >> Are you able to make the vmlinux/vmcore pair available to us? >> Short version: ############## It's the compressed kernel, including decompression stub, occupying the first 0x3ffe00 bytes of .text Long version: ############# I extracted the .text parts of the vmlinux and dump and diffed: vmlinux ------- .text: start: 0x0081c0 = 33216 size: 0x4a85d4 = 4883924 dd bs=1 count=4883924 skip=33216 if=vmlinux of=vmlinux_text dump ---- .text: start: 0xc01081c0 - 0xc0000000 + 0x94 = 1081940 size: 0x4a85d4 = 4883924 dd bs=1 count=4883924 skip=1081940 if=233_0128.dump of=dump_text First I thought I had made a mistake, because, to begin with, it's all different, but then, after 0x3ffe00 bytes (that'd correspond to virtual address 0xc01081c0+0x3ffe00=0xc0507fc0 in crash), it's all identical (except for __v7_setup_stack, which for some reason is placed in .text.) This made me run 'string' on the data in those first 0x3ffe00 bytes of .text and they looked kind of familiar, for example: 3baf -- System halted 3bc1 Attempting division by 0! 3bdb Uncompressing Linux... 3bf2 decompressor returned an error 3c11 done, booting the kernel. and that's when I realized it was the image from arch/arm/boot/compressed. I think you've loaded a new (capture?) kernel at the location where the original one was loaded before pulling out the dump. I also predict there will be a cut-off point at virtual address 0xc0507fc0, with all the code at lower addresses being bad and everything else being ok. If you need to you can probably replace those 0x3ffe00 bytes of the dump with those from the vmlinux.