Fantastic.  I liked how you used "objdump -r" to get the function call name.  I can do that on a system that does not have source code (specifially code related my loaded module) on it to narrow down the location of the panic.  I found the adding the "-d" in addition very helpful, "objdump -d -r <*.ko>"
 
Thank you very much for taking time to respond to my questions in such detail, much appreciated.
 
--Ahmed.


On Thu, Feb 14, 2013 at 2:05 PM, Dave Anderson <anderson@redhat.com> wrote:

Another way to determine what's being called is to dump
the relocation information.  Again, in the nfs_cache_destroy()
function below, there are two callq instructions, one at
the very beginning of the function at 0x0000000000016750,
and the other one at 0x0000000000016759:

 (gdb) disass nfs_cache_destroy
 Dump of assembler code for function nfs_cache_destroy:
    0x0000000000016750 <+0>:    callq  0x16755 <nfs_cache_destroy+5>
    0x0000000000016755 <+5>:    push   %rbp
    0x0000000000016756 <+6>:    mov    %rsp,%rbp
    0x0000000000016759 <+9>:    callq  0x1675e <nfs_cache_destroy+14>
    0x000000000001675e <+14>:   pop    %rbp
    0x000000000001675f <+15>:   retq
 End of assembler dump.
 (gdb)

The callq instructions consist of a one-byte "e8" opcode
followed by 4 bytes of relative displacement, so in the
example above, those 4 bytes would be located at addresses
0x0000000000016751 and 0x000000000001675a respectively.
(i.e., the callq site plus 1)

So then if you dump the relocation information from the
object file, you can see that __fentry__() is called at the
beginning of the function (generated by the ftrace facility,
and appearing at the beginning of all functions if it's
configured), but more importangly, sunrpc_destroy_cache_detail()
is the target of the second callq:

 $ objdump --reloc nfs.ko
 ... [ cut ] ...
 0000000000016741 R_X86_64_PC32     __fentry__-0x0000000000000004
 000000000001674a R_X86_64_PC32     sunrpc_init_cache_detail-0x0000000000000004
 0000000000016751 R_X86_64_PC32     __fentry__-0x0000000000000004
 000000000001675a R_X86_64_PC32     sunrpc_destroy_cache_detail-0x0000000000000004
 0000000000016761 R_X86_64_PC32     __fentry__-0x0000000000000004
 0000000000016769 R_X86_64_32S      .data+0x0000000000000720
 ...

which is what the list command showed:

 (gdb) list *0x0000000000016759
 0x16759 is in nfs_cache_destroy (fs/nfs/cache_lib.c:164).
 159            sunrpc_init_cache_detail(cd);
 160    }
 161
 162    void nfs_cache_destroy(struct cache_detail *cd)
 163    {
 164            sunrpc_destroy_cache_detail(cd);
 165    }
 (gdb)

The ftrace-generated call to __fentry__() will never be shown in
the text listing.

Dave

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility