7:55 a.m.
----- "Bob Montgomery" <bob.montgomery(a)hp.com> wrote:
This patch allows the dis -l command to show real source line
numbers
for module code, instead of this sort of thing:
crash-4.0.9> dis -l bnx2_poll_msix
include/linux/cpumask.h: 612
0xffffffffa008acc3 <bnx2_poll_msix>: push %rbp
0xffffffffa008acc4 <bnx2_poll_msix+1>: mov %rsp,%rbp
0xffffffffa008acc7 <bnx2_poll_msix+4>: push %r15
0xffffffffa008acc9 <bnx2_poll_msix+6>: push %r14
0xffffffffa008accb <bnx2_poll_msix+8>: push %r13
0xffffffffa008accd <bnx2_poll_msix+10>: mov %esi,%r13d
0xffffffffa008acd0 <bnx2_poll_msix+13>: push %r12
0xffffffffa008acd2 <bnx2_poll_msix+15>: xor %r12d,%r12d
0xffffffffa008acd5 <bnx2_poll_msix+18>: push %rbx
0xffffffffa008acd6 <bnx2_poll_msix+19>: mov %rdi,%rbx
0xffffffffa008acd9 <bnx2_poll_msix+22>: sub $0x8,%rsp
The problem has to do with a symbol table in the kernel containing an
abnormally huge text range because of a goofy vsyscall reference in the
file "arch/x86/kernel/hpet.c".
(gdb) p *(struct block *)0x8d664b0
$41 = {startaddr = 0xffffffff80225130, endaddr = 0xffffffffff6001b3,
This range pretty much covers the entire module address space and since
the kernel file is first in the object_files list, the search stops
there and returns the closest symbol from the kernel, instead of looking
on through the object list to the module of actual interest.
This patch causes the code to keep looking through all modules and
return the real best symbol thingy (you can tell how much I really know
about gdb's symtab stuff, ahem). Thanks (and blame) to John Wright for
suggesting the fix once I found the problem.
The patch applies to the gdb patch file in the crash directory.
John and I think that this code in gdb searches things too many times,
particularly with this patch, but it's a start since it seems to fix the
problem. Mayhaps some gdb experts can now help do it right.
With the patch, the above example becomes:
crash-4.0.9> dis -l bnx2_poll_msix
/build/buildd/linux-2.6-clim-2.6.29-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
3215
0xffffffffa008acc3 <bnx2_poll_msix>: push %rbp
0xffffffffa008acc4 <bnx2_poll_msix+1>: mov %rsp,%rbp
0xffffffffa008acc7 <bnx2_poll_msix+4>: push %r15
0xffffffffa008acc9 <bnx2_poll_msix+6>: push %r14
0xffffffffa008accb <bnx2_poll_msix+8>: push %r13
0xffffffffa008accd <bnx2_poll_msix+10>: mov %esi,%r13d
0xffffffffa008acd0 <bnx2_poll_msix+13>: push %r12
/build/buildd/linux-2.6-clim-2.6.29-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
3219
0xffffffffa008acd2 <bnx2_poll_msix+15>: xor %r12d,%r12d
/build/buildd/linux-2.6-clim-2.6.29-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
3215
0xffffffffa008acd5 <bnx2_poll_msix+18>: push %rbx
0xffffffffa008acd6 <bnx2_poll_msix+19>: mov %rdi,%rbx
0xffffffffa008acd9 <bnx2_poll_msix+22>: sub $0x8,%rsp
Outstanding!
BTW, I don't know if there *are* any gdb experts on this list...
If there are, I hope they can speak up!
I haven't tested this patch or taken the time to understand it,
but I'm taking it on good faith that it solves this bug, which
has been an elephant in the room since (I believe) the 2.6.21
timeframe.
In fact, Mike Snitzer had narrowed it down to the kernel patch that
seemingly caused it, and you've confirmed it:
https://www.redhat.com/archives/crash-utility/2009-January/msg00063.html
which was this one:
commit 7460ed2844ffad7141e30271c0c3da8336e66014
Author: john stultz <johnstul(a)us.ibm.com>
Date: Fri Feb 16 01:28:21 2007 -0800
[PATCH] time: x86_64: re-enable vsyscall support for x86_64
Cleanup and re-enable vsyscall gettimeofday using the generic clocksource
infrastructure.
[akpm osdl org: cleanup]
Signed-off-by: John Stultz <johnstul us ibm com>
Cc: Ingo Molnar <mingo elte hu>
Cc: Thomas Gleixner <tglx linutronix de>
Cc: Andi Kleen <ak muc de>
Cc: Roman Zippel <zippel linux-m68k org>
Signed-off-by: Andrew Morton <akpm linux-foundation org>
Signed-off-by: Linus Torvalds <torvalds linux-foundation org>
arch/x86_64/Kconfig | 4 +
arch/x86_64/kernel/hpet.c | 6 ++
arch/x86_64/kernel/time.c | 6 --
arch/x86_64/kernel/tsc.c | 7 ++
arch/x86_64/kernel/vmlinux.lds.S | 28 ++++------
arch/x86_64/kernel/vsyscall.c | 119 ++++++++++++++++++++++---------------
include/asm-x86_64/proto.h | 2 -
include/asm-x86_64/timex.h | 1 -
include/asm-x86_64/vsyscall.h | 29 ++--------
9 files changed, 104 insertions(+), 98 deletions(-)
Anyway, this is awesome -- I (and countless others) really appreciate it...
Thanks to you and John,
Dave
5:24 p.m.
New subject: Fix for source line numbers for x86_64 modules
On Thu, 2009-09-17 at 12:55 +0000, Dave Anderson wrote:
----- "Bob Montgomery" <bob.montgomery(a)hp.com>
wrote:
> This patch allows the dis -l command to show real source line numbers
> for module code, instead of this sort of thing:
>
I haven't tested this patch or taken the time to understand it,
but I'm taking it on good faith that it solves this bug, which
has been an elephant in the room since (I believe) the 2.6.21
timeframe.
It will be nice to get some independent verification.
In the meantime, we've found a couple of other things.
1) There is a class of module routine whose info does not
show up where this version of gdb checks, and those routines
will still not show line numbers. They are:
a) routines declared with __devexit
b) routines declared with __devinit
c) routines in module_exit() MACROs
d) routines in module_init() MACROs
In our particular 2.6.29-based test kernel, we load 70 modules.
Before this patch, we had 5507 routines that would not show
line numbers (reported with "include/linux/cpumask.h: 612 instead
of the real line).
After the patch, we have only 89 routines that won't show
line numbers correctly for the above reasons. For example:
...
ffffffffa006bc22 (t) e1000_exit_module include/linux/cpumask.h: 612
ffffffffa006bc45 (t) e1000_remove include/linux/cpumask.h: 612
ffffffffa0078188 (t) sha1_generic_mod_fini include/linux/cpumask.h: 612
ffffffffa008af54 (t) bnx2_cleanup include/linux/cpumask.h: 612
ffffffffa008af66 (t) bnx2_remove_one include/linux/cpumask.h: 612
ffffffffa008afbe (t) bnx2_init_one include/linux/cpumask.h: 612
ffffffffa00b3210 (t) cciss_init_one include/linux/cpumask.h: 612
...
2) John took a look at a new version of gdb, and it appears
to have some redesign in the symbol stuff that corrects
perhaps all of this. I'll let him summarize what he found.
In the meantime, I'll be interested to hear of any other
testing of this patch.
Bob Montgomery
1:41 p.m.
New subject: Fix for source line numbers for x86_64 modules
On Mon, Sep 21, 2009 at 04:24:58PM -0600, Bob Montgomery wrote:
On Thu, 2009-09-17 at 12:55 +0000, Dave Anderson wrote:
> ----- "Bob Montgomery" <bob.montgomery(a)hp.com> wrote:
>
> > This patch allows the dis -l command to show real source line numbers
> > for module code, instead of this sort of thing:
> >
> I haven't tested this patch or taken the time to understand it,
> but I'm taking it on good faith that it solves this bug, which
> has been an elephant in the room since (I believe) the 2.6.21
> timeframe.
It will be nice to get some independent verification.
In the meantime, we've found a couple of other things.
1) There is a class of module routine whose info does not
show up where this version of gdb checks, and those routines
will still not show line numbers. They are:
a) routines declared with __devexit
b) routines declared with __devinit
c) routines in module_exit() MACROs
d) routines in module_init() MACROs
In our particular 2.6.29-based test kernel, we load 70 modules.
Before this patch, we had 5507 routines that would not show
line numbers (reported with "include/linux/cpumask.h: 612 instead
of the real line).
After the patch, we have only 89 routines that won't show
line numbers correctly for the above reasons. For example:
...
ffffffffa006bc22 (t) e1000_exit_module include/linux/cpumask.h: 612
ffffffffa006bc45 (t) e1000_remove include/linux/cpumask.h: 612
ffffffffa0078188 (t) sha1_generic_mod_fini include/linux/cpumask.h: 612
ffffffffa008af54 (t) bnx2_cleanup include/linux/cpumask.h: 612
ffffffffa008af66 (t) bnx2_remove_one include/linux/cpumask.h: 612
ffffffffa008afbe (t) bnx2_init_one include/linux/cpumask.h: 612
ffffffffa00b3210 (t) cciss_init_one include/linux/cpumask.h: 612
...
2) John took a look at a new version of gdb, and it appears
to have some redesign in the symbol stuff that corrects
perhaps all of this. I'll let him summarize what he found.
Sorry for the delay. I'm testing this by running gdb directly on the
running kernel, using /proc/kcore as the core file. To load symbols
from a module, I'm using this script:
#!/bin/bash
#
# Usage: ./gdbline module image
#
# Outputs an add-symbol-file line suitable for pasting into gdb to examine
# a loaded module.
#
cd /sys/module/$1/sections
echo -n add-symbol-file $2 `/bin/cat .text`
for section in .[a-z]* *; do
if [ $section != ".text" ]; then
echo -n " -s" $section `/bin/cat $section`
fi
done
echo
(This is based on [1], only without a multiline command, since that will
corrupt the heap until some CVS commits today [2].)
So, with that out of the way, this is what I get for cciss with gdb
6.8-3 on Debian/amd64:
$ sudo gdb /usr/lib/debug/vmlinux-2.6.29-clim-3-amd64 /proc/kcore
[sudo] password for jswright:
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
warning: core file may not match specified executable file.
Core was generated by `root=/dev/cciss/c0d0p1 ro console=ttyS1,115200n8 mem=4G
crashkernel=128M@16M qui'.
[New process 0]
#0 0x0000000000000000 in ?? ()
(gdb) add-symbol-file
/usr/lib/debug/lib/modules/2.6.29-clim-3-amd64/kernel/drivers/block/cciss.ko
0xffffffffa007b000 -s .bss 0xffffffffa008feb0 -s .data 0xffffffffa0086f10 -s .devexit.text
0xffffffffa0082502 -s .devinit.text 0xffffffffa0081210 -s .exit.text 0xffffffffa00828cc -s
.gnu.linkonce.this_module 0xffffffffa008fcb0 -s .init.text 0xffffffffa0006000 -s
.note.gnu.build-id 0xffffffffa008292c -s .parainstructions 0xffffffffa0084b18 -s .rodata
0xffffffffa0082950 -s .rodata.str1.1 0xffffffffa0083280 -s .smp_locks 0xffffffffa0084bc8
-s .strtab 0xffffffffa0086208 -s .symtab 0xffffffffa0084c60 -s __bug_table
0xffffffffa0084bd8
add symbol table from file
"/usr/lib/debug/lib/modules/2.6.29-clim-3-amd64/kernel/drivers/block/cciss.ko"
at
.text_addr = 0xffffffffa007b000
.bss_addr = 0xffffffffa008feb0
.data_addr = 0xffffffffa0086f10
.devexit.text_addr = 0xffffffffa0082502
.devinit.text_addr = 0xffffffffa0081210
.exit.text_addr = 0xffffffffa00828cc
.gnu.linkonce.this_module_addr = 0xffffffffa008fcb0
.init.text_addr = 0xffffffffa0006000
.note.gnu.build-id_addr = 0xffffffffa008292c
.parainstructions_addr = 0xffffffffa0084b18
.rodata_addr = 0xffffffffa0082950
.rodata.str1.1_addr = 0xffffffffa0083280
.smp_locks_addr = 0xffffffffa0084bc8
.strtab_addr = 0xffffffffa0086208
.symtab_addr = 0xffffffffa0084c60
__bug_table_addr = 0xffffffffa0084bd8
(y or n) y
Reading symbols from
/usr/lib/debug/lib/modules/2.6.29-clim-3-amd64/kernel/drivers/block/cciss.ko...warning:
section .strtab not found in
/usr/lib/debug/lib/modules/2.6.29-clim-3-amd64/kernel/drivers/block/cciss.ko
warning: section .symtab not found in
/usr/lib/debug/lib/modules/2.6.29-clim-3-amd64/kernel/drivers/block/cciss.ko
done.
(gdb) info line cciss_init_one
No line number information available for address
0xffffffffa0081210 <cciss_init_one>
With gdb 6.8.50.20090628-4 on Debian/amd64:
# gdb /usr/lib/debug/vmlinux-2.6.29-clim-3-amd64 /proc/kcore
GNU gdb (GDB) 6.8.50.20090628-cvs-debian
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
warning: core file may not match specified executable file.
Core was generated by `root=/dev/cciss/c0d0p1 ro console=ttyS1,115200n8 mem=4G
crashkernel=128M@16M qui'.
#0 0x0000000000000000 in ?? ()
(gdb) add-symbol-file
/usr/lib/debug/lib/modules/2.6.29-clim-3-amd64/kernel/drivers/block/cciss.ko
0xffffffffa007b000 -s .bss 0xffffffffa008feb0 -s .data 0xffffffffa0086f10 -s .devexit.text
0xffffffffa0082502 -s .devinit.text 0xffffffffa0081210 -s .exit.text 0xffffffffa00828cc -s
.gnu.linkonce.this_module 0xffffffffa008fcb0 -s .init.text 0xffffffffa0006000 -s
.note.gnu.build-id 0xffffffffa008292c -s .parainstructions 0xffffffffa0084b18 -s .rodata
0xffffffffa0082950 -s .rodata.str1.1 0xffffffffa0083280 -s .smp_locks 0xffffffffa0084bc8
-s .strtab 0xffffffffa0086208 -s .symtab 0xffffffffa0084c60 -s __bug_table
0xffffffffa0084bd8
add symbol table from file
"/usr/lib/debug/lib/modules/2.6.29-clim-3-amd64/kernel/drivers/block/cciss.ko"
at
.text_addr = 0xffffffffa007b000
.bss_addr = 0xffffffffa008feb0
.data_addr = 0xffffffffa0086f10
.devexit.text_addr = 0xffffffffa0082502
.devinit.text_addr = 0xffffffffa0081210
.exit.text_addr = 0xffffffffa00828cc
.gnu.linkonce.this_module_addr = 0xffffffffa008fcb0
.init.text_addr = 0xffffffffa0006000
.note.gnu.build-id_addr = 0xffffffffa008292c
.parainstructions_addr = 0xffffffffa0084b18
.rodata_addr = 0xffffffffa0082950
.rodata.str1.1_addr = 0xffffffffa0083280
.smp_locks_addr = 0xffffffffa0084bc8
.strtab_addr = 0xffffffffa0086208
.symtab_addr = 0xffffffffa0084c60
__bug_table_addr = 0xffffffffa0084bd8
(y or n) y
Reading symbols from
/usr/lib/debug/lib/modules/2.6.29-clim-3-amd64/kernel/drivers/block/cciss.ko...warning:
section .strtab not found in
/usr/lib/debug/lib/modules/2.6.29-clim-3-amd64/kernel/drivers/block/cciss.ko
warning: section .symtab not found in
/usr/lib/debug/lib/modules/2.6.29-clim-3-amd64/kernel/drivers/block/cciss.ko
done.
(gdb) info line cciss_init_one
Line 3597 of "drivers/block/cciss.c"
starts at address 0xffffffffa0081210 <cciss_init_one>
and ends at 0xffffffffa0081227 <cciss_init_one+23>.
(gdb) dir /usr/src/linux-source-2.6.29-clim
Source directories searched: /usr/src/linux-source-2.6.29-clim:$cdir:$cwd
(gdb) info line drivers/block/cciss.c:3597
Line 3597 of "drivers/block/cciss.c"
starts at address 0xffffffffa0081210 <cciss_init_one>
and ends at 0xffffffffa0081227 <cciss_init_one+23>.
(gdb) l cciss_init_one
3592 * stealing all these major device numbers.
3593 * returns the number of block devices registered.
3594 */
3595 static int __devinit cciss_init_one(struct pci_dev *pdev,
3596 const struct pci_device_id *ent)
3597 {
3598 int i;
3599 int j = 0;
3600 int rc;
3601 int dac, return_code;
I haven't systematically tried this on every module symbol, but it seems
to work for every one I have tried in the 6.8.50.20090628-4 version, but
not in 6.8-4. This changeset [3] seems to be the one that fixes things.
But it's not a trivial backport to gdb-6.1. If I actually understood
the problem better, maybe I could just pull the relevant bits out of
that patch...
[1]: http://lkml.indiana.edu/hypermail/linux/kernel/0406.0/0802.html
[2]: http://sourceware.org/bugzilla/show_bug.cgi?id=10684
[3]:
http://sourceware.org/git/gitweb.cgi?p=gdb.git;a=commitdiff;h=b5830bae4e6...
--
+----------------------------------------------------------+
| John Wright <john.wright(a)hp.com> |
| HP Mission Critical OS Enablement & Solution Test (MOST) |
+----------------------------------------------------------+
5651
days inactive
5657
days old
devel@lists.crash-utility.osci.io
2 comments
3 participants
participants (3)
-
Bob Montgomery -
Dave Anderson -
Wright, John (Telco Linux, Fort Collins)