8:16 a.m.
Hi Bob,
OK, I'm currently provisioning a fresh Fedora 16 system and
I'll retry it.
I wonder if it has something to do with this crash-6.0.1 update:
- If the "--mod <directory-tree>" command line option, or the
setting of the CRASH_MODULE_PATH environment variable, or the
"mod -S <directory-tree>" point to a tree that contains only the
separate debuginfo "<module>.ko.debug" files, then those
debuginfo files will be used as the internal "add-symbol-file"
arguments to the embedded gdb module. Without the patch, it was
only acceptable to point to a directory tree that contained the
base "<module>.ko" files, and the separate debuginfo files
were found automatically based upon the directory path to the
base module file. This will allow an alternate module-debuginfo
directory tree to be set up like so:
# cd <directory>
# rpm2cpio kernel-debuginfo-<release>.rpm | cpio -idv
Having done that, the <directory> may be used with the "--mod",
command line argument, or as the CRASH_MODULE_PATH environment
variable, or as the "mod -S <directory> argument.
(anderson(a)redhat.com)
I'll check it myself, but I wonder what happens if you pass "mod -S"
the directory tree containing the <module>.ko.debug files instead of
the base <module>.ko files? It seems to be acting as if line between
the base <module>.ko and its <module>.ko.debug is not being recognized.
But then again, why would it work when individually loaded?
BTW, how's the kmem patch looking? I'm guessing it's more involved
than you first thought?
Thanks,
Dave
----- Original Message -----
On Thu, 2012-01-26 at 15:24 -0500, Dave Anderson wrote:
>
> ----- Original Message -----
> >
> >
> > ----- Original Message -----
> > > I've reverted back to crash-5.1.9 and applied my kmem patch to
> > > that for
> > > our use here. We're using a 3.1-based kernel, and need the
> > > kmem patch
> > > so crash can deal with the change in CONFIG_SLAB, but we're
> > > building
> > > with gcc-4.4.5 and don't really need the new gdb in
> > > crash-6.0.2, and
> > > crash-6.0.2 is not giving module source line numbers for us
> > > with "dis -l".
> > >
> > > This is just a heads up. I don't know why 6.0.2 is failing
> > > this, and
> > > since I found the last module source line number problem, it's
> > > not my
> > > turn ;-)
> > >
> > > Bob Montgomery
> >
> > What kernel version? I'll try to reproduce it.
Here's your test on my system:
crash-6.0.2> help | grep version
crash-6.0.2 version: 6.0.2 gdb version: 7.3.1
crash-6.0.2> help -k | grep proc_version
proc_version: Linux version 3.1.4-clim-2-amd64 (buildd@hpdebuild)
(gcc
version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011
crash-6.0.2> mod -s bnx2
/usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko
MODULE NAME SIZE OBJECT FILE
ffffffffa01f85e0 bnx2 65655
/usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
6265
0xffffffffa01f32e1 <bnx2_open>: push %rbp
0xffffffffa01f32e2 <bnx2_open+1>: mov %rsp,%rbp
0xffffffffa01f32e5 <bnx2_open+4>: push %r15
0xffffffffa01f32e7 <bnx2_open+6>: push %r14
0xffffffffa01f32e9 <bnx2_open+8>: push %r13
0xffffffffa01f32eb <bnx2_open+10>: push %r12
0xffffffffa01f32ed <bnx2_open+12>: mov %rdi,%r12
0xffffffffa01f32f0 <bnx2_open+15>: push %rbx
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
6266
0xffffffffa01f32f1 <bnx2_open+16>: lea 0x640(%rdi),%rbx
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
6265
0xffffffffa01f32f8 <bnx2_open+23>: sub $0x8,%rsp
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
6269
Ummmm. Ooops? Is it time to prepare my embarrassing explanation of
my
prior claim???
=====================
First start over and do what I actually was doing when I saw the
problem:
$ crash-6.0.2 --no_kmem_cache dump.201201062015 kernel_link
...
crash-6.0.2> help | grep version
crash-6.0.2 version: 6.0.2 gdb version: 7.3.1
crash> help -k | grep proc_version
proc_version: Linux version 3.1.4-clim-2-amd64 (buildd@hpdebuild)
(gcc
version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011
crash-6.0.2> mod -S /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64
...
crash-6.0.2> mod | grep sunrpc
ffffffffa01cab80 auth_rpcgss
40572
/usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko
ffffffffa03a7070 sunrpc
202679
/usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/net/sunrpc/sunrpc.ko
crash-6.0.2> dis -l rpc_sleep_on_priority
0xffffffffa038f71b <rpc_sleep_on_priority>: push %rbp
0xffffffffa038f71c <rpc_sleep_on_priority+1>: mov %rsp,%rbp
0xffffffffa038f71f <rpc_sleep_on_priority+4>: push %r12
0xffffffffa038f721 <rpc_sleep_on_priority+6>: mov %ecx,%r12d
0xffffffffa038f724 <rpc_sleep_on_priority+9>: push %rbx
0xffffffffa038f725 <rpc_sleep_on_priority+10>: mov %rdi,%rbx
0xffffffffa038f728 <rpc_sleep_on_priority+13>: sub $0x10,%rsp
0xffffffffa038f72c <rpc_sleep_on_priority+17>: mov
0x70(%rsi),%rax
0xffffffffa038f730 <rpc_sleep_on_priority+21>: test $0x4,%al
0xffffffffa038f732 <rpc_sleep_on_priority+23>: jne
0xffffffffa038f738
0xffffffffa038f734 <rpc_sleep_on_priority+25>: ud2
Well, that's a relief...
Oh, by the way, now that I've used mod -S to load all modules instead
of
individually loading bnx2.ko:
crash-6.0.2> mod | grep bnx2
ffffffffa01f85e0 bnx2 65655
/usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko
crash-6.0.2> dis -l bnx2_open
0xffffffffa01f32e1 <bnx2_open>: push %rbp
0xffffffffa01f32e2 <bnx2_open+1>: mov %rsp,%rbp
0xffffffffa01f32e5 <bnx2_open+4>: push %r15
0xffffffffa01f32e7 <bnx2_open+6>: push %r14
0xffffffffa01f32e9 <bnx2_open+8>: push %r13
0xffffffffa01f32eb <bnx2_open+10>: push %r12
0xffffffffa01f32ed <bnx2_open+12>: mov %rdi,%r12
0xffffffffa01f32f0 <bnx2_open+15>: push %rbx
0xffffffffa01f32f1 <bnx2_open+16>: lea 0x640(%rdi),%rbx
0xffffffffa01f32f8 <bnx2_open+23>: sub $0x8,%rsp
0xffffffffa01f32fc <bnx2_open+27>: callq 0xffffffff8129477b
<netif_carrier_off>
No source lines.
===================
If I do the same steps on my 5.1.9 version of crash, I get:
crash> help | grep version
crash version: 5.1.9 gdb version: 7.0
crash> help -k | grep proc_version
proc_version: Linux version 3.1.4-clim-2-amd64 (buildd@hpdebuild)
(gcc
version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011
crash> mod -S /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64
...
crash> dis -l rpc_sleep_on_priority
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/net/sunrpc/sched.c:
350
0xffffffffa038f71b <rpc_sleep_on_priority>: push %rbp
0xffffffffa038f71c <rpc_sleep_on_priority+1>: mov %rsp,%rbp
0xffffffffa038f71f <rpc_sleep_on_priority+4>: push %r12
0xffffffffa038f721 <rpc_sleep_on_priority+6>: mov %ecx,%r12d
0xffffffffa038f724 <rpc_sleep_on_priority+9>: push %rbx
0xffffffffa038f725 <rpc_sleep_on_priority+10>: mov %rdi,%rbx
0xffffffffa038f728 <rpc_sleep_on_priority+13>: sub $0x10,%rsp
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/arch/x86/include/asm/bitops.h:
312
0xffffffffa038f72c <rpc_sleep_on_priority+17>: mov
0x70(%rsi),%rax
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/net/sunrpc/sched.c:
352
0xffffffffa038f730 <rpc_sleep_on_priority+21>: test $0x4,%al
0xffffffffa038f732 <rpc_sleep_on_priority+23>: jne
0xffffffffa038f738 <rpc_sleep_on_priority+29>
0xffffffffa038f734 <rpc_sleep_on_priority+25>: ud2a
So 5.1.9 works, and 6.0.2 fails if I load all modules with mod -S.
That's interesting.
Bob Montgomery