3 p.m.
----- "Eric W. Biederman" <ebiederm(a)xmission.com> wrote:
Mike Snitzer <snitzer(a)gmail.com> writes:
> [I've trimmed the wide cc distribution that was inherited when I
> forked a different thread]
>
> On Mon, Jan 12, 2009 at 10:19 PM, Eric W. Biederman
> <ebiederm(a)xmission.com> wrote:
>> "Mike Snitzer" <snitzer(a)gmail.com> writes:
>>>
>>> Now if only I could fix line numbers when debugging crashes in x86_64
>>> modules with the crash utility! :)
>>
>> It's a userspace problem...
>>
>> All of the little usability things are userspace problems.
>>
>> I won't claim that it is trivial because it is a userspace problem, at the
same
>> time there is no reason to wait for any kernel features to merge etc. Someone
>> just has to scratch an itch and go fix it.
>
> Yes, the crash utility (userspace) is clearly having problems getting
> line number for symbols in x86_64 modules. But I finally took some
> time to bisect the point in the kernel where the crash utility first
> started to fail, it appears to be:
>
> commit 7460ed2844ffad7141e30271c0c3da8336e66014
> Author: john stultz <johnstul(a)us.ibm.com>
> Date: Fri Feb 16 01:28:21 2007 -0800
>
> I used version 4.0-7.6 of the crash utility to test if each commit was
> good or bad. I simply checked if ext3's module had correct line
> number info for the ext3_get_blocks_handle symbol with: sym
> ext3_get_blocks_handle
Weird. That patch doesn't appear to affect anything in that area.
So my stab in the dark is that there is something in vmlinux that
crash doesn't know how to cope with.
Actually it's not a problem with the vmlinux file, but rather with kernel
module object files. The crash utility has an embedded gdb module which
is invoked as "gdb vmlinux", and to get line numbers, the crash utility
simply uses the relevant built-in gdb function to get them. And line
numbers work fine with the base kernel code from the vmlinux file.
The debuginfo data of kernel modules can be subsequently added to the
crash session by doing a gdb "add-symbol-file" command for any or all
kernel modules. But getting correct line number information for kernel
modules has been a crap-shoot in the past, depending upon architecture
and/or kernel version. For example, they don't work with 2.6.9-based
RHEL4 x86_64 kernel modules, but work fine with 2.6.18-based RHEL5 x86_64
kernels.
Looking at Mike's suspect kernel patch list, I don't see anything that
would have any relationship to the issue. Perhaps there was a build tool
change during the same timeframe?
Dave
3:22 p.m.
New subject: BISECTED: Re: source line numbers with x86_64 modules?
On Tue, 27 Jan 2009 16:00:19 -0500 (EST)
Dave Anderson <anderson(a)redhat.com> wrote:
Actually it's not a problem with the vmlinux file, but rather with
kernel module object files. The crash utility has an embedded gdb
module which is invoked as "gdb vmlinux", and to get line numbers,
the crash utility simply uses the relevant built-in gdb function to
get them. And line numbers work fine with the base kernel code from
the vmlinux file.
if the entire goal is to get a pinpoint of where the crash is, just
using scripts/markup_oops.pl will do the right thing.
--
Arjan van de Ven Intel Open Source Technology Centre
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
3:55 p.m.
New subject: BISECTED: Re: source line numbers with x86_64 modules?
Dave Anderson <anderson(a)redhat.com> writes:
Actually it's not a problem with the vmlinux file, but rather
with kernel
module object files. The crash utility has an embedded gdb module which
is invoked as "gdb vmlinux", and to get line numbers, the crash utility
simply uses the relevant built-in gdb function to get them. And line
numbers work fine with the base kernel code from the vmlinux file.
The debuginfo data of kernel modules can be subsequently added to the
crash session by doing a gdb "add-symbol-file" command for any or all
kernel modules. But getting correct line number information for kernel
modules has been a crap-shoot in the past, depending upon architecture
and/or kernel version. For example, they don't work with 2.6.9-based
RHEL4 x86_64 kernel modules, but work fine with 2.6.18-based RHEL5 x86_64
kernels.
Looking at Mike's suspect kernel patch list, I don't see anything that
would have any relationship to the issue. Perhaps there was a build tool
change during the same timeframe?
It look like Mike just built a series of kernels and had a problem,
which should preclude a tool change.
That said. Does this feature of crash work in 2.6.29? If not is
there enough interest to track this down, and fix it if it is a kernel
bug?
If we are going to be using these tools we need them working on the
latest and greatest kernels, not some weird enterprise branch, for
fuddy duddies.
Eric
5:41 p.m.
New subject: BISECTED: Re: source line numbers with x86_64 modules?
On Tue, Jan 27, 2009 at 4:55 PM, Eric W. Biederman
<ebiederm(a)xmission.com> wrote:
Dave Anderson <anderson(a)redhat.com> writes:
> Actually it's not a problem with the vmlinux file, but rather with kernel
> module object files. The crash utility has an embedded gdb module which
> is invoked as "gdb vmlinux", and to get line numbers, the crash utility
> simply uses the relevant built-in gdb function to get them. And line
> numbers work fine with the base kernel code from the vmlinux file.
>
> The debuginfo data of kernel modules can be subsequently added to the
> crash session by doing a gdb "add-symbol-file" command for any or all
> kernel modules. But getting correct line number information for kernel
> modules has been a crap-shoot in the past, depending upon architecture
> and/or kernel version. For example, they don't work with 2.6.9-based
> RHEL4 x86_64 kernel modules, but work fine with 2.6.18-based RHEL5 x86_64
> kernels.
>
> Looking at Mike's suspect kernel patch list, I don't see anything that
> would have any relationship to the issue. Perhaps there was a build tool
> change during the same timeframe?
It look like Mike just built a series of kernels and had a problem,
which should preclude a tool change.
That is correct, I just built/booted/tested a series of kernels one
after the other as part of the standard git bisect procedure. No
tools were changed.
That said. Does this feature of crash work in 2.6.29? If not is
there enough interest to track this down, and fix it if it is a kernel
bug?
If we are going to be using these tools we need them working on the
latest and greatest kernels, not some weird enterprise branch, for
fuddy duddies.
This feature of crash does not work with 2.6.29, nor does it work with
any kernel I've tried >= 2.6.21. Andi Kleen shared with me that he
sees the same problem with a recent crash and 2.6.28.
AFAIK I found the regression point relative to linux (commit:
7460ed28). The verdict is clearly still out on where the actual bug
lives (linux vs crash).
Mike
5782
days inactive
5782
days old
devel@lists.crash-utility.osci.io
3 comments
4 participants
participants (4)
-
Arjan van de Ven -
Dave Anderson -
Eric W. Biederman -
Mike Snitzer