Hi, I searched the archives and found that you've discussed an issue I'm seeing with x86_64 kernels where crash doesn't have line numbers for modules' symbols: https://www.redhat.com/archives/crash-utility/2008-January/msg00021.html I'm using crash-4.0-6.3 on a RHEL5U1 x86_64 system with a custom 2.6.22.19 kernel. Given that the RHEL5U1 x86_64 kernels clearly do provide accurate line numbers for modules, has anyone identified how that is? I have to believe the redhat kernel is patched to fix this issue. I looked over the various redhat patches that are applied to RHEL5's 2.6.18 sources but can't see a patch that stands out as specifically addressing this x86_64 issue. But I could easily be overlooking some patch. please advise, thanks. Mike ps. please cc me as I've not yet been able to join the list

On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson <anderson(a)redhat.com&gt; wrote: > Mike Snitzer wrote: > > >> Hi, >> >> I searched the archives and found that you've discussed an issue I'm >> seeing with x86_64 kernels where crash doesn't have line numbers for >> modules' symbols: >> https://www.redhat.com/archives/crash-utility/2008-January/msg00021.html >> >> I'm using crash-4.0-6.3 on a RHEL5U1 x86_64 system with a custom >> 2.6.22.19 kernel. Given that the RHEL5U1 x86_64 kernels clearly do >> provide accurate line numbers for modules, has anyone identified how >> that is? I have to believe the redhat kernel is patched to fix this >> issue. >> >> I looked over the various redhat patches that are applied to RHEL5's >> 2.6.18 sources but can't see a patch that stands out as specifically >> addressing this x86_64 issue. But I could easily be overlooking some >> patch. >> >> please advise, thanks. >> Mike >> >> ps. please cc me as I've not yet been able to join the list >> >> >> > You're on the list now... > > Anyway, there are no kernel patches applied to RHEL5 kernels > to make it work that I'm aware of. That being said, I don't > have an answer as to why they don't work for modules on your > custom kernel (or why they don't work in RHEL4 kernels...). > I configured my x86_64 kernel to be relocatable. Could this be the reason for modules' line number debug data being off? From: http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5 "Using /proc/kallsyms or a --reloc=[size] as a command line argument is similar to using a System.map file, in that it results in the loss of the use of line number debug data."

Still doesn't explain why RHEL4 isn't happy in this regard either... I'm trying a non-relocatable variant of my kernel now and will report back with the results.

On Wed, Apr 30, 2008 at 7:41 PM, Dave Anderson <anderson(a)redhat.com&gt; wrote: > Mike Snitzer wrote: > > >> On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson <anderson(a)redhat.com&gt; >> >>> Anyway, there are no kernel patches applied to RHEL5 kernels >>> to make it work that I'm aware of. That being said, I don't >>> have an answer as to why they don't work for modules on your >>> custom kernel (or why they don't work in RHEL4 kernels...). >>> >>> >>> >> I configured my x86_64 kernel to be relocatable. Could this be the >> reason for modules' line number debug data being off? From: >> http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5 >> >> "Using /proc/kallsyms or a --reloc=[size] as a command line argument >> is similar to using a System.map file, in that it results in the loss >> of the use of line number debug data." >> >> >> > No, that particular issue only applies to the base kernel, and is > irrelevant with respect to modules. > (Note that the RHEL5 kernel is relocatable as well.) > OK, yeah making the base kernel non-relocatable didn't help. > What you might confirm is whether you're getting any debuginfo data when > you load the module? Are you sure that your modules are being built with -g? > How would I confirm that I'm getting debuginfo data when I 'mod -s ext3'? I'm using an spec file that is based heavily (at least relative to debuginfo) on the redhat kernel-2.6.spec but I'll have to verify that I'm building with -g... Mike

On Wed, Apr 30, 2008 at 8:21 PM, Dave Anderson <anderson(a)redhat.com&gt; wrote: > Mike Snitzer wrote: > > >> On Wed, Apr 30, 2008 at 7:41 PM, Dave Anderson <anderson(a)redhat.com&gt; >> > wrote: > >> >>> Mike Snitzer wrote: >>> >>> >>> >>> >>>> On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson <anderson(a)redhat.com&gt; >>>> >>>> >>>> >> >> >>>>> Anyway, there are no kernel patches applied to RHEL5 kernels >>>>> to make it work that I'm aware of. That being said, I don't >>>>> have an answer as to why they don't work for modules on your >>>>> custom kernel (or why they don't work in RHEL4 kernels...). >>>>> >>>>> >>>>> >>>>> >>>>> >>>> I configured my x86_64 kernel to be relocatable. Could this be the >>>> reason for modules' line number debug data being off? From: >>>> http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5 >>>> >>>> "Using /proc/kallsyms or a --reloc=[size] as a command line argument >>>> is similar to using a System.map file, in that it results in the loss >>>> of the use of line number debug data." >>>> >>>> >>>> >>>> >>>> >>> No, that particular issue only applies to the base kernel, and is >>> irrelevant with respect to modules. >>> (Note that the RHEL5 kernel is relocatable as well.) >>> >>> >>> >> OK, yeah making the base kernel non-relocatable didn't help. >> >> >> >> >>> What you might confirm is whether you're getting any debuginfo data >>> > when > >>> you load the module? Are you sure that your modules are being built >>> > with -g? > >>> >> How would I confirm that I'm getting debuginfo data when I 'mod -s >> ext3'? I'm using an spec file that is based heavily (at least >> relative to debuginfo) on the redhat kernel-2.6.spec but I'll have to >> verify that I'm building with -g... >> >> Mike >> >> >> > So your build procedure ends up with a kernel-debuginfo package that > contains a set of "split" <module-name>.ko.debug files? > Yes. > If you take the module filename displayed after the "mod -s" is done, > and run it through "readelf -a", you'll see a .gnu_debuglink section. > My kernel's ext3.ko has: [23] .gnu_debuglink PROGBITS 0000000000000000 00036558 0000000000000014 0000000000000000 0 0 4 which would appear fairly comparable to what the RHEL5 kernel's ext3.ko has: [22] .gnu_debuglink PROGBITS 0000000000000000 00037418 0000000000000014 0000000000000000 0 0 4 > That /lib/modules/<path-to>/module.ko module will have > a link to its partner in > /usr/lib/debug/lib/modules/<path-to>/module.ko.debug. > readelf -x $SECTION /lib/modules/<path-to>/ext3.ko shows that .gnu_debuglink points to ext3.ko.debug as expected. Given the "so far so good" nature of what you're helping me verify here it still begs the question: so why doesn't it work on my 2.6.22.19 x86_64 kernel? ;) Where does the line number debug data get stored in the module.ko.debug? In my case the source comes up as "include/linux/clocksource.h: 121" for all kernel modules' symbols. Any further pointers would be greatly appreciated, thanks a lot for your time already! Mike