11:13 a.m.
Something has changed with the vmlinux debuginfo file such that
the currently-embedded gdb-7.3.1 version in the crash utility
can no longer can find the debuginfo data for text symbols in
Linux 3.9 kernels.
I don't know whether it's due to a change in the kernel build procedure
arguments or maybe the tools used, because there is no such problem as
recently as 3.8.8-100.fc17.
Taking crash out of the picture, the simple gdb "whatis" command
should show the debuginfo data for kernel text symbols. For example,
here things work OK with gdb-7.5.1-34.el7:
# gdb /usr/lib/debug/lib/modules/3.9.0-0.55.el7.x86_64/vmlinux
GNU gdb (GDB) Red Hat Enterprise Linux (7.5.1-34.el7)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/lib/debug/lib/modules/3.9.0-0.55.el7.x86_64/vmlinux...done.
(gdb) whatis sys_read
type = long int (unsigned int, char *, size_t)
(gdb) whatis schedule
type = void (void)
(gdb) whatis kmem_cache_alloc
type = void *(struct kmem_cache *, gfp_t)
(gdb)
But here with gdb-7.3.1, the debuginfo data for text symbols
cannot be determined:
# /root/gdb-7.3.1/gdb/gdb /usr/lib/debug/lib/modules/3.9.0-0.55.el7.x86_64/vmlinux
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/lib/debug/lib/modules/3.9.0-0.55.el7.x86_64/vmlinux...done.
(gdb) whatis sys_read
type = <text variable, no debug info>
(gdb) whatis schedule
type = <text variable, no debug info>
(gdb) whatis kmem_cache_alloc
type = <text variable, no debug info>
(gdb)
This in turn causes problems with the crash utility, for example,
during initialization there is an x86_64-only "crash: cannot determine
thread return address" message:
# crash
crash 6.1.6-1.el7
Copyright (C) 2002-2013 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
crash: cannot determine thread return address
KERNEL: /usr/lib/debug/lib/modules/3.9.0-0.55.el7.x86_64/vmlinux
DUMPFILE: /dev/crash
CPUS: 4
DATE: Tue Apr 30 11:42:23 2013
UPTIME: 01:29:16
LOAD AVERAGE: 0.08, 0.03, 0.05
TASKS: 136
NODENAME: hp-dl320g5-02.lab.bos.redhat.com
RELEASE: 3.9.0-0.55.el7.x86_64
VERSION: #1 SMP Mon Apr 29 14:09:40 EDT 2013
MACHINE: x86_64 (2394 Mhz)
MEMORY: 8 GB
PID: 4467
COMMAND: "crash"
TASK: ffff8802164b0000 [THREAD_INFO: ffff8802197fc000]
CPU: 1
STATE: TASK_RUNNING (ACTIVE)
crash>
The "whatis" command reflects the gdb issue:
crash> whatis sys_read
<text variable, no debug info> sys_read;
crash> whatis schedule
<text variable, no debug info> schedule;
crash> whatis kmem_cache_alloc
<text variable, no debug info> kmem_cache_alloc;
crash>
And the "dis -l" command does not show line numbers:
crash> dis -l sys_read
0xffffffff8119b350 <sys_read>: data32 data32 data32 xchg %ax,%ax
0xffffffff8119b355 <sys_read+5>: push %rbp
0xffffffff8119b356 <sys_read+6>: mov %rsp,%rbp
0xffffffff8119b359 <sys_read+9>: push %r14
0xffffffff8119b35b <sys_read+11>: push %r13
0xffffffff8119b35d <sys_read+13>: mov %rdx,%r13
0xffffffff8119b360 <sys_read+16>: push %r12
0xffffffff8119b362 <sys_read+18>: mov %rsi,%r12
0xffffffff8119b365 <sys_read+21>: lea -0x28(%rbp),%rsi
0xffffffff8119b369 <sys_read+25>: push %rbx
0xffffffff8119b36a <sys_read+26>: sub $0x10,%rsp
0xffffffff8119b36e <sys_read+30>: callq 0xffffffff811b5cb0
...
For whatever reason, when loading kernel module debuginfo data into
a crash session, even though gdb fails to determine what a text symbol
is, it does find its line number (?):
crash> mod -s xfs
MODULE NAME SIZE OBJECT FILE
ffffffffa0410fa0 xfs 844348
/lib/modules/3.9.0-0.55.el7.x86_64/kernel/fs/xfs/xfs.ko
crash> whatis xfs_end_bio
<text variable, no debug info> xfs_end_bio;
crash> dis -l xfs_end_bio
/usr/src/debug/kernel-3.9.0-0.55.el7/linux-3.9.0-0.55.el7.x86_64/fs/xfs/xfs_aops.c: 383
0xffffffffa037cd20 <xfs_end_bio>: data32 data32 data32 xchg %ax,%ax
0xffffffffa037cd25 <xfs_end_bio+5>: push %rbp
/usr/src/debug/kernel-3.9.0-0.55.el7/linux-3.9.0-0.55.el7.x86_64/fs/xfs/xfs_aops.c: 387
0xffffffffa037cd26 <xfs_end_bio+6>: xor %edx,%edx
/usr/src/debug/kernel-3.9.0-0.55.el7/linux-3.9.0-0.55.el7.x86_64/fs/xfs/xfs_aops.c: 383
0xffffffffa037cd28 <xfs_end_bio+8>: mov %rsp,%rbp
0xffffffffa037cd2b <xfs_end_bio+11>: push %rbx
...
As it turns out, I am in the process of doing the always-painful
gdb upgrade procedure, which will be required for supporting the
ARM64 architecture. The FSF finally released gdb-7.6 last Friday,
and I am working on forward-porting the gdb-7.3.1.patch file to
gdb-7.6.
So unless there's another simple gdb-7.3.1 fix that can somehow
be a workaround until gdb-7.6 is folded in, the new version of
gdb is probably going to be the ultimate solution.
Dave