Something has changed with the vmlinux debuginfo file such that the currently-embedded gdb-7.3.1 version in the crash utility can no longer can find the debuginfo data for text symbols in Linux 3.9 kernels. I don't know whether it's due to a change in the kernel build procedure arguments or maybe the tools used, because there is no such problem as recently as 3.8.8-100.fc17. Taking crash out of the picture, the simple gdb "whatis" command should show the debuginfo data for kernel text symbols. For example, here things work OK with gdb-7.5.1-34.el7: # gdb /usr/lib/debug/lib/modules/3.9.0-0.55.el7.x86_64/vmlinux GNU gdb (GDB) Red Hat Enterprise Linux (7.5.1-34.el7) Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/lib/debug/lib/modules/3.9.0-0.55.el7.x86_64/vmlinux...done. (gdb) whatis sys_read type = long int (unsigned int, char *, size_t) (gdb) whatis schedule type = void (void) (gdb) whatis kmem_cache_alloc type = void *(struct kmem_cache *, gfp_t) (gdb) But here with gdb-7.3.1, the debuginfo data for text symbols cannot be determined: # /root/gdb-7.3.1/gdb/gdb /usr/lib/debug/lib/modules/3.9.0-0.55.el7.x86_64/vmlinux GNU gdb (GDB) 7.3.1 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/lib/debug/lib/modules/3.9.0-0.55.el7.x86_64/vmlinux...done. (gdb) whatis sys_read type = <text variable, no debug info> (gdb) whatis schedule type = <text variable, no debug info> (gdb) whatis kmem_cache_alloc type = <text variable, no debug info> (gdb) This in turn causes problems with the crash utility, for example, during initialization there is an x86_64-only "crash: cannot determine thread return address" message: # crash crash 6.1.6-1.el7 Copyright (C) 2002-2013 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.3.1 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... crash: cannot determine thread return address KERNEL: /usr/lib/debug/lib/modules/3.9.0-0.55.el7.x86_64/vmlinux DUMPFILE: /dev/crash CPUS: 4 DATE: Tue Apr 30 11:42:23 2013 UPTIME: 01:29:16 LOAD AVERAGE: 0.08, 0.03, 0.05 TASKS: 136 NODENAME: hp-dl320g5-02.lab.bos.redhat.com RELEASE: 3.9.0-0.55.el7.x86_64 VERSION: #1 SMP Mon Apr 29 14:09:40 EDT 2013 MACHINE: x86_64 (2394 Mhz) MEMORY: 8 GB PID: 4467 COMMAND: "crash" TASK: ffff8802164b0000 [THREAD_INFO: ffff8802197fc000] CPU: 1 STATE: TASK_RUNNING (ACTIVE) crash> The "whatis" command reflects the gdb issue: crash> whatis sys_read <text variable, no debug info> sys_read; crash> whatis schedule <text variable, no debug info> schedule; crash> whatis kmem_cache_alloc <text variable, no debug info> kmem_cache_alloc; crash> And the "dis -l" command does not show line numbers: crash> dis -l sys_read 0xffffffff8119b350 <sys_read>: data32 data32 data32 xchg %ax,%ax 0xffffffff8119b355 <sys_read+5>: push %rbp 0xffffffff8119b356 <sys_read+6>: mov %rsp,%rbp 0xffffffff8119b359 <sys_read+9>: push %r14 0xffffffff8119b35b <sys_read+11>: push %r13 0xffffffff8119b35d <sys_read+13>: mov %rdx,%r13 0xffffffff8119b360 <sys_read+16>: push %r12 0xffffffff8119b362 <sys_read+18>: mov %rsi,%r12 0xffffffff8119b365 <sys_read+21>: lea -0x28(%rbp),%rsi 0xffffffff8119b369 <sys_read+25>: push %rbx 0xffffffff8119b36a <sys_read+26>: sub $0x10,%rsp 0xffffffff8119b36e <sys_read+30>: callq 0xffffffff811b5cb0 ... For whatever reason, when loading kernel module debuginfo data into a crash session, even though gdb fails to determine what a text symbol is, it does find its line number (?): crash> mod -s xfs MODULE NAME SIZE OBJECT FILE ffffffffa0410fa0 xfs 844348 /lib/modules/3.9.0-0.55.el7.x86_64/kernel/fs/xfs/xfs.ko crash> whatis xfs_end_bio <text variable, no debug info> xfs_end_bio; crash> dis -l xfs_end_bio /usr/src/debug/kernel-3.9.0-0.55.el7/linux-3.9.0-0.55.el7.x86_64/fs/xfs/xfs_aops.c: 383 0xffffffffa037cd20 <xfs_end_bio>: data32 data32 data32 xchg %ax,%ax 0xffffffffa037cd25 <xfs_end_bio+5>: push %rbp /usr/src/debug/kernel-3.9.0-0.55.el7/linux-3.9.0-0.55.el7.x86_64/fs/xfs/xfs_aops.c: 387 0xffffffffa037cd26 <xfs_end_bio+6>: xor %edx,%edx /usr/src/debug/kernel-3.9.0-0.55.el7/linux-3.9.0-0.55.el7.x86_64/fs/xfs/xfs_aops.c: 383 0xffffffffa037cd28 <xfs_end_bio+8>: mov %rsp,%rbp 0xffffffffa037cd2b <xfs_end_bio+11>: push %rbx ... As it turns out, I am in the process of doing the always-painful gdb upgrade procedure, which will be required for supporting the ARM64 architecture. The FSF finally released gdb-7.6 last Friday, and I am working on forward-porting the gdb-7.3.1.patch file to gdb-7.6. So unless there's another simple gdb-7.3.1 fix that can somehow be a workaround until gdb-7.6 is folded in, the new version of gdb is probably going to be the ultimate solution. Dave