Hi Everyone, I am just trying load a kernel and System.map file into crash. crash vmlinux System.map I get: crash 5.1.7 Copyright (C) 2002-2011 Red Hat, Inc. Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.0 Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"... crash: vmlinux: no debugging data available However, the System.map file reads fine and “nm vmlinux” shows the symbols as well. A side note that this is a linux 2.4 kernel. Is linux 2.4 not supported or what am I doing wrong?

Hi Dave, thanks for your answer. The basic procedure forgotten, I assumed -g would be included. Never mind, I compile the kernel with -g. However, crash still doesn't like it:

einoud@superchi:~/Pretoria/pretoria/kernel2.4/build/emPC/GREYHOUND/lin ux-2.4.37.9$ crash 5.1.8 Copyright (C) 2002-2011 Red Hat, Inc. Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.0 Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"... crash: read error: kernel virtual address: c032e840 type: "xtime" I did some stepping through crash, maybe that'll be helpful as to see what happened? crash 5.1.8 Copyright (C) 2002-2011 Red Hat, Inc. Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.0 Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"... please wait... (patching 13083 gdb minimal_symbol values) Breakpoint 1, readmem (addr=3224561728, memtype=1, buffer=0x851c5d8, size=8, type=0x840c9c6 "xtime", error_handle=1) at memory.c:1849 1849 { (gdb) step 1856 if (CRASHDEBUG(4)) (gdb) step 1849 { (gdb) step 1856 if (CRASHDEBUG(4)) (gdb) step 1863 if (size <= 0) { (gdb) step 1861 bufptr = (char *)buffer; (gdb) step 1863 if (size <= 0) { (gdb) step 1870 fd = REMOTE_MEMSRC() ? pc->sockfd : (ACTIVE() ? pc->mfd : pc->dfd); (gdb) step 1875 switch (memtype) (gdb) p fd $1 = 6 (gdb) step 1891 if (LKCD_DUMPFILE()) (gdb) step 1894 if (!IS_KVADDR(addr)) { (gdb) step generic_is_kvaddr (addr=3224561728) at memory.c:1739 1739 { (gdb) step 1741 } (gdb) step generic_is_kvaddr (addr=3224561728) at memory.c:1739 1739 { (gdb) step generic_is_kvaddr (addr=3224561728) at memory.c:1741 1741 } (gdb) step readmem (addr=3224561728, memtype=1, buffer=0x851c5d8, size=8, type=0x840c9c6 "xtime", error_handle=1) at memory.c:1921 1921 if (!kvtop(CURRENT_CONTEXT(), addr, &paddr, 0)) { (gdb) step 1910 switch (memtype) (gdb) step 1921 if (!kvtop(CURRENT_CONTEXT(), addr, &paddr, 0)) { (gdb) step kvtop (tc=0x0, kvaddr=3224561728, paddr=0xbfffdc20, verbose=0) at memory.c:2615 2615 { (gdb) step 2618 return (machdep->kvtop(tc ? tc : CURRENT_CONTEXT(), kvaddr, (gdb) step 2615 { (gdb) step 2618 return (machdep->kvtop(tc ? tc : CURRENT_CONTEXT(), kvaddr, (gdb) step x86_kvtop (tc=0x0, kvaddr=3224561728, paddr=0xbfffdc20, verbose=0) at x86.c:2753 2753 if (!IS_KVADDR(kvaddr)) (gdb) step 2743 { (gdb) step 2753 if (!IS_KVADDR(kvaddr)) (gdb) step generic_is_kvaddr (addr=3224561728) at memory.c:1739 1739 { (gdb) step 1741 } (gdb) step generic_is_kvaddr (addr=0) at memory.c:1739 1739 { (gdb) step generic_is_kvaddr (addr=0) at memory.c:1741 1741 } (gdb) step x86_kvtop (tc=0x0, kvaddr=3224561728, paddr=0xbfffdc20, verbose=0) at x86.c:2756 2756 if (XEN_HYPER_MODE()) { (gdb) step 2763 if (!vt->vmalloc_start) { (gdb) step 2764 *paddr = VTOP(kvaddr); (gdb) step 2765 return TRUE; (gdb) step 2862 } (gdb) step kvtop (tc=0x0, kvaddr=3224561728, paddr=0xbfffdc20, verbose=0) at memory.c:2620 2620 } (gdb) step readmem (addr=3336256, memtype=1, buffer=0x851c5d8, size=8, type=0x840c9c6 "xtime", error_handle=1) at memory.c:1947 1947 cnt = PAGESIZE() - PAGEOFFSET(paddr); (gdb) step 1952 if (CRASHDEBUG(8)) (gdb) step 1956 if (memtype == KVADDR) (gdb) step 1957 pc->curcmd_flags |= MEMTYPE_KVADDR; (gdb) step 1961 switch (READMEM(fd, bufptr, cnt, (gdb) step read_dev_mem (fd=6, bufptr=0x851c5d8, cnt=8, addr=3224561728, paddr=3336256) at memory.c:2027 2027 { (gdb) step 2030 if (!machdep->verify_paddr(paddr)) { (gdb) step generic_verify_paddr (paddr=3336256) at memory.c:2020 2020 } (gdb) step read_dev_mem (fd=6, bufptr=0x851c5d8, cnt=8, addr=3224561728, paddr=3336256) at memory.c:2043 2043 if (vt->high_memory && (paddr >= (physaddr_t)(VTOP(vt->high_memory)))) { (gdb) step 2049 if (lseek(fd, (off_t)paddr, SEEK_SET) == -1) (gdb) step 2026 read_dev_mem(int fd, void *bufptr, int cnt, ulong addr, physaddr_t paddr) (gdb) step 2053 errno = 0; (gdb) step 2054 readcnt = read(fd, bufptr, cnt); (gdb) step read (fd=6, bufptr=0x851c5d8, cnt=8, addr=3224561728, paddr=3336256) at /usr/include/bits/unistd.h:45 45 return __read_alias (__fd, __buf, __nbytes); (gdb) step read_dev_mem (fd=6, bufptr=0x851c5d8, cnt=8, addr=3224561728, paddr=3336256) at memory.c:2056 2056 if ((readcnt != cnt) && CRASHDEBUG(4)) { (gdb) step 2054 readcnt = read(fd, bufptr, cnt); (gdb) step read (fd=6, bufptr=0x851c5d8, cnt=8, addr=3224561728, paddr=3336256) at /usr/include/bits/unistd.h:45 45 return __read_alias (__fd, __buf, __nbytes); (gdb) step read_dev_mem (fd=6, bufptr=0x851c5d8, cnt=8, addr=3224561728, paddr=3336256) at memory.c:2056 2056 if ((readcnt != cnt) && CRASHDEBUG(4)) { (gdb) step 2089 if (readcnt != cnt) (gdb) step 2079 if ((readcnt != cnt) && readcnt && (machdep->flags & DEVMEMRD) && (gdb) step 2093 } (gdb) step readmem (addr=<value optimized out>, memtype=1, buffer=0x851c5d8, size=8, type=0x840c9c6 "xtime", error_handle=1) at memory.c:1962 1962 (memtype == PHYSADDR) || (memtype == XENMACHADDR) ? 0 : addr, paddr)) (gdb) step 1970 if (PRINT_ERROR_MESSAGE) (gdb) step 1971 error(INFO, READ_ERRMSG, memtype_string(memtype, 0), addr, type); (gdb) step memtype_string (memtype=1, debug=0) at memory.c:2325 2325 switch (memtype) (gdb) step 2331 sprintf(membuf, debug ? "KVADDR" : "kernel virtual"); (gdb) step 2340 sprintf(membuf, debug ? "FILEADDR" : "dumpfile"); (gdb) step sprintf (memtype=<value optimized out>, debug=0) at /usr/include/bits/stdio2.h:34 34 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, (gdb) step memtype_string (memtype=<value optimized out>, debug=<value optimized out>) at memory.c:2351 2351 } (gdb) step __error (type=1, fmt=0x83c90d8 "read error: %s address: %llx type: \"%s\"\n") at tools.c:41 41 { (gdb) step 48 if (CRASHDEBUG(1) || (pc->flags & DROP_CORE)) { (gdb) step 41 { (gdb) step 45 void *retaddr[NUMBER_STACKFRAMES] = { 0 }; (gdb) step 48 if (CRASHDEBUG(1) || (pc->flags & DROP_CORE)) { (gdb) step 45 void *retaddr[NUMBER_STACKFRAMES] = { 0 }; (gdb) step 48 if (CRASHDEBUG(1) || (pc->flags & DROP_CORE)) { (gdb) step 40 __error(int type, char *fmt, ...) (gdb) step 55 (void)vsnprintf(buf, BUFSIZE, fmt, ap); (gdb) step vsnprintf (type=1, fmt=0x83c90d8 "read error: %s address: %llx type: \"%s\"\n") at /usr/include/bits/stdio2.h:78 78 return __builtin___vsnprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1, (gdb) step __error (type=1, fmt=0x83c90d8 "read error: %s address: %llx type: \"%s\"\n") at tools.c:58 58 if (!fmt && FATAL_ERROR(type)) { (gdb) step 63 end_of_line = FATAL_ERROR(type) && !(pc->flags & RUNTIME); (gdb) step 65 if ((new_line = (buf[0] == '\n'))) (gdb) step 67 else if (pc->flags & PLEASE_WAIT) (gdb) step 70 if (type == CONT) (gdb) step 67 else if (pc->flags & PLEASE_WAIT) (gdb) step 70 if (type == CONT) (gdb) step 75 if (pc->stdpipe) { (gdb) step 85 fprintf(stdout, "%s%s%s %s%s", (gdb) step fprintf (type=1, fmt=0x83c90d8 "read error: %s address: %llx type: \"%s\"\n") at /usr/include/bits/stdio2.h:98 98 return __fprintf_chk (__stream, __USE_FORTIFY_LEVEL - 1, __fmt, (gdb) step crash: read error: kernel virtual address: c032e840 type: "xtime" __error (type=1, fmt=0x83c90d8 "read error: %s address: %llx type: \"%s\"\n") at tools.c:92 92 fflush(stdout); (gdb) step 95 if ((fp != stdout) && (fp != pc->stdpipe)) { (gdb) step 105 if ((pc->flags & DROP_CORE) && (type != NOTE)) { (gdb) step 111 switch (type) (gdb) step 133 } (gdb) step readmem (addr=3224561728, memtype=1, buffer=0x851c5d8, size=8, type=0x840c9c6 "xtime", error_handle=<value optimized out>) at memory.c:1992 1992 switch (error_handle) (gdb) step 1995 if ((pc->flags & DEVMEM) && (kt->flags & PRE_KERNEL_INIT) && (gdb) step 1996 devmem_is_restricted() && switch_to_proc_kcore()) (gdb) step devmem_is_restricted (addr=3224561728, memtype=1, buffer=0x851c5d8, size=8, type=0x840c9c6 "xtime", error_handle=1) at memory.c:2137 2137 if (STREQ(pc->live_memsrc, "/dev/mem") && (gdb) step readmem (addr=3224561728, memtype=1, buffer=0x851c5d8, size=8, type=0x840c9c6 "xtime", error_handle=1) at memory.c:1996 1996 devmem_is_restricted() && switch_to_proc_kcore()) (gdb) step devmem_is_restricted (addr=3224561728, memtype=1, buffer=0x851c5d8, size=8, type=0x840c9c6 "xtime", error_handle=1) at memory.c:2138 2138 kernel_symbol_exists("devmem_is_allowed") && (gdb) step kernel_symbol_exists (symbol=0x83c63dc "devmem_is_allowed") at symbols.c:4722 4722 if ((sp = symname_hash_search(symbol))) (gdb) step symname_hash_search (name=0x83c63dc "devmem_is_allowed") at symbols.c:889 889 sp = st->symname_hash[SYMNAME_HASH_INDEX(name)]; (gdb) step 891 while (sp) { (gdb) step 892 if (STREQ(sp->name, name)) (gdb) step 894 sp = sp->name_hash_next; (gdb) step 891 while (sp) { (gdb) step 892 if (STREQ(sp->name, name)) (gdb) step 894 sp = sp->name_hash_next; (gdb) step 891 while (sp) { (gdb) step 892 if (STREQ(sp->name, name)) (gdb) step 894 sp = sp->name_hash_next; (gdb) step 891 while (sp) { (gdb) step 892 if (STREQ(sp->name, name)) (gdb) step 894 sp = sp->name_hash_next; (gdb) step 891 while (sp) { (gdb) step 892 if (STREQ(sp->name, name)) (gdb) step 894 sp = sp->name_hash_next; (gdb) step 891 while (sp) { (gdb) step 892 if (STREQ(sp->name, name)) (gdb) step 894 sp = sp->name_hash_next; (gdb) step 891 while (sp) { (gdb) step 892 if (STREQ(sp->name, name)) (gdb) step 894 sp = sp->name_hash_next; (gdb) step 891 while (sp) { (gdb) step 892 if (STREQ(sp->name, name)) (gdb) step 894 sp = sp->name_hash_next; (gdb) step 891 while (sp) { (gdb) step 892 if (STREQ(sp->name, name)) (gdb) step 894 sp = sp->name_hash_next; (gdb) step 891 while (sp) { (gdb) step 892 if (STREQ(sp->name, name)) (gdb) step 894 sp = sp->name_hash_next; (gdb) step 891 while (sp) { (gdb) step 892 if (STREQ(sp->name, name)) (gdb) step 894 sp = sp->name_hash_next; (gdb) step 891 while (sp) { (gdb) step 892 if (STREQ(sp->name, name)) (gdb) step 894 sp = sp->name_hash_next; (gdb) step 10 891 while (sp) { (gdb) step 20 894 sp = sp->name_hash_next; (gdb) step 891 while (sp) { (gdb) step 100 Program exited with code 01. Thanks, Reinoud. -----Original Message----- From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson Sent: Thursday, September 15, 2011 6:07 AM To: Discussion list for crash utility usage, maintenance and development Subject: Re: [Crash-utility] crash cannot read the symbols ----- Original Message ----- > Hi Everyone, > > I am just trying load a kernel and System.map file into crash. > > crash vmlinux System.map > > I get: > > crash 5.1.7 > Copyright (C) 2002-2011 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) > 1999-2006 > Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu Limited > Copyright > (C) 2006, 2007 VA Linux Systems Japan K.K. > Copyright (C) 2005 NEC Corporation > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public > License, and you are welcome to change it and/or distribute copies > of it under certain conditions. Enter "help copying" to see the > conditions. > This program has absolutely no warranty. Enter "help warranty" for > details. > > GNU gdb (GDB) 7.0 > Copyright (C) 2009 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show > copying" > and "show warranty" for details. > This GDB was configured as "i686-pc-linux-gnu"... > > crash: vmlinux: no debugging data available > > However, the System.map file reads fine and “nm vmlinux” shows the > symbols as well. > > A side note that this is a linux 2.4 kernel. > Is linux 2.4 not supported or what am I doing wrong? Linux 2.4 is supported, but either the vmlinux file was not originally built with the -g CFLAG, or it had the debuginfo data stripped. Without the debuginfo data, the vmlinux is useless. If by chance it is a RHEL3 2-4-based kernel, there is an associated vmlinux.debug file that comes with the separate kernel-debuginfo rpm package, so for example, you'd end up with two files: vmlinux-2.4.21-47.EL and vmlinux-2.4.21-47.EL.debug that would be installed here: /boot/vmlinux-2.4.21-47.EL /usr/lib/debug/boot/vmlinux-2.4.21-47.EL.debug If the files are found in locations above, running on a live system they would be found automatically when you enter: $ crash Alternatively you could just throw both files on the command line: $ crash <path-to>/vmlinux-2.4.21-47.EL <path-to>/vmlinux-2.4.21-47.EL.debug That all being said, since you are trying "crash vmlinux" I'm guessing that you are using a vmlinux file from another distro, or perhaps building your own kernel. If building your own kernel, the CFLAGS in the top-level Makefile needs to contain -g. In linux 2.4, I don't believe that there was a CONFIG_DEBUG_INFO option that could be configured to do it automatically, so you may have to change it manually, and then rebuild the kernel. Presuming that the original kernel that you're trying to analyze was booted from the same kernel version but was not built with -g, then you would also have to put the System.map file from the original kernel on the command line as you have done. If the vmlinux file is from another distro, mention that, and perhaps someone from the list can help. Dave -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

Hmm, the /dev/mem does not reflect the kernel and symbols I am trying to read, because I do not have a core dump of the crash. I just tried to read the kernel and modules in crash to read it.

For live system analysis, the physical memory source must be one of the following devices: /dev/mem /proc/kcore /dev/crash If the live system kernel was configured with CONFIG_STRICT_DEVMEM, then /dev/mem cannot be used. If the live system kernel was configured without CONFIG_PROC_KCORE, or if /proc/kcore is non-functional, then /proc/kcore cannot be used. The third alternative is this /dev/crash driver. Presuming that /lib/modules/`uname -r`/build points to a kernel build tree or kernel "devel" package tree, the module can simply be built and installed like so: # make ... # insmod crash.ko Once installed, the /dev/crash driver will be used by default for live system crash sessions.