# crash-ups vmlinux /dev/shm/vmcore-4.15.18-sp crash-ups 7.2.3++ Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: kernel relocated [426MB]: patching 60983 gdb minimal_symbol values KERNEL: vmlinux DUMPFILE: /dev/shm/vmcore-4.15.18-sp CPUS: 2 DATE: Wed Jun 13 15:44:26 2018 UPTIME: 00:04:26 LOAD AVERAGE: 1.78, 0.64, 0.23 TASKS: 109 NODENAME: rhel75b RELEASE: 4.15.18-sparse_ex VERSION: #3 SMP Tue Jun 12 17:41:53 EDT 2018 MACHINE: x86_64 (2400 Mhz) MEMORY: 2 GB PANIC: "sysrq: SysRq : Trigger a crash" PID: 2801 COMMAND: "bash" TASK: ffff89bbbcb0cc40 [THREAD_INFO: ffff89bbbcb0cc40] CPU: 1 STATE: TASK_RUNNING (SYSRQ) crash-ups> kmem -i PAGES TOTAL PERCENTAGE TOTAL MEM 480806 1.8 GB ---- FREE 30085 117.5 MB 6% of TOTAL MEM USED 450721 1.7 GB 93% of TOTAL MEM SHARED 8172 31.9 MB 1% of TOTAL MEM BUFFERS 1 4 KB 0% of TOTAL MEM *** Error in `crash-ups': corrupted double-linked list: 0x00000000062550f0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x7b8bd)[0x7f73b9f1d8bd] /lib64/libc.so.6(+0x7d3a5)[0x7f73b9f1f3a5] /lib64/libc.so.6(+0x7e87c)[0x7f73b9f2087c] /lib64/libc.so.6(realloc+0x1b2)[0x7f73b9f22752] crash-ups(xrealloc+0x1d)[0x78448d] crash-ups[0x6e7564] crash-ups[0x6f5e74] crash-ups[0x6fb194] crash-ups[0x6ffc48] crash-ups[0x700078] crash-ups[0x6777e1] crash-ups[0x679290] crash-ups[0x67292d] crash-ups(lookup_static_symbol_aux+0x7b)[0x672a8b] crash-ups(lookup_symbol_in_language+0x215)[0x672da5] crash-ups[0x610ca3] crash-ups(c_parse_internal+0xcf2)[0x6130f2] crash-ups(c_parse+0x159)[0x615a39] crash-ups[0x6ce90a] crash-ups(parse_exp_1+0x55)[0x6ceb05] crash-ups(parse_expression+0x19)[0x6ceb59] crash-ups(gdb_command_funnel+0x375)[0x676c35] crash-ups(gdb_interface+0xe5)[0x4f1b25] crash-ups(get_symbol_type+0x92)[0x52b262] crash-ups(get_array_length+0x492)[0x52de32] crash-ups[0x4945aa] crash-ups(cmd_kmem+0xd7a)[0x4a90ea] crash-ups(exec_command+0x2cc)[0x4681ec] crash-ups(main_loop+0x1ea)[0x46847a] crash-ups[0x6a4803] crash-ups(catch_errors+0x7a)[0x6a30aa] crash-ups[0x6a5796] crash-ups(catch_errors+0x7a)[0x6a30aa] crash-ups(gdb_main_entry+0x47)[0x6a5af7] crash-ups(main+0x775)[0x4667d5] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f73b9ec3c05] crash-ups[0x467a7e] ======= Memory map: ======== 00400000-00ae8000 r-xp 00000000 fd:02 1796492019 /home/share/git/crash/crash-upstream/crash 00ce7000-00ce8000 r--p 006e7000 fd:02 1796492019 /home/share/git/crash/crash-upstream/crash 00ce8000-00d0d000 rw-p 006e8000 fd:02 1796492019 /home/share/git/crash/crash-upstream/crash 00d0d000-00fc9000 rw-p 00000000 00:00 0 02bad000-06668000 rw-p 00000000 00:00 0 [heap] 7f73a4000000-7f73a4021000 rw-p 00000000 00:00 0 7f73a4021000-7f73a8000000 ---p 00000000 00:00 0 7f73aa0d5000-7f73aad96000 rw-p 00000000 00:00 0 7f73ab0e9000-7f73ab728000 rw-p 00000000 00:00 0 7f73ab728000-7f73ab904000 r--p 07a38000 fd:02 720972858 /home/share/dumparea/testdata/x86_64/sparse_ex.smp-4.15.18/vmlinux 7f73ab904000-7f73abe9e000 rw-p 00000000 00:00 0 7f73abe9e000-7f73ac107000 r--p 070dc000 fd:02 720972858 /home/share/dumparea/testdata/x86_64/sparse_ex.smp-4.15.18/vmlinux 7f73ac107000-7f73b2e82000 rw-p 00000000 00:00 0 7f73b2f93000-7f73b323f000 rw-p 00000000 00:00 0 7f73b323f000-7f73b9768000 r--p 00000000 fd:00 67182613 /usr/lib/locale/locale-archive 7f73b9768000-7f73b977d000 r-xp 00000000 fd:00 33554545 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f73b977d000-7f73b997c000 ---p 00015000 fd:00 33554545 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f73b997c000-7f73b997d000 r--p 00014000 fd:00 33554545 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f73b997d000-7f73b997e000 rw-p 00015000 fd:00 33554545 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f73b997e000-7f73b9a67000 r-xp 00000000 fd:00 33609599 /usr/lib64/libstdc++.so.6.0.19 7f73b9a67000-7f73b9c67000 ---p 000e9000 fd:00 33609599 /usr/lib64/libstdc++.so.6.0.19 7f73b9c67000-7f73b9c6f000 r--p 000e9000 fd:00 33609599 /usr/lib64/libstdc++.so.6.0.19 7f73b9c6f000-7f73b9c71000 rw-p 000f1000 fd:00 33609599 /usr/lib64/libstdc++.so.6.0.19 7f73b9c71000-7f73b9c86000 rw-p 00000000 00:00 0 7f73b9c86000-7f73b9c9d000 r-xp 00000000 fd:00 33609567 /usr/lib64/libpthread-2.17.so 7f73b9c9d000-7f73b9e9c000 ---p 00017000 fd:00 33609567 /usr/lib64/libpthread-2.17.so 7f73b9e9c000-7f73b9e9d000 r--p 00016000 fd:00 33609567 /usr/lib64/libpthread-2.17.so 7f73b9e9d000-7f73b9e9e000 rw-p 00017000 fd:00 33609567 /usr/lib64/libpthread-2.17.so 7f73b9e9e000-7f73b9ea2000 rw-p 00000000 00:00 0 7f73b9ea2000-7f73ba05a000 r-xp 00000000 fd:00 33609541 /usr/lib64/libc-2.17.so 7f73ba05a000-7f73ba25a000 ---p 001b8000 fd:00 33609541 /usr/lib64/libc-2.17.so 7f73ba25a000-7f73ba25e000 r--p 001b8000 fd:00 33609541 /usr/lib64/libc-2.17.so 7f73ba25e000-7f73ba260000 rw-p 001bc000 fd:00 33609541 /usr/lib64/libc-2.17.so 7f73ba260000-7f73ba265000 rw-p 00000000 00:00 0 7f73ba265000-7f73ba26a000 r-xp 00000000 fd:00 33610340 /usr/lib64/libsnappy.so.1.1.4 7f73ba26a000-7f73ba469000 ---p 00005000 fd:00 33610340 /usr/lib64/libsnappy.so.1.1.4 7f73ba469000-7f73ba46a000 r--p 00004000 fd:00 33610340 /usr/lib64/libsnappy.so.1.1.4 7f73ba46a000-7f73ba46b000 rw-p 00005000 fd:00 33610340 /usr/lib64/libsnappy.so.1.1.4 7f73ba46b000-7f73ba48b000 r-xp 00000000 fd:00 33610046 /usr/lib64/liblzo2.so.2.0.0 7f73ba48b000-7f73ba68a000 ---p 00020000 fd:00 33610046 /usr/lib64/liblzo2.so.2.0.0 7f73ba68a000-7f73ba68b000 r--p 0001f000 fd:00 33610046 /usr/lib64/liblzo2.so.2.0.0 7f73ba68b000-7f73ba68c000 rw-p 00020000 fd:00 33610046 /usr/lib64/liblzo2.so.2.0.0 7f73ba68c000-7f73ba6b1000 r-xp 00000000 fd:00 33609657 /usr/lib64/liblzma.so.5.2.2 7f73ba6b1000-7f73ba8b0000 ---p 00025000 fd:00 33609657 /usr/lib64/liblzma.so.5.2.2 7f73ba8b0000-7f73ba8b1000 r--p 00024000 fd:00 33609657 /usr/lib64/liblzma.so.5.2.2 7f73ba8b1000-7f73ba8b2000 rw-p 00025000 fd:00 33609657 /usr/lib64/liblzma.so.5.2.2 7f73ba8b2000-7f73ba9b3000 r-xp 00000000 fd:00 33609549 /usr/lib64/libm-2.17.so 7f73ba9b3000-7f73babb2000 ---p 00101000 fd:00 33609549 /usr/lib64/libm-2.17.so 7f73babb2000-7f73babb3000 r--p 00100000 fd:00 33609549 /usr/lib64/libm-2.17.so 7f73babb3000-7f73babb4000 rw-p 00101000 fd:00 33609549 /usr/lib64/libm-2.17.so 7f73babb4000-7f73babc9000 r-xp 00000000 fd:00 33609654 /usr/lib64/libz.so.1.2.7 7f73babc9000-7f73badc8000 ---p 00015000 fd:00 33609654 /usr/lib64/libz.so.1.2.7 7f73badc8000-7f73badc9000 r--p 00014000 fd:00 33609654 /usr/lib64/libz.so.1.2.7 7f73badc9000-7f73badca000 rw-p 00015000 fd:00 33609654 /usr/lib64/libz.so.1.2.7 7f73badca000-7f73badef000 r-xp 00000000 fd:00 33609627 /usr/lib64/libtinfo.so.5.9 7f73badef000-7f73bafef000 ---p 00025000 fd:00 33609627 /usr/lib64/libtinfo.so.5.9 7f73bafef000-7f73baff3000 r--p 00025000 fd:00 33609627 /usr/lib64/libtinfo.so.5.9 7f73baff3000-7f73baff4000 rw-p 00029000 fd:00 33609627 /usr/lib64/libtinfo.so.5.9 7f73baff4000-7f73bb01a000 r-xp 00000000 fd:00 33609617 /usr/lib64/libncurses.so.5.9 7f73bb01a000-7f73bb219000 ---p 00026000 fd:00 33609617 /usr/lib64/libncurses.so.5.9 7f73bb219000-7f73bb21a000 r--p 00025000 fd:00 33609617 /usr/lib64/libncurses.so.5.9 7f73bb21a000-7f73bb21b000 rw-p 00026000 fd:00 33609617 /usr/lib64/libncurses.so.5.9 7f73bb21b000-7f73bb21d000 r-xp 00000000 fd:00 33609547 /usr/lib64/libdl-2.17.so 7f73bb21d000-7f73bb41d000 ---p 00002000 fd:00 33609547 /usr/lib64/libdl-2.17.so 7f73bb41d000-7f73bb41e000 r--p 00002000 fd:00 33609547 /usr/lib64/libdl-2.17.so 7f73bb41e000-7f73bb41f000 rw-p 00003000 fd:00 33609547 /usr/lib64/libdl-2.17.so Aborted (core dumped) # gdb crash-ups core.25884 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /home/share/dumparea/testdata/x86_64/sparse_ex.smp-4.15.18/crash-ups...done. [New LWP 25884] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `crash-ups vmlinux /dev/shm/vmcore-4.15.18-sp'. Program terminated with signal 6, Aborted. #0 0x00007f73b9ed71f7 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7.x86_64 libgcc-4.8.5-16.el7.x86_64 libstdc++-4.8.5-16.el7.x86_64 lzo-2.06-8.el7.x86_64 ncurses-libs-5.9-13.20130511.el7.x86_64 snappy-1.1.0-3.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) bt #0 0x00007f73b9ed71f7 in raise () from /lib64/libc.so.6 #1 0x00007f73b9ed88e8 in abort () from /lib64/libc.so.6 #2 0x00007f73b9f16f47 in __libc_message () from /lib64/libc.so.6 #3 0x00007f73b9f1d8bd in malloc_consolidate () from /lib64/libc.so.6 #4 0x00007f73b9f1f3a5 in _int_malloc () from /lib64/libc.so.6 #5 0x00007f73b9f2087c in _int_realloc () from /lib64/libc.so.6 #6 0x00007f73b9f22752 in realloc () from /lib64/libc.so.6 #7 0x000000000078448d in xrealloc (ptr=, size=1024) at ./common/common-utils.c:69 #8 0x00000000006e7564 in add_file_name (lh=lh@entry=0x6255000, name=name@entry=0x7f73aba8c950 "rcupdate.h", dir_index=2, mod_time=0, length=0) at dwarf2read.c:15002 #9 0x00000000006f5e74 in dwarf_decode_line_header (offset=, cu=cu@entry=0x6254e00) at dwarf2read.c:15162 #10 0x00000000006fb194 in handle_DW_AT_stmt_list (comp_dir=0x7f73ab72ae6c
, cu=0x6254e00, die=0x41a9280) at dwarf2read.c:7949 #11 read_file_scope (cu=0x6254e00, die=0x41a9280) at dwarf2read.c:8007 #12 process_die (die=0x41a9280, cu=cu@entry=0x6254e00) at dwarf2read.c:7201 #13 0x00000000006ffc48 in process_full_comp_unit (pretend_language=language_minimal, per_cu=0x35da9f0) at dwarf2read.c:7005 #14 process_queue () at dwarf2read.c:6570 #15 dw2_do_instantiate_symtab (per_cu=) at dwarf2read.c:2295 #16 0x00000000006fff57 in psymtab_to_symtab_1 (pst=pst@entry=0x31fee10) at dwarf2read.c:6664 #17 0x0000000000700078 in dwarf2_read_symtab (self=0x31fee10, objfile=0x2f53cf0) at dwarf2read.c:6459 #18 0x00000000006777e1 in psymtab_to_symtab (objfile=objfile@entry=0x2f53cf0, pst=pst@entry=0x31fee10) at psymtab.c:781 #19 0x0000000000679290 in lookup_symbol_aux_psymtabs (objfile=0x2f53cf0, block_index=1, name=0x3d79ae0 "nr_swapper_spaces", domain=VAR_DOMAIN) at psymtab.c:515 #20 0x000000000067292d in lookup_symbol_aux_quick (objfile=objfile@entry=0x2f53cf0, kind=kind@entry=1, name=name@entry=0x3d79ae0 "nr_swapper_spaces", domain=domain@entry=VAR_DOMAIN) at symtab.c:1645 #21 0x0000000000672a8b in lookup_static_symbol_aux (name=name@entry=0x3d79ae0 "nr_swapper_spaces", domain=domain@entry=VAR_DOMAIN) at symtab.c:1407 #22 0x0000000000672da5 in lookup_symbol_aux (is_a_field_of_this=0x0, language=, domain=VAR_DOMAIN, block=0x0, name=0x3d79ae0 "nr_swapper_spaces") at symtab.c:1387 #23 lookup_symbol_in_language (name=name@entry=0x3d79ae0 "nr_swapper_spaces", block=0x0, domain=VAR_DOMAIN, lang=, is_a_field_of_this=0x0) at symtab.c:1213 #24 0x0000000000672df7 in lookup_symbol (name=name@entry=0x3d79ae0 "nr_swapper_spaces", block=, block@entry=0x0, domain=domain@entry=VAR_DOMAIN, is_a_field_of_this=) at symtab.c:1241 #25 0x0000000000610ca3 in classify_name (block=0x0) at c-exp.y:2766 #26 0x00000000006130f2 in c_lex () at c-exp.y:2934 #27 c_parse_internal () at c-exp.c:1937 #28 0x0000000000615a39 in c_parse () at c-exp.y:3064 #29 0x00000000006ce90a in parse_exp_in_context (stringptr=stringptr@entry=0x7ffc58df1d48, pc=pc@entry=0, block=block@entry=0x0, comma=comma@entry=0, out_subexp=out_subexp@entry=0x0, void_context_p=0) at parse.c:1234 #30 0x00000000006ceb05 in parse_exp_1 (stringptr=stringptr@entry=0x7ffc58df1d98, pc=pc@entry=0, block=block@entry=0x0, comma=comma@entry=0) at parse.c:1136 #31 0x00000000006ceb59 in parse_expression (string=0x7ffc58df2040 "nr_swapper_spaces") at parse.c:1279 #32 0x0000000000676c35 in gdb_get_symbol_type (req=0x7ffc58df1f20) at symtab.c:5714 #33 gdb_command_funnel (req=req@entry=0x7ffc58df1f20) at symtab.c:5214 #34 0x00000000004f1b25 in gdb_interface (req=req@entry=0x7ffc58df1f20) at gdb_interface.c:397 #35 0x000000000052b262 in get_symbol_type (name=name@entry=0x7ffc58df2040 "nr_swapper_spaces", member=member@entry=0x0, caller_req=caller_req@entry=0x7ffc58df1f20) at symbols.c:5697 #36 0x000000000052de32 in get_array_length (s=s@entry=0x85334f "nr_swapper_spaces", two_dim=two_dim@entry=0x0, entry_size=entry_size@entry=0) at symbols.c:8251 #37 0x00000000004945aa in dump_kmeminfo () at memory.c:8248 #38 0x00000000004a90ea in cmd_kmem () at memory.c:5018 #39 0x00000000004681ec in exec_command () at main.c:879 #40 0x000000000046847a in main_loop () at main.c:826 #41 0x00000000006a4803 in captured_command_loop (data=data@entry=0x0) at main.c:258 #42 0x00000000006a30aa in catch_errors (func=func@entry=0x6a47f0 , func_args=func_args@entry=0x0, errstring=errstring@entry=0x8f3b9f "", mask=mask@entry=6) at exceptions.c:557 #43 0x00000000006a5796 in captured_main (data=data@entry=0x7ffc58e274a0) at main.c:1064 #44 0x00000000006a30aa in catch_errors (func=func@entry=0x6a4ad0 , func_args=func_args@entry=0x7ffc58e274a0, errstring=errstring@entry=0x8f3b9f "", mask=mask@entry=6) at exceptions.c:557 #45 0x00000000006a5af7 in gdb_main (args=0x7ffc58e274a0) at main.c:1079 #46 gdb_main_entry (argc=, argv=argv@entry=0x7ffc58e27608) at main.c:1099 #47 0x00000000004f0d14 in gdb_main_loop (argc=, argc@entry=3, argv=argv@entry=0x7ffc58e27608) at gdb_interface.c:76 #48 0x00000000004667d5 in main (argc=3, argv=0x7ffc58e27608) at main.c:707 (gdb) Another case with the same vmcore: # crash-ups vmlinux /dev/shm/vmcore-4.15.18-sp crash-ups 7.2.3++ Copyright (C) 2002-2017 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: kernel relocated [426MB]: patching 60983 gdb minimal_symbol values KERNEL: vmlinux DUMPFILE: /dev/shm/vmcore-4.15.18-sp CPUS: 2 DATE: Wed Jun 13 15:44:26 2018 UPTIME: 00:04:26 LOAD AVERAGE: 1.78, 0.64, 0.23 TASKS: 109 NODENAME: rhel75b RELEASE: 4.15.18-sparse_ex VERSION: #3 SMP Tue Jun 12 17:41:53 EDT 2018 MACHINE: x86_64 (2400 Mhz) MEMORY: 2 GB PANIC: "sysrq: SysRq : Trigger a crash" PID: 2801 COMMAND: "bash" TASK: ffff89bbbcb0cc40 [THREAD_INFO: ffff89bbbcb0cc40] CPU: 1 STATE: TASK_RUNNING (SYSRQ) crash-ups> kmem -i PAGES TOTAL PERCENTAGE TOTAL MEM 480806 1.8 GB ---- FREE 30085 117.5 MB 6% of TOTAL MEM USED 450721 1.7 GB 93% of TOTAL MEM SHARED 8172 31.9 MB 1% of TOTAL MEM BUFFERS 1 4 KB 0% of TOTAL MEM