From: Dave Anderson <anderson(a)redhat.com&gt; Subject: Re: [Crash-utility] crash "bt" and "dmesg" show different messages To: "Discussion list for crash utility usage, maintenance and development" <crash-utility(a)redhat.com&gt; Date: Monday, November 29, 2010, 10:03 PM ----- "Yuming Cheng" <chengyuming_ah(a)yahoo.com.cn&gt; wrote: > Hi all, > > When using kdump, I find crash "bt" and "dmesg" give different info. > which one is more reliable ? > > Thanks, > ---cym In this case, the dmesg output is more helpful because it contains the exception frame.  It's pretty clear that the neigh_cleanup_and_release() has called a destructor function, but the address stored in neigh->parms->neigh_destructor (as stored in RAX) contains a bogus address of 0000000000000001: static void neigh_cleanup_and_release(struct neighbour *neigh) {         if (neigh->parms->neigh_destructor)                 neigh->parms->neigh_destructor(neigh);         __neigh_notify(neigh, RTM_DELNEIGH, 0);         neigh_release(neigh); } crash> dis -r neigh_cleanup_and_release+0x13 0xffffffff8022108d <neigh_cleanup_and_release>: push   %rbx 0xffffffff8022108e <neigh_cleanup_and_release+0x1>:     mov    0x10(%rdi),%rax 0xffffffff80221092 <neigh_cleanup_and_release+0x5>:     mov    %rdi,%rbx 0xffffffff80221095 <neigh_cleanup_and_release+0x8>:     mov    0x18(%rax),%rax 0xffffffff80221099 <neigh_cleanup_and_release+0xc>:     test   %rax,%rax 0xffffffff8022109c <neigh_cleanup_and_release+0xf>:     je     0xffffffff802210a0 <neigh_cleanup_and_release+0x13> 0xffffffff8022109e <neigh_cleanup_and_release+0x11>:    callq  *%rax 0xffffffff802210a0 <neigh_cleanup_and_release+0x13>:    lock decl 0x70(%rbx) If you do a "bt -e" I would guess that the exception frame would be found and displayed, but it *should* have been displayed in-line by the "bt" command. I can't tell you why it was not displayed by "bt" unless I have the dumpfile.  You also didn't mention what version of crash you were running -- there have been a few fixes for "missing" exception frames. If you want to make the dumpfile available to me, I can take a look at it. Dave > > dmesg > /****************************************/ > Unable to handle kernel NULL pointer dereference at 0000000000000001 > RIP: [<0000000000000001>] > PGD 323c6f067 PUD 323f13067 PMD 0 > Oops: 0010 [1] SMP > last sysfs file: /devices/pci0000:00/0000:00:00.0/irq > CPU 6 > Modules linked in: igb(U) bonding ipv6 xfrm_nalgo crypto_api autofs4 > hidp rfcomm l2cap bluetooth lockd sunrpc dm_mirror dm_multipath > scsi_dh video hwmon backlight sbs i2c_ec button battery asus_acpi > acpi_memhotplug ac parport_pc lp parport sg ixgbe pcspkr i2c_i801 > serio_raw i2c_core 8021q dca dm_raid45 dm_message dm_region_hash > dm_log dm_mod dm_mem_cache ahci libata shpchp mptsas mptscsih mptbase > scsi_transport_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd > ehci_hcd > Pid: 8894, comm: ifconfig Tainted: G      2.6.18-164.el5debug #1 > RIP: 0010:[<0000000000000001>]  [<0000000000000001>] > RSP: 0018:ffff810323dd9cf0  EFLAGS: 00010202 > RAX: 0000000000000001 RBX: ffff81032a8e5b68 RCX: 0000000000000000 > RDX: 0000000000000006 RSI: 0000000000000001 RDI: ffff81032a8e5b68 > RBP: ffff81033aabc850 R08: 0000000000000002 R09: 0000000000000001 > R10: ffff81032a8e5c30 R11: ffffffff80049ee3 R12: ffff81032a8e5ba8 > R13: 0000000000000006 R14: ffff8103238be000 R15: ffffffff8846ad00 > FS:  00002ba7032083f0(0000) GS:ffff810113a9e4c8(0000) > knlGS:0000000000000000 > CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: 0000000000000001 CR3: 00000003242b4000 CR4: 00000000000006e0 > Process ifconfig (pid: 8894, threadinfo ffff810323dd8000, task > ffff810323f461c0) > Stack:  ffffffff8023de3b ffff81032a8e5b68 ffffffff8023e0d6 > ffffffff8023e122 >  ffffffff88468eb0 ffff8103238be000 ffffffff8846ad00 ffff8103238be000 >  0000000000000000 ffffffff8846ae98 ffffffff8023e12d 0000000000000002 > Call Trace: >  [<ffffffff8023de3b>] neigh_cleanup_and_release+0x13/0x2c >  [<ffffffff8023e0d6>] neigh_flush_dev+0x9d/0xc3 >  [<ffffffff88439acb>] :ipv6:ndisc_netdev_event+0x30/0x3d >  [<ffffffff8006ae76>] notifier_call_chain+0x20/0x32 >  [<ffffffff80238c52>] dev_close+0x6e/0x72 >  [<ffffffff80237d24>] dev_change_flags+0x5a/0x119 >  [<ffffffff8026cb77>] devinet_ioctl+0x235/0x59c >  [<ffffffff8022f0e3>] sock_ioctl+0x1c7/0x1eb >  [<ffffffff8004465d>] do_ioctl+0x21/0x6b >  [<ffffffff80031f07>] vfs_ioctl+0x45d/0x4bf >  [<ffffffff800c0b9d>] audit_syscall_entry+0x180/0x1b3 >  [<ffffffff8004ef9e>] sys_ioctl+0x59/0x78 >  [<ffffffff800602a6>] tracesys+0xd5/0xdf > > /****************************************/ > crash btcrash> bt > PID: 8894   TASK: ffff810323f461c0  CPU: 6   COMMAND: "ifconfig" >  #0 [ffff810323dd9a50] crash_kexec at ffffffff800b6eae >  #1 [ffff810323dd9b10] __die at ffffffff80069087 >  #2 [ffff810323dd9b50] do_page_fault at ffffffff8006ad73 >  #3 [ffff810323dd9c40] error_exit at ffffffff80060e9d >  #4 [ffff810323dd9c78] skb_dequeue at ffffffff80049ee3 >  #5 [ffff810323dd9cf0] neigh_cleanup_and_release at ffffffff8023de3b >  #6 [ffff810323dd9d00] neigh_flush_dev at ffffffff8023e0d6 >  #7 [ffff810323dd9d40] neigh_ifdown at ffffffff8023e12d >  #8 [ffff810323dd9d80] ndisc_netdev_event at ffffffff88439acb >  #9 [ffff810323dd9d90] notifier_call_chain at ffffffff8006ae76 > #10 [ffff810323dd9db0] dev_close at ffffffff80238c52 > #11 [ffff810323dd9dc0] dev_change_flags at ffffffff80237d24 > #12 [ffff810323dd9df0] devinet_ioctl at ffffffff8026cb77 > #13 [ffff810323dd9e90] sock_ioctl at ffffffff8022f0e3 > #14 [ffff810323dd9eb0] do_ioctl at ffffffff8004465d > #15 [ffff810323dd9ed0] vfs_ioctl at ffffffff80031f07 > #16 [ffff810323dd9f40] sys_ioctl at ffffffff8004ef9e > #17 [ffff810323dd9f80] tracesys at ffffffff800602a6 (via system_call) >     RIP: 0000003749ccc557  RSP: 00007fff470ea238  RFLAGS: 00000206 >     RAX: ffffffffffffffda  RBX: ffffffff800602a6  RCX: > ffffffffffffffff >     RDX: 00007fff470ea240  RSI: 0000000000008914  RDI: > 0000000000000004 >     RBP: 0000000000000000   R8: 00007fff470ea244   R9: > 0000000000000002 >     R10: 0000000000000001  R11: 0000000000000206  R12: > 00007fff470ea360 >     R13: 00000000fffffffe  R14: 00007fff470ea530  R15: > 0000000000000004 >     ORIG_RAX: 0000000000000010  CS: 0033  SS: 002b > > > > > >        > > -- > Crash-utility mailing list > Crash-utility(a)redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility