Hi, I was investigating a 64 bit linux kernel dump . I have following doubts regarding usage of crash. 1) I wanted to access the intermediate kernel stack frames. To know the status of the frame and the point of failure. When I tried to access a stack frame I get an error message "crash: prohibited gdb command: frame". Can you please let me know if there is any other way of accessing the kernel stack frames using crash.

2) When I run bt in crash, I get a stack trace. Another person from a different team reported a slightly different stack trace to mine. Below are the stack traces. The register contents are quite different between the two My stack trace PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet" #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486 #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230 #2 [ffff88031ce75b20] oops_end at ffffffff8100fa38 #3 [ffff88031ce75b50] no_context at ffffffff8102d801 #4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9 #5 [ffff88031ce75c70] bad_area at ffffffff8102da41 #6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19 #7 [ffff88031ce75cf0] page_fault at ffffffff812d7425 #8 [ffff88031ce75d78] n_tty_read at ffffffff811f03b3 #9 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e #10 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8 #11 [ffff88031ce75f40] sys_read at ffffffff810ebe48 #12 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2 RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212 RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000 RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000 RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0 ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b Reported stack trace PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet" #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486 #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230 #2 [ffff88031ce75ad8] n_tty_read at ffffffff811f03b3 #3 [ffff88031ce75b20] oops_end at ffffffff8100fa38 #4 [ffff88031ce75b50] no_context at ffffffff8102d801 #5 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9 #6 [ffff88031ce75c20] native_sched_clock at ffffffff810120aa #7 [ffff88031ce75c70] bad_area at ffffffff8102da41 #8 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19 #9 [ffff88031ce75cf0] page_fault at ffffffff812d7425 [exception RIP: n_tty_read+1420] RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044 RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044 R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #10 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e #11 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8 #12 [ffff88031ce75f40] sys_read at ffffffff810ebe48 #13 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2 3) I want to retrieve the address of a data structure in the current context. How can it be done? I tried using struct command, but it did not help

4) When I run the command readelf -a vmcore, I get an error message "readelf: Error: Not an ELF file - it has the wrong magic bytes at the start." Please help regarding the above queries. Thanks and Regards Shashidhara Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please notify us immediately at mailmaster(a)mphasis.com and delete this mail from your records. -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

3) I want to retrieve the address of a data structure in the current context. How can it be done? I tried using struct command, but it did not help

> I need to find out the virtual address of the structure tty of type struct tty_struct, which is passed as an argument to the function n_read_tty. Below is the corresponding stack trace.

>PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet" > > #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486 > #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230 > #2 [ffff88031ce75b20] oops_end at ffffffff8100fa38 > #3 [ffff88031ce75b50] no_context at ffffffff8102d801 > #4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9 > #5 [ffff88031ce75c70] bad_area at ffffffff8102da41 > #6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19 > #7 [ffff88031ce75cf0] page_fault at ffffffff812d7425 > #8 [ffff88031ce75d78] n_tty_read at ffffffff811f03b3 > #9 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e > #10 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8 > #11 [ffff88031ce75f40] sys_read at ffffffff810ebe48 > #12 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2 > RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212 > RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000 > RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000 > RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000 > R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000 > R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0 > ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b

> I have another query I tried to convert the vmcore file to ELF format using "makedumpfile -E -d 31 -x vmlinux_temp vmcore dumpfile" . For which I got an error message " '-E' option is disable, because vmcore is kdump compressed format.

>Please guide me further

Hi, I was investigating a 64 bit linux kernel dump . I have following doubts regarding usage of crash. 1) I wanted to access the intermediate kernel stack frames. To know the status of the frame and the point of failure. When I tried to access a stack frame I get an error message “ crash: prohibited gdb command: frame ”. Can you please let me know if there is any other way of accessing the kernel stack frames using crash.

2) When I run bt in crash, I get a stack trace. Another person from a different team reported a slightly different stack trace to mine. Below are the stack traces. The register contents are quite different between the two My stack trace PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet" #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486 #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230 #2 [ffff88031ce75b20] oops_end at ffffffff8100fa38 #3 [ffff88031ce75b50] no_context at ffffffff8102d801 #4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9 #5 [ffff88031ce75c70] bad_area at ffffffff8102da41 #6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19 #7 [ffff88031ce75cf0] page_fault at ffffffff812d7425 #8 [ffff88031ce75d78] n_tty_read at ffffffff811f03b3 #9 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e #10 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8 #11 [ffff88031ce75f40] sys_read at ffffffff810ebe48 #12 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2 RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212 RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000 RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000 RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0 ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b Reported stack trace PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet" #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486 #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230 #2 [ffff88031ce75ad8] n_tty_read at ffffffff811f03b3 #3 [ffff88031ce75b20] oops_end at ffffffff8100fa38 #4 [ffff88031ce75b50] no_context at ffffffff8102d801 #5 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9 #6 [ffff88031ce75c20] native_sched_clock at ffffffff810120aa #7 [ffff88031ce75c70] bad_area at ffffffff8102da41 #8 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19 #9 [ffff88031ce75cf0] page_fault at ffffffff812d7425 [exception RIP: n_tty_read+1420] RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044 RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044 R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #10 [ffff88031ce75ec0] #10 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e #11 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8 #12 [ffff88031ce75f40] sys_read at ffffffff810ebe48 #13 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2

3) I want to retrieve the address of a data structure in the current context. How can it be done? I tried using struct command, but it did not help

4) When I run the command readelf -a vmcore, I get an error message ”readelf: Error: Not an ELF file - it has the wrong magic bytes at the start.”

First I would update your crash utility so that you have the exception frame dump that was a result of the page fault, because it's possible that the tty structure pointer is in the register dump. But anyway, without knowing the kernel version, it's hard to pinpoint exactly which instruction in n_tty_read() generated the page fault. Was the bad address generated because the tty structure pointer was NULL? And again, with an updated crash utility, you'll get more information w/respect to the register contents at the time of the page fault, and also you might get some help finding it with "bt -F". I'm not sure where the tty structure gets allocated from -- is it statically-allocated, or is it allocated from one of the "size-xxx" slab caches, etc...

Hi, Thank you Dave for your time and help. As suggested, I will update my crash utility first and then go about analyzing the dump. >>> I believe that something like this might work?: >>> $ makedumpfile -c -d 31 -x vmlinux_temp vmcore-old vmcore-new I tried to use the command you suggested " makedumpfile -c -d 31 -x vmlinux_temp vmcore vmcore-new " . I got an error message " The kernel version is not supported.The created dumpfile may be incomplete. check_release: Can't get the kernel version" Should I update makedumpfile utility as well? Or just updating crash will do?

Hi Dave, I updated the makedumpfile utility from 1.3.5 to 1.3.7 . When I run the below command makedumpfile -c -d 31 -x vmlinux_temp vmcore vmcore-new The kernel version is not supported. The created dumpfile may be incomplete. check_release: Can't get the kernel version. makedumpfile Failed.

Is there any other way to extract the ELF style vmcore file from the kdump compressed format. Please guide me.

Hi Dave, Thanks for the help. I have some doubts regarding kdump and crash utility. I am analyzing a vmcore dump caused by an oops in customer location using crash utility.The oops report is below [345132.723424] BUG: unable to handle kernel NULL pointer dereference at 0000000000000005 [345132.724928] IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818 [345132.726100] PGD 2c8e03067 PUD 2cbd88067 PMD 0 [345132.727187] Oops: 0000 [#1] SMP [345132.727879] last sysfs file: /sys/block/loop7/dev [345132.728935] CPU 1 [345132.729396] Modules linked in: xt_tcpudp iptable_filter ip_tables x_tables strmfs_mod bond0 ipmi_devintf hpwdt sctp ipv6 crc32c libcrc32c loop ipmi_si tpm_tis ipmi_msghandler hpilo tpm tpm_bios psmouse serio_raw shpchp pci_hotplug container processor evdev ext3 jbd mbcache dm_mirror dm_region_hash dm_log dm_snapshot dm_mod sg sr_mod cdrom ide_pci_generic ide_core usbhid hid ata_piix ata_generic libata ehci_hcd bnx2 uhci_hcd e1000e cciss scsi_mod button thermal fan thermal_sys edd [last unloaded: scsi_wait_scan] [345132.739511] Pid: 13366, comm: telnet Not tainted 2.6.32-cdma-18-amd64 #1 ProLiant DL380 G6 [345132.741423] RIP: 0010:[<ffffffff811f03b3>] [<ffffffff811f03b3>] n_tty_read+0x58c/0x818 [345132.743220] RSP: 0018:ffff88031ce75da8 EFLAGS: 00010246 [345132.744469] RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044 [345132.746061] RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c [345132.747726] RBP: ffff88031ce75eb8 R08: 0000000000000000 R09: 0000000000000000 [345132.749391] R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044 [345132.750981] R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff [345132.752650] FS: 00007ffff7fee6f0(0000) GS:ffff880033020000(0000) knlGS:0000000000000000 [345132.754569] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [345132.755915] CR2: 0000000000000005 CR3: 000000030c408000 CR4: 00000000000006e0 [345132.757579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [345132.759169] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [345132.760778] Process telnet (pid: 13366, threadinfo ffff88031ce74000, task ffff88031b60d580) [345132.762707] Stack: [345132.763162] ffff88031b60d580 ffff88031b60d580 ffff88031b60d580 ffff88031b60d580 [345132.764791] <0> 000000000061c02b 0000000000000000 0000000000000000 000000000061c02a [345132.766510] <0> ffff8802de651a40 ffff8802cbd549c0 ffff8802cbd54c90 ffff8802cbd54d1c [345132.768270] Call Trace: [345132.768877] [<ffffffff81045f84>] ? default_wake_function+0x0/0xf [345132.770309] [<ffffffff811ebf7e>] tty_read+0x7d/0xba [345132.771526] [<ffffffff810ebcc8>] vfs_read+0xab/0x167 [345132.772541] [<ffffffff810ebe48>] sys_read+0x47/0x6f [345132.773526] [<ffffffff8100bbc2>] system_call_fastpath+0x16/0x1b [345132.774652] Code: 00 41 8b 85 5c 02 00 00 48 8b 9d 78 ff ff ff f0 0f b3 03 45 19 f6 49 63 95 5c 02 00 00 49 8b 85 50 02 00 00 48 8b bd 48 ff ff ff <0f> be 1c 10 e8 fc 6b 0e 00 48 89 c6 41 8b 85 5c 02 00 00 41 ff [345132.778840] RIP [<ffffffff811f03b3>] n_tty_read+0x58c/0x818 [345132.780107] RSP <ffff88031ce75da8> [345132.780969] CR2: 0000000000000005 [345132.781786] hpwdt: New timer passed in is 120 seconds. [345132.782942] hpwdt: timer reset to 120 for kdump After analysis, we figured out that the crash occurs in the function n_read_tty of kernel-source/drivers/char/n_tty.c . The oops occurred on linux kernel 2.6.32. Below is the code fragment where the page fault occurred. The page fault occurs when executing the statement c = tty->read_buf[tty->read_tail] . /* N.B. avoid overrun if nr == 0 */ while (nr && tty->read_cnt) { int eol; eol = test_and_clear_bit(tty->read_tail, tty->read_flags); c = tty->read_buf[tty->read_tail]; // page fault statement after analyzing oops spin_lock_irqsave(&tty->read_lock, flags); tty->read_tail = ((tty->read_tail+1) & (N_TTY_BUF_SIZE-1)); tty->read_cnt--; if (eol) { /* this test should be redundant: * we shouldn't be reading data if * canon_data is 0 */ if (--tty->canon_data < 0) tty->canon_data = 0; } spin_unlock_irqrestore(&tty->read_lock, flags); Below is the contents of the structure tty_struct ( at the time of oops ). This was passed as an argument to the function n_read_tty(). tty_struct ffff8802cbd54800 struct tty_struct { ... magic = 21505, driver = 0xffff88031b54ea00, ops = 0xffffffff8130f650, name = "pts9\000\...", driver_data = 0xffff88029c8a9668, icanon = 1 '\001', read_buf = 0xffff8802cbfe6000 "", read_head = 0, read_tail = 0, read_cnt = 0, read_flags = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, canon_data = 0, ...................................... As per crash utility the field read_cnt is 0 when kernel oopsed.In that case, the statement while (nr && tty->read_cnt) in the above code fragment should have failed. This leads me to think that there was some other thread/task in kernel which should have updated the read_cnt field in parallel. However the crash utility reports that the runqueue of all CPUs at the time of crash as idle. Except CPU1 which was executing the user program telnet in kernel context ( system call ). Below is the runqueue output. CPU 0 RUNQUEUE: ffff880033012d80 CURRENT: PID: 0 TASK: ffffffff814204b0 COMMAND: "swapper" RT PRIO_ARRAY: ffff880033012e98 [no tasks queued] CFS RB_ROOT: ffff880033012e10 [no tasks queued] CPU 1 RUNQUEUE: ffff880033032d80 CURRENT: PID: 13366 TASK: ffff88031b60d580 COMMAND: "telnet" RT PRIO_ARRAY: ffff880033032e98 [no tasks queued] CFS RB_ROOT: ffff880033032e10 [no tasks queued] CPU 2 RUNQUEUE: ffff880033052d80 CURRENT: PID: 0 TASK: ffff88031e0e3540 COMMAND: "swapper" RT PRIO_ARRAY: ffff880033052e98 [no tasks queued] CFS RB_ROOT: ffff880033052e10 [no tasks queued] CPU 3 RUNQUEUE: ffff880033072d80 CURRENT: PID: 0 TASK: ffff88031e113580 COMMAND: "swapper" RT PRIO_ARRAY: ffff880033072e98 [no tasks queued] CFS RB_ROOT: ffff880033072e10 [no tasks queued] How is this logically possible. Crash reports there are no tasks running currently. Or before the oops trigger and kdump capturing the memory image, some process/thread ran which could have updated the data structure. I wanted to know if this scenario is possible. I kindly request your suggestion/guidance. Please let me know if you need any other details.

BTW, are you sure about that? Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown below, and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail is 0, then the statement above would be simply be reading tty->read_buf[0], or virtual address 0xffff8802cbfe6000. But the oops shows it faulting on a virtual address of "5": BUG: unable to handle kernel NULL pointer dereference at 0000000000000005

BTW, are you sure about that? Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown

and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail

then the statement above would be simply be reading tty->read_buf[0],

virtual address 0xffff8802cbfe6000. But the oops shows it faulting on

virtual address of "5": BUG: unable to handle kernel NULL pointer dereference at

And for that matter, since tty->read_cnt is 0 above, your original question as to how that code path was taken to begin with is also valid.

Yeah, although the contents tty->read_buf are hard to explain. It gets allocated during n_tty_open() and freed during n_tty_close(). And at the beginning of n_tty_read() there's: BUG_ON(!tty->read_buf); and the dump-time contents show a buffer allocated: crash> tty_struct ffff8802cbd54800 struct tty_struct { ... magic = 21505, driver = 0xffff88031b54ea00, ops = 0xffffffff8130f650, name = "pts9\000\...", driver_data = 0xffff88029c8a9668, icanon = 1 '\001', read_buf = 0xffff8802cbfe6000 "", read_head = 0, read_tail = 0, read_cnt = 0, ... but it's a NULL pointer when read during the function?

Hi Dave, Thanks again for the response. When I check the address of tty->read_buf using kmem command in crash, below is the output. Which shows that the memory is still not freed. If the tty_close handler had been invoked, before accessing tty->read_buf we should not get this output. Please correct me if I am erring. kmem 0xffff8802cbfe6000 CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE ffff88031f8039c0 size-4096 4096 6322 6393 6393 4k SLAB MEMORY TOTAL ALLOCATED FREE ffff8802ac4811c0 ffff8802cbfe6000 1 1 0 FREE / [ALLOCATED] [ffff8802cbfe6000] PAGE PHYSICAL MAPPING INDEX CNT FLAGS ffffea0009c9fa50 2cbfe6000 0 0 1 200000000000080 I would like to thank every one for your time and effort in analyzing this issue. Thanks and Regards Shashidhara

-----Original Message----- From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson Sent: Friday, June 24, 2011 7:10 PM To: Discussion list for crash utility usage,maintenance and development Subject: Re: [Crash-utility] Unable to switch stack frames while using crash ---- Original Message ----- >> >> BTW, are you sure about that? >> >> Presuming that the "tty" pointer is ffff8802cbd54800 as you've >> shown below, >> and therefore tty->read_buf is 0xffff8802cbfe6000 and >> tty->read_tail is 0, >> then the statement above would be simply be reading >> tty->read_buf[0], or >> virtual address 0xffff8802cbfe6000. But the oops shows it faulting on a >> virtual address of "5": Well, as it turns out, you have every reason to be sure about that... Anyway, I don't understand why line numbers are not available with your kernel: > crash> dis -rl n_tty_read+0x58c > dis: line numbers are not available ... [ cut ] ... > 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d > 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx > 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax > 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi > 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx But nonetheless, there is only on movsbl instruction in n_tty_read(), and looking at a RHEL6 kernel, you were correct in your original determination of the faulting instruction: crash> dis n_tty_read | grep movsbl 0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx crash> dis -rl 0xffffffff812f88c9 | tail ... [ cut ] ... /usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/drivers/char/n_tty.c: 1821 0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx crash> which is here: 1814 if (tty->icanon) { 1815 /* N.B. avoid overrun if nr == 0 */ 1816 while (nr && tty->read_cnt) { 1817 int eol; 1818 1819 eol = test_and_clear_bit(tty->read_tail, 1820 tty->read_flags); 1821 c = tty->read_buf[tty->read_tail]; The tty_struct offsets are these: crash> tty_struct -o struct tty_struct { ... [ cut ]... [0x250] char *read_buf; [0x258] int read_head; [0x25c] int read_tail; ... And you can see in the previous instructions the tty->read_buf (0x250) and tty->read_tail (0x25c) offsets being added to the tty_struct base address that is located in R13: > 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d > 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx > 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax > 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi > 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx And as you originally reported, the tty_struct address in R13 is ffff8802cbd54800: PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet" #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486 #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230 #2 [ffff88031ce75b20] oops_end at ffffffff8100fa38 #3 [ffff88031ce75b50] no_context at ffffffff8102d801 #4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9 #5 [ffff88031ce75c70] bad_area at ffffffff8102da41 #6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19 #7 [ffff88031ce75cf0] page_fault at ffffffff812d7425 [exception RIP: n_tty_read+1420] RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044 RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044 R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e #9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8 #10 [ffff88031ce75f40] sys_read at ffffffff810ebe48 #11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2 RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212 RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000 RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000 RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0 ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b But for whatever reason -- and I cannot explain it -- after these instructions were done: > 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx > 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax the RDX register ended up with 0000000000000005, and the RAX register with a 0000000000000000, leading to the: BUG: unable to handle kernel NULL pointer dereference at 0000000000000005 But when you display the tty_struct at ffff8802cbd54800, you see the read_buf and read_tail with seemingly legitimate values: crash> tty_struct ffff8802cbd54800 struct tty_struct { ... magic = 21505, driver = 0xffff88031b54ea00, ops = 0xffffffff8130f650, name = "pts9\000\...", driver_data = 0xffff88029c8a9668, icanon = 1 '\001', read_buf = 0xffff8802cbfe6000 "", read_head = 0, read_tail = 0, read_cnt = 0, ... So everything in your analysis was correct, but how it is possible that the RDX and RAX registers to have ended up with 0 and 5 is hard to explain. And for that matter, since tty->read_cnt is 0 above, your original question as to how that code path was taken to begin with is also valid. So I don't know -- anybody on the list ever seen anything like this? Stumped, Dave ----- Original Message ----- > Hi Dave, > > Thank you so much for your help. > > Below is the output of dis -rl n_tty_read+0x58c > > crash> dis -rl n_tty_read+0x58c > dis: line numbers are not available > 0xffffffff811efe27 <n_tty_read>: push %rbp > 0xffffffff811efe28 <n_tty_read+1>: mov %gs:0xb500,%rax > 0xffffffff811efe31 <n_tty_read+10>: mov %rsp,%rbp > 0xffffffff811efe34 <n_tty_read+13>: push %r15 > 0xffffffff811efe36 <n_tty_read+15>: push %r14 > 0xffffffff811efe38 <n_tty_read+17>: push %r13 > 0xffffffff811efe3a <n_tty_read+19>: mov %rdi,%r13 > 0xffffffff811efe3d <n_tty_read+22>: lea -0x70(%rbp),%rdi > 0xffffffff811efe41 <n_tty_read+26>: push %r12 > 0xffffffff811efe43 <n_tty_read+28>: push %rbx > 0xffffffff811efe44 <n_tty_read+29>: lea 0x490(%r13),%rbx > 0xffffffff811efe4b <n_tty_read+36>: sub $0xe8,%rsp > 0xffffffff811efe52 <n_tty_read+43>: mov %rax,-0x98(%rbp) > 0xffffffff811efe59 <n_tty_read+50>: mov %rcx,-0x78(%rbp) > 0xffffffff811efe5d <n_tty_read+54>: xor %eax,%eax > 0xffffffff811efe5f <n_tty_read+56>: mov $0xa,%ecx > 0xffffffff811efe64 <n_tty_read+61>: mov %rdx,-0xd8(%rbp) > 0xffffffff811efe6b <n_tty_read+68>: mov %rsi,-0xd0(%rbp) > 0xffffffff811efe72 <n_tty_read+75>: mov %rdx,-0x40(%rbp) > 0xffffffff811efe76 <n_tty_read+79>: rep stos %eax,%es:(%rdi) > 0xffffffff811efe78 <n_tty_read+81>: lea 0x1c0(%r13),%rax > 0xffffffff811efe7f <n_tty_read+88>: lea 0x1c8(%r13),%rcx > 0xffffffff811efe86 <n_tty_read+95>: mov %rbx,-0xc0(%rbp) > 0xffffffff811efe8d <n_tty_read+102>: lea 0xd8(%r13),%rbx > 0xffffffff811efe94 <n_tty_read+109>: movq > $0xffffffff81045f84,-0x60(%rbp) > 0xffffffff811efe9c <n_tty_read+117>: movq $0x0,-0xa8(%rbp) > 0xffffffff811efea7 <n_tty_read+128>: mov -0x98(%rbp),%rdx > 0xffffffff811efeae <n_tty_read+135>: mov %rax,-0xc8(%rbp) > 0xffffffff811efeb5 <n_tty_read+142>: mov -0x98(%rbp),%rax > 0xffffffff811efebc <n_tty_read+149>: mov %rcx,-0x90(%rbp) > 0xffffffff811efec3 <n_tty_read+156>: lea 0x51c(%r13),%rcx > 0xffffffff811efeca <n_tty_read+163>: mov %rbx,-0x80(%rbp) > 0xffffffff811efece <n_tty_read+167>: mov %rdx,-0x68(%rbp) > 0xffffffff811efed2 <n_tty_read+171>: lea 0x268(%r13),%rdx > 0xffffffff811efed9 <n_tty_read+178>: mov %rcx,-0xb8(%rbp) > 0xffffffff811efee0 <n_tty_read+185>: mov %rax,-0xf8(%rbp) > 0xffffffff811efee7 <n_tty_read+192>: mov %rax,-0x100(%rbp) > 0xffffffff811efeee <n_tty_read+199>: mov %rdx,-0x88(%rbp) > 0xffffffff811efef5 <n_tty_read+206>: mov %rax,-0x108(%rbp) > 0xffffffff811efefc <n_tty_read+213>: mov %rax,-0x110(%rbp) > 0xffffffff811eff03 <n_tty_read+220>: cmpq $0x0,0x250(%r13) > 0xffffffff811eff0b <n_tty_read+228>: jne 0xffffffff811eff11 > <n_tty_read+234> > 0xffffffff811eff0d <n_tty_read+230>: ud2a > 0xffffffff811eff0f <n_tty_read+232>: jmp 0xffffffff811eff0f > <n_tty_read+232> > 0xffffffff811eff11 <n_tty_read+234>: mov -0xd0(%rbp),%rdx > 0xffffffff811eff18 <n_tty_read+241>: mov 0x20(%rdx),%rax > 0xffffffff811eff1c <n_tty_read+245>: cmpq > $0xffffffff811ed61f,0x18(%rax) > 0xffffffff811eff24 <n_tty_read+253>: je 0xffffffff811effef > <n_tty_read+456> > 0xffffffff811eff2a <n_tty_read+259>: mov -0xf8(%rbp),%rcx > 0xffffffff811eff31 <n_tty_read+266>: mov 0x478(%rcx),%rax > 0xffffffff811eff38 <n_tty_read+273>: cmp %r13,0x180(%rax) > 0xffffffff811eff3f <n_tty_read+280>: jne 0xffffffff811effef > <n_tty_read+456> > 0xffffffff811eff45 <n_tty_read+286>: mov 0xc8(%r13),%rdx > 0xffffffff811eff4c <n_tty_read+293>: test %rdx,%rdx > 0xffffffff811eff4f <n_tty_read+296>: jne 0xffffffff811eff64 > <n_tty_read+317> > 0xffffffff811eff51 <n_tty_read+298>: mov $0xffffffff8139c972,%rdi > 0xffffffff811eff58 <n_tty_read+305>: xor %eax,%eax > 0xffffffff811eff5a <n_tty_read+307>: callq 0xffffffff812d4abf > <printk> > 0xffffffff811eff5f <n_tty_read+312>: jmpq 0xffffffff811effef > <n_tty_read+456> > 0xffffffff811eff64 <n_tty_read+317>: mov -0xf8(%rbp),%rbx > 0xffffffff811eff6b <n_tty_read+324>: mov 0x1e0(%rbx),%rax > 0xffffffff811eff72 <n_tty_read+331>: cmp %rdx,0x238(%rax) > 0xffffffff811eff79 <n_tty_read+338>: je 0xffffffff811effef > <n_tty_read+456> > 0xffffffff811eff7b <n_tty_read+340>: mov -0x98(%rbp),%rax > 0xffffffff811eff82 <n_tty_read+347>: testb $0x10,0x48a(%rax) > 0xffffffff811eff89 <n_tty_read+354>: jne 0xffffffff811f0611 > <n_tty_read+2026> > 0xffffffff811eff8f <n_tty_read+360>: mov 0x480(%rax),%rax > 0xffffffff811eff96 <n_tty_read+367>: cmpq $0x1,0x288(%rax) > 0xffffffff811eff9e <n_tty_read+375>: jne 0xffffffff811f0604 > <n_tty_read+2013> > 0xffffffff811effa4 <n_tty_read+381>: jmpq 0xffffffff811f0611 > <n_tty_read+2026> > 0xffffffff811effa9 <n_tty_read+386>: mov -0x98(%rbp),%rcx > 0xffffffff811effb0 <n_tty_read+393>: mov $0x1,%edx > 0xffffffff811effb5 <n_tty_read+398>: mov $0x15,%esi > 0xffffffff811effba <n_tty_read+403>: mov 0x1e0(%rcx),%rax > 0xffffffff811effc1 <n_tty_read+410>: mov 0x238(%rax),%rdi > 0xffffffff811effc8 <n_tty_read+417>: callq 0xffffffff8105953a > <kill_pgrp> > 0xffffffff811effcd <n_tty_read+422>: mov %gs:0xb508,%rdx > 0xffffffff811effd6 <n_tty_read+431>: lea -0x1fc8(%rdx),%rax > 0xffffffff811effdd <n_tty_read+438>: lock orb $0x4,-0x1fc8(%rdx) > 0xffffffff811effe5 <n_tty_read+446>: mov $0xfffffe00,%eax > 0xffffffff811effea <n_tty_read+451>: jmpq 0xffffffff811f0616 > <n_tty_read+2031> > 0xffffffff811effef <n_tty_read+456>: testb $0x10,0x21c(%r13) > 0xffffffff811efff7 <n_tty_read+464>: je 0xffffffff811f000f > <n_tty_read+488> > 0xffffffff811efff9 <n_tty_read+466>: movl $0x0,-0xb0(%rbp) > 0xffffffff811f0003 <n_tty_read+476>: movl $0x0,-0xac(%rbp) > 0xffffffff811f000d <n_tty_read+486>: jmp 0xffffffff811f0076 > <n_tty_read+591> > 0xffffffff811f000f <n_tty_read+488>: mov 0x70(%r13),%rdx > 0xffffffff811f0013 <n_tty_read+492>: movzbl 0x16(%rdx),%eax > 0xffffffff811f0017 <n_tty_read+496>: imul $0x19,%eax,%eax > 0xffffffff811f001a <n_tty_read+499>: mov %eax,-0xac(%rbp) > 0xffffffff811f0020 <n_tty_read+505>: movzbl 0x17(%rdx),%edx > 0xffffffff811f0024 <n_tty_read+509>: test %edx,%edx > 0xffffffff811f0026 <n_tty_read+511>: mov %edx,-0xb0(%rbp) > 0xffffffff811f002c <n_tty_read+517>: je 0xffffffff811f0082 > <n_tty_read+603> > 0xffffffff811f002e <n_tty_read+519>: test %eax,%eax > 0xffffffff811f0030 <n_tty_read+521>: je 0xffffffff811f003e > <n_tty_read+535> > 0xffffffff811f0032 <n_tty_read+523>: movw $0x1,0x21e(%r13) > 0xffffffff811f003c <n_tty_read+533>: jmp 0xffffffff811f0076 > <n_tty_read+591> > 0xffffffff811f003e <n_tty_read+535>: mov -0x90(%rbp),%rbx > 0xffffffff811f0045 <n_tty_read+542>: cmp %rbx,0x1c8(%r13) > 0xffffffff811f004c <n_tty_read+549>: je 0xffffffff811f0068 > <n_tty_read+577> > 0xffffffff811f004e <n_tty_read+551>: movzwl 0x21e(%r13),%eax > 0xffffffff811f0056 <n_tty_read+559>: mov $0x7fffffffffffffff,%r15 > 0xffffffff811f0060 <n_tty_read+569>: cmp -0xb0(%rbp),%eax > 0xffffffff811f0066 <n_tty_read+575>: jle 0xffffffff811f00b7 > <n_tty_read+656> > 0xffffffff811f0068 <n_tty_read+577>: mov -0xb0(%rbp),%eax > 0xffffffff811f006e <n_tty_read+583>: mov %ax,0x21e(%r13) > 0xffffffff811f0076 <n_tty_read+591>: mov $0x7fffffffffffffff,%r15 > 0xffffffff811f0080 <n_tty_read+601>: jmp 0xffffffff811f00b7 > <n_tty_read+656> > 0xffffffff811f0082 <n_tty_read+603>: movslq -0xac(%rbp),%r15 > 0xffffffff811f0089 <n_tty_read+610>: cmpl $0x0,-0xac(%rbp) > 0xffffffff811f0090 <n_tty_read+617>: mov $0x0,%eax > 0xffffffff811f0095 <n_tty_read+622>: movw $0x1,0x21e(%r13) > 0xffffffff811f009f <n_tty_read+632>: movl $0x1,-0xb0(%rbp) > 0xffffffff811f00a9 <n_tty_read+642>: movl $0x0,-0xac(%rbp) > 0xffffffff811f00b3 <n_tty_read+652>: cmove %rax,%r15 > 0xffffffff811f00b7 <n_tty_read+656>: mov -0xd0(%rbp),%rdx > 0xffffffff811f00be <n_tty_read+663>: testb $0x8,0x39(%rdx) > 0xffffffff811f00c2 <n_tty_read+667>: je 0xffffffff811f00e4 > <n_tty_read+701> > 0xffffffff811f00c4 <n_tty_read+669>: mov -0xc0(%rbp),%rdi > 0xffffffff811f00cb <n_tty_read+676>: callq 0xffffffff812d5ec7 > <mutex_trylock> > 0xffffffff811f00d0 <n_tty_read+681>: test %eax,%eax > 0xffffffff811f00d2 <n_tty_read+683>: jne 0xffffffff811f0104 > <n_tty_read+733> > 0xffffffff811f00d4 <n_tty_read+685>: movq > $0xfffffffffffffff5,-0xa8(%rbp) > 0xffffffff811f00df <n_tty_read+696>: jmpq 0xffffffff811f05eb > <n_tty_read+1988> > 0xffffffff811f00e4 <n_tty_read+701>: mov -0xc0(%rbp),%rdi > 0xffffffff811f00eb <n_tty_read+708>: callq 0xffffffff812d6358 > <mutex_lock_interruptible> > 0xffffffff811f00f0 <n_tty_read+713>: test %eax,%eax > 0xffffffff811f00f2 <n_tty_read+715>: je 0xffffffff811f0104 > <n_tty_read+733> > 0xffffffff811f00f4 <n_tty_read+717>: movq > $0xfffffffffffffe00,-0xa8(%rbp) > 0xffffffff811f00ff <n_tty_read+728>: jmpq 0xffffffff811f05eb > <n_tty_read+1988> > 0xffffffff811f0104 <n_tty_read+733>: mov 0xec(%r13),%al > 0xffffffff811f010b <n_tty_read+740>: mov -0xc8(%rbp),%rdi > 0xffffffff811f0112 <n_tty_read+747>: lea -0x70(%rbp),%rsi > 0xffffffff811f0116 <n_tty_read+751>: shr $0x3,%al > 0xffffffff811f0119 <n_tty_read+754>: mov %eax,%ecx > 0xffffffff811f011b <n_tty_read+756>: and $0x1,%ecx > 0xffffffff811f011e <n_tty_read+759>: mov %ecx,-0x9c(%rbp) > 0xffffffff811f0124 <n_tty_read+765>: callq 0xffffffff8106201b > <add_wait_queue> > 0xffffffff811f0129 <n_tty_read+770>: movslq -0xb0(%rbp),%rbx > 0xffffffff811f0130 <n_tty_read+777>: movslq -0xac(%rbp),%rax > 0xffffffff811f0137 <n_tty_read+784>: mov -0xd8(%rbp),%rdx > 0xffffffff811f013e <n_tty_read+791>: inc %rdx > 0xffffffff811f0141 <n_tty_read+794>: mov %rbx,-0xe0(%rbp) > 0xffffffff811f0148 <n_tty_read+801>: mov %rax,-0xe8(%rbp) > 0xffffffff811f014f <n_tty_read+808>: mov %rdx,-0xf0(%rbp) > 0xffffffff811f0156 <n_tty_read+815>: jmpq 0xffffffff811f0522 > <n_tty_read+1787> > 0xffffffff811f015b <n_tty_read+820>: cmpl $0x0,-0x9c(%rbp) > 0xffffffff811f0162 <n_tty_read+827>: je 0xffffffff811f01ef > <n_tty_read+968> > 0xffffffff811f0168 <n_tty_read+833>: mov 0xf8(%r13),%rax > 0xffffffff811f016f <n_tty_read+840>: cmpb $0x0,0xed(%rax) > 0xffffffff811f0176 <n_tty_read+847>: je 0xffffffff811f01ef > <n_tty_read+968> > 0xffffffff811f0178 <n_tty_read+849>: mov -0xd8(%rbp),%rcx > 0xffffffff811f017f <n_tty_read+856>: cmp %rcx,-0x40(%rbp) > 0xffffffff811f0183 <n_tty_read+860>: jne 0xffffffff811f052d > <n_tty_read+1798> > 0xffffffff811f0189 <n_tty_read+866>: lea 0x68(%rax),%rdi > 0xffffffff811f018d <n_tty_read+870>: callq 0xffffffff812d6fb8 > <_spin_lock_irqsave> > 0xffffffff811f0192 <n_tty_read+875>: mov 0xf8(%r13),%rdi > 0xffffffff811f0199 <n_tty_read+882>: mov %rax,%rsi > 0xffffffff811f019c <n_tty_read+885>: mov 0xed(%rdi),%bl > 0xffffffff811f01a2 <n_tty_read+891>: movb $0x0,0xed(%rdi) > 0xffffffff811f01a9 <n_tty_read+898>: add $0x68,%rdi > 0xffffffff811f01ad <n_tty_read+902>: callq 0xffffffff812d70c1 > <_spin_unlock_irqrestore> > 0xffffffff811f01b2 <n_tty_read+907>: mov -0x40(%rbp),%r12 > 0xffffffff811f01b6 <n_tty_read+911>: lea -0x31(%rbp),%rsi > 0xffffffff811f01ba <n_tty_read+915>: mov $0x1,%edx > 0xffffffff811f01bf <n_tty_read+920>: mov %r13,%rdi > 0xffffffff811f01c2 <n_tty_read+923>: mov %bl,-0x31(%rbp) > 0xffffffff811f01c5 <n_tty_read+926>: lea 0x1(%r12),%rax > 0xffffffff811f01ca <n_tty_read+931>: mov %rax,-0x40(%rbp) > 0xffffffff811f01ce <n_tty_read+935>: callq 0xffffffff812008ac > <tty_audit_add_data> > 0xffffffff811f01d3 <n_tty_read+940>: mov -0x31(%rbp),%al > 0xffffffff811f01d6 <n_tty_read+943>: mov %r12,%rcx > 0xffffffff811f01d9 <n_tty_read+946>: callq 0xffffffff811949a0 > <__put_user_1> > 0xffffffff811f01de <n_tty_read+951>: test %eax,%eax > 0xffffffff811f01e0 <n_tty_read+953>: jne 0xffffffff811f043d > <n_tty_read+1558> > 0xffffffff811f01e6 <n_tty_read+959>: decq -0x78(%rbp) > 0xffffffff811f01ea <n_tty_read+963>: jmpq 0xffffffff811f052d > <n_tty_read+1798> > 0xffffffff811f01ef <n_tty_read+968>: mov $0x1,%eax > 0xffffffff811f01f4 <n_tty_read+973>: mov -0x100(%rbp),%rbx > 0xffffffff811f01fb <n_tty_read+980>: xchg %rax,(%rbx) > 0xffffffff811f01fe <n_tty_read+983>: mov -0x40(%rbp),%rcx > 0xffffffff811f0202 <n_tty_read+987>: mov -0xd8(%rbp),%rax > 0xffffffff811f0209 <n_tty_read+994>: mov -0xe0(%rbp),%rbx > 0xffffffff811f0210 <n_tty_read+1001>: sub %rcx,%rax > 0xffffffff811f0213 <n_tty_read+1004>: lea (%rax,%rbx,1),%rdx > 0xffffffff811f0217 <n_tty_read+1008>: movzwl 0x21e(%r13),%eax > 0xffffffff811f021f <n_tty_read+1016>: cmp %rax,%rdx > 0xffffffff811f0222 <n_tty_read+1019>: jge 0xffffffff811f0240 > <n_tty_read+1049> > 0xffffffff811f0224 <n_tty_read+1021>: test %rdx,%rdx > 0xffffffff811f0227 <n_tty_read+1024>: jle 0xffffffff811f0240 > <n_tty_read+1049> > 0xffffffff811f0229 <n_tty_read+1026>: mov -0xd8(%rbp),%eax > 0xffffffff811f022f <n_tty_read+1032>: sub %cx,%ax > 0xffffffff811f0232 <n_tty_read+1035>: add -0xb0(%rbp),%eax > 0xffffffff811f0238 <n_tty_read+1041>: mov %ax,0x21e(%r13) > 0xffffffff811f0240 <n_tty_read+1049>: mov %r13,%rdi > 0xffffffff811f0243 <n_tty_read+1052>: callq 0xffffffff811f37f3 > <tty_flush_to_ldisc> > 0xffffffff811f0248 <n_tty_read+1057>: testb $0x10,0x21c(%r13) > 0xffffffff811f0250 <n_tty_read+1065>: je 0xffffffff811f0261 > <n_tty_read+1082> > 0xffffffff811f0252 <n_tty_read+1067>: cmpl $0x0,0x478(%r13) > 0xffffffff811f025a <n_tty_read+1075>: jne 0xffffffff811f026f > <n_tty_read+1096> > 0xffffffff811f025c <n_tty_read+1077>: jmpq 0xffffffff811f0621 > <n_tty_read+2042> > 0xffffffff811f0261 <n_tty_read+1082>: cmpl $0x0,0x260(%r13) > 0xffffffff811f0269 <n_tty_read+1090>: jle 0xffffffff811f0621 > <n_tty_read+2042> > 0xffffffff811f026f <n_tty_read+1096>: mov -0x110(%rbp),%rax > 0xffffffff811f0276 <n_tty_read+1103>: movq $0x0,(%rax) > 0xffffffff811f027d <n_tty_read+1110>: cmpl $0x0,-0x9c(%rbp) > 0xffffffff811f0284 <n_tty_read+1117>: mov -0x40(%rbp),%rax > 0xffffffff811f0288 <n_tty_read+1121>: je 0xffffffff811f0376 > <n_tty_read+1359> > 0xffffffff811f028e <n_tty_read+1127>: cmp -0xd8(%rbp),%rax > 0xffffffff811f0295 <n_tty_read+1134>: jne 0xffffffff811f0376 > <n_tty_read+1359> > 0xffffffff811f029b <n_tty_read+1140>: jmpq 0xffffffff811f033b > <n_tty_read+1300> > 0xffffffff811f02a0 <n_tty_read+1145>: mov -0xd0(%rbp),%rdi > 0xffffffff811f02a7 <n_tty_read+1152>: callq 0xffffffff811eb980 > <tty_hung_up_p> > 0xffffffff811f02ac <n_tty_read+1157>: test %eax,%eax > 0xffffffff811f02ae <n_tty_read+1159>: jne 0xffffffff811f052d > <n_tty_read+1798> > 0xffffffff811f02b4 <n_tty_read+1165>: test %r15,%r15 > 0xffffffff811f02b7 <n_tty_read+1168>: je 0xffffffff811f052d > <n_tty_read+1798> > 0xffffffff811f02bd <n_tty_read+1174>: mov -0xd0(%rbp),%rdx > 0xffffffff811f02c4 <n_tty_read+1181>: testb $0x8,0x39(%rdx) > 0xffffffff811f02c8 <n_tty_read+1185>: je 0xffffffff811f02da > <n_tty_read+1203> > 0xffffffff811f02ca <n_tty_read+1187>: movq > $0xfffffffffffffff5,-0xa8(%rbp) > 0xffffffff811f02d5 <n_tty_read+1198>: jmpq 0xffffffff811f052d > <n_tty_read+1798> > 0xffffffff811f02da <n_tty_read+1203>: mov -0x108(%rbp),%rcx > 0xffffffff811f02e1 <n_tty_read+1210>: mov 0x8(%rcx),%rax > 0xffffffff811f02e5 <n_tty_read+1214>: testb $0x4,0x10(%rax) > 0xffffffff811f02e9 <n_tty_read+1218>: je 0xffffffff811f02fb > <n_tty_read+1236> > 0xffffffff811f02eb <n_tty_read+1220>: movq > $0xfffffffffffffe00,-0xa8(%rbp) > 0xffffffff811f02f6 <n_tty_read+1231>: jmpq 0xffffffff811f052d > <n_tty_read+1798> > 0xffffffff811f02fb <n_tty_read+1236>: mov $0xfff,%eax > 0xffffffff811f0300 <n_tty_read+1241>: sub 0x260(%r13),%eax > 0xffffffff811f0307 <n_tty_read+1248>: test %eax,%eax > 0xffffffff811f0309 <n_tty_read+1250>: jg 0xffffffff811f0324 > <n_tty_read+1277> > 0xffffffff811f030b <n_tty_read+1252>: xor %eax,%eax > 0xffffffff811f030d <n_tty_read+1254>: testb $0x10,0x21c(%r13) > 0xffffffff811f0315 <n_tty_read+1262>: je 0xffffffff811f0324 > <n_tty_read+1277> > 0xffffffff811f0317 <n_tty_read+1264>: xor %eax,%eax > 0xffffffff811f0319 <n_tty_read+1266>: cmpl $0x0,0x478(%r13) > 0xffffffff811f0321 <n_tty_read+1274>: sete %al > 0xffffffff811f0324 <n_tty_read+1277>: mov %r15,%rdi > 0xffffffff811f0327 <n_tty_read+1280>: mov %eax,0xf0(%r13) > 0xffffffff811f032e <n_tty_read+1287>: callq 0xffffffff812d5a02 > <schedule_timeout> > 0xffffffff811f0333 <n_tty_read+1292>: mov %rax,%r15 > 0xffffffff811f0336 <n_tty_read+1295>: jmpq 0xffffffff811f0522 > <n_tty_read+1787> > 0xffffffff811f033b <n_tty_read+1300>: mov -0xf0(%rbp),%rbx > 0xffffffff811f0342 <n_tty_read+1307>: lea -0x31(%rbp),%rsi > 0xffffffff811f0346 <n_tty_read+1311>: mov $0x1,%edx > 0xffffffff811f034b <n_tty_read+1316>: mov %r13,%rdi > 0xffffffff811f034e <n_tty_read+1319>: movb $0x0,-0x31(%rbp) > 0xffffffff811f0352 <n_tty_read+1323>: mov %rbx,-0x40(%rbp) > 0xffffffff811f0356 <n_tty_read+1327>: callq 0xffffffff812008ac > <tty_audit_add_data> > 0xffffffff811f035b <n_tty_read+1332>: mov -0x31(%rbp),%al > 0xffffffff811f035e <n_tty_read+1335>: mov -0xd8(%rbp),%rcx > 0xffffffff811f0365 <n_tty_read+1342>: callq 0xffffffff811949a0 > <__put_user_1> > 0xffffffff811f036a <n_tty_read+1347>: test %eax,%eax > 0xffffffff811f036c <n_tty_read+1349>: jne 0xffffffff811f043d > <n_tty_read+1558> > 0xffffffff811f0372 <n_tty_read+1355>: decq -0x78(%rbp) > 0xffffffff811f0376 <n_tty_read+1359>: testb $0x10,0x21c(%r13) > 0xffffffff811f037e <n_tty_read+1367>: jne 0xffffffff811f0456 > <n_tty_read+1583> > 0xffffffff811f0384 <n_tty_read+1373>: jmpq 0xffffffff811f047a > <n_tty_read+1619> > 0xffffffff811f0389 <n_tty_read+1378>: mov 0x25c(%r13),%eax > 0xffffffff811f0390 <n_tty_read+1385>: mov -0x88(%rbp),%rbx > 0xffffffff811f0397 <n_tty_read+1392>: lock btr %eax,(%rbx) > 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d > 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx > 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax > 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi > 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx > > Below is the output of bt -a command in crash > > bt -a > PID: 0 TASK: ffffffff814204b0 CPU: 0 COMMAND: "swapper" > #0 [ffff880033007e80] crash_nmi_callback at ffffffff8101fbc9 > #1 [ffff880033007e90] notifier_call_chain at ffffffff81065893 > #2 [ffff880033007ed0] atomic_notifier_call_chain at ffffffff810658dd > #3 [ffff880033007ee0] notify_die at ffffffff8106597f > #4 [ffff880033007f10] do_nmi at ffffffff8100dc5d > #5 [ffff880033007f50] nmi at ffffffff812d76b0 > [exception RIP: mwait_idle+163] > RIP: ffffffff81013029 RSP: ffffffff813e3eb8 RFLAGS: 00000246 > RAX: 0000000000000000 RBX: ffffffff813e3fd8 RCX: 0000000000000000 > RDX: 0000000000000000 RSI: ffffffff813e3fd8 RDI: ffffffff81522308 > RBP: ffffffff813e3ec8 R8: 0000000000000000 R9: ffff88003306e290 > R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368 > R13: ffffffff814ccb30 R14: ffffffff814cdfa0 R15: ffffffff813e3fa8 > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 > --- <NMI exception stack> --- > #6 [ffffffff813e3eb8] mwait_idle at ffffffff81013029 > #7 [ffffffff813e3ed0] cpu_idle at ffffffff8100af21 > > PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet" > #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486 > #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230 > #2 [ffff88031ce75b20] oops_end at ffffffff8100fa38 > #3 [ffff88031ce75b50] no_context at ffffffff8102d801 > #4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9 > #5 [ffff88031ce75c70] bad_area at ffffffff8102da41 > #6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19 > #7 [ffff88031ce75cf0] page_fault at ffffffff812d7425 > [exception RIP: n_tty_read+1420] > RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246 > RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044 > RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c > RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000 > R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044 > R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 > #8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e > #9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8 > #10 [ffff88031ce75f40] sys_read at ffffffff810ebe48 > #11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2 > RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212 > RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000 > RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000 > RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000 > R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000 > R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0 > ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b > > PID: 0 TASK: ffff88031e0e3540 CPU: 2 COMMAND: "swapper" > #0 [ffff880033047e80] crash_nmi_callback at ffffffff8101fbc9 > #1 [ffff880033047e90] notifier_call_chain at ffffffff81065893 > #2 [ffff880033047ed0] atomic_notifier_call_chain at ffffffff810658dd > #3 [ffff880033047ee0] notify_die at ffffffff8106597f > #4 [ffff880033047f10] do_nmi at ffffffff8100dc5d > #5 [ffff880033047f50] nmi at ffffffff812d76b0 > [exception RIP: mwait_idle+163] > RIP: ffffffff81013029 RSP: ffff88031e0e5ef8 RFLAGS: 00000246 > RAX: 0000000000000000 RBX: ffff88031e0e5fd8 RCX: 0000000000000000 > RDX: 0000000000000000 RSI: ffff88031e0e5fd8 RDI: ffffffff81522308 > RBP: ffff88031e0e5f08 R8: 0000000000000000 R9: ffff88003302e290 > R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368 > R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 > --- <NMI exception stack> --- > #6 [ffff88031e0e5ef8] mwait_idle at ffffffff81013029 > #7 [ffff88031e0e5f10] cpu_idle at ffffffff8100af21 > > PID: 0 TASK: ffff88031e113580 CPU: 3 COMMAND: "swapper" > #0 [ffff880033067e80] crash_nmi_callback at ffffffff8101fbc9 > #1 [ffff880033067e90] notifier_call_chain at ffffffff81065893 > #2 [ffff880033067ed0] atomic_notifier_call_chain at ffffffff810658dd > #3 [ffff880033067ee0] notify_die at ffffffff8106597f > #4 [ffff880033067f10] do_nmi at ffffffff8100dc5d > #5 [ffff880033067f50] nmi at ffffffff812d76b0 > [exception RIP: mwait_idle+163] > RIP: ffffffff81013029 RSP: ffff88031e115ef8 RFLAGS: 00000246 > RAX: 0000000000000000 RBX: ffff88031e115fd8 RCX: 0000000000000000 > RDX: 0000000000000000 RSI: ffff88031e115fd8 RDI: ffffffff81522308 > RBP: ffff88031e115f08 R8: 0000000000000000 R9: 0000000000000000 > R10: 0000000000000800 R11: 0000000000000000 R12: ffffffff8147e368 > R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 > --- <NMI exception stack> --- > #6 [ffff88031e115ef8] mwait_idle at ffffffff81013029 > #7 [ffff88031e115f10] cpu_idle at ffffffff8100af21 > > Please let me know if you need any other details. > > Thanks and Regards > Shashidhara > > > -----Original Message----- > From: crash-utility-bounces(a)redhat.com > [mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson > Sent: Thursday, June 23, 2011 9:35 PM > To: Discussion list for crash utility usage,maintenance and > development > Subject: Re: [Crash-utility] Unable to switch stack frames while > using > crash > > > > ----- Original Message ----- > > > BTW, are you sure about that? > > > > Presuming that the "tty" pointer is ffff8802cbd54800 as you've > > shown > below, > > and therefore tty->read_buf is 0xffff8802cbfe6000 and > > tty->read_tail > is 0, > > then the statement above would be simply be reading > > tty->read_buf[0], > or > > virtual address 0xffff8802cbfe6000. But the oops shows it faulting > > on > a > > virtual address of "5": > > > > BUG: unable to handle kernel NULL pointer dereference at > 0000000000000005 > > Just for my own sanity, can you either attach the > "drivers/char/n_tty.c" > > from *your* specific kernel, or get the source-code/line-number data > from > the embedded gdb module? > > If you don't have the n_tty.c file readily available, you can get > the > source-code/line-number data of a particular function by doing > something > like this: > > Get the line number of the beginning of n_tty_read(), which in my > kernel > is at 1698 -- your's will probably be different: > > crash> gdb list n_tty_read > 1695 * This code must be sure never to sleep through a hangup. > 1696 */ > 1697 > 1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file > *file, > 1699 unsigned char __user *buf, size_t nr) > 1700 { > 1701 unsigned char __user *b = buf; > 1702 DECLARE_WAITQUEUE(wait, current); > 1703 int c; > 1704 int minimum, time; > crash> > > Then get the line number of the next function in the file, which is > n_tty_write(): > > crash> gdb list n_tty_write > 1918 * lock themselves) > 1919 */ > 1920 > 1921 static ssize_t n_tty_write(struct tty_struct *tty, struct file > *file, > 1922 const unsigned char *buf, size_t nr) > 1923 { > 1924 const unsigned char *b = buf; > 1925 DECLARE_WAITQUEUE(wait, current); > 1926 int c; > 1927 ssize_t retval = 0; > > And then dump the whole n_tty_read() function (plus some extra > stuff): > > crash> gdb list 1698,1920 > 1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file > *file, > 1699 unsigned char __user *buf, size_t nr) > 1700 { > 1701 unsigned char __user *b = buf; > 1702 DECLARE_WAITQUEUE(wait, current); > 1703 int c; > 1704 int minimum, time; > 1705 ssize_t retval = 0; > 1706 ssize_t size; > 1707 long timeout; > 1708 unsigned long flags; > 1709 int packet; > 1710 > 1711 do_it_again: > 1712 > 1713 BUG_ON(!tty->read_buf); > 1714 > 1715 c = job_control(tty, file); > 1716 if (c < 0) > 1717 return c; > 1718 > 1719 minimum = time = 0; > 1720 timeout = MAX_SCHEDULE_TIMEOUT; > 1721 if (!tty->icanon) { > 1722 time = (HZ / 10) * TIME_CHAR(tty); > 1723 minimum = MIN_CHAR(tty); > ... > > And lastly, since the crash occurred at > > IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818 > > Do this: > > crash> dis -rl n_tty_read+0x58c > ... > > And then post all of that data. > > Dave > > > -- > Crash-utility mailing list > Crash-utility(a)redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility > > Information transmitted by this e-mail is proprietary to MphasiS, > its > associated companies and/ or its customers and is intended > for use only by the individual or entity to which it is addressed, > and > may contain information that is privileged, confidential or > exempt from disclosure under applicable law. If you are not the > intended recipient or it appears that this mail has been forwarded > to you without proper authority, you are notified that any use or > dissemination of this information in any manner is strictly > prohibited. In such cases, please notify us immediately at > mailmaster(a)mphasis.com and delete this mail from your records. > > > -- > Crash-utility mailing list > Crash-utility(a)redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please notify us immediately at mailmaster(a)mphasis.com and delete this mail from your records. -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility