Re: [Crash-utility] Unable to switch stack frames while using crash

Hi Dave, Thanks again for the response. When I check the address of tty->read_buf using kmem command in crash, below is the output. Which shows that the memory is still not freed. If the tty_close handler had been invoked, before accessing tty->read_buf we should not get this output. Please correct me if I am erring. kmem 0xffff8802cbfe6000 CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE ffff88031f8039c0 size-4096 4096 6322 6393 6393 4k SLAB MEMORY TOTAL ALLOCATED FREE ffff8802ac4811c0 ffff8802cbfe6000 1 1 0 FREE / [ALLOCATED] [ffff8802cbfe6000] PAGE PHYSICAL MAPPING INDEX CNT FLAGS ffffea0009c9fa50 2cbfe6000 0 0 1 200000000000080 I would like to thank every one for your time and effort in analyzing this issue. Thanks and Regards Shashidhara -----Original Message----- From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson Sent: Friday, June 24, 2011 7:10 PM To: Discussion list for crash utility usage,maintenance and development Subject: Re: [Crash-utility] Unable to switch stack frames while using crash ---- Original Message ----- below, is 0, or on a Well, as it turns out, you have every reason to be sure about that... Anyway, I don't understand why line numbers are not available with your kernel: ... [ cut ] ... But nonetheless, there is only on movsbl instruction in n_tty_read(), and looking at a RHEL6 kernel, you were correct in your original determination of the faulting instruction: crash> dis n_tty_read | grep movsbl 0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx crash> dis -rl 0xffffffff812f88c9 | tail ... [ cut ] ... /usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/drivers/char/n_tty.c: 1821 0xffffffff812f88c9 <n_tty_read+0x2c9>: movsbl (%rdx,%rax,1),%ebx crash> which is here: 1814 if (tty->icanon) { 1815 /* N.B. avoid overrun if nr == 0 */ 1816 while (nr && tty->read_cnt) { 1817 int eol; 1818 1819 eol = test_and_clear_bit(tty->read_tail, 1820 tty->read_flags); 1821 c = tty->read_buf[tty->read_tail]; The tty_struct offsets are these: crash> tty_struct -o struct tty_struct { ... [ cut ]... [0x250] char *read_buf; [0x258] int read_head; [0x25c] int read_tail; ... And you can see in the previous instructions the tty->read_buf (0x250) and tty->read_tail (0x25c) offsets being added to the tty_struct base address that is located in R13: And as you originally reported, the tty_struct address in R13 is ffff8802cbd54800: PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet" #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486 #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230 #2 [ffff88031ce75b20] oops_end at ffffffff8100fa38 #3 [ffff88031ce75b50] no_context at ffffffff8102d801 #4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9 #5 [ffff88031ce75c70] bad_area at ffffffff8102da41 #6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19 #7 [ffff88031ce75cf0] page_fault at ffffffff812d7425 [exception RIP: n_tty_read+1420] RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044 RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044 R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 #8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e #9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8 #10 [ffff88031ce75f40] sys_read at ffffffff810ebe48 #11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2 RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212 RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000 RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000 RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000 R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0 ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b But for whatever reason -- and I cannot explain it -- after these instructions were done: the RDX register ended up with 0000000000000005, and the RAX register with a 0000000000000000, leading to the: BUG: unable to handle kernel NULL pointer dereference at 0000000000000005 But when you display the tty_struct at ffff8802cbd54800, you see the read_buf and read_tail with seemingly legitimate values: crash> tty_struct ffff8802cbd54800 struct tty_struct { ... magic = 21505, driver = 0xffff88031b54ea00, ops = 0xffffffff8130f650, name = "pts9\000\...", driver_data = 0xffff88029c8a9668, icanon = 1 '\001', read_buf = 0xffff8802cbfe6000 "", read_head = 0, read_tail = 0, read_cnt = 0, ... So everything in your analysis was correct, but how it is possible that the RDX and RAX registers to have ended up with 0 and 5 is hard to explain. And for that matter, since tty->read_cnt is 0 above, your original question as to how that code path was taken to begin with is also valid. So I don't know -- anybody on the list ever seen anything like this? Stumped, Dave ----- Original Message ----- > Hi Dave, > > Thank you so much for your help. > > Below is the output of dis -rl n_tty_read+0x58c > > 0xffffffff811efe27 <n_tty_read>: push %rbp > 0xffffffff811efe28 <n_tty_read+1>: mov %gs:0xb500,%rax > 0xffffffff811efe31 <n_tty_read+10>: mov %rsp,%rbp > 0xffffffff811efe34 <n_tty_read+13>: push %r15 > 0xffffffff811efe36 <n_tty_read+15>: push %r14 > 0xffffffff811efe38 <n_tty_read+17>: push %r13 > 0xffffffff811efe3a <n_tty_read+19>: mov %rdi,%r13 > 0xffffffff811efe3d <n_tty_read+22>: lea -0x70(%rbp),%rdi > 0xffffffff811efe41 <n_tty_read+26>: push %r12 > 0xffffffff811efe43 <n_tty_read+28>: push %rbx > 0xffffffff811efe44 <n_tty_read+29>: lea 0x490(%r13),%rbx > 0xffffffff811efe4b <n_tty_read+36>: sub $0xe8,%rsp > 0xffffffff811efe52 <n_tty_read+43>: mov %rax,-0x98(%rbp) > 0xffffffff811efe59 <n_tty_read+50>: mov %rcx,-0x78(%rbp) > 0xffffffff811efe5d <n_tty_read+54>: xor %eax,%eax > 0xffffffff811efe5f <n_tty_read+56>: mov $0xa,%ecx > 0xffffffff811efe64 <n_tty_read+61>: mov %rdx,-0xd8(%rbp) > 0xffffffff811efe6b <n_tty_read+68>: mov %rsi,-0xd0(%rbp) > 0xffffffff811efe72 <n_tty_read+75>: mov %rdx,-0x40(%rbp) > 0xffffffff811efe76 <n_tty_read+79>: rep stos %eax,%es:(%rdi) > 0xffffffff811efe78 <n_tty_read+81>: lea 0x1c0(%r13),%rax > 0xffffffff811efe7f <n_tty_read+88>: lea 0x1c8(%r13),%rcx > 0xffffffff811efe86 <n_tty_read+95>: mov %rbx,-0xc0(%rbp) > 0xffffffff811efe8d <n_tty_read+102>: lea 0xd8(%r13),%rbx > 0xffffffff811efe94 <n_tty_read+109>: movq > $0xffffffff81045f84,-0x60(%rbp) > 0xffffffff811efe9c <n_tty_read+117>: movq $0x0,-0xa8(%rbp) > 0xffffffff811efea7 <n_tty_read+128>: mov -0x98(%rbp),%rdx > 0xffffffff811efeae <n_tty_read+135>: mov %rax,-0xc8(%rbp) > 0xffffffff811efeb5 <n_tty_read+142>: mov -0x98(%rbp),%rax > 0xffffffff811efebc <n_tty_read+149>: mov %rcx,-0x90(%rbp) > 0xffffffff811efec3 <n_tty_read+156>: lea 0x51c(%r13),%rcx > 0xffffffff811efeca <n_tty_read+163>: mov %rbx,-0x80(%rbp) > 0xffffffff811efece <n_tty_read+167>: mov %rdx,-0x68(%rbp) > 0xffffffff811efed2 <n_tty_read+171>: lea 0x268(%r13),%rdx > 0xffffffff811efed9 <n_tty_read+178>: mov %rcx,-0xb8(%rbp) > 0xffffffff811efee0 <n_tty_read+185>: mov %rax,-0xf8(%rbp) > 0xffffffff811efee7 <n_tty_read+192>: mov %rax,-0x100(%rbp) > 0xffffffff811efeee <n_tty_read+199>: mov %rdx,-0x88(%rbp) > 0xffffffff811efef5 <n_tty_read+206>: mov %rax,-0x108(%rbp) > 0xffffffff811efefc <n_tty_read+213>: mov %rax,-0x110(%rbp) > 0xffffffff811eff03 <n_tty_read+220>: cmpq $0x0,0x250(%r13) > 0xffffffff811eff0b <n_tty_read+228>: jne 0xffffffff811eff11 > <n_tty_read+234> > 0xffffffff811eff0d <n_tty_read+230>: ud2a > 0xffffffff811eff0f <n_tty_read+232>: jmp 0xffffffff811eff0f > <n_tty_read+232> > 0xffffffff811eff11 <n_tty_read+234>: mov -0xd0(%rbp),%rdx > 0xffffffff811eff18 <n_tty_read+241>: mov 0x20(%rdx),%rax > 0xffffffff811eff1c <n_tty_read+245>: cmpq > $0xffffffff811ed61f,0x18(%rax) > 0xffffffff811eff24 <n_tty_read+253>: je 0xffffffff811effef > <n_tty_read+456> > 0xffffffff811eff2a <n_tty_read+259>: mov -0xf8(%rbp),%rcx > 0xffffffff811eff31 <n_tty_read+266>: mov 0x478(%rcx),%rax > 0xffffffff811eff38 <n_tty_read+273>: cmp %r13,0x180(%rax) > 0xffffffff811eff3f <n_tty_read+280>: jne 0xffffffff811effef > <n_tty_read+456> > 0xffffffff811eff45 <n_tty_read+286>: mov 0xc8(%r13),%rdx > 0xffffffff811eff4c <n_tty_read+293>: test %rdx,%rdx > 0xffffffff811eff4f <n_tty_read+296>: jne 0xffffffff811eff64 > <n_tty_read+317> > 0xffffffff811eff51 <n_tty_read+298>: mov $0xffffffff8139c972,%rdi > 0xffffffff811eff58 <n_tty_read+305>: xor %eax,%eax > 0xffffffff811eff5a <n_tty_read+307>: callq 0xffffffff812d4abf > <printk> > 0xffffffff811eff5f <n_tty_read+312>: jmpq 0xffffffff811effef > <n_tty_read+456> > 0xffffffff811eff64 <n_tty_read+317>: mov -0xf8(%rbp),%rbx > 0xffffffff811eff6b <n_tty_read+324>: mov 0x1e0(%rbx),%rax > 0xffffffff811eff72 <n_tty_read+331>: cmp %rdx,0x238(%rax) > 0xffffffff811eff79 <n_tty_read+338>: je 0xffffffff811effef > <n_tty_read+456> > 0xffffffff811eff7b <n_tty_read+340>: mov -0x98(%rbp),%rax > 0xffffffff811eff82 <n_tty_read+347>: testb $0x10,0x48a(%rax) > 0xffffffff811eff89 <n_tty_read+354>: jne 0xffffffff811f0611 > <n_tty_read+2026> > 0xffffffff811eff8f <n_tty_read+360>: mov 0x480(%rax),%rax > 0xffffffff811eff96 <n_tty_read+367>: cmpq $0x1,0x288(%rax) > 0xffffffff811eff9e <n_tty_read+375>: jne 0xffffffff811f0604 > <n_tty_read+2013> > 0xffffffff811effa4 <n_tty_read+381>: jmpq 0xffffffff811f0611 > <n_tty_read+2026> > 0xffffffff811effa9 <n_tty_read+386>: mov -0x98(%rbp),%rcx > 0xffffffff811effb0 <n_tty_read+393>: mov $0x1,%edx > 0xffffffff811effb5 <n_tty_read+398>: mov $0x15,%esi > 0xffffffff811effba <n_tty_read+403>: mov 0x1e0(%rcx),%rax > 0xffffffff811effc1 <n_tty_read+410>: mov 0x238(%rax),%rdi > 0xffffffff811effc8 <n_tty_read+417>: callq 0xffffffff8105953a > <kill_pgrp> > 0xffffffff811effcd <n_tty_read+422>: mov %gs:0xb508,%rdx > 0xffffffff811effd6 <n_tty_read+431>: lea -0x1fc8(%rdx),%rax > 0xffffffff811effdd <n_tty_read+438>: lock orb $0x4,-0x1fc8(%rdx) > 0xffffffff811effe5 <n_tty_read+446>: mov $0xfffffe00,%eax > 0xffffffff811effea <n_tty_read+451>: jmpq 0xffffffff811f0616 > <n_tty_read+2031> > 0xffffffff811effef <n_tty_read+456>: testb $0x10,0x21c(%r13) > 0xffffffff811efff7 <n_tty_read+464>: je 0xffffffff811f000f > <n_tty_read+488> > 0xffffffff811efff9 <n_tty_read+466>: movl $0x0,-0xb0(%rbp) > 0xffffffff811f0003 <n_tty_read+476>: movl $0x0,-0xac(%rbp) > 0xffffffff811f000d <n_tty_read+486>: jmp 0xffffffff811f0076 > <n_tty_read+591> > 0xffffffff811f000f <n_tty_read+488>: mov 0x70(%r13),%rdx > 0xffffffff811f0013 <n_tty_read+492>: movzbl 0x16(%rdx),%eax > 0xffffffff811f0017 <n_tty_read+496>: imul $0x19,%eax,%eax > 0xffffffff811f001a <n_tty_read+499>: mov %eax,-0xac(%rbp) > 0xffffffff811f0020 <n_tty_read+505>: movzbl 0x17(%rdx),%edx > 0xffffffff811f0024 <n_tty_read+509>: test %edx,%edx > 0xffffffff811f0026 <n_tty_read+511>: mov %edx,-0xb0(%rbp) > 0xffffffff811f002c <n_tty_read+517>: je 0xffffffff811f0082 > <n_tty_read+603> > 0xffffffff811f002e <n_tty_read+519>: test %eax,%eax > 0xffffffff811f0030 <n_tty_read+521>: je 0xffffffff811f003e > <n_tty_read+535> > 0xffffffff811f0032 <n_tty_read+523>: movw $0x1,0x21e(%r13) > 0xffffffff811f003c <n_tty_read+533>: jmp 0xffffffff811f0076 > <n_tty_read+591> > 0xffffffff811f003e <n_tty_read+535>: mov -0x90(%rbp),%rbx > 0xffffffff811f0045 <n_tty_read+542>: cmp %rbx,0x1c8(%r13) > 0xffffffff811f004c <n_tty_read+549>: je 0xffffffff811f0068 > <n_tty_read+577> > 0xffffffff811f004e <n_tty_read+551>: movzwl 0x21e(%r13),%eax > 0xffffffff811f0056 <n_tty_read+559>: mov $0x7fffffffffffffff,%r15 > 0xffffffff811f0060 <n_tty_read+569>: cmp -0xb0(%rbp),%eax > 0xffffffff811f0066 <n_tty_read+575>: jle 0xffffffff811f00b7 > <n_tty_read+656> > 0xffffffff811f0068 <n_tty_read+577>: mov -0xb0(%rbp),%eax > 0xffffffff811f006e <n_tty_read+583>: mov %ax,0x21e(%r13) > 0xffffffff811f0076 <n_tty_read+591>: mov $0x7fffffffffffffff,%r15 > 0xffffffff811f0080 <n_tty_read+601>: jmp 0xffffffff811f00b7 > <n_tty_read+656> > 0xffffffff811f0082 <n_tty_read+603>: movslq -0xac(%rbp),%r15 > 0xffffffff811f0089 <n_tty_read+610>: cmpl $0x0,-0xac(%rbp) > 0xffffffff811f0090 <n_tty_read+617>: mov $0x0,%eax > 0xffffffff811f0095 <n_tty_read+622>: movw $0x1,0x21e(%r13) > 0xffffffff811f009f <n_tty_read+632>: movl $0x1,-0xb0(%rbp) > 0xffffffff811f00a9 <n_tty_read+642>: movl $0x0,-0xac(%rbp) > 0xffffffff811f00b3 <n_tty_read+652>: cmove %rax,%r15 > 0xffffffff811f00b7 <n_tty_read+656>: mov -0xd0(%rbp),%rdx > 0xffffffff811f00be <n_tty_read+663>: testb $0x8,0x39(%rdx) > 0xffffffff811f00c2 <n_tty_read+667>: je 0xffffffff811f00e4 > <n_tty_read+701> > 0xffffffff811f00c4 <n_tty_read+669>: mov -0xc0(%rbp),%rdi > 0xffffffff811f00cb <n_tty_read+676>: callq 0xffffffff812d5ec7 > <mutex_trylock> > 0xffffffff811f00d0 <n_tty_read+681>: test %eax,%eax > 0xffffffff811f00d2 <n_tty_read+683>: jne 0xffffffff811f0104 > <n_tty_read+733> > 0xffffffff811f00d4 <n_tty_read+685>: movq > $0xfffffffffffffff5,-0xa8(%rbp) > 0xffffffff811f00df <n_tty_read+696>: jmpq 0xffffffff811f05eb > <n_tty_read+1988> > 0xffffffff811f00e4 <n_tty_read+701>: mov -0xc0(%rbp),%rdi > 0xffffffff811f00eb <n_tty_read+708>: callq 0xffffffff812d6358 > <mutex_lock_interruptible> > 0xffffffff811f00f0 <n_tty_read+713>: test %eax,%eax > 0xffffffff811f00f2 <n_tty_read+715>: je 0xffffffff811f0104 > <n_tty_read+733> > 0xffffffff811f00f4 <n_tty_read+717>: movq > $0xfffffffffffffe00,-0xa8(%rbp) > 0xffffffff811f00ff <n_tty_read+728>: jmpq 0xffffffff811f05eb > <n_tty_read+1988> > 0xffffffff811f0104 <n_tty_read+733>: mov 0xec(%r13),%al > 0xffffffff811f010b <n_tty_read+740>: mov -0xc8(%rbp),%rdi > 0xffffffff811f0112 <n_tty_read+747>: lea -0x70(%rbp),%rsi > 0xffffffff811f0116 <n_tty_read+751>: shr $0x3,%al > 0xffffffff811f0119 <n_tty_read+754>: mov %eax,%ecx > 0xffffffff811f011b <n_tty_read+756>: and $0x1,%ecx > 0xffffffff811f011e <n_tty_read+759>: mov %ecx,-0x9c(%rbp) > 0xffffffff811f0124 <n_tty_read+765>: callq 0xffffffff8106201b > <add_wait_queue> > 0xffffffff811f0129 <n_tty_read+770>: movslq -0xb0(%rbp),%rbx > 0xffffffff811f0130 <n_tty_read+777>: movslq -0xac(%rbp),%rax > 0xffffffff811f0137 <n_tty_read+784>: mov -0xd8(%rbp),%rdx > 0xffffffff811f013e <n_tty_read+791>: inc %rdx > 0xffffffff811f0141 <n_tty_read+794>: mov %rbx,-0xe0(%rbp) > 0xffffffff811f0148 <n_tty_read+801>: mov %rax,-0xe8(%rbp) > 0xffffffff811f014f <n_tty_read+808>: mov %rdx,-0xf0(%rbp) > 0xffffffff811f0156 <n_tty_read+815>: jmpq 0xffffffff811f0522 > <n_tty_read+1787> > 0xffffffff811f015b <n_tty_read+820>: cmpl $0x0,-0x9c(%rbp) > 0xffffffff811f0162 <n_tty_read+827>: je 0xffffffff811f01ef > <n_tty_read+968> > 0xffffffff811f0168 <n_tty_read+833>: mov 0xf8(%r13),%rax > 0xffffffff811f016f <n_tty_read+840>: cmpb $0x0,0xed(%rax) > 0xffffffff811f0176 <n_tty_read+847>: je 0xffffffff811f01ef > <n_tty_read+968> > 0xffffffff811f0178 <n_tty_read+849>: mov -0xd8(%rbp),%rcx > 0xffffffff811f017f <n_tty_read+856>: cmp %rcx,-0x40(%rbp) > 0xffffffff811f0183 <n_tty_read+860>: jne 0xffffffff811f052d > <n_tty_read+1798> > 0xffffffff811f0189 <n_tty_read+866>: lea 0x68(%rax),%rdi > 0xffffffff811f018d <n_tty_read+870>: callq 0xffffffff812d6fb8 > <_spin_lock_irqsave> > 0xffffffff811f0192 <n_tty_read+875>: mov 0xf8(%r13),%rdi > 0xffffffff811f0199 <n_tty_read+882>: mov %rax,%rsi > 0xffffffff811f019c <n_tty_read+885>: mov 0xed(%rdi),%bl > 0xffffffff811f01a2 <n_tty_read+891>: movb $0x0,0xed(%rdi) > 0xffffffff811f01a9 <n_tty_read+898>: add $0x68,%rdi > 0xffffffff811f01ad <n_tty_read+902>: callq 0xffffffff812d70c1 > <_spin_unlock_irqrestore> > 0xffffffff811f01b2 <n_tty_read+907>: mov -0x40(%rbp),%r12 > 0xffffffff811f01b6 <n_tty_read+911>: lea -0x31(%rbp),%rsi > 0xffffffff811f01ba <n_tty_read+915>: mov $0x1,%edx > 0xffffffff811f01bf <n_tty_read+920>: mov %r13,%rdi > 0xffffffff811f01c2 <n_tty_read+923>: mov %bl,-0x31(%rbp) > 0xffffffff811f01c5 <n_tty_read+926>: lea 0x1(%r12),%rax > 0xffffffff811f01ca <n_tty_read+931>: mov %rax,-0x40(%rbp) > 0xffffffff811f01ce <n_tty_read+935>: callq 0xffffffff812008ac > <tty_audit_add_data> > 0xffffffff811f01d3 <n_tty_read+940>: mov -0x31(%rbp),%al > 0xffffffff811f01d6 <n_tty_read+943>: mov %r12,%rcx > 0xffffffff811f01d9 <n_tty_read+946>: callq 0xffffffff811949a0 > <__put_user_1> > 0xffffffff811f01de <n_tty_read+951>: test %eax,%eax > 0xffffffff811f01e0 <n_tty_read+953>: jne 0xffffffff811f043d > <n_tty_read+1558> > 0xffffffff811f01e6 <n_tty_read+959>: decq -0x78(%rbp) > 0xffffffff811f01ea <n_tty_read+963>: jmpq 0xffffffff811f052d > <n_tty_read+1798> > 0xffffffff811f01ef <n_tty_read+968>: mov $0x1,%eax > 0xffffffff811f01f4 <n_tty_read+973>: mov -0x100(%rbp),%rbx > 0xffffffff811f01fb <n_tty_read+980>: xchg %rax,(%rbx) > 0xffffffff811f01fe <n_tty_read+983>: mov -0x40(%rbp),%rcx > 0xffffffff811f0202 <n_tty_read+987>: mov -0xd8(%rbp),%rax > 0xffffffff811f0209 <n_tty_read+994>: mov -0xe0(%rbp),%rbx > 0xffffffff811f0210 <n_tty_read+1001>: sub %rcx,%rax > 0xffffffff811f0213 <n_tty_read+1004>: lea (%rax,%rbx,1),%rdx > 0xffffffff811f0217 <n_tty_read+1008>: movzwl 0x21e(%r13),%eax > 0xffffffff811f021f <n_tty_read+1016>: cmp %rax,%rdx > 0xffffffff811f0222 <n_tty_read+1019>: jge 0xffffffff811f0240 > <n_tty_read+1049> > 0xffffffff811f0224 <n_tty_read+1021>: test %rdx,%rdx > 0xffffffff811f0227 <n_tty_read+1024>: jle 0xffffffff811f0240 > <n_tty_read+1049> > 0xffffffff811f0229 <n_tty_read+1026>: mov -0xd8(%rbp),%eax > 0xffffffff811f022f <n_tty_read+1032>: sub %cx,%ax > 0xffffffff811f0232 <n_tty_read+1035>: add -0xb0(%rbp),%eax > 0xffffffff811f0238 <n_tty_read+1041>: mov %ax,0x21e(%r13) > 0xffffffff811f0240 <n_tty_read+1049>: mov %r13,%rdi > 0xffffffff811f0243 <n_tty_read+1052>: callq 0xffffffff811f37f3 > <tty_flush_to_ldisc> > 0xffffffff811f0248 <n_tty_read+1057>: testb $0x10,0x21c(%r13) > 0xffffffff811f0250 <n_tty_read+1065>: je 0xffffffff811f0261 > <n_tty_read+1082> > 0xffffffff811f0252 <n_tty_read+1067>: cmpl $0x0,0x478(%r13) > 0xffffffff811f025a <n_tty_read+1075>: jne 0xffffffff811f026f > <n_tty_read+1096> > 0xffffffff811f025c <n_tty_read+1077>: jmpq 0xffffffff811f0621 > <n_tty_read+2042> > 0xffffffff811f0261 <n_tty_read+1082>: cmpl $0x0,0x260(%r13) > 0xffffffff811f0269 <n_tty_read+1090>: jle 0xffffffff811f0621 > <n_tty_read+2042> > 0xffffffff811f026f <n_tty_read+1096>: mov -0x110(%rbp),%rax > 0xffffffff811f0276 <n_tty_read+1103>: movq $0x0,(%rax) > 0xffffffff811f027d <n_tty_read+1110>: cmpl $0x0,-0x9c(%rbp) > 0xffffffff811f0284 <n_tty_read+1117>: mov -0x40(%rbp),%rax > 0xffffffff811f0288 <n_tty_read+1121>: je 0xffffffff811f0376 > <n_tty_read+1359> > 0xffffffff811f028e <n_tty_read+1127>: cmp -0xd8(%rbp),%rax > 0xffffffff811f0295 <n_tty_read+1134>: jne 0xffffffff811f0376 > <n_tty_read+1359> > 0xffffffff811f029b <n_tty_read+1140>: jmpq 0xffffffff811f033b > <n_tty_read+1300> > 0xffffffff811f02a0 <n_tty_read+1145>: mov -0xd0(%rbp),%rdi > 0xffffffff811f02a7 <n_tty_read+1152>: callq 0xffffffff811eb980 > <tty_hung_up_p> > 0xffffffff811f02ac <n_tty_read+1157>: test %eax,%eax > 0xffffffff811f02ae <n_tty_read+1159>: jne 0xffffffff811f052d > <n_tty_read+1798> > 0xffffffff811f02b4 <n_tty_read+1165>: test %r15,%r15 > 0xffffffff811f02b7 <n_tty_read+1168>: je 0xffffffff811f052d > <n_tty_read+1798> > 0xffffffff811f02bd <n_tty_read+1174>: mov -0xd0(%rbp),%rdx > 0xffffffff811f02c4 <n_tty_read+1181>: testb $0x8,0x39(%rdx) > 0xffffffff811f02c8 <n_tty_read+1185>: je 0xffffffff811f02da > <n_tty_read+1203> > 0xffffffff811f02ca <n_tty_read+1187>: movq > $0xfffffffffffffff5,-0xa8(%rbp) > 0xffffffff811f02d5 <n_tty_read+1198>: jmpq 0xffffffff811f052d > <n_tty_read+1798> > 0xffffffff811f02da <n_tty_read+1203>: mov -0x108(%rbp),%rcx > 0xffffffff811f02e1 <n_tty_read+1210>: mov 0x8(%rcx),%rax > 0xffffffff811f02e5 <n_tty_read+1214>: testb $0x4,0x10(%rax) > 0xffffffff811f02e9 <n_tty_read+1218>: je 0xffffffff811f02fb > <n_tty_read+1236> > 0xffffffff811f02eb <n_tty_read+1220>: movq > $0xfffffffffffffe00,-0xa8(%rbp) > 0xffffffff811f02f6 <n_tty_read+1231>: jmpq 0xffffffff811f052d > <n_tty_read+1798> > 0xffffffff811f02fb <n_tty_read+1236>: mov $0xfff,%eax > 0xffffffff811f0300 <n_tty_read+1241>: sub 0x260(%r13),%eax > 0xffffffff811f0307 <n_tty_read+1248>: test %eax,%eax > 0xffffffff811f0309 <n_tty_read+1250>: jg 0xffffffff811f0324 > <n_tty_read+1277> > 0xffffffff811f030b <n_tty_read+1252>: xor %eax,%eax > 0xffffffff811f030d <n_tty_read+1254>: testb $0x10,0x21c(%r13) > 0xffffffff811f0315 <n_tty_read+1262>: je 0xffffffff811f0324 > <n_tty_read+1277> > 0xffffffff811f0317 <n_tty_read+1264>: xor %eax,%eax > 0xffffffff811f0319 <n_tty_read+1266>: cmpl $0x0,0x478(%r13) > 0xffffffff811f0321 <n_tty_read+1274>: sete %al > 0xffffffff811f0324 <n_tty_read+1277>: mov %r15,%rdi > 0xffffffff811f0327 <n_tty_read+1280>: mov %eax,0xf0(%r13) > 0xffffffff811f032e <n_tty_read+1287>: callq 0xffffffff812d5a02 > <schedule_timeout> > 0xffffffff811f0333 <n_tty_read+1292>: mov %rax,%r15 > 0xffffffff811f0336 <n_tty_read+1295>: jmpq 0xffffffff811f0522 > <n_tty_read+1787> > 0xffffffff811f033b <n_tty_read+1300>: mov -0xf0(%rbp),%rbx > 0xffffffff811f0342 <n_tty_read+1307>: lea -0x31(%rbp),%rsi > 0xffffffff811f0346 <n_tty_read+1311>: mov $0x1,%edx > 0xffffffff811f034b <n_tty_read+1316>: mov %r13,%rdi > 0xffffffff811f034e <n_tty_read+1319>: movb $0x0,-0x31(%rbp) > 0xffffffff811f0352 <n_tty_read+1323>: mov %rbx,-0x40(%rbp) > 0xffffffff811f0356 <n_tty_read+1327>: callq 0xffffffff812008ac > <tty_audit_add_data> > 0xffffffff811f035b <n_tty_read+1332>: mov -0x31(%rbp),%al > 0xffffffff811f035e <n_tty_read+1335>: mov -0xd8(%rbp),%rcx > 0xffffffff811f0365 <n_tty_read+1342>: callq 0xffffffff811949a0 > <__put_user_1> > 0xffffffff811f036a <n_tty_read+1347>: test %eax,%eax > 0xffffffff811f036c <n_tty_read+1349>: jne 0xffffffff811f043d > <n_tty_read+1558> > 0xffffffff811f0372 <n_tty_read+1355>: decq -0x78(%rbp) > 0xffffffff811f0376 <n_tty_read+1359>: testb $0x10,0x21c(%r13) > 0xffffffff811f037e <n_tty_read+1367>: jne 0xffffffff811f0456 > <n_tty_read+1583> > 0xffffffff811f0384 <n_tty_read+1373>: jmpq 0xffffffff811f047a > <n_tty_read+1619> > 0xffffffff811f0389 <n_tty_read+1378>: mov 0x25c(%r13),%eax > 0xffffffff811f0390 <n_tty_read+1385>: mov -0x88(%rbp),%rbx > 0xffffffff811f0397 <n_tty_read+1392>: lock btr %eax,(%rbx) > 0xffffffff811f039b <n_tty_read+1396>: sbb %r14d,%r14d > 0xffffffff811f039e <n_tty_read+1399>: movslq 0x25c(%r13),%rdx > 0xffffffff811f03a5 <n_tty_read+1406>: mov 0x250(%r13),%rax > 0xffffffff811f03ac <n_tty_read+1413>: mov -0xb8(%rbp),%rdi > 0xffffffff811f03b3 <n_tty_read+1420>: movsbl (%rax,%rdx,1),%ebx > > Below is the output of bt -a command in crash > > bt -a > PID: 0 TASK: ffffffff814204b0 CPU: 0 COMMAND: "swapper" > #0 [ffff880033007e80] crash_nmi_callback at ffffffff8101fbc9 > #1 [ffff880033007e90] notifier_call_chain at ffffffff81065893 > #2 [ffff880033007ed0] atomic_notifier_call_chain at ffffffff810658dd > #3 [ffff880033007ee0] notify_die at ffffffff8106597f > #4 [ffff880033007f10] do_nmi at ffffffff8100dc5d > #5 [ffff880033007f50] nmi at ffffffff812d76b0 > [exception RIP: mwait_idle+163] > RIP: ffffffff81013029 RSP: ffffffff813e3eb8 RFLAGS: 00000246 > RAX: 0000000000000000 RBX: ffffffff813e3fd8 RCX: 0000000000000000 > RDX: 0000000000000000 RSI: ffffffff813e3fd8 RDI: ffffffff81522308 > RBP: ffffffff813e3ec8 R8: 0000000000000000 R9: ffff88003306e290 > R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368 > R13: ffffffff814ccb30 R14: ffffffff814cdfa0 R15: ffffffff813e3fa8 > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 > --- <NMI exception stack> --- > #6 [ffffffff813e3eb8] mwait_idle at ffffffff81013029 > #7 [ffffffff813e3ed0] cpu_idle at ffffffff8100af21 > > PID: 13366 TASK: ffff88031b60d580 CPU: 1 COMMAND: "telnet" > #0 [ffff88031ce759d0] machine_kexec at ffffffff81024486 > #1 [ffff88031ce75a40] crash_kexec at ffffffff8107e230 > #2 [ffff88031ce75b20] oops_end at ffffffff8100fa38 > #3 [ffff88031ce75b50] no_context at ffffffff8102d801 > #4 [ffff88031ce75ba0] __bad_area_nosemaphore at ffffffff8102d9c9 > #5 [ffff88031ce75c70] bad_area at ffffffff8102da41 > #6 [ffff88031ce75ca0] do_page_fault at ffffffff8102dd19 > #7 [ffff88031ce75cf0] page_fault at ffffffff812d7425 > [exception RIP: n_tty_read+1420] > RIP: ffffffff811f03b3 RSP: ffff88031ce75da8 RFLAGS: 00010246 > RAX: 0000000000000000 RBX: ffff8802cbd54a68 RCX: 000000000061c044 > RDX: 0000000000000005 RSI: ffff88031ce75e87 RDI: ffff8802cbd54d1c > RBP: ffff88031ce75eb8 R8: 0000000000000000 R9: 0000000000000000 > R10: 0000000000616680 R11: 0000000000000246 R12: 000000000061c044 > R13: ffff8802cbd54800 R14: 0000000000000000 R15: 7fffffffffffffff > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 > #8 [ffff88031ce75ec0] tty_read at ffffffff811ebf7e > #9 [ffff88031ce75f10] vfs_read at ffffffff810ebcc8 > #10 [ffff88031ce75f40] sys_read at ffffffff810ebe48 > #11 [ffff88031ce75f80] system_call_fastpath at ffffffff8100bbc2 > RIP: 00007ffff716b9e0 RSP: 00007fffffffdfc0 RFLAGS: 00010212 > RAX: 0000000000000000 RBX: ffffffff8100bbc2 RCX: 0000000000000000 > RDX: 0000000000001ff6 RSI: 000000000061c02a RDI: 0000000000000000 > RBP: 0000000000001ff6 R8: 0000000000000000 R9: 0000000000000000 > R10: 0000000000616680 R11: 0000000000000246 R12: 0000000000000000 > R13: 0000000000000001 R14: 000000000061c02a R15: 00000000006178a0 > ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b > > PID: 0 TASK: ffff88031e0e3540 CPU: 2 COMMAND: "swapper" > #0 [ffff880033047e80] crash_nmi_callback at ffffffff8101fbc9 > #1 [ffff880033047e90] notifier_call_chain at ffffffff81065893 > #2 [ffff880033047ed0] atomic_notifier_call_chain at ffffffff810658dd > #3 [ffff880033047ee0] notify_die at ffffffff8106597f > #4 [ffff880033047f10] do_nmi at ffffffff8100dc5d > #5 [ffff880033047f50] nmi at ffffffff812d76b0 > [exception RIP: mwait_idle+163] > RIP: ffffffff81013029 RSP: ffff88031e0e5ef8 RFLAGS: 00000246 > RAX: 0000000000000000 RBX: ffff88031e0e5fd8 RCX: 0000000000000000 > RDX: 0000000000000000 RSI: ffff88031e0e5fd8 RDI: ffffffff81522308 > RBP: ffff88031e0e5f08 R8: 0000000000000000 R9: ffff88003302e290 > R10: 0000000000012d80 R11: 0000000000000000 R12: ffffffff8147e368 > R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 > --- <NMI exception stack> --- > #6 [ffff88031e0e5ef8] mwait_idle at ffffffff81013029 > #7 [ffff88031e0e5f10] cpu_idle at ffffffff8100af21 > > PID: 0 TASK: ffff88031e113580 CPU: 3 COMMAND: "swapper" > #0 [ffff880033067e80] crash_nmi_callback at ffffffff8101fbc9 > #1 [ffff880033067e90] notifier_call_chain at ffffffff81065893 > #2 [ffff880033067ed0] atomic_notifier_call_chain at ffffffff810658dd > #3 [ffff880033067ee0] notify_die at ffffffff8106597f > #4 [ffff880033067f10] do_nmi at ffffffff8100dc5d > #5 [ffff880033067f50] nmi at ffffffff812d76b0 > [exception RIP: mwait_idle+163] > RIP: ffffffff81013029 RSP: ffff88031e115ef8 RFLAGS: 00000246 > RAX: 0000000000000000 RBX: ffff88031e115fd8 RCX: 0000000000000000 > RDX: 0000000000000000 RSI: ffff88031e115fd8 RDI: ffffffff81522308 > RBP: ffff88031e115f08 R8: 0000000000000000 R9: 0000000000000000 > R10: 0000000000000800 R11: 0000000000000000 R12: ffffffff8147e368 > R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 > --- <NMI exception stack> --- > #6 [ffff88031e115ef8] mwait_idle at ffffffff81013029 > #7 [ffff88031e115f10] cpu_idle at ffffffff8100af21 > > Please let me know if you need any other details. > > Thanks and Regards > Shashidhara > > > -----Original Message----- > From: crash-utility-bounces(a)redhat.com > [mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson > Sent: Thursday, June 23, 2011 9:35 PM > To: Discussion list for crash utility usage,maintenance and > development > Subject: Re: [Crash-utility] Unable to switch stack frames while using > crash > > > > ----- Original Message ----- > > > BTW, are you sure about that? > > > > Presuming that the "tty" pointer is ffff8802cbd54800 as you've shown > below, > > and therefore tty->read_buf is 0xffff8802cbfe6000 and tty->read_tail > is 0, > > then the statement above would be simply be reading > > tty->read_buf[0], > or > > virtual address 0xffff8802cbfe6000. But the oops shows it faulting > > on > a > > virtual address of "5": > > > > BUG: unable to handle kernel NULL pointer dereference at > 0000000000000005 > > Just for my own sanity, can you either attach the > "drivers/char/n_tty.c" > > from *your* specific kernel, or get the source-code/line-number data > from > the embedded gdb module? > > If you don't have the n_tty.c file readily available, you can get the > source-code/line-number data of a particular function by doing > something > like this: > > Get the line number of the beginning of n_tty_read(), which in my > kernel > is at 1698 -- your's will probably be different: > > crash> gdb list n_tty_read > 1695 * This code must be sure never to sleep through a hangup. > 1696 */ > 1697 > 1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file > *file, > 1699 unsigned char __user *buf, size_t nr) > 1700 { > 1701 unsigned char __user *b = buf; > 1702 DECLARE_WAITQUEUE(wait, current); > 1703 int c; > 1704 int minimum, time; > crash> > > Then get the line number of the next function in the file, which is > n_tty_write(): > > crash> gdb list n_tty_write > 1918 * lock themselves) > 1919 */ > 1920 > 1921 static ssize_t n_tty_write(struct tty_struct *tty, struct file > *file, > 1922 const unsigned char *buf, size_t nr) > 1923 { > 1924 const unsigned char *b = buf; > 1925 DECLARE_WAITQUEUE(wait, current); > 1926 int c; > 1927 ssize_t retval = 0; > > And then dump the whole n_tty_read() function (plus some extra stuff): > > crash> gdb list 1698,1920 > 1698 static ssize_t n_tty_read(struct tty_struct *tty, struct file > *file, > 1699 unsigned char __user *buf, size_t nr) > 1700 { > 1701 unsigned char __user *b = buf; > 1702 DECLARE_WAITQUEUE(wait, current); > 1703 int c; > 1704 int minimum, time; > 1705 ssize_t retval = 0; > 1706 ssize_t size; > 1707 long timeout; > 1708 unsigned long flags; > 1709 int packet; > 1710 > 1711 do_it_again: > 1712 > 1713 BUG_ON(!tty->read_buf); > 1714 > 1715 c = job_control(tty, file); > 1716 if (c < 0) > 1717 return c; > 1718 > 1719 minimum = time = 0; > 1720 timeout = MAX_SCHEDULE_TIMEOUT; > 1721 if (!tty->icanon) { > 1722 time = (HZ / 10) * TIME_CHAR(tty); > 1723 minimum = MIN_CHAR(tty); > ... > > And lastly, since the crash occurred at > > IP: [<ffffffff811f03b3>] n_tty_read+0x58c/0x818 > > Do this: > > crash> dis -rl n_tty_read+0x58c > ... > > And then post all of that data. > > Dave > > > -- > Crash-utility mailing list > Crash-utility(a)redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility > > Information transmitted by this e-mail is proprietary to MphasiS, its > associated companies and/ or its customers and is intended > for use only by the individual or entity to which it is addressed, and > may contain information that is privileged, confidential or > exempt from disclosure under applicable law. If you are not the > intended recipient or it appears that this mail has been forwarded > to you without proper authority, you are notified that any use or > dissemination of this information in any manner is strictly > prohibited. In such cases, please notify us immediately at > mailmaster(a)mphasis.com and delete this mail from your records. > > > -- > Crash-utility mailing list > Crash-utility(a)redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility Information transmitted by this e-mail is proprietary to MphasiS, its associated companies and/ or its customers and is intended for use only by the individual or entity to which it is addressed, and may contain information that is privileged, confidential or exempt from disclosure under applicable law. If you are not the intended recipient or it appears that this mail has been forwarded to you without proper authority, you are notified that any use or dissemination of this information in any manner is strictly prohibited. In such cases, please notify us immediately at mailmaster(a)mphasis.com and delete this mail from your records.