Hi Rachita,

I've figured out why the x86_64 interrupt-stack-to-process-stack
transition is showing a bogus exception frame.  It's not kdump
or jprobes -- I think it may have been introduced with the DWARF
CFI changes.

Anyway, in older x86_64 kernels, when an interrupt was taken,
the pt_regs exception frame would be laid down on the current stack,
and the rdi register would contain a pointer to it.  Then the stack
pointer would be switched to the per-cpu interrupt stack.  (Actually
it is switched to a point 64 bytes from the top of the interrupt
stack, presumably for cache line purposes).  The first thing
done after having been switched to the interrupt stack is to push
the rdi register, which again, contains a pointer to the exception
frame on the other stack.  Then it calls the interrupt handler.

Here's the "old" code, where the last 4 instructions in the macro
shown below perform the steps outlined above:

1. get the per-cpu interrupt stack address,
2. move it into rsp -- which effectively switches stacks,
3. then the rdi register is pushed,
4. and the interrupt handler called:

        .macro interrupt func
        CFI_STARTPROC   simple
        CFI_DEF_CFA     rsp,(SS-RDI)
        CFI_REL_OFFSET  rsp,(RSP-ORIG_RAX)
        CFI_REL_OFFSET  rip,(RIP-ORIG_RAX)
        cld
#ifdef CONFIG_DEBUG_INFO
        SAVE_ALL
        movq %rsp,%rdi
        /*
         * Setup a stack frame pointer.  This allows gdb to trace
         * back to the original stack.
         */
        movq %rsp,%rbp
        CFI_DEF_CFA_REGISTER    rbp
#else
        SAVE_ARGS
        leaq -ARGOFFSET(%rsp),%rdi      # arg1 for handler
#endif
        testl $3,CS(%rdi)
        je 1f
        swapgs
1:      addl $1,%gs:pda_irqcount        # RED-PEN should check preempt count
        movq %gs:pda_irqstackptr,%rax
        cmoveq %rax,%rsp
        pushq %rdi                      # save old stack
        call \func
        .endm

However, in current x86_64 kernels, the interrupt macro has changed
to look like this:

        .macro interrupt func
        cld
        SAVE_ARGS
        leaq -ARGOFFSET(%rsp),%rdi      # arg1 for handler
        pushq %rbp
        CFI_ADJUST_CFA_OFFSET   8
        CFI_REL_OFFSET          rbp, 0
        movq %rsp,%rbp
        CFI_DEF_CFA_REGISTER    rbp
        testl $3,CS(%rdi)
        je 1f
        swapgs
1:      incl    %gs:pda_irqcount        # RED-PEN should check preempt count
        cmoveq %gs:pda_irqstackptr,%rsp
        push    %rbp                    # backlink for old unwinder
        /*
         * We entered an interrupt context - irqs are off:
         */
        TRACE_IRQS_OFF
        call \func
        .endm

Note that rdi still contains the pt_regs pointer, as evidenced by
the "testl $3,CS(%rdi)" instruction, which is checking the CS register
contents in the pt_regs for whether it was operating in user-space
when the interrupt occurred.  But more importantly, note that just
prior to calling the handler, it does a "push %rbp" instead of a
"pushq %rdi" like it used to.

I'm pretty sure it's being done purposely, because instead of the
having "old unwinder" dumping kernel text addresses starting inside
of the pt_regs exception frame, it bumps the starting point up to
whatever's contained in $rbp, which is above the exception frame
on the old stack.  So it would avoid dumping text return addresses
that happen to be sitting in the pt_regs register dump.

Just to verify, I patched the current kernel to push rdi instead
of rbp.  Again, here's what the unpatched alt-sysrq-c backtrace
looks like:

crash> bt
PID: 0      TASK: ffff81003fe48100  CPU: 1   COMMAND: "swapper"
 #0 [ffff81003fe6bb40] crash_kexec at ffffffff800ab798
 #1 [ffff81003fe6bbc8] mwait_idle at ffffffff80055375
 #2 [ffff81003fe6bc00] sysrq_handle_crashdump at ffffffff80192fdc
 #3 [ffff81003fe6bc10] __handle_sysrq at ffffffff80192dae
 #4 [ffff81003fe6bc50] kbd_event at ffffffff8018db52
 #5 [ffff81003fe6bca0] input_event at ffffffff801e9b6d
 #6 [ffff81003fe6bcd0] hidinput_hid_event at ffffffff801e4299
 #7 [ffff81003fe6bd00] hid_process_event at ffffffff801df639
 #8 [ffff81003fe6bd40] hid_input_report at ffffffff801df9a7
 #9 [ffff81003fe6bdc0] hid_irq_in at ffffffff801e0d8e
#10 [ffff81003fe6bde0] usb_hcd_giveback_urb at ffffffff801d33a2
#11 [ffff81003fe6be10] uhci_giveback_urb at ffffffff8817b724
#12 [ffff81003fe6be50] uhci_scan_schedule at ffffffff8817be07
#13 [ffff81003fe6bed0] uhci_irq at ffffffff8817dc08
#14 [ffff81003fe6bf10] usb_hcd_irq at ffffffff801d3d91
#15 [ffff81003fe6bf20] handle_IRQ_event at ffffffff800106fd
#16 [ffff81003fe6bf50] __do_IRQ at ffffffff800b520c
#17 [ffff81003fe6bf58] __do_softirq at ffffffff80011bfa
#18 [ffff81003fe6bf90] do_IRQ at ffffffff8006a729
--- <IRQ stack> ---
#19 [ffff81003fe65e70] ret_from_intr at ffffffff8005ba89
    [exception RIP: cpu_idle+149]
    RIP: ffffffff800473a7  RSP: ffffffff8042e220  RFLAGS: ffffffff80074153
    RAX: ffffffffffffff16  RBX: 0000000000000000  RCX: ffffffff80055375
    RDX: 0000000000000010  RSI: 0000000000000246  RDI: ffff81003fe65ef0
    RBP: ffff81003fe64000   R8: ffffffff8034e818   R9: 0000000000000001
    R10: 0000000000000000  R11: 0000000000000000  R12: 000000000000003f
    R13: ffff810037d0c008  R14: 0000000000000246  R15: 0000000000000001
    ORIG_RAX: 0000000000000018  CS: 0020  SS: 0000
bt: WARNING: possibly bogus exception frame
crash>

And when the kernel is patched to push rdi instead, the
"old" behavior is emulated:

crash> bt
PID: 0      TASK: ffffffff8034ce60  CPU: 0   COMMAND: "swapper"
 #0 [ffffffff8047eb40] crash_kexec at ffffffff800ab798
 #1 [ffffffff8047ebc8] mwait_idle at ffffffff80055375
 #2 [ffffffff8047ec00] sysrq_handle_crashdump at ffffffff80192fdc
 #3 [ffffffff8047ec10] __handle_sysrq at ffffffff80192dae
 #4 [ffffffff8047ec50] kbd_event at ffffffff8018db52
 #5 [ffffffff8047eca0] input_event at ffffffff801e9b6d
 #6 [ffffffff8047ecd0] hidinput_hid_event at ffffffff801e4299
 #7 [ffffffff8047ecd8] ip_route_input at ffffffff8003662f
 #8 [ffffffff8047ed00] hid_process_event at ffffffff801df639
 #9 [ffffffff8047ed40] hid_input_report at ffffffff801df9a7
#10 [ffffffff8047edc0] hid_irq_in at ffffffff801e0d8e
#11 [ffffffff8047ede0] usb_hcd_giveback_urb at ffffffff801d33a2
#12 [ffffffff8047ee10] uhci_giveback_urb at ffffffff88126724
#13 [ffffffff8047ee50] uhci_scan_schedule at ffffffff88126e07
#14 [ffffffff8047eed0] uhci_irq at ffffffff88128c08
#15 [ffffffff8047ef10] usb_hcd_irq at ffffffff801d3d91
#16 [ffffffff8047ef20] handle_IRQ_event at ffffffff800106fd
#17 [ffffffff8047ef50] __do_IRQ at ffffffff800b520c
#18 [ffffffff8047ef58] __do_softirq at ffffffff80011bfa
#19 [ffffffff8047ef90] do_IRQ at ffffffff8006a729
--- <IRQ stack> ---
#20 [ffffffff80437ee8] ret_from_intr at ffffffff8005ba89
    [exception RIP: mwait_idle+54]
    RIP: ffffffff80055375  RSP: ffffffff80437f90  RFLAGS: 00000246
    RAX: 0000000000000000  RBX: 0000000000099000  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: 0000000000000001  RDI: ffffffff8034e818
    RBP: 0000000000099000   R8: ffffffff80436000   R9: 000000000000003e
    R10: ffff810037d0c038  R11: ffff81003f48e580  R12: ffff810037fef7a0
    R13: 0000000000000000  R14: ffffffff8034d050  R15: 0000000002246128
    ORIG_RAX: ffffffffffffff16  CS: 0010  SS: 0018
#21 [ffffffff80437f90] cpu_idle at ffffffff800473a7
crash>

Anyway, we'll have to come up with a differentiator so that
both types of interrupt-stack-linkages are handled.  It looks
like the rbp value is fixed with relationship to the exception
frame, so something can be done.

Just FYI,
  Dave