Re: [Crash-utility] arm64: odd backtrace?

----- Original Message ----- Support for IRQ stacks was only recently put in place in crash-7.1.5, and obviously backtraces for a crash-while-in-user-space task is not working correctly. Unfortunately the only test kdump I have on hand only has IRQ stack transitions from kernel space. I tried to create a kdump from a system running user-space commands on our 4.5.0-based kernel, but as luck would have it, kdump fails to work. (it never even reaches the secondary kernel for some reason, even though the kdump facility says it's functional) Obviously there's a problem in arm64_unwind_frame() trying to make the transition, and it returns FALSE because of the NULL fp and therefore INSTACK(frame->fp, bt)) fails. The function is trying to emulate the kernel's unwind_frame() function, which also would return -EINVAL because of the fp. But I'm not sure whether that fp value has been set correctly because of the first, seemingly bogus, exception frame that it's showing. As you have seen, kernel space exceptions look like this, where the fp, sp and pc values are legitimate, so it prints "-- <IRQ stack> --", and transitions to the exception frame on the process stack: crash> set debug 1 debug: 1 crash> bt PID: 0 TASK: fffffe035b0aae00 CPU: 3 COMMAND: "swapper/3" fffffe03fe183d58: fffffe0000137ee4 (crash_save_cpu on IRQ stack) #0 [fffffe03fe183d60] crash_save_cpu at fffffe0000137ee4 #1 [fffffe03fe183dc0] handle_IPI at fffffe000008e8d4 #2 [fffffe03fe183f80] gic_handle_irq at fffffe00000824c8 #3 [fffffe03fe183fd0] el1_irq at fffffe0000083520 bt: arm64_unwind_frame: switch stacks: fp: fffffe035b0f3f30 sp: fffffe035b0f3e10 pc: fffffe000008611c --- <IRQ stack> --- pt_regs: fffffe035b0f3e10 PC: fffffe000008611c [arch_cpu_idle+60] LR: fffffe0000086118 [arch_cpu_idle+56] SP: fffffe035b0f3f30 PSTATE: 60000145 X29: fffffe035b0f3f30 X28: 0000000000000000 X27: fffffe0000084170 X26: fffffe0000bf13dc X25: fffffe0000cf4000 X24: fffffe035b0f0000 X23: 0000000000000001 X22: fffffe0000b94c48 X21: 0000000000000003 X20: fffffe0000cf6000 X19: fffffe0000cf6028 X18: 000002aabb090050 X17: 000003ff9131a228 X16: fffffe000026dba4 X15: 00000000000000bf X14: 004894597490a924 X13: 0000000000000000 X12: 0000000000000010 X11: 0000000000000067 X10: 0000000000000ab0 X9: fffffe035b0f0000 X8: fffffe035b0ab910 X7: 0000000000007b17 X6: 000000000001c690 X5: 0000001515d0302c X4: 0100000000000000 X3: fffffe03fe184c8c X2: fffffe03fe184c80 X1: 0000000000000000 X0: fffffe035b0f0000 ORIG_X0: fffffe035b0f0000 SYSCALLNO: fffffe0000b94c48 #4 [fffffe035b0f3e10] arch_cpu_idle at fffffe000008611c #5 [fffffe035b0f3f40] default_idle_call at fffffe00000f81cc #6 [fffffe035b0f3f70] cpu_startup_entry at fffffe00000f8320 #7 [fffffe035b0f3f80] secondary_start_kernel at fffffe000008e338 crash> In your sample, it certainly doesn't appear that the first exception frame found on the IRQ stack is legitimate, and probably should not pass the test in arm64_is_kernel_exception_frame(), but it does: Maybe that is the cause of the bogus "fp"? Anyway, since the orig_sp is from a fixed location at the top of the IRQ stack, It then manages to make its way back to the "dhry" process stack, where this exception frame "looks" legitimate: But I'm not sure what happens when an arm64 IRQ exception occurs when the task is running in user space. Does it lay an exception frame down on the process stack and then make the transition? (and therefore the user-space frame above is legitimate?) Or does the user-space frame get laid down directly on the IRQ stack? Unfortunately I don't know enough about arm64 exception handling. In any case, the bt should display "-- <IRQ stack> ...", and them dump the user-to-kernel-space exception frame, wherever it lies, i.e., either on the normal process stack or (maybe?) on the IRQ stack. Anyway, can you make the vmlinux/vmcore pair available for me to download? You can send the details to me offline. Thanks, Dave