Re: [Crash-utility] [PATCH v5 0/4] arm64: more improvement of bt -f

Hi Takahiro, I applied patches 1/2 and 2/2 from the v5 patchset. But I can't believe the results are what you intended?

On arm64, the link register (LR) holds a return address, which is the one just after a branch instruction. So using a saved lr as PC for backtracing might cause some confusion. For example, in kernel/entry.S, work_resched: ... bl schedule ret_to_user: ... The current code shows "ret_o_user", instead of "work_resched", as a caller of schedule(). This patch corrects a PC by decrementing it by 4. But please note that this change may also make people a bit confused because a value of LR in the stack dump of "bt -f" doesn't match with an address in one-line summary. #2 [ffffcc7511407eb0] schedule at ffff0000d628aee0 ffffcc7511407eb0: ffffcc6d22f23080 ffff0000d5b44d6c <= LR ffffcc7511407ec0: ffffcc7511407ed0 0000000000000000 #3 [ffffcc7511407ed0] work_resched at ffff0000d5b44d68 <= correcrted PC Signed-off-by: AKASHI Takahiro <takahiro akashi linaro org>

From a kernel debugging perspective, all we really need is a basic unwinder,

For example, taking the 4.6 vmcore that you gave to me, here is the current crash utility's output of "bt -a", where the crashing task entered crash_kexec() via the sysrq-c page fault exception, and the tasks on the other cpus have all entered crash_save_cpu() on their IRQ stack as a result of the shutdown IPI, one from user-space and the others from the kernel: crash> bt -a PID: 0 TASK: ffff000008dcd900 CPU: 0 COMMAND: "swapper/0" #0 [ffff800022f42e50] crash_save_cpu at ffff00000812ae44 #1 [ffff800022f43010] handle_IPI at ffff00000808e718 #2 [ffff800022f43040] gic_handle_irq at ffff0000080815f8 #3 [ffff800022f43080] el1_irq at ffff000008084720 --- <IRQ stack> --- PC: ffff0000080857c0 [arch_cpu_idle+16] LR: ffff0000080857bc [arch_cpu_idle+12] SP: ffff000008dc3f10 PSTATE: 60400149 X29: ffff000008dc3f10 X28: ffff000008dc0000 X27: 0000000000000000 X26: ffff000008dc5000 X25: ffff000008dc5c1c X24: ffff000008dc5000 X23: ffff000008dc0000 X22: ffff000008bd0270 X21: ffff000008dc0000 X20: ffff000008dc5b88 X19: 0000000000000000 X18: 00000000a632f641 X17: 0000ffff7da57880 X16: ffff0000081d9838 X15: 00000000383a0a79 X14: 00000000b2b0b162 X13: 000000005f2cbeec X12: 0000000000045a9e X11: ffff8000213bd800 X10: 0000000000000850 X9: ffff000008dc0000 X8: 000000010000aa07 X7: 000000000000003d X6: 0015752a00000000 X5: 0100000000000000 X4: 00000000000001c0 X3: 0000000000000000 X2: 0000000000000001 X1: 0000000000000000 X0: 0000000000000000 ORIG_X0: 0000000000000000 SYSCALLNO: 7fffffffffffffff #4 [ffff000008dc3f10] arch_cpu_idle at ffff0000080857c0 #5 [ffff000008dc3f20] cpu_startup_entry at ffff0000080f26cc #6 [ffff000008dc3f80] rest_init at ffff0000087c7930 #7 [ffff000008dc3fa0] start_kernel at ffff000008b10b70 PID: 0 TASK: ffff8000218c0c80 CPU: 1 COMMAND: "swapper/1" #0 [ffff800022f56e50] crash_save_cpu at ffff00000812ae44 #1 [ffff800022f57010] handle_IPI at ffff00000808e718 #2 [ffff800022f57040] gic_handle_irq at ffff0000080815f8 #3 [ffff800022f57080] el1_irq at ffff000008084720 --- <IRQ stack> --- PC: ffff0000080857c0 [arch_cpu_idle+16] LR: ffff0000080857bc [arch_cpu_idle+12] SP: ffff8000218cff60 PSTATE: 60000349 X29: ffff8000218cff60 X28: ffff8000218cc000 X27: 0000000000000000 X26: ffff000008dc5000 X25: ffff000008dc5c1c X24: ffff000008dc5000 X23: ffff8000218cc000 X22: ffff000008bd0270 X21: ffff8000218cc000 X20: ffff000008dc5b88 X19: 0000000000000000 X18: 000000000000016d X17: 0000007f8b122780 X16: ffffffc0001adf68 X15: 0000000000000005 X14: 0000000c80096000 X13: ffff8000212b6600 X12: 0000000000047ae2 X11: ffff800021089980 X10: 0000000000000850 X9: ffff8000218cc000 X8: 000000010000aa07 X7: 0000000000000243 X6: 0015752a00000000 X5: 0100000000000000 X4: 00000000000003c0 X3: 0000000000000000 X2: 0000000000000001 X1: 0000000000000000 X0: 0000000000000000 ORIG_X0: 0000000000000000 SYSCALLNO: 7fffffffffffffff #4 [ffff8000218cff60] arch_cpu_idle at ffff0000080857c0 #5 [ffff8000218cff70] cpu_startup_entry at ffff0000080f26cc #6 [ffff8000218cffd0] secondary_start_kernel at ffff00000808e1ec PID: 1324 TASK: ffff80002018be80 CPU: 2 COMMAND: "dhry" #0 [ffff800022f6ae50] crash_save_cpu at ffff00000812ae44 #1 [ffff800022f6b010] handle_IPI at ffff00000808e718 #2 [ffff800022f6b040] gic_handle_irq at ffff0000080815f8 #3 [ffff800022f6b080] el0_irq_naked at ffff000008084c4c --- <IRQ stack> --- PC: 00000000004016a4 LR: 00000000004016a4 SP: 0000ffffc10c40a0 X29: 0000ffffc10c40a0 X28: 0000000000000000 X27: 0000000000000000 X26: 0000000000000000 X25: 0000000000402138 X24: 00000000004021f0 X23: 0000000000000000 X22: 0000000000000000 X21: 00000000004001a0 X20: 0000000000000000 X19: 0000000000000000 X18: 0000000000000000 X17: 0000000000000001 X16: 0000000000000000 X15: 0000000000493000 X14: 0000000000498000 X13: ffffffffffffffff X12: 0000000000000005 X11: 000000000000001e X10: 0101010101010101 X9: fffffffff59a9190 X8: 7f7f7f7f7f7f7f7f X7: 1f535226301f2b4c X6: 00000003001d1000 X5: 00101d0003000000 X4: 0000000000000000 X3: 4952545320454d4f X2: 0000000010c35b40 X1: 0000000000000011 X0: 0000000010c35b40 ORIG_X0: 0000000000498700 SYSCALLNO: ffffffffffffffff PSTATE: 20000000 PID: 1223 TASK: ffff800020ef5780 CPU: 3 COMMAND: "sh" #0 [ffff800020b0ba70] crash_kexec at ffff00000812b0ac #1 [ffff800020b0ba90] die at ffff000008088ce8 #2 [ffff800020b0bad0] __do_kernel_fault at ffff000008098fac #3 [ffff800020b0bb00] do_page_fault at ffff000008096814 #4 [ffff800020b0bb60] do_translation_fault at ffff00000809690c #5 [ffff800020b0bb70] do_mem_abort at ffff00000808128c #6 [ffff800020b0bd40] el1_da at ffff000008084568 PC: ffff000008457fc8 [sysrq_handle_crash+32] LR: ffff000008457fb4 [sysrq_handle_crash+12] SP: ffff800020b0bd40 PSTATE: 60400149 X29: ffff800020b0bd40 X28: ffff800020b08000 X27: ffff0000087e2000 X26: 0000000000000040 X25: 0000000000000120 X24: 0000000000000015 X23: 0000000000000000 X22: 0000000000000009 X21: ffff000008e071b0 X20: 0000000000000063 X19: ffff000008dda000 X18: 0000000000000006 X17: 0000ffff83d266c0 X16: ffff0000081c68b8 X15: ffff000008e6cc95 X14: 0000ffff83cc28f0 X13: ffff000008e6c758 X12: ffff000008dda7a0 X11: 0000000000000106 X10: 0000000000000002 X9: 0000000000000001 X8: 0000000000000106 X7: 0000000000000000 X6: ffff800022f337a4 X5: 00000000000047d4 X4: 0000000000000000 X3: 0000000000000000 X2: ffff000008dda7b8 X1: 0000000000000000 X0: 0000000000000001 ORIG_X0: ffff000008dda000 SYSCALLNO: ffff80002104d418 #7 [ffff800020b0bd50] __handle_sysrq at ffff000008458644 #8 [ffff800020b0bd90] write_sysrq_trigger at ffff000008458ac0 #9 [ffff800020b0bdb0] proc_reg_write at ffff00000822541c #10 [ffff800020b0bdd0] __vfs_write at ffff0000081c4774 #11 [ffff800020b0be50] vfs_write at ffff0000081c5588 #12 [ffff800020b0be90] sys_write at ffff0000081c68fc #13 [ffff800020b0bed0] el0_svc_naked at ffff000008084e70 PC: 0000ffff83d7b1e8 LR: 0000ffff83d2d700 SP: 0000fffffe197460 X29: 0000fffffe197460 X28: 0000000000000000 X27: 0000000000000000 X26: 000000001e9e40c8 X25: 00000000004c9be8 X24: 00000000004ec000 X23: 0000000000000001 X22: 0000000000000002 X21: 0000ffff83e02500 X20: 000000001e9da808 X19: 0000000000000002 X18: 0000000000000015 X17: 0000ffff83d266c0 X16: 0000000000000000 X15: 0015a60d0a000000 X14: 0000ffff83cc28f0 X13: 0000000000000018 X12: 0000000000000008 X11: 0101010101010101 X10: 7f7f7f7f7f7f7f7f X9: fefefefeff00ff62 X8: 0000000000000040 X7: fefefefefefefefe X6: 000000001e9da808 X5: 0000ffff83d055c8 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 000000001e9da808 X0: 0000000000000001 ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE: 20000000 PID: 0 TASK: ffff8000218c3200 CPU: 4 COMMAND: "swapper/4" #0 [ffff800022f92e50] crash_save_cpu at ffff00000812ae44 #1 [ffff800022f93010] handle_IPI at ffff00000808e718 #2 [ffff800022f93040] gic_handle_irq at ffff0000080815f8 #3 [ffff800022f93080] el1_irq at ffff000008084720 --- <IRQ stack> --- PC: ffff0000080857c0 [arch_cpu_idle+16] LR: ffff0000080857bc [arch_cpu_idle+12] SP: ffff8000218dbf60 PSTATE: 60000349 X29: ffff8000218dbf60 X28: ffff8000218d8000 X27: 0000000000000000 X26: ffff000008dc5000 X25: ffff000008dc5c1c X24: ffff000008dc5000 X23: ffff8000218d8000 X22: ffff000008bd0270 X21: ffff8000218d8000 X20: ffff000008dc5b88 X19: 0000000000000000 X18: 000000000000016d X17: 0000007f8b122780 X16: ffffffc0001adf68 X15: 0000000000000000 X14: 0000000000000000 X13: 00000000000003e7 X12: 0000000000000016 X11: ffff0000087ef470 X10: 0000000000000850 X9: ffff8000218d8000 X8: ffff8000218c3ab0 X7: 7fffffffffffffff X6: 0015752a00000000 X5: 0100000000000000 X4: 00000000000003c0 X3: 0000000000000000 X2: 0000000000000001 X1: 0000000000000000 X0: 0000000000000000 ORIG_X0: 0000000000000000 SYSCALLNO: 7fffffffffffffff #4 [ffff8000218dbf60] arch_cpu_idle at ffff0000080857c0 #5 [ffff8000218dbf70] cpu_startup_entry at ffff0000080f26cc #6 [ffff8000218dbfd0] secondary_start_kernel at ffff00000808e1ec PID: 0 TASK: ffff8000218c3e80 CPU: 5 COMMAND: "swapper/5" #0 [ffff800022fa6e50] crash_save_cpu at ffff00000812ae44 #1 [ffff800022fa7010] handle_IPI at ffff00000808e718 #2 [ffff800022fa7040] gic_handle_irq at ffff0000080815f8 #3 [ffff800022fa7080] el1_irq at ffff000008084720 --- <IRQ stack> --- PC: ffff0000080857c0 [arch_cpu_idle+16] LR: ffff0000080857bc [arch_cpu_idle+12] SP: ffff8000218dff60 PSTATE: 60000349 X29: ffff8000218dff60 X28: ffff8000218dc000 X27: 0000000000000000 X26: ffff000008dc5000 X25: ffff000008dc5c1c X24: ffff000008dc5000 X23: ffff8000218dc000 X22: ffff000008bd0270 X21: ffff8000218dc000 X20: ffff000008dc5b88 X19: 0000000000000000 X18: 000000000000016d X17: 0000007f8b122780 X16: ffffffc0001adf68 X15: 0000000000000007 X14: 0000000000000000 X13: 000000000000039d X12: 0000000000000000 X11: ffff0000087ef470 X10: 0000000000000850 X9: ffff8000218dc000 X8: 000000010000a479 X7: 7fffffffffffffff X6: 0016694e00000000 X5: 0100000000000000 X4: 00000000000003c0 X3: 0000000000000000 X2: 0000000000000001 X1: 0000000000000000 X0: 0000000000000000 ORIG_X0: 0000000000000000 SYSCALLNO: 7fffffffffffffff #4 [ffff8000218dff60] arch_cpu_idle at ffff0000080857c0 #5 [ffff8000218dff70] cpu_startup_entry at ffff0000080f26cc #6 [ffff8000218dffd0] secondary_start_kernel at ffff00000808e1ec PID: 0 TASK: ffff8000218c4b00 CPU: 6 COMMAND: "swapper/6" #0 [ffff800022fbae50] crash_save_cpu at ffff00000812ae44 #1 [ffff800022fbb010] handle_IPI at ffff00000808e718 #2 [ffff800022fbb040] gic_handle_irq at ffff0000080815f8 #3 [ffff800022fbb080] el1_irq at ffff000008084720 --- <IRQ stack> --- PC: ffff0000080857c0 [arch_cpu_idle+16] LR: ffff0000080857bc [arch_cpu_idle+12] SP: ffff8000218ebf60 PSTATE: 60000349 X29: ffff8000218ebf60 X28: ffff8000218e8000 X27: 0000000000000000 X26: ffff000008dc5000 X25: ffff000008dc5c1c X24: ffff000008dc5000 X23: ffff8000218e8000 X22: ffff000008bd0270 X21: ffff8000218e8000 X20: ffff000008dc5b88 X19: 0000000000000000 X18: 000000000000016d X17: 0000007f8b122780 X16: ffffffc0001adf68 X15: 000000000000000a X14: 0000000000000000 X13: 00000000000003a1 X12: 0000000000000000 X11: ffff8000213a0d00 X10: 0000000000000850 X9: ffff8000218e8000 X8: ffff8000218c53b0 X7: 00000000000001c4 X6: 002887fa00000000 X5: 0100000000000000 X4: 00000000000003c0 X3: 0000000000000000 X2: 0000000000000001 X1: 0000000000000000 X0: 0000000000000000 ORIG_X0: 0000000000000000 SYSCALLNO: 7fffffffffffffff #4 [ffff8000218ebf60] arch_cpu_idle at ffff0000080857c0 #5 [ffff8000218ebf70] cpu_startup_entry at ffff0000080f26cc #6 [ffff8000218ebfd0] secondary_start_kernel at ffff00000808e1ec PID: 0 TASK: ffff8000218c5780 CPU: 7 COMMAND: "swapper/7" #0 [ffff800022fcee50] crash_save_cpu at ffff00000812ae44 #1 [ffff800022fcf010] handle_IPI at ffff00000808e718 #2 [ffff800022fcf040] gic_handle_irq at ffff0000080815f8 #3 [ffff800022fcf080] el1_irq at ffff000008084720 --- <IRQ stack> --- PC: ffff0000080857c0 [arch_cpu_idle+16] LR: ffff0000080857bc [arch_cpu_idle+12] SP: ffff8000218eff60 PSTATE: 60400149 X29: ffff8000218eff60 X28: ffff8000218ec000 X27: 0000000000000000 X26: ffff000008dc5000 X25: ffff000008dc5c1c X24: ffff000008dc5000 X23: ffff8000218ec000 X22: ffff000008bd0270 X21: ffff8000218ec000 X20: ffff000008dc5b88 X19: 0000000000000000 X18: fffffffffffffffb X17: 0000ffff9dc8ba48 X16: ffff0000081da048 X15: 003b9aca00000000 X14: 0000000000000000 X13: 0000000000000001 X12: 0000000000000000 X11: ffff800021089980 X10: 0000000000000850 X9: ffff8000218ec000 X8: 000000010000a280 X7: ffff8000209b39c8 X6: ffff8000209b39c9 X5: 0100000000000000 X4: 00000000000001c0 X3: 0000000000000000 X2: 0000000000000001 X1: 0000000000000000 X0: 0000000000000000 ORIG_X0: 0000000000000000 SYSCALLNO: 7fffffffffffffff #4 [ffff8000218eff60] arch_cpu_idle at ffff0000080857c0 #5 [ffff8000218eff70] cpu_startup_entry at ffff0000080f26cc #6 [ffff8000218effd0] secondary_start_kernel at ffff00000808e1ec crash> And that's precisely what we want to see. (other than perhaps the in-kernel exception frame's dump of the ORIG_X0 and SYSCALL registers) Now, with v5's patches 1/2 and 2/2 patches applied, this is what gets displayed: crash> bt -a PID: 0 TASK: ffff000008dcd900 CPU: 0 COMMAND: "swapper/0" #0 [ffff000008dc3f10] arch_cpu_idle at ffff0000080857c0 #1 [ffff000008dc3f20] cpu_startup_entry at ffff0000080f26cc #2 [ffff000008dc3f80] rest_init at ffff0000087c7930 #3 [ffff000008dc3fa0] start_kernel at ffff000008b10b70 PID: 0 TASK: ffff8000218c0c80 CPU: 1 COMMAND: "swapper/1" #0 [ffff8000218cff60] arch_cpu_idle at ffff0000080857c0 #1 [ffff8000218cff70] cpu_startup_entry at ffff0000080f26cc #2 [ffff8000218cffd0] secondary_start_kernel at ffff00000808e1ec PID: 1324 TASK: ffff80002018be80 CPU: 2 COMMAND: "dhry" PC: 00000000004016a4 LR: 00000000004016a4 SP: 0000ffffc10c40a0 X29: 0000ffffc10c40a0 X28: 0000000000000000 X27: 0000000000000000 X26: 0000000000000000 X25: 0000000000402138 X24: 00000000004021f0 X23: 0000000000000000 X22: 0000000000000000 X21: 00000000004001a0 X20: 0000000000000000 X19: 0000000000000000 X18: 0000000000000000 X17: 0000000000000001 X16: 0000000000000000 X15: 0000000000493000 X14: 0000000000498000 X13: ffffffffffffffff X12: 0000000000000005 X11: 000000000000001e X10: 0101010101010101 X9: fffffffff59a9190 X8: 7f7f7f7f7f7f7f7f X7: 1f535226301f2b4c X6: 00000003001d1000 X5: 00101d0003000000 X4: 0000000000000000 X3: 4952545320454d4f X2: 0000000010c35b40 X1: 0000000000000011 X0: 0000000010c35b40 ORIG_X0: 0000000000498700 SYSCALLNO: ffffffffffffffff PSTATE: 20000000 #0 [user space] PID: 1223 TASK: ffff800020ef5780 CPU: 3 COMMAND: "sh" #0 [ffff800020b0ba70] crash_kexec at ffff00000812b0ac #1 [ffff800020b0ba90] die at ffff000008088ce8 #2 [ffff800020b0bad0] __do_kernel_fault at ffff000008098fac #3 [ffff800020b0bb00] do_page_fault at ffff000008096814 #4 [ffff800020b0bb60] do_translation_fault at ffff00000809690c #5 [ffff800020b0bb70] do_mem_abort at ffff00000808128c #6 [ffff800020b0bc20] el1_da at ffff000008084568 --- <Exception in kernel> --- PC: ffff000008457fc8 [sysrq_handle_crash+32] LR: ffff000008457fb4 [sysrq_handle_crash+12] SP: ffff800020b0bd40 PSTATE: 60400149 X29: ffff800020b0bd40 X28: ffff800020b08000 X27: ffff0000087e2000 X26: 0000000000000040 X25: 0000000000000120 X24: 0000000000000015 X23: 0000000000000000 X22: 0000000000000009 X21: ffff000008e071b0 X20: 0000000000000063 X19: ffff000008dda000 X18: 0000000000000006 X17: 0000ffff83d266c0 X16: ffff0000081c68b8 X15: ffff000008e6cc95 X14: 0000ffff83cc28f0 X13: ffff000008e6c758 X12: ffff000008dda7a0 X11: 0000000000000106 X10: 0000000000000002 X9: 0000000000000001 X8: 0000000000000106 X7: 0000000000000000 X6: ffff800022f337a4 X5: 00000000000047d4 X4: 0000000000000000 X3: 0000000000000000 X2: ffff000008dda7b8 X1: 0000000000000000 X0: 0000000000000001 #7 [ffff800020b0bd40] sysrq_handle_crash at ffff000008457fc8 #8 [ffff800020b0bd50] __handle_sysrq at ffff000008458644 #9 [ffff800020b0bd90] write_sysrq_trigger at ffff000008458ac0 #10 [ffff800020b0bdb0] proc_reg_write at ffff00000822541c #11 [ffff800020b0bdd0] __vfs_write at ffff0000081c4774 #12 [ffff800020b0be50] vfs_write at ffff0000081c5588 #13 [ffff800020b0be90] sys_write at ffff0000081c68fc #14 [ffff800020b0bed0] el0_svc_naked at ffff000008084e70 --- <Exception in user> --- PC: 0000ffff83d7b1e8 LR: 0000ffff83d2d700 SP: 0000fffffe197460 X29: 0000fffffe197460 X28: 0000000000000000 X27: 0000000000000000 X26: 000000001e9e40c8 X25: 00000000004c9be8 X24: 00000000004ec000 X23: 0000000000000001 X22: 0000000000000002 X21: 0000ffff83e02500 X20: 000000001e9da808 X19: 0000000000000002 X18: 0000000000000015 X17: 0000ffff83d266c0 X16: 0000000000000000 X15: 0015a60d0a000000 X14: 0000ffff83cc28f0 X13: 0000000000000018 X12: 0000000000000008 X11: 0101010101010101 X10: 7f7f7f7f7f7f7f7f X9: fefefefeff00ff62 X8: 0000000000000040 X7: fefefefefefefefe X6: 000000001e9da808 X5: 0000ffff83d055c8 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 000000001e9da808 X0: 0000000000000001 ORIG_X0: 0000000000000001 SYSCALLNO: 40 PSTATE: 20000000 PID: 0 TASK: ffff8000218c3200 CPU: 4 COMMAND: "swapper/4" #0 [ffff8000218dbf60] arch_cpu_idle at ffff0000080857c0 #1 [ffff8000218dbf70] cpu_startup_entry at ffff0000080f26cc #2 [ffff8000218dbfd0] secondary_start_kernel at ffff00000808e1ec PID: 0 TASK: ffff8000218c3e80 CPU: 5 COMMAND: "swapper/5" #0 [ffff8000218dff60] arch_cpu_idle at ffff0000080857c0 #1 [ffff8000218dff70] cpu_startup_entry at ffff0000080f26cc #2 [ffff8000218dffd0] secondary_start_kernel at ffff00000808e1ec PID: 0 TASK: ffff8000218c4b00 CPU: 6 COMMAND: "swapper/6" #0 [ffff8000218ebf60] arch_cpu_idle at ffff0000080857c0 #1 [ffff8000218ebf70] cpu_startup_entry at ffff0000080f26cc #2 [ffff8000218ebfd0] secondary_start_kernel at ffff00000808e1ec PID: 0 TASK: ffff8000218c5780 CPU: 7 COMMAND: "swapper/7" #0 [ffff8000218eff60] arch_cpu_idle at ffff0000080857c0 #1 [ffff8000218eff70] cpu_startup_entry at ffff0000080f26cc #2 [ffff8000218effd0] secondary_start_kernel at ffff00000808e1ec crash> What happened? This can't possibly be what you intended, is it? Dave -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility