Re: [Crash-utility] [PATCH] ppc64: fix 'bt' command for vmcore captured with fadump.

----- Original Message ----- > > On Monday 23 January 2017 11:43 PM, Dave Anderson wrote: >> ----- Original Message ----- >>> On Saturday 21 January 2017 02:00 AM, Dave Anderson wrote: >>>> ----- Original Message ----- >>>> >>>> ... [cut] ... >>>> >>>>>> Also, the exception frame doesn't even show the [bracketed] type of >>>>>> exception >>>>>> that occurred -- it's just a register dump followed by the remainder of >>>>>> the >>>>>> backtrace. Upon a quick glance, it's not obvious that they are even >>>>>> active >>>>>> tasks. And traditionally, all of the other architectures have always >>>>>> dumped >>>>>> a full trace. >>>>>> >>>>>> I'm not sure what the mechanism is for shutting down the non-active >>>>>> FADUMP tasks, so that's why I asked if you could restrict this change >>>>>> to just those types of dumps. (For that matter, is it even possible to >>>>>> differentiate a real kdump from an FADUMP dumpfile -- aside from a >>>>> Hi Dave, >>>>> >>>>> Differentiating a kdump and fadump dumpfile is not possible except that >>>>> the >>>>> stack search would invariably fail and ptregs are guaranteed to be saved >>>>> by >>>>> firmware in case of fadump. Posted v2 that doesn't change bt output for >>>>> anything >>>>> but active tasks in case of fadump.. >>>> Ok, so let me get this straight. The only difference I see with the v2 >>>> patch >>>> is that fadump non-panicking active tasks change from something like >>>> this: >>>> >>>> PID: 0 TASK: c000000000e7f6d0 CPU: 0 COMMAND: "swapper" >>>> #0 [c000000000f2ba30] (null) at 3aae291c67 (unreliable) >>>> #1 [c000000000f2bae0] .tick_dev_program_event at c0000000000d16fc >>>> #2 [c000000000f2bb90] .__hrtimer_start_range_ns at c0000000000c4bcc >>>> #3 [c000000000f2bcb0] .tick_nohz_stop_sched_tick at c0000000000d2d30 >>>> #4 [c000000000f2bdc0] .cpu_idle at c000000000015bf0 >>>> #5 [c000000000f2be70] .rest_init at c000000000009de4 >>>> #6 [c000000000f2bef0] .start_kernel at c000000000850eb4 >>>> #7 [c000000000f2bf90] .start_here_common at c0000000000083d8 >>>> >>>> to this: >>>> >>>> PID: 0 TASK: c000000000e7f6d0 CPU: 0 COMMAND: "swapper" >>>> #0 [c000000000f2bd50] (null) at 0 (unreliable) >>>> #1 [c000000000f2bdc0] .cpu_idle at c000000000015bf0 >>>> #2 [c000000000f2be70] .rest_init at c000000000009de4 >>>> #3 [c000000000f2bef0] .start_kernel at c000000000850eb4 >>>> #4 [c000000000f2bf90] .start_here_common at c0000000000083d8 >>>> >>>> But with your v1 patch, you also dumped the exception frame: >>>> >>>> PID: 0 TASK: c000000000e7f6d0 CPU: 0 COMMAND: "swapper" >>>> R0: 0000000000000000 R1: c000000000f2bd50 R2: >>>> c000000000f27628 >>>> R3: 0000000000000000 R4: 0000000000000000 R5: >>>> 8000000002144400 >>>> R6: 800000001314c4f8 R7: 0000000000000000 R8: >>>> 0000000000000000 >>>> R9: ffffffffffffffff R10: 0000000000000000 R11: >>>> 80003fbff901700c >>>> R12: 0000000000000000 R13: c000000000ff2500 R14: >>>> 0000000001a3fa58 >>>> R15: 00000000002230a8 R16: 0000000000223150 R17: >>>> 0000000000223144 >>>> R18: 0000000000c8a098 R19: 0000000002b13a58 R20: >>>> 0000000000000000 >>>> R21: 0000000002b135d8 R22: 0000000002b13530 R23: >>>> 0000000002280000 >>>> R24: 0000000002b135f0 R25: c000000000fd5c48 R26: >>>> c0000000010942f0 >>>> R27: c0000000010942f0 R28: c0000000005fd168 R29: >>>> 0000000000000008 >>>> R30: c000000000eb1d68 R31: c000000000f28080 >>>> NIP: c000000000055730 MSR: 8000000000009032 OR3: >>>> 0000000000000000 >>>> CTR: 0000000000000000 LR: c000000000057350 XER: >>>> 0000000000000000 >>>> CCR: 0000000024000048 MQ: 0000000000000000 DAR: >>>> 000001000ad763b0 >>>> DSISR: 0000000000000000 Syscall Result: 0000000000000000 >>>> NIP [c000000000055730] .plpar_hcall_norets >>>> LR [c000000000057350] .pseries_shared_idle_sleep >>>> #0 [c000000000f2bd50] (null) at 0 (unreliable) >>>> #1 [c000000000f2bdc0] .cpu_idle at c000000000015bf0 >>>> #2 [c000000000f2be70] .rest_init at c000000000009de4 >>>> #3 [c000000000f2bef0] .start_kernel at c000000000850eb4 >>>> #4 [c000000000f2bf90] .start_here_common at c0000000000083d8 >>>> >>>> Again, I don't understand how the non-panicking active tasks are stopped >>>> by the fadump facility, but is it because you cannot differentiate kdumps >>>> from fadumps that you don't show the exception frame with the v2 patch? >>> Hi Dave, >>> >>> The crashing cpu makes rtas call ibm,os-term to the firmware which >>> saves the regs info of all online cpus. AFAIK, there is no exception frame >>> marker (which we are using to detect one) set for this stack frames >>> by the kernel. With v1, I was printing the registers without looking for >>> exception frame marker, if the registers are saved.. >>> >>>> Would it be possible to also show the exception frame type in brackets >>>> and >>>> the register dump for those fadump non-panicking active tasks? >>>> >>> Hmmm.. Let me have a hard look at this. >>> Will try and improve this.. >> Hari, >> >> I was tinkering around with ppc64_get_dumpfile_stack_frame() from your v2 >> patch, >> and this seems to work: >> >> else { >> *ksp = pt_regs->gpr[1]; >> if (IS_KVADDR(*ksp)) { >> readmem(*ksp+16, KVADDR, nip, sizeof(ulong), >> "Regs NIP value", FAULT_ON_ERROR); >> + ppc64_print_regs(pt_regs); >> return TRUE; >> } else { >> if (IN_TASK_VMA(bt_in->task, *ksp)) >> fprintf(fp, "%0lx: Task is running in user >> space\n", >> bt_in->task); >> else >> fprintf(fp, "%0lx: Invalid Stack Pointer >> %0lx\n", >> bt_in->task, *ksp); >> *nip = pt_regs->nip; >> ppc64_print_regs(pt_regs); >> return FALSE; >> } >> } >> >> And if the task were to have been running in userspace, it already dumps >> the >> registers in the "else" section above. >> >> I see that the regs->trap is 0, so I understand now that there's nothing to >> translate w/respect to the exception frame type, but a follow-up >> translation >> of the NIP and LR would at least show that there was some kind of hypercall >> involved. (Whether it can be firmly determined whether FADUMP was >> responsible >> is another question) >> >> > Hi Dave, > > I did think of it but I was wary considering two register prints like below, > if there is an exception frame.. > > PID: 2121 TASK: c0000001af90c600 CPU: 2 COMMAND: "sshd" > R0: c0000000003e5280 R1: c0000001ae047a30 R2: > c000000000fd5a00 > R3: 0000000000000001 R4: 000000000000019e R5: > 000000000000000f > R6: 0000000000000004 R7: c0000001ae047bb8 R8: > 00000000000b3d9f > R9: 00000000000000f0 R10: 0000000000000678 R11: > c0000000008e0f38 > R12: c0000000003e6310 R13: c00000000b781200 R14: > 0000000000000000 > R15: 0000000000000000 R16: 000001000b7dad70 R17: > 000000005dfd3c08 > R18: 000000005dfd2838 R19: 00003ffff81eb620 R20: > 000000005df74128 > R21: 000001000b7d89a0 R22: 000000000000de4c R23: > 000000005df73b30 > R24: 000000005dfd3c88 R25: 00003ffff81eb428 R26: > c0000001ae047bb8 > R27: c0000001b17f4d80 R28: c000000000c60580 R29: > 000000000000019e > R30: 000000000000000f R31: 000000000000090b > NIP: 00003fffb6ac8400 MSR: 800000000000d033 OR3: > 0000000000000000 > CTR: c0000000003e6310 LR: c0000000003e493c XER: > 0000000020000000 > CCR: 0000000024004824 MQ: 0000000000000000 DAR: > 000001000b7e1640 > DSISR: 0000000002000000 Syscall Result: 0000000000000000 > #0 [c0000001ae047a30] (null) at c0000000fd783c00 (unreliable) > #1 [c0000001ae047a70] avc_has_perm at c0000000003e5280 > #2 [c0000001ae047b60] sock_has_perm at c0000000003e6238 > #3 [c0000001ae047be0] security_socket_sendmsg at c0000000003e28fc > #4 [c0000001ae047c30] sock_sendmsg at c00000000072d53c > #5 [c0000001ae047c60] sock_write_iter at c00000000072d644 > #6 [c0000001ae047d00] __vfs_write at c0000000002ed97c > #7 [c0000001ae047d90] vfs_write at c0000000002ef328 > #8 [c0000001ae047de0] sys_write at c0000000002f0f00 > #9 [c0000001ae047e30] system_call at c00000000000b184 > System Call [c00] exception frame: > R0: 0000000000000004 R1: 00003ffff81eb220 R2: > 00003fffb6b99800 > R3: 0000000000000003 R4: 000001000b80e3c0 R5: > 0000000000000034 > R6: 00003ffff81eb2e4 R7: 000000000000021e R8: > 0000000000000000 > R9: 0000000000000000 R10: 0000000000000000 R11: > 0000000000000000 > R12: 0000000000000000 R13: 00003fffb6497730 R14: > 0000000000000000 > R15: 0000000000000000 R16: 000001000b7dad70 R17: > 000000005dfd3c08 > R18: 000000005dfd2838 R19: 00003ffff81eb620 R20: > 000000005df74128 > R21: 000001000b7d89a0 R22: 000000000000de4c R23: > 000000005df73b30 > R24: 000000005dfd3c88 R25: 00003ffff81eb428 R26: > 00003ffff81eb430 > R27: 00003ffff81eb420 R28: 00003ffff81eb424 R29: > 00003ffff81eb2e4 > R30: 000001000b80e3c0 R31: 0000000000000034 > NIP: 00003fffb6ac8400 MSR: 800000000000d033 OR3: > 0000000000000003 > CTR: 0000000000000000 LR: 000000005df1c3e4 XER: > 0000000000000000 > CCR: 0000000044004824 MQ: 0000000000000001 DAR: > 00003fffb729c590 > DSISR: 000000000a000000 Syscall Result: 0000000000000000 > > > instead of this.. > > PID: 2121 TASK: c0000001af90c600 CPU: 2 COMMAND: "sshd" > #0 [c0000001ae047a30] (null) at c0000000fd783c00 (unreliable) > #1 [c0000001ae047a70] avc_has_perm at c0000000003e5280 > #2 [c0000001ae047b60] sock_has_perm at c0000000003e6238 > #3 [c0000001ae047be0] security_socket_sendmsg at c0000000003e28fc > #4 [c0000001ae047c30] sock_sendmsg at c00000000072d53c > #5 [c0000001ae047c60] sock_write_iter at c00000000072d644 > #6 [c0000001ae047d00] __vfs_write at c0000000002ed97c > #7 [c0000001ae047d90] vfs_write at c0000000002ef328 > #8 [c0000001ae047de0] sys_write at c0000000002f0f00 > #9 [c0000001ae047e30] system_call at c00000000000b184 > System Call [c00] exception frame: > R0: 0000000000000004 R1: 00003ffff81eb220 R2: > 00003fffb6b99800 > R3: 0000000000000003 R4: 000001000b80e3c0 R5: > 0000000000000034 > R6: 00003ffff81eb2e4 R7: 000000000000021e R8: > 0000000000000000 > R9: 0000000000000000 R10: 0000000000000000 R11: > 0000000000000000 > R12: 0000000000000000 R13: 00003fffb6497730 R14: > 0000000000000000 > R15: 0000000000000000 R16: 000001000b7dad70 R17: > 000000005dfd3c08 > R18: 000000005dfd2838 R19: 00003ffff81eb620 R20: > 000000005df74128 > R21: 000001000b7d89a0 R22: 000000000000de4c R23: > 000000005df73b30 > R24: 000000005dfd3c88 R25: 00003ffff81eb428 R26: > 00003ffff81eb430 > R27: 00003ffff81eb420 R28: 00003ffff81eb424 R29: > 00003ffff81eb2e4 > R30: 000001000b80e3c0 R31: 0000000000000034 > NIP: 00003fffb6ac8400 MSR: 800000000000d033 OR3: > 0000000000000003 > CTR: 0000000000000000 LR: 000000005df1c3e4 XER: > 0000000000000000 > CCR: 0000000044004824 MQ: 0000000000000001 DAR: > 00003fffb729c590 > DSISR: 000000000a000000 Syscall Result: 0000000000000000 > > > On second thought, that may not be bad after all?? > So, I am ok with the change you propose. Hmmm, except that in the "sshd" sample showing the firmware-generated eframe, and which the task was presumably running in kernel space when firmware took over (?), it has a userspace NIP of 00003fffb6ac8400. What's happening there?