2:33 p.m.
Han,
This is much worse than I thought. When you said "sometimes", you must
mean "all of the time" with respect to the active tasks? Because that's
what I see here.
I provisioned a ppc64le machine, set up kdump to create a compressed kdump,
and crashed the machine with sysrq-c. This is what I see with "bt -a":
crash> bt -a
PID: 12674 TASK: c00000002cc08810 CPU: 0 COMMAND: "bash"
PID: 0 TASK: c0000001ee020000 CPU: 1 COMMAND: "swapper/1"
PID: 0 TASK: c0000001ee021370 CPU: 2 COMMAND: "swapper/2"
PID: 0 TASK: c0000001ee0226e0 CPU: 3 COMMAND: "swapper/3"
PID: 0 TASK: c0000001ee023a50 CPU: 4 COMMAND: "swapper/4"
PID: 0 TASK: c0000001ee024dc0 CPU: 5 COMMAND: "swapper/5"
PID: 0 TASK: c0000001ee026130 CPU: 6 COMMAND: "swapper/6"
PID: 0 TASK: c0000001ee0274a0 CPU: 7 COMMAND: "swapper/7"
crash>
As far as I can tell, it does not even use the registers in the
notes, but rather defaults to searching the relevant stacks for
".crash_kexec". But on ppc64le, that symbol no longer exists as
it does on a big-endian ppc64:
crash> sym -q crash_kexec
c000000000051760 (t) crash_kexec_prepare_cpus
c0000000000519c0 (T) crash_kexec_secondary
c00000000016e8a0 (T) crash_kexec
crash>
On the big-endian ppc64 machines, there was this construct where
the text symbols started with a ".", and there were data symbols
that pointed to them:
crash> sym -q crash_kexec
c000000000050050 (t) .crash_kexec_prepare_cpus
c0000000000502b0 (T) .crash_kexec_secondary
c000000000172160 (T) .crash_kexec
c0000000012ab5f0 (d) crash_kexec_prepare_cpus
c0000000012ab600 (D) crash_kexec_secondary
c0000000012b9238 (D) crash_kexec
crash> rd c0000000012b9238
c0000000012b9238: c000000000172160 ......!`
crash>
So therefore backtraces are guaranteed to fail for the active tasks on ppc64le,
because the stack searching code does this:
for (i = 0, up = (ulong *)bt->stackbuf;
i < (bt->stacktop - bt->stackbase)/sizeof(ulong); i++, up++) {
sym = closest_symbol(*up);
if (STREQ(sym, ".netconsole_netdump") ||
STREQ(sym, ".netpoll_start_netdump") ||
STREQ(sym, ".start_disk_dump") ||
STREQ(sym, ".crash_kexec") ||
STREQ(sym, ".crash_fadump") ||
STREQ(sym, ".disk_dump")) {
*nip = *up;
*ksp = bt->stackbase +
((char *)(up) - 16 - bt->stackbuf);
return TRUE;
}
}
where no ".crash_kexec" symbol exists in ppc64le kernels.
So Han, can you find out who in IBM should be responsible for supporting
ppc64le in the crash utility? Or is it you?
Thanks,
Dave
----- Original Message -----
----- Original Message -----
>
> ----- Original Message -----
> > Hello,
> >
> > I just noticed that on ppc64le, sometimes "bt" cannot find the stack
> > info of current process. For example, there is a vmcore captured by
> > kdump on a ppc64le system, which running with a kernel version 3.10. The
> > vmcore was captured when kernel oopsed. There is no stack info found by
> > bt:
>
> Hello Han,
>
> I've never worked on the backtrace code for ppc64, as it was written
> by (and maintained by) IBM. From the debug messages, what happened is
> that the starting IP/SP hooks are not being found. The crash command
> sequence presumably looks like this:
>
> cmd_bt
> back_trace
> get_kdump_regs
> get_netdump_regs
> get_netdump_regs_ppc64 (should setup bt->machdep to point to
> NT_PRSTATUS note)
> ppc64_get_stack_frame
> ppc64_get_dumpfile_stack_frame
> ppc64_kdump_stack_frame (should get IP/SP pair based upon
> NT_PRSTATUS note contents)
> ppc64_back_trace_cmd
> ppc64_back_trace
>
> ppc64_kdump_stack_frame() should pull the starting NIP/KSP values from the
> pt_regs structure in the per-cpu NT_PRSTATUS note, but it appears that it
> is not,
> leaving the registers at their initialized values of NULL.
>
> This causes the failure later on when ppc64_back_trace_cmd() is called, and
> which
> prints the "=> PC: 0 () FP: 0" debug message shown below, and later on
> ppc64_back_trace()
> prints the "cannot find the stack info." debug message.
>
> Without the dumpfile, I can't offer much else. Can you verify the crash
> utility
> stack trail above, and if it is as I suspect, figure out why
> ppc64_kdump_stack_frame()
> is failing? Or what other path it is taking?
Actually, if this is a compressed kdump, ppc64_kdump_stack_frame() will not
be
called, and the register access is done inside
ppc64_get_dumpfile_stack_frame().
The ppc64_get_dumpfile_stack_frame() function first grabs the registers from
the pt_regs
structure in the per-cpu NT_PRSTATUS note, but then also checks the hard and
soft IRQ
stacks, and the hardware interrupt stack, for known instances of kernel dump
functions,
which would override the pt_regs contents. If nothing is found on those
stacks,
the registers from the NT_PRSTATUS note are used.
Dave
>
> >
> > crash 7.0.9-2.ael7b
> > Copyright (C) 2002-2014 Red Hat, Inc.
> > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> > Copyright (C) 1999-2006 Hewlett-Packard Co
> > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> > Copyright (C) 2005, 2011 NEC Corporation
> > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > This program is free software, covered by the GNU General Public License,
> > and you are welcome to change it and/or distribute copies of it under
> > certain conditions. Enter "help copying" to see the conditions.
> > This program has absolutely no warranty. Enter "help warranty" for
> > details.
> >
> > GNU gdb (GDB) 7.6
> > Copyright (C) 2013 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later
> > <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law. Type "show
> > copying"
> > and "show warranty" for details.
> > This GDB was configured as "powerpc64le-unknown-linux-gnu"...
> >
> > KERNEL: /usr/lib/debug/lib/modules/3.10.0-221.ael7b.ppc64le/vmlinux
> > DUMPFILE: /var/crash/127.0.0.1-2015.01.15-22:19:14/vmcore [PARTIAL
> > DUMP]
> > CPUS: 16
> > DATE: Thu Jan 15 21:18:16 2015
> > UPTIME: 17:53:43
> > LOAD AVERAGE: 213.58, 213.23, 212.70
> > TASKS: 1383
> > NODENAME: thymelp2.isst.aus.stglabs.ibm.com
> > RELEASE: 3.10.0-221.ael7b.ppc64le
> > VERSION: #1 SMP Wed Jan 7 09:27:09 EST 2015
> > MACHINE: ppc64le (3425 Mhz)
> > MEMORY: 15 GB
> > PANIC: "Oops: Kernel access of bad area, sig: 11 [#1]" (check
log
> > for
> > details)
> > PID: 1970
> > COMMAND: "cat"
> > TASK: c0000003130874a0 [THREAD_INFO: c00000005069c000]
> > CPU: 5
> > STATE: TASK_RUNNING (PANIC)
> >
> > crash> set debug 99
> > debug: 99
> > crash> bt
> > PID: 1970 TASK: c0000003130874a0 CPU: 5 COMMAND: "cat"
> > GETBUF(16384 -> 0)
> > <readmem: c00000005069c000, KVADDR, "stack contents", 16384,
(ROE),
> > 10a81570>
> > <read_diskdump: addr: c00000005069c000 paddr: 5069c000 cnt: 16384>
> > read_diskdump: paddr/pfn: 5069c000/5069 -> cache physical page: 50690000
> > c00000005069c018: do_no_restart_syscall
> > c00000005069e870: blk_throtl_bio+240
> > c00000005069e990: clone_endio
> > c00000005069ea00: generic_make_request_checks+836
> > c00000005069eab8: hardware_interrupt_common+128
> > c00000005069eac0: generic_make_request+36
> > c00000005069eb10: mempool_alloc_slab+36
> > c00000005069eb30: mempool_alloc+256
> > c00000005069eb50: mempool_alloc_slab+36
> > c00000005069ebc0: get_request+948
> > c00000005069ec00: __split_and_process_bio+1408
> > c00000005069ec20: autoremove_wake_function
> > c00000005069ec80: find_busiest_group+544
> > c00000005069edf0: load_balance+684
> > c00000005069ee10: blk_throtl_bio+240
> > c00000005069ee70: find_busiest_group+544
> > c00000005069eee0: dequeue_task_fair+968
> > c00000005069ef30: clone_endio
> > c00000005069ef50: get_page_from_freelist+1436
> > c00000005069f0a0: pSeries_cause_ipi_mux+112
> > c00000005069f0c0: smp_send_reschedule+164
> > c00000005069f0e0: default_wake_function+708
> > c00000005069f160: __wake_up_locked+116
> > c00000005069f1b0: ep_poll_callback+444
> > c00000005069f250: run_posix_cpu_timers+104
> > c00000005069f2c0: hvterm_raw_put_chars+64
> > c00000005069f2e0: hvc_console_print+336
> > c00000005069f3a8: initial_stab+2048
> > c00000005069f3b0: crash_save_cpu+252
> > c00000005069f488: cik_cp_resume+13476
> > c00000005069f490: dev_get_drvdata
> > c00000005069f580: default_machine_kexec+332
> > c00000005069f610: pSeries_machine_kexec+60
> > c00000005069f680: machine_kexec+56
> > c00000005069f6a0: crash_kexec+312
> > c00000005069f6f0: dev_attr_show+64
> > c00000005069f748: cik_cp_resume+13476
> > c00000005069f750: dev_get_drvdata
> > c00000005069f7f0: radeon_hwmon_show_temp+72
> > c00000005069f800: slb_miss_realmode+80
> > c00000005069f808: dev_get_drvdata
> > c00000005069f810: radeon_hwmon_show_temp+32
> > c00000005069f890: die+840
> > c00000005069f930: bad_page_fault+224
> > c00000005069f948: radeon_hwmon_show_temp+72
> > c00000005069f9a0: handle_page_fault+44
> > c00000005069fa00: dev_attr_show+64
> > c00000005069fa58: cik_cp_resume+13476
> > c00000005069fa60: dev_get_drvdata
> > c00000005069fb00: radeon_hwmon_show_temp+72
> > c00000005069fb10: slb_miss_realmode+80
> > c00000005069fb18: dev_get_drvdata
> > c00000005069fb20: radeon_hwmon_show_temp+32
> > c00000005069fb60: handle_mm_fault+1724
> > c00000005069fb80: sysfs_open_file
> > c00000005069fbd0: handle_page_fault+16
> > c00000005069fc90: alloc_pages_current+416
> > c00000005069fd00: dev_attr_show+64
> > c00000005069fd30: sysfs_read_file+220
> > c00000005069fde0: sys_read+304
> > c00000005069fe40: syscall_exit
> > [3fffd0d6fe88] back_trace:
> > task: c0000003130874a0
> > flags: 0
> > instptr: 0
> > stkptr: 0
> > bptr: 0
> > stackbase: c00000005069c000
> > stacktop: c0000000506a0000
> > tc: 1003c7b9fa8 (1970, c0000003130874a0)
> > hp: 0
> > ref: 0
> > stackbuf: 10a81570
> > textlist: 0
> > frameptr: 0
> > call_target: none
> > eframe_ip: 0
> > debug: 0
> > radix: 0
> > cpumask: 0
> > => PC: 0 () FP: 0
> > GETBUF(248 -> 1)
> > GETBUF(1500 -> 2)
> > cannot find the stack info.
> > FREEBUF(2)
> > FREEBUF(1)
> > crash>
> >
> >
> > Is this a problem?
> >
> > Thanks in advance!
> >
> > --
> > Crash-utility mailing list
> > Crash-utility(a)redhat.com
> > https://www.redhat.com/mailman/listinfo/crash-utility
> >
>