is crash 7.1.8 meant to support CONFIG_DEBUG_INFO_SPLIT=y / CONFIG_DEBUG_INFO_DWARF4=y kernel builds?
by Jason Vas Dias
Hi -
I have these kernel config options set for a successful kernel build :
$ grep DEBUG_INFO .config
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
CONFIG_DEBUG_INFO_SPLIT=y
CONFIG_DEBUG_INFO_DWARF4=y
This splits debug_info sections into separate ${X}.dwo files for each kernel
object produced.
I modified several files and did a 'make bzImage' , which succeeded,
and installed the kernel, and tried to run crash-7.1.8 to inspect a
few things, but it says:
<quote><pre>
$ crash vmlinuz /proc/kcore
....
gdb called without error_hook: Dwarf Error: wrong version in
compilation unit header (is 0, should be 2, 3, or 4) [in module
/usr/build/linux/linux-4.10/arch/x86/kernel/head64.dwo]
Dwarf Error: wrong version in compilation unit header (is 0, should be
2, 3, or 4) [in module
/usr/build/linux/linux-4.10/arch/x86/kernel/head64.dwo]
crash: vmlinux: no debugging data available
</pre></quote>
But the files still exist from the build:
-rw-r--r-- 1 root root 47784 Feb 21 20:07
/usr/build/linux/linux-4.10/arch/x86/kernel/head64.dwo
-rw-r--r-- 1 root root 17072 Feb 21 20:07
/usr/build/linux/linux-4.10/arch/x86/kernel/head64.o
in the same directory as the head64.c file .
I thought the whole point of 'vmlinux' was that it contained the debug_info ?
Do I need to re-link a vmlinux.dbg with all the *.dwo files
corresponding to each '*.o' file vmlinux contains , and vmlinux?
If so, then I don't see much point in the 'CONFIG_DEBUG_INFO_SPLIT=y'
option. Shouldn't a 'vmlinux.dwo' file be produced, containing all
concatenated
debug_info sections for 'vmlinux' ?
I guess crash just doesn't support
CONFIG_DEBUG_INFO_SPLIT=y / CONFIG_DEBUG_INFO_DWARF4=y ?
Sorry for the newbie type question. Please respond to :
jason.vas.dias(a)gmail.com
I'm not a member of the list.
Thanks & Regards,
Jason
7 years, 9 months
Re: [Crash-utility] Crash-utility Digest, Vol 137, Issue 18
by ぷ风过无痕??
>> >> PID: 30444 TASK: ffff8820454e9520 CPU: 19 COMMAND: "python"
>> >> #0 [ffff88203de8b4f0] machine_kexec at ffffffff8103d1ab
>> >> #1 [ffff88203de8b550] crash_kexec at ffffffff810cc4a2
>> >> #2 [ffff88203de8b620] oops_end at ffffffff8153d540
>> >> #3 [ffff88203de8b650] no_context at ffffffff8104e8cb
>> >> #4 [ffff88203de8b6a0] __bad_area_nosemaphore at ffffffff8104eb55
>> >> #5 [ffff88203de8b6f0] bad_area at ffffffff8104ec7e
>> >> #6 [ffff88203de8b720] __do_page_fault at ffffffff8104f483
>> >> #7 [ffff88203de8b840] do_page_fault at ffffffff8153f48e
>> >> #8 [ffff88203de8b870] page_fault at ffffffff8153c835
[exception RIP: list_del+27]
RIP: ffffffff812a3f8b RSP: ffff88203de8b928 RFLAGS: 00010046
RAX: 0000000000000006 RBX: ffffea006f040028 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffea006f040028
RBP: ffff88203de8b938 R8: ffffea006f040028 R9: 0000000000000000
R10: 0000000000000000 R11: 000000000000000a R12: 0000000000000000
R13: ffff880000021de8 R14: 0000000000000020 R15: ffff880000019d80
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
>> >> #9 [ffff88203de8b940] __rmqueue at ffffffff811336c3
>> >> #10 [ffff88203de8b9e0] get_page_from_freelist at ffffffff81135350
>> >> #11 [ffff88203de8bb10] __alloc_pages_nodemask at ffffffff81136ff9
>> >> #12 [ffff88203de8bc60] alloc_pages_vma at ffffffff8117068a
>> >> #13 [ffff88203de8bcb0] handle_pte_fault at ffffffff81152afd
>> >> #14 [ffff88203de8bd90] handle_mm_fault at ffffffff81153179
>> >> #15 [ffff88203de8be00] __do_page_fault at ffffffff8104f156
>> >> #16 [ffff88203de8bf20] do_page_fault at ffffffff8153f48e
>> >> #17 [ffff88203de8bf50] page_fault at ffffffff8153c835
Hi, Dave. I followed what you said.
sym do_linear_fault , do_non_linear_fault, do_swap_page , do_wp_page
And I only found that do_wp_page's symbol exist, but I also found that __do_fault's symbol exist. According to the source code, I found that only two functions(do_linear_fault , do_non_linear_fault) will call __do_fault() . So how to exclude these two functions(do_linear_fault , do_non_linear_fault)? By the way, the result of 'dis -l handle_pte_fault |grep call |grep do' is as follow, you can see the __do_fault().
crash> dis -l handle_pte_fault|grep call | grep do
0xffffffff8115244b <handle_pte_fault+139>: callq 0xffffffff81151e90 <__do_fault>
0xffffffff811524b2 <handle_pte_fault+242>: callq 0xffffffff81151e90 <__do_fault>
0xffffffff81152688 <handle_pte_fault+712>: callq 0xffffffff81151570 <do_wp_page>
0xffffffff81152d0c <handle_pte_fault+2380>: callq 0xffffffff81151570 <do_wp_page>
0xffffffff81152d5c <handle_pte_fault+2460>: callq 0xffffffff8115cc80 <do_page_add_anon_rmap>
7 years, 9 months
[ANNOUNCE] crash version 7.1.8 is available
by Dave Anderson
Download from: http://people.redhat.com/anderson
or
https://github.com/crash-utility/crash/releases
The github master branch serves as a development branch that will contain
all patches that are queued for the next release:
$ git clone git://github.com/crash-utility/crash.git
Changelog:
- Fix for Linux 4.6 commit b03a017bebc403d40aa53a092e79b3020786537d,
which introduced the new slab management type OBJFREELIST_SLAB.
In this mode, the freelist can be an object, and if the slab is full,
there is no freelist. On the next free, an object is recycled to be
used as the freelist but not cleaned-up. This patch will go through
only known freed objects, and will prevent "kmem -S" errors that
indicate "invalid/corrupt freelist entry" on kernels configured
with CONFIG_SLAB.
(thgarnie(a)google.com)
- Fix for the initialization-time loading of kernel module symbols
if the kernel crashed while running a module's initcall. Without
the patch, the crash session fails during initialation with a message
similar to "crash: store_module_symbols_v2: total: 7 mcnt: 8".
(rabinv(a)axis.com)
- Fix for a segmentation violation during session inialization when
running against a 32-bit MIPS ELF kdump or compressed kdump if a
per-cpu NT_PRSTATUS note cannot be be gathered from the dumpfile
header. Without the the patch, a segmentation violation occurs after
the message "WARNING: cannot find NT_PRSTATUS note for cpu: <number>"
is displayed.
(rabinv(a)axis.com)
- The 32-bit MIPS PGD_ORDER() macro expects __PGD_ORDER to be signed,
which it isn't now since the internal machdep->pagesize is unsigned.
Without this patch, module loading fails during initialization on a
kernel that has a page size of 16KB, with messages that indicate
"please wait... (gathering module symbol data)" followed by
"crash: invalid size request: 0 type: pgd page".
(rabinv(a)axis.com)
- For ARM64 dumpfiles with VMCOREINFO, verify the new "VA_BITS" number
against the calculated number.
(anderson(a)redhat.com)
- Fix for the ARM64 "bt" command in Linux 4.10 and later kernels that
are configured with CONFIG_THREAD_INFO_IN_TASK. Without the patch,
the "bt" command will fail for active tasks in dumpfiles that were
generated by the kdump facility.
(takahiro.akashi(a)linaro.org)
- Fix for Linux 4.10 commit 7fd8329ba502ef76dd91db561c7aed696b2c7720
"taint/module: Clean up global and module taint flags handling".
Without the patch, when running against Linux 4.10-rc1 and later
kernels, the crash utility fails during session initialization with
the message "crash: invalid structure size: tnt".
(panand(a)redhat.com)
- Fix for support of /proc/kcore as the live memory source in Linux 4.8
and later x86_64 kernels configured with CONFIG_RANDOMIZE_BASE, which
randomizes the unity-mapping PAGE_OFFSET value. Without the patch,
the crash session fails during session initialization with the error
message "crash: seek error: kernel virtual address: <address>
type: page_offset_base".
(anderson(a)redhat.com)
- Update to the module taint flags handling patch above to account for
the change in size of the module.taints flag from an int to a long,
while allowing for a kernel backport that keeps it as an int.
(anderson(a)redhat.com)
- Prepare for the kernel's "taint_flag.true" and "taint_flag.false"
member names to be changed to "c_true" and "c_false", which fixes
build problems when an out-of-tree module defines "true" or "false".
(anderson(a)redhat.com)
- Prevent the livepatch taint flag check during the system banner
display from generating a fatal session-killing error if relevant
kernel symbol names or data structures change in the future (again).
(anderson(a)redhat.com)
- Fix for the PPC64 "bt" command for non-panicking active tasks in
FADUMP-generated dumpfiles (Firmware Assisted Dump facility).
Without the patch, backtraces of those tasks may be of the form
"#0 [c0000000700b3a90] (null) at c0000000700b3b50 (unreliable)".
This patch uses and displays the ptregs register set saved in the
dumpfile header for the non-panicking active tasks.
(hbathini(a)linux.vnet.ibm.com)
- Fix for a possible segmentation violation when analyzing Linux 4.6
and earlier x86_64 kernels configured with CONFIG_RANDOMIZE_BASE.
A segmentation violation may occur during session initialization,
just after the patching of the gdb minimal_symbol values message,
depending upon the value of KERNEL_IMAGE_SIZE, which was variable
in the earlier KASLR kernels. This patch sets the KERNEL_IMAGE_SIZE
default value to 1GB for those earlier kernels, and also adds a
new "--machdep kernel_image_size=<value>" option that can be
used to override the default KERNEL_IMAGE_SIZE value if necessary.
(anderson(a)redhat.com)
- Fix the bracketing of the x86_64 FILL_PML4() macro.
(anderson(a)redhat.com)
- Fix for the "tree -t radix", "irq", and "files -p" command options
in Linux 4.6 and later kernels due to upstream changes in the radix
tree facility. Without the patch, the commands will fail with the
message "radix trees do not exist or have changed their format".
(hirofumi(a)mail.parknet.co.jp)
- Fix for the "trace.c" extension module. The kernel buffer referenced
by "max_tr_ring_buffer" is not available in all configurations of the
kernel so the unitialized max_tr_ring_buffer variable should not be
used. A similar check existed previously before the recent rework of
the trace extension module to support multiple buffers.
(rabinv(a)axix.com)
- Clarification in the display of CONFIG_SLUB object addresses that are
displayed by the "kmem" command when SLAB_RED_ZONE has been enabled.
By default, CONFIG_SLUB object addresses that are displayed by the
"kmem" command will point to the SLAB_RED_ZONE padding inserted at
the beginning of the object. As an alternative, a new "redzone"
environment variable has been addedd that can be toggled on or off.
If "set redzone off" is entered, the object addresses will point to
the address that gets returned to the allocator.
(hirofumi(a)mail.parknet.co.jp, anderson(a)redhat.com)
- Fix for the "CURRENT" value displayed by the "timer -r" command.
Without the patch, if the target machine has been up for a long
enough time, an arithmetic overflow will occur and the time value
displayed will be incorrect.
(shane.seymour(a)hpe.com)
- Fix for 32-bit X86 kernels configured with CONFIG_RANDOMIZE_BASE.
Without the patch, an invalid kernel PAGE_OFFSET value is calculated
and as a result the session fails during session initialization just
after the patching of the gdb minimal_symbol values message, showing
the warning message "WARNING: cannot read linux_banner string",
followed by "crash: /vmlinux and /dev/crash do not match!". This
patch also adds a new "--machdep page_offset=<value>" option that
can be used if the CONFIG_PAGE_OFFSET value is not the default
address of 0xc0000000.
(anderson(a)redhat.com)
- Introduction of a new PPC64-only "mach -o" option that dumps the OPAL
"Open Power Abstraction Layer" console buffer.
(ankit(a)linux.vnet.ibm.com)
- Fix for the "bt" command on Linux 4.9 and later 32-bit X86 kernels
containing kernel commit 0100301bfdf56a2a370c7157b5ab0fbf9313e1cd,
subject "sched/x86: Rewrite the switch_to() code". Without the
patch, backtraces for inactive (sleeping) tasks fail with the message
"bt: invalid structure member offset: task_struct_thread_eip".
(anderson(a)redhat.com)
- Fix for a "[-Wmisleading-indentation]" compiler warning and the
associated bug that is generated by lkcd_x86_trace.c when building
32-bit X86 with "make warn" with gcc-6.3.1.
(anderson(a)redhat.com)
- Fix for an invalid "bt" warning on a 32-bit X86 idle/swapper task.
Without the patch, the backtrace displays the "cannot resolve stack
trace" warning, dumps the backtrace, and then the text symbols:
crash> bt
PID: 0 TASK: f0962180 CPU: 6 COMMAND: "swapper/6"
bt: cannot resolve stack trace:
#0 [f095ff1c] __schedule at c0b6ef8d
#1 [f095ff58] schedule at c0b6f4a9
#2 [f095ff64] schedule_preempt_disabled at c0b6f728
#3 [f095ff6c] cpu_startup_entry at c04b0310
#4 [f095ff94] start_secondary at c04468c0
bt: text symbols on stack:
[f095ff1c] __schedule at c0b6ef8d
[f095ff58] schedule at c0b6f4ae
[f095ff64] schedule_preempt_disabled at c0b6f72d
[f095ff6c] cpu_startup_entry at c04b0315
[f095ff94] start_secondary at c04468c5
crash>
The backtrace shown is actually correct.
(anderson(a)redhat.com)
- Another fix for a similar "bt: cannot resolve stack trace" warning
on a 32-bit X86 idle/swapper task, but when running on cpu 0.
(anderson(a)redhat.com)
- Remove two one-time warning messages that are displayed when running
the "bt" command on Linux 4.2 and later 32-bit X86 kernels. Without
the patch, the first "bt" command that is executed will be preceded
by "bt: WARNING: "system_call" symbol does not exist", followed by
"bt: WARNING: neither "ret_from_sys_call" nor "syscall_badsys"
symbols exist".
(anderson(a)redhat.com)
- Fix for Linux 3.15 and later 32-bit X86 kernels containing kernel
commit 198d208df4371734ac4728f69cb585c284d20a15, titled "x86: Keep
thread_info on thread stack in x86_32". Without the patch, incorrect
addresses of each per-cpu hardirq_stack and softirq_stack were saved
for usage by the "bt" command.
(hirofumi(a)mail.parknet.co.jp, anderson(a)redhat.com)
- Additional fix for Linux 3.15 and later 32-bit X86 kernels containing
kernel commit 198d208df4371734ac4728f69cb585c284d20a15, titled "x86:
Keep thread_info on thread stack in x86_32". The patch fixes the
stack transition symbol from "handle_IRQ" to "handle_irq" for usage
by the "bt" command.
(anderson(a)redhat.com)
- Fix for 32-bit X86 kernels to determine the active task in a dumpfile
in the situation where the task was running on its soft IRQ stack,
took a hard IRQ, and then the system crashed while it was running on
its hard IRQ stack.
(hirofumi(a)mail.parknet.co.jp)
- Allow the "--kaslr=<offset>" and/or "--kaslr=auto" command line
options to be used with the 32-bit X86 architecture.
(anderson(a)redhat.com)
- Removed -Werror from the bfd and opcode library builds.
(anderson(a)redhat.com)
7 years, 9 months
Re: [Crash-utility] Crash-utility Digest, Vol 137, Issue 17
by ぷ风过无痕??
>> >> crash> bt 30444
>> >> PID: 30444 TASK: ffff8820454e9520 CPU: 19 COMMAND: "python"
>> >> #0 [ffff88203de8b4f0] machine_kexec at ffffffff8103d1ab
>> >> #1 [ffff88203de8b550] crash_kexec at ffffffff810cc4a2
>> >> #2 [ffff88203de8b620] oops_end at ffffffff8153d540
>> >> #3 [ffff88203de8b650] no_context at ffffffff8104e8cb
>> >> #4 [ffff88203de8b6a0] __bad_area_nosemaphore at ffffffff8104eb55
>> >> #5 [ffff88203de8b6f0] bad_area at ffffffff8104ec7e
>> >> #6 [ffff88203de8b720] __do_page_fault at ffffffff8104f483
>> >> #7 [ffff88203de8b840] do_page_fault at ffffffff8153f48e
>> >> #8 [ffff88203de8b870] page_fault at ffffffff8153c835
[exception RIP: list_del+27]
RIP: ffffffff812a3f8b RSP: ffff88203de8b928 RFLAGS: 00010046
RAX: 0000000000000006 RBX: ffffea006f040028 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffea006f040028
RBP: ffff88203de8b938 R8: ffffea006f040028 R9: 0000000000000000
R10: 0000000000000000 R11: 000000000000000a R12: 0000000000000000
R13: ffff880000021de8 R14: 0000000000000020 R15: ffff880000019d80
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000
>> >> #9 [ffff88203de8b940] __rmqueue at ffffffff811336c3
>> >> #10 [ffff88203de8b9e0] get_page_from_freelist at ffffffff81135350
>> >> #11 [ffff88203de8bb10] __alloc_pages_nodemask at ffffffff81136ff9
>> >> #12 [ffff88203de8bc60] alloc_pages_vma at ffffffff8117068a
>> >> #13 [ffff88203de8bcb0] handle_pte_fault at ffffffff81152afd
>> >> #14 [ffff88203de8bd90] handle_mm_fault at ffffffff81153179
>> >> #15 [ffff88203de8be00] __do_page_fault at ffffffff8104f156
>> >> #16 [ffff88203de8bf20] do_page_fault at ffffffff8153f48e
>> >> #17 [ffff88203de8bf50] page_fault at ffffffff8153c835
Hi, everyone, this is the backtrace from the vmcore in my server. I remember there are some important functions between #13(handle_pte_fault) and #12(handle_pte_fault), like do_linear_fault(), do_nolinear_fault(), do_swap_page(), do_wp_page(). Why these functions cannot be printed in the backtrace? Is the compiler optimized for these functions？If so,how can I make sure the accurate function execution path. By the way, the crash version is crash 7.1.0-6.el6, and I use the centos-6.7 in my server.
Thanks,
Wind
7 years, 9 months
[PATCH]: Implementation of opalmsg command in crash-utility
by Ankit Kumar
This patch creates 'opalmsg' command which can be used to dump opal console log
in crash. Currently it uses hard-coded start address and size of opal console
log buffer.
As this command is specific to POWER, it is coded as such.
Here is the sample output of 'opalmsg' command:
crash> opalmsg
[ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-mukesh-dirty-df9a248 starting...
[ 65.219065872,5] initial console log level: memory 7, driver 5
[ 65.219068917,6] CPU: P8 generation processor(max 8 threads/core)
[ 65.219071681,7] CPU: Boot CPU PIR is 0x0060 PVR is 0x004d0200
[ 65.219074685,7] CPU: Initial max PIR set to 0x1fff
[ 65.219602559,5] OPAL table: 0x300c7440 .. 0x300c78d0, branch table: 0x30002000
[ 65.219607955,5] FDT: Parsing fdt @0xff00000
[ 65.225380389,5] XSCOM: chip 0x8 at 0x3fc4000000000 [P8 DD2.0]
[ 65.225387919,6] XSTOP: XSCOM addr = 0x2010c82, FIR bit = 31
[ 491.377710151,7] PHB#0022: LINK: Link is up
[ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to host
[ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done
Log for dump collected on non OPAL ppc machine:
crash> opalmsg
Dump was not captured on an OPAL based machine
crash> help
btop help opalmsg set vm
On Non POWER machine 'opalmsg' command won't be available as opalmsg is
specific to POWER.
crash> opalmsg
crash: command not found: opalmsg
crash> help opalmsg
NAME
opalmsg - dump opal console log buffer
SYNOPSIS
opalmsg
DESCRIPTION
The opalmsg command retrieves, formats and dumps the OPAL console ring
buffer to help understand the state of OPAL firmware when the dump was
taken
EXAMPLES
crash> opalmsg
[ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-df9a248 starting...
[ 65.225387919,6] XSTOP: XSCOM addr = 0x2010c82, FIR bit = 31
[ 491.377710151,7] PHB#0022: LINK: Link is up
[ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to host
[ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done
Signed-off-by: Ankit Kumar <ankit(a)linux.vnet.ibm.com>
---
defs.h | 2 ++
global_data.c | 3 ++
help.c | 25 ++++++++++++++++
memory.c | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 121 insertions(+)
diff --git a/defs.h b/defs.h
index 68ed4d4..c81154a 100644
--- a/defs.h
+++ b/defs.h
@@ -4623,6 +4623,7 @@ void cmd_template(void); /* tools.c */
void cmd_alias(void); /* cmdline.c */
void cmd_repeat(void); /* cmdline.c */
void cmd_rd(void); /* memory.c */
+void cmd_opalmsg(void); /* memory.c */
void cmd_wr(void); /* memory.c */
void cmd_ptov(void); /* memory.c */
void cmd_vtop(void); /* memory.c */
@@ -5189,6 +5190,7 @@ extern char *help_ptob[];
extern char *help_ptov[];
extern char *help_quit[];
extern char *help_rd[];
+extern char *help_opalmsg[];
extern char *help_repeat[];
extern char *help_runq[];
extern char *help_ipcs[];
diff --git a/global_data.c b/global_data.c
index 998aaae..6bf3406 100644
--- a/global_data.c
+++ b/global_data.c
@@ -102,6 +102,9 @@ struct command_table_entry linux_command_table[] = {
{"q", cmd_quit, help_quit, MINIMAL},
{"tree", cmd_tree, help_tree, REFRESH_TASK_TABLE},
{"rd", cmd_rd, help_rd, MINIMAL},
+#if defined(PPC) || defined(PPC64)
+ {"opalmsg", cmd_opalmsg, help_opalmsg, 0},
+#endif
{"repeat", cmd_repeat, help_repeat, 0},
{"runq", cmd_runq, help_runq, REFRESH_TASK_TABLE},
{"search", cmd_search, help_search, 0},
diff --git a/help.c b/help.c
index 35cd941..75d771e 100644
--- a/help.c
+++ b/help.c
@@ -1717,6 +1717,30 @@ char *help_rd[] = {
NULL
};
+char *help_opalmsg[] = {
+"opalmsg",
+"dump opal console log buffer",
+"",
+" The opalmsg command retrieves, formats and dumps the OPAL console ring",
+" buffer to help understand the state of OPAL firmware when the dump was",
+" taken",
+"\nEXAMPLES",
+"crash> opalmsg",
+"[ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-df9a248 starting...",
+"[ 65.219065872,5] initial console log level: memory 7, driver 5",
+"[ 65.219068917,6] CPU: P8 generation processor(max 8 threads/core)",
+"[ 65.219071681,7] CPU: Boot CPU PIR is 0x0060 PVR is 0x004d0200",
+"[ 65.219074685,7] CPU: Initial max PIR set to 0x1fff",
+"[ 65.219602559,5] OPAL table: 0x300c7440 .. 0x300c78d0, branch table: 0x30002000",
+"[ 65.219607955,5] FDT: Parsing fdt @0xff00000",
+"[ 65.225380389,5] XSCOM: chip 0x8 at 0x3fc4000000000 [P8 DD2.0]",
+"[ 65.225387919,6] XSTOP: XSCOM addr = 0x2010c82, FIR bit = 31",
+"[ 491.377710151,7] PHB#0022: LINK: Link is up",
+"[ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to host",
+"[ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done",
+NULL
+};
+
char *help_wr[] = {
"wr",
"write memory",
@@ -3778,6 +3802,7 @@ char *help_alias[] = {
" builtin for foreach",
" builtin size *",
" builtin dmesg log",
+" builtin opalmsg opalmsg",
" builtin lsmod mod",
" builtin last ps -l",
" ",
diff --git a/memory.c b/memory.c
index 774d090..e7b8495 100644
--- a/memory.c
+++ b/memory.c
@@ -1419,6 +1419,97 @@ struct memloc { /* common holder of read memory */
uint64_t limit64;
};
+/*
+ * Definitions derived from OPAL. These need to track corresponding values in
+ * https://github.com/open-power/skiboot/blob/master/include/mem-map.h
+ */
+#define SKIBOOT_CONSOLE_DUMP_START 0x31000000
+#define SKIBOOT_CONSOLE_DUMP_SIZE 0x40000
+#define SKIBOOT_BASE 0x30000000
+
+void
+cmd_opalmsg(void)
+{
+ int i, a;
+ size_t typesz, sz;
+ void *location;
+ char readtype[20];
+ char *addrtype;
+ struct memloc mem;
+ int displayed, per_line;
+ int lost;
+ ulong error_handle;
+ struct opal {
+ unsigned long long base;
+ unsigned long long entry;
+ } opal;
+ long count = SKIBOOT_CONSOLE_DUMP_SIZE;
+ ulonglong addr = SKIBOOT_CONSOLE_DUMP_START;
+
+ if (CRASHDEBUG(4))
+ fprintf(fp, "<addr: %llx count: %ld (%s)>\n",
+ addr, count, "PHYSADDR");
+
+ /*
+ * Are we on an OPAL platform?
+ * struct opal of BSS section and hence default value will be ZERO(0)
+ * opal_init() in the kernel initializes this structure based on
+ * the platform. Use it as a key to determine whether the dump
+ * was taken on an OPAL based system or not.
+ */
+ if (symbol_exists("opal")) {
+ get_symbol_data("opal", sizeof(struct opal), &opal);
+ if (opal.base != SKIBOOT_BASE) {
+ fprintf(fp, "Dump was not captured on an OPAL based machine");
+ return;
+ }
+ } else {
+ fprintf(fp, "Dump was not captured on an OPAL based machine");
+ return;
+ }
+
+ BZERO(&mem, sizeof(struct memloc));
+ lost = typesz = per_line = 0;
+ location = NULL;
+
+ /* ASCII */
+ typesz = SIZEOF_8BIT;
+ location = &mem.u8;
+ sprintf(readtype, "ascii");
+ per_line = 256;
+ displayed = 0;
+
+ error_handle = FAULT_ON_ERROR;
+
+ for (i = a = 0; i < count; i++) {
+ if (!readmem(addr, PHYSADDR, location, typesz,
+ readtype, error_handle)) {
+ addr += typesz;
+ lost += 1;
+ continue;
+ }
+
+ if (isprint(mem.u8)) {
+ if ((a % per_line) == 0) {
+ if (displayed && i)
+ fprintf(fp, "\n");
+ }
+ fprintf(fp, "%c", mem.u8);
+ displayed++;
+ a++;
+ } else {
+ if (count == ASCII_UNLIMITED)
+ return;
+ a = 0;
+ }
+
+ addr += typesz;
+ }
+
+ if (lost != count)
+ fprintf(fp, "\n");
+}
+
static void
display_memory(ulonglong addr, long count, ulong flag, int memtype, void *opt)
{
--
2.7.4
7 years, 9 months
[PATCH v2]: Adds support in crash-utility to dump opal console buffer for PPC64
by Ankit Kumar
This patch adds support in crash-utility to dump opal console buffer for PPC64
by adding -o option in mach command. Currently start address and size of opal
console log buffer is hard-coded.
As this option is specific to POWER, it is coded as such.
Here is the sample output of 'mach' command with '-o' option:
crash> mach -o
[ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-mukesh-dirty-df9a248 starting...
[ 65.219065872,5] initial console log level: memory 7, driver 5
[ 65.219068917,6] CPU: P8 generation processor(max 8 threads/core)
[ 65.219071681,7] CPU: Boot CPU PIR is 0x0060 PVR is 0x004d0200
[ 65.219074685,7] CPU: Initial max PIR set to 0x1fff
[ 65.219602559,5] OPAL table: 0x300c7440 .. 0x300c78d0, branch table: 0x30002000
[ 65.219607955,5] FDT: Parsing fdt @0xff00000
[ 65.225380389,5] XSCOM: chip 0x8 at 0x3fc4000000000 [P8 DD2.0]
[ 65.225387919,6] XSTOP: XSCOM addr = 0x2010c82, FIR bit = 31
[ 491.377710151,7] PHB#0022: LINK: Link is up
[ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to host
[ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done
Log for dump collected on non OPAL ppc machine:
crash> mach -o
Dump was not captured on an OPAL based machine
crash> help mach
NAME
mach - machine specific data
SYNOPSIS
mach [-m | -c | -o -[xd]]
DESCRIPTION
This command displays data specific to a machine type.
...
-o Display opal console log in crash (ppc64 only).
Display opal console log in crash:
crash> mach -o
[ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-df9a248 starting...
[ 65.219065872,5] initial console log level: memory 7, driver 5
[ 65.219068917,6] CPU: P8 generation processor(max 8 threads/core)
[ 65.219071681,7] CPU: Boot CPU PIR is 0x0060 PVR is 0x004d0200
[ 65.219074685,7] CPU: Initial max PIR set to 0x1fff
[ 65.219607955,5] FDT: Parsing fdt @0xff00000
[ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to host
[ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done
Signed-off-by: Ankit Kumar <ankit(a)linux.vnet.ibm.com>
---
Changelog since v1:(Worked on suggestion given by Dave Anderson)
- Implemented opalmsg as part of 'mach -o' option.
help.c | 14 ++++++++-
ppc64.c | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 116 insertions(+), 2 deletions(-)
diff --git a/help.c b/help.c
index 1503a7c..45890b2 100644
--- a/help.c
+++ b/help.c
@@ -2290,7 +2290,7 @@ NULL
char *help_mach[] = {
"mach",
"machine specific data",
-"[-m | -c -[xd]]",
+"[-m | -c | -o -[xd]]",
" This command displays data specific to a machine type.\n",
" -m Display the physical memory map (x86, x86_64 and ia64 only).",
" -c Display each cpu's cpuinfo structure (x86, x86_64 and ia64 only).",
@@ -2298,6 +2298,7 @@ char *help_mach[] = {
" Display the hwrpb_struct, and each cpu's percpu_struct (alpha only).",
" -x override default output format with hexadecimal format.",
" -d override default output format with decimal format.",
+" -o Display opal console log in crash (ppc64 only).",
"\nEXAMPLES",
" %s> mach",
" MACHINE TYPE: i686",
@@ -2324,6 +2325,17 @@ char *help_mach[] = {
" 00000000fec00000 - 00000000fec90000 E820_RESERVED",
" 00000000fee00000 - 00000000fee10000 E820_RESERVED",
" 00000000ffb00000 - 0000000100000000 E820_RESERVED",
+" ",
+" Display opal console log in crash:\n",
+" %s> mach -o",
+" [ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-df9a248 starting...",
+" [ 65.219065872,5] initial console log level: memory 7, driver 5",
+" [ 65.219068917,6] CPU: P8 generation processor(max 8 threads/core)",
+" [ 65.219071681,7] CPU: Boot CPU PIR is 0x0060 PVR is 0x004d0200",
+" [ 65.219074685,7] CPU: Initial max PIR set to 0x1fff",
+" [ 65.219607955,5] FDT: Parsing fdt @0xff00000",
+" [ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to host",
+" [ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done",
NULL
};
diff --git a/ppc64.c b/ppc64.c
index 8dd0465..21c9827 100644
--- a/ppc64.c
+++ b/ppc64.c
@@ -2733,6 +2733,106 @@ ppc64_get_smp_cpus(void)
return get_cpus_online();
}
+
+/*
+ * Definitions derived from OPAL. These need to track corresponding values in
+ * https://github.com/open-power/skiboot/blob/master/include/mem-map.h
+ */
+#define SKIBOOT_CONSOLE_DUMP_START 0x31000000
+#define SKIBOOT_CONSOLE_DUMP_SIZE 0x40000
+#define SKIBOOT_BASE 0x30000000
+#define ASCII_UNLIMITED ((ulong)(-1) >> 1)
+
+void
+opalmsg(void)
+{
+ struct memloc {
+ uint8_t u8;
+ uint16_t u16;
+ uint32_t u32;
+ uint64_t u64;
+ uint64_t limit64;
+ };
+ struct opal {
+ unsigned long long base;
+ unsigned long long entry;
+ } opal;
+ int i, a;
+ size_t typesz, sz;
+ void *location;
+ char readtype[20];
+ char *addrtype;
+ struct memloc mem;
+ int displayed, per_line;
+ int lost;
+ ulong error_handle;
+ long count = SKIBOOT_CONSOLE_DUMP_SIZE;
+ ulonglong addr = SKIBOOT_CONSOLE_DUMP_START;
+
+ if (CRASHDEBUG(4))
+ fprintf(fp, "<addr: %llx count: %ld (%s)>\n",
+ addr, count, "PHYSADDR");
+
+ /*
+ * OPAL based platform check
+ * struct opal of BSS section and hence default value will be ZERO(0)
+ * opal_init() in the kernel initializes this structure based on
+ * the platform. Use it as a key to determine whether the dump
+ * was taken on an OPAL based system or not.
+ */
+ if (symbol_exists("opal")) {
+ get_symbol_data("opal", sizeof(struct opal), &opal);
+ if (opal.base != SKIBOOT_BASE) {
+ fprintf(fp, "Dump was captured on Non PowerNV Machine");
+ return;
+ }
+ } else {
+ fprintf(fp, "Dump was captured on Non PowerNV Machine");
+ return;
+ }
+
+ BZERO(&mem, sizeof(struct memloc));
+ lost = typesz = per_line = 0;
+ location = NULL;
+
+ /* ASCII */
+ typesz = SIZEOF_8BIT;
+ location = &mem.u8;
+ sprintf(readtype, "ascii");
+ per_line = 256;
+ displayed = 0;
+
+ error_handle = FAULT_ON_ERROR;
+
+ for (i = a = 0; i < count; i++) {
+ if (!readmem(addr, PHYSADDR, location, typesz,
+ readtype, error_handle)) {
+ addr += typesz;
+ lost += 1;
+ continue;
+ }
+
+ if (isprint(mem.u8)) {
+ if ((a % per_line) == 0) {
+ if (displayed && i)
+ fprintf(fp, "\n");
+ }
+ fprintf(fp, "%c", mem.u8);
+ displayed++;
+ a++;
+ } else {
+ if (count == ASCII_UNLIMITED)
+ return;
+ a = 0;
+ }
+
+ addr += typesz;
+ }
+
+ if (lost != count)
+ fprintf(fp, "\n");
+}
+
/*
* Machine dependent command.
*/
@@ -2741,7 +2841,7 @@ ppc64_cmd_mach(void)
{
int c;
- while ((c = getopt(argcnt, args, "cm")) != EOF) {
+ while ((c = getopt(argcnt, args, "cmo")) != EOF) {
switch(c)
{
case 'c':
@@ -2749,6 +2849,8 @@ ppc64_cmd_mach(void)
fprintf(fp, "PPC64: '-%c' option is not supported\n",
c);
break;
+ case 'o':
+ return opalmsg();
default:
argerrs++;
break;
--
2.7.4
7 years, 9 months
[PATCH] Fix x86 initialization for {hard, soft}irq_ctx
by OGAWA Hirofumi
Current code is setting {hard,soft}irq_ctx[] to (irq_ctx **), because
per_cpu symbol itself is pointer of specified type (irq_ctx *).
But, I wonder how this works in past, the code is expecting
{hard,soft}_ctx[] are (irq_ctx *). This fixes by deref per_cpu in
initialization, and set expected pointers.
Tested on i386 v3.10.
---
task.c | 28 ++++++++++++++++++++++++----
1 file changed, 24 insertions(+), 4 deletions(-)
diff -puN task.c~ia32-irq-stack-fix task.c
--- crash-64/task.c~ia32-irq-stack-fix 2017-02-08 17:37:33.126367767 +0900
+++ crash-64-hirofumi/task.c 2017-02-08 17:46:06.900703326 +0900
@@ -570,10 +570,20 @@ irqstacks_init(void)
if ((hard_sp = per_cpu_symbol_search("per_cpu__hardirq_ctx"))) {
if ((kt->flags & SMP) && (kt->flags & PER_CPU_OFF)) {
for (i = 0; i < NR_CPUS; i++) {
+ ulong ptr;
+
if (!kt->__per_cpu_offset[i])
continue;
- tt->hardirq_ctx[i] = hard_sp->value +
- kt->__per_cpu_offset[i];
+ ptr = hard_sp->value + kt->__per_cpu_offset[i];
+
+ if (!readmem(ptr, KVADDR, &ptr,
+ sizeof(void *), "hardirq ctx",
+ RETURN_ON_ERROR)) {
+ error(INFO, "cannot read hardirq_ctx[%d] at %lx\n",
+ i, ptr);
+ continue;
+ }
+ tt->hardirq_ctx[i] = ptr;
}
} else
tt->hardirq_ctx[0] = hard_sp->value;
@@ -604,10 +614,20 @@ irqstacks_init(void)
if ((soft_sp = per_cpu_symbol_search("per_cpu__softirq_ctx"))) {
if ((kt->flags & SMP) && (kt->flags & PER_CPU_OFF)) {
for (i = 0; i < NR_CPUS; i++) {
+ ulong ptr;
+
if (!kt->__per_cpu_offset[i])
continue;
- tt->softirq_ctx[i] = soft_sp->value +
- kt->__per_cpu_offset[i];
+ ptr = soft_sp->value + kt->__per_cpu_offset[i];
+
+ if (!readmem(ptr, KVADDR, &ptr,
+ sizeof(void *), "softirq ctx",
+ RETURN_ON_ERROR)) {
+ error(INFO, "cannot read softirq_ctx[%d] at %lx\n",
+ i, ptr);
+ continue;
+ }
+ tt->softirq_ctx[i] = ptr;
}
} else
tt->softirq_ctx[0] = soft_sp->value;
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
7 years, 9 months
[PATCH] overflow calculating time in timer output after long uptime
by Seymour, Shane M
Hi Dave,
I've been staring at an issue in crash all afternoon trying to work out what happened to the current clock value printed by timer -r. I've got output from the timer -r command that looks like (the vmcore has 315+ days of uptime) this:
CPU: 0 HRTIMER_CPU_BASE: ffff883f7fa0d7a0
CLOCK: 0 HRTIMER_CLOCK_BASE: ffff883f7fa0d7e0 [ktime_get]
CURRENT
9017759896290448
SOFTEXPIRES EXPIRES HRTIMER FUNCTION
27464498630000000 27464498630000000 ffff883f7fa0da40 ffffffff810d0440 <tick_sched_timer>
27464498643441992 27464498643491992 ffff880243f5bd60 ffffffff8109ae70 <hrtimer_wakeup>
27464498690255938 27464498690755937 ffff8842d7c1b908 ffffffff8109ae70 <hrtimer_wakeup>
...
The time under the CURRENT heading is way less than the softexpires/expires which really didn't look correct so I dug into the crash code and found that the calculation in dump_hrtimer_clock_base() can overflow for a largish value of current_time (I did the calculations involved in working out what current_time was by hand to make sure that they would overflow and they did, but I haven't included those workings). The value above works out to be the amount over the maximum value of an unsigned 64bit in that would have been calculated (i.e. it wrapped).
When I changed it from:
now = current_time * 1000000000LL / machdep->hz + offset;
to
now = current_time * (1000000000LL / machdep->hz) + offset;
It works as I expected it to. I've tested with that change and the fixed output is:
CPU: 0 HRTIMER_CPU_BASE: ffff883f7fa0d7a0
CLOCK: 0 HRTIMER_CLOCK_BASE: ffff883f7fa0d7e0 [ktime_get]
CURRENT
27464503970000000
SOFTEXPIRES EXPIRES HRTIMER FUNCTION
27464498630000000 27464498630000000 ffff883f7fa0da40 ffffffff810d0440 <tick_sched_timer>
27464498643441992 27464498643491992 ffff880243f5bd60 ffffffff8109ae70 <hrtimer_wakeup>
27464498690255938 27464498690755937 ffff8842d7c1b908 ffffffff8109ae70 <hrtimer_wakeup>
There's a second one in dump_hrtimer_base():
now = current_time * 1000000000LL / machdep->hz;
that may have similar issues but it wasn't impacting on the vmcore I was looking at but I changed it anyway.
Thanks
Shane
P.S. cut and pasted the diff output not sure if it's going to work correctly or not because of that.
---
--- a/kernel.c 2016-11-30 13:56:29.000000000 -0500
+++ b/kernel.c 2017-02-06 02:23:35.179164828 -0500
@@ -7263,7 +7263,7 @@ dump_hrtimer_clock_base(const void *hrti
offset = 0;
if (VALID_MEMBER(hrtimer_clock_base_offset))
offset = ktime_to_ns(base + OFFSET(hrtimer_clock_base_offset));
- now = current_time * 1000000000LL / machdep->hz + offset;
+ now = current_time * (1000000000LL / machdep->hz) + offset;
dump_active_timers(base, now);
}
@@ -7285,7 +7285,7 @@ dump_hrtimer_base(const void *hrtimer_ba
/* get current time(uptime) */
get_uptime(NULL, ¤t_time);
- now = current_time * 1000000000LL / machdep->hz;
+ now = current_time * (1000000000LL / machdep->hz);
dump_active_timers(base, now);
}
7 years, 9 months
Re: [Crash-utility] [PATCH] Fix for "kmem <addr>" for kernels configured with CONFIG_SLUB and SLAB_RED_ZONE.
by OGAWA Hirofumi
Dave Anderson <anderson(a)redhat.com> writes:
>> If SLAB_RED_ZONE is enabled, slub adds guard zone of sizeof(void *)
>> onto head of slab page (red zone padding of left of object) on v4.6 or
>> later.
>>
>> Without this fix, like following SUPERBLK and [allocate addr] has
>> difference.
>>
>> crash> mount
>> MOUNT SUPERBLK TYPE DEVNAME DIRNAME
>> ffff88013ae58040 ffff88013ac35698 rootfs rootfs /
>> [...]
>> crash> kmem ffff88013ac35698
>> CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE
>> ffff88013ac05bc0 kmalloc-4096 4096 118 126 18 32k
>> SLAB MEMORY NODE TOTAL ALLOCATED FREE
>> ffffea0004eb0c00 ffff88013ac30000 0 7 7 0
>> FREE / [ALLOCATED]
>> [ffff88013ac35690]
>> [...]
>
> Hi Ogawa,
>
> When you enter "kmem ffff88013ac35698", I am presuming that it it
> shows the [ALLOCATED] object at ffff88013ac35690 as shown above. If
> that's true, then in my opinion, it's doing the right thing.
>
> I understand that the kmalloc caller receives ffff88013ac35698, but
> with respect to the slab subsystem itself, the actual address of the
> slab object is ffff88013ac35690. I worry about the potential
> consequences of making such a wholesale change to the kmem command.
Hm, [addr] is not dumping slab address. It is dumping objects on slab
actually. So I think rather confuses users. (BTW, slab address is
represented as si->slab, and changes is against to vaddr)
Another example is,
crash> kmem ffffea0004d11600
CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE
ffff8801382271c0 ext4_inode_cache 2200 864 864 72 32k
SLAB MEMORY NODE TOTAL ALLOCATED FREE
ffffea0004d11600 ffff880134458000 0 12 12 0
FREE / [ALLOCATED]
[ffff880134458000]
[ffff8801344589e8]
[ffff8801344593d0]
[ffff880134459db8]
[ffff88013445a7a0]
[ffff88013445b188]
[ffff88013445bb70]
[ffff88013445c558]
[ffff88013445cf40]
[ffff88013445d928]
[ffff88013445e310]
[ffff88013445ecf8]
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffffea0004d11600 134458000 0 0 1 8000000000004080 slab,head
crash> struct ext4_inode_info.vfs_inode.i_op ffff880134458000
vfs_inode.i_op = 0xffffffffffffffff,
crash> struct ext4_inode_info.vfs_inode.i_op ffff8801344589e8
vfs_inode.i_op = 0xffffffffffffffff,
Specified slab address as argument of kmem command. It lists objects up.
But as said, all objects are actually having 8 bytes diff. Correct
result are the following.
crash> kmem ffffea0004d11600
CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE
ffff8801382271c0 ext4_inode_cache 2200 864 864 72 32k
SLAB MEMORY NODE TOTAL ALLOCATED FREE
ffffea0004d11600 ffff880134458000 0 12 12 0
FREE / [ALLOCATED]
[ffff880134458008]
[ffff8801344589f0]
[ffff8801344593d8]
[ffff880134459dc0]
[ffff88013445a7a8]
[ffff88013445b190]
[ffff88013445bb78]
[ffff88013445c560]
[ffff88013445cf48]
[ffff88013445d930]
[ffff88013445e318]
[ffff88013445ed00]
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ffffea0004d11600 134458000 0 0 1 8000000000004080 slab,head
crash> struct ext4_inode_info.vfs_inode.i_op ffff880134458008
vfs_inode.i_op = 0xffffffff81c2fe40 <ext4_fast_symlink_inode_operations>,
crash> struct ext4_inode_info.vfs_inode.i_op ffff8801344589f0
vfs_inode.i_op = 0xffffffff81c2fe40 <ext4_fast_symlink_inode_operations>,
Thanks.
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
7 years, 9 months
[PATCH] trace: don't use uninited max_tr_ring_buffer
by Rabin Vincent
From: Rabin Vincent <rabinv(a)axis.com>
This max buffer is not available in all configurations of the kernel so
we shouldn't attempt to use the unitialized max_tr_ring_buffer variable.
A similar check existed previously before the recent rework of the trace
extension to support multiple buffers.
---
extensions/trace.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/extensions/trace.c b/extensions/trace.c
index 51df98d..22039a9 100644
--- a/extensions/trace.c
+++ b/extensions/trace.c
@@ -487,6 +487,9 @@ static int ftrace_init_trace(struct trace_instance *ti, ulong instance_addr)
ti->pages) < 0)
goto out_fail;
+ if (!ti->max_tr_ring_buffer)
+ return 0;
+
ti->max_tr_buffers = calloc(sizeof(*ti->max_tr_buffers), nr_cpu_ids);
if (ti->max_tr_buffers == NULL)
goto out_fail;
--
2.1.4
7 years, 9 months
