9:04 a.m.
This patch creates 'opalmsg' command which can be used to dump opal console log
in crash. Currently it uses hard-coded start address and size of opal console
log buffer.
As this command is specific to POWER, it is coded as such.
Here is the sample output of 'opalmsg' command:
crash> opalmsg
[ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-mukesh-dirty-df9a248 starting...
[ 65.219065872,5] initial console log level: memory 7, driver 5
[ 65.219068917,6] CPU: P8 generation processor(max 8 threads/core)
[ 65.219071681,7] CPU: Boot CPU PIR is 0x0060 PVR is 0x004d0200
[ 65.219074685,7] CPU: Initial max PIR set to 0x1fff
[ 65.219602559,5] OPAL table: 0x300c7440 .. 0x300c78d0, branch table: 0x30002000
[ 65.219607955,5] FDT: Parsing fdt @0xff00000
[ 65.225380389,5] XSCOM: chip 0x8 at 0x3fc4000000000 [P8 DD2.0]
[ 65.225387919,6] XSTOP: XSCOM addr = 0x2010c82, FIR bit = 31
[ 491.377710151,7] PHB#0022: LINK: Link is up
[ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to host
[ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done
Log for dump collected on non OPAL ppc machine:
crash> opalmsg
Dump was not captured on an OPAL based machine
crash> help
btop help opalmsg set vm
On Non POWER machine 'opalmsg' command won't be available as opalmsg is
specific to POWER.
crash> opalmsg
crash: command not found: opalmsg
crash> help opalmsg
NAME
opalmsg - dump opal console log buffer
SYNOPSIS
opalmsg
DESCRIPTION
The opalmsg command retrieves, formats and dumps the OPAL console ring
buffer to help understand the state of OPAL firmware when the dump was
taken
EXAMPLES
crash> opalmsg
[ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-df9a248 starting...
[ 65.225387919,6] XSTOP: XSCOM addr = 0x2010c82, FIR bit = 31
[ 491.377710151,7] PHB#0022: LINK: Link is up
[ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to host
[ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done
Signed-off-by: Ankit Kumar <ankit(a)linux.vnet.ibm.com>
---
defs.h | 2 ++
global_data.c | 3 ++
help.c | 25 ++++++++++++++++
memory.c | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 121 insertions(+)
diff --git a/defs.h b/defs.h
index 68ed4d4..c81154a 100644
--- a/defs.h
+++ b/defs.h
@@ -4623,6 +4623,7 @@ void cmd_template(void); /* tools.c */
void cmd_alias(void); /* cmdline.c */
void cmd_repeat(void); /* cmdline.c */
void cmd_rd(void); /* memory.c */
+void cmd_opalmsg(void); /* memory.c */
void cmd_wr(void); /* memory.c */
void cmd_ptov(void); /* memory.c */
void cmd_vtop(void); /* memory.c */
@@ -5189,6 +5190,7 @@ extern char *help_ptob[];
extern char *help_ptov[];
extern char *help_quit[];
extern char *help_rd[];
+extern char *help_opalmsg[];
extern char *help_repeat[];
extern char *help_runq[];
extern char *help_ipcs[];
diff --git a/global_data.c b/global_data.c
index 998aaae..6bf3406 100644
--- a/global_data.c
+++ b/global_data.c
@@ -102,6 +102,9 @@ struct command_table_entry linux_command_table[] = {
{"q", cmd_quit, help_quit, MINIMAL},
{"tree", cmd_tree, help_tree, REFRESH_TASK_TABLE},
{"rd", cmd_rd, help_rd, MINIMAL},
+#if defined(PPC) || defined(PPC64)
+ {"opalmsg", cmd_opalmsg, help_opalmsg, 0},
+#endif
{"repeat", cmd_repeat, help_repeat, 0},
{"runq", cmd_runq, help_runq, REFRESH_TASK_TABLE},
{"search", cmd_search, help_search, 0},
diff --git a/help.c b/help.c
index 35cd941..75d771e 100644
--- a/help.c
+++ b/help.c
@@ -1717,6 +1717,30 @@ char *help_rd[] = {
NULL
};
+char *help_opalmsg[] = {
+"opalmsg",
+"dump opal console log buffer",
+"",
+" The opalmsg command retrieves, formats and dumps the OPAL console ring",
+" buffer to help understand the state of OPAL firmware when the dump was",
+" taken",
+"\nEXAMPLES",
+"crash> opalmsg",
+"[ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-df9a248 starting...",
+"[ 65.219065872,5] initial console log level: memory 7, driver 5",
+"[ 65.219068917,6] CPU: P8 generation processor(max 8 threads/core)",
+"[ 65.219071681,7] CPU: Boot CPU PIR is 0x0060 PVR is 0x004d0200",
+"[ 65.219074685,7] CPU: Initial max PIR set to 0x1fff",
+"[ 65.219602559,5] OPAL table: 0x300c7440 .. 0x300c78d0, branch table:
0x30002000",
+"[ 65.219607955,5] FDT: Parsing fdt @0xff00000",
+"[ 65.225380389,5] XSCOM: chip 0x8 at 0x3fc4000000000 [P8 DD2.0]",
+"[ 65.225387919,6] XSTOP: XSCOM addr = 0x2010c82, FIR bit = 31",
+"[ 491.377710151,7] PHB#0022: LINK: Link is up",
+"[ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to host",
+"[ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done",
+NULL
+};
+
char *help_wr[] = {
"wr",
"write memory",
@@ -3778,6 +3802,7 @@ char *help_alias[] = {
" builtin for foreach",
" builtin size *",
" builtin dmesg log",
+" builtin opalmsg opalmsg",
" builtin lsmod mod",
" builtin last ps -l",
" ",
diff --git a/memory.c b/memory.c
index 774d090..e7b8495 100644
--- a/memory.c
+++ b/memory.c
@@ -1419,6 +1419,97 @@ struct memloc { /* common holder of read memory
*/
uint64_t limit64;
};
+/*
+ * Definitions derived from OPAL. These need to track corresponding values in
+ * https://github.com/open-power/skiboot/blob/master/include/mem-map.h
+ */
+#define SKIBOOT_CONSOLE_DUMP_START 0x31000000
+#define SKIBOOT_CONSOLE_DUMP_SIZE 0x40000
+#define SKIBOOT_BASE 0x30000000
+
+void
+cmd_opalmsg(void)
+{
+ int i, a;
+ size_t typesz, sz;
+ void *location;
+ char readtype[20];
+ char *addrtype;
+ struct memloc mem;
+ int displayed, per_line;
+ int lost;
+ ulong error_handle;
+ struct opal {
+ unsigned long long base;
+ unsigned long long entry;
+ } opal;
+ long count = SKIBOOT_CONSOLE_DUMP_SIZE;
+ ulonglong addr = SKIBOOT_CONSOLE_DUMP_START;
+
+ if (CRASHDEBUG(4))
+ fprintf(fp, "<addr: %llx count: %ld (%s)>\n",
+ addr, count, "PHYSADDR");
+
+ /*
+ * Are we on an OPAL platform?
+ * struct opal of BSS section and hence default value will be ZERO(0)
+ * opal_init() in the kernel initializes this structure based on
+ * the platform. Use it as a key to determine whether the dump
+ * was taken on an OPAL based system or not.
+ */
+ if (symbol_exists("opal")) {
+ get_symbol_data("opal", sizeof(struct opal), &opal);
+ if (opal.base != SKIBOOT_BASE) {
+ fprintf(fp, "Dump was not captured on an OPAL based machine");
+ return;
+ }
+ } else {
+ fprintf(fp, "Dump was not captured on an OPAL based machine");
+ return;
+ }
+
+ BZERO(&mem, sizeof(struct memloc));
+ lost = typesz = per_line = 0;
+ location = NULL;
+
+ /* ASCII */
+ typesz = SIZEOF_8BIT;
+ location = &mem.u8;
+ sprintf(readtype, "ascii");
+ per_line = 256;
+ displayed = 0;
+
+ error_handle = FAULT_ON_ERROR;
+
+ for (i = a = 0; i < count; i++) {
+ if (!readmem(addr, PHYSADDR, location, typesz,
+ readtype, error_handle)) {
+ addr += typesz;
+ lost += 1;
+ continue;
+ }
+
+ if (isprint(mem.u8)) {
+ if ((a % per_line) == 0) {
+ if (displayed && i)
+ fprintf(fp, "\n");
+ }
+ fprintf(fp, "%c", mem.u8);
+ displayed++;
+ a++;
+ } else {
+ if (count == ASCII_UNLIMITED)
+ return;
+ a = 0;
+ }
+
+ addr += typesz;
+ }
+
+ if (lost != count)
+ fprintf(fp, "\n");
+}
+
static void
display_memory(ulonglong addr, long count, ulong flag, int memtype, void *opt)
{
--
2.7.4
3:41 p.m.
New subject: [PATCH]: Implementation of opalmsg command in crash-utility
----- Original Message -----
This patch creates 'opalmsg' command which can be used to
dump opal console log
in crash. Currently it uses hard-coded start address and size of opal console log
buffer.
As this command is specific to POWER, it is coded as such.
Hi Ankit,
There's no way I'm going to add such a subsystem- and machine-specific command to
the standard
set of generic commands. The base command set is almost never extended, where, for
example,
the "ipcs" and "tree" commands were added in 2012, and are the only
ones added since the
crash utility was originally introduced.
Here are a couple suggestions:
(1) Make it an extension module, which can be hosted on the extensions page here:
http://people.redhat.com/anderson/extensions.html
(2) Make it an option to the "mach" command, which does advertise several
machine-specific
options:
crash> help mach
NAME
mach - machine specific data
SYNOPSIS
mach [-m | -c -[xd]]
DESCRIPTION
This command displays data specific to a machine type.
-m Display the physical memory map (x86, x86_64 and ia64 only).
-c Display each cpu's cpuinfo structure (x86, x86_64 and ia64 only).
Display each cpu's x8664_pda structure (x86_64 only),
Display the hwrpb_struct, and each cpu's percpu_struct (alpha only).
-x override default output format with hexadecimal format.
-d override default output format with decimal format.
EXAMPLES
...
Your choice, but at least "mach -o" or whatever would allow it to be contained
in the crash binary. Sound reasonable?
Dave
Here is the sample output of 'opalmsg' command:
crash> opalmsg
[ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-mukesh-dirty-df9a248
starting...
[ 65.219065872,5] initial console log level: memory 7, driver 5
[ 65.219068917,6] CPU: P8 generation processor(max 8 threads/core)
[ 65.219071681,7] CPU: Boot CPU PIR is 0x0060 PVR is 0x004d0200
[ 65.219074685,7] CPU: Initial max PIR set to 0x1fff
[ 65.219602559,5] OPAL table: 0x300c7440 .. 0x300c78d0, branch table:
0x30002000
[ 65.219607955,5] FDT: Parsing fdt @0xff00000
[ 65.225380389,5] XSCOM: chip 0x8 at 0x3fc4000000000 [P8 DD2.0]
[ 65.225387919,6] XSTOP: XSCOM addr = 0x2010c82, FIR bit = 31
[ 491.377710151,7] PHB#0022: LINK: Link is up
[ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to host
[ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done
Log for dump collected on non OPAL ppc machine:
crash> opalmsg
Dump was not captured on an OPAL based machine
crash> help
btop help opalmsg set vm
On Non POWER machine 'opalmsg' command won't be available as opalmsg is
specific to POWER.
crash> opalmsg
crash: command not found: opalmsg
crash> help opalmsg
NAME
opalmsg - dump opal console log buffer
SYNOPSIS
opalmsg
DESCRIPTION
The opalmsg command retrieves, formats and dumps the OPAL console ring
buffer to help understand the state of OPAL firmware when the dump was
taken
EXAMPLES
crash> opalmsg
[ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-df9a248 starting...
[ 65.225387919,6] XSTOP: XSCOM addr = 0x2010c82, FIR bit = 31
[ 491.377710151,7] PHB#0022: LINK: Link is up
[ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to host
[ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done
Signed-off-by: Ankit Kumar <ankit(a)linux.vnet.ibm.com>
---
defs.h | 2 ++
global_data.c | 3 ++
help.c | 25 ++++++++++++++++
memory.c | 91
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 121 insertions(+)
diff --git a/defs.h b/defs.h
index 68ed4d4..c81154a 100644
--- a/defs.h
+++ b/defs.h
@@ -4623,6 +4623,7 @@ void cmd_template(void); /* tools.c */
void cmd_alias(void); /* cmdline.c */
void cmd_repeat(void); /* cmdline.c */
void cmd_rd(void); /* memory.c */
+void cmd_opalmsg(void); /* memory.c */
void cmd_wr(void); /* memory.c */
void cmd_ptov(void); /* memory.c */
void cmd_vtop(void); /* memory.c */
@@ -5189,6 +5190,7 @@ extern char *help_ptob[];
extern char *help_ptov[];
extern char *help_quit[];
extern char *help_rd[];
+extern char *help_opalmsg[];
extern char *help_repeat[];
extern char *help_runq[];
extern char *help_ipcs[];
diff --git a/global_data.c b/global_data.c
index 998aaae..6bf3406 100644
--- a/global_data.c
+++ b/global_data.c
@@ -102,6 +102,9 @@ struct command_table_entry linux_command_table[] = {
{"q", cmd_quit, help_quit, MINIMAL},
{"tree", cmd_tree, help_tree, REFRESH_TASK_TABLE},
{"rd", cmd_rd, help_rd, MINIMAL},
+#if defined(PPC) || defined(PPC64)
+ {"opalmsg", cmd_opalmsg, help_opalmsg, 0},
+#endif
{"repeat", cmd_repeat, help_repeat, 0},
{"runq", cmd_runq, help_runq, REFRESH_TASK_TABLE},
{"search", cmd_search, help_search, 0},
diff --git a/help.c b/help.c
index 35cd941..75d771e 100644
--- a/help.c
+++ b/help.c
@@ -1717,6 +1717,30 @@ char *help_rd[] = {
NULL
};
+char *help_opalmsg[] = {
+"opalmsg",
+"dump opal console log buffer",
+"",
+" The opalmsg command retrieves, formats and dumps the OPAL console ring",
+" buffer to help understand the state of OPAL firmware when the dump was",
+" taken",
+"\nEXAMPLES",
+"crash> opalmsg",
+"[ 65.219056911,5] SkiBoot skiboot-5.4.0-218-ge0225cc-df9a248
starting...",
+"[ 65.219065872,5] initial console log level: memory 7, driver 5",
+"[ 65.219068917,6] CPU: P8 generation processor(max 8 threads/core)",
+"[ 65.219071681,7] CPU: Boot CPU PIR is 0x0060 PVR is 0x004d0200",
+"[ 65.219074685,7] CPU: Initial max PIR set to 0x1fff",
+"[ 65.219602559,5] OPAL table: 0x300c7440 .. 0x300c78d0, branch table:
0x30002000",
+"[ 65.219607955,5] FDT: Parsing fdt @0xff00000",
+"[ 65.225380389,5] XSCOM: chip 0x8 at 0x3fc4000000000 [P8 DD2.0]",
+"[ 65.225387919,6] XSTOP: XSCOM addr = 0x2010c82, FIR bit = 31",
+"[ 491.377710151,7] PHB#0022: LINK: Link is up",
+"[ 494.026291523,7] BT: seq 0x25 netfn 0x0a cmd 0x48: Message sent to
host",
+"[ 494.027636927,7] BT: seq 0x25 netfn 0x0a cmd 0x48: IPMI MSG done",
+NULL
+};
+
char *help_wr[] = {
"wr",
"write memory",
@@ -3778,6 +3802,7 @@ char *help_alias[] = {
" builtin for foreach",
" builtin size *",
" builtin dmesg log",
+" builtin opalmsg opalmsg",
" builtin lsmod mod",
" builtin last ps -l",
" ",
diff --git a/memory.c b/memory.c
index 774d090..e7b8495 100644
--- a/memory.c
+++ b/memory.c
@@ -1419,6 +1419,97 @@ struct memloc { /* common holder of
read memory */
uint64_t limit64;
};
+/*
+ * Definitions derived from OPAL. These need to track corresponding values
in
+ * https://github.com/open-power/skiboot/blob/master/include/mem-map.h
+ */
+#define SKIBOOT_CONSOLE_DUMP_START 0x31000000
+#define SKIBOOT_CONSOLE_DUMP_SIZE 0x40000
+#define SKIBOOT_BASE 0x30000000
+
+void
+cmd_opalmsg(void)
+{
+ int i, a;
+ size_t typesz, sz;
+ void *location;
+ char readtype[20];
+ char *addrtype;
+ struct memloc mem;
+ int displayed, per_line;
+ int lost;
+ ulong error_handle;
+ struct opal {
+ unsigned long long base;
+ unsigned long long entry;
+ } opal;
+ long count = SKIBOOT_CONSOLE_DUMP_SIZE;
+ ulonglong addr = SKIBOOT_CONSOLE_DUMP_START;
+
+ if (CRASHDEBUG(4))
+ fprintf(fp, "<addr: %llx count: %ld (%s)>\n",
+ addr, count, "PHYSADDR");
+
+ /*
+ * Are we on an OPAL platform?
+ * struct opal of BSS section and hence default value will be ZERO(0)
+ * opal_init() in the kernel initializes this structure based on
+ * the platform. Use it as a key to determine whether the dump
+ * was taken on an OPAL based system or not.
+ */
+ if (symbol_exists("opal")) {
+ get_symbol_data("opal", sizeof(struct opal), &opal);
+ if (opal.base != SKIBOOT_BASE) {
+ fprintf(fp, "Dump was not captured on an OPAL based machine");
+ return;
+ }
+ } else {
+ fprintf(fp, "Dump was not captured on an OPAL based machine");
+ return;
+ }
+
+ BZERO(&mem, sizeof(struct memloc));
+ lost = typesz = per_line = 0;
+ location = NULL;
+
+ /* ASCII */
+ typesz = SIZEOF_8BIT;
+ location = &mem.u8;
+ sprintf(readtype, "ascii");
+ per_line = 256;
+ displayed = 0;
+
+ error_handle = FAULT_ON_ERROR;
+
+ for (i = a = 0; i < count; i++) {
+ if (!readmem(addr, PHYSADDR, location, typesz,
+ readtype, error_handle)) {
+ addr += typesz;
+ lost += 1;
+ continue;
+ }
+
+ if (isprint(mem.u8)) {
+ if ((a % per_line) == 0) {
+ if (displayed && i)
+ fprintf(fp, "\n");
+ }
+ fprintf(fp, "%c", mem.u8);
+ displayed++;
+ a++;
+ } else {
+ if (count == ASCII_UNLIMITED)
+ return;
+ a = 0;
+ }
+
+ addr += typesz;
+ }
+
+ if (lost != count)
+ fprintf(fp, "\n");
+}
+
static void
display_memory(ulonglong addr, long count, ulong flag, int memtype, void
*opt)
{
--
2.7.4
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
5:12 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
----- Original Message -----
Current code is setting {hard,soft}irq_ctx[] to (irq_ctx **), because
per_cpu symbol itself is pointer of specified type (irq_ctx *).
But, I wonder how this works in past, the code is expecting
{hard,soft}_ctx[] are (irq_ctx *). This fixes by deref per_cpu in
initialization, and set expected pointers.
Tested on i386 v3.10.
You are correct -- apparently it never did work.
The reason why I never noticed with RHEL kernels is that up until linux-2.6.29,
the IRQ stacks were declared as a static NR_CPUS-bounded array:
#ifdef CONFIG_4KSTACKS
/*
* per-CPU IRQ handling contexts (thread information and stack)
*/
union irq_ctx {
struct thread_info tinfo;
u32 stack[THREAD_SIZE/sizeof(u32)];
};
static union irq_ctx *hardirq_ctx[NR_CPUS] __read_mostly;
static union irq_ctx *softirq_ctx[NR_CPUS] __read_mostly;
static char softirq_stack[NR_CPUS * THREAD_SIZE] __page_aligned_bss;
static char hardirq_stack[NR_CPUS * THREAD_SIZE] __page_aligned_bss;
Then in linux-2.6.30, they were changed to be per-cpu variables:
#ifdef CONFIG_4KSTACKS
/*
* per-CPU IRQ handling contexts (thread information and stack)
*/
union irq_ctx {
struct thread_info tinfo;
u32 stack[THREAD_SIZE/sizeof(u32)];
} __attribute__((aligned(PAGE_SIZE)));
static DEFINE_PER_CPU(union irq_ctx *, hardirq_ctx);
static DEFINE_PER_CPU(union irq_ctx *, softirq_ctx);
static DEFINE_PER_CPU_PAGE_ALIGNED(union irq_ctx, hardirq_stack);
static DEFINE_PER_CPU_PAGE_ALIGNED(union irq_ctx, softirq_stack);
We never ran into the bug in 32-bit x86 RHEL6 (2.6.32-based) kernels because
we did not configure CONFIG_4KSTACKS.
Later on, the CONFIG_4KSTACKS restriction looks like it was removed, but we
stopped supporting 32-bit x86 in RHEL7. And nobody ever reported it from any
other x86 kernel sources.
I also note that in linux-3.15 the data structure name and variables were
changed to:
struct irq_stack {
u32 stack[THREAD_SIZE/sizeof(u32)];
} __aligned(THREAD_SIZE);
DECLARE_PER_CPU(struct irq_stack *, hardirq_stack);
DECLARE_PER_CPU(struct irq_stack *, softirq_stack);
So your patch would need more work for 3.15 and later kernels.
Thanks,
Dave
---
task.c | 28 ++++++++++++++++++++++++----
1 file changed, 24 insertions(+), 4 deletions(-)
diff -puN task.c~ia32-irq-stack-fix task.c
--- crash-64/task.c~ia32-irq-stack-fix 2017-02-08 17:37:33.126367767 +0900
+++ crash-64-hirofumi/task.c 2017-02-08 17:46:06.900703326 +0900
@@ -570,10 +570,20 @@ irqstacks_init(void)
if ((hard_sp = per_cpu_symbol_search("per_cpu__hardirq_ctx"))) {
if ((kt->flags & SMP) && (kt->flags & PER_CPU_OFF)) {
for (i = 0; i < NR_CPUS; i++) {
+ ulong ptr;
+
if (!kt->__per_cpu_offset[i])
continue;
- tt->hardirq_ctx[i] = hard_sp->value +
- kt->__per_cpu_offset[i];
+ ptr = hard_sp->value + kt->__per_cpu_offset[i];
+
+ if (!readmem(ptr, KVADDR, &ptr,
+ sizeof(void *), "hardirq ctx",
+ RETURN_ON_ERROR)) {
+ error(INFO, "cannot read hardirq_ctx[%d] at %lx\n",
+ i, ptr);
+ continue;
+ }
+ tt->hardirq_ctx[i] = ptr;
}
} else
tt->hardirq_ctx[0] = hard_sp->value;
@@ -604,10 +614,20 @@ irqstacks_init(void)
if ((soft_sp = per_cpu_symbol_search("per_cpu__softirq_ctx"))) {
if ((kt->flags & SMP) && (kt->flags & PER_CPU_OFF)) {
for (i = 0; i < NR_CPUS; i++) {
+ ulong ptr;
+
if (!kt->__per_cpu_offset[i])
continue;
- tt->softirq_ctx[i] = soft_sp->value +
- kt->__per_cpu_offset[i];
+ ptr = soft_sp->value + kt->__per_cpu_offset[i];
+
+ if (!readmem(ptr, KVADDR, &ptr,
+ sizeof(void *), "softirq ctx",
+ RETURN_ON_ERROR)) {
+ error(INFO, "cannot read softirq_ctx[%d] at %lx\n",
+ i, ptr);
+ continue;
+ }
+ tt->softirq_ctx[i] = ptr;
}
} else
tt->softirq_ctx[0] = soft_sp->value;
--
OGAWA Hirofumi <hirofumi mail parknet co jp>
9:12 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
Dave Anderson <anderson(a)redhat.com> writes:
but we stopped supporting 32-bit x86 in RHEL7. And nobody ever
reported it from any other x86 kernel sources.
Ah, I see. I didn't know RHEL7 dropped support of x86.
I also note that in linux-3.15 the data structure name and variables
were
changed to:
struct irq_stack {
u32 stack[THREAD_SIZE/sizeof(u32)];
} __aligned(THREAD_SIZE);
DECLARE_PER_CPU(struct irq_stack *, hardirq_stack);
DECLARE_PER_CPU(struct irq_stack *, softirq_stack);
So your patch would need more work for 3.15 and later kernels.
Right. That patch is for "per_cpu + irq_ctx" kernel.
Supporting irq_stack itself would be easy. But current x86 crash doesn't
work for v4.9 (still using irq_stack), this is because current kernel
changed switch_to() stack frame format (need to do similar with x86_64).
Thanks.
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
9:21 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
----- Original Message -----
Dave Anderson <anderson(a)redhat.com> writes:
> but we stopped supporting 32-bit x86 in RHEL7. And nobody ever
> reported it from any other x86 kernel sources.
Ah, I see. I didn't know RHEL7 dropped support of x86.
And because of that, I've pretty much dropped tracking later x86 kernels
in the crash utility, so it's not surprising that there may be other issues.
> I also note that in linux-3.15 the data structure name and variables were
> changed to:
>
> struct irq_stack {
> u32 stack[THREAD_SIZE/sizeof(u32)];
> } __aligned(THREAD_SIZE);
>
> DECLARE_PER_CPU(struct irq_stack *, hardirq_stack);
> DECLARE_PER_CPU(struct irq_stack *, softirq_stack);
>
> So your patch would need more work for 3.15 and later kernels.
Right. That patch is for "per_cpu + irq_ctx" kernel.
Supporting irq_stack itself would be easy. But current x86 crash doesn't
work for v4.9 (still using irq_stack), this is because current kernel
changed switch_to() stack frame format (need to do similar with x86_64).
Our last emails crossed, where I at least got the hardirq_ctx[] array to
set up correctly. But I would appreciate any other work you'd like to
do with the x86 architecture.
Thanks,
Dave
9:17 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
----- Original Message -----
----- Original Message -----
>
> Current code is setting {hard,soft}irq_ctx[] to (irq_ctx **), because
> per_cpu symbol itself is pointer of specified type (irq_ctx *).
>
> But, I wonder how this works in past, the code is expecting
> {hard,soft}_ctx[] are (irq_ctx *). This fixes by deref per_cpu in
> initialization, and set expected pointers.
>
> Tested on i386 v3.10.
>
You are correct -- apparently it never did work.
The reason why I never noticed with RHEL kernels is that up until
linux-2.6.29,
the IRQ stacks were declared as a static NR_CPUS-bounded array:
#ifdef CONFIG_4KSTACKS
/*
* per-CPU IRQ handling contexts (thread information and stack)
*/
union irq_ctx {
struct thread_info tinfo;
u32 stack[THREAD_SIZE/sizeof(u32)];
};
static union irq_ctx *hardirq_ctx[NR_CPUS] __read_mostly;
static union irq_ctx *softirq_ctx[NR_CPUS] __read_mostly;
static char softirq_stack[NR_CPUS * THREAD_SIZE] __page_aligned_bss;
static char hardirq_stack[NR_CPUS * THREAD_SIZE] __page_aligned_bss;
Then in linux-2.6.30, they were changed to be per-cpu variables:
#ifdef CONFIG_4KSTACKS
/*
* per-CPU IRQ handling contexts (thread information and stack)
*/
union irq_ctx {
struct thread_info tinfo;
u32 stack[THREAD_SIZE/sizeof(u32)];
} __attribute__((aligned(PAGE_SIZE)));
static DEFINE_PER_CPU(union irq_ctx *, hardirq_ctx);
static DEFINE_PER_CPU(union irq_ctx *, softirq_ctx);
static DEFINE_PER_CPU_PAGE_ALIGNED(union irq_ctx, hardirq_stack);
static DEFINE_PER_CPU_PAGE_ALIGNED(union irq_ctx, softirq_stack);
We never ran into the bug in 32-bit x86 RHEL6 (2.6.32-based) kernels because
we did not configure CONFIG_4KSTACKS.
Later on, the CONFIG_4KSTACKS restriction looks like it was removed, but we
stopped supporting 32-bit x86 in RHEL7. And nobody ever reported it from any
other x86 kernel sources.
I also note that in linux-3.15 the data structure name and variables were
changed to:
struct irq_stack {
u32 stack[THREAD_SIZE/sizeof(u32)];
} __aligned(THREAD_SIZE);
DECLARE_PER_CPU(struct irq_stack *, hardirq_stack);
DECLARE_PER_CPU(struct irq_stack *, softirq_stack);
So your patch would need more work for 3.15 and later kernels.
Thanks,
Dave
Hi Ogawa,
Please review the attached update to your patch, which adds support for
the irq_ctx-to-irq_stack transition.
Thanks,
Dave
10:34 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
Dave Anderson <anderson(a)redhat.com> writes:
Hi Ogawa,
Hi,
Please review the attached update to your patch, which adds support
for
the irq_ctx-to-irq_stack transition.
- if ((hard_sp = per_cpu_symbol_search("per_cpu__hardirq_ctx"))) {
+ if ((hard_sp = per_cpu_symbol_search("per_cpu__hardirq_ctx")) ||
+ (hard_sp = per_cpu_symbol_search("per_cpu__hardirq_stack"))) {
if ((kt->flags & SMP) && (kt->flags & PER_CPU_OFF)) {
for (i = 0; i < NR_CPUS; i++) {
+ ulong ptr;
+
[...]
Right, this will work, maybe, several stuffs. However hardirq_tasks[] is
missing, so this will not be fully. For now, I don't have dump of other
than v4.9 (stack format was changed), so I'm not sure which commands are
not working (user of hardirq_tasks[]).
Thanks.
+ if (MEMBER_EXISTS("irq_ctx", "tinfo")) {
+ for (i = 0; i < NR_CPUS; i++) {
+ if (!(tt->hardirq_ctx[i]))
+ continue;
- if (!readmem(tt->hardirq_ctx[i], KVADDR, thread_info_buf,
- SIZE(irq_ctx), "hardirq thread_union",
- RETURN_ON_ERROR)) {
- error(INFO, "cannot read hardirq_ctx[%d] at %lx\n",
- i, tt->hardirq_ctx[i]);
- continue;
- }
+ if (!readmem(tt->hardirq_ctx[i], KVADDR, thread_info_buf,
+ SIZE(irq_ctx), "hardirq thread_union",
+ RETURN_ON_ERROR)) {
+ error(INFO, "cannot read hardirq_ctx[%d] at %lx\n",
+ i, tt->hardirq_ctx[i]);
+ continue;
+ }
- tt->hardirq_tasks[i] =
- ULONG(thread_info_buf+OFFSET(thread_info_task));
+ tt->hardirq_tasks[i] =
+ ULONG(thread_info_buf+OFFSET(thread_info_task));
+ }
}
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
7:21 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
----- Original Message -----
Dave Anderson <anderson(a)redhat.com> writes:
> Hi Ogawa,
Hi,
> Please review the attached update to your patch, which adds support for
> the irq_ctx-to-irq_stack transition.
> - if ((hard_sp = per_cpu_symbol_search("per_cpu__hardirq_ctx"))) {
> + if ((hard_sp = per_cpu_symbol_search("per_cpu__hardirq_ctx")) ||
> + (hard_sp = per_cpu_symbol_search("per_cpu__hardirq_stack"))) {
> if ((kt->flags & SMP) && (kt->flags & PER_CPU_OFF)) {
> for (i = 0; i < NR_CPUS; i++) {
> + ulong ptr;
> +
[...]
Right, this will work, maybe, several stuffs. However hardirq_tasks[] is
missing, so this will not be fully. For now, I don't have dump of other
than v4.9 (stack format was changed), so I'm not sure which commands are
not working (user of hardirq_tasks[]).
I haven't looked at the recent kernel to check out how the linkage is done
without the temporary thread_info at the bottom of the hard/soft irq stacks,
but looking at a live 4.9 system, it looks like the first word at the base of
the hard and soft irq stack now contains a pointer into the stack of the
last caller? If that's true, then stkptr_to_task() could be used to
populate the hardirq_tasks[] and softirq_tasks[] -- but the initialization
would have to be delayed until near the end of task_init().
Dave
Thanks.
> + if (MEMBER_EXISTS("irq_ctx", "tinfo")) {
> + for (i = 0; i < NR_CPUS; i++) {
> + if (!(tt->hardirq_ctx[i]))
> + continue;
>
> - if (!readmem(tt->hardirq_ctx[i], KVADDR, thread_info_buf,
> - SIZE(irq_ctx), "hardirq thread_union",
> - RETURN_ON_ERROR)) {
> - error(INFO, "cannot read hardirq_ctx[%d] at %lx\n",
> - i, tt->hardirq_ctx[i]);
> - continue;
> - }
> + if (!readmem(tt->hardirq_ctx[i], KVADDR, thread_info_buf,
> + SIZE(irq_ctx), "hardirq thread_union",
> + RETURN_ON_ERROR)) {
> + error(INFO, "cannot read hardirq_ctx[%d] at %lx\n",
> + i, tt->hardirq_ctx[i]);
> + continue;
> + }
>
> - tt->hardirq_tasks[i] =
> - ULONG(thread_info_buf+OFFSET(thread_info_task));
> + tt->hardirq_tasks[i] =
> + ULONG(thread_info_buf+OFFSET(thread_info_task));
> + }
> }
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
2:17 a.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
Dave Anderson <anderson(a)redhat.com> writes:
> > Please review the attached update to your patch, which adds
support for
> > the irq_ctx-to-irq_stack transition.
> > - if ((hard_sp = per_cpu_symbol_search("per_cpu__hardirq_ctx"))) {
> > + if ((hard_sp = per_cpu_symbol_search("per_cpu__hardirq_ctx")) ||
> > + (hard_sp = per_cpu_symbol_search("per_cpu__hardirq_stack"))) {
> > if ((kt->flags & SMP) && (kt->flags & PER_CPU_OFF)) {
> > for (i = 0; i < NR_CPUS; i++) {
> > + ulong ptr;
> > +
>
> [...]
>
> Right, this will work, maybe, several stuffs. However hardirq_tasks[] is
> missing, so this will not be fully. For now, I don't have dump of other
> than v4.9 (stack format was changed), so I'm not sure which commands are
> not working (user of hardirq_tasks[]).
I haven't looked at the recent kernel to check out how the linkage is done
without the temporary thread_info at the bottom of the hard/soft irq stacks,
but looking at a live 4.9 system, it looks like the first word at the base of
the hard and soft irq stack now contains a pointer into the stack of the
last caller? If that's true, then stkptr_to_task() could be used to
populate the hardirq_tasks[] and softirq_tasks[] -- but the initialization
would have to be delayed until near the end of task_init().
Right. At least on v4.9, with quick check, looks like the x86 and x86_64
has different (x86 is for hardirq/softirq, x86_64 is for
hardirq/exception).
linux/arch/x86/kernel/dumpstack_{32,64}.c is how get previous stack from
sp.
x86:
in_hardirq_stack(stack, info)
info->next_sp = (unsigned long *)*begin;
in_softirq_stack(stack, info)
info->next_sp = (unsigned long *)*begin;
both is same as you said (at start of stack pages).
x86_64
in_exception_stack(stack, info)
info->next_sp = (unsigned long *)regs->sp;
in_irq_stack(stack, info)
info->next_sp = (unsigned long *)*(end - 1);
exception has pt_regs at top of stack (at end of stack pages)
irq has previous sp at top of stack (at end of stack pages)
But I'm still not checking when switched to this format, and how
consolidate/coding those in crash.
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
10:50 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
Hi Ogawa,
I've checked in a patch for storing the correct 32-bit x86 hard/soft IRQ stack
addresses:
https://github.com/crash-utility/crash/commit/5959759058e53035fb26b9f2b59...
Fix for Linux 3.15 and later 32-bit X86 kernels containing kernel
commit 198d208df4371734ac4728f69cb585c284d20a15, titled "x86: Keep
thread_info on thread stack in x86_32". Without the patch, incorrect
addresses of each per-cpu hardirq_stack and softirq_stack were saved
for usage by the "bt" command.
(hirofumi(a)mail.parknet.co.jp, anderson(a)redhat.com)
I don't have a vmcore that I can test whether the stack transition still works,
but at least it's got the correct addresses, and I believe I have the correct
values set up in restore_stack().
Also, there are also several other 32-bit x86 backtrace-related patches that I've
checked in the last few days for other issues that you may have run into:
commit 222e784fb2d16f28ab702a7f9a1889e0b18ca152
Author: Dave Anderson <anderson(a)redhat.com>
Date: Fri Feb 10 15:07:58 2017 -0500
Fix for 32-bit X86 kernels configured with CONFIG_RANDOMIZE_BASE.
Without the patch, an invalid kernel PAGE_OFFSET value is calculated
and as a result the session fails during session initialization just
after the patching of the gdb minimal_symbol values message, showing
the warning message "WARNING: cannot read linux_banner string",
followed by "crash: /vmlinux and /dev/crash do not match!". This
patch also adds a new "--machdep page_offset=<value>" option that
can be used if the CONFIG_PAGE_OFFSET value is not the default
address of 0xc0000000.
(anderson(a)redhat.com)
commit 69b577e423c1adb5f06d19bccc790063a500aac4
Author: Dave Anderson <anderson(a)redhat.com>
Date: Tue Feb 14 14:25:37 2017 -0500
Fix for the "bt" command on Linux 4.9 and later 32-bit X86 kernels
containing kernel commit 0100301bfdf56a2a370c7157b5ab0fbf9313e1cd,
subject "sched/x86: Rewrite the switch_to() code". Without the
patch, backtraces for inactive (sleeping) tasks fail with the message
"bt: invalid structure member offset: task_struct_thread_eip".
(anderson(a)redhat.com)
commit ec1a9b967da19acaa0ec6b6a904d16267ba6a409
Author: Dave Anderson <anderson(a)redhat.com>
Date: Tue Feb 14 15:23:01 2017 -0500
Fix for a "[-Wmisleading-indentation]" compiler warning and the
associated bug that is generated by lkcd_x86_trace.c when building
32-bit X86 with "make warn" with gcc-6.3.1.
(anderson(a)redhat.com)
commit 63dd57685e5e897ae343a213b58cb1e6ae4ed282
Author: Dave Anderson <anderson(a)redhat.com>
Date: Tue Feb 14 16:23:56 2017 -0500
Fix for an invalid "bt" warning on a 32-bit X86 idle (swapper) task.
Without the patch, the backtrace displays the "cannot resolve stack
trace" warning, dumps the backtrace, and then the text symbols:
crash> bt
PID: 0 TASK: f0962180 CPU: 6 COMMAND: "swapper/6"
bt: cannot resolve stack trace:
#0 [f095ff1c] __schedule at c0b6ef8d
#1 [f095ff58] schedule at c0b6f4a9
#2 [f095ff64] schedule_preempt_disabled at c0b6f728
#3 [f095ff6c] cpu_startup_entry at c04b0310
#4 [f095ff94] start_secondary at c04468c0
bt: text symbols on stack:
[f095ff1c] __schedule at c0b6ef8d
[f095ff58] schedule at c0b6f4ae
[f095ff64] schedule_preempt_disabled at c0b6f72d
[f095ff6c] cpu_startup_entry at c04b0315
[f095ff94] start_secondary at c04468c5
crash>
The backtrace shown is actually correct.
(anderson(a)redhat.com)
commit 8323490fd962ba1b6aa745deea2ec817d837862d
Author: Dave Anderson <anderson(a)redhat.com>
Date: Wed Feb 15 15:04:07 2017 -0500
Another fix for a similar "bt: cannot resolve stack trace" warning
on a 32-bit X86 idle/swapper task, but when running on cpu 0.
(anderson(a)redhat.com)
commit 345da6dc904cb88a3abfae36a990150a2036b95d
Author: Dave Anderson <anderson(a)redhat.com>
Date: Wed Feb 15 15:39:13 2017 -0500
Remove two one-time warning messages that are displayed when running
the "bt" command on Linux 4.2 and later 32-bit X86 kernels. Without
the patch, the first "bt" command that is executed will be preceded
by "bt: WARNING: "system_call" symbol does not exist", followed
by
"bt: WARNING: neither "ret_from_sys_call" nor
"syscall_badsys"
symbols exist".
(anderson(a)redhat.com)
It's been quite some time since I've done any 32-bit x86 maintenance,
and any help/time you can afford would be appreciated.
Thanks,
Dave
----- Original Message -----
Dave Anderson <anderson(a)redhat.com> writes:
>> > Please review the attached update to your patch, which adds support for
>> > the irq_ctx-to-irq_stack transition.
>> > - if ((hard_sp = per_cpu_symbol_search("per_cpu__hardirq_ctx")))
{
>> > + if ((hard_sp = per_cpu_symbol_search("per_cpu__hardirq_ctx"))
||
>> > + (hard_sp =
per_cpu_symbol_search("per_cpu__hardirq_stack"))) {
>> > if ((kt->flags & SMP) && (kt->flags &
PER_CPU_OFF)) {
>> > for (i = 0; i < NR_CPUS; i++) {
>> > + ulong ptr;
>> > +
>>
>> [...]
>>
>> Right, this will work, maybe, several stuffs. However hardirq_tasks[] is
>> missing, so this will not be fully. For now, I don't have dump of other
>> than v4.9 (stack format was changed), so I'm not sure which commands are
>> not working (user of hardirq_tasks[]).
>
> I haven't looked at the recent kernel to check out how the linkage is done
> without the temporary thread_info at the bottom of the hard/soft irq
> stacks,
> but looking at a live 4.9 system, it looks like the first word at the base
> of
> the hard and soft irq stack now contains a pointer into the stack of the
> last caller? If that's true, then stkptr_to_task() could be used to
> populate the hardirq_tasks[] and softirq_tasks[] -- but the initialization
> would have to be delayed until near the end of task_init().
Right. At least on v4.9, with quick check, looks like the x86 and x86_64
has different (x86 is for hardirq/softirq, x86_64 is for
hardirq/exception).
linux/arch/x86/kernel/dumpstack_{32,64}.c is how get previous stack from
sp.
x86:
in_hardirq_stack(stack, info)
info->next_sp = (unsigned long *)*begin;
in_softirq_stack(stack, info)
info->next_sp = (unsigned long *)*begin;
both is same as you said (at start of stack pages).
x86_64
in_exception_stack(stack, info)
info->next_sp = (unsigned long *)regs->sp;
in_irq_stack(stack, info)
info->next_sp = (unsigned long *)*(end - 1);
exception has pt_regs at top of stack (at end of stack pages)
irq has previous sp at top of stack (at end of stack pages)
But I'm still not checking when switched to this format, and how
consolidate/coding those in crash.
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
6:49 a.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
Dave Anderson <anderson(a)redhat.com> writes:
Hi Ogawa,
Hi,
It's been quite some time since I've done any 32-bit x86
maintenance,
and any help/time you can afford would be appreciated.
I've tried some on latest git with i386 linux-v4.9. And the results are
seems not working yet. I'm not checking the detail of crash failure yet,
sorry.
[In latest few commits, using "handle_IRQ" symbol though, at least v4.9
kernel doesn't have "handle_IRQ". So I changed from "handle_IRQ"
to
"do_IRQ".]
The following is some cases of "kernel stack dump" and "bt".
crash> log
[ 2.493782] ------------[ cut here ]------------
[ 2.493783] kernel BUG at
/devel/linux/works/tux3/linux-tux3/drivers/ata/libata-sff.c:1573!
[ 2.493784] invalid opcode: 0000 [#1] PREEMPT SMP
[ 2.493785] Modules linked in:
[ 2.493789] CPU: 3 PID: 45 Comm: kworker/3:1 Not tainted
4.9.0-03202-g815a856e470a-dirty #15
[ 2.493789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014
[ 2.493799] Workqueue: ata_sff ata_sff_pio_task
[ 2.493800] task: f57d3a00 task.stack: f4cd8000
[ 2.493803] EIP: 0060:[<c172b046>] EFLAGS: 00210002 CPU: 3
[ 2.493804] EIP is at __ata_sff_port_intr+0x146/0x150
[ 2.493805] EAX: f57d3a00 EBX: f3a98000 ECX: 00000041 EDX: 0000c06a
[ 2.493806] ESI: f3a9954c EDI: 00000000 EBP: f56d3f28 ESP: f56d3f14
[ 2.493806] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 2.493807] CR0: 80050033 CR2: ffffffff CR3: 020bb000 CR4: 000006d0
[ 2.493809] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 2.493809] DR6: fffe0ff0 DR7: 00000400
[ 2.493809] Stack:
[ 2.493812] 00000000 41000000 f3a98000 f3a9954c 00000000 f56d3f4c c172b338 00000000
[ 2.493814] 00000001 00000000 c172b3f7 00001458 00000001 f3fc047c f56d3f80 c172b57b
[ 2.493817] 00200086 f3fc047c 00000000 00dcd658 00000001 00000001 f3a9954c 00000000
[ 2.493817] Call Trace:
[ 2.493820] <IRQ>
[ 2.493820] [<c172b338>] ata_bmdma_port_intr+0x28/0xd0
[ 2.493822] [<c172b3f7>] ? ata_bmdma_interrupt+0x17/0x200
[ 2.493824] [<c172b57b>] ata_bmdma_interrupt+0x19b/0x200
[ 2.493829] [<c10cf378>] __handle_irq_event_percpu+0x38/0x470
[ 2.493831] [<c10cf827>] ? handle_irq_event+0x27/0x50
[ 2.493832] [<c10cf7ce>] handle_irq_event_percpu+0x1e/0x50
[ 2.493833] [<c10cf82e>] handle_irq_event+0x2e/0x50
[ 2.493834] [<c10d295d>] handle_edge_irq+0xad/0x130
[ 2.493835] [<c10d28b0>] ? handle_level_irq+0x110/0x110
[ 2.493837] [<c101cfbd>] handle_irq+0x9d/0xd0
[ 2.493838] <EOI>
[ 2.493841] <IRQ>
[ 2.493841] [<c1a0630e>] do_IRQ+0x5e/0x130
[ 2.493842] [<c1a067b0>] ? __irqentry_text_end+0x3/0x3
[ 2.493844] [<c1a059b1>] common_interrupt+0x31/0x38
[ 2.493845] [<c1a067b0>] ? __irqentry_text_end+0x3/0x3
[ 2.493846] [<c1a06835>] ? __do_softirq+0x85/0x565
[ 2.493848] [<c1a067b0>] ? __irqentry_text_end+0x3/0x3
[ 2.493849] [<c101cf12>] do_softirq_own_stack+0x32/0x40
[ 2.493852] <EOI>
[ 2.493852] [<c10631d9>] irq_exit+0xa9/0xc0
[ 2.493853] [<c1a06418>] smp_apic_timer_interrupt+0x38/0x50
[ 2.493853] [<c1a05c86>] apic_timer_interrupt+0x32/0x38
[ 2.493855] [<c1a04819>] ? _raw_spin_unlock_irq+0x29/0x50
[ 2.493857] [<c172ade8>] ata_sff_pio_task+0x78/0x190
[ 2.493858] [<c1078a37>] process_one_work+0x207/0x700
[ 2.493860] [<c107899b>] ? process_one_work+0x16b/0x700
[ 2.493861] [<c1078f71>] worker_thread+0x41/0x470
[ 2.493862] [<c1078f30>] ? process_one_work+0x700/0x700
[ 2.493864] [<c107ed46>] kthread+0xa6/0xc0
[ 2.493866] [<c107eca0>] ? kthread_park+0x50/0x50
[ 2.493867] [<c107eca0>] ? kthread_park+0x50/0x50
[ 2.493868] [<c1a05177>] ret_from_fork+0x1b/0x28
[ 2.493889] Code: c0 74 1a 83 8e 80 00 00 00 02 c7 83 20 2d 00 00 04 00 00 00 e9 39 ff
ff ff 8d 74 26 00 83 83 94 29 00 00 01 31 f6 e9 67 ff ff ff <0f> 0b 90 8d b4 26 00
00 00 00 55 89 e5 3e 8d 74 26 00 31 c9 e8
[ 2.493891] EIP: [<c172b046>]
[ 2.493892] __ata_sff_port_intr+0x146/0x150
[ 2.493892] SS:ESP 0068:f56d3f14
[ 2.493895] ---[ end trace 362d4d3f42fbe108 ]---
[ 2.493896] Kernel panic - not syncing: Fatal exception in interrupt
[ 2.494103] Kernel Offset: disabled
[ 2.509246] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
crash> bt
PID: 0 TASK: c1da8b00 CPU: 0 COMMAND: "swapper/0"
#0 [c1da1f60] __schedule at c19fe305
#1 [c1da1fa0] schedule at c19febb3
#2 [c1da1fac] schedule_preempt_disabled at c19ff0a2
#3 [c1da1fb4] cpu_startup_entry at c10a9580
crash> bt 45
PID: 45 TASK: f57d3a00 CPU: 3 COMMAND: "kworker/3:1"
bt: cannot resolve stack trace:
bt: Task in user space -- no backtrace
In above case, looks like failed to detect panic cpu, and "bt 45" also
not working.
crash> log
[ 2.406255] ------------[ cut here ]------------
[ 2.406391] kernel BUG at
/devel/linux/works/tux3/linux-tux3/drivers/ata/libata-sff.c:1573!
[ 2.406565] invalid opcode: 0000 [#1] PREEMPT SMP
[ 2.407285] Modules linked in:
[ 2.407442] CPU: 1 PID: 31 Comm: kworker/1:1 Not tainted
4.9.0-03202-g815a856e470a-dirty #15
[ 2.407643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014
[ 2.407986] Workqueue: ata_sff ata_sff_pio_task
[ 2.408145] task: f505e080 task.stack: f5060000
[ 2.408296] EIP: 0060:[<c172b046>] EFLAGS: 00210006 CPU: 1
[ 2.408441] EIP is at __ata_sff_port_intr+0x146/0x150
[ 2.408640] EAX: f505e080 EBX: f3da0000 ECX: 00000041 EDX: 0000c06a
[ 2.408865] ESI: f3da154c EDI: 00000000 EBP: f5a73f28 ESP: f5a73f14
[ 2.409077] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 2.409266] CR0: 80050033 CR2: ffffffff CR3: 020bb000 CR4: 000006d0
[ 2.409464] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 2.409621] DR6: fffe0ff0 DR7: 00000400
[ 2.409772] Stack:
[ 2.409952] 00000000 41000000 f3da0000 f3da154c 00000000 f5a73f4c c172b338 00000000
[ 2.410374] 00000001 00000000 c172b3f7 00001458 00000001 f42b532c f5a73f80 c172b57b
[ 2.410775] 00200086 f42b532c 00000000 00dcd658 00000001 00000001 f3da154c 00000000
[ 2.411140] Call Trace:
[ 2.411252] <IRQ>
[ 2.411287] [<c172b338>] ata_bmdma_port_intr+0x28/0xd0
[ 2.411453] [<c172b3f7>] ? ata_bmdma_interrupt+0x17/0x200
[ 2.411597] [<c172b57b>] ata_bmdma_interrupt+0x19b/0x200
[ 2.411747] [<c10cf378>] __handle_irq_event_percpu+0x38/0x470
[ 2.411925] [<c10cf827>] ? handle_irq_event+0x27/0x50
[ 2.412108] [<c10cf7ce>] handle_irq_event_percpu+0x1e/0x50
[ 2.412313] [<c10cf82e>] handle_irq_event+0x2e/0x50
[ 2.412511] [<c10d295d>] handle_edge_irq+0xad/0x130
[ 2.412713] [<c10d28b0>] ? handle_level_irq+0x110/0x110
[ 2.413029] [<c101cfbd>] handle_irq+0x9d/0xd0
[ 2.413196] <EOI>
[ 2.413237] [<c1a0630e>] do_IRQ+0x5e/0x130
[ 2.413397] [<c1a059b1>] common_interrupt+0x31/0x38
[ 2.413536] [<c1a04819>] ? _raw_spin_unlock_irq+0x29/0x50
[ 2.413691] [<c172ade8>] ata_sff_pio_task+0x78/0x190
[ 2.413832] [<c1078a37>] process_one_work+0x207/0x700
[ 2.419333] [<c107899b>] ? process_one_work+0x16b/0x700
[ 2.419477] [<c1078f71>] worker_thread+0x41/0x470
[ 2.419615] [<c1078f30>] ? process_one_work+0x700/0x700
[ 2.419758] [<c107ed46>] kthread+0xa6/0xc0
[ 2.419887] [<c107eca0>] ? kthread_park+0x50/0x50
[ 2.420022] [<c107eca0>] ? kthread_park+0x50/0x50
[ 2.420159] [<c1a05177>] ret_from_fork+0x1b/0x28
[ 2.420295] Code: c0 74 1a 83 8e 80 00 00 00 02 c7 83 20 2d 00 00 04 00 00 00 e9 39 ff
ff ff 8d 74 26 00 83 83 94 29 00 00 01 31 f6 e9 67 ff ff ff <0f> 0b 90 8d b4 26 00
00 00 00 55 89 e5 3e 8d 74 26 00 31 c9 e8
[ 2.422387] EIP: [<c172b046>]
[ 2.422431] __ata_sff_port_intr+0x146/0x150
[ 2.422585] SS:ESP 0068:f5a73f14
[ 2.422711] ---[ end trace 8465c4f061d2e6ad ]---
[ 2.422845] Kernel panic - not syncing: Fatal exception in interrupt
[ 2.423137] Kernel Offset: disabled
[ 2.423264] ---[ end Kernel panic - not syncing: Fatal exception in interrupt
crash> bt
PID: 31 TASK: f505e080 CPU: 1 COMMAND: "kworker/1:1"
#0 [f5a73de8] die at c101dc04
#1 [f5a73dfc] notify_die at c10809fc
#2 [f5a73e24] do_error_trap at c101ab1c
#3 [f5a73ec4] do_invalid_op at c101b2cf
#4 [f5a73ed4] error_code (via invalid_op) at c1a060fd
EAX: f505e080 EBX: f3da0000 ECX: 00000041 EDX: 0000c06a EBP: f5a73f28
DS: 007b ESI: f3da154c ES: 007b EDI: 00000000 GS: ffffb2b0
CS: 0060 EIP: c172b046 ERR: ffffffff EFLAGS: 00210006
#5 [f5a73f08] __ata_sff_port_intr at c172b046
#6 [f5a73f2c] ata_bmdma_port_intr at c172b333
#7 [f5a73f50] ata_bmdma_interrupt at c172b576
#8 [f5a73f84] __handle_irq_event_percpu at c10cf376
#9 [f5a73fc0] handle_irq_event_percpu at c10cf7c9
#10 [f5a73fd4] handle_irq_event at c10cf829
#11 [f5a73fe8] handle_edge_irq at c10d2958
--- <hard IRQ> ---
bt: cannot resolve stack trace:
bt: text symbols on stack:
Thanks.
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
3:26 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
----- Original Message -----
Dave Anderson <anderson(a)redhat.com> writes:
> Hi Ogawa,
Hi,
> It's been quite some time since I've done any 32-bit x86 maintenance,
> and any help/time you can afford would be appreciated.
I've tried some on latest git with i386 linux-v4.9. And the results are
seems not working yet. I'm not checking the detail of crash failure yet,
sorry.
[In latest few commits, using "handle_IRQ" symbol though, at least v4.9
kernel doesn't have "handle_IRQ". So I changed from "handle_IRQ"
to
"do_IRQ".]
Actually, it supposed to be the function that was running on the process stack
prior to the transition to the IRQ stack. So it should be "handle_irq" here
in arch/x86/kernel/irq_32.c:
bool handle_irq(struct irq_desc *desc, struct pt_regs *regs)
{
int overflow = check_stack_overflow();
if (IS_ERR_OR_NULL(desc))
return false;
if (user_mode(regs) || !execute_on_irq_stack(overflow, desc)) {
if (unlikely(overflow))
print_stack_overflow();
generic_handle_irq_desc(desc);
}
return true;
}
The following is some cases of "kernel stack dump" and "bt".
crash> log
[ 2.493782] ------------[ cut here ]------------
[ 2.493783] kernel BUG at
/devel/linux/works/tux3/linux-tux3/drivers/ata/libata-sff.c:1573!
[ 2.493784] invalid opcode: 0000 [#1] PREEMPT SMP
[ 2.493785] Modules linked in:
[ 2.493789] CPU: 3 PID: 45 Comm: kworker/3:1 Not tainted
4.9.0-03202-g815a856e470a-dirty #15
[ 2.493789] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014
[ 2.493799] Workqueue: ata_sff ata_sff_pio_task
[ 2.493800] task: f57d3a00 task.stack: f4cd8000
[ 2.493803] EIP: 0060:[<c172b046>] EFLAGS: 00210002 CPU: 3
[ 2.493804] EIP is at __ata_sff_port_intr+0x146/0x150
[ 2.493805] EAX: f57d3a00 EBX: f3a98000 ECX: 00000041 EDX: 0000c06a
[ 2.493806] ESI: f3a9954c EDI: 00000000 EBP: f56d3f28 ESP: f56d3f14
[ 2.493806] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 2.493807] CR0: 80050033 CR2: ffffffff CR3: 020bb000 CR4: 000006d0
[ 2.493809] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 2.493809] DR6: fffe0ff0 DR7: 00000400
[ 2.493809] Stack:
[ 2.493812] 00000000 41000000 f3a98000 f3a9954c 00000000 f56d3f4c
c172b338 00000000
[ 2.493814] 00000001 00000000 c172b3f7 00001458 00000001 f3fc047c
f56d3f80 c172b57b
[ 2.493817] 00200086 f3fc047c 00000000 00dcd658 00000001 00000001
f3a9954c 00000000
[ 2.493817] Call Trace:
[ 2.493820] <IRQ>
[ 2.493820] [<c172b338>] ata_bmdma_port_intr+0x28/0xd0
[ 2.493822] [<c172b3f7>] ? ata_bmdma_interrupt+0x17/0x200
[ 2.493824] [<c172b57b>] ata_bmdma_interrupt+0x19b/0x200
[ 2.493829] [<c10cf378>] __handle_irq_event_percpu+0x38/0x470
[ 2.493831] [<c10cf827>] ? handle_irq_event+0x27/0x50
[ 2.493832] [<c10cf7ce>] handle_irq_event_percpu+0x1e/0x50
[ 2.493833] [<c10cf82e>] handle_irq_event+0x2e/0x50
[ 2.493834] [<c10d295d>] handle_edge_irq+0xad/0x130
[ 2.493835] [<c10d28b0>] ? handle_level_irq+0x110/0x110
[ 2.493837] [<c101cfbd>] handle_irq+0x9d/0xd0
[ 2.493838] <EOI>
[ 2.493841] <IRQ>
[ 2.493841] [<c1a0630e>] do_IRQ+0x5e/0x130
[ 2.493842] [<c1a067b0>] ? __irqentry_text_end+0x3/0x3
[ 2.493844] [<c1a059b1>] common_interrupt+0x31/0x38
[ 2.493845] [<c1a067b0>] ? __irqentry_text_end+0x3/0x3
[ 2.493846] [<c1a06835>] ? __do_softirq+0x85/0x565
[ 2.493848] [<c1a067b0>] ? __irqentry_text_end+0x3/0x3
[ 2.493849] [<c101cf12>] do_softirq_own_stack+0x32/0x40
[ 2.493852] <EOI>
[ 2.493852] [<c10631d9>] irq_exit+0xa9/0xc0
[ 2.493853] [<c1a06418>] smp_apic_timer_interrupt+0x38/0x50
[ 2.493853] [<c1a05c86>] apic_timer_interrupt+0x32/0x38
[ 2.493855] [<c1a04819>] ? _raw_spin_unlock_irq+0x29/0x50
[ 2.493857] [<c172ade8>] ata_sff_pio_task+0x78/0x190
[ 2.493858] [<c1078a37>] process_one_work+0x207/0x700
[ 2.493860] [<c107899b>] ? process_one_work+0x16b/0x700
[ 2.493861] [<c1078f71>] worker_thread+0x41/0x470
[ 2.493862] [<c1078f30>] ? process_one_work+0x700/0x700
[ 2.493864] [<c107ed46>] kthread+0xa6/0xc0
[ 2.493866] [<c107eca0>] ? kthread_park+0x50/0x50
[ 2.493867] [<c107eca0>] ? kthread_park+0x50/0x50
[ 2.493868] [<c1a05177>] ret_from_fork+0x1b/0x28
[ 2.493889] Code: c0 74 1a 83 8e 80 00 00 00 02 c7 83 20 2d 00 00 04 00 00
00 e9 39 ff ff ff 8d 74 26 00 83 83 94 29 00 00 01 31 f6 e9 67 ff ff ff <0f>
0b 90 8d b4 26 00 00 00 00 55 89 e5 3e 8d 74 26 00 31 c9 e8
[ 2.493891] EIP: [<c172b046>]
[ 2.493892] __ata_sff_port_intr+0x146/0x150
[ 2.493892] SS:ESP 0068:f56d3f14
[ 2.493895] ---[ end trace 362d4d3f42fbe108 ]---
[ 2.493896] Kernel panic - not syncing: Fatal exception in interrupt
[ 2.494103] Kernel Offset: disabled
[ 2.509246] ---[ end Kernel panic - not syncing: Fatal exception in
interrupt
crash> bt
PID: 0 TASK: c1da8b00 CPU: 0 COMMAND: "swapper/0"
#0 [c1da1f60] __schedule at c19fe305
#1 [c1da1fa0] schedule at c19febb3
#2 [c1da1fac] schedule_preempt_disabled at c19ff0a2
#3 [c1da1fb4] cpu_startup_entry at c10a9580
crash> bt 45
PID: 45 TASK: f57d3a00 CPU: 3 COMMAND: "kworker/3:1"
bt: cannot resolve stack trace:
bt: Task in user space -- no backtrace
In above case, looks like failed to detect panic cpu, and "bt 45" also
not working.
crash> log
[ 2.406255] ------------[ cut here ]------------
[ 2.406391] kernel BUG at
/devel/linux/works/tux3/linux-tux3/drivers/ata/libata-sff.c:1573!
[ 2.406565] invalid opcode: 0000 [#1] PREEMPT SMP
[ 2.407285] Modules linked in:
[ 2.407442] CPU: 1 PID: 31 Comm: kworker/1:1 Not tainted
4.9.0-03202-g815a856e470a-dirty #15
[ 2.407643] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014
[ 2.407986] Workqueue: ata_sff ata_sff_pio_task
[ 2.408145] task: f505e080 task.stack: f5060000
[ 2.408296] EIP: 0060:[<c172b046>] EFLAGS: 00210006 CPU: 1
[ 2.408441] EIP is at __ata_sff_port_intr+0x146/0x150
[ 2.408640] EAX: f505e080 EBX: f3da0000 ECX: 00000041 EDX: 0000c06a
[ 2.408865] ESI: f3da154c EDI: 00000000 EBP: f5a73f28 ESP: f5a73f14
[ 2.409077] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[ 2.409266] CR0: 80050033 CR2: ffffffff CR3: 020bb000 CR4: 000006d0
[ 2.409464] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[ 2.409621] DR6: fffe0ff0 DR7: 00000400
[ 2.409772] Stack:
[ 2.409952] 00000000 41000000 f3da0000 f3da154c 00000000 f5a73f4c
c172b338 00000000
[ 2.410374] 00000001 00000000 c172b3f7 00001458 00000001 f42b532c
f5a73f80 c172b57b
[ 2.410775] 00200086 f42b532c 00000000 00dcd658 00000001 00000001
f3da154c 00000000
[ 2.411140] Call Trace:
[ 2.411252] <IRQ>
[ 2.411287] [<c172b338>] ata_bmdma_port_intr+0x28/0xd0
[ 2.411453] [<c172b3f7>] ? ata_bmdma_interrupt+0x17/0x200
[ 2.411597] [<c172b57b>] ata_bmdma_interrupt+0x19b/0x200
[ 2.411747] [<c10cf378>] __handle_irq_event_percpu+0x38/0x470
[ 2.411925] [<c10cf827>] ? handle_irq_event+0x27/0x50
[ 2.412108] [<c10cf7ce>] handle_irq_event_percpu+0x1e/0x50
[ 2.412313] [<c10cf82e>] handle_irq_event+0x2e/0x50
[ 2.412511] [<c10d295d>] handle_edge_irq+0xad/0x130
[ 2.412713] [<c10d28b0>] ? handle_level_irq+0x110/0x110
[ 2.413029] [<c101cfbd>] handle_irq+0x9d/0xd0
[ 2.413196] <EOI>
[ 2.413237] [<c1a0630e>] do_IRQ+0x5e/0x130
[ 2.413397] [<c1a059b1>] common_interrupt+0x31/0x38
[ 2.413536] [<c1a04819>] ? _raw_spin_unlock_irq+0x29/0x50
[ 2.413691] [<c172ade8>] ata_sff_pio_task+0x78/0x190
[ 2.413832] [<c1078a37>] process_one_work+0x207/0x700
[ 2.419333] [<c107899b>] ? process_one_work+0x16b/0x700
[ 2.419477] [<c1078f71>] worker_thread+0x41/0x470
[ 2.419615] [<c1078f30>] ? process_one_work+0x700/0x700
[ 2.419758] [<c107ed46>] kthread+0xa6/0xc0
[ 2.419887] [<c107eca0>] ? kthread_park+0x50/0x50
[ 2.420022] [<c107eca0>] ? kthread_park+0x50/0x50
[ 2.420159] [<c1a05177>] ret_from_fork+0x1b/0x28
[ 2.420295] Code: c0 74 1a 83 8e 80 00 00 00 02 c7 83 20 2d 00 00 04 00 00
00 e9 39 ff ff ff 8d 74 26 00 83 83 94 29 00 00 01 31 f6 e9 67 ff ff ff <0f>
0b 90 8d b4 26 00 00 00 00 55 89 e5 3e 8d 74 26 00 31 c9 e8
[ 2.422387] EIP: [<c172b046>]
[ 2.422431] __ata_sff_port_intr+0x146/0x150
[ 2.422585] SS:ESP 0068:f5a73f14
[ 2.422711] ---[ end trace 8465c4f061d2e6ad ]---
[ 2.422845] Kernel panic - not syncing: Fatal exception in interrupt
[ 2.423137] Kernel Offset: disabled
[ 2.423264] ---[ end Kernel panic - not syncing: Fatal exception in
interrupt
crash> bt
PID: 31 TASK: f505e080 CPU: 1 COMMAND: "kworker/1:1"
#0 [f5a73de8] die at c101dc04
#1 [f5a73dfc] notify_die at c10809fc
#2 [f5a73e24] do_error_trap at c101ab1c
#3 [f5a73ec4] do_invalid_op at c101b2cf
#4 [f5a73ed4] error_code (via invalid_op) at c1a060fd
EAX: f505e080 EBX: f3da0000 ECX: 00000041 EDX: 0000c06a EBP: f5a73f28
DS: 007b ESI: f3da154c ES: 007b EDI: 00000000 GS: ffffb2b0
CS: 0060 EIP: c172b046 ERR: ffffffff EFLAGS: 00210006
#5 [f5a73f08] __ata_sff_port_intr at c172b046
#6 [f5a73f2c] ata_bmdma_port_intr at c172b333
#7 [f5a73f50] ata_bmdma_interrupt at c172b576
#8 [f5a73f84] __handle_irq_event_percpu at c10cf376
#9 [f5a73fc0] handle_irq_event_percpu at c10cf7c9
#10 [f5a73fd4] handle_irq_event at c10cf829
#11 [f5a73fe8] handle_edge_irq at c10d2958
--- <hard IRQ> ---
bt: cannot resolve stack trace:
bt: text symbols on stack:
Thanks.
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
OK thanks. Please send patches as required.
Dave
3:39 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
Dave Anderson <anderson(a)redhat.com> writes:
Actually, it supposed to be the function that was running on the
process stack
prior to the transition to the IRQ stack. So it should be "handle_irq" here
in arch/x86/kernel/irq_32.c:
I see.
[...]
> crash> bt
> PID: 0 TASK: c1da8b00 CPU: 0 COMMAND: "swapper/0"
> #0 [c1da1f60] __schedule at c19fe305
> #1 [c1da1fa0] schedule at c19febb3
> #2 [c1da1fac] schedule_preempt_disabled at c19ff0a2
> #3 [c1da1fb4] cpu_startup_entry at c10a9580
> crash> bt 45
> PID: 45 TASK: f57d3a00 CPU: 3 COMMAND: "kworker/3:1"
> bt: cannot resolve stack trace:
> bt: Task in user space -- no backtrace
>
> In above case, looks like failed to detect panic cpu, and "bt 45" also
> not working.
crash> bt 45
PID: 45 TASK: f57d3a00 CPU: 3 COMMAND: "kworker/3:1"
bt: cannot resolve stack trace:
bt: Task in user space -- no backtrace
[...]
> crash> bt
> PID: 31 TASK: f505e080 CPU: 1 COMMAND: "kworker/1:1"
> #0 [f5a73de8] die at c101dc04
> #1 [f5a73dfc] notify_die at c10809fc
> #2 [f5a73e24] do_error_trap at c101ab1c
> #3 [f5a73ec4] do_invalid_op at c101b2cf
> #4 [f5a73ed4] error_code (via invalid_op) at c1a060fd
> EAX: f505e080 EBX: f3da0000 ECX: 00000041 EDX: 0000c06a EBP: f5a73f28
> DS: 007b ESI: f3da154c ES: 007b EDI: 00000000 GS: ffffb2b0
> CS: 0060 EIP: c172b046 ERR: ffffffff EFLAGS: 00210006
> #5 [f5a73f08] __ata_sff_port_intr at c172b046
> #6 [f5a73f2c] ata_bmdma_port_intr at c172b333
> #7 [f5a73f50] ata_bmdma_interrupt at c172b576
> #8 [f5a73f84] __handle_irq_event_percpu at c10cf376
> #9 [f5a73fc0] handle_irq_event_percpu at c10cf7c9
> #10 [f5a73fd4] handle_irq_event at c10cf829
> #11 [f5a73fe8] handle_edge_irq at c10d2958
> --- <hard IRQ> ---
> bt: cannot resolve stack trace:
> bt: text symbols on stack:
crash> bt
PID: 31 TASK: f505e080 CPU: 1 COMMAND: "kworker/1:1"
#0 [f5a73de8] die at c101dc04
#1 [f5a73dfc] notify_die at c10809fc
#2 [f5a73e24] do_error_trap at c101ab1c
#3 [f5a73ec4] do_invalid_op at c101b2cf
#4 [f5a73ed4] error_code (via invalid_op) at c1a060fd
EAX: f505e080 EBX: f3da0000 ECX: 00000041 EDX: 0000c06a EBP: f5a73f28
DS: 007b ESI: f3da154c ES: 007b EDI: 00000000 GS: ffffb2b0
CS: 0060 EIP: c172b046 ERR: ffffffff EFLAGS: 00210006
#5 [f5a73f08] __ata_sff_port_intr at c172b046
#6 [f5a73f2c] ata_bmdma_port_intr at c172b333
#7 [f5a73f50] ata_bmdma_interrupt at c172b576
#8 [f5a73f84] __handle_irq_event_percpu at c10cf376
#9 [f5a73fc0] handle_irq_event_percpu at c10cf7c9
#10 [f5a73fd4] handle_irq_event at c10cf829
#11 [f5a73fe8] handle_edge_irq at c10d2958
--- <hard IRQ> ---
bt: cannot resolve stack trace:
bt: text symbols on stack:
OK thanks. Please send patches as required.
Like above, bt is still not working yet though. Attached the patch to fix typo.
Thanks.
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
3:44 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
----- Original Message -----
Dave Anderson <anderson(a)redhat.com> writes:
> Actually, it supposed to be the function that was running on the process stack
> prior to the transition to the IRQ stack. So it should be "handle_irq"
here
> in arch/x86/kernel/irq_32.c:
I see.
[...]
>> crash> bt
>> PID: 0 TASK: c1da8b00 CPU: 0 COMMAND: "swapper/0"
>> #0 [c1da1f60] __schedule at c19fe305
>> #1 [c1da1fa0] schedule at c19febb3
>> #2 [c1da1fac] schedule_preempt_disabled at c19ff0a2
>> #3 [c1da1fb4] cpu_startup_entry at c10a9580
>> crash> bt 45
>> PID: 45 TASK: f57d3a00 CPU: 3 COMMAND: "kworker/3:1"
>> bt: cannot resolve stack trace:
>> bt: Task in user space -- no backtrace
>>
>> In above case, looks like failed to detect panic cpu, and "bt 45"
also
>> not working.
crash> bt 45
PID: 45 TASK: f57d3a00 CPU: 3 COMMAND: "kworker/3:1"
bt: cannot resolve stack trace:
bt: Task in user space -- no backtrace
[...]
>> crash> bt
>> PID: 31 TASK: f505e080 CPU: 1 COMMAND: "kworker/1:1"
>> #0 [f5a73de8] die at c101dc04
>> #1 [f5a73dfc] notify_die at c10809fc
>> #2 [f5a73e24] do_error_trap at c101ab1c
>> #3 [f5a73ec4] do_invalid_op at c101b2cf
>> #4 [f5a73ed4] error_code (via invalid_op) at c1a060fd
>> EAX: f505e080 EBX: f3da0000 ECX: 00000041 EDX: 0000c06a EBP:
>> f5a73f28
>> DS: 007b ESI: f3da154c ES: 007b EDI: 00000000 GS:
>> ffffb2b0
>> CS: 0060 EIP: c172b046 ERR: ffffffff EFLAGS: 00210006
>> #5 [f5a73f08] __ata_sff_port_intr at c172b046
>> #6 [f5a73f2c] ata_bmdma_port_intr at c172b333
>> #7 [f5a73f50] ata_bmdma_interrupt at c172b576
>> #8 [f5a73f84] __handle_irq_event_percpu at c10cf376
>> #9 [f5a73fc0] handle_irq_event_percpu at c10cf7c9
>> #10 [f5a73fd4] handle_irq_event at c10cf829
>> #11 [f5a73fe8] handle_edge_irq at c10d2958
>> --- <hard IRQ> ---
>> bt: cannot resolve stack trace:
>> bt: text symbols on stack:
crash> bt
PID: 31 TASK: f505e080 CPU: 1 COMMAND: "kworker/1:1"
#0 [f5a73de8] die at c101dc04
#1 [f5a73dfc] notify_die at c10809fc
#2 [f5a73e24] do_error_trap at c101ab1c
#3 [f5a73ec4] do_invalid_op at c101b2cf
#4 [f5a73ed4] error_code (via invalid_op) at c1a060fd
EAX: f505e080 EBX: f3da0000 ECX: 00000041 EDX: 0000c06a EBP: f5a73f28
DS: 007b ESI: f3da154c ES: 007b EDI: 00000000 GS: ffffb2b0
CS: 0060 EIP: c172b046 ERR: ffffffff EFLAGS: 00210006
#5 [f5a73f08] __ata_sff_port_intr at c172b046
#6 [f5a73f2c] ata_bmdma_port_intr at c172b333
#7 [f5a73f50] ata_bmdma_interrupt at c172b576
#8 [f5a73f84] __handle_irq_event_percpu at c10cf376
#9 [f5a73fc0] handle_irq_event_percpu at c10cf7c9
#10 [f5a73fd4] handle_irq_event at c10cf829
#11 [f5a73fe8] handle_edge_irq at c10d2958
--- <hard IRQ> ---
bt: cannot resolve stack trace:
bt: text symbols on stack:
> OK thanks. Please send patches as required.
Like above, bt is still not working yet though. Attached the patch to fix
typo.
Thanks.
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
Already checked in!
https://github.com/crash-utility/crash/commit/e71102d356de70488549184a57e...
Thanks,
Dave
3:01 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
Dave Anderson <anderson(a)redhat.com> writes:
> >> crash> bt
> >> PID: 0 TASK: c1da8b00 CPU: 0 COMMAND: "swapper/0"
> >> #0 [c1da1f60] __schedule at c19fe305
> >> #1 [c1da1fa0] schedule at c19febb3
> >> #2 [c1da1fac] schedule_preempt_disabled at c19ff0a2
> >> #3 [c1da1fb4] cpu_startup_entry at c10a9580
> >> crash> bt 45
> >> PID: 45 TASK: f57d3a00 CPU: 3 COMMAND: "kworker/3:1"
> >> bt: cannot resolve stack trace:
> >> bt: Task in user space -- no backtrace
> >>
> >> In above case, looks like failed to detect panic cpu, and "bt 45"
also
> >> not working.
>
> crash> bt 45
> PID: 45 TASK: f57d3a00 CPU: 3 COMMAND: "kworker/3:1"
> bt: cannot resolve stack trace:
> bt: Task in user space -- no backtrace
Debugged this case. The root cause is nested stack of softirq =>
hardirq. Now doesn't handle it correctly, and the patch attacked.
BTW, with this patch, "bt -t" seems to be working at least. "bt" is
failed sometime by confusion of stack-frame detection, this one is
harder to fix.
[BTW, current x86_get_pc() uses inactive_task_frame_ret_addr to get
pc. However, inactive_task_frame is only valid if task is sleeping
state. (running task may overwrite inactive_task_frame already.) I'm
not sure whether we should check is_task_active() or not. Even if
checking is_task_active(), we can't get pc correctly anyway.]
Thanks.
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
3:42 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
----- Original Message -----
Dave Anderson <anderson(a)redhat.com> writes:
>> >> crash> bt
>> >> PID: 0 TASK: c1da8b00 CPU: 0 COMMAND: "swapper/0"
>> >> #0 [c1da1f60] __schedule at c19fe305
>> >> #1 [c1da1fa0] schedule at c19febb3
>> >> #2 [c1da1fac] schedule_preempt_disabled at c19ff0a2
>> >> #3 [c1da1fb4] cpu_startup_entry at c10a9580
>> >> crash> bt 45
>> >> PID: 45 TASK: f57d3a00 CPU: 3 COMMAND: "kworker/3:1"
>> >> bt: cannot resolve stack trace:
>> >> bt: Task in user space -- no backtrace
>> >>
>> >> In above case, looks like failed to detect panic cpu, and "bt
45" also
>> >> not working.
>>
>> crash> bt 45
>> PID: 45 TASK: f57d3a00 CPU: 3 COMMAND: "kworker/3:1"
>> bt: cannot resolve stack trace:
>> bt: Task in user space -- no backtrace
Debugged this case. The root cause is nested stack of softirq =>
hardirq. Now doesn't handle it correctly, and the patch attacked.
BTW, with this patch, "bt -t" seems to be working at least. "bt" is
failed sometime by confusion of stack-frame detection, this one is
harder to fix.
OK thanks, I'll give this patch a test run.
[BTW, current x86_get_pc() uses inactive_task_frame_ret_addr to get
pc. However, inactive_task_frame is only valid if task is sleeping
state. (running task may overwrite inactive_task_frame already.) I'm
not sure whether we should check is_task_active() or not. Even if
checking is_task_active(), we can't get pc correctly anyway.]
Well, x86_get_pc() should only be called in the case of sleeping
tasks because each dumpfile type has its own function to try to
find the active task registers. For example, on a kdump:
cmd_bt()
back_trace()
get_kdump_regs()
get_netdump_regs()
get_netdump_regs_x86()
get_netdump_regs_x86() *should* find the starting point hooks.
If it fails to do so, it will default to machdep->get_stack_frame()
and ultimately x86_get_pc(). So if it gets there, the backtrace
is pretty much guaranteed to be invalid.
Thanks,
Dave
Thanks.
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
[Text Documents:p1-fix-x86-nest-stack.patch]
4:10 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
Dave Anderson <anderson(a)redhat.com> writes:
> [BTW, current x86_get_pc() uses inactive_task_frame_ret_addr to
get
> pc. However, inactive_task_frame is only valid if task is sleeping
> state. (running task may overwrite inactive_task_frame already.) I'm
> not sure whether we should check is_task_active() or not. Even if
> checking is_task_active(), we can't get pc correctly anyway.]
Well, x86_get_pc() should only be called in the case of sleeping
tasks because each dumpfile type has its own function to try to
find the active task registers. For example, on a kdump:
cmd_bt()
back_trace()
get_kdump_regs()
get_netdump_regs()
get_netdump_regs_x86()
get_netdump_regs_x86() *should* find the starting point hooks.
If it fails to do so, it will default to machdep->get_stack_frame()
and ultimately x86_get_pc(). So if it gets there, the backtrace
is pretty much guaranteed to be invalid.
Ah, this might be because of the above hardirq_tasks[] bug.
Thanks.
--
OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
4:41 p.m.
New subject: [PATCH] Fix x86 initialization for {hard, soft}irq_ctx
----- Original Message -----
----- Original Message -----
> Dave Anderson <anderson(a)redhat.com> writes:
>
> >> >> crash> bt
> >> >> PID: 0 TASK: c1da8b00 CPU: 0 COMMAND:
"swapper/0"
> >> >> #0 [c1da1f60] __schedule at c19fe305
> >> >> #1 [c1da1fa0] schedule at c19febb3
> >> >> #2 [c1da1fac] schedule_preempt_disabled at c19ff0a2
> >> >> #3 [c1da1fb4] cpu_startup_entry at c10a9580
> >> >> crash> bt 45
> >> >> PID: 45 TASK: f57d3a00 CPU: 3 COMMAND:
"kworker/3:1"
> >> >> bt: cannot resolve stack trace:
> >> >> bt: Task in user space -- no backtrace
> >> >>
> >> >> In above case, looks like failed to detect panic cpu, and "bt
45"
> >> >> also
> >> >> not working.
> >>
> >> crash> bt 45
> >> PID: 45 TASK: f57d3a00 CPU: 3 COMMAND: "kworker/3:1"
> >> bt: cannot resolve stack trace:
> >> bt: Task in user space -- no backtrace
>
> Debugged this case. The root cause is nested stack of softirq =>
> hardirq. Now doesn't handle it correctly, and the patch attacked.
>
> BTW, with this patch, "bt -t" seems to be working at least. "bt"
is
> failed sometime by confusion of stack-frame detection, this one is
> harder to fix.
OK thanks, I'll give this patch a test run.
Queued for crash-7.1.8:
https://github.com/crash-utility/crash/commit/fc9c517acddda878319774298cc...
Thanks,
Dave
>
> [BTW, current x86_get_pc() uses inactive_task_frame_ret_addr to get
> pc. However, inactive_task_frame is only valid if task is sleeping
> state. (running task may overwrite inactive_task_frame already.) I'm
> not sure whether we should check is_task_active() or not. Even if
> checking is_task_active(), we can't get pc correctly anyway.]
Well, x86_get_pc() should only be called in the case of sleeping
tasks because each dumpfile type has its own function to try to
find the active task registers. For example, on a kdump:
cmd_bt()
back_trace()
get_kdump_regs()
get_netdump_regs()
get_netdump_regs_x86()
get_netdump_regs_x86() *should* find the starting point hooks.
If it fails to do so, it will default to machdep->get_stack_frame()
and ultimately x86_get_pc(). So if it gets there, the backtrace
is pretty much guaranteed to be invalid.
Thanks,
Dave
> Thanks.
> --
> OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp>
>
>
>
> [Text Documents:p1-fix-x86-nest-stack.patch]
>
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
8:02 a.m.
New subject: [Skiboot] [PATCH]: Implementation of opalmsg command in crash-utility
Ankit Kumar <ankit(a)linux.vnet.ibm.com> writes:
+/*
+ * Definitions derived from OPAL. These need to track corresponding values in
+ * https://github.com/open-power/skiboot/blob/master/include/mem-map.h
+ */
+#define SKIBOOT_CONSOLE_DUMP_START 0x31000000
+#define SKIBOOT_CONSOLE_DUMP_SIZE 0x40000
Where did you get this size from?
INMEM_CON_LEN in skiboot is 0x100000
Even if booting a kdump kernel, you're going to have a device tree with
the appropriate nodes present to *programatically* find out where the
console is. You should certainly do that.
We've never said that the skiboot memory map is ABI, so it *may* change.
--
Stewart Smith
OPAL Architect, IBM.
2771
days inactive
2781
days old
devel@lists.crash-utility.osci.io
18 comments
4 participants
participants (4)
-
Ankit Kumar -
Dave Anderson -
OGAWA Hirofumi -
Stewart Smith