11:04 a.m.
There are just too many kinds of panic types are categorized under
the same Oops: xxxx, makes this field really ambiguous and not so useful
PANIC: "Oops: 0000 [#1] SMP " (check log for details)
this patch separated 3 kinds of panicmsg out, as the most happening cases
among the machines managed by me; the match string are copied
from kernel source code exactly, after applied, I got panicmsg like:
include/linux/kernel.h:#define HW_ERR
panicmsg: "[Hardware Error]: CPU 7: Machine Check Exception: 5 Bank 11:
f200003f000100b2"
drivers/char/sysrq.c:__handle_sysrq
panicmsg: "SysRq : Trigger a crash"
arch/x86/kernel/traps.c:do_general_protection
panicmsg: "general protection fault: 8800 [#1] SMP"
arch/x86/mm/fault.c:show_fault_oops
panicmsg: "BUG: unable to handle kernel paging request at
00001248a68eb328"
We need to move the SysRq matching lines to before matching "Oops", because
SysRq lines usually also has the Oops, need to take precedence for SysRq.
Signed-off-by: Derek Che <drc(a)yahoo-inc.com>
---
task.c | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
diff --git a/task.c b/task.c
index 4214d7f..1530e7b 100644
--- a/task.c
+++ b/task.c
@@ -5509,19 +5509,31 @@ get_panicmsg(char *buf)
}
rewind(pc->tmpfile);
while (!msg_found && fgets(buf, BUFSIZE, pc->tmpfile)) {
- if (strstr(buf, "Oops: ") ||
- strstr(buf, "kernel BUG at"))
- msg_found = TRUE;
+ if (strstr(buf, "[Hardware Error]: "))
+ msg_found = TRUE;
+ }
+ rewind(pc->tmpfile);
+ while (!msg_found && fgets(buf, BUFSIZE, pc->tmpfile)) {
+ if (strstr(buf, "general protection fault"))
+ msg_found = TRUE;
}
rewind(pc->tmpfile);
while (!msg_found && fgets(buf, BUFSIZE, pc->tmpfile)) {
if (strstr(buf, "SysRq : Netdump") ||
strstr(buf, "SysRq : Trigger a crashdump") ||
- strstr(buf, "SysRq : Crash")) {
+ strstr(buf, "SysRq : Crash") ||
+ strstr(buf, "SysRq : Trigger a crash")) {
pc->flags |= SYSRQ;
msg_found = TRUE;
}
}
+ rewind(pc->tmpfile);
+ while (!msg_found && fgets(buf, BUFSIZE, pc->tmpfile)) {
+ if (strstr(buf, "Oops: ") ||
+ strstr(buf, "kernel BUG at") ||
+ strstr(buf, "BUG: unable to handle kernel "))
+ msg_found = TRUE;
+ }
rewind(pc->tmpfile);
while (!msg_found && fgets(buf, BUFSIZE, pc->tmpfile)) {
if (strstr(buf, "sysrq") &&
2:52 p.m.
I ran a test on ~200 dumpfiles, and for the most part, the patch is
quite useful in replacing the "Oops" message with something more
helpful.
However, the "[Hardware Error]" check should be the very last thing checked.
Actually, I'm not even sure whether it should be checked at all, because there
are dozens of pr_emerg(HW_ERR ...) calls in the kernel, and it appears that they
don't all necessarily cause the kernel to crash.
For example, this sample vmcore currently correctly shows that the kernel has
crashed due to a BUG in mm/slab.c:
crash> sys
KERNEL: 2.6.32-220.el6.x86_64_slab_page_corruption/vmlinux.gz
DUMPFILE: 2.6.32-220.el6.x86_64_slab_page_corruption/musa_vmcore [PARTIAL DUMP]
CPUS: 32
DATE: Thu Feb 14 09:14:12 2013
UPTIME: 14:18:49
LOAD AVERAGE: 2.23, 1.94, 2.04
TASKS: 1621
NODENAME: musa
RELEASE: 2.6.32-220.el6.x86_64
VERSION: #1 SMP Wed Nov 9 08:03:13 EST 2011
MACHINE: x86_64 (2599 Mhz)
MEMORY: 128 GB
PANIC: "kernel BUG at mm/slab.c:533!"
crash> bt
PID: 159 TASK: ffff881018c2eac0 CPU: 28 COMMAND: "events/28"
#0 [ffff881018c359f0] machine_kexec at ffffffff81031fcb
#1 [ffff881018c35a50] crash_kexec at ffffffff810b8f72
#2 [ffff881018c35b20] oops_end at ffffffff814f04b0
#3 [ffff881018c35b50] die at ffffffff8100f26b
#4 [ffff881018c35b80] do_trap at ffffffff814efda4
#5 [ffff881018c35be0] do_invalid_op at ffffffff8100ce35
#6 [ffff881018c35c80] invalid_op at ffffffff8100bedb
[exception RIP: free_block+357]
RIP: ffffffff8115ffd5 RSP: ffff881018c35d30 RFLAGS: 00010006
RAX: ffffea00321db658 RBX: ffff880f5bc52c80 RCX: 0000000000000002
RDX: 004000000000006c RSI: ffff880fb58e9ac0 RDI: ffff880e51a1d000
RBP: ffff881018c35d80 R8: ffff880fb58e9ac0 R9: 0000000000000000
R10: 000000000000000c R11: 0000000000000000 R12: 0000000000000006
R13: ffff880ffaa95828 R14: 0000000000000002 R15: ffffea0000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
#7 [ffff881018c35d88] drain_array at ffffffff81160211
#8 [ffff881018c35dd8] cache_reap at ffffffff81161210
#9 [ffff881018c35e38] worker_thread at ffffffff8108b2b0
#10 [ffff881018c35ee8] kthread at ffffffff81090886
#11 [ffff881018c35f48] kernel_thread at ffffffff8100c14a
crash>
With your patch applied, it incorrectly shows this:
crash> sys
KERNEL: 2.6.32-220.el6.x86_64_slab_page_corruption/vmlinux.gz
DUMPFILE: 2.6.32-220.el6.x86_64_slab_page_corruption/musa_vmcore [PARTIAL DUMP]
CPUS: 32
DATE: Thu Feb 14 09:14:12 2013
UPTIME: 14:18:49
LOAD AVERAGE: 2.23, 1.94, 2.04
TASKS: 1621
NODENAME: musa
RELEASE: 2.6.32-220.el6.x86_64
VERSION: #1 SMP Wed Nov 9 08:03:13 EST 2011
MACHINE: x86_64 (2599 Mhz)
MEMORY: 128 GB
PANIC: "[Hardware Error]: Machine check events logged"
I don't have a problem with the other parts of the patch.
I'll move the hardware error check to the bottom, and only use it if there
are no other relevant strings found, and then re-test that configuration.
Dave
----- Original Message -----
There are just too many kinds of panic types are categorized under
the same Oops: xxxx, makes this field really ambiguous and not so useful
PANIC: "Oops: 0000 [#1] SMP " (check log for details)
this patch separated 3 kinds of panicmsg out, as the most happening cases
among the machines managed by me; the match string are copied
from kernel source code exactly, after applied, I got panicmsg like:
include/linux/kernel.h:#define HW_ERR
panicmsg: "[Hardware Error]: CPU 7: Machine Check Exception: 5 Bank
11: f200003f000100b2"
drivers/char/sysrq.c:__handle_sysrq
panicmsg: "SysRq : Trigger a crash"
arch/x86/kernel/traps.c:do_general_protection
panicmsg: "general protection fault: 8800 [#1] SMP"
arch/x86/mm/fault.c:show_fault_oops
panicmsg: "BUG: unable to handle kernel paging request at
00001248a68eb328"
We need to move the SysRq matching lines to before matching "Oops", because
SysRq lines usually also has the Oops, need to take precedence for SysRq.
Signed-off-by: Derek Che <drc(a)yahoo-inc.com>
---
task.c | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
diff --git a/task.c b/task.c
index 4214d7f..1530e7b 100644
--- a/task.c
+++ b/task.c
@@ -5509,19 +5509,31 @@ get_panicmsg(char *buf)
}
rewind(pc->tmpfile);
while (!msg_found && fgets(buf, BUFSIZE, pc->tmpfile)) {
- if (strstr(buf, "Oops: ") ||
- strstr(buf, "kernel BUG at"))
- msg_found = TRUE;
+ if (strstr(buf, "[Hardware Error]: "))
+ msg_found = TRUE;
+ }
+ rewind(pc->tmpfile);
+ while (!msg_found && fgets(buf, BUFSIZE, pc->tmpfile)) {
+ if (strstr(buf, "general protection fault"))
+ msg_found = TRUE;
}
rewind(pc->tmpfile);
while (!msg_found && fgets(buf, BUFSIZE, pc->tmpfile)) {
if (strstr(buf, "SysRq : Netdump") ||
strstr(buf, "SysRq : Trigger a crashdump") ||
- strstr(buf, "SysRq : Crash")) {
+ strstr(buf, "SysRq : Crash") ||
+ strstr(buf, "SysRq : Trigger a crash")) {
pc->flags |= SYSRQ;
msg_found = TRUE;
}
}
+ rewind(pc->tmpfile);
+ while (!msg_found && fgets(buf, BUFSIZE, pc->tmpfile)) {
+ if (strstr(buf, "Oops: ") ||
+ strstr(buf, "kernel BUG at") ||
+ strstr(buf, "BUG: unable to handle kernel "))
+ msg_found = TRUE;
+ }
rewind(pc->tmpfile);
while (!msg_found && fgets(buf, BUFSIZE, pc->tmpfile)) {
if (strstr(buf, "sysrq") &&
11:33 a.m.
On Tuesday, February 3, 2015 12:53 PM, Dave Anderson <anderson(a)redhat.com> wrote:
I'll move the hardware error check to the bottom, and only use it
if there
are no other relevant strings found, and then re-test that configuration.
how about match this string "Machine Check Exception:" ? or can I use a pattern
matching ?this is a memory failure at bank 12, usually indicates we need to replace this
memory in bank 12,
here we internally have a tool depends on crash to analyze kernel crash and find out
reasons and solutions, I hope the
"[Hardware Error]: CPU 14: Machine Check Exception: 5 Bank 12: fe00014b001000c3"
line
can be matched, instead of the less useful
"Kernel panic - not syncing: Fatal machine check on current CPU" currently
selected,
<0>[Hardware Error]: CPU 14: Machine Check Exception: 5 Bank 12: fe00014b001000c3
<0>[Hardware Error]: RIP !INEXACT! 10:<ffffffff810ace8a>
{tick_check_idle+0xca/0xe0}
<0>[Hardware Error]: TSC 52e41ed579869d ADDR 53a92b000 MISC 908424000803e8c
<0>[Hardware Error]: PROCESSOR 0:306e4 TIME 1423045186 SOCKET 0 APIC 5
<0>[Hardware Error]: Some CPUs didn't answer in synchronization
<0>[Hardware Error]: Machine check: Invalid
<0>Kernel panic - not syncing: Fatal machine check on current CPU
<4>Pid: 0, comm: swapper Tainted: G M ---------------
2.6.32-431.23.3.el6.YAHOO.20140804.x86_64 #1
<4>Call Trace:
<4> <#MC> [<ffffffff8152866c>] ? panic+0xa7/0x16f
<4> [<ffffffff8102880f>] ? mce_panic+0x20f/0x230
<4> [<ffffffff81029c87>] ? do_machine_check+0x7a7/0xaf0
<4> [<ffffffff810ace8a>] ? tick_check_idle+0xca/0xe0
<4> [<ffffffff8152bc9c>] ? machine_check+0x1c/0x30
<4> [<ffffffff810ace8a>] ? tick_check_idle+0xca/0xe0
<4> <<EOE>> <IRQ> [<ffffffff8107a51c>] ?
irq_enter+0x6c/0x80
<4> [<ffffffff8102b1d3>] ? smp_threshold_interrupt+0x13/0x40
<4> [<ffffffff8100bd13>] ? threshold_interrupt+0x13/0x20
<4> <EOI> [<ffffffff812e14ee>] ? intel_idle+0xde/0x170
<4> [<ffffffff812e14d1>] ? intel_idle+0xc1/0x170
<4> [<ffffffff814274a7>] ? cpuidle_idle_call+0xa7/0x140
<4> [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
<4> [<ffffffff8152219c>] ? start_secondary+0x2ac/0x2ef
Thanks,
- Derek
1:13 p.m.
----- Original Message -----
On Tuesday, February 3, 2015 12:53 PM, Dave Anderson <anderson(a)redhat.com>
wrote:
> I'll move the hardware error check to the bottom, and only use it if there
> are no other relevant strings found, and then re-test that configuration.
how about match this string "Machine Check Exception:" ? or can I use a
pattern matching ?this is a memory failure at bank 12, usually indicates we
need to replace this memory in bank 12,
here we internally have a tool depends on crash to analyze kernel crash and
find out reasons and solutions, I hope the
"[Hardware Error]: CPU 14: Machine Check Exception: 5 Bank 12:
fe00014b001000c3" line can be matched, instead of the less useful
"Kernel panic - not syncing: Fatal machine check on current CPU" currently
selected,
Why? The kernel panic "Fatal machine check" message above gives you the actual
reason
for the panic. And even if you were to alternatively display the first pr_emerg(HW_ERR)
string, the error message continues with 5 subsequent pr_cont() or pr_emerg(HW_ERR) lines
that you would have to check the log for the details.
I have added several more search strings to your original patch to pretty much prevent
any more generic "Oops ..." messages. But if the sequence gets as far as
checking for
the pr_emerg("Kernel panic - ...", I don't want to override that string. It
*is* the
"panic" string that's supposed to be displayed there.
Dave
<0>[Hardware Error]: CPU 14: Machine Check Exception: 5 Bank
12: fe00014b001000c3
<0>[Hardware Error]: RIP !INEXACT! 10:<ffffffff810ace8a>
{tick_check_idle+0xca/0xe0}
<0>[Hardware Error]: TSC 52e41ed579869d ADDR 53a92b000 MISC 908424000803e8c
<0>[Hardware Error]: PROCESSOR 0:306e4 TIME 1423045186 SOCKET 0 APIC 5
<0>[Hardware Error]: Some CPUs didn't answer in synchronization
<0>[Hardware Error]: Machine check: Invalid
<0>Kernel panic - not syncing: Fatal machine check on current CPU
<4>Pid: 0, comm: swapper Tainted: G M ---------------
2.6.32-431.23.3.el6.YAHOO.20140804.x86_64 #1
<4>Call Trace:
<4> <#MC> [<ffffffff8152866c>] ? panic+0xa7/0x16f
<4> [<ffffffff8102880f>] ? mce_panic+0x20f/0x230
<4> [<ffffffff81029c87>] ? do_machine_check+0x7a7/0xaf0
<4> [<ffffffff810ace8a>] ? tick_check_idle+0xca/0xe0
<4> [<ffffffff8152bc9c>] ? machine_check+0x1c/0x30
<4> [<ffffffff810ace8a>] ? tick_check_idle+0xca/0xe0
<4> <<EOE>> <IRQ> [<ffffffff8107a51c>] ?
irq_enter+0x6c/0x80
<4> [<ffffffff8102b1d3>] ? smp_threshold_interrupt+0x13/0x40
<4> [<ffffffff8100bd13>] ? threshold_interrupt+0x13/0x20
<4> <EOI> [<ffffffff812e14ee>] ? intel_idle+0xde/0x170
<4> [<ffffffff812e14d1>] ? intel_idle+0xc1/0x170
<4> [<ffffffff814274a7>] ? cpuidle_idle_call+0xa7/0x140
<4> [<ffffffff81009fc6>] ? cpu_idle+0xb6/0x110
<4> [<ffffffff8152219c>] ? start_secondary+0x2ac/0x2ef
Thanks,
- Derek
8:31 a.m.
----- Original Message -----
There are just too many kinds of panic types are categorized under
the same Oops: xxxx, makes this field really ambiguous and not so useful
PANIC: "Oops: 0000 [#1] SMP " (check log for details)
this patch separated 3 kinds of panicmsg out, as the most happening cases
among the machines managed by me; the match string are copied
from kernel source code exactly, after applied, I got panicmsg like:
include/linux/kernel.h:#define HW_ERR
panicmsg: "[Hardware Error]: CPU 7: Machine Check Exception: 5 Bank
11: f200003f000100b2"
drivers/char/sysrq.c:__handle_sysrq
panicmsg: "SysRq : Trigger a crash"
arch/x86/kernel/traps.c:do_general_protection
panicmsg: "general protection fault: 8800 [#1] SMP"
arch/x86/mm/fault.c:show_fault_oops
panicmsg: "BUG: unable to handle kernel paging request at
00001248a68eb328"
We need to move the SysRq matching lines to before matching "Oops", because
SysRq lines usually also has the Oops, need to take precedence for SysRq.
Signed-off-by: Derek Che <drc(a)yahoo-inc.com>
Hi Derek,
As I mentioned earlier, in addition to checking for the general
protection faults, in my testing I found several other instances
where the "Oops" message could be replaced with the more meaningful
messages that preceded it, such as double faults, divide errors,
stack segment faults, "Kernel BUG" (with a capital K), "Unable to
handle kernel ..." (with a capital U), etc. I also added a few
break instructions after a search-for message was found instead
of continuing to parse the kernel log.
However, the machine check string search does follow the "kernel panic - "
check, which I understand you would prefer to be the opposite. The
fatal error string searches that are being made come from from die()
calls, or from other message sources that are part of the kernel crash
sequence. On the other hand, the machine check messages are generated
from a stream of pr_emerg(HW_ERR) calls, and are not necessarily
(although likely) crash precedents. But since the kernel panic
message does contain the "Fatal machine check" message, the reason
behind the crash is readily evident.
I appreciate your getting the ball rolling here, as it was certainly
due for an update/improvement.
Queued for crash-7.1.0:
https://github.com/crash-utility/crash/commit/c3840016bf1770b6b1cf571202f...
Thanks,
Dave
2:16 a.m.
New subject: crash: pipe operation failed
Hi Anderson,
I wonder if you are aware of this pipe failure and is it worth to be fixed? if `|` is in
quote, crash should not try to interpret it, right?
crash> ps |awk 'NR <= 10 || $5 ~ /U/'crash: pipe operation failed
crash: pipe operation failed
crash>
Thanks,
- Derek
8:41 a.m.
New subject: crash: pipe operation failed
----- Original Message -----
Hi Anderson,
I wonder if you are aware of this pipe failure and is it worth to be fixed?
if `|` is in quote, crash should not try to interpret it, right?
crash> ps |awk 'NR <= 10 || $5 ~ /U/'crash: pipe operation failed
crash: pipe operation failed
crash>
Thanks,
- Derek
No, not aware. I'll take a look.
Thanks,
Dave
10:36 a.m.
New subject: crash: pipe operation failed
----- Original Message -----
----- Original Message -----
>
>
> Hi Anderson,
>
> I wonder if you are aware of this pipe failure and is it worth to be fixed?
> if `|` is in quote, crash should not try to interpret it, right?
>
> crash> ps |awk 'NR <= 10 || $5 ~ /U/'crash: pipe operation failed
> crash: pipe operation failed
> crash>
>
> Thanks,
> - Derek
No, not aware. I'll take a look.
Thanks,
Dave
Hi Derek,
A fix is queued for crash-7.1.1:
https://github.com/crash-utility/crash/commit/fe6679f131728bb368242a6eb0b...
Thanks,
Dave
11:56 a.m.
New subject: crash: pipe operation failed
On Wednesday, February 25, 2015 8:36 AM, Dave Anderson <anderson(a)redhat.com> wrote:
Hi Derek,
A fix is queued for crash-7.1.1:
https://github.com/crash-utility/crash/commit/fe6679f131728bb368242a6eb0b...
That's very nice! Do you also usually add Reporter: in the commit message? :-)
like the linux kernel community does
Thanks,
Dave
12:57 p.m.
New subject: crash: pipe operation failed
----- Original Message -----
On Wednesday, February 25, 2015 8:36 AM, Dave Anderson <anderson(a)redhat.com>
wrote:
> Hi Derek,
> A fix is queued for crash-7.1.1:
>
>
https://github.com/crash-utility/crash/commit/fe6679f131728bb368242a6eb0b...
That's very nice! Do you also usually add Reporter: in the commit message? :-)
like the linux kernel community does
Sorry, no, I never have done that.
Dave
1:19 p.m.
New subject: crash: pipe operation failed
On Wednesday, February 25, 2015 10:57 AM, Dave Anderson <anderson(a)redhat.com>
wrote:
>
> On Wednesday, February 25, 2015 8:36 AM, Dave Anderson <anderson(a)redhat.com>
> wrote:
> > Hi Derek,
> > A fix is queued for crash-7.1.1:
> >
> >
https://github.com/crash-utility/crash/commit/fe6679f131728bb368242a6eb0b...
>
>
> That's very nice! Do you also usually add Reporter: in the commit message? :-)
> like the linux kernel community does
Sorry, no, I never have done that.
I've just tested command like this, it works! Kudo
crash-7.1.1-rc1> kmem -s | awk 'NR == 1 || /ext4/'
CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE
ffff88062ab4a000 ext4_groupinfo_4k 136 6620 7020 234 4k
ffff88062b55ac00 ext4_inode_cache 1040 36865 61721 1991 32k
ffff88062b55ab00 ext4_xattr 88 691 736 16 4k
ffff88062b55aa00 ext4_free_data 64 3851 4672 73 4k
ffff88062b55a900 ext4_allocation_context 128 259 512 16 4k
ffff88062b55a800 ext4_io_end 80 10481 11424 224 4k
ffff88062b55a700 ext4_extent_status 40 27764 43962 431 4k
3412
days inactive
3434
days old
devel@lists.crash-utility.osci.io
10 comments
4 participants
participants (4)
-
Dave Anderson -
Derek Cheng -
Derek Cheng
-
drc＠yahoo-inc.com