3:35 p.m.
I have a vmcore generated on RHEL6.1 that newer versions of crash have
trouble analyzing (5.1.1-2.el6 seems to work ok).
I can provide additional binary files if needed, just let me know what
convention best suits the list (ftp, private email attachment, etc.)
Crash Version: OS: Result:
crash 5.1.8 Debian wheezy faults
crash 5.1.7-1.el6 RHEL6.2 Alpha faults
crash 5.1.1-2.el6 RHEL6.1 ok
Kernel:
2.6.32-131.0.15.el6.exp10.bz16586.x86_64 (2.6.32-131.0.15 + a fix for
Red Hat bz-707268)
Interesting warnings when starting crash:
WARNING: sparsemem: invalid section number: 137438888923
WARNING: sparsemem: invalid section number: 137438888923
First fault, null pointer deference:
please wait... (determining panic task)
Program received signal SIGSEGV, Segmentation fault.
x86_64_get_dumpfile_stack_frame (rsp=0x7fffffffcc58, rip=0x7fffffffcc50,
bt_in=0x7fffffffcce0) at x86_64.c:4183
4183 ur_rip = ULONG(user_regs +
(gdb) p user_regs
$1 = 0x0
Workaround, check that bt->machdep is not NULL:
diff -Nupr crash-5.1.8/x86_64.c crash-5.1.8.new/x86_64.c
--- crash-5.1.8/x86_64.c 2011-09-16 15:01:12.000000000 -0400
+++ crash-5.1.8.new/x86_64.c 2011-09-28 14:12:45.347188571 -0400
@@ -4178,7 +4178,7 @@ x86_64_get_dumpfile_stack_frame(struct b
goto skip_stage;
}
}
- } else if (ELF_NOTES_VALID()) {
+ } else if (ELF_NOTES_VALID() && bt->machdep) {
user_regs = bt->machdep;
ur_rip = ULONG(user_regs +
OFFSET(user_regs_struct_rip));
Second fault, a curiously large n_descsz in elf note header:
please wait... (determining panic task)
Program received signal SIGSEGV, Segmentation fault.
get_regs_from_note (note=0xd26472 "\b", ip=0x7fffffffc4e0,
sp=0x7fffffffc4e8)
at netdump.c:2221
2221 *sp = ULONG(user_regs + offset_sp);
(gdb) p *(Elf64_Nhdr *)note
$1 = {n_namesz = 8, n_descsz = 3438804992, n_type = 8}
Workaround, do not attempt reading registers from elf notes (this chunk
of code was not present in crash 5.1.1):
diff -Nupr crash-5.1.8/netdump.c crash-5.1.8.new/netdump.c
--- crash-5.1.8/netdump.c 2011-09-16 15:01:12.000000000 -0400
+++ crash-5.1.8.new/netdump.c 2011-09-28 14:14:43.687183734 -0400
@@ -2286,7 +2286,7 @@ get_netdump_regs_x86_64(struct bt_info *
bt->machdep = (void *)user_regs;
}
-
+#if 0
if (ELF_NOTES_VALID() &&
(bt->flags & BT_DUMPFILE_SEARCH) && DISKDUMP_DUMPFILE()
&&
(note = (Elf64_Nhdr *)
@@ -2305,7 +2305,7 @@ get_netdump_regs_x86_64(struct bt_info *
bt->machdep = (void *)user_regs;
}
-
+#endif
machdep->get_stack_frame(bt, ripp, rspp); }
Given the warning messages at the beginning of the process, I'm sure if
I'm dealing with a corrupted or incomplete vmcore image. Let me know
what additional info could be useful if this seems worth debugging
further.
Thanks,
-- Joe Lawrence
4:14 p.m.
Hi Joe,
It pretty clear it's due to this change in 5.1.5:
- Implemented the capability of using the NT_PRSTATUS ELF note data
that is saved in version 4 compressed kdump headers to determine the
starting stack and instruction pointer hooks for x86 and x86_64
backtraces when they cannot be determined in the traditional manners.
(wang.chao(a)cn.fujitsu.com, wency(a)cn.fujitsu.com)
What happens if you run it like so:
$ crash --no_elf_notes vmlinux vmcore
As far as this message:
WARNING: sparsemem: invalid section number: 137438888923
That should be outside the realm of Fujitsu's ELF notes patch. Does this kernel
have some kind of Stratus VM modification?
Dave
----- Original Message -----
Crash faults when determining panic task
I have a vmcore generated on RHEL6.1 that newer versions of crash
have trouble analyzing (5.1.1-2.el6 seems to work ok) .
I can provide additional binary files if needed, just let me know
what convention best suits the list (ftp, private email attachment,
etc.)
Crash Version : OS: Result:
crash 5.1.8 Debian wheezy faults
crash 5.1.7-1.el6 RHEL6.2 Alpha faults
crash 5.1.1-2.el6 RHEL6.1 ok
Kernel:
2.6.32-131.0.15.el6.exp10.bz16586.x86_64 ( 2.6.32-131.0.15 + a fix
for Red Hat bz - 707268)
Interesting warnings when starting crash:
WARNING: sparsemem: invalid section number: 137438888923
WARNING: sparsemem: invalid section number: 137438888923
First fault, null pointer deference:
please wait... (determining panic task)
Program received signal SIGSEGV, Segmentation fault.
x86_64_get_dumpfile_stack_frame (rsp=0x7fffffffcc58,
rip=0x7fffffffcc50,
bt_in=0x7fffffffcce0) at x86_64.c:4183
4183 ur_rip = ULONG(user_regs +
(gdb) p user_regs
$1 = 0x0
Workaround, check that bt->machdep is not NULL:
diff -Nupr crash-5.1.8/x86_64.c crash-5.1.8.new/x86_64.c
--- crash-5.1.8/x86_64.c 2011-09-16 15:01:12.000000000 -0400
+++ crash-5.1.8.new/x86_64.c 2011-09-28 14:12:45.347188571 -0400
@@ -4178,7 +4178,7 @@ x86_64_get_dumpfile_stack_frame(struct b
goto skip_stage;
}
}
- } else if (ELF_NOTES_VALID()) {
+ } else if (ELF_NOTES_VALID() && bt->machdep) {
user_regs = bt->machdep;
ur_rip = ULONG(user_regs +
OFFSET(user_regs_struct_rip));
Second fault, a curiously large n_descsz in elf note header:
please wait... (determining panic task)
Program received signal SIGSEGV, Segmentation fault.
get_regs_from_note (note=0xd26472 "\b", ip=0x7fffffffc4e0,
sp=0x7fffffffc4e8)
at netdump.c:2221
2221 *sp = ULONG(user_regs + offset_sp);
(gdb) p *(Elf64_Nhdr *)note
$1 = {n_namesz = 8, n_descsz = 3438804992, n_type = 8}
Workaround, do not attempt reading registers from elf notes (this
chunk of code was not present in crash 5.1.1):
diff -Nupr crash-5.1.8/netdump.c crash-5.1.8.new/netdump.c
--- crash-5.1.8/netdump.c 2011-09-16 15:01:12.000000000 -0400
+++ crash-5.1.8.new/netdump.c 2011-09-28 14:14:43.687183734 -0400
@@ -2286,7 +2286,7 @@ get_netdump_regs_x86_64(struct bt_info *
bt->machdep = (void *)user_regs;
}
-
+#if 0
if (ELF_NOTES_VALID() &&
(bt->flags & BT_DUMPFILE_SEARCH) && DISKDUMP_DUMPFILE() &&
(note = (Elf64_Nhdr *)
@@ -2305,7 +2305,7 @@ get_netdump_regs_x86_64(struct bt_info *
bt->machdep = (void *)user_regs;
}
-
+#endif
machdep->get_stack_frame(bt, ripp, rspp); }
Given the warning messages at the beginning of the process, I'm sure
if I' m dealing with a corrupted or incomplete vmcore image. Let me
know what additional info could be useful if this seems worth
debugging further.
Thanks,
-- Joe Lawrence
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
9:52 a.m.
Dave,
Adding --no_elf_notes to the crash invocation does indeed start crash
with without issue. Do you think that I am dealing with a
corrupted/incomplete vmcore (as evident in that extremely large n_descsz
value) or is this a bug that crash could more gracefully handle?
As far as the kernel is concerned,
2.6.32-131.0.15.el6.exp10.bz16586.x86_64 was a stock RH 2.6.32-131.0.15
with an added patch for handling an MD Raid bug (RHBZ-707268). Stratus
does load a driver to track dirty VM pages for harvesting purposes, but
does not change general VM behavior.
FWIW, this is the only vmcore that I've seen ELF note faulting or
invalid section numbers.
Thanks,
-- Joe
-----Original Message-----
From: crash-utility-bounces(a)redhat.com
[mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson
Sent: Wednesday, September 28, 2011 5:15 PM
To: Discussion list for crash utility usage,maintenance and development
Subject: Re: [Crash-utility] Crash faults when determining panic task
Hi Joe,
It pretty clear it's due to this change in 5.1.5:
- Implemented the capability of using the NT_PRSTATUS ELF note
data
that is saved in version 4 compressed kdump headers to
determine the
starting stack and instruction pointer hooks for x86 and
x86_64
backtraces when they cannot be determined in the traditional
manners.
(wang.chao(a)cn.fujitsu.com, wency(a)cn.fujitsu.com)
What happens if you run it like so:
$ crash --no_elf_notes vmlinux vmcore
As far as this message:
WARNING: sparsemem: invalid section number: 137438888923
That should be outside the realm of Fujitsu's ELF notes patch. Does
this kernel
have some kind of Stratus VM modification?
Dave
----- Original Message -----
Crash faults when determining panic task
I have a vmcore generated on RHEL6.1 that newer versions of crash
have trouble analyzing (5.1.1-2.el6 seems to work ok) .
I can provide additional binary files if needed, just let me know
what convention best suits the list (ftp, private email attachment,
etc.)
Crash Version : OS: Result:
crash 5.1.8 Debian wheezy faults
crash 5.1.7-1.el6 RHEL6.2 Alpha faults
crash 5.1.1-2.el6 RHEL6.1 ok
Kernel:
2.6.32-131.0.15.el6.exp10.bz16586.x86_64 ( 2.6.32-131.0.15 + a fix
for Red Hat bz - 707268)
Interesting warnings when starting crash:
WARNING: sparsemem: invalid section number: 137438888923
WARNING: sparsemem: invalid section number: 137438888923
First fault, null pointer deference:
please wait... (determining panic task)
Program received signal SIGSEGV, Segmentation fault.
x86_64_get_dumpfile_stack_frame (rsp=0x7fffffffcc58,
rip=0x7fffffffcc50,
bt_in=0x7fffffffcce0) at x86_64.c:4183
4183 ur_rip = ULONG(user_regs +
(gdb) p user_regs
$1 = 0x0
Workaround, check that bt->machdep is not NULL:
diff -Nupr crash-5.1.8/x86_64.c crash-5.1.8.new/x86_64.c
--- crash-5.1.8/x86_64.c 2011-09-16 15:01:12.000000000 -0400
+++ crash-5.1.8.new/x86_64.c 2011-09-28 14:12:45.347188571 -0400
@@ -4178,7 +4178,7 @@ x86_64_get_dumpfile_stack_frame(struct b
goto skip_stage;
}
}
- } else if (ELF_NOTES_VALID()) {
+ } else if (ELF_NOTES_VALID() && bt->machdep) {
user_regs = bt->machdep;
ur_rip = ULONG(user_regs +
OFFSET(user_regs_struct_rip));
Second fault, a curiously large n_descsz in elf note header:
please wait... (determining panic task)
Program received signal SIGSEGV, Segmentation fault.
get_regs_from_note (note=0xd26472 "\b", ip=0x7fffffffc4e0,
sp=0x7fffffffc4e8)
at netdump.c:2221
2221 *sp = ULONG(user_regs + offset_sp);
(gdb) p *(Elf64_Nhdr *)note
$1 = {n_namesz = 8, n_descsz = 3438804992, n_type = 8}
Workaround, do not attempt reading registers from elf notes (this
chunk of code was not present in crash 5.1.1):
diff -Nupr crash-5.1.8/netdump.c crash-5.1.8.new/netdump.c
--- crash-5.1.8/netdump.c 2011-09-16 15:01:12.000000000 -0400
+++ crash-5.1.8.new/netdump.c 2011-09-28 14:14:43.687183734 -0400
@@ -2286,7 +2286,7 @@ get_netdump_regs_x86_64(struct bt_info *
bt->machdep = (void *)user_regs;
}
-
+#if 0
if (ELF_NOTES_VALID() &&
(bt->flags & BT_DUMPFILE_SEARCH) && DISKDUMP_DUMPFILE() &&
(note = (Elf64_Nhdr *)
@@ -2305,7 +2305,7 @@ get_netdump_regs_x86_64(struct bt_info *
bt->machdep = (void *)user_regs;
}
-
+#endif
machdep->get_stack_frame(bt, ripp, rspp); }
Given the warning messages at the beginning of the process, I'm sure
if I' m dealing with a corrupted or incomplete vmcore image. Let me
know what additional info could be useful if this seems worth
debugging further.
Thanks,
-- Joe Lawrence
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
1:08 p.m.
----- Original Message -----
Dave,
Adding --no_elf_notes to the crash invocation does indeed start crash
with without issue. Do you think that I am dealing with a
corrupted/incomplete vmcore (as evident in that extremely large n_descsz
value) or is this a bug that crash could more gracefully handle?
Hi Joe,
It should absolutely handle it more gracefully, but I'm not sure whether
the vmcore is corrupt. It's difficult to debug this from afar, but hopefully
you can help me out a little bit. (I'm also cc'ing the authors of this code
directly, to see if they can shed a little light on the matter.)
W/respect to your first patch that checks for a non-NULL bt->machdep
in x86_64_get_dumpfile_stack_frame(), can you tell me how that happened
exactly? Before that function was called, it should have come through
get_netdump_regs_x86_64(), which would -- or would not -- have set
bt->machdep here:
if (ELF_NOTES_VALID() &&
(bt->flags & BT_DUMPFILE_SEARCH) && DISKDUMP_DUMPFILE()
&&
(note = (Elf64_Nhdr *)
diskdump_get_prstatus_percpu(bt->tc->processor))) {
user_regs = get_regs_from_note((char *)note, &rip, &rsp);
if (CRASHDEBUG(1))
netdump_print("ELF prstatus rsp: %lx rip: %lx\n",
rsp, rip);
*rspp = rsp;
*ripp = rip;
if (*ripp && *rspp)
bt->flags |= BT_KDUMP_ELF_REGS;
bt->machdep = (void *)user_regs;
}
If it did *not* set bt->machdep above, then it must have been because
diskdump_get_prstatus_percpu() below returned a NULL pointer?
void *
diskdump_get_prstatus_percpu(int cpu)
{
return dd->nt_prstatus_percpu[cpu];
}
If you bring up crash with at least debug level 1 like this:
$ crash -d1 vmlinux vmcore
you will see a dump of the array of dd->nt_prstatus_percpu[] note
pointers. Alternatively during run-time, you can see the same output
by entering "help -n".
Can you confirm:
(1) what "cpu" value was passed to the function (presumably it was
legitimate), and
(2) whether dd->nt_prstatus_percpu[cpu] was NULL?
Secondly, w/respect to the bogus note->n_descsz value, was the note
pointer containing it one of those listed in the dd->nt_prstatus[]
array? If not, what was the "cpu" value passed to diskdump_get_prstatus_cpu()
that time?
And also, what is the output of:
crash> help -k | grep _map:
On my workstation, I see this:
crash> help -k | grep _map:
cpu_possible_map: 0 1 2 3 4 5 6 7
cpu_present_map: 0 1 2 3 4 5 6 7
cpu_online_map: 0 1 2 3 4 5 6 7
crash>
I'm wondering if your dump shows a system with some of the lower
cpus taken offline?
Thanks,
Dave
As far as the kernel is concerned,
2.6.32-131.0.15.el6.exp10.bz16586.x86_64 was a stock RH
2.6.32-131.0.15
with an added patch for handling an MD Raid bug (RHBZ-707268). Stratus
does load a driver to track dirty VM pages for harvesting purposes, but
does not change general VM behavior.
FWIW, this is the only vmcore that I've seen ELF note faulting or
invalid section numbers.
Thanks,
-- Joe
-----Original Message-----
From: crash-utility-bounces(a)redhat.com
[mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson
Sent: Wednesday, September 28, 2011 5:15 PM
To: Discussion list for crash utility usage,maintenance and
development
Subject: Re: [Crash-utility] Crash faults when determining panic task
Hi Joe,
It pretty clear it's due to this change in 5.1.5:
- Implemented the capability of using the NT_PRSTATUS ELF note data
that is saved in version 4 compressed kdump headers to determine the
starting stack and instruction pointer hooks for x86 and x86_64
backtraces when they cannot be determined in the traditional manners.
(wang.chao(a)cn.fujitsu.com, wency(a)cn.fujitsu.com)
What happens if you run it like so:
$ crash --no_elf_notes vmlinux vmcore
As far as this message:
WARNING: sparsemem: invalid section number: 137438888923
That should be outside the realm of Fujitsu's ELF notes patch. Does this kernel
have some kind of Stratus VM modification?
Dave
----- Original Message -----
>
> Crash faults when determining panic task
>
> I have a vmcore generated on RHEL6.1 that newer versions of crash
> have trouble analyzing (5.1.1-2.el6 seems to work ok) .
>
>
>
> I can provide additional binary files if needed, just let me know
> what convention best suits the list (ftp, private email attachment,
> etc.)
>
>
>
> Crash Version : OS: Result:
>
> crash 5.1.8 Debian wheezy faults
>
> crash 5.1.7-1.el6 RHEL6.2 Alpha faults
>
> crash 5.1.1-2.el6 RHEL6.1 ok
>
>
> Kernel:
>
> 2.6.32-131.0.15.el6.exp10.bz16586.x86_64 ( 2.6.32-131.0.15 + a fix
> for Red Hat bz - 707268)
>
>
> Interesting warnings when starting crash:
>
> WARNING: sparsemem: invalid section number: 137438888923
>
> WARNING: sparsemem: invalid section number: 137438888923
>
>
> First fault, null pointer deference:
>
> please wait... (determining panic task)
>
> Program received signal SIGSEGV, Segmentation fault.
>
> x86_64_get_dumpfile_stack_frame (rsp=0x7fffffffcc58,
> rip=0x7fffffffcc50,
>
> bt_in=0x7fffffffcce0) at x86_64.c:4183
>
> 4183 ur_rip = ULONG(user_regs +
>
> (gdb) p user_regs
>
> $1 = 0x0
>
>
> Workaround, check that bt->machdep is not NULL:
>
> diff -Nupr crash-5.1.8/x86_64.c crash-5.1.8.new/x86_64.c
>
> --- crash-5.1.8/x86_64.c 2011-09-16 15:01:12.000000000 -0400
>
> +++ crash-5.1.8.new/x86_64.c 2011-09-28 14:12:45.347188571 -0400
>
> @@ -4178,7 +4178,7 @@ x86_64_get_dumpfile_stack_frame(struct b
>
> goto skip_stage;
>
> }
>
> }
>
> - } else if (ELF_NOTES_VALID()) {
>
> + } else if (ELF_NOTES_VALID() && bt->machdep) {
>
> user_regs = bt->machdep;
>
> ur_rip = ULONG(user_regs +
>
> OFFSET(user_regs_struct_rip));
>
>
> Second fault, a curiously large n_descsz in elf note header:
>
> please wait... (determining panic task)
>
> Program received signal SIGSEGV, Segmentation fault.
>
> get_regs_from_note (note=0xd26472 "\b", ip=0x7fffffffc4e0,
> sp=0x7fffffffc4e8)
>
> at netdump.c:2221
>
> 2221 *sp = ULONG(user_regs + offset_sp);
>
> (gdb) p *(Elf64_Nhdr *)note
>
> $1 = {n_namesz = 8, n_descsz = 3438804992, n_type = 8}
>
>
> Workaround, do not attempt reading registers from elf notes (this
> chunk of code was not present in crash 5.1.1):
>
> diff -Nupr crash-5.1.8/netdump.c crash-5.1.8.new/netdump.c
>
> --- crash-5.1.8/netdump.c 2011-09-16 15:01:12.000000000 -0400
>
> +++ crash-5.1.8.new/netdump.c 2011-09-28 14:14:43.687183734 -0400
>
> @@ -2286,7 +2286,7 @@ get_netdump_regs_x86_64(struct bt_info *
>
>
>
> bt->machdep = (void *)user_regs;
>
> }
>
> -
>
> +#if 0
>
> if (ELF_NOTES_VALID() &&
>
> (bt->flags & BT_DUMPFILE_SEARCH) && DISKDUMP_DUMPFILE() &&
>
> (note = (Elf64_Nhdr *)
>
> @@ -2305,7 +2305,7 @@ get_netdump_regs_x86_64(struct bt_info *
>
>
>
> bt->machdep = (void *)user_regs;
>
> }
>
> -
>
> +#endif
>
> machdep->get_stack_frame(bt, ripp, rspp); }
>
>
> Given the warning messages at the beginning of the process, I'm sure
> if I' m dealing with a corrupted or incomplete vmcore image. Let me
> know what additional info could be useful if this seems worth
> debugging further.
>
>
>
> Thanks,
>
> -- Joe Lawrence
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
3:11 p.m.
Hi Dave,
I hope I have captured everything you asked for here, if remote debugging over e-mail is
too tedious, I can arrange to post a vmlinux/vmcore on our FTP site (roughly 600MB
together).
*** Setup some breakpoints to watch bt->machdep:
get_netdump_regs_x86_64(struct bt_info *bt, ulong *ripp, ulong *rspp)
{
...
if (((NETDUMP_DUMPFILE() || KDUMP_DUMPFILE()) &&
VALID_STRUCT(user_regs_struct) && (bt->task == tt->panic_task))
||
(KDUMP_DUMPFILE() && (kt->flags & DWARF_UNWIND) &&
(bt->flags & BT_DUMPFILE_SEARCH))) {
...
2287 bt->machdep = (void *)user_regs;
...
if (ELF_NOTES_VALID() &&
(bt->flags & BT_DUMPFILE_SEARCH) && DISKDUMP_DUMPFILE() &&
(note = (Elf64_Nhdr *)
diskdump_get_prstatus_percpu(bt->tc->processor))) {
...
2306 bt->machdep = (void *)user_regs;
...
(gdb) break get_netdump_regs_x86_64
Breakpoint 1 at 0x519740: file netdump.c, line 2238.
(gdb) break netdump.c:2287
Breakpoint 2 at 0x519970: file netdump.c, line 2287.
(gdb) break netdump.c:2306
Breakpoint 3 at 0x5199e7: file netdump.c, line 2306.
(gdb) r
please wait... (determining panic task)
Breakpoint 1, get_netdump_regs_x86_64 (bt=0x7fffffffcd70, ripp=0x7fffffffcce0,
rspp=0x7fffffffcce8) at netdump.c:2238
2238 {
(gdb) c
Continuing.
Breakpoint 3, get_netdump_regs_x86_64 (bt=0x7fffffffcd70, ripp=0x7fffffffcce0,
rspp=0x7fffffffcce8) at netdump.c:2306
2306 bt->machdep = (void *)user_regs;
(gdb) p user_regs
$1 = 0xd14084 ""
(gdb) c
Continuing.
Breakpoint 1, get_netdump_regs_x86_64 (bt=0x7fffffffcd70, ripp=0x7fffffffcce0,
rspp=0x7fffffffcce8) at netdump.c:2238
2238 {
(gdb) c
Continuing.
Program received signal SIGSEGV, Segmentation fault.
x86_64_get_dumpfile_stack_frame (rsp=0x7fffffffcce8, rip=0x7fffffffcce0,
bt_in=0x7fffffffcd70) at x86_64.c:4183
4183 ur_rip = ULONG(user_regs +
*** So in its second invocation, get_netdump_regs_x86_64() never sets bt->machdep (only
breakpoint 1 fired)
*** Let's see what diskdump_get_prstatus_percpu() is returning
(gdb) break diskdump_get_prstatus_percpu
Breakpoint 1 at 0x526070: file diskdump.c, line 1451.
(gdb) r
please wait... (determining panic task)
Breakpoint 1, diskdump_get_prstatus_percpu (cpu=0) at diskdump.c:1451
1451 return dd->nt_prstatus_percpu[cpu];
(gdb) display dd->nt_prstatus_percpu[0]@16
1: dd->nt_prstatus_percpu[0]@16 = {0xd1c000, 0x0, 0x0, 0xd26472, 0xbf35ab2,
0xd26472, 0x200000012, 0xd1c850, 0xd1c600, 0x1010000012b,
0xffffffff814e4fa0, 0x14e4fa0, 0x4270, 0x0, 0x0, 0x0}
(gdb) c
Continuing.
Breakpoint 1, diskdump_get_prstatus_percpu (cpu=1) at diskdump.c:1451
1451 return dd->nt_prstatus_percpu[cpu];
1: dd->nt_prstatus_percpu[0]@16 = {0xd1c000, 0x0, 0x0, 0xd26472, 0xbf35ab2,
0xd26472, 0x200000012, 0xd1c850, 0xd1c600, 0x1010000012b,
0xffffffff814e4fa0, 0x14e4fa0, 0x4270, 0x0, 0x0, 0x0}
*** See crash -d1 vmlinux vmcore output at the bottom of the mail, particularly the part
that says...
crash: page excluded: kernel virtual address: ffffffff81bb3b00 type: "cpu number
(per_cpu)"
crash: get_cpus_present: present: 16
*** Bogus note->n_descsz value
*** Apply first patch to get us further into ELF Note processing
From inside netdump.c :: get_regs_from_note() at the point of the
fault, I don't see dd->nt_prstatus[], for dd is now type *diskdump_data... The
*note passed in can be found in dd->nt_prstatus_percpu[] however...
please wait... (determining panic task)
Program received signal SIGSEGV, Segmentation fault.
get_regs_from_note (note=0xd26472 "\b", ip=0x7fffffffc590, sp=0x7fffffffc598)
at netdump.c:2221
2221 *sp = ULONG(user_regs + offset_sp);
(gdb) p/x *((Elf64_Nhdr *)note)
$1 = {n_namesz = 0x8, n_descsz = 0xccf80000, n_type = 0x8}
(gdb) p dd->nt_prstatus_percpu[0]@16
$2 = {0xd1c000, 0x0, 0x0, 0xd26472, 0xbf35ab2, 0xd26472, 0x200000012,
0xd1c850, 0xd1c600, 0x1010000012b, 0xffffffff814e4fa0, 0x14e4fa0, 0x4270,
0x0, 0x0, 0x0}
(gdb) ptype dd
type = struct diskdump_data {
char *filename;
ulong flags;
int dfd;
FILE *ofp;
int machine_type;
struct disk_dump_header *header;
struct disk_dump_sub_header *sub_header;
struct kdump_sub_header *sub_header_kdump;
size_t data_offset;
int block_size;
int block_shift;
char *bitmap;
int bitmap_len;
char *dumpable_bitmap;
int byte;
int bit;
char *compressed_page;
char *curbufptr;
unsigned char *notes_buf;
void **nt_prstatus_percpu;
uint num_prstatus_notes;
struct page_cache_hdr page_cache_hdr[16];
char *page_cache_buf;
int evict_index;
ulong evictions;
ulong cached_reads;
ulong *valid_pages;
ulong accesses;
} *
*** Unpatched crash -d1 vmlinux vmcore output:
crash 5.1.8
Copyright (C) 2002-2011 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
compressed kdump: header->utsname.machine: x86_64
diskdump_data:
filename: vmcore
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
dfd: 3
ofp: 0
machine_type: 62 (EM_X86_64)
header: 2cc1fe0
signature: "KDUMP "
header_version: 4
utsname:
sysname: Linux
nodename: bahamut.mno.stratus.com
release: 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
version: #1 SMP Thu Jun 16 13:13:45 EDT 2011
machine: x86_64
domainname: sraeng
timestamp:
tv_sec: 4e4fe6e3
tv_usec: 0
status: 0 ()
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 288
max_mapnr: 4718592
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 1
tasks[nr_cpus]: 0
sub_header: 0 (n/a)
sub_header_kdump: 2cc2ff0
phys_base: 0
dump_level: 31 (0x1f)
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
offset_vmcoreinfo: 11bc
size_vmcoreinfo: 1392
OSRELEASE=2.6.32-131.0.15.el6.exp10.bz16586.x86_64
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffffffff81a2e8c0
SYMBOL(node_online_map)=ffffffff81ba0860
SYMBOL(swapper_pg_dir)=ffffffff81a25000
SYMBOL(_stext)=ffffffff81000198
SYMBOL(vmlist)=ffffffff81ee60b8
SYMBOL(mem_section)=ffffffff81ef03c0
LENGTH(mem_section)=4096
SIZE(mem_section)=32
OFFSET(mem_section.section_mem_map)=0
SIZE(page)=56
SIZE(pglist_data)=212416
SIZE(zone)=34496
SIZE(free_area)=88
SIZE(list_head)=16
SIZE(nodemask_t)=64
OFFSET(page.flags)=0
OFFSET(page._count)=8
OFFSET(page.mapping)=24
OFFSET(page.lru)=40
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=212288
OFFSET(pglist_data.node_start_pfn)=212312
OFFSET(pglist_data.node_spanned_pages)=212328
OFFSET(pglist_data.node_id)=212336
OFFSET(zone.free_area)=32864
OFFSET(zone.vm_stat)=34032
OFFSET(zone.spanned_pages)=34344
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=8
OFFSET(vm_struct.addr)=8
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=ffffffff81a37210
SYMBOL(log_end)=ffffffff81d5b820
SYMBOL(log_buf_len)=ffffffff81a37208
SYMBOL(logged_chars)=ffffffff81ddb920
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=11
NUMBER(PG_swapcache)=16
SYMBOL(phys_base)=ffffffff81a2d010
SYMBOL(init_level4_pgt)=ffffffff81a25000
SYMBOL(node_data)=ffffffff81b9cda0
LENGTH(node_data)=512
CRASHTIME=1313859299
offset_note: 1040
size_note: 1780
num_prstatus_notes: 1
notes_buf: 2cc4000
notes[0]: 2cc4000
NT_PRSTATUS_offset: 1040
data_offset: 122000
block_size: 4096
block_shift: 12
bitmap: 7fa5296fc010
bitmap_len: 1179648
dumpable_bitmap: 7fa528890010
byte: 0
bit: 0
compressed_page: 2cdeb30
curbufptr: 0
page_cache_hdr[0]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cceb20
pg_hit_count: 0
page_cache_hdr[1]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2ccfb20
pg_hit_count: 0
page_cache_hdr[2]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cd0b20
pg_hit_count: 0
page_cache_hdr[3]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cd1b20
pg_hit_count: 0
page_cache_hdr[4]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cd2b20
pg_hit_count: 0
page_cache_hdr[5]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cd3b20
pg_hit_count: 0
page_cache_hdr[6]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cd4b20
pg_hit_count: 0
page_cache_hdr[7]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cd5b20
pg_hit_count: 0
page_cache_hdr[8]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cd6b20
pg_hit_count: 0
page_cache_hdr[9]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cd7b20
pg_hit_count: 0
page_cache_hdr[10]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cd8b20
pg_hit_count: 0
page_cache_hdr[11]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cd9b20
pg_hit_count: 0
page_cache_hdr[12]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cdab20
pg_hit_count: 0
page_cache_hdr[13]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cdbb20
pg_hit_count: 0
page_cache_hdr[14]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cdcb20
pg_hit_count: 0
page_cache_hdr[15]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 2cddb20
pg_hit_count: 0
page_cache_buf: 2cceb20
evict_index: 0
evictions: 0
accesses: 0
cached_reads: 0
valid_pages: 2ccc710
crash: pv_init_ops exists: ARCH_PVOPS
compressed kdump: phys_base: 0
gdb vmlinux
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
cpu_possible_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
cpu_present_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
cpu_online_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
base kernel version: 2.6.32
verify_namelist:
dumpfile /proc/version:
Linux version 2.6.32-131.0.15.el6.exp10.bz16586.x86_64 (root(a)druk.mno.stratus.com) (gcc
version 4.4.5 20110214 (Red Hat 4.4.5-6) (GCC) ) #1 SMP Thu Jun 16 13:13:45 EDT 2011
vmlinux:
Linux version 2.6.32-131.0.15.el6.exp10.bz16586.x86_64 (root(a)druk.mno.stratus.com) (gcc
version 4.4.5 20110214 (Red Hat 4.4.5-6) (GCC) ) #1 SMP Thu Jun 16 13:13:45 EDT 2011
crash: page excluded: kernel virtual address: ffffffff81bb3b00 type: "cpu number
(per_cpu)"
crash: get_cpus_present: present: 16
crash: page excluded: kernel virtual address: ffffffff81bb3b00 type: "cpu number
(per_cpu)"
crash: get_cpus_present: present: 16
IRQ stack link register: undetermined
PAGESIZE=4096
mem_section_size = 32768
NR_SECTION_ROOTS = 4096
NR_MEM_SECTIONS = 524288
SECTIONS_PER_ROOT = 128
SECTION_ROOT_MASK = 0x7f
PAGES_PER_SECTION = 32768
node_online_map: [3, 0, 0, 0, 0, 0, 0, 0] -> nodes online: 2
node_table[0]:
id: 0
pgdat: ffff880000020040
size: 0
present: 0
mem_map: ffffea0000000000
start_paddr: 0
start_mapnr: 0
WARNING: sparsemem: invalid section number: 137438888923
WARNING: sparsemem: invalid section number: 137438888923
crash: invalid kernel virtual address: 0 type: "readstring characters"
crash: invalid kernel virtual address: 0 type: "readstring characters"
node_table[1]:
id: 1
pgdat: ffff880280000040
size: 2097152
present: 2097152
mem_map: ffffea0008c00000
start_paddr: 280000000
start_mapnr: 2621440
NOTE: page_hash_table does not exist in this kernel
^Mplease wait... (gathering kmem slab cache data)
kmem_cache_downsize: SIZE(kmem_cache_s): 36968 cache_cache.buffer_size: 32896
kmem_cache_downsize: nr_node_ids: 2
^M ^MNOTE: unwind_table structure has
changed, or does not exist in this kernel
init_unwind_table: DWARF_UNWIND_EH_FRAME
^Mplease wait... (gathering module symbol data)^M
^M^Mplease wait... (gathering task table data)^M
^Mcrash: get_cpus_online: online: 16
^Mplease wait... (determining panic task)
crash: get_active_set_panic_task: failed
Thanks,
-- Joe Lawrence
4:03 p.m.
----- Original Message -----
Hi Dave,
I hope I have captured everything you asked for here, if remote
debugging over e-mail is too tedious, I can arrange to post a
vmlinux/vmcore on our FTP site (roughly 600MB together).
Sure, you can do that if you'd like.
But anyway, the crash -d1 output is illuminating. You've got a 16-cpu
system, with all cpus online. But the compressed kdump header only
saw 1 cpu when it was created:
header: 2cc1fe0
signature: "KDUMP "
header_version: 4
utsname:
sysname: Linux
nodename: bahamut.mno.stratus.com
release: 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
version: #1 SMP Thu Jun 16 13:13:45 EDT 2011
machine: x86_64
domainname: sraeng
timestamp:
tv_sec: 4e4fe6e3
tv_usec: 0
status: 0 ()
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 288
max_mapnr: 4718592
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 1 <== should be 16
tasks[nr_cpus]: 0
and farther on, here is the array of note pointers that I
was asking about:
num_prstatus_notes: 1
notes_buf: 2cc4000
notes[0]: 2cc4000
Since dumpfile header's nr_cpus was 1, the array has only one entry.
I cannot explain that. But the crash utility can only deal with what
it finds in the dumpfile.
Furthermore, these two error messages indicate that memory containing
per-cpu data was "excluded" are of prime importance here:
crash: page excluded: kernel virtual address: ffffffff81bb3b00 type:
"cpu number (per_cpu)"
crash: page excluded: kernel virtual address: ffffffff81bb3b00 type: "cpu number
(per_cpu)"
The fact that the page was specifically "page excluded" is troubling,
because it should *never* have been filtered by makedumpfile -d<level>.
But, since the dumpfile indicates that the crucial per-cpu page was filtered,
so there's nothing that the crash utility can do about it.
I'm guessing that, even though you are able to get to a prompt
with --no_elf_notes, any command that depends upon per-cpu data
would fail. Although, it might be interesting to know *which*
cpu was in play when those two error messages were displayed
in x86_64_per_cpu_init() and x86_64_get_smp_cpus(). There is
a loop in both functions -- can you dump out which cpu's
per-cpu data was inaccessible?
Thanks,
Dave
>
>
> *** Setup some breakpoints to watch bt->machdep:
>
> get_netdump_regs_x86_64(struct bt_info *bt, ulong *ripp, ulong *rspp)
> {
> ...
>
> if (((NETDUMP_DUMPFILE() || KDUMP_DUMPFILE()) &&
> VALID_STRUCT(user_regs_struct) && (bt->task == tt->panic_task)) ||
> (KDUMP_DUMPFILE() && (kt->flags & DWARF_UNWIND) &&
> (bt->flags & BT_DUMPFILE_SEARCH))) {
> ...
> 2287 bt->machdep = (void *)user_regs;
> ...
>
> if (ELF_NOTES_VALID() &&
> (bt->flags & BT_DUMPFILE_SEARCH) && DISKDUMP_DUMPFILE() &&
> (note = (Elf64_Nhdr *)
> diskdump_get_prstatus_percpu(bt->tc->processor))) {
> ...
> 2306 bt->machdep = (void *)user_regs;
> ...
>
> (gdb) break get_netdump_regs_x86_64
> Breakpoint 1 at 0x519740: file netdump.c, line 2238.
> (gdb) break netdump.c:2287
> Breakpoint 2 at 0x519970: file netdump.c, line 2287.
> (gdb) break netdump.c:2306
> Breakpoint 3 at 0x5199e7: file netdump.c, line 2306.
> (gdb) r
>
> please wait... (determining panic task)
> Breakpoint 1, get_netdump_regs_x86_64 (bt=0x7fffffffcd70,
> ripp=0x7fffffffcce0,
> rspp=0x7fffffffcce8) at netdump.c:2238
> 2238 {
> (gdb) c
> Continuing.
>
> Breakpoint 3, get_netdump_regs_x86_64 (bt=0x7fffffffcd70,
> ripp=0x7fffffffcce0,
> rspp=0x7fffffffcce8) at netdump.c:2306
> 2306 bt->machdep = (void *)user_regs;
> (gdb) p user_regs
> $1 = 0xd14084 ""
> (gdb) c
> Continuing.
>
> Breakpoint 1, get_netdump_regs_x86_64 (bt=0x7fffffffcd70,
> ripp=0x7fffffffcce0,
> rspp=0x7fffffffcce8) at netdump.c:2238
> 2238 {
> (gdb) c
> Continuing.
>
> Program received signal SIGSEGV, Segmentation fault.
> x86_64_get_dumpfile_stack_frame (rsp=0x7fffffffcce8,
> rip=0x7fffffffcce0,
> bt_in=0x7fffffffcd70) at x86_64.c:4183
> 4183 ur_rip = ULONG(user_regs +
>
>
> *** So in its second invocation, get_netdump_regs_x86_64() never sets
> bt->machdep (only breakpoint 1 fired)
>
> *** Let's see what diskdump_get_prstatus_percpu() is returning
>
> (gdb) break diskdump_get_prstatus_percpu
> Breakpoint 1 at 0x526070: file diskdump.c, line 1451.
> (gdb) r
> please wait... (determining panic task)
> Breakpoint 1, diskdump_get_prstatus_percpu (cpu=0) at diskdump.c:1451
> 1451 return dd->nt_prstatus_percpu[cpu];
> (gdb) display dd->nt_prstatus_percpu[0]@16
> 1: dd->nt_prstatus_percpu[0]@16 = {0xd1c000, 0x0, 0x0, 0xd26472,
> 0xbf35ab2,
> 0xd26472, 0x200000012, 0xd1c850, 0xd1c600, 0x1010000012b,
> 0xffffffff814e4fa0, 0x14e4fa0, 0x4270, 0x0, 0x0, 0x0}
> (gdb) c
> Continuing.
>
> Breakpoint 1, diskdump_get_prstatus_percpu (cpu=1) at diskdump.c:1451
> 1451 return dd->nt_prstatus_percpu[cpu];
> 1: dd->nt_prstatus_percpu[0]@16 = {0xd1c000, 0x0, 0x0, 0xd26472,
> 0xbf35ab2,
> 0xd26472, 0x200000012, 0xd1c850, 0xd1c600, 0x1010000012b,
> 0xffffffff814e4fa0, 0x14e4fa0, 0x4270, 0x0, 0x0, 0x0}
>
>
> *** See crash -d1 vmlinux vmcore output at the bottom of the mail,
> particularly the part that says...
>
> crash: page excluded: kernel virtual address: ffffffff81bb3b00 type:
> "cpu number (per_cpu)"
> crash: get_cpus_present: present: 16
>
>
>
> *** Bogus note->n_descsz value
> *** Apply first patch to get us further into ELF Note processing
>
> >From inside netdump.c :: get_regs_from_note() at the point of the
> >fault, I don't see dd->nt_prstatus[], for dd is now type
> >*diskdump_data... The *note passed in can be found in
> >dd->nt_prstatus_percpu[] however...
>
> please wait... (determining panic task)
> Program received signal SIGSEGV, Segmentation fault.
> get_regs_from_note (note=0xd26472 "\b", ip=0x7fffffffc590,
> sp=0x7fffffffc598)
> at netdump.c:2221
> 2221 *sp = ULONG(user_regs + offset_sp);
> (gdb) p/x *((Elf64_Nhdr *)note)
> $1 = {n_namesz = 0x8, n_descsz = 0xccf80000, n_type = 0x8}
> (gdb) p dd->nt_prstatus_percpu[0]@16
> $2 = {0xd1c000, 0x0, 0x0, 0xd26472, 0xbf35ab2, 0xd26472, 0x200000012,
> 0xd1c850, 0xd1c600, 0x1010000012b, 0xffffffff814e4fa0, 0x14e4fa0,
> 0x4270,
> 0x0, 0x0, 0x0}
> (gdb) ptype dd
> type = struct diskdump_data {
> char *filename;
> ulong flags;
> int dfd;
> FILE *ofp;
> int machine_type;
> struct disk_dump_header *header;
> struct disk_dump_sub_header *sub_header;
> struct kdump_sub_header *sub_header_kdump;
> size_t data_offset;
> int block_size;
> int block_shift;
> char *bitmap;
> int bitmap_len;
> char *dumpable_bitmap;
> int byte;
> int bit;
> char *compressed_page;
> char *curbufptr;
> unsigned char *notes_buf;
> void **nt_prstatus_percpu;
> uint num_prstatus_notes;
> struct page_cache_hdr page_cache_hdr[16];
> char *page_cache_buf;
> int evict_index;
> ulong evictions;
> ulong cached_reads;
> ulong *valid_pages;
> ulong accesses;
> } *
>
>
>
> *** Unpatched crash -d1 vmlinux vmcore output:
>
> crash 5.1.8
> Copyright (C) 2002-2011 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> compressed kdump: header->utsname.machine: x86_64
> diskdump_data:
> filename: vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 3
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: 2cc1fe0
> signature: "KDUMP "
> header_version: 4
> utsname:
> sysname: Linux
> nodename: bahamut.mno.stratus.com
> release: 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> version: #1 SMP Thu Jun 16 13:13:45 EDT 2011
> machine: x86_64
> domainname: sraeng
> timestamp:
> tv_sec: 4e4fe6e3
> tv_usec: 0
> status: 0 ()
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 288
> max_mapnr: 4718592
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1
> tasks[nr_cpus]: 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 2cc2ff0
> phys_base: 0
> dump_level: 31 (0x1f)
>
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
> offset_vmcoreinfo: 11bc
> size_vmcoreinfo: 1392
> OSRELEASE=2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> PAGESIZE=4096
> SYMBOL(init_uts_ns)=ffffffff81a2e8c0
> SYMBOL(node_online_map)=ffffffff81ba0860
> SYMBOL(swapper_pg_dir)=ffffffff81a25000
> SYMBOL(_stext)=ffffffff81000198
> SYMBOL(vmlist)=ffffffff81ee60b8
> SYMBOL(mem_section)=ffffffff81ef03c0
> LENGTH(mem_section)=4096
> SIZE(mem_section)=32
> OFFSET(mem_section.section_mem_map)=0
> SIZE(page)=56
> SIZE(pglist_data)=212416
> SIZE(zone)=34496
> SIZE(free_area)=88
> SIZE(list_head)=16
> SIZE(nodemask_t)=64
> OFFSET(page.flags)=0
> OFFSET(page._count)=8
> OFFSET(page.mapping)=24
> OFFSET(page.lru)=40
> OFFSET(pglist_data.node_zones)=0
> OFFSET(pglist_data.nr_zones)=212288
> OFFSET(pglist_data.node_start_pfn)=212312
> OFFSET(pglist_data.node_spanned_pages)=212328
> OFFSET(pglist_data.node_id)=212336
> OFFSET(zone.free_area)=32864
> OFFSET(zone.vm_stat)=34032
> OFFSET(zone.spanned_pages)=34344
> OFFSET(free_area.free_list)=0
> OFFSET(list_head.next)=0
> OFFSET(list_head.prev)=8
> OFFSET(vm_struct.addr)=8
> LENGTH(zone.free_area)=11
> SYMBOL(log_buf)=ffffffff81a37210
> SYMBOL(log_end)=ffffffff81d5b820
> SYMBOL(log_buf_len)=ffffffff81a37208
> SYMBOL(logged_chars)=ffffffff81ddb920
> LENGTH(free_area.free_list)=5
> NUMBER(NR_FREE_PAGES)=0
> NUMBER(PG_lru)=5
> NUMBER(PG_private)=11
> NUMBER(PG_swapcache)=16
> SYMBOL(phys_base)=ffffffff81a2d010
> SYMBOL(init_level4_pgt)=ffffffff81a25000
> SYMBOL(node_data)=ffffffff81b9cda0
> LENGTH(node_data)=512
> CRASHTIME=1313859299
> offset_note: 1040
> size_note: 1780
num_prstatus_notes: 1
notes_buf: 2cc4000
notes[0]: 2cc4000
> NT_PRSTATUS_offset: 1040
>
> data_offset: 122000
> block_size: 4096
> block_shift: 12
> bitmap: 7fa5296fc010
> bitmap_len: 1179648
> dumpable_bitmap: 7fa528890010
> byte: 0
> bit: 0
> compressed_page: 2cdeb30
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cceb20
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2ccfb20
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd0b20
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd1b20
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd2b20
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd3b20
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd4b20
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd5b20
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd6b20
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd7b20
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd8b20
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd9b20
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cdab20
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cdbb20
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cdcb20
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cddb20
> pg_hit_count: 0
>
> page_cache_buf: 2cceb20
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: 2ccc710
> crash: pv_init_ops exists: ARCH_PVOPS
> compressed kdump: phys_base: 0
> gdb vmlinux
> GNU gdb (GDB) 7.0
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> cpu_possible_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> cpu_present_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> cpu_online_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> base kernel version: 2.6.32
> verify_namelist:
> dumpfile /proc/version:
> Linux version 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> (root(a)druk.mno.stratus.com) (gcc version 4.4.5 20110214 (Red Hat
> 4.4.5-6) (GCC) ) #1 SMP Thu Jun 16 13:13:45 EDT 2011
> vmlinux:
> Linux version 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> (root(a)druk.mno.stratus.com) (gcc version 4.4.5 20110214 (Red Hat
> 4.4.5-6) (GCC) ) #1 SMP Thu Jun 16 13:13:45 EDT 2011
>
> crash: page excluded: kernel virtual address: ffffffff81bb3b00 type:
> "cpu number (per_cpu)"
> crash: get_cpus_present: present: 16
> crash: page excluded: kernel virtual address: ffffffff81bb3b00 type:
> "cpu number (per_cpu)"
> crash: get_cpus_present: present: 16
> IRQ stack link register: undetermined
> PAGESIZE=4096
> mem_section_size = 32768
> NR_SECTION_ROOTS = 4096
> NR_MEM_SECTIONS = 524288
> SECTIONS_PER_ROOT = 128
> SECTION_ROOT_MASK = 0x7f
> PAGES_PER_SECTION = 32768
> node_online_map: [3, 0, 0, 0, 0, 0, 0, 0] -> nodes online: 2
> node_table[0]:
> id: 0
> pgdat: ffff880000020040
> size: 0
> present: 0
> mem_map: ffffea0000000000
> start_paddr: 0
> start_mapnr: 0
> WARNING: sparsemem: invalid section number: 137438888923
> WARNING: sparsemem: invalid section number: 137438888923
> crash: invalid kernel virtual address: 0 type: "readstring
> characters"
> crash: invalid kernel virtual address: 0 type: "readstring
> characters"
> node_table[1]:
> id: 1
> pgdat: ffff880280000040
> size: 2097152
> present: 2097152
> mem_map: ffffea0008c00000
> start_paddr: 280000000
> start_mapnr: 2621440
> NOTE: page_hash_table does not exist in this kernel
> ^Mplease wait... (gathering kmem slab cache data)
> kmem_cache_downsize: SIZE(kmem_cache_s): 36968
> cache_cache.buffer_size: 32896
> kmem_cache_downsize: nr_node_ids: 2
> ^M ^MNOTE: unwind_table structure has changed, or does not exist in
> this kernel
> init_unwind_table: DWARF_UNWIND_EH_FRAME
> ^Mplease wait... (gathering module symbol data)^M ^M^Mplease wait...
> (gathering task table data)^M ^Mcrash: get_cpus_online: online: 16
> ^Mplease wait... (determining panic task)
> crash: get_active_set_panic_task: failed
>
>
> Thanks,
>
> -- Joe Lawrence
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
6:19 p.m.
Hi Dave,
Before the patch mentioned below, makedumpfile hardcoded nr_cpus to 1 in
the header.
..........................
author Ken'ichi Ohmichi <oomichi(a)mxs.nes.nec.co.jp>
Thu, 11 Nov 2010 03:53:16 +0000 (12:53 +0900)
Before applying this patch, makedumpfile sets "1" to nr_cpus in kdump
main header always even if a machine has multiple cpus.
As the result, the subcommand "help -n" of the crash utility prints
an invalid value "1" as nr_cpus.
...........................
Could this be part of Joe's issue?
Thanks,
Jeff Hagen
-----Original Message-----
From: crash-utility-bounces(a)redhat.com
[mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson
Sent: Thursday, September 29, 2011 2:04 PM
To: Discussion list for crash utility usage,maintenance and development
Subject: Re: [Crash-utility] Crash faults when determining panic task
----- Original Message -----
Hi Dave,
I hope I have captured everything you asked for here, if remote
debugging over e-mail is too tedious, I can arrange to post a
vmlinux/vmcore on our FTP site (roughly 600MB together).
Sure, you can do that if you'd like.
But anyway, the crash -d1 output is illuminating. You've got a 16-cpu
system, with all cpus online. But the compressed kdump header only
saw 1 cpu when it was created:
header: 2cc1fe0
signature: "KDUMP "
header_version: 4
utsname:
sysname: Linux
nodename: bahamut.mno.stratus.com
release: 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
version: #1 SMP Thu Jun 16 13:13:45 EDT 2011
machine: x86_64
domainname: sraeng
timestamp:
tv_sec: 4e4fe6e3
tv_usec: 0
status: 0 ()
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 288
max_mapnr: 4718592
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 1 <== should be 16
tasks[nr_cpus]: 0
and farther on, here is the array of note pointers that I
was asking about:
num_prstatus_notes: 1
notes_buf: 2cc4000
notes[0]: 2cc4000
Since dumpfile header's nr_cpus was 1, the array has only one entry.
I cannot explain that. But the crash utility can only deal with what
it finds in the dumpfile.
Furthermore, these two error messages indicate that memory containing
per-cpu data was "excluded" are of prime importance here:
crash: page excluded: kernel virtual address: ffffffff81bb3b00 type:
"cpu number (per_cpu)"
crash: page excluded: kernel virtual address: ffffffff81bb3b00 type:
"cpu number (per_cpu)"
The fact that the page was specifically "page excluded" is troubling,
because it should *never* have been filtered by makedumpfile -d<level>.
But, since the dumpfile indicates that the crucial per-cpu page was
filtered,
so there's nothing that the crash utility can do about it.
I'm guessing that, even though you are able to get to a prompt
with --no_elf_notes, any command that depends upon per-cpu data
would fail. Although, it might be interesting to know *which*
cpu was in play when those two error messages were displayed
in x86_64_per_cpu_init() and x86_64_get_smp_cpus(). There is
a loop in both functions -- can you dump out which cpu's
per-cpu data was inaccessible?
Thanks,
Dave
>
>
> *** Setup some breakpoints to watch bt->machdep:
>
> get_netdump_regs_x86_64(struct bt_info *bt, ulong *ripp, ulong *rspp)
> {
> ...
>
> if (((NETDUMP_DUMPFILE() || KDUMP_DUMPFILE()) &&
> VALID_STRUCT(user_regs_struct) && (bt->task == tt->panic_task)) ||
> (KDUMP_DUMPFILE() && (kt->flags & DWARF_UNWIND) &&
> (bt->flags & BT_DUMPFILE_SEARCH))) {
> ...
> 2287 bt->machdep = (void *)user_regs;
> ...
>
> if (ELF_NOTES_VALID() &&
> (bt->flags & BT_DUMPFILE_SEARCH) && DISKDUMP_DUMPFILE() &&
> (note = (Elf64_Nhdr *)
> diskdump_get_prstatus_percpu(bt->tc->processor))) {
> ...
> 2306 bt->machdep = (void *)user_regs;
> ...
>
> (gdb) break get_netdump_regs_x86_64
> Breakpoint 1 at 0x519740: file netdump.c, line 2238.
> (gdb) break netdump.c:2287
> Breakpoint 2 at 0x519970: file netdump.c, line 2287.
> (gdb) break netdump.c:2306
> Breakpoint 3 at 0x5199e7: file netdump.c, line 2306.
> (gdb) r
>
> please wait... (determining panic task)
> Breakpoint 1, get_netdump_regs_x86_64 (bt=0x7fffffffcd70,
> ripp=0x7fffffffcce0,
> rspp=0x7fffffffcce8) at netdump.c:2238
> 2238 {
> (gdb) c
> Continuing.
>
> Breakpoint 3, get_netdump_regs_x86_64 (bt=0x7fffffffcd70,
> ripp=0x7fffffffcce0,
> rspp=0x7fffffffcce8) at netdump.c:2306
> 2306 bt->machdep = (void *)user_regs;
> (gdb) p user_regs
> $1 = 0xd14084 ""
> (gdb) c
> Continuing.
>
> Breakpoint 1, get_netdump_regs_x86_64 (bt=0x7fffffffcd70,
> ripp=0x7fffffffcce0,
> rspp=0x7fffffffcce8) at netdump.c:2238
> 2238 {
> (gdb) c
> Continuing.
>
> Program received signal SIGSEGV, Segmentation fault.
> x86_64_get_dumpfile_stack_frame (rsp=0x7fffffffcce8,
> rip=0x7fffffffcce0,
> bt_in=0x7fffffffcd70) at x86_64.c:4183
> 4183 ur_rip = ULONG(user_regs +
>
>
> *** So in its second invocation, get_netdump_regs_x86_64() never sets
> bt->machdep (only breakpoint 1 fired)
>
> *** Let's see what diskdump_get_prstatus_percpu() is returning
>
> (gdb) break diskdump_get_prstatus_percpu
> Breakpoint 1 at 0x526070: file diskdump.c, line 1451.
> (gdb) r
> please wait... (determining panic task)
> Breakpoint 1, diskdump_get_prstatus_percpu (cpu=0) at diskdump.c:1451
> 1451 return dd->nt_prstatus_percpu[cpu];
> (gdb) display dd->nt_prstatus_percpu[0]@16
> 1: dd->nt_prstatus_percpu[0]@16 = {0xd1c000, 0x0, 0x0, 0xd26472,
> 0xbf35ab2,
> 0xd26472, 0x200000012, 0xd1c850, 0xd1c600, 0x1010000012b,
> 0xffffffff814e4fa0, 0x14e4fa0, 0x4270, 0x0, 0x0, 0x0}
> (gdb) c
> Continuing.
>
> Breakpoint 1, diskdump_get_prstatus_percpu (cpu=1) at diskdump.c:1451
> 1451 return dd->nt_prstatus_percpu[cpu];
> 1: dd->nt_prstatus_percpu[0]@16 = {0xd1c000, 0x0, 0x0, 0xd26472,
> 0xbf35ab2,
> 0xd26472, 0x200000012, 0xd1c850, 0xd1c600, 0x1010000012b,
> 0xffffffff814e4fa0, 0x14e4fa0, 0x4270, 0x0, 0x0, 0x0}
>
>
> *** See crash -d1 vmlinux vmcore output at the bottom of the mail,
> particularly the part that says...
>
crash: page excluded: kernel virtual address: ffffffff81bb3b00 type:
> "cpu number (per_cpu)"
> crash: get_cpus_present: present: 16
>
>
>
> *** Bogus note->n_descsz value
> *** Apply first patch to get us further into ELF Note processing
>
> >From inside netdump.c :: get_regs_from_note() at the point of the
> >fault, I don't see dd->nt_prstatus[], for dd is now type
> >*diskdump_data... The *note passed in can be found in
> >dd->nt_prstatus_percpu[] however...
>
> please wait... (determining panic task)
> Program received signal SIGSEGV, Segmentation fault.
> get_regs_from_note (note=0xd26472 "\b", ip=0x7fffffffc590,
> sp=0x7fffffffc598)
> at netdump.c:2221
> 2221 *sp = ULONG(user_regs + offset_sp);
> (gdb) p/x *((Elf64_Nhdr *)note)
> $1 = {n_namesz = 0x8, n_descsz = 0xccf80000, n_type = 0x8}
> (gdb) p dd->nt_prstatus_percpu[0]@16
> $2 = {0xd1c000, 0x0, 0x0, 0xd26472, 0xbf35ab2, 0xd26472, 0x200000012,
> 0xd1c850, 0xd1c600, 0x1010000012b, 0xffffffff814e4fa0, 0x14e4fa0,
> 0x4270,
> 0x0, 0x0, 0x0}
> (gdb) ptype dd
> type = struct diskdump_data {
> char *filename;
> ulong flags;
> int dfd;
> FILE *ofp;
> int machine_type;
> struct disk_dump_header *header;
> struct disk_dump_sub_header *sub_header;
> struct kdump_sub_header *sub_header_kdump;
> size_t data_offset;
> int block_size;
> int block_shift;
> char *bitmap;
> int bitmap_len;
> char *dumpable_bitmap;
> int byte;
> int bit;
> char *compressed_page;
> char *curbufptr;
> unsigned char *notes_buf;
> void **nt_prstatus_percpu;
> uint num_prstatus_notes;
> struct page_cache_hdr page_cache_hdr[16];
> char *page_cache_buf;
> int evict_index;
> ulong evictions;
> ulong cached_reads;
> ulong *valid_pages;
> ulong accesses;
> } *
>
>
>
> *** Unpatched crash -d1 vmlinux vmcore output:
>
> crash 5.1.8
> Copyright (C) 2002-2011 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> compressed kdump: header->utsname.machine: x86_64
> diskdump_data:
> filename: vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 3
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: 2cc1fe0
> signature: "KDUMP "
> header_version: 4
> utsname:
> sysname: Linux
> nodename: bahamut.mno.stratus.com
> release: 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> version: #1 SMP Thu Jun 16 13:13:45 EDT 2011
> machine: x86_64
> domainname: sraeng
> timestamp:
> tv_sec: 4e4fe6e3
> tv_usec: 0
> status: 0 ()
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 288
> max_mapnr: 4718592
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1
> tasks[nr_cpus]: 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 2cc2ff0
> phys_base: 0
> dump_level: 31 (0x1f)
>
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUD
E_USER_DATA|DUMP_EXCLUDE_FREE)
> offset_vmcoreinfo: 11bc
> size_vmcoreinfo: 1392
> OSRELEASE=2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> PAGESIZE=4096
> SYMBOL(init_uts_ns)=ffffffff81a2e8c0
> SYMBOL(node_online_map)=ffffffff81ba0860
> SYMBOL(swapper_pg_dir)=ffffffff81a25000
> SYMBOL(_stext)=ffffffff81000198
> SYMBOL(vmlist)=ffffffff81ee60b8
> SYMBOL(mem_section)=ffffffff81ef03c0
> LENGTH(mem_section)=4096
> SIZE(mem_section)=32
> OFFSET(mem_section.section_mem_map)=0
> SIZE(page)=56
> SIZE(pglist_data)=212416
> SIZE(zone)=34496
> SIZE(free_area)=88
> SIZE(list_head)=16
> SIZE(nodemask_t)=64
> OFFSET(page.flags)=0
> OFFSET(page._count)=8
> OFFSET(page.mapping)=24
> OFFSET(page.lru)=40
> OFFSET(pglist_data.node_zones)=0
> OFFSET(pglist_data.nr_zones)=212288
> OFFSET(pglist_data.node_start_pfn)=212312
> OFFSET(pglist_data.node_spanned_pages)=212328
> OFFSET(pglist_data.node_id)=212336
> OFFSET(zone.free_area)=32864
> OFFSET(zone.vm_stat)=34032
> OFFSET(zone.spanned_pages)=34344
> OFFSET(free_area.free_list)=0
> OFFSET(list_head.next)=0
> OFFSET(list_head.prev)=8
> OFFSET(vm_struct.addr)=8
> LENGTH(zone.free_area)=11
> SYMBOL(log_buf)=ffffffff81a37210
> SYMBOL(log_end)=ffffffff81d5b820
> SYMBOL(log_buf_len)=ffffffff81a37208
> SYMBOL(logged_chars)=ffffffff81ddb920
> LENGTH(free_area.free_list)=5
> NUMBER(NR_FREE_PAGES)=0
> NUMBER(PG_lru)=5
> NUMBER(PG_private)=11
> NUMBER(PG_swapcache)=16
> SYMBOL(phys_base)=ffffffff81a2d010
> SYMBOL(init_level4_pgt)=ffffffff81a25000
> SYMBOL(node_data)=ffffffff81b9cda0
> LENGTH(node_data)=512
> CRASHTIME=1313859299
> offset_note: 1040
> size_note: 1780
num_prstatus_notes: 1
notes_buf: 2cc4000
notes[0]: 2cc4000
> NT_PRSTATUS_offset: 1040
>
> data_offset: 122000
> block_size: 4096
> block_shift: 12
> bitmap: 7fa5296fc010
> bitmap_len: 1179648
> dumpable_bitmap: 7fa528890010
> byte: 0
> bit: 0
> compressed_page: 2cdeb30
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cceb20
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2ccfb20
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd0b20
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd1b20
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd2b20
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd3b20
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd4b20
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd5b20
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd6b20
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd7b20
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd8b20
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd9b20
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cdab20
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cdbb20
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cdcb20
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cddb20
> pg_hit_count: 0
>
> page_cache_buf: 2cceb20
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: 2ccc710
> crash: pv_init_ops exists: ARCH_PVOPS
> compressed kdump: phys_base: 0
> gdb vmlinux
> GNU gdb (GDB) 7.0
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> cpu_possible_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> cpu_present_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> cpu_online_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> base kernel version: 2.6.32
> verify_namelist:
> dumpfile /proc/version:
> Linux version 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> (root(a)druk.mno.stratus.com) (gcc version 4.4.5 20110214 (Red Hat
> 4.4.5-6) (GCC) ) #1 SMP Thu Jun 16 13:13:45 EDT 2011
> vmlinux:
> Linux version 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> (root(a)druk.mno.stratus.com) (gcc version 4.4.5 20110214 (Red Hat
> 4.4.5-6) (GCC) ) #1 SMP Thu Jun 16 13:13:45 EDT 2011
>
crash: page excluded: kernel virtual address: ffffffff81bb3b00 type:
> "cpu number (per_cpu)"
> crash: get_cpus_present: present: 16
crash: page excluded: kernel virtual address: ffffffff81bb3b00 type:
> "cpu number (per_cpu)"
> crash: get_cpus_present: present: 16
> IRQ stack link register: undetermined
> PAGESIZE=4096
> mem_section_size = 32768
> NR_SECTION_ROOTS = 4096
> NR_MEM_SECTIONS = 524288
> SECTIONS_PER_ROOT = 128
> SECTION_ROOT_MASK = 0x7f
> PAGES_PER_SECTION = 32768
> node_online_map: [3, 0, 0, 0, 0, 0, 0, 0] -> nodes online: 2
> node_table[0]:
> id: 0
> pgdat: ffff880000020040
> size: 0
> present: 0
> mem_map: ffffea0000000000
> start_paddr: 0
> start_mapnr: 0
> WARNING: sparsemem: invalid section number: 137438888923
> WARNING: sparsemem: invalid section number: 137438888923
> crash: invalid kernel virtual address: 0 type: "readstring
> characters"
> crash: invalid kernel virtual address: 0 type: "readstring
> characters"
> node_table[1]:
> id: 1
> pgdat: ffff880280000040
> size: 2097152
> present: 2097152
> mem_map: ffffea0008c00000
> start_paddr: 280000000
> start_mapnr: 2621440
> NOTE: page_hash_table does not exist in this kernel
> ^Mplease wait... (gathering kmem slab cache data)
> kmem_cache_downsize: SIZE(kmem_cache_s): 36968
> cache_cache.buffer_size: 32896
> kmem_cache_downsize: nr_node_ids: 2
> ^M ^MNOTE: unwind_table structure has changed, or does not exist in
> this kernel
> init_unwind_table: DWARF_UNWIND_EH_FRAME
> ^Mplease wait... (gathering module symbol data)^M ^M^Mplease wait...
> (gathering task table data)^M ^Mcrash: get_cpus_online: online: 16
> ^Mplease wait... (determining panic task)
> crash: get_active_set_panic_task: failed
>
>
> Thanks,
>
> -- Joe Lawrence
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
8:27 a.m.
----- Original Message -----
Hi Dave,
Before the patch mentioned below, makedumpfile hardcoded nr_cpus to 1 in
the header.
..........................
author Ken'ichi Ohmichi <oomichi(a)mxs.nes.nec.co.jp>
Thu, 11 Nov 2010 03:53:16 +0000 (12:53 +0900)
Before applying this patch, makedumpfile sets "1" to nr_cpus in kdump
main header always even if a machine has multiple cpus.
As the result, the subcommand "help -n" of the crash utility prints
an invalid value "1" as nr_cpus.
...........................
Could this be part of Joe's issue?
Thanks,
Jeff Hagen
Interesting -- and depending upon which version of kexec-tools he has
installed, it certainly could affect the display of "nr_cpus" shown
by "help -n".
But as it turns out, the setting of "nr_cpus" in the header is irrelevant
with respect to compressed kdumps. It was used by the original diskdump-format
dumpfiles, upon which the compressed kdump dumpfile format is based. So that's
a red herring, regardless of whether Ken'ichi's patch was applied.
What's unexplainable here is the dump of the note information:
size_note: 1780
num_prstatus_notes: 1
notes_buf: 2cc4000
notes[0]: 2cc4000
The determination of the number of ELF nt_prstatus notes is based
upon the contents of the kdump_sub_header, where "size_note" describes
a single buffer in the dumpfile that contains an array of nt_prstatus
notes. Each note consists of a small Elf64_Nhdr header, a name string,
and a register dump. Here's an example of one taken from an ELF-format
kdump:
Elf64_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 336
n_type: 1 (NT_PRSTATUS)
0000000000000000 0000000000000000
0000000000000000 0000000000000000
000000000000544d 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000001 00007fffb894e76f
00007fffb894e170 ffff88012969ebf0
ffff880128541f88 ffff880108870a00
0000000000000000 0000000000000000
ffffffff8184f8f0 0000000000000000
ffff880108870ab0 0000000000000003
0000000000000004 ffffffff81ad7fd0
ffff8801280e44b0 ffffffffffffffff
ffffffff8108d378 0000000000000010
0000000000010202 ffff88012854de68
0000000000000018 00007fa3283af700
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
So rounded up, each note is roughly ~350 bytes. So, while a "size_note"
of 1780 bytes wouldn't be large enough to contain the notes for 16 cpus,
it would seem to contain more than 1 note. (???) But the note-gathering
code was only able to come up a "num_prstatus_notes" of 1.
It would interesting to find out what happened in the x86_process_elf_notes()
function.
Dave
-----Original Message-----
From: crash-utility-bounces(a)redhat.com
[mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson
Sent: Thursday, September 29, 2011 2:04 PM
To: Discussion list for crash utility usage,maintenance and
development
Subject: Re: [Crash-utility] Crash faults when determining panic task
----- Original Message -----
>
> Hi Dave,
>
> I hope I have captured everything you asked for here, if remote
> debugging over e-mail is too tedious, I can arrange to post a
> vmlinux/vmcore on our FTP site (roughly 600MB together).
Sure, you can do that if you'd like.
But anyway, the crash -d1 output is illuminating. You've got a
16-cpu
system, with all cpus online. But the compressed kdump header only
saw 1 cpu when it was created:
> header: 2cc1fe0
> signature: "KDUMP "
> header_version: 4
> utsname:
> sysname: Linux
> nodename: bahamut.mno.stratus.com
> release: 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> version: #1 SMP Thu Jun 16 13:13:45 EDT 2011
> machine: x86_64
> domainname: sraeng
> timestamp:
> tv_sec: 4e4fe6e3
> tv_usec: 0
> status: 0 ()
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 288
> max_mapnr: 4718592
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1 <== should be 16
> tasks[nr_cpus]: 0
and farther on, here is the array of note pointers that I
was asking about:
> num_prstatus_notes: 1
> notes_buf: 2cc4000
> notes[0]: 2cc4000
Since dumpfile header's nr_cpus was 1, the array has only one entry.
I cannot explain that. But the crash utility can only deal with what
it finds in the dumpfile.
Furthermore, these two error messages indicate that memory containing
per-cpu data was "excluded" are of prime importance here:
> crash: page excluded: kernel virtual address: ffffffff81bb3b00
> type:
"cpu number (per_cpu)"
> crash: page excluded: kernel virtual address: ffffffff81bb3b00
> type:
"cpu number (per_cpu)"
The fact that the page was specifically "page excluded" is troubling,
because it should *never* have been filtered by makedumpfile
-d<level>.
But, since the dumpfile indicates that the crucial per-cpu page was
filtered,
so there's nothing that the crash utility can do about it.
I'm guessing that, even though you are able to get to a prompt
with --no_elf_notes, any command that depends upon per-cpu data
would fail. Although, it might be interesting to know *which*
cpu was in play when those two error messages were displayed
in x86_64_per_cpu_init() and x86_64_get_smp_cpus(). There is
a loop in both functions -- can you dump out which cpu's
per-cpu data was inaccessible?
Thanks,
Dave
>
>
> *** Setup some breakpoints to watch bt->machdep:
>
> get_netdump_regs_x86_64(struct bt_info *bt, ulong *ripp, ulong
> *rspp)
> {
> ...
>
> if (((NETDUMP_DUMPFILE() || KDUMP_DUMPFILE()) &&
> VALID_STRUCT(user_regs_struct) && (bt->task == tt->panic_task)) ||
> (KDUMP_DUMPFILE() && (kt->flags & DWARF_UNWIND) &&
> (bt->flags & BT_DUMPFILE_SEARCH))) {
> ...
> 2287 bt->machdep = (void *)user_regs;
> ...
>
> if (ELF_NOTES_VALID() &&
> (bt->flags & BT_DUMPFILE_SEARCH) && DISKDUMP_DUMPFILE() &&
> (note = (Elf64_Nhdr *)
> diskdump_get_prstatus_percpu(bt->tc->processor))) {
> ...
> 2306 bt->machdep = (void *)user_regs;
> ...
>
> (gdb) break get_netdump_regs_x86_64
> Breakpoint 1 at 0x519740: file netdump.c, line 2238.
> (gdb) break netdump.c:2287
> Breakpoint 2 at 0x519970: file netdump.c, line 2287.
> (gdb) break netdump.c:2306
> Breakpoint 3 at 0x5199e7: file netdump.c, line 2306.
> (gdb) r
>
> please wait... (determining panic task)
> Breakpoint 1, get_netdump_regs_x86_64 (bt=0x7fffffffcd70,
> ripp=0x7fffffffcce0,
> rspp=0x7fffffffcce8) at netdump.c:2238
> 2238 {
> (gdb) c
> Continuing.
>
> Breakpoint 3, get_netdump_regs_x86_64 (bt=0x7fffffffcd70,
> ripp=0x7fffffffcce0,
> rspp=0x7fffffffcce8) at netdump.c:2306
> 2306 bt->machdep = (void *)user_regs;
> (gdb) p user_regs
> $1 = 0xd14084 ""
> (gdb) c
> Continuing.
>
> Breakpoint 1, get_netdump_regs_x86_64 (bt=0x7fffffffcd70,
> ripp=0x7fffffffcce0,
> rspp=0x7fffffffcce8) at netdump.c:2238
> 2238 {
> (gdb) c
> Continuing.
>
> Program received signal SIGSEGV, Segmentation fault.
> x86_64_get_dumpfile_stack_frame (rsp=0x7fffffffcce8,
> rip=0x7fffffffcce0,
> bt_in=0x7fffffffcd70) at x86_64.c:4183
> 4183 ur_rip = ULONG(user_regs +
>
>
> *** So in its second invocation, get_netdump_regs_x86_64() never
> sets
> bt->machdep (only breakpoint 1 fired)
>
> *** Let's see what diskdump_get_prstatus_percpu() is returning
>
> (gdb) break diskdump_get_prstatus_percpu
> Breakpoint 1 at 0x526070: file diskdump.c, line 1451.
> (gdb) r
> please wait... (determining panic task)
> Breakpoint 1, diskdump_get_prstatus_percpu (cpu=0) at
> diskdump.c:1451
> 1451 return dd->nt_prstatus_percpu[cpu];
> (gdb) display dd->nt_prstatus_percpu[0]@16
> 1: dd->nt_prstatus_percpu[0]@16 = {0xd1c000, 0x0, 0x0, 0xd26472,
> 0xbf35ab2,
> 0xd26472, 0x200000012, 0xd1c850, 0xd1c600, 0x1010000012b,
> 0xffffffff814e4fa0, 0x14e4fa0, 0x4270, 0x0, 0x0, 0x0}
> (gdb) c
> Continuing.
>
> Breakpoint 1, diskdump_get_prstatus_percpu (cpu=1) at
> diskdump.c:1451
> 1451 return dd->nt_prstatus_percpu[cpu];
> 1: dd->nt_prstatus_percpu[0]@16 = {0xd1c000, 0x0, 0x0, 0xd26472,
> 0xbf35ab2,
> 0xd26472, 0x200000012, 0xd1c850, 0xd1c600, 0x1010000012b,
> 0xffffffff814e4fa0, 0x14e4fa0, 0x4270, 0x0, 0x0, 0x0}
>
>
> *** See crash -d1 vmlinux vmcore output at the bottom of the mail,
> particularly the part that says...
>
> crash: page excluded: kernel virtual address: ffffffff81bb3b00
> type:
> "cpu number (per_cpu)"
> crash: get_cpus_present: present: 16
>
>
>
> *** Bogus note->n_descsz value
> *** Apply first patch to get us further into ELF Note processing
>
> >From inside netdump.c :: get_regs_from_note() at the point of the
> >fault, I don't see dd->nt_prstatus[], for dd is now type
> >*diskdump_data... The *note passed in can be found in
> >dd->nt_prstatus_percpu[] however...
>
> please wait... (determining panic task)
> Program received signal SIGSEGV, Segmentation fault.
> get_regs_from_note (note=0xd26472 "\b", ip=0x7fffffffc590,
> sp=0x7fffffffc598)
> at netdump.c:2221
> 2221 *sp = ULONG(user_regs + offset_sp);
> (gdb) p/x *((Elf64_Nhdr *)note)
> $1 = {n_namesz = 0x8, n_descsz = 0xccf80000, n_type = 0x8}
> (gdb) p dd->nt_prstatus_percpu[0]@16
> $2 = {0xd1c000, 0x0, 0x0, 0xd26472, 0xbf35ab2, 0xd26472,
> 0x200000012,
> 0xd1c850, 0xd1c600, 0x1010000012b, 0xffffffff814e4fa0, 0x14e4fa0,
> 0x4270,
> 0x0, 0x0, 0x0}
> (gdb) ptype dd
> type = struct diskdump_data {
> char *filename;
> ulong flags;
> int dfd;
> FILE *ofp;
> int machine_type;
> struct disk_dump_header *header;
> struct disk_dump_sub_header *sub_header;
> struct kdump_sub_header *sub_header_kdump;
> size_t data_offset;
> int block_size;
> int block_shift;
> char *bitmap;
> int bitmap_len;
> char *dumpable_bitmap;
> int byte;
> int bit;
> char *compressed_page;
> char *curbufptr;
> unsigned char *notes_buf;
> void **nt_prstatus_percpu;
> uint num_prstatus_notes;
> struct page_cache_hdr page_cache_hdr[16];
> char *page_cache_buf;
> int evict_index;
> ulong evictions;
> ulong cached_reads;
> ulong *valid_pages;
> ulong accesses;
> } *
>
>
>
> *** Unpatched crash -d1 vmlinux vmcore output:
>
> crash 5.1.8
> Copyright (C) 2002-2011 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it
> under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> compressed kdump: header->utsname.machine: x86_64
> diskdump_data:
> filename: vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 3
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: 2cc1fe0
> signature: "KDUMP "
> header_version: 4
> utsname:
> sysname: Linux
> nodename: bahamut.mno.stratus.com
> release: 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> version: #1 SMP Thu Jun 16 13:13:45 EDT 2011
> machine: x86_64
> domainname: sraeng
> timestamp:
> tv_sec: 4e4fe6e3
> tv_usec: 0
> status: 0 ()
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 288
> max_mapnr: 4718592
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1
> tasks[nr_cpus]: 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 2cc2ff0
> phys_base: 0
> dump_level: 31 (0x1f)
>
(DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUD
E_USER_DATA|DUMP_EXCLUDE_FREE)
> offset_vmcoreinfo: 11bc
> size_vmcoreinfo: 1392
> OSRELEASE=2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> PAGESIZE=4096
> SYMBOL(init_uts_ns)=ffffffff81a2e8c0
> SYMBOL(node_online_map)=ffffffff81ba0860
> SYMBOL(swapper_pg_dir)=ffffffff81a25000
> SYMBOL(_stext)=ffffffff81000198
> SYMBOL(vmlist)=ffffffff81ee60b8
> SYMBOL(mem_section)=ffffffff81ef03c0
> LENGTH(mem_section)=4096
> SIZE(mem_section)=32
> OFFSET(mem_section.section_mem_map)=0
> SIZE(page)=56
> SIZE(pglist_data)=212416
> SIZE(zone)=34496
> SIZE(free_area)=88
> SIZE(list_head)=16
> SIZE(nodemask_t)=64
> OFFSET(page.flags)=0
> OFFSET(page._count)=8
> OFFSET(page.mapping)=24
> OFFSET(page.lru)=40
> OFFSET(pglist_data.node_zones)=0
> OFFSET(pglist_data.nr_zones)=212288
> OFFSET(pglist_data.node_start_pfn)=212312
> OFFSET(pglist_data.node_spanned_pages)=212328
> OFFSET(pglist_data.node_id)=212336
> OFFSET(zone.free_area)=32864
> OFFSET(zone.vm_stat)=34032
> OFFSET(zone.spanned_pages)=34344
> OFFSET(free_area.free_list)=0
> OFFSET(list_head.next)=0
> OFFSET(list_head.prev)=8
> OFFSET(vm_struct.addr)=8
> LENGTH(zone.free_area)=11
> SYMBOL(log_buf)=ffffffff81a37210
> SYMBOL(log_end)=ffffffff81d5b820
> SYMBOL(log_buf_len)=ffffffff81a37208
> SYMBOL(logged_chars)=ffffffff81ddb920
> LENGTH(free_area.free_list)=5
> NUMBER(NR_FREE_PAGES)=0
> NUMBER(PG_lru)=5
> NUMBER(PG_private)=11
> NUMBER(PG_swapcache)=16
> SYMBOL(phys_base)=ffffffff81a2d010
> SYMBOL(init_level4_pgt)=ffffffff81a25000
> SYMBOL(node_data)=ffffffff81b9cda0
> LENGTH(node_data)=512
> CRASHTIME=1313859299
> offset_note: 1040
> size_note: 1780
> num_prstatus_notes: 1
> notes_buf: 2cc4000
> notes[0]: 2cc4000
> NT_PRSTATUS_offset: 1040
>
> data_offset: 122000
> block_size: 4096
> block_shift: 12
> bitmap: 7fa5296fc010
> bitmap_len: 1179648
> dumpable_bitmap: 7fa528890010
> byte: 0
> bit: 0
> compressed_page: 2cdeb30
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cceb20
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2ccfb20
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd0b20
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd1b20
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd2b20
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd3b20
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd4b20
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd5b20
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd6b20
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd7b20
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd8b20
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cd9b20
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cdab20
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cdbb20
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cdcb20
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2cddb20
> pg_hit_count: 0
>
> page_cache_buf: 2cceb20
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: 2ccc710
> crash: pv_init_ops exists: ARCH_PVOPS
> compressed kdump: phys_base: 0
> gdb vmlinux
> GNU gdb (GDB) 7.0
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> cpu_possible_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> cpu_present_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> cpu_online_map: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> base kernel version: 2.6.32
> verify_namelist:
> dumpfile /proc/version:
> Linux version 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> (root(a)druk.mno.stratus.com) (gcc version 4.4.5 20110214 (Red Hat
> 4.4.5-6) (GCC) ) #1 SMP Thu Jun 16 13:13:45 EDT 2011
> vmlinux:
> Linux version 2.6.32-131.0.15.el6.exp10.bz16586.x86_64
> (root(a)druk.mno.stratus.com) (gcc version 4.4.5 20110214 (Red Hat
> 4.4.5-6) (GCC) ) #1 SMP Thu Jun 16 13:13:45 EDT 2011
>
> crash: page excluded: kernel virtual address: ffffffff81bb3b00
> type:
> "cpu number (per_cpu)"
> crash: get_cpus_present: present: 16
> crash: page excluded: kernel virtual address: ffffffff81bb3b00
> type:
> "cpu number (per_cpu)"
> crash: get_cpus_present: present: 16
> IRQ stack link register: undetermined
> PAGESIZE=4096
> mem_section_size = 32768
> NR_SECTION_ROOTS = 4096
> NR_MEM_SECTIONS = 524288
> SECTIONS_PER_ROOT = 128
> SECTION_ROOT_MASK = 0x7f
> PAGES_PER_SECTION = 32768
> node_online_map: [3, 0, 0, 0, 0, 0, 0, 0] -> nodes online: 2
> node_table[0]:
> id: 0
> pgdat: ffff880000020040
> size: 0
> present: 0
> mem_map: ffffea0000000000
> start_paddr: 0
> start_mapnr: 0
> WARNING: sparsemem: invalid section number: 137438888923
> WARNING: sparsemem: invalid section number: 137438888923
> crash: invalid kernel virtual address: 0 type: "readstring
> characters"
> crash: invalid kernel virtual address: 0 type: "readstring
> characters"
> node_table[1]:
> id: 1
> pgdat: ffff880280000040
> size: 2097152
> present: 2097152
> mem_map: ffffea0008c00000
> start_paddr: 280000000
> start_mapnr: 2621440
> NOTE: page_hash_table does not exist in this kernel
> ^Mplease wait... (gathering kmem slab cache data)
> kmem_cache_downsize: SIZE(kmem_cache_s): 36968
> cache_cache.buffer_size: 32896
> kmem_cache_downsize: nr_node_ids: 2
> ^M ^MNOTE: unwind_table structure has changed, or does not exist in
> this kernel
> init_unwind_table: DWARF_UNWIND_EH_FRAME
> ^Mplease wait... (gathering module symbol data)^M ^M^Mplease
> wait...
> (gathering task table data)^M ^Mcrash: get_cpus_online: online: 16
> ^Mplease wait... (determining panic task)
> crash: get_active_set_panic_task: failed
>
>
> Thanks,
>
> -- Joe Lawrence
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
8:38 a.m.
----- Original Message -----
What's unexplainable here is the dump of the note information:
> size_note: 1780
> num_prstatus_notes: 1
> notes_buf: 2cc4000
> notes[0]: 2cc4000
Not to mention, that there's no explanation for why makedumpfile would filter
out this crucial page:
crash: page excluded: kernel virtual address: ffffffff81bb3b00 type:
"cpu number (per_cpu)"
crash: page excluded: kernel virtual address: ffffffff81bb3b00 type: "cpu number
(per_cpu)"
I wouldn't think that *any* page in the mapped kernel-text/static-data
region would be subject to filtering -- certainly not the one containing
the per-cpu data information.
Dave
9:59 a.m.
----- Original Message -----
What's unexplainable here is the dump of the note information:
> size_note: 1780
> num_prstatus_notes: 1
> notes_buf: 2cc4000
> notes[0]: 2cc4000
The determination of the number of ELF nt_prstatus notes is based
upon the contents of the kdump_sub_header, where "size_note" describes
a single buffer in the dumpfile that contains an array of nt_prstatus
notes. Each note consists of a small Elf64_Nhdr header, a name string,
and a register dump. Here's an example of one taken from an ELF-format
kdump:
Elf64_Nhdr:
n_namesz: 5 ("CORE")
n_descsz: 336
n_type: 1 (NT_PRSTATUS)
0000000000000000 0000000000000000
0000000000000000 0000000000000000
000000000000544d 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000001 00007fffb894e76f
00007fffb894e170 ffff88012969ebf0
ffff880128541f88 ffff880108870a00
0000000000000000 0000000000000000
ffffffff8184f8f0 0000000000000000
ffff880108870ab0 0000000000000003
0000000000000004 ffffffff81ad7fd0
ffff8801280e44b0 ffffffffffffffff
ffffffff8108d378 0000000000000010
0000000000010202 ffff88012854de68
0000000000000018 00007fa3283af700
0000000000000000 0000000000000000
0000000000000000 0000000000000000
0000000000000000 0000000000000000
So rounded up, each note is roughly ~350 bytes. So, while a "size_note"
of 1780 bytes wouldn't be large enough to contain the notes for 16 cpus,
it would seem to contain more than 1 note. (???) But the note-gathering
code was only able to come up a "num_prstatus_notes" of 1.
It would interesting to find out what happened in the x86_process_elf_notes()
function.
Digging into this a bit more -- the array of notes in the dumpfile
also includes the VMCOREINFO note (n_type 0), which is roughly ~1400
bytes in length. So given that all of the notes consume 1780 bytes in
Joe's dump, it looks like there is one NT_PRSTATUS note and one VMCOREINFO
note.
Joe, do you know if the non-crashing cpus were in some kind of
bizarre state such that they would not respond to the shutdown NMI?
I suppose in that case, there would be only the one NT_PRSTATUS
note for the crashing cpu (plus the VMCOREINFO note).
In any case, so far I've got two patches queued to help address
the two segmentation violations generated by a scenario such as
this. First Joe's patch:
--- x86_64.c 28 Sep 2011 18:09:54 -0000 1.187
+++ x86_64.c 29 Sep 2011 19:17:09 -0000 1.188
@@ -4181,7 +4181,7 @@
goto skip_stage;
}
}
- } else if (ELF_NOTES_VALID()) {
+ } else if (ELF_NOTES_VALID() && bt->machdep) {
user_regs = bt->machdep;
ur_rip = ULONG(user_regs +
OFFSET(user_regs_struct_rip));
And then this preventative measure to prevent a bogus ELF
note pointer being passed back:
--- diskdump.c 20 Sep 2011 20:41:14 -0000 1.38
+++ diskdump.c 30 Sep 2011 14:55:11 -0000
@@ -1467,6 +1467,9 @@
void *
diskdump_get_prstatus_percpu(int cpu)
{
+ if ((cpu < 0) || (cpu >= dd->num_prstatus_notes))
+ return NULL;
+
return dd->nt_prstatus_percpu[cpu];
}
Dave
1:38 p.m.
It would interesting to find out what happened in the
x86_process_elf_notes()
function.
*** Breakpoints in x86_process_elf_notes()...
(gdb) break diskdump.c:245
Breakpoint 1 at 0x52379b: file diskdump.c, line 245.
(gdb) r
Breakpoint 1, x86_process_elf_notes (note_ptr=0xd1e000, size_note=1780)
at diskdump.c:245
245 note64 = note_ptr + tot;
(gdb) p *(Elf64_Nhdr *)(note_ptr + tot)
$1 = {n_namesz = 5, n_descsz = 336, n_type = 1}
(gdb) c
Continuing.
Breakpoint 1, x86_process_elf_notes (note_ptr=0xd1e000, size_note=1780)
at diskdump.c:245
245 note64 = note_ptr + tot;
(gdb) p *(Elf64_Nhdr *)(note_ptr + tot)
$2 = {n_namesz = 11, n_descsz = 1392, n_type = 0}
(gdb) c
Continuing.
Breakpoint 1, x86_process_elf_notes (note_ptr=0xd1e000, size_note=1780)
at diskdump.c:245
245 note64 = note_ptr + tot;
(gdb) p *(Elf64_Nhdr *)(note_ptr + tot)
$3 = {n_namesz = 0, n_descsz = 0, n_type = 0}
(gdb) c
Continuing.
> crash: page excluded: kernel virtual address: ffffffff81bb3b00
type:
"cpu number (per_cpu)"
> crash: page excluded: kernel virtual address: ffffffff81bb3b00
type:
"cpu number (per_cpu)"
[snip]
loop in both functions -- can you dump out which cpu's
per-cpu data was inaccessible?
(gdb) break memory.c:1976
Breakpoint 1 at 0x4722ff: file memory.c, line 1976.
(gdb) set arg -d1 vmlinux vmcore
(gdb) r
Breakpoint 1, readmem (addr=18446744071591115520, memtype=1,
buffer=0x7fffffff5b5c, size=4, type=0x7c7744 "cpu number (per_cpu)",
error_handle=6) at memory.c:1976
1976 error(INFO, PAGE_EXCLUDED_ERRMSG,
memtype_string(memtype, 0), addr, type);
(gdb) up
#1 0x00000000004e5871 in x86_64_get_smp_cpus () at x86_64.c:4674
4674 if (!readmem(sp->value + kt->__per_cpu_offset[i],
(gdb) p cpunumber
$1 = 15
(gdb) p cpus
$2 = 16
(gdb) p i
$3 = 16
(gdb) p/x kt->__per_cpu_offset[0]@17
$4 = {0xffff880028200000, 0xffff880028240000, 0xffff880028280000,
0xffff8800282c0000, 0xffff880287400000, 0xffff880287440000,
0xffff880287480000, 0xffff8802874c0000, 0xffff880028300000,
0xffff880028340000, 0xffff880028380000, 0xffff8800283c0000,
0xffff880287500000, 0xffff880287540000, 0xffff880287580000,
0xffff8802875c0000, 0xffffffff81ba6000}
Joe, do you know if the non-crashing cpus were in some kind of
bizarre state such that they would not respond to the shutdown NMI?
I suppose in that case, there would be only the one NT_PRSTATUS
note for the crashing cpu (plus the VMCOREINFO note).
The other CPUs are almost all sitting idle, a few are running I/O.
In any case, so far I've got two patches queued to help address
the two segmentation violations generated by a scenario such as
this.
Patches applied and verified no segmentation faults.
I have uploaded this vmcore/vmlinux to our FTP site (details to come in private mail).
Thanks,
-- Joe Lawrence
2:33 p.m.
----- Original Message -----
RE: [Crash-utility] Crash faults when determining panic task
> It would interesting to find out what happened in the
> x86_process_elf_notes() function.
Thanks for your help debugging this -- the dumpfile contains pretty
much what I expected:
(1) a single NT_PRSTATUS note (n_type 1, n_descsz 336)
(2) followed by the VMCOREINFO note (n_type 0, n_descsz 1392), and
(?) zero-filled dumpfile data (n_type 0, n_descsz 0)
By comparison, if I add this debug printf() to x86_process_elf_notes()
and run it against an 8-way compressed kdump:
--- diskdump.c 30 Sep 2011 15:09:56 -0000 1.39
+++ diskdump.c 30 Sep 2011 18:58:28 -0000
@@ -243,7 +243,7 @@
for (tot = 0; tot < size_note; tot += len) {
if (machine_type("X86_64")) {
note64 = note_ptr + tot;
-
+ fprintf(fp, "n_type: %d n_descsz: %d\n", note64->n_type,
note64->n_descsz);
if (note64->n_type == NT_PRSTATUS) {
dd->nt_prstatus_percpu[num] = note64;
num++;
I see this:
...
This program has absolutely no warranty. Enter "help warranty" for details.
n_type: 1 n_descsz: 336
n_type: 1 n_descsz: 336
n_type: 1 n_descsz: 336
n_type: 1 n_descsz: 336
n_type: 1 n_descsz: 336
n_type: 1 n_descsz: 336
n_type: 1 n_descsz: 336
n_type: 1 n_descsz: 336
n_type: 0 n_descsz: 1373
GNU gdb (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.
...
So there's no extra zero-filled dumpfile location that gets checked,
i.e., it cleanly works its way through the dumpfile's notes region.
I don't know why that's not true with your dumpfile.
And, as it turns out, the per-cpu readmem() complaint is perfectly
legitimate -- I see the same thing on the compressed kdump example
above. It's just that the loop has gone beyond the end of the per-cpu
data -- in your case, it's trying to read non-existent per-cpu
data for the non-existent cpu 16. So that's not a problem...
I still don't understand why the dumpfile doesn't have the other
15 NT_PRSTATUS notes, but until that patch was added into crash-5.1.5,
we never cared, and it would never have been noticed. When I accepted
that patch, I was apprehensive that something like this might happen,
which is why I insisted that they also add the "--no_elf_notes"
option as a pre-emptive workaround:
https://www.redhat.com/archives/crash-utility/2011-April/msg00030.html
Finally, in the interest of paranoia, give the user the capability
of *not* using this facility. In main.c, create a "--no_elf_notes"
option (similar to "--zero_excluded"), and have it set a NO_ELF_NOTES
bit in the globally-accessible "*diskdump_flags".
So anyway, that all being the case, and with the two patches applied,
we've pretty much solved your problem from the crash utility's
perspective. Perhaps there's a kernel kdump or makedumpfile issue,
but that's beyond the scope of this mailing list.
Thanks again,
Dave
*** Breakpoints in x86_process_elf_notes()...
(gdb) break diskdump.c:245
Breakpoint 1 at 0x52379b: file diskdump.c, line 245.
(gdb) r
Breakpoint 1, x86_process_elf_notes (note_ptr=0xd1e000,
size_note=1780)
at diskdump.c:245
245 note64 = note_ptr + tot;
(gdb) p *(Elf64_Nhdr *)(note_ptr + tot)
$1 = {n_namesz = 5, n_descsz = 336, n_type = 1}
(gdb) c
Continuing.
Breakpoint 1, x86_process_elf_notes (note_ptr=0xd1e000,
size_note=1780)
at diskdump.c:245
245 note64 = note_ptr + tot;
(gdb) p *(Elf64_Nhdr *)(note_ptr + tot)
$2 = {n_namesz = 11, n_descsz = 1392, n_type = 0}
(gdb) c
Continuing.
Breakpoint 1, x86_process_elf_notes (note_ptr=0xd1e000,
size_note=1780)
at diskdump.c:245
245 note64 = note_ptr + tot;
(gdb) p *(Elf64_Nhdr *)(note_ptr + tot)
$3 = {n_namesz = 0, n_descsz = 0, n_type = 0}
(gdb) c
Continuing.
>> crash: page excluded: kernel virtual address: ffffffff81bb3b00
>> type:
"cpu number (per_cpu)"
>> crash: page excluded: kernel virtual address: ffffffff81bb3b00
>> type:
"cpu number (per_cpu)"
> [snip]
> loop in both functions -- can you dump out which cpu's
> per-cpu data was inaccessible?
(gdb) break memory.c:1976
Breakpoint 1 at 0x4722ff: file memory.c, line 1976.
(gdb) set arg -d1 vmlinux vmcore
(gdb) r
Breakpoint 1, readmem (addr=18446744071591115520, memtype=1,
buffer=0x7fffffff5b5c, size=4, type=0x7c7744 "cpu number (per_cpu)",
error_handle=6) at memory.c:1976
1976 error(INFO, PAGE_EXCLUDED_ERRMSG, memtype_string(memtype, 0),
addr, type);
(gdb) up
#1 0x00000000004e5871 in x86_64_get_smp_cpus () at x86_64.c:4674
4674 if (!readmem(sp->value + kt->__per_cpu_offset[i],
(gdb) p cpunumber
$1 = 15
(gdb) p cpus
$2 = 16
(gdb) p i
$3 = 16
(gdb) p/x kt->__per_cpu_offset[0]@17
$4 = {0xffff880028200000, 0xffff880028240000, 0xffff880028280000,
0xffff8800282c0000, 0xffff880287400000, 0xffff880287440000,
0xffff880287480000, 0xffff8802874c0000, 0xffff880028300000,
0xffff880028340000, 0xffff880028380000, 0xffff8800283c0000,
0xffff880287500000, 0xffff880287540000, 0xffff880287580000,
0xffff8802875c0000, 0xffffffff81ba6000}
> Joe, do you know if the non-crashing cpus were in some kind of
> bizarre state such that they would not respond to the shutdown NMI?
> I suppose in that case, there would be only the one NT_PRSTATUS
> note for the crashing cpu (plus the VMCOREINFO note).
The other CPUs are almost all sitting idle, a few are running I/O.
> In any case, so far I've got two patches queued to help address
> the two segmentation violations generated by a scenario such as
> this.
Patches applied and verified no segmentation faults.
I have uploaded this vmcore/vmlinux to our FTP site (details to come
in private mail).
Thanks,
-- Joe Lawrence
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
4658
days inactive
4660
days old
devel@lists.crash-utility.osci.io
11 comments
3 participants
participants (3)
-
Dave Anderson -
Hagen, Jeffrey -
Lawrence, Joe