11:30 a.m.
Ah, it must be because of the common user-kernel virtual address space on s390x? I can
accept a patch if it's s390 only.
Dave
Sent from my Verizon Wireless 4G LTE smartphone
-------- Original message --------
From: Dave Anderson <anderson(a)redhat.com>
Date: 08/03/2015 11:18 AM (GMT-05:00)
To: Michael Holzheu <holzheu(a)linux.vnet.ibm.com>
Cc: Heiko Carstens <heiko.carstens(a)de.ibm.com>, "Discussion list for crash
utility usage, maintenance and development" <crash-utility(a)redhat.com>
Subject: Re: [Crash-utility] [PATCH] crash: Do not use bt -t flag in
panic_search()
----- Original Message -----
Hi Dave,
I got a dump where a process "gmain" was incorrectly marked as running:
crash> ps | grep gmain
> 217 1 5 8bec23420 IN 0.0 463276 18240 gmain
The reason was that the "brute force" way parsing the "bt -t -o"
output in panic_search() found the symbol "panic" on the stack:
crash> bt -t -o 8bec23420
PID: 217 TASK: 8bec23420 CPU: 5 COMMAND: "gmain"
START: __schedule at 83f650
[ 8b662b900] (null) at 0
[ 8b662b950] (null) at 0
[ 8b662b978] __schedule at 83f650
[ 8b662b990] (null) at 0
...
[ 8b662bb18] (null) at 0
[ 8b662bb40] panic at 83679a <<<<<--------------
[ 8b662bb58] _ehead at 280da
I guess the obvious question is why "panic" was on the stack?
The real stack trace was as follows:
crash> bt 8bec23420
Detaching after fork from child process 15508.
PID: 217 TASK: 8bec23420 CPU: 5 COMMAND: "gmain"
#0 [8b662b8f0] __schedule at 83f650
#1 [8b662b958] schedule at 83fade
#2 [8b662b970] schedule_hrtimeout_range_clock at 842fc8
#3 [8b662ba10] poll_schedule_timeout at 2c6e8a
#4 [8b662ba30] do_sys_poll at 2c8604
#5 [8b662be40] sys_poll at 2c8852
#6 [8b662bea8] system_call at 843a66
IMHO the "-t" method is quite risky (at least on s390). What about using
the "normal" stack backtrace without the "-t" bt option?
That really worries me -- introducing the usage of normal backtrace on all tasks
instead of simply walking the stack memory looking for text addresses is a huge
change.
Dave
---
task.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--- a/task.c
+++ b/task.c
@@ -6633,7 +6633,7 @@ panic_search(void)
fd = &foreach_data;
fd->keys = 1;
fd->keyword_array[0] = FOREACH_BT;
- fd->flags |= (FOREACH_t_FLAG|FOREACH_o_FLAG);
+ fd->flags |= FOREACH_o_FLAG;
dietask = lasttask = NO_TASK;
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
12:05 p.m.
New subject: [PATCH] crash: Do not use bt -t flag in panic_search()
On Mon, 03 Aug 2015 12:30:22 -0400
Dave Anderson <anderson(a)prospeed.net> wrote:
Ah, it must be because of the common user-kernel virtual address space on s390x?
Yes we have overlapping virtual address ranges for kernel and user space and user
space registers are stored at the start of the kernel stack.
But I am not sure, if that is the reason here (have to verify this).
Doesn't the -t option check all the stack memory and looks for valid symbol
addresses? So IMO everything found there could be by chance the address
of panic. Why is this no problem on x86?
Michael
12:16 p.m.
New subject: [PATCH] crash: Do not use bt -t flag in panic_search()
----- Original Message -----
On Mon, 03 Aug 2015 12:30:22 -0400
Dave Anderson <anderson(a)prospeed.net> wrote:
>
>
> Ah, it must be because of the common user-kernel virtual address space on
> s390x?
Yes we have overlapping virtual address ranges for kernel and user space and user
space registers are stored at the start of the kernel stack.
But I am not sure, if that is the reason here (have to verify this).
Doesn't the -t option check all the stack memory and looks for valid symbol
addresses? So IMO everything found there could be by chance the address
of panic. Why is this no problem on x86?
I'm not sure what you mean. How could it be there by chance? Are you talking
about a highly unlikely situation where a user-space exception frame at the top
of the kernel stack happens to have a register containing a kernel virtual address
that's within the panic() function?
Dave
Michael
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
9:57 a.m.
New subject: [PATCH] crash: Do not use bt -t flag in panic_search()
On Mon, 3 Aug 2015 13:16:19 -0400 (EDT)
Dave Anderson <anderson(a)redhat.com> wrote:
----- Original Message -----
> On Mon, 03 Aug 2015 12:30:22 -0400
> Dave Anderson <anderson(a)prospeed.net> wrote:
>
> >
> >
> > Ah, it must be because of the common user-kernel virtual address space on
> > s390x?
>
> Yes we have overlapping virtual address ranges for kernel and user space and user
> space registers are stored at the start of the kernel stack.
>
> But I am not sure, if that is the reason here (have to verify this).
>
> Doesn't the -t option check all the stack memory and looks for valid symbol
> addresses? So IMO everything found there could be by chance the address
> of panic. Why is this no problem on x86?
I'm not sure what you mean. How could it be there by chance? Are you talking
about a highly unlikely situation where a user-space exception frame at the top
of the kernel stack happens to have a register containing a kernel virtual address
that's within the panic() function?
My assumption was that in "print_stack_text_syms()" in the following loop
everything on the stack is checked for kernel symbols:
for (i = (esp - bt->stackbase)/sizeof(ulong);
i < LONGS_PER_STACK; i++) {
Besides of function call return addresses there are local variables on the
stack. If a variable "by chance" contains an address inside the
"panic"
function, we will incorrectly assume that we have the panic task.
Is this understanding wrong?
Michael
10:05 a.m.
New subject: [PATCH] crash: Do not use bt -t flag in panic_search()
----- Original Message -----
On Mon, 3 Aug 2015 13:16:19 -0400 (EDT)
Dave Anderson <anderson(a)redhat.com> wrote:
>
>
> ----- Original Message -----
> > On Mon, 03 Aug 2015 12:30:22 -0400
> > Dave Anderson <anderson(a)prospeed.net> wrote:
> >
> > >
> > >
> > > Ah, it must be because of the common user-kernel virtual address space on
> > > s390x?
> >
> > Yes we have overlapping virtual address ranges for kernel and user space and
user
> > space registers are stored at the start of the kernel stack.
> >
> > But I am not sure, if that is the reason here (have to verify this).
> >
> > Doesn't the -t option check all the stack memory and looks for valid
symbol
> > addresses? So IMO everything found there could be by chance the address
> > of panic. Why is this no problem on x86?
>
> I'm not sure what you mean. How could it be there by chance? Are you talking
> about a highly unlikely situation where a user-space exception frame at the top
> of the kernel stack happens to have a register containing a kernel virtual address
> that's within the panic() function?
My assumption was that in "print_stack_text_syms()" in the following loop
everything on the stack is checked for kernel symbols:
for (i = (esp - bt->stackbase)/sizeof(ulong);
i < LONGS_PER_STACK; i++) {
Besides of function call return addresses there are local variables on the
stack. If a variable "by chance" contains an address inside the
"panic"
function, we will incorrectly assume that we have the panic task.
Is this understanding wrong?
Michael
No, you are right -- so let's prevent that functionality from ever being called
during the initialization of s390x live dumps.
It sounds like the sample vmcore core was taken by the the firmware since the
system was hung. Do the standalone facility and zgetdump *both* set the
live magic word at KVADDR 0?
Dave
3250
days inactive
3257
days old
devel@lists.crash-utility.osci.io
4 comments
3 participants
participants (3)
-
Dave Anderson -
Dave Anderson -
Michael Holzheu