2:32 a.m.
Hi Dave,
I‘m working on arm64 kdump by crash-7.2.7++.
There is a potential segmentation violation due to an invalid exception frame before
transitioning to the process stack when try using the bt command's "-S
<stack-address>" options.
For example, take the sp argument from the log:
[ 84.048650] pc : _raw_spin_lock+0x30/0x88
[ 84.048658] lr : lowmem_scan+0x45c/0xbd0
[ 84.048661] sp : ffffff800c42ba00 pstate : 20c00145
A segmentation violation is generated as below:
crash> bt -S ffffff800c42ba00 108
PID: 108 TASK: ffffffcd74122000 CPU: 5 COMMAND: "rtmm_reclaim"
bt: WARNING: cannot determine starting stack frame for task ffffffcd74122000
Program received signal SIGSEGV, Segmentation fault.
0x000055555572de2b in arm64_is_kernel_exception_frame (bt=0x7fffffffd640,
stkptr=18446743644915693792) at arm64.c:1785
warning: Source file is more recent than executable.
1785 if (INSTACK(regs->sp, bt) && INSTACK(regs->regs[29], bt)
&&
(gdb) bt
#0 0x000055555572de2b in arm64_is_kernel_exception_frame (bt=0x7fffffffd640,
stkptr=18446743644915693792) at arm64.c:1785
#1 0x000055555572ffaf in arm64_back_trace_cmd (bt=0x7fffffffd640) at arm64.c:2594
#2 0x00005555556ef0c4 in back_trace (bt=0x7fffffffd640) at kernel.c:3164
#3 0x00005555556ed624 in cmd_bt () at kernel.c:2833
#4 0x000055555564a73b in exec_command () at main.c:879
#5 0x000055555564a515 in main_loop () at main.c:826
#6 0x00005555558b5b43 in captured_command_loop (data=data@entry=0x0) at main.c:258
#7 0x00005555558b46ca in catch_errors (func=func@entry=0x5555558b5b30
<captured_command_loop>, func_args=func_args@entry=0x0,
errstring=errstring@entry=0x555555b0b728 "", mask=mask@entry=6) at
exceptions.c:557
#8 0x00005555558b6c42 in captured_main (data=data@entry=0x7fffffffdfb0) at main.c:1064
#9 0x00005555558b46ca in catch_errors (func=func@entry=0x5555558b5e70
<captured_main>, func_args=func_args@entry=0x7fffffffdfb0,
errstring=errstring@entry=0x555555b0b728 "", mask=mask@entry=6) at
exceptions.c:557
#10 0x00005555558b702e in gdb_main (args=0x7fffffffdfb0) at main.c:1079
#11 gdb_main_entry (argc=<optimized out>, argv=<optimized out>) at
main.c:1099
#12 0x000055555570fc53 in gdb_main_loop (argc=2, argv=0x7fffffffe148) at
gdb_interface.c:76
#13 0x000055555564a1e0 in main (argc=4, argv=0x7fffffffe148) at main.c:707
(gdb) p /x *(struct bt_info *) 0x7fffffffd640
$4 = {task = 0xffffffcd74122000, flags = 0x0, instptr = 0x0, stkptr = 0xffffff800c42ba00,
bptr = 0x0, stackbase = 0xffffff800c428000,
stacktop = 0xffffff800c42c000, stackbuf = 0x555555f23ae0, tc = 0x5555596e1778, hp =
0x7fffffffd5f0, textlist = 0x0, ref = 0x0, frameptr = 0x0,
call_target = 0x0, machdep = 0x0, debug = 0x0, eframe_ip = 0x0, radix = 0x0, cpumask =
0x0}
The stackframe.fp(0xffffff9c29e4f8e0) is larger than the stacktop address, so lead to
segmentation violation gernarated by accessing regs->sp:
(gdb) p /x 18446743644915693792//stkptr
$5 = 0xffffff9c29e4f8e0
(gdb) p /x 0xffffff9c29e4f8e0-0xffffff800c428000//STACK_OFFSET_TYPE(stkptr)
$6 = 0x1c1da278e0
(gdb) p /x regs
$7 = 0x55717394b3c0
(gdb) p *(struct arm64_pt_regs *) 0x55717394b3c0
Cannot access memory at address 0x55717394b3c0
For fix this, I think it must be add a condition
"arm64_in_exception_text(stackframe.pc) && INSTACK(stackframe.fp, bt)"
to avoid
an invalid exception frame before transitioning to the process stack.
The patch file has been upload to attachment.
Thanks for your review. I’m looking forward to your favourable reply!
Best regards,
Qiwu
#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
This e-mail and its attachments contain confidential information from XIAOMI, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!******/#
9:28 a.m.
----- Original Message -----
Hi Dave,
I‘m working on arm64 kdump by crash-7.2.7++.
There is a potential segmentation violation due to an invalid exception frame
before
transitioning to the process stack when try using the bt command's "-S
<stack-address>" options.
For example, take the sp argument from the log:
[ 84.048650] pc : _raw_spin_lock+0x30/0x88
[ 84.048658] lr : lowmem_scan+0x45c/0xbd0
[ 84.048661] sp : ffffff800c42ba00 pstate : 20c00145
A segmentation violation is generated as below:
crash> bt -S ffffff800c42ba00 108
PID: 108 TASK: ffffffcd74122000 CPU: 5 COMMAND: "rtmm_reclaim"
bt: WARNING: cannot determine starting stack frame for task ffffffcd74122000
Program received signal SIGSEGV, Segmentation fault.
0x000055555572de2b in arm64_is_kernel_exception_frame (bt=0x7fffffffd640,
stkptr=18446743644915693792) at arm64.c:1785
warning: Source file is more recent than executable.
1785 if (INSTACK(regs->sp, bt) && INSTACK(regs->regs[29], bt)
&&
(gdb) bt
#0 0x000055555572de2b in arm64_is_kernel_exception_frame (bt=0x7fffffffd640,
stkptr=18446743644915693792) at arm64.c:1785
#1 0x000055555572ffaf in arm64_back_trace_cmd (bt=0x7fffffffd640) at
arm64.c:2594
#2 0x00005555556ef0c4 in back_trace (bt=0x7fffffffd640) at kernel.c:3164
#3 0x00005555556ed624 in cmd_bt () at kernel.c:2833
#4 0x000055555564a73b in exec_command () at main.c:879
#5 0x000055555564a515 in main_loop () at main.c:826
#6 0x00005555558b5b43 in captured_command_loop (data=data@entry=0x0) at
main.c:258
#7 0x00005555558b46ca in catch_errors (func=func@entry=0x5555558b5b30
<captured_command_loop>, func_args=func_args@entry=0x0,
errstring=errstring@entry=0x555555b0b728 "", mask=mask@entry=6) at
exceptions.c:557
#8 0x00005555558b6c42 in captured_main (data=data@entry=0x7fffffffdfb0) at
main.c:1064
#9 0x00005555558b46ca in catch_errors (func=func@entry=0x5555558b5e70
<captured_main>, func_args=func_args@entry=0x7fffffffdfb0,
errstring=errstring@entry=0x555555b0b728 "", mask=mask@entry=6) at
exceptions.c:557
#10 0x00005555558b702e in gdb_main (args=0x7fffffffdfb0) at main.c:1079
#11 gdb_main_entry (argc=<optimized out>, argv=<optimized out>) at
main.c:1099
#12 0x000055555570fc53 in gdb_main_loop (argc=2, argv=0x7fffffffe148) at
gdb_interface.c:76
#13 0x000055555564a1e0 in main (argc=4, argv=0x7fffffffe148) at main.c:707
(gdb) p /x *(struct bt_info *) 0x7fffffffd640
$4 = {task = 0xffffffcd74122000, flags = 0x0, instptr = 0x0, stkptr =
0xffffff800c42ba00, bptr = 0x0, stackbase = 0xffffff800c428000,
stacktop = 0xffffff800c42c000, stackbuf = 0x555555f23ae0, tc =
0x5555596e1778, hp = 0x7fffffffd5f0, textlist = 0x0, ref = 0x0, frameptr =
0x0,
call_target = 0x0, machdep = 0x0, debug = 0x0, eframe_ip = 0x0, radix =
0x0, cpumask = 0x0}
The stackframe.fp(0xffffff9c29e4f8e0) is larger than the stacktop address, so
lead to segmentation violation gernarated by accessing regs->sp:
(gdb) p /x 18446743644915693792//stkptr
$5 = 0xffffff9c29e4f8e0
(gdb) p /x 0xffffff9c29e4f8e0-0xffffff800c428000//STACK_OFFSET_TYPE(stkptr)
$6 = 0x1c1da278e0
(gdb) p /x regs
$7 = 0x55717394b3c0
(gdb) p *(struct arm64_pt_regs *) 0x55717394b3c0
Cannot access memory at address 0x55717394b3c0
For fix this, I think it must be add a condition
"arm64_in_exception_text(stackframe.pc) && INSTACK(stackframe.fp, bt)"
to avoid
an invalid exception frame before transitioning to the process stack.
The patch file has been upload to attachment.
Thanks for your review. I’m looking forward to your favourable reply!
Best regards,
Qiwu
I cannot reproduce this using when I use "bt -S" with the "sp" address
found in
the log. When I do so, the backtrace is identical to when I enter "bt" alone,
so I'm not clear on why yours is behaving differently?
Is this a kdump-generated dumpfile?
Have you looked into why you get the "bt: WARNING: cannot determine
starting stack frame for task ffffffcd74122000" message?
You didn't show the results of your patch -- if you apply it, does the
backtrace get displayed correctly?
Since the "bt -S" option is almost never used. Would it be possible to
restrict your patch to fix/verify things in the section where it handles
the bt->hp->sp setting?
2555 } else if (bt->hp && bt->hp->esp) {
2556 if (arm64_on_irq_stack(bt->tc->processor,
bt->hp->esp)) {
2557 arm64_set_irq_stack(bt);
2558 bt->flags |= BT_IRQSTACK;
2559 }
2560 stackframe.fp = GET_STACK_ULONG(bt->hp->esp - 8);
2561 stackframe.pc = bt->hp->eip ?
2562 bt->hp->eip : GET_STACK_ULONG(bt->hp->esp);
2563 stackframe.sp = bt->hp->esp + 8;
2564 bt->flags &= ~BT_REGS_NOT_FOUND;
2565 }
It seems that you could check whether your stackframe.fp is larger than the stacktop
address, and for that matter what is the stackframe.pc that you see?
Dave
9:39 a.m.
----- Original Message -----
> The stackframe.fp(0xffffff9c29e4f8e0) is larger than the
stacktop address, so
> lead to segmentation violation gernarated by accessing regs->sp:
> (gdb) p /x 18446743644915693792//stkptr
> $5 = 0xffffff9c29e4f8e0
> (gdb) p /x 0xffffff9c29e4f8e0-0xffffff800c428000//STACK_OFFSET_TYPE(stkptr)
> $6 = 0x1c1da278e0
> (gdb) p /x regs
> $7 = 0x55717394b3c0
> (gdb) p *(struct arm64_pt_regs *) 0x55717394b3c0
> Cannot access memory at address 0x55717394b3c0
>
> For fix this, I think it must be add a condition
"arm64_in_exception_text(stackframe.pc) && INSTACK(stackframe.fp, bt)"
> to avoid an invalid exception frame before transitioning to the process stack.
Or alternatively, would it be better to have arm64_is_kernel_exception_frame()
verify that the "regs" assignment is legitimate, and if not, just return FALSE?
Dave
5:09 a.m.
New subject: [External Mail]Re: [PATCH] Fix a potential segfault for the ARM64 "bt -S <stack-address>" command
Hi Dave,
1. Is this a kdump-generated dumpfile?
It's a kdump-generated dumpfile for arm64.
2. Have you looked into why you get the "bt: WARNING: cannot determine starting stack
frame for task ffffffcd74122000" message?
Because kernel didn't enable crash_notes symbol to save active task regs.
3. You didn't show the results of your patch -- if you apply it, does the backtrace
get displayed correctly?
From the result of my patch, it shows bade stack frame for sp address
0xffffff800c42ba00.
crash> bt -S ffffff800c42ba00 108
PID: 108 TASK: ffffffcd74122000 CPU: 5 COMMAND: "rtmm_reclaim"
bt: WARNING: cannot determine starting stack frame for task ffffffcd74122000
#0 [ffffff9c29e4fa10] (null) at fffffffffffffffc
4. Since the "bt -S" option is almost never used. Would it be possible to
restrict your patch to fix/verify things in the section where it handles the
bt->hp->sp setting?
I add the following change to print where it handles the bt->hp->sp setting:
--- a/arm64.c
+++ b/arm64.c
@@ -2542,7 +2542,8 @@ arm64_back_trace_cmd(struct bt_info *bt)
* x+8: contains stackframe.pc -- text return address
* x+16: is the stackframe.sp address
*/
-
+ fprintf(stderr, "bt:flags=%llx, bptr=%lx, eip=%lx, esp=%lx, stkptr=%lx,
instptr=%lx, frameptr=%lx\n",
+ bt->flags, bt->bptr, bt->hp->eip, bt->hp->esp,
bt->stkptr, bt->instptr, bt->frameptr);
if (bt->flags & BT_KDUMP_ADJUST) {
if (arm64_on_irq_stack(bt->tc->processor, bt->bptr)) {
arm64_set_irq_stack(bt);
@@ -2572,6 +2573,7 @@ arm64_back_trace_cmd(struct bt_info *bt)
stackframe.fp = bt->frameptr;
}
+ fprintf(stderr, "stackframe:sp=%lx, pc=%lx, fp=%lx\n", stackframe.sp,
stackframe.pc, stackframe.fp);
if (bt->flags & BT_TEXT_SYMBOLS) {
arm64_print_text_symbols(bt, &stackframe, ofp);
if (BT_REFERENCE_FOUND(bt)) {
The result shows as below:
crash> bt -S ffffff800c42ba00 108
bt:flags=4000000000000, bptr=0, eip=0, esp=ffffff800c42ba00, stkptr=ffffff800c42ba00,
instptr=0, frameptr=0
stackframe:sp=ffffff800c42ba08, pc=0, fp=ffffff9c29e4fa10
It seems invalid stackframe.sp and pc calculated by GET_STACK_ULONG(bt->hp->esp). I
think it must be resulted from invalid bt->stackbuf address.
(gdb) p /x *(struct bt_info *) 0x7fffffffd640
$4 = {task = 0xffffffcd74122000, flags = 0x0, instptr = 0x0, stkptr = 0xffffff800c42ba00,
bptr = 0x0, stackbase = 0xffffff800c428000,
stacktop = 0xffffff800c42c000, stackbuf = 0x555555f23ae0, tc = 0x5555596e1778, hp =
0x7fffffffd5f0, textlist = 0x0, ref = 0x0, frameptr = 0x0,
call_target = 0x0, machdep = 0x0, debug = 0x0, eframe_ip = 0x0, radix = 0x0, cpumask =
0x0}
so this is the reason for that matter what is the stackframe.pc and stackframe.fp.
Best regards,
Qiwu
-----Original Message-----
From: Dave Anderson <anderson(a)redhat.com>
Sent: Monday, November 4, 2019 11:39 PM
To: Discussion list for crash utility usage, maintenance and development
<crash-utility(a)redhat.com>
Cc: 陈启武 <chenqiwu(a)xiaomi.com>
Subject: [External Mail]Re: [Crash-utility] [PATCH] Fix a potential segfault for the ARM64
"bt -S <stack-address>" command
----- Original Message -----
> The stackframe.fp(0xffffff9c29e4f8e0) is larger than the
stacktop
> address, so lead to segmentation violation gernarated by accessing regs->sp:
> (gdb) p /x 18446743644915693792//stkptr
> $5 = 0xffffff9c29e4f8e0
> (gdb) p /x
> 0xffffff9c29e4f8e0-0xffffff800c428000//STACK_OFFSET_TYPE(stkptr)
> $6 = 0x1c1da278e0
> (gdb) p /x regs
> $7 = 0x55717394b3c0
> (gdb) p *(struct arm64_pt_regs *) 0x55717394b3c0 Cannot access
> memory at address 0x55717394b3c0
>
> For fix this, I think it must be add a condition
"arm64_in_exception_text(stackframe.pc) && INSTACK(stackframe.fp, bt)"
> to avoid an invalid exception frame before transitioning to the process stack.
Or alternatively, would it be better to have arm64_is_kernel_exception_frame() verify that
the "regs" assignment is legitimate, and if not, just return FALSE?
Dave
#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
This e-mail and its attachments contain confidential information from XIAOMI, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!******/#
8:28 a.m.
New subject: [External Mail]Re: [PATCH] Fix a potential segfault for the ARM64 "bt -S <stack-address>" command
Ok, so since this is simply a fix to prevent a SIGSEGV, then my alternative
suggestion to have arm64_is_kernel_exception_frame() return FALSE if the
"regs" address assignment is invalid should suffice.
Thanks,
Dave
----- Original Message -----
Hi Dave,
1. Is this a kdump-generated dumpfile?
It's a kdump-generated dumpfile for arm64.
2. Have you looked into why you get the "bt: WARNING: cannot determine
starting stack frame for task ffffffcd74122000" message?
Because kernel didn't enable crash_notes symbol to save active task regs.
3. You didn't show the results of your patch -- if you apply it, does the
backtrace get displayed correctly?
From the result of my patch, it shows bade stack frame for sp address
0xffffff800c42ba00.
crash> bt -S ffffff800c42ba00 108
PID: 108 TASK: ffffffcd74122000 CPU: 5 COMMAND: "rtmm_reclaim"
bt: WARNING: cannot determine starting stack frame for task ffffffcd74122000
#0 [ffffff9c29e4fa10] (null) at fffffffffffffffc
4. Since the "bt -S" option is almost never used. Would it be possible to
restrict your patch to fix/verify things in the section where it handles the
bt->hp->sp setting?
I add the following change to print where it handles the bt->hp->sp setting:
--- a/arm64.c
+++ b/arm64.c
@@ -2542,7 +2542,8 @@ arm64_back_trace_cmd(struct bt_info *bt)
* x+8: contains stackframe.pc -- text return address
* x+16: is the stackframe.sp address
*/
-
+ fprintf(stderr, "bt:flags=%llx, bptr=%lx, eip=%lx, esp=%lx,
stkptr=%lx, instptr=%lx, frameptr=%lx\n",
+ bt->flags, bt->bptr, bt->hp->eip, bt->hp->esp,
bt->stkptr,
bt->instptr, bt->frameptr);
if (bt->flags & BT_KDUMP_ADJUST) {
if (arm64_on_irq_stack(bt->tc->processor, bt->bptr)) {
arm64_set_irq_stack(bt);
@@ -2572,6 +2573,7 @@ arm64_back_trace_cmd(struct bt_info *bt)
stackframe.fp = bt->frameptr;
}
+ fprintf(stderr, "stackframe:sp=%lx, pc=%lx, fp=%lx\n", stackframe.sp,
stackframe.pc, stackframe.fp);
if (bt->flags & BT_TEXT_SYMBOLS) {
arm64_print_text_symbols(bt, &stackframe, ofp);
if (BT_REFERENCE_FOUND(bt)) {
The result shows as below:
crash> bt -S ffffff800c42ba00 108
bt:flags=4000000000000, bptr=0, eip=0, esp=ffffff800c42ba00,
stkptr=ffffff800c42ba00, instptr=0, frameptr=0
stackframe:sp=ffffff800c42ba08, pc=0, fp=ffffff9c29e4fa10
It seems invalid stackframe.sp and pc calculated by
GET_STACK_ULONG(bt->hp->esp). I think it must be resulted from invalid
bt->stackbuf address.
(gdb) p /x *(struct bt_info *) 0x7fffffffd640
$4 = {task = 0xffffffcd74122000, flags = 0x0, instptr = 0x0, stkptr =
0xffffff800c42ba00, bptr = 0x0, stackbase = 0xffffff800c428000,
stacktop = 0xffffff800c42c000, stackbuf = 0x555555f23ae0, tc =
0x5555596e1778, hp = 0x7fffffffd5f0, textlist = 0x0, ref = 0x0, frameptr =
0x0,
call_target = 0x0, machdep = 0x0, debug = 0x0, eframe_ip = 0x0, radix =
0x0, cpumask = 0x0}
so this is the reason for that matter what is the stackframe.pc and
stackframe.fp.
Best regards,
Qiwu
-----Original Message-----
From: Dave Anderson <anderson(a)redhat.com>
Sent: Monday, November 4, 2019 11:39 PM
To: Discussion list for crash utility usage, maintenance and development
<crash-utility(a)redhat.com>
Cc: 陈启武 <chenqiwu(a)xiaomi.com>
Subject: [External Mail]Re: [Crash-utility] [PATCH] Fix a potential segfault
for the ARM64 "bt -S <stack-address>" command
----- Original Message -----
> > The stackframe.fp(0xffffff9c29e4f8e0) is larger than the stacktop
> > address, so lead to segmentation violation gernarated by accessing
> > regs->sp:
> > (gdb) p /x 18446743644915693792//stkptr
> > $5 = 0xffffff9c29e4f8e0
> > (gdb) p /x
> > 0xffffff9c29e4f8e0-0xffffff800c428000//STACK_OFFSET_TYPE(stkptr)
> > $6 = 0x1c1da278e0
> > (gdb) p /x regs
> > $7 = 0x55717394b3c0
> > (gdb) p *(struct arm64_pt_regs *) 0x55717394b3c0 Cannot access
> > memory at address 0x55717394b3c0
> >
> > For fix this, I think it must be add a condition
> > "arm64_in_exception_text(stackframe.pc) && INSTACK(stackframe.fp,
bt)"
> > to avoid an invalid exception frame before transitioning to the process
> > stack.
Or alternatively, would it be better to have
arm64_is_kernel_exception_frame() verify that the "regs" assignment is
legitimate, and if not, just return FALSE?
Dave
#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！
This e-mail and its attachments contain confidential information from
XIAOMI, which is intended only for the person or entity whose address is
listed above. Any use of the information contained herein in any way
(including, but not limited to, total or partial disclosure, reproduction,
or dissemination) by persons other than the intended recipient(s) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!******/#
1848
days inactive
1849
days old
devel@lists.crash-utility.osci.io
4 comments
2 participants
participants (2)
-
Dave Anderson -
陈启武