2:37 p.m.
Hi Dave
The following patch adds support for DWARF CFI based stack unwinding
for crash. Since this method uses the call frame instructions for
unwinding, it generates better backtraces than the existing backtrace
mechanism. So when we have the unwind info available, this new method
will be called, else we fall back to the existing mechanism.
... <this section moved below>
Please provide your suggestions and comments.
Thanks
Rachita
Hi Rachita,
I've only been able to test this on a live system that has __start_unwind
and __end_unwind symbols, so I don't know what a backtrace with an
in-kernel exception frame, or a backtrace with a transition to the x86_64
IRQ stack or x86_64 exception stacks, would look like. If you have
an example, I'd be interested in seeing how they get handled.
But what's there, i.e., with only user-mode-to-kernel-mode exception
frames being displayed, looks pretty good!
I'd like to tinker with it a bit, and fold most of what you've
done into 4.0-3.8. Then you can use that as a source base to
continue.
I do have a few questions/comments.
This netdump.c patch doesn't make sense to me -- we really don't
want to be doing any ELFSTORE operations in the debug-only
netdump_memory_dump() debug function:
@@ -771,8 +772,11 @@ netdump_memory_dump(FILE *fp)
dump_Elf64_Phdr(nd->load64 + i, ELFREAD);
offset64 = nd->notes64->p_offset;
for (tot = 0; tot < nd->notes64->p_filesz; tot += len) {
+ if (has_unwind_info)
+ len = dump_Elf64_Nhdr(offset64, ELFSTORE);
+ else
len = dump_Elf64_Nhdr(offset64, ELFREAD);
- offset64 += len;
+ offset64 += len;
}
break;
}
This patch below looks to only be necessary in dumpfiles, but it seems
like, given that the x86_64 user_regs_struct is unavailable in 2.6
vmlinux files, that the initializations in get_netdump_regs_x86_64()
would never get done -- because VALID_STRUCT(user_regs_struct) would
fail, right?
@@ -1562,8 +1566,10 @@ get_netdump_regs_x86_64(struct bt_info *
if (is_task_active(bt->task))
bt->flags |= BT_DUMPFILE_SEARCH;
- if ((NETDUMP_DUMPFILE() || KDUMP_DUMPFILE()) &&
- VALID_STRUCT(user_regs_struct) && (bt->task == tt->panic_task))
{
+ if (((NETDUMP_DUMPFILE() || KDUMP_DUMPFILE()) &&
+ VALID_STRUCT(user_regs_struct) && (bt->task ==
tt->panic_task)) ||
+ (KDUMP_DUMPFILE() && has_unwind_info && (bt->flags &
+ BT_DUMPFILE_SEARCH))) {
if (nd->num_prstatus_notes > 1)
note = (Elf64_Nhdr *)
nd->nt_prstatus_percpu[bt->tc->processor];
@@ -1574,9 +1580,21 @@ get_netdump_regs_x86_64(struct bt_info *
len = roundup(len + note->n_namesz, 4);
len = roundup(len + note->n_descsz, 4);
+ if KDUMP_DUMPFILE() {
+ ASSIGN_SIZE(user_regs_struct) = 27 * sizeof(unsigned long);
+ ASSIGN_OFFSET(user_regs_struct_rsp) = 19 * sizeof(unsigned long);
+ ASSIGN_OFFSET(user_regs_struct_rip) = 16 * sizeof(unsigned long);
+ }
user_regs = ((char *)note + len)
- SIZE(user_regs_struct) - sizeof(long);
+ if KDUMP_DUMPFILE() {
+ *rspp = *(ulong *)(user_regs + OFFSET(user_regs_struct_rsp));
+ *ripp = *(ulong *)(user_regs + OFFSET(user_regs_struct_rip));
+ if (*ripp && *rspp)
+ return;
+ }
+
But then again, perhaps you never needed the user_regs_struct_rsp and
user_regs_struct_rip offsets in your test scenario?
There does seem to be some unnecessary "kernel-port" left-overs that
should be pruned. Like the __get_user_nocheck(), __get_user_size()
and __get_user_asm() definitions are superfluous, since they're only
needed by __get_user(), which is not used.
I'll also make it compilable in other than x86_64 environments,
because the unwind_x86_64.c code should be #ifdef'd X86_64.
There still are a couple of things which need to be done, viz
1. Extend to obtaining unwind info from modules as well(currently
doing only for the kernel)
Shouldn't pose a major problem -- just requires following the links
from the kernel table AFAICT. But that leads to another question...
What happens, in dwarf_backtrace(), when it encounters a module
frame? My guess is that the call to unwind() will fail, and then
the loop will bail out prematurely, and end up truncating the
trace output? We'll need some type of error-handling there I
would think.
2. Currently reading the unwind info from eh_frame section only(ie
__start_unwind to __end_unwind). Need to add facility to read from
the .debug_frame(if .debug_frame is present in cases where .eh_frame
is absent. Will have to read from the vmlinux if we want to read the
.debug_frame info)
Definitely -- we've got a plethora of kernels that have the CFI stuff
in the vmlinux file, but not in the kernel.
3. Add FRAME_POINTER support.
Personally, I don't much care about this...
And don't forget about x86 support!
And we should probably -- for now anyway -- make it possible to
turn this capability on and off at will. Also, for now, it should
probably default to off until we pound on it a bit. For example,
things like "bt -l" is not supported in these scheme.
But, I can't emphasize this enough -- this is a nice piece of work
that you've done here.
I'll try to get something out in the next couple of days.
Thanks,
Dave
3:09 a.m.
New subject: [RFC] Crash patch for DWARF CFI based unwind support
On Tue, Oct 17, 2006 at 03:37:16PM -0400, Dave Anderson wrote:
<...snipped the rest for another mail...>
> Thanks,
> Dave
Thanks for all the comments..
Thanks
Rachita
This netdump.c patch doesn't make sense to me -- we really don't
want to be doing any ELFSTORE operations in the debug-only
netdump_memory_dump() debug function:
@@ -771,8 +772,11 @@ netdump_memory_dump(FILE *fp)
dump_Elf64_Phdr(nd->load64 + i, ELFREAD);
offset64 = nd->notes64->p_offset;
for (tot = 0; tot < nd->notes64->p_filesz; tot += len) {
+ if (has_unwind_info)
+ len = dump_Elf64_Nhdr(offset64, ELFSTORE);
+ else
len = dump_Elf64_Nhdr(offset64, ELFREAD);
- offset64 += len;
+ offset64 += len;
}
break;
}
Yes, you are right! I missed this one..
The store is already being done by the is_kdump().
This patch below looks to only be necessary in dumpfiles, but it seems
like, given that the x86_64 user_regs_struct is unavailable in 2.6
vmlinux files, that the initializations in get_netdump_regs_x86_64()
would never get done -- because VALID_STRUCT(user_regs_struct) would
fail, right?
@@ -1562,8 +1566,10 @@ get_netdump_regs_x86_64(struct bt_info *
if (is_task_active(bt->task))
bt->flags |= BT_DUMPFILE_SEARCH;
- if ((NETDUMP_DUMPFILE() || KDUMP_DUMPFILE()) &&
- VALID_STRUCT(user_regs_struct) && (bt->task ==
tt->panic_task)) {
+ if (((NETDUMP_DUMPFILE() || KDUMP_DUMPFILE()) &&
+ VALID_STRUCT(user_regs_struct) && (bt->task ==
tt->panic_task)) ||
+ (KDUMP_DUMPFILE() && has_unwind_info && (bt->flags
&
+ BT_DUMPFILE_SEARCH))) {
I add this last check to workaround the user_regs_struct not getting
initialized problem. Thats why I omit the VALID_STRUCT() check, and let it
pass if we have the unwind_info.
This should be a temporary arrangement till we fix the user_regs_struct
problem. The idea here is to read the register states for all the active
tasks from the NT_PRSTATUS section of the dumpfile for kdump(in the case
where we have the has_unwind_info set). The other case for kdump where we
do not have the unwind_info stays as it is.
if (nd->num_prstatus_notes > 1)
note = (Elf64_Nhdr *)
nd->nt_prstatus_percpu[bt->tc->processor];
@@ -1574,9 +1580,21 @@ get_netdump_regs_x86_64(struct bt_info *
len = roundup(len + note->n_namesz, 4);
len = roundup(len + note->n_descsz, 4);
+ if KDUMP_DUMPFILE() {
+ ASSIGN_SIZE(user_regs_struct) = 27 * sizeof(unsigned long);
+ ASSIGN_OFFSET(user_regs_struct_rsp) = 19 * sizeof(unsigned
long);
+ ASSIGN_OFFSET(user_regs_struct_rip) = 16 * sizeof(unsigned
long);
+ }
user_regs = ((char *)note + len)
- SIZE(user_regs_struct) - sizeof(long);
+ if KDUMP_DUMPFILE() {
+ *rspp = *(ulong *)(user_regs + OFFSET(user_regs_struct_rsp));
+ *ripp = *(ulong *)(user_regs + OFFSET(user_regs_struct_rip));
+ if (*ripp && *rspp)
+ return;
+ }
+
But then again, perhaps you never needed the user_regs_struct_rsp and
user_regs_struct_rip offsets in your test scenario?
I needed them., thats why had to sort of hard code them here :)
But again, this is temporary till we fix the user_regs_struct issue :)
There does seem to be some unnecessary "kernel-port" left-overs that
should be pruned. Like the __get_user_nocheck(), __get_user_size()
and __get_user_asm() definitions are superfluous, since they're only
needed by __get_user(), which is not used.
Actually __get_user is a TBD.
9:16 a.m.
New subject: [RFC] Crash patch for DWARF CFI based unwind support
On Tue, Oct 17, 2006 at 03:37:16PM -0400, Dave Anderson wrote:
> Hi Dave
>
> The following patch adds support for DWARF CFI based stack unwinding
> for crash. Since this method uses the call frame instructions for
> unwinding, it generates better backtraces than the existing backtrace
> mechanism. So when we have the unwind info available, this new method
> will be called, else we fall back to the existing mechanism.
>
> ... <this section moved below>
>
> Please provide your suggestions and comments.
>
> Thanks
> Rachita
Hi Rachita,
I've only been able to test this on a live system that has __start_unwind
and __end_unwind symbols, so I don't know what a backtrace with an
in-kernel exception frame, or a backtrace with a transition to the x86_64
IRQ stack or x86_64 exception stacks, would look like. If you have
an example, I'd be interested in seeing how they get handled.
Hi Dave
I was trying to use LKDTM to create various scenarios for crash dump.
To start with is the case of panic() in an interrupt context. Here I
am inducing a panic in handle_IRQ_event(), where I am registering a
jprobe. jp_handle_irq_event() is the jprobe handler which in turn calls
lkdtm_handler(). Running crash on the dump gives the following:
crash> bt
PID: 3898 TASK: ffff81022e988e20 CPU: 0 COMMAND: "slapd"
#0 [ffffffff8064bcf8] crash_kexec at ffffffff80152211
#1 [ffffffff8064bd40] machine_kexec at ffffffff8011a739
#2 [ffffffff8064bd80] crash_kexec at ffffffff8015222d
#3 [ffffffff8064be08] crash_kexec at ffffffff80152211
#4 [ffffffff8064be30] bust_spinlocks at ffffffff8011fd6d
#5 [ffffffff8064be40] panic at ffffffff80131410
#6 [ffffffff8064beb0] cdrom_pc_intr at ffffffff802ebe68
#7 [ffffffff8064bef0] ide_intr at ffffffff802df26f
#8 [ffffffff8064bf30] lkdtm_handler at ffffffff8800230d
#9 [ffffffff8064bf40] jp_handle_irq_event at ffffffff880023e8
#10 [ffffffff8064bf50] __do_IRQ at ffffffff801544f4
#11 [ffffffff8064bf58] __do_softirq at ffffffff80136b8f
#12 [ffffffff8064bf90] do_IRQ at ffffffff8010bda1
--- <IRQ stack> ---
#13 [ffff810229fd5f80] ret_from_intr at ffffffff80109b95
[exception RIP: unknown or invalid address]
RIP: 0000000000000000 RSP: 0000000000000000 RFLAGS: 00000000
RAX: ffffffffffffffff RBX: 00002afe35608c98 RCX: 00002afe359f7be4
RDX: 0000000000000033 RSI: 0000000000000202 RDI: 00007fff754bfbe0
RBP: 000000000000000a R8: 000055555590bca0 R9: 0000000000000000
R10: 00002afe35608c98 R11: 0000000000000001 R12: 0000000000000000
R13: 0000000000000000 R14: 00002afe3597b1e0 R15: 000055555590b760
ORIG_RAX: 000000000000002b CS: 0000 SS: 0000
bt: WARNING: possibly bogus exception frame
RIP: 00002afe359f7be4 RSP: 00007fff754bfbe0 RFLAGS: 00000202
RAX: 00002afe35608c98 RBX: 000055555590b760 RCX: 0000000000000001
RDX: 00002afe35608c98 RSI: 0000000000000000 RDI: 000055555590bca0
RBP: ffffffff80109c0b R8: 000000000000000a R9: 0000000000000000
R10: 0000000000000000 R11: 00002afe3597b1e0 R12: 000055555590b760
R13: 00007fff754bfd38 R14: 0000000000000001 R15: 000055555590b760
ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
crash> set unwind on
unwind: on
crash> bt
PID: 3898 TASK: ffff81022e988e20 CPU: 0 COMMAND: "slapd"
#0 [ffffffff8064bd88] crash_kexec at ffffffff80152211
#1 [ffffffff8064be48] panic at ffffffff80131410
#2 [ffffffff8064bf38] lkdtm_handler at ffffffff8800230d
--- <IRQ stack> ---
#3 [ffff810229fd5f80] ret_from_intr at ffffffff80109b95
[exception RIP: unknown or invalid address]
RIP: 0000000000000000 RSP: 0000000000000000 RFLAGS: 00000000
RAX: ffffffffffffffff RBX: 00002afe35608c98 RCX: 00002afe359f7be4
RDX: 0000000000000033 RSI: 0000000000000202 RDI: 00007fff754bfbe0
RBP: 000000000000000a R8: 000055555590bca0 R9: 0000000000000000
R10: 00002afe35608c98 R11: 0000000000000001 R12: 0000000000000000
R13: 0000000000000000 R14: 00002afe3597b1e0 R15: 000055555590b760
ORIG_RAX: 000000000000002b CS: 0000 SS: 0000
bt: WARNING: possibly bogus exception frame
#4 [ffff810229fd5f80] common_interrupt at ffffffff80109b95
RIP: 00002afe359f7be4 RSP: 00007fff754bfbe0 RFLAGS: 00000202
RAX: 00002afe35608c98 RBX: 000055555590b760 RCX: 0000000000000001
RDX: 00002afe35608c98 RSI: 0000000000000000 RDI: 000055555590bca0
RBP: ffffffff80109c0b R8: 000000000000000a R9: 0000000000000000
R10: 0000000000000000 R11: 00002afe3597b1e0 R12: 000055555590b760
R13: 00007fff754bfd38 R14: 0000000000000001 R15: 000055555590b760
ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
crash>
Comments?
In the stacktrace with 'unwind on', I was expecting to see jp_handle_irq_event
appear too (as frame 3)..Could my using a module to register the probe be the
reason ?
Thanks
Rachita
10:46 a.m.
New subject: [RFC] Crash patch for DWARF CFI based unwind support
Rachita Kothiyal wrote:
> Hi Rachita,
>
> I've only been able to test this on a live system that has __start_unwind
> and __end_unwind symbols, so I don't know what a backtrace with an
> in-kernel exception frame, or a backtrace with a transition to the x86_64
> IRQ stack or x86_64 exception stacks, would look like. If you have
> an example, I'd be interested in seeing how they get handled.
Hi Dave
I was trying to use LKDTM to create various scenarios for crash dump.
To start with is the case of panic() in an interrupt context. Here I
am inducing a panic in handle_IRQ_event(), where I am registering a
jprobe. jp_handle_irq_event() is the jprobe handler which in turn calls
lkdtm_handler(). Running crash on the dump gives the following:
crash> bt
PID: 3898 TASK: ffff81022e988e20 CPU: 0 COMMAND: "slapd"
#0 [ffffffff8064bcf8] crash_kexec at ffffffff80152211
#1 [ffffffff8064bd40] machine_kexec at ffffffff8011a739
#2 [ffffffff8064bd80] crash_kexec at ffffffff8015222d
#3 [ffffffff8064be08] crash_kexec at ffffffff80152211
#4 [ffffffff8064be30] bust_spinlocks at ffffffff8011fd6d
#5 [ffffffff8064be40] panic at ffffffff80131410
#6 [ffffffff8064beb0] cdrom_pc_intr at ffffffff802ebe68
#7 [ffffffff8064bef0] ide_intr at ffffffff802df26f
#8 [ffffffff8064bf30] lkdtm_handler at ffffffff8800230d
#9 [ffffffff8064bf40] jp_handle_irq_event at ffffffff880023e8
#10 [ffffffff8064bf50] __do_IRQ at ffffffff801544f4
#11 [ffffffff8064bf58] __do_softirq at ffffffff80136b8f
#12 [ffffffff8064bf90] do_IRQ at ffffffff8010bda1
--- <IRQ stack> ---
#13 [ffff810229fd5f80] ret_from_intr at ffffffff80109b95
[exception RIP: unknown or invalid address]
RIP: 0000000000000000 RSP: 0000000000000000 RFLAGS: 00000000
RAX: ffffffffffffffff RBX: 00002afe35608c98 RCX: 00002afe359f7be4
RDX: 0000000000000033 RSI: 0000000000000202 RDI: 00007fff754bfbe0
RBP: 000000000000000a R8: 000055555590bca0 R9: 0000000000000000
R10: 00002afe35608c98 R11: 0000000000000001 R12: 0000000000000000
R13: 0000000000000000 R14: 00002afe3597b1e0 R15: 000055555590b760
ORIG_RAX: 000000000000002b CS: 0000 SS: 0000
bt: WARNING: possibly bogus exception frame
RIP: 00002afe359f7be4 RSP: 00007fff754bfbe0 RFLAGS: 00000202
RAX: 00002afe35608c98 RBX: 000055555590b760 RCX: 0000000000000001
RDX: 00002afe35608c98 RSI: 0000000000000000 RDI: 000055555590bca0
RBP: ffffffff80109c0b R8: 000000000000000a R9: 0000000000000000
R10: 0000000000000000 R11: 00002afe3597b1e0 R12: 000055555590b760
R13: 00007fff754bfd38 R14: 0000000000000001 R15: 000055555590b760
ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
crash> set unwind on
unwind: on
crash> bt
PID: 3898 TASK: ffff81022e988e20 CPU: 0 COMMAND: "slapd"
#0 [ffffffff8064bd88] crash_kexec at ffffffff80152211
#1 [ffffffff8064be48] panic at ffffffff80131410
#2 [ffffffff8064bf38] lkdtm_handler at ffffffff8800230d
--- <IRQ stack> ---
#3 [ffff810229fd5f80] ret_from_intr at ffffffff80109b95
[exception RIP: unknown or invalid address]
RIP: 0000000000000000 RSP: 0000000000000000 RFLAGS: 00000000
RAX: ffffffffffffffff RBX: 00002afe35608c98 RCX: 00002afe359f7be4
RDX: 0000000000000033 RSI: 0000000000000202 RDI: 00007fff754bfbe0
RBP: 000000000000000a R8: 000055555590bca0 R9: 0000000000000000
R10: 00002afe35608c98 R11: 0000000000000001 R12: 0000000000000000
R13: 0000000000000000 R14: 00002afe3597b1e0 R15: 000055555590b760
ORIG_RAX: 000000000000002b CS: 0000 SS: 0000
bt: WARNING: possibly bogus exception frame
#4 [ffff810229fd5f80] common_interrupt at ffffffff80109b95
RIP: 00002afe359f7be4 RSP: 00007fff754bfbe0 RFLAGS: 00000202
RAX: 00002afe35608c98 RBX: 000055555590b760 RCX: 0000000000000001
RDX: 00002afe35608c98 RSI: 0000000000000000 RDI: 000055555590bca0
RBP: ffffffff80109c0b R8: 000000000000000a R9: 0000000000000000
R10: 0000000000000000 R11: 00002afe3597b1e0 R12: 000055555590b760
R13: 00007fff754bfd38 R14: 0000000000000001 R15: 000055555590b760
ORIG_RAX: ffffffffffffffff CS: 0033 SS: 002b
crash>
Comments?
Not really. I don't understand why IRQ exception frame contents
look the way they do. The crash utility's "irq_eframe" value,
which gets assigned from a fixed location at the top of the
interrupt stack, is obviously pointing to a bogus exception frame.
There might be something different in newer kernels, although I
don't see it in 2.6.18?
If you do a "help -m", the very last line shows the base address
of each per-cpu interrupt stack, as in:
crash> help -m
...
ibase[cpus]:
ffffffff80453d00 0000010037e28000 0000010037e60000 0000010037e94000
crash>
Each of the interrupt stacks are 16k long, so you can
look at them with a "rd" with a count of 2048. The linkage
between the interrupt stack and the stack from where it came
is supposed to be at the first word just below the top 64 words
on that stack.
So if I take cpu 2's interrupt stack from above, which is at
10037e60000, and dump it using "rd -s", I see this:
crash> rd -s 10037e60000 2048
...
10037e63f90: 0000007fbffff4b0 0000007fbffff410
10037e63fa0: 0000000000000046 0000000000000046
10037e63fb0: apic_timer_interrupt+0x85 000001003741df58
10037e63fc0: 0000000000000000 0000000000000000
10037e63fd0: 0000000000000000 0000000000000000
10037e63fe0: 0000000000000000 0000000000000000
10037e63ff0: 0000000000000000 0000000000000000
crash>
The linkage to the previous stack is at IRQ stack address
10037e63fb8, which points back to a process stack address of
1003741df58. That process stack address is also a pointer
to the exception frame that was laid down there:
crash> pt_regs 000001003741df58
struct pt_regs {
r15 = 0x0,
r14 = 0x1003741df58,
r13 = 0x7fbfffe840,
r12 = 0xa8,
rbp = 0x17dbfc0,
rbx = 0xffffffff801103ce,
r11 = 0x246,
r10 = 0x31d072f928,
r9 = 0x4587d9,
r8 = 0x3741df58,
rax = 0xffffffffffffffda,
rcx = 0xffffffffffffffff,
rdx = 0xa8,
rsi = 0x17dbfc0,
rdi = 0x4,
orig_rax = 0x0,
rip = 0x31d05b85b2,
cs = 0x33,
eflags = 0x246,
rsp = 0x7fbfffe808,
ss = 0x2b
}
So the last time an interrupt occurred on this
cpu, a program was running in user space (cs 0x33)
when a timer interrupt occurred.
If it had occurred in kernel mode, a kernel mode
exception frame should be there, So for example,
if I take cpu 3's interrupt stack from above, it
shows this:
crash> rd -s 0000010037e94000 2048
...
10037e97f90: 0000000000000000 0000000000000000 ................
10037e97fa0: 0000000000000046 0000000000000046 F.......F.......
10037e97fb0: apic_timer_interrupt+0x85 0000010037e83e98 .........>.7....
10037e97fc0: 0000000000000000 0000000000000000 ................
10037e97fd0: 0000000000000000 0000000000000000 ................
10037e97fe0: 0000000000000000 0000000000000000 ................
10037e97ff0: 0000000000000000 0000000000000000 ................
crash>
And in this case, the timer interrupt occurred while running
in kernel mode (cs 0x10):
crash> pt_regs 0000010037e83e98
struct pt_regs {
r15 = 0x0,
r14 = 0x0,
r13 = 0x0,
r12 = 0x0,
rbp = 0x1,
rbx = 0x0,
r11 = 0x5,
r10 = 0x0,
r9 = 0x8,
r8 = 0x10037e82000,
rax = 0x0,
rcx = 0x0,
rdx = 0x0,
rsi = 0x1003f047030,
rdi = 0x1000102a7e0,
orig_rax = 0xffffffffffffffef,
rip = 0xffffffff8010e84c,
cs = 0x10,
eflags = 0x246,
rsp = 0x10037e83f48,
ss = 0x18
}
crash>
The rip above shows that the cpu was in idle:
crash> sym 0xffffffff8010e84c
ffffffff8010e84c (t) mwait_idle+0x56 include/asm/thread_info.h: 63
crash>
So I cannot explain what's going on in your kernel,
where it looks like that basic stack-linkage assumption
is being violated, or has changed somehow? Maybe kdump
or jprobes has done something? I don't know what else to
tell you.
In the stacktrace with 'unwind on', I was expecting to see jp_handle_irq_event
appear too (as frame 3)..Could my using a module to register the probe be the
reason ?
I don't know. Maybe since the jprobes procedure mucks with
the text, it somehow alter the unwind mechanism?
Helpless,
Dave
8:32 a.m.
New subject: [RFC] Crash patch for DWARF CFI based unwind support
Hi Dave
The following patch is a consolidated cleanup and minor fixes attempt.
Specifically it does the following:
1. Saves register values(rip, rsp, rbp) got from the exception frame.
These are used to unwind the process stack. Added a 'bptr' field
in the bt_info structure to save the rbp value from the eframe.
2. Added checks for offset being zero while resolving the text address
to a symbol name.
3. Added dwarf_print_stack_entry() as a parallel to x86_print_stack_entry().
Could probably move the offset checking routine as a separate function.
Would save us some lines of code. Pushed that for later.
4. Fixed 'levels' in the backtrace output.
Thanks
Rachita
Signed-off-by: Rachita Kothiyal <rachita(a)in.ibm.com>
---
defs.h | 3
unwind_x86_32_64.c | 77 ++++++++++++++++++++++---
x86_64.c | 163 ++++++-----------------------------------------------
3 files changed, 91 insertions(+), 152 deletions(-)
diff -puN x86_64.c~cfi_backtrace_minor_fixes x86_64.c
--- crash-4.0-3.9/x86_64.c~cfi_backtrace_minor_fixes 2006-11-15 18:29:04.848359480 +0530
+++ crash-4.0-3.9-rachita/x86_64.c 2006-11-15 19:00:40.687148176 +0530
@@ -2548,6 +2548,7 @@ x86_64_dwarf_back_trace_cmd(struct bt_in
irq_eframe = 0;
last_process_stack_eframe = 0;
bt->call_target = NULL;
+ bt->bptr = 0;
rsp = bt->stkptr;
if (!rsp) {
error(INFO, "cannot determine starting stack pointer\n");
@@ -2617,31 +2618,9 @@ in_exception_stack:
stacktop = bt->stacktop - SIZE(pt_regs);
- if ((kt->flags & DWARF_UNWIND) && !done)
- done = dwarf_backtrace(bt, stacktop);
-
- for (i = (rsp - bt->stackbase)/sizeof(ulong);
- !done && (rsp < stacktop); i++, rsp += sizeof(ulong)) {
-
- up = (ulong *)(&bt->stackbuf[i*sizeof(ulong)]);
-
- if (!is_kernel_text(*up))
- continue;
-
- switch (x86_64_print_stack_entry(bt, ofp, level, i,*up))
- {
- case BACKTRACE_ENTRY_AND_EFRAME_DISPLAYED:
- rsp += SIZE(pt_regs);
- i += SIZE(pt_regs)/sizeof(ulong);
- case BACKTRACE_ENTRY_DISPLAYED:
- level++;
- break;
- case BACKTRACE_ENTRY_IGNORED:
- break;
- case BACKTRACE_COMPLETE:
- done = TRUE;
- break;
- }
+ if ((kt->flags & DWARF_UNWIND) && !done) {
+ level = dwarf_backtrace(bt, level, stacktop);
+ done = TRUE;
}
cs = x86_64_exception_frame(EFRAME_PRINT|EFRAME_CS, 0,
@@ -2702,32 +2681,10 @@ in_exception_stack:
stacktop = bt->stacktop - 64; /* from kernel code */
- if ((kt->flags & DWARF_UNWIND) && !done)
- done = dwarf_backtrace(bt, stacktop);
-
- for (i = (rsp - bt->stackbase)/sizeof(ulong);
- !done && (rsp < stacktop); i++, rsp += sizeof(ulong)) {
-
- up = (ulong *)(&bt->stackbuf[i*sizeof(ulong)]);
-
- if (!is_kernel_text(*up))
- continue;
-
- switch (x86_64_print_stack_entry(bt, ofp, level, i,*up))
- {
- case BACKTRACE_ENTRY_AND_EFRAME_DISPLAYED:
- rsp += SIZE(pt_regs);
- i += SIZE(pt_regs)/sizeof(ulong);
- case BACKTRACE_ENTRY_DISPLAYED:
- level++;
- break;
- case BACKTRACE_ENTRY_IGNORED:
- break;
- case BACKTRACE_COMPLETE:
- done = TRUE;
- break;
- }
- }
+ if ((kt->flags & DWARF_UNWIND) && !done) {
+ level = dwarf_backtrace(bt, level, stacktop);
+ done = TRUE;
+ }
if (!BT_REFERENCE_CHECK(bt))
fprintf(fp, "--- <IRQ stack> ---\n");
@@ -2792,21 +2749,6 @@ in_exception_stack:
}
/*
- * For a normally blocked task, hand-create the first level.
- */
- if (!done && !(kt->flags & DWARF_UNWIND) &&
- !(bt->flags & (BT_TEXT_SYMBOLS|BT_EXCEPTION_STACK|BT_IRQSTACK)) &&
- STREQ(closest_symbol(bt->instptr), "thread_return")) {
- bt->flags |= BT_SCHEDULE;
- i = (rsp - bt->stackbase)/sizeof(ulong);
- x86_64_print_stack_entry(bt, ofp, level,
- i, bt->instptr);
- bt->flags &= ~(ulonglong)BT_SCHEDULE;
- rsp += sizeof(ulong);
- level++;
- }
-
- /*
* Dump the IRQ exception frame from the process stack.
* If the CS register indicates a user exception frame,
* then set done to TRUE to avoid the process stack walk-through.
@@ -2814,8 +2756,7 @@ in_exception_stack:
*/
if (irq_eframe) {
bt->flags |= BT_EXCEPTION_FRAME;
- i = (irq_eframe - bt->stackbase)/sizeof(ulong);
- x86_64_print_stack_entry(bt, ofp, level, i, bt->instptr);
+ level = dwarf_print_stack_entry(bt, level);
bt->flags &= ~(ulonglong)BT_EXCEPTION_FRAME;
cs = x86_64_exception_frame(EFRAME_PRINT|EFRAME_CS, 0,
bt->stackbuf + (irq_eframe - bt->stackbase), bt, ofp);
@@ -2833,81 +2774,10 @@ in_exception_stack:
/*
* Walk the process stack.
*/
- if ((kt->flags & DWARF_UNWIND) && !done)
- done = dwarf_backtrace(bt, bt->stacktop);
-
- for (i = (rsp - bt->stackbase)/sizeof(ulong);
- !done && (rsp < bt->stacktop); i++, rsp += sizeof(ulong)) {
-
- up = (ulong *)(&bt->stackbuf[i*sizeof(ulong)]);
-
- if (!is_kernel_text(*up))
- continue;
-
- if ((bt->flags & BT_CHECK_CALLER)) {
- /*
- * A non-zero offset value from the value_search()
- * lets us know if it's a real text return address.
- */
- spt = value_search(*up, &offset);
- /*
- * sp gets the syment of the function that the text
- * routine above called before leaving its return
- * address on the stack -- if it can be determined.
- */
- sp = x86_64_function_called_by((*up)-5);
-
- if (sp == NULL) {
- /*
- * We were unable to get the called function.
- * If the text address had an offset, then
- * it must have made an indirect call, and
- * can't have called our target function.
- */
- if (offset) {
- if (CRASHDEBUG(1))
- fprintf(ofp,
- "< ignoring %s() -- makes indirect call and NOT
%s()>\n",
- spt->name,
- bt->call_target);
- continue;
- }
- } else if ((machdep->flags & SCHED_TEXT) &&
- STREQ(bt->call_target, "schedule") &&
- STREQ(sp->name, "__sched_text_start")) {
- ; /* bait and switch */
- } else if (!STREQ(sp->name, bt->call_target)) {
- /*
- * We got function called by the text routine,
- * but it's not our target function.
- */
- if (CRASHDEBUG(2))
- fprintf(ofp,
- "< ignoring %s() -- calls %s() and NOT %s()>\n",
- spt->name, sp->name,
- bt->call_target);
- continue;
- }
- }
-
- switch (x86_64_print_stack_entry(bt, ofp, level, i,*up))
- {
- case BACKTRACE_ENTRY_AND_EFRAME_DISPLAYED:
- last_process_stack_eframe = rsp + 8;
- if (x86_64_print_eframe_location(last_process_stack_eframe, level, ofp))
- level++;
- rsp += SIZE(pt_regs);
- i += SIZE(pt_regs)/sizeof(ulong);
- case BACKTRACE_ENTRY_DISPLAYED:
- level++;
- break;
- case BACKTRACE_ENTRY_IGNORED:
- break;
- case BACKTRACE_COMPLETE:
- done = TRUE;
- break;
- }
- }
+ if ((kt->flags & DWARF_UNWIND) && !done) {
+ level = dwarf_backtrace(bt, level, bt->stacktop);
+ done = TRUE;
+ }
if (!irq_eframe && !is_kernel_thread(bt->tc->task) &&
(GET_STACKBASE(bt->tc->task) == bt->stackbase)) {
@@ -3193,6 +3063,13 @@ x86_64_exception_frame(ulong flags, ulon
x86_64_do_bt_reference_check(bt, r15, NULL);
}
+ /* Remember the rip and rsp for unwinding the process stack */
+ if (kt->flags & DWARF_UNWIND){
+ bt->instptr = rip;
+ bt->stkptr = rsp;
+ bt->bptr = rbp;
+ }
+
if (kvaddr)
FREEBUF(pt_regs_buf);
diff -puN unwind_x86_32_64.c~cfi_backtrace_minor_fixes unwind_x86_32_64.c
--- crash-4.0-3.9/unwind_x86_32_64.c~cfi_backtrace_minor_fixes 2006-11-15
18:35:50.519688064 +0530
+++ crash-4.0-3.9-rachita/unwind_x86_32_64.c 2006-11-15 19:31:45.705622272 +0530
@@ -1036,9 +1036,8 @@ dump_local_unwind_tables(void)
int
-dwarf_backtrace(struct bt_info *bt, ulong stacktop)
+dwarf_backtrace(struct bt_info *bt, int level, ulong stacktop)
{
- int n = 0;
unsigned long bp, offset;
struct syment *sp;
char *name;
@@ -1051,7 +1050,7 @@ dwarf_backtrace(struct bt_info *bt, ulon
UNW_PC(frame) = bt->instptr;
/* read rbp from stack for non active tasks */
- if (!(bt->flags & BT_DUMPFILE_SEARCH) ) {
+ if (!(bt->flags & BT_DUMPFILE_SEARCH) && !bt->bptr) {
// readmem(frame->regs.rsp, KVADDR, &bp,
readmem(UNW_SP(frame), KVADDR, &bp,
sizeof(unsigned long), "reading bp", FAULT_ON_ERROR);
@@ -1084,7 +1083,7 @@ dwarf_backtrace(struct bt_info *bt, ulon
name = sp->name;
- fprintf(fp, " #0 [%016lx] %s at %016lx \n", UNW_SP(frame), name,
UNW_PC(frame));
+ fprintf(fp, " #%d [%016lx] %s at %016lx \n", level, UNW_SP(frame), name,
UNW_PC(frame));
if (CRASHDEBUG(2))
fprintf(fp, " < SP: %lx PC: %lx FP: %lx >\n", UNW_SP(frame),
@@ -1092,7 +1091,12 @@ dwarf_backtrace(struct bt_info *bt, ulon
while ((UNW_SP(frame) < stacktop)
&& !unwind(frame) && UNW_PC(frame)) {
- n++;
+ /* To prevent rip pushed on IRQ stack being reported both
+ * both on the IRQ and process stacks
+ */
+ if ((bt->flags & BT_IRQSTACK) && (UNW_SP(frame) >= stacktop - 16))
+ break;
+ level++;
sp = value_search(UNW_PC(frame), &offset);
if (!sp) {
if (CRASHDEBUG(1))
@@ -1101,9 +1105,24 @@ dwarf_backtrace(struct bt_info *bt, ulon
UNW_PC(frame));
break;
}
+
+ /*
+ * If offset is zero, it means we have crossed over to the next
+ * function. Recalculate by adjusting the text address
+ */
+ if (!offset) {
+ sp = value_search(UNW_PC(frame) - 1, &offset);
+ if (!sp) {
+ if (CRASHDEBUG(1))
+ fprintf(fp,
+ "unwind: cannot find symbol for PC: %lx\n",
+ UNW_PC(frame)-1);
+ goto bailout;
+ }
+ }
name = sp->name;
- fprintf(fp, "%s#%d [%016lx] %s at %016lx \n", n < 10 ? " " :
"",
- n, UNW_SP(frame), name, UNW_PC(frame));
+ fprintf(fp, "%s#%d [%016lx] %s at %016lx \n", level < 10 ? " " :
"",
+ level, UNW_SP(frame), name, UNW_PC(frame));
if (CRASHDEBUG(2))
fprintf(fp, " < SP: %lx PC: %lx FP: %lx >\n", UNW_SP(frame),
@@ -1112,7 +1131,49 @@ dwarf_backtrace(struct bt_info *bt, ulon
bailout:
FREEBUF(frame);
- return TRUE;
+ return ++level;
+}
+
+int dwarf_print_stack_entry(struct bt_info *bt, int level)
+{
+ int n = 0;
+ unsigned long offset;
+ struct syment *sp;
+ char *name;
+ struct unwind_frame_info *frame;
+
+ frame = (struct unwind_frame_info *)GETBUF(sizeof(struct unwind_frame_info));
+ UNW_SP(frame) = bt->stkptr;
+ UNW_PC(frame) = bt->instptr;
+
+ sp = value_search(UNW_PC(frame), &offset);
+ if (!sp) {
+ if (CRASHDEBUG(1))
+ fprintf(fp, "unwind: cannot find symbol for PC: %lx\n",
+ UNW_PC(frame));
+ goto bailout;
+ }
+
+ /*
+ * If offset is zero, it means we have crossed over to the next
+ * function. Recalculate by adjusting the text address
+ */
+ if (!offset) {
+ sp = value_search(UNW_PC(frame) - 1, &offset);
+ if (!sp) {
+ if (CRASHDEBUG(1))
+ fprintf(fp,
+ "unwind: cannot find symbol for PC: %lx\n",
+ UNW_PC(frame)-1);
+ goto bailout;
+ }
+ }
+ name = sp->name;
+ fprintf(fp, " #%d [%016lx] %s at %016lx \n", level, UNW_SP(frame), name,
UNW_PC(frame));
+
+bailout:
+ FREEBUF(frame);
+ return level;
}
void
diff -puN defs.h~cfi_backtrace_minor_fixes defs.h
--- crash-4.0-3.9/defs.h~cfi_backtrace_minor_fixes 2006-11-15 18:51:28.030164808 +0530
+++ crash-4.0-3.9-rachita/defs.h 2006-11-15 18:51:46.272391568 +0530
@@ -645,6 +645,7 @@ struct bt_info {
ulonglong flags;
ulong instptr;
ulong stkptr;
+ ulong bptr;
ulong stackbase;
ulong stacktop;
char *stackbuf;
@@ -3626,7 +3627,7 @@ struct machine_specific {
* unwind_x86_32_64.c
*/
void init_unwind_table(void);
-int dwarf_backtrace(struct bt_info *, ulong);
+int dwarf_backtrace(struct bt_info *, int, ulong);
void dwarf_debug(struct bt_info *);
#endif
_
9:05 a.m.
New subject: [RFC] Crash patch for DWARF CFI based unwind support
Rachita Kothiyal wrote:
Hi Dave
The following patch is a consolidated cleanup and minor fixes attempt.
Specifically it does the following:
1. Saves register values(rip, rsp, rbp) got from the exception frame.
These are used to unwind the process stack. Added a 'bptr' field
in the bt_info structure to save the rbp value from the eframe.
2. Added checks for offset being zero while resolving the text address
to a symbol name.
3. Added dwarf_print_stack_entry() as a parallel to x86_print_stack_entry().
Could probably move the offset checking routine as a separate function.
Would save us some lines of code. Pushed that for later.
4. Fixed 'levels' in the backtrace output.
Thanks
Rachita
Signed-off-by: Rachita Kothiyal <rachita(a)in.ibm.com>
---
Hi Rachita,
Great -- I'll tinker with this today and get out a new release ASAP.
One question re: this hanging thread:
> So, in other words, if we hardwire the user_regs_struct so that
> it uses the NT_PRSTATUS registers all the time, then we get
> the second (preferred/better) budget back trace when unwind
> is off.
>
> That being the case, I argue for hardwiring them all the time.
Yes, we can(should) do that for all the active tasks.
Rachita
Correct me if I'm wrong here, but...
If, during machdep_init(POST_GBD), if we force-initialize
(hardwire when necessary) the relevant user_regs_struct items,
then we could change this line from:
if (((NETDUMP_DUMPFILE() || KDUMP_DUMPFILE()) &&
VALID_STRUCT(user_regs_struct) && (bt->task == tt->panic_task))
||
(KDUMP_DUMPFILE() && (kt->flags & DWARF_UNWIND) &&
(bt->flags & BT_DUMPFILE_SEARCH))) {
to:
if (((NETDUMP_DUMPFILE() || KDUMP_DUMPFILE()) &&
VALID_STRUCT(user_regs_struct) && (bt->task == tt->panic_task))
||
(KDUMP_DUMPFILE() && (bt->flags & BT_DUMPFILE_SEARCH))) {
To be absolutely safe in a backwards-compatibility sense, perhaps
we should only do the hardwiring if it's a KDUMP_DUMPFILE().
Make sense?
Dave
defs.h | 3
unwind_x86_32_64.c | 77 ++++++++++++++++++++++---
x86_64.c | 163 ++++++-----------------------------------------------
3 files changed, 91 insertions(+), 152 deletions(-)
diff -puN x86_64.c~cfi_backtrace_minor_fixes x86_64.c
--- crash-4.0-3.9/x86_64.c~cfi_backtrace_minor_fixes 2006-11-15 18:29:04.848359480
+0530
+++ crash-4.0-3.9-rachita/x86_64.c 2006-11-15 19:00:40.687148176 +0530
@@ -2548,6 +2548,7 @@ x86_64_dwarf_back_trace_cmd(struct bt_in
irq_eframe = 0;
last_process_stack_eframe = 0;
bt->call_target = NULL;
+ bt->bptr = 0;
rsp = bt->stkptr;
if (!rsp) {
error(INFO, "cannot determine starting stack pointer\n");
@@ -2617,31 +2618,9 @@ in_exception_stack:
stacktop = bt->stacktop - SIZE(pt_regs);
- if ((kt->flags & DWARF_UNWIND) && !done)
- done = dwarf_backtrace(bt, stacktop);
-
- for (i = (rsp - bt->stackbase)/sizeof(ulong);
- !done && (rsp < stacktop); i++, rsp += sizeof(ulong)) {
-
- up = (ulong *)(&bt->stackbuf[i*sizeof(ulong)]);
-
- if (!is_kernel_text(*up))
- continue;
-
- switch (x86_64_print_stack_entry(bt, ofp, level, i,*up))
- {
- case BACKTRACE_ENTRY_AND_EFRAME_DISPLAYED:
- rsp += SIZE(pt_regs);
- i += SIZE(pt_regs)/sizeof(ulong);
- case BACKTRACE_ENTRY_DISPLAYED:
- level++;
- break;
- case BACKTRACE_ENTRY_IGNORED:
- break;
- case BACKTRACE_COMPLETE:
- done = TRUE;
- break;
- }
+ if ((kt->flags & DWARF_UNWIND) && !done) {
+ level = dwarf_backtrace(bt, level, stacktop);
+ done = TRUE;
}
cs = x86_64_exception_frame(EFRAME_PRINT|EFRAME_CS, 0,
@@ -2702,32 +2681,10 @@ in_exception_stack:
stacktop = bt->stacktop - 64; /* from kernel code */
- if ((kt->flags & DWARF_UNWIND) && !done)
- done = dwarf_backtrace(bt, stacktop);
-
- for (i = (rsp - bt->stackbase)/sizeof(ulong);
- !done && (rsp < stacktop); i++, rsp += sizeof(ulong)) {
-
- up = (ulong *)(&bt->stackbuf[i*sizeof(ulong)]);
-
- if (!is_kernel_text(*up))
- continue;
-
- switch (x86_64_print_stack_entry(bt, ofp, level, i,*up))
- {
- case BACKTRACE_ENTRY_AND_EFRAME_DISPLAYED:
- rsp += SIZE(pt_regs);
- i += SIZE(pt_regs)/sizeof(ulong);
- case BACKTRACE_ENTRY_DISPLAYED:
- level++;
- break;
- case BACKTRACE_ENTRY_IGNORED:
- break;
- case BACKTRACE_COMPLETE:
- done = TRUE;
- break;
- }
- }
+ if ((kt->flags & DWARF_UNWIND) && !done) {
+ level = dwarf_backtrace(bt, level, stacktop);
+ done = TRUE;
+ }
if (!BT_REFERENCE_CHECK(bt))
fprintf(fp, "--- <IRQ stack> ---\n");
@@ -2792,21 +2749,6 @@ in_exception_stack:
}
/*
- * For a normally blocked task, hand-create the first level.
- */
- if (!done && !(kt->flags & DWARF_UNWIND) &&
- !(bt->flags & (BT_TEXT_SYMBOLS|BT_EXCEPTION_STACK|BT_IRQSTACK))
&&
- STREQ(closest_symbol(bt->instptr), "thread_return")) {
- bt->flags |= BT_SCHEDULE;
- i = (rsp - bt->stackbase)/sizeof(ulong);
- x86_64_print_stack_entry(bt, ofp, level,
- i, bt->instptr);
- bt->flags &= ~(ulonglong)BT_SCHEDULE;
- rsp += sizeof(ulong);
- level++;
- }
-
- /*
* Dump the IRQ exception frame from the process stack.
* If the CS register indicates a user exception frame,
* then set done to TRUE to avoid the process stack walk-through.
@@ -2814,8 +2756,7 @@ in_exception_stack:
*/
if (irq_eframe) {
bt->flags |= BT_EXCEPTION_FRAME;
- i = (irq_eframe - bt->stackbase)/sizeof(ulong);
- x86_64_print_stack_entry(bt, ofp, level, i, bt->instptr);
+ level = dwarf_print_stack_entry(bt, level);
bt->flags &= ~(ulonglong)BT_EXCEPTION_FRAME;
cs = x86_64_exception_frame(EFRAME_PRINT|EFRAME_CS, 0,
bt->stackbuf + (irq_eframe - bt->stackbase), bt, ofp);
@@ -2833,81 +2774,10 @@ in_exception_stack:
/*
* Walk the process stack.
*/
- if ((kt->flags & DWARF_UNWIND) && !done)
- done = dwarf_backtrace(bt, bt->stacktop);
-
- for (i = (rsp - bt->stackbase)/sizeof(ulong);
- !done && (rsp < bt->stacktop); i++, rsp += sizeof(ulong)) {
-
- up = (ulong *)(&bt->stackbuf[i*sizeof(ulong)]);
-
- if (!is_kernel_text(*up))
- continue;
-
- if ((bt->flags & BT_CHECK_CALLER)) {
- /*
- * A non-zero offset value from the value_search()
- * lets us know if it's a real text return address.
- */
- spt = value_search(*up, &offset);
- /*
- * sp gets the syment of the function that the text
- * routine above called before leaving its return
- * address on the stack -- if it can be determined.
- */
- sp = x86_64_function_called_by((*up)-5);
-
- if (sp == NULL) {
- /*
- * We were unable to get the called function.
- * If the text address had an offset, then
- * it must have made an indirect call, and
- * can't have called our target function.
- */
- if (offset) {
- if (CRASHDEBUG(1))
- fprintf(ofp,
- "< ignoring %s() -- makes indirect call and NOT
%s()>\n",
- spt->name,
- bt->call_target);
- continue;
- }
- } else if ((machdep->flags & SCHED_TEXT) &&
- STREQ(bt->call_target, "schedule")
&&
- STREQ(sp->name, "__sched_text_start")) {
- ; /* bait and switch */
- } else if (!STREQ(sp->name, bt->call_target)) {
- /*
- * We got function called by the text routine,
- * but it's not our target function.
- */
- if (CRASHDEBUG(2))
- fprintf(ofp,
- "< ignoring %s() -- calls %s() and NOT
%s()>\n",
- spt->name, sp->name,
- bt->call_target);
- continue;
- }
- }
-
- switch (x86_64_print_stack_entry(bt, ofp, level, i,*up))
- {
- case BACKTRACE_ENTRY_AND_EFRAME_DISPLAYED:
- last_process_stack_eframe = rsp + 8;
- if (x86_64_print_eframe_location(last_process_stack_eframe,
level, ofp))
- level++;
- rsp += SIZE(pt_regs);
- i += SIZE(pt_regs)/sizeof(ulong);
- case BACKTRACE_ENTRY_DISPLAYED:
- level++;
- break;
- case BACKTRACE_ENTRY_IGNORED:
- break;
- case BACKTRACE_COMPLETE:
- done = TRUE;
- break;
- }
- }
+ if ((kt->flags & DWARF_UNWIND) && !done) {
+ level = dwarf_backtrace(bt, level, bt->stacktop);
+ done = TRUE;
+ }
if (!irq_eframe && !is_kernel_thread(bt->tc->task) &&
(GET_STACKBASE(bt->tc->task) == bt->stackbase)) {
@@ -3193,6 +3063,13 @@ x86_64_exception_frame(ulong flags, ulon
x86_64_do_bt_reference_check(bt, r15, NULL);
}
+ /* Remember the rip and rsp for unwinding the process stack */
+ if (kt->flags & DWARF_UNWIND){
+ bt->instptr = rip;
+ bt->stkptr = rsp;
+ bt->bptr = rbp;
+ }
+
if (kvaddr)
FREEBUF(pt_regs_buf);
diff -puN unwind_x86_32_64.c~cfi_backtrace_minor_fixes unwind_x86_32_64.c
--- crash-4.0-3.9/unwind_x86_32_64.c~cfi_backtrace_minor_fixes 2006-11-15
18:35:50.519688064 +0530
+++ crash-4.0-3.9-rachita/unwind_x86_32_64.c 2006-11-15 19:31:45.705622272 +0530
@@ -1036,9 +1036,8 @@ dump_local_unwind_tables(void)
int
-dwarf_backtrace(struct bt_info *bt, ulong stacktop)
+dwarf_backtrace(struct bt_info *bt, int level, ulong stacktop)
{
- int n = 0;
unsigned long bp, offset;
struct syment *sp;
char *name;
@@ -1051,7 +1050,7 @@ dwarf_backtrace(struct bt_info *bt, ulon
UNW_PC(frame) = bt->instptr;
/* read rbp from stack for non active tasks */
- if (!(bt->flags & BT_DUMPFILE_SEARCH) ) {
+ if (!(bt->flags & BT_DUMPFILE_SEARCH) && !bt->bptr) {
// readmem(frame->regs.rsp, KVADDR, &bp,
readmem(UNW_SP(frame), KVADDR, &bp,
sizeof(unsigned long), "reading bp", FAULT_ON_ERROR);
@@ -1084,7 +1083,7 @@ dwarf_backtrace(struct bt_info *bt, ulon
name = sp->name;
- fprintf(fp, " #0 [%016lx] %s at %016lx \n", UNW_SP(frame), name,
UNW_PC(frame));
+ fprintf(fp, " #%d [%016lx] %s at %016lx \n", level, UNW_SP(frame),
name, UNW_PC(frame));
if (CRASHDEBUG(2))
fprintf(fp, " < SP: %lx PC: %lx FP: %lx >\n",
UNW_SP(frame),
@@ -1092,7 +1091,12 @@ dwarf_backtrace(struct bt_info *bt, ulon
while ((UNW_SP(frame) < stacktop)
&& !unwind(frame) && UNW_PC(frame)) {
- n++;
+ /* To prevent rip pushed on IRQ stack being reported both
+ * both on the IRQ and process stacks
+ */
+ if ((bt->flags & BT_IRQSTACK) && (UNW_SP(frame) >=
stacktop - 16))
+ break;
+ level++;
sp = value_search(UNW_PC(frame), &offset);
if (!sp) {
if (CRASHDEBUG(1))
@@ -1101,9 +1105,24 @@ dwarf_backtrace(struct bt_info *bt, ulon
UNW_PC(frame));
break;
}
+
+ /*
+ * If offset is zero, it means we have crossed over to the next
+ * function. Recalculate by adjusting the text address
+ */
+ if (!offset) {
+ sp = value_search(UNW_PC(frame) - 1, &offset);
+ if (!sp) {
+ if (CRASHDEBUG(1))
+ fprintf(fp,
+ "unwind: cannot find symbol for PC:
%lx\n",
+ UNW_PC(frame)-1);
+ goto bailout;
+ }
+ }
name = sp->name;
- fprintf(fp, "%s#%d [%016lx] %s at %016lx \n", n < 10 ?
" " : "",
- n, UNW_SP(frame), name, UNW_PC(frame));
+ fprintf(fp, "%s#%d [%016lx] %s at %016lx \n", level < 10 ?
" " : "",
+ level, UNW_SP(frame), name, UNW_PC(frame));
if (CRASHDEBUG(2))
fprintf(fp, " < SP: %lx PC: %lx FP: %lx >\n",
UNW_SP(frame),
@@ -1112,7 +1131,49 @@ dwarf_backtrace(struct bt_info *bt, ulon
bailout:
FREEBUF(frame);
- return TRUE;
+ return ++level;
+}
+
+int dwarf_print_stack_entry(struct bt_info *bt, int level)
+{
+ int n = 0;
+ unsigned long offset;
+ struct syment *sp;
+ char *name;
+ struct unwind_frame_info *frame;
+
+ frame = (struct unwind_frame_info *)GETBUF(sizeof(struct unwind_frame_info));
+ UNW_SP(frame) = bt->stkptr;
+ UNW_PC(frame) = bt->instptr;
+
+ sp = value_search(UNW_PC(frame), &offset);
+ if (!sp) {
+ if (CRASHDEBUG(1))
+ fprintf(fp, "unwind: cannot find symbol for PC: %lx\n",
+ UNW_PC(frame));
+ goto bailout;
+ }
+
+ /*
+ * If offset is zero, it means we have crossed over to the next
+ * function. Recalculate by adjusting the text address
+ */
+ if (!offset) {
+ sp = value_search(UNW_PC(frame) - 1, &offset);
+ if (!sp) {
+ if (CRASHDEBUG(1))
+ fprintf(fp,
+ "unwind: cannot find symbol for PC:
%lx\n",
+ UNW_PC(frame)-1);
+ goto bailout;
+ }
+ }
+ name = sp->name;
+ fprintf(fp, " #%d [%016lx] %s at %016lx \n", level, UNW_SP(frame),
name, UNW_PC(frame));
+
+bailout:
+ FREEBUF(frame);
+ return level;
}
void
diff -puN defs.h~cfi_backtrace_minor_fixes defs.h
--- crash-4.0-3.9/defs.h~cfi_backtrace_minor_fixes 2006-11-15 18:51:28.030164808
+0530
+++ crash-4.0-3.9-rachita/defs.h 2006-11-15 18:51:46.272391568 +0530
@@ -645,6 +645,7 @@ struct bt_info {
ulonglong flags;
ulong instptr;
ulong stkptr;
+ ulong bptr;
ulong stackbase;
ulong stacktop;
char *stackbuf;
@@ -3626,7 +3627,7 @@ struct machine_specific {
* unwind_x86_32_64.c
*/
void init_unwind_table(void);
-int dwarf_backtrace(struct bt_info *, ulong);
+int dwarf_backtrace(struct bt_info *, int, ulong);
void dwarf_debug(struct bt_info *);
#endif
_
9:48 a.m.
New subject: [RFC] Crash patch for DWARF CFI based unwind support
On Wed, Nov 15, 2006 at 10:05:35AM -0500, Dave Anderson wrote:
Hi Rachita,
Great -- I'll tinker with this today and get out a new release ASAP.
One question re: this hanging thread:
> > So, in other words, if we hardwire the user_regs_struct so that
> > it uses the NT_PRSTATUS registers all the time, then we get
> > the second (preferred/better) budget back trace when unwind
> > is off.
> >
> > That being the case, I argue for hardwiring them all the time.
>
> Yes, we can(should) do that for all the active tasks.
>
> Rachita
>
Correct me if I'm wrong here, but...
If, during machdep_init(POST_GBD), if we force-initialize
(hardwire when necessary) the relevant user_regs_struct items,
Hi Dave
Yes, this should serve the purpose.
Thanks
Rachita
then we could change this line from:
if (((NETDUMP_DUMPFILE() || KDUMP_DUMPFILE()) &&
VALID_STRUCT(user_regs_struct) && (bt->task == tt->panic_task))
||
(KDUMP_DUMPFILE() && (kt->flags & DWARF_UNWIND) &&
(bt->flags & BT_DUMPFILE_SEARCH))) {
to:
if (((NETDUMP_DUMPFILE() || KDUMP_DUMPFILE()) &&
VALID_STRUCT(user_regs_struct) && (bt->task == tt->panic_task))
||
(KDUMP_DUMPFILE() && (bt->flags & BT_DUMPFILE_SEARCH))) {
To be absolutely safe in a backwards-compatibility sense, perhaps
we should only do the hardwiring if it's a KDUMP_DUMPFILE().
Make sense?
Dave
>
>
> defs.h | 3
> unwind_x86_32_64.c | 77 ++++++++++++++++++++++---
> x86_64.c | 163 ++++++-----------------------------------------------
> 3 files changed, 91 insertions(+), 152 deletions(-)
>
> diff -puN x86_64.c~cfi_backtrace_minor_fixes x86_64.c
> --- crash-4.0-3.9/x86_64.c~cfi_backtrace_minor_fixes 2006-11-15
18:29:04.848359480 +0530
> +++ crash-4.0-3.9-rachita/x86_64.c 2006-11-15 19:00:40.687148176 +0530
> @@ -2548,6 +2548,7 @@ x86_64_dwarf_back_trace_cmd(struct bt_in
> irq_eframe = 0;
> last_process_stack_eframe = 0;
> bt->call_target = NULL;
> + bt->bptr = 0;
> rsp = bt->stkptr;
> if (!rsp) {
> error(INFO, "cannot determine starting stack pointer\n");
> @@ -2617,31 +2618,9 @@ in_exception_stack:
>
> stacktop = bt->stacktop - SIZE(pt_regs);
>
> - if ((kt->flags & DWARF_UNWIND) && !done)
> - done = dwarf_backtrace(bt, stacktop);
> -
> - for (i = (rsp - bt->stackbase)/sizeof(ulong);
> - !done && (rsp < stacktop); i++, rsp += sizeof(ulong))
{
> -
> - up = (ulong *)(&bt->stackbuf[i*sizeof(ulong)]);
> -
> - if (!is_kernel_text(*up))
> - continue;
> -
> - switch (x86_64_print_stack_entry(bt, ofp, level, i,*up))
> - {
> - case BACKTRACE_ENTRY_AND_EFRAME_DISPLAYED:
> - rsp += SIZE(pt_regs);
> - i += SIZE(pt_regs)/sizeof(ulong);
> - case BACKTRACE_ENTRY_DISPLAYED:
> - level++;
> - break;
> - case BACKTRACE_ENTRY_IGNORED:
> - break;
> - case BACKTRACE_COMPLETE:
> - done = TRUE;
> - break;
> - }
> + if ((kt->flags & DWARF_UNWIND) && !done) {
> + level = dwarf_backtrace(bt, level, stacktop);
> + done = TRUE;
> }
>
> cs = x86_64_exception_frame(EFRAME_PRINT|EFRAME_CS, 0,
> @@ -2702,32 +2681,10 @@ in_exception_stack:
>
> stacktop = bt->stacktop - 64; /* from kernel code */
>
> - if ((kt->flags & DWARF_UNWIND) && !done)
> - done = dwarf_backtrace(bt, stacktop);
> -
> - for (i = (rsp - bt->stackbase)/sizeof(ulong);
> - !done && (rsp < stacktop); i++, rsp +=
sizeof(ulong)) {
> -
> - up = (ulong *)(&bt->stackbuf[i*sizeof(ulong)]);
> -
> - if (!is_kernel_text(*up))
> - continue;
> -
> - switch (x86_64_print_stack_entry(bt, ofp, level, i,*up))
> - {
> - case BACKTRACE_ENTRY_AND_EFRAME_DISPLAYED:
> - rsp += SIZE(pt_regs);
> - i += SIZE(pt_regs)/sizeof(ulong);
> - case BACKTRACE_ENTRY_DISPLAYED:
> - level++;
> - break;
> - case BACKTRACE_ENTRY_IGNORED:
> - break;
> - case BACKTRACE_COMPLETE:
> - done = TRUE;
> - break;
> - }
> - }
> + if ((kt->flags & DWARF_UNWIND) && !done) {
> + level = dwarf_backtrace(bt, level, stacktop);
> + done = TRUE;
> + }
>
> if (!BT_REFERENCE_CHECK(bt))
> fprintf(fp, "--- <IRQ stack> ---\n");
> @@ -2792,21 +2749,6 @@ in_exception_stack:
> }
>
> /*
> - * For a normally blocked task, hand-create the first level.
> - */
> - if (!done && !(kt->flags & DWARF_UNWIND) &&
> - !(bt->flags & (BT_TEXT_SYMBOLS|BT_EXCEPTION_STACK|BT_IRQSTACK))
&&
> - STREQ(closest_symbol(bt->instptr), "thread_return")) {
> - bt->flags |= BT_SCHEDULE;
> - i = (rsp - bt->stackbase)/sizeof(ulong);
> - x86_64_print_stack_entry(bt, ofp, level,
> - i, bt->instptr);
> - bt->flags &= ~(ulonglong)BT_SCHEDULE;
> - rsp += sizeof(ulong);
> - level++;
> - }
> -
> - /*
> * Dump the IRQ exception frame from the process stack.
> * If the CS register indicates a user exception frame,
> * then set done to TRUE to avoid the process stack walk-through.
> @@ -2814,8 +2756,7 @@ in_exception_stack:
> */
> if (irq_eframe) {
> bt->flags |= BT_EXCEPTION_FRAME;
> - i = (irq_eframe - bt->stackbase)/sizeof(ulong);
> - x86_64_print_stack_entry(bt, ofp, level, i, bt->instptr);
> + level = dwarf_print_stack_entry(bt, level);
> bt->flags &= ~(ulonglong)BT_EXCEPTION_FRAME;
> cs = x86_64_exception_frame(EFRAME_PRINT|EFRAME_CS, 0,
> bt->stackbuf + (irq_eframe - bt->stackbase), bt,
ofp);
> @@ -2833,81 +2774,10 @@ in_exception_stack:
> /*
> * Walk the process stack.
> */
> - if ((kt->flags & DWARF_UNWIND) && !done)
> - done = dwarf_backtrace(bt, bt->stacktop);
> -
> - for (i = (rsp - bt->stackbase)/sizeof(ulong);
> - !done && (rsp < bt->stacktop); i++, rsp += sizeof(ulong))
{
> -
> - up = (ulong *)(&bt->stackbuf[i*sizeof(ulong)]);
> -
> - if (!is_kernel_text(*up))
> - continue;
> -
> - if ((bt->flags & BT_CHECK_CALLER)) {
> - /*
> - * A non-zero offset value from the value_search()
> - * lets us know if it's a real text return address.
> - */
> - spt = value_search(*up, &offset);
> - /*
> - * sp gets the syment of the function that the text
> - * routine above called before leaving its return
> - * address on the stack -- if it can be determined.
> - */
> - sp = x86_64_function_called_by((*up)-5);
> -
> - if (sp == NULL) {
> - /*
> - * We were unable to get the called function.
> - * If the text address had an offset, then
> - * it must have made an indirect call, and
> - * can't have called our target function.
> - */
> - if (offset) {
> - if (CRASHDEBUG(1))
> - fprintf(ofp,
> - "< ignoring %s() -- makes indirect call and NOT
%s()>\n",
> - spt->name,
> - bt->call_target);
> - continue;
> - }
> - } else if ((machdep->flags & SCHED_TEXT) &&
> - STREQ(bt->call_target, "schedule")
&&
> - STREQ(sp->name, "__sched_text_start"))
{
> - ; /* bait and switch */
> - } else if (!STREQ(sp->name, bt->call_target)) {
> - /*
> - * We got function called by the text routine,
> - * but it's not our target function.
> - */
> - if (CRASHDEBUG(2))
> - fprintf(ofp,
> - "< ignoring %s() -- calls %s() and NOT
%s()>\n",
> - spt->name, sp->name,
> - bt->call_target);
> - continue;
> - }
> - }
> -
> - switch (x86_64_print_stack_entry(bt, ofp, level, i,*up))
> - {
> - case BACKTRACE_ENTRY_AND_EFRAME_DISPLAYED:
> - last_process_stack_eframe = rsp + 8;
> - if (x86_64_print_eframe_location(last_process_stack_eframe,
level, ofp))
> - level++;
> - rsp += SIZE(pt_regs);
> - i += SIZE(pt_regs)/sizeof(ulong);
> - case BACKTRACE_ENTRY_DISPLAYED:
> - level++;
> - break;
> - case BACKTRACE_ENTRY_IGNORED:
> - break;
> - case BACKTRACE_COMPLETE:
> - done = TRUE;
> - break;
> - }
> - }
> + if ((kt->flags & DWARF_UNWIND) && !done) {
> + level = dwarf_backtrace(bt, level, bt->stacktop);
> + done = TRUE;
> + }
>
> if (!irq_eframe && !is_kernel_thread(bt->tc->task)
&&
> (GET_STACKBASE(bt->tc->task) == bt->stackbase)) {
> @@ -3193,6 +3063,13 @@ x86_64_exception_frame(ulong flags, ulon
> x86_64_do_bt_reference_check(bt, r15, NULL);
> }
>
> + /* Remember the rip and rsp for unwinding the process stack */
> + if (kt->flags & DWARF_UNWIND){
> + bt->instptr = rip;
> + bt->stkptr = rsp;
> + bt->bptr = rbp;
> + }
> +
> if (kvaddr)
> FREEBUF(pt_regs_buf);
>
> diff -puN unwind_x86_32_64.c~cfi_backtrace_minor_fixes unwind_x86_32_64.c
> --- crash-4.0-3.9/unwind_x86_32_64.c~cfi_backtrace_minor_fixes 2006-11-15
18:35:50.519688064 +0530
> +++ crash-4.0-3.9-rachita/unwind_x86_32_64.c 2006-11-15 19:31:45.705622272 +0530
> @@ -1036,9 +1036,8 @@ dump_local_unwind_tables(void)
>
>
> int
> -dwarf_backtrace(struct bt_info *bt, ulong stacktop)
> +dwarf_backtrace(struct bt_info *bt, int level, ulong stacktop)
> {
> - int n = 0;
> unsigned long bp, offset;
> struct syment *sp;
> char *name;
> @@ -1051,7 +1050,7 @@ dwarf_backtrace(struct bt_info *bt, ulon
> UNW_PC(frame) = bt->instptr;
>
> /* read rbp from stack for non active tasks */
> - if (!(bt->flags & BT_DUMPFILE_SEARCH) ) {
> + if (!(bt->flags & BT_DUMPFILE_SEARCH) && !bt->bptr) {
> // readmem(frame->regs.rsp, KVADDR, &bp,
> readmem(UNW_SP(frame), KVADDR, &bp,
> sizeof(unsigned long), "reading bp",
FAULT_ON_ERROR);
> @@ -1084,7 +1083,7 @@ dwarf_backtrace(struct bt_info *bt, ulon
>
>
> name = sp->name;
> - fprintf(fp, " #0 [%016lx] %s at %016lx \n", UNW_SP(frame), name,
UNW_PC(frame));
> + fprintf(fp, " #%d [%016lx] %s at %016lx \n", level, UNW_SP(frame),
name, UNW_PC(frame));
>
> if (CRASHDEBUG(2))
> fprintf(fp, " < SP: %lx PC: %lx FP: %lx >\n",
UNW_SP(frame),
> @@ -1092,7 +1091,12 @@ dwarf_backtrace(struct bt_info *bt, ulon
>
> while ((UNW_SP(frame) < stacktop)
> && !unwind(frame) && UNW_PC(frame))
{
> - n++;
> + /* To prevent rip pushed on IRQ stack being reported both
> + * both on the IRQ and process stacks
> + */
> + if ((bt->flags & BT_IRQSTACK) && (UNW_SP(frame) >=
stacktop - 16))
> + break;
> + level++;
> sp = value_search(UNW_PC(frame), &offset);
> if (!sp) {
> if (CRASHDEBUG(1))
> @@ -1101,9 +1105,24 @@ dwarf_backtrace(struct bt_info *bt, ulon
> UNW_PC(frame));
> break;
> }
> +
> + /*
> + * If offset is zero, it means we have crossed over to the next
> + * function. Recalculate by adjusting the text address
> + */
> + if (!offset) {
> + sp = value_search(UNW_PC(frame) - 1, &offset);
> + if (!sp) {
> + if (CRASHDEBUG(1))
> + fprintf(fp,
> + "unwind: cannot find symbol for PC:
%lx\n",
> + UNW_PC(frame)-1);
> + goto bailout;
> + }
> + }
> name = sp->name;
> - fprintf(fp, "%s#%d [%016lx] %s at %016lx \n", n < 10 ?
" " : "",
> - n, UNW_SP(frame), name, UNW_PC(frame));
> + fprintf(fp, "%s#%d [%016lx] %s at %016lx \n", level <
10 ? " " : "",
> + level, UNW_SP(frame), name, UNW_PC(frame));
>
> if (CRASHDEBUG(2))
> fprintf(fp, " < SP: %lx PC: %lx FP: %lx
>\n", UNW_SP(frame),
> @@ -1112,7 +1131,49 @@ dwarf_backtrace(struct bt_info *bt, ulon
>
> bailout:
> FREEBUF(frame);
> - return TRUE;
> + return ++level;
> +}
> +
> +int dwarf_print_stack_entry(struct bt_info *bt, int level)
> +{
> + int n = 0;
> + unsigned long offset;
> + struct syment *sp;
> + char *name;
> + struct unwind_frame_info *frame;
> +
> + frame = (struct unwind_frame_info *)GETBUF(sizeof(struct
unwind_frame_info));
> + UNW_SP(frame) = bt->stkptr;
> + UNW_PC(frame) = bt->instptr;
> +
> + sp = value_search(UNW_PC(frame), &offset);
> + if (!sp) {
> + if (CRASHDEBUG(1))
> + fprintf(fp, "unwind: cannot find symbol for PC:
%lx\n",
> + UNW_PC(frame));
> + goto bailout;
> + }
> +
> + /*
> + * If offset is zero, it means we have crossed over to the next
> + * function. Recalculate by adjusting the text address
> + */
> + if (!offset) {
> + sp = value_search(UNW_PC(frame) - 1, &offset);
> + if (!sp) {
> + if (CRASHDEBUG(1))
> + fprintf(fp,
> + "unwind: cannot find symbol for PC:
%lx\n",
> + UNW_PC(frame)-1);
> + goto bailout;
> + }
> + }
> + name = sp->name;
> + fprintf(fp, " #%d [%016lx] %s at %016lx \n", level, UNW_SP(frame),
name, UNW_PC(frame));
> +
> +bailout:
> + FREEBUF(frame);
> + return level;
> }
>
> void
> diff -puN defs.h~cfi_backtrace_minor_fixes defs.h
> --- crash-4.0-3.9/defs.h~cfi_backtrace_minor_fixes 2006-11-15
18:51:28.030164808 +0530
> +++ crash-4.0-3.9-rachita/defs.h 2006-11-15 18:51:46.272391568 +0530
> @@ -645,6 +645,7 @@ struct bt_info {
> ulonglong flags;
> ulong instptr;
> ulong stkptr;
> + ulong bptr;
> ulong stackbase;
> ulong stacktop;
> char *stackbuf;
> @@ -3626,7 +3627,7 @@ struct machine_specific {
> * unwind_x86_32_64.c
> */
> void init_unwind_table(void);
> -int dwarf_backtrace(struct bt_info *, ulong);
> +int dwarf_backtrace(struct bt_info *, int, ulong);
> void dwarf_debug(struct bt_info *);
>
> #endif
> _
6925
days inactive
6954
days old
devel@lists.crash-utility.osci.io
6 comments
2 participants
participants (2)
-
Dave Anderson -
Rachita Kothiyal