4:08 p.m.
- Tentatively scheduled as base version for RHEL4-U5.
- The current 2.6.18 x86_64 kernel has changed the IRQ-stack-to-
process-stack linkage, where until now the link value was a pointer
to the exception frame on the process stack, but has been changed
to point to a location on the process stack above the exception
frame. Because of that, after displaying the trace data from the
IRQ stack, "bt" would then display an invalid exception frame,
which was reported as a "possibly bogus exception frame".
(anderson(a)redhat.com)
- Also in x86_64 kernels, fix for the "bt" command. When the backtrace
started on the NMI exception stack, it was displaying the correct
exception frame data, but was erroneously reporting that it was a
"possibly bogus exception frame". (anderson(a)redhat.com)
- And again in x86_64 kernels, fix for the "bt" command. When making
the transition from the IRQ stack back to the process stack, when
the IRQ stack entry was made via the relatively new "call_softirq"
entry point. In that case, there is no exception frame on the
process stack, because it's essentially just a cross-stack call
from do_softirq(). However, a bogus exception frame was being
displayed, along with a "possibly bogus exception frame" message;
and if the RIP value in the truly bogus exception frame happened
to fall in the user virtual address range, the remainder of the
process stack trace was not displayed at all. (anderson(a)redhat.com)
- Fix for 2.6.18-era ia64 DISCONTIGMEM kernels, which would fail
during initialization with the error message: "crash: invalid
(optional) structure member offsets: pglist_data_node_next or
pglist_data_pgdat_next". (anderson(a)redhat.com)
- Adapted Olivier Daudel's nifty enhancement to the "struct" command,
which allows the single "struct.member" argument to optionally be
expressed in a "struct.member[,member,member] format, in order to
display multiple members of a given structure. This also applies to
the "union" and "*" commands, as all three functions have now been
combined into one behind the scenes. Fixed the display for applying
a minus count, and given that it opened up a the door to a number of
entry errors, I also added additional error-catching/handling to avoid
the display of incorrect structure data.
(olivier.daudel(a)u-paris10.fr, anderson(a)redhat.com)
- Fixed three sources of potential segmentation violations when using
the "bt" command when the experimental dwarf CFI unwind backtrace
facility was turned on. (anderson(a)redhat.com)
- Added a new machdep_init(POST_VM) call, which is currently only being
used by the x86_64 architecture; it calls init_unwind_table(), which
has to be done after vm_init() in order to access the unwind tables
of kernel modules. (anderson(a)redhat.com)
Download from: http://people.redhat.com/anderson
7:15 a.m.
On Thu, Nov 02, 2006 at 05:08:35PM -0500, Dave Anderson wrote:
- Tentatively scheduled as base version for RHEL4-U5.
- The current 2.6.18 x86_64 kernel has changed the IRQ-stack-to-
process-stack linkage, where until now the link value was a pointer
to the exception frame on the process stack, but has been changed
to point to a location on the process stack above the exception
frame. Because of that, after displaying the trace data from the
IRQ stack, "bt" would then display an invalid exception frame,
which was reported as a "possibly bogus exception frame".
(anderson(a)redhat.com)
Hi Dave
With 4.0-3.8 and older versions of crash, I used to see this message
"possibly bogus exception frame" on starting crash. That seems to have
gone now with crash-4.0-3.9. However, I am still getting this message
when I do a bt on the latest crash(kdump generated vmcore).
4.0-3.8 and older versions:
WARNING: possibly bogus exception frame
WARNING: possibly bogus exception frame
KERNEL: /home/rachita/linux-2.6.18-panic-intcontext/vmlinux
DUMPFILE: /kdump/rachita/unwind/vmcore-panic-intcontext
CPUS: 2
DATE: Fri Oct 27 20:43:37 2006
UPTIME: 00:05:12
LOAD AVERAGE: 1.15, 0.81, 0.36
TASKS: 90
NODENAME: llm22
RELEASE: 2.6.18panic-intcontext
VERSION: #9 SMP Fri Oct 27 20:35:38 IST 2006
MACHINE: x86_64 (3600 Mhz)
MEMORY: 8.9 GB
PANIC: ""
PID: 0
COMMAND: "swapper"
TASK: ffffffff805564c0 (1 of 2) [THREAD_INFO: ffffffff806f6000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
crash> bt
PID: 0 TASK: ffffffff805564c0 CPU: 0 COMMAND: "swapper"
#0 [ffffffff8064bce8] crash_kexec at ffffffff80152225
#1 [ffffffff8064bd30] machine_kexec at ffffffff8011a739
#2 [ffffffff8064bd70] crash_kexec at ffffffff80152241
#3 [ffffffff8064bdf8] crash_kexec at ffffffff80152225
#4 [ffffffff8064be20] bust_spinlocks at ffffffff8011fd6d
#5 [ffffffff8064be30] panic at ffffffff80131420
#6 [ffffffff8064bef8] hrtimer_run_queues at ffffffff80145f6e
#7 [ffffffff8064bf20] handle_IRQ_event at ffffffff80154432
#8 [ffffffff8064bf50] __do_IRQ at ffffffff8015451f
#9 [ffffffff8064bf58] __do_softirq at ffffffff80136ba3
#10 [ffffffff8064bf90] do_IRQ at ffffffff8010bda1
--- <IRQ stack> ---
#11 [ffffffff806f7f20] ret_from_intr at ffffffff80109b95
[exception RIP: cpu_idle+149]
RIP: ffffffff8010890f RSP: 000000000008e000 RFLAGS: ffffffff8070379c
RAX: ffffffffffffffff RBX: 0000000000000000 RCX: ffffffff80108968
RDX: 0000000000000010 RSI: 0000000000000246 RDI: ffffffff806f7fa0
RBP: ffffffff806f6000 R8: ffffffff80557db8 R9: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: ffffffff803951dc R15: 000000000008e000
ORIG_RAX: 0000000000000018 CS: 20800 SS: 0000
bt: WARNING: possibly bogus exception frame
#12 [ffffffff806f7fd0] x86_64_start_kernel at ffffffff80703296
crash>
On crash-4.0-3.9
crash> bt
PID: 0 TASK: ffffffff805564c0 CPU: 0 COMMAND: "swapper"
#0 [ffffffff8064bce8] crash_kexec at ffffffff80152225
#1 [ffffffff8064bd30] machine_kexec at ffffffff8011a739
#2 [ffffffff8064bd70] crash_kexec at ffffffff80152241
#3 [ffffffff8064bdf8] crash_kexec at ffffffff80152225
#4 [ffffffff8064be20] bust_spinlocks at ffffffff8011fd6d
#5 [ffffffff8064be30] panic at ffffffff80131420
#6 [ffffffff8064bef8] hrtimer_run_queues at ffffffff80145f6e
#7 [ffffffff8064bf20] handle_IRQ_event at ffffffff80154432
#8 [ffffffff8064bf50] __do_IRQ at ffffffff8015451f
#9 [ffffffff8064bf58] __do_softirq at ffffffff80136ba3
#10 [ffffffff8064bf90] do_IRQ at ffffffff8010bda1
--- <IRQ stack> ---
#11 [ffffffff806f7f20] ret_from_intr at ffffffff80109b95
[exception RIP: cpu_idle+149]
RIP: ffffffff8010890f RSP: 000000000008e000 RFLAGS: ffffffff8070379c
RAX: ffffffffffffffff RBX: 0000000000000000 RCX: ffffffff80108968
RDX: 0000000000000010 RSI: 0000000000000246 RDI: ffffffff806f7fa0
RBP: ffffffff806f6000 R8: ffffffff80557db8 R9: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: ffffffff803951dc R15: 000000000008e000
ORIG_RAX: 0000000000000018 CS: 20800 SS: 0000
bt: WARNING: possibly bogus exception frame
#12 [ffffffff806f7fd0] x86_64_start_kernel at ffffffff80703296
On doing a 'help -m' I find that irq_eframe_link is zero..is that ok?
Thanks
Rachita
8:58 a.m.
Rachita Kothiyal wrote:
Hi Dave
With 4.0-3.8 and older versions of crash, I used to see this message
"possibly bogus exception frame" on starting crash. That seems to have
gone now with crash-4.0-3.9. However, I am still getting this message
when I do a bt on the latest crash(kdump generated vmcore).
On crash-4.0-3.9
crash> bt
PID: 0 TASK: ffffffff805564c0 CPU: 0 COMMAND: "swapper"
#0 [ffffffff8064bce8] crash_kexec at ffffffff80152225
#1 [ffffffff8064bd30] machine_kexec at ffffffff8011a739
#2 [ffffffff8064bd70] crash_kexec at ffffffff80152241
#3 [ffffffff8064bdf8] crash_kexec at ffffffff80152225
#4 [ffffffff8064be20] bust_spinlocks at ffffffff8011fd6d
#5 [ffffffff8064be30] panic at ffffffff80131420
#6 [ffffffff8064bef8] hrtimer_run_queues at ffffffff80145f6e
#7 [ffffffff8064bf20] handle_IRQ_event at ffffffff80154432
#8 [ffffffff8064bf50] __do_IRQ at ffffffff8015451f
#9 [ffffffff8064bf58] __do_softirq at ffffffff80136ba3
#10 [ffffffff8064bf90] do_IRQ at ffffffff8010bda1
--- <IRQ stack> ---
#11 [ffffffff806f7f20] ret_from_intr at ffffffff80109b95
[exception RIP: cpu_idle+149]
RIP: ffffffff8010890f RSP: 000000000008e000 RFLAGS: ffffffff8070379c
RAX: ffffffffffffffff RBX: 0000000000000000 RCX: ffffffff80108968
RDX: 0000000000000010 RSI: 0000000000000246 RDI: ffffffff806f7fa0
RBP: ffffffff806f6000 R8: ffffffff80557db8 R9: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: ffffffff803951dc R15: 000000000008e000
ORIG_RAX: 0000000000000018 CS: 20800 SS: 0000
bt: WARNING: possibly bogus exception frame
#12 [ffffffff806f7fd0] x86_64_start_kernel at ffffffff80703296
On doing a 'help -m' I find that irq_eframe_link is zero..is that ok?
Thanks
Rachita
Clearly the exception frame is bogus (RSP and RFLAGS), so
if your kernel's ".macro interrupt func" pushes rpb instead
of rdi prior to calling the interrupt handler, then the
irq_eframe_link shouldn't be zero.
Do a "dis common_interrupt" -- in a RHEL5 kernel it looks like
this:
crash> dis common_interrupt
0xffffffff8005b968 <common_interrupt>: cld
0xffffffff8005b969 <common_interrupt+1>: sub $0x48,%rsp
0xffffffff8005b96d <common_interrupt+5>: mov %rdi,0x40(%rsp)
0xffffffff8005b972 <common_interrupt+10>: mov %rsi,0x38(%rsp)
0xffffffff8005b977 <common_interrupt+15>: mov %rdx,0x30(%rsp)
0xffffffff8005b97c <common_interrupt+20>: mov %rcx,0x28(%rsp)
0xffffffff8005b981 <common_interrupt+25>: mov %rax,0x20(%rsp)
0xffffffff8005b986 <common_interrupt+30>: mov %r8,0x18(%rsp)
0xffffffff8005b98b <common_interrupt+35>: mov %r9,0x10(%rsp)
0xffffffff8005b990 <common_interrupt+40>: mov %r10,0x8(%rsp)
0xffffffff8005b995 <common_interrupt+45>: mov %r11,(%rsp)
0xffffffff8005b999 <common_interrupt+49>: lea
0xffffffffffffffd0(%rsp),%rdi
0xffffffff8005b99e <common_interrupt+54>: push %rbp
0xffffffff8005b99f <common_interrupt+55>: mov %rsp,%rbp
0xffffffff8005b9a2 <common_interrupt+58>: testl $0x3,0x88(%rdi)
0xffffffff8005b9ac <common_interrupt+68>: je 0xffffffff8005b9b1
<common_interrupt+73>
0xffffffff8005b9ae <common_interrupt+70>: invlpg %ax
0xffffffff8005b9b1 <common_interrupt+73>: incl %gs:0x28
0xffffffff8005b9b9 <common_interrupt+81>: cmove %gs:0x30,%rsp
0xffffffff8005b9c3 <common_interrupt+91>: push %rbp
0xffffffff8005b9c4 <common_interrupt+92>: callq 0xffffffff8006a57b
<do_IRQ>
crash>
If "crash --machdep irq_eframe_link=40 ..." works, then
something in x86_64_irq_eframe_link_init() needs to be
looked at.
Dave
9:46 a.m.
On Wed, Nov 08, 2006 at 09:58:53AM -0500, Dave Anderson wrote:
Rachita Kothiyal wrote:
>
>
> Hi Dave
>
> With 4.0-3.8 and older versions of crash, I used to see this message
> "possibly bogus exception frame" on starting crash. That seems to have
> gone now with crash-4.0-3.9. However, I am still getting this message
> when I do a bt on the latest crash(kdump generated vmcore).
>
>
> On crash-4.0-3.9
>
> crash> bt
> PID: 0 TASK: ffffffff805564c0 CPU: 0 COMMAND: "swapper"
> #0 [ffffffff8064bce8] crash_kexec at ffffffff80152225
> #1 [ffffffff8064bd30] machine_kexec at ffffffff8011a739
> #2 [ffffffff8064bd70] crash_kexec at ffffffff80152241
> #3 [ffffffff8064bdf8] crash_kexec at ffffffff80152225
> #4 [ffffffff8064be20] bust_spinlocks at ffffffff8011fd6d
> #5 [ffffffff8064be30] panic at ffffffff80131420
> #6 [ffffffff8064bef8] hrtimer_run_queues at ffffffff80145f6e
> #7 [ffffffff8064bf20] handle_IRQ_event at ffffffff80154432
> #8 [ffffffff8064bf50] __do_IRQ at ffffffff8015451f
> #9 [ffffffff8064bf58] __do_softirq at ffffffff80136ba3
> #10 [ffffffff8064bf90] do_IRQ at ffffffff8010bda1
> --- <IRQ stack> ---
> #11 [ffffffff806f7f20] ret_from_intr at ffffffff80109b95
> [exception RIP: cpu_idle+149]
> RIP: ffffffff8010890f RSP: 000000000008e000 RFLAGS: ffffffff8070379c
> RAX: ffffffffffffffff RBX: 0000000000000000 RCX: ffffffff80108968
> RDX: 0000000000000010 RSI: 0000000000000246 RDI: ffffffff806f7fa0
> RBP: ffffffff806f6000 R8: ffffffff80557db8 R9: 0000000000000001
> R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
> R13: 0000000000000000 R14: ffffffff803951dc R15: 000000000008e000
> ORIG_RAX: 0000000000000018 CS: 20800 SS: 0000
> bt: WARNING: possibly bogus exception frame
> #12 [ffffffff806f7fd0] x86_64_start_kernel at ffffffff80703296
>
> On doing a 'help -m' I find that irq_eframe_link is zero..is that ok?
>
> Thanks
> Rachita
Clearly the exception frame is bogus (RSP and RFLAGS), so
if your kernel's ".macro interrupt func" pushes rpb instead
of rdi prior to calling the interrupt handler, then the
irq_eframe_link shouldn't be zero.
Do a "dis common_interrupt" -- in a RHEL5 kernel it looks like
this:
crash> dis common_interrupt
0xffffffff8005b968 <common_interrupt>: cld
0xffffffff8005b969 <common_interrupt+1>: sub $0x48,%rsp
0xffffffff8005b96d <common_interrupt+5>: mov %rdi,0x40(%rsp)
0xffffffff8005b972 <common_interrupt+10>: mov %rsi,0x38(%rsp)
0xffffffff8005b977 <common_interrupt+15>: mov %rdx,0x30(%rsp)
0xffffffff8005b97c <common_interrupt+20>: mov %rcx,0x28(%rsp)
0xffffffff8005b981 <common_interrupt+25>: mov %rax,0x20(%rsp)
0xffffffff8005b986 <common_interrupt+30>: mov %r8,0x18(%rsp)
0xffffffff8005b98b <common_interrupt+35>: mov %r9,0x10(%rsp)
0xffffffff8005b990 <common_interrupt+40>: mov %r10,0x8(%rsp)
0xffffffff8005b995 <common_interrupt+45>: mov %r11,(%rsp)
0xffffffff8005b999 <common_interrupt+49>: lea
0xffffffffffffffd0(%rsp),%rdi
0xffffffff8005b99e <common_interrupt+54>: push %rbp
0xffffffff8005b99f <common_interrupt+55>: mov %rsp,%rbp
0xffffffff8005b9a2 <common_interrupt+58>: testl $0x3,0x88(%rdi)
0xffffffff8005b9ac <common_interrupt+68>: je 0xffffffff8005b9b1
<common_interrupt+73>
0xffffffff8005b9ae <common_interrupt+70>: invlpg %ax
0xffffffff8005b9b1 <common_interrupt+73>: incl %gs:0x28
0xffffffff8005b9b9 <common_interrupt+81>: cmove %gs:0x30,%rsp
0xffffffff8005b9c3 <common_interrupt+91>: push %rbp
0xffffffff8005b9c4 <common_interrupt+92>: callq 0xffffffff8006a57b
<do_IRQ>
crash>
If "crash --machdep irq_eframe_link=40 ..." works, then
something in x86_64_irq_eframe_link_init() needs to be
looked at.
Hi Dave
The dis common_interrupt looks exactly like above and with the
--machdep irq_eframe_link=40 in the commandline I dont see the bogus
frames in the bt.
Thanks
Rachita
Dave
9:51 a.m.
Rachita Kothiyal wrote:
> If "crash --machdep irq_eframe_link=40 ..." works, then
> something in x86_64_irq_eframe_link_init() needs to be
> looked at.
Hi Dave
The dis common_interrupt looks exactly like above and with the
--machdep irq_eframe_link=40 in the commandline I dont see the bogus
frames in the bt.
Thanks
Rachita
Right -- so you'll have to debug x86_64_irq_eframe_link_init().
and figure out why it's failing to make it to the bottom, where
the link gets initialized.
Dave
12:50 p.m.
On Wed, Nov 08, 2006 at 10:51:00AM -0500, Dave Anderson wrote:
Rachita Kothiyal wrote:
>
> > If "crash --machdep irq_eframe_link=40 ..." works, then
> > something in x86_64_irq_eframe_link_init() needs to be
> > looked at.
>
> Hi Dave
>
> The dis common_interrupt looks exactly like above and with the
> --machdep irq_eframe_link=40 in the commandline I dont see the bogus
> frames in the bt.
>
> Thanks
> Rachita
>
Right -- so you'll have to debug x86_64_irq_eframe_link_init().
and figure out why it's failing to make it to the bottom, where
the link gets initialized.
Hi Dave
It turns out that the column width of the window I was running crash on was the problem!
With a width of 80, the instruction at <common_interrupt+49> seems to
be wrapped around. This was causing incomplete instruction getting read into
the buffer on a fgets in x86_64_irq_eframe_link_init(). It wasnt reaching
till the 'push rbp' instruction at all and this left the irq_eframe_link
uninitialised. On increasing the column size it worked fine.
crash> dis common_interrupt
0xffffffff80109b34 <common_interrupt>: cld
0xffffffff80109b35 <common_interrupt+1>: sub $0x48,%rsp
0xffffffff80109b39 <common_interrupt+5>: mov %rdi,0x40(%rsp)
0xffffffff80109b3e <common_interrupt+10>: mov %rsi,0x38(%rsp)
0xffffffff80109b43 <common_interrupt+15>: mov %rdx,0x30(%rsp)
0xffffffff80109b48 <common_interrupt+20>: mov %rcx,0x28(%rsp)
0xffffffff80109b4d <common_interrupt+25>: mov %rax,0x20(%rsp)
0xffffffff80109b52 <common_interrupt+30>: mov %r8,0x18(%rsp)
0xffffffff80109b57 <common_interrupt+35>: mov %r9,0x10(%rsp)
0xffffffff80109b5c <common_interrupt+40>: mov %r10,0x8(%rsp)
0xffffffff80109b61 <common_interrupt+45>: mov %r11,(%rsp)
0xffffffff80109b65 <common_interrupt+49>:
lea 0xffffffffffffffd0(%rsp),%rdi
0xffffffff80109b6a <common_interrupt+54>: push %rbp
0xffffffff80109b6b <common_interrupt+55>: mov %rsp,%rbp
Thanks
Rachita
1 p.m.
Rachita Kothiyal wrote:
On Wed, Nov 08, 2006 at 10:51:00AM -0500, Dave Anderson wrote:
> Rachita Kothiyal wrote:
>
> >
> > > If "crash --machdep irq_eframe_link=40 ..." works, then
> > > something in x86_64_irq_eframe_link_init() needs to be
> > > looked at.
> >
> > Hi Dave
> >
> > The dis common_interrupt looks exactly like above and with the
> > --machdep irq_eframe_link=40 in the commandline I dont see the bogus
> > frames in the bt.
> >
> > Thanks
> > Rachita
> >
>
> Right -- so you'll have to debug x86_64_irq_eframe_link_init().
> and figure out why it's failing to make it to the bottom, where
> the link gets initialized.
Hi Dave
It turns out that the column width of the window I was running crash on was the problem!
With a width of 80, the instruction at <common_interrupt+49> seems to
be wrapped around. This was causing incomplete instruction getting read into
the buffer on a fgets in x86_64_irq_eframe_link_init(). It wasnt reaching
till the 'push rbp' instruction at all and this left the irq_eframe_link
uninitialised. On increasing the column size it worked fine.
crash> dis common_interrupt
0xffffffff80109b34 <common_interrupt>: cld
0xffffffff80109b35 <common_interrupt+1>: sub $0x48,%rsp
0xffffffff80109b39 <common_interrupt+5>: mov %rdi,0x40(%rsp)
0xffffffff80109b3e <common_interrupt+10>: mov %rsi,0x38(%rsp)
0xffffffff80109b43 <common_interrupt+15>: mov %rdx,0x30(%rsp)
0xffffffff80109b48 <common_interrupt+20>: mov %rcx,0x28(%rsp)
0xffffffff80109b4d <common_interrupt+25>: mov %rax,0x20(%rsp)
0xffffffff80109b52 <common_interrupt+30>: mov %r8,0x18(%rsp)
0xffffffff80109b57 <common_interrupt+35>: mov %r9,0x10(%rsp)
0xffffffff80109b5c <common_interrupt+40>: mov %r10,0x8(%rsp)
0xffffffff80109b61 <common_interrupt+45>: mov %r11,(%rsp)
0xffffffff80109b65 <common_interrupt+49>:
lea 0xffffffffffffffd0(%rsp),%rdi
0xffffffff80109b6a <common_interrupt+54>: push %rbp
0xffffffff80109b6b <common_interrupt+55>: mov %rsp,%rbp
Thanks
Rachita
Unbelievable -- nice catch!
I would have thought since the output of the disassembly
was changed to a temporary file instead of stdout, that there
wouldn't be any line-wrap applied by gdb behind the scenes.
And as luck would have it, I did my testing in a window
larger than 80-columns...
Back to the drawing board.
Thanks again for finding this.
Dave
6705
days inactive
6711
days old
devel@lists.crash-utility.osci.io
6 comments
2 participants
participants (2)
-
Dave Anderson -
Rachita Kothiyal