Re: [Crash-utility] Kernel Crash Analysis on Android
by Shankar, AmarX
Hi Dave,
Thanks for your info regarding kexec tool.
I am unable to download kexec from below link.
http://www.kernel.org/pub/linux/kernel/people/horms/kexec-tools/kexec-too...
It says HTTP 404 Page Not Found.
Could you please guide me on this?
Thanks & Regards,
Amar Shankar
> On Wed, Mar 21, 2012 at 06:00:00PM +0000, Shankar, AmarX wrote:
>
> > I want to do kernel crash Analysis on Android Merrifield Target.
> >
> > Could someone please help me how to do it?
>
> Merrifield is pretty much similar than Medfield, e.g it has x86 core. So I
> guess you can follow the instructions how to setup kdump on x86 (see
> Documentation/kdump/kdump.txt) unless you already have that configured.
>
> crash should support this directly presuming you have vmlinux/vmcore files to
> feed it. You can configure crash to support x86 on x86_64 host by running:
>
> % make target=X86
> & make
>
> (or something along those lines).
Right -- just the first make command will suffice, i.e., when running
on an x86_64 host:
$ wget http://people.redhat.com/anderson/crash-6.0.4.tar.gz
$ tar xzf crash-6.0.4.tar.gz
...
$ cd crash-6.0.4
$ make target=X86
...
$ ./crash <path-to>/vmlinux <path-to>/vmcore
Dave
From: Shankar, AmarX
Sent: Wednesday, March 21, 2012 11:30 PM
To: 'crash-utility(a)redhat.com'
Subject: Kernel Crash Analysis on Android
Hi,
I want to do kernel crash Analysis on Android Merrifield Target.
Could someone please help me how to do it?
Thanks & Regards,
Amar Shankar
10 months, 1 week
[PATCH] kmem, snap: iomem/ioport display and vmcore snapshot support
by HATAYAMA Daisuke
Some days ago I was in a situation that I had to convert vmcore in
kvmdump format into ELF since some extension module we have locally
can be used only on relatively old crash utility, around version 4,
but such old crash utility cannot handle kvmdump format.
To do the conversion in handy, I used snap command with some modifications
so that it tries to use iomem information in vmcore instead of host's
/proc/iomem. This patch is its cleaned-up version.
In this development, I naturally got down to also making an interface
for an access to resource objects, and so together with the snap
command's patch, I also extended kmem command for iomem/ioport
support. Actually:
kmem -r displays /proc/iomem
crash> kmem -r
00000000-0000ffff : reserved
00010000-0009dbff : System RAM
0009dc00-0009ffff : reserved
000c0000-000c7fff : Video ROM
...
and kmem -R displays /proc/ioport
crash> kmem -R
0000-001f : dma1
0020-0021 : pic1
0040-0043 : timer0
0050-0053 : timer1
...
Looking into old version of kernel source code back, resource structure
has been unchanged since linux-2.4.0. I borrowed the way of walking on
resouce tree in this patch from the lastest v3.3-rc series, but I
guess the logic is also applicable to old kernels. I expect Dave's
regression testsuite.
Also, there would be another command more sutable for iomem/ioport.
If necessay, I'll repost the patch.
---
HATAYAMA Daisuke (4):
Add vmcore snapshot support
Add kmem -r and -R options
Add dump iomem/ioport functions; a helper for resource objects
Add a helper function for iterating resource objects
defs.h | 9 ++++
extensions/snap.c | 54 ++++++++++++++++++++++-
help.c | 2 +
memory.c | 122 +++++++++++++++++++++++++++++++++++++++++++++++++++--
4 files changed, 180 insertions(+), 7 deletions(-)
--
Thanks.
HATAYAMA Daisuke
10 months, 1 week
Re: [Crash-utility] question about phys_base
by Dave Anderson
----- Original Message -----
> >
> > OK, so then I don't understand what you mean by "may be the same"?
> >
> > You didn't answer my original question, but if I understand you correctly,
> > it would be impossible for the qemu host to create a PT_LOAD segment that
> > describes an x86_64 guest's __START_KERNEL_map region, because the host
> > doesn't know that what kind of kernel the guest is running.
>
> Yes. Even if the guest is linux, it is still impossible to do it. Because
> the guest maybe in the second kernel.
>
> qemu-dump walks all guest's page table and collect virtual address and
> physical address mapping. If the page is not used by guest, the virtual is set
> to 0. I create PT_LOAD according to such mapping. So if the guest is linux,
> there may be a PT_LOAD segment that describes __START_KERNEL_map region.
> But the information stored in PT_LOAD maybe for the second kernel. If crash
> uses it, crash will see the second kernel, not the first kernel.
Just to be clear -- what do you mean by the "second" kernel? Do you
mean that a guest kernel crashed guest, and did a kdump operation,
and that second kdump kernel failed somehow, and now you're trying
to do a "virsh dump" on the kdump kernel?
Dave
10 months, 1 week
question about phys_base
by Wen Congyang
Hi, Dave
I am implementing a new dump command in the qemu. The vmcore's
format is elf(like kdump). And I try to provide phys_base in
the PT_LOAD. But if the os uses the first vcpu do kdump, the
value of phys_base is wrong.
I find a function x86_64_virt_phys_base() in crash's code.
Is it OK to call this function first? If the function
successes, we do not calculate phys_base according to PT_LOAD.
Thanks
Wen Congyang
10 months, 1 week
[PATCH] runq: search current task's runqueue explicitly
by HATAYAMA Daisuke
Currently, runq sub-command doesn't consider CFS runqueue's current
task removed from CFS runqueue. Due to this, the remaining CFS
runqueus that follow the current task's is not displayed. This patch
fixes this by making runq sub-command search current task's runqueue
explicitly.
Note that CFS runqueue exists for each task group, and so does CFS
runqueue's current task, and the above search needs to be done
recursively.
Test
====
On vmcore I made 7 task groups:
root group --- A --- AA --- AAA
+ +- AAB
|
+- AB --- ABA
+- ABB
and then I ran three CPU bound tasks, which is exactly the same as
int main(void) { for (;;) continue; return 0; }
for each task group, including root group; so total 24 tasks. For
readability, I annotated each task name with its belonging group name.
For example, loop.ABA belongs to task group ABA.
Look at CPU0 collumn below. [before] lacks 8 tasks and [after]
successfully shows all tasks on the runqueue, which is identical to
the result of [sched debug] that is expected to ouput correct result.
I'll send this vmcore later.
[before]
crash> runq | cat
CPU 0 RUNQUEUE: ffff88000a215f80
CURRENT: PID: 28263 TASK: ffff880037aaa040 COMMAND: "loop.ABA"
RT PRIO_ARRAY: ffff88000a216098
[no tasks queued]
CFS RB_ROOT: ffff88000a216010
[120] PID: 28262 TASK: ffff880037cc40c0 COMMAND: "loop.ABA"
<cut>
[after]
crash_fix> runq
CPU 0 RUNQUEUE: ffff88000a215f80
CURRENT: PID: 28263 TASK: ffff880037aaa040 COMMAND: "loop.ABA"
RT PRIO_ARRAY: ffff88000a216098
[no tasks queued]
CFS RB_ROOT: ffff88000a216010
[120] PID: 28262 TASK: ffff880037cc40c0 COMMAND: "loop.ABA"
[120] PID: 28271 TASK: ffff8800787a8b40 COMMAND: "loop.ABB"
[120] PID: 28272 TASK: ffff880037afd580 COMMAND: "loop.ABB"
[120] PID: 28245 TASK: ffff8800785e8b00 COMMAND: "loop.AB"
[120] PID: 28246 TASK: ffff880078628ac0 COMMAND: "loop.AB"
[120] PID: 28241 TASK: ffff880078616b40 COMMAND: "loop.AA"
[120] PID: 28239 TASK: ffff8800785774c0 COMMAND: "loop.AA"
[120] PID: 28240 TASK: ffff880078617580 COMMAND: "loop.AA"
[120] PID: 28232 TASK: ffff880079b5d4c0 COMMAND: "loop.A"
<cut>
[sched debug]
crash> runq -d
CPU 0
[120] PID: 28232 TASK: ffff880079b5d4c0 COMMAND: "loop.A"
[120] PID: 28239 TASK: ffff8800785774c0 COMMAND: "loop.AA"
[120] PID: 28240 TASK: ffff880078617580 COMMAND: "loop.AA"
[120] PID: 28241 TASK: ffff880078616b40 COMMAND: "loop.AA"
[120] PID: 28245 TASK: ffff8800785e8b00 COMMAND: "loop.AB"
[120] PID: 28246 TASK: ffff880078628ac0 COMMAND: "loop.AB"
[120] PID: 28262 TASK: ffff880037cc40c0 COMMAND: "loop.ABA"
[120] PID: 28263 TASK: ffff880037aaa040 COMMAND: "loop.ABA"
[120] PID: 28271 TASK: ffff8800787a8b40 COMMAND: "loop.ABB"
[120] PID: 28272 TASK: ffff880037afd580 COMMAND: "loop.ABB"
<cut>
Diff stat
=========
defs.h | 1 +
task.c | 37 +++++++++++++++++--------------------
2 files changed, 18 insertions(+), 20 deletions(-)
Thanks.
HATAYAMA, Daisuke
10 months, 1 week
[RFC] makedumpfile, crash: LZO compression support
by HATAYAMA Daisuke
Hello,
This is a RFC patch set that adds LZO compression support to
makedumpfile and crash utility. LZO is as good as in size but by far
better in speed than ZLIB, leading to reducing down time during
generation of crash dump and refiltering.
How to build:
1. Get LZO library, which is provided as lzo-devel package on recent
linux distributions, and is also available on author's website:
http://www.oberhumer.com/opensource/lzo/.
2. Apply the patch set to makedumpfile v1.4.0 and crash v6.0.0.
3. Build both using make. But for crash, do the following now:
$ make CFLAGS="-llzo2"
How to use:
I've newly used -l option for lzo compression in this patch. So for
example, do as follows:
$ makedumpfile -l vmcore dumpfile
$ crash vmlinux dumpfile
Request of configure-like feature for crash utility:
I would like configure-like feature on crash utility for users to
select wheather to add LZO feature actually or not in build-time,
that is: ./configure --enable-lzo or ./configure --disable-lzo.
The reason is that support staff often downloads and installs the
latest version of crash utility on machines where lzo library is not
provided.
Looking at the source code, it looks to me that crash does some kind
of configuration processing in a local manner, around configure.c,
and I guess it's difficult to use autoconf tools directly.
Or is there another better way?
Performance Comparison:
Sample Data
Ideally, I must have measured the performance for many enough
vmcores generated from machines that was actually running, but now
I don't have enough sample vmcores, I couldn't do so. So this
comparison doesn't answer question on I/O time improvement. This
is TODO for now.
Instead, I choosed worst and best cases regarding compression
ratio and speed only. Specifically, the former is /dev/urandom and
the latter is /dev/zero.
I get the sample data of 10MB, 100MB and 1GB by doing like this:
$ dd bs=4096 count=$((1024*1024*1024/4096)) if=/dev/urandom of=urandom.1GB
How to measure
Then I performed compression for each block, 4096 bytes, and
measured total compression time and output size. See attached
mycompress.c.
Result
See attached file result.txt.
Discussion
For both kinds of data, lzo's compression was considerably quicker
than zlib's. Compression ratio is about 37% for urandom data, and
about 8.5% for zero data. Actual situation of physical memory
would be in between the two cases, and so I guess average
compression time ratio is between 37% and 8.5%.
Although beyond the topic of this patch set, we can estimate worst
compression time on more data size since compression is performed
block size wise and the compression time increases
linearly. Estimated worst time on 2TB memory is about 15 hours for
lzo and about 40 hours for zlib. In this case, compressed data
size is larger than the original, so they are really not used,
compression time is fully meaningless. I think compression must be
done in parallel, and I'll post such patch later.
Diffstat
* makedumpfile
diskdump_mod.h | 3 +-
makedumpfile.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++------
makedumpfile.h | 12 +++++++
3 files changed, 101 insertions(+), 12 deletions(-)
* crash
defs.h | 1 +
diskdump.c | 20 +++++++++++++++++++-
diskdump.h | 3 ++-
3 files changed, 22 insertions(+), 2 deletions(-)
TODO
* evaluation including I/O time using actual vmcores
Thanks.
HATAYAMA, Daisuke
10 months, 1 week
Re: [Crash-utility] [RFI] Support Fujitsu's sadump dump format
by tachibana＠mxm.nes.nec.co.jp
Hi Hatayama-san,
On 2011/06/29 12:12:18 +0900, HATAYAMA Daisuke <d.hatayama(a)jp.fujitsu.com> wrote:
> From: Dave Anderson <anderson(a)redhat.com>
> Subject: Re: [Crash-utility] [RFI] Support Fujitsu's sadump dump format
> Date: Tue, 28 Jun 2011 08:57:42 -0400 (EDT)
>
> >
> >
> > ----- Original Message -----
> >> Fujitsu has stand-alone dump mechanism based on firmware level
> >> functionality, which we call SADUMP, in short.
> >>
> >> We've maintained utility tools internally but now we're thinking that
> >> the best is crash utility and makedumpfile supports the sadump format
> >> for the viewpoint of both portability and maintainability.
> >>
> >> We'll be of course responsible for its maintainance in a continuous
> >> manner. The sadump dump format is very similar to diskdump format and
> >> so kdump (compressed) format, so we estimate patch set would be a
> >> relatively small size.
> >>
> >> Could you tell me whether crash utility and makedumpfile can support
> >> the sadump format? If OK, we'll start to make patchset.
I think it's not bad to support sadump by makedumpfile. However I have
several questions.
- Do you want to use makedumpfile to make an existing file that sadump has
dumped small?
- It isn't possible to support the same form as kdump-compressed format
now, is it?
- When the information that makedumpfile reads from a note of /proc/vmcore
(or a header of kdump-compressed format) is added by an extension of
makedumpfile, do you need to modify sadump?
Thanks
tachibana
> >
> > Sure, yes, the crash utility can always support another dumpfile format.
> >
>
> Thanks. It helps a lot.
>
> > It's unclear to me how similar SADUMP is to diskdump/compressed-kdump.
> > Does your internal version patch diskdump.c, or do you maintain your
> > own "sadump.c"? I ask because if your patchset is at all intrusive,
> > I'd prefer it be kept in its own file, primarily for maintainability,
> > but also because SADUMP is essentially a black-box to anybody outside
> > Fujitsu.
>
> What I meant when I used ``similar'' is both literally and
> logically. The format consists of diskdump header-like header, two
> kinds of bitmaps used for the same purpose as those in diskump format,
> and memory data. They can be handled in common with the existing data
> structure, diskdump_data, non-intrusively, so I hope they are placed
> in diskdump.c.
>
> On the other hand, there's a code to be placed at such specific
> area. sadump is triggered depending on kdump's progress and so
> register values to be contained in vmcore varies according to the
> progress: If crash_notes has been initialized when sadump is
> triggered, sadump packs the register values in crash_notes; if not
> yet, packs registers gathered by firmware. This is sadump specific
> processing, so I think putting it in specific sadump.c file is a
> natural and reasonable choise.
>
> Anyway, I have not made any patch set for this. I'll post a patch set
> when I complete.
>
> Again, thanks a lot for the positive answer.
>
> Thanks.
> HATAYAMA, Daisuke
>
>
> _______________________________________________
> kexec mailing list
> kexec(a)lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec
10 months, 1 week
x86_64: Function parameters from stack frames
by Alexandr Terekhov
Hello,
I'd like to introduce a small patch for the crash tool (I described an idea behind it https://www.redhat.com/archives/crash-utility/2013-March/msg00164.html).
* crash version 6.1.6
* arch x86_64
You can find it here https://github.com/hziSot/crash-stack-parser
So, here are some results from a real system:
=======> CPU #0
bin> bt
PID: 0 TASK: ffffffff81a8d020 CPU: 0 COMMAND: "swapper"
#0 [ffff880045603920] machine_kexec at ffffffff8103284b
#1 [ffff880045603980] crash_kexec at ffffffff810ba972
#2 [ffff880045603a50] oops_end at ffffffff81501860
#3 [ffff880045603a80] no_context at ffffffff81043bfb
#4 [ffff880045603ad0] __bad_area_nosemaphore at ffffffff81043e85
#5 [ffff880045603b20] bad_area_nosemaphore at ffffffff81043f53
#6 [ffff880045603b30] __do_page_fault at ffffffff810446b1
#7 [ffff880045603c50] do_page_fault at ffffffff8150383e
#8 [ffff880045603c80] page_fault at ffffffff81500bf5
#9 [ffff880045603d60] activate_task at ffffffff81053953
#10 [ffff880045603d70] try_to_wake_up at ffffffff810600c0
#11 [ffff880045603de0] default_wake_function at ffffffff810602c2
#12 [ffff880045603df0] __wake_up_common at ffffffff8104e369
#13 [ffff880045603e40] complete at ffffffff81053347
#14 [ffff880045603e70] qla24xx_msix_default at ffffffffa00a8f74 [qla2xxx]
#15 [ffff880045603ed0] handle_IRQ_event at ffffffff810dbb10
#16 [ffff880045603f20] handle_edge_irq at ffffffff810de29e
#17 [ffff880045603f60] handle_irq at ffffffff8100df09
#18 [ffff880045603f80] do_IRQ at ffffffff815060bc
--- <IRQ stack> ---
#19 [ffffffff81a01da8] ret_from_intr at ffffffff8100ba53
[exception RIP: intel_idle+222]
RIP: ffffffff812cdc0e RSP: ffffffff81a01e58 RFLAGS: 00000206
RAX: 0000000000000000 RBX: ffffffff81a01ec8 RCX: 0000000000000000
RDX: 000000000000116c RSI: 0000000000000000 RDI: 0000000000441165
RBP: ffffffff8100ba4e R8: 0000000000000005 R9: 000000000000006d
R10: 000000c26ede6d76 R11: 0000000000000000 R12: ffff8800456115a0
R13: 0000000000000000 R14: 000000c12ce6a080 R15: ffff8800456116a0
ORIG_RAX: ffffffffffffff9d CS: 0010 SS: 0018
#20 [ffffffff81a01ed0] cpuidle_idle_call at ffffffff81407997
#21 [ffffffff81a01ef0] cpu_idle at ffffffff81009e06
bin> bt -H
Backtrace:
# 0: [RSP: 0xffff880045603980, RIP: 0xffffffff8103284b] machine_kexec (struct kimage * arg = 0xffff88082659b400)
# 1: [RSP: 0xffff880045603a50, RIP: 0xffffffff810ba972] crash_kexec (struct pt_regs * arg = 0xffff880045603c88)
# 2: [RSP: 0xffff880045603a80, RIP: 0xffffffff81501860] oops_end (long unsigned int arg = unknown, struct pt_regs * arg = 0xffff880045603c88, int arg = 0x9)
# 3: [RSP: 0xffff880045603ad0, RIP: 0xffffffff81043bfb] no_context (struct pt_regs * arg = 0xffff880045603c88, long unsigned int arg = 0x10, long unsigned int arg = 0x0)
# 4: [RSP: 0xffff880045603b20, RIP: 0xffffffff81043e85] __bad_area_nosemaphore (struct pt_regs * arg = 0xffff880045603c88, long unsigned int arg = 0x10, long unsigned int arg = 0x0, int arg = 0x30001)
# 5: [RSP: 0xffff880045603b30, RIP: 0xffffffff81043f53] bad_area_nosemaphore (struct pt_regs * arg = 0xffff880045603c88, long unsigned int arg = 0x10, long unsigned int arg = 0x0)
# 6: [RSP: 0xffff880045603c50, RIP: 0xffffffff810446b1] __do_page_fault (struct pt_regs * arg = 0xffff880045603c88, long unsigned int arg = 0x4b2, long unsigned int arg = 0x10)
# 7: [RSP: 0xffff880045603c80, RIP: 0xffffffff8150383e] do_page_fault (struct pt_regs * arg = 0xffff880045603c88, long unsigned int arg = 0xffffffffffffffff)
# 8: [RSP: 0xffff880045603d30, RIP: 0xffffffff81500bf5] page_fault ()
# 9: [RSP: 0xffff880045603d60, RIP: 0xffffffff81053916] enqueue_task (struct rq * arg = 0xffff8800456d6680, struct task_struct * arg = 0xffff88082b83eae0, int arg = 0x1)
# 10: [RSP: 0xffff880045603d70, RIP: 0xffffffff81053953] activate_task (struct rq * arg = 0xffff8800456d6680, struct task_struct * arg = 0xffff88082b83eae0, int arg = 0x1)
# 11: [RSP: 0xffff880045603de0, RIP: 0xffffffff810600c0] try_to_wake_up (struct task_struct * arg = 0xffff88082b83eae0, unsigned int arg = 0x3, int arg = 0x0)
# 12: [RSP: 0xffff880045603df0, RIP: 0xffffffff810602c2] default_wake_function (wait_queue_t * arg = 0xffff8800456c3b40, unsigned int arg = 0x3, int arg = 0x0, void * arg = 0xc1416b9c06)
# 13: [RSP: 0xffff880045603e40, RIP: 0xffffffff8104e369] __wake_up_common (wait_queue_head_t * arg = 0xffff880825e6a2a8, unsigned int arg = 0x3, int arg = 0x1, int arg = 0x0, void * arg = 0x0)
# 14: [RSP: 0xffff880045603e70, RIP: 0xffffffff81053347] complete (struct completion * arg = 0xffff880825e6a2a0)
# 15: [RSP: 0xffff880045603ed0, RIP: 0xffffffffa00a8f74] qla24xx_msix_default (int arg = 0x3e, void * arg = 0xffff880826bd55c0)
# 16: [RSP: 0xffff880045603f20, RIP: 0xffffffff810dbb10] handle_IRQ_event (unsigned int arg = 0x3e, struct irqaction * arg = 0xffff880826bd5840)
# 17: [RSP: 0xffff880045603f60, RIP: 0xffffffff810de29e] handle_edge_irq (unsigned int arg = 0x3e, struct irq_desc * arg = 0xffff880824563ec0)
# 18: [RSP: 0xffff880045603f80, RIP: 0xffffffff8100df09] handle_irq (unsigned int arg = 0x3e, struct pt_regs * arg = 0xffffffff81a01da8)
# 19: [RSP: 0xffff880045603fb0, RIP: 0xffffffff815060bc] do_IRQ (struct pt_regs * arg = 0xffffffff81a01da8)
# 20: [RSP: 0xffffffff81a01e50, RIP: 0xffffffff8100ba53] common_interrupt ()
# 21: [RSP: 0xffffffff81a01ed0, RIP: 0xffffffff812cdc0e] intel_idle (struct cpuidle_device * arg = 0xffff88004561dcc0, struct cpuidle_state * arg = 0xffff88004561de50)
# 22: [RSP: 0xffffffff81a01ef0, RIP: 0xffffffff81407997] cpuidle_idle_call ()
Some of the arguments are not accurate, but many of them are correct:
bin> p/x *(( struct pt_regs * ) 0xffffffff81a01da8 )
$54 = {
r15 = 0xffff8800456116a0,
r14 = 0xc12ce6a080,
r13 = 0x0,
r12 = 0xffff8800456115a0,
bp = 0xffffffff8100ba4e,
bx = 0xffffffff81a01ec8,
r11 = 0x0,
r10 = 0xc26ede6d76,
r9 = 0x6d,
r8 = 0x5,
ax = 0x0,
cx = 0x0,
dx = 0x116c,
si = 0x0,
di = 0x441165,
orig_ax = 0xffffffffffffff9d,
ip = 0xffffffff812cdc0e,
cs = 0x10,
flags = 0x206,
sp = 0xffffffff81a01e58,
ss = 0x18
}
bin> p/x *((struct irq_desc * ) 0xffff880824563ec0)
$55 = {
irq = 0x3e,
timer_rand_state = 0x0,
kstat_irqs = 0xffff880825d7f6c0,
irq_2_iommu = 0xffff880825e70880,
handle_irq = 0xffffffff810de1c0,
chip = 0xffffffff81a934e0,
msi_desc = 0xffff880826bd56c0,
handler_data = 0x0,
chip_data = 0xffff880825e708a0,
action = 0xffff880826bd5840,
status = 0x1000100,
depth = 0x0,
wake_depth = 0x0,
irq_count = 0x2cd,
last_unhandled = 0x0,
irqs_unhandled = 0x0,
lock = {
raw_lock = {
slock = 0x5bc05bc
}
},
affinity = 0xffff880826a0f800,
affinity_hint = 0x0,
node = 0x0,
pending_mask = 0xffff88082b765a00,
threads_active = {
counter = 0x0
},
wait_for_threads = {
lock = {
raw_lock = {
slock = 0x0
}
},
task_list = {
next = 0xffff880824563f60,
prev = 0xffff880824563f60
}
},
dir = 0xffff880824563d40,
name = 0xffffffff8177f1ce
}
=======> CPU #6
bin> bt
PID: 0 TASK: ffff88082b83eae0 CPU: 6 COMMAND: "swapper"
#0 [ffff8800456c7e90] crash_nmi_callback at ffffffff81029df6
#1 [ffff8800456c7ea0] notifier_call_chain at ffffffff815038f5
#2 [ffff8800456c7ee0] atomic_notifier_call_chain at ffffffff8150395a
#3 [ffff8800456c7ef0] notify_die at ffffffff810981ee
#4 [ffff8800456c7f20] do_nmi at ffffffff81501573
#5 [ffff8800456c7f50] nmi at ffffffff81500e80
[exception RIP: _spin_lock_irq+37]
RIP: ffffffff81500615 RSP: ffff8800456c3940 RFLAGS: 00000097
RAX: 00000000000008a4 RBX: ffff8800456d6680 RCX: 000000000000c14e
RDX: 00000000000008a3 RSI: 0000000000000001 RDI: ffff8800456d6680
RBP: ffff8800456c3940 R8: 0000000000013679 R9: 00000000fffffffa
R10: 0000000000000002 R11: 0000000000000002 R12: ffff8800456d6680
R13: 0000000000000001 R14: 00000000ffffffff R15: 00000000ffffffff
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
--- <NMI exception stack> ---
#6 [ffff8800456c3940] _spin_lock_irq at ffffffff81500615
#7 [ffff8800456c3948] dequeue_task_idle at ffffffff81053f19
#8 [ffff8800456c3968] dequeue_task at ffffffff8105385e
#9 [ffff8800456c3998] deactivate_task at ffffffff810538a3
#10 [ffff8800456c39a8] thread_return at ffffffff814fded9
#11 [ffff8800456c3a68] schedule_timeout at ffffffff814febf2
#12 [ffff8800456c3b18] wait_for_common at ffffffff814fe8f3
#13 [ffff8800456c3ba8] wait_for_completion_timeout at ffffffff814fe9e3
#14 [ffff8800456c3bb8] qla2x00_mailbox_command at ffffffffa0098d1a [qla2xxx]
#15 [ffff8800456c3c68] qla2x00_issue_iocb_timeout at ffffffffa009cc3b [qla2xxx]
#16 [ffff8800456c3cf8] qla2x00_issue_iocb at ffffffffa009ccd4 [qla2xxx]
#17 [ffff8800456c3d08] qla24xx_abort_command at ffffffffa009f5b8 [qla2xxx]
#18 [ffff8800456c3d68] qla24xx_bsg_timeout at ffffffffa00c0ba6 [qla2xxx]
#19 [ffff8800456c3da8] fc_bsg_job_timeout at ffffffffa00718f7 [scsi_transport_fc]
#20 [ffff8800456c3dd8] blk_rq_timed_out at ffffffff8125dccb
#21 [ffff8800456c3df8] blk_rq_timed_out_timer at ffffffff8125df88
#22 [ffff8800456c3e48] run_timer_softirq at ffffffff8107e927
#23 [ffff8800456c3ed8] __do_softirq at ffffffff81073f51
#24 [ffff8800456c3f48] call_softirq at ffffffff8100c24c
#25 [ffff8800456c3f60] do_softirq at ffffffff8100de85
#26 [ffff8800456c3f80] irq_exit at ffffffff81073d35
#27 [ffff8800456c3f90] smp_apic_timer_interrupt at ffffffff815061b0
#28 [ffff8800456c3fb0] apic_timer_interrupt at ffffffff8100bc13
--- <IRQ stack> ---
#29 [ffff88082b847db8] apic_timer_interrupt at ffffffff8100bc13
[exception RIP: intel_idle+222]
RIP: ffffffff812cdc0e RSP: ffff88082b847e68 RFLAGS: 00000206
RAX: 0000000000000000 RBX: ffff88082b847ed8 RCX: 0000000000000000
RDX: 0000000000000f69 RSI: 0000000000000000 RDI: 00000000003c35fa
RBP: ffffffff8100bc0e R8: 0000000000000005 R9: 000000000000006d
R10: 000000c130c95aec R11: 0000000000000000 R12: ffff8800456d15a0
R13: 0000000000000000 R14: 000000c12d23a980 R15: ffff8800456d16a0
ORIG_RAX: ffffffffffffff10 CS: 0010 SS: 0018
#30 [ffff88082b847ee0] cpuidle_idle_call at ffffffff81407997
#31 [ffff88082b847f00] cpu_idle at ffffffff81009e06
bin> bt -H
# 0: [RSP: 0xffff8800456c7ea0, RIP: 0xffffffff81029df6] crash_nmi_callback (struct notifier_block * arg = 0xffffffff81aa5260, long unsigned int arg = 0xc, void * arg = 0xffff8800456c7ef8)
# 1: [RSP: 0xffff8800456c7ee0, RIP: 0xffffffff815038f5] notifier_call_chain (struct notifier_block ** arg = 0xffffffff81ea87a8, long unsigned int arg = 0xc, void * arg = 0xffff8800456c7ef8, int arg = 0xffffffff, int * arg = 0x0)
# 2: [RSP: 0xffff8800456c7ef0, RIP: 0xffffffff8150395a] atomic_notifier_call_chain (struct atomic_notifier_head * arg = 0xffffffff81ea87a0, long unsigned int arg = 0xc, void * arg = 0xffff8800456c7ef8)
# 3: [RSP: 0xffff8800456c7f20, RIP: 0xffffffff810981ee] notify_die (enum die_val arg = 0xc, const char * arg = 0xffffffff8177d9a2, struct pt_regs * arg = 0xffff8800456c7f58, long int arg = 0x0, int arg = 0x2, int arg = 0x2)
# 4: [RSP: 0xffff8800456c7f50, RIP: 0xffffffff81501573] do_nmi (struct pt_regs * arg = 0xffff8800456c7f58, long int arg = 0xffffffffffffffff)
# 5: [RSP: 0xffff8800456c8000, RIP: 0xffffffff81500e80] nmi ()
# 6: [RSP: 0xffff8800456c3948, RIP: 0xffffffff81500615] _spin_lock_irq (spinlock_t * arg = 0xffff8800456d6680)
# 7: [RSP: 0xffff8800456c3968, RIP: 0xffffffff81053f19] dequeue_task_idle (struct rq * arg = 0xffff8800456d6680, struct task_struct * arg = 0xffff88082b83eae0, int arg = 0x1)
# 8: [RSP: 0xffff8800456c3998, RIP: 0xffffffff8105385e] dequeue_task (struct rq * arg = 0xffff8800456d6680, struct task_struct * arg = 0xffff88082b83eae0, int arg = 0x1)
# 9: [RSP: 0xffff8800456c39a8, RIP: 0xffffffff810538a3] deactivate_task (struct rq * arg = 0xffff8800456d6680, struct task_struct * arg = 0xffff88082b83eae0, int arg = 0x1)
# 10: [RSP: 0xffff8800456c3a68, RIP: 0xffffffff814fded9] thread_return ()
# 11: [RSP: 0xffff8800456c3b18, RIP: 0xffffffff814febf2] schedule_timeout (long int arg = 0x7530)
# 12: [RSP: 0xffff8800456c3ba8, RIP: 0xffffffff814fe8f3] wait_for_common (struct completion * arg = 0xffff880825e6a2a0, long int arg = 0x7530, int arg = 0x2)
# 13: [RSP: 0xffff8800456c3bb8, RIP: 0xffffffff814fe9e3] wait_for_completion_timeout (struct completion * arg = 0xffff880825e6a2a0, long unsigned int arg = 0x7530)
# 14: [RSP: 0xffff8800456c3c68, RIP: 0xffffffffa0098d1a] qla2x00_mailbox_command (scsi_qla_host_t * arg = 0xffff880824562de0, mbx_cmd_t * arg = 0xffff8800456c3c70)
# 15: [RSP: 0xffff8800456c3cf8, RIP: 0xffffffffa009cc3b] qla2x00_issue_iocb_timeout (scsi_qla_host_t * arg = 0xffff880824562de0, void * arg = 0xffff880824561100, dma_addr_t arg = 0x824561100, size_t arg = 0x0, uint32_t arg = 0x1e)
# 16: [RSP: 0xffff8800456c3d08, RIP: 0xffffffffa009ccd4] qla2x00_issue_iocb (scsi_qla_host_t * arg = 0xffff880824562de0, void * arg = 0xffff880824561100, dma_addr_t arg = 0x824561100, size_t arg = 0x0)
# 17: [RSP: 0xffff8800456c3d68, RIP: 0xffffffffa009f5b8] qla24xx_abort_command (srb_t * arg = 0xffff8808145db480)
# 18: [RSP: 0xffff8800456c3da8, RIP: 0xffffffffa00c0ba6] qla24xx_bsg_timeout (struct fc_bsg_job * arg = 0xffff8808268a2740)
# 19: [RSP: 0xffff8800456c3dd8, RIP: 0xffffffffa00718f7] fc_bsg_job_timeout (struct request * arg = 0xffff88082446b360)
# 20: [RSP: 0xffff8800456c3df8, RIP: 0xffffffff8125dccb] blk_rq_timed_out (struct request * arg = 0xffff88082446b360)
# 21: [RSP: 0xffff8800456c3e48, RIP: 0xffffffff8125df88] blk_rq_timed_out_timer (long unsigned int arg = 0xffff88082477cea0)
# 22: [RSP: 0xffff8800456c3ed8, RIP: 0xffffffff8107e927] run_timer_softirq (struct softirq_action * arg = 0xffffffff81a83088)
# 23: [RSP: 0xffff8800456c3f48, RIP: 0xffffffff81073f51] __do_softirq ()
# 24: [RSP: 0xffff8800456c3f60, RIP: 0xffffffff8100c24c] call_softirq ()
# 25: [RSP: 0xffff8800456c3f80, RIP: 0xffffffff8100de85] do_softirq ()
# 26: [RSP: 0xffff8800456c3f90, RIP: 0xffffffff81073d35] irq_exit ()
# 27: [RSP: 0xffff8800456c3fb0, RIP: 0xffffffff815061b0] smp_apic_timer_interrupt (struct pt_regs * arg = 0xffff88082b847db8)
# 28: [RSP: 0xffff88082b847e60, RIP: 0xffffffff8100bc13] apic_timer_interrupt ()
# 29: [RSP: 0xffff88082b847ee0, RIP: 0xffffffff812cdc0e] intel_idle (struct cpuidle_device * arg = 0xffff8800456ddcc0, struct cpuidle_state * arg = 0xffff8800456dde50)
# 30: [RSP: 0xffff88082b847f00, RIP: 0xffffffff81407997] cpuidle_idle_call ()
Please feel free to submit any found issues via email or github.
11 years
PPC64: vtop of module and user-space virtual addresses fails on 3.10 kernels
by Dave Anderson
This is a request for some help from the IBM'ers on the list...
Starting somewhere in the 3.10 timeframe (I believe), the
virtual-to-physical translation of kernel modules no longer
works for ppc64. User-space vtop also seems to be broken.
For example, here's an example using a 3.10.0-0.rc4.59.el7.ppc64
kernel, which shows the "WARNING: cannot access vmalloc'd module
memory" message during initialization, and I also show the results
of a "vtop" on the first and last module addresses in the kernel's
"modules" list:
# crash
crash 7.0.1
Copyright (C) 2002-2013 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64-unknown-linux-gnu"...
WARNING: cannot access vmalloc'd module memory
KERNEL: /usr/lib/debug/lib/modules/3.10.0-0.rc4.59.el7.ppc64/vmlinux
DUMPFILE: /dev/crash
CPUS: 28
DATE: Mon Jun 24 13:35:45 2013
UPTIME: 01:54:23
LOAD AVERAGE: 0.00, 0.17, 0.23
TASKS: 318
NODENAME: ibm-p730-04-lp4.rhts.eng.bos.redhat.com
RELEASE: 3.10.0-0.rc4.59.el7.ppc64
VERSION: #1 SMP Mon Jun 3 14:42:22 EDT 2013
MACHINE: ppc64 (3550 Mhz)
MEMORY: 8 GB
PID: 30000
COMMAND: "crash"
TASK: c000000192860000 [THREAD_INFO: c0000001c6c00000]
CPU: 5
STATE: TASK_RUNNING (ACTIVE)
crash> p modules
modules = $1 = {
next = 0xd00000000d019080,
prev = 0xd000000001016f10
}
crash> vtop 0xd00000000d019080
VIRTUAL PHYSICAL
d00000000d019080 (not mapped)
PAGE DIRECTORY: c0000000011d0000
L4: c0000000011d0000 => c0000001fba80000
PMD: c0000001fba80000 => c0000001fba70000
PMD: c0000001fba80000 => fba76808
PTE: fba76808 => 0
crash> vtop 0xd000000001016f10
VIRTUAL PHYSICAL
d000000001016f10 (not mapped)
PAGE DIRECTORY: c0000000011d0000
L4: c0000000011d0000 => c0000001fba80000
PMD: c0000001fba80000 => c0000001fba70000
PMD: c0000001fba80000 => fba70808
PTE: fba70808 => 0
crash>
I'm not at all familiar with ppc64 page table walk-throughs,
but what little debugging I've tried has yielded nothing other
than module-address-translation that end up with PTE's that
contain zero like the above.
Also, the vtop of user-space addresses also seems to be completely
disfunctional. Taking the live system above, if I take the user-space
address of the page-table page buffer used by the crash utility itself,
and try to do a vtop on it, it yields this obviously bogus result:
crash> help -m | grep ptbl:
ptbl: 100133d6f10
crash> vtop 100133d6f10
VIRTUAL PHYSICAL
100133d6f10 (not mapped)
PAGE DIRECTORY: c0000001e0940000
L4: c0000001e0940008 => 0
VMA START END FLAGS FILE
c00000001b7d0000 10012fe0000 100167b0000 100073
crash>
It seems to have been something recently introduced, as here's
a 3.9.0-0.rc8.54.el7.ppc64 kernel, which works just fine:
# crash
crash 7.0.1
Copyright (C) 2002-2013 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64-unknown-linux-gnu"...
KERNEL: /usr/lib/debug/lib/modules/3.9.0-0.rc8.54.el7.ppc64/vmlinux
DUMPFILE: /dev/crash
CPUS: 28
DATE: Mon Jun 24 13:50:38 2013
UPTIME: 00:12:20
LOAD AVERAGE: 0.22, 0.17, 0.16
TASKS: 316
NODENAME: ibm-p730-04-lp4.rhts.eng.bos.redhat.com
RELEASE: 3.9.0-0.rc8.54.el7.ppc64
VERSION: #1 SMP Mon Apr 22 18:30:40 EDT 2013
MACHINE: ppc64 (3550 Mhz)
MEMORY: 8 GB
PID: 7035
COMMAND: "crash"
TASK: c0000001ddf00000 [THREAD_INFO: c0000001bb780000]
CPU: 4
STATE: TASK_RUNNING (ACTIVE)
crash> p modules
modules = $1 = {
next = 0xd00000000c729100,
prev = 0xd000000000fe6ff8
}
crash> vtop 0xd00000000c729100
VIRTUAL PHYSICAL
d00000000c729100 1cfb29100
PAGE DIRECTORY: c000000001190000
L4: c000000001190000 => c0000001fba80000
PMD: c0000001fba80000 => c0000001fba60000
PMD: c0000001fba80000 => fba66390
PTE: fba66390 => 73ec88000395
PAGE: 1cfb20000
PTE PHYSICAL FLAGS
73ec88000395 1cfb20000 (PRESENT|RW|COHERENT|DIRTY|ACCESSED)
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
c0000000042c26f0 1cfb20000 0 0 1 73c0400000000
crash> vtop 0xd000000000fe6ff8
VIRTUAL PHYSICAL
d000000000fe6ff8 1eba16ff8
PAGE DIRECTORY: c000000001190000
L4: c000000001190000 => c0000001fba80000
PMD: c0000001fba80000 => c0000001fba60000
PMD: c0000001fba80000 => fba607f0
PTE: fba607f0 => 7ae848000395
PAGE: 1eba10000
PTE PHYSICAL FLAGS
7ae848000395 1eba10000 (PRESENT|RW|COHERENT|DIRTY|ACCESSED)
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
c000000004482338 1eba10000 0 0 1 7ac0400000000
crash>
And the user-space vtop example works as expected:
crash> help -m | grep ptbl:
ptbl: 1001d208c80
crash> vtop 1001d208c80
VIRTUAL PHYSICAL
1001d208c80 1a1b18c80
PAGE DIRECTORY: c0000001ec8a9c00
L4: c0000001ec8a9c08 => c0000001f0a50000
PMD: c0000001f0a50008 => c0000001c7f20000
PMD: c0000001f0a50008 => c7f26900
PTE: c7f26900 => 686c48000393
PAGE: 1a1b10000
PTE PHYSICAL FLAGS
686c48000393 1a1b10000 (PRESENT|USER|COHERENT|DIRTY|ACCESSED)
VMA START END FLAGS FILE
c0000001eb523af0 1001ce20000 100206f0000 100073
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
c000000003fe26b8 1a1b10000 c0000001e93f4c41 1001d20 1 6840400080068
crash>
Any ideas?
Thanks,
Dave
11 years, 2 months
