Re: [Crash-utility] Debugging Xen Hypervisor with 'crash' question...
Roger Cruz wrote:
Sorry if this is an obvious question but I’m new to the ‘crash’
utility.
I read Anderson’s white paper on crash and didn’t find any references
to how to use ‘crash’ to debug the hypervisor. I have crash running and
accessing Domain 0’s kernel tasks and other variables, so I am
comfortable thinking that I have the right setup. I start crash with:
#crash xen-syms /dom0/proc/vmcore
And get the following output
#crash xen-syms /dom0/proc/vmcore
crash 4.0-4.7
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: xen-syms
DUMPFILE: /dom0/proc/vmcore
CPUS: 4
DOMAINS: 4
UPTIME: 00:01:30
MACHINE: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2327 Mhz)
MEMORY: 4 GB
PCPU-ID: 2
PCPU: ff1bbfb4
VCPU-ID: 0
VCPU: ffbe6080 (VCPU_RUNNING)
DOMAIN-ID: 0
DOMAIN: ff238080 (DOMAIN_RUNNING)
STATE: CRASH
I would like to know what commands there are to examine the memory
management system or any other internal data structures. Also, how do I
look at a stack trace in the hypervisor for a crash. I tried the ‘gdb
where’ command and it said no stack.
Enter "help" -- it shows the commands when running against
a xen-syms hypervisor:
crash> help
* dumpinfo list sched vcpu
alias eval log search vcpus
ascii exit p set whatis
bt extend pcpus struct wr
dis foreach pte sym q
domain gdb rd sys
doms help repeat union
crash version: 4.0-4.7 gdb version: 6.1
For help on any command above, enter "help <command>".
For help on input options, enter "help input".
For help on output options, enter "help output".
crash>
Then for any particular command, enter "help <command>",
so for backtrace options, enter "help bt". I do note
that some of the common commands between running crash
on a vmlinux and a xen-syms show the help data for the
command as if it were running against a vmlinux, and
as such, some advertised options may not work on a
xen-syms session.
A limited set of gdb commands are runnable, although the
embedded gdb module has no clue of the vmcore file; it's
invoked internally as "gdb xen-syms".
I'm presuming that the crash occurred within the hypervisor
as opposed to the (vmlinux) kernel? If it happened within
kernel code, substitute the xen-syms argument with the
vmlinux of the dom0 kernel, and you will be presented
with a different set of commands.
Dave
Thanks in advance.
Roger Cruz
Principal SW Engineer
Marathon Technologies Corp.
978-489-1153