Roger Cruz wrote: Enter "help" -- it shows the commands when running against a xen-syms hypervisor: crash> help * dumpinfo list sched vcpu alias eval log search vcpus ascii exit p set whatis bt extend pcpus struct wr dis foreach pte sym q domain gdb rd sys doms help repeat union crash version: 4.0-4.7 gdb version: 6.1 For help on any command above, enter "help <command>". For help on input options, enter "help input". For help on output options, enter "help output". crash> Then for any particular command, enter "help <command>", so for backtrace options, enter "help bt". I do note that some of the common commands between running crash on a vmlinux and a xen-syms show the help data for the command as if it were running against a vmlinux, and as such, some advertised options may not work on a xen-syms session. A limited set of gdb commands are runnable, although the embedded gdb module has no clue of the vmcore file; it's invoked internally as "gdb xen-syms". I'm presuming that the crash occurred within the hypervisor as opposed to the (vmlinux) kernel? If it happened within kernel code, substitute the xen-syms argument with the vmlinux of the dom0 kernel, and you will be presented with a different set of commands. Dave

Roger Cruz wrote: I see what you're talking about, but I'm afraid I can't shed any light on the matter. I have never personally debugged any hypervisor crashes. Support for xen-syms analysis in the crash utility was not done done by me, but implemented wholly by Itsuro Oda (oda(a)valinux.co.jp) and Fumihiko Kakuma (kakuma(a)valinux.co.jp). I am also curious as to how the xen trace shown in the serial port output is related to the backtrace shown by "bt". Perhaps they can help explain it? (Oda-san is on this list) No. The closest to that is "bt -f", which dumps the stack contents of each frame, but it will require disassembling the affected functions to figure out where in that stack frame the local variable exists.

Hi, 'bt' is done by crash command itself, not using a gdb function. 'bt' for x86 xen-hypervisor ends when it meets "handle_exception" now. I don't implement to handle an exception frame yet. In this case 'bt' shows only the backtrace after BUG() occur. To get the backtrace before BUG(), examin the register values of the exception (it can be seen at just after "ff1bbd7c") and you can do 'bt pcpu-id -I eip -S esp' to show the backtrace. In this case it is not necessary to examin the register values fortunately because the console output shows them at the exeption occur (ie. at the BUG() called). Try 'bt 2 -I ff10e4c5 -S ff1bbdb0'. (it is not different from the backtrace shown by the console output very much) FYI, it can be shown by 'log' command. Thanks. Itsuro Oda On Thu, 11 Oct 2007 17:27:46 -0400 "Roger Cruz" <rcruz(a)marathontechnologies.com&gt; wrote: > I see. I was kind of hoping that since 'crash' was using GDB some of > that functionality would have carried up. A while ago, I worked in > getting a coredump function into GDB for VxWorks and it was not too hard > to do. I would think that a similar functionality already existed for > Xen coredumps. I can't promise that I will do this since I'm under the > gun to get something working here, but I could try to tackle something > similar based on your work and what Itsuro and Fumihiko have already > done. > > R. > > -----Original Message----- > From: crash-utility-bounces(a)redhat.com > [mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson > Sent: Thursday, October 11, 2007 5:24 PM > To: Discussion list for crash utility usage, maintenance and > development; oda(a)valinux.co.jp > Subject: Re: [Crash-utility] Debugging Xen Hypervisor with 'crash' > question... > > Roger Cruz wrote: > > Dave, thanks for your reply. > > > > I am trying to find out why a change I made to the hypervisor is > > crashing. Using print statements is not a workable solution so I'm > > trying the coredump approach. I have crash working on a coredump but > > the backtrace information does not match what Xen puts out on its > serial > > port (I know Xen's it's not 100% correct, but at least it does show > some > > of the actual routines executed in the calltrace). > > I see what you're talking about, but I'm afraid I can't shed > any light on the matter. I have never personally debugged any > hypervisor crashes. > > Support for xen-syms analysis in the crash utility was not done > done by me, but implemented wholly by Itsuro Oda (oda(a)valinux.co.jp) > and Fumihiko Kakuma (kakuma(a)valinux.co.jp). I am also curious as > to how the xen trace shown in the serial port output is related > to the backtrace shown by "bt". > > Perhaps they can help explain it? (Oda-san is on this list) > > > > > Additionally, I want to select the context that allows me to walk up > and > > down the stack and look at the source code and its local variables. I > > can do this with a typical GDB coredump file so I was looking for an > > equivalent functionality with 'crash'. Does crash support that? > > No. The closest to that is "bt -f", which dumps the stack contents > of each frame, but it will require disassembling the affected > functions to figure out where in that stack frame the local variable > exists. > > > > > I have tried to run gdb against the /proc/vmcore but it complains > about > > an unrecognized format, so it looks like my only choice right now is > > 'crash', unless someone has figured out how to make GDB work with Xen > > cores. > > > > Thank you again! > > Roger > > > > > > This GDB was configured as "i686-pc-linux-gnu"... > > > > KERNEL: xen-syms > > DUMPFILE: /dom0/proc/vmcore > > CPUS: 4 > > DOMAINS: 6 > > UPTIME: 01:51:01 > > MACHINE: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2327 Mhz) > > MEMORY: 4 GB > > PCPU-ID: 2 > > PCPU: ff1bbfb4 > > VCPU-ID: 0 > > VCPU: ffbdf080 (VCPU_RUNNING) > > DOMAIN-ID: 1 > > DOMAIN: ff1a8080 (DOMAIN_RUNNING) > > STATE: CRASH > > > > crash> bt > > PCPU: 2 VCPU: ffbdf080 > > #0 [ff1bbcd4] kexec_crash_save_cpu at ff10a8ad > > #1 [ff1bbcdc] kexec_crash at ff10a9f0 > > #2 [ff1bbcec] panic at ff11b9c2 > > #3 [ff1bbd1c] do_invalid_op at ff13314b > > #4 [ff1bbd7c] handle_exception at ff1734bc > > > > > > > > (XEN) Xen BUG at page_alloc.c:902 > > (XEN) ----[ Xen-3.1.0 x86_32p debug=n Not tainted ]---- > > (XEN) Xen stack trace from esp=ff1bbdb0: > > (XEN) ff17dd3c e8000081 00000000 00000003 2dad0067 00000000 > f839ea18 > > f839ea18 > > (XEN) ffbd0900 00000000 ff19b0a0 ff108df1 f839ea18 00000000 > 001269c1 > > 00000022 > > (XEN) ffbdfd30 ff14382e ffbdf080 ffbdfd30 ff23e900 ff23e900 > 0000000f > > ff23e900 > > (XEN) ff1a8080 ff115457 ffbe6080 ffc01000 ffbdf080 ff1a8080 > ff1a9708 > > ff23e900 > > (XEN) ff1af104 000f060e 00000000 00000000 ffbd4080 001269c1 > 00009708 > > ff1a8080 > > (XEN) 00000614 ffbc9000 00220001 ff159d31 00000001 00000001 > 00000000 > > 0000060e > > (XEN) 00000000 ff1a96d4 ffbdfdfc ffbdfdf8 ff1bf080 ff14b8d0 > ff1a9708 > > f9efbefd > > (XEN) 0000060e 07b86000 00000000 269c1000 00000001 00000014 > fe867ce8 > > ff1bbfb4 > > (XEN) 00000014 ff1bbfb4 ff1ad1f0 ff13f088 00000001 f95dfba0 > 00000001 > > 47868c00 > > (XEN) 00000000 0000060e f2e792f6 0000060e 00fee003 ffffffff > ff107c90 > > 00000012 > > (XEN) 00000003 00000000 ffbdf080 ff15d2d4 ff1bbfb4 fffe00b0 > ff1bbfb4 > > 0000060e > > (XEN) 00000004 00000000 ffbdf080 f9c8f8bf ff144f0d ffbdf080 > 00000000 > > ffbdf080 > > (XEN) 0000060e ff1a96d4 ffbdf080 ffffffff ff14b7df ffbdf080 > ffffffff > > ffbdfd74 > > (XEN) ff14cbb7 ffbdfd74 ffbdf080 ffbdf080 ff144ffa ffbdf080 > ffbdf080 > > f95df858 > > (XEN) ff1575ef ffbdf080 ff1a9708 ff23e900 ff1153ae f9efbefd > 0000060e > > ff1af100 > > (XEN) 00000002 ff1bbfb4 ff1bbfb4 00000001 47868c00 00000000 > f95dfb54 > > ff15e717 > > (XEN) ff1bbfb4 00000001 f95dfba0 00000001 47868c00 00000000 > f95dfb54 > > 00000014 > > (XEN) 00f0000b ff249285 00000008 00000282 f95dfb44 00000010 > 00000000 > > 00000000 > > (XEN) 00000000 00000000 00000002 ffbdf080 > > (XEN) Xen call trace: > > (XEN) [<ff10e4c5>] free_domheap_pages+0x105/0x2d0 > > (XEN) [<ff19b0a0>] dmi_decode+0x20/0xd0 > > (XEN) [<ff108df1>] do_grant_table_op+0x1161/0x1910 > > (XEN) [<ff14382e>] hvm_io_assist+0x13e/0x1400 > > (XEN) [<ff1a8080>] zap_low_mappings+0x80/0xa0 > > (XEN) [<ff115457>] add_entry+0x57/0x140 > > (XEN) [<ff1a8080>] zap_low_mappings+0x80/0xa0 > > (XEN) [<ff1a8080>] zap_low_mappings+0x80/0xa0 > > (XEN) [<ff159d31>] vmx_io_instruction+0x421/0xc30 > > (XEN) [<ff14b8d0>] pt_thaw_time+0x70/0x80 > > (XEN) [<ff13f088>] hvm_do_hypercall+0xb8/0x1e0 > > (XEN) [<ff107c90>] do_grant_table_op+0x0/0x1910 > > (XEN) [<ff15d2d4>] vmx_vmexit_handler+0x334/0x1760 > > (XEN) [<ff144f0d>] is_isa_irq_masked+0x2d/0x90 > > (XEN) [<ff14b7df>] pt_update_irq+0x9f/0x120 > > (XEN) [<ff14cbb7>] vlapic_has_interrupt+0x37/0x60 > > (XEN) [<ff144ffa>] cpu_has_pending_irq+0x3a/0x60 > > (XEN) [<ff1575ef>] vmx_intr_assist+0x3f/0x2e0 > > (XEN) [<ff1153ae>] timer_softirq_action+0xce/0xf0 > > (XEN) [<ff15e717>] vmx_asm_vmexit_handler+0x17/0x20 > > (XEN) > > (XEN) > > (XEN) **************************************** > > (XEN) Panic on CPU 2: > > (XEN) Xen BUG at page_alloc.c:902 > > (XEN) **************************************** > > (XEN) > > > > -----Original Message----- > > From: crash-utility-bounces(a)redhat.com > > [mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson > > Sent: Thursday, October 11, 2007 2:51 PM > > To: Discussion list for crash utility usage, maintenance and > development > > Subject: Re: [Crash-utility] Debugging Xen Hypervisor with 'crash' > > question... > > > > Roger Cruz wrote: > > > >>Sorry if this is an obvious question but I'm new to the 'crash' > > > > utility. > > > >> I read Anderson's white paper on crash and didn't find any references > > > > > >>to how to use 'crash' to debug the hypervisor. I have crash running > > > > and > > > >>accessing Domain 0's kernel tasks and other variables, so I am > >>comfortable thinking that I have the right setup. I start crash with: > >> > >> > >> > >>#crash xen-syms /dom0/proc/vmcore > >> > >> > >> > >>And get the following output > >> > >> > >> > >>#crash xen-syms /dom0/proc/vmcore > >> > >> > >> > >>crash 4.0-4.7 > >> > >>Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc. > >> > >>Copyright (C) 2004, 2005, 2006 IBM Corporation > >> > >>Copyright (C) 1999-2006 Hewlett-Packard Co > >> > >>Copyright (C) 2005, 2006 Fujitsu Limited > >> > >>Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > >> > >>Copyright (C) 2005 NEC Corporation > >> > >>Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > >> > >>Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > >> > >>This program is free software, covered by the GNU General Public > > > > License, > > > >>and you are welcome to change it and/or distribute copies of it under > >> > >>certain conditions. Enter "help copying" to see the conditions. > >> > >>This program has absolutely no warranty. Enter "help warranty" for > > > > details. > > > >> > >> > >>GNU gdb 6.1 > >> > >>Copyright 2004 Free Software Foundation, Inc. > >> > >>GDB is free software, covered by the GNU General Public License, and > > > > you are > > > >>welcome to change it and/or distribute copies of it under certain > > > > conditions. > > > >>Type "show copying" to see the conditions. > >> > >>There is absolutely no warranty for GDB. Type "show warranty" for > > > > details. > > > >>This GDB was configured as "i686-pc-linux-gnu"... > >> > >> > >> > >> KERNEL: xen-syms > >> > >> DUMPFILE: /dom0/proc/vmcore > >> > >> CPUS: 4 > >> > >> DOMAINS: 4 > >> > >> UPTIME: 00:01:30 > >> > >> MACHINE: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2327 Mhz) > >> > >> MEMORY: 4 GB > >> > >> PCPU-ID: 2 > >> > >> PCPU: ff1bbfb4 > >> > >> VCPU-ID: 0 > >> > >> VCPU: ffbe6080 (VCPU_RUNNING) > >> > >>DOMAIN-ID: 0 > >> > >> DOMAIN: ff238080 (DOMAIN_RUNNING) > >> > >> STATE: CRASH > >> > >> > >> > >>I would like to know what commands there are to examine the memory > >>management system or any other internal data structures. Also, how do > > > > I > > > >>look at a stack trace in the hypervisor for a crash. I tried the 'gdb > > > > > >>where' command and it said no stack. > >> > > > > > > Enter "help" -- it shows the commands when running against > > a xen-syms hypervisor: > > > > crash> help > > > > * dumpinfo list sched vcpu > > alias eval log search vcpus > > ascii exit p set whatis > > bt extend pcpus struct wr > > dis foreach pte sym q > > domain gdb rd sys > > doms help repeat union > > > > crash version: 4.0-4.7 gdb version: 6.1 > > For help on any command above, enter "help <command>". > > For help on input options, enter "help input". > > For help on output options, enter "help output". > > > > crash> > > > > Then for any particular command, enter "help <command>", > > so for backtrace options, enter "help bt". I do note > > that some of the common commands between running crash > > on a vmlinux and a xen-syms show the help data for the > > command as if it were running against a vmlinux, and > > as such, some advertised options may not work on a > > xen-syms session. > > > > A limited set of gdb commands are runnable, although the > > embedded gdb module has no clue of the vmcore file; it's > > invoked internally as "gdb xen-syms". > > > > I'm presuming that the crash occurred within the hypervisor > > as opposed to the (vmlinux) kernel? If it happened within > > kernel code, substitute the xen-syms argument with the > > vmlinux of the dom0 kernel, and you will be presented > > with a different set of commands. > > > > Dave > > > > > >> > >> > >>Thanks in advance. > >> > >> > >> > >> > >> > >>Roger Cruz > >> > >>Principal SW Engineer > >> > >>Marathon Technologies Corp. > >> > >>978-489-1153 > >> > > > > > > -- > > Crash-utility mailing list > > Crash-utility(a)redhat.com > > https://www.redhat.com/mailman/listinfo/crash-utility > > > > -- > > Crash-utility mailing list > > Crash-utility(a)redhat.com > > https://www.redhat.com/mailman/listinfo/crash-utility > > > -- > Crash-utility mailing list > Crash-utility(a)redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility -- Itsuro ODA <oda(a)valinux.co.jp&gt;