xen guest linux : paravirtulized Vsles11-1:/usr/src/linux # grep CONFIG_DEBUG_INFO .config -n 4574:CONFIG_DEBUG_INFO=y Vsles11-1:/usr/src/linux # ls ..tmp_kallsyms1.o.cmd .tmp_System.map .version Makefile crypto kernel scripts ..tmp_kallsyms2.o.cmd .tmp_kallsyms1.S .vmlinux.cmd Module.symvers drivers lib security ..tmp_vmlinux1.cmd .tmp_kallsyms1.o .vmlinux.o.cmd README firmware mm sound ..tmp_vmlinux2.cmd .tmp_kallsyms2.S COPYING README.SUSE fs modules.builtin tools .config .tmp_kallsyms2.o CREDITS REPORTING-BUGS include modules.order usr .config.old .tmp_versions Documentation System.map init net virt .mailmap .tmp_vmlinux1 Kbuild arch ipc perfmon vmlinux .missing-syscalls.d .tmp_vmlinux2 MAINTAINERS block kdb samples vmlinux.o Vsles11-1:/usr/src/linux # SLES11:/etc/xen # xm dump-core 4 /opt/sles11_1.mem Dumping core of domain: 4 ... SLES11:/etc/xen # cd /opt/ SLES11:/opt # ls .Trash-0 centos_guest huawei kdump linux-3.0 novell sles11_1.mem sys_use user xen_study 10-17-12:52 gnome job kernel lost+found openoffice.org sles11_ana tar_rpm vmlinux Life hardware kde3 kvm network openoffice.org3 sles11_guest tencent winxp_guest SLES11:/opt # crash vmlinux sles11_1.mem crash 4.0-7.6 Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc. Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb 6.1 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... crash: cannot resolve "end_pfn" use lastest crash version: SLES11:/opt/tar_rpm/crash-5.1.9 # ./crash /opt/vmlinux /opt/sles11_1.mem crash 5.1.9 Copyright (C) 2002-2011 Red Hat, Inc. Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.0 Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... crash: cannot find mfn 0 (0x0) in page index crash: cannot read/find pte page --

SLES11:/opt/tar_rpm/crash-5.1.9 # ./crash -d8 /opt/kdump/vmlinux /opt/kdump/sles11_1.mem > /tmp/crash_output3 output see attachement. Thank your response.

> Date: Wed, 26 Oct 2011 08:57:01 -0400 > From: anderson(a)redhat.com > To: crash-utility(a)redhat.com > Subject: Re: [Crash-utility] [help]crash can't anaylyse xen guest > paravirtulized linux vmcore > > > > ----- Original Message ----- > > > > > > xen guest linux : paravirtulized > > > > Vsles11-1:/usr/src/linux # grep CONFIG_DEBUG_INFO .config -n > > 4574:CONFIG_DEBUG_INFO=y > > Vsles11-1:/usr/src/linux # ls > > ..tmp_kallsyms1.o.cmd .tmp_System.map .version Makefile crypto > > kernel > > scripts > > ..tmp_kallsyms2.o.cmd .tmp_kallsyms1.S .vmlinux.cmd > > Module.symvers > > drivers lib security > > ..tmp_vmlinux1.cmd .tmp_kallsyms1.o .vmlinux.o.cmd README > > firmware mm > > sound > > ..tmp_vmlinux2.cmd .tmp_kallsyms2.S COPYING README.SUSE fs > > modules.builtin tools > > .config .tmp_kallsyms2.o CREDITS REPORTING-BUGS include > > modules.order > > usr > > .config.old .tmp_versions Documentation System.map init net virt > > .mailmap .tmp_vmlinux1 Kbuild arch ipc perfmon vmlinux > > .missing-syscalls.d .tmp_vmlinux2 MAINTAINERS block kdb samples > > vmlinux.o > > Vsles11-1:/usr/src/linux # > > > > SLES11:/etc/xen # xm dump-core 4 /opt/sles11_1.mem > > Dumping core of domain: 4 ... > > > > SLES11:/etc/xen # cd /opt/ > > SLES11:/opt # ls > > .Trash-0 centos_guest huawei kdump linux-3.0 novell sles11_1.mem > > sys_use user xen_study > > 10-17-12:52 gnome job kernel lost+found openoffice.org sles11_ana > > tar_rpm vmlinux > > Life hardware kde3 kvm network openoffice.org3 sles11_guest > > tencent > > winxp_guest > > > > > > SLES11:/opt # crash vmlinux sles11_1.mem > > > > crash 4.0-7.6 > > Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, > > Inc. > > Copyright (C) 2004, 2005, 2006 IBM Corporation > > Copyright (C) 1999-2006 Hewlett-Packard Co > > Copyright (C) 2005, 2006 Fujitsu Limited > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > > Copyright (C) 2005 NEC Corporation > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > This program is free software, covered by the GNU General Public > > License, > > and you are welcome to change it and/or distribute copies of it > > under > > certain conditions. Enter "help copying" to see the conditions. > > This program has absolutely no warranty. Enter "help warranty" > > for > > details. > > > > GNU gdb 6.1 > > Copyright 2004 Free Software Foundation, Inc. > > GDB is free software, covered by the GNU General Public License, > > and > > you are > > welcome to change it and/or distribute copies of it under certain > > conditions. > > Type "show copying" to see the conditions. > > There is absolutely no warranty for GDB. Type "show warranty" for > > details. > > This GDB was configured as "x86_64-unknown-linux-gnu"... > > > > > > crash: cannot resolve "end_pfn" > > > > use lastest crash version: > > SLES11:/opt/tar_rpm/crash-5.1.9 # ./crash /opt/vmlinux > > /opt/sles11_1.mem > > > > crash 5.1.9 > > Copyright (C) 2002-2011 Red Hat, Inc. > > Copyright (C) 2004, 2005, 2006 IBM Corporation > > Copyright (C) 1999-2006 Hewlett-Packard Co > > Copyright (C) 2005, 2006 Fujitsu Limited > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > > Copyright (C) 2005 NEC Corporation > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > > This program is free software, covered by the GNU General Public > > License, > > and you are welcome to change it and/or distribute copies of it > > under > > certain conditions. Enter "help copying" to see the conditions. > > This program has absolutely no warranty. Enter "help warranty" > > for > > details. > > > > GNU gdb (GDB) 7.0 > > Copyright (C) 2009 Free Software Foundation, Inc. > > License GPLv3+: GNU GPL version 3 or later > > <http://gnu.org/licenses/gpl.html> > > This is free software: you are free to change and redistribute > > it. > > There is NO WARRANTY, to the extent permitted by law. Type "show > > copying" > > and "show warranty" for details. > > This GDB was configured as "x86_64-unknown-linux-gnu"... > > > > crash: cannot find mfn 0 (0x0) in page index > > > > crash: cannot read/find pte page > > -- > > Using the latest version of crash, what is the output of: > > $ crash -d8 vmlinux sles11_1.mem > > It's going to quite verbose -- you probably want to save it as a > file > like this: > > $ crash -d8 vmlinux sles11_1.mem > /tmp/output > > and then attach it to the mail message. > > Dave > > -- > Crash-utility mailing list > Crash-utility(a)redhat.com > https://www.redhat.com/mailman/listinfo/crash-utility -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

Date: Wed, 26 Oct 2011 11:54:42 -0400 From: anderson(a)redhat.com To: crash-utility(a)redhat.com Subject: Re: [Crash-utility] [help]crash can't anaylyse xen guest paravirtulized linux vmcore ----- Original Message ----- > > > ----- Original Message ----- > > > > > > SLES11:/opt/tar_rpm/crash-5.1.9 # ./crash -d8 /opt/kdump/vmlinux > > /opt/kdump/sles11_1.mem > /tmp/crash_output3 > > > > > > output see attachement. > > > > > > > > Thank your response. > > It's been a long time since I've looked at the particulars of a Xen dumpfile > given that Red Hat has discontinued Xen for KVM, so I'm a bit rusty. In any > case, upon the first memory read, while trying to walk through the page table > data to resolve a kernel virtual address, it comes to a dead end. I have no > idea why, nor do I have any further suggestions. > > Can the SLES-qualified members of the list confirm that "xm dump" still creates > dumpfiles that crash can handle? > > Dave FWIW, it's actually making the second kernel virtual address read: $ grep pgd: crash_output3 [ffffffff806f7360] pgd: 9af04067 mfn: 9af04 pgd_index: 1fe [ffffffff80674690] pgd: 9af04067 mfn: 9af04 pgd_index: 1fe $ where, depending upon the kernel version, ffffffff806f7360 should be "end_pfn" or "max_pfn". It ended up resolving that first virtual address to a page table and page, but the data there doesn't make sense: $ grep "end pfn:" crash_output3 end pfn: 0 $ So even though it appeared to work OK, it's apparently bogus. Then I believe that the second virtual address of ffffffff80674690 should be "p2m_top", and while trying to walk the page tables for that virtual address, it failed. If you do this: $ nm -Bn vmlinux | grep -e ffffffff806f7360 -e ffffffff80674690 you can at least verify that. But unfortunately it doesn't really help understand what's happening. Dave -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

Red Hat never supported running 64-bit Xen guests on a 32-bit dom0.

Date: Thu, 27 Oct 2011 09:45:00 -0400 From: anderson(a)redhat.com To: crash-utility(a)redhat.com Subject: Re: [Crash-utility] [help]crash can't anaylyse xen guest paravirtulized linux vmcore ----- Original Message ----- > > On the off-chance that you've hit this particular one (there's no > clear indication) - taking a dump of a 64-bit guest from a 32-bit > host will not work. libxc bug that gave me no end of grief. Interesting... I noticed that he used a 32-bit HVM guest an example of a successful xm dump. Red Hat never supported running 64-bit Xen guests on a 32-bit dom0. Dav!

-- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

Date: Thu, 27 Oct 2011 10:41:00 -0400 From: anderson(a)redhat.com To: crash-utility(a)redhat.com Subject: Re: [Crash-utility] [help]crash can't anaylyse xen guest paravirtulized linux vmcore ----- Original Message ----- > > > It is wrong in last mail , correct as follow: > > > xen and dom0 is x86_64 > > > when para x86_64 domU, problem happens > when hvm 32bit domU , crash can work correctly. Just for sanity's sake, log into the x86_64 domU, run crash on the live system, and enter: crash> sym phys_to_machine_mapping crash> sym max_pfn Then log out, and compare the output above with the vmlinux file that you're using with the dumpfile: $ nm -Bn | grep -e max_pfn -e phys_to_machine_mapping Dave -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

Date: Wed, 26 Oct 2011 11:54:42 -0400 From: anderson(a)redhat.com To: crash-utility(a)redhat.com Subject: Re: [Crash-utility] [help]crash can't anaylyse xen guest paravirtulized linux vmcore ----- Original Message ----- > > > ----- Original Message ----- > > > > > > SLES11:/opt/tar_rpm/crash-5.1.9 # ./crash -d8 /opt/kdump/vmlinux > > /opt/kdump/sles11_1.mem > /tmp/crash_output3 > > > > > > output see attachement. > > > > > > > > Thank your response. > > It's been a long time since I've looked at the particulars of a Xen dumpfile > given that Red Hat has discontinued Xen for KVM, so I'm a bit rusty. In any > case, upon the first memory read, while trying to walk through the page table > data to resolve a kernel virtual address, it comes to a dead end. I have no > idea why, nor do I have any further suggestions. > > Can the SLES-qualified members of the list confirm that "xm dump" still creates > dumpfiles that crash can handle? > > Dave FWIW, it's actually making the second kernel virtual address read: $ grep pgd: crash_output3 [ffffffff806f7360] pgd: 9af04067 mfn: 9af04 pgd_index: 1fe [ffffffff80674690] pgd: 9af04067 mfn: 9af04 pgd_index: 1fe $ where, depending upon the kernel version, ffffffff806f7360 should be "end_pfn" or "max_pfn". It ended up resolving that first virtual address to a page table and page, but the data there doesn't make sense: $ grep "end pfn:" crash_output3 end pfn: 0 $ So even though it appeared to work OK, it's apparently bogus. Then I believe that the second virtual address of ffffffff80674690 should be "p2m_top", and while trying to walk the page tables for that virtual address, it failed. If you do this: $ nm -Bn vmlinux | grep -e ffffffff806f7360 -e ffffffff80674690 you can at least verify that. But unfortunately it doesn't really help understand what's happening. Dave -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility

Date: Thu, 27 Oct 2011 09:34:56 -0400 From: anderson(a)redhat.com To: crash-utility(a)redhat.com Subject: Re: [Crash-utility] [help]crash can't anaylyse xen guest paravirtulized linux vmcore ----- Original Message ----- > > > my crash can analyse xen hvm guest linux memory dump core : > > > SLES11:/mnt/sles10.2_crash # crash32 -s vmlinux-debug sles10.img > WARNING: kernel version inconsistency between vmlinux and dumpfile > > crash32> bt > PID: 0 TASK: c02d8280 CPU: 0 COMMAND: "swapper" > #0 [c0329f9c] schedule at c0288d6d > #1 [c0329ffc] is386 at c0100194 OK, so you're saying that it's only an issue with SLES paravirtualized kernels? I compared the output of "crash -d8" output using a 2.6.18-152.el5xen kernel (fully virtualized with the old Xen patch), and the sequence is pretty much identical, other than the .xen_p2m and .xen_pages Elf64_Shdr's are in a different order in the dumpfile. But that shouldn't make a difference, because crash doesn't expect them in any particular order. So again, the only thing I can see is that the page that contains the "max_pfn" symbol is full of zeroes -- which is obviously wrong -- and then while walking the page tables to find the page containing the "phys_to_machine_mapping" symbol, it finds the page table that should contain a PTE that points to the page containing it, but the PTE entry is zero. Although other PTEs in the page table page do contain legitimate PTEs. The only thing I can think of is that the vmlinux file that you're using doesn't match the dumpfile, i.e., the "max_pfn" and "phys_to_machine_mapping" symbols from the vmlinux file are not the same as the kernel represented by the dumpfile? Dave -- Crash-utility mailing list Crash-utility(a)redhat.com https://www.redhat.com/mailman/listinfo/crash-utility