Re: [Crash-utility] how to generate xen hypervisor core though kdump or other way?
by Daniel Kiper
Hi Dave,
[...]
> That has always been the case -- at least up until the most
> recent version of Xen (3.1-era) that Red Hat supported -- where
> a kdump vmcore that is taken on the dom0 host can be analyzed
> either from the viewpoint of the dom0 vmlinux kernel or from the
> viewpoint of the xen-syms hypervisor. And for that matter, given
> that it's a dump of all physical memory, you can also analyze any
> of the guest vmlinux kernels if you know what the value of the
> pfn_to_mfn_list_list pfn is:
>
> $ crash -h
> ...
>
> --p2m_mfn pfn
> When a Xen Hypervisor or its dom0 kernel crashes, the dumpfile
> is typically analyzed with either the Xen hypervisor or the dom0
> kernel. It is also possible to analyze any of the guest domU
> kernels if the pfn_to_mfn_list_list pfn value of the guest kernel
> is passed on the command line along with its NAMELIST and the
> dumpfile.
To be honest I have never not attempted to use this option.
I was focusing on Xen dump analysis only. It looks that
it is time to do that now.
> So anyway, Hu shows that the vmlinux/vmcore pair works OK, but
> the xen-syms/vmcore pair is not working with his more recent
> version of Xen (4.1.3). I would have thought that your recent
> patch set would have addressed his Xen version?
It did. I suppose that there is an issue with finding
dom0 image in crash dump. Probably I will investigate
it further in 2-3 weeks.
> On the other hand, I cannot confirm whether SLES does something
> differently such that their Xen hypervisor is patched to such a
> degree that it doesn't work with crash-6.1.0?
As above.
Daniel
11 years, 8 months
Re: [Crash-utility] using crash for ARM
by Dave Anderson
Paawan,
Right -- it looks to be a bug, presuming that ARM is using 1MB pages
for the unity-mapped region:
crash> vtop c0000000 | grep PAGE:
PAGE: 11000 (1MB)
crash> vtop c0100000 | grep PAGE:
PAGE: 11000 (1MB)
crash> vtop c0200000 | grep PAGE:
PAGE: 211000 (1MB)
crash> vtop c0300000 | grep PAGE:
PAGE: 211000 (1MB)
crash> vtop c0400000 | grep PAGE:
PAGE: 411000 (1MB)
crash> vtop c0500000 | grep PAGE:
PAGE: 411000 (1MB)
crash>
And confusing the issue even more, when the virtual memory is read,
the data at c0000000 and c0100000 is different (as expected), but the
virtual-to-physical translation "paddr:" value indicates different
physical addresses that what are shown above:
crash> set debug 4
debug: 4
crash> rd c0000000
<addr: c0000000 count: 1 flag: 488 (KVADDR)>
<readmem: c0000000, KVADDR, "32-bit KVADDR", 4, (FOE), ff8bea60>
<read_kdump: addr: c0000000 paddr: 0 cnt: 4>
c0000000: ea000012 ....
crash> rd c0100000
<addr: c0100000 count: 1 flag: 488 (KVADDR)>
<readmem: c0100000, KVADDR, "32-bit KVADDR", 4, (FOE), ff8bea60>
<read_kdump: addr: c0100000 paddr: 100000 cnt: 4>
c0100000: 3694c21d ...6
crash>
In other words, above it shows that c0000000 is at physical address 0,
and that c0100000 is at 100000, i.e., not 11000 as shown by the
verbose vtop display.
I'll leave it up to the crash utility's ARM maintainers to come up
with the proper answer and fix for the verbose vtop display. I've
cc'd them with this email. Also please use the crash-utility(a)redhat.com
mailing list for these kinds of discussions.
Thanks,
Dave
----- Original Message -----
>
> Hi Dave,
>
> The section translation inside 'arm_vtop' function and getting to the
> physical address looks beyond my understanding.
>
> if (verbose)
> fprintf(fp, "PAGE DIRECTORY: %lx\n", (ulong)pgd);
>
> /*
> * pgd_offset(pgd, vaddr)
> */
> page_dir = pgd + PGD_OFFSET(vaddr) * 2;
> FILL_PGD(PAGEBASE(pgd), KVADDR, PGDIR_SIZE());
> pgd_pte = ULONG(machdep->pgd + PGDIR_OFFSET(page_dir));
>
> if (verbose)
> fprintf(fp, " PGD: %s => %lx\n",
> mkstring(buf, VADDR_PRLEN, RJUST | LONG_HEX,
> MKSTR((ulong)page_dir)), pgd_pte);
>
> if (!pgd_pte)
> return FALSE;
>
> /*
> * pmd_offset(pgd, vaddr)
> *
> * Here PMD is folded into a PGD.
> */
> pmd_pte = pgd_pte;
> page_middle = page_dir;
>
> if (verbose)
> fprintf(fp, " PMD: %s => %lx\n",
> mkstring(buf, VADDR_PRLEN, RJUST | LONG_HEX,
> MKSTR((ulong)page_middle)), pmd_pte);
>
> if ((pmd_pte & PMD_TYPE_MASK) == PMD_TYPE_SECT) {
> if (verbose) {
> fprintf(fp, " PAGE: %s (1MB)\n\n",
> mkstring(buf, VADDR_PRLEN, RJUST | LONG_HEX,
> MKSTR(PAGEBASE(pmd_pte))));
> }
> *paddr = PAGEBASE(pmd_pte) + (vaddr & ~_SECTION_PAGE_MASK);
> return TRUE;
> }
>
>
>
>
>
> if you look at
> page_dir = pgd + PGD_OFFSET(vaddr) * 2;
> FILL_PGD(PAGEBASE(pgd), KVADDR, PGDIR_SIZE());
> pgd_pte = ULONG(machdep->pgd + PGDIR_OFFSET(page_dir));
>
>
> the final first level discriptor is not coming properly for
> kernel virtual address with [20st counting from 0....n] bit set.
>
>
>
>
> if you look at the following output:
>
>
> crash> vtop 0xc010005c
> VIRTUAL PHYSICAL
> c010005c 8210005c
>
> PAGE DIRECTORY: c0004000
> PGD: c0007000 => 8200040e
> PMD: c0007000 => 8200040e
> PAGE: 82000000 (1MB)
>
>
> PAGE PHYSICAL MAPPING INDEX CNT FLAGS
> c103c000 82000000 c1d16f28 3 54 8027c
> crash> vtop 0xc000005c
> VIRTUAL PHYSICAL
> c000005c 8200005c
>
> PAGE DIRECTORY: c0004000
> PGD: c0007000 => 8200040e
> PMD: c0007000 => 8200040e
> PAGE: 82000000 (1MB)
>
>
> PAGE PHYSICAL MAPPING INDEX CNT FLAGS
> c103c000 82000000 c1d16f28 3 54 8027c
>
>
>
> virtual to physical mapping is fine, because it doesnt even go
> through arm_vtop() function.
> as it is straight one to one mapping, you use VTOP macro.
> so basically section code snippet which I sent you never come into
> picture.
> if you have a look at the both kernel virtual addresses
>
>
> 0xc010005c and
> 0xc000005c
> both are 1MB apart. but still following contents are same.
>
> PAGE DIRECTORY: c0004000
> PGD: c0007000 => 8200040e
> PMD: c0007000 => 8200040e
>
> PAGE: 82000000 (1MB)
>
>
> I shall clarify more, If you require more info.
> I hope I am not missing anything here.
>
>
>
> PS:
> If I want to have correct physical address corresponding to
> 0xc010005c
>
>
> then I have to do workaround for section level physical addresses as
> follows.
> page_dir = pgd + PGD_OFFSET(vaddr) * 2;
> if (bit(vaddr,20)) //if bit is set then move to the next pgd */
> page_dir = page_dir + 1;
> FILL_PGD(PAGEBASE(pgd), KVADDR, PGDIR_SIZE());
> pgd_pte = ULONG(machdep->pgd + PGDIR_OFFSET(page_dir));
>
>
>
> Regards,
> Oza.
>
>
>
>
>
>
> From: Dave Anderson <anderson(a)redhat.com>
> To: paawan oza <paawan1982(a)yahoo.com>
> Cc: "Discussion list for crash utility usage, maintenance and
> development" <crash-utility(a)redhat.com>
> Sent: Tuesday, 14 August 2012 7:46 PM
> Subject: Re: [Crash-utility] using crash for ARM
>
>
>
> ----- Original Message -----
> >
> >
> > strings ./vmlinux | grep "Linux version"
> > Linux version 3.0.15+ (oza@lc-blr-291) (gcc version 4.4.3 (GCC) )
> > #4 PREEMPT Mon Aug 13 12:02:58 IST 2012
> >
> > cat /proc/version
> > Linux version 3.0.15+ (oza@lc-blr-291) (gcc version 4.4.3 (GCC) )
> > #4 PREEMPT Mon Aug 13 12:02:58 IST 2012
>
> Looks OK -- so you'll have to debug verify_namelist(). It does a
> popen("/usr/bin/strings ./vmlinux") and searches for the linux_banner
> string here:
>
> while (fgets(buffer, BUFSIZE-1, pipe)) {
> if (!strstr(buffer, "Linux version 2.") &&
> !strstr(buffer, "Linux version 3."))
> continue;
>
> and when it finds it, it picks out the gcc version number. I don't
> know why it's not working in your case.
>
> Also, since your vmlinux file is identical to /proc/version, why are
> you using a System.map argument again?
>
> >
> > crash> ptype struct kmem_cache
> > type = struct kmem_cache {
> > struct array_cache *array[1];
> > unsigned int batchcount;
> > unsigned int limit;
> > unsigned int shared;
> > unsigned int buffer_size;
> > u32 reciprocal_buffer_size;
> > unsigned int flags;
> > unsigned int num;
> > unsigned int gfporder;
> > gfp_t gfpflags;
> > size_t colour;
> > unsigned int colour_off;
> > struct kmem_cache *slabp_cache;
> > unsigned int slab_size;
> > unsigned int dflags;
> > void (*ctor)(void *);
> > const char *name;
> > struct list_head next;
> > struct kmem_list3 *nodelists[1];
> > }
>
> OK, in this case you'll have to debug get_array_length(), and figure
> out why it's not finding the lookfor2 string -- which should be set
> to "*array[" -- in the ptype command output. Again, I don't know
> why it's not finding it.
>
> Dave
>
>
>
11 years, 9 months
Re: [Crash-utility] how to generate xen hypervisor core though kdump or other way?
by Daniel Kiper
Hi,
> crash 5.1.9
> Copyright (C) 2002-2011 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for details.
>
> crash: vmlinux-3.0.13-0.27-xen.gz: original filename unknown
> Use "-f vmlinux-3.0.13-0.27-xen.gz" on command line to prevent this message.
>
> GNU gdb (GDB) 7.0
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> KERNEL: vmlinux-3.0.13-0.27-xen.gz
> DEBUGINFO: ../lib/debug/boot/vmlinux-3.0.13-0.27-xen.debug
> DUMPFILE: /mnt/winimg/crash/2012-09-30-19:03/vmcore
> CPUS: 2
> DATE: Sun Sep 30 19:02:52 2012
> UPTIME: 07:45:24
> LOAD AVERAGE: 0.32, 0.23, 0.14
> TASKS: 207
> NODENAME: HjCloud
> RELEASE: 3.0.13-0.27-xen
> VERSION: #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b)
> MACHINE: x86_64 (2793 Mhz)
> MEMORY: 3.6 GB
> PANIC: "[27924.896523] Oops: 0002 [#1] SMP " (check log for details)
> PID: 7450
> COMMAND: "bash"
> TASK: ffff8800cfbf61c0 [THREAD_INFO: ffff8800cba86000]
> CPU: 1
> STATE: TASK_RUNNING (PANIC)
[...]
It looks that crash tool reads only dom0 part.
However, it looks that there is also some info
about Xen hypervisor state, too. I heard that
it is possible to do that in that way but personally
I did not such things. I think that you should
first check SLES documentation (man crash?)/groups/...
As I know this distribution use strongly customized
things (e.g. Linux Kernel) and they could behave
differently then vanilla one.
I am going to check that once but
now I am working on other things.
Daniel
11 years, 9 months
Re: [Crash-utility] how to generate xen hypervisor core though kdump or other way?
by Daniel Kiper
Hi,
> crash 5.1.9
> Copyright (C) 2002-2011 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for details.
>
> crash: vmlinux-3.0.13-0.27-xen.gz: original filename unknown
> Use "-f vmlinux-3.0.13-0.27-xen.gz" on command line to prevent this message.
>
> GNU gdb (GDB) 7.0
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> KERNEL: vmlinux-3.0.13-0.27-xen.gz
> DEBUGINFO: ../lib/debug/boot/vmlinux-3.0.13-0.27-xen.debug
> DUMPFILE: /mnt/winimg/crash/2012-09-30-19:03/vmcore
> CPUS: 2
> DATE: Sun Sep 30 19:02:52 2012
> UPTIME: 07:45:24
> LOAD AVERAGE: 0.32, 0.23, 0.14
> TASKS: 207
> NODENAME: HjCloud
> RELEASE: 3.0.13-0.27-xen
> VERSION: #1 SMP Wed Feb 15 13:33:49 UTC 2012 (d73692b)
> MACHINE: x86_64 (2793 Mhz)
> MEMORY: 3.6 GB
> PANIC: "[27924.896523] Oops: 0002 [#1] SMP " (check log for details)
> PID: 7450
> COMMAND: "bash"
> TASK: ffff8800cfbf61c0 [THREAD_INFO: ffff8800cba86000]
> CPU: 1
> STATE: TASK_RUNNING (PANIC)
[...]
One more question. What is the size of crash dump file?
Is it almost equal to:
<machine_memory_size> - <memory_reserved_for_crash_kernel>
Daniel
11 years, 9 months
Re: [Crash-utility] how to generate xen hypervisor core though kdump or other way?
by Daniel Kiper
Hi,
> Now I updated crash tool, but the issue still exists, It reports as follow:
[...]
As I remember "crash xen-syms-dbg-4.1.3_02-0.5.1 /mnt/disk/crash/2012-09-30-19\:03/vmcore"
should be sufficient. If it still does not work
please send me output from following command:
# readelf -Wa /mnt/disk/crash/2012-09-30-19\:03/vmcore
Daniel
11 years, 9 months
how to generate xen hypervisor core though kdump or other way?
by hu jun
Hi guys:
My server is Suse Linux SLES11 sp2 verison ,
HjCloud:~ # uname -a
Linux HjCloud 3.0.38-0.5-xen #1 SMP Fri Aug 3 09:02:17 UTC 2012 (358029e) x86_64 x86_64 x86_64 GNU/Linux
I know to get dom0 kdump core by "echo c >/proc/sysrq-trigger ",
but don't know how to get xen hypervisor core , then how to analysis the hypervisor core by Crash software.
Can you help me?
11 years, 9 months
