On X86 host, how to build crash tool which can analysis pwoerpc vmcore
by Liu, Jianbo (James)
Hi Experts:
I got vmcore file on powerpc board(P2020 or P2040), but failed to do crash analysis on x86 host, it showed we need to build a powerpc supported crash binary.
On the README of the latest crash-7.1.3, it state to get crash binary which can support powerpc32 vmcore analysis, we need to build with `make target=PPC` on PPC64 host, but currently, it's hard to find this kind of host to do build, if there are some other switch/method can let us to build powerpc supported binary on x86/x86_64 host?
Thanks a lot for your time!!
Best Regards,
James
Liu Jianbo | WIND RIVER | Senior Engineer - Technical Support
Tel 86 28 65318098 | Cell 86 13558641588 | Fax 86 28 65319983
[Description: Description: Wind River Support Network]<https://support.windriver.com/>
[Description: Description: Wind River Knowledge Forum]<https://ask.windriver.com/>
[Description: Description: Submit a Technical Service Request]<https://windriver.force.com/support>
[Description: Description: WIND]<http://www.windriver.com/>
10 years, 11 months
[PATCH] crash: allow also block devices as dump file input
by Michael Holzheu
On s390 we have stand-alone dump tools that write the dump directly to a
block device partition. It is possible to use the partition device node
instead of a dump file when running crash. When we do that, everything works
fine but we get the following (a bit misleading) warning:
WARNING: /dev/sda1: may be truncated or incomplete
PT_LOAD p_offset: 16384
p_filesz: 5497558138880
bytes required: 5497558155264
dumpfile size: 0
Fix this and print the following info message for block devices:
NOTE: /dev/sda1: No dump complete check for block devices
Signed-off-by: Michael Holzheu <holzheu(a)linux.vnet.ibm.com>
---
netdump.c | 5 +++++
1 file changed, 5 insertions(+)
--- a/netdump.c
+++ b/netdump.c
@@ -605,6 +605,11 @@ check_dumpfile_size(char *file)
if (stat64(file, &stat) < 0)
return;
+ if (S_ISBLK(stat.st_mode)) {
+ error(NOTE, "%s: No dump complete check for block devices\n",
+ file);
+ return;
+ }
for (i = 0; i < nd->num_pt_load_segments; i++) {
pls = &nd->pt_load_segments[i];
10 years, 11 months
crash tool - problem with new Xen linear virtual mapped sparse p2m list
by Daniel Kiper
Hi all,
Some time ago Linux kernel commit 054954eb051f35e74b75a566a96fe756015352c8
(xen: switch to linear virtual mapped sparse p2m list) introduced linear
virtual mapped sparse p2m list. It fixed some issues, however, it broke
crash tool too. I tried to fix this issue but the problem looks more
difficult than I expected.
Let's focus on "crash vmcore vmlinux". vmcore file was generated from dom0.
"crash vmcore xen-syms" works without any issue.
At first sight problem looks simple. Just add a function which reads p2m list
from vmcore and voila. I have done it. Then another issue arisen.
Please take a look at following backtrace:
#24426 0x000000000048b0f6 in readmem (addr=18446683600570023936, memtype=1, buffer=0x3c0a060, size=4096,
type=0x900b2f "xen_p2m_addr page", error_handle=2) at memory.c:2157
#24427 0x000000000050f746 in __xen_pvops_m2p_vma (machine=5323599872, mfn=1299707) at kernel.c:9050
#24428 0x000000000050edb7 in __xen_m2p (machine=5323599872, mfn=1299707) at kernel.c:8867
#24429 0x000000000050e948 in xen_m2p (machine=5323599872) at kernel.c:8796
#24430 0x0000000000528fca in x86_64_kvtop_xen_wpt (tc=0x0, kvaddr=18446683600570023936, paddr=0x7fff51c7c100, verbose=0)
at x86_64.c:1997
#24431 0x0000000000528890 in x86_64_kvtop (tc=0x0, kvaddr=18446683600570023936, paddr=0x7fff51c7c100, verbose=0)
at x86_64.c:1887
#24432 0x000000000048d708 in kvtop (tc=0x0, kvaddr=18446683600570023936, paddr=0x7fff51c7c100, verbose=0) at memory.c:2900
#24433 0x000000000048b0f6 in readmem (addr=18446683600570023936, memtype=1, buffer=0x3c0a060, size=4096,
type=0x900b2f "xen_p2m_addr page", error_handle=2) at memory.c:2157
#24434 0x000000000050f746 in __xen_pvops_m2p_vma (machine=5323599872, mfn=1299707) at kernel.c:9050
#24435 0x000000000050edb7 in __xen_m2p (machine=5323599872, mfn=1299707) at kernel.c:8867
#24436 0x000000000050e948 in xen_m2p (machine=5323599872) at kernel.c:8796
#24437 0x0000000000528fca in x86_64_kvtop_xen_wpt (tc=0x0, kvaddr=18446683600570023936, paddr=0x7fff51c7ca60, verbose=0)
at x86_64.c:1997
#24438 0x0000000000528890 in x86_64_kvtop (tc=0x0, kvaddr=18446683600570023936, paddr=0x7fff51c7ca60, verbose=0)
at x86_64.c:1887
#24439 0x000000000048d708 in kvtop (tc=0x0, kvaddr=18446683600570023936, paddr=0x7fff51c7ca60, verbose=0) at memory.c:2900
#24440 0x000000000048b0f6 in readmem (addr=18446683600570023936, memtype=1, buffer=0x3c0a060, size=4096,
type=0x900b2f "xen_p2m_addr page", error_handle=2) at memory.c:2157
#24441 0x000000000050f746 in __xen_pvops_m2p_vma (machine=6364917760, mfn=1553935) at kernel.c:9050
#24442 0x000000000050edb7 in __xen_m2p (machine=6364917760, mfn=1553935) at kernel.c:8867
#24443 0x000000000050e948 in xen_m2p (machine=6364917760) at kernel.c:8796
#24444 0x0000000000528fca in x86_64_kvtop_xen_wpt (tc=0x0, kvaddr=18446744072099176512, paddr=0x7fff51c7d3c0, verbose=0)
at x86_64.c:1997
#24445 0x0000000000528890 in x86_64_kvtop (tc=0x0, kvaddr=18446744072099176512, paddr=0x7fff51c7d3c0, verbose=0)
at x86_64.c:1887
#24446 0x000000000048d708 in kvtop (tc=0x0, kvaddr=18446744072099176512, paddr=0x7fff51c7d3c0, verbose=0) at memory.c:2900
#24447 0x000000000048b0f6 in readmem (addr=18446744072099176512, memtype=1, buffer=0xfbb500, size=768,
type=0x8fd772 "module struct", error_handle=6) at memory.c:2157
#24448 0x00000000004fb0ab in module_init () at kernel.c:3355
As you can see module_init() calls readmem() which attempts to read virtual address
which lies outside of kernel text mapping (0xffffffff80000000 - 0xffffffffa0000000).
In this case addr=18446744072099176512 == 0xffffffffa003a040 which is known as module
mapping space. readmem() needs physical address, so, it calls kvtop() then kvtop()
calls x86_64_kvtop(). x86_64_kvtop() is not able to easily calculate, using simple
arithmetic like in case of kernel text mapping space, physical address from virtual
address. Hence it calls x86_64_kvtop_xen_wpt() to calculate it by traversing page
tables. x86_64_kvtop_xen_wpt() needs to do some m2p translation so it calls xen_m2p()
which calls __xen_m2p() and finally it calls __xen_pvops_m2p_vma() (my function which
tries to read linear virtual mapped sparse p2m list). Then __xen_pvops_m2p_vma() calls
readmem() which tries to read addr=18446683600570023936 == 0xffffc90000000000 which
is VMA used for m2p list. Well, once again physical address must be calculated by
traversing page tables. However, this requires access to m2p list which leads to
another readmem() call. Starting from here we are in the loop. After thousands of
repetitions crash dies due to stack overflow. Not nice... :-(((
Do we have any viable fix for this issue? I considered a few but I have not found prefect one.
1) In theory we can use p2m tree to solve that problem because it is available in parallel
with VMA mapped p2m right now. However, this is temporary solution and it will be phased
out sooner or later. We need long term solution.
2) As I saw crash tool creates xkd->p2m_mfn_frame_list from Xen crash_xen_info_t.dom0_pfn_to_mfn_frame_list_list
in dom0 case. Potentially this would solve the problem for dom0 crash dumps but it does
not solve the problem for PV guests in general. We need something which is more generic.
3) The best thing which I was able to think of is to put list of PFNs containing p2m list
somewhere in ELF notes. Then readmem() called from __xen_pvops_m2p_vma() could use this
physical addresses calculated from PFNs somehow. However, this is also not perfect because
it requires changes in kernel, and/or xl and crash. Additionally, this solution increases
ELF notes. Every 1 GiB of memory will add 2 MiB of PFNs to ELF note. Probably there is
a chance that we can employ a compression method to reduce ELF note size but...
So, as you can see there is not perfect solution for this issue.
Hmmm... Should we mix solution #2 and #3? Is it better thing?
Daniel
10 years, 11 months
[PATCH v3 0/2] arm/arm64: less heuristics and qemu dump support
by Andrew Jones
The first patch allows us to rely on one less heuristic by taking
advantage of information recently made available to us through the
kernel image. The second patch enables analysis of qemu generated
dump files.
v3:
- Forgot the type == 'A' check needed to avoid a ton of unnecessary
strcmps [drew]
v2:
- Drop "arm64: relax symbol filters" patch and just stash the value of
the one absolute symbol we need "_kernel_flags_le". [Dave Anderson]
Also, while working on this, I testing several dumps. The following are
my results.
arm/aarch64 kvm guest kdump testing (P - PASS, F - FAIL). Testing done
with a latest mainline crash utility patched with these patches and a
latest mainline qemu with patches for dump generation.
.-----------------------------------------------------------------------.
| Host | arm32 | arm64 | arm64 | arm64 |
|---------------------------------------|-------|-------|-------|-------|
| Guest | arm32 | arm64 | arm64 | arm32 |
|---------------------------------------|-------|-------|-------|-------|
| Pagesize| 4K | 4K | 64K | 4K |
|=======================================================================|
| kdump in guest | F[1] | P[2] | P[3] | F[1] |
|---------------------------------------|-------|-------|-------|-------|
| qmp-dump-guest-memory <filename>[4] | P | P | P | P |
|---------------------------------------|-------|-------|-------|-------|
| qmp-dump-guest-memory -z <filename>[5]| F[8] | P | P | F[8] |
|---------------------------------------|-------|-------|-------|-------|
| qmp-dump-guest-memory -l <filename>[6]| F[8] | P | P | F[8] |
|---------------------------------------|-------|-------|-------|-------|
| qmp-dump-guest-memory -s <filename>[7]| F[8] | P | P | F[8] |
.-----------------------------------------------------------------------.
[1] Kernel v4.4-rc1 crashes with a NULL pointer dereference at virtual
address 00000000 in a memcpy (crash_kexec/machine_kexec/fncpy/memcpy).
Needs kernel debugging.
[2] Not sure about mainline, but works with the RHEL kernel,
makedumpfile does not yet support arm64 with 4K pages, but using
'core_collector cp' in /etc/kdump.conf allows saving an uncompressed
elf file.
[3] Not sure about mainline, but works with the RHEL kernel,
uses makedumpfile, thus generates a makedumpfile formatted file
using zlib compression.
[4] No format specified, creates an uncompressed elf formatted file.
[5] makedumpfile format, with zlib compression
[6] makedumpfile format, with lzo compression
[7] makedumpfile format, with snappy compression
[8] The crash utility doesn't seem to like arm32 dumps in makedumpfile
format. Looks like the physical page bitmap is all zeros? Needs
qemu and crash debugging.
Andrew Jones (2):
arm64: read pagesize
arm/arm64: read elf notes for qemu generated cores
arm.c | 22 ++++++++++++++++++++++
arm64.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
defs.h | 2 ++
netdump.c | 21 +++++++++++++++++++++
4 files changed, 97 insertions(+)
--
2.4.3
10 years, 11 months
Re: [Crash-utility] The problems when running SuSE 12 on VirtualBox
by Dave Anderson
Can you show the full output of "crash -d8"?
Sent from my Verizon Wireless 4G LTE smartphone
-------- Original message --------
From: Nan Xiao <xiaonan830818(a)gmail.com>
Date: 11/16/2015 8:55 PM (GMT-05:00)
To: "Discussion list for crash utility usage, maintenance and development" <crash-utility(a)redhat.com>
Subject: Re: [Crash-utility] The problems when running SuSE 12 on VirtualBox
Hi Dietmar & Dave,
Thanks very much for your time and answer! But for question 3, there
is still no answer.
I just execute "crash" with no parameters in the command line, and want to debug
Dom0:
# crash
......
WARNING: could not find MAGIC_START!
WARNING: cannot read linux_banner string
crash: /var/tmp/vmlinux-3.12.49-6-xen.gz_JWv5bU and /dev/mem do not match!
......
So why does this operation fail?
Best Regards
Nan Xiao
On Mon, Nov 16, 2015 at 10:48 PM, Dave Anderson <anderson(a)redhat.com> wrote:
>
>
> ----- Original Message -----
>> Am Montag 16 November 2015, 14:01:35 schrieb Nan Xiao:
>> > Hi Dave,
>> >
>> > >> (2) "machine type mismatch"
>> > >> WARNING: machine type mismatch:
>> > >> crash utility: X86_64
>> > >> /var/tmp/xen-4.5.1_10-1.gz_E5lLat: X86
>> > >>
>> > >> But according to "uname -a":
>> > >>
>> > >> # uname -a
>> > >> Linux linux-6ev3 3.12.49-6-xen #1 SMP Mon Oct 26 16:05:37 UTC 2015
>> > >> (11560c3) x86_64 x86_64 x86_64 GNU/Linux
>> > >>
>> > >> The running SuSE is also a 64-bit OS, so whether can remove these
>> > >> warnings?
>> >
>> > > The mismatch message says that "/var/tmp/xen-4.5.1_10-1.gz_E5lLat" is an X86
>> > > 32-bit binary. You cannot use a 32-bit xen binary with a 64-bit crash utility
>> > > binary.
>> >
>> > > Run "gunzip xen-4.5.1_10-1.gz", and then run "file" on the uncompressed file.
>> > > If it's a 32-bit binary, and the dumpfile came from that kernel, then you need
>> > > to use a 32-bit crash utility binary. If the dumpfile came from a 64-bit kernel,
>> > > then you're using the wrong xen binary.
>> >
>> > According to this
>> > post(http://lists.xen.org/archives/html/xen-users/2015-10/msg00141.html),
>> > although the xen file is 32-bit, it is the correct file for running on
>> > 64-bit OS. So I am not sure it should be considered as a bug.
>>
>> No it's not a bug.
>> The file you mentioned is only used for booting the hypervisor. For debugging
>> the hypervisor a statically linked hypervisor file has to be used.
>>
>> On SLES11 SP4 I have for booting the xen hypervisor:
>> # file xen-4.4.2_12-23.1
>> xen-4.4.2_12-23.1_x2: ELF 32-bit LSB executable, Intel 80386, version 1
>> (SYSV), statically linked, stripped
>>
>> For working with crash and the vmcore:
>> # file xen-syms-4.4.2_12-23.1
>> xen-syms-4.4.2_12-23.1_x2: ELF 64-bit LSB executable, x86-64, version 1
>> (SYSV), statically linked, not stripped
>
>
> Hi Dietmar,
>
> Thanks for clarifying that. I should have noticed he was trying to use the
> actual xen binary instead of the xen-syms file -- which is specified in the
> crash(8) man page.
>
> And I wasn't even aware that the actual xen binary is a 32-bit image -- good
> to know...
>
> Thanks,
> Dave
>
>
>
>
>
>
>>
>> >
>> > >> (3)
>> > >>
>> > >> WARNING: could not find MAGIC_START!
>> > >> WARNING: cannot read linux_banner string
>> > >> crash: /var/tmp/vmlinux-3.12.49-6-xen.gz_FjRFMJ and /dev/mem do not
>> > >> match!
>> > >>
>> > >> How can I fix this issue and run crash on SuSE?
>> >
>> > > For starters, make sure the kernel, the dumpfile, and the crash binary
>> > > are all
>> > > 32-bit or all 64-bit, whichever is appropriate.
>> >
>> > # which crash
>> > /usr/bin/crash
>> >
>> > # file /usr/bin/crash
>> > /usr/bin/crash: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
>> > dynamically linked (uses shared libs), for GNU/Linux 3.0.0,
>> > BuildID[sha1]=e6f9921f98c5b4daedd8dac798c139b77676b588, stripped
>> >
>> > Using "crash /boot/vmlinux-3.12.49-6-xen.gz /proc/kcore" works OK:
>>
>> This is expected as this is similar to the xen behaviour.
>> For booting:
>> # file vmlinuz-3.0.101-63-xen
>> vmlinuz-3.0.101-63-xen: Linux/x86 Kernel, Setup Version 0x20b, bzImage,
>> Version 3.0.101, Version 3.0.101-63, RO-rootFS, swap_dev 0x3, Normal VGA
>>
>> And for debugging:
>> # file vmlinux-3.0.101-63-xen
>> vmlinux-3.0.101-63-xen: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
>> statically linked, not stripped
>>
>> Dietmar.
>>
>> >
>> > # file /proc/kcore
>> > /proc/kcore: ELF 64-bit LSB core file x86-64, version 1 (SYSV),
>> > SVR4-style, from 'root=UUID=ed63d5d7-1d2f-45ea-a77b-5b3ae05c1c2e
>> > resume=/dev/sda1 splash=silent q'
>> >
>> > # crash /boot/vmlinux-3.12.49-6-xen.gz /proc/kcore
>> >
>> > crash 7.1.3
>> > Copyright (C) 2002-2014 Red Hat, Inc.
>> > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
>> > Copyright (C) 1999-2006 Hewlett-Packard Co
>> > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
>> > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
>> > Copyright (C) 2005, 2011 NEC Corporation
>> > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
>> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
>> > This program is free software, covered by the GNU General Public License,
>> > and you are welcome to change it and/or distribute copies of it under
>> > certain conditions. Enter "help copying" to see the conditions.
>> > This program has absolutely no warranty. Enter "help warranty" for
>> > details.
>> >
>> > GNU gdb (GDB) 7.6
>> > Copyright (C) 2013 Free Software Foundation, Inc.
>> > License GPLv3+: GNU GPL version 3 or later
>> > <http://gnu.org/licenses/gpl.html>
>> > This is free software: you are free to change and redistribute it.
>> > There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>> > and "show warranty" for details.
>> > This GDB was configured as "x86_64-unknown-linux-gnu"...
>> >
>> > KERNEL: /boot/vmlinux-3.12.49-6-xen.gz
>> > DEBUGINFO: /usr/lib/debug/boot/vmlinux-3.12.49-6-xen.debug
>> > DUMPFILE: /proc/kcore
>> > CPUS: 1
>> > DATE: Mon Nov 16 00:59:02 2015
>> > UPTIME: 04:55:39
>> > LOAD AVERAGE: 0.08, 0.03, 0.05
>> > TASKS: 241
>> > NODENAME: linux-6ev3
>> > RELEASE: 3.12.49-6-xen
>> > VERSION: #1 SMP Mon Oct 26 16:05:37 UTC 2015 (11560c3)
>> > MACHINE: x86_64 (2596 Mhz)
>> > MEMORY: 855.2 MB
>> > PID: 7472
>> > COMMAND: "crash"
>> > TASK: ffff880019ab6540 [THREAD_INFO: ffff880002dcc000]
>> > CPU: 0
>> > STATE: TASK_RUNNING (ACTIVE)
>> >
>> > So I think all the files are 64-bits.
>> >
>> > Thanks!
>> > Best Regards
>> > Nan Xiao
>> >
>> >
>> > On Fri, Nov 13, 2015 at 10:54 PM, Dave Anderson <anderson(a)redhat.com>
>> > wrote:
>> > >
>> > >
>> > > ----- Original Message -----
>> > >> Hi all,
>> > >>
>> > >> Firstly, I am not sure whether posting this issue on this mailing list
>> > >> is appropriate, if not,
>> > >> please forgive me, thanks!
>> > >>
>> > >> I install SuSE Enterprise Linux 12 (Xen version) on VirtualBox, and want
>> > >> to use crash.
>> > >> Executing "crash" command, and the output likes this:
>> > >>
>> > >> # crash
>> > >>
>> > >> crash 7.1.3
>> > >> Copyright (C) 2002-2014 Red Hat, Inc.
>> > >> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
>> > >> Copyright (C) 1999-2006 Hewlett-Packard Co
>> > >> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
>> > >> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
>> > >> Copyright (C) 2005, 2011 NEC Corporation
>> > >> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
>> > >> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
>> > >> This program is free software, covered by the GNU General Public
>> > >> License,
>> > >> and you are welcome to change it and/or distribute copies of it under
>> > >> certain conditions. Enter "help copying" to see the conditions.
>> > >> This program has absolutely no warranty. Enter "help warranty" for
>> > >> details.
>> > >>
>> > >> crash: /boot/symtypes-3.12.49-6-xen.gz: original filename unknown
>> > >> Use "-f /boot/symtypes-3.12.49-6-xen.gz" on command line to prevent
>> > >> this message.
>> > >>
>> > >> crash: /boot/symtypes-3.12.49-6-default.gz: original filename unknown
>> > >> Use "-f /boot/symtypes-3.12.49-6-default.gz" on command line to
>> > >> prevent this message.
>> > >>
>> > >> crash: /boot/xen-4.5.1_10-1.gz: original filename unknown
>> > >> Use "-f /boot/xen-4.5.1_10-1.gz" on command line to prevent this
>> > >> message.
>> > >>
>> > >> WARNING: machine type mismatch:
>> > >> crash utility: X86_64
>> > >> /var/tmp/xen-4.5.1_10-1.gz_E5lLat: X86
>> > >>
>> > >> crash: /boot/xen-4.5.gz: original filename unknown
>> > >> Use "-f /boot/xen-4.5.gz" on command line to prevent this message.
>> > >>
>> > >> WARNING: machine type mismatch:
>> > >> crash utility: X86_64
>> > >> /var/tmp/xen-4.5.gz_0BL7f2: X86
>> > >>
>> > >> crash: /boot/xen-4.gz: original filename unknown
>> > >> Use "-f /boot/xen-4.gz" on command line to prevent this message.
>> > >>
>> > >> WARNING: machine type mismatch:
>> > >> crash utility: X86_64
>> > >> /var/tmp/xen-4.gz_PW97tB: X86
>> > >>
>> > >> crash: /boot/xen-dbg-4.5.1_10-1.gz: original filename unknown
>> > >> Use "-f /boot/xen-dbg-4.5.1_10-1.gz" on command line to prevent this
>> > >> message.
>> > >>
>> > >> WARNING: machine type mismatch:
>> > >> crash utility: X86_64
>> > >> /var/tmp/xen-dbg-4.5.1_10-1.gz_zcILOa: X86
>> > >>
>> > >> crash: /boot/xen-dbg-4.5.gz: original filename unknown
>> > >> Use "-f /boot/xen-dbg-4.5.gz" on command line to prevent this message.
>> > >>
>> > >> WARNING: machine type mismatch:
>> > >> crash utility: X86_64
>> > >> /var/tmp/xen-dbg-4.5.gz_AEpJfK: X86
>> > >>
>> > >> crash: /boot/xen-dbg-4.gz: original filename unknown
>> > >> Use "-f /boot/xen-dbg-4.gz" on command line to prevent this message.
>> > >>
>> > >> WARNING: machine type mismatch:
>> > >> crash utility: X86_64
>> > >> /var/tmp/xen-dbg-4.gz_np36Nj: X86
>> > >>
>> > >> crash: /boot/xen-dbg.gz: original filename unknown
>> > >> Use "-f /boot/xen-dbg.gz" on command line to prevent this message.
>> > >>
>> > >> WARNING: machine type mismatch:
>> > >> crash utility: X86_64
>> > >> /var/tmp/xen-dbg.gz_1tAiuT: X86
>> > >>
>> > >> crash: /boot/xen.gz: original filename unknown
>> > >> Use "-f /boot/xen.gz" on command line to prevent this message.
>> > >>
>> > >> WARNING: machine type mismatch:
>> > >> crash utility: X86_64
>> > >> /var/tmp/xen.gz_htZLuv: X86
>> > >>
>> > >> crash: /boot/symvers-3.12.49-6-xen.gz: original filename unknown
>> > >> Use "-f /boot/symvers-3.12.49-6-xen.gz" on command line to prevent
>> > >> this message.
>> > >>
>> > >> GNU gdb (GDB) 7.6
>> > >> Copyright (C) 2013 Free Software Foundation, Inc.
>> > >> License GPLv3+: GNU GPL version 3 or later
>> > >> <http://gnu.org/licenses/gpl.html>
>> > >> This is free software: you are free to change and redistribute it.
>> > >> There is NO WARRANTY, to the extent permitted by law. Type "show
>> > >> copying"
>> > >> and "show warranty" for details.
>> > >> This GDB was configured as "x86_64-unknown-linux-gnu"...
>> > >>
>> > >> WARNING: could not find MAGIC_START!
>> > >> WARNING: cannot read linux_banner string
>> > >> crash: /var/tmp/vmlinux-3.12.49-6-xen.gz_FjRFMJ and /dev/mem do not
>> > >> match!
>> > >>
>> > >> Usage:
>> > >>
>> > >> crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
>> > >> crash [OPTION]... [NAMELIST] (live system
>> > >> form)
>> > >>
>> > >> Enter "crash -h" for details.
>> > >>
>> > >> My question are as follows:
>> > >>
>> > >> (1) There are so may warnings like this:
>> > >>
>> > >> crash: xxxxxx: original filename unknown
>> > >> Use "-f xxxxxx" on command line to prevent this message.
>> > >>
>> > >> Do I need to add somay "-f xxxxxx" options on command line? Is there
>> > >> any better method?
>> > >
>> > > The message above simply mean that the original filename of the
>> > > compressed
>> > > file was not stored in the file's header. That means that whoever
>> > > created
>> > > the compressed file used "gzip -n" or "--no-name":
>> > >
>> > > $ man gzip
>> > > ... [ cut ] ...
>> > >
>> > > -n --no-name
>> > > When compressing, do not save the original file name and time
>> > > stamp by default. (The original name is always saved if the
>> > > name
>> > > had to be truncated.) When decompressing, do not restore the
>> > > original file name if present (remove only the gzip suffix from
>> > > the compressed file name) and do not restore the original time
>> > > stamp
>> > > if present (copy it from the compressed file). This option is
>> > > the
>> > > default when decompressing.
>> > > ...
>> > >
>> > >> (2) "machine type mismatch"
>> > >> WARNING: machine type mismatch:
>> > >> crash utility: X86_64
>> > >> /var/tmp/xen-4.5.1_10-1.gz_E5lLat: X86
>> > >>
>> > >> But according to "uname -a":
>> > >>
>> > >> # uname -a
>> > >> Linux linux-6ev3 3.12.49-6-xen #1 SMP Mon Oct 26 16:05:37 UTC 2015
>> > >> (11560c3) x86_64 x86_64 x86_64 GNU/Linux
>> > >>
>> > >> The running SuSE is also a 64-bit OS, so whether can remove these
>> > >> warnings?
>> > >
>> > > The mismatch message says that "/var/tmp/xen-4.5.1_10-1.gz_E5lLat" is an
>> > > X86
>> > > 32-bit binary. You cannot use a 32-bit xen binary with a 64-bit crash
>> > > utility
>> > > binary.
>> > >
>> > > Run "gunzip xen-4.5.1_10-1.gz", and then run "file" on the uncompressed
>> > > file.
>> > > If it's a 32-bit binary, and the dumpfile came from that kernel, then you
>> > > need
>> > > to use a 32-bit crash utility binary. If the dumpfile came from a 64-bit
>> > > kernel,
>> > > then you're using the wrong xen binary.
>> > >
>> > >> (3)
>> > >>
>> > >> WARNING: could not find MAGIC_START!
>> > >> WARNING: cannot read linux_banner string
>> > >> crash: /var/tmp/vmlinux-3.12.49-6-xen.gz_FjRFMJ and /dev/mem do not
>> > >> match!
>> > >>
>> > >> How can I fix this issue and run crash on SuSE?
>> > >
>> > > For starters, make sure the kernel, the dumpfile, and the crash binary are all
>> > > 32-bit or all 64-bit, whichever is appropriate.
>> > >
>> > > Dave
>> > >
>> > >>
>> > >> Thanks very much in advance!
>> > >> Best Regards
>> > >> Nan Xiao
>> > >>
>> > >> --
>> > >> Crash-utility mailing list
>> > >> Crash-utility(a)redhat.com
>> > >> https://www.redhat.com/mailman/listinfo/crash-utility
>> > >>
>> > >
>> > > --
>> > > Crash-utility mailing list
>> > > Crash-utility(a)redhat.com
>> > > https://www.redhat.com/mailman/listinfo/crash-utility
>> >
>> > --
>> > Crash-utility mailing list
>> > Crash-utility(a)redhat.com
>> > https://www.redhat.com/mailman/listinfo/crash-utility
>> >
>>
>> --
>> Company details: http://ts.fujitsu.com/imprint.html
>>
>> --
>> Crash-utility mailing list
>> Crash-utility(a)redhat.com
>> https://www.redhat.com/mailman/listinfo/crash-utility
>>
>
> --
> Crash-utility mailing list
> Crash-utility(a)redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
10 years, 11 months
[PATCH v2 0/2] arm/arm64: less heuristics and qemu dump support
by Andrew Jones
The first patch allows us to rely on one less heuristic by taking
advantage of information recently made available to us through the
kernel image. The second patch enables analysis of qemu generated
dump files.
v2:
- Drop "arm64: relax symbol filters" patch and just stash the value of
the one absolute symbol we need "_kernel_flags_le". [Dave Anderson]
Also, while working on this, I testing several dumps. The following are
my results.
arm/aarch64 kvm guest kdump testing (P - PASS, F - FAIL). Testing done
with a latest mainline crash utility patched with these patches and a
latest mainline qemu with patches for dump generation.
.-----------------------------------------------------------------------.
| Host | arm32 | arm64 | arm64 | arm64 |
|---------------------------------------|-------|-------|-------|-------|
| Guest | arm32 | arm64 | arm64 | arm32 |
|---------------------------------------|-------|-------|-------|-------|
| Pagesize| 4K | 4K | 64K | 4K |
|=======================================================================|
| kdump in guest | F[1] | P[2] | P[3] | F[1] |
|---------------------------------------|-------|-------|-------|-------|
| qmp-dump-guest-memory <filename>[4] | P | P | P | P |
|---------------------------------------|-------|-------|-------|-------|
| qmp-dump-guest-memory -z <filename>[5]| F[8] | P | P | F[8] |
|---------------------------------------|-------|-------|-------|-------|
| qmp-dump-guest-memory -l <filename>[6]| F[8] | P | P | F[8] |
|---------------------------------------|-------|-------|-------|-------|
| qmp-dump-guest-memory -s <filename>[7]| F[8] | P | P | F[8] |
.-----------------------------------------------------------------------.
[1] Kernel v4.4-rc1 crashes with a NULL pointer dereference at virtual
address 00000000 in a memcpy (crash_kexec/machine_kexec/fncpy/memcpy).
Needs kernel debugging.
[2] Not sure about mainline, but works with the RHEL kernel,
makedumpfile does not yet support arm64 with 4K pages, but using
'core_collector cp' in /etc/kdump.conf allows saving an uncompressed
elf file.
[3] Not sure about mainline, but works with the RHEL kernel,
uses makedumpfile, thus generates a makedumpfile formatted file
using zlib compression.
[4] No format specified, creates an uncompressed elf formatted file.
[5] makedumpfile format, with zlib compression
[6] makedumpfile format, with lzo compression
[7] makedumpfile format, with snappy compression
[8] The crash utility doesn't seem to like arm32 dumps in makedumpfile
format. Looks like the physical page bitmap is all zeros? Needs
qemu and crash debugging.
Andrew Jones (3):
arm64: relax symbol filters
arm64: read pagesize
arm/arm64: read elf notes for qemu generated cores
arm.c | 22 ++++++++++++++++++++++
arm64.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
defs.h | 4 ++++
netdump.c | 21 +++++++++++++++++++++
4 files changed, 101 insertions(+), 2 deletions(-)
--
2.4.3
Andrew Jones (2):
arm64: read pagesize
arm/arm64: read elf notes for qemu generated cores
arm.c | 22 ++++++++++++++++++++++
arm64.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
defs.h | 2 ++
netdump.c | 21 +++++++++++++++++++++
4 files changed, 97 insertions(+)
--
2.4.3
10 years, 11 months
[PATCH 0/3] arm/arm64: less heuristics and qemu dump support
by Andrew Jones
The first two patches allow us to rely on one less heuristic by taking
advantage of information recently made available to us through the
kernel image. The second patch enables analysis of qemu generated
dump files.
Also, while working on this, I testing several dumps. The following are
my results.
arm/aarch64 kvm guest kdump testing (P - PASS, F - FAIL). Testing done
with a latest mainline crash utility patched with these patches and a
latest mainline qemu with patches for dump generation.
.-----------------------------------------------------------------------.
| Host | arm32 | arm64 | arm64 | arm64 |
|---------------------------------------|-------|-------|-------|-------|
| Guest | arm32 | arm64 | arm64 | arm32 |
|---------------------------------------|-------|-------|-------|-------|
| Pagesize| 4K | 4K | 64K | 4K |
|=======================================================================|
| kdump in guest | F[1] | P[2] | P[3] | F[1] |
|---------------------------------------|-------|-------|-------|-------|
| qmp-dump-guest-memory <filename>[4] | P | P | P | P |
|---------------------------------------|-------|-------|-------|-------|
| qmp-dump-guest-memory -z <filename>[5]| F[8] | P | P | F[8] |
|---------------------------------------|-------|-------|-------|-------|
| qmp-dump-guest-memory -l <filename>[6]| F[8] | P | P | F[8] |
|---------------------------------------|-------|-------|-------|-------|
| qmp-dump-guest-memory -s <filename>[7]| F[8] | P | P | F[8] |
.-----------------------------------------------------------------------.
[1] Kernel v4.4-rc1 crashes with a NULL pointer dereference at virtual
address 00000000 in a memcpy (crash_kexec/machine_kexec/fncpy/memcpy).
Needs kernel debugging.
[2] Not sure about mainline, but works with the RHEL kernel,
makedumpfile does not yet support arm64 with 4K pages, but using
'core_collector cp' in /etc/kdump.conf allows saving an uncompressed
elf file.
[3] Not sure about mainline, but works with the RHEL kernel,
uses makedumpfile, thus generates a makedumpfile formatted file
using zlib compression.
[4] No format specified, creates an uncompressed elf formatted file.
[5] makedumpfile format, with zlib compression
[6] makedumpfile format, with lzo compression
[7] makedumpfile format, with snappy compression
[8] The crash utility doesn't seem to like arm32 dumps in makedumpfile
format. Looks like the physical page bitmap is all zeros? Needs
qemu and crash debugging.
Andrew Jones (3):
arm64: relax symbol filters
arm64: read pagesize
arm/arm64: read elf notes for qemu generated cores
arm.c | 22 ++++++++++++++++++++++
arm64.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--
defs.h | 4 ++++
netdump.c | 21 +++++++++++++++++++++
4 files changed, 101 insertions(+), 2 deletions(-)
--
2.4.3
10 years, 11 months
Support for dynamic allocations of 'struct fpu'
by Atsushi Kumagai
Hello,
This is a follow-up report of the issue I found in the release test
for makedumpfile-1.5.9. The original report was posted in kexec-ML:
http://lists.infradead.org/pipermail/kexec/2015-October/014620.html
> o Support new kernels
> - The supported kernel is updated to 4.1 in this version.
> At first I'm going to extend the supported version to 4.2, but
> I found an issue that makedumpfile seems to exclude necessary pages
> by mistake on linux 4.2. When crash-7.1.3 reads that filtered dump file,
> the following message is shown.
>
> crash: page excluded: kernel virtual address: f3fe0000 type: "fill_task_struct"
>
> This will be fixed in the next version.
I looked for the cause of this issue and found that it doesn't seem to be an
issue of makedumpfile, it seems to be a crash side issue.
The size of task_struct is decided dynamically by FPU registers since linux 4.2
due to:
commit 5aaeb5c01c5b6c0be7b7aadbf3ace9f3a4458c3d
Author: Ingo Molnar <mingo(a)kernel.org>
Date: Fri Jul 17 12:28:12 2015 +0200
x86/fpu, sched: Introduce CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT and use it on x86
and
commit 0c8c0f03e3a292e031596484275c14cf39c0ab7a
Author: Dave Hansen <dave(a)sr71.net>
Date: Fri Jul 17 12:28:11 2015 +0200
86/fpu, sched: Dynamically allocate 'struct fpu'
This change will cause a difference between dwarf info and the actual size
like below:
(This is an example in linux 4.2 on x86_64)
- dwarf info
$ dwarfdump vmlinux | grep -A 2 task_struct
DW_AT_name "task_struct"
DW_AT_byte_size 0x00001940 // 6464 byte
DW_AT_decl_file 0x0000001a include/linux/sched.h
- actual size
crash> p arch_task_struct_size
arch_task_struct_size = $1 = 2880
crash>
I don't think crash handle this change, so crash can read an irrelevant
page when trying to read a task_struct. If the dump is filtered by makedumpfile
and the page just behind the task_struct is excluded, the message I reported
will be shown.
To fix the size_table for the crash's initialization is easy, we should
just update it by arch_task_struct_size like:
diff --git a/task.c b/task.c
index 8956fb5..ee94d4e 100755
--- a/task.c
+++ b/task.c
@@ -284,6 +284,17 @@ task_init(void)
MEMBER_OFFSET_INIT(pid_pid_chain, "pid", "pid_chain");
STRUCT_SIZE_INIT(task_struct, "task_struct");
+ int task_struct_size;
+ if (kernel_symbol_exists("arch_task_struct_size") &&
+ readmem(symbol_value("arch_task_struct_size"), KVADDR,
+ &task_struct_size, sizeof(int),
+ "arch_task_struct_size", RETURN_ON_ERROR)) {
+ ASSIGN_SIZE(task_struct) = task_struct_size;
+ if (CRASHDEBUG(1))
+ fprintf(fp, "\downsize_task_struct: %ld to %ld\n",
+ STRUCT_SIZE("task_struct"),
+ SIZE(task_struct));
+ }
However, struct command always refer to dwarf info since it's probably
designed for general purpose, I can't come up with good way to fix it.
Do you have any comments ?
Thanks,
Atsushi Kumagai
10 years, 11 months
The problems when running SuSE 12 on VirtualBox
by Nan Xiao
Hi all,
Firstly, I am not sure whether posting this issue on this mailing list
is appropriate, if not,
please forgive me, thanks!
I install SuSE Enterprise Linux 12 (Xen version) on VirtualBox, and
want to use crash.
Executing "crash" command, and the output likes this:
# crash
crash 7.1.3
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
crash: /boot/symtypes-3.12.49-6-xen.gz: original filename unknown
Use "-f /boot/symtypes-3.12.49-6-xen.gz" on command line to prevent
this message.
crash: /boot/symtypes-3.12.49-6-default.gz: original filename unknown
Use "-f /boot/symtypes-3.12.49-6-default.gz" on command line to
prevent this message.
crash: /boot/xen-4.5.1_10-1.gz: original filename unknown
Use "-f /boot/xen-4.5.1_10-1.gz" on command line to prevent this message.
WARNING: machine type mismatch:
crash utility: X86_64
/var/tmp/xen-4.5.1_10-1.gz_E5lLat: X86
crash: /boot/xen-4.5.gz: original filename unknown
Use "-f /boot/xen-4.5.gz" on command line to prevent this message.
WARNING: machine type mismatch:
crash utility: X86_64
/var/tmp/xen-4.5.gz_0BL7f2: X86
crash: /boot/xen-4.gz: original filename unknown
Use "-f /boot/xen-4.gz" on command line to prevent this message.
WARNING: machine type mismatch:
crash utility: X86_64
/var/tmp/xen-4.gz_PW97tB: X86
crash: /boot/xen-dbg-4.5.1_10-1.gz: original filename unknown
Use "-f /boot/xen-dbg-4.5.1_10-1.gz" on command line to prevent this message.
WARNING: machine type mismatch:
crash utility: X86_64
/var/tmp/xen-dbg-4.5.1_10-1.gz_zcILOa: X86
crash: /boot/xen-dbg-4.5.gz: original filename unknown
Use "-f /boot/xen-dbg-4.5.gz" on command line to prevent this message.
WARNING: machine type mismatch:
crash utility: X86_64
/var/tmp/xen-dbg-4.5.gz_AEpJfK: X86
crash: /boot/xen-dbg-4.gz: original filename unknown
Use "-f /boot/xen-dbg-4.gz" on command line to prevent this message.
WARNING: machine type mismatch:
crash utility: X86_64
/var/tmp/xen-dbg-4.gz_np36Nj: X86
crash: /boot/xen-dbg.gz: original filename unknown
Use "-f /boot/xen-dbg.gz" on command line to prevent this message.
WARNING: machine type mismatch:
crash utility: X86_64
/var/tmp/xen-dbg.gz_1tAiuT: X86
crash: /boot/xen.gz: original filename unknown
Use "-f /boot/xen.gz" on command line to prevent this message.
WARNING: machine type mismatch:
crash utility: X86_64
/var/tmp/xen.gz_htZLuv: X86
crash: /boot/symvers-3.12.49-6-xen.gz: original filename unknown
Use "-f /boot/symvers-3.12.49-6-xen.gz" on command line to prevent
this message.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: could not find MAGIC_START!
WARNING: cannot read linux_banner string
crash: /var/tmp/vmlinux-3.12.49-6-xen.gz_FjRFMJ and /dev/mem do not match!
Usage:
crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
Enter "crash -h" for details.
My question are as follows:
(1) There are so may warnings like this:
crash: xxxxxx: original filename unknown
Use "-f xxxxxx" on command line to prevent this message.
Do I need to add somay "-f xxxxxx" options on command line? Is there
any better method?
(2) "machine type mismatch"
WARNING: machine type mismatch:
crash utility: X86_64
/var/tmp/xen-4.5.1_10-1.gz_E5lLat: X86
But according to "uname -a":
# uname -a
Linux linux-6ev3 3.12.49-6-xen #1 SMP Mon Oct 26 16:05:37 UTC 2015
(11560c3) x86_64 x86_64 x86_64 GNU/Linux
The running SuSE is also a 64-bit OS, so whether can remove these warnings?
(3)
WARNING: could not find MAGIC_START!
WARNING: cannot read linux_banner string
crash: /var/tmp/vmlinux-3.12.49-6-xen.gz_FjRFMJ and /dev/mem do not match!
How can I fix this issue and run crash on SuSE?
Thanks very much in advance!
Best Regards
Nan Xiao
10 years, 11 months
Traversing hlist_node linked structures
by Nikolay Borisov
Hello,
I'd like to ask whether it's possible to traverse the chains of a
hashtable which is defined via the hashtable.h infrastructure? If
there is no support for this currently I guess crash has to be taught
to understand how to do the maths to extract the container structures
from embedded hlist_nodes, similar to how it does it for list_head
type of structs? And unless this is done doing the arithmetic manually
is the only way?
Regards,
Nikolay
10 years, 11 months
