4.0-3.21 cross crash support.
by Maxim Syrchin
Good news everyone,
cross-crash patch for crash-4.0.3.21 is available now on
http://sourceforge.net/projects/crosscrash/.
Current version allows the analysis of linux kernel core dumps by Linux
Kernel Crash Dump (LKCD) up to version 9 LKCD format.
The arch support matrix is below:
host arch | dump arch | test results
i386 | i386 | OK
i386 | x86_64 | OK
i386 | PPC32 | OK
i386 | PPC64 | not tested
x86_64 | all | not tested*
PPC32 | all | not tested*
PPC64 | all | not tested
* - non-cross usage is available and works
IA64, alpha, s390(x) architectures are not fixed for cross analyse
Any comments are welcome.
Thanks,
Maxim Syrchin,
17 years, 7 months
crash version 4.0-4.1 is available
by Dave Anderson
- Implemented dependable backtraces for the x86_64 architecture. (!!!)
This feature builds upon the current "low_budget" backtrace function,
and also required the fix for the BUG()/ud2a disassembly problem
addressed in 4.0-3.22. It does not require kernel unwind support,
but rather it calculates function framesizes by disassembling the
code from the beginning of the function to the point where it calls
the next function, parsing for add or sub instructions on the rsp,
and for push and pop instructions, thereby determining the framesize
of the function at the point of the call. This is similar to what is
done for x86, but requires far less hackery. You will notice a slight
hitch the first time a "bt" is done on a task, but for each text
return address in any backtrace, its framesize is cached for all
subsequent instances. It also accounts for backtrace text return
addresses from the .text.lock section, by appending "(via function)"
to the end of the frame line. Also, because it layers on top of the
current backtrace code, it does not compromise the capability of
switching between the process, IRQ, and exception stacks. That all
being said, 100% accuracy cannot be guaranteed. But for the ~30
sample dumpfiles I keep around for x86_64 testing, I cannot find any
obviously invalid backtraces. However, if there is any doubt, the
"bt -o" option will perform backtraces using the "old" manner; and
"bt -O" will force the old manner to always be used. Of course the
"bt -t" and "bt -T" options are still available. It's interesting to
redirect the output of "foreach bt" to a file using this version, and
then compare it with the output from an older version.
(anderson(a)redhat.com)
- Fix for s390 and s390x backtrace commands to recognize the kernel
structure name change from "runqueue" to "rq".
(holzheu(a)linux.vnet.ibm.com)
- Merged fourth round of "xencrash" patches, which allows a crash
session to alternatively be brought up against the xen-syms
binary instead of a vmlinux kernel. This patch enhances the
"doms" command display contents, and adds support to access the
ia64 frame table virtual address space so that the page_info table
can be accessed. (oda(a)valinux.co.jp)
Download from: http://people.redhat.com/anderson
17 years, 7 months
xencrash some enhancement
by Itsuro ODA
Hi,
The attached patch includes some enhancements of xencrash:
* "doms" adds some itmes to show (as follows)
----------------------------------------------------------------
DID DOMAIN ST T MAXPAGE TOTPAGE VCPU SHARED_I P2M_MFN
32753 ffbf8080 RU O 0 0 0 0 ----
32754 ffbfa080 RU X 0 0 0 0 ----
32767 ffbfc080 RU I 0 0 2 0 ----
>* 0 ff198080 RU 0 ffffffff 32900 2 ff194000 18d0
4 ffbee080 RU U 4000 4000 2 ff18d000 3eb92
5 ff186080 RU U 4000 4000 2 ff184000 298d3
---------------------------------------------------------------
* IA64: support to access frame table virtual address space.
(so the page_info table can be accessed now.)
The patch is for crash-4.0-3.22.
Thanks.
--
Itsuro ODA <oda(a)valinux.co.jp>
17 years, 7 months
Heads up -- live system "failure" on later x86_64 Fedora kernels
by Dave Anderson
If you see this when running on live 2.6.19 or later Fedora x86_64 kernels:
# crash
crash 4.0-3.22
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
crash: read error: kernel virtual address: ffffffff8062a180 type: "xtime"
#
It's not a crash bug. And it can be worked around by entering:
# crash /dev/mem
The problem is with the intermingling of upstream and Red Hat kernels,
where in the 2.6.18+ timeframe, Andi Kleen made the x86_64 e820
map __initdata. That's OK for upstream kernels, because the x86_64
e820 map is not used during runtime.
But because Red Hat kernels impose a restriction on /dev/mem
to only allow access to the first 256 pages of RAM, an x86_64
version of page_is_ram() is also necessary for Red Hat kernels.
Furthermore, for live system analysis, I had to create a Red Hat
"/dev/crash" driver to access live kernel memory, and it also uses
page_is_ram() as a memory qualifier -- which ends up looking at
reallocated garbage data as the e820 map, and fails.
Interestingly enough, the bug opens the flood-gates for /dev/mem
because of this:
/*
* devmem_is_allowed() checks to see if /dev/mem access to a certain address is
* valid. The argument is a physical page number.
*
*
* On x86-64, access has to be given to the first megabyte of ram because that area
* contains bios code and data regions used by X and dosemu and similar apps.
* Access has to be given to non-kernel-ram areas as well, these contain the PCI
* mmio resources as well as potential bios/acpi data regions.
*/
int devmem_is_allowed(unsigned long pagenr)
{
if (pagenr <= 256)
return 1;
if (!page_is_ram(pagenr))
return 1;
return 0;
}
If a page frame above 256 is requested it's only supposed to be
acceptible it if it's *not* RAM -- but since page_is_ram() is failing,
*anything* is accepted.
And so this inadvertantly allows /dev/mem to be used by crash,
as is done for non-Red Hat kernels.
Conceivably there could be a scenario where the garbage data
may contain something that would cause page_is_ram() to
return successfully, in which case even using the crash /dev/mem
command line argument could potentially cause crash to fail. But in
that case, there is no workaround.
In any case, the bug will be fixed in subsequent Fedora kernels.
Dave
17 years, 7 months
[PATCH] Runqueue fix for s390
by holzheu
Hi Dave,
For s390(x) we check the runqueues to find out, which task is currently
running on a cpu.
In 2.6.19 "struct runqueue" was renamed to "struct rq". This patch uses
"struct rq" in case "struct runqueue" is not available in the vmlinux
debug info.
---
s390.c | 16 ++++++++++++----
s390x.c | 16 ++++++++++++----
2 files changed, 24 insertions(+), 8 deletions(-)
diff -Naurp crash-4.0-3.22/s390.c crash-4.0-3.22-s390-rq-fix/s390.c
--- crash-4.0-3.22/s390.c 2007-04-10 16:27:20.000000000 +0200
+++ crash-4.0-3.22-s390-rq-fix/s390.c 2007-04-24 17:01:20.000000000 +0200
@@ -538,7 +538,7 @@ s390_has_cpu(unsigned long task)
/* Linux 2.6 */
unsigned long runqueue_addr, runqueue_offset;
unsigned long cpu_offset, per_cpu_offset_addr, running_task;
- char runqueue[4096];
+ char rq[4096];
int cpu;
cpu = s390_cpu_of_task(task);
@@ -549,10 +549,18 @@ s390_has_cpu(unsigned long task)
&cpu_offset, sizeof(long),"per_cpu_offset",
FAULT_ON_ERROR);
runqueue_addr=runqueue_offset + cpu_offset;
- readmem(runqueue_addr,KVADDR,&runqueue,sizeof(runqueue),
- "runqueue", FAULT_ON_ERROR);
- running_task = *((unsigned long*)&runqueue[MEMBER_OFFSET(
+ readmem(runqueue_addr,KVADDR,&rq, sizeof(rq), "rq",
+ FAULT_ON_ERROR);
+
+ if (STRUCT_EXISTS("runqueue"))
+ running_task = *((unsigned long*)&rq[MEMBER_OFFSET(
"runqueue", "curr")]);
+ else if (STRUCT_EXISTS("rq"))
+ running_task = *((unsigned long*)&rq[MEMBER_OFFSET(
+ "rq", "curr")]);
+ else
+ error(WARNING, "Unable to determine running task!\n");
+
if(running_task == task)
return TRUE;
else
diff -Naurp crash-4.0-3.22/s390x.c crash-4.0-3.22-s390-rq-fix/s390x.c
--- crash-4.0-3.22/s390x.c 2007-04-10 16:27:20.000000000 +0200
+++ crash-4.0-3.22-s390-rq-fix/s390x.c 2007-04-24 17:02:56.000000000 +0200
@@ -572,7 +572,7 @@ s390x_has_cpu(unsigned long task)
/* Linux 2.6 */
unsigned long runqueue_addr, runqueue_offset;
unsigned long cpu_offset, per_cpu_offset_addr, running_task;
- char runqueue[4096];
+ char rq[4096];
int cpu;
cpu = s390x_cpu_of_task(task);
@@ -583,10 +583,18 @@ s390x_has_cpu(unsigned long task)
&cpu_offset, sizeof(long),"per_cpu_offset",
FAULT_ON_ERROR);
runqueue_addr=runqueue_offset + cpu_offset;
- readmem(runqueue_addr,KVADDR,&runqueue,sizeof(runqueue),
- "runqueue", FAULT_ON_ERROR);
- running_task = *((unsigned long*)&runqueue[MEMBER_OFFSET(
+ readmem(runqueue_addr,KVADDR,&rq,sizeof(rq), "rq",
+ FAULT_ON_ERROR);
+
+ if (STRUCT_EXISTS("runqueue"))
+ running_task = *((unsigned long*)&rq[MEMBER_OFFSET(
"runqueue", "curr")]);
+ else if (STRUCT_EXISTS("rq"))
+ running_task = *((unsigned long*)&rq[MEMBER_OFFSET(
+ "rq", "curr")]);
+ else
+ error(WARNING, "Unable to determine running task!\n");
+
if(running_task == task)
return TRUE;
else
17 years, 7 months
Finding thread/task idle time
by Nipul Gandhi
Hi all -
When looking at a vmcore, is there anyway to
determine how long a thread has been idle ?
When debugging hangs, it would be very important to
know this....if example, if a thread is blocked
in biowait()/submit_bio() for more than a few milliseconds.
Any insight appreciated....An example would be
ideal.
Thanks in advance,
Nipul
--
------------
Nipul Gandhi
x1306
VERITAS
17 years, 7 months
Fw: [Fastboot] makedumpfile-1.1.3: linux-2.6.21 support
by Ken'ichi Ohmichi
Hi all,
Today, I released a new makedumpfile related to the following thread.
Re: [Crash-utility] handling missing kdump pages in diskdump format
Thank you for the cooperations.
Thanks
Ken'ichi Ohmichi
To: fastboot-ml <fastboot(a)lists.linux-foundation.org>
From: "Ken'ichi Ohmichi" <oomichi(a)mxs.nes.nec.co.jp>
Date: Fri, 13 Apr 2007 14:59:01 +0900
Cc: Dave Anderson <anderson(a)redhat.com>
Subject: [Fastboot] makedumpfile-1.1.3: linux-2.6.21 support
Hi.
makedumpfile version 1.1.3 is released.
Please test it. Your opinions/comments are welcome.
Explanation of makedumpfile:
To shorten the size of the dumpfile and the time of creating the
dumpfile, makedumpfile copies only the necessary pages for analysis
to the dumpfile from /proc/vmcore. You can specify the kind of
unnecessary pages with dump_level. If you want to shorten the size
further, enable the compression of the page data.
Changelog:
o linux-2.6.21 is supported.
On linux-2.6.21, a member free_pages of struct zone has been removed.
Then, the number of free pages is taken from zone.vm_stat[0] if a
member free_pages of struct zone is not present.
The patch was provided by Sachin P. Sant, Thanks.
o The descriptors of the excluded zero-filled pages are changed so that
the crash utility (v4.0-3.22, or later) can distinguish between the
excluded zero-filled pages and other excluded pages.
If the crash utility accesses the excluded zero-filled page, it returns
zero-filled page. If accessing other excluded page, it displays the
error message and fails.
The patch was provided by Bob Montgomery, Thanks.
o A member "dump_level" is added into struct kdump_sub_header so that
the crash utility (v4.0-3.22, or later) can display the dump_level
in case of kdump-compressed format.
You can download the latest makedumpfile(ver. 1.1.3) from the following URL.
Details of the change are written on the CVS page of the following site.
https://sourceforge.net/projects/makedumpfile/
Method of installation:
You can compile the makedumpfile command as follows;
1. "tar -zxvf makedumpfile-1.1.3.tar.gz"
2. "cd makedumpfile"
3. "make; make install"
Usage:
makedumpfile [-c] [-E] [-d dump_level] [-x vmlinux] dump_mem dump_file
Example:
If you want to exclude pages filled by zero, cache pages, user pages
and free pages and to enable compression, please execute the following
command.
# makedumpfile -c -d 31 -x vmlinux /proc/vmcore dumpfile
Todo:
- ia64 DISCONTIGMEM support
Thanks
Ken'ichi Ohmichi
_______________________________________________
fastboot mailing list
fastboot(a)lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/fastboot
------- ここまでが転送メッセージ
17 years, 7 months
crash version 4.0-3.22 is available
by Dave Anderson
- In kernel version 2.6.20 a "__bug_table" section has been added
to the kernel for x86 and x86_64, which contains the encoding for
the filename and line number information associated with each
instance of a kernel BUG(). Prior to that, x86 and x86_64 kernels
may have contained the filename/line-number encoding in the bytes
following the BUG()'s "ud2a" instruction. When disassembled, the
output would display a series of nonsensical instructions, or perhaps
one or more "(bad)" instruction lines, before eventually getting
back in sync with the actual instruction stream. Whether the
encoded bytes were included depends upon the kernel version,
whether CONFIG_DEBUG_BUGVERBOSE was configured, or whether an
"#if 1" surrounding the BUG() definition was manually changed.
This version of crash determines whether the encoded bytes exist,
and if so, the embedded gdb disassembler has been modified to
skip over those bytes, resulting in correct "dis" command output.
If necessary, a "dis -b" option has been added to override the
pre-calculated encoded byte count value. (anderson(a)redhat.com)
- Fix for the x86 backtrace code to also recognize the encoded
filename and line number information potentially following
"ud2a" instructions generated by kernel BUG() calls. In order
to determine the framesize of a function, the backtrace code
does its own text disassembly to count instances of push, pop,
and stack register increments/decrements. Without this patch,
the framesize calculation may either be too small or too large,
depending upon the contents of the encoded data following the
BUG()'s ud2a instruction. Therefore, it is possible that one or
more bogus frames are selected and displayed, and/or one or more
legitimate frames are skipped over. For example, when it affected
the framesize calculation of schedule(), backtraces of all non-active
tasks ending up in schedule() would be invalid. Here's an example in
which the schedule() framesize was miscalulated:
PID: 1292 TASK: ed78a000 CPU: 0 COMMAND: "setroubleshootd"
#0 [c07fdba8] schedule at c05f370e
#1 [c07fdcb4] __journal_file_buffer at ee05126d
#2 [c07fdcd8] __journal_file_buffer at ee05126d
#3 [c07fdd08] ext3_mark_iloc_dirty at ee08837d
#4 [c07fdd38] journal_dirty_metadata at ee052a13
#5 [c07fdd80] __find_get_block at c0463f59
#6 [c07fddac] __find_get_block at c0463f59
#7 [c07fddf0] find_get_page at c0444294
#8 [c07fddfc] filemap_nopage at c0446cf5
#9 [c07fde6c] find_extend_vma at c0454132
#10 [c07fde7c] get_futex_key at c042f9f6
#11 [c07fde94] futex_wake at c042fe2a
#12 [c07fdeb8] do_futex at c0430a19
#13 [c07fdfac] sys_poll at c047254b
#14 [c07fdfb8] system_call at c0404cf8
EAX: ffffffda EBX: 09f3da18 ECX: 00000002 EDX: 00000064
DS: 007b ESI: 00000064 ES: 007b EDI: 00342ff4
SS: 007b ESP: bfe76d04 EBP: bfe76d18
CS: 0073 EIP: 0094a402 ERR: 000000a8 EFLAGS: 00200246
With the fix, it looks like this:
PID: 1292 TASK: ed78a000 CPU: 0 COMMAND: "setroubleshootd"
#0 [c07fdba8] schedule at c05f370e
#1 [c07fdc0c] schedule_timeout at c05f3e7c
#2 [c07fdc30] do_sys_poll at c047243e
#3 [c07fdfac] sys_poll at c047254b
#4 [c07fdfb8] system_call at c0404cf8
EAX: ffffffda EBX: 09f3da18 ECX: 00000002 EDX: 00000064
DS: 007b ESI: 00000064 ES: 007b EDI: 00342ff4
SS: 007b ESP: bfe76d04 EBP: bfe76d18
CS: 0073 EIP: 0094a402 ERR: 000000a8 EFLAGS: 00200246
In the example above, the schedule() framesize was miscalculated
because the post-ud2a text contained the filename pointer address
c060fe0b, and the "60" was decoded as a "pusha" instruction; that
occurred twice, each time incrementing the framesize by 32 bytes.
(anderson(a)redhat.com)
- Added preparations for an upcoming version update to kdump's
associated makedumpfile utility, which will return an error if a
read attempt of a page that has been explicitly excluded is made.
Until now, a zero-filled page was returned. To maintain the
current behavior of returning a zero-filled page when accessing
an excluded page, three options are available:
1) use the "--zero_excluded" crash command line option.
2) during runtime, enter "set zero_excluded on".
3) enter "set zero_excluded on" in your .crashrc file.
(anderson(a)redhat.com, oomichi(a)mxs.nes.nec.co.jp, bob.montgomery(a)hp.com)
- Implemented "help -n" debug output function for compressed diskdump
and compressed kdump dumpfiles. As is done for the other dumpfile
formats, the core file's header information along with any other
run-time dumpfile data is displayed. (anderson(a)redhat.com)
- If the page-exclusion "dump_level" of a compressed diskdump, a
compressed kdump, or an ELF diskdump dumpfile exists and can be
determined, its value and bitmask translation will be displayed as
part of the "help -n" dumpfile debug output. Also, has been done
with partial ELF diskdumps, if a compressed diskdump or compressed
kdump can be confirmed as a partial dump, the "[PARTIAL DUMP]"
indicator will follow the dumpfile name during initialization and by
the "sys" command. (anderson(a)redhat.com, oomichi(a)mxs.nes.nec.co.jp,
indou.takao(a)jp.fujitsu.com, akiyama.nobuyuk(a)jp.fujitsu.com)
- Support for xendumps of fully-virtualized x86_64 relocatable
kernels. Without the patch, the physical base address was not
being determined, and the session would fail during initialization
with the error message: " crash: vmlinux and core do not match!"
(anderson(a)redhat.com)
- Fix for 4.0-3.21 "BOOKE" ppc.c patch, which failed to compile.
(antipov(a)ru.mvista.com)
Download from: http://people.redhat.com/anderson
17 years, 7 months
crash version 4.0-3.21 is available
by Dave Anderson
- Introduced support for upstream xensource ELF format dumpfiles,
which will replace the current xendump format in xen 3.0.5. The
new xen format uses ELF in a non-standard manner such that memory
contents are defined in section headers instead of the traditional
manner of using program headers. Testing has been on paravirtualized
x86, x86 PAE, x86_64 and ia64 dumpfiles. Fully-virtualized dumpfiles
have not been tested. (anderson(a)redhat.com)
- A number of "xencrash" (where the session is run against a xen-syms
binary) fixes have been applied:
1) "bt" did not switch from the ia64 MCA stack to the vcpu stack.
2) "bt" caused an infinite loop if ar_bspstore contained an illegal
value.
3) "bt" shows unnecessary unwind warning message. (ia64)
4) "man log" caused crash to fail with a segmentation violation.
5) "man log" did not have an example.
(oda(a)valinux.co.jp)
- Fix for "vtop" on x86 PAE kernels, which could abort upon reaching
the PTE translation section, showing the error message: "vtop:
cannot determine the swap location". (anderson(a)redhat.com)
- Fix for "vm -p" or "vtop" on 2.6 x86 PAE kernels, which could show
incorrect swap offsets, because the swap type/offset encoding was
moved to the high word of the 64-bit PTE. (anderson(a)redhat.com)
- Fix for "vm -p" on x86_64 kernels when a PTE referenced a swap
location, it would show "(not mapped)" instead of the swap location.
(anderson(a)redhat.com)
- In current 2.6 kernels, it is now possible to recognize ppc BOOKE
processors, which is the current default in crash. If the processor
is confirmed to not be BOOKE, then page table translation is done
differently. (antipov(a)ru.mvista.com)
- Fix for live system analysis of Ubuntu kernels due to a mismatch
between /proc/version and the linux_banner string. This was due
to an appendage to the linux_banner string in Ubuntu kernels.
(asid(a)hp.com)
- Fix for 2.6.21 kernels that fail during initialization with the
message: "crash: invalid (optional) structure member offsets:
zone_struct_free_pages or zone_free_pages". This was due to the
removal of the zone struct's "free_pages" member; instead the
zone's "vm_stat[NR_FREE_PAGES]" value is used. (anderson(a)redhat.com)
Download from: http://people.redhat.com/anderson
17 years, 7 months
Re: [Crash-utility] handling missing kdump pages in diskdump format
by Ken'ichi Ohmichi
Hi Vivek,
2007/03/06 12:02:46 +0530, Vivek Goyal <vgoyal(a)in.ibm.com> wrote:
>On Thu, Mar 01, 2007 at 09:15:05PM +0900, Ken'ichi Ohmichi wrote:
>>
>> (added Vivek Goyal)
>> (added redhat-kexec-kdump-ml)
>>
>> Hi everyone,
>>
>> If Bob's proposal (the end of this mail) is merged into makedumpfile,
>> the analysis behavior of ELF dumpfile is different from kdump-compressed
>> dumpfile's as follows.
>> When reading ELF dumpfile, the crash utility treats the excluded pages
>> as zero-filled pages. But, as to kdump-compressed dumpfile, the crash
>> utility will display the warning message to mean "These pages are
>> excluded by partial dump" when it accesses the excluded pages.
>>
>
>I quickly went through this thread. I don't know much about the diskdump
>format but from a layman's perspective, instead of keeping the zero pages
>and compressing them, why not extend diskdump format and maintain another
>bitmap which signifies the valid zero pages but they are not physically
>part of the core file? I think overall it might reduce dump size.
Thank you for the comment.
I think that it is unnecessary to extend diskdump format.
Instead of having all the compressed zero-pages, it is enough that
page descriptors of all zero-pages point same zero-page.
In this implement, the size of dumpfile increases by only one page
(4K, 16K, etc.). The attached patch is for it.
>Secondly, for ELF format core files, probably we can disable the behavior of
>excluding zero filled pages from dumpfile and then "crash" behavior can
>be consistent. (Return zero filled page only for valid data otherwise crib).
I want to use the feature of excluding zero-pages, because our systems
(x86_64) have many zero-pages immediately after system booting.
Bob is researching for the behavior of crash on ELF format dumpfiles.
I would like to wait for his report.
Thanks
Ken'ichi Ohmichi
17 years, 7 months
