handling missing kdump pages in diskdump format
by Bob Montgomery
I've been experimenting with the makedumpfile utility for kdump on ia64.
One of my experiments was to verify that a page that should have been
missing indeed was missing. I used crash 4.0-3.8 to look for a user
page that should have been omitted from the dump.
crash> x/xg 0xe0000040fc00c000
0xe0000040fc00c000: 0x0000000000000000
On a full dump from makedumpfile as well as on a straight copy of
vmcore, crash reports this:
crash> x/xg 0xe0000040fc00c000
0xe0000040fc00c000: 0x00010102464c457f
The dumpfiles created by makedumpfile appear to crash as diskdump files,
and crash appears to excuse missing pages and report 0x0 contents here:
diskdump.c:read_diskdump, line 454:
if (!page_is_dumpable(pfn)) {
memset(bufptr, 0, cnt);
return cnt;
Shouldn't there be some indication that a requested page is missing as
opposed to being legitimately full of zeros?
Bob Montgomery
17 years, 9 months
crash can not read ia64 lkcd v9 dump
by Olaf Hering
crash 4.0-3.9 can not read a dump file from an ia64 box running 2.6.5.
Is this supposed to work?
crash -s boot/System.map-2.6.5-7.267-sn2 boot/Kerntypes-2.6.5-7.267-sn2 dump.3
crash: diskdump: dump does not have panic dump header
crash: diskdump: dump does not have panic dump header
dump_header:
dh_magic_number: a8190173618f23ed (DUMP_MAGIC_NUMBER)
dh_version: 9 (LKCD_DUMP_V9)
dh_header_size: 742
dh_dump_level: 3 (DUMP_LEVEL_HEADER|DUMP_LEVEL_KERN)
dh_page_size: 16384
dh_memory_size: 20993310
dh_memory_start: e000000000000000
dh_memory_end: a8190173618f23ed
dh_num_pages: 336091
dh_panic_string: sysrq
dh_time: Mon Aug 21 08:32:07 2006
dh_utsname_sysname: Linux
dh_utsname_nodename: cognac
dh_utsname_release: 2.6.5-7.267-sn2
dh_utsname_version: #1 SMP Wed Jun 21 10:50:51 UTC 2006
dh_utsname_machine: ia64
dh_utsname_domainname:
dh_current_task: e00009b0287a8000
dh_dump_compress: 2 (DUMP_COMPRESS_GZIP)
dh_dump_flags: 80000004 ()
dh_dump_device: 0
crash: LKCD machine specific dump header doesn't match crash version: 3b200 761f8
crash: traceback of currently executing threads may not work
gdb boot/Kerntypes-2.6.5-7.267-sn2
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "ia64-unknown-linux-gnu"...
GETBUF(248 -> 0)
GETBUF(1500 -> 1)
please wait... (patching 20920 gdb minimal_symbol values) ^M ^M FREEBUF(1)
FREEBUF(0)
<readmem: a0000001005fa0d0, KVADDR, "kernel_config_data", 32768, (ROE), 60000000002e8310>
crash: seek error: kernel virtual address: a0000001005fa0d0 type: "kernel_config_data"
WARNING: cannot read kernel_config_data
<readmem: a000000100acea80, KVADDR, "xtime", 16, (FOE), 600000000006fc70>
crash: seek error: kernel virtual address: a000000100acea80 type: "xtime"
18 years
support for older gcc
by Daniel Li
It seems there is a regression in recent versions of crash which
disabled the fix put into crash 4.0-3.9 to deal with the fact older
version gcc was not providing enough debug info in vmlinux (see the
email threads on Oct 4 and Oct 5). Is there any plan to add the fix back in?
Thanks,
Daniel
18 years
xencrash-0.2 and crash version 4.0-3.13
by Dave Anderson
Hello Itsuro,
Please use the current crash utility version for all future
"xencrash" patches. As noted in the release announcement,
version 4.0-3.13 contains an adaptation of your most recent
patch. Please verify that the modifications that I made to your
patch are to your satisfaction.
I do note, using the vmcores I received today from Magnus Damm, which
were created using his latest xen kernel kdump patch and kexec-tools
patches, that running crash with the xen-syms file fails during initialization.
I was going to look into why, but I will leave that up to you.
In the meantime, I will work on getting crash to work with Magnus's new
NT_XEN_KDUMP_CR3 ELF note format for analyzing dom0 vmlinux dumps.
Thanks very much,
Dave
18 years
crash version 4.0-3.13 is available
by Dave Anderson
- Adapted the "xencrash-0.2" patch described here:
https://www.redhat.com/archives/crash-utility/2006-November/msg00036.html
This functionality consists of three inter-dependent parts, all of
while are still under development:
1) the kexec-tools user package
2) the kdump kernel patch for xen
3) the crash utility
The end result will be a single crash binary that can be used with
either the xen dom0 vmlinux kernel, or with the xen-syms hypervisor binary,
with the common vmcore created when either of those two entities crash.
(oda(a)valinux.co.jp, anderson(a)redhat.com)
- Fixed the initialization-time, and "sys" command, displays of the system
memory size when memory nodes have holes. Without this patch, more memory
than what is installed may be displayed. (anderson(a)redhat.com)
Download from: http://people.redhat.com/anderson
18 years
Crash shows - Invalid Memory size
by Sharyathi Nagesh
Hi
We encountered this problem with Crash showing invalid memory when ran
on live machine:
======================================
[root@venuslp11 ~]# free -m
total used free shared buffers cached
Mem: 2151 1594 557 0 583 795
^^^^^^
-/+ buffers/cache: 215 1935
Swap: 1983 0 1983
[root@venuslp11 ~]# cat /proc/ppc64/lparcfg | grep DesMem
DesMem=2304
^^^^^^
[root@venuslp11 ~]# rpm -q crash
crash-4.0-3.7
[root@venuslp11 ~]# crash
...
KERNEL: /usr/lib/debug/lib/modules/2.6.18-1.2732.el5/vmlinux
DUMPFILE: /dev/mem
CPUS: 2
DATE: Fri Oct 27 07:58:54 2006
UPTIME: 01:12:34
LOAD AVERAGE: 1.52, 1.05, 0.48
TASKS: 98
NODENAME: venuslp11.upt.austin.ibm.com
RELEASE: 2.6.18-1.2732.el5
VERSION: #1 SMP Tue Oct 17 18:24:27 EDT 2006
MACHINE: ppc64 (2301 Mhz)
MEMORY: 3.2 GB
^^^^^^^^
PID: 25097
COMMAND: "crash"
TASK: c000000000fedbe0 [THREAD_INFO: c00000000b190000]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
crash>
==================================
As I looked into the code I found:
The differences are observed because of the different way in which
they(proc and crash) are implemented to calculate Total Memory.
In /proc/meminfo it traverse through the memory counting each page and it has
different routines to calculate No of pages in highmem, init section, bootmem etc.
Which may be difficult to implement with Crash.
Instead we can look into sys file
implementation(/sys/devices/system/node/node<n>/meminfo). Here the Total Page is
got not from unsigned long node_spanned_pages but from long node_present_pages.
The definitions of node_present_pages says 'total number of physical pages'
while node_spanned_pages says 'total size of physical page range, including holes'.
This is observed because of way Node 2 is spread in the machine its
pfn(physical frame number) starts from 0 while that of 0th and 1st node
starts from 4096 and 8192 pfns respectively. so node3->spanned_pages has
double counted value from even the node 0 and node 1. Hence I feel its
better to use present_pages which has only the pages from the node
excluding the holes.
======================================
The patch to fix the problem:
Let me know of your opinion..
18 years, 1 month
Re: [Crash-utility] xencrash-0.2
by Dave Anderson
> Hi,
>
> Though I wish it is merged into crash finally,
> I understand it is early to merge at this time
> since kdump for xen hypervisor (and also dom0cut) is
> in still unstable-tree of xen development. So xencrash
> is usefull only for xen developers now.
>
> I think it is a good timing to merge when kdump for
> xen hypervisor become stable and a distribution such as
> RedHat adopt it.
>
> Thanks.
> Itsuro ODA
>
Hello Itsuro,
I understand that this facility can only be used in
a xen development environment. However, I prefer to
merge your patch as soon as possible.
As far as I can tell your work is mostly complete,
except for ia64 support. So I would presume that further
changes will be relatively small updates to your current
patch. Correct me if I am wrong.
While working with your patch, one of the modifications
I've done is to make it compile on architectures other than
x86 and x86_64. In order to do that, I plan to #ifdef the
hypervisor-only code such that it only gets compiled for
architectures that support it, since there's no reason to
have non-hypervisor architectures contain the hypervisor-only
code.
The manner in which you have structured the xen integration
into the crash utility is very safe, and remarkably clean.
So, unless you object, I will be merging it -- with a few
modifications that I'm confident that you will not find
objectionable -- and then your subsequent patches can be
made against the most recent crash version.
Thanks,
Dave
18 years, 1 month
RE: [Crash-utility] modules and data / bss initialization
by Castor Fu
Yup -- it's easy enough to work around when one knows
what one wants, but it's kind of painful. I have no idea why people
removed all this functionality from insmod. I suppose someone wants
to hide more info?
If nobody else has fixed this I'll either do it myself or get someone else to do it in
the next week or so.
-----Original Message-----
From: crash-utility-bounces(a)redhat.com [mailto:crash-utility-bounces@redhat.com]On Behalf Of Dave Anderson
Sent: Monday, November 20, 2006 2:04 PM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] modules and data / bss initialization
Castor Fu wrote:
Under linux 2.6 it seems that crash is not figuring out the locations
of any sections of modules other than the text section.
This means that things like 'sym modvar1' can find the correct location
but 'p modvar1' does not work.
I guess that there are a couple of possibilities here:
1. if CONFIG_KALLSYMS is set, one could look through the sect_attrs
for each of the sections and initialize the segments.
2. if CONFIG_KALLSYMS is not set, we would have to try to match
offsets of some variables from the object file with the existing offsets
in the kernel.
Clearly [2] is the 'better' solution because it doesn't
require CONFIG_KALLSYMS to be set, but it seems like more work.
Has anyone done either of these yet? If not does it seem like I've described the
problem correctly?
Thanks,
castor
It's been years since that code has been tinkered with,
and since I don't debug much module code, I wasn't even
aware that there was a 2.6-only difference. I thought that
the add-symbol-file operation would properly assign the
module's msymbols with the correct addresses for data
symbols, which it does do correctly for the text symbols.
(I think that's the issue with the "p" command, i.e., the
embedded gdb is still using the offset value from
the object file and failing...)
Anyway, you can still do a "whatis modvar1", get the data type,
and then either do a "struct" command with the address, or
come up with a cobbled-together gdb print command, right?
Dave
18 years, 1 month
modules and data / bss initialization
by Castor Fu
Under linux 2.6 it seems that crash is not figuring out the locations
of any sections of modules other than the text section.
This means that things like 'sym modvar1' can find the correct location
but 'p modvar1' does not work.
I guess that there are a couple of possibilities here:
1. if CONFIG_KALLSYMS is set, one could look through the sect_attrs
for each of the sections and initialize the segments.
2. if CONFIG_KALLSYMS is not set, we would have to try to match
offsets of some variables from the object file with the existing offsets
in the kernel.
Clearly [2] is the 'better' solution because it doesn't
require CONFIG_KALLSYMS to be set, but it seems like more work.
Has anyone done either of these yet? If not does it seem like I've described the
problem correctly?
Thanks,
castor
18 years, 1 month
xencrash-0.2
by Itsuro ODA
Hi,
This is an update of crash analisys tool for Xen hypervisor.
(posted 2006/10/30)
The main enhance is x86_64 support and minor changes are
the following:
* now command name is just "crash"
It recognizes as Xen hypervisor analysis if
- the name of namelist starts "xen-syms", or
- --hyper option specified
* BT_XEN_HYPER_MODE flag is removed
* checking by "make warn" done
Note: This is still based on crash-4.0-3.8
To use xencrash:
- get crash-4.0-3.8 from
http://people.redhat.com/anderson/crash-4.0-3.8.tar.gz
- tar zxf crash-4.0-3.8.tar.gz
- cd crash-4.0-3.8
- zcat xencrash-4.0-3.8-0.2.patch.gz | patch -p1
- make
usage: crash xen-syms dump-file
where dump-file is either
- vmcore (whole memory dump image) get by kdump, or
- xendump (the part of xen hypervisor) cut by dom0cut
support architecture is x86 and x86_64.
The main direction of our development will be IA-64 support.
Thanks.
--
Itsuro ODA <oda(a)valinux.co.jp>
18 years, 1 month
