function arguments
by Shahar Luxenburg
Hi,
I'm using crash utility on RHEL 5 on x86. According to the TODO list 'bt'
command should show function arguments: "Display arguments in the stack
trace. At present, we do not have support for PPC64 and x86_64".
Unfortunately, no function arguments are presented using 'bt' command. Am I
missing something?
If that's the case, I'd like to add that feature (for x86 architecture at
first). Code from kdb can be ported to the crash utility for that purpose.
kdb is analyzing the code using basic blocks, understanding where the
registers are being saved (either to other registers or memory location) and
if their values can be retrieved, it prints them. Can I use debug
information for that purpose? I can look at the eh_frame section to
understand where ebx, edi, esi and ebp are saved but I can not find
information, for each frame, what is the fate of eax, edx and ecx - If I'd
know that, then either they were saved in memory (stack location) - it's
easy to get them, or they were saved in another register, than I can use the
debug information to understand if they were saved on the stack at some
'younger' frame.
Any hint will be welcomed.
Regards,
Shahar.
18 years
crash and libvirt, and more
by Richard W.M. Jones
Hi:
I don't know if you're aware of this, but libvirt[1] recently added a
call which allows you to snoop on the live memory of guests,
virDomainMemoryPeek[2]:
int virDomainMemoryPeek (virDomainPtr dom,
unsigned long long start, /* start address */
size_t size, /* size (bytes) */
void * buffer, /* return buffer */
unsigned int flags);
This would allow, in theory, for crash to debug running guests. I had
a look at the crash code and it doesn't seem like it would be too hard
to add this.
We [the libvirt team] only support this for QEMU & KVM guests at the
moment, but we plan to support this call for Xen in the near future.
Also, the call only works on virtual memory addresses (in other words,
the address is translated through the guest's page tables), but in
practice that isn't too bad because the common configuration for Linux
is to map all of physical memory at some address, eg. 0xc0000000 on
i386. Also the peek operation is read-only.
So if you are interested, let me know, and I will attempt a patch.
- - -
Now, the bigger picture ...
For some months now we've been attempting to write system
administrator tools to mimic common sysadmin commands, except that
they work on guests. For example 'virt-ps <guest>' lists out the
process table in <guest>. It runs from the host and works by snooping
guest memory using virDomainMemoryPeek.
We have had some success, although it's been quite a lot harder than
we imagined it would be. At the moment we have 'virt-dmesg',
'virt-uname', 'virt-ifconfig' and 'virt-ps', plus a handful of custom
commands, working to a greater or lesser extent.
However I wasn't aware before that crash could already do this
(particularly 'log', 'ps', 'mount' and 'net' commands), and in fact
crash has a lot more complete support for these commands than we do.
So it makes sense to use crash to do this, instead of continuing with
our separate implementation, if we can make it work.
I think there are two things that we'd need to add to crash in order
to get this working:
(i) Scripting. I'm aware that there are two scripting projects for
crash out there already, but it looked fairly immature and/or
unsupported. However, not too hard to pull these projects up to
standard and/or add some scripting support, or use expect.
(ii) Getting the debug symbols.
Item (ii) is the big deal for us. Our current virt-* tools can work
with a wide range of kernels.
What we do is to download the kernel-debuginfo packages beforehand,
extract only the tiny amount of debug info we actually need from
vmlinux, and build a 'kernel database'. (We're using dwarves to get
the layout of the dozen or so structures that we care about). It
turns out that it's quite easy to heuristically determine the version
of a running kernel, and from that we can look up the structures in
the kernel database at runtime.
Upshot is that we support currently ~ 350 kernels with a database
which is a modest 1 MB in size, and probably could be made smaller
with very little effort.
The problem I haven't yet resolved with using crash is that we need a
matching, identical vmlinux image (ie. 50-100 MB) per guest kernel
version. In the case where we see a kernel version we've not seen
before, we may have to download this and store it somewhere.
The alternative seems to involve some really deep hacking inside gdb,
perhaps so it can be persuaded to use only partial debug info?
I don't know if you have any thoughts about (ii).
Rich.
[1] http://libvirt.org/
[2] http://libvirt.org/html/libvirt-libvirt.html#virDomainMemoryPeek
--
Richard Jones, Emerging Technologies, Red Hat http://et.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/
18 years
crash version 4.0-7.1 is available
by Dave Anderson
- Fix to address RT kernel's renaming of the address_space.nrpages
member to address_space.__nrpages. Without the patch, "kmem -i"
would fail with the error message "kmem: invalid structure member
offset: address_space_nrpages". (bwalle(a)suse.de)
- For crash utility debug backtraces displayed in error conditions,
the usage of __builtin_return_address() has been replaced with the
backtrace() function. This prevents crashes if the Makefile is
modified to compile with -O2. (bwalle(a)suse.de, anderson(a)redhat.com)
- Fix for ia64 hypervisor backtraces when the entries in the cpu map
are not contiguous. (takebe_akio(a)jp.fujitsu.com)
- Fix to make shell-escaped commands in a crash input file direct
their output properly. Without the patch, if the output of an
input file was redirected to a file or pipe, the output of any
shell-escaped commands in the input file only went to stdout.
(anderson(a)redhat.com)
- Fix to allow the usage of the "-i inputfile" command line option
when operating from an init script. Without the patch, the crash
session would fail during initialization with the error message:
"crash: /dev/tty: No such device or address". (anderson(a)redhat.com)
- Fix for the "kmem -P <address>" option, where <address> is an
invalid physical address. Without the patch, the command causes
a segmentation violation on an ia64; on other architectures an
unnecessary mem_map header is displayed prior to the error message.
(wency(a)cn.fujitso.com)
- Fix for a potential endless cascade of SIGFPE exceptions during
session initialization when a vmlinux and vmcore do not match,
and a correct System.map or a non-debug vmlinux file is not supplied.
Doing that is is allowable, but is certainly not recommended. In
this case, and incorrect kernel HZ value of 0 was calculated and used
for the initial "UPTIME:" display. (anderson(a)redhat.com)
- More gracefully handle a nonsensical "search -u <address>" command
attempt on a kernel thread or any context with no user address space.
Without the patch, the error message was related to a failed user
virtual address translation attempt; with the patch it now indicates:
"search: current context has no user address space".
(anderson(a)redhat.com)
- Reworked the "search" command for usage with the Xen Hypervisor.
When attempted on a Xen hypervisor, and depending upon the arguments
used, a segmentation violation, a nonsensical error message, or if
neither of the aforementioned, the command could appear to work but
not necessarily find the search target value even though it was there
in the specified memory range. To address the various shortcomings,
the following restrictions have been put in place for usage with the
Xen hypervisor:
(1) A starting virtual address must be supplied either symbolically
or with the "-s <address>" option.
(2) The (nonsensical) "-u" option is no longer accepted.
(3) The "-k" option is no longer accepted.
(4) When cycling through virtual memory, as soon as an address
cannot be read, the search will be quietly suspended. To
determine where the search was suspended, enter "set debug 1",
and then re-run the command.
(anderson(a)redhat.com)
- Fix for initialization-time segmentation violation due to a module
allocating and creating an exported symbol list outside of its own
virtual address space, and then overwriting its own symbol list
pointer. (anderson(a)redhat.com)
- Implementation of a "--minimal" command line option, which brings
up a crash session that is restricted to the "log", "dis", "rd",
"sym", "eval" and "exit" commands. This option may provide a way to
extract some minimal/quick information from a corrupted or truncated
dumpfile, or in situations where one of the several kernel subsystem
initialization routines, which are not called, would abort the
crash session. (sharyath(a)in.ibm.com, sachinp(a)in.ibm.com)
Download from: http://people.redhat.com/anderson
18 years
[RFC] Provide minimal debugging to crash
by Sharyathi Nagesh
Hi
Sachin and myself have worked on this patch. In case the memory dump is
corrupted crash tool exits from execution and we don't have any means to
debug the problem. This feature provides minimalistic features to crash
and help in debugging even if dump is corrupted.
Description of the patch:
This patch will provide the minimal functionality in crash tool.
This patch has been generated against crash-4.0.6.2.
This feature will read the dump, even if it is corrupted. It
provides minimal debugging features, as provided by lcrash tool earlier.
It provides access to dmesg, symbol information, disassembly and memory
read features.
You can get into minimal mode by providing "--minimal" option while
starting crash
eg: crash --minimal <vmlinux> <dump>
========================================================================
Testing:
This patch has been tested on the corrupted dump images on z-series
machine.
'help' command in the minimal mode gives out put in a line, not
like the way out put is generated in the normal mode of operation.
Please go through the features and let us know your thoughts
========================================================================
Thanks
Yeehaw
18 years
cannot resolve: "end_pfn" with recent rawhide kernel
by Jeff Layton
Attempting to open a core from a recent x86_64 rawhide kernel and am
getting this error:
crash: cannot resolve: "end_pfn"
kernel 2.6.27-0.244.rc2.git1.fc10.x86_64
crash-4.0-7.x86_64
The host is a FV xen guest and the core was generated with "xm
dump-core". Let me know if you need other info (or if I should open a
BZ).
Thanks,
--
Jeff Layton <jlayton(a)redhat.com>
18 years
Re: Crash-utility Digest, Vol 35, Issue 6
by Durga Prasad
>
> > Hi Dave,
> > I attempted to try your suggested commands, but it tends to fail on my
> > machine:
> >
> >
> > libl041:/var/log/dump # crash -s vmlinux.sles10sp2.smp.x86_64
> > 2008-08-08-23\:27/vmcore
> > crash> struct ext3_xattr_search
> > struct: invalid data structure reference: ext3_xattr_search
> > crash> mod -s ext3
> > MODULE NAME SIZE OBJECT FILE
> > ffffffff8856a500 ext3 153360
> > /lib/modules/2.6.16.60-0.21-smp/kernel/fs/ext3/ext3.ko
> > crash> struct ext3_xattr_search
> > struct: invalid data structure reference: ext3_xattr_search
> > crash> struct ext3_xattr_info
> > struct: invalid data structure reference: ext3_xattr_info
> >
> >
> > crash> sys
> > KERNEL: vmlinux.sles10sp2.smp.x86_64
> > DUMPFILE: 2008-08-08-23:27/vmcore
> > CPUS: 4
> > DATE: Fri Aug 8 23:20:09 2008
> > UPTIME: 01:31:48
> > LOAD AVERAGE: 2.48, 1.32, 0.56
> > TASKS: 157
> > NODENAME: libl041
> > RELEASE: 2.6.16.60-0.21-smp
> > VERSION: #1 SMP Tue May 6 12:41:02 UTC 2008
> > MACHINE: x86_64 (1995 Mhz)
> > MEMORY: 3.9 GB
> > PANIC: "Oops: 0000 [1] SMP " (check log for details)
> >
> > crash version: 4.0-4.10 gdb version: 6.1
> >
> > I tried the same after compiling and installing the latest crash version
> > (crash version: 4.0-6.3 gdb version: 6.1) with no difference in
> results.
> >
> > What would you suggest?
> >
> > Thanks
> > Durga
>
> The first thing to verify is whether those two structures existed
> in the 2.6.16-era timeframe?
>
> Dave
>
> The 2.6.16.60-021 kernel DO have the said structures.
Regards
18 years
Re: Xen Hypervisor analysis
by Itsuro ODA
Hi,
On Fri, Aug 08, 2008 at 11:15:17AM -0400, Dave Anderson wrote:
>
> Gentlemen,
>
> Our QA guys have started spending some time with Xen hypervisor
> crash analysis, and one obvious shortcoming is the implementation
> of the "search" command.
>
> I don't see how it ever could have worked correctly, but in
> its current state, it either causes segmentation violations,
> or quietly runs but doesn't find anything.
I'm sorry. I have missed testing it actually.
> The primary problem is that the search command code is very
> linux-specific, and so the search command or command-options
> typically go off into the weeds trying to handle the addresses
> as linux virtual addresses.
>
> Here's what I plan to do for the Xen hypervisor search command:
>
> 1. Disallow the "-u" option, which is nonsensical.
> 2. Disallow the "-k" option, because of the various identity-mapped
> and virtual address ranges used by the various architectures
> is more than I care to deal with.
> 3. Do not translate the virtual addresses to physical before
> reading the memory, i.e., just keep the addresses virtual.
> 4. Require either a "-s <address>" starting address or a Xen hypervisor
> symbol as the starting address.
> 5. While cycling through memory, as soon as a page is encountered
> that cannot be read, the search will be stopped.
It sounds reasonable.
> I'm hoping to get a new release out shortly, and you can see what
> I plan to do. If you guys have a desire to extend the capability
> beyond 1-5 above, send me patches.
>
> Thanks,
> Dave Anderson
Thanks.
--
Itsuro ODA <oda(a)valinux.co.jp>
18 years
Viewing module symbols using crash
by Durga Prasad
Hi,
[1]:
I need to debug a kernel module (a device driver).
I added the desired module to crash using mod -S <path to module>. The
driver is compiled with -g option.
I could not see the structures defined within the module.
Am I missing something?
System info:
KERNEL: vmlinux.sles10sp2.smp.x86_64
DUMPFILE: 2008-08-08-18:57/vmcore
CPUS: 4
RELEASE: 2.6.16.60-0.21-smp
VERSION: #1 SMP Tue May 6 12:41:02 UTC 2008
MACHINE: x86_64 (1995 Mhz)
MEMORY: 3.9 GB
[2]:
I could get into live kernel debugging using crash on RHEL 5 after
installing relevant kernel-debug packages. How do I do the same on SLES 10?
As of now, I have been compiling the kernel with CONFIG_DEBUG_INFO set. I
can use the resulting vmlinux for post crash analysis, but not for live
analysis. Any light on how to accomplish this would be appreciated.
Thanks
Durga Prasad
18 years
Kdump compressed format
by Bernhard Walle
Hi,
is there any documentation how the kdump compressed format (or the
diskdump format, from what it is derived) looks like? Beside of the
source code of the tools that support it. ;-)
Thanks for any help!
Bernhard
--
Bernhard Walle, SUSE LINUX Products GmbH, Architecture Development
18 years
Xen Hypervisor analysis
by Dave Anderson
Gentlemen,
Our QA guys have started spending some time with Xen hypervisor
crash analysis, and one obvious shortcoming is the implementation
of the "search" command.
I don't see how it ever could have worked correctly, but in
its current state, it either causes segmentation violations,
or quietly runs but doesn't find anything.
The primary problem is that the search command code is very
linux-specific, and so the search command or command-options
typically go off into the weeds trying to handle the addresses
as linux virtual addresses.
Here's what I plan to do for the Xen hypervisor search command:
1. Disallow the "-u" option, which is nonsensical.
2. Disallow the "-k" option, because of the various identity-mapped
and virtual address ranges used by the various architectures
is more than I care to deal with.
3. Do not translate the virtual addresses to physical before
reading the memory, i.e., just keep the addresses virtual.
4. Require either a "-s <address>" starting address or a Xen hypervisor
symbol as the starting address.
5. While cycling through memory, as soon as a page is encountered
that cannot be read, the search will be stopped.
I'm hoping to get a new release out shortly, and you can see what
I plan to do. If you guys have a desire to extend the capability
beyond 1-5 above, send me patches.
Thanks,
Dave Anderson
18 years
