Crash physical search on live session not recommended :-)
by Bob Montgomery
While testing my search patch, I kicked off an unconstrained physical
search on a live session and hung the machine so thoroughly that it
required a visit to the machine room to physically unplug it to get the
remote console back up. Coincidence? Or should physical address search
on a live session be constrained somehow for safety?
Bob Montgomery
13 years, 9 months
[PATCH] Expanded search capability for crash
by Bob Montgomery
This patch adds options to crash's search command that allow searches
for character strings (-c), unsigned hexadecimal integer values (-w),
and unsigned hexadecimal short values (-h). The integer and short
values are searched on integer and short alignments respectively.
Strings are searched across contiguous page boundaries, which can cause
possibly confusing behavior since contiguity is affected by the type of
search. For example, a page-crossing string found in a user virtual
address search might not be found in a kernel virtual address search if
the kernel pages mapped to the user address space are not contiguous.
The patch is for crash-5.1.2 and has been tested on an x86_64 system.
Some last minute caveats on with search due to the parser (worth putting
in help?):
All numbers are interpreted as hex numbers, regardless of starting
character.
Double quotes are magic to crash's parser, so strings of almost anything
can be enclosed in "", for example: "can't".
Single quotes mean nothing, so searching for 'dog' is the same as
searching for "'dog'".
Bob Montgomery
13 years, 9 months
problem with crash utility in ubuntu
by Amer Aljaedi
Hi ,
I have tried to use crash 4.1.0 in ubuntu 10.04 kernel (2.6.32.25), when I
don't specify the memory dump file ,it works perfectly as it reads the
memory from crash driver (crash model), it shows below:
root@o:/# crash /boot/System.map-2.6.32-25-generic
/home/amer/ubuntu-lucid/debian/build/build-generic/vmlinux
crash 4.1.0
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
SYSTEM MAP: /boot/System.map-2.6.32-25-generic
DEBUG KERNEL: /home/amer/ubuntu-lucid/debian/build/build-generic/vmlinux
(2.6.32-25-generic)
DUMPFILE: */dev/crash
* CPUS: 1
DATE: Wed Feb 23 11:40:40 2011
UPTIME: 09:17:34
LOAD AVERAGE: 0.89, 0.74, 0.78
TASKS: 258
NODENAME: o
RELEASE: 2.6.32-25-generic
VERSION: #45-Ubuntu SMP Sat Oct 16 19:48:22 UTC 2010
MACHINE: i686 (1339 Mhz)
MEMORY: 511.5 MB
PID: 2703
COMMAND: "crash"
TASK: cf27bfc0 [THREAD_INFO: dfbc0000]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
* but when I entered the dump file argument I got error (the memory image
is obtained from /dev/crash by dd tool), it shows below: *
root@o:/# crash /boot/System.map-2.6.32-25-generic
/home/amer/ubuntu-lucid/debian/build/build-generic/vmlinux
home/amer/image.dd
crash 4.1.0
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
*crash: home/amer/image.dd: not a supported file format
*
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist]
[dumpfile]
Enter "crash -h" for details.
could you please give some advice to resolve this problem.
Thanks,
amer
13 years, 9 months
Re: [Crash-utility] Heads up Linux 2.6.38-rc4 compile problems.
by Mike Snitzer
Hey Eric,
On Mon, Feb 14, 2011 at 12:34 AM, Eric W. Biederman
<ebiederm(a)xmission.com> wrote:
>
> And for completeness. When I was rebooting v2.6.38-rc4 to start running
> 795abaf1e4e188c4171e3cd3dbb11a9fcacaf505 I hit this.
>
> Sigh. I wish crash worked on something besides redhats enterprise
> kernels. Then I could use the system core file I have to do more than
> extract the dmesg.
Then you should cc crash-utility(a)redhat.com (now cc'd) and work with
Dave Anderson (e.g. get him your vmlinux and core files, which version
of crash you're using and how it fails). Dave does an amazing job of
working through crash issues which are reported against upstream
kernels -- the key first step is the report.
Mike
13 years, 9 months
How to work with crash utility.
by nishant mungse
Hello all,
I am new to this crash utility and have installed crash-5.1.1 on fedora11.
Actually i searched a lot on net but i am unable to understand that. I tried
many thing but it failed.
I have written a module and it is giving oops after mke2fs and wann to use
crash utility to solve this problem. Will anyone guide from basic steps how
to use crash. Please help me out.
Regards,
Nishant.
13 years, 9 months
Re: [Crash-utility] RFC: string search for crash
by Dave Anderson
----- Original Message -----
> On Wed, 2011-02-02 at 14:49 +0000, Dave Anderson wrote:
> >
> > ----- Original Message -----
> > > Here is a patch to add string search to crash-5.1.1.
> > >
> > > It requires the my previous patch for the parse_line routine.
> > >
> > > It searches for the specified strings, and for half or more of the
> > > string appearing at the start and end of search blocks (usually pages)
> > > in case the string spans a page boundary.
> > >
>
> > First, this is a really nifty feature...
>
> Thanks.
>
> > Second, I appreciate that you created a new string_search() function
> > instead of attempting to merge it into the existing search()
> > function.
>
> Way too ugly otherwise.
>
> >
> > Third, given that you're searching for specific string, why not show
> > the actual byte-aligned address as the starting point, and then just
> > display the string contents until the next non-ASCII character, or some
> > other delimiter like a 48 byte limit or whatever?
>
> That's actually the easier thing to do and just a preference decision.
> I like looking at aligned addresses for using with subsequent rd
> commands. And it's also sometimes useful to see some pre-string context.
> But rounding to the previous long aligned address is really random with
> respect to pre-string context. There's probably a better way to provide
> that.
Actually, rounding back down to the previous long-aligned address might
be reasonable.
>
> > I understand that the page-crossing issue is a PITA, but with respect
> > to a user searching memory for strings, the page-crossing issue is
> > seemingly irrelevant. In other words, why this:
> >
> > > ffff880125f72000: ger_event.......................................
> >
> > instead of displaying something like this:
> >
> > ffff880125f71ffd: danger_event
> >
> > So I guess I'm just wondering why the "dan" at the end of the page
> > cannot be displayed? It just doesn't seem user-friendly to force
> > the user to understand the half-of-the-string-at-a-page-boundary
> > business. Since you do recognize that string exists and crosses
> > a page boundary, shouldn't you be able to display the first part?
>
> I started working on my search stuff directly on dumpfiles, which meant
> I was searching through a collection of physical pages. When searching
> physical pages, there is no reason to believe that the end of page N has
> anything logically to do with the beginning of page N+1. So looking
> across to the next page for the continuation of a string didn't seem
> strictly correct or useful. (This was never even a consideration when
> searching for a long or int; they don't cross page boundaries.)
Right, certainly it doesn't make sense with physical pages, and to a
somewhat lesser extent, with unity-mapped pages. But with user virtual,
the mapped kernel virtual region (x86_64 and ia64), and vmalloc addresses,
strings would cross page boundaries. So I wonder whether it's worth the
effort to do something like I suggest since you always know what kind of
memory is being searched? I know, I know, it's easy for *me* to say...
But that leads to another issue. As I mentioned before, the -p
implementation I've got uses its own function, so to marry that
with the string search capability kind of mucks with your patch.
I'm wondering whether your string search function could be called
from both the virtual and physical search functions near the bottom
where the page is searched? Just something to think about.
>
> If I'm searching by virtual address, then N+1 does follow N, but I don't
> know that at the time I'm working on page N. The current search
> strategy is "choose a page, load the page, search the page". So if I
> find a promising first part of a string at the end of a page, the search
> loop would need a memory to carry my position in the string(s) across to
> the next page, if I can determine that the next page follows
> contiguously in the current search type. Not impossible, but not
> trivial, I think. Possibly easier to pre-read the following page to the
> length of the longest search string onto the end of an extended page
> buffer and then search for the full string to the end of that extended
> page buffer before loading the next page (and part of the one after
> that). But you still have to deal with the possiblity that the "next"
> page isn't contiguous to the one you're working on.
Right, perhaps the "last-page-contents-and-its-address" could always be saved?
And you could always attempt to read the next page if you see the beginning
of a search string.
>
> When you called out the example above, a first-of-page hit on
> "ger_event" when searching for the string "danger", my first thought was
> that you had found a bug. The "dan" at the end of the previous page
> should also have met the criteria for half of the string "danger", and
> showed up as an end-of-page hit. But (whew!), here's what's in memory
> there:
>
> crash-5.1.1str> rd ffff880125f71ff0 8
> ffff880125f71ff0: 000000000ed7ab73 676972745f64656c s.......led_trig
> ffff880125f72000: 6e6576655f726567 0000000000000074 ger_event.......
> ffff880125f72010: 0000000000000000 0000000000000000 ................
> ffff880125f72020: 0000000000000000 0000000000000000 ................
>
> Which serves as a reminder that the "part of a string" match is just
> there to suggest possibilities that might turn out to be false
> positives.
>
> > Lastly, I'm still planning to add the remaining "search" command
> > updates for the -KVM flags and the "missing" x86_64 segment bug,
> > and so I may need you to re-work this patch to fit into an updated
> > version of search(). I've been delayed getting crash-5.1.2 out
> > the door because of all of the recent patch postings, and it may
> > be worth waiting to get this piece in until 5.1.3 -- is that OK
> > with you?
>
> No rush.
OK good.
Thanks,
Dave
13 years, 9 months
[ANNOUNCE] crash version 5.1.2 is available
by Dave Anderson
- Added the /dev/crash memory driver Makefile and source file to
the crash package for live system analysis when the target
system's kernel was configured with CONFIG_STRICT_DEVMEM, or was
not configured with CONFIG_PROC_KCORE, or whose /proc/kcore is
simply not functional. The driver can be built and installed by
entering the memory_driver subdirectory and entering "make", and
then "insmod crash.ko". If the module is successfully installed,
it will be used by default for live crash sessions.
(anderson(a)redhat.com)
- Fix for the "extend -u" command option. Without the patch, after
the successful unloading of a crash extension module, there may be
an invalid error message that indicates "extend: <module>.so and
<module>.so are different".
(anderson(a)redhat.com)
- Implement support for Xen version 4 hypervisor dumpfiles:
1. Accept the "__per_cpu_shift" symbol. Without the patch, crash
initialization fails on X86, X86_64 and IA64 dumpfiles.
2. Make the x86_64 XEN_VIRT_START value a variable dependent upon
the Xen version. Without the patch, crash initialization fails
on x86_64 dumpfiles.
3. In Xen version 4, "init_tss" is a per-cpu symbol. Without this
patch, crash fails during initialization with the error message
"crash: cannot resolve init_tss" on X86 and X86_64 dumpfiles.
4. Each domain can have a different number of max VCPUs in Xen
version 4. Prepare for this by converting the static array into
a dynamic one.
5. If the size of the vcpu array in struct domain is known, use it
to size the dynamically allocated vcpu array in crash. This
enables crash to initialize domains with a different number of
VCPUs than specified by the XEN_HYPER_MAX_VIRT_CPUS macro.
6. The "vcpu" field changed from a fixed array to a pointer to an
array. The size of the array is stored in the (newly introduced)
"max_vcpus" field. Modify xen_hyper_store_domain_context() to
account for this change.
7. The command line options (such as opt_sched) are discarded after
boot in Xen version 4, so they are no longer available. Use the
"ops" variable (if it exists) to determine the active scheduler.
Without this patch, crash fails during initialization with the
error message "crash: schedule data not found".
(ptesarik(a)suse.cz)
- Created a new extension module API:
int load_module_symbols_helper(char *module);
It takes a kernel module name as an argument, and performs the same
procedure as the command "mod -s <module>".
(nakayama.ts(a)ncos.nec.co.jp, anderson(a)redhat.com)
- When loading debuginfo data for kernel module object files with the
"mod" command or with the new load_module_symbols_helper() extension
module API, search a non-standard directory path by specifying the
directory tree in the CRASH_MODULE_PATH shell environment variable.
(nakayama.ts(a)ncos.nec.co.jp, anderson(a)redhat.com)
- Created a new extension module API:
int get_kernel_config(char *conf_name, char **str);
The kernel must be configured with CONFIG_IKCONFIG. It takes a
kernel configuration item string, which may be of the format
"CONFIG_XXX" or just "XXX", and a pointer to a char *. The function
returns:
IKCONFIG_Y: configuration item is built into the kernel
IKCONFIG_M: configuration item is part of a kernel module
IKCONFIG_STR: configuration item consists of a string
IKCONFIG_N: the kernel is not configured with CONFIG_IKCONFIG or
the configuration item is not configured.
If "Y", "M" or "STR", the configuration item's string representation
will be pointed to by the passed-in "str" pointer.
(nakayama.ts(a)ncos.nec.co.jp, anderson(a)redhat.com)
- Update of the "extensions/trace.c" extension module. Initially
designed to support 2.6.32 (RHEL6), it has been updated to support
kernels up to 2.6.38-rc1.
(laijs(a)cn.fujitsu.com)
- Fix for the internal parse_line() function to properly handle
multiple string arguments. Without the patch, the second, fourth,
etc., string argument would get broken up into individual tokens.
(bob.montgomery(a)hp.com)
- Updates to support ARM page table changes and PTE differences that
were introduced in 2.6.38 kernels. Deleted two irrelevant ARM-only
source code comments.
(ext-mika.1.westerberg(a)nokia.com)
- Removed two unused "struct syment" local variable declarations in
the gdb_add_symbol_file() function in gdb-7.0/gdb/symtab.c.
(ptesarik(a)suse.cz)
- Replaced the usage of the VOID_PTR() macro with the ULONG() macro for
structure member accesses in vm_area_dump(), vm_area_page_dump() and
next_upage().
(ptesarik(a)suse.cz)
- Implemented support for makedumpfile's "vmcore.flat" dumpfile format.
It is no longer necessary to revert the flat dumpfile back into an
ELF vmcore or compressed kdump vmcore with "makedumpfile -R", or with
the "makedumpfile-R.pl" script. Without the patch, attempting to use
a flat dumpfile fails with the message "crash: vmcore.flat: not a
supported file format".
(oomichi(a)mxs.nes.nec.co.jp)
- Moved the GDB_CONF_FLAGS logic from the Makefile into configure.c.
(ptesarik(a)suse.cz)
- Use memset() in the shift_string_right() utility function, and use
shift_string_right() and memset() in the mkstring() utility function.
(ptesarik(a)suse.cz)
- Added a new "search -p" option to search physical memory. Until now
only kernel virtual memory and user virtual memory of the current
context could be searched.
(anderson(a)redhat.com)
- Optimization of the "search -k" function resulting in a significant
time reduction when cycling through vmalloc memory space.
(anderson(a)redhat.com)
- Added a new "search -K" option, which searches a subset of kernel
virtual memory, restricting the search to kernel unity-mapped virtual
memory, and on the x86_64 and ia64 machines, their discretely mapped
kernel-text/static-data regions.
(anderson(a)redhat.com)
- Added a new "search -V" option, which searches a subset of kernel
virtual memory, restricting the search to kernel virtual memory
that was allocated by vmalloc(), kernel module memory, and the
virtual mem_map region if it exists on x86_64, ia64, ppc64 and
s390x machines.
(anderson(a)redhat.com)
- Reworked the functionality of the "search" command to break up kernel
virtual memory into machine-dependent address regions so that the
cycling through disparate virtual address regions can be done in a
more efficient manner.
(anderson(a)redhat.com)
- Fix for "search -k" on x86_64. Early 2.6 kernels did not have their
module virtual address space included in the kernel's "vmlist" array;
without the patch, module memory is not searched.
(anderson(a)redhat.com)
- Fix for "search -k" on x86_64. Without the patch, the kernel's
mapped kernel text/static-data region is not searched as a separate
region.
(anderson(a)redhat.com)
- Fix for "search -k" on architectures that may have a virtual memmap,
such as x86_64, ia64, ppc64 and s390x. Without the patch, virtual
memmap array memory is not searched.
(anderson(a)redhat.com)
- Fix for "search -k" on s390x. Without the patch, the command may
end prematurely with the error message "search: read error: kernel
virtual address: 0 type: entry".
(anderson(a)redhat.com)
- Fix for "search" command to prevent unnecessary read error messages
for memory pages that are not contained in a xendump dumpfile.
Without the patch, the output may be interspersed with messages that
indicate "search: cannot find mfn in page index".
(anderson(a)redhat.com)
- Fix for a compile failure of the embedded gdb-7.0/bfd/verilog.c file
due to a smarter -Werror flag used by gcc-4.6.
(anderson(a)redhat.com)
Download from: http://people.redhat.com/anderson
13 years, 9 months
How do debug the kdump kernel using a Xen dump-core file?
by Olaf Hering
Hello,
'xm dump-core -L <domain>' generates a coredump of a Xen4 HVM guest, and
crash is able to poke around in this coredump.
Thanks for that.
But: How can I use crash to debug the crash/kdump kernel itself?
I did a 'echo c > /proc/sysrq-trigger' and want to look at the crash
kernel, not the normal kernel.
Olaf
13 years, 9 months
[PATCH] Fix segmentation violation in symbol_search
by Petr Tesarik
Subject: Fix segmentation violation in symbol_search
Fix a possible segmentation violation in crash if a module name
is not NUL-terminated. Although store_module_symbols_v2 complains
about an overly long module name, there are several problems
with the current approach:
1. The maximum size is hard-wired in defs.h and the current
constant doesn't even match struct module's name field size
on any architecture.
2. If the string is too long, it is probably not NUL-terminated,
so we can't use strlen() on it.
3. Even though only the first MAX_MOD_NAME-1 bytes are copied
to struct load_module, the _MODULE_* pseudo-symbol names are
generated from the unabridged module name. As a consequence,
they are not found further on in the loop at the end of
store_module_symbols_v2, so lm->mod_symtable remains NULL
for that module. The symbol_search() function is not
prepared for that situation and tries to dereference that
NULL pointer here:
sp = lm->mod_symtable;
sp_end = lm->mod_symend;
for ( ; sp <= sp_end; sp++) {
if (!pseudos && MODULE_PSEUDO_SYMBOL(sp))
^^^^
Regards,
Petr Tesarik
13 years, 9 months
[PATCH 2/2] Fix ZERO_FILL flag to mkstring()
by Petr Tesarik
The ZERO_FILL flag should in fact be honoured during justification,
not for the formatting. This patch makes it possible to use
the ZERO_FILL flag for any type, not just LONG_DEC.
Signed-off-by: Petr Tesarik <ptesarik(a)suse.cz>
13 years, 9 months
