On Wed, Feb 25, 2009 at 7:45 PM, Dave Anderson <anderson@redhat.com> wrote:

----- "Adhiraj Joshi" <adhiraj@linsyssoft.com> wrote:

> Hi All,
>
> On my x86 machine with fedora 10, I get an error which says: "crash:
> read error: kernel virtual address:...". The running kernel is not the
> Fedora 10 kernel, I have installed a latest vanilla kernel (2.6.27.12)
> on my machine.
>
> On googling, I found that someone has reported a similar problem but
> which was on Fedora kernel and x86_64 arch. Here is the link to that bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=237383
>
> In my case, this is the output from the crash command:
>
> [root@maveric ~]# crash /lib/modules/`uname -r`/build/vmlinux
> crash 4.0-7
> Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb 6.1
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "i686-pc-linux-gnu"...
>
> crash: read error: kernel virtual address: c08b0ae8 type: "cpu_possible_map"
> WARNING: cannot read cpu_possible_map
> crash: read error: kernel virtual address: c0801a94 type: "cpu_present_map"
> WARNING: cannot read cpu_present_map
> crash: read error: kernel virtual address: c080155c type: "cpu_online_map"
> WARNING: cannot read cpu_online_map
> crash: read error: kernel virtual address: c08dd700 type: "xtime"
> [root@maveric ~]#
>
> This is the "uname -a" output:
> "Linux maveric 2.6.27.12-adhi #1 SMP Fri Feb 20 22:13:54 IST 2009 i686
> i686 i386 GNU/Linux"
>
> Does anyone has any idea on this?

Your kernel is configured with CONFIG_STRICT_DEVMEM, which disallows
the reading of /dev/mem above 1MB physical. That makes it useless for
the crash utility:

https://www.redhat.com/archives/crash-utility/2009-February/msg00003.html

Fedora and RHEL kernels contain the /dev/crash driver, which gets automatically
installed upon crash invocation on the live system. That driver, found in
the kernel file "drivers/char/crash.c", is a replacement for the (restricted)
/dev/mem driver.

For your kernel, you've got 3 options:

(1) Rebuild your kernel without the CONFIG_STRICT_DEVMEM restriction.
(2) Port the Fedora /dev/crash driver (./drivers/char/crash.c) to your kernel.
(3) Write a kretprobe module that tinkers with the return value of the
kernel's devmem_is_allowed() function such that it always returns 1.

And w/respect to each option:

(1) Obviously this is the course of least resistance.

(2) Porting the RHEL/Fedora /dev/crash driver is possible, but since page_is_ram()
is not EXPORT_GPL()'d in 2.6.27 upstream kernels, then its usage by the driver
would have to be removed. And if you were to make it EXPORT_GPL(), then it
makes far more sense to just remove the CONFIG_STRICT_DEVMEM instead.

(3) I'll append some information re: writing a kretprobe module, but like
porting the /dev/crash driver, it may require rebuilding your kernel anyway.
And if that's the case, just remove the CONFIG_STRICT_DEVMEM restriction.

Also, you should upgrade from 4.0-7. There have been several fixes
for 2.6.27 kernels since that version.

Dave

----------------------------------------------------------------------------

Writing a kretprobe for devmem_is_allowed():

There are are currently three types of probes: kprobes, jprobes, and
kretprobes (also called return probes), but for this purpose, a
kretprobe is required because it allows the return value of the
devmem_is_allowed() function to be modified such that it will
always return 1.

The kernel documentation contains both directions for building
a kretprobe module and example module files for each kprobe type.
Pre-2.6.25 kernel trees put sample cut-and-pastable module files
contained within the "./Documentation/kprobes.txt" file itself.
2.6.25 and later kernel trees locate them in the "./samples/kprobes"
directory.

Note that pre-2.6.25 kernels must have been configured with both
CONFIG_KPROBES, CONFIG_KALLSYMS and CONFIG_MODULES turned on.
Additionally, 2.6.25 and later kernels also need CONFIG_KRETPROBES,
In any case, if the target kernel configs preclude the installation
of the module, then it probably makes more sense to rebuild the
kernel with CONFIG_STRICT_DEVMEM turned off, and avoid this kprobe
approach entirely.

Take the "kretprobe-example.c" cut-and-pastable file from the
the "./Documentation/kprobes.txt" file, or the standalone
"./samples/kprobes/kretprobe_example.c" file, whichever is
appropriate, and modify the kretprobe_init() and ret_handler()
functions like so:

static int __init kretprobe_init(void)
{
int ret;

- my_kretprobe.kp.symbol_name = func_name;
+ my_kretprobe.kp.symbol_name = "devmem_is_allowed";
ret = register_kretprobe(&my_kretprobe);
if (ret < 0) {
printk(KERN_INFO "register_kretprobe failed, returned %d\n",
ret);
return -1;
}
printk(KERN_INFO "Planted return probe at %s: %p\n",
my_kretprobe.kp.symbol_name, my_kretprobe.kp.addr);
return 0;
}

and make the ret_handler() function to simply do this:

static int ret_handler(struct kretprobe_instance *ri, struct pt_regs *regs)
{
regs->ax = 1;
return 0;
}

Earlier kernel versions may have different pt_regs structure member
names for the return register, i.e., "regs->eax" or "regs->rax" for
the x86 and x86_64 architectures respectively.

Build the kretprobe module as directed in ./Documentation/kprobes.txt"
and insmod it, after which crash will run just fine with /dev/mem.

--
Crash-utility mailing list
Crash-utility@redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility