Re: [Crash-utility] Crash fails with the error: "crash: read error: kernel virtual address:..."
by Dave Anderson
----- "Dave Anderson" <anderson(a)redhat.com> wrote:
>
> Another issue with the port is that the driver was written for
> kernels prior to the x86/x86_64 merge. So the driver's
> "./include/asm-i386/crash.h" and "./include/asm-x86_64/crash.h"
> header files will need to be merged.
Actually that's already been done in the Fedora driver, so that's
not an issue -- I was thinking about the RHEL version, which is
based upon 2.6.18...
Also, the Fedora driver does EXPORT_SYMBOL_GPL(page_is_ram).
So if you can rebuild your kernel, you might just add that
instead of substituting e820_any_mapped().
Dave
15 years, 9 months
Re: [Crash-utility] Crash fails with the error: "crash: read error: kernel virtual address:..."
by Dave Anderson
----- "Adhiraj Joshi" <adhiraj(a)linsyssoft.com> wrote:
> Hi Dave,
>
> Thanks for the quick reply! I followed approach 1 of disabling
> CONFIG_STRICT_DEVMEM and the problem disappeared. Now a new problem
> :-) On invoking crash, I get the following warning:
> WARNING: cannot access vmalloc'd module memory
>
> Because of this I am not able to run commands like "mod":
> crash> mod
> mod: cannot access vmalloc'd module memory
> crash>
What you're running into now is yet another problem with the /dev/mem
driver on the 32-bit x86 architecture when used on a live system with
more than 896MB of physical memory. Here's the beginning of the
/dev/mem driver's read routine in ./drivers/char/mem.c:
static ssize_t read_mem(struct file * file, char __user * buf,
size_t count, loff_t *ppos)
{
unsigned long p = *ppos;
ssize_t read, sz;
char *ptr;
if (!valid_phys_addr_range(p, count))
return -EFAULT;
where on x86, valid_phys_addr_range() looks like this:
static inline int valid_phys_addr_range(unsigned long addr, size_t count)
{
if (addr + count > __pa(high_memory))
return 0;
return 1;
}
And "high_memory" is the dividing line between "lowmem" and "highmem",
which is 896MB physical on your machine. So any reference to a physical
address above 896MB is rejected.
Kernel module memory is vmalloc'd with the __GFP_HIGHMEM modifier set,
so the allocation is biased to use physical memory from the highmem zone.
That being the case, and since your machine apparently has physical memory
above 896MB, any crash utility request to read module memory that
exists up there fails. Hence the "cannot access vmalloc'd module
memory" warning. You'll also see failures to access user memory,
and the PTEs to translate them as well. But given that the crucial
kernel memory is always located in lowmem, at least you can run a
crash session.
> Should the approach 2 or 3 that you mentioned below solve this
> problem? I have upgraded crash to the latest version (4.0-7.7).
... [ snip ] ...
> > For your kernel, you've got 3 options:
> >
> > (1) Rebuild your kernel without the CONFIG_STRICT_DEVMEM restriction.
> > (2) Port the Fedora /dev/crash driver (./drivers/char/crash.c) to your kernel.
> > (3) Write a kretprobe module that tinkers with the return value of the
> > kernel's devmem_is_allowed() function such that it always returns 1.
(3) won't help since you can't get that far -- and since you've turned
off CONFIG_STRICT_DEVMEM, it's unnecessary anyway.
You also can't kludge/kretprobe valid_phys_addr_range() to return 1, because
later on in read_mem() the physical address is translated into a unity-mapped
kernel virtual address here:
ptr = xlate_dev_mem_ptr(p);
if (!ptr)
return -EFAULT;
if (copy_to_user(buf, ptr, sz)) {
unxlate_dev_mem_ptr(p, ptr);
return -EFAULT;
}
where for x86, xlate_dev_mem_ptr() simply does a __va() on the
physical address:
void *xlate_dev_mem_ptr(unsigned long phys)
{
void *addr;
unsigned long start = phys & PAGE_MASK;
/* If page is RAM, we can use __va. Otherwise ioremap and unmap. */
if (page_is_ram(start >> PAGE_SHIFT))
return __va(phys);
and __va() simply adds PAGE_OFFSET to the physical address to make a
unity-mapped kernel virtual address -- which does not apply to "highmem"
physical addresses.
So that basically leaves (2) as the best option. That driver uses
kmap() on the physical address, and so it can handle any physical
memory request.
The driver should be portable, one caveat being is that it
uses page_is_ram() as a physical memory qualifier, and therefore
requires that the function be EXPORT_SYMBOL()'d to build. And in
upstream kernels, it currently is not.
It looks like you may be able to replace page_is_ram() with the
e820_any_mapped() function -- which is exported -- and look for
the E820_RAM "type":
int
e820_any_mapped(u64 start, u64 end, unsigned type)
{
int i;
for (i = 0; i < e820.nr_map; i++) {
struct e820entry *ei = &e820.map[i];
if (type && ei->type != type)
continue;
if (ei->addr >= end || ei->addr + ei->size <= start)
continue;
return 1;
}
return 0;
}
EXPORT_SYMBOL_GPL(e820_any_mapped);
Or you can make page_is_ram() EXPORT_SYMBOL_GPL().
Or you can just comment out the page_is_ram() call in the crash
driver's crash.h file.
Another issue with the port is that the driver was written for
kernels prior to the x86/x86_64 merge. So the driver's
"./include/asm-i386/crash.h" and "./include/asm-x86_64/crash.h"
header files will need to be merged.
Or you can always boot your kernel with "mem=896M" on the
kernel command line, and the problem goes away... ;-)
Dave
15 years, 9 months
Re: [Crash-utility] Crash fails with the error: "crash: read error: kernel virtual address:..."
by Dave Anderson
----- "Adhiraj Joshi" <adhiraj(a)linsyssoft.com> wrote:
> Hi All,
>
> On my x86 machine with fedora 10, I get an error which says: "crash:
> read error: kernel virtual address:...". The running kernel is not the
> Fedora 10 kernel, I have installed a latest vanilla kernel (2.6.27.12)
> on my machine.
>
> On googling, I found that someone has reported a similar problem but
> which was on Fedora kernel and x86_64 arch. Here is the link to that bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=237383
>
> In my case, this is the output from the crash command:
>
> [root@maveric ~]# crash /lib/modules/`uname -r`/build/vmlinux
> crash 4.0-7
> Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb 6.1
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "i686-pc-linux-gnu"...
>
> crash: read error: kernel virtual address: c08b0ae8 type: "cpu_possible_map"
> WARNING: cannot read cpu_possible_map
> crash: read error: kernel virtual address: c0801a94 type: "cpu_present_map"
> WARNING: cannot read cpu_present_map
> crash: read error: kernel virtual address: c080155c type: "cpu_online_map"
> WARNING: cannot read cpu_online_map
> crash: read error: kernel virtual address: c08dd700 type: "xtime"
> [root@maveric ~]#
>
> This is the "uname -a" output:
> "Linux maveric 2.6.27.12-adhi #1 SMP Fri Feb 20 22:13:54 IST 2009 i686
> i686 i386 GNU/Linux"
>
> Does anyone has any idea on this?
Your kernel is configured with CONFIG_STRICT_DEVMEM, which disallows
the reading of /dev/mem above 1MB physical. That makes it useless for
the crash utility:
https://www.redhat.com/archives/crash-utility/2009-February/msg00003.html
Fedora and RHEL kernels contain the /dev/crash driver, which gets automatically
installed upon crash invocation on the live system. That driver, found in
the kernel file "drivers/char/crash.c", is a replacement for the (restricted)
/dev/mem driver.
For your kernel, you've got 3 options:
(1) Rebuild your kernel without the CONFIG_STRICT_DEVMEM restriction.
(2) Port the Fedora /dev/crash driver (./drivers/char/crash.c) to your kernel.
(3) Write a kretprobe module that tinkers with the return value of the
kernel's devmem_is_allowed() function such that it always returns 1.
And w/respect to each option:
(1) Obviously this is the course of least resistance.
(2) Porting the RHEL/Fedora /dev/crash driver is possible, but since page_is_ram()
is not EXPORT_GPL()'d in 2.6.27 upstream kernels, then its usage by the driver
would have to be removed. And if you were to make it EXPORT_GPL(), then it
makes far more sense to just remove the CONFIG_STRICT_DEVMEM instead.
(3) I'll append some information re: writing a kretprobe module, but like
porting the /dev/crash driver, it may require rebuilding your kernel anyway.
And if that's the case, just remove the CONFIG_STRICT_DEVMEM restriction.
Also, you should upgrade from 4.0-7. There have been several fixes
for 2.6.27 kernels since that version.
Dave
----------------------------------------------------------------------------
Writing a kretprobe for devmem_is_allowed():
There are are currently three types of probes: kprobes, jprobes, and
kretprobes (also called return probes), but for this purpose, a
kretprobe is required because it allows the return value of the
devmem_is_allowed() function to be modified such that it will
always return 1.
The kernel documentation contains both directions for building
a kretprobe module and example module files for each kprobe type.
Pre-2.6.25 kernel trees put sample cut-and-pastable module files
contained within the "./Documentation/kprobes.txt" file itself.
2.6.25 and later kernel trees locate them in the "./samples/kprobes"
directory.
Note that pre-2.6.25 kernels must have been configured with both
CONFIG_KPROBES, CONFIG_KALLSYMS and CONFIG_MODULES turned on.
Additionally, 2.6.25 and later kernels also need CONFIG_KRETPROBES,
In any case, if the target kernel configs preclude the installation
of the module, then it probably makes more sense to rebuild the
kernel with CONFIG_STRICT_DEVMEM turned off, and avoid this kprobe
approach entirely.
Take the "kretprobe-example.c" cut-and-pastable file from the
the "./Documentation/kprobes.txt" file, or the standalone
"./samples/kprobes/kretprobe_example.c" file, whichever is
appropriate, and modify the kretprobe_init() and ret_handler()
functions like so:
static int __init kretprobe_init(void)
{
int ret;
- my_kretprobe.kp.symbol_name = func_name;
+ my_kretprobe.kp.symbol_name = "devmem_is_allowed";
ret = register_kretprobe(&my_kretprobe);
if (ret < 0) {
printk(KERN_INFO "register_kretprobe failed, returned %d\n",
ret);
return -1;
}
printk(KERN_INFO "Planted return probe at %s: %p\n",
my_kretprobe.kp.symbol_name, my_kretprobe.kp.addr);
return 0;
}
and make the ret_handler() function to simply do this:
static int ret_handler(struct kretprobe_instance *ri, struct pt_regs *regs)
{
regs->ax = 1;
return 0;
}
Earlier kernel versions may have different pt_regs structure member
names for the return register, i.e., "regs->eax" or "regs->rax" for
the x86 and x86_64 architectures respectively.
Build the kretprobe module as directed in ./Documentation/kprobes.txt"
and insmod it, after which crash will run just fine with /dev/mem.
15 years, 9 months
Re: [Crash-utility] crash on ppc docs / pointers
by Dave Anderson
----- "Bryn M. Reeves" <bmr(a)redhat.com> wrote:
> Hi Folks,
>
> I have some experience using crash on x86/x86_64 cores but I'm pretty
> new to ppc64. I'm making some headway, but a bit confused by some
> things.
>
> E.g., I have a function that calls kfree, so when I disassemble it,
> I'm expecting to see something like:
>
> bl <address of .kfree>
>
> But instead I find:
>
> 0xd000000000bd3b00 bl 0xd000000000bd4120
>
> Where that address lies inside .init_module:
>
> crash> sym 0xd000000000bd4120
> d000000000bd4120 (T) .init_module+496
>
> .kfree is somewhere else entirely:
> crash> sym .kfree
> c0000000000ed0b4 (T) .kfree
>
> I'm pretty certain this is the call to kfree that I'm interested in
> (LR points at the next instruction and we oopsed inside kfree).
>
> Is there some sort of indirection here that I'm missing?
Apparently so, at least with respect to kernel module code calling into
the base kernel. Here's a call to kmalloc() in ext3_xattr_block_set(),
where kmalloc() is actually an inline in slab.h that calls __kmalloc():
crash> dis -l .ext3_xattr_block_set
...
/usr/src/debug/kernel-2.6.18/linux-2.6.18.ppc64/fs/ext3/xattr.c: 724
0xd00000000019671c <.ext3_xattr_block_set+548>: ld r9,40(r26)
include/linux/slab.h: 154
0xd000000000196720 <.ext3_xattr_block_set+552>: li r4,208
0xd000000000196724 <.ext3_xattr_block_set+556>: ld r3,32(r9)
0xd000000000196728 <.ext3_xattr_block_set+560>: bl 0xd000000000199858
...
and looking around that "bl" target, there's the target function
address of ".__kmalloc" a few words after it:
crash> rd -s 0xd000000000199858 20
d000000000199858: 3d82fffe398ce448 f8410028e96c0020
d000000000199868: e84c00287d6903a6 4e80042000000000
d000000000199878: .__kmalloc PPC64_CACHES+24336
d000000000199888: 3d82fffe398ce478 f8410028e96c0020
d000000000199898: e84c00287d6903a6 4e80042000000000
d0000000001998a8: .memset PPC64_CACHES+24336
d0000000001998b8: 3d82fffe398ce4a8 f8410028e96c0020
d0000000001998c8: e84c00287d6903a6 4e80042000000000
d0000000001998d8: .memcpy PPC64_CACHES+24336
d0000000001998e8: 3d82fffe398ce4d8 f8410028e96c0020
crash>
So I presume if you look at the data around 0xd000000000bd4120, you'll
most likely see ".kfree".
Dave
15 years, 9 months
crash on ppc docs / pointers
by Bryn M. Reeves
Hi Folks,
I have some experience using crash on x86/x86_64 cores but I'm pretty
new to ppc64. I'm making some headway, but a bit confused by some things.
E.g., I have a function that calls kfree, so when I disassemble it,
I'm expecting to see something like:
bl <address of .kfree>
But instead I find:
0xd000000000bd3b00 bl 0xd000000000bd4120
Where that address lies inside .init_module:
crash> sym 0xd000000000bd4120
d000000000bd4120 (T) .init_module+496
.kfree is somewhere else entirely:
crash> sym .kfree
c0000000000ed0b4 (T) .kfree
I'm pretty certain this is the call to kfree that I'm interested in
(LR points at the next instruction and we oopsed inside kfree).
Is there some sort of indirection here that I'm missing?
Regards,
Bryn.
15 years, 9 months
Crash fails with the error: "crash: read error: kernel virtual address:..."
by Adhiraj Joshi
Hi All,
On my x86 machine with fedora 10, I get an error which says: "crash: read
error: kernel virtual address:...". The running kernel is not the Fedora 10
kernel, I have installed a latest vanilla kernel (2.6.27.12) on my machine.
On googling, I found that someone has reported a similar problem but which
was on Fedora kernel and x86_64 arch. Here is the link to that bug:
https://bugzilla.redhat.com/show_bug.cgi?id=237383
In my case, this is the output from the crash command:
[root@maveric ~]# crash /lib/modules/`uname -r`/build/vmlinux
crash 4.0-7
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
crash: read error: kernel virtual address: c08b0ae8 type:
"cpu_possible_map"
WARNING: cannot read cpu_possible_map
crash: read error: kernel virtual address: c0801a94 type: "cpu_present_map"
WARNING: cannot read cpu_present_map
crash: read error: kernel virtual address: c080155c type: "cpu_online_map"
WARNING: cannot read cpu_online_map
crash: read error: kernel virtual address: c08dd700 type: "xtime"
[root@maveric ~]#
This is the "uname -a" output:
"Linux maveric 2.6.27.12-adhi #1 SMP Fri Feb 20 22:13:54 IST 2009 i686 i686
i386 GNU/Linux"
Does anyone has any idea on this?
Thanks and regards,
Adhiraj Joshi.
15 years, 9 months
Re: [Crash-utility] xen_domctl_arch_setup struct differs from Xen's ?
by Dave Anderson
----- "Jun Koi" <junkoi2004(a)gmail.com> wrote:
> Hi,
>
> I notice that xen_domctl_arch_setup structure is defined in xendump.h
> in a different way with the one from Xen: the compile option check on
> IA64 is disabled. Why is that?
>
> /* from xendump.h */
> typedef struct xen_domctl_arch_setup {
> uint64_t flags; /* XEN_DOMAINSETUP_* */
> /* #ifdef __ia64__ */
> uint64_t bp; /* mpaddr of boot param area */
> uint64_t maxmem; /* Highest memory address for MDT. */
> uint64_t xsi_va; /* Xen shared_info area virtual address.
> */
> uint32_t hypercall_imm; /* Break imm for Xen hypercalls. */
> /* #endif */
> } xen_domctl_arch_setup_t;
Actually I'd be amazed if crash still works with "xm save" type files.
It was more an experiment than anything else.
In any case, the read of the data structure above was only applicable
to ia64, and the data thrown away. It was just there to seek past the
data structure to the rest of the ia64-specific file format.
Dave
15 years, 9 months
Re: [Crash-utility] crash without namelist?
by Dave Anderson
----- "Jun Koi" <junkoi2004(a)gmail.com> wrote:
> Hi,
>
> It seems that currently crash doesnt work without a namelist file?
Currently??? Never has, never will...
>
> But I think sometimes we only have the System.map file in hand,
> without the namelist. Can we do anything with that? Because in many
> cases only some restricted operations are enough to debug the crashed
> dump.
The basic design of the crash utility is that it is essentially
a huge wrapper around the embedded gdb module, taking advantage
of gdb's allowance of an alternative user interface. So during
initialization, crash calls into the embedded gdb's main() function,
passing it the vmlinux file as an argument, and control comes back
to the crash code through gdb's command_loop_hook pointer.
Even if the basic code flow were re-written, again, making it a
completely different design, there are just so many requirements
that need structure member offsets in order to get off the ground,
I don't want to even bother thinking about them all. Look at any
of the per-architecture *_init(POST_GDB) functions, and try to
envision crash running running without the offsets that it needs.
Dave
15 years, 9 months
crash without namelist?
by Jun Koi
Hi,
It seems that currently crash doesnt work without a namelist file?
But I think sometimes we only have the System.map file in hand,
without the namelist. Can we do anything with that? Because in many
cases only some restricted operations are enough to debug the crashed
dump.
Thanks,
Jun
15 years, 9 months
xen_domctl_arch_setup struct differs from Xen's ?
by Jun Koi
Hi,
I notice that xen_domctl_arch_setup structure is defined in xendump.h
in a different way with the one from Xen: the compile option check on
IA64 is disabled. Why is that?
/* from xendump.h */
typedef struct xen_domctl_arch_setup {
uint64_t flags; /* XEN_DOMAINSETUP_* */
/* #ifdef __ia64__ */
uint64_t bp; /* mpaddr of boot param area */
uint64_t maxmem; /* Highest memory address for MDT. */
uint64_t xsi_va; /* Xen shared_info area virtual address. */
uint32_t hypercall_imm; /* Break imm for Xen hypercalls. */
/* #endif */
} xen_domctl_arch_setup_t;
Thanks,
Jun
15 years, 9 months
