8:58 a.m.
On the other hand, there's nothing to prevent the
ambitious developer from writing their own
/dev/crash driver that *does* have a write operation
in it, is there?
-- David Wright, Egenera, Inc.
-----Original Message-----
From: crash-utility-bounces(a)redhat.com
[mailto:crash-utility-bounces@redhat.com] On Behalf Of Dave Anderson
Sent: Thursday, March 06, 2008 9:37 AM
To: Discussion list for crash utility usage,maintenance and
development
Subject: Re: [Crash-utility] Unable to change the content of
memory usingcrash on a live system
Dheeraj Sangamkar wrote:
> I use crash 4.0-3.9 on a live 2.6.9-55 kernel on i386/i686 as root.
>
> crash> ls -l /dev/crash
> crw------- 1 root root 10, 61 Mar 5 21:57 /dev/crash
> crash> ls -l /dev/mem
> crw-r----- 1 root kmem 1, 1 Mar 5 16:49 /dev/mem
> crash> q
> [root@linux17081 ~]# ls -l /dev/crash /dev/mem
> ls: /dev/crash: No such file or directory
> crw-r----- 1 root kmem 1, 1 Mar 5 16:49 /dev/mem
> [root@linux17081 ~]# id
> uid=0(root) gid=0(root)
> groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)
>
> So, the /dev/crash file has write permission for me. The
>
> I am attempting to change the content of some memory.
>
> crash> struct request_queue 0xf7b933f8
> struct request_queue {
> queue_head = {
> <SNIP>
> ...
> }
>
> crash> struct -o request_queue | grep in_flight
> [476] unsigned int in_flight;
> crash> eval 0xf7b933f8 + 476
> hexadecimal: f7b935d4
> decimal: 4156110292 (-138857004)
> octal: 36756232724
> binary: 11110111101110010011010111010100
> crash> rd f7b935d4
> f7b935d4: fffffff1 ....
> crash> wr f7b935d4 0
> wr: cannot write to /dev/crash!
>
> I get the error above even if I change the ownership of
/dev/kmem to
> root:root
> crash> ls -l /dev/mem
> crw-r----- 1 root root 1, 1 Mar 5 16:49 /dev/mem
>
> Am I doing something wrong? How do I change the content of
memory on a
> live system using crash?
With Red Hat x86 and x86_64 kernels, you can't.
I feel your pain...
The crash utility traditionally has had the capability of writing
to /dev/mem, which can be a very useful, powerful (and dangerous)
tool for kernel debugging.
But Red Hat deemed the /dev/mem interface as a security hole,
and restricted the x86 and x86_64 /dev/mem drivers to just
the first 256 pages (1MB) of physical memory, making it useless
for the crash utility. They allowed me to create the /dev/crash
driver to replace it -- but it is effectively read-only because
the driver has no write file operations handler:
static struct file_operations crash_fops = {
owner: THIS_MODULE,
llseek: crash_llseek,
read: crash_read,
};
and so the kernel's vfs_write() returns EINVAL.
Changing the permission of /dev/mem won't help because it
isn't used by the crash utility when /dev/crash exists.
Sorry about that,
Dave
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
9:10 a.m.
New subject: Unable to change the content of memory usingcrash on a live system
"Wright, David" <dwright(a)egenera.com> writes:
On the other hand, there's nothing to prevent the
ambitious developer from writing their own
/dev/crash driver that *does* have a write operation
in it, is there?
Why bother? Just unbreak the /dev/mem interface and don't load the
crash module.
-Jeff
9:12 a.m.
New subject: Unable to change the content of memory usingcrash on a live system
Wright, David wrote:
On the other hand, there's nothing to prevent the
ambitious developer from writing their own
/dev/crash driver that *does* have a write operation
in it, is there?
-- David Wright, Egenera, Inc.
Nope. Or removing the restriction from the /dev/mem driver,
which runs into the per-arch devmem_is_allowed() function.
Here's the "arch/i386/mm/init.c" version:
/*
* devmem_is_allowed() checks to see if /dev/mem access to a certain address is
* valid. The argument is a physical page number.
*
* On x86, access has to be given to the first megabyte of ram because that area
* contains bios code and data regions used by X and dosemu and similar apps.
* Access has to be given to non-kernel-ram areas as well, these contain the PCI
* mmio resources as well as potential bios/acpi data regions.
*/
int devmem_is_allowed(unsigned long pagenr)
{
if (pagenr <= 256)
return 1;
if (!page_is_ram(pagenr))
return 1;
return 0;
}
Dave
2:03 p.m.
New subject: change the content of memory using crash on a live system -- use kprobes!
As it turns out, another way to work around the /dev/mem restriction
is with kprobes.
The devmem_is_allowed() function looks like this, and for
the purposes of using it by the crash utility, we'd like it
to return 1 always:
int devmem_is_allowed(unsigned long pagenr)
{
if (pagenr <= 256)
return 1;
if (!page_is_ram(pagenr))
return 1;
return 0;
}
I took the sample kretprobes.c file from Documentation/kprobes.txt
and set a kretprobe in devmem_is_allowed() that forces a return
value of 1:
static int ret_handler(struct kretprobe_instance *ri, struct pt_regs *regs)
{
regs->eax = 1;
return 0;
}
Here's the "kretprobes.c" module I used:
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/kprobes.h>
static const char *probed_func = "devmem_is_allowed";
/* Return-probe handler: force return value to be 1. */
static int ret_handler(struct kretprobe_instance *ri, struct pt_regs *regs)
{
regs->eax = 1;
return 0;
}
static struct kretprobe my_kretprobe = {
.handler = ret_handler,
/* Probe up to 20 instances concurrently. */
.maxactive = 20
};
static int __init kretprobe_init(void)
{
int ret;
my_kretprobe.kp.symbol_name = (char *)probed_func;
if ((ret = register_kretprobe(&my_kretprobe)) < 0) {
printk("register_kretprobe failed, returned %d\n", ret);
return -1;
}
printk("Planted return probe at %p\n", my_kretprobe.kp.addr);
return 0;
}
static void __exit kretprobe_exit(void)
{
unregister_kretprobe(&my_kretprobe);
printk("kretprobe unregistered\n");
/* nmissed > 0 suggests that maxactive was set too low. */
printk("Missed probing %d instances of %s\n",
my_kretprobe.nmissed, probed_func);
}
module_init(kretprobe_init)
module_exit(kretprobe_exit)
MODULE_LICENSE("GPL");
And then build it with the supplied Makefile snippet:
obj-m := kretprobes.o
KDIR := /lib/modules/$(shell uname -r)/build
PWD := $(shell pwd)
default:
$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
clean:
rm -f *.mod.c *.ko *.o
Load the module, and then while it's running, "crash /dev/mem" will
override its default usage of "/dev/crash" and just work.
This was on a RHEL5 kernel, but it should work for RHEL4 as well:
$ crash /dev/mem
crash 4.0-6.1
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: /usr/lib/debug/lib/modules/2.6.18-53.el5/vmlinux
DUMPFILE: /dev/mem
CPUS: 2
DATE: Thu Mar 6 14:43:06 2008
UPTIME: 23 days, 04:50:13
LOAD AVERAGE: 0.14, 0.20, 0.20
TASKS: 175
NODENAME: crash.boston.redhat.com
RELEASE: 2.6.18-53.el5
VERSION: #1 SMP Wed Oct 10 16:34:02 EDT 2007
MACHINE: i686 (1993 Mhz)
MEMORY: 511.5 MB
PID: 15518
COMMAND: "crash"
TASK: cb0ffaa0 [THREAD_INFO: d976c000]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
crash> p panic_on_oops
panic_on_oops = $2 = 1
crash> wr panic_on_oops 2
crash> p panic_on_oops
panic_on_oops = $3 = 2
crash>
Dave
1:04 p.m.
New subject: change the content of memory using crash on a live system -- use kprobes!
Thanks a million Dave,
I will try using the kprobes approach.
Dheeraj
On Fri, Mar 7, 2008 at 1:33 AM, Dave Anderson <anderson(a)redhat.com> wrote:
As it turns out, another way to work around the /dev/mem restriction
is with kprobes.
The devmem_is_allowed() function looks like this, and for
the purposes of using it by the crash utility, we'd like it
to return 1 always:
int devmem_is_allowed(unsigned long pagenr)
{
if (pagenr <= 256)
return 1;
if (!page_is_ram(pagenr))
return 1;
return 0;
}
I took the sample kretprobes.c file from Documentation/kprobes.txt
and set a kretprobe in devmem_is_allowed() that forces a return
value of 1:
static int ret_handler(struct kretprobe_instance *ri, struct pt_regs
*regs)
{
regs->eax = 1;
return 0;
}
Here's the "kretprobes.c" module I used:
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/kprobes.h>
static const char *probed_func = "devmem_is_allowed";
/* Return-probe handler: force return value to be 1. */
static int ret_handler(struct kretprobe_instance *ri, struct pt_regs
*regs)
{
regs->eax = 1;
return 0;
}
static struct kretprobe my_kretprobe = {
.handler = ret_handler,
/* Probe up to 20 instances concurrently. */
.maxactive = 20
};
static int __init kretprobe_init(void)
{
int ret;
my_kretprobe.kp.symbol_name = (char *)probed_func;
if ((ret = register_kretprobe(&my_kretprobe)) < 0) {
printk("register_kretprobe failed, returned %d\n", ret);
return -1;
}
printk("Planted return probe at %p\n", my_kretprobe.kp.addr);
return 0;
}
static void __exit kretprobe_exit(void)
{
unregister_kretprobe(&my_kretprobe);
printk("kretprobe unregistered\n");
/* nmissed > 0 suggests that maxactive was set too low. */
printk("Missed probing %d instances of %s\n",
my_kretprobe.nmissed, probed_func);
}
module_init(kretprobe_init)
module_exit(kretprobe_exit)
MODULE_LICENSE("GPL");
And then build it with the supplied Makefile snippet:
obj-m := kretprobes.o
KDIR := /lib/modules/$(shell uname -r)/build
PWD := $(shell pwd)
default:
$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
clean:
rm -f *.mod.c *.ko *.o
Load the module, and then while it's running, "crash /dev/mem" will
override its default usage of "/dev/crash" and just work.
This was on a RHEL5 kernel, but it should work for RHEL4 as well:
$ crash /dev/mem
crash 4.0-6.1
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you
are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: /usr/lib/debug/lib/modules/2.6.18-53.el5/vmlinux
DUMPFILE: /dev/mem
CPUS: 2
DATE: Thu Mar 6 14:43:06 2008
UPTIME: 23 days, 04:50:13
LOAD AVERAGE: 0.14, 0.20, 0.20
TASKS: 175
NODENAME: crash.boston.redhat.com
RELEASE: 2.6.18-53.el5
VERSION: #1 SMP Wed Oct 10 16:34:02 EDT 2007
MACHINE: i686 (1993 Mhz)
MEMORY: 511.5 MB
PID: 15518
COMMAND: "crash"
TASK: cb0ffaa0 [THREAD_INFO: d976c000]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
crash> p panic_on_oops
panic_on_oops = $2 = 1
crash> wr panic_on_oops 2
crash> p panic_on_oops
panic_on_oops = $3 = 2
crash>
Dave
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
1:44 p.m.
New subject: change the content of memory using crash on a live system -- use kprobes!
Dheeraj Sangamkar wrote:
Thanks a million Dave,
I will try using the kprobes approach.
One thing I neglected to mention re: the usage of /dev/mem.
Another restriction it has always had (not just in Red Hat
kernels) is this:
static ssize_t read_mem(struct file * file, char __user * buf,
size_t count, loff_t *ppos)
{
unsigned long p = *ppos;
ssize_t read, sz;
char *ptr;
if (!valid_phys_addr_range(p, count))
return -EFAULT;
where valid_phys_addr_range() on an x86 restricts access
to lowmem, i.e. the first 896MB of physical memory:
static inline int valid_phys_addr_range(unsigned long addr, size_t count)
{
if (addr + count > __pa(high_memory))
return 0;
return 1;
}
So if you need to look at a piece of memory above that,
"crash /dev/mem" will fail. Normally that's not a problem,
if you're trying to access unity-mapped kernel virtual
memory, i.e., where static kernel data, slab data, etc.
exist. But if it's a user-space address, or a mapped vmalloc()
address, and the PTE's needed to translate that virtual
address exist in highmem, then the physical location
cannot be determined.
That's always been a weakness with the /dev/mem driver,
and again, I don't exactly know why. The /dev/crash
driver doesn't make the restriction, and AFAICT if the
valid_phys_addr_range() could be also be kprobe'd to return
a 1, then it would mimic /dev/crash. But it gets inlined
and it's certainly not nearly as simple as the devmem_is_allowed()
probe. It may be possible, but I don't know how...
Anyway, getting back to your original example:
crash> struct request_queue 0xf7b933f8
struct request_queue {
queue_head = {
<SNIP>
...
}
It's not clear to me whether 0xf7b933f8 is a vmalloc()
address (like from a module) or right on the edge of
being at the very top of unity-mapped memory. If it's
a mapped vmalloc() address, then the kprobes hack
won't work "out of the box". You may have first run
crash with /dev/crash, do a vtop on the vmalloc address
to find its physical address, and then *only* if the
physical address is in lowmem, will you be able to
then subsequently do the kprobes workaround, using
"wr -p" to specify the physical address instead of
the virtual address.
Or reboot your kernel with "mem=896m" to keep all
physical memory below 896MB.
Clear as mud?
Dave
Dave
6108
days inactive
6109
days old
devel@lists.crash-utility.osci.io
5 comments
4 participants
participants (4)
-
Dave Anderson -
Dheeraj Sangamkar -
Jeff Moyer -
Wright, David