9:35 a.m.
"Graham, Simon" wrote:
I've been experimenting with the latest crash version on a system
running Xen (3.0.2-2 with kernel 2.6.16) and am
experiencing problems when attempting to debug the system live; debugging of saved guest
files and guest crash dumps
works well, it's just live debugging of dom0 that is problematical.
The symptom is that I get the error:
WARNING: cannot read linux_banner string
When I debug crash, I can see that it is finding the correct unstripped kernel from which
it finds the address of the
linux_banner variable _but_ when it attempts to read the string from /dev/mem it gets all
zeroes back. I have
confirmed that the address is correct by using gdb against /proc/kcore - gdb reads the
string quite nicely from the
virtual address (happens to be 0xc0450000, I can see that crash translates this to
physical address 450000 and gets
all zeroes when it reads this location from /dev/mem).
Hi Graham,
The xen /dev/mem driver read_mem() function does this:
if ((v = ioremap(p, sz)) == NULL) {
/*
* Some programs (e.g., dmidecode) groove off into weird RAM
* areas where no tables can possibly exist (because Xen will
* have stomped on them!). These programs get rather upset if
* we let them know that Xen failed their access, so we fake
* out a read of all zeroes. :-)
*/
if (clear_user(buf, count))
return -EFAULT;
read += count;
break;
}
when ioremap() returns NULL. Here's the NULL-returner in __ioremap():
/*
* Don't allow anybody to remap normal RAM that we're using..
*/
if (is_local_lowmem(phys_addr)) {
char *t_addr, *t_end;
struct page *page;
t_addr = bus_to_virt(phys_addr);
t_end = t_addr + (size - 1);
for(page = virt_to_page(t_addr); page <= virt_to_page(t_end); page++)
if(!PageReserved(page))
return NULL;
domid = DOMID_SELF;
}
So /dev/mem cannot be used. So to get crash to work on a live
x86 xen0 system (FC5), I've had to use the RHEL4 crash.c module
(/dev/crash) to make an end-run around /dev/mem. RHEL4's x86 and
x86_64 /dev/mem driver only allows reads of the first 256 pages of
physical memory. And now Xen has the restriction above.
FC5 has a CONFIG_CRASH configurable:
$ diff kernel-2.6.15-i686-guest.config.orig kernel-2.6.15-i686-guest.config
2526c2526
< # CONFIG_CRASH is not set
---
CONFIG_CRASH=m
2965d2964
< # CONFIG_CRASH is not set
$
If you're building from an upstream source, you could build the
crash.c module from FC5/RHEL4 into that kernel, or perhaps
patch the xen /dev/mem driver above to remove the local_lowmem()
restriction, or something like that...
Dave
I'm pretty sure that the various files in /boot for dom0 all match the running
system. Output from `crash -d2'
attached in case its useful...
So - the question is - what's the obvious thing I am missing!
Thanks,
Simon
crash 4.0-2.24
Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005 Fujitsu Limited
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
find_booted_kernel: search for [Linux version 2.6.16-xen0 (sgraham(a)anna.sn.stratus.com)
(gcc version 3.4.4 20050721
(Red Hat 3.4.4-2)) #3 Sat May 6 19:04:09 EDT 2006
]
mount_points[0]: / (83a1ac0)
mount_points[1]: /proc (83a1ad0)
mount_points[2]: /dev (83a1ae0)
mount_points[3]: / (83a1af0)
mount_points[4]: /dev (83a1b00)
mount_points[5]: /proc (83a1b10)
mount_points[6]: /sys (83a1b20)
mount_points[7]: /dev/pts (83a1b30)
mount_points[8]: /boot (83a1b48)
mount_points[9]: /dev/shm (83a1b58)
searchdirs[9]: /usr/lib/debug/lib/modules/2.6.16-xen0/
searchdirs[0]: /usr/src/linux/
searchdirs[1]: /boot/
searchdirs[2]: /boot/efi/redhat
searchdirs[3]: /boot/efi/EFI/redhat
searchdirs[4]: /
searchdirs[5]: /usr/src/kernels/
searchdirs[6]: /usr/src/debug/
searchdirs[7]: /usr/src/redhat/BUILD/kernel-2.6.16/linux/
searchdirs[8]: /usr/src/redhat/BUILD/kernel-2.6.16/linux-2.6.16/
find_booted_kernel: check: /boot/xen-syms-3.0.2-2
find_booted_kernel: check: /boot/vmlinux-syms-2.6.16-xenU
find_booted_kernel: check: /boot/vmlinux-syms-2.6.16-xen0
find_booted_kernel: found: /boot/vmlinux-syms-2.6.16-xen0
get_live_memory_source: /dev/mem
/proc/version:
Linux version 2.6.16-xen0 (sgraham(a)anna.sn.stratus.com) (gcc version 3.4.4 20050721 (Red
Hat 3.4.4-2)) #3 Sat May 6
19:04:09 EDT 2006
/boot/vmlinux-syms-2.6.16-xen0:
Linux version 2.6.16-xen0 (sgraham(a)anna.sn.stratus.com) (gcc version 3.4.4 20050721 (Red
Hat 3.4.4-2)) #3 Sat May 6
19:04:09 EDT 2006
gdb /boot/vmlinux-syms-2.6.16-xen0
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
WARNING: cannot read linux_banner string
/proc/version:
Linux version 2.6.16-xen0 (sgraham(a)anna.sn.stratus.com) (gcc version 3.4.4 20050721 (Red
Hat 3.4.4-2)) #3 Sat May 6
19:04:09 EDT 2006
linux_banner:
crash: /boot/vmlinux-syms-2.6.16-xen0 and /dev/mem do not match!
Usage:
crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
Enter "crash -h" for details.
-----------------------------------------------------------------------------------------------------------
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility