8:15 p.m.
I'm trying to explore crash dumps under these conditions:
- VMware workstation 7.1.5 build-491717
- CentOS 5.7 +updates, as of today
- kernel 2.6.18-274.17.1.el5 x86
- crash 6.0.2
I've sucessfully enabled kdump to generate crash dumps, but the
'crash' utility can neither find the vmlinux image for the live
system, nor match the crash dump with the vmlinux image I direct
it to.
Misc RPMs installed:
[root@172-20-1-25 modules]# uname -r; rpm -qa | grep kernel
2.6.18-274.17.1.el5
kernel-2.6.18-274.17.1.el5
kernel-debug-2.6.18-274.17.1.el5
kernel-debuginfo-2.6.18-274.17.1.el5.centos.plus
kernel-debuginfo-common-2.6.18-274.17.1.el5.centos.plus
kernel-devel-2.6.18-274.17.1.el5
Here are various invocations I've attempted; can anyone suggest what's
causing these failures, and how to work around them?
Please let me know if there are any details I can provide to assist.
Additional notes:
- I've tries both crashkernel=64M@16M and crashkernel=128M@16M as
kernel arguments, with the same results.
- I've attempted a symlink to match the debug modules directly to
the name of the kernel, but that didn't change the results:
2.6.18-274.17.1.el5.centos.plus -> 2.6.18-274.17.1.el5
-------------------------------
Against the live system:
[root@172-20-1-25 modules]# crash
crash 6.0.2
[...]
crash: cannot find booted kernel -- please enter namelist argument
-------------------------------
Against the vmlinux file supplied by the kernel-debuginfo RPM:
[root@172-20-1-25 modules]# crash
/usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux
crash 6.0.2
[...]
WARNING: /usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux
and /proc/version do not match!
WARNING: /proc/version indicates kernel version: 2.6.18-274.17.1.el5
crash: please use the vmlinux file for that kernel version, or try using
the System.map for that kernel version as an additional argument.
-------------------------------
Against my crash dump:
[root@172-20-1-25 modules]# crash
/usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux
/home/crash/127.0.0.1-2012-01-26-19\:19\:19/vmcore
crash 6.0.2
[...]
crash: cannot determine base kernel version
WARNING: cannot read linux_banner string
crash: /usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux and
/home/crash/127.0.0.1-2012-01-26-19:19:19/vmcore do not match!
--
Brian Reichert <reichert(a)numachi.com>
BSD admin/developer at large
3:54 p.m.
----- Original Message -----
I'm trying to explore crash dumps under these conditions:
- VMware workstation 7.1.5 build-491717
- CentOS 5.7 +updates, as of today
- kernel 2.6.18-274.17.1.el5 x86
- crash 6.0.2
I've sucessfully enabled kdump to generate crash dumps, but the
'crash' utility can neither find the vmlinux image for the live
system, nor match the crash dump with the vmlinux image I direct
it to.
Misc RPMs installed:
[root@172-20-1-25 modules]# uname -r; rpm -qa | grep kernel
2.6.18-274.17.1.el5
kernel-2.6.18-274.17.1.el5
kernel-debug-2.6.18-274.17.1.el5
kernel-debuginfo-2.6.18-274.17.1.el5.centos.plus
kernel-debuginfo-common-2.6.18-274.17.1.el5.centos.plus
kernel-devel-2.6.18-274.17.1.el5
Here are various invocations I've attempted; can anyone suggest what's
causing these failures, and how to work around them?
Please let me know if there are any details I can provide to assist.
Additional notes:
- I've tries both crashkernel=64M@16M and crashkernel=128M@16M as
kernel arguments, with the same results.
- I've attempted a symlink to match the debug modules directly to
the name of the kernel, but that didn't change the results:
2.6.18-274.17.1.el5.centos.plus -> 2.6.18-274.17.1.el5
-------------------------------
Against the live system:
[root@172-20-1-25 modules]# crash
crash 6.0.2
[...]
crash: cannot find booted kernel -- please enter namelist argument
-------------------------------
Against the vmlinux file supplied by the kernel-debuginfo RPM:
[root@172-20-1-25 modules]# crash
/usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux
crash 6.0.2
[...]
WARNING: /usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux
and /proc/version do not match!
WARNING: /proc/version indicates kernel version: 2.6.18-274.17.1.el5
crash: please use the vmlinux file for that kernel version, or try using
the System.map for that kernel version as an additional
argument.
There is an inconsistency between the 2.6.18-274.17.1.el5.centos.plus kernel
and the running kernel 2.6.18-274.17.1.el5. I don't know what the
"centos.plus"
vmlinux kernel is, but its symbols apparently do no match the 2.6.18-274.17.1.el5
kernel that is running. Is the system running a bona fide RHEL5 2.6.18-274.17.1.el5
kernel, and you're using a centos-generated kernel-debuginfo package? In other
words,
in a real RHEL5 environment there should be
"kernel-debuginfo-2.6.18-274.17.1.el5" and
associated "kernel-debuginfo-common-2.6.18-274.17.1.el5" packages that would
have
the vmlinux file for the running 2.6.18-274.17.1.el5 kernel. So again, I don't have
a
clue about what those "centos.plus" packages are all about...
Anyway, you could try what the error message suggests. There is probably a
"/boot/System.map-2.6.18-274.17.1.el5" on that system, and presuming so,
try this on the live system:
$ crash usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux
/boot/System.map-2.6.18-274.17.1.el5
-------------------------------
Against my crash dump:
[root@172-20-1-25 modules]# crash
/usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux
/home/crash/127.0.0.1-2012-01-26-19\:19\:19/vmcore
crash 6.0.2
[...]
crash: cannot determine base kernel version
WARNING: cannot read linux_banner string
crash: /usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux
and /home/crash/127.0.0.1-2012-01-26-19:19:19/vmcore do not match!
Same thing is happening with the vmcore. In the case of a vmcore, it reads
the "linux_banner" string (what /proc/version shows on a live system) at the
address where the "centos.plus" vmlinux file says it is, but after doing so,
it doesn't match up. (so it's reading the wrong location)
If you do this:
$ strings /home/crash/127.0.0.1-2012-01-26-19:19:19/vmcore | grep "Linux
version"
I'm guessing you won't see the "centos.plus" string?
As with the live system, you could try throwing the /boot/System.map-<version> file
on the command line.
In any case, if you add a debug count on the command line (the larger the number
the more debug data gets displayed), it may illuminate the failure, i.e., by
entering:
$ crash -d7 /path-to/vmlinux /path-to/vmcore
Dave
3:21 p.m.
On Thu, Jan 26, 2012 at 04:54:58PM -0500, Dave Anderson wrote:
There is an inconsistency between the 2.6.18-274.17.1.el5.centos.plus
kernel
and the running kernel 2.6.18-274.17.1.el5. I don't know what the
"centos.plus"
vmlinux kernel is, but its symbols apparently do no match the 2.6.18-274.17.1.el5
kernel that is running.
Is the system running a bona fide RHEL5 2.6.18-274.17.1.el5
kernel, and you're using a centos-generated kernel-debuginfo package?
No, this is a CentOS 5.7 installation. These RPMs came from the CentOS
official repos. I was concerned by the apparrent disrepancy in the
names of the files.
Anyway, you could try what the error message suggests. There is
probably a
"/boot/System.map-2.6.18-274.17.1.el5" on that system, and presuming so,
try this on the live system:
I'll go looking for such a file, and explore your other suggestions.
Thanks for the feedback!
Dave
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Brian Reichert <reichert(a)numachi.com>
BSD admin/developer at large
11:03 a.m.
----- Original Message -----
On Thu, Jan 26, 2012 at 04:54:58PM -0500, Dave Anderson wrote:
> There is an inconsistency between the
> 2.6.18-274.17.1.el5.centos.plus kernel
> and the running kernel 2.6.18-274.17.1.el5. I don't know what the
> "centos.plus"
> vmlinux kernel is, but its symbols apparently do no match the
> 2.6.18-274.17.1.el5
> kernel that is running.
> Is the system running a bona fide RHEL5 2.6.18-274.17.1.el5
> kernel, and you're using a centos-generated kernel-debuginfo
> package?
No, this is a CentOS 5.7 installation. These RPMs came from the CentOS
official repos. I was concerned by the apparrent disrepancy in the
names of the files.
> Anyway, you could try what the error message suggests. There is probably a
> "/boot/System.map-2.6.18-274.17.1.el5" on that system, and presuming so,
> try this on the live system:
I'll go looking for such a file, and explore your other suggestions.
Thanks for the feedback!
I note that here:
http://debuginfo.centos.org/5/x86_64/
they have both the "centos.plus" version that you have installed, as well as
what would appear to be the one that you actually want:
kernel-debuginfo-2.6.18-274.17.1.el5.centos.plus.x86_64.rpm 11-Jan-2012 18:57 200M
kernel-debuginfo-2.6.18-274.17.1.el5.x86_64.rpm 11-Jan-2012 01:59 179M
I would install the kernel-debuginfo-2.6.18-274.17.1.el5.x86_64.rpm
and its associated kernel-debuginfo-common-2.6.18-274.7.1.el5.x86_64.rpm
packages.
Dave
5:29 p.m.
On Fri, Jan 27, 2012 at 12:03:44PM -0500, Dave Anderson wrote:
I note that here:
http://debuginfo.centos.org/5/x86_64/
they have both the "centos.plus" version that you have installed, as well as
what would appear to be the one that you actually want:
kernel-debuginfo-2.6.18-274.17.1.el5.centos.plus.x86_64.rpm 11-Jan-2012 18:57 200M
kernel-debuginfo-2.6.18-274.17.1.el5.x86_64.rpm 11-Jan-2012 01:59 179M
I would install the kernel-debuginfo-2.6.18-274.17.1.el5.x86_64.rpm
and its associated kernel-debuginfo-common-2.6.18-274.7.1.el5.x86_64.rpm
packages.
Weird. All I did was this:
yum --enablerepo="debug" -y install kernel-debuginfo
And *.centos.plus is what came out of the repo. :/
Dave
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Brian Reichert <reichert(a)numachi.com>
BSD admin/developer at large
5:26 p.m.
On Thu, Jan 26, 2012 at 04:54:58PM -0500, Dave Anderson wrote:
As with the live system, you could try throwing the
/boot/System.map-<version> file
on the command line.
For the record, this suggestion did work, thanks.
Against the vmlinux file supplied by the kernel-debuginfo RPM:
crash -S /boot/System.map-2.6.18-274.17.1.el5 \
/usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux
I can analyze my crash thusly:
crash -S /boot/System.map-2.6.18-274.17.1.el5 \
/usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux \
/home/crash/127.0.0.1-2012-01-27-12\:22\:05/vmcore
So, I'm off to the races. Hurray!
Dave
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Brian Reichert <reichert(a)numachi.com>
BSD admin/developer at large
5383
days inactive
5383
days old
devel@lists.crash-utility.osci.io
5 comments
2 participants
participants (2)
-
Brian Reichert -
Dave Anderson