9:06 a.m.
Whenever a crash session fails with a "do not match" message
such as this:
crash: vmlinux-2.6.27.19-5-default.debug and vmcore do not match!
it is because the virtual addresses of the kernel symbols in the
vmlinux file do not match the virtual addresses of the kernel that
crashed.
When you run with "crash -d<number>" you will see the first few
attempts to gather kernel data (kernel_config_data, xtime, the
cpu_possible, cpu_present and cpu_online maps, the utsname structure,
and then the "linux_banner" string. The linux_banner string is what
you would see in /proc/version on the live system, for example:
# cat /proc/version
Linux version 2.6.32-220.el6.x86_64 (mockbuild(a)x86-004.build.bos.redhat.com) (gcc version
4.4.5 20110214 (Red Hat 4.4.5-6) (GCC) ) #1 SMP Wed Nov 9 08:03:13 EST 2011
#
If the crash utility does not find the correct string at the address
specified in the vmlinux file, then it makes no sense to continue,
and so it quits with the "do not match" message.
Accordingly, one of your examples ends like this, where the
vmlinux file indicates that the linux_banner string is located
at kernel virtual address ffffffff804a7000. That address was
translated to physical address 4a7000, and then read from the
compressed kdump vmcore. But the linux_banner string read from
that address in the vmcore contains garbage:
...
<readmem: ffffffff804a7000, KVADDR, "readstring characters", 1499, (ROE|Q),
7fff35eeaa70>
<read_diskdump: addr: ffffffff804a7000 paddr: 4a7000 cnt: 1499>
read_diskdump: paddr/pfn: 4a7000/4a7 -> physical page is cached: 4a7000
linux_banner:
�x;�����
crash: vmlinux-2.6.27.19-5-default.debug and vmcore do not match!
...
I cannot answer for SUSE, but if I am not mistaken, they build a
debug kernel (i.e, built with -g) for their distrobution kernels
after-the-fact, and when that is done, the kernel virtual addresses
are not exactly the same. To work around that, you can pass the
System.map file of the actual crashed kernel -- which has the *correct"
kernel virtual addresses -- on the crash command line. And when
you did that, things apparently worked correctly.
In other words, this failed:
# crash vmlinux-2.6.27.19-5-default vmlinux-2.6.27.19-5-default.debug vmcore
But this worked:
# crash vmlinux-2.6.27.19-5-default System.map-2.6.27.19-5-default vmcore
And it worked because the crash utility takes the (correct) symbol values
found in the System.map file, and "patches" the symbol values that were
initially read from the vmlinux file(s).
Complicating matters a bit is the usage of a vmlinux file and its associated
vmlinux.debug file. In old RHEL3 kernels, Red Hat would split the vmlinux
file into its stripped version and a .debug version, the same way as is
done with user-space programs. But from RHEL4 onwards, the that was no longer
done, and there is a single vmlinux file that contains all the debuginfo
data, and which is the same kernel as the distribution kernel. In SLES kernels,
I am not sure whether that was true in the example above, or whether the .debug
kernel is the kernel that they build after-the-fact?
In any case, I note that there a several novell.com addresses that are cc'd
with this email, and there are many SLES users and Novell/Suse employees
on this list who can give you the specifics of how they do things.
Dave
----- Original Message -----
on another machine,I use all kinds of crash command :
Vsles11sp1 /mnt/disk/crash/2012-03-24-18:57 #crash
vmlinux-2.6.27.19-5-default.gz System.map-2.6.27.19-5-default vmcore
crash 6.0.4
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html >
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
crash: /var/tmp/vmlinux-2.6.27.19-5-default.gz_RYx9JC: no debugging
data available
Vsles11sp1 /mnt/disk/crash/2012-03-24-18:57 #gunzip vmlinux-2.
6.27.19-5-default.gz
Vsles11sp1 /mnt/disk/crash/2012-03-24-18:57 # crash
vmlinux-2.6.27.19-5-default vmlinux-2.6.27.19-5-default.debug vmcore
crash 6.0.4
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html >
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: could not find MAGIC_START!
crash: vmlinux-2.6.27.19-5-default.debug and vmcore do not match!
Usage:
crash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
Enter "crash -h" for details.
Vsles11sp1 /mnt/disk/crash/2012-03-24-18:57 #cat README.txt
Kernel crashdump
----------------
Crash time : 2012-03-24 10:57 (+0000)
Kernel version : 2.6.27.19-5-default
Host : sles11
Dump level : 0
Dump format : compressed
but I use another command is OK, what's wrong ?
Vsles11sp1 /mnt/disk/crash/2012-03-24-18:57 #crash
vmlinux-2.6.27.19-5-default System.map-2.6.27.19-5-default vmcore
crash 6.0.4
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html >
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
SYSTEM MAP: System.map-2.6.27.19-5-default
DEBUG KERNEL: vmlinux-2.6.27.19-5-default (2.6.27.19-5-default)
DUMPFILE: vmcore
CPUS: 2
DATE: Sat Mar 24 18:57:31 2012
UPTIME: 00:13:30
LOAD AVERAGE: 0.27, 0.30, 0.28
TASKS: 190
NODENAME: sles11
RELEASE: 2.6.27.19-5-default
VERSION: #6 SMP Sat Nov 12 08:58:08 CST 2011
MACHINE: x86_64 (2393 Mhz)
MEMORY: 3.9 GB
PANIC: "SysRq : Trigger a crashdump"
PID: 4700
COMMAND: "bash"
TASK: ffff88007f8f6080 [THREAD_INFO: ffff88007f802000]
CPU: 0
STATE: TASK_RUNNING (SYSRQ)
crash>
From: hujun_hotmail
Sent: Saturday, March 24, 2012 8:07 PM
To: crash mailist
Subject: [Crash-utility] crash can live analysis but vmcore can't.
hello experts:
My linux system : Suse SLES 11 , crash can live analysis, but can't
open vmcore generated by kdump.
why?
sles11:/mnt/sda12/crash/2012-03-24-18:57 # crash
crash 6.0.4
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html >
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
KERNEL: /boot/vmlinuz-2.6.27.19-5-default.vmlinux-debug
DUMPFILE: /dev/mem
CPUS: 2
DATE: Sat Mar 24 19:10:45 2012
UPTIME: 00:07:17
LOAD AVERAGE: 0.16, 0.19, 0.12
TASKS: 189
NODENAME: sles11
RELEASE: 2.6.27.19-5-default
VERSION: #6 SMP Sat Nov 12 08:58:08 CST 2011
MACHINE: x86_64 (2393 Mhz)
MEMORY: 3.9 GB
PID: 4590
COMMAND: "crash"
TASK: ffff8801230fa340 [THREAD_INFO: ffff880131c86000]
CPU: 0
STATE: TASK_RUNNING (ACTIVE)
crash>
sles11:/mnt/sda12/crash/2012-03-24-18:57 # crash
vmlinux-2.6.27.19-5-default.gz vmlinux-2.6.27.19-5-default.debug
vmcore
crash 6.0.4
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public
License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for
details.
please wait... (uncompressing vmlinux-2.6.27.19-5-default.gz)
compressed kdump: header->utsname.machine:
compressed kdump: memory bitmap offset: 2000
diskdump_data:
filename: vmcore
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
dfd: 4
ofp: 0
machine_type: 62 (EM_X86_64)
header: dee020
signature: "KDUMP "
header_version: 1
utsname:
sysname:
nodename:
release:
version:
machine:
domainname:
timestamp:
tv_sec: 0
tv_usec: 0
status: 0 ()
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 80
max_mapnr: 1294336
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 1
tasks[nr_cpus]: 0
sub_header: 0 (n/a)
sub_header_kdump: def030
phys_base: 0
dump_level: 0 (0x0)
data_offset: 52000
block_size: 4096
block_shift: 12
bitmap: 7f812de69010
bitmap_len: 327680
dumpable_bitmap: 7f812de18010
byte: 0
bit: 0
compressed_page: e00a40
curbufptr: 0
page_cache_hdr[0]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df0a30
pg_hit_count: 0
page_cache_hdr[1]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df1a30
pg_hit_count: 0
page_cache_hdr[2]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df2a30
pg_hit_count: 0
page_cache_hdr[3]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df3a30
pg_hit_count: 0
page_cache_hdr[4]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df4a30
pg_hit_count: 0
page_cache_hdr[5]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df5a30
pg_hit_count: 0
page_cache_hdr[6]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df6a30
pg_hit_count: 0
page_cache_hdr[7]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df7a30
pg_hit_count: 0
page_cache_hdr[8]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df8a30
pg_hit_count: 0
page_cache_hdr[9]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: df9a30
pg_hit_count: 0
page_cache_hdr[10]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfaa30
pg_hit_count: 0
page_cache_hdr[11]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfba30
pg_hit_count: 0
page_cache_hdr[12]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfca30
pg_hit_count: 0
page_cache_hdr[13]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfda30
pg_hit_count: 0
page_cache_hdr[14]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dfea30
pg_hit_count: 0
page_cache_hdr[15]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: dffa30
pg_hit_count: 0
page_cache_buf: df0a30
evict_index: 0
evictions: 0
accesses: 0
cached_reads: 0
valid_pages: df0040
readmem: read_diskdump()
[?1034hcompressed kdump: phys_base: 0
gdb vmlinux-2.6.27.19-5-default.debug
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html >
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
GETBUF(248 -> 0)
GETBUF(1500 -> 1)
please wait... (patching 35285 gdb minimal_symbol values)
FREEBUF(1)
FREEBUF(0)
<readmem: ffffffff804b8210, KVADDR, "kernel_config_data", 32768,
(ROE), 17e92b0>
<read_diskdump: addr: ffffffff804b8210 paddr: 4b8210 cnt: 3568>
read_diskdump: paddr/pfn: 4b8210/4b8 -> cache physical page: 4b8000
<read_diskdump: addr: ffffffff804b9000 paddr: 4b9000 cnt: 4096>
read_diskdump: paddr/pfn: 4b9000/4b9 -> cache physical page: 4b9000
<read_diskdump: addr: ffffffff804ba000 paddr: 4ba000 cnt: 4096>
read_diskdump: paddr/pfn: 4ba000/4ba -> cache physical page: 4ba000
<read_diskdump: addr: ffffffff804bb000 paddr: 4bb000 cnt: 4096>
read_diskdump: paddr/pfn: 4bb000/4bb -> cache physical page: 4bb000
<read_diskdump: addr: ffffffff804bc000 paddr: 4bc000 cnt: 4096>
read_diskdump: paddr/pfn: 4bc000/4bc -> cache physical page: 4bc000
<read_diskdump: addr: ffffffff804bd000 paddr: 4bd000 cnt: 4096>
read_diskdump: paddr/pfn: 4bd000/4bd -> cache physical page: 4bd000
<read_diskdump: addr: ffffffff804be000 paddr: 4be000 cnt: 4096>
read_diskdump: paddr/pfn: 4be000/4be -> cache physical page: 4be000
<read_diskdump: addr: ffffffff804bf000 paddr: 4bf000 cnt: 4096>
read_diskdump: paddr/pfn: 4bf000/4bf -> cache physical page: 4bf000
<read_diskdump: addr: ffffffff804c0000 paddr: 4c0000 cnt: 528>
read_diskdump: paddr/pfn: 4c0000/4c0 -> cache physical page: 4c0000
WARNING: could not find MAGIC_START!
GETBUF(248 -> 0)
FREEBUF(0)
GETBUF(64 -> 0)
<readmem: ffffffff80a59910, KVADDR, "possible", 64, (ROE), da00e0>
<read_diskdump: addr: ffffffff80a59910 paddr: a59910 cnt: 64>
read_diskdump: paddr/pfn: a59910/a59 -> cache physical page: a59000
cpu_possible_map:
<readmem: ffffffff8095d950, KVADDR, "present", 64, (ROE), da00e0>
<read_diskdump: addr: ffffffff8095d950 paddr: 95d950 cnt: 64>
read_diskdump: paddr/pfn: 95d950/95d -> cache physical page: 95d000
cpu_present_map: 2 3 6 7 10 11 14 15 18 19 22 23 26 27 30 31 34 35 38
39 42 43 46 47 50 51 54 55 58 59 62 63 66 67 70 71 74 75 78 79 82 83
86 87 90 91 94 95 98 99 102 103 106 107 110 111 114 115 118 119 122
123 126 127 130 131 134 135 138 139 142 143 146 147 150 151 154 155
158 159 162 163 166 167 170 171 174 175 178 179 182 183 186 187 190
191 194 195 198 199 202 203 206 207 210 211 214 215 218 219 222 223
226 227 230 231 234 235 238 239 242 243 246 247 250 251 254 255 258
259 262 263 266 267 270 271 274 275 278 279 282 283 286 287 290 291
294 295 298 299 302 303 306 307 310 311 314 315 318 319 322 323 326
327 330 331 334 335 338 339 342 343 346 347 350 351 354 355 358 359
362 363 366 367 370 371 374 375 378 379 382 383 386 387 390 391 394
395 398 399 402 403 406 407 410 411 414 415 418 419 422 423 426 427
430 431 434 435 438 439 442 443 446 447 450 451 454 455 458 459 462
463 466 467 470 471 474 475 478 479 482 483 486 487 490 491 494 495
498 499 502 503 506 507 510 511
<readmem: ffffffff808cbe90, KVADDR, "online", 64, (ROE), da00e0>
<read_diskdump: addr: ffffffff808cbe90 paddr: 8cbe90 cnt: 64>
read_diskdump: paddr/pfn: 8cbe90/8cb -> cache physical page: 8cb000
cpu_online_map:
FREEBUF(0)
<readmem: ffffffff80abaf00, KVADDR, "xtime", 16, (FOE), c5e430>
<read_diskdump: addr: ffffffff80abaf00 paddr: abaf00 cnt: 16>
read_diskdump: paddr/pfn: abaf00/aba -> cache physical page: aba000
<readmem: ffffffff806dc894, KVADDR, "init_uts_ns", 390, (ROE),
c5ea1c>
<read_diskdump: addr: ffffffff806dc894 paddr: 6dc894 cnt: 390>
read_diskdump: paddr/pfn: 6dc894/6dc -> cache physical page: 6dc000
base kernel version: 0.0.0
<readmem: ffffffff804a7000, KVADDR, "accessible check", 8, (ROE|Q),
7fff35eeba88>
<read_diskdump: addr: ffffffff804a7000 paddr: 4a7000 cnt: 8>
read_diskdump: paddr/pfn: 4a7000/4a7 -> cache physical page: 4a7000
<readmem: ffffffff804a7000, KVADDR, "readstring characters", 1499,
(ROE|Q), 7fff35eeaa70>
<read_diskdump: addr: ffffffff804a7000 paddr: 4a7000 cnt: 1499>
read_diskdump: paddr/pfn: 4a7000/4a7 -> physical page is cached:
4a7000
linux_banner:
紉;���
crash: vmlinux-2.6.27.19-5-default.debug and vmcore do not match!
Usage:
crash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
Enter "crash -h" for details.
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
--
Crash-utility mailing list
Crash-utility(a)redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility