5:51 p.m.
It seems the problem is not one with guest dump, but the version of SLES.
After upgrading my NATIVE SLES 9 system to SP 3, exactly the same
problem happened while trying to use 'crash' on the live system, with a
debug linux kernel ('vmlinux.dbg' below) built on the same system from
matching 'kernel-source' package. (During this upgrade, the linux kernel
changed from 2.6.5-7.97-smp to 2.6.5-7.244-smp, the same as that on the
guest.)
Has anyone else seen this?
tst037:/boot # ./crash vmlinux-2.6.5-7.244-smp vmlinux.dbg /dev/mem
crash 4.0-4.6
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: could not find MAGIC_START!
KERNEL: vmlinux-2.6.5-7.244-smp
DEBUG KERNEL: vmlinux.dbg (2.6.5-7.244-default)
DUMPFILE: /dev/mem
CPUS: 4
DATE: Tue Aug 28 10:57:37 2007
UPTIME: 213503982284 days, 23:42:22
LOAD AVERAGE: 0.00, 0.01, 0.22
TASKS: 104
NODENAME: tst037
RELEASE: 2.6.5-7.244-smp
VERSION: #1 SMP Mon Dec 12 18:32:25 UTC 2005
MACHINE: x86_64 (2793 Mhz)
MEMORY: 1015808 GB
PID: 14432
COMMAND: "crash"
TASK: 1012fbb0db8 [THREAD_INFO: 5]
CPU: 3
STATE: (unknown)
crash> bt
PID: 14432 TASK: 1012fbb0db8 CPU: 3 COMMAND: "crash"
(active)
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 ffffffff803d2800 RU 0.0 4398751044160
4389851194656 [200^L]
0 -1 1 10037e1e9a0 RU 0.0 4398425096768
8800221827680 [200^L]
0 -1 2 10037e1e080 RU 0.0 4398425096768
8800221827680 [200^L]
0 -1 3 100bff6b2d0 RU 0.0 4398425096768
8800221827680 [200^L]
0 0 3 ffffffff803d2808 ?? 0.0 0 0 [swapper]
1 0 3 10037e1f2c8 ?? 0.0 640 320 init
2 -1 3 100bff6a9b8 ?? 0.0 0 0 [migration/0]
3 -1 3 100bff6a098 ?? 0.0 0 0 [ksoftirqd/0]
4 -1 3 100bfe552e8 ?? 0.0 0 0 [migration/1]
5 -1 3 100bfe549c8 ?? 0.0 0 0 [ksoftirqd/1]
6 -1 3 100bfe540a8 ?? 0.0 0 0 [migration/2]
7 -1 3 100bfe5f2f8 ?? 0.0 0 0 [ksoftirqd/2]
8 -1 3 100bfe5e9d8 ?? 0.0 0 0 [migration/3]
9 -1 3 100bfe5e0b8 ?? 0.0 0 0 [ksoftirqd/3]
10 -1 3 1000a7ef308 ?? 0.0 0 0 [events/0]
11 -1 3 1000a7ee9e8 ?? 0.0 0 0 [events/1]
12 -1 3 1000a7ee0c8 ?? 0.0 0 0 [events/2]
13 -1 3 1000a7f9318 ?? 0.0 0 0 [events/3]
14 -1 3 1000a7f89f8 ?? 0.0 0 0 [khelper]
15 -1 3 1000a7f80d8 ?? 0.0 0 0 [kacpid]
82 -1 3 10037c7f328 ?? 0.0 0 0 [kblockd/0]
83 -1 3 10037c7ea08 ?? 0.0 0 0 [kblockd/1]
84 -1 3 10037c7e0e8 ?? 0.0 0 0 [kblockd/2]
85 -1 3 10037d0d338 ?? 0.0 0 0 [kblockd/3]
110 -1 3 10037d0ca18 ?? 0.0 0 0 [pdflush]
111 -1 3 10037d5b358 ?? 0.0 0 0 [pdflush]
112 -1 3 10037d36108 ?? 0.0 0 0 [kswapd0]
113 -1 3 10037d63368 ?? 0.0 0 0 [aio/0]
114 -1 3 10037d5aa38 ?? 0.0 0 0 [aio/1]
115 -1 3 10037d62a48 ?? 0.0 0 0 [aio/2]
116 -1 3 10037d5a118 ?? 0.0 0 0 [aio/3]
1355 -1 3 10037dbaa68 ?? 0.0 0 0 [kseriod]
2050 -1 3 1013fde0238 ?? 0.0 0 0 [scsi_eh_0]
2224 -1 3 1013fde5488 ?? 0.0 0 0 [reiserfs/0]
2225 -1 3 1013fc2d448 ?? 0.0 0 0 [reiserfs/1]
2226 -1 3 1013fc954d8 ?? 0.0 0 0 [reiserfs/2]
2227 -1 3 10037dcc168 ?? 0.0 0 0 [reiserfs/3]
4282 -1 3 10132400a58 ?? 0.0 0 0 [khubd]
4865 -1 3 10132a2f398 ?? 0.0 2560 608 hwscand
6086 -1 3 10037dbea98 ?? 0.0 2512 692 dhcpcd
7143 -1 3 10132eb9338 ?? 0.0 2484 620 irqbalance
8276 -1 3 1012dff89e8 ?? 0.0 5716 924 syslogd
8282 -1 3 1012d1ff678 ?? 0.0 2748 1120 klogd
8340 -1 3 1012d2129c8 ?? 0.0 4580 724 portmap
8342 -1 3 1013971b2d8 ?? 0.0 21460 2336 slpd
8343 -1 3 1012dc17358 ?? 0.1 22396 3696 resmgrd
8612 -1 3 1012197f398 ?? 0.1 31732 7452 cupsd
8625 -1 3 1012f8ed3c8 ?? 0.1 30476 3176 sshd
8744 -1 3 1012dc16118 ?? 0.1 23640 3268 powersaved
8797 -1 3 1012dff80c8 ?? 0.2 99996 7952 slapd
8820 -1 3 10132a2e158 ?? 0.2 99996 7952 slapd
8982 -1 3 1012156aa48 ?? 0.2 99996 7952 slapd
9523 -1 3 1012e03e0f8 ?? 0.0 20104 2376 master
9539 -1 3 10129c60328 ?? 0.1 22640 3996 qmgr
9555 -1 3 1012d66ccf8 ?? 0.2 99996 7952 slapd
9558 -1 3 1012b6c0338 ?? 0.2 99996 7952 slapd
9656 -1 3 10132eb8a18 ?? 0.0 17884 2040 cron
9661 -1 3 1012f8ecaa8 ?? 0.1 35180 4028 nscd
9662 -1 3 1012cc595e8 ?? 0.1 35180 4028 nscd
9663 -1 3 10037dbf3b8 ?? 0.1 35180 4028 nscd
9664 -1 3 1012ebf75a8 ?? 0.1 35180 4028 nscd
9665 -1 3 10122aa6bc8 ?? 0.1 35180 4028 nscd
9666 -1 3 1013fcdd598 ?? 0.1 35180 4028 nscd
9667 -1 3 10122aa62a8 ?? 0.1 35180 4028 nscd
9773 -1 3 10129c60c48 ?? 0.0 8936 980 kdm
9780 -1 3 10122a3f3b8 ?? 0.5 40900 24864 X
9783 -1 3 1012dcdcd98 ?? 0.1 35220 3400 kdm
9832 -1 3 1013a841678 ?? 0.0 2720 768 mingetty
9833 -1 3 10037dcca88 ?? 0.0 2720 768 mingetty
9834 -1 3 1013a840d58 ?? 0.0 2720 768 mingetty
9835 -1 3 1012d66c3d8 ?? 0.0 2720 768 mingetty
9836 -1 3 1012fa8e428 ?? 0.0 2720 768 mingetty
9837 -1 3 1013fcdcc78 ?? 0.0 2720 768 mingetty
9865 -1 3 10132a2ea78 ?? 0.0 8000 1808 kde
9898 -1 3 101347db548 ?? 0.0 6288 932 gpg-agent
9899 -1 3 1012a4203f8 ?? 0.0 21196 2256 ssh-agent
9923 -1 3 10129542378 ?? 0.3 60192 13740 kdeinit
9926 -1 3 1013971a098 ?? 0.3 60972 13780 kdeinit
9928 -1 3 10132eb80f8 ?? 0.3 63436 15452 kdeinit
9931 -1 3 1012156a128 ?? 0.4 78752 21480 kdeinit
9939 -1 3 101226cf2c8 ?? 0.2 35848 8404 artsd
10009 -1 3 1012d034a08 ?? 0.5 94832 27044 kdeinit
10010 -1 3 101223d1688 ?? 0.0 2480 484 kwrapper
10012 -1 3 101226ce088 ?? 0.4 70116 20084 kdeinit
10013 -1 3 1012b8f8d88 ?? 0.4 78556 23520 kdeinit
10015 -1 3 101347dac28 ?? 0.4 70120 22384 kdeinit
10017 -1 3 101237a14f8 ?? 0.4 69476 20816 kdeinit
10019 -1 3 1012e03ea18 ?? 0.5 79308 25180 kdeinit
10021 -1 3 101295435b8 ?? 0.5 86836 25992 kdeinit
10022 -1 3 10134215378 ?? 0.3 62000 14332 kdeinit
10026 -1 3 1013fde0b58 ?? 0.4 75996 23024 kdeinit
10030 -1 3 10037dbe178 ?? 0.5 77916 26192 kdeinit
10031 -1 3 1013f34aad8 ?? 0.4 72728 22036 kpowersave
10032 -1 3 101295963e8 ?? 0.5 79788 26072 suseplugger
10033 -1 3 1013fde1478 ?? 0.4 71832 22452 susewatcher
10035 -1 3 1012156b368 ?? 0.4 75900 23428 kdeinit
10040 -1 3 1013a840438 ?? 0.5 79788 26072 suseplugger
10050 -1 3 10037dbb388 ?? 0.5 81420 25236 kdeinit
10051 -1 3 1012fbb0498 ?? 0.1 8420 2760 bash
10087 -1 3 1013f34b3f8 ?? 0.1 38132 3560 sshd
10090 -1 3 1012a421638 ?? 0.1 9324 2976 bash
10196 -1 3 1012fbb16d8 ?? 0.4 64700 19300 kdesktop_lock
14432 -1 3 1012fbb0db8 ?? 2.4 74380 125620 crash
14437 -1 3 1012d66d618 ?? 0.0 4284 1252 crash
25011 -1 3 1012b8f96a8 ?? 0.0 16920 2180 pickup
crash> bt 1012d1ff678
PID: 8282 TASK: 1012d1ff678 CPU: 3 COMMAND: "klogd"
bt: invalid kernel virtual address: 11 type: "stack contents"
bt: read of stack at 11 failed
crash> task 1012d1ff678
PID: 8282 TASK: 1012d1ff678 CPU: 3 COMMAND: "klogd"
struct task_struct {
state = 1104534274048,
thread_info = 0x11,
usage = {
counter = 320
},
flags = 0,
ptrace = 498216206335,
lock_depth = 120,
prio = 0,
static_prio = 1048832,
run_list = {
next = 0x200200,
prev = 0x0
},
array = 0x11de3f11,
sleep_avg = 6,
interactive_credit = 252089242482,
timestamp = 252089242482,
last_ran = 0,
activated = 0,
policy = 18446744073709551615,
cpus_allowed = 18446744073709551615,
time_slice = 7,
first_time_slice = 1,
tasks = {
next = 0x1012d212a58,
prev = 0x1012dff8a78
crash> q