[Crash-utility] crash and sles 9 GUEST dumps

After finding out how to get crash working with native sles 9 LKCD format dumps -- namely, build and use a debug vmlinux with appropriate flags to feed to crash -- I started looking into using crash on kernel dumps created for sles 9 guest domains. As compared to the LKCD format of native sles 9 dumps, those dumps are created using the new non-standard ELF format with section headers instead of program headers, which is the case with the xenctrl library now. Such formats are working for RHAS4U4 64bit guests, while I had to make minor modification to make it work for RHAS4U4 32bit guests as well. However, when it comes to sles 9 guests, crash seems to be having problems locating the stacks for each thread, with the exception of the CURRENT thread. (see below) It may well be that the stack pointers were not saved properly for sles 9 guests by the Xen library in the dump. I'll take a look into the dump and the xen library code to see if that is the case... Or is this the case of crash not looking in the right places for those stack pointers? Thanks, Daniel /dumps/new/sles/64bit$ /home/dli/bin/crash vmlinux-2.6.5-7.244-smp vmlinux.dbg DUMP10.1.230.112 crash 4.0-4.5 Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007 Red Hat, Inc. Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb 6.1 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: could not find MAGIC_START! please wait... (gathering task table data) crash: invalid kernel virtual address: 13 type: "fill_thread_info" crash: invalid kernel virtual address: f type: "fill_thread_info" crash: invalid kernel virtual address: 6 type: "fill_thread_info" crash: invalid kernel virtual address: c type: "fill_thread_info" crash: invalid kernel virtual address: 13 type: "fill_thread_info" crash: invalid kernel virtual address: c type: "fill_thread_info" crash: invalid kernel virtual address: 18 type: "fill_thread_info" crash: invalid kernel virtual address: f type: "fill_thread_info" crash: invalid kernel virtual address: 13 type: "fill_thread_info" crash: invalid kernel virtual address: 13 type: "fill_thread_info" crash: invalid kernel virtual address: 12 type: "fill_thread_info" crash: invalid kernel virtual address: 11 type: "fill_thread_info" crash: invalid kernel virtual address: 15 type: "fill_thread_info" crash: invalid kernel virtual address: f type: "fill_thread_info" crash: invalid kernel virtual address: 12 type: "fill_thread_info" crash: invalid kernel virtual address: 6e type: "fill_thread_info" crash: invalid kernel virtual address: 22 type: "fill_thread_info" crash: invalid kernel virtual address: 13 type: "fill_thread_info" crash: invalid kernel virtual address: f type: "fill_thread_info" crash: invalid kernel virtual address: c type: "fill_thread_info" crash: invalid kernel virtual address: c type: "fill_thread_info" crash: invalid kernel virtual address: 13 type: "fill_thread_info" crash: invalid kernel virtual address: f type: "fill_thread_info" crash: invalid kernel virtual address: f type: "fill_thread_info" crash: invalid kernel virtual address: 11 type: "fill_thread_info" crash: invalid kernel virtual address: 10 type: "fill_thread_info" crash: invalid kernel virtual address: c type: "fill_thread_info" crash: invalid kernel virtual address: 14 type: "fill_thread_info" crash: invalid kernel virtual address: 13 type: "fill_thread_info" crash: invalid kernel virtual address: 18 type: "fill_thread_info" crash: invalid kernel virtual address: f type: "fill_thread_info" crash: invalid kernel virtual address: e type: "fill_thread_info" crash: invalid kernel virtual address: f type: "fill_thread_info" please wait... (determining panic task) bt: invalid kernel virtual address: 13 type: "stack contents" bt: read of stack at 13 failed bt: invalid kernel virtual address: f type: "stack contents" bt: read of stack at f failed bt: invalid kernel virtual address: 6 type: "stack contents" bt: read of stack at 6 failed bt: invalid kernel virtual address: c type: "stack contents" bt: read of stack at c failed bt: invalid kernel virtual address: 13 type: "stack contents" bt: read of stack at 13 failed bt: invalid kernel virtual address: c type: "stack contents" bt: read of stack at c failed bt: invalid kernel virtual address: 18 type: "stack contents" bt: read of stack at 18 failed bt: invalid kernel virtual address: f type: "stack contents" bt: read of stack at f failed bt: invalid kernel virtual address: 13 type: "stack contents" bt: read of stack at 13 failed bt: invalid kernel virtual address: 13 type: "stack contents" bt: read of stack at 13 failed bt: invalid kernel virtual address: 12 type: "stack contents" bt: read of stack at 12 failed bt: invalid kernel virtual address: 11 type: "stack contents" bt: read of stack at 11 failed bt: invalid kernel virtual address: 15 type: "stack contents" bt: read of stack at 15 failed bt: invalid kernel virtual address: f type: "stack contents" bt: read of stack at f failed bt: invalid kernel virtual address: 12 type: "stack contents" bt: read of stack at 12 failed bt: invalid kernel virtual address: 6e type: "stack contents" bt: read of stack at 6e failed bt: invalid kernel virtual address: 22 type: "stack contents" bt: read of stack at 22 failed bt: invalid kernel virtual address: 13 type: "stack contents" bt: read of stack at 13 failed bt: invalid kernel virtual address: f type: "stack contents" bt: read of stack at f failed bt: invalid kernel virtual address: c type: "stack contents" bt: read of stack at c failed bt: invalid kernel virtual address: c type: "stack contents" bt: read of stack at c failed bt: invalid kernel virtual address: 13 type: "stack contents" bt: read of stack at 13 failed bt: invalid kernel virtual address: f type: "stack contents" bt: read of stack at f failed bt: invalid kernel virtual address: f type: "stack contents" bt: read of stack at f failed bt: invalid kernel virtual address: 11 type: "stack contents" bt: read of stack at 11 failed bt: invalid kernel virtual address: 10 type: "stack contents" bt: read of stack at 10 failed bt: invalid kernel virtual address: c type: "stack contents" bt: read of stack at c failed bt: invalid kernel virtual address: 14 type: "stack contents" bt: read of stack at 14 failed bt: invalid kernel virtual address: 13 type: "stack contents" bt: read of stack at 13 failed bt: invalid kernel virtual address: 18 type: "stack contents" bt: read of stack at 18 failed bt: invalid kernel virtual address: f type: "stack contents" bt: read of stack at f failed bt: invalid kernel virtual address: e type: "stack contents" bt: read of stack at e failed bt: invalid kernel virtual address: f type: "stack contents" bt: read of stack at f failed KERNEL: vmlinux-2.6.5-7.244-smp DEBUG KERNEL: vmlinux.dbg (2.6.5-7.244-default) DUMPFILE: DUMP10.1.230.112 CPUS: 1 DATE: Thu Jul 26 14:34:46 2007 UPTIME: 213503982284 days, 21:34:00 LOAD AVERAGE: 0.01, 0.12, 0.07 TASKS: 34 NODENAME: linux RELEASE: 2.6.5-7.244-smp VERSION: #1 SMP Mon Dec 12 18:32:25 UTC 2005 MACHINE: x86_64 (2793 Mhz) MEMORY: 1015808 GB PANIC: "" PID: 0 COMMAND: " " TASK: ffffffff803d2800 (1 of 2) [THREAD_INFO: ffffffff80590000] CPU: 0 STATE: TASK_RUNNING (ACTIVE) WARNING: panic task not found crash> bt PID: 0 TASK: ffffffff803d2800 CPU: 0 COMMAND: "<80>^L" #0 [ffffffff80591ef0] schedule at ffffffff801394e4 #1 [ffffffff80591f98] default_idle at ffffffff8010f1c0 #2 [ffffffff80591fc8] cpu_idle at ffffffff8010f65a crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM 4389578663200 [<80>^L] 0 0 0 ffffffff803d2808 ?? 0.0 0 0 [swapper] 1 0 0 10017e1f2c8 ?? 0.1 640 304 init 2 -1 0 10017e1e9a8 ?? 0.0 0 0 [migration/0] 3 -1 0 10017e1e088 ?? 0.0 0 0 [ksoftirqd/0] 4 -1 0 10001b712d8 ?? 0.0 0 0 [events/0] 5 -1 0 10001b709b8 ?? 0.0 0 0 [khelper] 6 -1 0 10001b70098 ?? 0.0 0 0 [kacpid] 25 -1 0 10017dd72e8 ?? 0.0 0 0 [kblockd/0] 47 -1 0 10017dd69c8 ?? 0.0 0 0 [pdflush] 48 -1 0 10017dd60a8 ?? 0.0 0 0 [pdflush] 49 -1 0 100178272f8 ?? 0.0 0 0 [kswapd0] 50 -1 0 100178269d8 ?? 0.0 0 0 [aio/0] 1295 -1 0 100178260b8 ?? 0.0 0 0 [kseriod] 2077 -1 0 10017897308 ?? 0.0 0 0 [reiserfs/0] 2744 -1 0 10014de9488 ?? 0.0 0 0 [khubd] 3077 -1 0 10015aa13c8 ?? 0.2 2560 608 hwscand 3693 -1 0 100164e1348 ?? 0.2 3568 816 syslogd 3696 -1 0 10015180208 ?? 0.3 2744 1112 klogd 3721 -1 0 10015b0eab8 ?? 0.2 3536 628 resmgrd 3722 -1 0 10015e6e1c8 ?? 0.2 4564 640 portmap 3803 -1 0 10015d49368 ?? 0.6 20036 2340 master 3814 -1 0 10015daea58 ?? 0.6 20100 2312 pickup 3815 -1 0 10016c5a0d8 ?? 0.6 20144 2364 qmgr 3861 -1 0 10016ca2a08 ?? 0.7 26800 2932 sshd 4022 -1 0 10014c42b48 ?? 0.2 6804 924 cron 4057 -1 0 100178960c8 ?? 0.2 2484 612 agetty 4058 -1 0 10016c5b318 ?? 0.5 21864 1772 login 4059 -1 0 10016ca3328 ?? 0.2 7012 936 mingetty 4060 -1 0 10015fb5398 ?? 0.2 7012 936 mingetty 4061 -1 0 10014cc6238 ?? 0.2 7012 936 mingetty 4062 -1 0 10015b0f3d8 ?? 0.2 7012 936 mingetty 4063 -1 0 100151e7458 ?? 0.2 7012 936 mingetty 4152 -1 0 10016a180f8 ?? 0.8 12716 2992 bash crash> bt 10015180208 PID: 3696 TASK: 10015180208 CPU: 0 COMMAND: "klogd" *bt: invalid kernel virtual address: 12 type: "stack contents"* bt: read of stack at 12 failed crash> task 10015180208 PID: 3696 TASK: 10015180208 CPU: 0 COMMAND: "klogd" struct task_struct { *state = 1099873050624,* * thread_info = 0x12,* usage = { counter = 320 }, flags = 0, ptrace = 502511173631, lock_depth = 120, prio = 0, static_prio = 1048832, run_list = { next = 0x200200, prev = 0x0 }, array = 0x50fe72e6, sleep_avg = 1, interactive_credit = 67616128664, timestamp = 67616128664, last_ran = 0, activated = 0, policy = 18446744073709551615, cpus_allowed = 18446744073709551615, time_slice = 150, first_time_slice = 0, tasks = { next = 0x10015b0eb48, prev = 0x100164e13d8 }, ptrace_children = { next = 0x100151802a8, prev = 0x100151802a8 }, ptrace_list = { next = 0x100151802b8, prev = 0x100151802b8 }, mm = 0x1001546c500, active_mm = 0x1001546c500, binfmt = 0xffffffff803e70c0, exit_state = 0, exit_code = 0, exit_signal = 17, pdeath_signal = 0, personality = 0, did_exec = 0, pid = 3696, tgid = 3696, real_parent = 0x10017e1f2c0, parent = 0x10017e1f2c0, children = { next = 0x10015180320, prev = 0x10015180320 }, sibling = { next = 0x10015b0ebe0, prev = 0x100164e1470 }, group_leader = 0x10015180200, pids = {{ pid_chain = { next = 0x10015180370, prev = 0x10015180370 }, pidptr = 0x10015180360, pid = { nr = 3696, count = { counter = 1 }, task = 0x10015180200, task_list = { next = 0x10015180348, prev = 0x10015180348 }, hash_chain = { next = 0x10017827470, prev = 0x10016ca2b80 } } }, { pid_chain = { next = 0x100151803b8, prev = 0x100151803b8 }, pidptr = 0x100151803a8, pid = { nr = 3696, count = { counter = 1 }, task = 0x10015180200, task_list = { next = 0x10015180390, prev = 0x10015180390 }, hash_chain = { next = 0x100178274b8, prev = 0x10016ca2bc8 } } }, { pid_chain = { next = 0x10015180400, prev = 0x10015180400 }, pidptr = 0x100151803f0, pid = { nr = 3696, count = { counter = 1 }, task = 0x10015180200, task_list = { next = 0x100151803d8, prev = 0x100151803d8 }, hash_chain = { next = 0x10001949240, prev = 0x10016ca2c10 } } }, { pid_chain = { next = 0x10015180448, prev = 0x10015180448 }, pidptr = 0x10015180438, pid = { nr = 3696, count = { counter = 1 }, task = 0x10015180200, task_list = { next = 0x10015180420, prev = 0x10015180420 }, hash_chain = { next = 0x10001949340, prev = 0x10016ca2c58 } } }}, wait_chldexit = { lock = { lock = 1 }, task_list = { next = 0x10015180470, prev = 0x10015180470 } }, vfork_done = 0x0, set_child_tid = 0x2a95894b90, clear_child_tid = 0x2a95894b90, rt_priority = 0, it_real_value = 0, it_prof_value = 0, it_virt_value = 0, it_real_incr = 0, it_prof_incr = 0, it_virt_incr = 0, real_timer = { entry = { next = 0x100100, prev = 0x200200 }, expires = 29143, lock = { lock = 1 }, magic = 1267182958, function = 0xffffffff80141b50 <it_real_fn>, data = 1099865522688, base = 0x0 }, utime = 0, stime = 4, cutime = 0, cstime = 0, nvcsw = 13, nivcsw = 2, cnvcsw = 0, cnivcsw = 0, start_time = 53888910424, min_flt = 105, maj_flt = 0, cmin_flt = 0, cmaj_flt = 0, uid = 0, euid = 0, suid = 0, fsuid = 0, gid = 0, egid = 0, sgid = 0, fsgid = 0, group_info = 0xffffffff803e2a00, cap_effective = 4294967039, cap_inheritable = 0, cap_permitted = 4294967039, keep_capabilities = 0, user = 0xffffffff803e29a0, rlim = {{ rlim_cur = 18446744073709551615, rlim_max = 18446744073709551615 }, { rlim_cur = 18446744073709551615, rlim_max = 18446744073709551615 }, { rlim_cur = 18446744073709551615, rlim_max = 18446744073709551615 }, { rlim_cur = 8388608, rlim_max = 18446744073709551615 }, { rlim_cur = 0, rlim_max = 18446744073709551615 }, { rlim_cur = 18446744073709551615, rlim_max = 18446744073709551615 }, { rlim_cur = 3071, rlim_max = 3071 }, { rlim_cur = 1024, rlim_max = 1024 }, { rlim_cur = 18446744073709551615, rlim_max = 18446744073709551615 }, { rlim_cur = 18446744073709551615, rlim_max = 18446744073709551615 }, { rlim_cur = 18446744073709551615, rlim_max = 18446744073709551615 }, { rlim_cur = 1024, rlim_max = 1024 }, { rlim_cur = 819200, rlim_max = 819200 }}, used_math = 0, rcvd_sigterm = 0, oomkilladj = 0, comm = "klogd\000roc\000\000\000\000\000\000", link_count = 0, total_link_count = 0, sysvsem = { undo_list = 0x0 }, thread = { rsp0 = 1099873058120, rsp = 548682070920, userrsp = 182897429248, fs = 0, gs = 0, es = 0, ds = 0, fsindex = 0, gsindex = 0, debugreg0 = 0, debugreg1 = 0, debugreg2 = 0, debugreg3 = 0, debugreg6 = 0, debugreg7 = 0, cr2 = 0, trap_no = 0, error_code = 0, i387 = { fxsave = { cwd = 0, swd = 0, twd = 0, fop = 0, rip = 0, rdp = 281470681751424, mxcsr = 0, mxcsr_mask = 0, st_space = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, xmm_space = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 , 0, 0, 0}, padding = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0} } }, ioperm = 0, io_bitmap_ptr = 0x0, tls_array = {0, 0, 0} }, fs = 0x10014c7a180, files = 0x10001a114c0, namespace = 0x100154fb900, signal = 0x10015184600, sighand = 0x0, blocked = { sig = {0} }, real_blocked = { sig = {1099865524632} }, pending = { list = { next = 0x10015180998, prev = 0x0 }, signal = { sig = {0} } }, sas_ss_sp = 0, sas_ss_size = 0, notifier = 0, notifier_data = 0x0, notifier_mask = 0x0, security = 0x600000005, parent_exec_id = 1, self_exec_id = 1, alloc_lock = { lock = 1 }, proc_lock = { lock = 0 }, switch_lock = { lock = 0 }, journal_info = 0x0, reclaim_state = 0x10015469180, proc_dentry = 0x0, backing_dev_info = 0x10015b40940, io_context = 0x0, ptrace_message = 0, last_siginfo = 0x0, io_wait = 0xac9, rchar = 2292, wchar = 3, syscr = 32, syscw = 475, acct_rss_mem1 = 2743, acct_vm_mem1 = 4, acct_stimexpd = 4294967297, ckrm_tsklock = { lock = 0 }, ckrm_celock = { lock = 0 }, ce_data = 0xffffffff804f3f20, taskclass = 0x100164e1bc8, taskclass_link = { next = 0x10015b0f338, prev = 0xffffffff80537940 }, cpu_class = 0x0, demand_stat = { run = 0, total = 61218488692, last_sleep = 32000000, recalc_interval = 0, cpu_demand = 105133020 }, delays = { waitcpu_total = 3647587, runcpu_total = 23603870, iowait_total = 0, mem_iowait_total = 4294967311, runs = 0, num_iowaits = 0, num_memwaits = 0, splpar_total = 1431654400 }, map_base = 0, mempolicy = 0x0, il_next = 0, audit = 0x10 } crash>