April 2006 - Crash-utility - Crash Utility List Archives

by Dave Anderson

20 years

4
12
0 / 0

by Dave Anderson

changes the pid_hash chain from: static struct hlist_head *pid_hash[PIDTYPE_MAX]; to: static struct hlist_head *pid_hash; The pid_hash infrastructure is what the crash utility uses to gather the set of running tasks during initialization. The change above breaks it, and crash will fail with an error message stating: crash: cannot determine pid_hash array dimensions As the PID handling has changed over the years, there have been several task-gathering functions in crash, the latest of which is refresh_hlist_task_table(). I don't know whether it will be possible to shoe-horn the new scheme to fit this current crash function, or whether yet another new one will have to be written. If it's the latter, it may take a while... Dave

20 years

3
4
0 / 0

Re: [Crash-utility] crash warnings

by Dave Anderson

Badari, What happens when you disassemble sys_read() on a live system running that particular kernel? Use "dis sys_read" as the crash command. Thanks, Dave

20 years

2
2
0 / 0

Re: [Crash-utility] crash warnings

by Dave Anderson

> > Given that "crash" runs fine on live machine, I am going to assume > > that its a problem with kdump format for now :( > > > > No -- wait -- please don't! ;-) > > I now have the dumpfile Badari has been referring to in this thread. While there are a couple crash-utility-related items to address, the primary problem may in fact be related to kdump. This issue is that crash is getting confused because of bizarre disassembly of kernel text. Here's an example, which takes crash out of the picture -- I will simply use gdb alone. If I run gdb alone on the vmlinux file, and disassemble a kernel function, everything looks normal: # gdb vmlinux GNU gdb Red Hat Linux (6.3.0.0-1.63rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1". (gdb) disassemble sys_read Dump of assembler code for function sys_read: 0xffffffff8017b991 <sys_read+0>: push %r13 0xffffffff8017b993 <sys_read+2>: mov %rsi,%r13 0xffffffff8017b996 <sys_read+5>: push %r12 0xffffffff8017b998 <sys_read+7>: mov $0xfffffffffffffff7,%r12 0xffffffff8017b99f <sys_read+14>: push %rbp 0xffffffff8017b9a0 <sys_read+15>: mov %rdx,%rbp 0xffffffff8017b9a3 <sys_read+18>: push %rbx 0xffffffff8017b9a4 <sys_read+19>: sub $0x18,%rsp 0xffffffff8017b9a8 <sys_read+23>: lea 0x14(%rsp),%rsi 0xffffffff8017b9ad <sys_read+28>: callq 0xffffffff8017bf23 <fget_light> 0xffffffff8017b9b2 <sys_read+33>: test %rax,%rax 0xffffffff8017b9b5 <sys_read+36>: mov %rax,%rbx 0xffffffff8017b9b8 <sys_read+39>: je 0xffffffff8017b9f1 <sys_read+96> 0xffffffff8017b9ba <sys_read+41>: mov 0x38(%rax),%rax 0xffffffff8017b9be <sys_read+45>: lea 0x8(%rsp),%rcx 0xffffffff8017b9c3 <sys_read+50>: mov %rbp,%rdx 0xffffffff8017b9c6 <sys_read+53>: mov %r13,%rsi 0xffffffff8017b9c9 <sys_read+56>: mov %rbx,%rdi 0xffffffff8017b9cc <sys_read+59>: mov %rax,0x8(%rsp) 0xffffffff8017b9d1 <sys_read+64>: callq 0xffffffff8017b52f <vfs_read> 0xffffffff8017b9d6 <sys_read+69>: mov %rax,%r12 0xffffffff8017b9d9 <sys_read+72>: mov 0x8(%rsp),%rax 0xffffffff8017b9de <sys_read+77>: mov %rax,0x38(%rbx) 0xffffffff8017b9e2 <sys_read+81>: cmpl $0x0,0x14(%rsp) 0xffffffff8017b9e7 <sys_read+86>: je 0xffffffff8017b9f1 <sys_read+96> 0xffffffff8017b9e9 <sys_read+88>: mov %rbx,%rdi 0xffffffff8017b9ec <sys_read+91>: callq 0xffffffff8017be05 <fput> 0xffffffff8017b9f1 <sys_read+96>: add $0x18,%rsp 0xffffffff8017b9f5 <sys_read+100>: mov %r12,%rax 0xffffffff8017b9f8 <sys_read+103>: pop %rbx 0xffffffff8017b9f9 <sys_read+104>: pop %rbp 0xffffffff8017b9fa <sys_read+105>: pop %r12 0xffffffff8017b9fc <sys_read+107>: pop %r13 0xffffffff8017b9fe <sys_read+109>: retq End of assembler dump. (gdb) Now, if I bring the suspect kdump vmcore into the picture, check this out: # gdb vmlinux vmcore GNU gdb Red Hat Linux (6.3.0.0-1.63rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1". #0 mwait_idle () at include/asm/thread_info.h:63 63 include/asm/thread_info.h: No such file or directory. in include/asm/thread_info.h (gdb) disassemble sys_read Dump of assembler code for function sys_read: 0xffffffff8017b991 <sys_read+0>: add %cl,0xffffffffffffff83(%rax) 0xffffffff8017b994 <sys_read+3>: (bad) 0xffffffff8017b995 <sys_read+4>: sbb %cl,0xffffffffffffff89(%rax) 0xffffffff8017b998 <sys_read+7>: callq 0xffffffffdc5916f8 0xffffffff8017b99d <sys_read+12>: pop %r13 0xffffffff8017b99f <sys_read+14>: retq 0xffffffff8017b9a0 <sys_read+15>: mov 1862306(%rip),%eax # 0xffffffff80342448 <files_stat+8> 0xffffffff8017b9a6 <sys_read+21>: retq 0xffffffff8017b9a7 <sys_read+22>: push %rbx 0xffffffff8017b9a8 <sys_read+23>: cmp %rdi,(%rdi) 0xffffffff8017b9ab <sys_read+26>: mov %rdi,%rbx 0xffffffff8017b9ae <sys_read+29>: je 0xffffffff8017b9db <sys_read+74> 0xffffffff8017b9b0 <sys_read+31>: mov $0xffffffff80456b80,%rdi 0xffffffff8017b9b7 <sys_read+38>: callq 0xffffffff802d0aad <__down_interruptible+75> 0xffffffff8017b9bc <sys_read+43>: mov (%rbx),%rdx 0xffffffff8017b9bf <sys_read+46>: mov 0x8(%rbx),%rax 0xffffffff8017b9c3 <sys_read+50>: mov %rax,0x8(%rdx) 0xffffffff8017b9c7 <sys_read+54>: mov %rdx,(%rax) 0xffffffff8017b9ca <sys_read+57>: mov %rbx,0x8(%rbx) 0xffffffff8017b9ce <sys_read+61>: mov %rbx,(%rbx) 0xffffffff8017b9d1 <sys_read+64>: movl $0x1,2994597(%rip) # 0xffffffff80456b80 <files_lock> 0xffffffff8017b9db <sys_read+74>: pop %rbx 0xffffffff8017b9dc <sys_read+75>: retq 0xffffffff8017b9dd <sys_read+76>: push %rbx 0xffffffff8017b9de <sys_read+77>: mov %rdi,%rbx 0xffffffff8017b9e1 <sys_read+80>: lock decl 0x28(%rdi) 0xffffffff8017b9e5 <sys_read+84>: sete %al 0xffffffff8017b9e8 <sys_read+87>: test %al,%al 0xffffffff8017b9ea <sys_read+89>: je 0xffffffff8017ba21 <sys_write+34> 0xffffffff8017b9ec <sys_read+91>: mov 2787309(%rip),%rax # 0xffffffff804241e0 <security_ops> 0xffffffff8017b9f3 <sys_read+98>: callq *0x1e8(%rax) 0xffffffff8017b9f9 <sys_read+104>: mov %rbx,%rdi 0xffffffff8017b9fc <sys_read+107>: callq 0xffffffff8017b9a7 <sys_read+22> End of assembler dump. (gdb) I see exactly the same behaviour using the embedded gdb-6.1 in the crash utility. I'm digging into it more as we speak, but I just wanted to throw this out for now. Thanks, Dave

20 years

1
0
0 / 0

Re: [Crash-utility] crash warnings

by Dave Anderson

Hi Badari, Just to be clear, you stated: > crash -d1 shows me > > crash: get_netdump_panic_task: crashing_cpu: 1 > crash: get_netdump_panic_task: failed > crash: get_active_set_panic_task: failed > and I presumed: > Now, since after the "crash: get_active_set_panic_task: failed" > message, there are no more error messages, so panic_search() > must have determined that the idle thread (swapper) on cpu 0 was > in fact the running task that took the sysrq. > But, are there any other error messages after the "get_active_set_panic_task: failed"? Like for example: crash: panic_search: failed (via foreach bt) crash: get_panic_context: panic task not found or did you see: crash: panic_search: ffffffff80335340 (via foreach bt) Thanks, Dave

20 years

1
0
0 / 0

crash warnings

by Badari Pulavarty

Hi, I get following crash warnings on x86-64 machine. Wondering why ? And also, its not showing stacks correctly. Thanks, Badari # ./crash /var/log/dump/2006-04-24-08:02/vmcore /usr/src/linux/vmlinux crash 4.0-2.23 Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc. Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005 Fujitsu Limited Copyright (C) 2005 NEC Corporation Copyright (C) 1999, 2002 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb 6.1 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame WARNING: possibly bogus exception frame KERNEL: /usr/src/linux/vmlinux DUMPFILE: /var/log/dump/2006-04-24-08:02/vmcore CPUS: 2 DATE: Mon Apr 24 08:02:03 2006 UPTIME: 00:06:31 LOAD AVERAGE: 0.00, 0.00, 0.00 TASKS: 63 NODENAME: elm3a242 RELEASE: 2.6.16-20-smp VERSION: #1 SMP Mon Apr 10 04:51:13 UTC 2006 MACHINE: x86_64 (3000 Mhz) MEMORY: 4.6 GB PANIC: "SysRq : Trigger a crashdump" PID: 0 COMMAND: "swapper" TASK: ffffffff80335340 (1 of 2) [THREAD_INFO: ffffffff8045c000] CPU: 0 STATE: TASK_RUNNING (ACTIVE) crash> bt PID: 0 TASK: ffffffff80335340 CPU: 0 COMMAND: "swapper" #0 [ffffffff8045dee8] schedule at ffffffff802cf6fa

20 years

2
5
0 / 0

[RFC] Patch to fix broken net command

by Rachita Kothiyal

Hi Dave Following changes in the 2.6.14 kernel in the files_struct, the 'net' command was broken. The following patch fixes it. I have done basic testing on kernels before and after 2.6.14 to ensure that backward compatibility is maintained. Kindly review and provide your comments. Thanks Rachita o From 2.6.14 kernel onwards, there were changes introduced in the files_struct. This patch accomodates those kernel changes in crash to keep the 'net' command working. Signed-off-by: Rachita Kothiyal <rachita(a)in.ibm.com> --- net.c | 57 ++++++++++++++++++++++++++++++++++++++------------------- 1 files changed, 38 insertions(+), 19 deletions(-) diff -puN net.c~fix-net-command net.c --- crash-4.0-2.23/net.c~fix-net-command 2006-04-17 19:16:18.000000000 +0530 +++ crash-4.0-2.23-rachita/net.c 2006-04-17 20:41:16.000000000 +0530 @@ -1082,7 +1082,7 @@ dump_sockets(ulong flag, struct referenc void dump_sockets_workhorse(ulong task, ulong flag, struct reference *ref) { - ulong files_struct_addr = 0; + ulong files_struct_addr = 0, fdtable_addr = 0; int max_fdset = 0; int max_fds = 0; ulong open_fds_addr = 0; @@ -1114,32 +1114,51 @@ dump_sockets_workhorse(ulong task, ulong sizeof(void *), "task files contents", FAULT_ON_ERROR); if (files_struct_addr) { - readmem(files_struct_addr + OFFSET(files_struct_max_fdset), - KVADDR, &max_fdset, sizeof(int), - "files_struct max_fdset", FAULT_ON_ERROR); - - readmem(files_struct_addr + OFFSET(files_struct_max_fds), - KVADDR, &max_fds, sizeof(int), "files_struct max_fds", - FAULT_ON_ERROR); - } + if (VALID_MEMBER(files_struct_max_fdset)) { + readmem(files_struct_addr + OFFSET(files_struct_max_fdset), + KVADDR, &max_fdset, sizeof(int), + "files_struct max_fdset", FAULT_ON_ERROR); + readmem(files_struct_addr + OFFSET(files_struct_max_fds), + KVADDR, &max_fds, sizeof(int), "files_struct max_fds", + FAULT_ON_ERROR); + } + else if (VALID_MEMBER(files_struct_fdt)) { + readmem(files_struct_addr + OFFSET(files_struct_fdt), KVADDR, + &fdtable_addr, sizeof(void *), "fdtable buffer", + FAULT_ON_ERROR); + readmem(fdtable_addr + OFFSET(fdtable_max_fdset), + KVADDR, &max_fdset, sizeof(int), + "fdtable_struct max_fdset", FAULT_ON_ERROR); + readmem(fdtable_addr + OFFSET(fdtable_max_fds), + KVADDR, &max_fds, sizeof(int), "fdtable_struct max_fds", + FAULT_ON_ERROR); + } + } - if (!files_struct_addr || (max_fdset == 0) || (max_fds == 0)) { + if ((VALID_MEMBER(files_struct_fdt) && !fdtable_addr) || + !files_struct_addr || (max_fdset == 0) || (max_fds == 0)) { if (!NET_REFERENCE_CHECK(ref)) fprintf(fp, "No open sockets.\n"); return; } - readmem(files_struct_addr + OFFSET(files_struct_open_fds), KVADDR, - &open_fds_addr, sizeof(void *), "files_struct open_fds addr", - FAULT_ON_ERROR); + if (VALID_MEMBER(fdtable_open_fds)){ + readmem(fdtable_addr + OFFSET(fdtable_open_fds), KVADDR, + &open_fds_addr, sizeof(void *), "files_struct open_fds addr", + FAULT_ON_ERROR); + readmem(fdtable_addr + OFFSET(fdtable_fd), KVADDR, &fd, + sizeof(void *), "files_struct fd addr", FAULT_ON_ERROR); + } else { + readmem(files_struct_addr + OFFSET(files_struct_open_fds), KVADDR, + &open_fds_addr, sizeof(void *), "files_struct open_fds addr", + FAULT_ON_ERROR); + readmem(files_struct_addr + OFFSET(files_struct_fd), KVADDR, &fd, + sizeof(void *), "files_struct fd addr", FAULT_ON_ERROR); + } if (open_fds_addr) - readmem(open_fds_addr, KVADDR, &open_fds, sizeof(fd_set), - "files_struct open_fds", FAULT_ON_ERROR); - - readmem(files_struct_addr + OFFSET(files_struct_fd), KVADDR, &fd, - sizeof(void *), "files_struct fd addr", FAULT_ON_ERROR); - + readmem(open_fds_addr, KVADDR, &open_fds, sizeof(fd_set), + "files_struct open_fds", FAULT_ON_ERROR); if (!open_fds_addr || !fd) { if (!NET_REFERENCE_CHECK(ref)) fprintf(fp, "No open sockets.\n"); _

20 years

2
1
0 / 0

Need help to get crash to work on kdump geneated vmcore

by Sampathkumar, Kishore (STSD)

Hi, I have been working on back-porting kexec/kdump functionality from 2.6.13 vanilla kernel to RHEL4, Update 2 kernel (version 2.6.9-22). As part of that, when I try to invoke "crash" on a vmcore generated, I get the following error in "crash" tool: [root crash-4.0-2.23]# ./crash /boot/System.map-2.6.9-kdump-1 /usr/src/redhat/BUILD/kernel-2.6.9-kdump-1/linux- 2.6.9/vmlinux /root/vmcore crash 4.0-2.23 Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc. Copyright (C) 2004, 2005, 2006 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005 Fujitsu Limited Copyright (C) 2005 NEC Corporation Copyright (C) 1999, 2002 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb 6.1 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... crash: read error: kernel virtual address: ffffffff813a8200 type: "xtime" [root crash-4.0-2.23]# [root crash-4.0-2.23]# I tried with both crash-4.0-2.15 as well as crash-4.0-2.23. Same result. - Should I be re-building crash as part of the back-port? - Is crash capable of handing vmcore's generated by kdump? - Can someone suggest how I can make progress to get crash to work for this? Thanks, - Kishore

20 years

2
3
0 / 0

crash version 4.0-2.23 is available

by Dave Anderson

- Fix for "kmem -[sS]" command in 2.6.15 kernels which introduced per-NUMA node slab chains. Without this patch the command fails with a "kmem: invalid structure member offset: kmem_cache_s_lists" error message. (sharyath(a)in.ibm.com) - Fix for this initialization error on 2.6.16 kernels indicating: "crash: cannot determine idle task addresses from init_tasks[] or runqueues[]" followed by "crash: cannot resolve init_task_union" error messages. This was due to the introduction of a runqueue.cpu member that conflicted with an old cpu member in RHEL3-specific O(1) scheduler code. (anderson(a)redhat.com) - Fix for "kmem -i" in newer 2.6 kernels where the new ZONE_DMA32 bumps up the value of ZONE_HIGHMEM, causing a potential segmentation violation. (anderson(a)redhat.com) - Fix for "kmem -i", where the PG_slab bit determination has been fixed so that the correct number of slab pages is displayed. (anderson(a)redhat.com) - Fix for "swap" command and "kmem -i" option on 64-bit 2.6.15 kernels which could fail with a crash internal buffer dump followed by these messages: "swap: cannot allocate any more memory!" or "kmem: cannot allocate any more memory!". This was due to the swap_info_struct.max member being downsized from a long to an int. (anderson(a)redhat.com) - Continued xen support: this kernel runs on live x86 xen0 and xenU kernels, on xenU core dumps, and on xenU "xm save" files. This support is for x86 kernels with writable page tables only. Minimal support for running on live x86_64 xen0 kernels with writable page tables is also in place, but does not allow access to user virtual memory as of yet. (anderson(a)redhat.com) Download from: http://people.redhat.com/anderson

20 years, 1 month

1
0
0 / 0

Issues with kmem -i

by Sharyathi Nagesh

On executing kmem -i over dump we are seeing these results crash> kmem -i PAGES TOTAL PERCENTAGE TOTAL MEM 128778 503 MB ---- FREE 44244 172.8 MB 34% of TOTAL MEM USED 84534 330.2 MB 65% of TOTAL MEM SHARED 45690 178.5 MB 35% of TOTAL MEM BUFFERS 4371 17.1 MB 3% of TOTAL MEM CACHED 51358 200.6 MB 39% of TOTAL MEM SLAB 0 0 0% of TOTAL MEM TOTAL HIGH 0 0 0% of TOTAL MEM This is due to the fact that o mi->get_slabs field is turning out to be zero in dump_mem_map() function. This is happening as vt->PG_slab field is not getting updated with value(10 in this case) in vm_init(). This is happening as pte field is taken from page structure. o Probable solutions: Hard code value vt->PG_slab ?? o The output is breaking as totalhigh_pages is coming out zero, So part of code in function dump_kmeminfo() pct = freehighmem_pages ? (freehighmem_pages * 100)/totalhigh_pages : 0; Cause Arithmetic calculation error. Still looking into the problem. Regards Sharyathi Nagesh

20 years, 1 month

2
5
0 / 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Crash-utility April 2006