June 2012 - Crash-utility - Crash Utility List Archives

ppc: work out irq and fix pte translation display

by Toshikazu Nakayama

This patch set updates ppc "irq" and PTE translation display of book3e. - Support "irq" command for recent kernel The ppc's "irq" could not work well for recent kernel at all because ppc_init() always made SIZE(irq_desc_t) invalid by itself. And also, "irq" options were not supported entirely. Stash legacy ppc_dump_irq() behind generic_dump_irq() so that "irq" works on recent kernel and follow toward mainline updates lightly by using generic irq functions. - Fix PTE translation for book3e Although book3e uses multiple PTE bits for one attribute, ppc_translate_pte() can not handle such attributes well. And further privilege (kernel) RW protection bit assign of book3e are different from user's ones. Fix and add these translation features in ppc_translate_pte(). - results of update Before: irq can not work at all. crash> help -m | grep dump_irq dump_irq: ppc_dump_irq() crash> irq irq: cannot determine number of IRQs After: "irq" command work well crash> irq 36 IRQ: 36 STATUS: 24004 (IRQ_LEVEL) HANDLER: ebc08004 typename: c07c7044 " OpenPIC " startup: c00ca754 <default_startup> shutdown: c00ca6f0 <default_shutdown> enable: c00ca7b8 <default_enable> disable: c00ca5a8 <default_disable> : ACTION: eac56e80 handler: c043e398 <serial8250_interrupt> flags: 8080 name: c07c4534 "serial" dev_id: ebe3bec0 next: 0 DEPTH: 0 crash> irq -s CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 16: 0 0 0 0 0 0 0 0 OpenPIC bman-err,qman-err,error_rx,fman-err,fman-err,pme-err 17: 0 0 0 0 0 0 0 0 OpenPIC ds3232 : crash> irq -c 0 -s CPU0 16: 0 OpenPIC bman-err,qman-err,error_rx,fman-err,fman-err,pme-err 17: 0 OpenPIC ds3232 : crash> irq -a IRQ NAME AFFINITY 16 bman-err,qman-err,error_rx,fman-err,fman-err,pme-err 0-7 17 ds3232 0-7 : Before/After: PTE translation is updated like crash> vtop f987c01c [kernel vaddr] PTE PHYSICAL FLAGS ff974241255 ff974000 (PRESENT|USER|RW|COHERENT|DIRTY|ACCESSED) * multiple bits attribute USER or RW(non privilege) is not set in 255h => Fix as below PTE PHYSICAL FLAGS ff974241255 ff974000 (PRESENT|K-RW|COHERENT|DIRTY|ACCESSED) Thanks, Toshi Toshikazu Nakayama (6): ppc: update dump_irq() ppc: rework nr_irqs initialization ppc: add show_interrupts ppc: add get_irq_affinity ppc: fix the handling of PTE flags in translate_pte ppc: handle privilege level rw access from pte defs.h | 7 ++++- ppc.c | 65 ++++++++++++++++++++++++++++++++++++++++++++------------------- 2 files changed, 50 insertions(+), 22 deletions(-)

12 years, 5 months

2
1
0 / 0

Re: [Crash-utility] Crash-utility Digest, Vol 81, Issue 20

by Anthony Booker

Dave, Thanks for the reply, please excuse the rough formatting, I'll turn the digest off so future exchanges will be better attributed. On 06/28/2012 09:00 AM, crash-utility-request(a)redhat.com wrote: > What you see is what you get. > > It's a somewhat painful process, where each item in the gdb-<version>.patch > file contains something that needs to be addressed to maintain backwards > compatibility. Each time I do it, I just walk through each patch in that > file, and either it applies directly, or it involves some amount of pain, > given the complexity of gdb. I can appreciate that it's not an easy task, in my limited experience with the source for gdb it seems "venerable and eccentric" :) which must make rebasing a challenge. From my point of view that challenge is increased because I don't immediately have a grasp on the purpose of the patch and therefore how to react to failures to apply some changes. For instance the version I'm looking at would seem to be a snapshot of approximate 7.4.0 heritage, which has done away with some of the patch sites, like rltypedefs.h and cli-cmds.c. The former seems, to me, to be something I can ignore for now; readline makes things nicer to use but isn't vital. However cli-cmds seems more fundamental to basic functionality. All of which is just to say that experience has taught me that before I dive in and try to understand the mating of two unfamiliar components, it's worth asking if anyone has a guide book handy. > But the short answer is, why? Because the kernel I'm working with, while sharing a lot of the current x86_64 architecture, doesn't identify itself as such. I've been given access to gdb that's been hacked on to support userspace debug, but I want to play with the kernel. I can't immediately share the source I have, but since it's for internal development only I think my only problem is I get to update crash. So we can continue to use it until we can pass on what's useful to whoever's interested in it. I'm also working on the opposite end, to see if I can get information on the changes made to gdb and determine whether I can backport that into 7.3.1 which, if my understanding is correct, will also allow me to rig crash for my use until the official gdb support gets out. > Dave

12 years, 5 months

2
1
0 / 0

PPC64 compiler question

by Dave Anderson

While doing some testing I ran into a surprising crash-build bug on ppc64, where on a RHEL5 ppc64 host, a simple "make" command would build a 32-bit PPC crash utility. But on RHEL6, it built as 64-bit PPC64 crash utility as expected. (This was due to a crash-6.0.3 change to support "make target=PPC" on a ppc64 host). Anyway, the reason is that in the RHEL5 environment, if an application is compiled without specifying -m32 or -m64, it builds a 32-bit binary, whereas on RHEL6 it builds a 64-bit binary by default: RHEL5: # cat doit.c main() {} # gcc doit.c # file a.out a.out: ELF 32-bit MSB executable, PowerPC or cisco 4500, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), not stripped # gcc --version gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-52) Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # uname -r 2.6.18-308.el5 # RHEL6: # cat doit.c main() {} # gcc doit.c # file a.out a.out: ELF 64-bit MSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped # gcc --version gcc (GCC) 4.4.6 20110731 (Red Hat 4.4.6-3) Copyright (C) 2010 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # uname -r 2.6.32-220.el6.ppc64 # What controls that? Is it the compiler, some environment setting, or what? Dave

12 years, 5 months

3
4
0 / 0

Pointers on updating the gdb version crash uses.

by Anthony Booker

Hi, I have the chance to look at using crash in a new, to me, situation but there's a catch: I think I need to rebuild it based on a newer gdb version. I've started picking apart some of the gdb patch but I'm wondering if there's a repository maintained anywhere that will help me understand more of the purpose of the patchset, or maybe some guide that is a better approach to the problem. So my question is, anyone have the time and inclination to chip in on the smartest way to bring up crash on a newer gdb? Cheers, Anthony Booker.

12 years, 5 months

2
1
0 / 0

ppc: fix backtrace problem when NT_PRSTATUS was not saved

by Toshikazu Nakayama

Hi Dave, I've reworked patch about ppc "bt" problem discussed in https://www.redhat.com/archives/crash-utility/2012-June/msg00029.html A patch is now getting to work only for ppc targets, and notes are relocatecd in first ppc's "bt", not in initialization-time. And I've tested both "bt" diskdump by makedumpfile and netdump by /proc/vmcore with patched kernel like void crash_save_cpu(struct pt_regs *regs, int cpu) if ((cpu % 2)) return; I was worried about beeing able to catch all of non-contiguous NT_PRSTATUS notes, alternate with invalid odd notes in every process well, so I modified test case this time (that's no problem, everything OK). [After initialization: notes are not relocated yet] crash> help -D | grep notes num_prstatus_notes: 4 notes_buf: 107a3518 notes[0]: 107a3518 notes[1]: 107a3638 notes[2]: 107a3758 notes[3]: 107a3878 [Relocate in first bt(only once) and display correctly] crash> bt PID: 1000 TASK: eb0cf800 CPU: 4 COMMAND: "bash" bt: WARNING: cpu#1: not saved crash_notes bt: WARNING: cpu#3: not saved crash_notes bt: WARNING: cpu#5: not saved crash_notes bt: WARNING: cpu#7: not saved crash_notes R0: 00000001 R1: ea11be60 R2: eb0cf800 R3: 00000063 R4: 00000000 R5: ffffffff R6: c043ba2c R7: 00000000 R8: 00008000 R9: 00000000 R10: 00000000 R11: ea11be70 R12: 28242444 R13: 100b8448 R14: 100b07b8 R15: 100b0894 R16: 00000000 R17: 00000000 R18: 00000000 R19: 1006d270 R20: 00000000 R21: 100f02e0 R22: 00000000 R23: 00000001 R24: c08f1ac8 R25: 00029002 R26: c08f1bac R27: c08d0000 R28: 00000000 R29: c09ada48 R30: 00000063 R31: ea11be60 NIP: c0423378 MSR: 00021002 OR3: c09ada48 CTR: c0423344 LR: c0423d8c XER: 00000000 CCR: 28242444 MQ: 00008000 DAR: 00000000 DSISR: 00800000 Syscall Result: ea11be60 NIP [00000000c0423378] sysrq_handle_crash LR [00000000c0423d8c] __handle_sysrq => PC: c0423378 (sysrq_handle_crash+52) FP: ea11be60 #0 [ea11be60] sysrq_handle_crash at c0423378 #1 [ea11be70] __handle_sysrq at c0423d8c #2 [ea11bea0] write_sysrq_trigger at c0423ed0 #3 [ea11beb0] proc_reg_write at c01a79a4 #4 [ea11bee0] vfs_write at c014ba44 #5 [ea11bf00] sys_write at c014bcd8 #6 [ea11bf40] ret_from_syscall at c0013834 syscall [c00] exception frame: R0: 00000004 R1: bfe54e00 R2: 48027f80 R3: 00000001 R4: 4811d000 R5: 00000002 R6: 00000000 R7: 00000001 R8: 00000000 R9: 00000000 R10: 00000000 R11: 00000200 R12: 28242442 R13: 100b8448 R14: 100b07b8 R15: 100b0894 R16: 00000000 R17: 00000000 R18: 00000000 R19: 1006d270 R20: 00000000 R21: 100f02e0 R22: 1006b4d0 R23: 00000001 R24: 100b0000 R25: 1009b474 R26: bfe54ec8 R27: 00000002 R28: 0fefa400 R29: 4811d000 R30: 0fef9ff4 R31: 00000002 NIP: 0fe61494 MSR: 0002f902 OR3: 00000001 CTR: 0fe06fec LR: 0fe07050 XER: 00000000 CCR: 28242444 MQ: 00000000 DAR: 4811d000 DSISR: 00800000 Syscall Result: 00000000 [notes has been relocated: Counts num_prstatus_notes even if NULL] crash> help -D | grep notes num_prstatus_notes: 8 notes_buf: 107a3518 notes[0]: 107a3518 notes[1]: 0 notes[2]: 107a3638 notes[3]: 0 notes[4]: 107a3758 notes[5]: 0 notes[6]: 107a3878 notes[7]: 0 Thanks, Toshi

12 years, 5 months

2
3
0 / 0

Pages excluded in dump RHEL 6.2 - set to ffffffffffffffff

by Oberman, Laurence (GSE MSE)

I have a dump from RHEL 6.2 with all these masks set to all f's, so obviously crash excludes them. crash: page excluded: kernel virtual address: ffffffffffffffff type: "possible" WARNING: cannot read cpu_possible_map crash: page excluded: kernel virtual address: ffffffffffffffff type: "present" WARNING: cannot read cpu_present_map crash: page excluded: kernel virtual address: ffffffffffffffff type: "online" WARNING: cannot read cpu_online_map typedef struct cpumask { DECLARE_BITMAP(bits, NR_CPUS); } cpumask_t; #ifndef CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS /* These strip const, as traditionally they weren't const. */ #define cpu_possible_map (*(cpumask_t *)cpu_possible_mask) #define cpu_online_map (*(cpumask_t *)cpu_online_mask) #define cpu_present_map (*(cpumask_t *)cpu_present_mask) #define cpu_active_map (*(cpumask_t *)cpu_active_mask) Are these deprecated now and is that why they are fixed at all f's. [root@centos62 crash]# crash System.map vmlinux va102.2012-06-14.vmcore crash 6.0.7 Copyright (C) 2002-2012 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.3.1 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"... crash: page excluded: kernel virtual address: ffffffffffffffff type: "possible" WARNING: cannot read cpu_possible_map crash: page excluded: kernel virtual address: ffffffffffffffff type: "present" WARNING: cannot read cpu_present_map crash: page excluded: kernel virtual address: ffffffffffffffff type: "online" WARNING: cannot read cpu_online_map crash: page excluded: kernel virtual address: ffffffffffffffff type: "cpu_present_map" crash: page excluded: kernel virtual address: ffffffffffffffff type: "cpu_present_map" crash: page excluded: kernel virtual address: ffffffffffffffff type: "cpu_online_map" SYSTEM MAP: System.map DEBUG KERNEL: vmlinux (2.6.32-220.7.1.el6) DUMPFILE: va102.2012-06-14.vmcore [PARTIAL DUMP] CPUS: 32 DATE: Thu Jun 14 05:24:17 2012 UPTIME: 11:13:23 LOAD AVERAGE: 215.00, 196.78, 135.81 TASKS: 1778 NODENAME: va1splunkindex02.va1.swtor.int RELEASE: 2.6.32-220.7.1.el6.x86_64 VERSION: #1 SMP Tue Mar 6 13:54:51 EST 2012 MACHINE: x86_64 (2799 Mhz) MEMORY: 48 GB PANIC: "Oops: 0002 [#1] SMP " (check log for details) PID: 2428 COMMAND: "bash" TASK: ffff88063874ab40 [THREAD_INFO: ffff8806dcd1a000] CPU: 0 STATE: TASK_RUNNING (PANIC) Thanks Laurence

12 years, 5 months

2
4
0 / 0

Handle the NT_PRSTATUS lost for the "bt" command

by Toshikazu Nakayama

The purpose of this patch is to work out "bt" command for the diskdump which NT_PRSTATUS note could not be saved by IPI lost. I think IPI is possibly lost by panic under the serious crashed condition. I noticed that "bt" failed in my ppc environment when the NT_PRSTATUS notes are lost on some CPUs while IPI delivery. Then, I made CPU map for prstatus in diskdump more correctable by checking a validation of crash_notes field. I've tested this problem by patching kernel like, - kernel/kexec.c void crash_save_cpu(struct pt_regs *regs, int cpu) { + if (current->pid == 0) + /* this cpu was idle; nothing to capture */ + return; It looks terrible and impractical test case but actually I met this code in my using distro's kernel. I couldn't reproduce actual IPI lost case, then fortunately, use this as a example of the causes if IPI could not be delivered to other CPUs. => Taking diskdump by sysrq+c and makedumpfile. crash> help -D | grep notes num_prstatus_notes: 1 notes_buf: 10ba91a8 notes[0]: 10ba91a8 crash> help -k | grep cpus cpus: 8 cpus_override: (null) crash> bt PID: 1001 TASK: ea62b000 CPU: 2 COMMAND: "bash" Segmentation fault Since seven idle cpus did not save NT_PRSTATUS note, crash could not handle CPU#2's note where is located as CPU#0's. With this patch, crash get to work out with correct CPU map to prstatus. WARNING: catch lost crash_notes at cpu#0 WARNING: catch lost crash_notes at cpu#1 WARNING: catch lost crash_notes at cpu#3 WARNING: catch lost crash_notes at cpu#4 WARNING: catch lost crash_notes at cpu#5 WARNING: catch lost crash_notes at cpu#6 WARNING: catch lost crash_notes at cpu#7 crash.fix> help -D | grep notes num_prstatus_notes: 1 notes_buf: 107a3378 notes[2]: 107a3378 crash.fix> help -k | grep cpus cpus: 8 cpus_override: (null) crash.fix> bt PID: 1001 TASK: ea62b000 CPU: 2 COMMAND: "bash" R0: 00000001 R1: eb793e60 R2: ea62b000 R3: 00000063 R4: 00000000 R5: ffffffff R6: c043ba2c R7: 00000000 R8: 00008000 R9: 00000000 R10: 00000000 R11: eb793e70 R12: 28242444 R13: 100b8448 R14: 100b07b8 R15: 100b0894 R16: 00000000 R17: 00000000 R18: 00000000 R19: 1006d270 R20: 00000000 R21: 100f0430 R22: 00000000 R23: 00000001 R24: c08f1ac8 R25: 00029002 R26: c08f1bac R27: c08d0000 R28: 00000000 R29: c09ada48 R30: 00000063 R31: eb793e60 NIP: c0423378 MSR: 00021002 OR3: c09ada48 CTR: c0423344 LR: c0423d8c XER: 00000000 CCR: 28242444 MQ: 00008000 DAR: 00000000 DSISR: 00800000 Syscall Result: eb793e60 NIP [00000000c0423378] sysrq_handle_crash LR [00000000c0423d8c] __handle_sysrq #0 [eb793e60] sysrq_handle_crash at c0423378 : snip Thanks, Toshi

12 years, 5 months

2
7
0 / 0

[PATCH] Update trace extension for per-cpu buffer size

by Rabin Vincent

Since Linux v3.4 (specifically, commit 438ced1720b584000 "ring-buffer: Add per_cpu ring buffer control files"), the trace buffer size is now per-cpu. The patch below updates the trace extension to handle this. Rabin diff --git a/extensions/trace.c b/extensions/trace.c index 831cc77..224c662 100644 --- a/extensions/trace.c +++ b/extensions/trace.c @@ -24,6 +24,7 @@ static int nr_cpu_ids; * lockless ring_buffer and old non-lockless ring_buffer are both supported. */ static int lockless_ring_buffer; +static int per_cpu_buffer_sizes; #define koffset(struct, member) struct##_##member##_offset @@ -37,6 +38,7 @@ static int koffset(ring_buffer, buffers); static int koffset(ring_buffer_per_cpu, cpu); static int koffset(ring_buffer_per_cpu, pages); +static int koffset(ring_buffer_per_cpu, nr_pages); static int koffset(ring_buffer_per_cpu, head_page); static int koffset(ring_buffer_per_cpu, tail_page); static int koffset(ring_buffer_per_cpu, commit_page); @@ -71,6 +73,7 @@ struct ring_buffer_per_cpu { ulong real_head_page; int head_page_index; + unsigned int nr_pages; ulong *pages; ulong *linear_pages; @@ -144,7 +147,14 @@ static int init_offsets(void) init_offset(trace_array, buffer); init_offset(tracer, name); - init_offset(ring_buffer, pages); + if (MEMBER_EXISTS("ring_buffer_per_cpu", "nr_pages")) { + per_cpu_buffer_sizes = 1; + if (verbose) + fprintf(fp, "per cpu buffer sizes\n"); + } + + if (!per_cpu_buffer_sizes) + init_offset(ring_buffer, pages); init_offset(ring_buffer, flags); init_offset(ring_buffer, cpus); init_offset(ring_buffer, buffers); @@ -155,6 +165,8 @@ static int init_offsets(void) fprintf(fp, "lockless\n"); } + if (per_cpu_buffer_sizes) + init_offset(ring_buffer_per_cpu, nr_pages); init_offset(ring_buffer_per_cpu, cpu); init_offset(ring_buffer_per_cpu, pages); init_offset(ring_buffer_per_cpu, head_page); @@ -362,6 +374,10 @@ static int ftrace_init_buffers(struct ring_buffer_per_cpu *buffers, buffer_read_value(reader_page); buffer_read_value(overrun); buffer_read_value(entries); + if (per_cpu_buffer_sizes) { + buffer_read_value(nr_pages); + pages = buffers[i].nr_pages; + } #undef buffer_read_value if (ftrace_init_pages(buffers + i, pages) < 0) -- 1.7.9.5

12 years, 5 months

3
3
0 / 0

multiple structs with the same name

by Ben Myers

Hi Folks, I have a second question related to crash. If there are multiple instances of a structure in the kernel which have the same name, how do I print them out? Mark discovered that as of 7ff9554b there is also a new 'struct log' in printk.c. crash> struct log struct log { u64 ts_nsec; u16 len; u16 text_len; u16 dict_len; u16 level; } SIZE: 16 XFS also has a 'struct log' in fs/xfs/xfs_log_priv.h. How do we choose which will print? Thanks, Ben

12 years, 5 months

4
7
0 / 0

Fix bug of list -h

by qiaonuohan

Hello Dave, When investigating list command, I found the "-h" is involved but not discussed in help page. And then I tried to use, but I find some problems with it. The first, I made patch to show it. The "ld->start" should be the pointer to the structure list_head. And the second one, I am not sure about the reason why you don't display the node related to the address user input. I will take the tasks of task_struct as an example. crash> task_struct.tasks ffff8800371a0ac0 tasks = { next = 0xffffffff81a8d468, prev = 0xffff88004a9e0f88 } crash> list task_struct.tasks -s task_struct.tasks -h ffff8800371a0ac0 ffffffff81a8d020 tasks = { next = 0xffff88004eaf1908, prev = 0xffff8800371a0f08 } ffff88004eaf14c0 tasks = { next = 0xffff88004eaf0ec8, prev = 0xffffffff81a8d468 } ... As the command shows, the command neglect the task_struct at ffff8800371a0ac0. I don't know why it is omitted. And if only one node is in the list, why prints "(empty)"? readmem(ld->start, KVADDR, &ld->start, sizeof(void *), "LIST_HEAD contents", FAULT_ON_ERROR); if (ld->start == ld->end) { fprintf(fp, "(empty)\n"); return; } -- -- Regards Qiao Nuohan

12 years, 5 months

3
10
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Crash-utility June 2012