Re: [Crash-utility] [PATCH v3 2/2] arm/arm64: read elf notes for qemu generated cores

----- Original Message ----- > > > ----- Original Message ----- > > QEMU can generate both non-makedumpfile (just elf) and makedumpfile > > formatted kdumps. In neither case will crash_notes have prstatus, as > > crash_kexec doesn't run in the kernel, however the elf notes will > > contain the prstatus, and we can dig them out of there. > > I don't have a lot of ARM and ARM64 dumpfiles, but just doing a > quick sanity test of your patch, I came across this ARM dumpfile, > which I believe may be a QEMU-generated ELF vmcore. I'm not sure, > but it only has 1 NT_PRSTATUS note for the 1 online cpu (of 5 cpus).

> > But anyway, note that as expected, it cannot find the registers in the > kernel's uninitialized crash_notes -- here without your patch:

> > $ crash vmcore.pae vmlinux.pae.gz > > crash 7.1.4rc15 > Copyright (C) 2002-2014 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > Copyright (C) 1999-2006 Hewlett-Packard Co > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > Copyright (C) 2005, 2011 NEC Corporation > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public License, > and you are welcome to change it and/or distribute copies of it under > certain conditions. Enter "help copying" to see the conditions. > This program has absolutely no warranty. Enter "help warranty" for > details. > > GNU gdb (GDB) 7.6 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "--host=x86_64-unknown-linux-gnu > --target=arm-elf-linux"... > > WARNING: invalid note (n_type != NT_PRSTATUS) > WARNING: cannot retrieve registers for active tasks > > KERNEL: vmlinux.pae.gz > DUMPFILE: vmcore.pae > CPUS: 5 [OFFLINE: 4] > DATE: Sun Jun 8 18:27:39 2014 > UPTIME: 00:03:22 > LOAD AVERAGE: 0.16, 0.16, 0.07 > TASKS: 51 > NODENAME: buildroot > RELEASE: 3.13.5 > VERSION: #3 SMP Mon Jun 9 05:58:39 CST 2014 > MACHINE: armv7l (unknown Mhz) > MEMORY: 256 MB > PANIC: "SysRq : Trigger a crash" > PID: 732 > COMMAND: "sh" > TASK: 8bcead00 [THREAD_INFO: 8ad32000] > CPU: 0 > STATE: TASK_RUNNING (SYSRQ) > > crash> bt -a > PID: 732 TASK: 8bcead00 CPU: 0 COMMAND: "sh" > bt: WARNING: cannot determine starting stack frame for task 8bcead00 > > PID: 0 TASK: 8bc561c0 CPU: 1 COMMAND: "swapper/1" > bt: WARNING: cannot determine starting stack frame for task 8bc561c0 > > PID: 0 TASK: 8bc56580 CPU: 2 COMMAND: "swapper/2" > bt: WARNING: cannot determine starting stack frame for task 8bc56580 > > PID: 0 TASK: 8bc56940 CPU: 3 COMMAND: "swapper/3" > bt: WARNING: cannot determine starting stack frame for task 8bc56940 > > PID: 0 TASK: 8bc56d00 CPU: 4 COMMAND: "swapper/4" > bt: WARNING: cannot determine starting stack frame for task 8bc56d00 > crash> > > > With your patch applied, it generates a SIGSEGV in arm_get_crash_notes(): > > $ ./crash /usr/dumps/ARM/vmcore.pae /usr/dumps/ARM/vmlinux.pae.gz > > crash 7.1.4rc15 > Copyright (C) 2002-2014 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation > Copyright (C) 1999-2006 Hewlett-Packard Co > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > Copyright (C) 2005, 2011 NEC Corporation > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public License, > and you are welcome to change it and/or distribute copies of it under > certain conditions. Enter "help copying" to see the conditions. > This program has absolutely no warranty. Enter "help warranty" for > details. > > GNU gdb (GDB) 7.6 > Copyright (C) 2013 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show copying" > and "show warranty" for details. > This GDB was configured as "--host=x86_64-unknown-linux-gnu > --target=arm-elf-linux"... > > Segmentation fault (core dumped) > $ > > I haven't debugged it other than determining that the "note" it looks to > have found the single note OK, but then upon continuation the next time > through the loop, the "note" pointer is valid at line 597, but your > function sets it back to NULL, and therefore it craps out at line 622: > > 597 note = (Elf32_Nhdr *)buf; > 598 p = buf + sizeof(Elf32_Nhdr); > 599 > 600 /* > 601 * dumpfiles created with qemu won't have > crash_notes, but there will > 602 * be elf notes. > 603 */ > 604 if (note->n_namesz == 0 && (DISKDUMP_DUMPFILE() || > KDUMP_DUMPFILE())) { > 605 if (DISKDUMP_DUMPFILE()) > 606 note = > diskdump_get_prstatus_percpu(i); > 607 else if (KDUMP_DUMPFILE()) > 608 note = > netdump_get_prstatus_percpu(i); > 609 if (note) { > 610 /* > 611 * SIZE(note_buf) accounts for a > "final note", which is a > 612 * trailing empty elf note header. > 613 */ > 614 long notesz = SIZE(note_buf) - > sizeof(Elf32_Nhdr); > 615 > 616 if (sizeof(Elf32_Nhdr) + > roundup(note->n_namesz, 4) + > 617 note->n_descsz == notesz) > 618 BCOPY((char *)note, buf, > notesz); > 619 } > 620 } > 621 > 622 if (note->n_type != NT_PRSTATUS) { > 623 error(WARNING, "invalid note (n_type != > NT_PRSTATUS)\n"); > 624 goto fail; > 625 } > > Not sure how you want to handle that, probably just bail out the same way > if note becomes NULL? If I add this to arm_get_crash_notes(), just after your new function: if (!note) { error(WARNING, "cannot find NT_PRSTATUS note for cpu: %d\n", i); continue; } I get this: $ ./crash /usr/dumps/ARM/vmcore.pae* /usr/dumps/ARM/vmlinux.pae.gz crash 7.1.4rc15 Copyright (C) 2002-2014 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=arm-elf-linux"... WARNING: cannot find NT_PRSTATUS note for cpu: 1 WARNING: cannot find NT_PRSTATUS note for cpu: 2 WARNING: cannot find NT_PRSTATUS note for cpu: 3 WARNING: cannot find NT_PRSTATUS note for cpu: 4 KERNEL: /usr/dumps/ARM/vmlinux.pae.gz DUMPFILE: /usr/dumps/ARM/vmcore.pae CPUS: 5 [OFFLINE: 4] DATE: Sun Jun 8 18:27:39 2014 UPTIME: 00:03:22 LOAD AVERAGE: 0.16, 0.16, 0.07 TASKS: 51 NODENAME: buildroot RELEASE: 3.13.5 VERSION: #3 SMP Mon Jun 9 05:58:39 CST 2014 MACHINE: armv7l (unknown Mhz) MEMORY: 256 MB PANIC: "SysRq : Trigger a crash" PID: 732 COMMAND: "sh" TASK: 8bcead00 [THREAD_INFO: 8ad32000] CPU: 0 STATE: TASK_RUNNING (SYSRQ) crash> bt -a PID: 732 TASK: 8bcead00 CPU: 0 COMMAND: "sh" #0 [<80265064>] (sysrq_handle_crash) from [<80265810>] #1 [<80265810>] (__handle_sysrq) from [<80265928>] #2 [<80265928>] (write_sysrq_trigger) from [<80112120>] #3 [<80112120>] (proc_reg_write) from [<800c9840>] #4 [<800c9840>] (vfs_write) from [<800c9be4>] #5 [<800c9be4>] (sys_write) from [<8000e3e0>] pc : [<76e9cfdc>] lr : [<0000f998>] psr: 600d0010 sp : 7eab862c ip : 00000000 fp : 000a82a4 r10: 00000020 r9 : 000a8294 r8 : 00000001 r7 : 00000004 r6 : 000a9bf0 r5 : 00000001 r4 : 000a7d88 r3 : 00000000 r2 : 00000002 r1 : 000a9bf0 r0 : 00000001 Flags: nZCv IRQs on FIQs on Mode USER_32 ISA ARM PID: 0 TASK: 8bc561c0 CPU: 1 COMMAND: "swapper/1" bt: WARNING: cannot determine starting stack frame for task 8bc561c0 PID: 0 TASK: 8bc56580 CPU: 2 COMMAND: "swapper/2" bt: WARNING: cannot determine starting stack frame for task 8bc56580 PID: 0 TASK: 8bc56940 CPU: 3 COMMAND: "swapper/3" bt: WARNING: cannot determine starting stack frame for task 8bc56940 PID: 0 TASK: 8bc56d00 CPU: 4 COMMAND: "swapper/4" bt: WARNING: cannot determine starting stack frame for task 8bc56d00 crash> Note that if I did "goto fail" instead of "continue", I lose the good cpu 0 backtrace from the NT_PRSTATUS that your patch found, so doing it this way is the best of both worlds.