Download from: http://people.redhat.com/anderson or https://github.com/crash-utility/crash/releases The master branch serves as a development branch that will contain all patches that are queued for the next release: $ git clone git://github.com/crash-utility/crash.git Changelog: - Support for "irq" and "irq -u" on the S390 and S390X architectures if they are running Linux 3.12 and later kernels. Older kernels without GENERIC_HARDIRQ support will fail with the error message "irq: cannot determine number of IRQs". (sebott(a)linux.vnet.ibm.com) - Fix for the handling of multiple ramdump images. Without the patch, entering more than one ramdump image on the command line may result in a segmentation violation. (oza(a)broadcom.com) - Implemented the capability of building crash as an x86_64 binary for analyzing little-endian PPC64 dumpfiles on an x86_64 host, which can be done by entering "make target=PPC64". After the initial build is complete, subsequent builds can be done by entering "make" alone. (anderson(a)redhat.com) - Fix for the "crash --log <dumpfile>" option on both of the PPC64 architectures. Without the patch, the command fails with the message "crash: seek error: physical address: <address> type: log_buf pointer", followed by "crash: cannot read log_buf value". This bug was introduced in crash-7.0.0 by a patch that added support for the PPC64 BOOK3E processor family. (anderson(a)redhat.com) - Fix for a misleading fatal error message if a 32-bit crash binary built on an X86_64 host with "make target=X86" or "make target=ARM" is used on a live X86_64 system without specifying a vmlinux namelist. Without the patch, the session fails with the message "crash: cannot find booted kernel -- please enter namelist argument". The error message will be "crash: compiled for the X86 architecture" or "crash: compiled for the ARM architecture". (anderson(a)redhat.com) - Fix for finding the starting stack and instruction pointer hooks for the active tasks in x86_64 ELF or compressed dumpfiles created by the KVM "virsh dump --memory-only" facility. Without the patch, the backtraces of active tasks may show an invalid starting frame that indicates "__schedule". The fix displays the exception RIP and dumps the register contents that are stored in the dumpfile header. If the active task was operating in the kernel, the backtrace continues from there; if the task was operating in user-space, the backtrace is complete at that point. (anderson(a)redhat.com) - Fix for the "waitq" command when it is passed the address of a wait_queue_head_t structure. Without the patch, if the entries on the list are dynamically-created __wait_queue structures on kernel stacks, the tasks owning the kernel stack are not displayed. (anderson(a)redhat.com) - Implemented a new "net -n [pid|task]" option that displays the list of network devices with respect the network namespace of the current context, or that of a task specified by the optional "pid" or "task" argument. The former "net -n <address>" option that translates an IPv4 address expressed as a decimal or hexadecimal value into a standard numbers-and-dots notation has been changed to "net -N". (vvs(a)parallels.com) - Fix for the kernel virtual address to symbol name translation for special text region delimiter symbols declared in vmlinux.lds.S with VMLINUX_SYMBOL(), such as __sched_text_start, __lock_text_start, __kprobes_text_start, __entry_text_start and __irqentry_text_start. Without the patch, if the addresses of those symbols are the same value as the first "real" symbol in those text regions, commands such as "dis" and "sym" may show the "_text_start" symbol name instead of the desired text symbol name. (qiaonuohan(a)cn.fujitsu.com, anderson(a)redhat.com) - Enhancement of the "kmem -i" option to display memory overcommit information, which will be appended to the traditional output of the command. For example: crash> kmem -i PAGES TOTAL PERCENTAGE TOTAL MEM 1965332 7.5 GB ---- FREE 78080 305 MB 3% of TOTAL MEM USED 1887252 7.2 GB 96% of TOTAL MEM SHARED 789954 3 GB 40% of TOTAL MEM BUFFERS 110606 432.1 MB 5% of TOTAL MEM CACHED 1212645 4.6 GB 61% of TOTAL MEM SLAB 146563 572.5 MB 7% of TOTAL MEM TOTAL SWAP 1970175 7.5 GB ---- SWAP USED 5 20 KB 0% of TOTAL SWAP SWAP FREE 1970170 7.5 GB 99% of TOTAL SWAP COMMIT LIMIT 2952841 11.3 GB ---- COMMITTED 1150595 4.4 GB 38% of TOTAL LIMIT The COMMIT LIMIT and COMMITTED information is similar to that displayed by the CommitLimit and Committed_AS lines in /proc/meminfo. (atomlin(a)redhat.com) - Fix for the "kmem [-s|-S] <address>" command, and the "rd -S[S]" and "bt -F[F]" options. Without the patch, if the page structure associated with a memory address still contains a (stale) pointer to the address of a kmem_cache structure, but whose page.flags does not have the PG_slab bit set, the address is incorrectly presumed to be contained within that slab cache. As as result, the "kmem" command may display one or more messages indicating a "bad inuse counter", a "bad next pointer" or a "bad s_mem pointer", followed by an "address not found in cache" error message. The "rd -S[S]" and "bt -F[F]" commands may mislabel memory locations as belonging to slab caches. (anderson(a)redhat.com) - Added a new "vm -M <mm_struct>" option. When a task is exiting, the mm_struct address pointer in its task_struct is NULL'd out, and as a result, the "vm" command looks like this: crash> vm PID: 4563 TASK: ffff88049863f500 CPU: 8 COMMAND: "postgres" MM PGD RSS TOTAL_VM 0 0 0k 0k However, the mm_struct address can be retrieved from the task's kernel stack and entered manually with this option, which allows the "vm" command to attempt to dump the virtual memory data of the task. It may, or may not, work, depending upon how far the virtual memory deconstruction has proceeded. This option only verifies that the address entered is from the "mm_struct" slab cache, and that its mm_struct.mm_count is non-zero. (qiaonuohan(a)cn.fujitsu.com, anderson(a)redhat.com) - Fix for the X86_64 "bt" and "mach" commands when running against kernels that have the following Linux 3.18 commit, which addresses CVE-2014-9322. The kernel patch removes the per-cpu exception stack used for handling stack segment faults: commit 6f442be2fb22be02cafa606f1769fa1e6f894441 x86_64, traps: Stop using IST for #SS Without this patch, backtraces that originate on any of the other 4 per-cpu exception stacks will be mis-labeled at the transition point back to the previous stack. For example, backtraces that that originate on the NMI stack will indicate that they are coming from the "DOUBLEFAULT" stack. The patch examines all idt_table entries during initialization, looking for gate descriptors that have non-zero index values, and when found, pulls out out the handler function address; from that information, the exception stack name string array is properly initialized rather than being hard-coded. This fix also properly labels the exception stack names on x86_64 CONFIG_PREEMPT_RT realtime kernels, which only utilize 3 exception stacks instead of the traditional 5 (now 4 with this kernel commit), instead of just showing "RT". Also, without the patch, the "mach" command will mis-label the stack names when it displays the base addresses of each per-cpu exception stack. (anderson(a)redhat.com) - Additional output for the "help [-D|-n]" options on X86 and X86_64 architectures. For compressed kdumps, the elf_prstatus structure in each per-cpu NT_PRSTATUS note will be translated. For ELF kdumps, the elf_prstatus structure in each per-cpu NT_PRSTATUS note, and the QEMUCPUState structure in each per-cpu QEMU note, will be translated. (zhouwj-fnst(a)cn.fujitsu.com, anderson(a)redhat.com) - Implemented a new "bt -A" option for the S390X architecture, which adds support for displaying the new s390x vector registers. For ELF dumps, the registers are taken from the VX ELF notes; for s390 dumps. the registers are taken from memory. The option produces the same output as the -a option, but also displays the vector registers for all active tasks. (holzheu(a)linux.vnet.ibm.com) - Fix for the 32-bit ARM virtual-to-physical address translation of unity-mapped kernel virtual addresses in kernels configured with CONFIG_ARM_LPAE if the system's phys_base exceeds 4GB. (sdu.liu(a)huawei.com) - Fix for the "help [-D|-n]" option on 32-bit X86 kernels that use the 64-bit ELF vmcore format generated by "virsh dump --memory-only". Without the patch, the QEMUCPUState structures in QEMU notes are not translated. (qiaonuohan(a)cn.fujitsu.com) - Additional output for the "help [-D|-n]" options on X86 and X86_64 architectures. For compressed kdumps generated by "virsh dump --memory-only", the QEMUCPUState structure in each per-cpu QEMU note will be translated, and the dumpfile offset address of each QEMU note will be displayed. (qiaonuohan(a)cn.fujitsu.com, anderson(a)redhat.com) - Introduction of support for the 32-bit MIPS architecture. This initial support is restricted to 32-bit MIPS kernels that are configured as little-endian. With respect to dumpfile types, only ELF vmcores are recognized. In addition to building crash as a 32-bit MIPS binary, it is also possible to build crash as an x86 binary on an x86 or x86_64 host so that crash analysis of MIPS dumpfiles can be performed on an x86 or x86_64 host. The x86 binary can be built by entering "make target=MIPS" for the initial build; subsequent builds with MIPS support can be accomplished by entering "make" alone. (rabin(a)rab.in) - Added support for big-endian 32-bit MIPS kernels. Only native MIPS crash binaries may be built with big-endian support; running the "make target=MIPS" build option on an x86 or x86_64 host creates x86 binaries with little-endian support only. (rabin(a)rab.in) - Update the "ps" help page to reflect that the "ps -l" option may be based upon the task_struct's sched_entity.last_arrival. Without the patch, it indicates that either the task_struct's last_run or timestamp value are used. (anderson(a)redhat.com) - Fix for the "kmem -z" option output to change the zone structure's pages_scanned field from a signed to an unsigned long integer. (Alexandr_Terekhov(a)epam.com) - Fix for "kmem -z" option on Linux 2.6.30 and later kernels. Without the patch, the zone structure's all_unreclaimable and pages_scanned fields are not dumped. (anderson(a)redhat.com) - Fix for the PPC64 "bt" command on both big-endian and little-endian architectures. Without the patch, backtraces of the active tasks may be "empty" on little-endian machines, or show a one-liner of the form: "#0 [c0000005f4db7a60] (null) at 501 (unreliable)" on big-endian machines. (anderson(a)redhat.com) - Additional output for the "help [-D|-n]" options for the PPC64 architecture. For compressed kdump and ELF kdump dumpfiles, the elf_prstatus structure in each per-cpu NT_PRSTATUS note will be translated. (anderson(a)redhat.com) - The "help -r" option has been extended to dump the PPC64 registers stored in each per-cpu NT_PRSTATUS note in compressed kdump and ELF kdump dumpfiles. (anderson(a)redhat.com) - Prevent "help -r" and "help -[D|n]" from generating a segmentation violation when attempting to access non-existent NT_PRSTATUS notes for offline cpus in ELF or compressed kdumps. (anderson(a)redhat.com) - Fix for the "kmem -V" option output to change the display of the vm_event_states fields from signed to unsigned long integers. (adobriyan(a)gmail.com) - Fix to allow the "ps -G" qualifier to be used in conjunction with the "ps -p" option. Without the patch, "ps -G -p" fails with the error message "ps: do_list: hash queue is in use?" (anderson(a)redhat.com) - Fix for the "runq" command on kernels that are configured with CONFIG_RT_GROUP_SCHED=n. Without the patch, real-time tasks queued on a per-cpu rt_rq.rt_prio_array will not be displayed under the "RT PRIO_ARRAY" header. (mty.shibata(a)gmail.com) - Fix for a regression introduced in crash-7.0.9 when running on a live 32-bit ARM machine. Without the patch, a segmentation violation is generated during session initialization. (anderson(a)redhat.com) - Enhancement of the "PANIC:" message displayed by the initial system banner and by the "sys" command. Without the patch, many panic types are categorized under the same generic message of the form: PANIC: "Oops: 0000 [#1] SMP " (check log for details) or in other types of crashes, no message is displayed at all. With this patch, a more comprehensive search is made of the kernel log for a more informative panic message. (drc(a)yahoo-inc.com, anderson(a)redhat.com) - Add appropriate checks for the MIPS architecture to allow extension modules to be loaded with the "extend" command. (rabin(a)rab.in) - Update the extensions/trace.c extension module to account for the movement of the ftrace_event_call.name member into an anonymous union in Linux 3.15, commit de7b2973903c6cc50b31ee5682a69b2219b9919d. (rabin(a)rab.in) - Added support for VMware .vmss suspended state files as dumpfiles. Similar to all other supported dumpfile types, it is invoked as: $ crash vmlinux <vmname>.vmss A "<vmname>.vmss" file created by the VMware vSphere ESX hypervisor contains a header and the full memory image. A "<vmname>.vmss" file created by the VMware Workstation facility only contains the header, and must be accompanied by a companion "<vmname>.vmem" memory image that is located in the same directory as the "<vmname>.vmss" file. (hfu(a)vmware.com)